2024-04-12T20:49:49.657 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-12T20:49:49.657 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653602 --name vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} --owner scheduled_vshankar@teuthology -- /home/teuthworker/archive/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653602/orig.config.yaml 2024-04-12T20:49:49.713 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-12T20:49:49.809 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653602 branch: wip-vshankar-testing-20240411.061452 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} email: vshankar@redhat.com first_in_suite: false job_id: '7653602' kernel: client: branch: testing flavor: default kdb: true sha1: 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240411.061452 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 324baf8ddb91dea46cac50cb535c306115791355 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.dir_split install: ceph: flavor: default sha1: 324baf8ddb91dea46cac50cb535c306115791355 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240411.061452 sha1: 324baf8ddb91dea46cac50cb535c306115791355 owner: scheduled_vshankar@teuthology priority: 120 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7105 sha1: 324baf8ddb91dea46cac50cb535c306115791355 sleep_before_teardown: 0 subset: 463/1024 suite: fs suite_branch: wip-vshankar-testing-20240411.061452 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_324baf8ddb91dea46cac50cb535c306115791355/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 324baf8ddb91dea46cac50cb535c306115791355 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGgfawhttM8bakmawDbfA9v4zT/9TeqL0Lt52aAgle1oTNGgFfwkL7RmsOF7pZe2yHP6o7fBjtE4tM2zyOycpL8= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBNBlcblySRBQY2/aZag/T4UJXMCQi37rXNF4jcTOJDpTdYl1FjC5jEx3Ydvnts6ayviYpIMVDwaaVtov+joco= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1WzkEwGmFJJny5HFZIHuvHkIkqY/QRzGdln/z9GLFLEQF3E2OzLi9TEhRIVaz6gMXVONK4gdUh/VMBOf1iio8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-12_03:11:38 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-12T20:49:49.809 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_324baf8ddb91dea46cac50cb535c306115791355/qa; will attempt to use it 2024-04-12T20:49:49.810 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_324baf8ddb91dea46cac50cb535c306115791355/qa/tasks 2024-04-12T20:49:49.810 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-12T20:49:49.812 INFO:teuthology.task.internal:Checking packages... 2024-04-12T20:49:49.836 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '324baf8ddb91dea46cac50cb535c306115791355' 2024-04-12T20:49:49.836 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-12T20:49:49.836 INFO:teuthology.packaging:ref: None 2024-04-12T20:49:49.837 INFO:teuthology.packaging:tag: None 2024-04-12T20:49:49.837 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240411.061452 2024-04-12T20:49:49.837 INFO:teuthology.packaging:sha1: 324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T20:49:49.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240411.061452 2024-04-12T20:49:50.068 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2879.g324baf8d 2024-04-12T20:49:50.071 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-12T20:49:50.078 INFO:teuthology.task.internal:no buildpackages task found 2024-04-12T20:49:50.078 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-12T20:49:50.093 INFO:teuthology.task.internal:Saving configuration 2024-04-12T20:49:50.112 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-12T20:49:50.118 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-12T20:49:50.143 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653602', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 20:44:44.682044', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGgfawhttM8bakmawDbfA9v4zT/9TeqL0Lt52aAgle1oTNGgFfwkL7RmsOF7pZe2yHP6o7fBjtE4tM2zyOycpL8='} 2024-04-12T20:49:50.166 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653602', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 20:44:44.680467', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHBNBlcblySRBQY2/aZag/T4UJXMCQi37rXNF4jcTOJDpTdYl1FjC5jEx3Ydvnts6ayviYpIMVDwaaVtov+joco='} 2024-04-12T20:49:50.187 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-04-12_03:11:38-fs-wip-vshankar-testing-20240411.061452-testing-default-smithi/7653602', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-12 20:44:44.683327', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI1WzkEwGmFJJny5HFZIHuvHkIkqY/QRzGdln/z9GLFLEQF3E2OzLi9TEhRIVaz6gMXVONK4gdUh/VMBOf1iio8='} 2024-04-12T20:49:50.188 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-12T20:49:50.194 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-12T20:49:50.194 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-12T20:49:50.194 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-12T20:49:50.194 INFO:teuthology.run_tasks:Running task console_log... 2024-04-12T20:49:50.297 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6e36d81160>, signals=[15]) 2024-04-12T20:49:50.298 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-12T20:49:50.304 INFO:teuthology.task.internal:Opening connections... 2024-04-12T20:49:50.304 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-04-12T20:49:50.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T20:49:50.398 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-04-12T20:49:50.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T20:49:50.489 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-04-12T20:49:50.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T20:49:50.577 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-12T20:49:50.583 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-04-12T20:49:50.601 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-04-12T20:49:50.601 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-04-12T20:49:50.661 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-04-12T20:49:50.662 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T20:49:50.663 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-12T20:49:50.663 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T20:49:50.663 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-04-12T20:49:50.684 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-04-12T20:49:50.702 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-04-12T20:49:50.702 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-04-12T20:49:50.758 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-04-12T20:49:50.758 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-04-12T20:49:50.758 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-12T20:49:50.759 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T20:49:50.760 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-04-12T20:49:50.784 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-04-12T20:49:50.801 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-04-12T20:49:50.801 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-04-12T20:49:50.859 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-04-12T20:49:50.859 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-04-12T20:49:50.859 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-04-12T20:49:50.859 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-04-12T20:49:50.859 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-04-12T20:49:50.859 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-04-12T20:49:50.859 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-12T20:49:50.859 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-04-12T20:49:50.860 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-04-12T20:49:50.860 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-12T20:49:50.860 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-04-12T20:49:50.860 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-12T20:49:50.860 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-12T20:49:50.860 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-12T20:49:50.860 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-04-12T20:49:50.884 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-12T20:49:50.894 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-12T20:49:50.899 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-12T20:49:50.900 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-04-12T20:49:50.902 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-04-12T20:49:50.904 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-04-12T20:49:50.920 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-12T20:49:50.927 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-12T20:49:50.927 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-04-12T20:49:50.962 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-04-12T20:49:50.964 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-04-12T20:49:51.043 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-12T20:49:51.122 INFO:teuthology.run_tasks:Running task kernel... 2024-04-12T20:49:51.136 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8'} 2024-04-12T20:49:51.137 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-12T20:49:51.137 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-12T20:49:51.137 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T20:49:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T20:49:51.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-12T20:49:51.405 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-12T20:49:51.405 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-12T20:49:51.405 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-04-12T20:49:51.422 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-361.el9.x86_64 2024-04-12T20:49:51.423 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-12T20:49:51.423 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-12T20:49:51.423 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum upgrade -y linux-firmware 2024-04-12T20:49:53.668 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 14 MB/s | 8.1 MB 00:00 2024-04-12T20:49:55.377 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.5 MB 00:00 2024-04-12T20:49:57.143 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 6.1 MB/s | 8.0 MB 00:01 2024-04-12T20:50:18.858 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 939 kB/s | 19 MB 00:20 2024-04-12T20:50:23.000 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-12T20:50:24.426 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 16 kB 00:00 2024-04-12T20:50:34.113 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 2.2 MB/s | 21 MB 00:09 2024-04-12T20:50:39.252 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr:Modular dependency problems: 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 1: conflicting requests 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 2: conflicting requests 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 3: conflicting requests 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 4: conflicting requests 2024-04-12T20:50:41.487 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 5: conflicting requests 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 6: conflicting requests 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 7: conflicting requests 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 8: conflicting requests 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 9: conflicting requests 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 10: conflicting requests 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 11: conflicting requests 2024-04-12T20:50:41.488 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 12: conflicting requests 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 13: conflicting requests 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 14: conflicting requests 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 15: conflicting requests 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 16: conflicting requests 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 17: conflicting requests 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 18: conflicting requests 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 19: conflicting requests 2024-04-12T20:50:41.489 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 20: conflicting requests 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 21: conflicting requests 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 22: conflicting requests 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 23: conflicting requests 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 24: conflicting requests 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 25: conflicting requests 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 26: conflicting requests 2024-04-12T20:50:41.490 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 27: conflicting requests 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 28: conflicting requests 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 29: conflicting requests 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 30: conflicting requests 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 31: conflicting requests 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 32: conflicting requests 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 33: conflicting requests 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.491 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 34: conflicting requests 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 35: conflicting requests 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 36: conflicting requests 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 37: conflicting requests 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 38: cannot install the best candidate for the job 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.492 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 39: conflicting requests 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 40: cannot install the best candidate for the job 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 41: cannot install the best candidate for the job 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.493 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 42: cannot install the best candidate for the job 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 43: cannot install the best candidate for the job 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 44: cannot install the best candidate for the job 2024-04-12T20:50:41.494 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.495 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.495 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.495 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.495 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 45: cannot install the best candidate for the job 2024-04-12T20:50:41.495 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.495 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.495 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-12T20:50:41.495 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-12T20:50:42.057 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-04-12T20:50:42.065 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-12T20:50:42.065 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-04-12T20:50:42.065 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-12T20:50:42.065 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-04-12T20:50:42.066 INFO:teuthology.orchestra.run.smithi081.stdout: linux-firmware noarch 20240219-143.el9 baseos 376 M 2024-04-12T20:50:42.066 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T20:50:42.066 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-04-12T20:50:42.066 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-04-12T20:50:42.066 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 1 Package 2024-04-12T20:50:42.066 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T20:50:42.066 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 376 M 2024-04-12T20:50:42.066 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-04-12T20:50:55.944 INFO:teuthology.orchestra.run.smithi081.stdout:linux-firmware-20240219-143.el9.noarch.rpm 27 MB/s | 376 MB 00:13 2024-04-12T20:50:55.945 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-04-12T20:50:55.945 INFO:teuthology.orchestra.run.smithi081.stdout:Total 27 MB/s | 376 MB 00:13 2024-04-12T20:50:58.970 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-04-12T20:50:59.093 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-04-12T20:50:59.093 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-04-12T20:51:02.298 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-04-12T20:51:02.298 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-04-12T20:51:09.814 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-04-12T20:51:10.212 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-12T20:51:10.297 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-12T20:51:11.271 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-04-12T20:51:11.271 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-04-12T20:51:11.714 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-04-12T20:51:11.714 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T20:51:11.714 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-04-12T20:51:11.714 INFO:teuthology.orchestra.run.smithi081.stdout: linux-firmware-20240219-143.el9.noarch 2024-04-12T20:51:11.714 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T20:51:11.714 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-04-12T20:51:12.014 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-12T20:51:12.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-12T20:51:12.163 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ 2024-04-12T20:51:12.163 DEBUG:teuthology.orchestra.run.smithi081:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-12T20:51:12.166 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi081.front.sepia.ceph.com to complete... 2024-04-12T20:51:46.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12 20:51:46 URL:https://3.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g3f60ed14f7c1-1.x86_64.rpm [357344123/357344123] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-12T20:51:46.038 INFO:teuthology.orchestra.run.smithi081.stderr:FINISHED --2024-04-12 20:51:46-- 2024-04-12T20:51:46.039 INFO:teuthology.orchestra.run.smithi081.stderr:Total wall clock time: 34s 2024-04-12T20:51:46.039 INFO:teuthology.orchestra.run.smithi081.stderr:Downloaded: 1 files, 341M in 33s (10.2 MB/s) 2024-04-12T20:51:46.041 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-12T20:51:46.041 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-12T20:51:48.086 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying... ######################################## 2024-04-12T20:51:48.486 INFO:teuthology.orchestra.run.smithi081.stdout:Preparing... ######################################## 2024-04-12T20:51:48.567 INFO:teuthology.orchestra.run.smithi081.stdout:Updating / installing... 2024-04-12T20:51:56.520 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-6.9.0_rc3_g3f60ed14f7c1-1 ######################################## 2024-04-12T20:52:11.597 INFO:teuthology.orchestra.run.smithi081.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-12T20:52:34.540 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-12T20:52:34.540 DEBUG:teuthology.orchestra.run.smithi081:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-12T20:52:34.613 INFO:teuthology.orchestra.run.smithi081.stdout:/boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:52:34.613 INFO:teuthology.orchestra.run.smithi081.stdout:/boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:52:34.613 INFO:teuthology.orchestra.run.smithi081.stdout:/boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:52:34.613 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/System.map 2024-04-12T20:52:34.613 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/config 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-12T20:52:34.614 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce 2024-04-12T20:52:34.615 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-12T20:52:34.616 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-12T20:52:34.617 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-12T20:52:34.618 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-12T20:52:34.619 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ata_generic.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-12T20:52:34.620 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-12T20:52:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-12T20:52:34.622 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sis.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-12T20:52:34.623 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/brd.ko 2024-04-12T20:52:34.624 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/loop.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/virtio_blk.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkfront.ko 2024-04-12T20:52:34.625 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-12T20:52:34.626 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-12T20:52:34.627 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/uv_mmtimer.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/virtio_console.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel 2024-04-12T20:52:34.628 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-12T20:52:34.629 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-12T20:52:34.630 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/edd.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-12T20:52:34.631 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-12T20:52:34.632 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-12T20:52:34.633 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-12T20:52:34.634 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-12T20:52:34.635 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-12T20:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-12T20:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-12T20:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-12T20:52:34.639 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-12T20:52:34.640 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-12T20:52:34.641 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-12T20:52:34.642 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-12T20:52:34.643 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-12T20:52:34.645 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-12T20:52:34.645 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-12T20:52:34.645 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-12T20:52:34.645 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-12T20:52:34.645 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-12T20:52:34.646 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-12T20:52:34.647 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-dev.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-12T20:52:34.648 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-12T20:52:34.649 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-12T20:52:34.650 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-12T20:52:34.651 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-12T20:52:34.652 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-12T20:52:34.653 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-12T20:52:34.654 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-12T20:52:34.655 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mfd-core.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-12T20:52:34.656 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-12T20:52:34.657 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-12T20:52:34.658 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet 2024-04-12T20:52:34.659 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere 2024-04-12T20:52:34.660 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-12T20:52:34.661 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade 2024-04-12T20:52:34.662 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco 2024-04-12T20:52:34.663 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex 2024-04-12T20:52:34.664 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf 2024-04-12T20:52:34.665 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-12T20:52:34.666 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-12T20:52:34.667 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-12T20:52:34.668 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-12T20:52:34.669 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-12T20:52:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-12T20:52:34.671 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/net_failover.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs 2024-04-12T20:52:34.672 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/dp83640.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-12T20:52:34.673 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-12T20:52:34.674 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slhc.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tun.ko 2024-04-12T20:52:34.675 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/virtio_net.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netfront.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-12T20:52:34.676 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-12T20:52:34.677 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-12T20:52:34.678 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-12T20:52:34.679 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl 2024-04-12T20:52:34.680 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-12T20:52:34.681 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-12T20:52:34.682 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-12T20:52:34.683 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-12T20:52:34.684 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-12T20:52:34.685 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-12T20:52:34.686 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-12T20:52:34.687 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-12T20:52:34.688 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-12T20:52:34.689 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal 2024-04-12T20:52:34.690 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-12T20:52:34.691 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm 2024-04-12T20:52:34.692 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class 2024-04-12T20:52:34.693 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-12T20:52:34.694 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial 2024-04-12T20:52:34.695 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-12T20:52:34.696 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-12T20:52:34.697 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-12T20:52:34.698 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-12T20:52:34.699 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-12T20:52:34.700 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_balloon.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_mmio.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/virtio/virtio_ring.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-12T20:52:34.701 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-12T20:52:34.702 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-12T20:52:34.703 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-12T20:52:34.704 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda 2024-04-12T20:52:34.705 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/fat.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/vfat.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-12T20:52:34.706 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/fuse.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-12T20:52:34.707 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2 2024-04-12T20:52:34.708 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-12T20:52:34.709 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-12T20:52:34.710 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-12T20:52:34.711 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-12T20:52:34.712 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-12T20:52:34.713 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/rcu/rcutorture.ko 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/asn1_encoder.ko 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-12T20:52:34.714 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-12T20:52:34.715 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/zstd/zstd_compress.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-12T20:52:34.716 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-12T20:52:34.717 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-12T20:52:34.718 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-12T20:52:34.719 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/failover.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dns_resolver/dns_resolver.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-12T20:52:34.720 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-12T20:52:34.721 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-12T20:52:34.722 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-12T20:52:34.723 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-12T20:52:34.724 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-12T20:52:34.725 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-12T20:52:34.726 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-12T20:52:34.727 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-12T20:52:34.728 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-12T20:52:34.729 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-12T20:52:34.730 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-12T20:52:34.731 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-12T20:52:34.732 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_quota.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tunnel.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-12T20:52:34.733 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-12T20:52:34.734 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-12T20:52:34.735 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-12T20:52:34.736 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink 2024-04-12T20:52:34.737 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-12T20:52:34.738 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-12T20:52:34.739 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-12T20:52:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-12T20:52:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-12T20:52:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/security/keys/trusted-keys/trusted.ko 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-12T20:52:34.743 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-12T20:52:34.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-12T20:52:34.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-12T20:52:34.744 INFO:teuthology.orchestra.run.smithi081.stdout:/lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-12T20:52:34.747 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:52:34.748 DEBUG:teuthology.orchestra.run.smithi081:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-12T20:52:34.769 INFO:teuthology.orchestra.run.smithi081.stdout:postinstall scriptlet (using /bin/sh): 2024-04-12T20:52:34.769 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-12T20:52:34.769 INFO:teuthology.orchestra.run.smithi081.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g3f60ed14f7c1 /lib/modules/6.9.0-rc3-g3f60ed14f7c1/vmlinuz 2024-04-12T20:52:34.769 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2024-04-12T20:52:34.769 INFO:teuthology.orchestra.run.smithi081.stdout:for file in vmlinuz System.map config; do 2024-04-12T20:52:34.769 INFO:teuthology.orchestra.run.smithi081.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1"; then 2024-04-12T20:52:34.769 INFO:teuthology.orchestra.run.smithi081.stdout: cp "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/${file}" "/boot/${file}-6.9.0-rc3-g3f60ed14f7c1" 2024-04-12T20:52:34.769 INFO:teuthology.orchestra.run.smithi081.stdout: fi 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:done 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -d "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" ] && \ 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" >/dev/null 2>&1; then 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout: cp -r "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/dtb" "/boot/dtb-6.9.0-rc3-g3f60ed14f7c1" 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep" ]; then 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout: /usr/sbin/depmod 6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-install remove 6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:52:34.770 INFO:teuthology.orchestra.run.smithi081.stdout:fi 2024-04-12T20:52:34.771 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 68.el9 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8264254 2024-04-12T20:52:34.845 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-04-12T20:52:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T20:52:34.847 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-04-12T20:52:34.847 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-12T20:52:34.847 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:52:34.848 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-12T20:52:35.371 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-04-12T20:52:36.956 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-12T20:52:36.990 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-04-12T20:52:37.001 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-04-12T20:52:37.015 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.PM7HmLA4LA 2024-04-12T20:52:37.015 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PM7HmLA4LA 2024-04-12T20:52:37.083 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.PM7HmLA4LA 2024-04-12T20:52:37.220 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.PM7HmLA4LA is 6KB 2024-04-12T20:52:37.269 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.PM7HmLA4LA 2024-04-12T20:52:37.284 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-04-12T20:52:37.348 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-12T20:52:37.349 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-12T20:52:37.349 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-12T20:52:37.349 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1.conf 2024-04-12T20:52:37.350 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:52:37.496 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-04-12T20:53:07.499 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-12T20:53:07.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-12T20:53:07.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T20:53:42.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-12T20:53:51.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-12T20:53:51.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T20:54:09.438 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-04-12T20:54:21.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-12T20:54:21.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T20:54:23.380 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-04-12T20:54:24.043 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-12T20:54:24.043 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-12T20:54:24.044 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-04-12T20:54:24.085 INFO:teuthology.orchestra.run.smithi081.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:54:24.085 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-12T20:54:24.085 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-12T20:54:24.085 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-12T20:54:25.087 DEBUG:teuthology.parallel:result is None 2024-04-12T20:54:25.087 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-12T20:54:25.096 INFO:teuthology.task.internal:Creating test directory... 2024-04-12T20:54:25.097 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T20:54:25.103 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T20:54:25.106 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-12T20:54:25.152 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-12T20:54:25.159 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-12T20:54:25.167 INFO:teuthology.task.internal:Creating archive directory... 2024-04-12T20:54:25.168 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T20:54:25.174 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T20:54:25.176 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-12T20:54:25.311 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-12T20:54:25.318 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-12T20:54:25.318 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T20:54:25.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T20:54:25.335 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-12T20:54:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T20:54:25.352 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T20:54:25.356 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T20:54:25.377 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-12T20:54:25.422 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T20:54:25.439 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T20:54:25.488 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T20:54:25.490 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T20:54:25.509 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T20:54:25.565 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-12T20:54:25.569 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-12T20:54:25.577 INFO:teuthology.task.internal:Configuring sudo... 2024-04-12T20:54:25.577 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T20:54:25.582 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T20:54:25.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-12T20:54:25.654 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-12T20:54:25.664 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-12T20:54:25.664 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T20:54:25.668 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T20:54:25.671 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-12T20:54:25.704 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T20:54:25.818 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T20:54:25.908 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T20:54:25.984 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T20:54:26.065 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T20:54:26.065 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T20:54:26.178 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T20:54:26.222 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T20:54:26.286 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T20:54:26.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T20:54:26.372 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T20:54:26.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T20:54:26.441 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T20:54:26.484 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-12T20:54:26.542 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T20:54:26.627 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-12T20:54:26.673 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T20:54:26.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-12T20:54:26.743 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-04-12T20:54:26.750 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-04-12T20:54:26.752 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-04-12T20:54:26.792 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T20:54:26.831 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T20:54:26.871 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-12T20:54:27.331 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-12T20:54:27.340 INFO:teuthology.task.internal:Starting timer... 2024-04-12T20:54:27.340 INFO:teuthology.run_tasks:Running task pcp... 2024-04-12T20:54:27.351 INFO:teuthology.run_tasks:Running task selinux... 2024-04-12T20:54:27.360 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-12T20:54:27.360 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-04-12T20:54:27.469 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-04-12T20:54:27.473 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-04-12T20:54:27.543 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-04-12T20:54:27.545 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-04-12T20:54:27.605 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-04-12T20:54:27.607 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-12T20:54:27.607 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-04-12T20:54:27.670 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-04-12T20:54:27.670 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-04-12T20:54:27.697 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-04-12T20:54:27.697 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-04-12T20:54:27.720 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-04-12T20:54:27.721 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-04-12T20:54:27.721 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-12T20:54:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T20:54:27.802 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-12T20:54:27.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T20:54:27.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-12T20:54:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-12T20:54:27.859 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-12T20:54:27.859 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-04-12T20:54:27.940 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-04-12T20:54:27.974 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-04-12T20:54:28.007 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-12T20:54:28.017 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-12T20:54:28.022 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-12T20:54:28.150 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-12T20:54:28.166 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-12T20:54:28.167 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi104.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-12T21:00:32.638 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-04-12T21:00:32.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-12T21:00:32.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:00:32.789 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-04-12T21:00:32.951 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-04-12T21:00:32.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-12T21:00:32.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:00:33.046 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-04-12T21:00:33.116 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-04-12T21:00:33.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-12T21:00:33.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-12T21:00:33.212 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-04-12T21:00:33.282 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-04-12T21:00:33.282 INFO:teuthology.run_tasks:Running task clock... 2024-04-12T21:00:33.299 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-12T21:00:33.299 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T21:00:33.300 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T21:00:33.307 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T21:00:33.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T21:00:33.309 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-12T21:00:33.310 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-12T21:00:33.340 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T21:00:33.355 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T21:00:33.357 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T21:00:33.372 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T21:00:33.386 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-04-12T21:00:33.400 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-04-12T21:00:33.402 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-04-12T21:00:33.406 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-12T21:00:33.416 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-04-12T21:00:33.416 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T21:00:33.432 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T21:00:33.433 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T21:00:33.448 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T21:00:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-12T21:00:33.476 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-04-12T21:00:33.479 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T21:00:33.480 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-04-12T21:00:33.480 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.480 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.480 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.480 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.492 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-04-12T21:00:33.495 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T21:00:33.495 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-04-12T21:00:33.495 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.495 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.495 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.495 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.543 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-04-12T21:00:33.581 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-04-12T21:00:33.629 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-12T21:00:33.684 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-12T21:00:33.818 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-04-12T21:00:33.827 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-12T21:00:33.827 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-04-12T21:00:33.827 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.827 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.827 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.827 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-12T21:00:33.829 INFO:teuthology.run_tasks:Running task install... 2024-04-12T21:00:33.837 DEBUG:teuthology.task.install:project ceph 2024-04-12T21:00:33.837 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '324baf8ddb91dea46cac50cb535c306115791355'}} 2024-04-12T21:00:33.837 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '324baf8ddb91dea46cac50cb535c306115791355'} 2024-04-12T21:00:33.837 INFO:teuthology.task.install:Using flavor: default 2024-04-12T21:00:33.843 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-12T21:00:33.843 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-12T21:00:33.844 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '324baf8ddb91dea46cac50cb535c306115791355', 'tag': None, 'wait_for_package': False} 2024-04-12T21:00:33.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:00:33.846 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '324baf8ddb91dea46cac50cb535c306115791355', 'tag': None, 'wait_for_package': False} 2024-04-12T21:00:33.846 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:00:33.847 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '324baf8ddb91dea46cac50cb535c306115791355', 'tag': None, 'wait_for_package': False} 2024-04-12T21:00:33.848 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:00:33.992 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/ 2024-04-12T21:00:33.992 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2879.g324baf8d 2024-04-12T21:00:34.003 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/ 2024-04-12T21:00:34.003 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2879.g324baf8d 2024-04-12T21:00:34.066 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/ 2024-04-12T21:00:34.066 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2879.g324baf8d 2024-04-12T21:00:34.149 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T21:00:34.150 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:00:34.150 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T21:00:34.187 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T21:00:34.187 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:00:34.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T21:00:34.208 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-12T21:00:34.208 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:00:34.208 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-12T21:00:34.217 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-12T21:00:34.217 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/324baf8ddb91dea46cac50cb535c306115791355/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T21:00:34.236 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-12T21:00:34.236 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/324baf8ddb91dea46cac50cb535c306115791355/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T21:00:34.251 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-12T21:00:34.251 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/324baf8ddb91dea46cac50cb535c306115791355/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-12T21:00:34.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T21:00:34.305 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T21:00:34.370 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T21:00:34.384 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T21:00:34.393 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-12T21:00:34.438 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-04-12T21:00:34.439 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-04-12T21:00:34.451 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-04-12T21:00:34.452 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-04-12T21:00:34.529 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-12T21:00:34.621 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-04-12T21:00:34.626 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-04-12T21:00:34.674 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-04-12T21:00:34.686 INFO:teuthology.orchestra.run.smithi104.stdout:74 files removed 2024-04-12T21:00:34.700 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-12T21:00:34.712 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-12T21:00:34.995 INFO:teuthology.orchestra.run.smithi081.stdout:74 files removed 2024-04-12T21:00:35.033 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-12T21:00:35.411 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 204 kB/s | 83 kB 00:00 2024-04-12T21:00:35.414 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 198 kB/s | 83 kB 00:00 2024-04-12T21:00:35.778 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-04-12T21:00:35.815 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-04-12T21:00:36.003 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 180 kB/s | 83 kB 00:00 2024-04-12T21:00:36.216 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-04-12T21:00:36.237 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-04-12T21:00:36.954 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 13 kB/s | 11 kB 00:00 2024-04-12T21:00:37.140 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 9.3 MB/s | 8.1 MB 00:00 2024-04-12T21:00:37.160 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 8.9 MB/s | 8.1 MB 00:00 2024-04-12T21:00:37.504 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-04-12T21:00:38.449 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 9.3 MB/s | 8.1 MB 00:00 2024-04-12T21:00:38.940 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.5 MB 00:00 2024-04-12T21:00:39.152 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.5 MB 00:00 2024-04-12T21:00:39.972 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-12T21:00:40.034 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-04-12T21:00:40.375 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.5 MB 00:00 2024-04-12T21:00:41.873 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-12T21:00:42.355 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-04-12T21:00:44.143 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:03 2024-04-12T21:00:48.599 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.2 MB 00:03 2024-04-12T21:00:50.480 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:06 2024-04-12T21:00:50.966 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-12T21:00:51.169 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.2 MB 00:03 2024-04-12T21:00:52.586 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-04-12T21:00:53.294 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 21 MB 00:02 2024-04-12T21:00:54.873 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-12T21:00:56.423 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 95 kB/s | 16 kB 00:00 2024-04-12T21:00:56.537 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 21 MB 00:03 2024-04-12T21:00:58.383 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-12T21:00:59.531 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 7.2 MB/s | 21 MB 00:02 2024-04-12T21:01:00.957 INFO:teuthology.orchestra.run.smithi104.stderr:Modular dependency problems: 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 1: conflicting requests 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 2: conflicting requests 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 3: conflicting requests 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 4: conflicting requests 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 5: conflicting requests 2024-04-12T21:01:00.958 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 6: conflicting requests 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 7: conflicting requests 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 8: conflicting requests 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 9: conflicting requests 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 10: conflicting requests 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 11: conflicting requests 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.959 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 12: conflicting requests 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 13: conflicting requests 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 14: conflicting requests 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 15: conflicting requests 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 16: conflicting requests 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 17: conflicting requests 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 18: conflicting requests 2024-04-12T21:01:00.960 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 19: conflicting requests 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 20: conflicting requests 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 21: conflicting requests 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 22: conflicting requests 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 23: conflicting requests 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 24: conflicting requests 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 25: conflicting requests 2024-04-12T21:01:00.961 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 26: conflicting requests 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 27: conflicting requests 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 28: conflicting requests 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 29: conflicting requests 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 30: conflicting requests 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 31: conflicting requests 2024-04-12T21:01:00.962 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 32: conflicting requests 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 33: conflicting requests 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 34: conflicting requests 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 35: conflicting requests 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 36: conflicting requests 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 37: conflicting requests 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.963 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 38: cannot install the best candidate for the job 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 39: conflicting requests 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.964 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 40: cannot install the best candidate for the job 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 41: cannot install the best candidate for the job 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.965 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 42: cannot install the best candidate for the job 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 43: cannot install the best candidate for the job 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.966 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 44: cannot install the best candidate for the job 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: Problem 45: cannot install the best candidate for the job 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-12T21:01:00.967 INFO:teuthology.orchestra.run.smithi104.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-12T21:01:01.353 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-12T21:01:01.354 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-12T21:01:01.368 INFO:teuthology.orchestra.run.smithi104.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-12T21:01:01.369 INFO:teuthology.orchestra.run.smithi104.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-12T21:01:01.370 INFO:teuthology.orchestra.run.smithi104.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-12T21:01:01.371 INFO:teuthology.orchestra.run.smithi104.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-12T21:01:01.373 INFO:teuthology.orchestra.run.smithi104.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-12T21:01:01.376 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-12T21:01:01.378 INFO:teuthology.orchestra.run.smithi104.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-12T21:01:01.472 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-04-12T21:01:01.484 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================= 2024-04-12T21:01:01.484 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-04-12T21:01:01.484 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================= 2024-04-12T21:01:01.484 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-04-12T21:01:01.484 INFO:teuthology.orchestra.run.smithi104.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 6.6 k 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 5.4 M 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 889 k 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 145 k 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 1.5 M 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 142 k 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 3.5 M 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 7.4 M 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 49 k 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 10 M 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 49 M 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 264 k 2024-04-12T21:01:01.485 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 762 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 224 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-shell x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 27 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 18 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 32 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 746 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 127 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-12T21:01:01.486 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 161 k 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 320 k 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 297 k 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 99 k 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 85 k 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.1 M 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 171 k 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager x86_64 1:1.47.4-1.el9 baseos 2.3 M 2024-04-12T21:01:01.487 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 baseos 1.8 M 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 baseos 42 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 baseos 249 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.4 M 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.1 M 2024-04-12T21:01:01.488 INFO:teuthology.orchestra.run.smithi104.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 21 M 2024-04-12T21:01:01.489 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 34 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 2.4 M 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 249 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 4.7 M 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 17 M 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 16 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 25 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-12T21:01:01.490 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 164 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 500 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 5.2 M 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-12T21:01:01.491 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-12T21:01:01.492 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 45 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 135 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-12T21:01:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-12T21:01:01.494 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-12T21:01:01.495 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-12T21:01:01.496 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:01:01.497 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-04-12T21:01:01.498 INFO:teuthology.orchestra.run.smithi104.stdout:========================================================================================= 2024-04-12T21:01:01.498 INFO:teuthology.orchestra.run.smithi104.stdout:Install 150 Packages 2024-04-12T21:01:01.498 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 24 Packages 2024-04-12T21:01:01.498 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:01:01.498 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 229 M 2024-04-12T21:01:01.498 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-04-12T21:01:01.587 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-04-12T21:01:02.784 INFO:teuthology.orchestra.run.smithi104.stdout:(1/174): ceph-19.0.0-2879.g324baf8d.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-04-12T21:01:03.277 INFO:teuthology.orchestra.run.smithi104.stdout:(2/174): ceph-fuse-19.0.0-2879.g324baf8d.el9.x8 1.8 MB/s | 889 kB 00:00 2024-04-12T21:01:03.384 INFO:teuthology.orchestra.run.smithi104.stdout:(3/174): ceph-immutable-object-cache-19.0.0-287 1.3 MB/s | 145 kB 00:00 2024-04-12T21:01:03.493 INFO:teuthology.orchestra.run.smithi104.stdout:(4/174): ceph-base-19.0.0-2879.g324baf8d.el9.x8 4.9 MB/s | 5.4 MB 00:01 2024-04-12T21:01:03.709 INFO:teuthology.orchestra.run.smithi104.stdout:(5/174): ceph-mgr-19.0.0-2879.g324baf8d.el9.x86 6.9 MB/s | 1.5 MB 00:00 2024-04-12T21:01:03.760 INFO:teuthology.orchestra.run.smithi104.stdout:(6/174): ceph-mds-19.0.0-2879.g324baf8d.el9.x86 6.3 MB/s | 2.4 MB 00:00 2024-04-12T21:01:04.204 INFO:teuthology.orchestra.run.smithi153.stderr:Modular dependency problems: 2024-04-12T21:01:04.204 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-04-12T21:01:04.204 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 1: conflicting requests 2024-04-12T21:01:04.204 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 2: conflicting requests 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 3: conflicting requests 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 4: conflicting requests 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 5: conflicting requests 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 6: conflicting requests 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.205 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 7: conflicting requests 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 8: conflicting requests 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 9: conflicting requests 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 10: conflicting requests 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 11: conflicting requests 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 12: conflicting requests 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 13: conflicting requests 2024-04-12T21:01:04.206 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 14: conflicting requests 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 15: conflicting requests 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 16: conflicting requests 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 17: conflicting requests 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 18: conflicting requests 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 19: conflicting requests 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.207 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 20: conflicting requests 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 21: conflicting requests 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 22: conflicting requests 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 23: conflicting requests 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 24: conflicting requests 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 25: conflicting requests 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 26: conflicting requests 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 27: conflicting requests 2024-04-12T21:01:04.208 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 28: conflicting requests 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 29: conflicting requests 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 30: conflicting requests 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 31: conflicting requests 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 32: conflicting requests 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 33: conflicting requests 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 34: conflicting requests 2024-04-12T21:01:04.209 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 35: conflicting requests 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 36: conflicting requests 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 37: conflicting requests 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 38: cannot install the best candidate for the job 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.210 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 39: conflicting requests 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 40: cannot install the best candidate for the job 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 41: cannot install the best candidate for the job 2024-04-12T21:01:04.211 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 42: cannot install the best candidate for the job 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 43: cannot install the best candidate for the job 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.212 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 44: cannot install the best candidate for the job 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: Problem 45: cannot install the best candidate for the job 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.213 INFO:teuthology.orchestra.run.smithi153.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-12T21:01:04.293 INFO:teuthology.orchestra.run.smithi104.stdout:(7/174): ceph-mon-19.0.0-2879.g324baf8d.el9.x86 8.0 MB/s | 4.7 MB 00:00 2024-04-12T21:01:04.602 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-12T21:01:04.603 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-12T21:01:04.617 INFO:teuthology.orchestra.run.smithi153.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-12T21:01:04.618 INFO:teuthology.orchestra.run.smithi153.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-12T21:01:04.619 INFO:teuthology.orchestra.run.smithi153.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-12T21:01:04.620 INFO:teuthology.orchestra.run.smithi153.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-12T21:01:04.622 INFO:teuthology.orchestra.run.smithi153.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-12T21:01:04.625 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-12T21:01:04.626 INFO:teuthology.orchestra.run.smithi153.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-12T21:01:04.721 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 6.6 k 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 5.4 M 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 889 k 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 145 k 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 1.5 M 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 142 k 2024-04-12T21:01:04.736 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 3.5 M 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 7.4 M 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 49 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 10 M 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 49 M 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 264 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 762 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 224 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 27 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 18 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 32 k 2024-04-12T21:01:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 746 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 127 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 161 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 320 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 297 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 99 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 85 k 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.1 M 2024-04-12T21:01:04.738 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 171 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager x86_64 1:1.47.4-1.el9 baseos 2.3 M 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 baseos 1.8 M 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 baseos 42 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 baseos 249 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-12T21:01:04.739 INFO:teuthology.orchestra.run.smithi153.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.4 M 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.1 M 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-12T21:01:04.740 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 21 M 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 34 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 2.4 M 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 249 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 4.7 M 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 17 M 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 16 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 25 k 2024-04-12T21:01:04.741 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 164 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-12T21:01:04.742 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 500 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 5.2 M 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-12T21:01:04.743 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 45 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 135 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-12T21:01:04.744 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-12T21:01:04.745 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-12T21:01:04.746 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-12T21:01:04.747 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-12T21:01:04.748 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout:========================================================================================= 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout:Install 150 Packages 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 24 Packages 2024-04-12T21:01:04.749 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:01:04.750 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 229 M 2024-04-12T21:01:04.750 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-04-12T21:01:05.459 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-04-12T21:01:05.652 INFO:teuthology.orchestra.run.smithi104.stdout:(8/174): ceph-osd-19.0.0-2879.g324baf8d.el9.x86 8.9 MB/s | 17 MB 00:01 2024-04-12T21:01:05.752 INFO:teuthology.orchestra.run.smithi104.stdout:(9/174): ceph-radosgw-19.0.0-2879.g324baf8d.el9 6.8 MB/s | 10 MB 00:01 2024-04-12T21:01:05.769 INFO:teuthology.orchestra.run.smithi104.stdout:(10/174): ceph-selinux-19.0.0-2879.g324baf8d.el 216 kB/s | 25 kB 00:00 2024-04-12T21:01:05.911 INFO:teuthology.orchestra.run.smithi104.stdout:(11/174): cephfs-mirror-19.0.0-2879.g324baf8d.e 1.5 MB/s | 224 kB 00:00 2024-04-12T21:01:06.103 INFO:teuthology.orchestra.run.smithi104.stdout:(12/174): ceph-common-19.0.0-2879.g324baf8d.el9 5.6 MB/s | 21 MB 00:03 2024-04-12T21:01:06.120 INFO:teuthology.orchestra.run.smithi104.stdout:(13/174): cephfs-shell-19.0.0-2879.g324baf8d.el 129 kB/s | 27 kB 00:00 2024-04-12T21:01:06.245 INFO:teuthology.orchestra.run.smithi104.stdout:(14/174): libcephfs-devel-19.0.0-2879.g324baf8d 224 kB/s | 32 kB 00:00 2024-04-12T21:01:06.287 INFO:teuthology.orchestra.run.smithi104.stdout:(15/174): libcephfs2-19.0.0-2879.g324baf8d.el9. 4.4 MB/s | 746 kB 00:00 2024-04-12T21:01:06.387 INFO:teuthology.orchestra.run.smithi104.stdout:(16/174): libcephsqlite-19.0.0-2879.g324baf8d.e 1.1 MB/s | 164 kB 00:00 2024-04-12T21:01:06.412 INFO:teuthology.orchestra.run.smithi104.stdout:(17/174): librados-devel-19.0.0-2879.g324baf8d. 1.0 MB/s | 127 kB 00:00 2024-04-12T21:01:06.538 INFO:teuthology.orchestra.run.smithi104.stdout:(18/174): libradosstriper1-19.0.0-2879.g324baf8 3.3 MB/s | 500 kB 00:00 2024-04-12T21:01:06.663 INFO:teuthology.orchestra.run.smithi104.stdout:(19/174): python3-ceph-argparse-19.0.0-2879.g32 361 kB/s | 45 kB 00:00 2024-04-12T21:01:06.813 INFO:teuthology.orchestra.run.smithi104.stdout:(20/174): python3-ceph-common-19.0.0-2879.g324b 900 kB/s | 135 kB 00:00 2024-04-12T21:01:06.964 INFO:teuthology.orchestra.run.smithi104.stdout:(21/174): python3-cephfs-19.0.0-2879.g324baf8d. 1.1 MB/s | 161 kB 00:00 2024-04-12T21:01:07.130 INFO:teuthology.orchestra.run.smithi104.stdout:(22/174): python3-rados-19.0.0-2879.g324baf8d.e 1.9 MB/s | 320 kB 00:00 2024-04-12T21:01:07.205 INFO:teuthology.orchestra.run.smithi104.stdout:(23/174): librgw2-19.0.0-2879.g324baf8d.el9.x86 6.6 MB/s | 5.2 MB 00:00 2024-04-12T21:01:07.289 INFO:teuthology.orchestra.run.smithi104.stdout:(24/174): python3-rbd-19.0.0-2879.g324baf8d.el9 1.8 MB/s | 297 kB 00:00 2024-04-12T21:01:07.348 INFO:teuthology.orchestra.run.smithi104.stdout:(25/174): python3-rgw-19.0.0-2879.g324baf8d.el9 698 kB/s | 99 kB 00:00 2024-04-12T21:01:07.432 INFO:teuthology.orchestra.run.smithi104.stdout:(26/174): rbd-fuse-19.0.0-2879.g324baf8d.el9.x8 601 kB/s | 85 kB 00:00 2024-04-12T21:01:07.590 INFO:teuthology.orchestra.run.smithi104.stdout:(27/174): rbd-nbd-19.0.0-2879.g324baf8d.el9.x86 1.1 MB/s | 171 kB 00:00 2024-04-12T21:01:07.715 INFO:teuthology.orchestra.run.smithi104.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2879.g 274 kB/s | 34 kB 00:00 2024-04-12T21:01:07.849 INFO:teuthology.orchestra.run.smithi104.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2879.g324baf8 1.0 MB/s | 142 kB 00:00 2024-04-12T21:01:07.907 INFO:teuthology.orchestra.run.smithi104.stdout:(30/174): rbd-mirror-19.0.0-2879.g324baf8d.el9. 5.5 MB/s | 3.1 MB 00:00 2024-04-12T21:01:08.105 INFO:teuthology.orchestra.run.smithi081.stderr:Modular dependency problems: 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 1: conflicting requests 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 2: conflicting requests 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 3: conflicting requests 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 4: conflicting requests 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 5: conflicting requests 2024-04-12T21:01:08.106 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 6: conflicting requests 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 7: conflicting requests 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 8: conflicting requests 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 9: conflicting requests 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 10: conflicting requests 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 11: conflicting requests 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.107 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 12: conflicting requests 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 13: conflicting requests 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 14: conflicting requests 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 15: conflicting requests 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 16: conflicting requests 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 17: conflicting requests 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 18: conflicting requests 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.108 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 19: conflicting requests 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 20: conflicting requests 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 21: conflicting requests 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 22: conflicting requests 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 23: conflicting requests 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 24: conflicting requests 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 25: conflicting requests 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.109 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 26: conflicting requests 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 27: conflicting requests 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 28: conflicting requests 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 29: conflicting requests 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 30: conflicting requests 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 31: conflicting requests 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 32: conflicting requests 2024-04-12T21:01:08.110 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 33: conflicting requests 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 34: conflicting requests 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 35: conflicting requests 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 36: conflicting requests 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 37: conflicting requests 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.111 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 38: cannot install the best candidate for the job 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 39: conflicting requests 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.112 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 40: cannot install the best candidate for the job 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 41: cannot install the best candidate for the job 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.113 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 42: cannot install the best candidate for the job 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 43: cannot install the best candidate for the job 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.114 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 44: cannot install the best candidate for the job 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: Problem 45: cannot install the best candidate for the job 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.115 INFO:teuthology.orchestra.run.smithi081.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-12T21:01:08.733 INFO:teuthology.orchestra.run.smithi104.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2879.g324ba 4.0 MB/s | 3.5 MB 00:00 2024-04-12T21:01:08.754 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-12T21:01:08.756 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-12T21:01:08.771 INFO:teuthology.orchestra.run.smithi153.stdout:(1/174): ceph-19.0.0-2879.g324baf8d.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-04-12T21:01:08.779 INFO:teuthology.orchestra.run.smithi081.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-12T21:01:08.780 INFO:teuthology.orchestra.run.smithi081.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-12T21:01:08.782 INFO:teuthology.orchestra.run.smithi081.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-12T21:01:08.783 INFO:teuthology.orchestra.run.smithi081.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-12T21:01:08.786 INFO:teuthology.orchestra.run.smithi081.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-12T21:01:08.790 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-12T21:01:08.791 INFO:teuthology.orchestra.run.smithi081.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-12T21:01:08.874 INFO:teuthology.orchestra.run.smithi104.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2879.g32 1.7 MB/s | 249 kB 00:00 2024-04-12T21:01:08.891 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-04-12T21:01:08.911 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-04-12T21:01:08.911 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-04-12T21:01:08.911 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-04-12T21:01:08.911 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-04-12T21:01:08.911 INFO:teuthology.orchestra.run.smithi081.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 6.6 k 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 5.4 M 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 889 k 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 145 k 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 1.5 M 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 142 k 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 3.5 M 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 7.4 M 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 49 k 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 10 M 2024-04-12T21:01:08.912 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 49 M 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 264 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 762 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 224 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-shell x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 27 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 18 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel x86_64 0.3.112-1.el8 CentOS-BaseOS 18 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 32 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 746 k 2024-04-12T21:01:08.913 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 127 k 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 161 k 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 320 k 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 297 k 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 99 k 2024-04-12T21:01:08.914 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 85 k 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.1 M 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 171 k 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager x86_64 1:1.47.4-1.el9 baseos 2.3 M 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 baseos 1.8 M 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 baseos 42 k 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 baseos 249 k 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-12T21:01:08.915 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.4 M 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 3.1 M 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-12T21:01:08.916 INFO:teuthology.orchestra.run.smithi081.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: xfsdump x86_64 3.1.12-4.el9 baseos 323 k 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 21 M 2024-04-12T21:01:08.917 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 34 k 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 2.4 M 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 249 k 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 4.7 M 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 17 M 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2879.g324baf8d.el9 ceph-noarch 16 k 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 25 k 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-12T21:01:08.918 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 164 k 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-12T21:01:08.919 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 500 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 5.2 M 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-12T21:01:08.920 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-12T21:01:08.921 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 45 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.0.0-2879.g324baf8d.el9 ceph 135 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-12T21:01:08.922 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-12T21:01:08.923 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-12T21:01:08.924 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-12T21:01:08.925 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-12T21:01:08.926 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-04-12T21:01:08.927 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-12T21:01:08.928 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:01:08.928 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-04-12T21:01:08.928 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-04-12T21:01:08.928 INFO:teuthology.orchestra.run.smithi081.stdout:Install 150 Packages 2024-04-12T21:01:08.928 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 24 Packages 2024-04-12T21:01:08.928 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:01:08.929 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 229 M 2024-04-12T21:01:08.929 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-04-12T21:01:09.000 INFO:teuthology.orchestra.run.smithi104.stdout:(33/174): ceph-mgr-rook-19.0.0-2879.g324baf8d.e 395 kB/s | 49 kB 00:00 2024-04-12T21:01:09.117 INFO:teuthology.orchestra.run.smithi104.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2879.g3 141 kB/s | 16 kB 00:00 2024-04-12T21:01:09.664 INFO:teuthology.orchestra.run.smithi153.stdout:(2/174): ceph-base-19.0.0-2879.g324baf8d.el9.x8 3.9 MB/s | 5.4 MB 00:01 2024-04-12T21:01:09.667 INFO:teuthology.orchestra.run.smithi104.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 4.2 MB/s | 7.4 MB 00:01 2024-04-12T21:01:09.697 INFO:teuthology.orchestra.run.smithi153.stdout:(3/174): ceph-fuse-19.0.0-2879.g324baf8d.el9.x8 961 kB/s | 889 kB 00:00 2024-04-12T21:01:09.755 INFO:teuthology.orchestra.run.smithi153.stdout:(4/174): ceph-immutable-object-cache-19.0.0-287 1.5 MB/s | 145 kB 00:00 2024-04-12T21:01:09.926 INFO:teuthology.orchestra.run.smithi104.stdout:(36/174): cephadm-19.0.0-2879.g324baf8d.el9.noa 2.9 MB/s | 762 kB 00:00 2024-04-12T21:01:10.014 INFO:teuthology.orchestra.run.smithi153.stdout:(5/174): ceph-mgr-19.0.0-2879.g324baf8d.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-04-12T21:01:10.026 INFO:teuthology.orchestra.run.smithi104.stdout:(37/174): cephfs-top-19.0.0-2879.g324baf8d.el9. 183 kB/s | 18 kB 00:00 2024-04-12T21:01:10.068 INFO:teuthology.orchestra.run.smithi104.stdout:(38/174): ceph-volume-19.0.0-2879.g324baf8d.el9 277 kB/s | 264 kB 00:00 2024-04-12T21:01:10.193 INFO:teuthology.orchestra.run.smithi104.stdout:(39/174): libaio-devel-0.3.112-1.el8.x86_64.rpm 105 kB/s | 18 kB 00:00 2024-04-12T21:01:10.318 INFO:teuthology.orchestra.run.smithi104.stdout:(40/174): fuse-2.9.9-15.el9.x86_64.rpm 321 kB/s | 80 kB 00:00 2024-04-12T21:01:10.419 INFO:teuthology.orchestra.run.smithi104.stdout:(41/174): jq-1.6-16.el9.x86_64.rpm 833 kB/s | 187 kB 00:00 2024-04-12T21:01:10.456 INFO:teuthology.orchestra.run.smithi104.stdout:(42/174): libgfortran-11.4.1-3.el9.x86_64.rpm 5.8 MB/s | 803 kB 00:00 2024-04-12T21:01:10.497 INFO:teuthology.orchestra.run.smithi104.stdout:(43/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 195 kB 00:00 2024-04-12T21:01:10.539 INFO:teuthology.orchestra.run.smithi104.stdout:(44/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 218 kB 00:00 2024-04-12T21:01:10.565 INFO:teuthology.orchestra.run.smithi104.stdout:(45/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-12T21:01:10.632 INFO:teuthology.orchestra.run.smithi104.stdout:(46/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-12T21:01:10.682 INFO:teuthology.orchestra.run.smithi104.stdout:(47/174): python3-cryptography-36.0.1-4.el9.x86 8.6 MB/s | 1.2 MB 00:00 2024-04-12T21:01:10.689 INFO:teuthology.orchestra.run.smithi153.stdout:(6/174): ceph-mon-19.0.0-2879.g324baf8d.el9.x86 7.0 MB/s | 4.7 MB 00:00 2024-04-12T21:01:10.724 INFO:teuthology.orchestra.run.smithi104.stdout:(48/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-04-12T21:01:10.749 INFO:teuthology.orchestra.run.smithi104.stdout:(49/174): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-12T21:01:10.781 INFO:teuthology.orchestra.run.smithi153.stdout:(7/174): ceph-mds-19.0.0-2879.g324baf8d.el9.x86 2.2 MB/s | 2.4 MB 00:01 2024-04-12T21:01:10.791 INFO:teuthology.orchestra.run.smithi104.stdout:(50/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-12T21:01:10.833 INFO:teuthology.orchestra.run.smithi104.stdout:(51/174): automake-1.16.2-8.el9.noarch.rpm 8.1 MB/s | 689 kB 00:00 2024-04-12T21:01:10.900 INFO:teuthology.orchestra.run.smithi104.stdout:(52/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-12T21:01:10.942 INFO:teuthology.orchestra.run.smithi104.stdout:(53/174): bison-3.7.4-5.el9.x86_64.rpm 6.1 MB/s | 943 kB 00:00 2024-04-12T21:01:10.976 INFO:teuthology.orchestra.run.smithi104.stdout:(54/174): elfutils-libelf-devel-0.190-2.el9.x86 284 kB/s | 21 kB 00:00 2024-04-12T21:01:11.026 INFO:teuthology.orchestra.run.smithi104.stdout:(55/174): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-04-12T21:01:11.043 INFO:teuthology.orchestra.run.smithi104.stdout:(56/174): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-04-12T21:01:11.119 INFO:teuthology.orchestra.run.smithi104.stdout:(57/174): flexiblas-openblas-openmp-3.0.4-8.el9 243 kB/s | 18 kB 00:00 2024-04-12T21:01:11.185 INFO:teuthology.orchestra.run.smithi104.stdout:(58/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 160 kB 00:00 2024-04-12T21:01:11.236 INFO:teuthology.orchestra.run.smithi104.stdout:(59/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-12T21:01:11.269 INFO:teuthology.orchestra.run.smithi104.stdout:(60/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 542 kB/s | 45 kB 00:00 2024-04-12T21:01:11.320 INFO:teuthology.orchestra.run.smithi104.stdout:(61/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 662 kB 00:00 2024-04-12T21:01:11.353 INFO:teuthology.orchestra.run.smithi104.stdout:(62/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 582 kB 00:00 2024-04-12T21:01:11.387 INFO:teuthology.orchestra.run.smithi104.stdout:(63/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 382 kB/s | 25 kB 00:00 2024-04-12T21:01:11.420 INFO:teuthology.orchestra.run.smithi104.stdout:(64/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 711 kB/s | 47 kB 00:00 2024-04-12T21:01:11.454 INFO:teuthology.orchestra.run.smithi104.stdout:(65/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-04-12T21:01:11.504 INFO:teuthology.orchestra.run.smithi104.stdout:(66/174): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-12T21:01:11.521 INFO:teuthology.orchestra.run.smithi104.stdout:(67/174): openblas-0.3.21-2.el9.x86_64.rpm 529 kB/s | 35 kB 00:00 2024-04-12T21:01:11.722 INFO:teuthology.orchestra.run.smithi104.stdout:(68/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 21 MB/s | 4.1 MB 00:00 2024-04-12T21:01:11.805 INFO:teuthology.orchestra.run.smithi104.stdout:(69/174): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-04-12T21:01:11.889 INFO:teuthology.orchestra.run.smithi104.stdout:(70/174): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-04-12T21:01:11.914 INFO:teuthology.orchestra.run.smithi104.stdout:(71/174): postgresql-contrib-13.14-1.el9.x86_64 8.2 MB/s | 911 kB 00:00 2024-04-12T21:01:11.956 INFO:teuthology.orchestra.run.smithi104.stdout:(72/174): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-04-12T21:01:12.081 INFO:teuthology.orchestra.run.smithi104.stdout:(73/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-04-12T21:01:12.182 INFO:teuthology.orchestra.run.smithi104.stdout:(74/174): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-12T21:01:12.223 INFO:teuthology.orchestra.run.smithi104.stdout:(75/174): python3-attrs-20.3.0-7.el9.noarch.rpm 617 kB/s | 87 kB 00:00 2024-04-12T21:01:12.299 INFO:teuthology.orchestra.run.smithi104.stdout:(76/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-04-12T21:01:12.424 INFO:teuthology.orchestra.run.smithi104.stdout:(77/174): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 6.0 MB 00:00 2024-04-12T21:01:12.449 INFO:teuthology.orchestra.run.smithi104.stdout:(78/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-04-12T21:01:12.525 INFO:teuthology.orchestra.run.smithi104.stdout:(79/174): python3-jsonpatch-1.21-16.el9.noarch. 255 kB/s | 26 kB 00:00 2024-04-12T21:01:12.558 INFO:teuthology.orchestra.run.smithi104.stdout:(80/174): python3-jsonpointer-2.0-4.el9.noarch. 176 kB/s | 19 kB 00:00 2024-04-12T21:01:12.594 INFO:teuthology.orchestra.run.smithi081.stdout:(1/174): ceph-19.0.0-2879.g324baf8d.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-12T21:01:12.642 INFO:teuthology.orchestra.run.smithi104.stdout:(81/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-04-12T21:01:12.667 INFO:teuthology.orchestra.run.smithi104.stdout:(82/174): python3-markupsafe-1.1.1-12.el9.x86_6 319 kB/s | 35 kB 00:00 2024-04-12T21:01:12.776 INFO:teuthology.orchestra.run.smithi104.stdout:(83/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-12T21:01:12.960 INFO:teuthology.orchestra.run.smithi104.stdout:(84/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-12T21:01:12.984 INFO:teuthology.orchestra.run.smithi104.stdout:(85/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 222 kB 00:00 2024-04-12T21:01:13.027 INFO:teuthology.orchestra.run.smithi104.stdout:(86/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-12T21:01:13.068 INFO:teuthology.orchestra.run.smithi104.stdout:(87/174): python3-prettytable-0.7.2-27.el9.noar 505 kB/s | 42 kB 00:00 2024-04-12T21:01:13.119 INFO:teuthology.orchestra.run.smithi104.stdout:(88/174): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-04-12T21:01:13.144 INFO:teuthology.orchestra.run.smithi104.stdout:(89/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-12T21:01:13.186 INFO:teuthology.orchestra.run.smithi104.stdout:(90/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-12T21:01:13.210 INFO:teuthology.orchestra.run.smithi081.stdout:(2/174): ceph-fuse-19.0.0-2879.g324baf8d.el9.x8 1.4 MB/s | 889 kB 00:00 2024-04-12T21:01:13.211 INFO:teuthology.orchestra.run.smithi104.stdout:(91/174): python3-pytz-2021.1-5.el9.noarch.rpm 761 kB/s | 51 kB 00:00 2024-04-12T21:01:13.253 INFO:teuthology.orchestra.run.smithi104.stdout:(92/174): python3-requests-oauthlib-1.3.0-12.el 804 kB/s | 54 kB 00:00 2024-04-12T21:01:13.327 INFO:teuthology.orchestra.run.smithi081.stdout:(3/174): ceph-immutable-object-cache-19.0.0-287 1.2 MB/s | 145 kB 00:00 2024-04-12T21:01:13.712 INFO:teuthology.orchestra.run.smithi104.stdout:(93/174): ceph-test-19.0.0-2879.g324baf8d.el9.x 6.1 MB/s | 49 MB 00:07 2024-04-12T21:01:13.738 INFO:teuthology.orchestra.run.smithi104.stdout:(94/174): python3-toml-0.10.2-6.el9.noarch.rpm 86 kB/s | 42 kB 00:00 2024-04-12T21:01:13.780 INFO:teuthology.orchestra.run.smithi104.stdout:(95/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 658 kB/s | 44 kB 00:00 2024-04-12T21:01:13.819 INFO:teuthology.orchestra.run.smithi081.stdout:(4/174): ceph-mds-19.0.0-2879.g324baf8d.el9.x86 4.8 MB/s | 2.4 MB 00:00 2024-04-12T21:01:13.888 INFO:teuthology.orchestra.run.smithi104.stdout:(96/174): qatlib-service-24.02.0-1.el9.x86_64.r 440 kB/s | 36 kB 00:00 2024-04-12T21:01:13.972 INFO:teuthology.orchestra.run.smithi104.stdout:(97/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 571 kB/s | 47 kB 00:00 2024-04-12T21:01:14.119 INFO:teuthology.orchestra.run.smithi081.stdout:(5/174): ceph-mgr-19.0.0-2879.g324baf8d.el9.x86 5.0 MB/s | 1.5 MB 00:00 2024-04-12T21:01:14.147 INFO:teuthology.orchestra.run.smithi104.stdout:(98/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-04-12T21:01:14.172 INFO:teuthology.orchestra.run.smithi104.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 508 kB/s | 221 kB 00:00 2024-04-12T21:01:14.197 INFO:teuthology.orchestra.run.smithi104.stdout:(100/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-04-12T21:01:14.222 INFO:teuthology.orchestra.run.smithi104.stdout:(101/174): uuid-1.6.2-55.el9.x86_64.rpm 769 kB/s | 58 kB 00:00 2024-04-12T21:01:14.248 INFO:teuthology.orchestra.run.smithi104.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-12T21:01:14.273 INFO:teuthology.orchestra.run.smithi104.stdout:(103/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 844 kB/s | 64 kB 00:00 2024-04-12T21:01:14.281 INFO:teuthology.orchestra.run.smithi153.stdout:(8/174): ceph-common-19.0.0-2879.g324baf8d.el9. 3.5 MB/s | 21 MB 00:06 2024-04-12T21:01:14.298 INFO:teuthology.orchestra.run.smithi104.stdout:(104/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 598 kB/s | 45 kB 00:00 2024-04-12T21:01:14.390 INFO:teuthology.orchestra.run.smithi153.stdout:(9/174): ceph-selinux-19.0.0-2879.g324baf8d.el9 233 kB/s | 25 kB 00:00 2024-04-12T21:01:14.407 INFO:teuthology.orchestra.run.smithi104.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-04-12T21:01:14.466 INFO:teuthology.orchestra.run.smithi104.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 3.2 MB/s | 548 kB 00:00 2024-04-12T21:01:14.499 INFO:teuthology.orchestra.run.smithi104.stdout:(107/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-04-12T21:01:14.524 INFO:teuthology.orchestra.run.smithi104.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 334 kB/s | 19 kB 00:00 2024-04-12T21:01:14.566 INFO:teuthology.orchestra.run.smithi104.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.9 MB/s | 309 kB 00:00 2024-04-12T21:01:14.592 INFO:teuthology.orchestra.run.smithi104.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-04-12T21:01:14.598 INFO:teuthology.orchestra.run.smithi153.stdout:(10/174): ceph-osd-19.0.0-2879.g324baf8d.el9.x8 4.3 MB/s | 17 MB 00:03 2024-04-12T21:01:14.617 INFO:teuthology.orchestra.run.smithi104.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-04-12T21:01:14.642 INFO:teuthology.orchestra.run.smithi104.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-12T21:01:14.667 INFO:teuthology.orchestra.run.smithi104.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-12T21:01:14.698 INFO:teuthology.orchestra.run.smithi153.stdout:(11/174): cephfs-mirror-19.0.0-2879.g324baf8d.e 2.2 MB/s | 224 kB 00:00 2024-04-12T21:01:14.703 INFO:teuthology.orchestra.run.smithi081.stdout:(6/174): ceph-mon-19.0.0-2879.g324baf8d.el9.x86 8.1 MB/s | 4.7 MB 00:00 2024-04-12T21:01:14.718 INFO:teuthology.orchestra.run.smithi104.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-04-12T21:01:14.740 INFO:teuthology.orchestra.run.smithi153.stdout:(12/174): cephfs-shell-19.0.0-2879.g324baf8d.el 652 kB/s | 27 kB 00:00 2024-04-12T21:01:14.760 INFO:teuthology.orchestra.run.smithi104.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-04-12T21:01:14.794 INFO:teuthology.orchestra.run.smithi104.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-04-12T21:01:14.824 INFO:teuthology.orchestra.run.smithi153.stdout:(13/174): libcephfs-devel-19.0.0-2879.g324baf8d 381 kB/s | 32 kB 00:00 2024-04-12T21:01:14.856 INFO:teuthology.orchestra.run.smithi104.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 335 kB/s | 32 kB 00:00 2024-04-12T21:01:14.886 INFO:teuthology.orchestra.run.smithi104.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-04-12T21:01:14.911 INFO:teuthology.orchestra.run.smithi104.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.1 MB/s | 172 kB 00:00 2024-04-12T21:01:14.978 INFO:teuthology.orchestra.run.smithi104.stdout:(120/174): libarrow-9.0.0-10.el9.x86_64.rpm 9.2 MB/s | 4.4 MB 00:00 2024-04-12T21:01:15.012 INFO:teuthology.orchestra.run.smithi104.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-12T21:01:15.028 INFO:teuthology.orchestra.run.smithi104.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 706 kB/s | 35 kB 00:00 2024-04-12T21:01:15.054 INFO:teuthology.orchestra.run.smithi104.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.1 MB/s | 303 kB 00:00 2024-04-12T21:01:15.079 INFO:teuthology.orchestra.run.smithi104.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-04-12T21:01:15.104 INFO:teuthology.orchestra.run.smithi104.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-12T21:01:15.129 INFO:teuthology.orchestra.run.smithi104.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-04-12T21:01:15.154 INFO:teuthology.orchestra.run.smithi104.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-04-12T21:01:15.180 INFO:teuthology.orchestra.run.smithi104.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-04-12T21:01:15.199 INFO:teuthology.orchestra.run.smithi153.stdout:(14/174): libcephfs2-19.0.0-2879.g324baf8d.el9. 1.9 MB/s | 746 kB 00:00 2024-04-12T21:01:15.205 INFO:teuthology.orchestra.run.smithi104.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-12T21:01:15.230 INFO:teuthology.orchestra.run.smithi104.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-04-12T21:01:15.281 INFO:teuthology.orchestra.run.smithi104.stdout:(131/174): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-04-12T21:01:15.299 INFO:teuthology.orchestra.run.smithi153.stdout:(15/174): libcephsqlite-19.0.0-2879.g324baf8d.e 1.6 MB/s | 164 kB 00:00 2024-04-12T21:01:15.306 INFO:teuthology.orchestra.run.smithi104.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 616 kB/s | 46 kB 00:00 2024-04-12T21:01:15.400 INFO:teuthology.orchestra.run.smithi153.stdout:(16/174): librados-devel-19.0.0-2879.g324baf8d. 1.2 MB/s | 127 kB 00:00 2024-04-12T21:01:15.411 INFO:teuthology.orchestra.run.smithi104.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 609 kB/s | 79 kB 00:00 2024-04-12T21:01:15.515 INFO:teuthology.orchestra.run.smithi104.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 277 kB/s | 58 kB 00:00 2024-04-12T21:01:15.557 INFO:teuthology.orchestra.run.smithi104.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-04-12T21:01:15.591 INFO:teuthology.orchestra.run.smithi104.stdout:(136/174): python3-kubernetes-26.1.0-2.el9.noar 2.7 MB/s | 1.0 MB 00:00 2024-04-12T21:01:15.615 INFO:teuthology.orchestra.run.smithi104.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-12T21:01:15.641 INFO:teuthology.orchestra.run.smithi104.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-04-12T21:01:15.650 INFO:teuthology.orchestra.run.smithi153.stdout:(17/174): libradosstriper1-19.0.0-2879.g324baf8 2.0 MB/s | 500 kB 00:00 2024-04-12T21:01:15.666 INFO:teuthology.orchestra.run.smithi104.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-12T21:01:15.691 INFO:teuthology.orchestra.run.smithi104.stdout:(140/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 669 kB/s | 90 kB 00:00 2024-04-12T21:01:15.717 INFO:teuthology.orchestra.run.smithi104.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-12T21:01:15.742 INFO:teuthology.orchestra.run.smithi104.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-04-12T21:01:15.767 INFO:teuthology.orchestra.run.smithi104.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-12T21:01:15.792 INFO:teuthology.orchestra.run.smithi104.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-04-12T21:01:15.817 INFO:teuthology.orchestra.run.smithi104.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-12T21:01:15.843 INFO:teuthology.orchestra.run.smithi104.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-12T21:01:15.877 INFO:teuthology.orchestra.run.smithi104.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 339 kB/s | 20 kB 00:00 2024-04-12T21:01:15.902 INFO:teuthology.orchestra.run.smithi104.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-12T21:01:15.936 INFO:teuthology.orchestra.run.smithi104.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-04-12T21:01:15.994 INFO:teuthology.orchestra.run.smithi104.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-12T21:01:16.153 INFO:teuthology.orchestra.run.smithi081.stdout:(7/174): ceph-base-19.0.0-2879.g324baf8d.el9.x8 1.4 MB/s | 5.4 MB 00:03 2024-04-12T21:01:16.158 INFO:teuthology.orchestra.run.smithi153.stdout:(18/174): ceph-radosgw-19.0.0-2879.g324baf8d.el 1.9 MB/s | 10 MB 00:05 2024-04-12T21:01:16.228 INFO:teuthology.orchestra.run.smithi104.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 9.9 MB/s | 2.3 MB 00:00 2024-04-12T21:01:16.267 INFO:teuthology.orchestra.run.smithi153.stdout:(19/174): python3-ceph-argparse-19.0.0-2879.g32 418 kB/s | 45 kB 00:00 2024-04-12T21:01:16.369 INFO:teuthology.orchestra.run.smithi104.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 13 MB/s | 1.8 MB 00:00 2024-04-12T21:01:16.478 INFO:teuthology.orchestra.run.smithi104.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 386 kB/s | 42 kB 00:00 2024-04-12T21:01:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:(20/174): python3-ceph-common-19.0.0-2879.g324b 599 kB/s | 135 kB 00:00 2024-04-12T21:01:16.595 INFO:teuthology.orchestra.run.smithi104.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 2.1 MB/s | 249 kB 00:00 2024-04-12T21:01:16.711 INFO:teuthology.orchestra.run.smithi081.stdout:(8/174): ceph-common-19.0.0-2879.g324baf8d.el9. 4.6 MB/s | 21 MB 00:04 2024-04-12T21:01:16.712 INFO:teuthology.orchestra.run.smithi104.stdout:(155/174): acl-2.3.1-4.el9.x86_64.rpm 612 kB/s | 71 kB 00:00 2024-04-12T21:01:16.717 INFO:teuthology.orchestra.run.smithi153.stdout:(21/174): python3-cephfs-19.0.0-2879.g324baf8d. 717 kB/s | 161 kB 00:00 2024-04-12T21:01:16.779 INFO:teuthology.orchestra.run.smithi104.stdout:(156/174): librados2-19.0.0-2879.g324baf8d.el9. 3.9 MB/s | 3.4 MB 00:00 2024-04-12T21:01:16.829 INFO:teuthology.orchestra.run.smithi081.stdout:(9/174): ceph-selinux-19.0.0-2879.g324baf8d.el9 218 kB/s | 25 kB 00:00 2024-04-12T21:01:16.846 INFO:teuthology.orchestra.run.smithi104.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 273 kB/s | 36 kB 00:00 2024-04-12T21:01:16.913 INFO:teuthology.orchestra.run.smithi104.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.4 MB/s | 192 kB 00:00 2024-04-12T21:01:16.993 INFO:teuthology.orchestra.run.smithi153.stdout:(22/174): python3-rados-19.0.0-2879.g324baf8d.e 1.1 MB/s | 320 kB 00:00 2024-04-12T21:01:17.021 INFO:teuthology.orchestra.run.smithi104.stdout:(159/174): libacl-2.3.1-4.el9.x86_64.rpm 213 kB/s | 23 kB 00:00 2024-04-12T21:01:17.122 INFO:teuthology.orchestra.run.smithi104.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 929 kB/s | 256 kB 00:00 2024-04-12T21:01:17.146 INFO:teuthology.orchestra.run.smithi081.stdout:(10/174): ceph-osd-19.0.0-2879.g324baf8d.el9.x8 6.9 MB/s | 17 MB 00:02 2024-04-12T21:01:17.147 INFO:teuthology.orchestra.run.smithi104.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 866 kB/s | 108 kB 00:00 2024-04-12T21:01:17.169 INFO:teuthology.orchestra.run.smithi153.stdout:(23/174): librgw2-19.0.0-2879.g324baf8d.el9.x86 3.4 MB/s | 5.2 MB 00:01 2024-04-12T21:01:17.227 INFO:teuthology.orchestra.run.smithi153.stdout:(24/174): python3-rgw-19.0.0-2879.g324baf8d.el9 1.7 MB/s | 99 kB 00:00 2024-04-12T21:01:17.237 INFO:teuthology.orchestra.run.smithi081.stdout:(11/174): cephfs-mirror-19.0.0-2879.g324baf8d.e 2.4 MB/s | 224 kB 00:00 2024-04-12T21:01:17.239 INFO:teuthology.orchestra.run.smithi104.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 155 kB 00:00 2024-04-12T21:01:17.260 INFO:teuthology.orchestra.run.smithi153.stdout:(25/174): python3-rbd-19.0.0-2879.g324baf8d.el9 1.1 MB/s | 297 kB 00:00 2024-04-12T21:01:17.264 INFO:teuthology.orchestra.run.smithi104.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-04-12T21:01:17.285 INFO:teuthology.orchestra.run.smithi153.stdout:(26/174): rbd-fuse-19.0.0-2879.g324baf8d.el9.x8 1.4 MB/s | 85 kB 00:00 2024-04-12T21:01:17.287 INFO:teuthology.orchestra.run.smithi081.stdout:(12/174): cephfs-shell-19.0.0-2879.g324baf8d.el 551 kB/s | 27 kB 00:00 2024-04-12T21:01:17.336 INFO:teuthology.orchestra.run.smithi153.stdout:(27/174): rbd-nbd-19.0.0-2879.g324baf8d.el9.x86 3.3 MB/s | 171 kB 00:00 2024-04-12T21:01:17.347 INFO:teuthology.orchestra.run.smithi081.stdout:(13/174): libcephfs-devel-19.0.0-2879.g324baf8d 551 kB/s | 32 kB 00:00 2024-04-12T21:01:17.356 INFO:teuthology.orchestra.run.smithi104.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 546 kB/s | 64 kB 00:00 2024-04-12T21:01:17.373 INFO:teuthology.orchestra.run.smithi104.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 264 kB/s | 29 kB 00:00 2024-04-12T21:01:17.386 INFO:teuthology.orchestra.run.smithi153.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2879.g 685 kB/s | 34 kB 00:00 2024-04-12T21:01:17.453 INFO:teuthology.orchestra.run.smithi153.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2879.g324baf8 2.1 MB/s | 142 kB 00:00 2024-04-12T21:01:17.463 INFO:teuthology.orchestra.run.smithi081.stdout:(14/174): libcephfs2-19.0.0-2879.g324baf8d.el9. 6.3 MB/s | 746 kB 00:00 2024-04-12T21:01:17.507 INFO:teuthology.orchestra.run.smithi104.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 8.1 MB/s | 1.2 MB 00:00 2024-04-12T21:01:17.522 INFO:teuthology.orchestra.run.smithi081.stdout:(15/174): libcephsqlite-19.0.0-2879.g324baf8d.e 2.8 MB/s | 164 kB 00:00 2024-04-12T21:01:17.589 INFO:teuthology.orchestra.run.smithi081.stdout:(16/174): librados-devel-19.0.0-2879.g324baf8d. 1.9 MB/s | 127 kB 00:00 2024-04-12T21:01:17.607 INFO:teuthology.orchestra.run.smithi104.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.2 MB/s | 2.2 MB 00:00 2024-04-12T21:01:17.665 INFO:teuthology.orchestra.run.smithi104.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-04-12T21:01:17.716 INFO:teuthology.orchestra.run.smithi104.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-04-12T21:01:17.739 INFO:teuthology.orchestra.run.smithi081.stdout:(17/174): libradosstriper1-19.0.0-2879.g324baf8 3.3 MB/s | 500 kB 00:00 2024-04-12T21:01:17.741 INFO:teuthology.orchestra.run.smithi104.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 414 kB/s | 31 kB 00:00 2024-04-12T21:01:17.791 INFO:teuthology.orchestra.run.smithi104.stdout:(171/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2024-04-12T21:01:17.833 INFO:teuthology.orchestra.run.smithi104.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-04-12T21:01:17.875 INFO:teuthology.orchestra.run.smithi104.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 891 kB/s | 74 kB 00:00 2024-04-12T21:01:18.434 INFO:teuthology.orchestra.run.smithi104.stdout:(174/174): librbd1-19.0.0-2879.g324baf8d.el9.x8 1.3 MB/s | 3.1 MB 00:02 2024-04-12T21:01:18.438 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-04-12T21:01:18.439 INFO:teuthology.orchestra.run.smithi104.stdout:Total 14 MB/s | 229 MB 00:16 2024-04-12T21:01:18.462 INFO:teuthology.orchestra.run.smithi153.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2879.g324ba 3.5 MB/s | 3.5 MB 00:01 2024-04-12T21:01:18.632 INFO:teuthology.orchestra.run.smithi081.stdout:(18/174): librgw2-19.0.0-2879.g324baf8d.el9.x86 5.9 MB/s | 5.2 MB 00:00 2024-04-12T21:01:18.699 INFO:teuthology.orchestra.run.smithi081.stdout:(19/174): python3-ceph-argparse-19.0.0-2879.g32 689 kB/s | 45 kB 00:00 2024-04-12T21:01:18.774 INFO:teuthology.orchestra.run.smithi081.stdout:(20/174): python3-ceph-common-19.0.0-2879.g324b 1.8 MB/s | 135 kB 00:00 2024-04-12T21:01:18.850 INFO:teuthology.orchestra.run.smithi081.stdout:(21/174): python3-cephfs-19.0.0-2879.g324baf8d. 2.1 MB/s | 161 kB 00:00 2024-04-12T21:01:18.933 INFO:teuthology.orchestra.run.smithi081.stdout:(22/174): python3-rados-19.0.0-2879.g324baf8d.e 3.8 MB/s | 320 kB 00:00 2024-04-12T21:01:18.937 INFO:teuthology.orchestra.run.smithi153.stdout:(31/174): rbd-mirror-19.0.0-2879.g324baf8d.el9. 1.8 MB/s | 3.1 MB 00:01 2024-04-12T21:01:19.000 INFO:teuthology.orchestra.run.smithi081.stdout:(23/174): python3-rbd-19.0.0-2879.g324baf8d.el9 4.4 MB/s | 297 kB 00:00 2024-04-12T21:01:19.066 INFO:teuthology.orchestra.run.smithi081.stdout:(24/174): python3-rgw-19.0.0-2879.g324baf8d.el9 1.5 MB/s | 99 kB 00:00 2024-04-12T21:01:19.095 INFO:teuthology.orchestra.run.smithi153.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2879.g32 1.5 MB/s | 249 kB 00:00 2024-04-12T21:01:19.158 INFO:teuthology.orchestra.run.smithi081.stdout:(25/174): rbd-fuse-19.0.0-2879.g324baf8d.el9.x8 938 kB/s | 85 kB 00:00 2024-04-12T21:01:19.188 INFO:teuthology.orchestra.run.smithi153.stdout:(33/174): ceph-mgr-rook-19.0.0-2879.g324baf8d.e 538 kB/s | 49 kB 00:00 2024-04-12T21:01:19.279 INFO:teuthology.orchestra.run.smithi153.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2879.g3 180 kB/s | 16 kB 00:00 2024-04-12T21:01:19.463 INFO:teuthology.orchestra.run.smithi153.stdout:(35/174): ceph-volume-19.0.0-2879.g324baf8d.el9 1.4 MB/s | 264 kB 00:00 2024-04-12T21:01:19.634 INFO:teuthology.orchestra.run.smithi081.stdout:(26/174): rbd-mirror-19.0.0-2879.g324baf8d.el9. 6.5 MB/s | 3.1 MB 00:00 2024-04-12T21:01:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:(27/174): rbd-nbd-19.0.0-2879.g324baf8d.el9.x86 2.0 MB/s | 171 kB 00:00 2024-04-12T21:01:19.784 INFO:teuthology.orchestra.run.smithi081.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2879.g 520 kB/s | 34 kB 00:00 2024-04-12T21:01:19.827 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-04-12T21:01:19.847 INFO:teuthology.orchestra.run.smithi153.stdout:(36/174): cephadm-19.0.0-2879.g324baf8d.el9.noa 1.9 MB/s | 762 kB 00:00 2024-04-12T21:01:19.910 INFO:teuthology.orchestra.run.smithi081.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2879.g324baf8 1.1 MB/s | 142 kB 00:00 2024-04-12T21:01:19.922 INFO:teuthology.orchestra.run.smithi153.stdout:(37/174): cephfs-top-19.0.0-2879.g324baf8d.el9. 243 kB/s | 18 kB 00:00 2024-04-12T21:01:20.072 INFO:teuthology.orchestra.run.smithi153.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 4.6 MB/s | 7.4 MB 00:01 2024-04-12T21:01:20.105 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-04-12T21:01:20.105 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-04-12T21:01:20.105 INFO:teuthology.orchestra.run.smithi153.stdout:(39/174): libaio-devel-0.3.112-1.el8.x86_64.rpm 96 kB/s | 18 kB 00:00 2024-04-12T21:01:20.147 INFO:teuthology.orchestra.run.smithi153.stdout:(40/174): fuse-2.9.9-15.el9.x86_64.rpm 1.1 MB/s | 80 kB 00:00 2024-04-12T21:01:20.215 INFO:teuthology.orchestra.run.smithi153.stdout:(41/174): jq-1.6-16.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-04-12T21:01:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:(42/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-04-12T21:01:20.372 INFO:teuthology.orchestra.run.smithi153.stdout:(43/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-04-12T21:01:20.377 INFO:teuthology.orchestra.run.smithi081.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2879.g324ba 7.6 MB/s | 3.5 MB 00:00 2024-04-12T21:01:20.455 INFO:teuthology.orchestra.run.smithi153.stdout:(44/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 218 kB 00:00 2024-04-12T21:01:20.489 INFO:teuthology.orchestra.run.smithi153.stdout:(45/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-04-12T21:01:20.556 INFO:teuthology.orchestra.run.smithi153.stdout:(46/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-12T21:01:20.626 INFO:teuthology.orchestra.run.smithi153.stdout:(47/174): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-12T21:01:20.685 INFO:teuthology.orchestra.run.smithi081.stdout:(31/174): ceph-radosgw-19.0.0-2879.g324baf8d.el 2.2 MB/s | 10 MB 00:04 2024-04-12T21:01:20.690 INFO:teuthology.orchestra.run.smithi153.stdout:(48/174): python3-cryptography-36.0.1-4.el9.x86 5.2 MB/s | 1.2 MB 00:00 2024-04-12T21:01:20.724 INFO:teuthology.orchestra.run.smithi153.stdout:(49/174): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-04-12T21:01:20.791 INFO:teuthology.orchestra.run.smithi153.stdout:(50/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.1 MB/s | 215 kB 00:00 2024-04-12T21:01:20.886 INFO:teuthology.orchestra.run.smithi081.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2879.g32 1.2 MB/s | 249 kB 00:00 2024-04-12T21:01:20.969 INFO:teuthology.orchestra.run.smithi081.stdout:(33/174): ceph-mgr-rook-19.0.0-2879.g324baf8d.e 598 kB/s | 49 kB 00:00 2024-04-12T21:01:21.073 INFO:teuthology.orchestra.run.smithi081.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2879.g3 160 kB/s | 16 kB 00:00 2024-04-12T21:01:21.220 INFO:teuthology.orchestra.run.smithi081.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 8.8 MB/s | 7.4 MB 00:00 2024-04-12T21:01:21.253 INFO:teuthology.orchestra.run.smithi081.stdout:(36/174): ceph-volume-19.0.0-2879.g324baf8d.el9 1.4 MB/s | 264 kB 00:00 2024-04-12T21:01:21.337 INFO:teuthology.orchestra.run.smithi081.stdout:(37/174): cephfs-top-19.0.0-2879.g324baf8d.el9. 221 kB/s | 18 kB 00:00 2024-04-12T21:01:21.370 INFO:teuthology.orchestra.run.smithi081.stdout:(38/174): cephadm-19.0.0-2879.g324baf8d.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-12T21:01:21.513 INFO:teuthology.orchestra.run.smithi081.stdout:(39/174): libaio-devel-0.3.112-1.el8.x86_64.rpm 101 kB/s | 18 kB 00:00 2024-04-12T21:01:21.538 INFO:teuthology.orchestra.run.smithi081.stdout:(40/174): fuse-2.9.9-15.el9.x86_64.rpm 483 kB/s | 80 kB 00:00 2024-04-12T21:01:21.655 INFO:teuthology.orchestra.run.smithi081.stdout:(41/174): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-04-12T21:01:21.706 INFO:teuthology.orchestra.run.smithi081.stdout:(42/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 803 kB 00:00 2024-04-12T21:01:21.739 INFO:teuthology.orchestra.run.smithi081.stdout:(43/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-12T21:01:21.773 INFO:teuthology.orchestra.run.smithi081.stdout:(44/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 218 kB 00:00 2024-04-12T21:01:21.806 INFO:teuthology.orchestra.run.smithi081.stdout:(45/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-12T21:01:21.857 INFO:teuthology.orchestra.run.smithi081.stdout:(46/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-12T21:01:21.907 INFO:teuthology.orchestra.run.smithi081.stdout:(47/174): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-12T21:01:21.940 INFO:teuthology.orchestra.run.smithi081.stdout:(48/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-04-12T21:01:21.968 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-04-12T21:01:21.968 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-04-12T21:01:21.974 INFO:teuthology.orchestra.run.smithi081.stdout:(49/174): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-12T21:01:22.024 INFO:teuthology.orchestra.run.smithi081.stdout:(50/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-04-12T21:01:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:(51/174): ceph-test-19.0.0-2879.g324baf8d.el9.x 6.2 MB/s | 49 MB 00:07 2024-04-12T21:01:22.292 INFO:teuthology.orchestra.run.smithi153.stdout:(52/174): automake-1.16.2-8.el9.noarch.rpm 440 kB/s | 689 kB 00:01 2024-04-12T21:01:22.600 INFO:teuthology.orchestra.run.smithi081.stdout:(51/174): ceph-test-19.0.0-2879.g324baf8d.el9.x 8.4 MB/s | 49 MB 00:05 2024-04-12T21:01:22.708 INFO:teuthology.orchestra.run.smithi153.stdout:(53/174): bison-3.7.4-5.el9.x86_64.rpm 492 kB/s | 943 kB 00:01 2024-04-12T21:01:22.800 INFO:teuthology.orchestra.run.smithi081.stdout:(52/174): boost-program-options-1.75.0-8.el9.x8 537 kB/s | 107 kB 00:00 2024-04-12T21:01:22.892 INFO:teuthology.orchestra.run.smithi153.stdout:(54/174): elfutils-libelf-devel-0.190-2.el9.x86 36 kB/s | 21 kB 00:00 2024-04-12T21:01:22.917 INFO:teuthology.orchestra.run.smithi081.stdout:(53/174): elfutils-libelf-devel-0.190-2.el9.x86 186 kB/s | 21 kB 00:00 2024-04-12T21:01:22.971 INFO:teuthology.orchestra.run.smithi081.stdout:(54/174): automake-1.16.2-8.el9.noarch.rpm 692 kB/s | 689 kB 00:00 2024-04-12T21:01:22.984 INFO:teuthology.orchestra.run.smithi153.stdout:(55/174): boost-program-options-1.75.0-8.el9.x8 146 kB/s | 107 kB 00:00 2024-04-12T21:01:23.068 INFO:teuthology.orchestra.run.smithi081.stdout:(55/174): bison-3.7.4-5.el9.x86_64.rpm 905 kB/s | 943 kB 00:01 2024-04-12T21:01:23.085 INFO:teuthology.orchestra.run.smithi081.stdout:(56/174): flexiblas-3.0.4-8.el9.x86_64.rpm 290 kB/s | 33 kB 00:00 2024-04-12T21:01:23.210 INFO:teuthology.orchestra.run.smithi081.stdout:(57/174): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-12T21:01:23.227 INFO:teuthology.orchestra.run.smithi081.stdout:(58/174): flexiblas-openblas-openmp-3.0.4-8.el9 129 kB/s | 18 kB 00:00 2024-04-12T21:01:23.402 INFO:teuthology.orchestra.run.smithi081.stdout:(59/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 259 kB/s | 45 kB 00:00 2024-04-12T21:01:23.469 INFO:teuthology.orchestra.run.smithi081.stdout:(60/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 620 kB/s | 160 kB 00:00 2024-04-12T21:01:23.526 INFO:teuthology.orchestra.run.smithi153.stdout:(56/174): flexiblas-3.0.4-8.el9.x86_64.rpm 52 kB/s | 33 kB 00:00 2024-04-12T21:01:23.693 INFO:teuthology.orchestra.run.smithi153.stdout:(57/174): flex-2.6.4-9.el9.x86_64.rpm 319 kB/s | 313 kB 00:00 2024-04-12T21:01:23.753 INFO:teuthology.orchestra.run.smithi081.stdout:(61/174): librdkafka-1.6.1-102.el9.x86_64.rpm 1.9 MB/s | 662 kB 00:00 2024-04-12T21:01:23.787 INFO:teuthology.orchestra.run.smithi081.stdout:(62/174): libtool-2.4.6-45.el9.x86_64.rpm 1.8 MB/s | 582 kB 00:00 2024-04-12T21:01:23.845 INFO:teuthology.orchestra.run.smithi081.stdout:(63/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 279 kB/s | 25 kB 00:00 2024-04-12T21:01:23.879 INFO:teuthology.orchestra.run.smithi081.stdout:(64/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 521 kB/s | 47 kB 00:00 2024-04-12T21:01:23.988 INFO:teuthology.orchestra.run.smithi081.stdout:(65/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.3 MB/s | 3.0 MB 00:00 2024-04-12T21:01:24.069 INFO:teuthology.orchestra.run.smithi153.stdout:(58/174): flexiblas-openblas-openmp-3.0.4-8.el9 34 kB/s | 18 kB 00:00 2024-04-12T21:01:24.071 INFO:teuthology.orchestra.run.smithi081.stdout:(66/174): lua-5.4.4-4.el9.x86_64.rpm 986 kB/s | 188 kB 00:00 2024-04-12T21:01:24.096 INFO:teuthology.orchestra.run.smithi081.stdout:(67/174): openblas-0.3.21-2.el9.x86_64.rpm 327 kB/s | 35 kB 00:00 2024-04-12T21:01:24.122 INFO:teuthology.orchestra.run.smithi081.stdout:(68/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-04-12T21:01:24.560 INFO:teuthology.orchestra.run.smithi153.stdout:(59/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 185 kB/s | 160 kB 00:00 2024-04-12T21:01:24.664 INFO:teuthology.orchestra.run.smithi081.stdout:(69/174): postgresql-13.14-1.el9.x86_64.rpm 2.9 MB/s | 1.6 MB 00:00 2024-04-12T21:01:24.727 INFO:teuthology.orchestra.run.smithi153.stdout:(60/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 69 kB/s | 45 kB 00:00 2024-04-12T21:01:24.894 INFO:teuthology.orchestra.run.smithi081.stdout:(70/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 5.2 MB/s | 4.1 MB 00:00 2024-04-12T21:01:24.939 INFO:teuthology.orchestra.run.smithi081.stdout:(71/174): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-04-12T21:01:25.039 INFO:teuthology.orchestra.run.smithi081.stdout:(72/174): openblas-openmp-0.3.21-2.el9.x86_64.r 4.9 MB/s | 4.7 MB 00:00 2024-04-12T21:01:25.081 INFO:teuthology.orchestra.run.smithi081.stdout:(73/174): postgresql-private-libs-13.14-1.el9.x 714 kB/s | 133 kB 00:00 2024-04-12T21:01:25.174 INFO:teuthology.orchestra.run.smithi081.stdout:(74/174): python3-attrs-20.3.0-7.el9.noarch.rpm 964 kB/s | 87 kB 00:00 2024-04-12T21:01:25.240 INFO:teuthology.orchestra.run.smithi081.stdout:(75/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-04-12T21:01:25.382 INFO:teuthology.orchestra.run.smithi081.stdout:(76/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-04-12T21:01:25.716 INFO:teuthology.orchestra.run.smithi081.stdout:(77/174): postgresql-server-13.14-1.el9.x86_64. 7.5 MB/s | 5.8 MB 00:00 2024-04-12T21:01:25.750 INFO:teuthology.orchestra.run.smithi081.stdout:(78/174): python3-jinja2-2.11.3-5.el9.noarch.rp 677 kB/s | 248 kB 00:00 2024-04-12T21:01:25.800 INFO:teuthology.orchestra.run.smithi081.stdout:(79/174): python3-jsonpatch-1.21-16.el9.noarch. 313 kB/s | 26 kB 00:00 2024-04-12T21:01:25.825 INFO:teuthology.orchestra.run.smithi081.stdout:(80/174): python3-jsonpointer-2.0-4.el9.noarch. 258 kB/s | 19 kB 00:00 2024-04-12T21:01:25.900 INFO:teuthology.orchestra.run.smithi081.stdout:(81/174): python3-markupsafe-1.1.1-12.el9.x86_6 468 kB/s | 35 kB 00:00 2024-04-12T21:01:25.934 INFO:teuthology.orchestra.run.smithi081.stdout:(82/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-04-12T21:01:26.051 INFO:teuthology.orchestra.run.smithi081.stdout:(83/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-04-12T21:01:26.094 INFO:teuthology.orchestra.run.smithi153.stdout:(61/174): librdkafka-1.6.1-102.el9.x86_64.rpm 432 kB/s | 662 kB 00:01 2024-04-12T21:01:26.168 INFO:teuthology.orchestra.run.smithi081.stdout:(84/174): python3-babel-2.9.1-2.el9.noarch.rpm 6.0 MB/s | 6.0 MB 00:00 2024-04-12T21:01:26.168 INFO:teuthology.orchestra.run.smithi153.stdout:(62/174): libtool-2.4.6-45.el9.x86_64.rpm 404 kB/s | 582 kB 00:01 2024-04-12T21:01:26.235 INFO:teuthology.orchestra.run.smithi081.stdout:(85/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-04-12T21:01:26.260 INFO:teuthology.orchestra.run.smithi081.stdout:(86/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 222 kB 00:00 2024-04-12T21:01:26.318 INFO:teuthology.orchestra.run.smithi081.stdout:(87/174): python3-prettytable-0.7.2-27.el9.noar 511 kB/s | 42 kB 00:00 2024-04-12T21:01:26.386 INFO:teuthology.orchestra.run.smithi081.stdout:(88/174): python3-protobuf-3.14.0-13.el9.noarch 2.1 MB/s | 269 kB 00:00 2024-04-12T21:01:26.436 INFO:teuthology.orchestra.run.smithi081.stdout:(89/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-12T21:01:26.611 INFO:teuthology.orchestra.run.smithi081.stdout:(90/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.2 MB/s | 5.1 MB 00:00 2024-04-12T21:01:26.628 INFO:teuthology.orchestra.run.smithi081.stdout:(91/174): python3-pytz-2021.1-5.el9.noarch.rpm 267 kB/s | 51 kB 00:00 2024-04-12T21:01:26.661 INFO:teuthology.orchestra.run.smithi081.stdout:(92/174): python3-pyasn1-modules-0.4.8-6.el9.no 1.0 MB/s | 279 kB 00:00 2024-04-12T21:01:26.711 INFO:teuthology.orchestra.run.smithi153.stdout:(63/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 41 kB/s | 25 kB 00:00 2024-04-12T21:01:26.712 INFO:teuthology.orchestra.run.smithi081.stdout:(93/174): python3-requests-oauthlib-1.3.0-12.el 539 kB/s | 54 kB 00:00 2024-04-12T21:01:26.745 INFO:teuthology.orchestra.run.smithi081.stdout:(94/174): python3-toml-0.10.2-6.el9.noarch.rpm 501 kB/s | 42 kB 00:00 2024-04-12T21:01:26.771 INFO:teuthology.orchestra.run.smithi081.stdout:(95/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 768 kB/s | 44 kB 00:00 2024-04-12T21:01:26.829 INFO:teuthology.orchestra.run.smithi081.stdout:(96/174): qatlib-service-24.02.0-1.el9.x86_64.r 625 kB/s | 36 kB 00:00 2024-04-12T21:01:26.836 INFO:teuthology.orchestra.run.smithi153.stdout:(64/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 71 kB/s | 47 kB 00:00 2024-04-12T21:01:26.863 INFO:teuthology.orchestra.run.smithi081.stdout:(97/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-04-12T21:01:26.922 INFO:teuthology.orchestra.run.smithi081.stdout:(98/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 519 kB/s | 47 kB 00:00 2024-04-12T21:01:26.980 INFO:teuthology.orchestra.run.smithi081.stdout:(99/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.6 MB/s | 305 kB 00:00 2024-04-12T21:01:27.014 INFO:teuthology.orchestra.run.smithi081.stdout:(100/174): uuid-1.6.2-55.el9.x86_64.rpm 636 kB/s | 58 kB 00:00 2024-04-12T21:01:27.098 INFO:teuthology.orchestra.run.smithi081.stdout:(101/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 807 kB/s | 94 kB 00:00 2024-04-12T21:01:27.123 INFO:teuthology.orchestra.run.smithi081.stdout:(102/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 592 kB/s | 64 kB 00:00 2024-04-12T21:01:27.182 INFO:teuthology.orchestra.run.smithi081.stdout:(103/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 544 kB/s | 45 kB 00:00 2024-04-12T21:01:27.353 INFO:teuthology.orchestra.run.smithi153.stdout:(65/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 701 kB/s | 3.0 MB 00:04 2024-04-12T21:01:27.373 INFO:teuthology.orchestra.run.smithi081.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 89 kB/s | 22 kB 00:00 2024-04-12T21:01:27.624 INFO:teuthology.orchestra.run.smithi081.stdout:(105/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-04-12T21:01:27.674 INFO:teuthology.orchestra.run.smithi081.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.8 MB/s | 548 kB 00:00 2024-04-12T21:01:27.703 INFO:teuthology.orchestra.run.smithi153.stdout:(66/174): lua-5.4.4-4.el9.x86_64.rpm 217 kB/s | 188 kB 00:00 2024-04-12T21:01:27.728 INFO:teuthology.orchestra.run.smithi153.stdout:(67/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 288 kB/s | 292 kB 00:01 2024-04-12T21:01:27.808 INFO:teuthology.orchestra.run.smithi081.stdout:(107/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.3 MB/s | 863 kB 00:00 2024-04-12T21:01:27.833 INFO:teuthology.orchestra.run.smithi081.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 213 kB/s | 19 kB 00:00 2024-04-12T21:01:27.883 INFO:teuthology.orchestra.run.smithi081.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 523 kB/s | 26 kB 00:00 2024-04-12T21:01:27.934 INFO:teuthology.orchestra.run.smithi081.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 988 kB/s | 49 kB 00:00 2024-04-12T21:01:27.967 INFO:teuthology.orchestra.run.smithi081.stdout:(111/174): gperftools-libs-2.9.1-2.el9.x86_64.r 903 kB/s | 309 kB 00:00 2024-04-12T21:01:27.993 INFO:teuthology.orchestra.run.smithi081.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-04-12T21:01:27.995 INFO:teuthology.orchestra.run.smithi153.stdout:(68/174): openblas-0.3.21-2.el9.x86_64.rpm 55 kB/s | 35 kB 00:00 2024-04-12T21:01:28.026 INFO:teuthology.orchestra.run.smithi081.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-04-12T21:01:28.101 INFO:teuthology.orchestra.run.smithi081.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-04-12T21:01:28.143 INFO:teuthology.orchestra.run.smithi081.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-12T21:01:28.169 INFO:teuthology.orchestra.run.smithi081.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 650 kB/s | 43 kB 00:00 2024-04-12T21:01:28.194 INFO:teuthology.orchestra.run.smithi081.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 651 kB/s | 32 kB 00:00 2024-04-12T21:01:28.219 INFO:teuthology.orchestra.run.smithi081.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 303 kB/s | 15 kB 00:00 2024-04-12T21:01:28.253 INFO:teuthology.orchestra.run.smithi081.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-12T21:01:28.295 INFO:teuthology.orchestra.run.smithi081.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-04-12T21:01:28.337 INFO:teuthology.orchestra.run.smithi081.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-04-12T21:01:28.362 INFO:teuthology.orchestra.run.smithi081.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 534 kB/s | 35 kB 00:00 2024-04-12T21:01:28.387 INFO:teuthology.orchestra.run.smithi081.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-04-12T21:01:28.488 INFO:teuthology.orchestra.run.smithi081.stdout:(124/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2024-04-12T21:01:28.513 INFO:teuthology.orchestra.run.smithi081.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-12T21:01:28.538 INFO:teuthology.orchestra.run.smithi081.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 215 kB/s | 11 kB 00:00 2024-04-12T21:01:28.564 INFO:teuthology.orchestra.run.smithi081.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 355 kB/s | 18 kB 00:00 2024-04-12T21:01:28.589 INFO:teuthology.orchestra.run.smithi081.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 464 kB/s | 23 kB 00:00 2024-04-12T21:01:28.606 INFO:teuthology.orchestra.run.smithi081.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 467 kB/s | 19 kB 00:00 2024-04-12T21:01:28.632 INFO:teuthology.orchestra.run.smithi081.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 479 kB/s | 20 kB 00:00 2024-04-12T21:01:28.674 INFO:teuthology.orchestra.run.smithi081.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-04-12T21:01:28.758 INFO:teuthology.orchestra.run.smithi081.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-04-12T21:01:28.783 INFO:teuthology.orchestra.run.smithi081.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 729 kB/s | 79 kB 00:00 2024-04-12T21:01:28.800 INFO:teuthology.orchestra.run.smithi081.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-04-12T21:01:28.875 INFO:teuthology.orchestra.run.smithi081.stdout:(135/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-04-12T21:01:28.892 INFO:teuthology.orchestra.run.smithi081.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-12T21:01:28.907 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-04-12T21:01:28.926 INFO:teuthology.orchestra.run.smithi081.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-12T21:01:28.959 INFO:teuthology.orchestra.run.smithi081.stdout:(138/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-04-12T21:01:28.985 INFO:teuthology.orchestra.run.smithi081.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 308 kB/s | 28 kB 00:00 2024-04-12T21:01:29.003 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-12T21:01:29.010 INFO:teuthology.orchestra.run.smithi081.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 370 kB/s | 31 kB 00:00 2024-04-12T21:01:29.029 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-12T21:01:29.035 INFO:teuthology.orchestra.run.smithi081.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-12T21:01:29.060 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-12T21:01:29.060 INFO:teuthology.orchestra.run.smithi081.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 723 kB/s | 36 kB 00:00 2024-04-12T21:01:29.070 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-12T21:01:29.086 INFO:teuthology.orchestra.run.smithi081.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-12T21:01:29.111 INFO:teuthology.orchestra.run.smithi081.stdout:(144/174): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-12T21:01:29.118 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-12T21:01:29.145 INFO:teuthology.orchestra.run.smithi081.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-12T21:01:29.170 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-12T21:01:29.178 INFO:teuthology.orchestra.run.smithi081.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-04-12T21:01:29.204 INFO:teuthology.orchestra.run.smithi081.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2024-04-12T21:01:29.205 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-12T21:01:29.237 INFO:teuthology.orchestra.run.smithi081.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-12T21:01:29.239 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-12T21:01:29.271 INFO:teuthology.orchestra.run.smithi081.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-04-12T21:01:29.281 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-12T21:01:29.355 INFO:teuthology.orchestra.run.smithi081.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-12T21:01:29.563 INFO:teuthology.orchestra.run.smithi081.stdout:(151/174): librados2-19.0.0-2879.g324baf8d.el9. 10 MB/s | 3.4 MB 00:00 2024-04-12T21:01:29.600 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-12T21:01:29.638 INFO:teuthology.orchestra.run.smithi081.stdout:(152/174): NetworkManager-1.47.4-1.el9.x86_64.r 8.2 MB/s | 2.3 MB 00:00 2024-04-12T21:01:29.656 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:01:29.706 INFO:teuthology.orchestra.run.smithi081.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 635 kB/s | 42 kB 00:00 2024-04-12T21:01:29.756 INFO:teuthology.orchestra.run.smithi081.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 5.0 MB/s | 249 kB 00:00 2024-04-12T21:01:29.779 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:01:29.816 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:01:29.831 INFO:teuthology.orchestra.run.smithi081.stdout:(155/174): NetworkManager-libnm-1.47.4-1.el9.x8 6.9 MB/s | 1.8 MB 00:00 2024-04-12T21:01:29.848 INFO:teuthology.orchestra.run.smithi081.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 782 kB/s | 71 kB 00:00 2024-04-12T21:01:29.882 INFO:teuthology.orchestra.run.smithi081.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 738 kB/s | 36 kB 00:00 2024-04-12T21:01:29.907 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:01:29.915 INFO:teuthology.orchestra.run.smithi081.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-04-12T21:01:29.937 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 12/198 2024-04-12T21:01:29.940 INFO:teuthology.orchestra.run.smithi081.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-04-12T21:01:29.966 INFO:teuthology.orchestra.run.smithi081.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 466 kB/s | 23 kB 00:00 2024-04-12T21:01:29.975 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-12T21:01:30.016 INFO:teuthology.orchestra.run.smithi081.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-04-12T21:01:30.041 INFO:teuthology.orchestra.run.smithi081.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-04-12T21:01:30.075 INFO:teuthology.orchestra.run.smithi081.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-04-12T21:01:30.101 INFO:teuthology.orchestra.run.smithi081.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-04-12T21:01:30.117 INFO:teuthology.orchestra.run.smithi081.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 693 kB/s | 29 kB 00:00 2024-04-12T21:01:30.152 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-12T21:01:30.185 INFO:teuthology.orchestra.run.smithi081.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 15 MB/s | 1.2 MB 00:00 2024-04-12T21:01:30.201 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-12T21:01:30.230 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:01:30.277 INFO:teuthology.orchestra.run.smithi081.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-04-12T21:01:30.334 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:01:30.352 INFO:teuthology.orchestra.run.smithi081.stdout:(168/174): librbd1-19.0.0-2879.g324baf8d.el9.x8 2.9 MB/s | 3.1 MB 00:01 2024-04-12T21:01:30.355 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 17/198 2024-04-12T21:01:30.386 INFO:teuthology.orchestra.run.smithi081.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-04-12T21:01:30.419 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 17/198 2024-04-12T21:01:30.460 INFO:teuthology.orchestra.run.smithi081.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.3 MB/s | 2.3 MB 00:00 2024-04-12T21:01:30.467 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-12T21:01:30.477 INFO:teuthology.orchestra.run.smithi081.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 248 kB/s | 31 kB 00:00 2024-04-12T21:01:30.490 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-12T21:01:30.503 INFO:teuthology.orchestra.run.smithi081.stdout:(172/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.7 MB/s | 323 kB 00:00 2024-04-12T21:01:30.506 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-12T21:01:30.561 INFO:teuthology.orchestra.run.smithi081.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-04-12T21:01:30.587 INFO:teuthology.orchestra.run.smithi081.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 689 kB/s | 74 kB 00:00 2024-04-12T21:01:30.601 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-04-12T21:01:30.601 INFO:teuthology.orchestra.run.smithi081.stdout:Total 11 MB/s | 229 MB 00:21 2024-04-12T21:01:30.653 INFO:teuthology.orchestra.run.smithi153.stdout:(69/174): postgresql-13.14-1.el9.x86_64.rpm 612 kB/s | 1.6 MB 00:02 2024-04-12T21:01:30.691 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-12T21:01:30.723 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-12T21:01:30.763 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-12T21:01:30.868 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-12T21:01:30.957 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-12T21:01:31.017 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-12T21:01:31.086 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:01:31.142 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:01:31.347 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:01:31.456 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-12T21:01:31.558 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-12T21:01:31.597 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-12T21:01:31.632 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-12T21:01:31.695 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/198 2024-04-12T21:01:31.731 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/198 2024-04-12T21:01:31.774 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/198 2024-04-12T21:01:31.807 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 32/198 2024-04-12T21:01:31.835 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-5.4.4-4.el9.x86_64 33/198 2024-04-12T21:01:31.903 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 34/198 2024-04-12T21:01:31.956 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 35/198 2024-04-12T21:01:32.004 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 36/198 2024-04-12T21:01:32.039 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 37/198 2024-04-12T21:01:32.115 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 38/198 2024-04-12T21:01:32.150 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:19.0.0-2879.g324baf8d.el9. 39/198 2024-04-12T21:01:32.195 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el 40/198 2024-04-12T21:01:32.239 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_6 41/198 2024-04-12T21:01:32.408 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-12T21:01:32.428 INFO:teuthology.orchestra.run.smithi153.stdout:(70/174): postgresql-contrib-13.14-1.el9.x86_64 513 kB/s | 911 kB 00:01 2024-04-12T21:01:32.454 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-12T21:01:32.527 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-12T21:01:32.661 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-12T21:01:32.722 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-12T21:01:33.195 INFO:teuthology.orchestra.run.smithi153.stdout:(71/174): postgresql-private-libs-13.14-1.el9.x 174 kB/s | 133 kB 00:00 2024-04-12T21:01:33.258 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-12T21:01:33.351 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-12T21:01:33.429 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-12T21:01:33.471 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-12T21:01:33.481 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-12T21:01:33.511 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-12T21:01:33.529 INFO:teuthology.orchestra.run.smithi153.stdout:(72/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 729 kB/s | 4.1 MB 00:05 2024-04-12T21:01:33.551 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-12T21:01:33.589 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_6 53/198 2024-04-12T21:01:33.619 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-12T21:01:33.842 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-12T21:01:33.988 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-12T21:01:34.054 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-12T21:01:34.098 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-12T21:01:34.221 INFO:teuthology.orchestra.run.smithi153.stdout:(73/174): openblas-openmp-0.3.21-2.el9.x86_64.r 744 kB/s | 4.7 MB 00:06 2024-04-12T21:01:34.425 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-04-12T21:01:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-12T21:01:34.426 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-12T21:01:34.460 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-12T21:01:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-04-12T21:01:34.670 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-04-12T21:01:34.674 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-12T21:01:34.725 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-12T21:01:34.761 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-12T21:01:34.808 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-12T21:01:34.844 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-12T21:01:34.895 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-12T21:01:34.937 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-12T21:01:35.021 INFO:teuthology.orchestra.run.smithi153.stdout:(74/174): python3-attrs-20.3.0-7.el9.noarch.rpm 109 kB/s | 87 kB 00:00 2024-04-12T21:01:35.069 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-12T21:01:35.117 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-12T21:01:35.139 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-12T21:01:35.181 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-12T21:01:35.274 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-12T21:01:35.314 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-12T21:01:35.354 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-12T21:01:35.380 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-12T21:01:35.404 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-12T21:01:35.476 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-12T21:01:35.588 INFO:teuthology.orchestra.run.smithi153.stdout:(75/174): protobuf-3.14.0-13.el9.x86_64.rpm 502 kB/s | 1.0 MB 00:02 2024-04-12T21:01:35.595 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-12T21:01:35.636 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-12T21:01:35.664 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-04-12T21:01:35.698 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 82/198 2024-04-12T21:01:35.737 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : uuid-1.6.2-55.el9.x86_64 83/198 2024-04-12T21:01:35.769 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 84/198 2024-04-12T21:01:35.799 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 85/198 2024-04-12T21:01:35.842 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 86/198 2024-04-12T21:01:35.875 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 87/198 2024-04-12T21:01:35.925 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 88/198 2024-04-12T21:01:35.976 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 89/198 2024-04-12T21:01:36.062 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 90/198 2024-04-12T21:01:36.114 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 91/198 2024-04-12T21:01:36.146 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 92/198 2024-04-12T21:01:36.193 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 93/198 2024-04-12T21:01:36.290 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-packaging-20.9-5.el9.noarch 94/198 2024-04-12T21:01:36.343 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 95/198 2024-04-12T21:01:36.496 INFO:teuthology.orchestra.run.smithi153.stdout:(76/174): python3-devel-3.9.17-2.el9.x86_64.rpm 270 kB/s | 245 kB 00:00 2024-04-12T21:01:37.060 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 96/198 2024-04-12T21:01:37.103 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 97/198 2024-04-12T21:01:37.136 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 98/198 2024-04-12T21:01:37.186 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-04-12T21:01:37.187 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-04-12T21:01:37.235 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 99/198 2024-04-12T21:01:37.285 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 100/198 2024-04-12T21:01:37.365 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 101/198 2024-04-12T21:01:37.396 INFO:teuthology.orchestra.run.smithi153.stdout:(77/174): python3-jinja2-2.11.3-5.el9.noarch.rp 276 kB/s | 248 kB 00:00 2024-04-12T21:01:37.413 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 102/198 2024-04-12T21:01:37.588 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.21-2.el9.x86_64 103/198 2024-04-12T21:01:37.627 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 104/198 2024-04-12T21:01:37.704 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 105/198 2024-04-12T21:01:37.997 INFO:teuthology.orchestra.run.smithi153.stdout:(78/174): python3-jsonpatch-1.21-16.el9.noarch. 43 kB/s | 26 kB 00:00 2024-04-12T21:01:38.342 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 106/198 2024-04-12T21:01:38.458 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 107/198 2024-04-12T21:01:38.522 INFO:teuthology.orchestra.run.smithi153.stdout:(79/174): python3-jsonpointer-2.0-4.el9.noarch. 36 kB/s | 19 kB 00:00 2024-04-12T21:01:39.339 INFO:teuthology.orchestra.run.smithi153.stdout:(80/174): python3-mako-1.1.4-6.el9.noarch.rpm 211 kB/s | 172 kB 00:00 2024-04-12T21:01:39.673 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 108/198 2024-04-12T21:01:39.746 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 109/198 2024-04-12T21:01:39.773 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 110/198 2024-04-12T21:01:39.939 INFO:teuthology.orchestra.run.smithi153.stdout:(81/174): python3-markupsafe-1.1.1-12.el9.x86_6 58 kB/s | 35 kB 00:00 2024-04-12T21:01:40.075 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 111/198 2024-04-12T21:01:40.103 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 112/198 2024-04-12T21:01:40.194 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 112/198 2024-04-12T21:01:40.225 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 113/198 2024-04-12T21:01:40.260 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 114/198 2024-04-12T21:01:40.731 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-12T21:01:40.764 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 116/198 2024-04-12T21:01:40.835 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 116/198 2024-04-12T21:01:40.857 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 117/198 2024-04-12T21:01:41.306 INFO:teuthology.orchestra.run.smithi153.stdout:(82/174): postgresql-server-13.14-1.el9.x86_64. 730 kB/s | 5.8 MB 00:08 2024-04-12T21:01:42.256 INFO:teuthology.orchestra.run.smithi153.stdout:(83/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 264 kB/s | 250 kB 00:00 2024-04-12T21:01:43.070 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:01:43.103 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:01:43.114 INFO:teuthology.orchestra.run.smithi153.stdout:(84/174): python3-babel-2.9.1-2.el9.noarch.rpm 754 kB/s | 6.0 MB 00:08 2024-04-12T21:01:43.173 INFO:teuthology.orchestra.run.smithi153.stdout:(85/174): python3-oauthlib-3.1.1-5.el9.noarch.r 242 kB/s | 222 kB 00:00 2024-04-12T21:01:43.639 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:01:43.673 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 119/198 2024-04-12T21:01:43.746 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 119/198 2024-04-12T21:01:43.746 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-12T21:01:43.746 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-12T21:01:43.746 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:01:43.774 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 120/198 2024-04-12T21:01:43.790 INFO:teuthology.orchestra.run.smithi153.stdout:(86/174): python3-prettytable-0.7.2-27.el9.noar 69 kB/s | 42 kB 00:00 2024-04-12T21:01:43.815 INFO:teuthology.orchestra.run.smithi153.stdout:(87/174): python3-packaging-20.9-5.el9.noarch.r 110 kB/s | 77 kB 00:00 2024-04-12T21:01:44.615 INFO:teuthology.orchestra.run.smithi153.stdout:(88/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 198 kB/s | 159 kB 00:00 2024-04-12T21:01:44.699 INFO:teuthology.orchestra.run.smithi153.stdout:(89/174): python3-protobuf-3.14.0-13.el9.noarch 297 kB/s | 269 kB 00:00 2024-04-12T21:01:44.723 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-04-12T21:01:44.813 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-12T21:01:44.862 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-12T21:01:44.909 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-12T21:01:44.954 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-12T21:01:45.049 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-12T21:01:45.163 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-12T21:01:45.219 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-12T21:01:45.249 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-12T21:01:45.341 INFO:teuthology.orchestra.run.smithi153.stdout:(90/174): python3-pytz-2021.1-5.el9.noarch.rpm 79 kB/s | 51 kB 00:00 2024-04-12T21:01:45.350 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-12T21:01:45.558 INFO:teuthology.orchestra.run.smithi153.stdout:(91/174): python3-pyasn1-modules-0.4.8-6.el9.no 296 kB/s | 279 kB 00:00 2024-04-12T21:01:45.749 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-12T21:01:45.797 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:01:45.951 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:01:45.974 INFO:teuthology.orchestra.run.smithi153.stdout:(92/174): python3-requests-oauthlib-1.3.0-12.el 85 kB/s | 54 kB 00:00 2024-04-12T21:01:45.981 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:01:46.109 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:01:46.142 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 12/198 2024-04-12T21:01:46.207 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-12T21:01:46.502 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-12T21:01:46.563 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-12T21:01:46.589 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:01:46.600 INFO:teuthology.orchestra.run.smithi153.stdout:(93/174): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-04-12T21:01:46.749 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:01:46.782 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 17/198 2024-04-12T21:01:46.882 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 17/198 2024-04-12T21:01:46.934 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-12T21:01:46.979 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-12T21:01:46.993 INFO:teuthology.orchestra.run.smithi153.stdout:(94/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 740 kB/s | 5.1 MB 00:07 2024-04-12T21:01:47.041 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-12T21:01:47.302 INFO:teuthology.orchestra.run.smithi153.stdout:(95/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 63 kB/s | 44 kB 00:00 2024-04-12T21:01:47.333 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-12T21:01:47.384 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-12T21:01:47.426 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-12T21:01:47.765 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-12T21:01:47.944 INFO:teuthology.orchestra.run.smithi153.stdout:(96/174): qatlib-service-24.02.0-1.el9.x86_64.r 56 kB/s | 36 kB 00:00 2024-04-12T21:01:47.969 INFO:teuthology.orchestra.run.smithi153.stdout:(97/174): qatlib-24.02.0-1.el9.x86_64.rpm 227 kB/s | 221 kB 00:00 2024-04-12T21:01:47.982 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-12T21:01:48.071 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-12T21:01:48.285 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:01:48.335 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:01:48.667 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:01:48.736 INFO:teuthology.orchestra.run.smithi153.stdout:(98/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 60 kB/s | 47 kB 00:00 2024-04-12T21:01:48.860 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-12T21:01:48.997 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-12T21:01:49.041 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-12T21:01:49.078 INFO:teuthology.orchestra.run.smithi153.stdout:(99/174): socat-1.7.4.1-5.el9.x86_64.rpm 275 kB/s | 305 kB 00:01 2024-04-12T21:01:49.091 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-12T21:01:49.195 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/198 2024-04-12T21:01:49.245 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/198 2024-04-12T21:01:49.309 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/198 2024-04-12T21:01:49.358 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 32/198 2024-04-12T21:01:49.398 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 33/198 2024-04-12T21:01:49.411 INFO:teuthology.orchestra.run.smithi153.stdout:(100/174): uuid-1.6.2-55.el9.x86_64.rpm 86 kB/s | 58 kB 00:00 2024-04-12T21:01:49.514 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 34/198 2024-04-12T21:01:49.581 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 35/198 2024-04-12T21:01:49.617 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 36/198 2024-04-12T21:01:49.686 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 37/198 2024-04-12T21:01:49.794 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 38/198 2024-04-12T21:01:49.803 INFO:teuthology.orchestra.run.smithi153.stdout:(101/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 129 kB/s | 94 kB 00:00 2024-04-12T21:01:49.837 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.0.0-2879.g324baf8d.el9. 39/198 2024-04-12T21:01:49.887 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el 40/198 2024-04-12T21:01:49.935 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_6 41/198 2024-04-12T21:01:50.057 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-12T21:01:50.095 INFO:teuthology.orchestra.run.smithi153.stdout:(102/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 93 kB/s | 64 kB 00:00 2024-04-12T21:01:50.116 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-12T21:01:50.162 INFO:teuthology.orchestra.run.smithi153.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 336 kB/s | 22 kB 00:00 2024-04-12T21:01:50.223 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-12T21:01:50.271 INFO:teuthology.orchestra.run.smithi153.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.8 MB/s | 863 kB 00:00 2024-04-12T21:01:50.402 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-12T21:01:50.430 INFO:teuthology.orchestra.run.smithi153.stdout:(105/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-04-12T21:01:50.467 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-12T21:01:50.497 INFO:teuthology.orchestra.run.smithi153.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.4 MB/s | 548 kB 00:00 2024-04-12T21:01:50.547 INFO:teuthology.orchestra.run.smithi153.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-04-12T21:01:50.614 INFO:teuthology.orchestra.run.smithi153.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.6 MB/s | 309 kB 00:00 2024-04-12T21:01:50.664 INFO:teuthology.orchestra.run.smithi153.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 520 kB/s | 26 kB 00:00 2024-04-12T21:01:50.714 INFO:teuthology.orchestra.run.smithi153.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-04-12T21:01:50.765 INFO:teuthology.orchestra.run.smithi153.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-12T21:01:50.814 INFO:teuthology.orchestra.run.smithi153.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-04-12T21:01:50.898 INFO:teuthology.orchestra.run.smithi153.stdout:(113/174): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-12T21:01:50.932 INFO:teuthology.orchestra.run.smithi153.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-12T21:01:50.974 INFO:teuthology.orchestra.run.smithi153.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-04-12T21:01:50.999 INFO:teuthology.orchestra.run.smithi153.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 5.3 MB/s | 548 kB 00:00 2024-04-12T21:01:51.024 INFO:teuthology.orchestra.run.smithi153.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-04-12T21:01:51.050 INFO:teuthology.orchestra.run.smithi153.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-12T21:01:51.075 INFO:teuthology.orchestra.run.smithi153.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-04-12T21:01:51.117 INFO:teuthology.orchestra.run.smithi153.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-04-12T21:01:51.142 INFO:teuthology.orchestra.run.smithi153.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-04-12T21:01:51.159 INFO:teuthology.orchestra.run.smithi153.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 844 kB/s | 35 kB 00:00 2024-04-12T21:01:51.166 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-12T21:01:51.193 INFO:teuthology.orchestra.run.smithi153.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-04-12T21:01:51.235 INFO:teuthology.orchestra.run.smithi153.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-04-12T21:01:51.274 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-12T21:01:51.276 INFO:teuthology.orchestra.run.smithi153.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-04-12T21:01:51.319 INFO:teuthology.orchestra.run.smithi153.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-04-12T21:01:51.377 INFO:teuthology.orchestra.run.smithi153.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.4 MB/s | 2.0 MB 00:00 2024-04-12T21:01:51.378 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-12T21:01:51.402 INFO:teuthology.orchestra.run.smithi153.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 277 kB/s | 23 kB 00:00 2024-04-12T21:01:51.428 INFO:teuthology.orchestra.run.smithi153.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-12T21:01:51.448 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-12T21:01:51.453 INFO:teuthology.orchestra.run.smithi153.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-12T21:01:51.476 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-12T21:01:51.506 INFO:teuthology.orchestra.run.smithi153.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 928 kB/s | 46 kB 00:00 2024-04-12T21:01:51.521 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-12T21:01:51.539 INFO:teuthology.orchestra.run.smithi153.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 9.2 MB/s | 1.0 MB 00:00 2024-04-12T21:01:51.564 INFO:teuthology.orchestra.run.smithi153.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-12T21:01:51.580 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-12T21:01:51.589 INFO:teuthology.orchestra.run.smithi153.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-12T21:01:51.615 INFO:teuthology.orchestra.run.smithi153.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-12T21:01:51.631 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_6 53/198 2024-04-12T21:01:51.640 INFO:teuthology.orchestra.run.smithi153.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-04-12T21:01:51.665 INFO:teuthology.orchestra.run.smithi153.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-12T21:01:51.672 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-12T21:01:51.691 INFO:teuthology.orchestra.run.smithi153.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-04-12T21:01:51.716 INFO:teuthology.orchestra.run.smithi153.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-04-12T21:01:51.741 INFO:teuthology.orchestra.run.smithi153.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-12T21:01:51.766 INFO:teuthology.orchestra.run.smithi153.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-12T21:01:51.792 INFO:teuthology.orchestra.run.smithi153.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-04-12T21:01:51.817 INFO:teuthology.orchestra.run.smithi153.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-12T21:01:51.842 INFO:teuthology.orchestra.run.smithi153.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-12T21:01:51.868 INFO:teuthology.orchestra.run.smithi153.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-12T21:01:51.910 INFO:teuthology.orchestra.run.smithi153.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-04-12T21:01:51.935 INFO:teuthology.orchestra.run.smithi153.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-04-12T21:01:51.960 INFO:teuthology.orchestra.run.smithi153.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-04-12T21:01:52.072 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-12T21:01:52.077 INFO:teuthology.orchestra.run.smithi153.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-12T21:01:52.248 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-12T21:01:52.322 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-12T21:01:52.376 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-12T21:01:52.573 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-12T21:01:52.633 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-12T21:01:52.727 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-12T21:01:52.820 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-12T21:01:52.866 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-12T21:01:52.906 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-12T21:01:52.972 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-12T21:01:53.013 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-12T21:01:53.076 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-12T21:01:53.134 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-12T21:01:53.285 INFO:teuthology.orchestra.run.smithi153.stdout:(150/174): librbd1-19.0.0-2879.g324baf8d.el9.x8 2.6 MB/s | 3.1 MB 00:01 2024-04-12T21:01:53.323 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-12T21:01:53.378 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-12T21:01:53.416 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-12T21:01:53.436 INFO:teuthology.orchestra.run.smithi153.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 15 MB/s | 2.3 MB 00:00 2024-04-12T21:01:53.465 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-12T21:01:53.569 INFO:teuthology.orchestra.run.smithi153.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 14 MB/s | 1.8 MB 00:00 2024-04-12T21:01:53.592 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-12T21:01:53.619 INFO:teuthology.orchestra.run.smithi153.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 836 kB/s | 42 kB 00:00 2024-04-12T21:01:53.631 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-12T21:01:53.675 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-12T21:01:53.686 INFO:teuthology.orchestra.run.smithi153.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 3.7 MB/s | 249 kB 00:00 2024-04-12T21:01:53.705 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-12T21:01:53.745 INFO:teuthology.orchestra.run.smithi153.stdout:(155/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-04-12T21:01:53.771 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-12T21:01:53.787 INFO:teuthology.orchestra.run.smithi153.stdout:(156/174): elfutils-debuginfod-client-0.190-2.e 876 kB/s | 36 kB 00:00 2024-04-12T21:01:53.854 INFO:teuthology.orchestra.run.smithi153.stdout:(157/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-04-12T21:01:53.926 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-12T21:01:53.929 INFO:teuthology.orchestra.run.smithi153.stdout:(158/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.3 MB/s | 256 kB 00:00 2024-04-12T21:01:53.996 INFO:teuthology.orchestra.run.smithi153.stdout:(159/174): libacl-2.3.1-4.el9.x86_64.rpm 346 kB/s | 23 kB 00:00 2024-04-12T21:01:54.054 INFO:teuthology.orchestra.run.smithi153.stdout:(160/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-04-12T21:01:54.099 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-12T21:01:54.122 INFO:teuthology.orchestra.run.smithi153.stdout:(161/174): librados2-19.0.0-2879.g324baf8d.el9. 1.6 MB/s | 3.4 MB 00:02 2024-04-12T21:01:54.154 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-12T21:01:54.164 INFO:teuthology.orchestra.run.smithi153.stdout:(162/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-04-12T21:01:54.192 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-04-12T21:01:54.206 INFO:teuthology.orchestra.run.smithi153.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-04-12T21:01:54.230 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 82/198 2024-04-12T21:01:54.231 INFO:teuthology.orchestra.run.smithi153.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 957 kB/s | 64 kB 00:00 2024-04-12T21:01:54.273 INFO:teuthology.orchestra.run.smithi153.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 428 kB/s | 29 kB 00:00 2024-04-12T21:01:54.282 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : uuid-1.6.2-55.el9.x86_64 83/198 2024-04-12T21:01:54.331 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 84/198 2024-04-12T21:01:54.365 INFO:teuthology.orchestra.run.smithi153.stdout:(166/174): openssl-3.0.7-27.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2024-04-12T21:01:54.373 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 85/198 2024-04-12T21:01:54.421 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 86/198 2024-04-12T21:01:54.448 INFO:teuthology.orchestra.run.smithi153.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 12 MB/s | 2.2 MB 00:00 2024-04-12T21:01:54.470 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 87/198 2024-04-12T21:01:54.539 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 88/198 2024-04-12T21:01:54.586 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 89/198 2024-04-12T21:01:54.590 INFO:teuthology.orchestra.run.smithi153.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.3 MB 00:00 2024-04-12T21:01:54.616 INFO:teuthology.orchestra.run.smithi153.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 2.7 MB/s | 465 kB 00:00 2024-04-12T21:01:54.658 INFO:teuthology.orchestra.run.smithi153.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 465 kB/s | 31 kB 00:00 2024-04-12T21:01:54.713 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 90/198 2024-04-12T21:01:54.716 INFO:teuthology.orchestra.run.smithi153.stdout:(171/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2024-04-12T21:01:54.781 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 91/198 2024-04-12T21:01:54.816 INFO:teuthology.orchestra.run.smithi153.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.8 MB/s | 1.1 MB 00:00 2024-04-12T21:01:54.817 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 92/198 2024-04-12T21:01:54.875 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 93/198 2024-04-12T21:01:55.015 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 94/198 2024-04-12T21:01:55.082 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 95/198 2024-04-12T21:01:55.342 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 120/198 2024-04-12T21:01:55.342 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-04-12T21:01:55.342 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-04-12T21:01:55.342 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-04-12T21:01:55.343 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-04-12T21:01:55.343 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-04-12T21:01:55.343 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-04-12T21:01:55.343 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-04-12T21:01:55.343 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:01:55.434 INFO:teuthology.orchestra.run.smithi153.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 104 kB/s | 74 kB 00:00 2024-04-12T21:01:55.500 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 121/198 2024-04-12T21:01:55.516 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 121/198 2024-04-12T21:01:55.516 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:01:55.516 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-12T21:01:55.516 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:01:55.998 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 96/198 2024-04-12T21:01:56.056 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 97/198 2024-04-12T21:01:56.090 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 98/198 2024-04-12T21:01:56.205 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 99/198 2024-04-12T21:01:56.268 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 100/198 2024-04-12T21:01:56.366 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 101/198 2024-04-12T21:01:56.408 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 102/198 2024-04-12T21:01:56.636 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.21-2.el9.x86_64 103/198 2024-04-12T21:01:56.671 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 104/198 2024-04-12T21:01:56.769 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 105/198 2024-04-12T21:01:57.003 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 122/198 2024-04-12T21:01:57.029 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 122/198 2024-04-12T21:01:57.030 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:01:57.030 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-12T21:01:57.030 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-12T21:01:57.030 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-12T21:01:57.030 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:01:57.254 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 123/198 2024-04-12T21:01:57.276 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 123/198 2024-04-12T21:01:57.276 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:01:57.277 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-12T21:01:57.277 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-12T21:01:57.277 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-12T21:01:57.277 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:01:57.575 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 106/198 2024-04-12T21:01:57.718 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 107/198 2024-04-12T21:01:57.721 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 124/198 2024-04-12T21:01:57.744 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 124/198 2024-04-12T21:01:57.744 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:01:57.744 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-12T21:01:57.744 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-12T21:01:57.744 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-12T21:01:57.744 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:01:57.859 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : automake-1.16.2-8.el9.noarch 125/198 2024-04-12T21:01:57.908 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 126/198 2024-04-12T21:01:57.964 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 127/198 2024-04-12T21:01:58.154 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 128/198 2024-04-12T21:01:58.205 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 129/198 2024-04-12T21:01:58.282 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 130/198 2024-04-12T21:01:58.371 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 131/198 2024-04-12T21:01:58.510 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 132/198 2024-04-12T21:01:58.546 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 133/198 2024-04-12T21:01:58.579 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 134/198 2024-04-12T21:01:58.610 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 135/198 2024-04-12T21:01:58.646 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-16.el9.x86_64 136/198 2024-04-12T21:01:58.674 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-15.el9.x86_64 137/198 2024-04-12T21:01:59.025 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 138/198 2024-04-12T21:01:59.055 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 138/198 2024-04-12T21:01:59.088 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.e 139/198 2024-04-12T21:01:59.148 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d. 140/198 2024-04-12T21:01:59.178 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 141/198 2024-04-12T21:01:59.265 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 108/198 2024-04-12T21:01:59.370 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 109/198 2024-04-12T21:01:59.414 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 110/198 2024-04-12T21:01:59.748 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 111/198 2024-04-12T21:01:59.782 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 112/198 2024-04-12T21:01:59.903 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 112/198 2024-04-12T21:01:59.938 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 113/198 2024-04-12T21:01:59.998 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 114/198 2024-04-12T21:02:00.206 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 141/198 2024-04-12T21:02:00.239 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 142/198 2024-04-12T21:02:00.525 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-12T21:02:00.560 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 116/198 2024-04-12T21:02:00.633 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 116/198 2024-04-12T21:02:00.661 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 117/198 2024-04-12T21:02:01.199 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 142/198 2024-04-12T21:02:01.222 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 143/198 2024-04-12T21:02:01.321 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 143/198 2024-04-12T21:02:01.493 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el 144/198 2024-04-12T21:02:01.524 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 145/198 2024-04-12T21:02:01.547 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 145/198 2024-04-12T21:02:01.547 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:01.547 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-12T21:02:01.547 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-12T21:02:01.547 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-12T21:02:01.547 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:02:01.591 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 146/198 2024-04-12T21:02:01.603 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 146/198 2024-04-12T21:02:01.700 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 147/198 2024-04-12T21:02:03.248 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:02:03.283 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:02:03.914 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:02:03.962 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 119/198 2024-04-12T21:02:04.126 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 119/198 2024-04-12T21:02:04.127 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-12T21:02:04.127 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-12T21:02:04.127 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:04.179 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 120/198 2024-04-12T21:02:06.191 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 148/198 2024-04-12T21:02:06.252 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 149/198 2024-04-12T21:02:06.357 INFO:teuthology.orchestra.run.smithi153.stdout:(174/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 800 kB/s | 16 MB 00:20 2024-04-12T21:02:06.362 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-04-12T21:02:06.362 INFO:teuthology.orchestra.run.smithi153.stdout:Total 3.7 MB/s | 229 MB 01:01 2024-04-12T21:02:07.132 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libtool-2.4.6-45.el9.x86_64 150/198 2024-04-12T21:02:07.173 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:02:07.196 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:02:07.196 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:07.196 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-12T21:02:07.196 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-12T21:02:07.196 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-12T21:02:07.196 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:02:07.241 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2879.g324ba 152/198 2024-04-12T21:02:07.262 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2879.g324ba 152/198 2024-04-12T21:02:07.262 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:07.262 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-12T21:02:07.262 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:02:07.307 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:02:07.329 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:02:07.330 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:07.330 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-12T21:02:07.330 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-12T21:02:07.330 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-12T21:02:07.330 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:02:07.633 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 154/198 2024-04-12T21:02:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 154/198 2024-04-12T21:02:07.656 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:07.656 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-12T21:02:07.656 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-12T21:02:07.656 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-12T21:02:07.656 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:02:07.676 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-04-12T21:02:07.708 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 155/198 2024-04-12T21:02:07.738 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 156/198 2024-04-12T21:02:07.790 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/198 2024-04-12T21:02:07.955 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-04-12T21:02:07.955 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-04-12T21:02:07.972 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 158/198 2024-04-12T21:02:08.029 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 159/198 2024-04-12T21:02:08.068 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-12T21:02:08.129 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-12T21:02:08.172 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-12T21:02:08.196 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_ 163/198 2024-04-12T21:02:08.229 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-12T21:02:08.258 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-12T21:02:08.304 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-12T21:02:08.373 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-12T21:02:08.407 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-12T21:02:09.027 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:09.064 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:09.084 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:09.899 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 170/198 2024-04-12T21:02:09.910 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-04-12T21:02:09.911 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-04-12T21:02:09.992 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-12T21:02:10.072 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flex-2.6.4-9.el9.x86_64 172/198 2024-04-12T21:02:10.103 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : bison-3.7.4-5.el9.x86_64 173/198 2024-04-12T21:02:10.129 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 174/198 2024-04-12T21:02:10.162 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-12T21:02:10.188 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-12T21:02:10.214 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-12T21:02:10.214 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-12T21:02:10.302 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-12T21:02:10.332 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-12T21:02:10.332 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-12T21:02:10.416 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-12T21:02:10.442 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-12T21:02:10.475 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-12T21:02:10.501 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-12T21:02:10.526 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-12T21:02:10.559 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-12T21:02:10.598 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-12T21:02:10.630 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-12T21:02:10.656 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-12T21:02:10.681 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-12T21:02:10.706 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-12T21:02:10.731 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-12T21:02:10.738 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:10.738 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:10.926 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:10.926 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-12T21:02:11.009 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-12T21:02:11.034 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-12T21:02:11.065 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-12T21:02:11.093 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-12T21:02:11.118 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-12T21:02:11.199 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-12T21:02:16.831 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 1/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 2/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 3/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 4/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2879.g324ba 5/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 6/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 7/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 8/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 9/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 12/198 2024-04-12T21:02:16.832 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 13/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 14/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_ 15/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 17/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_6 18/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 19/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 20/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el 21/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:19.0.0-2879.g324baf8d.el9. 22/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_6 23/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 24/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 25/198 2024-04-12T21:02:16.833 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 26/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 27/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 28/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 29/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d. 30/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 31/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 32/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 33/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el 34/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 35/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.e 36/198 2024-04-12T21:02:16.834 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 37/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 38/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 39/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-16.el9.x86_64 42/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/198 2024-04-12T21:02:16.835 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/198 2024-04-12T21:02:16.836 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 64/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 70/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-13.14-1.el9.x86_64 71/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 72/198 2024-04-12T21:02:16.837 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 73/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 74/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/198 2024-04-12T21:02:16.838 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/198 2024-04-12T21:02:16.839 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-12T21:02:16.840 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-12T21:02:16.841 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-12T21:02:16.842 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-12T21:02:16.843 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/198 2024-04-12T21:02:16.844 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 173/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 174/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 175/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 176/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 177/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 178/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 179/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 180/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 181/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 182/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 183/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 184/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 185/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 186/198 2024-04-12T21:02:16.845 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 187/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 188/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 189/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 190/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 191/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 192/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 193/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 194/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-12T21:02:16.846 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-12T21:02:16.882 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-04-12T21:02:16.919 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 120/198 2024-04-12T21:02:16.919 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-04-12T21:02:16.920 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-04-12T21:02:16.920 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-04-12T21:02:16.920 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-04-12T21:02:16.920 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-04-12T21:02:16.920 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-04-12T21:02:16.920 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-04-12T21:02:16.920 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:16.985 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-12T21:02:17.019 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-12T21:02:17.050 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-12T21:02:17.059 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-12T21:02:17.100 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-12T21:02:17.145 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 121/198 2024-04-12T21:02:17.153 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-12T21:02:17.187 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-12T21:02:17.196 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 121/198 2024-04-12T21:02:17.197 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:17.197 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-12T21:02:17.197 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:17.221 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-12T21:02:17.263 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-12T21:02:17.586 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-12T21:02:17.638 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:02:17.744 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:02:17.773 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:02:17.847 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:02:17.877 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 12/198 2024-04-12T21:02:17.918 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-12T21:02:18.099 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-12T21:02:18.150 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-12T21:02:18.178 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:02:18.284 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:02:18.312 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 17/198 2024-04-12T21:02:18.359 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 17/198 2024-04-12T21:02:18.399 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-12T21:02:18.422 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-12T21:02:18.437 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-12T21:02:18.580 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-12T21:02:18.611 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-12T21:02:18.643 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-12T21:02:18.740 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-12T21:02:18.787 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 122/198 2024-04-12T21:02:18.828 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-12T21:02:18.867 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 122/198 2024-04-12T21:02:18.867 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:18.867 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-12T21:02:18.867 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-12T21:02:18.867 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-12T21:02:18.868 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:18.889 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-12T21:02:18.962 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:02:19.006 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:02:19.121 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 123/198 2024-04-12T21:02:19.189 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 123/198 2024-04-12T21:02:19.189 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:19.189 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-12T21:02:19.189 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-12T21:02:19.189 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-12T21:02:19.190 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:19.219 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-12T21:02:19.338 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-12T21:02:19.440 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-12T21:02:19.469 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-12T21:02:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-12T21:02:19.569 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/198 2024-04-12T21:02:19.611 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/198 2024-04-12T21:02:19.663 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 124/198 2024-04-12T21:02:19.664 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/198 2024-04-12T21:02:19.696 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 32/198 2024-04-12T21:02:19.724 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-5.4.4-4.el9.x86_64 33/198 2024-04-12T21:02:19.729 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 124/198 2024-04-12T21:02:19.729 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:19.729 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-12T21:02:19.729 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-12T21:02:19.729 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-12T21:02:19.730 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:19.792 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 34/198 2024-04-12T21:02:19.846 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 35/198 2024-04-12T21:02:19.872 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : automake-1.16.2-8.el9.noarch 125/198 2024-04-12T21:02:19.876 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 36/198 2024-04-12T21:02:19.912 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 37/198 2024-04-12T21:02:19.935 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 126/198 2024-04-12T21:02:19.980 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 38/198 2024-04-12T21:02:20.014 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:19.0.0-2879.g324baf8d.el9. 39/198 2024-04-12T21:02:20.018 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 127/198 2024-04-12T21:02:20.057 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el 40/198 2024-04-12T21:02:20.086 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_6 41/198 2024-04-12T21:02:20.099 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-12T21:02:20.099 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:02:20.099 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: acl-2.3.1-4.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.100 INFO:teuthology.orchestra.run.smithi104.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-04-12T21:02:20.101 INFO:teuthology.orchestra.run.smithi104.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: automake-1.16.2-8.el9.noarch 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: bison-3.7.4-5.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.102 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:20.103 INFO:teuthology.orchestra.run.smithi104.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: flex-2.6.4-9.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: jq-1.6-16.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.104 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-12T21:02:20.105 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: lua-5.4.4-4.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-13.14-1.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-12T21:02:20.106 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-12T21:02:20.107 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-12T21:02:20.108 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-12T21:02:20.109 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-12T21:02:20.110 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-12T21:02:20.111 INFO:teuthology.orchestra.run.smithi104.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: re2-1:20211101-3.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-12T21:02:20.112 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-12T21:02:20.113 INFO:teuthology.orchestra.run.smithi104.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-12T21:02:20.114 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-04-12T21:02:20.114 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-04-12T21:02:20.155 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-12T21:02:20.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-12T21:02:20.253 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 128/198 2024-04-12T21:02:20.274 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-12T21:02:20.317 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 129/198 2024-04-12T21:02:20.403 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 130/198 2024-04-12T21:02:20.409 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-12T21:02:20.470 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-12T21:02:20.512 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 131/198 2024-04-12T21:02:20.619 DEBUG:teuthology.parallel:result is None 2024-04-12T21:02:20.680 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 132/198 2024-04-12T21:02:20.724 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 133/198 2024-04-12T21:02:20.766 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 134/198 2024-04-12T21:02:20.806 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 135/198 2024-04-12T21:02:20.862 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.6-16.el9.x86_64 136/198 2024-04-12T21:02:20.892 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-15.el9.x86_64 137/198 2024-04-12T21:02:21.002 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-12T21:02:21.090 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-12T21:02:21.184 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-12T21:02:21.218 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-12T21:02:21.228 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-12T21:02:21.258 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-12T21:02:21.298 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-12T21:02:21.345 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_6 53/198 2024-04-12T21:02:21.374 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-12T21:02:21.482 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 138/198 2024-04-12T21:02:21.521 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 138/198 2024-04-12T21:02:21.578 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.e 139/198 2024-04-12T21:02:21.617 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-12T21:02:21.684 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d. 140/198 2024-04-12T21:02:21.714 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 141/198 2024-04-12T21:02:21.753 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-12T21:02:21.835 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-12T21:02:21.879 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-12T21:02:22.020 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-12T21:02:22.081 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-12T21:02:22.136 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-12T21:02:22.208 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-12T21:02:22.240 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-12T21:02:22.293 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-12T21:02:22.340 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-12T21:02:22.375 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-12T21:02:22.419 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-12T21:02:22.461 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-12T21:02:22.595 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-12T21:02:22.663 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-12T21:02:22.800 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-12T21:02:22.954 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 141/198 2024-04-12T21:02:22.959 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-12T21:02:22.999 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 142/198 2024-04-12T21:02:23.142 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-12T21:02:23.180 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-12T21:02:23.211 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-12T21:02:23.237 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-12T21:02:23.261 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-12T21:02:23.336 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-12T21:02:23.463 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-12T21:02:23.503 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-12T21:02:23.530 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-04-12T21:02:23.564 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 82/198 2024-04-12T21:02:23.604 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : uuid-1.6.2-55.el9.x86_64 83/198 2024-04-12T21:02:23.635 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 84/198 2024-04-12T21:02:23.674 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 85/198 2024-04-12T21:02:23.699 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 86/198 2024-04-12T21:02:23.741 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 87/198 2024-04-12T21:02:23.782 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 88/198 2024-04-12T21:02:23.816 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 89/198 2024-04-12T21:02:23.904 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 90/198 2024-04-12T21:02:23.947 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 91/198 2024-04-12T21:02:23.978 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 92/198 2024-04-12T21:02:24.026 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 93/198 2024-04-12T21:02:24.104 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 142/198 2024-04-12T21:02:24.123 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 94/198 2024-04-12T21:02:24.132 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 143/198 2024-04-12T21:02:24.180 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 95/198 2024-04-12T21:02:24.277 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 143/198 2024-04-12T21:02:24.456 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el 144/198 2024-04-12T21:02:24.484 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 145/198 2024-04-12T21:02:24.562 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 145/198 2024-04-12T21:02:24.562 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:24.563 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-12T21:02:24.563 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-12T21:02:24.563 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-12T21:02:24.563 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:24.610 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 146/198 2024-04-12T21:02:24.648 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 146/198 2024-04-12T21:02:24.749 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 147/198 2024-04-12T21:02:24.913 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 96/198 2024-04-12T21:02:24.960 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 97/198 2024-04-12T21:02:24.993 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 98/198 2024-04-12T21:02:25.094 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 99/198 2024-04-12T21:02:25.152 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 100/198 2024-04-12T21:02:25.238 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 101/198 2024-04-12T21:02:25.279 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 102/198 2024-04-12T21:02:25.456 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 103/198 2024-04-12T21:02:25.485 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 104/198 2024-04-12T21:02:25.562 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 105/198 2024-04-12T21:02:26.213 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 106/198 2024-04-12T21:02:26.324 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 107/198 2024-04-12T21:02:27.549 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 108/198 2024-04-12T21:02:27.630 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 109/198 2024-04-12T21:02:27.656 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 110/198 2024-04-12T21:02:27.970 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 111/198 2024-04-12T21:02:28.003 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 112/198 2024-04-12T21:02:28.093 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 112/198 2024-04-12T21:02:28.124 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 113/198 2024-04-12T21:02:28.160 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 114/198 2024-04-12T21:02:28.643 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-12T21:02:28.673 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 116/198 2024-04-12T21:02:28.744 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 116/198 2024-04-12T21:02:28.766 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 117/198 2024-04-12T21:02:29.599 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 148/198 2024-04-12T21:02:29.686 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 149/198 2024-04-12T21:02:30.631 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libtool-2.4.6-45.el9.x86_64 150/198 2024-04-12T21:02:30.669 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:02:30.736 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:02:30.736 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:30.736 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-12T21:02:30.736 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-12T21:02:30.736 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-12T21:02:30.736 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:30.778 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2879.g324ba 152/198 2024-04-12T21:02:30.845 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2879.g324ba 152/198 2024-04-12T21:02:30.845 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:30.845 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-12T21:02:30.845 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:30.913 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:02:30.987 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:02:30.987 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:30.988 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-12T21:02:30.988 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-12T21:02:30.988 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-12T21:02:30.988 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:31.013 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:02:31.054 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:02:31.330 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 154/198 2024-04-12T21:02:31.413 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 154/198 2024-04-12T21:02:31.413 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:31.413 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-12T21:02:31.413 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-12T21:02:31.413 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-12T21:02:31.413 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:31.480 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 155/198 2024-04-12T21:02:31.514 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 156/198 2024-04-12T21:02:31.538 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/198 2024-04-12T21:02:31.594 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 118/198 2024-04-12T21:02:31.632 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 119/198 2024-04-12T21:02:31.713 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 119/198 2024-04-12T21:02:31.713 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-12T21:02:31.713 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-12T21:02:31.713 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:31.741 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 120/198 2024-04-12T21:02:31.797 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 158/198 2024-04-12T21:02:31.857 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 159/198 2024-04-12T21:02:31.916 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-12T21:02:31.980 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-12T21:02:32.025 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-12T21:02:32.056 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_ 163/198 2024-04-12T21:02:32.097 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-12T21:02:32.138 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-12T21:02:32.212 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-12T21:02:32.319 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-12T21:02:32.347 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-12T21:02:33.253 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:33.304 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:33.367 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:34.734 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 170/198 2024-04-12T21:02:34.846 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-12T21:02:34.950 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flex-2.6.4-9.el9.x86_64 172/198 2024-04-12T21:02:34.994 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : bison-3.7.4-5.el9.x86_64 173/198 2024-04-12T21:02:35.019 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 174/198 2024-04-12T21:02:35.053 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-12T21:02:35.087 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-12T21:02:35.129 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-12T21:02:35.130 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-12T21:02:35.275 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-12T21:02:35.317 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-12T21:02:35.317 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-12T21:02:35.455 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-12T21:02:35.499 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-12T21:02:35.545 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-12T21:02:35.595 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-12T21:02:35.652 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-12T21:02:35.722 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-12T21:02:35.800 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-12T21:02:35.849 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-12T21:02:35.882 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-12T21:02:35.916 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-12T21:02:35.949 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-12T21:02:35.982 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-12T21:02:36.019 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:36.020 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:36.350 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:36.350 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-12T21:02:36.481 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-12T21:02:36.519 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-12T21:02:36.573 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-12T21:02:36.618 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-12T21:02:36.662 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-12T21:02:36.782 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-12T21:02:37.045 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-12T21:02:37.046 INFO:teuthology.orchestra.run.smithi081.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-12T21:02:37.046 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:42.729 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 1/198 2024-04-12T21:02:42.729 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 2/198 2024-04-12T21:02:42.729 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 3/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 4/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2879.g324ba 5/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 6/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 7/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 8/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 9/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 12/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 13/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 14/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_ 15/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:02:42.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 17/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_6 18/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 19/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 20/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el 21/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.0.0-2879.g324baf8d.el9. 22/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_6 23/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 24/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 25/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 26/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 27/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 28/198 2024-04-12T21:02:42.731 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 29/198 2024-04-12T21:02:42.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d. 30/198 2024-04-12T21:02:42.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 31/198 2024-04-12T21:02:42.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 32/198 2024-04-12T21:02:42.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 33/198 2024-04-12T21:02:42.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el 34/198 2024-04-12T21:02:42.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 35/198 2024-04-12T21:02:42.746 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.e 36/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 37/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 38/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 39/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.6-16.el9.x86_64 42/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/198 2024-04-12T21:02:42.747 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 64/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/198 2024-04-12T21:02:42.748 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 70/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-13.14-1.el9.x86_64 71/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 72/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 73/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 74/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/198 2024-04-12T21:02:42.749 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/198 2024-04-12T21:02:42.750 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-12T21:02:42.751 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-12T21:02:42.752 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-12T21:02:42.753 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-12T21:02:42.753 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-12T21:02:42.753 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-12T21:02:42.753 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-12T21:02:42.753 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-12T21:02:42.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-12T21:02:42.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 173/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 174/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 175/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 176/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 177/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 178/198 2024-04-12T21:02:42.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 179/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 180/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 181/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 182/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 183/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 184/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 185/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 186/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 187/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 188/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 189/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 190/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 191/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 192/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 193/198 2024-04-12T21:02:42.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 194/198 2024-04-12T21:02:42.758 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-12T21:02:42.758 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-12T21:02:42.758 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-12T21:02:43.295 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 120/198 2024-04-12T21:02:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-04-12T21:02:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-04-12T21:02:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-04-12T21:02:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-04-12T21:02:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-04-12T21:02:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-04-12T21:02:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-04-12T21:02:43.296 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:43.469 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 121/198 2024-04-12T21:02:43.484 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 121/198 2024-04-12T21:02:43.484 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:43.484 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-12T21:02:43.485 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:44.964 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 122/198 2024-04-12T21:02:44.989 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 122/198 2024-04-12T21:02:44.989 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:44.989 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-12T21:02:44.989 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-12T21:02:44.989 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-12T21:02:44.989 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:45.215 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 123/198 2024-04-12T21:02:45.237 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 123/198 2024-04-12T21:02:45.237 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:45.237 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-12T21:02:45.237 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-12T21:02:45.237 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-12T21:02:45.237 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: acl-2.3.1-4.el9.x86_64 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-12T21:02:45.320 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.321 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: automake-1.16.2-8.el9.noarch 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: bison-3.7.4-5.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.322 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.323 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: flex-2.6.4-9.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: jq-1.6-16.el9.x86_64 2024-04-12T21:02:45.324 INFO:teuthology.orchestra.run.smithi081.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-12T21:02:45.325 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-12T21:02:45.326 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-13.14-1.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.327 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-12T21:02:45.328 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-12T21:02:45.329 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-12T21:02:45.330 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-12T21:02:45.331 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-3.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-12T21:02:45.332 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-12T21:02:45.333 INFO:teuthology.orchestra.run.smithi081.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-12T21:02:45.333 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:02:45.333 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-04-12T21:02:45.682 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 124/198 2024-04-12T21:02:45.704 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 124/198 2024-04-12T21:02:45.704 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:45.704 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-12T21:02:45.704 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-12T21:02:45.704 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-12T21:02:45.704 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:45.812 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : automake-1.16.2-8.el9.noarch 125/198 2024-04-12T21:02:45.869 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 126/198 2024-04-12T21:02:45.924 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 127/198 2024-04-12T21:02:46.110 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 128/198 2024-04-12T21:02:46.157 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 129/198 2024-04-12T21:02:46.236 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 130/198 2024-04-12T21:02:46.301 DEBUG:teuthology.parallel:result is None 2024-04-12T21:02:46.327 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 131/198 2024-04-12T21:02:46.465 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 132/198 2024-04-12T21:02:46.500 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 133/198 2024-04-12T21:02:46.542 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 134/198 2024-04-12T21:02:46.573 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 135/198 2024-04-12T21:02:46.608 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 136/198 2024-04-12T21:02:46.652 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 137/198 2024-04-12T21:02:47.047 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 138/198 2024-04-12T21:02:47.068 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 138/198 2024-04-12T21:02:47.100 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.e 139/198 2024-04-12T21:02:47.161 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d. 140/198 2024-04-12T21:02:47.191 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 141/198 2024-04-12T21:02:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 141/198 2024-04-12T21:02:48.268 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 142/198 2024-04-12T21:02:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 142/198 2024-04-12T21:02:49.268 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 143/198 2024-04-12T21:02:49.369 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 143/198 2024-04-12T21:02:49.523 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el 144/198 2024-04-12T21:02:49.545 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 145/198 2024-04-12T21:02:49.567 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 145/198 2024-04-12T21:02:49.568 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:49.568 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-12T21:02:49.568 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-12T21:02:49.568 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-12T21:02:49.568 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:49.604 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 146/198 2024-04-12T21:02:49.614 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 146/198 2024-04-12T21:02:49.714 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 147/198 2024-04-12T21:02:54.253 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 148/198 2024-04-12T21:02:54.315 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 149/198 2024-04-12T21:02:55.197 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libtool-2.4.6-45.el9.x86_64 150/198 2024-04-12T21:02:55.227 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:02:55.250 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:02:55.250 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:55.250 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-12T21:02:55.250 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-12T21:02:55.251 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-12T21:02:55.251 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:55.295 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2879.g324ba 152/198 2024-04-12T21:02:55.316 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2879.g324ba 152/198 2024-04-12T21:02:55.317 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:55.317 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-12T21:02:55.317 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:55.371 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:02:55.392 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:02:55.393 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:55.393 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-12T21:02:55.393 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-12T21:02:55.393 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-12T21:02:55.393 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:55.696 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 154/198 2024-04-12T21:02:55.718 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 154/198 2024-04-12T21:02:55.718 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-12T21:02:55.718 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-12T21:02:55.718 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-12T21:02:55.718 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-12T21:02:55.718 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:02:55.771 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 155/198 2024-04-12T21:02:55.810 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 156/198 2024-04-12T21:02:55.837 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/198 2024-04-12T21:02:56.019 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 158/198 2024-04-12T21:02:56.084 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 159/198 2024-04-12T21:02:56.123 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-12T21:02:56.176 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-12T21:02:56.219 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-12T21:02:56.252 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_ 163/198 2024-04-12T21:02:56.292 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-12T21:02:56.322 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-12T21:02:56.367 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-12T21:02:56.428 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-12T21:02:56.462 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-12T21:02:57.078 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:57.118 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:57.137 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-12T21:02:57.968 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 170/198 2024-04-12T21:02:58.070 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-12T21:02:58.152 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flex-2.6.4-9.el9.x86_64 172/198 2024-04-12T21:02:58.191 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : bison-3.7.4-5.el9.x86_64 173/198 2024-04-12T21:02:58.216 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 174/198 2024-04-12T21:02:58.241 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-12T21:02:58.267 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-12T21:02:58.293 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-12T21:02:58.293 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-12T21:02:58.382 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-12T21:02:58.419 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-12T21:02:58.419 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-12T21:02:58.503 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-12T21:02:58.529 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-12T21:02:58.563 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-12T21:02:58.588 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-12T21:02:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-12T21:02:58.646 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-12T21:02:58.686 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-12T21:02:58.718 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-12T21:02:58.768 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-12T21:02:58.793 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-12T21:02:58.818 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-12T21:02:58.843 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-12T21:02:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:58.850 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:59.046 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-12T21:02:59.047 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-12T21:02:59.129 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-12T21:02:59.154 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-12T21:02:59.186 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-12T21:02:59.214 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-12T21:02:59.239 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-12T21:02:59.302 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-12T21:03:04.795 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-12T21:03:04.795 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 1/198 2024-04-12T21:03:04.795 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 2/198 2024-04-12T21:03:04.795 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 3/198 2024-04-12T21:03:04.795 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 4/198 2024-04-12T21:03:04.795 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2879.g324ba 5/198 2024-04-12T21:03:04.795 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 6/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 7/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 8/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 9/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 10/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 11/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 12/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 13/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 14/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_ 15/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 16/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 17/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_6 18/198 2024-04-12T21:03:04.796 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86 19/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 20/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el 21/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:19.0.0-2879.g324baf8d.el9. 22/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_6 23/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 24/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 25/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 26/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 27/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 28/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 29/198 2024-04-12T21:03:04.797 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d. 30/198 2024-04-12T21:03:04.813 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noa 31/198 2024-04-12T21:03:04.813 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.n 32/198 2024-04-12T21:03:04.813 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2879.g324 33/198 2024-04-12T21:03:04.813 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el 34/198 2024-04-12T21:03:04.813 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 35/198 2024-04-12T21:03:04.813 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.e 36/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 37/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 38/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 39/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 40/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 41/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 42/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 43/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 44/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 45/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 46/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 47/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 48/198 2024-04-12T21:03:04.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 49/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 50/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 51/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : automake-1.16.2-8.el9.noarch 52/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : bison-3.7.4-5.el9.x86_64 53/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 54/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 55/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flex-2.6.4-9.el9.x86_64 56/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 57/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 58/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 59/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 60/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 61/198 2024-04-12T21:03:04.815 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 62/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 63/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 64/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 65/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 66/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 67/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 68/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 69/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 70/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-13.14-1.el9.x86_64 71/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 72/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 73/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 74/198 2024-04-12T21:03:04.816 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 75/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 76/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 77/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 78/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 79/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 80/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 81/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 82/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 83/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 84/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 85/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 86/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 87/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 88/198 2024-04-12T21:03:04.817 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 89/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 90/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 91/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 93/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 94/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 95/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 96/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 97/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 98/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 99/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 100/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 101/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 102/198 2024-04-12T21:03:04.818 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 103/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 104/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-12T21:03:04.819 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-12T21:03:04.820 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-12T21:03:04.821 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 151/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 153/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-12T21:03:04.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : acl-2.3.1-4.el9.x86_64 163/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : acl-2.3.1-3.el9.x86_64 164/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 166/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 167/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 168/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 169/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 170/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 171/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 172/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 173/198 2024-04-12T21:03:04.823 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 174/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 175/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 176/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 177/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 178/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 179/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 180/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 181/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 182/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 183/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 184/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 185/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 186/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 187/198 2024-04-12T21:03:04.824 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 188/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 189/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 190/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 191/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 192/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 193/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 194/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-12T21:03:04.825 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-12T21:03:08.116 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-12T21:03:08.116 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:03:08.116 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-04-12T21:03:08.116 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-12T21:03:08.116 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-12T21:03:08.116 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: acl-2.3.1-4.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.117 INFO:teuthology.orchestra.run.smithi153.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: automake-1.16.2-8.el9.noarch 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: bison-3.7.4-5.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-12T21:03:08.118 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.119 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-volume-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-shell-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: cephfs-top-2:19.0.0-2879.g324baf8d.el9.noarch 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: flex-2.6.4-9.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-12T21:03:08.120 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libaio-devel-0.3.112-1.el8.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-12T21:03:08.121 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: lua-5.4.4-4.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-12T21:03:08.122 INFO:teuthology.orchestra.run.smithi153.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-13.14-1.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-12T21:03:08.123 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-12T21:03:08.124 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-12T21:03:08.125 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-12T21:03:08.126 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-12T21:03:08.127 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.128 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:19.0.0-2879.g324baf8d.el9.x86_64 2024-04-12T21:03:08.129 INFO:teuthology.orchestra.run.smithi153.stdout: re2-1:20211101-3.el9.x86_64 2024-04-12T21:03:08.129 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-12T21:03:08.129 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-12T21:03:08.129 INFO:teuthology.orchestra.run.smithi153.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-12T21:03:08.129 INFO:teuthology.orchestra.run.smithi153.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-12T21:03:08.129 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-12T21:03:08.129 INFO:teuthology.orchestra.run.smithi153.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-12T21:03:08.130 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:03:08.130 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-04-12T21:03:08.581 DEBUG:teuthology.parallel:result is None 2024-04-12T21:03:08.582 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:03:08.807 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T21:03:08.878 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-2879.g324baf8d.el9 2024-04-12T21:03:08.879 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2879.g324baf8d.el9 2024-04-12T21:03:08.879 INFO:teuthology.task.install:The correct ceph version 19.0.0-2879.g324baf8d is installed. 2024-04-12T21:03:08.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:03:09.029 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T21:03:09.055 INFO:teuthology.orchestra.run.smithi104.stdout:19.0.0-2879.g324baf8d.el9 2024-04-12T21:03:09.055 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2879.g324baf8d.el9 2024-04-12T21:03:09.055 INFO:teuthology.task.install:The correct ceph version 19.0.0-2879.g324baf8d is installed. 2024-04-12T21:03:09.057 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:03:09.194 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-12T21:03:09.218 INFO:teuthology.orchestra.run.smithi153.stdout:19.0.0-2879.g324baf8d.el9 2024-04-12T21:03:09.218 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2879.g324baf8d.el9 2024-04-12T21:03:09.218 INFO:teuthology.task.install:The correct ceph version 19.0.0-2879.g324baf8d is installed. 2024-04-12T21:03:09.224 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-12T21:03:09.224 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:03:09.224 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T21:03:09.328 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:03:09.328 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T21:03:09.362 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:03:09.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-12T21:03:09.395 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-12T21:03:09.395 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:03:09.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T21:03:09.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T21:03:09.595 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:03:09.595 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T21:03:09.630 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T21:03:09.698 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:03:09.698 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-04-12T21:03:09.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-12T21:03:09.800 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-12T21:03:09.800 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:03:09.801 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T21:03:09.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T21:03:10.019 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:03:10.020 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T21:03:10.048 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T21:03:10.115 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:03:10.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-12T21:03:10.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-12T21:03:10.215 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-12T21:03:10.215 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:03:10.215 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T21:03:10.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T21:03:10.412 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:03:10.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T21:03:10.439 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T21:03:10.506 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:03:10.506 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-04-12T21:03:10.537 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-12T21:03:10.608 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-12T21:03:10.717 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '324baf8ddb91dea46cac50cb535c306115791355', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-12T21:03:10.717 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:03:10.718 INFO:tasks.cephadm:Cluster fsid is 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:03:10.718 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-12T21:03:10.718 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.81', 'mon.b': '172.21.15.104', 'mon.c': '172.21.15.153'} 2024-04-12T21:03:10.718 INFO:tasks.cephadm:First mon is mon.a on smithi081 2024-04-12T21:03:10.718 INFO:tasks.cephadm:First mgr is x 2024-04-12T21:03:10.718 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-12T21:03:10.719 DEBUG:teuthology.orchestra.run.smithi081:> sudo hostname $(hostname -s) 2024-04-12T21:03:10.804 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-04-12T21:03:10.831 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-04-12T21:03:10.864 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-12T21:03:10.864 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:03:11.083 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '324baf8ddb91dea46cac50cb535c306115791355', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78468/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2879-g324baf8d', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2879.g324baf8d'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-11 07:57:23.604569', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240411.061452', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-12T21:03:11.295 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240411.061452, sha1 324baf8ddb91dea46cac50cb535c306115791355 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:03:11.296 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/x86_64/flavors/default/cephadm 2024-04-12T21:03:11.296 DEBUG:teuthology.orchestra.run.smithi081:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:12.014 INFO:teuthology.orchestra.run.smithi081.stdout:-rw-r--r--. 1 ubuntu ubuntu 781508 Apr 12 21:03 /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:12.014 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:12.494 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 781508 Apr 12 21:03 /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:12.494 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240411.061452/324baf8ddb91dea46cac50cb535c306115791355/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:13.351 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 781508 Apr 12 21:03 /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:13.351 DEBUG:teuthology.orchestra.run.smithi081:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:13.411 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:13.430 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-12T21:03:13.465 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 on all hosts... 2024-04-12T21:03:13.465 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 pull 2024-04-12T21:03:13.472 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 pull 2024-04-12T21:03:13.474 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 pull 2024-04-12T21:03:14.253 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355... 2024-04-12T21:03:14.331 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355... 2024-04-12T21:03:14.716 INFO:teuthology.orchestra.run.smithi081.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355... 2024-04-12T21:03:41.824 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-04-12T21:03:41.824 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev)", 2024-04-12T21:03:41.825 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca", 2024-04-12T21:03:41.825 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-04-12T21:03:41.825 INFO:teuthology.orchestra.run.smithi153.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63" 2024-04-12T21:03:41.825 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-04-12T21:03:41.825 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-04-12T21:03:49.873 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-04-12T21:03:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev)", 2024-04-12T21:03:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca", 2024-04-12T21:03:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-04-12T21:03:49.873 INFO:teuthology.orchestra.run.smithi104.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63" 2024-04-12T21:03:49.874 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-04-12T21:03:49.874 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-04-12T21:03:54.894 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-12T21:03:54.894 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_version": "ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev)", 2024-04-12T21:03:54.894 INFO:teuthology.orchestra.run.smithi081.stdout: "image_id": "8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca", 2024-04-12T21:03:54.894 INFO:teuthology.orchestra.run.smithi081.stdout: "repo_digests": [ 2024-04-12T21:03:54.894 INFO:teuthology.orchestra.run.smithi081.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63" 2024-04-12T21:03:54.894 INFO:teuthology.orchestra.run.smithi081.stdout: ] 2024-04-12T21:03:54.895 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-12T21:03:54.964 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph 2024-04-12T21:03:55.065 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-04-12T21:03:55.100 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-04-12T21:03:55.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /etc/ceph 2024-04-12T21:03:55.237 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-04-12T21:03:55.269 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-04-12T21:03:55.300 INFO:tasks.cephadm:Writing seed config... 2024-04-12T21:03:55.301 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-12T21:03:55.301 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-12T21:03:55.302 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-12T21:03:55.303 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-12T21:03:55.304 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-12T21:03:55.305 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:03:55.305 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-12T21:03:55.364 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 11b8cc88-f910-11ee-bc8d-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-12T21:03:55.364 DEBUG:teuthology.orchestra.run.smithi081:mon.a> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.a.service 2024-04-12T21:03:55.412 DEBUG:teuthology.orchestra.run.smithi081:mgr.x> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mgr.x.service 2024-04-12T21:03:55.458 INFO:tasks.cephadm:Bootstrapping... 2024-04-12T21:03:55.458 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 -v bootstrap --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.81 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-12T21:03:55.736 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-04-12T21:03:55.736 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355', '-v', 'bootstrap', '--fsid', '11b8cc88-f910-11ee-bc8d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.81', '--skip-admin-label'] 2024-04-12T21:03:55.906 INFO:teuthology.orchestra.run.smithi081.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-12T21:03:55.906 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying podman|docker is present... 2024-04-12T21:03:55.967 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-12T21:03:55.967 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying lvm2 is present... 2024-04-12T21:03:55.968 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying time synchronization is in place... 2024-04-12T21:03:55.985 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-12T21:03:55.985 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-12T21:03:56.001 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-12T21:03:56.001 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-04-12T21:03:56.018 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout enabled 2024-04-12T21:03:56.041 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout active 2024-04-12T21:03:56.041 INFO:teuthology.orchestra.run.smithi081.stdout:Unit chronyd.service is enabled and running 2024-04-12T21:03:56.041 INFO:teuthology.orchestra.run.smithi081.stdout:Repeating the final host check... 2024-04-12T21:03:56.100 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-12T21:03:56.100 INFO:teuthology.orchestra.run.smithi081.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-12T21:03:56.100 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl is present 2024-04-12T21:03:56.100 INFO:teuthology.orchestra.run.smithi081.stdout:lvcreate is present 2024-04-12T21:03:56.116 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-12T21:03:56.116 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-12T21:03:56.131 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-12T21:03:56.132 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-04-12T21:03:56.148 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout enabled 2024-04-12T21:03:56.163 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout active 2024-04-12T21:03:56.163 INFO:teuthology.orchestra.run.smithi081.stdout:Unit chronyd.service is enabled and running 2024-04-12T21:03:56.163 INFO:teuthology.orchestra.run.smithi081.stdout:Host looks OK 2024-04-12T21:03:56.164 INFO:teuthology.orchestra.run.smithi081.stdout:Cluster fsid: 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:03:56.164 INFO:teuthology.orchestra.run.smithi081.stdout:Acquiring lock 140701851583392 on /run/cephadm/11b8cc88-f910-11ee-bc8d-c7b262605968.lock 2024-04-12T21:03:56.164 INFO:teuthology.orchestra.run.smithi081.stdout:Lock 140701851583392 acquired on /run/cephadm/11b8cc88-f910-11ee-bc8d-c7b262605968.lock 2024-04-12T21:03:56.164 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying IP 172.21.15.81 port 3300 ... 2024-04-12T21:03:56.165 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying IP 172.21.15.81 port 6789 ... 2024-04-12T21:03:56.165 INFO:teuthology.orchestra.run.smithi081.stdout:Base mon IP(s) is [172.21.15.81:3300, 172.21.15.81:6789], mon addrv is [v2:172.21.15.81:3300,v1:172.21.15.81:6789] 2024-04-12T21:03:56.173 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.81 metric 100 2024-04-12T21:03:56.173 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.81 metric 100 2024-04-12T21:03:56.181 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-12T21:03:56.181 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-12T21:03:56.181 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1783sec hoplimit 64 pref medium 2024-04-12T21:03:56.188 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-12T21:03:56.188 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-12T21:03:56.188 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-12T21:03:56.188 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-12T21:03:56.188 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-04-12T21:03:56.188 INFO:teuthology.orchestra.run.smithi081.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-12T21:03:56.189 INFO:teuthology.orchestra.run.smithi081.stdout:Mon IP `172.21.15.81` is in CIDR network `172.21.0.0/20` 2024-04-12T21:03:56.189 INFO:teuthology.orchestra.run.smithi081.stdout:Mon IP `172.21.15.81` is in CIDR network `172.21.0.0/20` 2024-04-12T21:03:56.189 INFO:teuthology.orchestra.run.smithi081.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-12T21:03:56.190 INFO:teuthology.orchestra.run.smithi081.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-12T21:03:56.191 INFO:teuthology.orchestra.run.smithi081.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355... 2024-04-12T21:04:05.284 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stdout 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca 2024-04-12T21:04:05.284 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355... 2024-04-12T21:04:05.284 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Getting image source signatures 2024-04-12T21:04:05.285 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:6bf82c0ee398070d04aa79e81829b86865c405c08ea2a15650b327ddec49ed8f 2024-04-12T21:04:05.285 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying blob sha256:9793b5b5582798bde394f4383d142e9f248a25e4ebfd72ee05ea9ec6251a877d 2024-04-12T21:04:05.285 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Copying config sha256:8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca 2024-04-12T21:04:05.285 INFO:teuthology.orchestra.run.smithi081.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-12T21:04:06.812 INFO:teuthology.orchestra.run.smithi081.stdout:ceph: stdout ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:04:06.812 INFO:teuthology.orchestra.run.smithi081.stdout:Ceph version: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:04:06.812 INFO:teuthology.orchestra.run.smithi081.stdout:Extracting ceph user uid/gid from container image... 2024-04-12T21:04:08.023 INFO:teuthology.orchestra.run.smithi081.stdout:stat: stdout 167 167 2024-04-12T21:04:08.024 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial keys... 2024-04-12T21:04:09.136 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQDIoRlm7ayfHBAAJ2l8pOxOvJ5NwBQB5zXQwg== 2024-04-12T21:04:10.301 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQDJoRlm4NrUJxAAxG7Snf03dEoFRnBvTzoneA== 2024-04-12T21:04:11.400 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph-authtool: stdout AQDKoRlmWL7sLRAAKBJUWCwa9kswjT3vez1jFw== 2024-04-12T21:04:11.402 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial monmap... 2024-04-12T21:04:12.416 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-12T21:04:12.416 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-12T21:04:12.416 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:12.416 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-12T21:04:12.416 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool for a [v2:172.21.15.81:3300,v1:172.21.15.81:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-12T21:04:12.416 INFO:teuthology.orchestra.run.smithi081.stdout:setting min_mon_release = quincy 2024-04-12T21:04:12.416 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: set fsid to 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:12.416 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-12T21:04:12.417 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:04:12.417 INFO:teuthology.orchestra.run.smithi081.stdout:Creating mon... 2024-04-12T21:04:13.482 INFO:teuthology.orchestra.run.smithi081.stdout:create mon.a on 2024-04-12T21:04:13.839 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-12T21:04:14.074 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-12T21:04:14.351 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968.target → /etc/systemd/system/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968.target. 2024-04-12T21:04:14.351 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968.target → /etc/systemd/system/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968.target. 2024-04-12T21:04:14.724 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.a 2024-04-12T21:04:14.724 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to reset failed state of unit ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.a.service: Unit ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.a.service not loaded. 2024-04-12T21:04:15.015 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968.target.wants/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.a.service → /etc/systemd/system/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@.service. 2024-04-12T21:04:15.624 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:15 smithi081 podman[31070]: 2024-04-12 21:04:15.298280438 +0000 UTC m=+0.031142861 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:04:15.624 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:15 smithi081 podman[31070]: 2024-04-12 21:04:15.62617018 +0000 UTC m=+0.359032652 container init 90df519c260d2cfc3bee253ceb4f35d35fed2c22250501f62156f5090d530f2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:04:15.727 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-12T21:04:15.727 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2024-04-12T21:04:15.744 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-12T21:04:15.744 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-04-12T21:04:15.744 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2024-04-12T21:04:15.744 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to enable service . firewalld.service is not available 2024-04-12T21:04:15.744 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mon to start... 2024-04-12T21:04:15.744 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mon... 2024-04-12T21:04:15.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:15 smithi081 podman[31070]: 2024-04-12 21:04:15.637705236 +0000 UTC m=+0.370567705 container start 90df519c260d2cfc3bee253ceb4f35d35fed2c22250501f62156f5090d530f2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:04:15.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:15 smithi081 bash[31070]: 90df519c260d2cfc3bee253ceb4f35d35fed2c22250501f62156f5090d530f2d 2024-04-12T21:04:15.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:15 smithi081 systemd[1]: Started Ceph mon.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:04:15.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:15 smithi081 ceph-mon[31088]: mkfs 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:15.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:15 smithi081 ceph-mon[31088]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-12T21:04:16.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-12T21:04:16.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: monmap epoch 1 2024-04-12T21:04:16.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:16.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: last_changed 2024-04-12T21:04:11.886065+0000 2024-04-12T21:04:16.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:04:16.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: min_mon_release 19 (squid) 2024-04-12T21:04:16.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: election_strategy: 1 2024-04-12T21:04:16.995 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:04:16.996 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: fsmap 2024-04-12T21:04:16.996 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: osdmap e1: 0 total, 0 up, 0 in 2024-04-12T21:04:16.996 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: mgrmap e1: no daemons active 2024-04-12T21:04:16.996 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:16 smithi081 ceph-mon[31088]: from='client.? 172.21.15.81:0/1011359805' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-12T21:04:17.006 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout cluster: 2024-04-12T21:04:17.008 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout id: 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout services: 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.704697s) 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout data: 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout pgs: 2024-04-12T21:04:17.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:17.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.326+0000 7f7568000700 1 Processor -- start 2024-04-12T21:04:17.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.326+0000 7f7568000700 1 -- start start 2024-04-12T21:04:17.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.327+0000 7f7568000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756010b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:17.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.327+0000 7f7568000700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f756007bdd0 con 0x7f7560108ed0 2024-04-12T21:04:17.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.328+0000 7f7565600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756010b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:17.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.328+0000 7f7565600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756010b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45494/0 (socket says 172.21.15.81:45494) 2024-04-12T21:04:17.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.328+0000 7f7565600700 1 -- 172.21.15.81:0/1793731365 learned_addr learned my addr 172.21.15.81:0/1793731365 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:17.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.328+0000 7f7565600700 1 -- 172.21.15.81:0/1793731365 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756010b800 con 0x7f7560108ed0 2024-04-12T21:04:17.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.329+0000 7f7565600700 1 --2- 172.21.15.81:0/1793731365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756010b2c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75540098a0 tx=0x7f7554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcec4458035b576 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.329+0000 7f755fe00700 1 -- 172.21.15.81:0/1793731365 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7554004160 con 0x7f7560108ed0 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.329+0000 7f755fe00700 1 -- 172.21.15.81:0/1793731365 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7554004300 con 0x7f7560108ed0 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.329+0000 7f755fe00700 1 -- 172.21.15.81:0/1793731365 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75540036a0 con 0x7f7560108ed0 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.330+0000 7f7568000700 1 -- 172.21.15.81:0/1793731365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 msgr2=0x7f756010b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.330+0000 7f7568000700 1 --2- 172.21.15.81:0/1793731365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756010b2c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f75540098a0 tx=0x7f7554009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.330+0000 7f7568000700 1 -- 172.21.15.81:0/1793731365 shutdown_connections 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.330+0000 7f7568000700 1 --2- 172.21.15.81:0/1793731365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756010b2c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.330+0000 7f7568000700 1 -- 172.21.15.81:0/1793731365 >> 172.21.15.81:0/1793731365 conn(0x7f7560102860 msgr2=0x7f7560104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.330+0000 7f7568000700 1 -- 172.21.15.81:0/1793731365 shutdown_connections 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.330+0000 7f7568000700 1 -- 172.21.15.81:0/1793731365 wait complete. 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.331+0000 7f7568000700 1 Processor -- start 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.331+0000 7f7568000700 1 -- start start 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.331+0000 7f7568000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756019dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:17.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.331+0000 7f7568000700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f756010bbb0 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.332+0000 7f7565600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756019dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.332+0000 7f7565600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756019dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45504/0 (socket says 172.21.15.81:45504) 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.332+0000 7f7565600700 1 -- 172.21.15.81:0/1011359805 learned_addr learned my addr 172.21.15.81:0/1011359805 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.332+0000 7f7565600700 1 -- 172.21.15.81:0/1011359805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756019e240 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.333+0000 7f7565600700 1 --2- 172.21.15.81:0/1011359805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756019dd00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7554000c00 tx=0x7f7554004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.333+0000 7f755e000700 1 -- 172.21.15.81:0/1011359805 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7554003db0 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.333+0000 7f755e000700 1 -- 172.21.15.81:0/1011359805 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7554011630 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.333+0000 7f755e000700 1 -- 172.21.15.81:0/1011359805 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7554011950 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.333+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f756019e4d0 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.333+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75601a1170 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.333+0000 7f755e000700 1 -- 172.21.15.81:0/1011359805 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f755401a460 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.334+0000 7f755e000700 1 -- 172.21.15.81:0/1011359805 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f755401b780 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.334+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7560064110 con 0x7f7560108ed0 2024-04-12T21:04:17.018 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.338+0000 7f755e000700 1 -- 172.21.15.81:0/1011359805 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f755401b980 con 0x7f7560108ed0 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.418+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f756002e1d0 con 0x7f7560108ed0 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.418+0000 7f755e000700 1 -- 172.21.15.81:0/1011359805 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f755401bb60 con 0x7f7560108ed0 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.419+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 msgr2=0x7f756019dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.419+0000 7f7568000700 1 --2- 172.21.15.81:0/1011359805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756019dd00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7554000c00 tx=0x7f7554004750 comp rx=0 tx=0).stop 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.420+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 shutdown_connections 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.420+0000 7f7568000700 1 --2- 172.21.15.81:0/1011359805 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7560108ed0 0x7f756019dd00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.420+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 >> 172.21.15.81:0/1011359805 conn(0x7f7560102860 msgr2=0x7f7560104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.420+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 shutdown_connections 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:16.420+0000 7f7568000700 1 -- 172.21.15.81:0/1011359805 wait complete. 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:mon is available 2024-04-12T21:04:17.019 INFO:teuthology.orchestra.run.smithi081.stdout:Assimilating anything we can from ceph.conf... 2024-04-12T21:04:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:17 smithi081 ceph-mon[31088]: from='client.? 172.21.15.81:0/1258021894' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-12T21:04:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:17 smithi081 ceph-mon[31088]: from='client.? 172.21.15.81:0/1258021894' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [global] 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout fsid = 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.81:3300,v1:172.21.15.81:6789] 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-12T21:04:18.338 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-12T21:04:18.339 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:18.344 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [osd] 2024-04-12T21:04:18.344 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.564+0000 7f9c1d600700 1 Processor -- start 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.564+0000 7f9c1d600700 1 -- start start 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.564+0000 7f9c1d600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1810b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.565+0000 7f9c1d600700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1807be80 con 0x7f9c18108cc0 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.565+0000 7f9c17e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1810b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.565+0000 7f9c17e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1810b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45518/0 (socket says 172.21.15.81:45518) 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.565+0000 7f9c17e00700 1 -- 172.21.15.81:0/1001724141 learned_addr learned my addr 172.21.15.81:0/1001724141 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.565+0000 7f9c17e00700 1 -- 172.21.15.81:0/1001724141 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1810b5f0 con 0x7f9c18108cc0 2024-04-12T21:04:18.345 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.566+0000 7f9c17e00700 1 --2- 172.21.15.81:0/1001724141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1810b0b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9c080098a0 tx=0x7f9c08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fe63b3329085c server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.566+0000 7f9c16a00700 1 -- 172.21.15.81:0/1001724141 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c08004160 con 0x7f9c18108cc0 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.566+0000 7f9c16a00700 1 -- 172.21.15.81:0/1001724141 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c08004300 con 0x7f9c18108cc0 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.566+0000 7f9c16a00700 1 -- 172.21.15.81:0/1001724141 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c080036a0 con 0x7f9c18108cc0 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.567+0000 7f9c1d600700 1 -- 172.21.15.81:0/1001724141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 msgr2=0x7f9c1810b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.567+0000 7f9c1d600700 1 --2- 172.21.15.81:0/1001724141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1810b0b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9c080098a0 tx=0x7f9c08009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.567+0000 7f9c1d600700 1 -- 172.21.15.81:0/1001724141 shutdown_connections 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.567+0000 7f9c1d600700 1 --2- 172.21.15.81:0/1001724141 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1810b0b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.567+0000 7f9c1d600700 1 -- 172.21.15.81:0/1001724141 >> 172.21.15.81:0/1001724141 conn(0x7f9c181026f0 msgr2=0x7f9c18104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.567+0000 7f9c1d600700 1 -- 172.21.15.81:0/1001724141 shutdown_connections 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.567+0000 7f9c1d600700 1 -- 172.21.15.81:0/1001724141 wait complete. 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.568+0000 7f9c1d600700 1 Processor -- start 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.568+0000 7f9c1d600700 1 -- start start 2024-04-12T21:04:18.346 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.568+0000 7f9c1d600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.569+0000 7f9c1d600700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1810ba60 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.569+0000 7f9c17e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.569+0000 7f9c17e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45522/0 (socket says 172.21.15.81:45522) 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.569+0000 7f9c17e00700 1 -- 172.21.15.81:0/1258021894 learned_addr learned my addr 172.21.15.81:0/1258021894 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.569+0000 7f9c17e00700 1 -- 172.21.15.81:0/1258021894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1819e370 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.569+0000 7f9c17e00700 1 --2- 172.21.15.81:0/1258021894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1819de30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9c08000c00 tx=0x7f9c08004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.570+0000 7f9c14c00700 1 -- 172.21.15.81:0/1258021894 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c08003db0 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.570+0000 7f9c14c00700 1 -- 172.21.15.81:0/1258021894 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c08011630 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.570+0000 7f9c14c00700 1 -- 172.21.15.81:0/1258021894 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c08011950 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.570+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c1819e600 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.570+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c181a12f0 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.570+0000 7f9c14c00700 1 -- 172.21.15.81:0/1258021894 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9c0801a460 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.570+0000 7f9c14c00700 1 -- 172.21.15.81:0/1258021894 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9c0801b780 con 0x7f9c18108cc0 2024-04-12T21:04:18.347 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.571+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c18064110 con 0x7f9c18108cc0 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.574+0000 7f9c14c00700 1 -- 172.21.15.81:0/1258021894 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9c0801ba20 con 0x7f9c18108cc0 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.651+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f9c18076410 con 0x7f9c18108cc0 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.659+0000 7f9c14c00700 1 -- 172.21.15.81:0/1258021894 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f9c18076410 con 0x7f9c18108cc0 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.660+0000 7f9c14c00700 1 -- 172.21.15.81:0/1258021894 <== mon.0 v2:172.21.15.81:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9c0801ad50 con 0x7f9c18108cc0 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.661+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 msgr2=0x7f9c1819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.661+0000 7f9c1d600700 1 --2- 172.21.15.81:0/1258021894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1819de30 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9c08000c00 tx=0x7f9c08004750 comp rx=0 tx=0).stop 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.661+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 shutdown_connections 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.661+0000 7f9c1d600700 1 --2- 172.21.15.81:0/1258021894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9c18108cc0 0x7f9c1819de30 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.661+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 >> 172.21.15.81:0/1258021894 conn(0x7f9c181026f0 msgr2=0x7f9c18103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.661+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 shutdown_connections 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:17.661+0000 7f9c1d600700 1 -- 172.21.15.81:0/1258021894 wait complete. 2024-04-12T21:04:18.348 INFO:teuthology.orchestra.run.smithi081.stdout:Generating new minimal ceph.conf... 2024-04-12T21:04:19.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:19 smithi081 ceph-mon[31088]: from='client.? 172.21.15.81:0/1705479682' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:04:19.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.945+0000 7f5305e00700 1 Processor -- start 2024-04-12T21:04:19.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.945+0000 7f5305e00700 1 -- start start 2024-04-12T21:04:19.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.946+0000 7f5305e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530010aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:19.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.946+0000 7f5305e00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f530010aff0 con 0x7f530010a600 2024-04-12T21:04:19.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.946+0000 7f52fea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530010aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:19.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.946+0000 7f52fea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530010aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45526/0 (socket says 172.21.15.81:45526) 2024-04-12T21:04:19.712 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.946+0000 7f52fea00700 1 -- 172.21.15.81:0/409444186 learned_addr learned my addr 172.21.15.81:0/409444186 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:19.713 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.947+0000 7f52fea00700 1 -- 172.21.15.81:0/409444186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530010b760 con 0x7f530010a600 2024-04-12T21:04:19.716 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.947+0000 7f52fea00700 1 --2- 172.21.15.81:0/409444186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530010aa20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f52f00098a0 tx=0x7f52f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b768ae35156a43a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:19.716 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.947+0000 7f52fd600700 1 -- 172.21.15.81:0/409444186 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52f0004160 con 0x7f530010a600 2024-04-12T21:04:19.716 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.947+0000 7f52fd600700 1 -- 172.21.15.81:0/409444186 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f52f0004300 con 0x7f530010a600 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.947+0000 7f52fd600700 1 -- 172.21.15.81:0/409444186 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52f0011b50 con 0x7f530010a600 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.948+0000 7f5305e00700 1 -- 172.21.15.81:0/409444186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 msgr2=0x7f530010aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.948+0000 7f5305e00700 1 --2- 172.21.15.81:0/409444186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530010aa20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f52f00098a0 tx=0x7f52f0009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.948+0000 7f5305e00700 1 -- 172.21.15.81:0/409444186 shutdown_connections 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.948+0000 7f5305e00700 1 --2- 172.21.15.81:0/409444186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530010aa20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.948+0000 7f5305e00700 1 -- 172.21.15.81:0/409444186 >> 172.21.15.81:0/409444186 conn(0x7f530007dc20 msgr2=0x7f53001084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.948+0000 7f5305e00700 1 -- 172.21.15.81:0/409444186 shutdown_connections 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.948+0000 7f5305e00700 1 -- 172.21.15.81:0/409444186 wait complete. 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.949+0000 7f5305e00700 1 Processor -- start 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.950+0000 7f5305e00700 1 -- start start 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.950+0000 7f5305e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.950+0000 7f5305e00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f530010eb10 con 0x7f530010a600 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.950+0000 7f52fea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:19.717 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.950+0000 7f52fea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530019e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45534/0 (socket says 172.21.15.81:45534) 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.950+0000 7f52fea00700 1 -- 172.21.15.81:0/1705479682 learned_addr learned my addr 172.21.15.81:0/1705479682 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.951+0000 7f52fea00700 1 -- 172.21.15.81:0/1705479682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530019ea30 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.951+0000 7f52fea00700 1 --2- 172.21.15.81:0/1705479682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530019e4f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f52f0000c00 tx=0x7f52f0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.951+0000 7f5304a00700 1 -- 172.21.15.81:0/1705479682 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52f0003b20 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.951+0000 7f5304a00700 1 -- 172.21.15.81:0/1705479682 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f52f0003cc0 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.951+0000 7f5304a00700 1 -- 172.21.15.81:0/1705479682 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f52f0024910 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.951+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530019ecc0 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.952+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53001a19b0 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.952+0000 7f5304a00700 1 -- 172.21.15.81:0/1705479682 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f52f0022020 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.952+0000 7f5304a00700 1 -- 172.21.15.81:0/1705479682 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f52f0011e00 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.953+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5300064110 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:18.956+0000 7f5304a00700 1 -- 172.21.15.81:0/1705479682 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f52f002ddf0 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.031+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f53001a1b80 con 0x7f530010a600 2024-04-12T21:04:19.718 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.032+0000 7f5304a00700 1 -- 172.21.15.81:0/1705479682 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f52f0032030 con 0x7f530010a600 2024-04-12T21:04:19.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.033+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 msgr2=0x7f530019e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:19.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.033+0000 7f5305e00700 1 --2- 172.21.15.81:0/1705479682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530019e4f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f52f0000c00 tx=0x7f52f0011c10 comp rx=0 tx=0).stop 2024-04-12T21:04:19.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.034+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 shutdown_connections 2024-04-12T21:04:19.719 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.034+0000 7f5305e00700 1 --2- 172.21.15.81:0/1705479682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f530010a600 0x7f530019e4f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:19.720 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.034+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 >> 172.21.15.81:0/1705479682 conn(0x7f530007dc20 msgr2=0x7f5300107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:19.720 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.034+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 shutdown_connections 2024-04-12T21:04:19.720 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:19.034+0000 7f5305e00700 1 -- 172.21.15.81:0/1705479682 wait complete. 2024-04-12T21:04:19.720 INFO:teuthology.orchestra.run.smithi081.stdout:Restarting the monitor... 2024-04-12T21:04:20.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:19 smithi081 systemd[1]: Stopping Ceph mon.a for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:04:20.524 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:20 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31084]: 2024-04-12T21:04:20.213+0000 7f4c9a200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-12T21:04:20.524 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:20 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31084]: 2024-04-12T21:04:20.213+0000 7f4c9a200700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-12T21:04:20.524 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:20 smithi081 podman[31330]: 2024-04-12 21:04:20.265682122 +0000 UTC m=+0.180469429 container died 90df519c260d2cfc3bee253ceb4f35d35fed2c22250501f62156f5090d530f2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:04:20.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:20 smithi081 podman[31330]: 2024-04-12 21:04:20.663733118 +0000 UTC m=+0.578520482 container cleanup 90df519c260d2cfc3bee253ceb4f35d35fed2c22250501f62156f5090d530f2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-12T21:04:20.810 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:20 smithi081 bash[31330]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a 2024-04-12T21:04:21.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:20 smithi081 podman[31344]: 2024-04-12 21:04:20.814437742 +0000 UTC m=+0.532778522 container remove 90df519c260d2cfc3bee253ceb4f35d35fed2c22250501f62156f5090d530f2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:04:21.620 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-12T21:04:21.621 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 systemd[1]: Stopped Ceph mon.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:04:21.621 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.a.service: Consumed 1.816s CPU time. 2024-04-12T21:04:21.621 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 systemd[1]: Starting Ceph mon.a for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:04:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 podman[31435]: 2024-04-12 21:04:21.623881271 +0000 UTC m=+0.117197332 container create 2d2703ba7ffb62344b6fca11c3f34967f293e5e3956e99abf0e4db65bd6498f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:04:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 podman[31435]: 2024-04-12 21:04:21.540159236 +0000 UTC m=+0.033475275 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:04:21.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 podman[31435]: 2024-04-12 21:04:21.852247279 +0000 UTC m=+0.345563347 container init 2d2703ba7ffb62344b6fca11c3f34967f293e5e3956e99abf0e4db65bd6498f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:04:21.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 podman[31435]: 2024-04-12 21:04:21.863124238 +0000 UTC m=+0.356440306 container start 2d2703ba7ffb62344b6fca11c3f34967f293e5e3956e99abf0e4db65bd6498f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2) 2024-04-12T21:04:21.948 INFO:teuthology.orchestra.run.smithi081.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-12T21:04:22.217 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: set uid:gid to 167:167 (ceph:ceph) 2024-04-12T21:04:22.217 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev), process ceph-mon, pid 2 2024-04-12T21:04:22.217 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: pidfile_write: ignore empty --pid-file 2024-04-12T21:04:22.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: load: jerasure load: lrc 2024-04-12T21:04:22.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: RocksDB version: 7.9.2 2024-04-12T21:04:22.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Git sha 0 2024-04-12T21:04:22.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Compile date 2024-04-11 06:46:50 2024-04-12T21:04:22.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: DB SUMMARY 2024-04-12T21:04:22.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: DB Session ID: 0AA3Y8GB1NS2ONU8QT2H 2024-04-12T21:04:22.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: CURRENT file: CURRENT 2024-04-12T21:04:22.218 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: IDENTITY file: IDENTITY 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104078 ; 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.error_if_exists: 0 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.create_if_missing: 0 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.paranoid_checks: 1 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-12T21:04:22.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.env: 0x55758fd6bcc0 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.fs: PosixFileSystem 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.info_log: 0x5575cfebdc40 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_file_opening_threads: 16 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.statistics: (nil) 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.use_fsync: 0 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_log_file_size: 0 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-12T21:04:22.220 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.keep_log_file_num: 1000 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.recycle_log_file_num: 0 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.allow_fallocate: 1 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.allow_mmap_reads: 0 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.allow_mmap_writes: 0 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.use_direct_reads: 0 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.create_missing_column_families: 0 2024-04-12T21:04:22.221 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.db_log_dir: 2024-04-12T21:04:22.222 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.wal_dir: 2024-04-12T21:04:22.222 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.advise_random_on_open: 1 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.db_write_buffer_size: 0 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.write_buffer_manager: 0x5575cfe8e1e0 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-12T21:04:22.224 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.rate_limiter: (nil) 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.wal_recovery_mode: 2 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.enable_thread_tracking: 0 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.enable_pipelined_write: 0 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.unordered_write: 0 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.row_cache: None 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.wal_filter: None 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-12T21:04:22.225 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.allow_ingest_behind: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.two_write_queues: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.manual_wal_flush: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.wal_compression: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.atomic_flush: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.log_readahead_size: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.best_efforts_recovery: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.allow_data_in_errors: 0 2024-04-12T21:04:22.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.db_host_id: __hostname__ 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_background_jobs: 2 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_background_compactions: -1 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_subcompactions: 1 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_total_wal_size: 0 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_open_files: -1 2024-04-12T21:04:22.227 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bytes_per_sync: 0 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_readahead_size: 0 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_background_flushes: -1 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Compression algorithms supported: 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: kZSTD supported: 0 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: kXpressCompression supported: 0 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: kLZ4HCCompression supported: 1 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: kZlibCompression supported: 1 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: kSnappyCompression supported: 1 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: kLZ4Compression supported: 1 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: kBZip2Compression supported: 0 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-12T21:04:22.228 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.merge_operator: 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_filter: None 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_filter_factory: None 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.sst_partitioner_factory: None 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5575cfebde00) 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout: cache_index_and_filter_blocks: 1 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout: pin_top_level_index_and_filter: 1 2024-04-12T21:04:22.229 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_type: 0 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: data_block_index_type: 0 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_shortening: 1 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: checksum: 4 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: no_block_cache: 0 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache: 0x5575cfec1090 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_name: BinnedLRUCache 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_options: 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: capacity : 536870912 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: num_shard_bits : 4 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: strict_capacity_limit : 0 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: high_pri_pool_ratio: 0.000 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_cache_compressed: (nil) 2024-04-12T21:04:22.230 INFO:journalctl@ceph.mon.a.smithi081.stdout: persistent_cache: (nil) 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_size: 4096 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_size_deviation: 10 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_restart_interval: 16 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: index_block_restart_interval: 1 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: metadata_block_size: 4096 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: partition_filters: 0 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: use_delta_encoding: 1 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: filter_policy: bloomfilter 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: whole_key_filtering: 1 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: verify_compression: 0 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: read_amp_bytes_per_bit: 0 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: format_version: 5 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: enable_index_compression: 1 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: block_align: 0 2024-04-12T21:04:22.231 INFO:journalctl@ceph.mon.a.smithi081.stdout: max_auto_readahead_size: 262144 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout: prepopulate_block_cache: 0 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout: initial_auto_readahead_size: 8192 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout: num_file_reads_for_auto_readahead: 2 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.write_buffer_size: 33554432 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_write_buffer_number: 2 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression: NoCompression 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression: Disabled 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.prefix_extractor: nullptr 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.num_levels: 7 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-12T21:04:22.232 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.level: 32767 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.strategy: 0 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-12T21:04:22.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.enabled: false 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.target_file_size_base: 67108864 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-12T21:04:22.234 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.arena_block_size: 1048576 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.disable_auto_compactions: 0 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-12T21:04:22.235 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.inplace_update_support: 0 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.bloom_locality: 0 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.max_successive_merges: 0 2024-04-12T21:04:22.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.paranoid_file_checks: 0 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.force_consistency_checks: 1 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.report_bg_io_stats: 0 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.ttl: 2592000 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.enable_blob_files: false 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.min_blob_size: 0 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.blob_file_size: 268435456 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-12T21:04:22.237 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.blob_file_starting_level: 0 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b4b9e9a2-464a-46f9-a360-2bb95d1de5d3 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712955861921970, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712955861923848, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99298, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97257, "index_size": 297, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13923, "raw_average_key_size": 49, "raw_value_size": 89640, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1712955861, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b4b9e9a2-464a-46f9-a360-2bb95d1de5d3", "db_session_id": "0AA3Y8GB1NS2ONU8QT2H", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712955861924003, "job": 1, "event": "recovery_finished"} 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5575cffb0000 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: DB pointer 0x5575cff9a000 2024-04-12T21:04:22.238 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** DB Stats ** 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** Compaction Stats [default] ** 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: L0 2/0 98.83 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-12T21:04:22.239 INFO:journalctl@ceph.mon.a.smithi081.stdout: Sum 2/0 98.83 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** Compaction Stats [default] ** 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-12T21:04:22.240 INFO:journalctl@ceph.mon.a.smithi081.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout: Cumulative compaction: 0.00 GB write, 14.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout: Interval compaction: 0.00 GB write, 14.84 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout: Block cache BinnedLRUCache@0x5575cfec1090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: starting mon.a rank 0 at public addrs [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] at bind addrs [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a@-1(???) e1 preinit fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a@-1(???).mds e1 new map 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a@-1(???).mds e1 print_map 2024-04-12T21:04:22.241 INFO:journalctl@ceph.mon.a.smithi081.stdout: e1 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout: legacy client fscid: -1 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout: 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout: No filesystems configured 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 bash[31435]: 2d2703ba7ffb62344b6fca11c3f34967f293e5e3956e99abf0e4db65bd6498f6 2024-04-12T21:04:22.242 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 systemd[1]: Started Ceph mon.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: monmap epoch 1 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: last_changed 2024-04-12T21:04:11.886065+0000 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: min_mon_release 19 (squid) 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: election_strategy: 1 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: fsmap 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: osdmap e1: 0 total, 0 up, 0 in 2024-04-12T21:04:22.243 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:21 smithi081 ceph-mon[31453]: mgrmap e1: no daemons active 2024-04-12T21:04:23.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.526+0000 7fd224a00700 1 Processor -- start 2024-04-12T21:04:23.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.526+0000 7fd224a00700 1 -- start start 2024-04-12T21:04:23.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.526+0000 7fd224a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd2201087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:23.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.526+0000 7fd224a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd220108da0 con 0x7fd2201083b0 2024-04-12T21:04:23.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.527+0000 7fd21de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd2201087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:23.153 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.527+0000 7fd21de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd2201087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45550/0 (socket says 172.21.15.81:45550) 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.527+0000 7fd21de00700 1 -- 172.21.15.81:0/362988481 learned_addr learned my addr 172.21.15.81:0/362988481 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.527+0000 7fd21de00700 1 -- 172.21.15.81:0/362988481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd220109560 con 0x7fd2201083b0 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.528+0000 7fd21de00700 1 --2- 172.21.15.81:0/362988481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd2201087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd2100098a0 tx=0x7fd210009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf41aa0582ff5b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.528+0000 7fd21ca00700 1 -- 172.21.15.81:0/362988481 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd210004160 con 0x7fd2201083b0 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.528+0000 7fd21ca00700 1 -- 172.21.15.81:0/362988481 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd210004300 con 0x7fd2201083b0 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.528+0000 7fd21ca00700 1 -- 172.21.15.81:0/362988481 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd210011b50 con 0x7fd2201083b0 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.529+0000 7fd224a00700 1 -- 172.21.15.81:0/362988481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 msgr2=0x7fd2201087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.529+0000 7fd224a00700 1 --2- 172.21.15.81:0/362988481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd2201087d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd2100098a0 tx=0x7fd210009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.529+0000 7fd224a00700 1 -- 172.21.15.81:0/362988481 shutdown_connections 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.529+0000 7fd224a00700 1 --2- 172.21.15.81:0/362988481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd2201087d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.529+0000 7fd224a00700 1 -- 172.21.15.81:0/362988481 >> 172.21.15.81:0/362988481 conn(0x7fd220103a10 msgr2=0x7fd220105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:23.154 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.529+0000 7fd224a00700 1 -- 172.21.15.81:0/362988481 shutdown_connections 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.529+0000 7fd224a00700 1 -- 172.21.15.81:0/362988481 wait complete. 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.530+0000 7fd224a00700 1 Processor -- start 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.530+0000 7fd224a00700 1 -- start start 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.531+0000 7fd224a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd22019c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.531+0000 7fd224a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd22010c910 con 0x7fd2201083b0 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.531+0000 7fd21de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd22019c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.531+0000 7fd21de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd22019c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:45558/0 (socket says 172.21.15.81:45558) 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.531+0000 7fd21de00700 1 -- 172.21.15.81:0/775379227 learned_addr learned my addr 172.21.15.81:0/775379227 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.531+0000 7fd21de00700 1 -- 172.21.15.81:0/775379227 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd22019c850 con 0x7fd2201083b0 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.532+0000 7fd21de00700 1 --2- 172.21.15.81:0/775379227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd22019c310 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd210000c00 tx=0x7fd210011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.532+0000 7fd216a00700 1 -- 172.21.15.81:0/775379227 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd210003b20 con 0x7fd2201083b0 2024-04-12T21:04:23.155 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.532+0000 7fd216a00700 1 -- 172.21.15.81:0/775379227 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd210003cc0 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.532+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd22019cae0 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.532+0000 7fd216a00700 1 -- 172.21.15.81:0/775379227 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd210024910 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.532+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd22019f7d0 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.532+0000 7fd216a00700 1 -- 172.21.15.81:0/775379227 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd210022020 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.533+0000 7fd216a00700 1 -- 172.21.15.81:0/775379227 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd210011e00 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.533+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd220064110 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.536+0000 7fd216a00700 1 -- 172.21.15.81:0/775379227 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd210042e50 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.612+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fd22019f9a0 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.617+0000 7fd216a00700 1 -- 172.21.15.81:0/775379227 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fd210031020 con 0x7fd2201083b0 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.618+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 msgr2=0x7fd22019c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.618+0000 7fd224a00700 1 --2- 172.21.15.81:0/775379227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd22019c310 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd210000c00 tx=0x7fd210011c10 comp rx=0 tx=0).stop 2024-04-12T21:04:23.156 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.618+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 shutdown_connections 2024-04-12T21:04:23.157 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.618+0000 7fd224a00700 1 --2- 172.21.15.81:0/775379227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd2201083b0 0x7fd22019c310 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:23.157 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.618+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 >> 172.21.15.81:0/775379227 conn(0x7fd220103a10 msgr2=0x7fd2201044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:23.157 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.618+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 shutdown_connections 2024-04-12T21:04:23.157 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:22.618+0000 7fd224a00700 1 -- 172.21.15.81:0/775379227 wait complete. 2024-04-12T21:04:23.157 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-12T21:04:23.194 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-12T21:04:23.194 INFO:teuthology.orchestra.run.smithi081.stdout:Creating mgr... 2024-04-12T21:04:23.196 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-12T21:04:23.197 INFO:teuthology.orchestra.run.smithi081.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-12T21:04:23.625 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mgr.x 2024-04-12T21:04:23.625 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Failed to reset failed state of unit ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mgr.x.service: Unit ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mgr.x.service not loaded. 2024-04-12T21:04:23.874 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968.target.wants/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@.service. 2024-04-12T21:04:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:23 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/775379227' entity='client.admin' 2024-04-12T21:04:24.282 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:23 smithi081 systemd[1]: Starting Ceph mgr.x for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:04:24.568 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:24 smithi081 podman[31668]: 2024-04-12 21:04:24.190217975 +0000 UTC m=+0.032534459 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:04:24.568 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:24 smithi081 podman[31668]: 2024-04-12 21:04:24.309509233 +0000 UTC m=+0.151825745 container create eb2441a6844341d108c9845e025ec89bbac0c4059c759f80688a78267342cc50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:04:24.568 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:24 smithi081 podman[31668]: 2024-04-12 21:04:24.571690035 +0000 UTC m=+0.414006550 container init eb2441a6844341d108c9845e025ec89bbac0c4059c759f80688a78267342cc50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-12T21:04:24.665 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-12T21:04:24.665 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2024-04-12T21:04:24.680 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-12T21:04:24.680 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-04-12T21:04:24.680 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2024-04-12T21:04:24.681 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to enable service . firewalld.service is not available 2024-04-12T21:04:24.699 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-12T21:04:24.699 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2024-04-12T21:04:24.715 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-12T21:04:24.715 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-04-12T21:04:24.715 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2024-04-12T21:04:24.715 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-12T21:04:24.715 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr to start... 2024-04-12T21:04:24.715 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr... 2024-04-12T21:04:24.838 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:24 smithi081 podman[31668]: 2024-04-12 21:04:24.581949032 +0000 UTC m=+0.424265512 container start eb2441a6844341d108c9845e025ec89bbac0c4059c759f80688a78267342cc50 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-12T21:04:24.839 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:24 smithi081 bash[31668]: eb2441a6844341d108c9845e025ec89bbac0c4059c759f80688a78267342cc50 2024-04-12T21:04:24.839 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:24 smithi081 systemd[1]: Started Ceph mgr.x for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:04:25.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:25 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:25.370+0000 7ff724c68200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:04:25.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:25 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:25.496+0000 7ff724c68200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:04:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:25 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1972789113' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T21:04:26.073 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:26.075 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-04-12T21:04:26.075 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "11b8cc88-f910-11ee-bc8d-c7b262605968", 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-12T21:04:26.076 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-12T21:04:26.077 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-12T21:04:26.078 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-12T21:04:26.078 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-12T21:04:26.078 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-12T21:04:26.078 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-12T21:04:26.078 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-12T21:04:26.078 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-12T21:04:26.084 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:26.084 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-12T21:04:26.084 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:26.084 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-12T21:04:26.084 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-12T21:04:26.084 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2024-04-12T21:04:15.716575+0000", 2024-04-12T21:04:26.085 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.251+0000 7f09d4a00700 1 Processor -- start 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.251+0000 7f09d4a00700 1 -- start start 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.252+0000 7f09d4a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d007c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.252+0000 7f09d4a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09d007c830 con 0x7f09d007de80 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.252+0000 7f09cde00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d007c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.252+0000 7f09cde00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d007c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37296/0 (socket says 172.21.15.81:37296) 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.252+0000 7f09cde00700 1 -- 172.21.15.81:0/328287011 learned_addr learned my addr 172.21.15.81:0/328287011 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.253+0000 7f09cde00700 1 -- 172.21.15.81:0/328287011 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d007c9b0 con 0x7f09d007de80 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.253+0000 7f09cde00700 1 --2- 172.21.15.81:0/328287011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d007c2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f09c00098a0 tx=0x7f09c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06ee64abfe9b4d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:26.086 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.253+0000 7f09cca00700 1 -- 172.21.15.81:0/328287011 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09c0004160 con 0x7f09d007de80 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.253+0000 7f09cca00700 1 -- 172.21.15.81:0/328287011 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f09c0004300 con 0x7f09d007de80 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.253+0000 7f09cca00700 1 -- 172.21.15.81:0/328287011 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09c0011b50 con 0x7f09d007de80 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.254+0000 7f09d4a00700 1 -- 172.21.15.81:0/328287011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 msgr2=0x7f09d007c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.254+0000 7f09d4a00700 1 --2- 172.21.15.81:0/328287011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d007c2f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f09c00098a0 tx=0x7f09c0009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.255+0000 7f09d4a00700 1 -- 172.21.15.81:0/328287011 shutdown_connections 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.255+0000 7f09d4a00700 1 --2- 172.21.15.81:0/328287011 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d007c2f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.255+0000 7f09d4a00700 1 -- 172.21.15.81:0/328287011 >> 172.21.15.81:0/328287011 conn(0x7f09d0103a00 msgr2=0x7f09d0105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.255+0000 7f09d4a00700 1 -- 172.21.15.81:0/328287011 shutdown_connections 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.255+0000 7f09d4a00700 1 -- 172.21.15.81:0/328287011 wait complete. 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.256+0000 7f09d4a00700 1 Processor -- start 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.256+0000 7f09d4a00700 1 -- start start 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.257+0000 7f09d4a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.257+0000 7f09d4a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f09d010e950 con 0x7f09d007de80 2024-04-12T21:04:26.087 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.257+0000 7f09cde00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.257+0000 7f09cde00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37306/0 (socket says 172.21.15.81:37306) 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.257+0000 7f09cde00700 1 -- 172.21.15.81:0/1972789113 learned_addr learned my addr 172.21.15.81:0/1972789113 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.257+0000 7f09cde00700 1 -- 172.21.15.81:0/1972789113 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d019eb40 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.258+0000 7f09cde00700 1 --2- 172.21.15.81:0/1972789113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d019e600 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f09c0011b60 tx=0x7f09c0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.258+0000 7f09c6a00700 1 -- 172.21.15.81:0/1972789113 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09c0003b20 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.258+0000 7f09c6a00700 1 -- 172.21.15.81:0/1972789113 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f09c0003cc0 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.258+0000 7f09c6a00700 1 -- 172.21.15.81:0/1972789113 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09c0024840 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.258+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d019edd0 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.258+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d01a1ac0 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.258+0000 7f09c6a00700 1 -- 172.21.15.81:0/1972789113 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f09c0022020 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.259+0000 7f09c6a00700 1 -- 172.21.15.81:0/1972789113 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f09c0011e00 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.259+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09d0064110 con 0x7f09d007de80 2024-04-12T21:04:26.088 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.262+0000 7f09c6a00700 1 -- 172.21.15.81:0/1972789113 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f09c0043e50 con 0x7f09d007de80 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.343+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f09d01a1fb0 con 0x7f09d007de80 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.344+0000 7f09c6a00700 1 -- 172.21.15.81:0/1972789113 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f09c0032030 con 0x7f09d007de80 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.345+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 msgr2=0x7f09d019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.345+0000 7f09d4a00700 1 --2- 172.21.15.81:0/1972789113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d019e600 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f09c0011b60 tx=0x7f09c0011c40 comp rx=0 tx=0).stop 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.345+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 shutdown_connections 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.345+0000 7f09d4a00700 1 --2- 172.21.15.81:0/1972789113 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f09d007de80 0x7f09d019e600 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.345+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 >> 172.21.15.81:0/1972789113 conn(0x7f09d0103a00 msgr2=0x7f09d0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.345+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 shutdown_connections 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:25.345+0000 7f09d4a00700 1 -- 172.21.15.81:0/1972789113 wait complete. 2024-04-12T21:04:26.089 INFO:teuthology.orchestra.run.smithi081.stdout:mgr not available, waiting (1/15)... 2024-04-12T21:04:26.631 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:26.329+0000 7ff724c68200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:04:26.631 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:26.450+0000 7ff724c68200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:04:26.631 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:26.634+0000 7ff724c68200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:04:26.902 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:26.759+0000 7ff724c68200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:04:26.902 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:26.868+0000 7ff724c68200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:04:27.465 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:27.211+0000 7ff724c68200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T21:04:27.465 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:27.353+0000 7ff724c68200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:04:27.807 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:27.468+0000 7ff724c68200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:04:27.807 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:27.592+0000 7ff724c68200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:04:27.807 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:27.810+0000 7ff724c68200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:04:29.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:28 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3441925074' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T21:04:29.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:29.151+0000 7ff724c68200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:04:29.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:29.326+0000 7ff724c68200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:04:29.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:29.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-04-12T21:04:29.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "11b8cc88-f910-11ee-bc8d-c7b262605968", 2024-04-12T21:04:29.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2024-04-12T21:04:29.510 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-12T21:04:29.511 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:29.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-12T21:04:29.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:29.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-12T21:04:29.512 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-12T21:04:29.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-12T21:04:29.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-12T21:04:29.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-12T21:04:29.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-12T21:04:29.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:29.520 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:29.521 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:29.522 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2024-04-12T21:04:15.716575+0000", 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.661+0000 7fb64e800700 1 Processor -- start 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.661+0000 7fb64e800700 1 -- start start 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.662+0000 7fb64e800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64810b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.662+0000 7fb64e800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb64807be60 con 0x7fb648108c60 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.662+0000 7fb64d400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64810b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.662+0000 7fb64d400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64810b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37318/0 (socket says 172.21.15.81:37318) 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.662+0000 7fb64d400700 1 -- 172.21.15.81:0/1582416639 learned_addr learned my addr 172.21.15.81:0/1582416639 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.663+0000 7fb64d400700 1 -- 172.21.15.81:0/1582416639 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64810b590 con 0x7fb648108c60 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.663+0000 7fb64d400700 1 --2- 172.21.15.81:0/1582416639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64810b050 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0098a0 tx=0x7fb63c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933312da2f6b638c server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:29.523 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.663+0000 7fb647e00700 1 -- 172.21.15.81:0/1582416639 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb63c004160 con 0x7fb648108c60 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.663+0000 7fb647e00700 1 -- 172.21.15.81:0/1582416639 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb63c004300 con 0x7fb648108c60 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.663+0000 7fb647e00700 1 -- 172.21.15.81:0/1582416639 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb63c011b50 con 0x7fb648108c60 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.664+0000 7fb64e800700 1 -- 172.21.15.81:0/1582416639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 msgr2=0x7fb64810b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.664+0000 7fb64e800700 1 --2- 172.21.15.81:0/1582416639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64810b050 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0098a0 tx=0x7fb63c009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.664+0000 7fb64e800700 1 -- 172.21.15.81:0/1582416639 shutdown_connections 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.664+0000 7fb64e800700 1 --2- 172.21.15.81:0/1582416639 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64810b050 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.664+0000 7fb64e800700 1 -- 172.21.15.81:0/1582416639 >> 172.21.15.81:0/1582416639 conn(0x7fb648102670 msgr2=0x7fb648104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.664+0000 7fb64e800700 1 -- 172.21.15.81:0/1582416639 shutdown_connections 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.664+0000 7fb64e800700 1 -- 172.21.15.81:0/1582416639 wait complete. 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.665+0000 7fb64e800700 1 Processor -- start 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.665+0000 7fb64e800700 1 -- start start 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.665+0000 7fb64e800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.665+0000 7fb64e800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb64810c560 con 0x7fb648108c60 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.666+0000 7fb64d400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:29.524 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.666+0000 7fb64d400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64819e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37324/0 (socket says 172.21.15.81:37324) 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.666+0000 7fb64d400700 1 -- 172.21.15.81:0/3441925074 learned_addr learned my addr 172.21.15.81:0/3441925074 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.666+0000 7fb64d400700 1 -- 172.21.15.81:0/3441925074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64819e8a0 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.666+0000 7fb64d400700 1 --2- 172.21.15.81:0/3441925074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64819e360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb63c011b60 tx=0x7fb63c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.667+0000 7fb646000700 1 -- 172.21.15.81:0/3441925074 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb63c003b20 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.667+0000 7fb646000700 1 -- 172.21.15.81:0/3441925074 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb63c003cc0 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.667+0000 7fb646000700 1 -- 172.21.15.81:0/3441925074 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb63c024840 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.667+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64819eb30 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.667+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6481a1820 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.667+0000 7fb646000700 1 -- 172.21.15.81:0/3441925074 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb63c022020 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.667+0000 7fb646000700 1 -- 172.21.15.81:0/3441925074 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb63c011e00 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.668+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb648064110 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.671+0000 7fb646000700 1 -- 172.21.15.81:0/3441925074 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb63c043e50 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.750+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb6481a1f00 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.751+0000 7fb646000700 1 -- 172.21.15.81:0/3441925074 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fb63c032020 con 0x7fb648108c60 2024-04-12T21:04:29.525 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.752+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 msgr2=0x7fb64819e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:29.526 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.752+0000 7fb64e800700 1 --2- 172.21.15.81:0/3441925074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64819e360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb63c011b60 tx=0x7fb63c011c40 comp rx=0 tx=0).stop 2024-04-12T21:04:29.526 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.752+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 shutdown_connections 2024-04-12T21:04:29.526 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.752+0000 7fb64e800700 1 --2- 172.21.15.81:0/3441925074 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb648108c60 0x7fb64819e360 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:29.526 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.752+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 >> 172.21.15.81:0/3441925074 conn(0x7fb648102670 msgr2=0x7fb6481030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:29.526 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.752+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 shutdown_connections 2024-04-12T21:04:29.526 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:28.752+0000 7fb64e800700 1 -- 172.21.15.81:0/3441925074 wait complete. 2024-04-12T21:04:29.526 INFO:teuthology.orchestra.run.smithi081.stdout:mgr not available, waiting (2/15)... 2024-04-12T21:04:29.859 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:29.745+0000 7ff724c68200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:04:29.859 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:29.862+0000 7ff724c68200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:04:30.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:30 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:30.118+0000 7ff724c68200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:04:30.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:30 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:30.361+0000 7ff724c68200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:04:31.050 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:30 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:30.735+0000 7ff724c68200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:04:31.050 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:31.053+0000 7ff724c68200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:04:31.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:31.360+0000 7ff724c68200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:04:32.576 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:32 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/636208258' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T21:04:32.576 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:32 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:32.422+0000 7ff724c68200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:04:32.895 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:32.895 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-04-12T21:04:32.895 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "11b8cc88-f910-11ee-bc8d-c7b262605968", 2024-04-12T21:04:32.895 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2024-04-12T21:04:32.895 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-12T21:04:32.896 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-12T21:04:32.897 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:32.897 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-12T21:04:32.897 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:32.897 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-12T21:04:32.897 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-12T21:04:32.897 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-12T21:04:32.905 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-12T21:04:32.906 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2024-04-12T21:04:15.716575+0000", 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-04-12T21:04:32.907 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.197+0000 7f712f200700 1 Processor -- start 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.197+0000 7f712f200700 1 -- start start 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.198+0000 7f712f200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713007c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.198+0000 7f712f200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f713007c870 con 0x7f713007de30 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.198+0000 7f712de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713007c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.198+0000 7f712de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713007c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37336/0 (socket says 172.21.15.81:37336) 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.198+0000 7f712de00700 1 -- 172.21.15.81:0/879352727 learned_addr learned my addr 172.21.15.81:0/879352727 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.199+0000 7f712de00700 1 -- 172.21.15.81:0/879352727 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713007c9f0 con 0x7f713007de30 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.199+0000 7f712de00700 1 --2- 172.21.15.81:0/879352727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713007c330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71200098a0 tx=0x7f7120009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5eb0a685c474380 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.199+0000 7f712d400700 1 -- 172.21.15.81:0/879352727 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7120004160 con 0x7f713007de30 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.199+0000 7f712d400700 1 -- 172.21.15.81:0/879352727 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7120004300 con 0x7f713007de30 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.199+0000 7f712d400700 1 -- 172.21.15.81:0/879352727 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7120011b50 con 0x7f713007de30 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.200+0000 7f712f200700 1 -- 172.21.15.81:0/879352727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 msgr2=0x7f713007c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:32.908 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.200+0000 7f712f200700 1 --2- 172.21.15.81:0/879352727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713007c330 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f71200098a0 tx=0x7f7120009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.200+0000 7f712f200700 1 -- 172.21.15.81:0/879352727 shutdown_connections 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.200+0000 7f712f200700 1 --2- 172.21.15.81:0/879352727 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713007c330 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.200+0000 7f712f200700 1 -- 172.21.15.81:0/879352727 >> 172.21.15.81:0/879352727 conn(0x7f7130103820 msgr2=0x7f7130105c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.200+0000 7f712f200700 1 -- 172.21.15.81:0/879352727 shutdown_connections 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.200+0000 7f712f200700 1 -- 172.21.15.81:0/879352727 wait complete. 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.201+0000 7f712f200700 1 Processor -- start 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.201+0000 7f712f200700 1 -- start start 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.202+0000 7f712f200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713019e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.202+0000 7f712f200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f713010e790 con 0x7f713007de30 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.202+0000 7f712de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713019e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.202+0000 7f712de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713019e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37340/0 (socket says 172.21.15.81:37340) 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.202+0000 7f712de00700 1 -- 172.21.15.81:0/636208258 learned_addr learned my addr 172.21.15.81:0/636208258 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.202+0000 7f712de00700 1 -- 172.21.15.81:0/636208258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713019e840 con 0x7f713007de30 2024-04-12T21:04:32.909 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.203+0000 7f712de00700 1 --2- 172.21.15.81:0/636208258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713019e300 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7120011b60 tx=0x7f7120011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.203+0000 7f7126a00700 1 -- 172.21.15.81:0/636208258 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7120003b20 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.203+0000 7f7126a00700 1 -- 172.21.15.81:0/636208258 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7120003cc0 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.203+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713019ead0 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.203+0000 7f7126a00700 1 -- 172.21.15.81:0/636208258 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7120024840 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.203+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71301a17c0 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.203+0000 7f7126a00700 1 -- 172.21.15.81:0/636208258 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7120022020 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.204+0000 7f7126a00700 1 -- 172.21.15.81:0/636208258 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7120011e00 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.204+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7130064110 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.207+0000 7f7126a00700 1 -- 172.21.15.81:0/636208258 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7120043e50 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.285+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f71301a1ea0 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.286+0000 7f7126a00700 1 -- 172.21.15.81:0/636208258 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f7120032030 con 0x7f713007de30 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.287+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 msgr2=0x7f713019e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:32.910 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.287+0000 7f712f200700 1 --2- 172.21.15.81:0/636208258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713019e300 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7120011b60 tx=0x7f7120011c40 comp rx=0 tx=0).stop 2024-04-12T21:04:32.911 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.288+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 shutdown_connections 2024-04-12T21:04:32.911 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.288+0000 7f712f200700 1 --2- 172.21.15.81:0/636208258 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f713007de30 0x7f713019e300 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:32.911 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.288+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 >> 172.21.15.81:0/636208258 conn(0x7f7130103820 msgr2=0x7f7130104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:32.911 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.288+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 shutdown_connections 2024-04-12T21:04:32.911 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:32.288+0000 7f712f200700 1 -- 172.21.15.81:0/636208258 wait complete. 2024-04-12T21:04:32.911 INFO:teuthology.orchestra.run.smithi081.stdout:mgr not available, waiting (3/15)... 2024-04-12T21:04:33.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:33.141+0000 7ff724c68200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:04:33.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:33.270+0000 7ff724c68200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:04:33.902 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:33.552+0000 7ff724c68200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: Activating manager daemon x 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: mgrmap e2: x(active, starting, since 0.00669636s) 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: Manager daemon x is now available 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T21:04:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' 2024-04-12T21:04:33.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' 2024-04-12T21:04:33.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:33 smithi081 ceph-mon[31453]: from='mgr.14100 172.21.15.81:0/1813576593' entity='mgr.x' 2024-04-12T21:04:35.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:35 smithi081 ceph-mon[31453]: mgrmap e3: x(active, since 1.02235s) 2024-04-12T21:04:36.426 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:36.426 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-04-12T21:04:36.427 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsid": "11b8cc88-f910-11ee-bc8d-c7b262605968", 2024-04-12T21:04:36.427 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "health": { 2024-04-12T21:04:36.429 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-12T21:04:36.429 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-12T21:04:36.429 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-12T21:04:36.429 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:36.429 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-12T21:04:36.429 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 0 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "a" 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-12T21:04:36.430 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:36.431 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-12T21:04:36.431 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-12T21:04:36.431 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-12T21:04:36.432 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-12T21:04:36.432 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-12T21:04:36.432 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-12T21:04:36.432 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:36.432 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-12T21:04:36.432 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-12T21:04:36.432 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-12T21:04:36.432 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-12T21:04:36.433 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-12T21:04:36.433 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-12T21:04:36.441 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "restful" 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ], 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "modified": "2024-04-12T21:04:15.716575+0000", 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout }, 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-12T21:04:36.442 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.488+0000 7efca5c00700 1 Processor -- start 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.488+0000 7efca5c00700 1 -- start start 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.489+0000 7efca5c00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca01087e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.489+0000 7efca5c00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efca0108db0 con 0x7efca01083c0 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.489+0000 7efc9ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca01087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.489+0000 7efc9ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca01087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34244/0 (socket says 172.21.15.81:34244) 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.489+0000 7efc9ea00700 1 -- 172.21.15.81:0/783447142 learned_addr learned my addr 172.21.15.81:0/783447142 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.490+0000 7efc9ea00700 1 -- 172.21.15.81:0/783447142 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca0109570 con 0x7efca01083c0 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.490+0000 7efc9ea00700 1 --2- 172.21.15.81:0/783447142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca01087e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efc900098a0 tx=0x7efc90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a6cdc2df824e4b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.490+0000 7efc9d600700 1 -- 172.21.15.81:0/783447142 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc90004160 con 0x7efca01083c0 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.490+0000 7efc9d600700 1 -- 172.21.15.81:0/783447142 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efc90004300 con 0x7efca01083c0 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.490+0000 7efc9d600700 1 -- 172.21.15.81:0/783447142 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc90011b50 con 0x7efca01083c0 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.491+0000 7efca5c00700 1 -- 172.21.15.81:0/783447142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 msgr2=0x7efca01087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:36.443 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.491+0000 7efca5c00700 1 --2- 172.21.15.81:0/783447142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca01087e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efc900098a0 tx=0x7efc90009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.492+0000 7efca5c00700 1 -- 172.21.15.81:0/783447142 shutdown_connections 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.492+0000 7efca5c00700 1 --2- 172.21.15.81:0/783447142 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca01087e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.492+0000 7efca5c00700 1 -- 172.21.15.81:0/783447142 >> 172.21.15.81:0/783447142 conn(0x7efca0103a40 msgr2=0x7efca0105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.492+0000 7efca5c00700 1 -- 172.21.15.81:0/783447142 shutdown_connections 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.492+0000 7efca5c00700 1 -- 172.21.15.81:0/783447142 wait complete. 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.493+0000 7efca5c00700 1 Processor -- start 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.493+0000 7efca5c00700 1 -- start start 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.494+0000 7efca5c00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca007db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.494+0000 7efca5c00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efca010c920 con 0x7efca01083c0 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.494+0000 7efc9ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca007db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.494+0000 7efc9ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca007db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34254/0 (socket says 172.21.15.81:34254) 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.494+0000 7efc9ea00700 1 -- 172.21.15.81:0/3002359587 learned_addr learned my addr 172.21.15.81:0/3002359587 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.495+0000 7efc9ea00700 1 -- 172.21.15.81:0/3002359587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca007e0c0 con 0x7efca01083c0 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.495+0000 7efc9ea00700 1 --2- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca007db80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc90009fb0 tx=0x7efc90011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:36.444 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.495+0000 7efc97400700 1 -- 172.21.15.81:0/3002359587 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc900038b0 con 0x7efca01083c0 2024-04-12T21:04:36.445 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.495+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca007c2b0 con 0x7efca01083c0 2024-04-12T21:04:36.445 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.495+0000 7efc97400700 1 -- 172.21.15.81:0/3002359587 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efc90011e50 con 0x7efca01083c0 2024-04-12T21:04:36.445 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.495+0000 7efc97400700 1 -- 172.21.15.81:0/3002359587 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc90019550 con 0x7efca01083c0 2024-04-12T21:04:36.445 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.495+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca007c7f0 con 0x7efca01083c0 2024-04-12T21:04:36.445 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.497+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca0064110 con 0x7efca01083c0 2024-04-12T21:04:36.445 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.497+0000 7efc97400700 1 -- 172.21.15.81:0/3002359587 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7efc90003a50 con 0x7efca01083c0 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.498+0000 7efc97400700 1 --2- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7efc84039450 0x7efc8403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.498+0000 7efc97400700 1 -- 172.21.15.81:0/3002359587 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efc9004d6b0 con 0x7efca01083c0 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.498+0000 7efc9e000700 1 --2- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7efc84039450 0x7efc8403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.499+0000 7efc9e000700 1 --2- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7efc84039450 0x7efc8403b910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efc8c006fd0 tx=0x7efc8c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.504+0000 7efc97400700 1 -- 172.21.15.81:0/3002359587 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc9000fda0 con 0x7efca01083c0 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.571+0000 7efc97400700 1 -- 172.21.15.81:0/3002359587 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7efc90019800 con 0x7efca01083c0 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.809+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7efca007cfe0 con 0x7efca01083c0 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.810+0000 7efc97400700 1 -- 172.21.15.81:0/3002359587 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7efc90044fb0 con 0x7efca01083c0 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.814+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7efc84039450 msgr2=0x7efc8403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.814+0000 7efca5c00700 1 --2- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7efc84039450 0x7efc8403b910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7efc8c006fd0 tx=0x7efc8c006e40 comp rx=0 tx=0).stop 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.814+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 msgr2=0x7efca007db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.814+0000 7efca5c00700 1 --2- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca007db80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc90009fb0 tx=0x7efc90011bf0 comp rx=0 tx=0).stop 2024-04-12T21:04:36.446 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.814+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 shutdown_connections 2024-04-12T21:04:36.447 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.814+0000 7efca5c00700 1 --2- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7efc84039450 0x7efc8403b910 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:36.447 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.814+0000 7efca5c00700 1 --2- 172.21.15.81:0/3002359587 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efca01083c0 0x7efca007db80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:36.447 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.814+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 >> 172.21.15.81:0/3002359587 conn(0x7efca0103a40 msgr2=0x7efca0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:36.447 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.815+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 shutdown_connections 2024-04-12T21:04:36.447 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:35.815+0000 7efca5c00700 1 -- 172.21.15.81:0/3002359587 wait complete. 2024-04-12T21:04:36.447 INFO:teuthology.orchestra.run.smithi081.stdout:mgr is available 2024-04-12T21:04:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:36 smithi081 ceph-mon[31453]: mgrmap e4: x(active, since 2s) 2024-04-12T21:04:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:36 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3002359587' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-12T21:04:37.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:37 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/170577061' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-12T21:04:38.268 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [global] 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout fsid = 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout [osd] 2024-04-12T21:04:38.269 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-12T21:04:38.270 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-12T21:04:38.270 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-12T21:04:38.270 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.070+0000 7ff637200700 1 Processor -- start 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.070+0000 7ff637200700 1 -- start start 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.071+0000 7ff637200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff638106eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.071+0000 7ff637200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff63807be20 con 0x7ff638106a90 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.071+0000 7ff635e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff638106eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.071+0000 7ff635e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff638106eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34258/0 (socket says 172.21.15.81:34258) 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.071+0000 7ff635e00700 1 -- 172.21.15.81:0/1811586871 learned_addr learned my addr 172.21.15.81:0/1811586871 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:38.276 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.072+0000 7ff635e00700 1 -- 172.21.15.81:0/1811586871 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6381073f0 con 0x7ff638106a90 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.072+0000 7ff635e00700 1 --2- 172.21.15.81:0/1811586871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff638106eb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6280098a0 tx=0x7ff628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5241f6db888d4bc server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.072+0000 7ff634a00700 1 -- 172.21.15.81:0/1811586871 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff628004160 con 0x7ff638106a90 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff634a00700 1 -- 172.21.15.81:0/1811586871 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff628004300 con 0x7ff638106a90 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff634a00700 1 -- 172.21.15.81:0/1811586871 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff628011b50 con 0x7ff638106a90 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff637200700 1 -- 172.21.15.81:0/1811586871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 msgr2=0x7ff638106eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff637200700 1 --2- 172.21.15.81:0/1811586871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff638106eb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6280098a0 tx=0x7ff628009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff637200700 1 -- 172.21.15.81:0/1811586871 shutdown_connections 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff637200700 1 --2- 172.21.15.81:0/1811586871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff638106eb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff637200700 1 -- 172.21.15.81:0/1811586871 >> 172.21.15.81:0/1811586871 conn(0x7ff638102670 msgr2=0x7ff638104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff637200700 1 -- 172.21.15.81:0/1811586871 shutdown_connections 2024-04-12T21:04:38.277 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.073+0000 7ff637200700 1 -- 172.21.15.81:0/1811586871 wait complete. 2024-04-12T21:04:38.278 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.074+0000 7ff637200700 1 Processor -- start 2024-04-12T21:04:38.278 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.075+0000 7ff637200700 1 -- start start 2024-04-12T21:04:38.278 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.075+0000 7ff637200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff63810c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:38.278 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.075+0000 7ff637200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff638108480 con 0x7ff638106a90 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.075+0000 7ff635e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff63810c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.075+0000 7ff635e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff63810c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34274/0 (socket says 172.21.15.81:34274) 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.075+0000 7ff635e00700 1 -- 172.21.15.81:0/170577061 learned_addr learned my addr 172.21.15.81:0/170577061 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.076+0000 7ff635e00700 1 -- 172.21.15.81:0/170577061 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63810ca70 con 0x7ff638106a90 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.076+0000 7ff635e00700 1 --2- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff63810c530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff628011b60 tx=0x7ff628011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.076+0000 7ff62ea00700 1 -- 172.21.15.81:0/170577061 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff628003b20 con 0x7ff638106a90 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.076+0000 7ff62ea00700 1 -- 172.21.15.81:0/170577061 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff628003cc0 con 0x7ff638106a90 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.076+0000 7ff62ea00700 1 -- 172.21.15.81:0/170577061 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff628024840 con 0x7ff638106a90 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.076+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63810cd00 con 0x7ff638106a90 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.077+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6381092c0 con 0x7ff638106a90 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.078+0000 7ff62ea00700 1 -- 172.21.15.81:0/170577061 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7ff628022020 con 0x7ff638106a90 2024-04-12T21:04:38.279 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.078+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63810ced0 con 0x7ff638106a90 2024-04-12T21:04:38.280 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.078+0000 7ff62ea00700 1 --2- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7ff61c0395b0 0x7ff61c03ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:38.280 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.078+0000 7ff62ea00700 1 -- 172.21.15.81:0/170577061 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff628011e00 con 0x7ff638106a90 2024-04-12T21:04:38.280 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.078+0000 7ff635400700 1 --2- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7ff61c0395b0 0x7ff61c03ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:38.280 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.079+0000 7ff635400700 1 --2- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7ff61c0395b0 0x7ff61c03ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff624006fd0 tx=0x7ff624006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:38.280 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.084+0000 7ff62ea00700 1 -- 172.21.15.81:0/170577061 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff63810ced0 con 0x7ff638106a90 2024-04-12T21:04:38.280 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.304+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7ff638197a50 con 0x7ff638106a90 2024-04-12T21:04:38.280 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.306+0000 7ff62ea00700 1 -- 172.21.15.81:0/170577061 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7ff62804d2f0 con 0x7ff638106a90 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.310+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7ff61c0395b0 msgr2=0x7ff61c03ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.310+0000 7ff637200700 1 --2- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7ff61c0395b0 0x7ff61c03ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff624006fd0 tx=0x7ff624006e40 comp rx=0 tx=0).stop 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.310+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 msgr2=0x7ff63810c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.310+0000 7ff637200700 1 --2- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff63810c530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff628011b60 tx=0x7ff628011c40 comp rx=0 tx=0).stop 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.311+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 shutdown_connections 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.311+0000 7ff637200700 1 --2- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7ff61c0395b0 0x7ff61c03ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.311+0000 7ff637200700 1 --2- 172.21.15.81:0/170577061 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff638106a90 0x7ff63810c530 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.311+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 >> 172.21.15.81:0/170577061 conn(0x7ff638102670 msgr2=0x7ff638194ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:38.281 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.311+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 shutdown_connections 2024-04-12T21:04:38.282 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:37.311+0000 7ff637200700 1 -- 172.21.15.81:0/170577061 wait complete. 2024-04-12T21:04:38.282 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling cephadm module... 2024-04-12T21:04:39.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:39 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/545258527' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-12T21:04:39.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:39 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: ignoring --setuser ceph since I am not root 2024-04-12T21:04:39.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:39 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: ignoring --setgroup ceph since I am not root 2024-04-12T21:04:39.886 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.851+0000 7fd405200700 1 Processor -- start 2024-04-12T21:04:39.886 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.852+0000 7fd405200700 1 -- start start 2024-04-12T21:04:39.886 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.852+0000 7fd405200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd40010b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:39.886 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.852+0000 7fd405200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd40007be10 con 0x7fd400108e90 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.852+0000 7fd3fde00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd40010b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.853+0000 7fd3fde00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd40010b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34284/0 (socket says 172.21.15.81:34284) 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.853+0000 7fd3fde00700 1 -- 172.21.15.81:0/315910166 learned_addr learned my addr 172.21.15.81:0/315910166 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.853+0000 7fd3fde00700 1 -- 172.21.15.81:0/315910166 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd40010b7c0 con 0x7fd400108e90 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.853+0000 7fd3fde00700 1 --2- 172.21.15.81:0/315910166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd40010b280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00098a0 tx=0x7fd3f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1c5fcf253b3ef8 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.853+0000 7fd3fca00700 1 -- 172.21.15.81:0/315910166 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3f0004160 con 0x7fd400108e90 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.853+0000 7fd3fca00700 1 -- 172.21.15.81:0/315910166 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd3f0004300 con 0x7fd400108e90 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.853+0000 7fd3fca00700 1 -- 172.21.15.81:0/315910166 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3f0011b50 con 0x7fd400108e90 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.854+0000 7fd405200700 1 -- 172.21.15.81:0/315910166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 msgr2=0x7fd40010b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.854+0000 7fd405200700 1 --2- 172.21.15.81:0/315910166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd40010b280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00098a0 tx=0x7fd3f0009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.854+0000 7fd405200700 1 -- 172.21.15.81:0/315910166 shutdown_connections 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.854+0000 7fd405200700 1 --2- 172.21.15.81:0/315910166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd40010b280 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.854+0000 7fd405200700 1 -- 172.21.15.81:0/315910166 >> 172.21.15.81:0/315910166 conn(0x7fd400102860 msgr2=0x7fd400104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:39.887 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.854+0000 7fd405200700 1 -- 172.21.15.81:0/315910166 shutdown_connections 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.854+0000 7fd405200700 1 -- 172.21.15.81:0/315910166 wait complete. 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.856+0000 7fd405200700 1 Processor -- start 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.856+0000 7fd405200700 1 -- start start 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.856+0000 7fd405200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd400108710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.856+0000 7fd405200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd40010c780 con 0x7fd400108e90 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.856+0000 7fd3fde00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd400108710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.857+0000 7fd3fde00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd400108710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34300/0 (socket says 172.21.15.81:34300) 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.857+0000 7fd3fde00700 1 -- 172.21.15.81:0/545258527 learned_addr learned my addr 172.21.15.81:0/545258527 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.857+0000 7fd3fde00700 1 -- 172.21.15.81:0/545258527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd400108c50 con 0x7fd400108e90 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.857+0000 7fd3fde00700 1 --2- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd400108710 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009fb0 tx=0x7fd3f0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.857+0000 7fd3f6a00700 1 -- 172.21.15.81:0/545258527 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3f00038b0 con 0x7fd400108e90 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.857+0000 7fd3f6a00700 1 -- 172.21.15.81:0/545258527 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd3f0011e50 con 0x7fd400108e90 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.857+0000 7fd3f6a00700 1 -- 172.21.15.81:0/545258527 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3f0019420 con 0x7fd400108e90 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.858+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd400106e40 con 0x7fd400108e90 2024-04-12T21:04:39.888 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.858+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd400107380 con 0x7fd400108e90 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.859+0000 7fd3f6a00700 1 -- 172.21.15.81:0/545258527 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fd3f0003a50 con 0x7fd400108e90 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.859+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd400064110 con 0x7fd400108e90 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.859+0000 7fd3f6a00700 1 --2- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7fd3e4039560 0x7fd3e403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.859+0000 7fd3f6a00700 1 -- 172.21.15.81:0/545258527 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd3f004d880 con 0x7fd400108e90 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.859+0000 7fd3fd400700 1 --2- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7fd3e4039560 0x7fd3e403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.860+0000 7fd3fd400700 1 --2- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7fd3e4039560 0x7fd3e403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec006fd0 tx=0x7fd3ec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:38.865+0000 7fd3f6a00700 1 -- 172.21.15.81:0/545258527 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd3f00172b0 con 0x7fd400108e90 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.131+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fd400107600 con 0x7fd400108e90 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.184+0000 7fd3f6a00700 1 -- 172.21.15.81:0/545258527 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fd3f004d2a0 con 0x7fd400108e90 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.187+0000 7fd3f6a00700 1 -- 172.21.15.81:0/545258527 <== mon.0 v2:172.21.15.81:3300/0 8 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7fd3f004fcc0 con 0x7fd400108e90 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7fd3e4039560 msgr2=0x7fd3e403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 --2- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7fd3e4039560 0x7fd3e403ba20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec006fd0 tx=0x7fd3ec006e40 comp rx=0 tx=0).stop 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 msgr2=0x7fd400108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:39.889 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 --2- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd400108710 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3f0009fb0 tx=0x7fd3f0011bf0 comp rx=0 tx=0).stop 2024-04-12T21:04:39.890 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 shutdown_connections 2024-04-12T21:04:39.890 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 --2- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7fd3e4039560 0x7fd3e403ba20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:39.890 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 --2- 172.21.15.81:0/545258527 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd400108e90 0x7fd400108710 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:39.890 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 >> 172.21.15.81:0/545258527 conn(0x7fd400102860 msgr2=0x7fd400103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:39.890 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 shutdown_connections 2024-04-12T21:04:39.890 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:39.189+0000 7fd405200700 1 -- 172.21.15.81:0/545258527 wait complete. 2024-04-12T21:04:40.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:39 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:39.988+0000 7fd6e051a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:04:40.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:40.111+0000 7fd6e051a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:04:40.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:40 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/545258527' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-12T21:04:40.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:40 smithi081 ceph-mon[31453]: mgrmap e5: x(active, since 5s) 2024-04-12T21:04:41.050 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:40.929+0000 7fd6e051a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:04:41.050 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:41.052+0000 7fd6e051a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:04:41.357 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:41 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3740489129' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-12T21:04:41.358 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:41.234+0000 7fd6e051a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:04:41.358 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:41.359+0000 7fd6e051a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:04:41.402 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.431+0000 7f7b3ea00700 1 Processor -- start 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.432+0000 7f7b3ea00700 1 -- start start 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.432+0000 7f7b3ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.432+0000 7f7b3ea00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3810afd0 con 0x7f7b3810a5e0 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.432+0000 7f7b37e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.432+0000 7f7b37e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34310/0 (socket says 172.21.15.81:34310) 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.432+0000 7f7b37e00700 1 -- 172.21.15.81:0/3493445265 learned_addr learned my addr 172.21.15.81:0/3493445265 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:41.403 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.433+0000 7f7b37e00700 1 -- 172.21.15.81:0/3493445265 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3810b790 con 0x7f7b3810a5e0 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.433+0000 7f7b37e00700 1 --2- 172.21.15.81:0/3493445265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3810aa00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7b280098a0 tx=0x7f7b28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f3e5d60f1eb732 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.434+0000 7f7b36a00700 1 -- 172.21.15.81:0/3493445265 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b28004160 con 0x7f7b3810a5e0 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.434+0000 7f7b36a00700 1 -- 172.21.15.81:0/3493445265 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7b28004300 con 0x7f7b3810a5e0 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.434+0000 7f7b36a00700 1 -- 172.21.15.81:0/3493445265 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b28011b50 con 0x7f7b3810a5e0 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.434+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3493445265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 msgr2=0x7f7b3810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.434+0000 7f7b3ea00700 1 --2- 172.21.15.81:0/3493445265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3810aa00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7b280098a0 tx=0x7f7b28009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.435+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3493445265 shutdown_connections 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.435+0000 7f7b3ea00700 1 --2- 172.21.15.81:0/3493445265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3810aa00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.435+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3493445265 >> 172.21.15.81:0/3493445265 conn(0x7f7b3807dc20 msgr2=0x7f7b381084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.435+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3493445265 shutdown_connections 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.435+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3493445265 wait complete. 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.436+0000 7f7b3ea00700 1 Processor -- start 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.436+0000 7f7b3ea00700 1 -- start start 2024-04-12T21:04:41.404 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.436+0000 7f7b3ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3819e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.436+0000 7f7b3ea00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3810eb40 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.436+0000 7f7b37e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3819e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.436+0000 7f7b37e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3819e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34314/0 (socket says 172.21.15.81:34314) 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.436+0000 7f7b37e00700 1 -- 172.21.15.81:0/3740489129 learned_addr learned my addr 172.21.15.81:0/3740489129 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.437+0000 7f7b37e00700 1 -- 172.21.15.81:0/3740489129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3819ea60 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.437+0000 7f7b37e00700 1 --2- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3819e520 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b28000c00 tx=0x7f7b28011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.437+0000 7f7b34c00700 1 -- 172.21.15.81:0/3740489129 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b28003b20 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.437+0000 7f7b34c00700 1 -- 172.21.15.81:0/3740489129 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7b28003cc0 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.437+0000 7f7b34c00700 1 -- 172.21.15.81:0/3740489129 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7b28024910 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.438+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3819ecf0 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.438+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b381a19e0 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.439+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b38064110 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.439+0000 7f7b34c00700 1 -- 172.21.15.81:0/3740489129 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f7b28022020 con 0x7f7b3810a5e0 2024-04-12T21:04:41.405 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.440+0000 7f7b34c00700 1 --2- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f7b1c039600 0x7f7b1c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.440+0000 7f7b34c00700 1 -- 172.21.15.81:0/3740489129 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7b28011e00 con 0x7f7b3810a5e0 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.440+0000 7f7b37400700 1 -- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f7b1c039600 msgr2=0x7f7b1c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3647696686 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.440+0000 7f7b37400700 1 --2- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f7b1c039600 0x7f7b1c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.444+0000 7f7b34c00700 1 -- 172.21.15.81:0/3740489129 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b2801ad80 con 0x7f7b3810a5e0 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.641+0000 7f7b37400700 1 -- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f7b1c039600 msgr2=0x7f7b1c03bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3647696686 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.641+0000 7f7b37400700 1 --2- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f7b1c039600 0x7f7b1c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.730+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f7b381a1bb0 con 0x7f7b3810a5e0 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.731+0000 7f7b34c00700 1 -- 172.21.15.81:0/3740489129 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f7b381a1bb0 con 0x7f7b3810a5e0 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f7b1c039600 msgr2=0x7f7b1c03bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 --2- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f7b1c039600 0x7f7b1c03bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 msgr2=0x7f7b3819e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 --2- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3819e520 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b28000c00 tx=0x7f7b28011c10 comp rx=0 tx=0).stop 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 shutdown_connections 2024-04-12T21:04:41.406 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 --2- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f7b1c039600 0x7f7b1c03bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:41.407 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 --2- 172.21.15.81:0/3740489129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b3810a5e0 0x7f7b3819e520 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:41.407 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 >> 172.21.15.81:0/3740489129 conn(0x7f7b3807dc20 msgr2=0x7f7b38107be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:41.407 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.734+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 shutdown_connections 2024-04-12T21:04:41.407 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:40.735+0000 7f7b3ea00700 1 -- 172.21.15.81:0/3740489129 wait complete. 2024-04-12T21:04:41.407 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for the mgr to restart... 2024-04-12T21:04:41.407 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr epoch 5... 2024-04-12T21:04:41.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:41.486+0000 7fd6e051a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:04:41.956 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:41.832+0000 7fd6e051a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T21:04:42.217 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:41.979+0000 7fd6e051a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:04:42.217 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:42.094+0000 7fd6e051a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:04:42.217 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:42.219+0000 7fd6e051a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:04:42.567 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:42.439+0000 7fd6e051a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:04:44.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:43 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:43.708+0000 7fd6e051a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:04:44.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:43 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:43.879+0000 7fd6e051a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:04:44.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:44 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:44.298+0000 7fd6e051a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:04:44.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:44 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:44.417+0000 7fd6e051a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:04:44.970 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:44 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:44.671+0000 7fd6e051a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:04:44.970 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:44 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:44.916+0000 7fd6e051a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:04:45.611 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:45 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:45.297+0000 7fd6e051a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:04:45.903 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:45 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:45.615+0000 7fd6e051a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:04:46.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:45 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:45.921+0000 7fd6e051a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:04:47.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:46.949+0000 7fd6e051a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:04:47.902 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:47.643+0000 7fd6e051a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:04:47.902 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:47.771+0000 7fd6e051a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:04:48.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:04:48.051+0000 7fd6e051a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:04:48.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: Active manager daemon x restarted 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: Activating manager daemon x 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: osdmap e2: 0 total, 0 up, 0 in 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: mgrmap e6: x(active, starting, since 0.00921337s) 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:04:48.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:48 smithi081 ceph-mon[31453]: Manager daemon x is now available 2024-04-12T21:04:49.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: Found migration_current of "None". Setting to last migration. 2024-04-12T21:04:49.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:04:49.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:04:49.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:04:49.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:04:49.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:04:49.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T21:04:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:04:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:04:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:49 smithi081 ceph-mon[31453]: mgrmap e7: x(active, since 1.01648s) 2024-04-12T21:04:49.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-04-12T21:04:49.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-12T21:04:49.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-12T21:04:49.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-04-12T21:04:49.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.958+0000 7f4252800700 1 Processor -- start 2024-04-12T21:04:49.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.958+0000 7f4252800700 1 -- start start 2024-04-12T21:04:49.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.959+0000 7f4252800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c07c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:49.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.959+0000 7f4252800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f424c07c850 con 0x7f424c07dda0 2024-04-12T21:04:49.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.959+0000 7f424b400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c07c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:49.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.959+0000 7f424b400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c07c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34324/0 (socket says 172.21.15.81:34324) 2024-04-12T21:04:49.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.959+0000 7f424b400700 1 -- 172.21.15.81:0/2647522460 learned_addr learned my addr 172.21.15.81:0/2647522460 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:49.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.960+0000 7f424b400700 1 -- 172.21.15.81:0/2647522460 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c07c9d0 con 0x7f424c07dda0 2024-04-12T21:04:49.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.960+0000 7f424b400700 1 --2- 172.21.15.81:0/2647522460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c07c310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f423c0098a0 tx=0x7f423c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7457711f602b9130 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:49.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.961+0000 7f424a000700 1 -- 172.21.15.81:0/2647522460 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f423c004160 con 0x7f424c07dda0 2024-04-12T21:04:49.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.961+0000 7f424a000700 1 -- 172.21.15.81:0/2647522460 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f423c004300 con 0x7f424c07dda0 2024-04-12T21:04:49.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.961+0000 7f424a000700 1 -- 172.21.15.81:0/2647522460 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f423c011b50 con 0x7f424c07dda0 2024-04-12T21:04:49.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.961+0000 7f4252800700 1 -- 172.21.15.81:0/2647522460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 msgr2=0x7f424c07c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:49.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.961+0000 7f4252800700 1 --2- 172.21.15.81:0/2647522460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c07c310 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f423c0098a0 tx=0x7f423c009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:49.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.961+0000 7f4252800700 1 -- 172.21.15.81:0/2647522460 shutdown_connections 2024-04-12T21:04:49.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.961+0000 7f4252800700 1 --2- 172.21.15.81:0/2647522460 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c07c310 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:49.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.961+0000 7f4252800700 1 -- 172.21.15.81:0/2647522460 >> 172.21.15.81:0/2647522460 conn(0x7f424c1039a0 msgr2=0x7f424c105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:49.760 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.962+0000 7f4252800700 1 -- 172.21.15.81:0/2647522460 shutdown_connections 2024-04-12T21:04:49.760 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.962+0000 7f4252800700 1 -- 172.21.15.81:0/2647522460 wait complete. 2024-04-12T21:04:49.760 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.963+0000 7f4252800700 1 Processor -- start 2024-04-12T21:04:49.760 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.963+0000 7f4252800700 1 -- start start 2024-04-12T21:04:49.761 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.964+0000 7f4252800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c19e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:49.761 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.964+0000 7f4252800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f424c10e850 con 0x7f424c07dda0 2024-04-12T21:04:49.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.964+0000 7f424b400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c19e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:49.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.964+0000 7f424b400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c19e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34334/0 (socket says 172.21.15.81:34334) 2024-04-12T21:04:49.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.964+0000 7f424b400700 1 -- 172.21.15.81:0/156352149 learned_addr learned my addr 172.21.15.81:0/156352149 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:49.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.964+0000 7f424b400700 1 -- 172.21.15.81:0/156352149 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c19ea00 con 0x7f424c07dda0 2024-04-12T21:04:49.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.965+0000 7f424b400700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c19e4c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f423c011b60 tx=0x7f423c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:49.762 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.965+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f423c003b20 con 0x7f424c07dda0 2024-04-12T21:04:49.765 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.965+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f423c003cc0 con 0x7f424c07dda0 2024-04-12T21:04:49.765 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.965+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f423c024840 con 0x7f424c07dda0 2024-04-12T21:04:49.765 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.965+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f424c19ec90 con 0x7f424c07dda0 2024-04-12T21:04:49.765 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.965+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f424c1a1980 con 0x7f424c07dda0 2024-04-12T21:04:49.765 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.966+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f423c022020 con 0x7f424c07dda0 2024-04-12T21:04:49.765 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.967+0000 7f4243e00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 0x7f423003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.967+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 --> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f423003c1d0 con 0x7f4230039600 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.967+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f423c011e00 con 0x7f424c07dda0 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.967+0000 7f424aa00700 1 -- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 msgr2=0x7f423003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3647696686 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:41.967+0000 7f424aa00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 0x7f423003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:42.168+0000 7f424aa00700 1 -- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 msgr2=0x7f423003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3647696686 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:42.168+0000 7f424aa00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 0x7f423003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:42.569+0000 7f424aa00700 1 -- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 msgr2=0x7f423003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3647696686 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:42.569+0000 7f424aa00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 0x7f423003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:43.370+0000 7f424aa00700 1 -- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 msgr2=0x7f423003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3647696686 2024-04-12T21:04:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:43.370+0000 7f424aa00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 0x7f423003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-12T21:04:49.770 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:44.973+0000 7f424aa00700 1 -- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 msgr2=0x7f423003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/3647696686 2024-04-12T21:04:49.771 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:44.973+0000 7f424aa00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 0x7f423003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-12T21:04:49.771 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:48.060+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f423c024a40 con 0x7f424c07dda0 2024-04-12T21:04:49.771 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:48.060+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 msgr2=0x7f423003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T21:04:49.771 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:48.060+0000 7f4243e00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 0x7f423003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:49.771 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.063+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f423c02ca80 con 0x7f424c07dda0 2024-04-12T21:04:49.771 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.064+0000 7f4243e00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f423003cef0 0x7f423003f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:49.772 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.064+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f423003c1d0 con 0x7f423003cef0 2024-04-12T21:04:49.772 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.067+0000 7f424aa00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f423003cef0 0x7f423003f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:49.772 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.069+0000 7f424aa00700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f423003cef0 0x7f423003f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4238003a10 tx=0x7f42380092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:49.772 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.071+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mgr.14120 v2:172.21.15.81:6800/1636479292 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f423003c1d0 con 0x7f423003cef0 2024-04-12T21:04:49.772 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.079+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f424c064110 con 0x7f423003cef0 2024-04-12T21:04:49.772 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.079+0000 7f4243e00700 1 -- 172.21.15.81:0/156352149 <== mgr.14120 v2:172.21.15.81:6800/1636479292 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f424c064110 con 0x7f423003cef0 2024-04-12T21:04:49.772 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.079+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f423003cef0 msgr2=0x7f423003f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:49.773 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.079+0000 7f4252800700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f423003cef0 0x7f423003f2e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4238003a10 tx=0x7f42380092b0 comp rx=0 tx=0).stop 2024-04-12T21:04:49.773 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.079+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 msgr2=0x7f424c19e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:49.773 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.079+0000 7f4252800700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c19e4c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f423c011b60 tx=0x7f423c011c40 comp rx=0 tx=0).stop 2024-04-12T21:04:49.773 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.080+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 shutdown_connections 2024-04-12T21:04:49.773 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.080+0000 7f4252800700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f423003cef0 0x7f423003f2e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:49.773 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.080+0000 7f4252800700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f424c07dda0 0x7f424c19e4c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:49.773 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.080+0000 7f4252800700 1 --2- 172.21.15.81:0/156352149 >> [v2:172.21.15.81:6800/3647696686,v1:172.21.15.81:6801/3647696686] conn(0x7f4230039600 0x7f423003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:49.774 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.080+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 >> 172.21.15.81:0/156352149 conn(0x7f424c1039a0 msgr2=0x7f424c104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:49.774 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.080+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 shutdown_connections 2024-04-12T21:04:49.774 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:49.080+0000 7f4252800700 1 -- 172.21.15.81:0/156352149 wait complete. 2024-04-12T21:04:49.775 INFO:teuthology.orchestra.run.smithi081.stdout:mgr epoch 5 is available 2024-04-12T21:04:49.775 INFO:teuthology.orchestra.run.smithi081.stdout:Setting orchestrator backend to cephadm... 2024-04-12T21:04:50.380 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:50 smithi081 ceph-mon[31453]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-12T21:04:50.380 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:50 smithi081 ceph-mon[31453]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-12T21:04:50.380 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:50 smithi081 ceph-mon[31453]: [12/Apr/2024:21:04:49] ENGINE Bus STARTING 2024-04-12T21:04:50.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:50 smithi081 ceph-mon[31453]: [12/Apr/2024:21:04:49] ENGINE Serving on http://172.21.15.81:8765 2024-04-12T21:04:50.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:50 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.383+0000 7f407fa00700 1 Processor -- start 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.383+0000 7f407fa00700 1 -- start start 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.384+0000 7f407fa00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407807c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.384+0000 7f407fa00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f407807c850 con 0x7f407807dea0 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.384+0000 7f407d000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407807c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.384+0000 7f407d000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407807c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34350/0 (socket says 172.21.15.81:34350) 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.384+0000 7f407d000700 1 -- 172.21.15.81:0/3455941411 learned_addr learned my addr 172.21.15.81:0/3455941411 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.385+0000 7f407d000700 1 -- 172.21.15.81:0/3455941411 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407807c9d0 con 0x7f407807dea0 2024-04-12T21:04:51.178 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.385+0000 7f407d000700 1 --2- 172.21.15.81:0/3455941411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407807c310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f406c009e80 tx=0x7f406c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e983345da6937d server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.385+0000 7f4077400700 1 -- 172.21.15.81:0/3455941411 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f406c017070 con 0x7f407807dea0 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.386+0000 7f4077400700 1 -- 172.21.15.81:0/3455941411 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f406c00b8c0 con 0x7f407807dea0 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.386+0000 7f4077400700 1 -- 172.21.15.81:0/3455941411 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f406c0045c0 con 0x7f407807dea0 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.386+0000 7f407fa00700 1 -- 172.21.15.81:0/3455941411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 msgr2=0x7f407807c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.386+0000 7f407fa00700 1 --2- 172.21.15.81:0/3455941411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407807c310 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f406c009e80 tx=0x7f406c00b330 comp rx=0 tx=0).stop 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.386+0000 7f407fa00700 1 -- 172.21.15.81:0/3455941411 shutdown_connections 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.386+0000 7f407fa00700 1 --2- 172.21.15.81:0/3455941411 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407807c310 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.386+0000 7f407fa00700 1 -- 172.21.15.81:0/3455941411 >> 172.21.15.81:0/3455941411 conn(0x7f4078103a00 msgr2=0x7f4078105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.387+0000 7f407fa00700 1 -- 172.21.15.81:0/3455941411 shutdown_connections 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.387+0000 7f407fa00700 1 -- 172.21.15.81:0/3455941411 wait complete. 2024-04-12T21:04:51.179 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.388+0000 7f407fa00700 1 Processor -- start 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.388+0000 7f407fa00700 1 -- start start 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.388+0000 7f407fa00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.388+0000 7f407fa00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f407810e990 con 0x7f407807dea0 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.388+0000 7f407d000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.388+0000 7f407d000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407819e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34362/0 (socket says 172.21.15.81:34362) 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.388+0000 7f407d000700 1 -- 172.21.15.81:0/2220829248 learned_addr learned my addr 172.21.15.81:0/2220829248 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.389+0000 7f407d000700 1 -- 172.21.15.81:0/2220829248 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407819eae0 con 0x7f407807dea0 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.389+0000 7f407d000700 1 --2- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407819e5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f406c0072a0 tx=0x7f406c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.389+0000 7f4075600700 1 -- 172.21.15.81:0/2220829248 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f406c020020 con 0x7f407807dea0 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.389+0000 7f4075600700 1 -- 172.21.15.81:0/2220829248 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f406c003b50 con 0x7f407807dea0 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.389+0000 7f4075600700 1 -- 172.21.15.81:0/2220829248 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f406c017040 con 0x7f407807dea0 2024-04-12T21:04:51.180 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.389+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f407819ed70 con 0x7f407807dea0 2024-04-12T21:04:51.181 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.389+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40781a1a60 con 0x7f407807dea0 2024-04-12T21:04:51.181 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.391+0000 7f4075600700 1 -- 172.21.15.81:0/2220829248 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f406c023020 con 0x7f407807dea0 2024-04-12T21:04:51.181 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.391+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4078064110 con 0x7f407807dea0 2024-04-12T21:04:51.181 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.391+0000 7f4075600700 1 --2- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f40600395b0 0x7f406003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:51.181 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.391+0000 7f4075600700 1 -- 172.21.15.81:0/2220829248 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f406c00f050 con 0x7f407807dea0 2024-04-12T21:04:51.181 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.391+0000 7f4077e00700 1 --2- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f40600395b0 0x7f406003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:51.181 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.392+0000 7f4077e00700 1 --2- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f40600395b0 0x7f406003ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4064006fd0 tx=0x7f4064006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.398+0000 7f4075600700 1 -- 172.21.15.81:0/2220829248 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f406c02a020 con 0x7f407807dea0 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.635+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f4078104bd0 con 0x7f40600395b0 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.646+0000 7f4075600700 1 -- 172.21.15.81:0/2220829248 <== mgr.14120 v2:172.21.15.81:6800/1636479292 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4078104bd0 con 0x7f40600395b0 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f40600395b0 msgr2=0x7f406003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 --2- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f40600395b0 0x7f406003ba70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4064006fd0 tx=0x7f4064006e40 comp rx=0 tx=0).stop 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 msgr2=0x7f407819e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 --2- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407819e5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f406c0072a0 tx=0x7f406c00bd50 comp rx=0 tx=0).stop 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 shutdown_connections 2024-04-12T21:04:51.182 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 --2- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f40600395b0 0x7f406003ba70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:51.183 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 --2- 172.21.15.81:0/2220829248 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f407807dea0 0x7f407819e5a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:51.183 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 >> 172.21.15.81:0/2220829248 conn(0x7f4078103a00 msgr2=0x7f40781044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:51.183 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 shutdown_connections 2024-04-12T21:04:51.183 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:50.649+0000 7f407fa00700 1 -- 172.21.15.81:0/2220829248 wait complete. 2024-04-12T21:04:51.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:51 smithi081 ceph-mon[31453]: [12/Apr/2024:21:04:49] ENGINE Serving on https://172.21.15.81:7150 2024-04-12T21:04:51.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:51 smithi081 ceph-mon[31453]: [12/Apr/2024:21:04:49] ENGINE Bus STARTED 2024-04-12T21:04:51.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:51 smithi081 ceph-mon[31453]: mgrmap e8: x(active, since 2s) 2024-04-12T21:04:51.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:51 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:04:51.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:51 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:04:52.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:52 smithi081 ceph-mon[31453]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:04:52.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-12T21:04:52.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.837+0000 7fa98ea00700 1 Processor -- start 2024-04-12T21:04:52.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.837+0000 7fa98ea00700 1 -- start start 2024-04-12T21:04:52.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.837+0000 7fa98ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa98807ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:52.755 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.837+0000 7fa98ea00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa98807f260 con 0x7fa98807e870 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.838+0000 7fa98d600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa98807ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.838+0000 7fa98d600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa98807ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34366/0 (socket says 172.21.15.81:34366) 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.838+0000 7fa98d600700 1 -- 172.21.15.81:0/2683853396 learned_addr learned my addr 172.21.15.81:0/2683853396 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.838+0000 7fa98d600700 1 -- 172.21.15.81:0/2683853396 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98807fa70 con 0x7fa98807e870 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.839+0000 7fa98d600700 1 --2- 172.21.15.81:0/2683853396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa98807ec90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0098a0 tx=0x7fa97c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a598a594faedb96 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.839+0000 7fa987e00700 1 -- 172.21.15.81:0/2683853396 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa97c004160 con 0x7fa98807e870 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.839+0000 7fa987e00700 1 -- 172.21.15.81:0/2683853396 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa97c004300 con 0x7fa98807e870 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.839+0000 7fa987e00700 1 -- 172.21.15.81:0/2683853396 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa97c011b50 con 0x7fa98807e870 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.840+0000 7fa98ea00700 1 -- 172.21.15.81:0/2683853396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 msgr2=0x7fa98807ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.840+0000 7fa98ea00700 1 --2- 172.21.15.81:0/2683853396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa98807ec90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0098a0 tx=0x7fa97c009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.840+0000 7fa98ea00700 1 -- 172.21.15.81:0/2683853396 shutdown_connections 2024-04-12T21:04:52.756 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.840+0000 7fa98ea00700 1 --2- 172.21.15.81:0/2683853396 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa98807ec90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.840+0000 7fa98ea00700 1 -- 172.21.15.81:0/2683853396 >> 172.21.15.81:0/2683853396 conn(0x7fa98807d8f0 msgr2=0x7fa9881084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.840+0000 7fa98ea00700 1 -- 172.21.15.81:0/2683853396 shutdown_connections 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.840+0000 7fa98ea00700 1 -- 172.21.15.81:0/2683853396 wait complete. 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.841+0000 7fa98ea00700 1 Processor -- start 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.841+0000 7fa98ea00700 1 -- start start 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.841+0000 7fa98ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa9881a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.841+0000 7fa98ea00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa988080e30 con 0x7fa98807e870 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.842+0000 7fa98d600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa9881a2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.842+0000 7fa98d600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa9881a2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34380/0 (socket says 172.21.15.81:34380) 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.842+0000 7fa98d600700 1 -- 172.21.15.81:0/1593428869 learned_addr learned my addr 172.21.15.81:0/1593428869 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.842+0000 7fa98d600700 1 -- 172.21.15.81:0/1593428869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9881a3090 con 0x7fa98807e870 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.842+0000 7fa98d600700 1 --2- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa9881a2b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0046e0 tx=0x7fa97c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.842+0000 7fa986000700 1 -- 172.21.15.81:0/1593428869 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa97c0039e0 con 0x7fa98807e870 2024-04-12T21:04:52.757 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.842+0000 7fa986000700 1 -- 172.21.15.81:0/1593428869 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa97c003b80 con 0x7fa98807e870 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.842+0000 7fa986000700 1 -- 172.21.15.81:0/1593428869 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa97c024760 con 0x7fa98807e870 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.843+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9881a3320 con 0x7fa98807e870 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.843+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9881a6010 con 0x7fa98807e870 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.844+0000 7fa986000700 1 -- 172.21.15.81:0/1593428869 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fa97c022020 con 0x7fa98807e870 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.844+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa988064110 con 0x7fa98807e870 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.844+0000 7fa986000700 1 --2- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fa970039600 0x7fa97003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.844+0000 7fa986000700 1 -- 172.21.15.81:0/1593428869 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa97c011e00 con 0x7fa98807e870 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.845+0000 7fa98cc00700 1 --2- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fa970039600 0x7fa97003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.845+0000 7fa98cc00700 1 --2- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fa970039600 0x7fa97003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa974006fd0 tx=0x7fa974006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:51.850+0000 7fa986000700 1 -- 172.21.15.81:0/1593428869 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa97c04d2e0 con 0x7fa98807e870 2024-04-12T21:04:52.758 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.076+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fa98802e010 con 0x7fa970039600 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.078+0000 7fa986000700 1 -- 172.21.15.81:0/1593428869 <== mgr.14120 v2:172.21.15.81:6800/1636479292 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fa98802e010 con 0x7fa970039600 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fa970039600 msgr2=0x7fa97003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 --2- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fa970039600 0x7fa97003bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa974006fd0 tx=0x7fa974006e40 comp rx=0 tx=0).stop 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 msgr2=0x7fa9881a2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 --2- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa9881a2b50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0046e0 tx=0x7fa97c011c10 comp rx=0 tx=0).stop 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 shutdown_connections 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 --2- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fa970039600 0x7fa97003bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 --2- 172.21.15.81:0/1593428869 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa98807e870 0x7fa9881a2b50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 >> 172.21.15.81:0/1593428869 conn(0x7fa98807d8f0 msgr2=0x7fa9881993f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 shutdown_connections 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:52.081+0000 7fa98ea00700 1 -- 172.21.15.81:0/1593428869 wait complete. 2024-04-12T21:04:52.759 INFO:teuthology.orchestra.run.smithi081.stdout:Generating ssh key... 2024-04-12T21:04:53.293 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-mon[31453]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:04:54.328 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: Generating public/private rsa key pair. 2024-04-12T21:04:54.328 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: Your identification has been saved in /tmp/tmpledjvkph/key. 2024-04-12T21:04:54.328 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: Your public key has been saved in /tmp/tmpledjvkph/key.pub. 2024-04-12T21:04:54.328 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: The key fingerprint is: 2024-04-12T21:04:54.328 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: SHA256:5FZYkyuVpvK7JAlrIN2ZYi3Y4vUVWK3XgpZZkDVYVds ceph-11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:54.328 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: The key's randomart image is: 2024-04-12T21:04:54.328 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: +---[RSA 3072]----+ 2024-04-12T21:04:54.328 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: | oB+++.. | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: | oo =*. o | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: | . .O+o. . E | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: | + o o.O++.. | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: |+ B.* .+S.. | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: |.+.+.o.o. | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: | . o.o .. | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: | . o. | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: | .. | 2024-04-12T21:04:54.329 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:04:53 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: +----[SHA256]-----+ 2024-04-12T21:04:54.629 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.326+0000 7fde1d800700 1 Processor -- start 2024-04-12T21:04:54.629 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.326+0000 7fde1d800700 1 -- start start 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.327+0000 7fde1d800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde18109db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.327+0000 7fde1d800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde1810a2f0 con 0x7fde18109990 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.327+0000 7fde16a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde18109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.327+0000 7fde16a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde18109db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34394/0 (socket says 172.21.15.81:34394) 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.327+0000 7fde16a00700 1 -- 172.21.15.81:0/1906508856 learned_addr learned my addr 172.21.15.81:0/1906508856 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.328+0000 7fde16a00700 1 -- 172.21.15.81:0/1906508856 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde18108060 con 0x7fde18109990 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.328+0000 7fde16a00700 1 --2- 172.21.15.81:0/1906508856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde18109db0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde080098a0 tx=0x7fde08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e17e84b6ef06bb server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.328+0000 7fde16000700 1 -- 172.21.15.81:0/1906508856 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde08004160 con 0x7fde18109990 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.328+0000 7fde16000700 1 -- 172.21.15.81:0/1906508856 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fde08004300 con 0x7fde18109990 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.328+0000 7fde16000700 1 -- 172.21.15.81:0/1906508856 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde08011b50 con 0x7fde18109990 2024-04-12T21:04:54.630 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.329+0000 7fde1d800700 1 -- 172.21.15.81:0/1906508856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 msgr2=0x7fde18109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.329+0000 7fde1d800700 1 --2- 172.21.15.81:0/1906508856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde18109db0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fde080098a0 tx=0x7fde08009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.329+0000 7fde1d800700 1 -- 172.21.15.81:0/1906508856 shutdown_connections 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.329+0000 7fde1d800700 1 --2- 172.21.15.81:0/1906508856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde18109db0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.329+0000 7fde1d800700 1 -- 172.21.15.81:0/1906508856 >> 172.21.15.81:0/1906508856 conn(0x7fde18103a00 msgr2=0x7fde18105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.329+0000 7fde1d800700 1 -- 172.21.15.81:0/1906508856 shutdown_connections 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.329+0000 7fde1d800700 1 -- 172.21.15.81:0/1906508856 wait complete. 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.330+0000 7fde1d800700 1 Processor -- start 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.331+0000 7fde1d800700 1 -- start start 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.331+0000 7fde1d800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde1819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.331+0000 7fde1d800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fde1810c990 con 0x7fde18109990 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.331+0000 7fde16a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde1819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.331+0000 7fde16a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde1819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34404/0 (socket says 172.21.15.81:34404) 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.331+0000 7fde16a00700 1 -- 172.21.15.81:0/2445443235 learned_addr learned my addr 172.21.15.81:0/2445443235 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:54.631 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.332+0000 7fde16a00700 1 -- 172.21.15.81:0/2445443235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1819eb70 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.332+0000 7fde16a00700 1 --2- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde1819e630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde08011b60 tx=0x7fde08011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.332+0000 7fde0f400700 1 -- 172.21.15.81:0/2445443235 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde08003b20 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.332+0000 7fde0f400700 1 -- 172.21.15.81:0/2445443235 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fde08003cc0 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.332+0000 7fde0f400700 1 -- 172.21.15.81:0/2445443235 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde08024840 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.332+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde1819ee00 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.333+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde181a1af0 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.334+0000 7fde0f400700 1 -- 172.21.15.81:0/2445443235 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fde08022020 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.334+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde1810cb10 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.334+0000 7fde0f400700 1 --2- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fddfc039600 0x7fddfc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.334+0000 7fde0f400700 1 -- 172.21.15.81:0/2445443235 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fde08011e00 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.334+0000 7fde0fe00700 1 --2- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fddfc039600 0x7fddfc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.335+0000 7fde0fe00700 1 --2- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fddfc039600 0x7fddfc03bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde00006fd0 tx=0x7fde00006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.340+0000 7fde0f400700 1 -- 172.21.15.81:0/2445443235 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fde0804d1a0 con 0x7fde18109990 2024-04-12T21:04:54.632 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.565+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fde18104ae0 con 0x7fddfc039600 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.975+0000 7fde0f400700 1 -- 172.21.15.81:0/2445443235 <== mgr.14120 v2:172.21.15.81:6800/1636479292 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fde18104ae0 con 0x7fddfc039600 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.977+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fddfc039600 msgr2=0x7fddfc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.977+0000 7fde1d800700 1 --2- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fddfc039600 0x7fddfc03bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fde00006fd0 tx=0x7fde00006e40 comp rx=0 tx=0).stop 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.977+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 msgr2=0x7fde1819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.977+0000 7fde1d800700 1 --2- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde1819e630 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fde08011b60 tx=0x7fde08011c40 comp rx=0 tx=0).stop 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.978+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 shutdown_connections 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.978+0000 7fde1d800700 1 --2- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fddfc039600 0x7fddfc03bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.978+0000 7fde1d800700 1 --2- 172.21.15.81:0/2445443235 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fde18109990 0x7fde1819e630 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.978+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 >> 172.21.15.81:0/2445443235 conn(0x7fde18103a00 msgr2=0x7fde181043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.978+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 shutdown_connections 2024-04-12T21:04:54.633 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:53.978+0000 7fde1d800700 1 -- 172.21.15.81:0/2445443235 wait complete. 2024-04-12T21:04:55.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:54 smithi081 ceph-mon[31453]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:04:55.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:54 smithi081 ceph-mon[31453]: Generating ssh key... 2024-04-12T21:04:55.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:54 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:04:55.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:54 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:04:56.100 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXqkKoEr5KYkIRFYpP4xG+T8r1weVu36b3NTQ1iIwlymlwRskBJSQRnc/ERDbjnVcQTvp6iIm+hsIR5ZG5FSshTF1+a33et+U5RF6N8KO/53Mreo0B86OXdCP/IzfXPgIl0oAsgohB+TCWQvxUoUIajhzsbD9+7YwZoznZhE67fWOdarzWBGUl4NjF60BaIAzdsejmbI2PN6Ie/EoK/XsRF1SLBFWTvh2k8Lv1g9shidVEo/b4lKv/OkP1L2jiJLSk9u2zbEpfKXyEFGyZrKlStvusU/X9qbfB5pspTwkuKmkAWrHNjKSMzVLCskb0k2F2JQY9L8ZUzQcZOMEDYR6rCPO0+98KHC2GvpgLb005VwBnsRJqkaaCcwLsmg5mOx9YGaDHW4wJttCniRjWyLh38JiB0KJxHBl2tOhopg4D7qV7hHKd/50Kim+KxWlNz6Up11zOMWCJnYGNoRIUQnmX+Y23qIehkI9XSizkxeqeStGVjCGM/2NR7/1NeBMQTyM= ceph-11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:04:56.106 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.149+0000 7fe2fc400700 1 Processor -- start 2024-04-12T21:04:56.106 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.149+0000 7fe2fc400700 1 -- start start 2024-04-12T21:04:56.106 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.150+0000 7fe2fc400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f407c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:56.106 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.150+0000 7fe2fc400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f407c830 con 0x7fe2f407de80 2024-04-12T21:04:56.106 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.150+0000 7fe2f9a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f407c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:56.106 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.150+0000 7fe2f9a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f407c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42264/0 (socket says 172.21.15.81:42264) 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.150+0000 7fe2f9a00700 1 -- 172.21.15.81:0/131508842 learned_addr learned my addr 172.21.15.81:0/131508842 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.151+0000 7fe2f9a00700 1 -- 172.21.15.81:0/131508842 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f407c9b0 con 0x7fe2f407de80 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.151+0000 7fe2f9a00700 1 --2- 172.21.15.81:0/131508842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f407c2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe2e80098a0 tx=0x7fe2e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53906047090185d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.152+0000 7fe2f3e00700 1 -- 172.21.15.81:0/131508842 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2e8004160 con 0x7fe2f407de80 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.152+0000 7fe2f3e00700 1 -- 172.21.15.81:0/131508842 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe2e8004300 con 0x7fe2f407de80 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.152+0000 7fe2f3e00700 1 -- 172.21.15.81:0/131508842 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2e8011b50 con 0x7fe2f407de80 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.152+0000 7fe2fc400700 1 -- 172.21.15.81:0/131508842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 msgr2=0x7fe2f407c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.152+0000 7fe2fc400700 1 --2- 172.21.15.81:0/131508842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f407c2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe2e80098a0 tx=0x7fe2e8009bb0 comp rx=0 tx=0).stop 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.153+0000 7fe2fc400700 1 -- 172.21.15.81:0/131508842 shutdown_connections 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.153+0000 7fe2fc400700 1 --2- 172.21.15.81:0/131508842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f407c2f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.153+0000 7fe2fc400700 1 -- 172.21.15.81:0/131508842 >> 172.21.15.81:0/131508842 conn(0x7fe2f4103a00 msgr2=0x7fe2f4105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.153+0000 7fe2fc400700 1 -- 172.21.15.81:0/131508842 shutdown_connections 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.153+0000 7fe2fc400700 1 -- 172.21.15.81:0/131508842 wait complete. 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.154+0000 7fe2fc400700 1 Processor -- start 2024-04-12T21:04:56.107 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.154+0000 7fe2fc400700 1 -- start start 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.154+0000 7fe2fc400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.154+0000 7fe2fc400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f410e950 con 0x7fe2f407de80 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.155+0000 7fe2f9a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.155+0000 7fe2f9a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42274/0 (socket says 172.21.15.81:42274) 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.155+0000 7fe2f9a00700 1 -- 172.21.15.81:0/901288317 learned_addr learned my addr 172.21.15.81:0/901288317 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.155+0000 7fe2f9a00700 1 -- 172.21.15.81:0/901288317 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f419eb40 con 0x7fe2f407de80 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.155+0000 7fe2f9a00700 1 --2- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f419e600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8011b60 tx=0x7fe2e8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.155+0000 7fe2f2000700 1 -- 172.21.15.81:0/901288317 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2e8003b20 con 0x7fe2f407de80 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.156+0000 7fe2f2000700 1 -- 172.21.15.81:0/901288317 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe2e8003cc0 con 0x7fe2f407de80 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.156+0000 7fe2f2000700 1 -- 172.21.15.81:0/901288317 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2e8024840 con 0x7fe2f407de80 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.156+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f419edd0 con 0x7fe2f407de80 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.156+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f41a1ac0 con 0x7fe2f407de80 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.157+0000 7fe2f2000700 1 -- 172.21.15.81:0/901288317 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe2e8022020 con 0x7fe2f407de80 2024-04-12T21:04:56.108 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.157+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f4064110 con 0x7fe2f407de80 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.157+0000 7fe2f2000700 1 --2- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fe2dc039600 0x7fe2dc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.157+0000 7fe2f2000700 1 -- 172.21.15.81:0/901288317 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe2e8011e00 con 0x7fe2f407de80 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.158+0000 7fe2f9000700 1 --2- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fe2dc039600 0x7fe2dc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.158+0000 7fe2f9000700 1 --2- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fe2dc039600 0x7fe2dc03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0006fd0 tx=0x7fe2e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.162+0000 7fe2f2000700 1 -- 172.21.15.81:0/901288317 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2e804d1c0 con 0x7fe2f407de80 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.387+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe2f4104ca0 con 0x7fe2dc039600 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.388+0000 7fe2f2000700 1 -- 172.21.15.81:0/901288317 <== mgr.14120 v2:172.21.15.81:6800/1636479292 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe2f4104ca0 con 0x7fe2dc039600 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fe2dc039600 msgr2=0x7fe2dc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 --2- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fe2dc039600 0x7fe2dc03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0006fd0 tx=0x7fe2e0006e40 comp rx=0 tx=0).stop 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 msgr2=0x7fe2f419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 --2- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f419e600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe2e8011b60 tx=0x7fe2e8011c40 comp rx=0 tx=0).stop 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 shutdown_connections 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 --2- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fe2dc039600 0x7fe2dc03bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:56.109 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 --2- 172.21.15.81:0/901288317 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe2f407de80 0x7fe2f419e600 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:04:56.110 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 >> 172.21.15.81:0/901288317 conn(0x7fe2f4103a00 msgr2=0x7fe2f4104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:04:56.110 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.391+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 shutdown_connections 2024-04-12T21:04:56.110 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:55.392+0000 7fe2fc400700 1 -- 172.21.15.81:0/901288317 wait complete. 2024-04-12T21:04:56.110 INFO:teuthology.orchestra.run.smithi081.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-12T21:04:56.110 INFO:teuthology.orchestra.run.smithi081.stdout:Adding key to root@localhost authorized_keys... 2024-04-12T21:04:56.110 INFO:teuthology.orchestra.run.smithi081.stdout:Adding host smithi081... 2024-04-12T21:04:56.615 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:56 smithi081 ceph-mon[31453]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:04:57.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:57 smithi081 ceph-mon[31453]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi081", "addr": "172.21.15.81", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:04:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:04:59 smithi081 ceph-mon[31453]: Deploying cephadm binary to smithi081 2024-04-12T21:05:03.951 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Added host 'smithi081' with addr '172.21.15.81' 2024-04-12T21:05:03.951 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.711+0000 7f256ce00700 1 Processor -- start 2024-04-12T21:05:03.951 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.711+0000 7f256ce00700 1 -- start start 2024-04-12T21:05:03.951 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.712+0000 7f256ce00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f256807c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.712+0000 7f256ce00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f256807c890 con 0x7f256807de50 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.712+0000 7f2565e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f256807c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.712+0000 7f2565e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f256807c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42276/0 (socket says 172.21.15.81:42276) 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.712+0000 7f2565e00700 1 -- 172.21.15.81:0/3773709332 learned_addr learned my addr 172.21.15.81:0/3773709332 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.713+0000 7f2565e00700 1 -- 172.21.15.81:0/3773709332 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256807ca10 con 0x7f256807de50 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.713+0000 7f2565e00700 1 --2- 172.21.15.81:0/3773709332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f256807c350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f25580098a0 tx=0x7f2558009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f809bceeaea65fd server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.713+0000 7f2564a00700 1 -- 172.21.15.81:0/3773709332 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2558004160 con 0x7f256807de50 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.713+0000 7f2564a00700 1 -- 172.21.15.81:0/3773709332 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2558004300 con 0x7f256807de50 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.713+0000 7f2564a00700 1 -- 172.21.15.81:0/3773709332 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2558011b50 con 0x7f256807de50 2024-04-12T21:05:03.952 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.714+0000 7f256ce00700 1 -- 172.21.15.81:0/3773709332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 msgr2=0x7f256807c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.714+0000 7f256ce00700 1 --2- 172.21.15.81:0/3773709332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f256807c350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f25580098a0 tx=0x7f2558009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.714+0000 7f256ce00700 1 -- 172.21.15.81:0/3773709332 shutdown_connections 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.714+0000 7f256ce00700 1 --2- 172.21.15.81:0/3773709332 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f256807c350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.714+0000 7f256ce00700 1 -- 172.21.15.81:0/3773709332 >> 172.21.15.81:0/3773709332 conn(0x7f2568103a40 msgr2=0x7f2568105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.714+0000 7f256ce00700 1 -- 172.21.15.81:0/3773709332 shutdown_connections 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.714+0000 7f256ce00700 1 -- 172.21.15.81:0/3773709332 wait complete. 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.715+0000 7f256ce00700 1 Processor -- start 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.715+0000 7f256ce00700 1 -- start start 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.716+0000 7f256ce00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f25681a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.716+0000 7f256ce00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f256810a580 con 0x7f256807de50 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.716+0000 7f2565e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f25681a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.716+0000 7f2565e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f25681a28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42282/0 (socket says 172.21.15.81:42282) 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.716+0000 7f2565e00700 1 -- 172.21.15.81:0/1020255676 learned_addr learned my addr 172.21.15.81:0/1020255676 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.716+0000 7f2565e00700 1 -- 172.21.15.81:0/1020255676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25681a2e10 con 0x7f256807de50 2024-04-12T21:05:03.953 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.717+0000 7f2565e00700 1 --2- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f25681a28d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2558011b60 tx=0x7f2558011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.717+0000 7f255ea00700 1 -- 172.21.15.81:0/1020255676 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2558003b20 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.717+0000 7f255ea00700 1 -- 172.21.15.81:0/1020255676 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2558003cc0 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.717+0000 7f255ea00700 1 -- 172.21.15.81:0/1020255676 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2558024840 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.717+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25681a30a0 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.717+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25681a5d90 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.718+0000 7f255ea00700 1 -- 172.21.15.81:0/1020255676 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f2558022020 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.718+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2568064110 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.719+0000 7f255ea00700 1 --2- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f254c039600 0x7f254c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.719+0000 7f255ea00700 1 -- 172.21.15.81:0/1020255676 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2558011e00 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.719+0000 7f2565400700 1 --2- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f254c039600 0x7f254c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.720+0000 7f2565400700 1 --2- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f254c039600 0x7f254c03bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2554006fd0 tx=0x7f2554006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.724+0000 7f255ea00700 1 -- 172.21.15.81:0/1020255676 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f255804d1a0 con 0x7f256807de50 2024-04-12T21:05:03.954 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:04:56.953+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi081", "addr": "172.21.15.81", "target": ["mon-mgr", ""]}) -- 0x7f25680008d0 con 0x7f254c039600 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.248+0000 7f255ea00700 1 -- 172.21.15.81:0/1020255676 <== mgr.14120 v2:172.21.15.81:6800/1636479292 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f25680008d0 con 0x7f254c039600 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f254c039600 msgr2=0x7f254c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 --2- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f254c039600 0x7f254c03bac0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2554006fd0 tx=0x7f2554006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 msgr2=0x7f25681a28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 --2- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f25681a28d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2558011b60 tx=0x7f2558011c40 comp rx=0 tx=0).stop 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 shutdown_connections 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 --2- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f254c039600 0x7f254c03bac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 --2- 172.21.15.81:0/1020255676 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f256807de50 0x7f25681a28d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 >> 172.21.15.81:0/1020255676 conn(0x7f2568103a40 msgr2=0x7f2568105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 shutdown_connections 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:03.252+0000 7f256ce00700 1 -- 172.21.15.81:0/1020255676 wait complete. 2024-04-12T21:05:03.955 INFO:teuthology.orchestra.run.smithi081.stdout:Deploying unmanaged mon service... 2024-04-12T21:05:04.586 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:04 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:05:04.586 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:04 smithi081 ceph-mon[31453]: Added host smithi081 2024-04-12T21:05:04.586 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:04 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:05:05.875 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.598+0000 7fb53fc00700 1 Processor -- start 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.598+0000 7fb53fc00700 1 -- start start 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.599+0000 7fb53fc00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53810aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.599+0000 7fb53fc00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb53810aff0 con 0x7fb53810a600 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.599+0000 7fb53d200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53810aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.599+0000 7fb53d200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53810aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42298/0 (socket says 172.21.15.81:42298) 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.599+0000 7fb53d200700 1 -- 172.21.15.81:0/1620832838 learned_addr learned my addr 172.21.15.81:0/1620832838 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.600+0000 7fb53d200700 1 -- 172.21.15.81:0/1620832838 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53810b760 con 0x7fb53810a600 2024-04-12T21:05:05.876 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.600+0000 7fb53d200700 1 --2- 172.21.15.81:0/1620832838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53810aa20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0098a0 tx=0x7fb52c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eba2ef927ddaefb server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.600+0000 7fb537400700 1 -- 172.21.15.81:0/1620832838 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb52c004160 con 0x7fb53810a600 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.600+0000 7fb537400700 1 -- 172.21.15.81:0/1620832838 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb52c004300 con 0x7fb53810a600 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.600+0000 7fb537400700 1 -- 172.21.15.81:0/1620832838 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb52c011b50 con 0x7fb53810a600 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.601+0000 7fb53fc00700 1 -- 172.21.15.81:0/1620832838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 msgr2=0x7fb53810aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.601+0000 7fb53fc00700 1 --2- 172.21.15.81:0/1620832838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53810aa20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0098a0 tx=0x7fb52c009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.601+0000 7fb53fc00700 1 -- 172.21.15.81:0/1620832838 shutdown_connections 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.601+0000 7fb53fc00700 1 --2- 172.21.15.81:0/1620832838 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53810aa20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.601+0000 7fb53fc00700 1 -- 172.21.15.81:0/1620832838 >> 172.21.15.81:0/1620832838 conn(0x7fb53807dc20 msgr2=0x7fb5381084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.601+0000 7fb53fc00700 1 -- 172.21.15.81:0/1620832838 shutdown_connections 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.601+0000 7fb53fc00700 1 -- 172.21.15.81:0/1620832838 wait complete. 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.602+0000 7fb53fc00700 1 Processor -- start 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.602+0000 7fb53fc00700 1 -- start start 2024-04-12T21:05:05.877 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.602+0000 7fb53fc00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53819e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.602+0000 7fb53fc00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb53810eb10 con 0x7fb53810a600 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.603+0000 7fb53d200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53819e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.603+0000 7fb53d200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53819e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42306/0 (socket says 172.21.15.81:42306) 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.603+0000 7fb53d200700 1 -- 172.21.15.81:0/2631154677 learned_addr learned my addr 172.21.15.81:0/2631154677 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.603+0000 7fb53d200700 1 -- 172.21.15.81:0/2631154677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53819ea30 con 0x7fb53810a600 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.603+0000 7fb53d200700 1 --2- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53819e4f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb52c000c00 tx=0x7fb52c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.603+0000 7fb535600700 1 -- 172.21.15.81:0/2631154677 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb52c003b20 con 0x7fb53810a600 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.604+0000 7fb535600700 1 -- 172.21.15.81:0/2631154677 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb52c003cc0 con 0x7fb53810a600 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.604+0000 7fb535600700 1 -- 172.21.15.81:0/2631154677 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb52c024910 con 0x7fb53810a600 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.604+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53819ecc0 con 0x7fb53810a600 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.604+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5381a19b0 con 0x7fb53810a600 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.605+0000 7fb535600700 1 -- 172.21.15.81:0/2631154677 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb52c022020 con 0x7fb53810a600 2024-04-12T21:05:05.878 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.605+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb538064110 con 0x7fb53810a600 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.605+0000 7fb535600700 1 --2- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fb520039600 0x7fb52003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.605+0000 7fb535600700 1 -- 172.21.15.81:0/2631154677 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb52c011e00 con 0x7fb53810a600 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.605+0000 7fb537e00700 1 --2- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fb520039600 0x7fb52003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.606+0000 7fb537e00700 1 --2- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fb520039600 0x7fb52003bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb524006fd0 tx=0x7fb524006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.611+0000 7fb535600700 1 -- 172.21.15.81:0/2631154677 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb52c04d130 con 0x7fb53810a600 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.860+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fb538108300 con 0x7fb520039600 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.868+0000 7fb535600700 1 -- 172.21.15.81:0/2631154677 <== mgr.14120 v2:172.21.15.81:6800/1636479292 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb538108300 con 0x7fb520039600 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.871+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fb520039600 msgr2=0x7fb52003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.871+0000 7fb53fc00700 1 --2- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fb520039600 0x7fb52003bac0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb524006fd0 tx=0x7fb524006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.871+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 msgr2=0x7fb53819e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.871+0000 7fb53fc00700 1 --2- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53819e4f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb52c000c00 tx=0x7fb52c011c10 comp rx=0 tx=0).stop 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.872+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 shutdown_connections 2024-04-12T21:05:05.879 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.872+0000 7fb53fc00700 1 --2- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7fb520039600 0x7fb52003bac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:05.880 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.872+0000 7fb53fc00700 1 --2- 172.21.15.81:0/2631154677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb53810a600 0x7fb53819e4f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:05.880 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.872+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 >> 172.21.15.81:0/2631154677 conn(0x7fb53807dc20 msgr2=0x7fb538107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:05.880 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.872+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 shutdown_connections 2024-04-12T21:05:05.880 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:04.872+0000 7fb53fc00700 1 -- 172.21.15.81:0/2631154677 wait complete. 2024-04-12T21:05:05.880 INFO:teuthology.orchestra.run.smithi081.stdout:Deploying unmanaged mgr service... 2024-04-12T21:05:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:05 smithi081 ceph-mon[31453]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:05:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:05 smithi081 ceph-mon[31453]: Saving service mon spec with placement count:5 2024-04-12T21:05:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:05 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:05:07.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:07 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:05:07.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:07 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:05:07.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:07 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:05:08.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-12T21:05:08.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.095+0000 7f1528a00700 1 Processor -- start 2024-04-12T21:05:08.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.095+0000 7f1528a00700 1 -- start start 2024-04-12T21:05:08.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.096+0000 7f1528a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f1524109db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:08.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.096+0000 7f1528a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f152410a2f0 con 0x7f1524109990 2024-04-12T21:05:08.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.096+0000 7f1521e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f1524109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:08.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.096+0000 7f1521e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f1524109db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50576/0 (socket says 172.21.15.81:50576) 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.096+0000 7f1521e00700 1 -- 172.21.15.81:0/1764236425 learned_addr learned my addr 172.21.15.81:0/1764236425 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.097+0000 7f1521e00700 1 -- 172.21.15.81:0/1764236425 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1524108060 con 0x7f1524109990 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.097+0000 7f1521e00700 1 --2- 172.21.15.81:0/1764236425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f1524109db0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f15140098a0 tx=0x7f1514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a887d57fb986af6a server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.097+0000 7f1520a00700 1 -- 172.21.15.81:0/1764236425 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1514004160 con 0x7f1524109990 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.097+0000 7f1520a00700 1 -- 172.21.15.81:0/1764236425 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1514004300 con 0x7f1524109990 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.097+0000 7f1520a00700 1 -- 172.21.15.81:0/1764236425 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1514011b20 con 0x7f1524109990 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.098+0000 7f1528a00700 1 -- 172.21.15.81:0/1764236425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 msgr2=0x7f1524109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.098+0000 7f1528a00700 1 --2- 172.21.15.81:0/1764236425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f1524109db0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f15140098a0 tx=0x7f1514009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.098+0000 7f1528a00700 1 -- 172.21.15.81:0/1764236425 shutdown_connections 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.098+0000 7f1528a00700 1 --2- 172.21.15.81:0/1764236425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f1524109db0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.098+0000 7f1528a00700 1 -- 172.21.15.81:0/1764236425 >> 172.21.15.81:0/1764236425 conn(0x7f1524103a00 msgr2=0x7f1524105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.099+0000 7f1528a00700 1 -- 172.21.15.81:0/1764236425 shutdown_connections 2024-04-12T21:05:08.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.099+0000 7f1528a00700 1 -- 172.21.15.81:0/1764236425 wait complete. 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.100+0000 7f1528a00700 1 Processor -- start 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.100+0000 7f1528a00700 1 -- start start 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.100+0000 7f1528a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f152419c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.100+0000 7f1528a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f152410c990 con 0x7f1524109990 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.101+0000 7f1521e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f152419c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.101+0000 7f1521e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f152419c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50586/0 (socket says 172.21.15.81:50586) 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.101+0000 7f1521e00700 1 -- 172.21.15.81:0/3831723618 learned_addr learned my addr 172.21.15.81:0/3831723618 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.101+0000 7f1521e00700 1 -- 172.21.15.81:0/3831723618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152419c950 con 0x7f1524109990 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.101+0000 7f1521e00700 1 --2- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f152419c410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f15140117e0 tx=0x7f1514011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.102+0000 7f151aa00700 1 -- 172.21.15.81:0/3831723618 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1514003bf0 con 0x7f1524109990 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.102+0000 7f1528a00700 1 -- 172.21.15.81:0/3831723618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f152419cbe0 con 0x7f1524109990 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.102+0000 7f151aa00700 1 -- 172.21.15.81:0/3831723618 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f151401a910 con 0x7f1524109990 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.102+0000 7f151aa00700 1 -- 172.21.15.81:0/3831723618 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1514024970 con 0x7f1524109990 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.102+0000 7f1528a00700 1 -- 172.21.15.81:0/3831723618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f152419d040 con 0x7f1524109990 2024-04-12T21:05:08.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.103+0000 7f151aa00700 1 -- 172.21.15.81:0/3831723618 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1514022020 con 0x7f1524109990 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.103+0000 7f150fe00700 1 -- 172.21.15.81:0/3831723618 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f152410cb10 con 0x7f1524109990 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.103+0000 7f151aa00700 1 --2- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1508039560 0x7f150803ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.103+0000 7f151aa00700 1 -- 172.21.15.81:0/3831723618 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f151404da50 con 0x7f1524109990 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.104+0000 7f1521400700 1 --2- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1508039560 0x7f150803ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.105+0000 7f1521400700 1 --2- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1508039560 0x7f150803ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1510006fd0 tx=0x7f1510006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.110+0000 7f151aa00700 1 -- 172.21.15.81:0/3831723618 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1514011e20 con 0x7f1524109990 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.351+0000 7f150fe00700 1 -- 172.21.15.81:0/3831723618 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1524063b80 con 0x7f1508039560 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.358+0000 7f151aa00700 1 -- 172.21.15.81:0/3831723618 <== mgr.14120 v2:172.21.15.81:6800/1636479292 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1524063b80 con 0x7f1508039560 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.361+0000 7f150fe00700 1 -- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1508039560 msgr2=0x7f150803ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.361+0000 7f150fe00700 1 --2- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1508039560 0x7f150803ba20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1510006fd0 tx=0x7f1510006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.361+0000 7f150fe00700 1 -- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 msgr2=0x7f152419c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.361+0000 7f150fe00700 1 --2- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f152419c410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f15140117e0 tx=0x7f1514011c10 comp rx=0 tx=0).stop 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.362+0000 7f150fe00700 1 -- 172.21.15.81:0/3831723618 shutdown_connections 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.362+0000 7f150fe00700 1 --2- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1508039560 0x7f150803ba20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.362+0000 7f150fe00700 1 --2- 172.21.15.81:0/3831723618 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1524109990 0x7f152419c410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:08.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.362+0000 7f150fe00700 1 -- 172.21.15.81:0/3831723618 >> 172.21.15.81:0/3831723618 conn(0x7f1524103a00 msgr2=0x7f1524104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:08.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.362+0000 7f150fe00700 1 -- 172.21.15.81:0/3831723618 shutdown_connections 2024-04-12T21:05:08.017 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:07.362+0000 7f150fe00700 1 -- 172.21.15.81:0/3831723618 wait complete. 2024-04-12T21:05:09.095 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:09 smithi081 ceph-mon[31453]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:05:09.095 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:09 smithi081 ceph-mon[31453]: Saving service mgr spec with placement count:2 2024-04-12T21:05:09.095 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:09 smithi081 ceph-mon[31453]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:09.095 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:09 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/455567211' entity='client.admin' 2024-04-12T21:05:09.351 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.579+0000 7f9a33a00700 1 Processor -- start 2024-04-12T21:05:09.351 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.579+0000 7f9a33a00700 1 -- start start 2024-04-12T21:05:09.352 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.580+0000 7f9a33a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c10b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:09.352 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.580+0000 7f9a33a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2c07be80 con 0x7f9a2c108e50 2024-04-12T21:05:09.352 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.580+0000 7f9a31000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c10b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:09.352 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.580+0000 7f9a31000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c10b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50602/0 (socket says 172.21.15.81:50602) 2024-04-12T21:05:09.352 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.580+0000 7f9a31000700 1 -- 172.21.15.81:0/1649532743 learned_addr learned my addr 172.21.15.81:0/1649532743 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:09.352 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.581+0000 7f9a31000700 1 -- 172.21.15.81:0/1649532743 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2c106c40 con 0x7f9a2c108e50 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.581+0000 7f9a31000700 1 --2- 172.21.15.81:0/1649532743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c10b240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0098a0 tx=0x7f9a1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bbe0bb584f3e98 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.582+0000 7f9a2b400700 1 -- 172.21.15.81:0/1649532743 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a1c004160 con 0x7f9a2c108e50 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.582+0000 7f9a2b400700 1 -- 172.21.15.81:0/1649532743 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a1c004300 con 0x7f9a2c108e50 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.582+0000 7f9a2b400700 1 -- 172.21.15.81:0/1649532743 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a1c011b20 con 0x7f9a2c108e50 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.583+0000 7f9a33a00700 1 -- 172.21.15.81:0/1649532743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 msgr2=0x7f9a2c10b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.583+0000 7f9a33a00700 1 --2- 172.21.15.81:0/1649532743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c10b240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0098a0 tx=0x7f9a1c009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.583+0000 7f9a33a00700 1 -- 172.21.15.81:0/1649532743 shutdown_connections 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.583+0000 7f9a33a00700 1 --2- 172.21.15.81:0/1649532743 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c10b240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:09.357 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.583+0000 7f9a33a00700 1 -- 172.21.15.81:0/1649532743 >> 172.21.15.81:0/1649532743 conn(0x7f9a2c102880 msgr2=0x7f9a2c104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.583+0000 7f9a33a00700 1 -- 172.21.15.81:0/1649532743 shutdown_connections 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.583+0000 7f9a33a00700 1 -- 172.21.15.81:0/1649532743 wait complete. 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.585+0000 7f9a33a00700 1 Processor -- start 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.585+0000 7f9a33a00700 1 -- start start 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.586+0000 7f9a33a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c19e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.586+0000 7f9a33a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9a2c107d60 con 0x7f9a2c108e50 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.586+0000 7f9a31000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c19e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.586+0000 7f9a31000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c19e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50618/0 (socket says 172.21.15.81:50618) 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.586+0000 7f9a31000700 1 -- 172.21.15.81:0/455567211 learned_addr learned my addr 172.21.15.81:0/455567211 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.586+0000 7f9a31000700 1 -- 172.21.15.81:0/455567211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a2c19e9e0 con 0x7f9a2c108e50 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.587+0000 7f9a31000700 1 --2- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c19e4a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0117e0 tx=0x7f9a1c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:09.358 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.587+0000 7f9a29600700 1 -- 172.21.15.81:0/455567211 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a1c003ba0 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.587+0000 7f9a29600700 1 -- 172.21.15.81:0/455567211 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a1c01a910 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.587+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a2c19ec70 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.587+0000 7f9a29600700 1 -- 172.21.15.81:0/455567211 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a1c0248f0 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.587+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a2c19f0d0 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.589+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a2c064110 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.589+0000 7f9a29600700 1 -- 172.21.15.81:0/455567211 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f9a1c022020 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.590+0000 7f9a29600700 1 --2- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f9a100395b0 0x7f9a1003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.590+0000 7f9a29600700 1 -- 172.21.15.81:0/455567211 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9a1c04dc00 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.590+0000 7f9a2be00700 1 --2- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f9a100395b0 0x7f9a1003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.591+0000 7f9a2be00700 1 --2- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f9a100395b0 0x7f9a1003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a14006fd0 tx=0x7f9a14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.595+0000 7f9a29600700 1 -- 172.21.15.81:0/455567211 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a1c01ab70 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.826+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f9a2c0636c0 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.830+0000 7f9a29600700 1 -- 172.21.15.81:0/455567211 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f9a1c0212c0 con 0x7f9a2c108e50 2024-04-12T21:05:09.359 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.833+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f9a100395b0 msgr2=0x7f9a1003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.833+0000 7f9a33a00700 1 --2- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f9a100395b0 0x7f9a1003ba70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9a14006fd0 tx=0x7f9a14006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.833+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 msgr2=0x7f9a2c19e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.833+0000 7f9a33a00700 1 --2- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c19e4a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c0117e0 tx=0x7f9a1c011ba0 comp rx=0 tx=0).stop 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.833+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 shutdown_connections 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.833+0000 7f9a33a00700 1 --2- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f9a100395b0 0x7f9a1003ba70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.833+0000 7f9a33a00700 1 --2- 172.21.15.81:0/455567211 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9a2c108e50 0x7f9a2c19e4a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.833+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 >> 172.21.15.81:0/455567211 conn(0x7f9a2c102880 msgr2=0x7f9a2c103380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.834+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 shutdown_connections 2024-04-12T21:05:09.360 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:08.834+0000 7f9a33a00700 1 -- 172.21.15.81:0/455567211 wait complete. 2024-04-12T21:05:11.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:11 smithi081 ceph-mon[31453]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:11.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:11 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:05:11.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:11 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3720073501' entity='client.admin' 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.554+0000 7f1fb4a00700 1 Processor -- start 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.554+0000 7f1fb4a00700 1 -- start start 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.554+0000 7f1fb4a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb01070c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.554+0000 7f1fb4a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb007bdf0 con 0x7f1fb0106ca0 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.554+0000 7f1fade00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb01070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.555+0000 7f1fade00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb01070c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50626/0 (socket says 172.21.15.81:50626) 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.555+0000 7f1fade00700 1 -- 172.21.15.81:0/2910739240 learned_addr learned my addr 172.21.15.81:0/2910739240 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.555+0000 7f1fade00700 1 -- 172.21.15.81:0/2910739240 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb0107600 con 0x7f1fb0106ca0 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.555+0000 7f1fade00700 1 --2- 172.21.15.81:0/2910739240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb01070c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00098a0 tx=0x7f1fa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=186a73c28802586b server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.555+0000 7f1faca00700 1 -- 172.21.15.81:0/2910739240 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fa0004160 con 0x7f1fb0106ca0 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.556+0000 7f1faca00700 1 -- 172.21.15.81:0/2910739240 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa0004300 con 0x7f1fb0106ca0 2024-04-12T21:05:11.472 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.556+0000 7f1faca00700 1 -- 172.21.15.81:0/2910739240 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fa0011b20 con 0x7f1fb0106ca0 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.556+0000 7f1fb4a00700 1 -- 172.21.15.81:0/2910739240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 msgr2=0x7f1fb01070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.556+0000 7f1fb4a00700 1 --2- 172.21.15.81:0/2910739240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb01070c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1fa00098a0 tx=0x7f1fa0009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.556+0000 7f1fb4a00700 1 -- 172.21.15.81:0/2910739240 shutdown_connections 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.556+0000 7f1fb4a00700 1 --2- 172.21.15.81:0/2910739240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb01070c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.556+0000 7f1fb4a00700 1 -- 172.21.15.81:0/2910739240 >> 172.21.15.81:0/2910739240 conn(0x7f1fb0102860 msgr2=0x7f1fb0104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.557+0000 7f1fb4a00700 1 -- 172.21.15.81:0/2910739240 shutdown_connections 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.557+0000 7f1fb4a00700 1 -- 172.21.15.81:0/2910739240 wait complete. 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.558+0000 7f1fb4a00700 1 Processor -- start 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.558+0000 7f1fb4a00700 1 -- start start 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.558+0000 7f1fb4a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.558+0000 7f1fb4a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb0108550 con 0x7f1fb0106ca0 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.558+0000 7f1fade00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.558+0000 7f1fade00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb019e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50632/0 (socket says 172.21.15.81:50632) 2024-04-12T21:05:11.473 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.558+0000 7f1fade00700 1 -- 172.21.15.81:0/3720073501 learned_addr learned my addr 172.21.15.81:0/3720073501 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.559+0000 7f1fade00700 1 -- 172.21.15.81:0/3720073501 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb019ea90 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.559+0000 7f1fade00700 1 --2- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb019e550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0011b60 tx=0x7f1fa0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.559+0000 7f1fa6a00700 1 -- 172.21.15.81:0/3720073501 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fa0003b60 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.559+0000 7f1fa6a00700 1 -- 172.21.15.81:0/3720073501 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa001a910 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.559+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb019ed20 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.560+0000 7f1fa6a00700 1 -- 172.21.15.81:0/3720073501 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1fa0024930 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.560+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb019f180 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.561+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb0064110 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.561+0000 7f1fa6a00700 1 -- 172.21.15.81:0/3720073501 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f1fa0022020 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.561+0000 7f1fa6a00700 1 --2- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1f94039600 0x7f1f9403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.562+0000 7f1fa6a00700 1 -- 172.21.15.81:0/3720073501 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1fa004d8a0 con 0x7f1fb0106ca0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.562+0000 7f1fad400700 1 --2- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1f94039600 0x7f1f9403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:11.474 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.562+0000 7f1fad400700 1 --2- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1f94039600 0x7f1f9403bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c006fd0 tx=0x7f1f9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.567+0000 7f1fa6a00700 1 -- 172.21.15.81:0/3720073501 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1fa000b3e0 con 0x7f1fb0106ca0 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.791+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f1fb00636c0 con 0x7f1fb0106ca0 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.797+0000 7f1fa6a00700 1 -- 172.21.15.81:0/3720073501 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f1fa002ddb0 con 0x7f1fb0106ca0 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1f94039600 msgr2=0x7f1f9403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 --2- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1f94039600 0x7f1f9403bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1f9c006fd0 tx=0x7f1f9c006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 msgr2=0x7f1fb019e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 --2- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb019e550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1fa0011b60 tx=0x7f1fa0011c40 comp rx=0 tx=0).stop 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 shutdown_connections 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 --2- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f1f94039600 0x7f1f9403bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 --2- 172.21.15.81:0/3720073501 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1fb0106ca0 0x7f1fb019e550 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 >> 172.21.15.81:0/3720073501 conn(0x7f1fb0102860 msgr2=0x7f1fb0195160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 shutdown_connections 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:10.801+0000 7f1fb4a00700 1 -- 172.21.15.81:0/3720073501 wait complete. 2024-04-12T21:05:11.475 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling the dashboard module... 2024-04-12T21:05:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:13 smithi081 ceph-mon[31453]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:13 smithi081 ceph-mon[31453]: from='mgr.14120 172.21.15.81:0/1284376594' entity='mgr.x' 2024-04-12T21:05:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:13 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3526715524' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-12T21:05:13.403 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:13 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: ignoring --setuser ceph since I am not root 2024-04-12T21:05:13.403 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:13 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: ignoring --setgroup ceph since I am not root 2024-04-12T21:05:14.247 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:14 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:14.053+0000 7f331d6c2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:05:14.247 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:14 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:14.176+0000 7f331d6c2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:05:14.549 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:14 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3526715524' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-12T21:05:14.550 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:14 smithi081 ceph-mon[31453]: mgrmap e9: x(active, since 25s) 2024-04-12T21:05:14.665 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.092+0000 7f28c8a00700 1 Processor -- start 2024-04-12T21:05:14.665 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.092+0000 7f28c8a00700 1 -- start start 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.092+0000 7f28c8a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c410b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.092+0000 7f28c8a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28c407be70 con 0x7f28c4108e30 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.093+0000 7f28c1e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c410b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.093+0000 7f28c1e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c410b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50640/0 (socket says 172.21.15.81:50640) 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.093+0000 7f28c1e00700 1 -- 172.21.15.81:0/3096441481 learned_addr learned my addr 172.21.15.81:0/3096441481 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.093+0000 7f28c1e00700 1 -- 172.21.15.81:0/3096441481 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c410b760 con 0x7f28c4108e30 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.094+0000 7f28c1e00700 1 --2- 172.21.15.81:0/3096441481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c410b220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28b40098a0 tx=0x7f28b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc12a8d9a9cfcf3f server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.094+0000 7f28c1400700 1 -- 172.21.15.81:0/3096441481 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28b4004160 con 0x7f28c4108e30 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.094+0000 7f28c1400700 1 -- 172.21.15.81:0/3096441481 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b4004300 con 0x7f28c4108e30 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.094+0000 7f28c1400700 1 -- 172.21.15.81:0/3096441481 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28b4011b20 con 0x7f28c4108e30 2024-04-12T21:05:14.666 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.095+0000 7f28c8a00700 1 -- 172.21.15.81:0/3096441481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 msgr2=0x7f28c410b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.095+0000 7f28c8a00700 1 --2- 172.21.15.81:0/3096441481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c410b220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28b40098a0 tx=0x7f28b4009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.095+0000 7f28c8a00700 1 -- 172.21.15.81:0/3096441481 shutdown_connections 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.095+0000 7f28c8a00700 1 --2- 172.21.15.81:0/3096441481 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c410b220 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.095+0000 7f28c8a00700 1 -- 172.21.15.81:0/3096441481 >> 172.21.15.81:0/3096441481 conn(0x7f28c4102860 msgr2=0x7f28c4104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.095+0000 7f28c8a00700 1 -- 172.21.15.81:0/3096441481 shutdown_connections 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.095+0000 7f28c8a00700 1 -- 172.21.15.81:0/3096441481 wait complete. 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.096+0000 7f28c8a00700 1 Processor -- start 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.096+0000 7f28c8a00700 1 -- start start 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.096+0000 7f28c8a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.096+0000 7f28c8a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28c410c7c0 con 0x7f28c4108e30 2024-04-12T21:05:14.671 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.097+0000 7f28c1e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.097+0000 7f28c1e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c419e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50652/0 (socket says 172.21.15.81:50652) 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.097+0000 7f28c1e00700 1 -- 172.21.15.81:0/3526715524 learned_addr learned my addr 172.21.15.81:0/3526715524 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.097+0000 7f28c1e00700 1 -- 172.21.15.81:0/3526715524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c419ea40 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.097+0000 7f28c1e00700 1 --2- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c419e500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f28b4011b60 tx=0x7f28b4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.097+0000 7f28baa00700 1 -- 172.21.15.81:0/3526715524 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28b4003b60 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.098+0000 7f28baa00700 1 -- 172.21.15.81:0/3526715524 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b401a910 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.098+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c419ecd0 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.098+0000 7f28baa00700 1 -- 172.21.15.81:0/3526715524 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28b4024930 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.098+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c419f130 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.099+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28c4064110 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.099+0000 7f28baa00700 1 -- 172.21.15.81:0/3526715524 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f28b4022020 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.099+0000 7f28baa00700 1 --2- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f28a8039600 0x7f28a803bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.099+0000 7f28baa00700 1 -- 172.21.15.81:0/3526715524 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f28b404d8e0 con 0x7f28c4108e30 2024-04-12T21:05:14.672 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.100+0000 7f28bbe00700 1 --2- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f28a8039600 0x7f28a803bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.100+0000 7f28bbe00700 1 --2- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f28a8039600 0x7f28a803bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f28ac006fd0 tx=0x7f28ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.105+0000 7f28baa00700 1 -- 172.21.15.81:0/3526715524 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28b400b3e0 con 0x7f28c4108e30 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:12.427+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f28c41a1f10 con 0x7f28c4108e30 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.241+0000 7f28baa00700 1 -- 172.21.15.81:0/3526715524 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f28b402ddb0 con 0x7f28c4108e30 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.247+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f28a8039600 msgr2=0x7f28a803bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.247+0000 7f28c8a00700 1 --2- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f28a8039600 0x7f28a803bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f28ac006fd0 tx=0x7f28ac006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.247+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 msgr2=0x7f28c419e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.247+0000 7f28c8a00700 1 --2- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c419e500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f28b4011b60 tx=0x7f28b4011c40 comp rx=0 tx=0).stop 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.247+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 shutdown_connections 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.247+0000 7f28c8a00700 1 --2- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f28a8039600 0x7f28a803bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.248+0000 7f28c8a00700 1 --2- 172.21.15.81:0/3526715524 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28c4108e30 0x7f28c419e500 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.248+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 >> 172.21.15.81:0/3526715524 conn(0x7f28c4102860 msgr2=0x7f28c41033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:14.673 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.248+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 shutdown_connections 2024-04-12T21:05:14.674 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:13.248+0000 7f28c8a00700 1 -- 172.21.15.81:0/3526715524 wait complete. 2024-04-12T21:05:15.095 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:14 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:14.992+0000 7f331d6c2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:05:15.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:15 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:15.122+0000 7f331d6c2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:05:15.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:15 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:15.311+0000 7f331d6c2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:05:15.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:15 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2459313133' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-12T21:05:15.888 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:15 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:15.437+0000 7f331d6c2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:05:15.888 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:15 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:15.549+0000 7f331d6c2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:05:15.888 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:15 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:15.891+0000 7f331d6c2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T21:05:16.150 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:16.037+0000 7f331d6c2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:05:16.150 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:16.153+0000 7f331d6c2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:05:16.247 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-04-12T21:05:16.247 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-12T21:05:16.247 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-12T21:05:16.247 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-12T21:05:16.247 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-12T21:05:16.247 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.254+0000 7f7bf6200700 1 Processor -- start 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.254+0000 7f7bf6200700 1 -- start start 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.254+0000 7f7bf6200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf007c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.254+0000 7f7bf6200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf007c830 con 0x7f7bf007de80 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.255+0000 7f7bef400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf007c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.255+0000 7f7bef400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf007c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58688/0 (socket says 172.21.15.81:58688) 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.255+0000 7f7bef400700 1 -- 172.21.15.81:0/2287787064 learned_addr learned my addr 172.21.15.81:0/2287787064 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.255+0000 7f7bef400700 1 -- 172.21.15.81:0/2287787064 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf007c9b0 con 0x7f7bf007de80 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.256+0000 7f7bef400700 1 --2- 172.21.15.81:0/2287787064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf007c2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7be00098a0 tx=0x7f7be0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37f4e6a27f041a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.256+0000 7f7bee000700 1 -- 172.21.15.81:0/2287787064 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7be0004160 con 0x7f7bf007de80 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.256+0000 7f7bee000700 1 -- 172.21.15.81:0/2287787064 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be0004300 con 0x7f7bf007de80 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.256+0000 7f7bee000700 1 -- 172.21.15.81:0/2287787064 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7be0011b20 con 0x7f7bf007de80 2024-04-12T21:05:16.248 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.257+0000 7f7bf6200700 1 -- 172.21.15.81:0/2287787064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 msgr2=0x7f7bf007c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.257+0000 7f7bf6200700 1 --2- 172.21.15.81:0/2287787064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf007c2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7be00098a0 tx=0x7f7be0009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.257+0000 7f7bf6200700 1 -- 172.21.15.81:0/2287787064 shutdown_connections 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.257+0000 7f7bf6200700 1 --2- 172.21.15.81:0/2287787064 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf007c2f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.257+0000 7f7bf6200700 1 -- 172.21.15.81:0/2287787064 >> 172.21.15.81:0/2287787064 conn(0x7f7bf0103a00 msgr2=0x7f7bf0105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.257+0000 7f7bf6200700 1 -- 172.21.15.81:0/2287787064 shutdown_connections 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.257+0000 7f7bf6200700 1 -- 172.21.15.81:0/2287787064 wait complete. 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.258+0000 7f7bf6200700 1 Processor -- start 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.259+0000 7f7bf6200700 1 -- start start 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.259+0000 7f7bf6200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.259+0000 7f7bf6200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf010e990 con 0x7f7bf007de80 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.259+0000 7f7bef400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.259+0000 7f7bef400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf019e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58704/0 (socket says 172.21.15.81:58704) 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.259+0000 7f7bef400700 1 -- 172.21.15.81:0/2459313133 learned_addr learned my addr 172.21.15.81:0/2459313133 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:16.249 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.260+0000 7f7bef400700 1 -- 172.21.15.81:0/2459313133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf019eac0 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.260+0000 7f7bef400700 1 --2- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf019e580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7be00117e0 tx=0x7f7be0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.260+0000 7f7be7e00700 1 -- 172.21.15.81:0/2459313133 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7be0003ba0 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.260+0000 7f7be7e00700 1 -- 172.21.15.81:0/2459313133 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be001a910 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.260+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf019ed50 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.260+0000 7f7be7e00700 1 -- 172.21.15.81:0/2459313133 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7be00248f0 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.260+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf019f1b0 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.262+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bf0064110 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.262+0000 7f7be7e00700 1 -- 172.21.15.81:0/2459313133 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f7be0022020 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.262+0000 7f7be7e00700 1 --2- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f7bd4039600 0x7f7bd403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.262+0000 7f7be7e00700 1 -- 172.21.15.81:0/2459313133 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7be004dfe0 con 0x7f7bf007de80 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.262+0000 7f7beea00700 1 -- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f7bd4039600 msgr2=0x7f7bd403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/1636479292 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.263+0000 7f7beea00700 1 --2- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f7bd4039600 0x7f7bd403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T21:05:16.250 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.268+0000 7f7be7e00700 1 -- 172.21.15.81:0/2459313133 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7be000f2e0 con 0x7f7bf007de80 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.464+0000 7f7beea00700 1 -- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f7bd4039600 msgr2=0x7f7bd403bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/1636479292 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.464+0000 7f7beea00700 1 --2- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f7bd4039600 0x7f7bd403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.536+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f7bf01a1ef0 con 0x7f7bf007de80 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.537+0000 7f7be7e00700 1 -- 172.21.15.81:0/2459313133 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f7be004d930 con 0x7f7bf007de80 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.539+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f7bd4039600 msgr2=0x7f7bd403bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.539+0000 7f7bf6200700 1 --2- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f7bd4039600 0x7f7bd403bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.539+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 msgr2=0x7f7bf019e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.539+0000 7f7bf6200700 1 --2- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf019e580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7be00117e0 tx=0x7f7be0011ba0 comp rx=0 tx=0).stop 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.539+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 shutdown_connections 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.539+0000 7f7bf6200700 1 --2- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f7bd4039600 0x7f7bd403bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.539+0000 7f7bf6200700 1 --2- 172.21.15.81:0/2459313133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7bf007de80 0x7f7bf019e580 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.539+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 >> 172.21.15.81:0/2459313133 conn(0x7f7bf0103a00 msgr2=0x7f7bf01044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.540+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 shutdown_connections 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:15.540+0000 7f7bf6200700 1 -- 172.21.15.81:0/2459313133 wait complete. 2024-04-12T21:05:16.251 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for the mgr to restart... 2024-04-12T21:05:16.252 INFO:teuthology.orchestra.run.smithi081.stdout:Waiting for mgr epoch 9... 2024-04-12T21:05:16.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:16.295+0000 7f331d6c2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:05:16.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:16.524+0000 7f331d6c2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:05:18.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:17 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:17.813+0000 7f331d6c2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:05:18.153 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:17 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:17.985+0000 7f331d6c2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:05:18.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:18 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:18.404+0000 7f331d6c2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:05:18.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:18 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:18.524+0000 7f331d6c2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:05:19.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:18 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:18.779+0000 7f331d6c2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:05:19.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:19 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:19.026+0000 7f331d6c2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:05:19.719 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:19 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:19.405+0000 7f331d6c2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:05:19.719 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:19 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:19.722+0000 7f331d6c2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:05:20.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:20 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:20.031+0000 7f331d6c2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:05:21.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:21.063+0000 7f331d6c2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:05:22.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:21.759+0000 7f331d6c2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:05:22.152 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:21.889+0000 7f331d6c2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:05:22.652 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:05:22.175+0000 7f331d6c2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: Active manager daemon x restarted 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: Activating manager daemon x 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: osdmap e3: 0 total, 0 up, 0 in 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: mgrmap e10: x(active, starting, since 0.00980717s) 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-12T21:05:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:22 smithi081 ceph-mon[31453]: Manager daemon x is now available 2024-04-12T21:05:23.547 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:05:23.548 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-12T21:05:23.548 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-12T21:05:23.548 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:23 smithi081 ceph-mon[31453]: mgrmap e11: x(active, since 1.0103s) 2024-04-12T21:05:23.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout { 2024-04-12T21:05:23.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-12T21:05:23.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-12T21:05:23.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout } 2024-04-12T21:05:23.827 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.888+0000 7f3727c00700 1 Processor -- start 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.888+0000 7f3727c00700 1 -- start start 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.888+0000 7f3727c00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.889+0000 7f3727c00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f372007e180 con 0x7f372007dd60 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.889+0000 7f3725200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.889+0000 7f3725200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58706/0 (socket says 172.21.15.81:58706) 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.889+0000 7f3725200700 1 -- 172.21.15.81:0/2541924681 learned_addr learned my addr 172.21.15.81:0/2541924681 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.889+0000 7f3725200700 1 -- 172.21.15.81:0/2541924681 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372007c810 con 0x7f372007dd60 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.890+0000 7f3725200700 1 --2- 172.21.15.81:0/2541924681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372007c2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37140098a0 tx=0x7f3714009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47e1a6073f66bd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.890+0000 7f371f400700 1 -- 172.21.15.81:0/2541924681 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3714004160 con 0x7f372007dd60 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.890+0000 7f371f400700 1 -- 172.21.15.81:0/2541924681 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3714004300 con 0x7f372007dd60 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.890+0000 7f371f400700 1 -- 172.21.15.81:0/2541924681 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3714011b20 con 0x7f372007dd60 2024-04-12T21:05:23.828 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.891+0000 7f3727c00700 1 -- 172.21.15.81:0/2541924681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 msgr2=0x7f372007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.891+0000 7f3727c00700 1 --2- 172.21.15.81:0/2541924681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372007c2d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37140098a0 tx=0x7f3714009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.891+0000 7f3727c00700 1 -- 172.21.15.81:0/2541924681 shutdown_connections 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.891+0000 7f3727c00700 1 --2- 172.21.15.81:0/2541924681 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372007c2d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.891+0000 7f3727c00700 1 -- 172.21.15.81:0/2541924681 >> 172.21.15.81:0/2541924681 conn(0x7f37201039a0 msgr2=0x7f3720105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.891+0000 7f3727c00700 1 -- 172.21.15.81:0/2541924681 shutdown_connections 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.891+0000 7f3727c00700 1 -- 172.21.15.81:0/2541924681 wait complete. 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.892+0000 7f3727c00700 1 Processor -- start 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.892+0000 7f3727c00700 1 -- start start 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.892+0000 7f3727c00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372010b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.892+0000 7f3727c00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f372010e870 con 0x7f372007dd60 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.893+0000 7f3725200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372010b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.893+0000 7f3725200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372010b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58708/0 (socket says 172.21.15.81:58708) 2024-04-12T21:05:23.829 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.893+0000 7f3725200700 1 -- 172.21.15.81:0/1106487067 learned_addr learned my addr 172.21.15.81:0/1106487067 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.893+0000 7f3725200700 1 -- 172.21.15.81:0/1106487067 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f372010bc30 con 0x7f372007dd60 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.893+0000 7f3725200700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372010b6f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37140117e0 tx=0x7f3714011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.893+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3714003ba0 con 0x7f372007dd60 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.894+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f371401a910 con 0x7f372007dd60 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.894+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f372010bec0 con 0x7f372007dd60 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.894+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37140248f0 con 0x7f372007dd60 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.894+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3720108400 con 0x7f372007dd60 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.895+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f3714022020 con 0x7f372007dd60 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.896+0000 7f371d600700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 0x7f370803bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.896+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 --> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f370803c230 con 0x7f3708039660 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.896+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f371404dc10 con 0x7f372007dd60 2024-04-12T21:05:23.830 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.896+0000 7f371fe00700 1 -- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 msgr2=0x7f370803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/1636479292 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:16.896+0000 7f371fe00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 0x7f370803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:17.097+0000 7f371fe00700 1 -- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 msgr2=0x7f370803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/1636479292 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:17.097+0000 7f371fe00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 0x7f370803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:17.498+0000 7f371fe00700 1 -- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 msgr2=0x7f370803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/1636479292 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:17.498+0000 7f371fe00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 0x7f370803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:18.299+0000 7f371fe00700 1 -- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 msgr2=0x7f370803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/1636479292 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:18.299+0000 7f371fe00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 0x7f370803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:19.901+0000 7f371fe00700 1 -- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 msgr2=0x7f370803bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.81:6800/1636479292 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:19.901+0000 7f371fe00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 0x7f370803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:22.185+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7f371402d3f0 con 0x7f372007dd60 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:22.186+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 msgr2=0x7f370803bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:22.186+0000 7f371d600700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 0x7f370803bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.186+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7f371404f670 con 0x7f372007dd60 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.187+0000 7f371d600700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f370803cff0 0x7f370803f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:23.831 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.187+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3714046c70 con 0x7f370803cff0 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.189+0000 7f371fe00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f370803cff0 0x7f370803f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.190+0000 7f371fe00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f370803cff0 0x7f370803f3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f370c003a10 tx=0x7f370c009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.191+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f3714046c70 con 0x7f370803cff0 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.198+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f372010c090 con 0x7f370803cff0 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.199+0000 7f371d600700 1 -- 172.21.15.81:0/1106487067 <== mgr.14152 v2:172.21.15.81:6800/3972789130 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f372010c090 con 0x7f370803cff0 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.199+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f370803cff0 msgr2=0x7f370803f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.199+0000 7f3727c00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f370803cff0 0x7f370803f3e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f370c003a10 tx=0x7f370c009270 comp rx=0 tx=0).stop 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.199+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 msgr2=0x7f372010b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.199+0000 7f3727c00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372010b6f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37140117e0 tx=0x7f3714011ba0 comp rx=0 tx=0).stop 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.200+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 shutdown_connections 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.200+0000 7f3727c00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f372007dd60 0x7f372010b6f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.200+0000 7f3727c00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f370803cff0 0x7f370803f3e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.200+0000 7f3727c00700 1 --2- 172.21.15.81:0/1106487067 >> [v2:172.21.15.81:6800/1636479292,v1:172.21.15.81:6801/1636479292] conn(0x7f3708039660 0x7f370803bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:23.832 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.200+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 >> 172.21.15.81:0/1106487067 conn(0x7f37201039a0 msgr2=0x7f3720104490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:23.833 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.200+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 shutdown_connections 2024-04-12T21:05:23.833 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:23.200+0000 7f3727c00700 1 -- 172.21.15.81:0/1106487067 wait complete. 2024-04-12T21:05:23.833 INFO:teuthology.orchestra.run.smithi081.stdout:mgr epoch 9 is available 2024-04-12T21:05:23.833 INFO:teuthology.orchestra.run.smithi081.stdout:Generating a dashboard self-signed certificate... 2024-04-12T21:05:24.605 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:24 smithi081 ceph-mon[31453]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-12T21:05:24.605 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:24 smithi081 ceph-mon[31453]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-12T21:05:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:25 smithi081 ceph-mon[31453]: [12/Apr/2024:21:05:23] ENGINE Bus STARTING 2024-04-12T21:05:25.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:25 smithi081 ceph-mon[31453]: [12/Apr/2024:21:05:24] ENGINE Serving on https://172.21.15.81:7150 2024-04-12T21:05:25.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:25 smithi081 ceph-mon[31453]: [12/Apr/2024:21:05:24] ENGINE Serving on http://172.21.15.81:8765 2024-04-12T21:05:25.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:25 smithi081 ceph-mon[31453]: [12/Apr/2024:21:05:24] ENGINE Bus STARTED 2024-04-12T21:05:25.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:25 smithi081 ceph-mon[31453]: mgrmap e12: x(active, since 2s) 2024-04-12T21:05:25.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:26.008 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.062+0000 7fbe46a00700 1 Processor -- start 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.063+0000 7fbe46a00700 1 -- start start 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.063+0000 7fbe46a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe4010b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.063+0000 7fbe46a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4007be30 con 0x7fbe40108e70 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.063+0000 7fbe3fe00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe4010b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.063+0000 7fbe3fe00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe4010b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37100/0 (socket says 172.21.15.81:37100) 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.063+0000 7fbe3fe00700 1 -- 172.21.15.81:0/79706496 learned_addr learned my addr 172.21.15.81:0/79706496 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.064+0000 7fbe3fe00700 1 -- 172.21.15.81:0/79706496 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4010b7a0 con 0x7fbe40108e70 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.064+0000 7fbe3fe00700 1 --2- 172.21.15.81:0/79706496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe4010b260 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbe300098a0 tx=0x7fbe30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461440ac9ef445d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.064+0000 7fbe3ea00700 1 -- 172.21.15.81:0/79706496 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe30004160 con 0x7fbe40108e70 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.065+0000 7fbe3ea00700 1 -- 172.21.15.81:0/79706496 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe30004300 con 0x7fbe40108e70 2024-04-12T21:05:26.009 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.065+0000 7fbe3ea00700 1 -- 172.21.15.81:0/79706496 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe30011b20 con 0x7fbe40108e70 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.065+0000 7fbe46a00700 1 -- 172.21.15.81:0/79706496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 msgr2=0x7fbe4010b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.065+0000 7fbe46a00700 1 --2- 172.21.15.81:0/79706496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe4010b260 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbe300098a0 tx=0x7fbe30009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.065+0000 7fbe46a00700 1 -- 172.21.15.81:0/79706496 shutdown_connections 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.065+0000 7fbe46a00700 1 --2- 172.21.15.81:0/79706496 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe4010b260 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.065+0000 7fbe46a00700 1 -- 172.21.15.81:0/79706496 >> 172.21.15.81:0/79706496 conn(0x7fbe40102860 msgr2=0x7fbe40104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.066+0000 7fbe46a00700 1 -- 172.21.15.81:0/79706496 shutdown_connections 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.066+0000 7fbe46a00700 1 -- 172.21.15.81:0/79706496 wait complete. 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.067+0000 7fbe46a00700 1 Processor -- start 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.067+0000 7fbe46a00700 1 -- start start 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.067+0000 7fbe46a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe401086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:26.010 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.067+0000 7fbe46a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4010c7a0 con 0x7fbe40108e70 2024-04-12T21:05:26.013 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.067+0000 7fbe3fe00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe401086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.067+0000 7fbe3fe00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe401086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37114/0 (socket says 172.21.15.81:37114) 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.067+0000 7fbe3fe00700 1 -- 172.21.15.81:0/3797283548 learned_addr learned my addr 172.21.15.81:0/3797283548 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.068+0000 7fbe3fe00700 1 -- 172.21.15.81:0/3797283548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe40108c30 con 0x7fbe40108e70 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.068+0000 7fbe3fe00700 1 --2- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe401086f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbe30011b60 tx=0x7fbe30011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.068+0000 7fbe3cc00700 1 -- 172.21.15.81:0/3797283548 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe30003b60 con 0x7fbe40108e70 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.068+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe40106e20 con 0x7fbe40108e70 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.068+0000 7fbe3cc00700 1 -- 172.21.15.81:0/3797283548 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe3001a910 con 0x7fbe40108e70 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.068+0000 7fbe3cc00700 1 -- 172.21.15.81:0/3797283548 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe30024930 con 0x7fbe40108e70 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.069+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe401072e0 con 0x7fbe40108e70 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.070+0000 7fbe3cc00700 1 -- 172.21.15.81:0/3797283548 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fbe30022020 con 0x7fbe40108e70 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.070+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe40064110 con 0x7fbe40108e70 2024-04-12T21:05:26.014 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.070+0000 7fbe3cc00700 1 --2- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbe24039600 0x7fbe2403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.070+0000 7fbe3cc00700 1 -- 172.21.15.81:0/3797283548 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbe3004da30 con 0x7fbe40108e70 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.070+0000 7fbe3f400700 1 --2- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbe24039600 0x7fbe2403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.071+0000 7fbe3f400700 1 --2- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbe24039600 0x7fbe2403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c006fd0 tx=0x7fbe2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.080+0000 7fbe3cc00700 1 -- 172.21.15.81:0/3797283548 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe3000b3e0 con 0x7fbe40108e70 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.302+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fbe40107ac0 con 0x7fbe24039600 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.364+0000 7fbe3cc00700 1 -- 172.21.15.81:0/3797283548 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fbe40107ac0 con 0x7fbe24039600 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.367+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbe24039600 msgr2=0x7fbe2403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.367+0000 7fbe46a00700 1 --2- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbe24039600 0x7fbe2403bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c006fd0 tx=0x7fbe2c006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.367+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 msgr2=0x7fbe401086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.367+0000 7fbe46a00700 1 --2- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe401086f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbe30011b60 tx=0x7fbe30011c40 comp rx=0 tx=0).stop 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.367+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 shutdown_connections 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.367+0000 7fbe46a00700 1 --2- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbe24039600 0x7fbe2403bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.367+0000 7fbe46a00700 1 --2- 172.21.15.81:0/3797283548 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe40108e70 0x7fbe401086f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:26.015 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.367+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 >> 172.21.15.81:0/3797283548 conn(0x7fbe40102860 msgr2=0x7fbe40103360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:26.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.368+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 shutdown_connections 2024-04-12T21:05:26.016 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:25.368+0000 7fbe46a00700 1 -- 172.21.15.81:0/3797283548 wait complete. 2024-04-12T21:05:26.016 INFO:teuthology.orchestra.run.smithi081.stdout:Creating initial admin user... 2024-04-12T21:05:26.560 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:26 smithi081 ceph-mon[31453]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:05:26.560 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:26.560 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:27.746 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$cKkctpK6a6HsMigfSZ3Wa.wPxUUKlNI5lmlGyIn4PcI3/hOe0EGIu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712955927, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-12T21:05:27.746 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.562+0000 7fd91a200700 1 Processor -- start 2024-04-12T21:05:27.746 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.563+0000 7fd91a200700 1 -- start start 2024-04-12T21:05:27.746 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.563+0000 7fd91a200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd914109db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:27.746 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.563+0000 7fd91a200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd91410a2f0 con 0x7fd914109990 2024-04-12T21:05:27.750 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.563+0000 7fd913400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd914109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:27.750 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.563+0000 7fd913400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd914109db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37130/0 (socket says 172.21.15.81:37130) 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.563+0000 7fd913400700 1 -- 172.21.15.81:0/2671016855 learned_addr learned my addr 172.21.15.81:0/2671016855 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.564+0000 7fd913400700 1 -- 172.21.15.81:0/2671016855 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd914108060 con 0x7fd914109990 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.564+0000 7fd913400700 1 --2- 172.21.15.81:0/2671016855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd914109db0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd9040098a0 tx=0x7fd904009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd67f9ed74f596a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.565+0000 7fd912000700 1 -- 172.21.15.81:0/2671016855 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd904004160 con 0x7fd914109990 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.565+0000 7fd912000700 1 -- 172.21.15.81:0/2671016855 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd904004300 con 0x7fd914109990 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.565+0000 7fd912000700 1 -- 172.21.15.81:0/2671016855 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd904011b20 con 0x7fd914109990 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.566+0000 7fd91a200700 1 -- 172.21.15.81:0/2671016855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 msgr2=0x7fd914109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.566+0000 7fd91a200700 1 --2- 172.21.15.81:0/2671016855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd914109db0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd9040098a0 tx=0x7fd904009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.566+0000 7fd91a200700 1 -- 172.21.15.81:0/2671016855 shutdown_connections 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.566+0000 7fd91a200700 1 --2- 172.21.15.81:0/2671016855 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd914109db0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.566+0000 7fd91a200700 1 -- 172.21.15.81:0/2671016855 >> 172.21.15.81:0/2671016855 conn(0x7fd914103a00 msgr2=0x7fd914105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.566+0000 7fd91a200700 1 -- 172.21.15.81:0/2671016855 shutdown_connections 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.566+0000 7fd91a200700 1 -- 172.21.15.81:0/2671016855 wait complete. 2024-04-12T21:05:27.751 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.567+0000 7fd91a200700 1 Processor -- start 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.567+0000 7fd91a200700 1 -- start start 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.567+0000 7fd91a200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd91419c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.567+0000 7fd91a200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd91410c990 con 0x7fd914109990 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.568+0000 7fd913400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd91419c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.568+0000 7fd913400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd91419c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37140/0 (socket says 172.21.15.81:37140) 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.568+0000 7fd913400700 1 -- 172.21.15.81:0/2621979173 learned_addr learned my addr 172.21.15.81:0/2621979173 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.568+0000 7fd913400700 1 -- 172.21.15.81:0/2621979173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd91419c930 con 0x7fd914109990 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.568+0000 7fd913400700 1 --2- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd91419c3f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd9040117e0 tx=0x7fd904011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.569+0000 7fd90be00700 1 -- 172.21.15.81:0/2621979173 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd904003bf0 con 0x7fd914109990 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.569+0000 7fd90be00700 1 -- 172.21.15.81:0/2621979173 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd90401a910 con 0x7fd914109990 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.569+0000 7fd90be00700 1 -- 172.21.15.81:0/2621979173 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd904024970 con 0x7fd914109990 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.569+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd91419cbc0 con 0x7fd914109990 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.569+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd91419d020 con 0x7fd914109990 2024-04-12T21:05:27.752 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.570+0000 7fd90be00700 1 -- 172.21.15.81:0/2621979173 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7fd904022020 con 0x7fd914109990 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.570+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd91410cb10 con 0x7fd914109990 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.571+0000 7fd90be00700 1 --2- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd8f8039610 0x7fd8f803bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.571+0000 7fd90be00700 1 -- 172.21.15.81:0/2621979173 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd90404dc10 con 0x7fd914109990 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.571+0000 7fd912a00700 1 --2- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd8f8039610 0x7fd8f803bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.572+0000 7fd912a00700 1 --2- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd8f8039610 0x7fd8f803bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd900006fd0 tx=0x7fd900006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.576+0000 7fd90be00700 1 -- 172.21.15.81:0/2621979173 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd904011e20 con 0x7fd914109990 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:26.818+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fd91419f9d0 con 0x7fd8f8039610 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.054+0000 7fd90be00700 1 -- 172.21.15.81:0/2621979173 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fd91419f9d0 con 0x7fd8f8039610 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd8f8039610 msgr2=0x7fd8f803bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 --2- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd8f8039610 0x7fd8f803bad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd900006fd0 tx=0x7fd900006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 msgr2=0x7fd91419c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 --2- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd91419c3f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd9040117e0 tx=0x7fd904011c10 comp rx=0 tx=0).stop 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 shutdown_connections 2024-04-12T21:05:27.753 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 --2- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd8f8039610 0x7fd8f803bad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:27.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 --2- 172.21.15.81:0/2621979173 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd914109990 0x7fd91419c3f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:27.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 >> 172.21.15.81:0/2621979173 conn(0x7fd914103a00 msgr2=0x7fd914104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:27.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 shutdown_connections 2024-04-12T21:05:27.754 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:27.057+0000 7fd91a200700 1 -- 172.21.15.81:0/2621979173 wait complete. 2024-04-12T21:05:27.754 INFO:teuthology.orchestra.run.smithi081.stdout:Fetching dashboard port number... 2024-04-12T21:05:28.362 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:28 smithi081 ceph-mon[31453]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:05:28.362 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:28.362 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:29.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:29 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/875813780' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-12T21:05:30.061 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stdout 8443 2024-04-12T21:05:30.062 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.364+0000 7f6edce00700 1 Processor -- start 2024-04-12T21:05:30.062 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.364+0000 7f6edce00700 1 -- start start 2024-04-12T21:05:30.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.365+0000 7f6edce00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed810b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:30.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.365+0000 7f6edce00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed807bdb0 con 0x7f6ed8108ef0 2024-04-12T21:05:30.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.365+0000 7f6ed5e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed810b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:30.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.365+0000 7f6ed5e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed810b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37142/0 (socket says 172.21.15.81:37142) 2024-04-12T21:05:30.063 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.365+0000 7f6ed5e00700 1 -- 172.21.15.81:0/1399950032 learned_addr learned my addr 172.21.15.81:0/1399950032 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:30.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.366+0000 7f6ed5e00700 1 -- 172.21.15.81:0/1399950032 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed810b820 con 0x7f6ed8108ef0 2024-04-12T21:05:30.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.366+0000 7f6ed5e00700 1 --2- 172.21.15.81:0/1399950032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed810b2e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80098a0 tx=0x7f6ec8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ffa0fd7f46fe84 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:30.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.367+0000 7f6ed5400700 1 -- 172.21.15.81:0/1399950032 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8004160 con 0x7f6ed8108ef0 2024-04-12T21:05:30.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.367+0000 7f6ed5400700 1 -- 172.21.15.81:0/1399950032 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ec8004300 con 0x7f6ed8108ef0 2024-04-12T21:05:30.065 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.367+0000 7f6ed5400700 1 -- 172.21.15.81:0/1399950032 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8011b20 con 0x7f6ed8108ef0 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.368+0000 7f6edce00700 1 -- 172.21.15.81:0/1399950032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 msgr2=0x7f6ed810b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.368+0000 7f6edce00700 1 --2- 172.21.15.81:0/1399950032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed810b2e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6ec80098a0 tx=0x7f6ec8009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.368+0000 7f6edce00700 1 -- 172.21.15.81:0/1399950032 shutdown_connections 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.368+0000 7f6edce00700 1 --2- 172.21.15.81:0/1399950032 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed810b2e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.368+0000 7f6edce00700 1 -- 172.21.15.81:0/1399950032 >> 172.21.15.81:0/1399950032 conn(0x7f6ed8102860 msgr2=0x7f6ed8104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.368+0000 7f6edce00700 1 -- 172.21.15.81:0/1399950032 shutdown_connections 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.368+0000 7f6edce00700 1 -- 172.21.15.81:0/1399950032 wait complete. 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.369+0000 7f6edce00700 1 Processor -- start 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.369+0000 7f6edce00700 1 -- start start 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.370+0000 7f6edce00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed8108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.370+0000 7f6edce00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed810c7c0 con 0x7f6ed8108ef0 2024-04-12T21:05:30.066 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.370+0000 7f6ed5e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed8108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:30.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.370+0000 7f6ed5e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed8108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37144/0 (socket says 172.21.15.81:37144) 2024-04-12T21:05:30.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.370+0000 7f6ed5e00700 1 -- 172.21.15.81:0/875813780 learned_addr learned my addr 172.21.15.81:0/875813780 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:30.069 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.370+0000 7f6ed5e00700 1 -- 172.21.15.81:0/875813780 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed8108cb0 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.371+0000 7f6ed5e00700 1 --2- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed8108770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8011b60 tx=0x7f6ec8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.371+0000 7f6ecea00700 1 -- 172.21.15.81:0/875813780 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8003b60 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.371+0000 7f6ecea00700 1 -- 172.21.15.81:0/875813780 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ec801a910 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.371+0000 7f6ecea00700 1 -- 172.21.15.81:0/875813780 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ec8024930 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.371+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ed8106ea0 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.371+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ed8107360 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.372+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ed8064110 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.372+0000 7f6ecea00700 1 -- 172.21.15.81:0/875813780 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f6ec8022020 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.373+0000 7f6ecea00700 1 --2- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6ebc039600 0x7f6ebc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.373+0000 7f6ecea00700 1 -- 172.21.15.81:0/875813780 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6ec804daf0 con 0x7f6ed8108ef0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.373+0000 7f6ecfe00700 1 --2- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6ebc039600 0x7f6ebc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:30.070 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.374+0000 7f6ecfe00700 1 --2- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6ebc039600 0x7f6ebc03bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0006fd0 tx=0x7f6ec0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.379+0000 7f6ecea00700 1 -- 172.21.15.81:0/875813780 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ec8024c50 con 0x7f6ed8108ef0 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.614+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f6ed81076a0 con 0x7f6ed8108ef0 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.615+0000 7f6ecea00700 1 -- 172.21.15.81:0/875813780 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f6ed81076a0 con 0x7f6ed8108ef0 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6ebc039600 msgr2=0x7f6ebc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 --2- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6ebc039600 0x7f6ebc03bac0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0006fd0 tx=0x7f6ec0006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 msgr2=0x7f6ed8108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 --2- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed8108770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8011b60 tx=0x7f6ec8011c40 comp rx=0 tx=0).stop 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 shutdown_connections 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 --2- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6ebc039600 0x7f6ebc03bac0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 --2- 172.21.15.81:0/875813780 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6ed8108ef0 0x7f6ed8108770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 >> 172.21.15.81:0/875813780 conn(0x7f6ed8102860 msgr2=0x7f6ed8103350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 shutdown_connections 2024-04-12T21:05:30.071 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:28.618+0000 7f6edce00700 1 -- 172.21.15.81:0/875813780 wait complete. 2024-04-12T21:05:30.108 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-12T21:05:30.109 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout disabled 2024-04-12T21:05:30.138 INFO:teuthology.orchestra.run.smithi081.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-12T21:05:30.138 INFO:teuthology.orchestra.run.smithi081.stdout:systemctl: stdout inactive 2024-04-12T21:05:30.138 INFO:teuthology.orchestra.run.smithi081.stdout:firewalld.service is not enabled 2024-04-12T21:05:30.138 INFO:teuthology.orchestra.run.smithi081.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-12T21:05:30.142 INFO:teuthology.orchestra.run.smithi081.stdout:Ceph Dashboard is now available at: 2024-04-12T21:05:30.143 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:30.143 INFO:teuthology.orchestra.run.smithi081.stdout: URL: https://smithi081.front.sepia.ceph.com:8443/ 2024-04-12T21:05:30.143 INFO:teuthology.orchestra.run.smithi081.stdout: User: admin 2024-04-12T21:05:30.143 INFO:teuthology.orchestra.run.smithi081.stdout: Password: c5wkf5eaeu 2024-04-12T21:05:30.143 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:30.144 INFO:teuthology.orchestra.run.smithi081.stdout:Saving cluster configuration to /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config directory 2024-04-12T21:05:30.146 INFO:teuthology.orchestra.run.smithi081.stdout:Enabling autotune for osd_memory_target 2024-04-12T21:05:30.301 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:30 smithi081 ceph-mon[31453]: mgrmap e13: x(active, since 6s) 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.791+0000 7f43ed400700 1 Processor -- start 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.791+0000 7f43ed400700 1 -- start start 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.792+0000 7f43ed400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e810a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.792+0000 7f43ed400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f43e810ad30 con 0x7f43e810a3d0 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.792+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e810a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.792+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e810a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37148/0 (socket says 172.21.15.81:37148) 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.792+0000 7f43e7e00700 1 -- 172.21.15.81:0/1015095837 learned_addr learned my addr 172.21.15.81:0/1015095837 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.793+0000 7f43e7e00700 1 -- 172.21.15.81:0/1015095837 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e810aeb0 con 0x7f43e810a3d0 2024-04-12T21:05:31.623 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.793+0000 7f43e7e00700 1 --2- 172.21.15.81:0/1015095837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e810a7f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f43d80098a0 tx=0x7f43d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0be11a3b9fb7c6d server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.793+0000 7f43e6a00700 1 -- 172.21.15.81:0/1015095837 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43d8004160 con 0x7f43e810a3d0 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.793+0000 7f43e6a00700 1 -- 172.21.15.81:0/1015095837 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43d8004300 con 0x7f43e810a3d0 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.793+0000 7f43e6a00700 1 -- 172.21.15.81:0/1015095837 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43d8011b20 con 0x7f43e810a3d0 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.794+0000 7f43ed400700 1 -- 172.21.15.81:0/1015095837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 msgr2=0x7f43e810a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.794+0000 7f43ed400700 1 --2- 172.21.15.81:0/1015095837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e810a7f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f43d80098a0 tx=0x7f43d8009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.794+0000 7f43ed400700 1 -- 172.21.15.81:0/1015095837 shutdown_connections 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.794+0000 7f43ed400700 1 --2- 172.21.15.81:0/1015095837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e810a7f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.794+0000 7f43ed400700 1 -- 172.21.15.81:0/1015095837 >> 172.21.15.81:0/1015095837 conn(0x7f43e807dbd0 msgr2=0x7f43e81084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.794+0000 7f43ed400700 1 -- 172.21.15.81:0/1015095837 shutdown_connections 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.794+0000 7f43ed400700 1 -- 172.21.15.81:0/1015095837 wait complete. 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.795+0000 7f43ed400700 1 Processor -- start 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.795+0000 7f43ed400700 1 -- start start 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.796+0000 7f43ed400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.796+0000 7f43ed400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f43e810e880 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.796+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.796+0000 7f43e7e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37150/0 (socket says 172.21.15.81:37150) 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.796+0000 7f43e7e00700 1 -- 172.21.15.81:0/948307309 learned_addr learned my addr 172.21.15.81:0/948307309 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.796+0000 7f43e7e00700 1 -- 172.21.15.81:0/948307309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e819eb90 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.797+0000 7f43e7e00700 1 --2- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e819e650 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43d80116a0 tx=0x7f43d8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.797+0000 7f43e4c00700 1 -- 172.21.15.81:0/948307309 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43d8003a40 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.797+0000 7f43e4c00700 1 -- 172.21.15.81:0/948307309 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43d8003be0 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.797+0000 7f43e4c00700 1 -- 172.21.15.81:0/948307309 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43d801a750 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.797+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e819ee20 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.797+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e819f280 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.798+0000 7f43e4c00700 1 -- 172.21.15.81:0/948307309 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f43d8024aa0 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.798+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e8064110 con 0x7f43e810a3d0 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.799+0000 7f43e4c00700 1 --2- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43cc0396b0 0x7f43cc03bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:31.625 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.799+0000 7f43e4c00700 1 -- 172.21.15.81:0/948307309 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f43d804df90 con 0x7f43e810a3d0 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.799+0000 7f43e7400700 1 --2- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43cc0396b0 0x7f43cc03bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.800+0000 7f43e7400700 1 --2- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43cc0396b0 0x7f43cc03bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f43d4006fd0 tx=0x7f43d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:30.804+0000 7f43e4c00700 1 -- 172.21.15.81:0/948307309 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43d8011e20 con 0x7f43e810a3d0 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.029+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f43e802e0e0 con 0x7f43e810a3d0 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.030+0000 7f43e4c00700 1 -- 172.21.15.81:0/948307309 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f43d8029de0 con 0x7f43e810a3d0 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43cc0396b0 msgr2=0x7f43cc03bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 --2- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43cc0396b0 0x7f43cc03bb70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f43d4006fd0 tx=0x7f43d4006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 msgr2=0x7f43e819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 --2- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e819e650 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43d80116a0 tx=0x7f43d8011ba0 comp rx=0 tx=0).stop 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 shutdown_connections 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 --2- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43cc0396b0 0x7f43cc03bb70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:31.626 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 --2- 172.21.15.81:0/948307309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f43e810a3d0 0x7f43e819e650 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:31.627 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 >> 172.21.15.81:0/948307309 conn(0x7f43e807dbd0 msgr2=0x7f43e8107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:31.627 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 shutdown_connections 2024-04-12T21:05:31.627 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:31.033+0000 7f43ed400700 1 -- 172.21.15.81:0/948307309 wait complete. 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.297+0000 7f0592800700 1 Processor -- start 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.297+0000 7f0592800700 1 -- start start 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.298+0000 7f0592800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c10a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.298+0000 7f0592800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f058c10ad30 con 0x7f058c10a3d0 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.298+0000 7f0591400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c10a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.298+0000 7f0591400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c10a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37162/0 (socket says 172.21.15.81:37162) 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.298+0000 7f0591400700 1 -- 172.21.15.81:0/421293110 learned_addr learned my addr 172.21.15.81:0/421293110 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.298+0000 7f0591400700 1 -- 172.21.15.81:0/421293110 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c10aeb0 con 0x7f058c10a3d0 2024-04-12T21:05:33.232 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.299+0000 7f0591400700 1 --2- 172.21.15.81:0/421293110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c10a7f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f05800098a0 tx=0x7f0580009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3c6757b2595d0f server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.299+0000 7f058be00700 1 -- 172.21.15.81:0/421293110 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0580004160 con 0x7f058c10a3d0 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.299+0000 7f058be00700 1 -- 172.21.15.81:0/421293110 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0580004300 con 0x7f058c10a3d0 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.299+0000 7f058be00700 1 -- 172.21.15.81:0/421293110 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0580011b20 con 0x7f058c10a3d0 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.300+0000 7f0592800700 1 -- 172.21.15.81:0/421293110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 msgr2=0x7f058c10a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.300+0000 7f0592800700 1 --2- 172.21.15.81:0/421293110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c10a7f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f05800098a0 tx=0x7f0580009bb0 comp rx=0 tx=0).stop 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.300+0000 7f0592800700 1 -- 172.21.15.81:0/421293110 shutdown_connections 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.300+0000 7f0592800700 1 --2- 172.21.15.81:0/421293110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c10a7f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.300+0000 7f0592800700 1 -- 172.21.15.81:0/421293110 >> 172.21.15.81:0/421293110 conn(0x7f058c07dbd0 msgr2=0x7f058c1084e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.300+0000 7f0592800700 1 -- 172.21.15.81:0/421293110 shutdown_connections 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.300+0000 7f0592800700 1 -- 172.21.15.81:0/421293110 wait complete. 2024-04-12T21:05:33.233 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.301+0000 7f0592800700 1 Processor -- start 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.301+0000 7f0592800700 1 -- start start 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.302+0000 7f0592800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.302+0000 7f0592800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f058c10e880 con 0x7f058c10a3d0 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.302+0000 7f0591400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.302+0000 7f0591400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37170/0 (socket says 172.21.15.81:37170) 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.302+0000 7f0591400700 1 -- 172.21.15.81:0/3979891958 learned_addr learned my addr 172.21.15.81:0/3979891958 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.302+0000 7f0591400700 1 -- 172.21.15.81:0/3979891958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c19eb90 con 0x7f058c10a3d0 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.303+0000 7f0591400700 1 --2- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c19e650 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0580011b60 tx=0x7f0580011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.303+0000 7f058a000700 1 -- 172.21.15.81:0/3979891958 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0580003b60 con 0x7f058c10a3d0 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.303+0000 7f058a000700 1 -- 172.21.15.81:0/3979891958 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f058001a910 con 0x7f058c10a3d0 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.303+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058c19ee20 con 0x7f058c10a3d0 2024-04-12T21:05:33.234 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.303+0000 7f058a000700 1 -- 172.21.15.81:0/3979891958 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0580024930 con 0x7f058c10a3d0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.303+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058c19f280 con 0x7f058c10a3d0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.304+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f058c064110 con 0x7f058c10a3d0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.304+0000 7f058a000700 1 -- 172.21.15.81:0/3979891958 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0580022020 con 0x7f058c10a3d0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.305+0000 7f058a000700 1 --2- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0574039700 0x7f057403bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.305+0000 7f058a000700 1 -- 172.21.15.81:0/3979891958 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f058004dab0 con 0x7f058c10a3d0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.305+0000 7f0590a00700 1 --2- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0574039700 0x7f057403bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.306+0000 7f0590a00700 1 --2- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0574039700 0x7f057403bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0578006fd0 tx=0x7f0578006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.311+0000 7f058a000700 1 -- 172.21.15.81:0/3979891958 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0580016260 con 0x7f058c10a3d0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.647+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f058c02e0e0 con 0x7f058c10a3d0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.653+0000 7f058a000700 1 -- 172.21.15.81:0/3979891958 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f058002de10 con 0x7f058c10a3d0 2024-04-12T21:05:33.235 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0574039700 msgr2=0x7f057403bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:33.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 --2- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0574039700 0x7f057403bbc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0578006fd0 tx=0x7f0578006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:33.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 msgr2=0x7f058c19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:33.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 --2- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c19e650 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0580011b60 tx=0x7f0580011c40 comp rx=0 tx=0).stop 2024-04-12T21:05:33.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 shutdown_connections 2024-04-12T21:05:33.237 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 --2- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0574039700 0x7f057403bbc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 --2- 172.21.15.81:0/3979891958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f058c10a3d0 0x7f058c19e650 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 >> 172.21.15.81:0/3979891958 conn(0x7f058c07dbd0 msgr2=0x7f058c107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 shutdown_connections 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr 2024-04-12T21:05:32.656+0000 7f0592800700 1 -- 172.21.15.81:0/3979891958 wait complete. 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout:Or, if you are only running a single cluster on this host: 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:33.238 INFO:teuthology.orchestra.run.smithi081.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-12T21:05:33.239 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:33.239 INFO:teuthology.orchestra.run.smithi081.stdout: ceph telemetry on 2024-04-12T21:05:33.239 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:33.239 INFO:teuthology.orchestra.run.smithi081.stdout:For more information see: 2024-04-12T21:05:33.239 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:33.239 INFO:teuthology.orchestra.run.smithi081.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-12T21:05:33.239 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:33.239 INFO:teuthology.orchestra.run.smithi081.stdout:Bootstrap complete. 2024-04-12T21:05:33.339 INFO:tasks.cephadm:Fetching config... 2024-04-12T21:05:33.340 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:05:33.340 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-12T21:05:33.394 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-12T21:05:33.394 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:05:33.394 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-12T21:05:33.437 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-12T21:05:33.437 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:05:33.437 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-12T21:05:33.569 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-12T21:05:33.570 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:05:33.570 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-12T21:05:33.616 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-12T21:05:33.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXqkKoEr5KYkIRFYpP4xG+T8r1weVu36b3NTQ1iIwlymlwRskBJSQRnc/ERDbjnVcQTvp6iIm+hsIR5ZG5FSshTF1+a33et+U5RF6N8KO/53Mreo0B86OXdCP/IzfXPgIl0oAsgohB+TCWQvxUoUIajhzsbD9+7YwZoznZhE67fWOdarzWBGUl4NjF60BaIAzdsejmbI2PN6Ie/EoK/XsRF1SLBFWTvh2k8Lv1g9shidVEo/b4lKv/OkP1L2jiJLSk9u2zbEpfKXyEFGyZrKlStvusU/X9qbfB5pspTwkuKmkAWrHNjKSMzVLCskb0k2F2JQY9L8ZUzQcZOMEDYR6rCPO0+98KHC2GvpgLb005VwBnsRJqkaaCcwLsmg5mOx9YGaDHW4wJttCniRjWyLh38JiB0KJxHBl2tOhopg4D7qV7hHKd/50Kim+KxWlNz6Up11zOMWCJnYGNoRIUQnmX+Y23qIehkI9XSizkxeqeStGVjCGM/2NR7/1NeBMQTyM= ceph-11b8cc88-f910-11ee-bc8d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T21:05:33.783 INFO:teuthology.orchestra.run.smithi081.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXqkKoEr5KYkIRFYpP4xG+T8r1weVu36b3NTQ1iIwlymlwRskBJSQRnc/ERDbjnVcQTvp6iIm+hsIR5ZG5FSshTF1+a33et+U5RF6N8KO/53Mreo0B86OXdCP/IzfXPgIl0oAsgohB+TCWQvxUoUIajhzsbD9+7YwZoznZhE67fWOdarzWBGUl4NjF60BaIAzdsejmbI2PN6Ie/EoK/XsRF1SLBFWTvh2k8Lv1g9shidVEo/b4lKv/OkP1L2jiJLSk9u2zbEpfKXyEFGyZrKlStvusU/X9qbfB5pspTwkuKmkAWrHNjKSMzVLCskb0k2F2JQY9L8ZUzQcZOMEDYR6rCPO0+98KHC2GvpgLb005VwBnsRJqkaaCcwLsmg5mOx9YGaDHW4wJttCniRjWyLh38JiB0KJxHBl2tOhopg4D7qV7hHKd/50Kim+KxWlNz6Up11zOMWCJnYGNoRIUQnmX+Y23qIehkI9XSizkxeqeStGVjCGM/2NR7/1NeBMQTyM= ceph-11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:05:33.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:33 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3979891958' entity='client.admin' 2024-04-12T21:05:33.825 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXqkKoEr5KYkIRFYpP4xG+T8r1weVu36b3NTQ1iIwlymlwRskBJSQRnc/ERDbjnVcQTvp6iIm+hsIR5ZG5FSshTF1+a33et+U5RF6N8KO/53Mreo0B86OXdCP/IzfXPgIl0oAsgohB+TCWQvxUoUIajhzsbD9+7YwZoznZhE67fWOdarzWBGUl4NjF60BaIAzdsejmbI2PN6Ie/EoK/XsRF1SLBFWTvh2k8Lv1g9shidVEo/b4lKv/OkP1L2jiJLSk9u2zbEpfKXyEFGyZrKlStvusU/X9qbfB5pspTwkuKmkAWrHNjKSMzVLCskb0k2F2JQY9L8ZUzQcZOMEDYR6rCPO0+98KHC2GvpgLb005VwBnsRJqkaaCcwLsmg5mOx9YGaDHW4wJttCniRjWyLh38JiB0KJxHBl2tOhopg4D7qV7hHKd/50Kim+KxWlNz6Up11zOMWCJnYGNoRIUQnmX+Y23qIehkI9XSizkxeqeStGVjCGM/2NR7/1NeBMQTyM= ceph-11b8cc88-f910-11ee-bc8d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T21:05:33.871 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXqkKoEr5KYkIRFYpP4xG+T8r1weVu36b3NTQ1iIwlymlwRskBJSQRnc/ERDbjnVcQTvp6iIm+hsIR5ZG5FSshTF1+a33et+U5RF6N8KO/53Mreo0B86OXdCP/IzfXPgIl0oAsgohB+TCWQvxUoUIajhzsbD9+7YwZoznZhE67fWOdarzWBGUl4NjF60BaIAzdsejmbI2PN6Ie/EoK/XsRF1SLBFWTvh2k8Lv1g9shidVEo/b4lKv/OkP1L2jiJLSk9u2zbEpfKXyEFGyZrKlStvusU/X9qbfB5pspTwkuKmkAWrHNjKSMzVLCskb0k2F2JQY9L8ZUzQcZOMEDYR6rCPO0+98KHC2GvpgLb005VwBnsRJqkaaCcwLsmg5mOx9YGaDHW4wJttCniRjWyLh38JiB0KJxHBl2tOhopg4D7qV7hHKd/50Kim+KxWlNz6Up11zOMWCJnYGNoRIUQnmX+Y23qIehkI9XSizkxeqeStGVjCGM/2NR7/1NeBMQTyM= ceph-11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:05:33.886 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXqkKoEr5KYkIRFYpP4xG+T8r1weVu36b3NTQ1iIwlymlwRskBJSQRnc/ERDbjnVcQTvp6iIm+hsIR5ZG5FSshTF1+a33et+U5RF6N8KO/53Mreo0B86OXdCP/IzfXPgIl0oAsgohB+TCWQvxUoUIajhzsbD9+7YwZoznZhE67fWOdarzWBGUl4NjF60BaIAzdsejmbI2PN6Ie/EoK/XsRF1SLBFWTvh2k8Lv1g9shidVEo/b4lKv/OkP1L2jiJLSk9u2zbEpfKXyEFGyZrKlStvusU/X9qbfB5pspTwkuKmkAWrHNjKSMzVLCskb0k2F2JQY9L8ZUzQcZOMEDYR6rCPO0+98KHC2GvpgLb005VwBnsRJqkaaCcwLsmg5mOx9YGaDHW4wJttCniRjWyLh38JiB0KJxHBl2tOhopg4D7qV7hHKd/50Kim+KxWlNz6Up11zOMWCJnYGNoRIUQnmX+Y23qIehkI9XSizkxeqeStGVjCGM/2NR7/1NeBMQTyM= ceph-11b8cc88-f910-11ee-bc8d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-12T21:05:33.934 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXqkKoEr5KYkIRFYpP4xG+T8r1weVu36b3NTQ1iIwlymlwRskBJSQRnc/ERDbjnVcQTvp6iIm+hsIR5ZG5FSshTF1+a33et+U5RF6N8KO/53Mreo0B86OXdCP/IzfXPgIl0oAsgohB+TCWQvxUoUIajhzsbD9+7YwZoznZhE67fWOdarzWBGUl4NjF60BaIAzdsejmbI2PN6Ie/EoK/XsRF1SLBFWTvh2k8Lv1g9shidVEo/b4lKv/OkP1L2jiJLSk9u2zbEpfKXyEFGyZrKlStvusU/X9qbfB5pspTwkuKmkAWrHNjKSMzVLCskb0k2F2JQY9L8ZUzQcZOMEDYR6rCPO0+98KHC2GvpgLb005VwBnsRJqkaaCcwLsmg5mOx9YGaDHW4wJttCniRjWyLh38JiB0KJxHBl2tOhopg4D7qV7hHKd/50Kim+KxWlNz6Up11zOMWCJnYGNoRIUQnmX+Y23qIehkI9XSizkxeqeStGVjCGM/2NR7/1NeBMQTyM= ceph-11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:05:33.949 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-12T21:05:34.385 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:05:37.051 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:37.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:37.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi081", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:05:37.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:37.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:05:37.052 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:37.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.389+0000 7f99a4600700 1 -- 172.21.15.81:0/2757674747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 msgr2=0x7f999c102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:37.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.389+0000 7f99a4600700 1 --2- 172.21.15.81:0/2757674747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 0x7f999c102930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9990009910 tx=0x7f9990009c20 comp rx=0 tx=0).stop 2024-04-12T21:05:37.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.389+0000 7f99a4600700 1 -- 172.21.15.81:0/2757674747 shutdown_connections 2024-04-12T21:05:37.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.389+0000 7f99a4600700 1 --2- 172.21.15.81:0/2757674747 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 0x7f999c102930 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.389+0000 7f99a4600700 1 -- 172.21.15.81:0/2757674747 >> 172.21.15.81:0/2757674747 conn(0x7f999c0fdab0 msgr2=0x7f999c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.389+0000 7f99a4600700 1 -- 172.21.15.81:0/2757674747 shutdown_connections 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.390+0000 7f99a4600700 1 -- 172.21.15.81:0/2757674747 wait complete. 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.391+0000 7f99a4600700 1 Processor -- start 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.391+0000 7f99a4600700 1 -- start start 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.391+0000 7f99a4600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 0x7f999c076770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.391+0000 7f99a4600700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f999c1085a0 con 0x7f999c102510 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.391+0000 7f99a1c00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 0x7f999c076770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.391+0000 7f99a1c00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 0x7f999c076770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34732/0 (socket says 172.21.15.81:34732) 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.392+0000 7f99a1c00700 1 -- 172.21.15.81:0/964705006 learned_addr learned my addr 172.21.15.81:0/964705006 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.392+0000 7f99a1c00700 1 -- 172.21.15.81:0/964705006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f999c076cb0 con 0x7f999c102510 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.392+0000 7f99a1c00700 1 --2- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 0x7f999c076770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9990005850 tx=0x7f9990005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.393+0000 7f999a000700 1 -- 172.21.15.81:0/964705006 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9990026070 con 0x7f999c102510 2024-04-12T21:05:37.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.393+0000 7f999a000700 1 -- 172.21.15.81:0/964705006 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99900044a0 con 0x7f999c102510 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.393+0000 7f999a000700 1 -- 172.21.15.81:0/964705006 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9990003ab0 con 0x7f999c102510 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.393+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f999c074ea0 con 0x7f999c102510 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.393+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f999c0753e0 con 0x7f999c102510 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.394+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f999c108d20 con 0x7f999c102510 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.394+0000 7f999a000700 1 -- 172.21.15.81:0/964705006 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f9990029070 con 0x7f999c102510 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.395+0000 7f999a000700 1 --2- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f99840396b0 0x7f998403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.395+0000 7f999a000700 1 -- 172.21.15.81:0/964705006 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f999001e070 con 0x7f999c102510 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.395+0000 7f99a1200700 1 --2- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f99840396b0 0x7f998403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.396+0000 7f99a1200700 1 --2- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f99840396b0 0x7f998403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9988006fd0 tx=0x7f9988006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:37.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.401+0000 7f999a000700 1 -- 172.21.15.81:0/964705006 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9990030020 con 0x7f999c102510 2024-04-12T21:05:37.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.628+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f999c190fd0 con 0x7f999c102510 2024-04-12T21:05:37.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.634+0000 7f999a000700 1 -- 172.21.15.81:0/964705006 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f9990013070 con 0x7f999c102510 2024-04-12T21:05:37.633 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.637+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f99840396b0 msgr2=0x7f998403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:37.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.637+0000 7f99a4600700 1 --2- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f99840396b0 0x7f998403bb70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9988006fd0 tx=0x7f9988006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:37.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.637+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 msgr2=0x7f999c076770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:37.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.637+0000 7f99a4600700 1 --2- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 0x7f999c076770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9990005850 tx=0x7f9990005030 comp rx=0 tx=0).stop 2024-04-12T21:05:37.634 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.638+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 shutdown_connections 2024-04-12T21:05:37.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.638+0000 7f99a4600700 1 --2- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f99840396b0 0x7f998403bb70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:37.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.638+0000 7f99a4600700 1 --2- 172.21.15.81:0/964705006 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f999c102510 0x7f999c076770 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:37.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.638+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 >> 172.21.15.81:0/964705006 conn(0x7f999c0fdab0 msgr2=0x7f999c0fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:37.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.638+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 shutdown_connections 2024-04-12T21:05:37.635 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:37.638+0000 7f99a4600700 1 -- 172.21.15.81:0/964705006 wait complete. 2024-04-12T21:05:38.345 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-12T21:05:38.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-12T21:05:38.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:38 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/964705006' entity='client.admin' 2024-04-12T21:05:38.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:05:38.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:39.419 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:05:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.350+0000 7f669a200700 1 -- 172.21.15.81:0/406481690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 msgr2=0x7f6694104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.350+0000 7f669a200700 1 --2- 172.21.15.81:0/406481690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 0x7f6694104040 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f668400b0a0 tx=0x7f668400b3b0 comp rx=0 tx=0).stop 2024-04-12T21:05:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.350+0000 7f669a200700 1 -- 172.21.15.81:0/406481690 shutdown_connections 2024-04-12T21:05:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.350+0000 7f669a200700 1 --2- 172.21.15.81:0/406481690 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 0x7f6694104040 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.350+0000 7f669a200700 1 -- 172.21.15.81:0/406481690 >> 172.21.15.81:0/406481690 conn(0x7f66940fdb00 msgr2=0x7f66940fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.350+0000 7f669a200700 1 -- 172.21.15.81:0/406481690 shutdown_connections 2024-04-12T21:05:41.347 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.350+0000 7f669a200700 1 -- 172.21.15.81:0/406481690 wait complete. 2024-04-12T21:05:41.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.351+0000 7f669a200700 1 Processor -- start 2024-04-12T21:05:41.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.351+0000 7f669a200700 1 -- start start 2024-04-12T21:05:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.352+0000 7f669a200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 0x7f6694076770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.352+0000 7f669a200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f66941086e0 con 0x7f6694103c20 2024-04-12T21:05:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.352+0000 7f6693400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 0x7f6694076770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.352+0000 7f6693400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 0x7f6694076770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:34748/0 (socket says 172.21.15.81:34748) 2024-04-12T21:05:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.352+0000 7f6693400700 1 -- 172.21.15.81:0/3303653884 learned_addr learned my addr 172.21.15.81:0/3303653884 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.352+0000 7f6693400700 1 -- 172.21.15.81:0/3303653884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6694076cb0 con 0x7f6694103c20 2024-04-12T21:05:41.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.352+0000 7f6693400700 1 --2- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 0x7f6694076770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6684007890 tx=0x7f6684009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.353+0000 7f668be00700 1 -- 172.21.15.81:0/3303653884 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6684017040 con 0x7f6694103c20 2024-04-12T21:05:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.353+0000 7f668be00700 1 -- 172.21.15.81:0/3303653884 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f668400bd50 con 0x7f6694103c20 2024-04-12T21:05:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.353+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6694074ea0 con 0x7f6694103c20 2024-04-12T21:05:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.353+0000 7f668be00700 1 -- 172.21.15.81:0/3303653884 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f668401d7a0 con 0x7f6694103c20 2024-04-12T21:05:41.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.353+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66940753e0 con 0x7f6694103c20 2024-04-12T21:05:41.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.354+0000 7f668be00700 1 -- 172.21.15.81:0/3303653884 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6684024020 con 0x7f6694103c20 2024-04-12T21:05:41.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.354+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6694108e60 con 0x7f6694103c20 2024-04-12T21:05:41.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.355+0000 7f668be00700 1 --2- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6678039700 0x7f667803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:41.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.355+0000 7f668be00700 1 -- 172.21.15.81:0/3303653884 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6684022020 con 0x7f6694103c20 2024-04-12T21:05:41.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.355+0000 7f6692a00700 1 --2- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6678039700 0x7f667803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:41.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.355+0000 7f6692a00700 1 --2- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6678039700 0x7f667803bbc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6680006fd0 tx=0x7f6680006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:41.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.360+0000 7f668be00700 1 -- 172.21.15.81:0/3303653884 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f668402b020 con 0x7f6694103c20 2024-04-12T21:05:41.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.590+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f6694191060 con 0x7f6678039700 2024-04-12T21:05:41.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.597+0000 7f668be00700 1 -- 172.21.15.81:0/3303653884 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6694191060 con 0x7f6678039700 2024-04-12T21:05:41.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.599+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6678039700 msgr2=0x7f667803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:41.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.599+0000 7f669a200700 1 --2- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6678039700 0x7f667803bbc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6680006fd0 tx=0x7f6680006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:41.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.599+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 msgr2=0x7f6694076770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:41.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.599+0000 7f669a200700 1 --2- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 0x7f6694076770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6684007890 tx=0x7f6684009790 comp rx=0 tx=0).stop 2024-04-12T21:05:41.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.600+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 shutdown_connections 2024-04-12T21:05:41.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.600+0000 7f669a200700 1 --2- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6678039700 0x7f667803bbc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:41.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.600+0000 7f669a200700 1 --2- 172.21.15.81:0/3303653884 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6694103c20 0x7f6694076770 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:41.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.600+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 >> 172.21.15.81:0/3303653884 conn(0x7f66940fdb00 msgr2=0x7f66940fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.600+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 shutdown_connections 2024-04-12T21:05:41.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:41.600+0000 7f669a200700 1 -- 172.21.15.81:0/3303653884 wait complete. 2024-04-12T21:05:42.388 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2024-04-12T21:05:42.388 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:05:42.388 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2024-04-12T21:05:42.406 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:05:42.406 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:05:42.463 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2024-04-12T21:05:42.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch host add smithi104 2024-04-12T21:05:42.880 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:05:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:42 smithi081 ceph-mon[31453]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:05:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:05:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:05:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:05:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:42 smithi081 ceph-mon[31453]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-12T21:05:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:42 smithi081 ceph-mon[31453]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:42 smithi081 ceph-mon[31453]: Updating smithi081:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:05:43.869 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:43 smithi081 ceph-mon[31453]: Updating smithi081:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:05:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.995+0000 7f4958000700 1 -- 172.21.15.81:0/1743358647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 msgr2=0x7f495010ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.995+0000 7f4958000700 1 --2- 172.21.15.81:0/1743358647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 0x7f495010ddb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4944009910 tx=0x7f4944009c20 comp rx=0 tx=0).stop 2024-04-12T21:05:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.995+0000 7f4958000700 1 -- 172.21.15.81:0/1743358647 shutdown_connections 2024-04-12T21:05:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.995+0000 7f4958000700 1 --2- 172.21.15.81:0/1743358647 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 0x7f495010ddb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.995+0000 7f4958000700 1 -- 172.21.15.81:0/1743358647 >> 172.21.15.81:0/1743358647 conn(0x7f4950078250 msgr2=0x7f495007a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.995+0000 7f4958000700 1 -- 172.21.15.81:0/1743358647 shutdown_connections 2024-04-12T21:05:44.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.995+0000 7f4958000700 1 -- 172.21.15.81:0/1743358647 wait complete. 2024-04-12T21:05:44.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.996+0000 7f4958000700 1 Processor -- start 2024-04-12T21:05:44.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.996+0000 7f4958000700 1 -- start start 2024-04-12T21:05:44.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.996+0000 7f4958000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 0x7f495019e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:44.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.996+0000 7f4958000700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4950113120 con 0x7f495010d9d0 2024-04-12T21:05:44.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.997+0000 7f4955600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 0x7f495019e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:44.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.997+0000 7f4955600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 0x7f495019e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60384/0 (socket says 172.21.15.81:60384) 2024-04-12T21:05:44.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.997+0000 7f4955600700 1 -- 172.21.15.81:0/3413334395 learned_addr learned my addr 172.21.15.81:0/3413334395 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:44.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.997+0000 7f4955600700 1 -- 172.21.15.81:0/3413334395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495019eb00 con 0x7f495010d9d0 2024-04-12T21:05:44.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.997+0000 7f4955600700 1 --2- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 0x7f495019e5c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4944006010 tx=0x7f4944005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:44.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.998+0000 7f494e000700 1 -- 172.21.15.81:0/3413334395 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4944026070 con 0x7f495010d9d0 2024-04-12T21:05:44.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.998+0000 7f494e000700 1 -- 172.21.15.81:0/3413334395 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f494400bbd0 con 0x7f495010d9d0 2024-04-12T21:05:44.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.998+0000 7f494e000700 1 -- 172.21.15.81:0/3413334395 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49440039d0 con 0x7f495010d9d0 2024-04-12T21:05:44.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.998+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495019edf0 con 0x7f495010d9d0 2024-04-12T21:05:44.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.998+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49501a2e90 con 0x7f495010d9d0 2024-04-12T21:05:44.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.999+0000 7f494e000700 1 -- 172.21.15.81:0/3413334395 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4944029070 con 0x7f495010d9d0 2024-04-12T21:05:44.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:44.999+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f495010a1f0 con 0x7f495010d9d0 2024-04-12T21:05:44.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:45.000+0000 7f494e000700 1 --2- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f49380396b0 0x7f493803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:44.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:45.000+0000 7f494e000700 1 -- 172.21.15.81:0/3413334395 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f494401e070 con 0x7f495010d9d0 2024-04-12T21:05:44.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:45.000+0000 7f4954c00700 1 --2- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f49380396b0 0x7f493803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:44.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:45.001+0000 7f4954c00700 1 --2- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f49380396b0 0x7f493803bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4940006fd0 tx=0x7f4940006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:45.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:45.005+0000 7f494e000700 1 -- 172.21.15.81:0/3413334395 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4944013070 con 0x7f495010d9d0 2024-04-12T21:05:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:45 smithi081 ceph-mon[31453]: Updating smithi081:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T21:05:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:45 smithi081 ceph-mon[31453]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:45.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:45.237+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}) -- 0x7f495019f920 con 0x7f49380396b0 2024-04-12T21:05:46.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:46 smithi081 ceph-mon[31453]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:05:47.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:47 smithi081 ceph-mon[31453]: Deploying cephadm binary to smithi104 2024-04-12T21:05:47.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:47 smithi081 ceph-mon[31453]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:49.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:49 smithi081 ceph-mon[31453]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:50.176 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.179+0000 7f494e000700 1 -- 172.21.15.81:0/3413334395 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f495019f920 con 0x7f49380396b0 2024-04-12T21:05:50.176 INFO:teuthology.orchestra.run.smithi081.stdout:Added host 'smithi104' with addr '172.21.15.104' 2024-04-12T21:05:50.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.183+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f49380396b0 msgr2=0x7f493803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:50.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.183+0000 7f4958000700 1 --2- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f49380396b0 0x7f493803bb70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4940006fd0 tx=0x7f4940006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:50.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.183+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 msgr2=0x7f495019e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:50.179 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.183+0000 7f4958000700 1 --2- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 0x7f495019e5c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4944006010 tx=0x7f4944005030 comp rx=0 tx=0).stop 2024-04-12T21:05:50.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.183+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 shutdown_connections 2024-04-12T21:05:50.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.183+0000 7f4958000700 1 --2- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f49380396b0 0x7f493803bb70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:50.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.183+0000 7f4958000700 1 --2- 172.21.15.81:0/3413334395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f495010d9d0 0x7f495019e5c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:50.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.183+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 >> 172.21.15.81:0/3413334395 conn(0x7f4950078250 msgr2=0x7f4950110480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:50.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.184+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 shutdown_connections 2024-04-12T21:05:50.180 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:50.184+0000 7f4958000700 1 -- 172.21.15.81:0/3413334395 wait complete. 2024-04-12T21:05:50.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch host ls --format=json 2024-04-12T21:05:51.331 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:05:51.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:51.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:51 smithi081 ceph-mon[31453]: Added host smithi104 2024-04-12T21:05:51.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:05:51.483 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:51 smithi081 ceph-mon[31453]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:51.483 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:52.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.446+0000 7fedcbc00700 1 -- 172.21.15.81:0/3206215200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 msgr2=0x7fedc410aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.446+0000 7fedcbc00700 1 --2- 172.21.15.81:0/3206215200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 0x7fedc410aab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009910 tx=0x7fedb4009c20 comp rx=0 tx=0).stop 2024-04-12T21:05:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.446+0000 7fedcbc00700 1 -- 172.21.15.81:0/3206215200 shutdown_connections 2024-04-12T21:05:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.446+0000 7fedcbc00700 1 --2- 172.21.15.81:0/3206215200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 0x7fedc410aab0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.446+0000 7fedcbc00700 1 -- 172.21.15.81:0/3206215200 >> 172.21.15.81:0/3206215200 conn(0x7fedc41001c0 msgr2=0x7fedc41025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.446+0000 7fedcbc00700 1 -- 172.21.15.81:0/3206215200 shutdown_connections 2024-04-12T21:05:53.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.446+0000 7fedcbc00700 1 -- 172.21.15.81:0/3206215200 wait complete. 2024-04-12T21:05:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.447+0000 7fedcbc00700 1 Processor -- start 2024-04-12T21:05:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.447+0000 7fedcbc00700 1 -- start start 2024-04-12T21:05:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.448+0000 7fedcbc00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 0x7fedc419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.448+0000 7fedcbc00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fedc410ed10 con 0x7fedc410a6d0 2024-04-12T21:05:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.448+0000 7fedc9200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 0x7fedc419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.448+0000 7fedc9200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 0x7fedc419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60394/0 (socket says 172.21.15.81:60394) 2024-04-12T21:05:53.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.448+0000 7fedc9200700 1 -- 172.21.15.81:0/1431426005 learned_addr learned my addr 172.21.15.81:0/1431426005 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:53.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.448+0000 7fedc9200700 1 -- 172.21.15.81:0/1431426005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc419a750 con 0x7fedc410a6d0 2024-04-12T21:05:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.449+0000 7fedc9200700 1 --2- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 0x7fedc419a210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fedb4006010 tx=0x7fedb4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.449+0000 7fedc2000700 1 -- 172.21.15.81:0/1431426005 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedb4026070 con 0x7fedc410a6d0 2024-04-12T21:05:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.449+0000 7fedc2000700 1 -- 172.21.15.81:0/1431426005 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedb400bbd0 con 0x7fedc410a6d0 2024-04-12T21:05:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.449+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc419aa40 con 0x7fedc410a6d0 2024-04-12T21:05:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.449+0000 7fedc2000700 1 -- 172.21.15.81:0/1431426005 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fedb4003910 con 0x7fedc410a6d0 2024-04-12T21:05:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.449+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc419eae0 con 0x7fedc410a6d0 2024-04-12T21:05:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.450+0000 7fedc2000700 1 -- 172.21.15.81:0/1431426005 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fedb4029070 con 0x7fedc410a6d0 2024-04-12T21:05:53.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.450+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc4105e10 con 0x7fedc410a6d0 2024-04-12T21:05:53.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.451+0000 7fedc2000700 1 --2- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feda8039650 0x7feda803bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:53.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.451+0000 7fedc2000700 1 -- 172.21.15.81:0/1431426005 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fedb401e070 con 0x7fedc410a6d0 2024-04-12T21:05:53.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.451+0000 7fedbbe00700 1 --2- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feda8039650 0x7feda803bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:53.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.452+0000 7fedbbe00700 1 --2- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feda8039650 0x7feda803bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fedac006fd0 tx=0x7fedac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:53.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.457+0000 7fedc2000700 1 -- 172.21.15.81:0/1431426005 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fedb4013070 con 0x7fedc410a6d0 2024-04-12T21:05:53.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:53 smithi081 ceph-mon[31453]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:53.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:53.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.687+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fedc419b570 con 0x7feda8039650 2024-04-12T21:05:53.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.689+0000 7fedc2000700 1 -- 172.21.15.81:0/1431426005 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fedc419b570 con 0x7feda8039650 2024-04-12T21:05:53.686 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:05:53.686 INFO:teuthology.orchestra.run.smithi081.stdout:[{"addr": "172.21.15.81", "hostname": "smithi081", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2024-04-12T21:05:53.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feda8039650 msgr2=0x7feda803bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:53.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 --2- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feda8039650 0x7feda803bb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fedac006fd0 tx=0x7fedac006e40 comp rx=0 tx=0).stop 2024-04-12T21:05:53.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 msgr2=0x7fedc419a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:53.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 --2- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 0x7fedc419a210 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fedb4006010 tx=0x7fedb4005030 comp rx=0 tx=0).stop 2024-04-12T21:05:53.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 shutdown_connections 2024-04-12T21:05:53.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 --2- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feda8039650 0x7feda803bb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:53.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 --2- 172.21.15.81:0/1431426005 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fedc410a6d0 0x7fedc419a210 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:53.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 >> 172.21.15.81:0/1431426005 conn(0x7fedc41001c0 msgr2=0x7fedc4100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:53.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.692+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 shutdown_connections 2024-04-12T21:05:53.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:53.693+0000 7fedcbc00700 1 -- 172.21.15.81:0/1431426005 wait complete. 2024-04-12T21:05:54.410 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-04-12T21:05:54.411 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:05:54.411 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-04-12T21:05:54.430 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:05:54.430 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:05:54.487 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-04-12T21:05:54.488 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch host add smithi153 2024-04-12T21:05:54.897 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:05:55.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:55 smithi081 ceph-mon[31453]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:05:55.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:55 smithi081 ceph-mon[31453]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:56.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.756+0000 7fcc5ee00700 1 -- 172.21.15.81:0/2297186675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 msgr2=0x7fcc58108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:05:56.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.756+0000 7fcc5ee00700 1 --2- 172.21.15.81:0/2297186675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 0x7fcc58108960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800b0a0 tx=0x7fcc4800b3b0 comp rx=0 tx=0).stop 2024-04-12T21:05:56.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.756+0000 7fcc5ee00700 1 -- 172.21.15.81:0/2297186675 shutdown_connections 2024-04-12T21:05:56.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.756+0000 7fcc5ee00700 1 --2- 172.21.15.81:0/2297186675 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 0x7fcc58108960 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:05:56.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.756+0000 7fcc5ee00700 1 -- 172.21.15.81:0/2297186675 >> 172.21.15.81:0/2297186675 conn(0x7fcc580fdfa0 msgr2=0x7fcc581003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:05:56.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.756+0000 7fcc5ee00700 1 -- 172.21.15.81:0/2297186675 shutdown_connections 2024-04-12T21:05:56.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.756+0000 7fcc5ee00700 1 -- 172.21.15.81:0/2297186675 wait complete. 2024-04-12T21:05:56.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.757+0000 7fcc5ee00700 1 Processor -- start 2024-04-12T21:05:56.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.758+0000 7fcc5ee00700 1 -- start start 2024-04-12T21:05:56.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.758+0000 7fcc5ee00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 0x7fcc5819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:56.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.758+0000 7fcc5ee00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5810cb40 con 0x7fcc58108580 2024-04-12T21:05:56.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.758+0000 7fcc57e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 0x7fcc5819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:56.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.758+0000 7fcc57e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 0x7fcc5819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38188/0 (socket says 172.21.15.81:38188) 2024-04-12T21:05:56.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.758+0000 7fcc57e00700 1 -- 172.21.15.81:0/417259158 learned_addr learned my addr 172.21.15.81:0/417259158 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:05:56.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.759+0000 7fcc57e00700 1 -- 172.21.15.81:0/417259158 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5819a600 con 0x7fcc58108580 2024-04-12T21:05:56.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.759+0000 7fcc57e00700 1 --2- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 0x7fcc5819a0c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcc48000f80 tx=0x7fcc48003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:56.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.759+0000 7fcc55600700 1 -- 172.21.15.81:0/417259158 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc48017040 con 0x7fcc58108580 2024-04-12T21:05:56.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.759+0000 7fcc55600700 1 -- 172.21.15.81:0/417259158 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc48022070 con 0x7fcc58108580 2024-04-12T21:05:56.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.759+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5819a890 con 0x7fcc58108580 2024-04-12T21:05:56.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.759+0000 7fcc55600700 1 -- 172.21.15.81:0/417259158 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc4801d440 con 0x7fcc58108580 2024-04-12T21:05:56.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.760+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5819e990 con 0x7fcc58108580 2024-04-12T21:05:56.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.761+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc58103bf0 con 0x7fcc58108580 2024-04-12T21:05:56.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.761+0000 7fcc55600700 1 -- 172.21.15.81:0/417259158 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fcc48004000 con 0x7fcc58108580 2024-04-12T21:05:56.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.761+0000 7fcc55600700 1 --2- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcc3c039700 0x7fcc3c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:05:56.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.761+0000 7fcc55600700 1 -- 172.21.15.81:0/417259158 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcc4800f050 con 0x7fcc58108580 2024-04-12T21:05:56.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.762+0000 7fcc4f400700 1 --2- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcc3c039700 0x7fcc3c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:05:56.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.762+0000 7fcc4f400700 1 --2- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcc3c039700 0x7fcc3c03bbc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcc40006fd0 tx=0x7fcc40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:05:56.763 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.767+0000 7fcc55600700 1 -- 172.21.15.81:0/417259158 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc48007e20 con 0x7fcc58108580 2024-04-12T21:05:56.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:05:56.998+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}) -- 0x7fcc5819b420 con 0x7fcc3c039700 2024-04-12T21:05:57.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:57.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:57.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:57.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:57.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:05:57.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:05:57.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:05:57.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-12T21:05:57.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:05:57.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:56 smithi081 ceph-mon[31453]: Updating smithi104:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:05:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:58 smithi081 ceph-mon[31453]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:05:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:58 smithi081 ceph-mon[31453]: Updating smithi104:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T21:05:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:58 smithi081 ceph-mon[31453]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:05:58.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:58.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:58.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:05:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:59 smithi081 ceph-mon[31453]: Deploying cephadm binary to smithi153 2024-04-12T21:05:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:05:59 smithi081 ceph-mon[31453]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:01 smithi081 ceph-mon[31453]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:01.685 INFO:teuthology.orchestra.run.smithi081.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-04-12T21:06:01.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.686+0000 7fcc55600700 1 -- 172.21.15.81:0/417259158 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fcc5819b420 con 0x7fcc3c039700 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.691+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcc3c039700 msgr2=0x7fcc3c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.691+0000 7fcc5ee00700 1 --2- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcc3c039700 0x7fcc3c03bbc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fcc40006fd0 tx=0x7fcc40006e40 comp rx=0 tx=0).stop 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.691+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 msgr2=0x7fcc5819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.691+0000 7fcc5ee00700 1 --2- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 0x7fcc5819a0c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcc48000f80 tx=0x7fcc48003ce0 comp rx=0 tx=0).stop 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.691+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 shutdown_connections 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.691+0000 7fcc5ee00700 1 --2- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcc3c039700 0x7fcc3c03bbc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.691+0000 7fcc5ee00700 1 --2- 172.21.15.81:0/417259158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcc58108580 0x7fcc5819a0c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.691+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 >> 172.21.15.81:0/417259158 conn(0x7fcc580fdfa0 msgr2=0x7fcc58076660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:01.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.692+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 shutdown_connections 2024-04-12T21:06:01.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:01.692+0000 7fcc5ee00700 1 -- 172.21.15.81:0/417259158 wait complete. 2024-04-12T21:06:02.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch host ls --format=json 2024-04-12T21:06:02.843 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:06:02.977 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:02.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:02 smithi081 ceph-mon[31453]: Added host smithi153 2024-04-12T21:06:02.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:02.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:02 smithi081 ceph-mon[31453]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:02.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:03.941 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:04.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.700+0000 7f3fbf400700 1 -- 172.21.15.81:0/1349673468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 msgr2=0x7f3fb8108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:04.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.700+0000 7f3fbf400700 1 --2- 172.21.15.81:0/1349673468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 0x7f3fb8108970 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009910 tx=0x7f3fac009c20 comp rx=0 tx=0).stop 2024-04-12T21:06:04.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.700+0000 7f3fbf400700 1 -- 172.21.15.81:0/1349673468 shutdown_connections 2024-04-12T21:06:04.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.700+0000 7f3fbf400700 1 --2- 172.21.15.81:0/1349673468 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 0x7f3fb8108970 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:04.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.700+0000 7f3fbf400700 1 -- 172.21.15.81:0/1349673468 >> 172.21.15.81:0/1349673468 conn(0x7f3fb80fdff0 msgr2=0x7f3fb8100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:04.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.700+0000 7f3fbf400700 1 -- 172.21.15.81:0/1349673468 shutdown_connections 2024-04-12T21:06:04.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.700+0000 7f3fbf400700 1 -- 172.21.15.81:0/1349673468 wait complete. 2024-04-12T21:06:04.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.702+0000 7f3fbf400700 1 Processor -- start 2024-04-12T21:06:04.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.702+0000 7f3fbf400700 1 -- start start 2024-04-12T21:06:04.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.702+0000 7f3fbf400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 0x7f3fb8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:04.698 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.702+0000 7f3fbf400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb810cb50 con 0x7f3fb8108590 2024-04-12T21:06:04.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.703+0000 7f3fbca00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 0x7f3fb8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:04.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.703+0000 7f3fbca00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 0x7f3fb8197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38206/0 (socket says 172.21.15.81:38206) 2024-04-12T21:06:04.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.703+0000 7f3fbca00700 1 -- 172.21.15.81:0/241797942 learned_addr learned my addr 172.21.15.81:0/241797942 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:06:04.699 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.703+0000 7f3fbca00700 1 -- 172.21.15.81:0/241797942 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb8198490 con 0x7f3fb8108590 2024-04-12T21:06:04.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.703+0000 7f3fbca00700 1 --2- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 0x7f3fb8197f50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3fac006010 tx=0x7f3fac005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:04.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.704+0000 7f3fb5600700 1 -- 172.21.15.81:0/241797942 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fac026070 con 0x7f3fb8108590 2024-04-12T21:06:04.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.704+0000 7f3fb5600700 1 -- 172.21.15.81:0/241797942 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fac00bc30 con 0x7f3fb8108590 2024-04-12T21:06:04.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.704+0000 7f3fb5600700 1 -- 172.21.15.81:0/241797942 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fac0039d0 con 0x7f3fb8108590 2024-04-12T21:06:04.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.704+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb8198720 con 0x7f3fb8108590 2024-04-12T21:06:04.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.704+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb819c820 con 0x7f3fb8108590 2024-04-12T21:06:04.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.705+0000 7f3fb5600700 1 -- 172.21.15.81:0/241797942 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3fac029070 con 0x7f3fb8108590 2024-04-12T21:06:04.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.705+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fb8103c00 con 0x7f3fb8108590 2024-04-12T21:06:04.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.705+0000 7f3fb5600700 1 --2- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3fa0039650 0x7f3fa003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:04.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.705+0000 7f3fb5600700 1 -- 172.21.15.81:0/241797942 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3fac01e070 con 0x7f3fb8108590 2024-04-12T21:06:04.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.706+0000 7f3fb7e00700 1 --2- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3fa0039650 0x7f3fa003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:04.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.706+0000 7f3fb7e00700 1 --2- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3fa0039650 0x7f3fa003bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4006fd0 tx=0x7f3fa4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:04.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.711+0000 7f3fb5600700 1 -- 172.21.15.81:0/241797942 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3fac013070 con 0x7f3fb8108590 2024-04-12T21:06:04.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.945+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3fb81992b0 con 0x7f3fa0039650 2024-04-12T21:06:04.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.948+0000 7f3fb5600700 1 -- 172.21.15.81:0/241797942 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f3fb81992b0 con 0x7f3fa0039650 2024-04-12T21:06:04.944 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:06:04.944 INFO:teuthology.orchestra.run.smithi081.stdout:[{"addr": "172.21.15.81", "hostname": "smithi081", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-04-12T21:06:04.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.950+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3fa0039650 msgr2=0x7f3fa003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:04.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.950+0000 7f3fbf400700 1 --2- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3fa0039650 0x7f3fa003bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4006fd0 tx=0x7f3fa4006e40 comp rx=0 tx=0).stop 2024-04-12T21:06:04.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.951+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 msgr2=0x7f3fb8197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:04.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.951+0000 7f3fbf400700 1 --2- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 0x7f3fb8197f50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3fac006010 tx=0x7f3fac005030 comp rx=0 tx=0).stop 2024-04-12T21:06:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.951+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 shutdown_connections 2024-04-12T21:06:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.951+0000 7f3fbf400700 1 --2- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3fa0039650 0x7f3fa003bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.951+0000 7f3fbf400700 1 --2- 172.21.15.81:0/241797942 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb8108590 0x7f3fb8197f50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.951+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 >> 172.21.15.81:0/241797942 conn(0x7f3fb80fdff0 msgr2=0x7f3fb80ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.951+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 shutdown_connections 2024-04-12T21:06:04.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:04.951+0000 7f3fbf400700 1 -- 172.21.15.81:0/241797942 wait complete. 2024-04-12T21:06:05.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:04 smithi081 ceph-mon[31453]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:05.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:05.648 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-12T21:06:05.648 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd crush tunables default 2024-04-12T21:06:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:05 smithi081 ceph-mon[31453]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:06:06.094 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:06:07.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:06 smithi081 ceph-mon[31453]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:07.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.916+0000 7f61f5000700 1 -- 172.21.15.81:0/2224798669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 msgr2=0x7f61f010ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:07.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.916+0000 7f61f5000700 1 --2- 172.21.15.81:0/2224798669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 0x7f61f010ddb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f61e0009910 tx=0x7f61e0009c20 comp rx=0 tx=0).stop 2024-04-12T21:06:07.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.916+0000 7f61f5000700 1 -- 172.21.15.81:0/2224798669 shutdown_connections 2024-04-12T21:06:07.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.916+0000 7f61f5000700 1 --2- 172.21.15.81:0/2224798669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 0x7f61f010ddb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:07.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.916+0000 7f61f5000700 1 -- 172.21.15.81:0/2224798669 >> 172.21.15.81:0/2224798669 conn(0x7f61f0078250 msgr2=0x7f61f007a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:07.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.916+0000 7f61f5000700 1 -- 172.21.15.81:0/2224798669 shutdown_connections 2024-04-12T21:06:07.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.916+0000 7f61f5000700 1 -- 172.21.15.81:0/2224798669 wait complete. 2024-04-12T21:06:07.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.917+0000 7f61f5000700 1 Processor -- start 2024-04-12T21:06:07.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.917+0000 7f61f5000700 1 -- start start 2024-04-12T21:06:07.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.918+0000 7f61f5000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 0x7f61f019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:07.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.918+0000 7f61f5000700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0113120 con 0x7f61f010d9d0 2024-04-12T21:06:07.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.918+0000 7f61ede00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 0x7f61f019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:07.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.918+0000 7f61ede00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 0x7f61f019e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:54824/0 (socket says 172.21.15.81:54824) 2024-04-12T21:06:07.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.918+0000 7f61ede00700 1 -- 172.21.15.81:0/1663754903 learned_addr learned my addr 172.21.15.81:0/1663754903 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:06:07.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.918+0000 7f61ede00700 1 -- 172.21.15.81:0/1663754903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f019eaa0 con 0x7f61f010d9d0 2024-04-12T21:06:07.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.919+0000 7f61ede00700 1 --2- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 0x7f61f019e560 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f61e0006010 tx=0x7f61e0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:07.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.919+0000 7f61e6a00700 1 -- 172.21.15.81:0/1663754903 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61e0026070 con 0x7f61f010d9d0 2024-04-12T21:06:07.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.919+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f019ed90 con 0x7f61f010d9d0 2024-04-12T21:06:07.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.919+0000 7f61e6a00700 1 -- 172.21.15.81:0/1663754903 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e000bbd0 con 0x7f61f010d9d0 2024-04-12T21:06:07.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.919+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f01a2e30 con 0x7f61f010d9d0 2024-04-12T21:06:07.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.919+0000 7f61e6a00700 1 -- 172.21.15.81:0/1663754903 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61e00039f0 con 0x7f61f010d9d0 2024-04-12T21:06:07.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.920+0000 7f61e6a00700 1 -- 172.21.15.81:0/1663754903 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f61e0029070 con 0x7f61f010d9d0 2024-04-12T21:06:07.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.920+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f010a1f0 con 0x7f61f010d9d0 2024-04-12T21:06:07.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.921+0000 7f61e6a00700 1 --2- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61d4039650 0x7f61d403bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:07.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.921+0000 7f61e6a00700 1 -- 172.21.15.81:0/1663754903 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f61e001e070 con 0x7f61f010d9d0 2024-04-12T21:06:07.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.921+0000 7f61ed400700 1 --2- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61d4039650 0x7f61d403bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:07.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.922+0000 7f61ed400700 1 --2- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61d4039650 0x7f61d403bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f61dc006fd0 tx=0x7f61dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:07.923 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:07.927+0000 7f61e6a00700 1 -- 172.21.15.81:0/1663754903 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61e0013070 con 0x7f61f010d9d0 2024-04-12T21:06:08.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.160+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f61f019f8c0 con 0x7f61f010d9d0 2024-04-12T21:06:08.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.828+0000 7f61e6a00700 1 -- 172.21.15.81:0/1663754903 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f61e000f980 con 0x7f61f010d9d0 2024-04-12T21:06:08.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.834+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61d4039650 msgr2=0x7f61d403bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:08.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.834+0000 7f61f5000700 1 --2- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61d4039650 0x7f61d403bb10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f61dc006fd0 tx=0x7f61dc006e40 comp rx=0 tx=0).stop 2024-04-12T21:06:08.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.835+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 msgr2=0x7f61f019e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:08.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.835+0000 7f61f5000700 1 --2- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 0x7f61f019e560 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f61e0006010 tx=0x7f61e0005030 comp rx=0 tx=0).stop 2024-04-12T21:06:08.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.835+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 shutdown_connections 2024-04-12T21:06:08.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.835+0000 7f61f5000700 1 --2- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61d4039650 0x7f61d403bb10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:08.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.835+0000 7f61f5000700 1 --2- 172.21.15.81:0/1663754903 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f61f010d9d0 0x7f61f019e560 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:08.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.835+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 >> 172.21.15.81:0/1663754903 conn(0x7f61f0078250 msgr2=0x7f61f0110480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:08.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.835+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 shutdown_connections 2024-04-12T21:06:08.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:08.836+0000 7f61f5000700 1 -- 172.21.15.81:0/1663754903 wait complete. 2024-04-12T21:06:08.834 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2024-04-12T21:06:09.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:09.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:09.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:09.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:09.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:06:09.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:09.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:09.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-12T21:06:09.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: Updating smithi153:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:09.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1663754903' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-12T21:06:09.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:08 smithi081 ceph-mon[31453]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:09.603 INFO:tasks.cephadm:Adding mon.a on smithi081 2024-04-12T21:06:09.604 INFO:tasks.cephadm:Adding mon.b on smithi104 2024-04-12T21:06:09.604 INFO:tasks.cephadm:Adding mon.c on smithi153 2024-04-12T21:06:09.604 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch apply mon '3;smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi153:172.21.15.153=c' 2024-04-12T21:06:09.834 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:10.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:09 smithi081 ceph-mon[31453]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:06:10.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:09 smithi081 ceph-mon[31453]: Updating smithi153:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.client.admin.keyring 2024-04-12T21:06:10.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:09 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1663754903' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-12T21:06:10.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:09 smithi081 ceph-mon[31453]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:06:10.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:10.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:10.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:11.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:10 smithi081 ceph-mon[31453]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:11.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 -- 172.21.15.153:0/3634043000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 msgr2=0x7f6950101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:11.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 --2- 172.21.15.153:0/3634043000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 0x7f6950101a00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6940009910 tx=0x7f6940009c20 comp rx=0 tx=0).stop 2024-04-12T21:06:11.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 -- 172.21.15.153:0/3634043000 shutdown_connections 2024-04-12T21:06:11.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 --2- 172.21.15.153:0/3634043000 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 0x7f6950101a00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:11.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 -- 172.21.15.153:0/3634043000 >> 172.21.15.153:0/3634043000 conn(0x7f69500fced0 msgr2=0x7f69500ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:11.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 -- 172.21.15.153:0/3634043000 shutdown_connections 2024-04-12T21:06:11.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 -- 172.21.15.153:0/3634043000 wait complete. 2024-04-12T21:06:11.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 Processor -- start 2024-04-12T21:06:11.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.289+0000 7f69587ce700 1 -- start start 2024-04-12T21:06:11.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.290+0000 7f69587ce700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 0x7f69501030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:11.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.290+0000 7f69587ce700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f695010b9f0 con 0x7f6950101620 2024-04-12T21:06:11.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.290+0000 7f695656a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 0x7f69501030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:11.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.290+0000 7f695656a700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 0x7f69501030c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.153:60860/0 (socket says 172.21.15.153:60860) 2024-04-12T21:06:11.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.290+0000 7f695656a700 1 -- 172.21.15.153:0/1328839629 learned_addr learned my addr 172.21.15.153:0/1328839629 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:06:11.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.291+0000 7f695656a700 1 -- 172.21.15.153:0/1328839629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6950104fb0 con 0x7f6950101620 2024-04-12T21:06:11.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.291+0000 7f695656a700 1 --2- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 0x7f69501030c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6940005850 tx=0x7f6940005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:11.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.291+0000 7f69477fe700 1 -- 172.21.15.153:0/1328839629 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6940026070 con 0x7f6950101620 2024-04-12T21:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.291+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6950103660 con 0x7f6950101620 2024-04-12T21:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.292+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6950103b20 con 0x7f6950101620 2024-04-12T21:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.292+0000 7f69477fe700 1 -- 172.21.15.153:0/1328839629 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f694000bbd0 con 0x7f6950101620 2024-04-12T21:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.292+0000 7f69477fe700 1 -- 172.21.15.153:0/1328839629 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6940003680 con 0x7f6950101620 2024-04-12T21:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.292+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6934005330 con 0x7f6950101620 2024-04-12T21:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.293+0000 7f69477fe700 1 -- 172.21.15.153:0/1328839629 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6940029070 con 0x7f6950101620 2024-04-12T21:06:11.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.293+0000 7f69477fe700 1 --2- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f693c039650 0x7f693c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.293+0000 7f69477fe700 1 -- 172.21.15.153:0/1328839629 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f694001e070 con 0x7f6950101620 2024-04-12T21:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.294+0000 7f6955d69700 1 --2- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f693c039650 0x7f693c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:11.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.294+0000 7f6955d69700 1 --2- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f693c039650 0x7f693c03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f694c006fd0 tx=0x7f694c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:11.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.307+0000 7f69477fe700 1 -- 172.21.15.153:0/1328839629 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6940013070 con 0x7f6950101620 2024-04-12T21:06:11.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.514+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}) -- 0x7f6934000c80 con 0x7f693c039650 2024-04-12T21:06:11.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.531+0000 7f69477fe700 1 -- 172.21.15.153:0/1328839629 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6934000c80 con 0x7f693c039650 2024-04-12T21:06:11.528 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-04-12T21:06:11.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f693c039650 msgr2=0x7f693c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 --2- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f693c039650 0x7f693c03bb10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f694c006fd0 tx=0x7f694c006e40 comp rx=0 tx=0).stop 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 msgr2=0x7f69501030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 --2- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 0x7f69501030c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6940005850 tx=0x7f6940005030 comp rx=0 tx=0).stop 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 shutdown_connections 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 --2- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f693c039650 0x7f693c03bb10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 --2- 172.21.15.153:0/1328839629 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6950101620 0x7f69501030c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 >> 172.21.15.153:0/1328839629 conn(0x7f69500fced0 msgr2=0x7f695006a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 shutdown_connections 2024-04-12T21:06:11.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:11.533+0000 7f69587ce700 1 -- 172.21.15.153:0/1328839629 wait complete. 2024-04-12T21:06:12.138 DEBUG:teuthology.orchestra.run.smithi104:mon.b> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.b.service 2024-04-12T21:06:12.141 DEBUG:teuthology.orchestra.run.smithi153:mon.c> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mon.c.service 2024-04-12T21:06:12.143 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-12T21:06:12.143 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-12T21:06:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi153:172.21.15.153=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:06:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: Saving service mon spec with placement smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi153:172.21.15.153=c;count:3 2024-04-12T21:06:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: Deploying daemon mon.c on smithi153 2024-04-12T21:06:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:12 smithi081 ceph-mon[31453]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:13.060 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:15.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.613+0000 7ffbb0cde700 1 -- 172.21.15.153:0/2969443249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 msgr2=0x7ffbac101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:15.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.613+0000 7ffbb0cde700 1 --2- 172.21.15.153:0/2969443249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 0x7ffbac101a00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c009910 tx=0x7ffb9c009c20 comp rx=0 tx=0).stop 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.613+0000 7ffbb0cde700 1 -- 172.21.15.153:0/2969443249 shutdown_connections 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.613+0000 7ffbb0cde700 1 --2- 172.21.15.153:0/2969443249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 0x7ffbac101a00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.613+0000 7ffbb0cde700 1 -- 172.21.15.153:0/2969443249 >> 172.21.15.153:0/2969443249 conn(0x7ffbac0fced0 msgr2=0x7ffbac0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.613+0000 7ffbb0cde700 1 -- 172.21.15.153:0/2969443249 shutdown_connections 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.613+0000 7ffbb0cde700 1 -- 172.21.15.153:0/2969443249 wait complete. 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.614+0000 7ffbb0cde700 1 Processor -- start 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.614+0000 7ffbb0cde700 1 -- start start 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.614+0000 7ffbb0cde700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 0x7ffbac198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:15.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.614+0000 7ffbb0cde700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ffbac10b9f0 con 0x7ffbac101620 2024-04-12T21:06:15.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.615+0000 7ffbaa59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 0x7ffbac198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:15.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.615+0000 7ffbaa59c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 0x7ffbac198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.153:41410/0 (socket says 172.21.15.153:41410) 2024-04-12T21:06:15.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.615+0000 7ffbaa59c700 1 -- 172.21.15.153:0/1885683902 learned_addr learned my addr 172.21.15.153:0/1885683902 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.615+0000 7ffbaa59c700 1 -- 172.21.15.153:0/1885683902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbac198580 con 0x7ffbac101620 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.616+0000 7ffbaa59c700 1 --2- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 0x7ffbac198040 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c005850 tx=0x7ffb9c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.616+0000 7ffb9b7fe700 1 -- 172.21.15.153:0/1885683902 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb9c026070 con 0x7ffbac101620 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.616+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbac198870 con 0x7ffbac101620 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.616+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbac19c910 con 0x7ffbac101620 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.616+0000 7ffb9b7fe700 1 -- 172.21.15.153:0/1885683902 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb9c00b6f0 con 0x7ffbac101620 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.616+0000 7ffb9b7fe700 1 -- 172.21.15.153:0/1885683902 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ffb9c030060 con 0x7ffbac101620 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.617+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb8c005330 con 0x7ffbac101620 2024-04-12T21:06:15.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.617+0000 7ffb9b7fe700 1 -- 172.21.15.153:0/1885683902 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7ffb9c029070 con 0x7ffbac101620 2024-04-12T21:06:15.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.618+0000 7ffb9b7fe700 1 --2- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ffb94039600 0x7ffb9403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:15.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.618+0000 7ffb9b7fe700 1 -- 172.21.15.153:0/1885683902 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ffb9c01e070 con 0x7ffbac101620 2024-04-12T21:06:15.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.618+0000 7ffba9d9b700 1 --2- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ffb94039600 0x7ffb9403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:15.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.619+0000 7ffba9d9b700 1 --2- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ffb94039600 0x7ffb9403bac0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffba0006fd0 tx=0x7ffba0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:15.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.633+0000 7ffb9b7fe700 1 -- 172.21.15.153:0/1885683902 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb9c013070 con 0x7ffbac101620 2024-04-12T21:06:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:15 smithi081 ceph-mon[31453]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:15.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.918+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ffb8c0058f0 con 0x7ffbac101620 2024-04-12T21:06:15.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.919+0000 7ffb9b7fe700 1 -- 172.21.15.153:0/1885683902 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7ffb9c00bb80 con 0x7ffbac101620 2024-04-12T21:06:15.920 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:06:15.921 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","modified":"2024-04-12T21:04:11.886065Z","created":"2024-04-12T21:04:11.886065Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-12T21:06:15.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ffb94039600 msgr2=0x7ffb9403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:15.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 --2- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ffb94039600 0x7ffb9403bac0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ffba0006fd0 tx=0x7ffba0006e40 comp rx=0 tx=0).stop 2024-04-12T21:06:15.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 msgr2=0x7ffbac198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:15.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 --2- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 0x7ffbac198040 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c005850 tx=0x7ffb9c005030 comp rx=0 tx=0).stop 2024-04-12T21:06:15.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 shutdown_connections 2024-04-12T21:06:15.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 --2- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ffb94039600 0x7ffb9403bac0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:15.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 --2- 172.21.15.153:0/1885683902 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ffbac101620 0x7ffbac198040 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:15.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 >> 172.21.15.153:0/1885683902 conn(0x7ffbac0fced0 msgr2=0x7ffbac06a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:15.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 shutdown_connections 2024-04-12T21:06:15.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:15.921+0000 7ffbb0cde700 1 -- 172.21.15.153:0/1885683902 wait complete. 2024-04-12T21:06:15.923 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-04-12T21:06:17.607 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-12T21:06:17.607 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-12T21:06:17.845 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: mon.a calling monitor election 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: mon.c calling monitor election 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: monmap epoch 2 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: last_changed 2024-04-12T21:06:15.983325+0000 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: min_mon_release 19 (squid) 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: election_strategy: 1 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: fsmap 2024-04-12T21:06:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:06:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: mgrmap e13: x(active, since 58s) 2024-04-12T21:06:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: overall HEALTH_OK 2024-04-12T21:06:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:21.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:22.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.185+0000 7f5622b53700 1 -- 172.21.15.153:0/4258634672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c108660 msgr2=0x7f560c005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:22.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.185+0000 7f5622b53700 1 --2- 172.21.15.153:0/4258634672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c108660 0x7f560c005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f561000fc90 tx=0x7f561000ffa0 comp rx=0 tx=0).stop 2024-04-12T21:06:22.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.186+0000 7f5622b53700 1 -- 172.21.15.153:0/4258634672 shutdown_connections 2024-04-12T21:06:22.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.186+0000 7f5622b53700 1 --2- 172.21.15.153:0/4258634672 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c108660 0x7f560c005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:22.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.186+0000 7f5622b53700 1 -- 172.21.15.153:0/4258634672 >> 172.21.15.153:0/4258634672 conn(0x7f561c0fe0c0 msgr2=0x7f561c1004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:22.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.186+0000 7f5622b53700 1 -- 172.21.15.153:0/4258634672 shutdown_connections 2024-04-12T21:06:22.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.186+0000 7f5622b53700 1 -- 172.21.15.153:0/4258634672 wait complete. 2024-04-12T21:06:22.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f5622b53700 1 Processor -- start 2024-04-12T21:06:22.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f5622b53700 1 -- start start 2024-04-12T21:06:22.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f5622b53700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 0x7f561c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:22.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f5622b53700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c198610 0x7f561c19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:22.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f5622b53700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f561c10a690 con 0x7f561c198610 2024-04-12T21:06:22.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f5622b53700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f561c10a510 con 0x7f561c108660 2024-04-12T21:06:22.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f56208ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 0x7f561c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:22.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f56208ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 0x7f561c1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58022/0 (socket says 172.21.15.153:58022) 2024-04-12T21:06:22.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.187+0000 7f56208ef700 1 -- 172.21.15.153:0/2615743433 learned_addr learned my addr 172.21.15.153:0/2615743433 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:06:22.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.188+0000 7f56208ef700 1 -- 172.21.15.153:0/2615743433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 msgr2=0x7f561c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-12T21:06:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.188+0000 7f56208ef700 1 -- 172.21.15.153:0/2615743433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 msgr2=0x7f561c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T21:06:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.188+0000 7f56208ef700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 0x7f561c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:06:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.188+0000 7f56208ef700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 0x7f561c1980d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T21:06:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.188+0000 7f561bfff700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c198610 0x7f561c19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.188+0000 7f561bfff700 1 -- 172.21.15.153:0/2615743433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 msgr2=0x7f561c1980d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T21:06:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.188+0000 7f561bfff700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 0x7f561c1980d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.188+0000 7f561bfff700 1 -- 172.21.15.153:0/2615743433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f561c198c50 con 0x7f561c198610 2024-04-12T21:06:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.189+0000 7f561bfff700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c198610 0x7f561c19c950 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f560800b4d0 tx=0x7f560800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.189+0000 7f5619ffb700 1 -- 172.21.15.153:0/2615743433 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5608011630 con 0x7f561c198610 2024-04-12T21:06:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.189+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f561c19ce90 con 0x7f561c198610 2024-04-12T21:06:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.189+0000 7f5619ffb700 1 -- 172.21.15.153:0/2615743433 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5608011cb0 con 0x7f561c198610 2024-04-12T21:06:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.189+0000 7f5619ffb700 1 -- 172.21.15.153:0/2615743433 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5608010e60 con 0x7f561c198610 2024-04-12T21:06:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.189+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561c19d470 con 0x7f561c198610 2024-04-12T21:06:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.191+0000 7f5619ffb700 1 -- 172.21.15.153:0/2615743433 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5608018020 con 0x7f561c198610 2024-04-12T21:06:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.191+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561c19d780 con 0x7f561c198610 2024-04-12T21:06:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.191+0000 7f5619ffb700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5600039390 0x7f560003b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.191+0000 7f5619ffb700 1 -- 172.21.15.153:0/2615743433 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f560804c250 con 0x7f561c198610 2024-04-12T21:06:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.192+0000 7f56208ef700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5600039390 0x7f560003b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.193+0000 7f56208ef700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5600039390 0x7f560003b850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f561001b4c0 tx=0x7f5610005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:22.205 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.209+0000 7f5619ffb700 1 -- 172.21.15.153:0/2615743433 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f561c19d780 con 0x7f561c198610 2024-04-12T21:06:22.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:22.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:22 smithi081 ceph-mon[31453]: Deploying daemon mon.b on smithi104 2024-04-12T21:06:22.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:22.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.494+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f561c1993e0 con 0x7f561c198610 2024-04-12T21:06:22.491 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:22.491 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:22 smithi153 ceph-mon[36569]: Deploying daemon mon.b on smithi104 2024-04-12T21:06:22.491 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:22.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.496+0000 7f5619ffb700 1 -- 172.21.15.153:0/2615743433 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f560801f020 con 0x7f561c198610 2024-04-12T21:06:22.492 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:06:22.492 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","modified":"2024-04-12T21:06:15.983325Z","created":"2024-04-12T21:04:11.886065Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-12T21:06:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5600039390 msgr2=0x7f560003b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5600039390 0x7f560003b850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f561001b4c0 tx=0x7f5610005da0 comp rx=0 tx=0).stop 2024-04-12T21:06:22.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c198610 msgr2=0x7f561c19c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c198610 0x7f561c19c950 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f560800b4d0 tx=0x7f560800b890 comp rx=0 tx=0).stop 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 shutdown_connections 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5600039390 0x7f560003b850 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f561c108660 0x7f561c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 --2- 172.21.15.153:0/2615743433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f561c198610 0x7f561c19c950 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 >> 172.21.15.153:0/2615743433 conn(0x7f561c0fe0c0 msgr2=0x7f561c0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 shutdown_connections 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:22.498+0000 7f5622b53700 1 -- 172.21.15.153:0/2615743433 wait complete. 2024-04-12T21:06:22.495 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-04-12T21:06:23.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:23 smithi153 ceph-mon[36569]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:23.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:23 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/2615743433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:06:23.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:23 smithi081 ceph-mon[31453]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:23.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:23 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/2615743433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:06:24.003 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-12T21:06:24.003 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph mon dump -f json 2024-04-12T21:06:24.248 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:06:25.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:25 smithi104 systemd[1]: Started Ceph mon.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.134+0000 7efc0be38700 1 -- 172.21.15.153:0/3891029756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efbf00035c0 msgr2=0x7efbf0005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.134+0000 7efc0be38700 1 --2- 172.21.15.153:0/3891029756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efbf00035c0 0x7efbf0005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7efc0406a4b0 tx=0x7efbfc00fe60 comp rx=0 tx=0).stop 2024-04-12T21:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.134+0000 7efc0be38700 1 -- 172.21.15.153:0/3891029756 shutdown_connections 2024-04-12T21:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.134+0000 7efc0be38700 1 --2- 172.21.15.153:0/3891029756 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efbf00035c0 0x7efbf0005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.134+0000 7efc0be38700 1 -- 172.21.15.153:0/3891029756 >> 172.21.15.153:0/3891029756 conn(0x7efc040fcf90 msgr2=0x7efc040ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:30.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.134+0000 7efc0be38700 1 -- 172.21.15.153:0/3891029756 shutdown_connections 2024-04-12T21:06:30.132 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.134+0000 7efc0be38700 1 -- 172.21.15.153:0/3891029756 wait complete. 2024-04-12T21:06:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.135+0000 7efc0be38700 1 Processor -- start 2024-04-12T21:06:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.135+0000 7efc0be38700 1 -- start start 2024-04-12T21:06:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.135+0000 7efc0be38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 0x7efc04101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.135+0000 7efc0be38700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc04101960 0x7efc04101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.135+0000 7efc0be38700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 0x7efc041a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.135+0000 7efc0be38700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efc0410a5e0 con 0x7efc04101960 2024-04-12T21:06:30.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.135+0000 7efc0be38700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efc0410a460 con 0x7efc041a44c0 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.135+0000 7efc0be38700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc0410a760 con 0x7efbf00035c0 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc09bd4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 0x7efc04101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc09bd4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 0x7efc04101420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38434/0 (socket says 172.21.15.153:38434) 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc09bd4700 1 -- 172.21.15.153:0/1139302307 learned_addr learned my addr 172.21.15.153:0/1139302307 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc09bd4700 1 -- 172.21.15.153:0/1139302307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 msgr2=0x7efc04101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc093d3700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc04101960 0x7efc04101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc09bd4700 1 -- 172.21.15.153:0/1139302307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 msgr2=0x7efc04101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc09bd4700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 0x7efc04101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc09bd4700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 0x7efc04101420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T21:06:30.135 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc0a3d5700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 0x7efc041a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc0a3d5700 1 -- 172.21.15.153:0/1139302307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 msgr2=0x7efc041a68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc0a3d5700 1 -- 172.21.15.153:0/1139302307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 msgr2=0x7efc041a68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc0a3d5700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 0x7efc041a68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc0a3d5700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 0x7efc041a68b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc093d3700 1 -- 172.21.15.153:0/1139302307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 msgr2=0x7efc04101420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc093d3700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 0x7efc04101420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc093d3700 1 -- 172.21.15.153:0/1139302307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 msgr2=0x7efc041a68b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc093d3700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 0x7efc041a68b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.136+0000 7efc093d3700 1 -- 172.21.15.153:0/1139302307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc041a6fe0 con 0x7efc04101960 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.137+0000 7efc093d3700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc04101960 0x7efc04101de0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efbf400b550 tx=0x7efbf400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:30.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.137+0000 7efbfaffd700 1 -- 172.21.15.153:0/1139302307 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4004230 con 0x7efc04101960 2024-04-12T21:06:30.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.137+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc041a72d0 con 0x7efc04101960 2024-04-12T21:06:30.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.137+0000 7efbfaffd700 1 -- 172.21.15.153:0/1139302307 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efbf40043d0 con 0x7efc04101960 2024-04-12T21:06:30.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.137+0000 7efbfaffd700 1 -- 172.21.15.153:0/1139302307 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf4011c20 con 0x7efc04101960 2024-04-12T21:06:30.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.137+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc041a7810 con 0x7efc04101960 2024-04-12T21:06:30.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.138+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc04064110 con 0x7efc04101960 2024-04-12T21:06:30.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.138+0000 7efbfaffd700 1 -- 172.21.15.153:0/1139302307 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7efbf40036a0 con 0x7efc04101960 2024-04-12T21:06:30.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.139+0000 7efbfaffd700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efbe8039ae0 0x7efbe803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:30.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.139+0000 7efbfaffd700 1 -- 172.21.15.153:0/1139302307 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efbf404c550 con 0x7efc04101960 2024-04-12T21:06:30.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.139+0000 7efc09bd4700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efbe8039ae0 0x7efbe803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:30.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.140+0000 7efc09bd4700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efbe8039ae0 0x7efbe803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efbfc005b40 tx=0x7efbfc005ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:30.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.146+0000 7efbfaffd700 1 -- 172.21.15.153:0/1139302307 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efbf4020720 con 0x7efc04101960 2024-04-12T21:06:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: mon.c calling monitor election 2024-04-12T21:06:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:06:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: mon.a calling monitor election 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-12T21:06:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: monmap epoch 3 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: last_changed 2024-04-12T21:06:25.021695+0000 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: min_mon_release 19 (squid) 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: election_strategy: 1 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: fsmap 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: mgrmap e13: x(active, since 67s) 2024-04-12T21:06:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: overall HEALTH_OK 2024-04-12T21:06:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:30.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.433+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7efc0402e200 con 0x7efc04101960 2024-04-12T21:06:30.430 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: mon.c calling monitor election 2024-04-12T21:06:30.430 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:06:30.430 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: mon.a calling monitor election 2024-04-12T21:06:30.430 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: monmap epoch 3 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: last_changed 2024-04-12T21:06:25.021695+0000 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: min_mon_release 19 (squid) 2024-04-12T21:06:30.431 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: election_strategy: 1 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: fsmap 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: mgrmap e13: x(active, since 67s) 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: overall HEALTH_OK 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:30.432 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:30.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.434+0000 7efbfaffd700 1 -- 172.21.15.153:0/1139302307 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7efbf4021df0 con 0x7efc04101960 2024-04-12T21:06:30.433 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-04-12T21:06:30.433 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":3,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","modified":"2024-04-12T21:06:25.021695Z","created":"2024-04-12T21:04:11.886065Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:3300","nonce":0},{"type":"v1","addr":"172.21.15.81:6789","nonce":0}]},"addr":"172.21.15.81:6789/0","public_addr":"172.21.15.81:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-12T21:06:30.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efbe8039ae0 msgr2=0x7efbe803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:30.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efbe8039ae0 0x7efbe803bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efbfc005b40 tx=0x7efbfc005ad0 comp rx=0 tx=0).stop 2024-04-12T21:06:30.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc04101960 msgr2=0x7efc04101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:30.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc04101960 0x7efc04101de0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efbf400b550 tx=0x7efbf400b910 comp rx=0 tx=0).stop 2024-04-12T21:06:30.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 shutdown_connections 2024-04-12T21:06:30.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efbe8039ae0 0x7efbe803bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:30.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efc041a44c0 0x7efc041a68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:30.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efbf00035c0 0x7efc04101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:30.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 --2- 172.21.15.153:0/1139302307 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efc04101960 0x7efc04101de0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:30.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.437+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 >> 172.21.15.153:0/1139302307 conn(0x7efc040fcf90 msgr2=0x7efc041074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:30.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.438+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 shutdown_connections 2024-04-12T21:06:30.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:30.438+0000 7efc0be38700 1 -- 172.21.15.153:0/1139302307 wait complete. 2024-04-12T21:06:30.435 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 3 2024-04-12T21:06:31.054 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-12T21:06:31.054 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph config generate-minimal-conf 2024-04-12T21:06:31.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:31 smithi153 ceph-mon[36569]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:31.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:31.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:31 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/1139302307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:06:31.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:31.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:31 smithi081 ceph-mon[31453]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:31.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:31.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:31 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/1139302307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:06:31.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:31.464 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:06:32.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mon.c calling monitor election 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mon.a calling monitor election 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:32.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: monmap epoch 3 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: last_changed 2024-04-12T21:06:25.021695+0000 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: min_mon_release 19 (squid) 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: election_strategy: 1 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: fsmap 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mgrmap e13: x(active, since 67s) 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: overall HEALTH_OK 2024-04-12T21:06:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/1139302307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-12T21:06:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:32.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: mon.b calling monitor election 2024-04-12T21:06:32.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-12T21:06:32.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-12T21:06:32.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-12T21:06:32.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: mon.b calling monitor election 2024-04-12T21:06:32.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: mon.c calling monitor election 2024-04-12T21:06:32.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: mon.a calling monitor election 2024-04-12T21:06:32.978 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: monmap epoch 3 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: last_changed 2024-04-12T21:06:25.021695+0000 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: min_mon_release 19 (squid) 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: election_strategy: 1 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: fsmap 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: mgrmap e13: x(active, since 69s) 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: overall HEALTH_OK 2024-04-12T21:06:32.979 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: Updating smithi153:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:32.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: Updating smithi104:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:32.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:32.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:32.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:32.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:32.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:32.980 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mon.b calling monitor election 2024-04-12T21:06:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mon.b calling monitor election 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mon.c calling monitor election 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mon.a calling monitor election 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: monmap epoch 3 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: last_changed 2024-04-12T21:06:25.021695+0000 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: min_mon_release 19 (squid) 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: election_strategy: 1 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-12T21:06:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: fsmap 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: mgrmap e13: x(active, since 69s) 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: overall HEALTH_OK 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: Updating smithi153:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: Updating smithi104:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: mon.b calling monitor election 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: Updating smithi081:/etc/ceph/ceph.conf 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: Updating smithi104:/etc/ceph/ceph.conf 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: Updating smithi153:/etc/ceph/ceph.conf 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: mon.b calling monitor election 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: mon.c calling monitor election 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: mon.a calling monitor election 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-12T21:06:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: monmap epoch 3 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: last_changed 2024-04-12T21:06:25.021695+0000 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: created 2024-04-12T21:04:11.886065+0000 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: min_mon_release 19 (squid) 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: election_strategy: 1 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: 0: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.a 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: 1: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.c 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: 2: [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon.b 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: fsmap 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: osdmap e4: 0 total, 0 up, 0 in 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: mgrmap e13: x(active, since 69s) 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: overall HEALTH_OK 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: Updating smithi153:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: Updating smithi104:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:33.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:33.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:33.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.460+0000 7f6a18e00700 1 -- 172.21.15.81:0/2072201425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a14074ee0 msgr2=0x7f6a140752c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:33.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.460+0000 7f6a18e00700 1 --2- 172.21.15.81:0/2072201425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a14074ee0 0x7f6a140752c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009910 tx=0x7f6a04009c20 comp rx=0 tx=0).stop 2024-04-12T21:06:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.460+0000 7f6a18e00700 1 -- 172.21.15.81:0/2072201425 shutdown_connections 2024-04-12T21:06:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.460+0000 7f6a18e00700 1 --2- 172.21.15.81:0/2072201425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a14074ee0 0x7f6a140752c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.460+0000 7f6a18e00700 1 -- 172.21.15.81:0/2072201425 >> 172.21.15.81:0/2072201425 conn(0x7f6a140fdff0 msgr2=0x7f6a14100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.460+0000 7f6a18e00700 1 -- 172.21.15.81:0/2072201425 shutdown_connections 2024-04-12T21:06:33.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.460+0000 7f6a18e00700 1 -- 172.21.15.81:0/2072201425 wait complete. 2024-04-12T21:06:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.461+0000 7f6a18e00700 1 Processor -- start 2024-04-12T21:06:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.461+0000 7f6a18e00700 1 -- start start 2024-04-12T21:06:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.461+0000 7f6a18e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a14074ee0 0x7f6a1419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.461+0000 7f6a18e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a1419a9f0 0x7f6a1419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.461+0000 7f6a18e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1419f2c0 0x7f6a1419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:33.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.461+0000 7f6a18e00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1410ed20 con 0x7f6a1419f2c0 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.461+0000 7f6a18e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1410eba0 con 0x7f6a1419a9f0 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.461+0000 7f6a18e00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6a1410eea0 con 0x7f6a14074ee0 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a14074ee0 0x7f6a1419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a14074ee0 0x7f6a1419a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:37248/0 (socket says 172.21.15.81:37248) 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11e00700 1 -- 172.21.15.81:0/3928691874 learned_addr learned my addr 172.21.15.81:0/3928691874 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11400700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a1419a9f0 0x7f6a1419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a12800700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1419f2c0 0x7f6a1419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11e00700 1 -- 172.21.15.81:0/3928691874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a1419a9f0 msgr2=0x7f6a1419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a1419a9f0 0x7f6a1419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11e00700 1 -- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1419f2c0 msgr2=0x7f6a1419f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1419f2c0 0x7f6a1419f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11e00700 1 -- 172.21.15.81:0/3928691874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a141a7030 con 0x7f6a14074ee0 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a11400700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a1419a9f0 0x7f6a1419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:06:33.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.462+0000 7f6a12800700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1419f2c0 0x7f6a1419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T21:06:33.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.463+0000 7f6a11e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a14074ee0 0x7f6a1419a4b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009fd0 tx=0x7f6a0400bde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:33.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.463+0000 7f6a0aa00700 1 -- 172.21.15.81:0/3928691874 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a04027070 con 0x7f6a14074ee0 2024-04-12T21:06:33.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.463+0000 7f6a0aa00700 1 -- 172.21.15.81:0/3928691874 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a040050d0 con 0x7f6a14074ee0 2024-04-12T21:06:33.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.463+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a141a7200 con 0x7f6a14074ee0 2024-04-12T21:06:33.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.463+0000 7f6a0aa00700 1 -- 172.21.15.81:0/3928691874 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a04022de0 con 0x7f6a14074ee0 2024-04-12T21:06:33.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.463+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a141a7730 con 0x7f6a14074ee0 2024-04-12T21:06:33.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.465+0000 7f6a0aa00700 1 -- 172.21.15.81:0/3928691874 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6a0402a070 con 0x7f6a14074ee0 2024-04-12T21:06:33.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.465+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a14064110 con 0x7f6a14074ee0 2024-04-12T21:06:33.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.465+0000 7f6a0aa00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69f8039ae0 0x7f69f803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:33.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.465+0000 7f6a0aa00700 1 -- 172.21.15.81:0/3928691874 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6a0404d8f0 con 0x7f6a14074ee0 2024-04-12T21:06:33.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.466+0000 7f6a11400700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69f8039ae0 0x7f69f803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:33.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.466+0000 7f6a11400700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69f8039ae0 0x7f69f803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009ec0 tx=0x7f6a00009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:33.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.471+0000 7f6a0aa00700 1 -- 172.21.15.81:0/3928691874 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a04013070 con 0x7f6a14074ee0 2024-04-12T21:06:33.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.698+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f6a141a79f0 con 0x7f6a14074ee0 2024-04-12T21:06:33.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.699+0000 7f6a0aa00700 1 -- 172.21.15.81:0/3928691874 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f6a04031020 con 0x7f6a14074ee0 2024-04-12T21:06:33.695 INFO:teuthology.orchestra.run.smithi081.stdout:# minimal ceph.conf for 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:33.695 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-04-12T21:06:33.696 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:06:33.696 INFO:teuthology.orchestra.run.smithi081.stdout: mon_host = [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-04-12T21:06:33.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.703+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69f8039ae0 msgr2=0x7f69f803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:33.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.703+0000 7f6a18e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69f8039ae0 0x7f69f803bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a00009ec0 tx=0x7f6a00009210 comp rx=0 tx=0).stop 2024-04-12T21:06:33.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a14074ee0 msgr2=0x7f6a1419a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:33.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a14074ee0 0x7f6a1419a4b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6a04009fd0 tx=0x7f6a0400bde0 comp rx=0 tx=0).stop 2024-04-12T21:06:33.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 shutdown_connections 2024-04-12T21:06:33.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69f8039ae0 0x7f69f803bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:33.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6a1419f2c0 0x7f6a1419f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:33.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6a14074ee0 0x7f6a1419a4b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:33.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 --2- 172.21.15.81:0/3928691874 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6a1419a9f0 0x7f6a1419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:33.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 >> 172.21.15.81:0/3928691874 conn(0x7f6a140fdff0 msgr2=0x7f6a14108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:33.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 shutdown_connections 2024-04-12T21:06:33.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:33.704+0000 7f6a18e00700 1 -- 172.21.15.81:0/3928691874 wait complete. 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: Updating smithi081:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: Reconfiguring mon.a (unknown last config time)... 2024-04-12T21:06:34.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: Reconfiguring daemon mon.a on smithi081 2024-04-12T21:06:34.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:33 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3928691874' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:34.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: Updating smithi081:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:34.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:34.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: Reconfiguring mon.a (unknown last config time)... 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: Reconfiguring daemon mon.a on smithi081 2024-04-12T21:06:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:33 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3928691874' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: Updating smithi081:/var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/config/ceph.conf 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: Reconfiguring mon.a (unknown last config time)... 2024-04-12T21:06:34.206 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:34.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:34.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:34.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: Reconfiguring daemon mon.a on smithi081 2024-04-12T21:06:34.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:33 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3928691874' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:34.438 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-12T21:06:34.439 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:06:34.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T21:06:34.530 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:06:34.530 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:06:34.670 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:06:34.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T21:06:34.705 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:06:34.705 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:06:34.776 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:06:34.776 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-04-12T21:06:34.811 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:06:34.811 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-12T21:06:34.881 INFO:tasks.cephadm:Adding mgr.x on smithi081 2024-04-12T21:06:34.881 INFO:tasks.cephadm:Adding mgr.y on smithi104 2024-04-12T21:06:34.881 INFO:tasks.cephadm:Adding mgr.z on smithi153 2024-04-12T21:06:34.882 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch apply mgr '3;smithi081=x;smithi104=y;smithi153=z' 2024-04-12T21:06:34.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:34 smithi081 ceph-mon[31453]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:34 smithi153 ceph-mon[36569]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:35.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:34 smithi104 ceph-mon[36624]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:35.156 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:06:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.482+0000 7febc517d700 1 -- 172.21.15.153:0/3982848861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0102640 msgr2=0x7febc0102a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.482+0000 7febc517d700 1 --2- 172.21.15.153:0/3982848861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0102640 0x7febc0102a60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7febb800b0a0 tx=0x7febb800b3b0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.482+0000 7febc517d700 1 -- 172.21.15.153:0/3982848861 shutdown_connections 2024-04-12T21:06:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.482+0000 7febc517d700 1 --2- 172.21.15.153:0/3982848861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0102640 0x7febc0102a60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.482+0000 7febc517d700 1 -- 172.21.15.153:0/3982848861 >> 172.21.15.153:0/3982848861 conn(0x7febc00fdc20 msgr2=0x7febc0100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.482+0000 7febc517d700 1 -- 172.21.15.153:0/3982848861 shutdown_connections 2024-04-12T21:06:36.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.482+0000 7febc517d700 1 -- 172.21.15.153:0/3982848861 wait complete. 2024-04-12T21:06:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.483+0000 7febc517d700 1 Processor -- start 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.483+0000 7febc517d700 1 -- start start 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.483+0000 7febc517d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febc0102640 0x7febc0076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.483+0000 7febc517d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0074ea0 0x7febc0075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.483+0000 7febc517d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febc0075860 0x7febc01a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.483+0000 7febc517d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7febc0108680 con 0x7febc0074ea0 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.483+0000 7febc517d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7febc0108500 con 0x7febc0102640 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febc517d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7febc0108800 con 0x7febc0075860 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febc0075860 0x7febc01a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febc0075860 0x7febc01a1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38504/0 (socket says 172.21.15.153:38504) 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 -- 172.21.15.153:0/3298419512 learned_addr learned my addr 172.21.15.153:0/3298419512 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbed9d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febc0102640 0x7febc0076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbe59c700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0074ea0 0x7febc0075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:36.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 -- 172.21.15.153:0/3298419512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febc0102640 msgr2=0x7febc0076850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febc0102640 0x7febc0076850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 -- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0074ea0 msgr2=0x7febc0075320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0074ea0 0x7febc0075320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 -- 172.21.15.153:0/3298419512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febc01a2490 con 0x7febc0075860 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbe59c700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0074ea0 0x7febc0075320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febbf59e700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febc0075860 0x7febc01a1ed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7febb000b550 tx=0x7febb000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7feba7fff700 1 -- 172.21.15.153:0/3298419512 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb0015020 con 0x7febc0075860 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.484+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febc01a2780 con 0x7febc0075860 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.485+0000 7feba7fff700 1 -- 172.21.15.153:0/3298419512 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febb0011d10 con 0x7febc0075860 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.485+0000 7feba7fff700 1 -- 172.21.15.153:0/3298419512 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febb000f2e0 con 0x7febc0075860 2024-04-12T21:06:36.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.485+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febc01a2d40 con 0x7febc0075860 2024-04-12T21:06:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.486+0000 7feba7fff700 1 -- 172.21.15.153:0/3298419512 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7febb0018020 con 0x7febc0075860 2024-04-12T21:06:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.486+0000 7feba7fff700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feba8039ae0 0x7feba803bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.486+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febc0064110 con 0x7febc0075860 2024-04-12T21:06:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.486+0000 7feba7fff700 1 -- 172.21.15.153:0/3298419512 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7febb004c2b0 con 0x7febc0075860 2024-04-12T21:06:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.486+0000 7febbed9d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feba8039ae0 0x7feba803bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:36.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.487+0000 7febbed9d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feba8039ae0 0x7feba803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7febb8015040 tx=0x7febb800b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:36.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.491+0000 7feba7fff700 1 -- 172.21.15.153:0/3298419512 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7febb001f020 con 0x7febc0075860 2024-04-12T21:06:36.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.703+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi081=x;smithi104=y;smithi153=z", "target": ["mon-mgr", ""]}) -- 0x7febc0062ef0 con 0x7feba8039ae0 2024-04-12T21:06:36.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.717+0000 7feba7fff700 1 -- 172.21.15.153:0/3298419512 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7febc0062ef0 con 0x7feba8039ae0 2024-04-12T21:06:36.713 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mgr update... 2024-04-12T21:06:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feba8039ae0 msgr2=0x7feba803bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feba8039ae0 0x7feba803bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7febb8015040 tx=0x7febb800b7b0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febc0075860 msgr2=0x7febc01a1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febc0075860 0x7febc01a1ed0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7febb000b550 tx=0x7febb000b910 comp rx=0 tx=0).stop 2024-04-12T21:06:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 shutdown_connections 2024-04-12T21:06:36.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feba8039ae0 0x7feba803bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7febc0075860 0x7febc01a1ed0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7febc0102640 0x7febc0076850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 --2- 172.21.15.153:0/3298419512 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7febc0074ea0 0x7febc0075320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 >> 172.21.15.153:0/3298419512 conn(0x7febc00fdc20 msgr2=0x7febc00fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 shutdown_connections 2024-04-12T21:06:36.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:06:36.718+0000 7febc517d700 1 -- 172.21.15.153:0/3298419512 wait complete. 2024-04-12T21:06:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:36 smithi104 ceph-mon[36624]: Reconfiguring mon.b (monmap changed)... 2024-04-12T21:06:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:36 smithi104 ceph-mon[36624]: Reconfiguring daemon mon.b on smithi104 2024-04-12T21:06:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:36 smithi104 ceph-mon[36624]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:36 smithi081 ceph-mon[31453]: Reconfiguring mon.b (monmap changed)... 2024-04-12T21:06:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:36.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:36.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:36 smithi081 ceph-mon[31453]: Reconfiguring daemon mon.b on smithi104 2024-04-12T21:06:36.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:36 smithi081 ceph-mon[31453]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:36.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:36.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:36.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:36 smithi153 ceph-mon[36569]: Reconfiguring mon.b (monmap changed)... 2024-04-12T21:06:36.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:36.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:36.995 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:36.995 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:36 smithi153 ceph-mon[36569]: Reconfiguring daemon mon.b on smithi104 2024-04-12T21:06:36.995 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:36 smithi153 ceph-mon[36569]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:37.275 DEBUG:teuthology.orchestra.run.smithi104:mgr.y> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mgr.y.service 2024-04-12T21:06:37.277 DEBUG:teuthology.orchestra.run.smithi153:mgr.z> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mgr.z.service 2024-04-12T21:06:37.279 INFO:tasks.cephadm:Deploying OSDs... 2024-04-12T21:06:37.279 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:06:37.279 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T21:06:37.337 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T21:06:37.337 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-04-12T21:06:37.390 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T21:06:37.390 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:37.390 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-12T21:06:37.391 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:37.391 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:37.391 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-12 21:05:38.091794088 +0000 2024-04-12T21:06:37.391 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-12 20:59:58.174684730 +0000 2024-04-12T21:06:37.391 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-12 20:59:58.174684730 +0000 2024-04-12T21:06:37.391 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-12 20:59:58.174684730 +0000 2024-04-12T21:06:37.391 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T21:06:37.473 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-12T21:06:37.473 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-12T21:06:37.473 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000362259 s, 1.4 MB/s 2024-04-12T21:06:37.476 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T21:06:37.528 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 6h/6d Inode: 652 Links: 1 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-12 21:05:38.210791853 +0000 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-12 20:59:58.777673446 +0000 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-12 20:59:58.777673446 +0000 2024-04-12T21:06:37.575 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-12 20:59:58.777673446 +0000 2024-04-12T21:06:37.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T21:06:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-12T21:06:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-12T21:06:37.657 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000331233 s, 1.5 MB/s 2024-04-12T21:06:37.661 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T21:06:37.719 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-04-12T21:06:37.767 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T21:06:37.767 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-12T21:06:37.767 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-12 21:05:38.305790069 +0000 2024-04-12T21:06:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-12 20:59:59.397661844 +0000 2024-04-12T21:06:37.768 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-12 20:59:59.397661844 +0000 2024-04-12T21:06:37.768 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-12 20:59:59.397661844 +0000 2024-04-12T21:06:37.768 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T21:06:37.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi081=x;smithi104=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:06:37.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: Saving service mgr spec with placement smithi081=x;smithi104=y;smithi153=z;count:3 2024-04-12T21:06:37.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:37.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:37.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:37.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: Reconfiguring mon.c (monmap changed)... 2024-04-12T21:06:37.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:37.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:37.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:37.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:37 smithi081 ceph-mon[31453]: Reconfiguring daemon mon.c on smithi153 2024-04-12T21:06:37.847 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-12T21:06:37.847 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-12T21:06:37.847 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000358446 s, 1.4 MB/s 2024-04-12T21:06:37.850 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T21:06:37.903 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-04-12T21:06:37.949 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T21:06:37.949 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:37.949 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 6h/6d Inode: 689 Links: 1 2024-04-12T21:06:37.950 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:37.950 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:37.950 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-12 21:05:38.385788566 +0000 2024-04-12T21:06:37.950 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-12 21:00:00.012650336 +0000 2024-04-12T21:06:37.950 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-12 21:00:00.012650336 +0000 2024-04-12T21:06:37.950 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-12 21:00:00.012650336 +0000 2024-04-12T21:06:37.950 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T21:06:37.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi081=x;smithi104=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:06:37.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: Saving service mgr spec with placement smithi081=x;smithi104=y;smithi153=z;count:3 2024-04-12T21:06:37.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:37.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:37.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:37.961 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: Reconfiguring mon.c (monmap changed)... 2024-04-12T21:06:37.962 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:37.962 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:37.962 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:37.962 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:37 smithi153 ceph-mon[36569]: Reconfiguring daemon mon.c on smithi153 2024-04-12T21:06:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi081=x;smithi104=y;smithi153=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:06:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: Saving service mgr spec with placement smithi081=x;smithi104=y;smithi153=z;count:3 2024-04-12T21:06:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: Reconfiguring mon.c (monmap changed)... 2024-04-12T21:06:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-12T21:06:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-12T21:06:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:37 smithi104 ceph-mon[36624]: Reconfiguring daemon mon.c on smithi153 2024-04-12T21:06:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-04-12T21:06:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-04-12T21:06:38.080 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000309265 s, 1.7 MB/s 2024-04-12T21:06:38.085 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T21:06:38.139 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-04-12T21:06:38.139 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T21:06:38.158 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T21:06:38.158 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-04-12T21:06:38.215 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T21:06:38.216 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:38.216 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-12T21:06:38.216 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:38.216 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:38.216 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-12 21:06:24.474709961 +0000 2024-04-12T21:06:38.217 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-12 20:58:57.878461015 +0000 2024-04-12T21:06:38.217 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-12 20:58:57.878461015 +0000 2024-04-12T21:06:38.217 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-12 20:58:57.878461015 +0000 2024-04-12T21:06:38.217 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T21:06:38.284 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-12T21:06:38.285 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-12T21:06:38.285 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000259442 s, 2.0 MB/s 2024-04-12T21:06:38.286 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T21:06:38.343 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-04-12T21:06:38.400 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T21:06:38.401 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:38.401 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-12T21:06:38.401 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:38.401 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:38.401 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-12 21:06:24.475709949 +0000 2024-04-12T21:06:38.401 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-12 20:58:58.214457100 +0000 2024-04-12T21:06:38.401 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-12 20:58:58.214457100 +0000 2024-04-12T21:06:38.401 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-12 20:58:58.214457100 +0000 2024-04-12T21:06:38.401 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T21:06:38.468 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-12T21:06:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-12T21:06:38.469 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000229315 s, 2.2 MB/s 2024-04-12T21:06:38.470 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T21:06:38.527 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-04-12T21:06:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T21:06:38.583 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:38.583 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-12T21:06:38.583 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:38.583 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:38.583 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-12 21:06:24.475709949 +0000 2024-04-12T21:06:38.584 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-12 20:58:58.556453115 +0000 2024-04-12T21:06:38.584 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-12 20:58:58.556453115 +0000 2024-04-12T21:06:38.584 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-12 20:58:58.556453115 +0000 2024-04-12T21:06:38.584 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T21:06:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-12T21:06:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-12T21:06:38.651 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000233625 s, 2.2 MB/s 2024-04-12T21:06:38.653 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T21:06:38.709 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-04-12T21:06:38.765 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T21:06:38.765 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:38.765 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-04-12T21:06:38.765 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:38.765 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:38.765 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-04-12 21:06:24.475709949 +0000 2024-04-12T21:06:38.766 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-04-12 20:58:58.868449480 +0000 2024-04-12T21:06:38.766 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-04-12 20:58:58.868449480 +0000 2024-04-12T21:06:38.766 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-04-12 20:58:58.868449480 +0000 2024-04-12T21:06:38.766 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T21:06:38.831 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-04-12T21:06:38.831 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-04-12T21:06:38.832 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000203396 s, 2.5 MB/s 2024-04-12T21:06:38.833 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T21:06:38.890 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-04-12T21:06:38.890 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-04-12T21:06:38.906 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-12T21:06:38.906 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 656 Links: 1 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-12 21:06:14.976565210 +0000 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-12 20:58:46.938135516 +0000 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-12 20:58:46.938135516 +0000 2024-04-12T21:06:38.962 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-12 20:58:46.938135516 +0000 2024-04-12T21:06:38.963 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-12T21:06:39.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:38 smithi153 ceph-mon[36569]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:39.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:39.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:39.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:39.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:39.028 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:39.030 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-12T21:06:39.030 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-12T21:06:39.030 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000210895 s, 2.4 MB/s 2024-04-12T21:06:39.032 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-12T21:06:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:38 smithi104 ceph-mon[36624]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:39.087 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-04-12T21:06:39.143 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-12T21:06:39.143 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:39.143 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 677 Links: 1 2024-04-12T21:06:39.143 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:39.143 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:39.143 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-12 21:06:14.977565192 +0000 2024-04-12T21:06:39.143 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-12 20:58:47.293129152 +0000 2024-04-12T21:06:39.143 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-12 20:58:47.293129152 +0000 2024-04-12T21:06:39.144 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-12 20:58:47.293129152 +0000 2024-04-12T21:06:39.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-12T21:06:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:38 smithi081 ceph-mon[31453]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-12T21:06:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-12T21:06:39.211 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000207303 s, 2.5 MB/s 2024-04-12T21:06:39.212 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-12T21:06:39.267 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-04-12T21:06:39.323 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-12T21:06:39.323 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:39.323 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 693 Links: 1 2024-04-12T21:06:39.323 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:39.324 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:39.324 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-12 21:06:14.977565192 +0000 2024-04-12T21:06:39.324 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-12 20:58:47.624123218 +0000 2024-04-12T21:06:39.324 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-12 20:58:47.624123218 +0000 2024-04-12T21:06:39.324 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-12 20:58:47.624123218 +0000 2024-04-12T21:06:39.324 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-12T21:06:39.388 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-12T21:06:39.389 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-12T21:06:39.389 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000218076 s, 2.3 MB/s 2024-04-12T21:06:39.390 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-12T21:06:39.446 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 715 Links: 1 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-04-12 21:06:14.977565192 +0000 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-04-12 20:58:47.957117250 +0000 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-04-12 20:58:47.957117250 +0000 2024-04-12T21:06:39.502 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-04-12 20:58:47.957117250 +0000 2024-04-12T21:06:39.503 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-12T21:06:39.568 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-04-12T21:06:39.568 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-04-12T21:06:39.568 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.00021239 s, 2.4 MB/s 2024-04-12T21:06:39.569 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-12T21:06:39.626 INFO:tasks.cephadm:Deploying osd.0 on smithi081 with /dev/vg_nvme/lv_4... 2024-04-12T21:06:39.626 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T21:06:40.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:40.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:40.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:06:40.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:40.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:40.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:39 smithi153 ceph-mon[36569]: Deploying daemon mgr.z on smithi153 2024-04-12T21:06:40.068 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:06:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:06:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:39 smithi104 ceph-mon[36624]: Deploying daemon mgr.z on smithi153 2024-04-12T21:06:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:06:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:39 smithi081 ceph-mon[31453]: Deploying daemon mgr.z on smithi153 2024-04-12T21:06:40.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:40 smithi081 ceph-mon[31453]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:40.986 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:40 smithi153 ceph-mon[36569]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:41.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:40 smithi104 ceph-mon[36624]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:41.552 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:41 smithi153 systemd[1]: Started Ceph mgr.z for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:06:42.052 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:41.660+0000 7f742a3b0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:06:42.053 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:41.754+0000 7f742a3b0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:06:42.552 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:42.399+0000 7f742a3b0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:42.491+0000 7f742a3b0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:42.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-mon[36569]: Deploying daemon mgr.y on smithi104 2024-04-12T21:06:42.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:42.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:06:42.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:42.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:42.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:42 smithi104 ceph-mon[36624]: Deploying daemon mgr.y on smithi104 2024-04-12T21:06:42.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:42.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:42.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-12T21:06:42.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:42.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:42.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:42 smithi081 ceph-mon[31453]: Deploying daemon mgr.y on smithi104 2024-04-12T21:06:42.783 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T21:06:42.783 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T21:06:42.783 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-12T21:06:42.783 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-12T21:06:42.783 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0351089 s, 299 MB/s 2024-04-12T21:06:42.783 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-12T21:06:43.052 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:42.632+0000 7f742a3b0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:06:43.053 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:42.727+0000 7f742a3b0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:06:43.053 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:42.809+0000 7f742a3b0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:06:43.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:43 smithi153 ceph-mon[36569]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:43.360 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:43.074+0000 7f742a3b0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T21:06:43.360 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:43.183+0000 7f742a3b0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:06:43.360 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:43.269+0000 7f742a3b0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:06:43.360 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:43.364+0000 7f742a3b0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:06:43.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:43 smithi081 ceph-mon[31453]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:43.527 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:43 smithi104 ceph-mon[36624]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:43.527 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:43 smithi104 systemd[1]: Starting Ceph mgr.y for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:06:43.527 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:43 smithi104 podman[37992]: 2024-04-12 21:06:43.43364824 +0000 UTC m=+0.017481046 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:06:43.579 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi081:vg_nvme/lv_4 2024-04-12T21:06:43.788 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:43 smithi104 podman[37992]: 2024-04-12 21:06:43.542154804 +0000 UTC m=+0.125987600 container create a2701ccbf323895b41e69c464d574083d2053e53533a1f340ba7a176479336e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:06:43.788 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:43 smithi104 podman[37992]: 2024-04-12 21:06:43.765741884 +0000 UTC m=+0.349574695 container init a2701ccbf323895b41e69c464d574083d2053e53533a1f340ba7a176479336e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-12T21:06:43.788 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:43 smithi104 podman[37992]: 2024-04-12 21:06:43.768441007 +0000 UTC m=+0.352273816 container start a2701ccbf323895b41e69c464d574083d2053e53533a1f340ba7a176479336e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T21:06:43.788 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:43 smithi104 ceph-mgr[38011]: -- 172.21.15.104:0/2721745754 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df69bd65a0 con 0x55df69baac00 2024-04-12T21:06:43.802 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:43.541+0000 7f742a3b0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:06:44.024 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:06:44.078 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:43 smithi104 bash[37992]: a2701ccbf323895b41e69c464d574083d2053e53533a1f340ba7a176479336e6 2024-04-12T21:06:44.078 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:43 smithi104 systemd[1]: Started Ceph mgr.y for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:06:44.341 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:44 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:44.322+0000 7f4f152f7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-12T21:06:44.802 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:44 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:44.543+0000 7f742a3b0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:06:44.803 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:44 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:44.680+0000 7f742a3b0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:06:44.828 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:44 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:44.423+0000 7f4f152f7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-12T21:06:45.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:45.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:44 smithi081 ceph-mon[31453]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:45.218 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:45 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:45.136+0000 7f4f152f7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-12T21:06:45.218 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.218 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.218 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.218 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.219 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:45.219 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:44 smithi104 ceph-mon[36624]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:45.302 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:45 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:45.035+0000 7f742a3b0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:06:45.303 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:45 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:45.129+0000 7f742a3b0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:06:45.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:45.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:45.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:44 smithi153 ceph-mon[36569]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:45.472 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:45 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:45.234+0000 7f4f152f7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-12T21:06:45.472 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:45 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:45.379+0000 7f4f152f7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-12T21:06:45.472 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:45 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:45.475+0000 7f4f152f7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-12T21:06:45.802 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:45 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:45.332+0000 7f742a3b0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:06:45.803 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:45 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:45.529+0000 7f742a3b0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:06:45.828 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:45 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:45.576+0000 7f4f152f7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 -- 172.21.15.81:0/327810666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 msgr2=0x7f3094105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 --2- 172.21.15.81:0/327810666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 0x7f3094105990 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3084009990 tx=0x7f3084009ca0 comp rx=0 tx=0).stop 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 -- 172.21.15.81:0/327810666 shutdown_connections 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 --2- 172.21.15.81:0/327810666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3094108800 0x7f309410ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 --2- 172.21.15.81:0/327810666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 0x7f3094105990 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 --2- 172.21.15.81:0/327810666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094105ed0 0x7f30941082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 -- 172.21.15.81:0/327810666 >> 172.21.15.81:0/327810666 conn(0x7f30940fcfb0 msgr2=0x7f30940ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 -- 172.21.15.81:0/327810666 shutdown_connections 2024-04-12T21:06:45.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.913+0000 7f3099e00700 1 -- 172.21.15.81:0/327810666 wait complete. 2024-04-12T21:06:45.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.914+0000 7f3099e00700 1 Processor -- start 2024-04-12T21:06:45.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.914+0000 7f3099e00700 1 -- start start 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3099e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 0x7f3094102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3099e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094105ed0 0x7f3094101550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3099e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3094108800 0x7f3094101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3099e00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f309410e750 con 0x7f3094108800 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3099e00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f309410e5d0 con 0x7f30941035a0 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3099e00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f309410e8d0 con 0x7f3094105ed0 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3092a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 0x7f3094102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3092a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 0x7f3094102f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:59096/0 (socket says 172.21.15.81:59096) 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3092a00700 1 -- 172.21.15.81:0/548096071 learned_addr learned my addr 172.21.15.81:0/548096071 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:06:45.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3092000700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094105ed0 0x7f3094101550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.915+0000 7f3093400700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3094108800 0x7f3094101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3092a00700 1 -- 172.21.15.81:0/548096071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094105ed0 msgr2=0x7f3094101550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3092a00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094105ed0 0x7f3094101550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3092a00700 1 -- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3094108800 msgr2=0x7f3094101a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3092a00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3094108800 0x7f3094101a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3092a00700 1 -- 172.21.15.81:0/548096071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3094102340 con 0x7f30941035a0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3092000700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094105ed0 0x7f3094101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3092a00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 0x7f3094102f00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f308400b2e0 tx=0x7f308400bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3084026070 con 0x7f30941035a0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3084003d70 con 0x7f30941035a0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.916+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30941a2f20 con 0x7f30941035a0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.917+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3084021410 con 0x7f30941035a0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.917+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30941a32c0 con 0x7f30941035a0 2024-04-12T21:06:45.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.918+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30940697a0 con 0x7f30941035a0 2024-04-12T21:06:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.918+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3084029070 con 0x7f30941035a0 2024-04-12T21:06:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.919+0000 7f3098a00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3078039890 0x7f307803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:06:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.919+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f308404dab0 con 0x7f30941035a0 2024-04-12T21:06:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.919+0000 7f3092000700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3078039890 0x7f307803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:06:45.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.920+0000 7f3092000700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3078039890 0x7f307803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3080011490 tx=0x7f3080009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:06:45.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:45.924+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3084013070 con 0x7f30941035a0 2024-04-12T21:06:46.087 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:45 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:45.820+0000 7f742a3b0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:06:46.129 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:45 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:45.847+0000 7f4f152f7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-12T21:06:46.129 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:45 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:45.955+0000 7f4f152f7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-12T21:06:46.129 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:46.040+0000 7f4f152f7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-12T21:06:46.129 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:46.132+0000 7f4f152f7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-12T21:06:46.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:46.166+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3094062ef0 con 0x7f3078039890 2024-04-12T21:06:46.340 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:46.105+0000 7f742a3b0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:06:46.340 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:46.344+0000 7f742a3b0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:06:46.528 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:46.305+0000 7f4f152f7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: Reconfiguring mgr.x (unknown last config time)... 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: Reconfiguring daemon mgr.x on smithi081 2024-04-12T21:06:46.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:06:46.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:06:46.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:06:46.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:46 smithi153 ceph-mon[36569]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:46.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: Reconfiguring mgr.x (unknown last config time)... 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: Reconfiguring daemon mgr.x on smithi081 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:06:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:46 smithi104 ceph-mon[36624]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:46.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.840 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: Reconfiguring mgr.x (unknown last config time)... 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: Reconfiguring daemon mgr.x on smithi081 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:06:46.841 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:06:46.842 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:46.842 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:46 smithi081 ceph-mon[31453]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:47.554 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:47 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:47.162+0000 7f742a3b0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:06:47.578 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:47 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:47.271+0000 7f4f152f7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-12T21:06:47.578 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:47 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:47.401+0000 7f4f152f7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-12T21:06:48.033 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:47 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:47.751+0000 7f4f152f7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-12T21:06:48.033 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:47 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:47.840+0000 7f4f152f7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-12T21:06:48.033 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:48.036+0000 7f4f152f7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:06:48.052 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:47 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:47.738+0000 7f742a3b0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:06:48.052 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:47 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:47.838+0000 7f742a3b0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:06:48.328 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:48.222+0000 7f4f152f7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-12T21:06:48.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: Standby manager daemon z started 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-12T21:06:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-mon[36624]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:06:48.552 INFO:journalctl@ceph.mgr.z.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-z[38449]: 2024-04-12T21:06:48.057+0000 7f742a3b0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: Standby manager daemon z started 2024-04-12T21:06:48.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-12T21:06:48.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:48 smithi153 ceph-mon[36569]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:06:48.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:06:48.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:06:48.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:06:48.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:06:48.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-12T21:06:48.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:06:48.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: Standby manager daemon z started 2024-04-12T21:06:48.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-12T21:06:48.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:48 smithi081 ceph-mon[31453]: from='mgr.? 172.21.15.153:0/2387276250' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:06:48.759 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:48.506+0000 7f4f152f7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-12T21:06:48.759 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:48 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:48.763+0000 7f4f152f7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-12T21:06:48.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:48.906+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7f3084009d10 con 0x7f30941035a0 2024-04-12T21:06:49.078 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:49 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:49.000+0000 7f4f152f7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-12T21:06:49.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:49 smithi153 ceph-mon[36569]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:49.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:49 smithi153 ceph-mon[36569]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-12T21:06:49.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-12T21:06:49.570 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:49 smithi081 ceph-mon[31453]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:49.571 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:49 smithi081 ceph-mon[31453]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-12T21:06:49.571 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-12T21:06:49.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:49 smithi104 ceph-mon[36624]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:49.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:49 smithi104 ceph-mon[36624]: mgrmap e14: x(active, since 86s), standbys: z 2024-04-12T21:06:49.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-12T21:06:50.078 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:49 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:49.797+0000 7f4f152f7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-12T21:06:50.672 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:50.363+0000 7f4f152f7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-12T21:06:50.673 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:50.459+0000 7f4f152f7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-12T21:06:50.673 INFO:journalctl@ceph.mgr.y.smithi104.stdout:Apr 12 21:06:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-y[38007]: 2024-04-12T21:06:50.676+0000 7f4f152f7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-12T21:06:51.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:06:51.297+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mon.2 v2:172.21.15.104:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f308402a3b0 con 0x7f30941035a0 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: Standby manager daemon y started 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2292159398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]: dispatch 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]: dispatch 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]': finished 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: osdmap e5: 1 total, 0 up, 1 in 2024-04-12T21:06:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:06:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: Standby manager daemon y started 2024-04-12T21:06:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T21:06:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:06:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T21:06:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:06:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2292159398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]: dispatch 2024-04-12T21:06:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]: dispatch 2024-04-12T21:06:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]': finished 2024-04-12T21:06:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: osdmap e5: 1 total, 0 up, 1 in 2024-04-12T21:06:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:06:51.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: Standby manager daemon y started 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: from='mgr.? 172.21.15.104:0/2779806981' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2292159398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]: dispatch 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]: dispatch 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5891e774-9745-4511-a67d-b0d5b30f48ad"}]': finished 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: osdmap e5: 1 total, 0 up, 1 in 2024-04-12T21:06:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:06:52.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:52 smithi153 ceph-mon[36569]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-12T21:06:52.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:06:52.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:52 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4276069681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:06:52.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:52 smithi104 ceph-mon[36624]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-12T21:06:52.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:06:52.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:52 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4276069681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:06:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:52 smithi081 ceph-mon[31453]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-12T21:06:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-12T21:06:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:52 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4276069681' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:06:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:53 smithi153 ceph-mon[36569]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:53.553 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:53 smithi081 ceph-mon[31453]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:53 smithi104 ceph-mon[36624]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:55 smithi104 ceph-mon[36624]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:55.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:55 smithi081 ceph-mon[31453]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:55.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:55 smithi153 ceph-mon[36569]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:57 smithi104 ceph-mon[36624]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:57.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:57 smithi081 ceph-mon[31453]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:57 smithi153 ceph-mon[36569]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:06:59 smithi104 ceph-mon[36624]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:59.586 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:06:59 smithi081 ceph-mon[31453]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:06:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:06:59 smithi153 ceph-mon[36569]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T21:07:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T21:07:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-12T21:07:00.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:01.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:01 smithi104 ceph-mon[36624]: Deploying daemon osd.0 on smithi081 2024-04-12T21:07:01.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:01 smithi104 ceph-mon[36624]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:01 smithi081 ceph-mon[31453]: Deploying daemon osd.0 on smithi081 2024-04-12T21:07:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:01 smithi081 ceph-mon[31453]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:01 smithi153 ceph-mon[36569]: Deploying daemon osd.0 on smithi081 2024-04-12T21:07:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:01 smithi153 ceph-mon[36569]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:03.458 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:03 smithi081 ceph-mon[31453]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:03 smithi104 ceph-mon[36624]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:03.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:03 smithi153 ceph-mon[36569]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:05.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:05 smithi081 ceph-mon[31453]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:05.604 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:05 smithi153 ceph-mon[36569]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:05 smithi104 ceph-mon[36624]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:07 smithi081 ceph-mon[31453]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:07 smithi153 ceph-mon[36569]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:07 smithi104 ceph-mon[36624]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:08.812 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:08.812 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:08.812 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:08.812 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:08 smithi081 ceph-mon[31453]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:09.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:09.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:09.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:09.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:08 smithi153 ceph-mon[36569]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:09.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:09.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:09.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:09.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:08 smithi104 ceph-mon[36624]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:11 smithi153 ceph-mon[36569]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:11 smithi104 ceph-mon[36624]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:11 smithi081 ceph-mon[31453]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:11.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.754+0000 7f3098a00700 1 -- 172.21.15.81:0/548096071 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3094062ef0 con 0x7f3078039890 2024-04-12T21:07:11.751 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 0 on host 'smithi081' 2024-04-12T21:07:11.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3078039890 msgr2=0x7f307803bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:11.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3078039890 0x7f307803bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3080011490 tx=0x7f3080009260 comp rx=0 tx=0).stop 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 msgr2=0x7f3094102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 0x7f3094102f00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f308400b2e0 tx=0x7f308400bbb0 comp rx=0 tx=0).stop 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 shutdown_connections 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3078039890 0x7f307803bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3094108800 0x7f3094101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f30941035a0 0x7f3094102f00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 --2- 172.21.15.81:0/548096071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094105ed0 0x7f3094101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 >> 172.21.15.81:0/548096071 conn(0x7f30940fcfb0 msgr2=0x7f3094104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 shutdown_connections 2024-04-12T21:07:11.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:07:11.757+0000 7f3099e00700 1 -- 172.21.15.81:0/548096071 wait complete. 2024-04-12T21:07:12.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:12 smithi081 ceph-mon[31453]: from='osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:07:12.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:12 smithi081 ceph-mon[31453]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:07:12.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:12.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:12.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:12.443 DEBUG:teuthology.orchestra.run.smithi081:osd.0> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.0.service 2024-04-12T21:07:12.447 INFO:tasks.cephadm:Deploying osd.1 on smithi104 with /dev/vg_nvme/lv_4... 2024-04-12T21:07:12.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T21:07:12.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:12 smithi153 ceph-mon[36569]: from='osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:07:12.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:12 smithi153 ceph-mon[36569]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:07:12.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:12.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:12.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:12.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:12 smithi104 ceph-mon[36624]: from='osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:07:12.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:12 smithi104 ceph-mon[36624]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-12T21:07:12.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:12.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:12.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:12.698 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.b/config 2024-04-12T21:07:13.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:13 smithi153 ceph-mon[36569]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:13 smithi153 ceph-mon[36569]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T21:07:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:13 smithi153 ceph-mon[36569]: from='osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:07:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:13 smithi153 ceph-mon[36569]: osdmap e6: 1 total, 0 up, 1 in 2024-04-12T21:07:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:13 smithi153 ceph-mon[36569]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:07:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:13 smithi104 ceph-mon[36624]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:13 smithi104 ceph-mon[36624]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T21:07:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:13 smithi104 ceph-mon[36624]: from='osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:07:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:13 smithi104 ceph-mon[36624]: osdmap e6: 1 total, 0 up, 1 in 2024-04-12T21:07:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:13 smithi104 ceph-mon[36624]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:07:13.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:13 smithi081 ceph-mon[31453]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:13.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:13 smithi081 ceph-mon[31453]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-12T21:07:13.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:13 smithi081 ceph-mon[31453]: from='osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:07:13.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:13 smithi081 ceph-mon[31453]: osdmap e6: 1 total, 0 up, 1 in 2024-04-12T21:07:13.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:13.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:13 smithi081 ceph-mon[31453]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:07:14.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:14 smithi081 ceph-mon[31453]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:07:14.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:14 smithi081 ceph-mon[31453]: osdmap e7: 1 total, 0 up, 1 in 2024-04-12T21:07:14.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:14.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:14.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:14 smithi081 ceph-mon[31453]: from='osd.0 ' entity='osd.0' 2024-04-12T21:07:14.388 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:07:14 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-0[46176]: 2024-04-12T21:07:14.208+0000 7f4a8f600700 -1 osd.0 0 waiting for initial osdmap 2024-04-12T21:07:14.388 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:07:14 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-0[46176]: 2024-04-12T21:07:14.224+0000 7f4a85400700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:07:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T21:07:14.505 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T21:07:14.506 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-12T21:07:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-12T21:07:14.506 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113138 s, 927 MB/s 2024-04-12T21:07:14.506 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-12T21:07:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:14 smithi153 ceph-mon[36569]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:07:14.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:14 smithi153 ceph-mon[36569]: osdmap e7: 1 total, 0 up, 1 in 2024-04-12T21:07:14.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:14.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:14.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:14 smithi153 ceph-mon[36569]: from='osd.0 ' entity='osd.0' 2024-04-12T21:07:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:14 smithi104 ceph-mon[36624]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:07:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:14 smithi104 ceph-mon[36624]: osdmap e7: 1 total, 0 up, 1 in 2024-04-12T21:07:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:14 smithi104 ceph-mon[36624]: from='osd.0 ' entity='osd.0' 2024-04-12T21:07:15.200 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-04-12T21:07:15.434 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.b/config 2024-04-12T21:07:15.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:15 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:07:15.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:15 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:07:15.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:15 smithi153 ceph-mon[36569]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:15.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:15.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:15 smithi153 ceph-mon[36569]: osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] boot 2024-04-12T21:07:15.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:15 smithi153 ceph-mon[36569]: osdmap e8: 1 total, 1 up, 1 in 2024-04-12T21:07:15.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:15.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:15 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:07:15.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:15 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:07:15.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:15 smithi104 ceph-mon[36624]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:15.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:15.556 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:15 smithi104 ceph-mon[36624]: osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] boot 2024-04-12T21:07:15.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:15 smithi104 ceph-mon[36624]: osdmap e8: 1 total, 1 up, 1 in 2024-04-12T21:07:15.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:15 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:07:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:15 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:07:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:15 smithi081 ceph-mon[31453]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-12T21:07:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:15 smithi081 ceph-mon[31453]: osd.0 [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] boot 2024-04-12T21:07:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:15 smithi081 ceph-mon[31453]: osdmap e8: 1 total, 1 up, 1 in 2024-04-12T21:07:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-12T21:07:17.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.021+0000 7f94f05ea700 1 -- 172.21.15.104:0/3664059622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 msgr2=0x7f94e8102f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:17.020 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.021+0000 7f94f05ea700 1 --2- 172.21.15.104:0/3664059622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 0x7f94e8102f30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f94d80099c0 tx=0x7f94d8009cd0 comp rx=0 tx=0).stop 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.021+0000 7f94f05ea700 1 -- 172.21.15.104:0/3664059622 shutdown_connections 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.021+0000 7f94f05ea700 1 --2- 172.21.15.104:0/3664059622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94e81046b0 0x7f94e810af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 --2- 172.21.15.104:0/3664059622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 0x7f94e8102f30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 --2- 172.21.15.104:0/3664059622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e8103d10 0x7f94e8104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 -- 172.21.15.104:0/3664059622 >> 172.21.15.104:0/3664059622 conn(0x7f94e80fe0b0 msgr2=0x7f94e81004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 -- 172.21.15.104:0/3664059622 shutdown_connections 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 -- 172.21.15.104:0/3664059622 wait complete. 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 Processor -- start 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 -- start start 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 0x7f94e8198dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e8103d10 0x7f94e8199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94e81046b0 0x7f94e81a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:07:17.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f94e810e770 con 0x7f94e81046b0 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f94e810e5f0 con 0x7f94e8102b10 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94f05ea700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f94e810e8f0 con 0x7f94e8103d10 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94ee386700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 0x7f94e8198dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94ee386700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 0x7f94e8198dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38470/0 (socket says 172.21.15.104:38470) 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.022+0000 7f94ee386700 1 -- 172.21.15.104:0/2005501495 learned_addr learned my addr 172.21.15.104:0/2005501495 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94edb85700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e8103d10 0x7f94e8199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94ee386700 1 -- 172.21.15.104:0/2005501495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e8103d10 msgr2=0x7f94e8199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94ee386700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e8103d10 0x7f94e8199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94ee386700 1 -- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94e81046b0 msgr2=0x7f94e81a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94ee386700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94e81046b0 0x7f94e81a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94ee386700 1 -- 172.21.15.104:0/2005501495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e81a0a30 con 0x7f94e8102b10 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94edb85700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e8103d10 0x7f94e8199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94ee386700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 0x7f94e8198dd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94d800c010 tx=0x7f94d8009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:07:17.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94df7fe700 1 -- 172.21.15.104:0/2005501495 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8027070 con 0x7f94e8102b10 2024-04-12T21:07:17.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94df7fe700 1 -- 172.21.15.104:0/2005501495 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d8004150 con 0x7f94e8102b10 2024-04-12T21:07:17.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94df7fe700 1 -- 172.21.15.104:0/2005501495 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d800f460 con 0x7f94e8102b10 2024-04-12T21:07:17.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e81a0cc0 con 0x7f94e8102b10 2024-04-12T21:07:17.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.023+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e81a11a0 con 0x7f94e8102b10 2024-04-12T21:07:17.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.024+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e80697a0 con 0x7f94e8102b10 2024-04-12T21:07:17.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.025+0000 7f94df7fe700 1 -- 172.21.15.104:0/2005501495 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f94d802a070 con 0x7f94e8102b10 2024-04-12T21:07:17.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.025+0000 7f94df7fe700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94d40a3c30 0x7f94d40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:07:17.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.026+0000 7f94df7fe700 1 -- 172.21.15.104:0/2005501495 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f94d80cf860 con 0x7f94e8102b10 2024-04-12T21:07:17.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.026+0000 7f94edb85700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94d40a3c30 0x7f94d40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:07:17.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.027+0000 7f94edb85700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94d40a3c30 0x7f94d40a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94e4005fd0 tx=0x7f94e4005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:07:17.026 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.030+0000 7f94df7fe700 1 -- 172.21.15.104:0/2005501495 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94d805eff0 con 0x7f94e8102b10 2024-04-12T21:07:17.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:17.242+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f94e8062ef0 con 0x7f94d40a3c30 2024-04-12T21:07:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: osdmap e9: 1 total, 1 up, 1 in 2024-04-12T21:07:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:07:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: osdmap e9: 1 total, 1 up, 1 in 2024-04-12T21:07:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:07:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: osdmap e9: 1 total, 1 up, 1 in 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:07:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:18.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:18 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi081 2024-04-12T21:07:18.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:18 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 17089M 2024-04-12T21:07:18.520 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:18 smithi104 ceph-mon[36624]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:07:18.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:18 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi081 2024-04-12T21:07:18.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:18 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 17089M 2024-04-12T21:07:18.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:18 smithi081 ceph-mon[31453]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:07:18.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:18 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi081 2024-04-12T21:07:18.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:18 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 17089M 2024-04-12T21:07:18.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:18 smithi153 ceph-mon[36569]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:07:19.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:19 smithi104 ceph-mon[36624]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:19.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:19 smithi081 ceph-mon[31453]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:19 smithi153 ceph-mon[36569]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:20.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:20 smithi104 ceph-mon[36624]: from='client.? 172.21.15.104:0/2568081069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]: dispatch 2024-04-12T21:07:20.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:20 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]: dispatch 2024-04-12T21:07:20.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:20 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]': finished 2024-04-12T21:07:20.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:20 smithi104 ceph-mon[36624]: osdmap e10: 2 total, 1 up, 2 in 2024-04-12T21:07:20.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:20 smithi081 ceph-mon[31453]: from='client.? 172.21.15.104:0/2568081069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]: dispatch 2024-04-12T21:07:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:20 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]: dispatch 2024-04-12T21:07:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:20 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]': finished 2024-04-12T21:07:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:20 smithi081 ceph-mon[31453]: osdmap e10: 2 total, 1 up, 2 in 2024-04-12T21:07:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:20.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:20 smithi153 ceph-mon[36569]: from='client.? 172.21.15.104:0/2568081069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]: dispatch 2024-04-12T21:07:20.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:20 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]: dispatch 2024-04-12T21:07:20.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:20 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "60afc2bd-b9e9-4489-9cbc-a756b944fdc4"}]': finished 2024-04-12T21:07:20.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:20 smithi153 ceph-mon[36569]: osdmap e10: 2 total, 1 up, 2 in 2024-04-12T21:07:20.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:21.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:21 smithi081 ceph-mon[31453]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:21.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:21 smithi081 ceph-mon[31453]: from='client.? 172.21.15.104:0/463275940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:07:21.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:21 smithi153 ceph-mon[36569]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:21.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:21 smithi153 ceph-mon[36569]: from='client.? 172.21.15.104:0/463275940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:07:21.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:21 smithi104 ceph-mon[36624]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:21.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:21 smithi104 ceph-mon[36624]: from='client.? 172.21.15.104:0/463275940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:07:23.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:23 smithi104 ceph-mon[36624]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:23.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:23 smithi081 ceph-mon[31453]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:23 smithi153 ceph-mon[36569]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:25.450 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:25 smithi104 ceph-mon[36624]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:25 smithi081 ceph-mon[31453]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:25 smithi153 ceph-mon[36569]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:27 smithi081 ceph-mon[31453]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T21:07:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:27.744 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:27 smithi104 ceph-mon[36624]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:27.744 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T21:07:27.744 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:27 smithi153 ceph-mon[36569]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:27.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-12T21:07:27.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:28.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:28 smithi081 ceph-mon[31453]: Deploying daemon osd.1 on smithi104 2024-04-12T21:07:28.697 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:28 smithi104 ceph-mon[36624]: Deploying daemon osd.1 on smithi104 2024-04-12T21:07:28.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:28 smithi153 ceph-mon[36569]: Deploying daemon osd.1 on smithi104 2024-04-12T21:07:29.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:29 smithi081 ceph-mon[31453]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:29.685 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:29 smithi104 ceph-mon[36624]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:29.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:29 smithi153 ceph-mon[36569]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:31.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:31 smithi081 ceph-mon[31453]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:31.700 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:31 smithi104 ceph-mon[36624]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:31 smithi153 ceph-mon[36569]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:33 smithi104 ceph-mon[36624]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:33.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:33 smithi081 ceph-mon[31453]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:33.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:33.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:33.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:33 smithi153 ceph-mon[36569]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:33.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:33.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:33.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:35.477 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:35 smithi104 ceph-mon[36624]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:35.477 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.477 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.477 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:35.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:35.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:35 smithi081 ceph-mon[31453]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:35.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:35.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:35.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:35 smithi153 ceph-mon[36569]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:35.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:35.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:35.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:35.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.830+0000 7f94df7fe700 1 -- 172.21.15.104:0/2005501495 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f94e8062ef0 con 0x7f94d40a3c30 2024-04-12T21:07:35.827 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 1 on host 'smithi104' 2024-04-12T21:07:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94d40a3c30 msgr2=0x7f94d40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94d40a3c30 0x7f94d40a60f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f94e4005fd0 tx=0x7f94e4005ee0 comp rx=0 tx=0).stop 2024-04-12T21:07:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 msgr2=0x7f94e8198dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 0x7f94e8198dd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f94d800c010 tx=0x7f94d8009f10 comp rx=0 tx=0).stop 2024-04-12T21:07:35.828 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 shutdown_connections 2024-04-12T21:07:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94d40a3c30 0x7f94d40a60f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f94e81046b0 0x7f94e81a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94e8102b10 0x7f94e8198dd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 --2- 172.21.15.104:0/2005501495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f94e8103d10 0x7f94e8199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 >> 172.21.15.104:0/2005501495 conn(0x7f94e80fe0b0 msgr2=0x7f94e81070b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:07:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 shutdown_connections 2024-04-12T21:07:35.829 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:07:35.832+0000 7f94f05ea700 1 -- 172.21.15.104:0/2005501495 wait complete. 2024-04-12T21:07:36.452 DEBUG:teuthology.orchestra.run.smithi104:osd.1> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.1.service 2024-04-12T21:07:36.455 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_4... 2024-04-12T21:07:36.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-12T21:07:36.700 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:07:37.073 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:37.073 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:37.073 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:37.073 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:36 smithi153 ceph-mon[36569]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:37.073 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:36 smithi153 ceph-mon[36569]: from='osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:07:37.073 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:36 smithi153 ceph-mon[36569]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:07:37.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:37.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:37.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:37.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:36 smithi104 ceph-mon[36624]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:37.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:36 smithi104 ceph-mon[36624]: from='osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:07:37.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:36 smithi104 ceph-mon[36624]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:07:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:36 smithi081 ceph-mon[31453]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:36 smithi081 ceph-mon[31453]: from='osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:07:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:36 smithi081 ceph-mon[31453]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-12T21:07:37.940 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:37 smithi104 ceph-mon[36624]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T21:07:37.940 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:37 smithi104 ceph-mon[36624]: osdmap e11: 2 total, 1 up, 2 in 2024-04-12T21:07:37.941 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:37.941 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:37 smithi104 ceph-mon[36624]: from='osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:07:37.941 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:37 smithi104 ceph-mon[36624]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:07:37.972 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:37 smithi153 ceph-mon[36569]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T21:07:37.972 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:37 smithi153 ceph-mon[36569]: osdmap e11: 2 total, 1 up, 2 in 2024-04-12T21:07:37.973 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:37.973 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:37 smithi153 ceph-mon[36569]: from='osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:07:37.973 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:37 smithi153 ceph-mon[36569]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:07:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:37 smithi081 ceph-mon[31453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-12T21:07:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:37 smithi081 ceph-mon[31453]: osdmap e11: 2 total, 1 up, 2 in 2024-04-12T21:07:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:37 smithi081 ceph-mon[31453]: from='osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:07:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:37 smithi081 ceph-mon[31453]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:07:38.586 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-12T21:07:38.586 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-12T21:07:38.586 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-12T21:07:38.586 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-12T21:07:38.586 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149082 s, 703 MB/s 2024-04-12T21:07:38.586 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-12T21:07:38.785 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:07:38 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-1[41216]: 2024-04-12T21:07:38.533+0000 7f4d10aa6700 -1 osd.1 0 waiting for initial osdmap 2024-04-12T21:07:38.785 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:07:38 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-1[41216]: 2024-04-12T21:07:38.538+0000 7f4d08891700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:07:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:38 smithi104 ceph-mon[36624]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:07:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:38 smithi104 ceph-mon[36624]: osdmap e12: 2 total, 1 up, 2 in 2024-04-12T21:07:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:38 smithi104 ceph-mon[36624]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:38 smithi104 ceph-mon[36624]: from='osd.1 ' entity='osd.1' 2024-04-12T21:07:39.132 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:38 smithi153 ceph-mon[36569]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:07:39.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:38 smithi153 ceph-mon[36569]: osdmap e12: 2 total, 1 up, 2 in 2024-04-12T21:07:39.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:39.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:39.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:38 smithi153 ceph-mon[36569]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:39.133 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:38 smithi153 ceph-mon[36569]: from='osd.1 ' entity='osd.1' 2024-04-12T21:07:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:38 smithi081 ceph-mon[31453]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:07:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:38 smithi081 ceph-mon[31453]: osdmap e12: 2 total, 1 up, 2 in 2024-04-12T21:07:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:38 smithi081 ceph-mon[31453]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-12T21:07:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:38 smithi081 ceph-mon[31453]: from='osd.1 ' entity='osd.1' 2024-04-12T21:07:39.159 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-04-12T21:07:39.400 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:07:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:07:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:07:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] boot 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: osdmap e13: 2 total, 2 up, 2 in 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:07:40.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:07:40.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:40.188 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] boot 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: osdmap e13: 2 total, 2 up, 2 in 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:40.189 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:07:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:07:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: osd.1 [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] boot 2024-04-12T21:07:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: osdmap e13: 2 total, 2 up, 2 in 2024-04-12T21:07:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-12T21:07:40.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:07:40.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:40.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:40.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:40.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 -- 172.21.15.153:0/124986951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff44105e00 msgr2=0x7eff441081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:40.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 --2- 172.21.15.153:0/124986951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff44105e00 0x7eff441081f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7eff38009990 tx=0x7eff38009ca0 comp rx=0 tx=0).stop 2024-04-12T21:07:40.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 -- 172.21.15.153:0/124986951 shutdown_connections 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 --2- 172.21.15.153:0/124986951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff44108730 0x7eff4410aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 --2- 172.21.15.153:0/124986951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff441034d0 0x7eff441058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 --2- 172.21.15.153:0/124986951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff44105e00 0x7eff441081f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 -- 172.21.15.153:0/124986951 >> 172.21.15.153:0/124986951 conn(0x7eff440fcea0 msgr2=0x7eff440ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 -- 172.21.15.153:0/124986951 shutdown_connections 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 -- 172.21.15.153:0/124986951 wait complete. 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 Processor -- start 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.960+0000 7eff4919d700 1 -- start start 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff4919d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441034d0 0x7eff4419aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff4919d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff44105e00 0x7eff4419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff4919d700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff44108730 0x7eff441a2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff4919d700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7eff4410e6c0 con 0x7eff44108730 2024-04-12T21:07:40.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff4919d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7eff4410e540 con 0x7eff44105e00 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff4919d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7eff4410e840 con 0x7eff441034d0 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441034d0 0x7eff4419aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441034d0 0x7eff4419aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36646/0 (socket says 172.21.15.153:36646) 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 -- 172.21.15.153:0/3441349469 learned_addr learned my addr 172.21.15.153:0/3441349469 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff437fe700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff44105e00 0x7eff4419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 -- 172.21.15.153:0/3441349469 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff44105e00 msgr2=0x7eff4419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff44105e00 0x7eff4419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 -- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff44108730 msgr2=0x7eff441a2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff44108730 0x7eff441a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 -- 172.21.15.153:0/3441349469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff441a2b20 con 0x7eff441034d0 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.961+0000 7eff43fff700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441034d0 0x7eff4419aec0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7eff3400b560 tx=0x7eff3400b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.962+0000 7eff417fa700 1 -- 172.21.15.153:0/3441349469 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff34015020 con 0x7eff441034d0 2024-04-12T21:07:40.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.962+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff441a2e10 con 0x7eff441034d0 2024-04-12T21:07:40.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.962+0000 7eff417fa700 1 -- 172.21.15.153:0/3441349469 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff34011d10 con 0x7eff441034d0 2024-04-12T21:07:40.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.962+0000 7eff417fa700 1 -- 172.21.15.153:0/3441349469 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff34010e50 con 0x7eff441034d0 2024-04-12T21:07:40.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.962+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff441a32f0 con 0x7eff441034d0 2024-04-12T21:07:40.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.963+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff440697a0 con 0x7eff441034d0 2024-04-12T21:07:40.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.964+0000 7eff417fa700 1 -- 172.21.15.153:0/3441349469 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7eff34018020 con 0x7eff441034d0 2024-04-12T21:07:40.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.964+0000 7eff417fa700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7eff2c0a3d50 0x7eff2c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:07:40.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.964+0000 7eff417fa700 1 -- 172.21.15.153:0/3441349469 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7eff340ce2f0 con 0x7eff441034d0 2024-04-12T21:07:40.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.965+0000 7eff437fe700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7eff2c0a3d50 0x7eff2c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:07:40.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.966+0000 7eff437fe700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7eff2c0a3d50 0x7eff2c0a6210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7eff3800b5c0 tx=0x7eff38005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:07:40.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:40.969+0000 7eff417fa700 1 -- 172.21.15.153:0/3441349469 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff3401f020 con 0x7eff441034d0 2024-04-12T21:07:41.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:40 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi104 2024-04-12T21:07:41.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:40 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 17080M 2024-04-12T21:07:41.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:40 smithi153 ceph-mon[36569]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:41.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:40 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi104 2024-04-12T21:07:41.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:40 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 17080M 2024-04-12T21:07:41.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:40 smithi081 ceph-mon[31453]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:41.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:41.178+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7eff44062ef0 con 0x7eff2c0a3d50 2024-04-12T21:07:41.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:40 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi104 2024-04-12T21:07:41.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:40 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 17080M 2024-04-12T21:07:41.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:40 smithi104 ceph-mon[36624]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:42.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:41 smithi153 ceph-mon[36569]: osdmap e14: 2 total, 2 up, 2 in 2024-04-12T21:07:42.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:41 smithi153 ceph-mon[36569]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:07:42.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:07:42.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:07:42.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:42.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:41 smithi081 ceph-mon[31453]: osdmap e14: 2 total, 2 up, 2 in 2024-04-12T21:07:42.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:41 smithi081 ceph-mon[31453]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:07:42.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:07:42.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:07:42.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:42.229 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:41 smithi104 ceph-mon[36624]: osdmap e14: 2 total, 2 up, 2 in 2024-04-12T21:07:42.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:41 smithi104 ceph-mon[36624]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:07:42.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:07:42.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:07:42.230 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:42 smithi081 ceph-mon[31453]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:43.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:42 smithi153 ceph-mon[36569]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:43.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:42 smithi104 ceph-mon[36624]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:44.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:43 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/2967838259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]: dispatch 2024-04-12T21:07:44.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:43 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]: dispatch 2024-04-12T21:07:44.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:43 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]': finished 2024-04-12T21:07:44.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:43 smithi081 ceph-mon[31453]: osdmap e15: 3 total, 2 up, 3 in 2024-04-12T21:07:44.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:07:44.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:43 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/2967838259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]: dispatch 2024-04-12T21:07:44.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:43 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]: dispatch 2024-04-12T21:07:44.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:43 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]': finished 2024-04-12T21:07:44.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:43 smithi153 ceph-mon[36569]: osdmap e15: 3 total, 2 up, 3 in 2024-04-12T21:07:44.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:07:44.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:43 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/2967838259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]: dispatch 2024-04-12T21:07:44.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:43 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]: dispatch 2024-04-12T21:07:44.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:43 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4a8e96d-ddca-4406-977f-543283eadef9"}]': finished 2024-04-12T21:07:44.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:43 smithi104 ceph-mon[36624]: osdmap e15: 3 total, 2 up, 3 in 2024-04-12T21:07:44.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:07:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:44 smithi081 ceph-mon[31453]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:44 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/2969300720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:07:45.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:44 smithi153 ceph-mon[36569]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:45.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:44 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/2969300720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:07:45.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:44 smithi104 ceph-mon[36624]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:45.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:44 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/2969300720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:07:47.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:47 smithi153 ceph-mon[36569]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:47.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:47 smithi104 ceph-mon[36624]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:47.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:47 smithi081 ceph-mon[31453]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:49.395 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:49 smithi153 ceph-mon[36569]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:49.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:49 smithi104 ceph-mon[36624]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:49.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:49 smithi081 ceph-mon[31453]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:51.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:51 smithi153 ceph-mon[36569]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T21:07:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:51 smithi104 ceph-mon[36624]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T21:07:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:51.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:51 smithi081 ceph-mon[31453]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:51.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-12T21:07:51.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:52.465 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:52 smithi153 ceph-mon[36569]: Deploying daemon osd.2 on smithi153 2024-04-12T21:07:52.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:52 smithi104 ceph-mon[36624]: Deploying daemon osd.2 on smithi153 2024-04-12T21:07:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:52 smithi081 ceph-mon[31453]: Deploying daemon osd.2 on smithi153 2024-04-12T21:07:53.471 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:53 smithi153 ceph-mon[36569]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:53.554 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:53 smithi081 ceph-mon[31453]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:53 smithi104 ceph-mon[36624]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:55 smithi104 ceph-mon[36624]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:55.582 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:55 smithi153 ceph-mon[36569]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:55.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:55 smithi081 ceph-mon[31453]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:57.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:57.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:57.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:57.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:56 smithi153 ceph-mon[36569]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:56 smithi104 ceph-mon[36624]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:57.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:57.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:57.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:07:57.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:56 smithi081 ceph-mon[31453]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:59.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:59 smithi153 ceph-mon[36569]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:59.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:59.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:59.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:07:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.407+0000 7eff417fa700 1 -- 172.21.15.153:0/3441349469 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7eff44062ef0 con 0x7eff2c0a3d50 2024-04-12T21:07:59.404 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 2 on host 'smithi153' 2024-04-12T21:07:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7eff2c0a3d50 msgr2=0x7eff2c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7eff2c0a3d50 0x7eff2c0a6210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7eff3800b5c0 tx=0x7eff38005fd0 comp rx=0 tx=0).stop 2024-04-12T21:07:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441034d0 msgr2=0x7eff4419aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:07:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441034d0 0x7eff4419aec0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7eff3400b560 tx=0x7eff3400b870 comp rx=0 tx=0).stop 2024-04-12T21:07:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 shutdown_connections 2024-04-12T21:07:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7eff2c0a3d50 0x7eff2c0a6210 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7eff44108730 0x7eff441a2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7eff441034d0 0x7eff4419aec0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 --2- 172.21.15.153:0/3441349469 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff44105e00 0x7eff4419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:07:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 >> 172.21.15.153:0/3441349469 conn(0x7eff440fcea0 msgr2=0x7eff44102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:07:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 shutdown_connections 2024-04-12T21:07:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:07:59.410+0000 7eff4919d700 1 -- 172.21.15.153:0/3441349469 wait complete. 2024-04-12T21:07:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:59 smithi104 ceph-mon[36624]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:07:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:59 smithi081 ceph-mon[31453]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:07:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:07:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:07:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:07:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:07:59.969 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.2.service 2024-04-12T21:07:59.971 INFO:tasks.cephadm:Deploying osd.3 on smithi081 with /dev/vg_nvme/lv_3... 2024-04-12T21:07:59.972 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T21:08:00.417 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:08:00.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:00.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:00.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:00.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:00 smithi153 ceph-mon[36569]: from='osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:08:00.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:00 smithi153 ceph-mon[36569]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:08:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:00 smithi081 ceph-mon[31453]: from='osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:08:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:00 smithi081 ceph-mon[31453]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:08:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:00 smithi104 ceph-mon[36624]: from='osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:08:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:00 smithi104 ceph-mon[36624]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-12T21:08:01.611 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:01 smithi081 ceph-mon[31453]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:08:01.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:01 smithi081 ceph-mon[31453]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T21:08:01.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:01 smithi081 ceph-mon[31453]: osdmap e16: 3 total, 2 up, 3 in 2024-04-12T21:08:01.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:01.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:01 smithi081 ceph-mon[31453]: from='osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:08:01.612 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:01 smithi081 ceph-mon[31453]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:08:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:01 smithi153 ceph-mon[36569]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:08:01.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:01 smithi153 ceph-mon[36569]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T21:08:01.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:01 smithi153 ceph-mon[36569]: osdmap e16: 3 total, 2 up, 3 in 2024-04-12T21:08:01.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:01.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:01 smithi153 ceph-mon[36569]: from='osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:08:01.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:01 smithi153 ceph-mon[36569]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:08:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:01 smithi104 ceph-mon[36624]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:08:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:01 smithi104 ceph-mon[36624]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-12T21:08:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:01 smithi104 ceph-mon[36624]: osdmap e16: 3 total, 2 up, 3 in 2024-04-12T21:08:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:01 smithi104 ceph-mon[36624]: from='osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:08:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:01 smithi104 ceph-mon[36624]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:08:02.394 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-2[41742]: 2024-04-12T21:08:02.108+0000 7fdcacceb700 -1 osd.2 0 waiting for initial osdmap 2024-04-12T21:08:02.394 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-2[41742]: 2024-04-12T21:08:02.113+0000 7fdca4ad6700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:08:02.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:02 smithi081 ceph-mon[31453]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:08:02.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:02 smithi081 ceph-mon[31453]: osdmap e17: 3 total, 2 up, 3 in 2024-04-12T21:08:02.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:02.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:02.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:02 smithi081 ceph-mon[31453]: from='osd.2 ' entity='osd.2' 2024-04-12T21:08:02.666 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:02 smithi153 ceph-mon[36569]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:08:02.666 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:02 smithi153 ceph-mon[36569]: osdmap e17: 3 total, 2 up, 3 in 2024-04-12T21:08:02.666 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:02.666 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:02.667 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:02 smithi153 ceph-mon[36569]: from='osd.2 ' entity='osd.2' 2024-04-12T21:08:02.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:02 smithi104 ceph-mon[36624]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:08:02.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:02 smithi104 ceph-mon[36624]: osdmap e17: 3 total, 2 up, 3 in 2024-04-12T21:08:02.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:02.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:02.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:02 smithi104 ceph-mon[36624]: from='osd.2 ' entity='osd.2' 2024-04-12T21:08:03.140 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T21:08:03.140 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T21:08:03.140 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-12T21:08:03.140 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-12T21:08:03.140 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0356983 s, 294 MB/s 2024-04-12T21:08:03.140 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-12T21:08:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:08:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:08:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:08:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:03.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:03.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] boot 2024-04-12T21:08:03.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: osdmap e18: 3 total, 3 up, 3 in 2024-04-12T21:08:03.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] boot 2024-04-12T21:08:03.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: osdmap e18: 3 total, 3 up, 3 in 2024-04-12T21:08:03.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: osd.2 [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] boot 2024-04-12T21:08:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: osdmap e18: 3 total, 3 up, 3 in 2024-04-12T21:08:03.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-12T21:08:03.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi081:vg_nvme/lv_3 2024-04-12T21:08:04.306 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:08:04.704 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:04 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi153 2024-04-12T21:08:04.704 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:04 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 17080M 2024-04-12T21:08:04.704 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:08:04.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:04 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi153 2024-04-12T21:08:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:04 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 17080M 2024-04-12T21:08:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:08:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:04 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi153 2024-04-12T21:08:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:04 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 17080M 2024-04-12T21:08:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:08:05.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:05 smithi081 ceph-mon[31453]: pgmap v89: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:05.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-12T21:08:05.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:05 smithi081 ceph-mon[31453]: osdmap e19: 3 total, 3 up, 3 in 2024-04-12T21:08:05.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:08:05.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:05 smithi153 ceph-mon[36569]: pgmap v89: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:05.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-12T21:08:05.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:05 smithi153 ceph-mon[36569]: osdmap e19: 3 total, 3 up, 3 in 2024-04-12T21:08:05.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:08:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:05 smithi104 ceph-mon[36624]: pgmap v89: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-12T21:08:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:05 smithi104 ceph-mon[36624]: osdmap e19: 3 total, 3 up, 3 in 2024-04-12T21:08:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-12T21:08:06.282 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50398]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:08:06.283 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50398]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.283 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50398]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.283 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50398]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.283 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50401]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:08:06.283 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50401]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.283 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50401]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.283 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50401]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.458 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44937]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:08:06.458 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44937]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.458 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44937]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.458 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44937]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.458 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44940]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:08:06.458 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44940]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.458 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44940]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.458 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44940]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 -- 172.21.15.81:0/3437457897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8105f10 msgr2=0x7f53a8108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 --2- 172.21.15.81:0/3437457897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8105f10 0x7f53a8108300 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5394009990 tx=0x7f5394009ca0 comp rx=0 tx=0).stop 2024-04-12T21:08:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 -- 172.21.15.81:0/3437457897 shutdown_connections 2024-04-12T21:08:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 --2- 172.21.15.81:0/3437457897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a8108840 0x7f53a810acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 --2- 172.21.15.81:0/3437457897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a81035e0 0x7f53a81059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 --2- 172.21.15.81:0/3437457897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8105f10 0x7f53a8108300 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 -- 172.21.15.81:0/3437457897 >> 172.21.15.81:0/3437457897 conn(0x7f53a80fcfd0 msgr2=0x7f53a80ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:08:06.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 -- 172.21.15.81:0/3437457897 shutdown_connections 2024-04-12T21:08:06.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.465+0000 7f53af600700 1 -- 172.21.15.81:0/3437457897 wait complete. 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.467+0000 7f53af600700 1 Processor -- start 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.467+0000 7f53af600700 1 -- start start 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.467+0000 7f53af600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a81035e0 0x7f53a8198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.467+0000 7f53af600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a8105f10 0x7f53a81992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.467+0000 7f53af600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8108840 0x7f53a81a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.467+0000 7f53af600700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f53a810e7a0 con 0x7f53a81035e0 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.467+0000 7f53af600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f53a810e620 con 0x7f53a8105f10 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.467+0000 7f53af600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53a810e920 con 0x7f53a8108840 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a81035e0 0x7f53a8198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f539fe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a8105f10 0x7f53a81992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a81035e0 0x7f53a8198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41524/0 (socket says 172.21.15.81:41524) 2024-04-12T21:08:06.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 -- 172.21.15.81:0/1384902164 learned_addr learned my addr 172.21.15.81:0/1384902164 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:08:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53ad600700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8108840 0x7f53a81a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:08:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 -- 172.21.15.81:0/1384902164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8108840 msgr2=0x7f53a81a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8108840 0x7f53a81a0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:06.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 -- 172.21.15.81:0/1384902164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a8105f10 msgr2=0x7f53a81992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a8105f10 0x7f53a81992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 -- 172.21.15.81:0/1384902164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53a81a09c0 con 0x7f53a81035e0 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f539fe00700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a8105f10 0x7f53a81992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.468+0000 7f53acc00700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a81035e0 0x7f53a8198d60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f539800ef40 tx=0x7f539800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.469+0000 7f53a6000700 1 -- 172.21.15.81:0/1384902164 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539800c9c0 con 0x7f53a81035e0 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.469+0000 7f53a6000700 1 -- 172.21.15.81:0/1384902164 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5398019070 con 0x7f53a81035e0 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.469+0000 7f53a6000700 1 -- 172.21.15.81:0/1384902164 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5398014480 con 0x7f53a81035e0 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.469+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53a81013d0 con 0x7f53a81035e0 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.469+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53a81018c0 con 0x7f53a81035e0 2024-04-12T21:08:06.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.470+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53a80697a0 con 0x7f53a81035e0 2024-04-12T21:08:06.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.474+0000 7f53a6000700 1 -- 172.21.15.81:0/1384902164 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f5398012050 con 0x7f53a81035e0 2024-04-12T21:08:06.471 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.475+0000 7f53a6000700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f538c0a3c30 0x7f538c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:08:06.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.476+0000 7f53a6000700 1 -- 172.21.15.81:0/1384902164 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f53980cf4a0 con 0x7f53a81035e0 2024-04-12T21:08:06.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.476+0000 7f539fe00700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f538c0a3c30 0x7f538c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:08:06.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.476+0000 7f539fe00700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f538c0a3c30 0x7f538c0a60f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f539400b2e0 tx=0x7f5394005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:08:06.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.479+0000 7f53a6000700 1 -- 172.21.15.81:0/1384902164 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5398010040 con 0x7f53a81035e0 2024-04-12T21:08:06.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45800]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:08:06.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45800]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45800]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45800]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45804]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:08:06.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45804]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45804]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45804]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45807]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:08:06.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45807]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45807]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45807]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45810]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:08:06.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45810]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45810]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 sudo[45810]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: osdmap e20: 3 total, 3 up, 3 in 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:06.525 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:06.526 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50410]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50410]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50410]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50410]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50413]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 sudo[50413]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T21:08:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: osdmap e20: 3 total, 3 up, 3 in 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:06.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:06.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:06.724+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f53a8102d40 con 0x7f538c0a3c30 2024-04-12T21:08:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44943]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-12T21:08:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44943]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44943]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44943]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44946]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-12T21:08:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44946]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-12T21:08:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44946]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-12T21:08:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 sudo[44946]: pam_unix(sudo:session): session closed for user root 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: osdmap e20: 3 total, 3 up, 3 in 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:07.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:07.253+0000 7f53a6000700 1 -- 172.21.15.81:0/1384902164 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5398099200 con 0x7f53a81035e0 2024-04-12T21:08:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: osdmap e21: 3 total, 3 up, 3 in 2024-04-12T21:08:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:08:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:08:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:08:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:07 smithi081 ceph-mon[31453]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: osdmap e21: 3 total, 3 up, 3 in 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:08:07.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:08:07.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:08:07.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:07.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:07 smithi153 ceph-mon[36569]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: pgmap v92: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: osdmap e21: 3 total, 3 up, 3 in 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:08:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:08:07.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:07.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:07 smithi104 ceph-mon[36624]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-12T21:08:08.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:08 smithi081 ceph-mon[31453]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:08.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:08 smithi153 ceph-mon[36569]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:08.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:08 smithi104 ceph-mon[36624]: pgmap v94: 1 pgs: 1 creating+peering; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:11.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:11 smithi153 ceph-mon[36569]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:11 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3221643922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]: dispatch 2024-04-12T21:08:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:11 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]: dispatch 2024-04-12T21:08:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:11 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]': finished 2024-04-12T21:08:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:11 smithi153 ceph-mon[36569]: osdmap e22: 4 total, 3 up, 4 in 2024-04-12T21:08:11.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:11 smithi104 ceph-mon[36624]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:11 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3221643922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]: dispatch 2024-04-12T21:08:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:11 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]: dispatch 2024-04-12T21:08:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:11 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]': finished 2024-04-12T21:08:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:11 smithi104 ceph-mon[36624]: osdmap e22: 4 total, 3 up, 4 in 2024-04-12T21:08:11.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:11 smithi081 ceph-mon[31453]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:11 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3221643922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]: dispatch 2024-04-12T21:08:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:11 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]: dispatch 2024-04-12T21:08:11.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:11 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb"}]': finished 2024-04-12T21:08:11.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:11 smithi081 ceph-mon[31453]: osdmap e22: 4 total, 3 up, 4 in 2024-04-12T21:08:11.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:12.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:12 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/14604003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:08:12.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:12 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/14604003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:08:12.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:12 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/14604003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:08:13.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:13 smithi153 ceph-mon[36569]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:13 smithi104 ceph-mon[36624]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:13.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:13 smithi081 ceph-mon[31453]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:15.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:15 smithi153 ceph-mon[36569]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:15.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:15 smithi104 ceph-mon[36624]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:15.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:15 smithi081 ceph-mon[31453]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:17 smithi153 ceph-mon[36569]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:17 smithi104 ceph-mon[36624]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:17.594 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:17 smithi081 ceph-mon[31453]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:19.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:19 smithi081 ceph-mon[31453]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:19.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:19 smithi104 ceph-mon[36624]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:19 smithi153 ceph-mon[36569]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:20.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T21:08:20.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T21:08:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:20.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-12T21:08:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:21.574 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:21 smithi081 ceph-mon[31453]: Deploying daemon osd.3 on smithi081 2024-04-12T21:08:21.574 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:21 smithi081 ceph-mon[31453]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:21.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:21 smithi104 ceph-mon[36624]: Deploying daemon osd.3 on smithi081 2024-04-12T21:08:21.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:21 smithi104 ceph-mon[36624]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:21 smithi153 ceph-mon[36569]: Deploying daemon osd.3 on smithi081 2024-04-12T21:08:21.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:21 smithi153 ceph-mon[36569]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:23.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:23 smithi104 ceph-mon[36624]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:23.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:23 smithi081 ceph-mon[31453]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:23.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:23 smithi153 ceph-mon[36569]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:25 smithi104 ceph-mon[36624]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:25.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:25 smithi081 ceph-mon[31453]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:25 smithi153 ceph-mon[36569]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:27.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:27 smithi104 ceph-mon[36624]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:27.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:27 smithi081 ceph-mon[31453]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:27 smithi153 ceph-mon[36569]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:28.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:28.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:28.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:28.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:28 smithi153 ceph-mon[36569]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:28.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:28.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:28.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:28.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:28 smithi104 ceph-mon[36624]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:28.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:28.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:28.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:28.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:28 smithi081 ceph-mon[31453]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:31.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:31 smithi153 ceph-mon[36569]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:31.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:31.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:31.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:31.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:31.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:31.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:31 smithi104 ceph-mon[36624]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:31.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:31.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:31.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:31.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:31.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:31.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:31 smithi081 ceph-mon[31453]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:31.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:31.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:31.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:31.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:31.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:32.212 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.216+0000 7f53a6000700 1 -- 172.21.15.81:0/1384902164 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f53a8102d40 con 0x7f538c0a3c30 2024-04-12T21:08:32.212 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 3 on host 'smithi081' 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f538c0a3c30 msgr2=0x7f538c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f538c0a3c30 0x7f538c0a60f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f539400b2e0 tx=0x7f5394005fb0 comp rx=0 tx=0).stop 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a81035e0 msgr2=0x7f53a8198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a81035e0 0x7f53a8198d60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f539800ef40 tx=0x7f539800c3a0 comp rx=0 tx=0).stop 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 shutdown_connections 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f538c0a3c30 0x7f538c0a60f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f53a8108840 0x7f53a81a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f53a81035e0 0x7f53a8198d60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 --2- 172.21.15.81:0/1384902164 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f53a8105f10 0x7f53a81992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:32.216 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.219+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 >> 172.21.15.81:0/1384902164 conn(0x7f53a80fcfd0 msgr2=0x7f53a8103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:08:32.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.220+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 shutdown_connections 2024-04-12T21:08:32.217 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:08:32.220+0000 7f53af600700 1 -- 172.21.15.81:0/1384902164 wait complete. 2024-04-12T21:08:32.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:32 smithi081 ceph-mon[31453]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:08:32.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:32.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:32.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:32.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:32 smithi153 ceph-mon[36569]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:08:32.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:32.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:32.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:32.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:32 smithi104 ceph-mon[36624]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-12T21:08:32.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:32.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:32.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:33.006 DEBUG:teuthology.orchestra.run.smithi081:osd.3> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.3.service 2024-04-12T21:08:33.011 INFO:tasks.cephadm:Deploying osd.4 on smithi104 with /dev/vg_nvme/lv_3... 2024-04-12T21:08:33.011 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T21:08:33.256 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.b/config 2024-04-12T21:08:33.499 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:33 smithi081 ceph-mon[31453]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:33.499 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:33 smithi081 ceph-mon[31453]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T21:08:33.500 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:33 smithi081 ceph-mon[31453]: osdmap e23: 4 total, 3 up, 4 in 2024-04-12T21:08:33.500 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:33 smithi081 ceph-mon[31453]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:08:33.500 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:33.500 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 12 21:08:33 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-3[52703]: 2024-04-12T21:08:33.323+0000 7fcc3b800700 -1 osd.3 0 waiting for initial osdmap 2024-04-12T21:08:33.501 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 12 21:08:33 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-3[52703]: 2024-04-12T21:08:33.346+0000 7fcc36600700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:08:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:33 smithi104 ceph-mon[36624]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:33 smithi104 ceph-mon[36624]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T21:08:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:33 smithi104 ceph-mon[36624]: osdmap e23: 4 total, 3 up, 4 in 2024-04-12T21:08:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:33 smithi104 ceph-mon[36624]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:08:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:33 smithi153 ceph-mon[36569]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:33.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:33 smithi153 ceph-mon[36569]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-12T21:08:33.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:33 smithi153 ceph-mon[36569]: osdmap e23: 4 total, 3 up, 4 in 2024-04-12T21:08:33.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:33 smithi153 ceph-mon[36569]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:08:33.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:34.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:34 smithi104 ceph-mon[36624]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:08:34.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:34 smithi104 ceph-mon[36624]: osdmap e24: 4 total, 3 up, 4 in 2024-04-12T21:08:34.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:34.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:34 smithi081 ceph-mon[31453]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:08:34.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:34 smithi081 ceph-mon[31453]: osdmap e24: 4 total, 3 up, 4 in 2024-04-12T21:08:34.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:34 smithi153 ceph-mon[36569]: from='osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:08:34.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:34 smithi153 ceph-mon[36569]: osdmap e24: 4 total, 3 up, 4 in 2024-04-12T21:08:34.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T21:08:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T21:08:35.312 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-12T21:08:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-12T21:08:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114388 s, 917 MB/s 2024-04-12T21:08:35.312 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-12T21:08:35.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:35 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:08:35.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:35 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:08:35.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:35 smithi104 ceph-mon[36624]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:35.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:35.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:35 smithi104 ceph-mon[36624]: osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] boot 2024-04-12T21:08:35.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:35 smithi104 ceph-mon[36624]: osdmap e25: 4 total, 4 up, 4 in 2024-04-12T21:08:35.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:35 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:08:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:35 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:08:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:35 smithi081 ceph-mon[31453]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:35 smithi081 ceph-mon[31453]: osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] boot 2024-04-12T21:08:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:35 smithi081 ceph-mon[31453]: osdmap e25: 4 total, 4 up, 4 in 2024-04-12T21:08:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:35.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:35 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:08:35.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:35 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:08:35.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:35 smithi153 ceph-mon[36569]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-12T21:08:35.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:35.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:35 smithi153 ceph-mon[36569]: osd.3 [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] boot 2024-04-12T21:08:35.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:35 smithi153 ceph-mon[36569]: osdmap e25: 4 total, 4 up, 4 in 2024-04-12T21:08:35.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-12T21:08:35.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-04-12T21:08:36.247 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.b/config 2024-04-12T21:08:36.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:36 smithi104 ceph-mon[36624]: osdmap e26: 4 total, 4 up, 4 in 2024-04-12T21:08:36.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:36 smithi081 ceph-mon[31453]: osdmap e26: 4 total, 4 up, 4 in 2024-04-12T21:08:36.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:36 smithi153 ceph-mon[36569]: osdmap e26: 4 total, 4 up, 4 in 2024-04-12T21:08:37.383 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:37 smithi104 ceph-mon[36624]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:37.383 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:37 smithi104 ceph-mon[36624]: osdmap e27: 4 total, 4 up, 4 in 2024-04-12T21:08:37.383 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:37 smithi104 ceph-mon[36624]: osdmap e28: 4 total, 4 up, 4 in 2024-04-12T21:08:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:37 smithi081 ceph-mon[31453]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:37 smithi081 ceph-mon[31453]: osdmap e27: 4 total, 4 up, 4 in 2024-04-12T21:08:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:37 smithi081 ceph-mon[31453]: osdmap e28: 4 total, 4 up, 4 in 2024-04-12T21:08:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:37 smithi153 ceph-mon[36569]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:37.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:37 smithi153 ceph-mon[36569]: osdmap e27: 4 total, 4 up, 4 in 2024-04-12T21:08:37.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:37 smithi153 ceph-mon[36569]: osdmap e28: 4 total, 4 up, 4 in 2024-04-12T21:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 -- 172.21.15.104:0/2005668774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc14103d10 msgr2=0x7fdc14104170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/2005668774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc14103d10 0x7fdc14104170 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdc080099c0 tx=0x7fdc08009cd0 comp rx=0 tx=0).stop 2024-04-12T21:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 -- 172.21.15.104:0/2005668774 shutdown_connections 2024-04-12T21:08:37.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/2005668774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc141046b0 0x7fdc1410af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/2005668774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc14102b10 0x7fdc14102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/2005668774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc14103d10 0x7fdc14104170 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 -- 172.21.15.104:0/2005668774 >> 172.21.15.104:0/2005668774 conn(0x7fdc140fe0b0 msgr2=0x7fdc141004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 -- 172.21.15.104:0/2005668774 shutdown_connections 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.909+0000 7fdc1a30b700 1 -- 172.21.15.104:0/2005668774 wait complete. 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc1a30b700 1 Processor -- start 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc1a30b700 1 -- start start 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc1a30b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc14102b10 0x7fdc14076800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc1a30b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc14103d10 0x7fdc14074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc1a30b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc141046b0 0x7fdc140753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc1a30b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410e840 con 0x7fdc14103d10 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc1a30b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410e6c0 con 0x7fdc141046b0 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc1a30b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1410e9c0 con 0x7fdc14102b10 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc188a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc141046b0 0x7fdc140753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:08:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc188a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc141046b0 0x7fdc140753e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:36764/0 (socket says 172.21.15.104:36764) 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc188a8700 1 -- 172.21.15.104:0/1419820309 learned_addr learned my addr 172.21.15.104:0/1419820309 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc13fff700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc14102b10 0x7fdc14076800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc188a8700 1 -- 172.21.15.104:0/1419820309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc14102b10 msgr2=0x7fdc14076800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.910+0000 7fdc188a8700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc14102b10 0x7fdc14076800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc188a8700 1 -- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc14103d10 msgr2=0x7fdc14074ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc188a8700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc14103d10 0x7fdc14074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc188a8700 1 -- 172.21.15.104:0/1419820309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc14075cb0 con 0x7fdc141046b0 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc188a8700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc141046b0 0x7fdc140753e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00b970 tx=0x7fdc0c00bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc117fa700 1 -- 172.21.15.104:0/1419820309 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c019070 con 0x7fdc141046b0 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc141a3070 con 0x7fdc141046b0 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc117fa700 1 -- 172.21.15.104:0/1419820309 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0c014440 con 0x7fdc141046b0 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc141a3600 con 0x7fdc141046b0 2024-04-12T21:08:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.911+0000 7fdc117fa700 1 -- 172.21.15.104:0/1419820309 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0075d0 con 0x7fdc141046b0 2024-04-12T21:08:37.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.912+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc140697a0 con 0x7fdc141046b0 2024-04-12T21:08:37.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.913+0000 7fdc117fa700 1 -- 172.21.15.104:0/1419820309 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fdc0c01c070 con 0x7fdc141046b0 2024-04-12T21:08:37.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.913+0000 7fdc117fa700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdbfc0a3c40 0x7fdbfc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:08:37.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.914+0000 7fdc117fa700 1 -- 172.21.15.104:0/1419820309 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fdc0c0ce640 con 0x7fdc141046b0 2024-04-12T21:08:37.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.914+0000 7fdc13fff700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdbfc0a3c40 0x7fdbfc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:08:37.911 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.915+0000 7fdc13fff700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdbfc0a3c40 0x7fdbfc0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc04006fd0 tx=0x7fdc04008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:08:37.914 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:37.918+0000 7fdc117fa700 1 -- 172.21.15.104:0/1419820309 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc0c09c240 con 0x7fdc141046b0 2024-04-12T21:08:38.124 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:38.127+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fdc14062ef0 con 0x7fdbfc0a3c40 2024-04-12T21:08:38.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi081 2024-04-12T21:08:38.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 8544M 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: osdmap e29: 4 total, 4 up, 4 in 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:08:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi081 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 8544M 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:38.752 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:38.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: osdmap e29: 4 total, 4 up, 4 in 2024-04-12T21:08:38.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:08:38.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:08:38.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:38.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi081 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 8544M 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: osdmap e29: 4 total, 4 up, 4 in 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:08:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:39.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:39 smithi104 ceph-mon[36624]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:08:39.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:39 smithi104 ceph-mon[36624]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:39 smithi153 ceph-mon[36569]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:08:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:39 smithi153 ceph-mon[36569]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:39.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:39 smithi081 ceph-mon[31453]: from='client.24233 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:08:39.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:39 smithi081 ceph-mon[31453]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:41 smithi153 ceph-mon[36569]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-12T21:08:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:41 smithi153 ceph-mon[36569]: from='client.? 172.21.15.104:0/798334483' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]: dispatch 2024-04-12T21:08:41.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:41 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]: dispatch 2024-04-12T21:08:41.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:41 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]': finished 2024-04-12T21:08:41.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:41 smithi153 ceph-mon[36569]: osdmap e30: 5 total, 4 up, 5 in 2024-04-12T21:08:41.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:08:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:41 smithi104 ceph-mon[36624]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-12T21:08:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:41 smithi104 ceph-mon[36624]: from='client.? 172.21.15.104:0/798334483' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]: dispatch 2024-04-12T21:08:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:41 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]: dispatch 2024-04-12T21:08:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:41 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]': finished 2024-04-12T21:08:41.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:41 smithi104 ceph-mon[36624]: osdmap e30: 5 total, 4 up, 5 in 2024-04-12T21:08:41.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:08:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:41 smithi081 ceph-mon[31453]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-12T21:08:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:41 smithi081 ceph-mon[31453]: from='client.? 172.21.15.104:0/798334483' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]: dispatch 2024-04-12T21:08:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:41 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]: dispatch 2024-04-12T21:08:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:41 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2eb029c7-091e-43bb-8085-4f736bcf6a62"}]': finished 2024-04-12T21:08:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:41 smithi081 ceph-mon[31453]: osdmap e30: 5 total, 4 up, 5 in 2024-04-12T21:08:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:08:42.729 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:42 smithi104 ceph-mon[36624]: from='client.? 172.21.15.104:0/1317301206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:08:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:42 smithi153 ceph-mon[36569]: from='client.? 172.21.15.104:0/1317301206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:08:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:42 smithi081 ceph-mon[31453]: from='client.? 172.21.15.104:0/1317301206' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:08:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:43 smithi153 ceph-mon[36569]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-12T21:08:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:43 smithi104 ceph-mon[36624]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-12T21:08:43.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:43 smithi081 ceph-mon[31453]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-04-12T21:08:45.726 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:45 smithi104 ceph-mon[36624]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-12T21:08:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:45 smithi153 ceph-mon[36569]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-12T21:08:45.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:45 smithi081 ceph-mon[31453]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-12T21:08:47.737 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:47 smithi104 ceph-mon[36624]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-12T21:08:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:47 smithi153 ceph-mon[36569]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-12T21:08:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:47 smithi081 ceph-mon[31453]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-12T21:08:48.759 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T21:08:48.760 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T21:08:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-12T21:08:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:08:49.791 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:49 smithi104 ceph-mon[36624]: Deploying daemon osd.4 on smithi104 2024-04-12T21:08:49.791 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:49 smithi104 ceph-mon[36624]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:08:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:49 smithi153 ceph-mon[36569]: Deploying daemon osd.4 on smithi104 2024-04-12T21:08:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:49 smithi153 ceph-mon[36569]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:08:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:49 smithi081 ceph-mon[31453]: Deploying daemon osd.4 on smithi104 2024-04-12T21:08:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:49 smithi081 ceph-mon[31453]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:08:50.792 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:50 smithi104 ceph-mon[36624]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:50.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:50 smithi153 ceph-mon[36569]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:50.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:50 smithi081 ceph-mon[31453]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:53.532 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:53 smithi104 ceph-mon[36624]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:53 smithi153 ceph-mon[36569]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:53.606 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:53 smithi081 ceph-mon[31453]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:55.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:55.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:55.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:55.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:54 smithi153 ceph-mon[36569]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:55.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:55.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:55.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:55.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:54 smithi104 ceph-mon[36624]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:55.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:55.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:08:55.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:55.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:54 smithi081 ceph-mon[31453]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:57.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.321+0000 7fdc117fa700 1 -- 172.21.15.104:0/1419820309 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fdc14062ef0 con 0x7fdbfc0a3c40 2024-04-12T21:08:57.318 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 4 on host 'smithi104' 2024-04-12T21:08:57.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdbfc0a3c40 msgr2=0x7fdbfc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:57.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdbfc0a3c40 0x7fdbfc0a6100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdc04006fd0 tx=0x7fdc04008040 comp rx=0 tx=0).stop 2024-04-12T21:08:57.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc141046b0 msgr2=0x7fdc140753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:08:57.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc141046b0 0x7fdc140753e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00b970 tx=0x7fdc0c00bd30 comp rx=0 tx=0).stop 2024-04-12T21:08:57.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 shutdown_connections 2024-04-12T21:08:57.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdbfc0a3c40 0x7fdbfc0a6100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:57.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdc141046b0 0x7fdc140753e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:57.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdc14102b10 0x7fdc14076800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:57.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 --2- 172.21.15.104:0/1419820309 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdc14103d10 0x7fdc14074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:08:57.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 >> 172.21.15.104:0/1419820309 conn(0x7fdc140fe0b0 msgr2=0x7fdc14107120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:08:57.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 shutdown_connections 2024-04-12T21:08:57.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:08:57.323+0000 7fdc1a30b700 1 -- 172.21.15.104:0/1419820309 wait complete. 2024-04-12T21:08:57.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:57 smithi153 ceph-mon[36569]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:57.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:57.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:57 smithi104 ceph-mon[36624]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:57.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:57 smithi081 ceph-mon[31453]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:57.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:57.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:57.962 DEBUG:teuthology.orchestra.run.smithi104:osd.4> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.4.service 2024-04-12T21:08:57.965 INFO:tasks.cephadm:Deploying osd.5 on smithi153 with /dev/vg_nvme/lv_3... 2024-04-12T21:08:57.965 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-12T21:08:58.211 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:08:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:58 smithi104 ceph-mon[36624]: from='osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:08:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:58 smithi104 ceph-mon[36624]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:08:58.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:58.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:58.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:58 smithi153 ceph-mon[36569]: from='osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:08:58.615 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:58 smithi153 ceph-mon[36569]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:08:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:08:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:58 smithi081 ceph-mon[31453]: from='osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:08:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:58 smithi081 ceph-mon[31453]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-12T21:08:59.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:59 smithi104 ceph-mon[36624]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:59.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:59 smithi104 ceph-mon[36624]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T21:08:59.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:59 smithi104 ceph-mon[36624]: from='osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:08:59.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:59 smithi104 ceph-mon[36624]: osdmap e31: 5 total, 4 up, 5 in 2024-04-12T21:08:59.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:08:59.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:08:59 smithi104 ceph-mon[36624]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:08:59.636 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:59 smithi153 ceph-mon[36569]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:59.636 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:59 smithi153 ceph-mon[36569]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T21:08:59.636 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:59 smithi153 ceph-mon[36569]: from='osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:08:59.636 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:59 smithi153 ceph-mon[36569]: osdmap e31: 5 total, 4 up, 5 in 2024-04-12T21:08:59.636 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:08:59.636 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:08:59 smithi153 ceph-mon[36569]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:08:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:59 smithi081 ceph-mon[31453]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:08:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:59 smithi081 ceph-mon[31453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-12T21:08:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:59 smithi081 ceph-mon[31453]: from='osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:08:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:59 smithi081 ceph-mon[31453]: osdmap e31: 5 total, 4 up, 5 in 2024-04-12T21:08:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:08:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:08:59 smithi081 ceph-mon[31453]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:09:00.289 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-12T21:09:00.289 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-12T21:09:00.289 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-12T21:09:00.289 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-12T21:09:00.289 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115185 s, 910 MB/s 2024-04-12T21:09:00.290 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-12T21:09:00.382 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 12 21:09:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-4[47770]: 2024-04-12T21:09:00.121+0000 7f301e600700 -1 osd.4 0 waiting for initial osdmap 2024-04-12T21:09:00.382 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 12 21:09:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-4[47770]: 2024-04-12T21:09:00.128+0000 7f30163eb700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:09:00.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:00 smithi153 ceph-mon[36569]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:09:00.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:00 smithi153 ceph-mon[36569]: osdmap e32: 5 total, 4 up, 5 in 2024-04-12T21:09:00.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:00.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:00.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:00 smithi153 ceph-mon[36569]: from='osd.4 ' entity='osd.4' 2024-04-12T21:09:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:00 smithi081 ceph-mon[31453]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:09:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:00 smithi081 ceph-mon[31453]: osdmap e32: 5 total, 4 up, 5 in 2024-04-12T21:09:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:00 smithi081 ceph-mon[31453]: from='osd.4 ' entity='osd.4' 2024-04-12T21:09:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:00 smithi104 ceph-mon[36624]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:09:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:00 smithi104 ceph-mon[36624]: osdmap e32: 5 total, 4 up, 5 in 2024-04-12T21:09:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:00 smithi104 ceph-mon[36624]: from='osd.4 ' entity='osd.4' 2024-04-12T21:09:00.903 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-04-12T21:09:01.155 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] boot 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: osdmap e33: 5 total, 5 up, 5 in 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:09:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] boot 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: osdmap e33: 5 total, 5 up, 5 in 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:01.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:01.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:01.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:09:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:09:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-12T21:09:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: osd.4 [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] boot 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: osdmap e33: 5 total, 5 up, 5 in 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:01.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:02 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi104 2024-04-12T21:09:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:02 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 8540M 2024-04-12T21:09:02.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:02 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi104 2024-04-12T21:09:02.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:02 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 8540M 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.775+0000 7f6468066700 1 -- 172.21.15.153:0/2977094625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460101a10 msgr2=0x7f6460101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.775+0000 7f6468066700 1 --2- 172.21.15.153:0/2977094625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460101a10 0x7f6460101e30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f645c009990 tx=0x7f645c009ca0 comp rx=0 tx=0).stop 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.775+0000 7f6468066700 1 -- 172.21.15.153:0/2977094625 shutdown_connections 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.775+0000 7f6468066700 1 --2- 172.21.15.153:0/2977094625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64601035b0 0x7f6460109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.775+0000 7f6468066700 1 --2- 172.21.15.153:0/2977094625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6460101a10 0x7f6460101e30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.775+0000 7f6468066700 1 --2- 172.21.15.153:0/2977094625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6460102c10 0x7f6460103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.775+0000 7f6468066700 1 -- 172.21.15.153:0/2977094625 >> 172.21.15.153:0/2977094625 conn(0x7f64600fcfb0 msgr2=0x7f64600ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 -- 172.21.15.153:0/2977094625 shutdown_connections 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 -- 172.21.15.153:0/2977094625 wait complete. 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 Processor -- start 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 -- start start 2024-04-12T21:09:02.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6460101a10 0x7f6460197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6460102c10 0x7f6460198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601035b0 0x7f646019f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f646010d670 con 0x7f6460101a10 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f646010d4f0 con 0x7f6460102c10 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.776+0000 7f6468066700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f646010d7f0 con 0x7f64601035b0 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601035b0 0x7f646019f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601035b0 0x7f646019f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45560/0 (socket says 172.21.15.153:45560) 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 -- 172.21.15.153:0/2183063339 learned_addr learned my addr 172.21.15.153:0/2183063339 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6465601700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6460102c10 0x7f6460198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 -- 172.21.15.153:0/2183063339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6460102c10 msgr2=0x7f6460198290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6460102c10 0x7f6460198290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 -- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6460101a10 msgr2=0x7f6460197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:02.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6460101a10 0x7f6460197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 -- 172.21.15.153:0/2183063339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646019f950 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6466603700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601035b0 0x7f646019f280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f645800ef30 tx=0x7f645800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6456ffd700 1 -- 172.21.15.153:0/2183063339 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64580092e0 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646019fbe0 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6456ffd700 1 -- 172.21.15.153:0/2183063339 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6458004bb0 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6456ffd700 1 -- 172.21.15.153:0/2183063339 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6458004280 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.777+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64601a0120 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.778+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64600697a0 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.779+0000 7f6456ffd700 1 -- 172.21.15.153:0/2183063339 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f645801b030 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.780+0000 7f6456ffd700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f644c0a3c40 0x7f644c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.780+0000 7f6456ffd700 1 -- 172.21.15.153:0/2183063339 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f64580cee60 con 0x7f64601035b0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.780+0000 7f6465e02700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f644c0a3c40 0x7f644c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:09:02.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.781+0000 7f6465e02700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f644c0a3c40 0x7f644c0a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f645c0098e0 tx=0x7f645c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:09:02.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.784+0000 7f6456ffd700 1 -- 172.21.15.153:0/2183063339 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6458022020 con 0x7f64601035b0 2024-04-12T21:09:02.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:02 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi104 2024-04-12T21:09:02.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:02 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 8540M 2024-04-12T21:09:02.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:02.992+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f6460062ef0 con 0x7f644c0a3c40 2024-04-12T21:09:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:03 smithi081 ceph-mon[31453]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:03 smithi081 ceph-mon[31453]: osdmap e34: 5 total, 5 up, 5 in 2024-04-12T21:09:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:09:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:09:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:03.719 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:03 smithi153 ceph-mon[36569]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:03.719 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:03 smithi153 ceph-mon[36569]: osdmap e34: 5 total, 5 up, 5 in 2024-04-12T21:09:03.719 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:09:03.719 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:09:03.719 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:03 smithi104 ceph-mon[36624]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:03 smithi104 ceph-mon[36624]: osdmap e34: 5 total, 5 up, 5 in 2024-04-12T21:09:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:09:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:09:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:04.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:04 smithi153 ceph-mon[36569]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:09:04.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:04 smithi081 ceph-mon[31453]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:09:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:04 smithi104 ceph-mon[36624]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:09:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:05 smithi153 ceph-mon[36569]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:05 smithi104 ceph-mon[36624]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:05 smithi081 ceph-mon[31453]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:06.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:06 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/2464761049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]: dispatch 2024-04-12T21:09:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:06 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]: dispatch 2024-04-12T21:09:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:06 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]': finished 2024-04-12T21:09:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:06 smithi153 ceph-mon[36569]: osdmap e35: 6 total, 5 up, 6 in 2024-04-12T21:09:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:06 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/3579148776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:09:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:06 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/2464761049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]: dispatch 2024-04-12T21:09:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:06 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]: dispatch 2024-04-12T21:09:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:06 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]': finished 2024-04-12T21:09:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:06 smithi104 ceph-mon[36624]: osdmap e35: 6 total, 5 up, 6 in 2024-04-12T21:09:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:06 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/3579148776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:09:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:06 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/2464761049' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]: dispatch 2024-04-12T21:09:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:06 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]: dispatch 2024-04-12T21:09:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:06 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fe772d4-5800-4a88-a101-1dbf64b4abf6"}]': finished 2024-04-12T21:09:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:06 smithi081 ceph-mon[31453]: osdmap e35: 6 total, 5 up, 6 in 2024-04-12T21:09:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:06 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/3579148776' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:09:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:07 smithi153 ceph-mon[36569]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:07 smithi104 ceph-mon[36624]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:07 smithi081 ceph-mon[31453]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:09.716 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:09 smithi153 ceph-mon[36569]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:09 smithi104 ceph-mon[36624]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:09.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:09 smithi081 ceph-mon[31453]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:11.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:11 smithi153 ceph-mon[36569]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:11 smithi104 ceph-mon[36624]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:11.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:11 smithi081 ceph-mon[31453]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:13.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:13 smithi153 ceph-mon[36569]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:13.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T21:09:13.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:13 smithi104 ceph-mon[36624]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T21:09:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:13 smithi081 ceph-mon[31453]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-12T21:09:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:14.501 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:14 smithi153 ceph-mon[36569]: Deploying daemon osd.5 on smithi153 2024-04-12T21:09:14.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:14 smithi104 ceph-mon[36624]: Deploying daemon osd.5 on smithi153 2024-04-12T21:09:14.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:14 smithi081 ceph-mon[31453]: Deploying daemon osd.5 on smithi153 2024-04-12T21:09:15.758 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:15 smithi153 ceph-mon[36569]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:15 smithi104 ceph-mon[36624]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:15 smithi081 ceph-mon[31453]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:16.770 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:16 smithi153 ceph-mon[36569]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:16.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:16 smithi104 ceph-mon[36624]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:16.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:16 smithi081 ceph-mon[31453]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:19 smithi153 ceph-mon[36569]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:19.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:19.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:19.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:19.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:19 smithi104 ceph-mon[36624]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:19.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:19.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:19.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:19 smithi081 ceph-mon[31453]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:20.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:20 smithi153 ceph-mon[36569]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:20.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:20 smithi104 ceph-mon[36624]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:20.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:20 smithi081 ceph-mon[31453]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:22.323 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 5 on host 'smithi153' 2024-04-12T21:09:22.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.326+0000 7f6456ffd700 1 -- 172.21.15.153:0/2183063339 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6460062ef0 con 0x7f644c0a3c40 2024-04-12T21:09:22.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f644c0a3c40 msgr2=0x7f644c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:22.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f644c0a3c40 0x7f644c0a6100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f645c0098e0 tx=0x7f645c005fb0 comp rx=0 tx=0).stop 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601035b0 msgr2=0x7f646019f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601035b0 0x7f646019f280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f645800ef30 tx=0x7f645800c3a0 comp rx=0 tx=0).stop 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 shutdown_connections 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f644c0a3c40 0x7f644c0a6100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601035b0 0x7f646019f280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6460101a10 0x7f6460197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 --2- 172.21.15.153:0/2183063339 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6460102c10 0x7f6460198290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 >> 172.21.15.153:0/2183063339 conn(0x7f64600fcfb0 msgr2=0x7f6460105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 shutdown_connections 2024-04-12T21:09:22.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:09:22.329+0000 7f6468066700 1 -- 172.21.15.153:0/2183063339 wait complete. 2024-04-12T21:09:22.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:09:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:22 smithi153 ceph-mon[36569]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:09:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:09:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:22 smithi104 ceph-mon[36624]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:09:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:09:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:22 smithi081 ceph-mon[31453]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-12T21:09:23.032 DEBUG:teuthology.orchestra.run.smithi153:osd.5> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.5.service 2024-04-12T21:09:23.035 INFO:tasks.cephadm:Deploying osd.6 on smithi081 with /dev/vg_nvme/lv_2... 2024-04-12T21:09:23.035 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T21:09:23.494 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:09:23.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:23 smithi153 ceph-mon[36569]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:23.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:23 smithi153 ceph-mon[36569]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T21:09:23.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:23 smithi153 ceph-mon[36569]: from='osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:09:23.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:23 smithi153 ceph-mon[36569]: osdmap e36: 6 total, 5 up, 6 in 2024-04-12T21:09:23.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:23.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:23 smithi153 ceph-mon[36569]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:09:23.785 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:23 smithi081 ceph-mon[31453]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:23.785 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:23 smithi081 ceph-mon[31453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T21:09:23.785 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:23 smithi081 ceph-mon[31453]: from='osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:09:23.785 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:23 smithi081 ceph-mon[31453]: osdmap e36: 6 total, 5 up, 6 in 2024-04-12T21:09:23.785 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:23.785 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:23 smithi081 ceph-mon[31453]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:09:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:23 smithi104 ceph-mon[36624]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:23 smithi104 ceph-mon[36624]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-12T21:09:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:23 smithi104 ceph-mon[36624]: from='osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:09:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:23 smithi104 ceph-mon[36624]: osdmap e36: 6 total, 5 up, 6 in 2024-04-12T21:09:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:23 smithi104 ceph-mon[36624]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:09:25.296 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 12 21:09:25 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-5[48702]: 2024-04-12T21:09:25.016+0000 7f6a873d2700 -1 osd.5 0 waiting for initial osdmap 2024-04-12T21:09:25.296 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 12 21:09:25 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-5[48702]: 2024-04-12T21:09:25.023+0000 7f6a7f1bd700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:09:25.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:25 smithi081 ceph-mon[31453]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:25.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:25 smithi081 ceph-mon[31453]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:09:25.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:25 smithi081 ceph-mon[31453]: osdmap e37: 6 total, 5 up, 6 in 2024-04-12T21:09:25.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:25.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:25.491 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:25 smithi081 ceph-mon[31453]: from='osd.5 ' entity='osd.5' 2024-04-12T21:09:25.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:25 smithi153 ceph-mon[36569]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:25.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:25 smithi153 ceph-mon[36569]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:09:25.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:25 smithi153 ceph-mon[36569]: osdmap e37: 6 total, 5 up, 6 in 2024-04-12T21:09:25.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:25.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:25.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:25 smithi153 ceph-mon[36569]: from='osd.5 ' entity='osd.5' 2024-04-12T21:09:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:25 smithi104 ceph-mon[36624]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-04-12T21:09:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:25 smithi104 ceph-mon[36624]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:09:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:25 smithi104 ceph-mon[36624]: osdmap e37: 6 total, 5 up, 6 in 2024-04-12T21:09:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:25 smithi104 ceph-mon[36624]: from='osd.5 ' entity='osd.5' 2024-04-12T21:09:26.458 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T21:09:26.458 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T21:09:26.458 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-12T21:09:26.458 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-12T21:09:26.458 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0346188 s, 303 MB/s 2024-04-12T21:09:26.459 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-12T21:09:26.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] boot 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: osdmap e38: 6 total, 6 up, 6 in 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:26.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] boot 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: osdmap e38: 6 total, 6 up, 6 in 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:26.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:26.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:26.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:26.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:09:26.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:09:26.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: osd.5 [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] boot 2024-04-12T21:09:26.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: osdmap e38: 6 total, 6 up, 6 in 2024-04-12T21:09:26.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-12T21:09:26.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:26.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:09:26.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:26.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:26.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:26.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:27.373 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi081:vg_nvme/lv_2 2024-04-12T21:09:27.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:27 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi153 2024-04-12T21:09:27.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:27 smithi104 ceph-mon[36624]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:27.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:27 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 8540M 2024-04-12T21:09:27.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:27 smithi104 ceph-mon[36624]: osdmap e39: 6 total, 6 up, 6 in 2024-04-12T21:09:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:27 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi153 2024-04-12T21:09:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:27 smithi081 ceph-mon[31453]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:27 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 8540M 2024-04-12T21:09:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:27 smithi081 ceph-mon[31453]: osdmap e39: 6 total, 6 up, 6 in 2024-04-12T21:09:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:27 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi153 2024-04-12T21:09:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:27 smithi153 ceph-mon[36569]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:27 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 8540M 2024-04-12T21:09:27.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:27 smithi153 ceph-mon[36569]: osdmap e39: 6 total, 6 up, 6 in 2024-04-12T21:09:27.813 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:09:28.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:28 smithi081 ceph-mon[31453]: osdmap e40: 6 total, 6 up, 6 in 2024-04-12T21:09:28.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:28 smithi153 ceph-mon[36569]: osdmap e40: 6 total, 6 up, 6 in 2024-04-12T21:09:28.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:28 smithi104 ceph-mon[36624]: osdmap e40: 6 total, 6 up, 6 in 2024-04-12T21:09:29.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:29 smithi081 ceph-mon[31453]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:29.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:29 smithi081 ceph-mon[31453]: osdmap e41: 6 total, 6 up, 6 in 2024-04-12T21:09:29.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:29 smithi153 ceph-mon[36569]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:29.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:29 smithi153 ceph-mon[36569]: osdmap e41: 6 total, 6 up, 6 in 2024-04-12T21:09:29.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.825+0000 7f6997800700 1 -- 172.21.15.81:0/830299133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69901035a0 msgr2=0x7f6990109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:29.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.825+0000 7f6997800700 1 --2- 172.21.15.81:0/830299133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69901035a0 0x7f6990109e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f698800b0a0 tx=0x7f698800b3b0 comp rx=0 tx=0).stop 2024-04-12T21:09:29.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.826+0000 7f6997800700 1 -- 172.21.15.81:0/830299133 shutdown_connections 2024-04-12T21:09:29.822 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.826+0000 7f6997800700 1 --2- 172.21.15.81:0/830299133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69901035a0 0x7f6990109e50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:29.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.826+0000 7f6997800700 1 --2- 172.21.15.81:0/830299133 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6990101a00 0x7f6990101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:29.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.826+0000 7f6997800700 1 --2- 172.21.15.81:0/830299133 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6990102c00 0x7f6990103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:29.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.826+0000 7f6997800700 1 -- 172.21.15.81:0/830299133 >> 172.21.15.81:0/830299133 conn(0x7f69900fcfa0 msgr2=0x7f69900ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:09:29.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.826+0000 7f6997800700 1 -- 172.21.15.81:0/830299133 shutdown_connections 2024-04-12T21:09:29.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.826+0000 7f6997800700 1 -- 172.21.15.81:0/830299133 wait complete. 2024-04-12T21:09:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.828+0000 7f6997800700 1 Processor -- start 2024-04-12T21:09:29.824 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.828+0000 7f6997800700 1 -- start start 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6997800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6990101a00 0x7f6990198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6997800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6990102c00 0x7f69901991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6997800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69901035a0 0x7f69901a0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6997800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f699010d5d0 con 0x7f6990101a00 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6997800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f699010d450 con 0x7f69901035a0 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6997800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f699010d750 con 0x7f6990102c00 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6994e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6990101a00 0x7f6990198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6995800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69901035a0 0x7f69901a0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6995800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69901035a0 0x7f69901a0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:35558/0 (socket says 172.21.15.81:35558) 2024-04-12T21:09:29.825 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6994e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6990101a00 0x7f6990198c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52598/0 (socket says 172.21.15.81:52598) 2024-04-12T21:09:29.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6995800700 1 -- 172.21.15.81:0/280665669 learned_addr learned my addr 172.21.15.81:0/280665669 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:09:29.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f698fe00700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6990102c00 0x7f69901991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6995800700 1 -- 172.21.15.81:0/280665669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6990102c00 msgr2=0x7f69901991c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6995800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6990102c00 0x7f69901991c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6995800700 1 -- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6990101a00 msgr2=0x7f6990198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.829+0000 7f6995800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6990101a00 0x7f6990198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.830+0000 7f6995800700 1 -- 172.21.15.81:0/280665669 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69901a0900 con 0x7f69901035a0 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.830+0000 7f6995800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69901035a0 0x7f69901a0230 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f698800b780 tx=0x7f6988004f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.830+0000 7f698d600700 1 -- 172.21.15.81:0/280665669 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988021070 con 0x7f69901035a0 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.830+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6990103e20 con 0x7f69901035a0 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.830+0000 7f698d600700 1 -- 172.21.15.81:0/280665669 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6988007b80 con 0x7f69901035a0 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.830+0000 7f698d600700 1 -- 172.21.15.81:0/280665669 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988005280 con 0x7f69901035a0 2024-04-12T21:09:29.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.830+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6990104360 con 0x7f69901035a0 2024-04-12T21:09:29.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:29 smithi104 ceph-mon[36624]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:29.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:29 smithi104 ceph-mon[36624]: osdmap e41: 6 total, 6 up, 6 in 2024-04-12T21:09:29.828 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.831+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69900697a0 con 0x7f69901035a0 2024-04-12T21:09:29.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.833+0000 7f698d600700 1 -- 172.21.15.81:0/280665669 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6988024070 con 0x7f69901035a0 2024-04-12T21:09:29.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.835+0000 7f698d600700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69780a3c40 0x7f69780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:09:29.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.835+0000 7f698d600700 1 -- 172.21.15.81:0/280665669 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f698800f020 con 0x7f69901035a0 2024-04-12T21:09:29.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.835+0000 7f6994e00700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69780a3c40 0x7f69780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:09:29.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.835+0000 7f6994e00700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69780a3c40 0x7f69780a6100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6990199b60 tx=0x7f6984009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:09:29.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:29.839+0000 7f698d600700 1 -- 172.21.15.81:0/280665669 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f698802b020 con 0x7f69901035a0 2024-04-12T21:09:30.073 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:30.077+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f69900ff110 con 0x7f69780a3c40 2024-04-12T21:09:30.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:30 smithi081 ceph-mon[31453]: osdmap e42: 6 total, 6 up, 6 in 2024-04-12T21:09:30.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:09:30.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:09:30.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:30.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:30 smithi153 ceph-mon[36569]: osdmap e42: 6 total, 6 up, 6 in 2024-04-12T21:09:30.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:09:30.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:09:30.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:30.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:30 smithi104 ceph-mon[36624]: osdmap e42: 6 total, 6 up, 6 in 2024-04-12T21:09:30.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:09:30.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:09:30.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:31.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:31 smithi081 ceph-mon[31453]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:09:31.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:31 smithi081 ceph-mon[31453]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-12T21:09:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:31 smithi153 ceph-mon[36569]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:09:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:31 smithi153 ceph-mon[36569]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-12T21:09:31.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:31 smithi104 ceph-mon[36624]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:09:31.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:31 smithi104 ceph-mon[36624]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 182 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-12T21:09:33.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:33 smithi081 ceph-mon[31453]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-12T21:09:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:33 smithi153 ceph-mon[36569]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-12T21:09:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:33 smithi104 ceph-mon[36624]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-12T21:09:34.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:34 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3545233815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]: dispatch 2024-04-12T21:09:34.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:34 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]: dispatch 2024-04-12T21:09:34.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:34 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]': finished 2024-04-12T21:09:34.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:34 smithi081 ceph-mon[31453]: osdmap e43: 7 total, 6 up, 7 in 2024-04-12T21:09:34.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:34 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3545233815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]: dispatch 2024-04-12T21:09:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:34 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]: dispatch 2024-04-12T21:09:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:34 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]': finished 2024-04-12T21:09:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:34 smithi153 ceph-mon[36569]: osdmap e43: 7 total, 6 up, 7 in 2024-04-12T21:09:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:34.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:34 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3545233815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]: dispatch 2024-04-12T21:09:34.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:34 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]: dispatch 2024-04-12T21:09:34.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:34 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77e17684-22e7-4e11-b27f-ffc4703182ba"}]': finished 2024-04-12T21:09:34.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:34 smithi104 ceph-mon[36624]: osdmap e43: 7 total, 6 up, 7 in 2024-04-12T21:09:34.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:35.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:35 smithi081 ceph-mon[31453]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-12T21:09:35.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:35 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2414103171' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:09:35.703 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:35 smithi153 ceph-mon[36569]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-12T21:09:35.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:35 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2414103171' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:09:35.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:35 smithi104 ceph-mon[36624]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-12T21:09:35.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:35 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2414103171' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:09:37.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:37 smithi081 ceph-mon[31453]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-12T21:09:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:37 smithi153 ceph-mon[36569]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-12T21:09:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:37 smithi104 ceph-mon[36624]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-12T21:09:39.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:39 smithi081 ceph-mon[31453]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:09:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:39 smithi153 ceph-mon[36569]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:09:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:39 smithi104 ceph-mon[36624]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-12T21:09:41.480 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:41 smithi081 ceph-mon[31453]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:41 smithi153 ceph-mon[36569]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:41 smithi104 ceph-mon[36624]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:43.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:43 smithi081 ceph-mon[31453]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:43.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T21:09:43.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:43 smithi153 ceph-mon[36569]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T21:09:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:43 smithi104 ceph-mon[36624]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-12T21:09:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:44.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:44 smithi081 ceph-mon[31453]: Deploying daemon osd.6 on smithi081 2024-04-12T21:09:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:44 smithi153 ceph-mon[36569]: Deploying daemon osd.6 on smithi081 2024-04-12T21:09:44.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:44 smithi104 ceph-mon[36624]: Deploying daemon osd.6 on smithi081 2024-04-12T21:09:45.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:45 smithi081 ceph-mon[31453]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:45 smithi153 ceph-mon[36569]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:45.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:45 smithi104 ceph-mon[36624]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:47 smithi153 ceph-mon[36569]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:47 smithi104 ceph-mon[36624]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:47.828 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:47 smithi081 ceph-mon[31453]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:49.678 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:49 smithi081 ceph-mon[31453]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:49 smithi153 ceph-mon[36569]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:49 smithi104 ceph-mon[36624]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:51.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:51 smithi081 ceph-mon[31453]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:51 smithi153 ceph-mon[36569]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:51 smithi104 ceph-mon[36624]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:52.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:52.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:52.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:52.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:52 smithi081 ceph-mon[31453]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:52 smithi153 ceph-mon[36569]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:52.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:52.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:52.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:52.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:52 smithi104 ceph-mon[36624]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:55.572 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:55 smithi081 ceph-mon[31453]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:55.572 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:55.572 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:55 smithi104 ceph-mon[36624]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:55.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:55 smithi153 ceph-mon[36569]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:55.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:55.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:56.468 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:56 smithi081 ceph-mon[31453]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:09:56.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.533+0000 7f698d600700 1 -- 172.21.15.81:0/280665669 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f69900ff110 con 0x7f69780a3c40 2024-04-12T21:09:56.531 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 6 on host 'smithi081' 2024-04-12T21:09:56.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69780a3c40 msgr2=0x7f69780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:56.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69780a3c40 0x7f69780a6100 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6990199b60 tx=0x7f6984009260 comp rx=0 tx=0).stop 2024-04-12T21:09:56.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69901035a0 msgr2=0x7f69901a0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:09:56.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69901035a0 0x7f69901a0230 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f698800b780 tx=0x7f6988004f40 comp rx=0 tx=0).stop 2024-04-12T21:09:56.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 shutdown_connections 2024-04-12T21:09:56.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f69780a3c40 0x7f69780a6100 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:56.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69901035a0 0x7f69901a0230 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:56.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6990101a00 0x7f6990198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:56.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 --2- 172.21.15.81:0/280665669 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6990102c00 0x7f69901991c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:09:56.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 >> 172.21.15.81:0/280665669 conn(0x7f69900fcfa0 msgr2=0x7f69900fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:09:56.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 shutdown_connections 2024-04-12T21:09:56.534 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:09:56.536+0000 7f6997800700 1 -- 172.21.15.81:0/280665669 wait complete. 2024-04-12T21:09:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:56 smithi104 ceph-mon[36624]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:09:56.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:56 smithi153 ceph-mon[36569]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-12T21:09:57.314 DEBUG:teuthology.orchestra.run.smithi081:osd.6> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.6.service 2024-04-12T21:09:57.319 INFO:tasks.cephadm:Deploying osd.7 on smithi104 with /dev/vg_nvme/lv_2... 2024-04-12T21:09:57.319 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T21:09:57.563 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.b/config 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: osdmap e44: 7 total, 6 up, 7 in 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:57.626 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: osdmap e44: 7 total, 6 up, 7 in 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.627 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: osdmap e44: 7 total, 6 up, 7 in 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:09:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:09:58.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:58 smithi104 ceph-mon[36624]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:09:58.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:58 smithi104 ceph-mon[36624]: osdmap e45: 7 total, 6 up, 7 in 2024-04-12T21:09:58.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:58.648 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:58 smithi081 ceph-mon[31453]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:09:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:58 smithi081 ceph-mon[31453]: osdmap e45: 7 total, 6 up, 7 in 2024-04-12T21:09:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:58 smithi153 ceph-mon[36569]: from='osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:09:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:58 smithi153 ceph-mon[36569]: osdmap e45: 7 total, 6 up, 7 in 2024-04-12T21:09:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:58.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:59.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:59 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:09:59.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:59 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:09:59.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:59 smithi081 ceph-mon[31453]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:59.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:59 smithi081 ceph-mon[31453]: osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] boot 2024-04-12T21:09:59.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:59 smithi081 ceph-mon[31453]: osdmap e46: 7 total, 7 up, 7 in 2024-04-12T21:09:59.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:09:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:59.504 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T21:09:59.504 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T21:09:59.505 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-12T21:09:59.505 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-12T21:09:59.505 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114836 s, 913 MB/s 2024-04-12T21:09:59.505 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-12T21:09:59.600 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:59 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:09:59.600 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:59 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:09:59.600 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:59 smithi104 ceph-mon[36624]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:59.601 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:59 smithi104 ceph-mon[36624]: osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] boot 2024-04-12T21:09:59.601 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:59 smithi104 ceph-mon[36624]: osdmap e46: 7 total, 7 up, 7 in 2024-04-12T21:09:59.601 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:09:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:09:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:59 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:09:59.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:59 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:09:59.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:59 smithi153 ceph-mon[36569]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 183 MiB used, 536 GiB / 536 GiB avail 2024-04-12T21:09:59.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:59 smithi153 ceph-mon[36569]: osd.6 [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] boot 2024-04-12T21:09:59.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:59 smithi153 ceph-mon[36569]: osdmap e46: 7 total, 7 up, 7 in 2024-04-12T21:09:59.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:09:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-12T21:10:00.075 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-04-12T21:10:00.321 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.b/config 2024-04-12T21:10:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:00 smithi081 ceph-mon[31453]: osdmap e47: 7 total, 7 up, 7 in 2024-04-12T21:10:00.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:00 smithi081 ceph-mon[31453]: overall HEALTH_OK 2024-04-12T21:10:00.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:00 smithi104 ceph-mon[36624]: osdmap e47: 7 total, 7 up, 7 in 2024-04-12T21:10:00.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:00 smithi104 ceph-mon[36624]: overall HEALTH_OK 2024-04-12T21:10:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:00 smithi153 ceph-mon[36569]: osdmap e47: 7 total, 7 up, 7 in 2024-04-12T21:10:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:00 smithi153 ceph-mon[36569]: overall HEALTH_OK 2024-04-12T21:10:01.402 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:01 smithi104 ceph-mon[36624]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:01 smithi081 ceph-mon[31453]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:01 smithi153 ceph-mon[36569]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3299863334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc102c10 msgr2=0x7fa9dc103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3299863334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc102c10 0x7fa9dc103070 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc0099c0 tx=0x7fa9cc009cd0 comp rx=0 tx=0).stop 2024-04-12T21:10:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3299863334 shutdown_connections 2024-04-12T21:10:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3299863334 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9dc1035b0 0x7fa9dc109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3299863334 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc101a10 0x7fa9dc101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:01.947 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3299863334 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc102c10 0x7fa9dc103070 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3299863334 >> 172.21.15.104:0/3299863334 conn(0x7fa9dc0fcfb0 msgr2=0x7fa9dc0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3299863334 shutdown_connections 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.949+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3299863334 wait complete. 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 Processor -- start 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 -- start start 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9dc101a10 0x7fa9dc198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc102c10 0x7fa9dc199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc1035b0 0x7fa9dc1a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10d6f0 con 0x7fa9dc101a10 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10d570 con 0x7fa9dc1035b0 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10d870 con 0x7fa9dc102c10 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc1035b0 0x7fa9dc1a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:01.948 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc1035b0 0x7fa9dc1a04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:34270/0 (socket says 172.21.15.104:34270) 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 -- 172.21.15.104:0/3000913609 learned_addr learned my addr 172.21.15.104:0/3000913609 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e12a6700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc102c10 0x7fa9dc199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 -- 172.21.15.104:0/3000913609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc102c10 msgr2=0x7fa9dc199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc102c10 0x7fa9dc199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 -- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9dc101a10 msgr2=0x7fa9dc198ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9dc101a10 0x7fa9dc198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 -- 172.21.15.104:0/3000913609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9dc1a0b40 con 0x7fa9dc1035b0 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e22a8700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc1035b0 0x7fa9dc1a04a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400b550 tx=0x7fa9d400b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9d2ffd700 1 -- 172.21.15.104:0/3000913609 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4015020 con 0x7fa9dc1035b0 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9dc1a0e30 con 0x7fa9dc1035b0 2024-04-12T21:10:01.949 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.950+0000 7fa9d2ffd700 1 -- 172.21.15.104:0/3000913609 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4011d10 con 0x7fa9dc1035b0 2024-04-12T21:10:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.951+0000 7fa9d2ffd700 1 -- 172.21.15.104:0/3000913609 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d400f2e0 con 0x7fa9dc1035b0 2024-04-12T21:10:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.951+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9dc1a1360 con 0x7fa9dc1035b0 2024-04-12T21:10:01.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.952+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9dc0697a0 con 0x7fa9dc1035b0 2024-04-12T21:10:01.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.953+0000 7fa9d2ffd700 1 -- 172.21.15.104:0/3000913609 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa9d4018020 con 0x7fa9dc1035b0 2024-04-12T21:10:01.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.954+0000 7fa9d2ffd700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa9c80a3c40 0x7fa9c80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:01.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.954+0000 7fa9d2ffd700 1 -- 172.21.15.104:0/3000913609 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fa9d40cf230 con 0x7fa9dc1035b0 2024-04-12T21:10:01.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.955+0000 7fa9e1aa7700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa9c80a3c40 0x7fa9c80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:01.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.955+0000 7fa9e1aa7700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa9c80a3c40 0x7fa9c80a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc102a70 tx=0x7fa9d8009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:10:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:01.958+0000 7fa9d2ffd700 1 -- 172.21.15.104:0/3000913609 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9d401f020 con 0x7fa9dc1035b0 2024-04-12T21:10:02.171 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:02.175+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa9dc06aea0 con 0x7fa9c80a3c40 2024-04-12T21:10:03.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi081 2024-04-12T21:10:03.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 5696M 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:03.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:03 smithi081 ceph-mon[31453]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:03.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi081 2024-04-12T21:10:03.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 5696M 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:03 smithi153 ceph-mon[36569]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi081 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 5696M 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:03.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:03.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:03.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:03.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:03.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:03.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:03.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:03 smithi104 ceph-mon[36624]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:05.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:05 smithi104 ceph-mon[36624]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:05.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:05 smithi104 ceph-mon[36624]: from='client.? 172.21.15.104:0/2771564041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]: dispatch 2024-04-12T21:10:05.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:05 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]: dispatch 2024-04-12T21:10:05.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:05 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]': finished 2024-04-12T21:10:05.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:05 smithi104 ceph-mon[36624]: osdmap e48: 8 total, 7 up, 8 in 2024-04-12T21:10:05.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:05.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:05 smithi081 ceph-mon[31453]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:05.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:05 smithi081 ceph-mon[31453]: from='client.? 172.21.15.104:0/2771564041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]: dispatch 2024-04-12T21:10:05.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:05 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]: dispatch 2024-04-12T21:10:05.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:05 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]': finished 2024-04-12T21:10:05.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:05 smithi081 ceph-mon[31453]: osdmap e48: 8 total, 7 up, 8 in 2024-04-12T21:10:05.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:05 smithi153 ceph-mon[36569]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:05 smithi153 ceph-mon[36569]: from='client.? 172.21.15.104:0/2771564041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]: dispatch 2024-04-12T21:10:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:05 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]: dispatch 2024-04-12T21:10:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:05 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "535d287c-497d-47a4-99e0-0d72ace39cd4"}]': finished 2024-04-12T21:10:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:05 smithi153 ceph-mon[36569]: osdmap e48: 8 total, 7 up, 8 in 2024-04-12T21:10:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:06.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:06 smithi104 ceph-mon[36624]: from='client.? 172.21.15.104:0/4178809426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:10:06.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:06 smithi081 ceph-mon[31453]: from='client.? 172.21.15.104:0/4178809426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:10:06.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:06 smithi153 ceph-mon[36569]: from='client.? 172.21.15.104:0/4178809426' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:10:07.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:07 smithi104 ceph-mon[36624]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:07.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:07 smithi081 ceph-mon[31453]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:07 smithi153 ceph-mon[36569]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:09.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:09 smithi104 ceph-mon[36624]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:09 smithi081 ceph-mon[31453]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:09 smithi153 ceph-mon[36569]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:11.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:11 smithi104 ceph-mon[36624]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:11.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:11 smithi081 ceph-mon[31453]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:11.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:11 smithi153 ceph-mon[36569]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:12.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T21:10:12.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T21:10:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-12T21:10:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:13.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:13 smithi104 ceph-mon[36624]: Deploying daemon osd.7 on smithi104 2024-04-12T21:10:13.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:13 smithi104 ceph-mon[36624]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:13.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:13 smithi081 ceph-mon[31453]: Deploying daemon osd.7 on smithi104 2024-04-12T21:10:13.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:13 smithi081 ceph-mon[31453]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:13 smithi153 ceph-mon[36569]: Deploying daemon osd.7 on smithi104 2024-04-12T21:10:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:13 smithi153 ceph-mon[36569]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:15.533 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:15 smithi104 ceph-mon[36624]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:15.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:15 smithi081 ceph-mon[31453]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:15 smithi153 ceph-mon[36569]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:17.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:17 smithi104 ceph-mon[36624]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:17.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:17 smithi081 ceph-mon[31453]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:17.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:17 smithi153 ceph-mon[36569]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:18 smithi104 ceph-mon[36624]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:19.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:19.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:19.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:19.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:18 smithi081 ceph-mon[31453]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:19.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:19.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:19.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:19.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:18 smithi153 ceph-mon[36569]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:21.300 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.303+0000 7fa9d2ffd700 1 -- 172.21.15.104:0/3000913609 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa9dc06aea0 con 0x7fa9c80a3c40 2024-04-12T21:10:21.300 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 7 on host 'smithi104' 2024-04-12T21:10:21.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa9c80a3c40 msgr2=0x7fa9c80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:21.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa9c80a3c40 0x7fa9c80a6100 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc102a70 tx=0x7fa9d8009250 comp rx=0 tx=0).stop 2024-04-12T21:10:21.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc1035b0 msgr2=0x7fa9dc1a04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:21.302 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc1035b0 0x7fa9dc1a04a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400b550 tx=0x7fa9d400b860 comp rx=0 tx=0).stop 2024-04-12T21:10:21.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 shutdown_connections 2024-04-12T21:10:21.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa9c80a3c40 0x7fa9c80a6100 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:21.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9dc1035b0 0x7fa9dc1a04a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:21.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa9dc101a10 0x7fa9dc198ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:21.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 --2- 172.21.15.104:0/3000913609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9dc102c10 0x7fa9dc199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:21.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 >> 172.21.15.104:0/3000913609 conn(0x7fa9dc0fcfb0 msgr2=0x7fa9dc06a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:10:21.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 shutdown_connections 2024-04-12T21:10:21.303 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:10:21.306+0000 7fa9e3d0b700 1 -- 172.21.15.104:0/3000913609 wait complete. 2024-04-12T21:10:21.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:21 smithi104 ceph-mon[36624]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:21.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:21.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:21.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:21 smithi081 ceph-mon[31453]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:21.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:21.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:21 smithi153 ceph-mon[36569]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:22.016 DEBUG:teuthology.orchestra.run.smithi104:osd.7> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.7.service 2024-04-12T21:10:22.021 INFO:tasks.cephadm:Deploying osd.8 on smithi153 with /dev/vg_nvme/lv_2... 2024-04-12T21:10:22.022 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-12T21:10:22.269 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:10:22.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:22.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:22.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:22 smithi153 ceph-mon[36569]: from='osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:10:22.711 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:22 smithi153 ceph-mon[36569]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:10:22.711 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:22 smithi153 ceph-mon[36569]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:22 smithi104 ceph-mon[36624]: from='osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:10:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:22 smithi104 ceph-mon[36624]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:10:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:22 smithi104 ceph-mon[36624]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:22 smithi081 ceph-mon[31453]: from='osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:10:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:22 smithi081 ceph-mon[31453]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-12T21:10:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:22 smithi081 ceph-mon[31453]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:23 smithi081 ceph-mon[31453]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T21:10:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:23 smithi081 ceph-mon[31453]: from='osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:10:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:23 smithi081 ceph-mon[31453]: osdmap e49: 8 total, 7 up, 8 in 2024-04-12T21:10:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:23 smithi081 ceph-mon[31453]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:10:23.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:23 smithi153 ceph-mon[36569]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T21:10:23.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:23 smithi153 ceph-mon[36569]: from='osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:10:23.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:23 smithi153 ceph-mon[36569]: osdmap e49: 8 total, 7 up, 8 in 2024-04-12T21:10:23.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:23.925 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:23 smithi153 ceph-mon[36569]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:10:24.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:23 smithi104 ceph-mon[36624]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-12T21:10:24.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:23 smithi104 ceph-mon[36624]: from='osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:10:24.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:23 smithi104 ceph-mon[36624]: osdmap e49: 8 total, 7 up, 8 in 2024-04-12T21:10:24.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:24.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:23 smithi104 ceph-mon[36624]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:10:24.079 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 12 21:10:23 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-7[54359]: 2024-04-12T21:10:23.633+0000 7f6eacaa4700 -1 osd.7 0 waiting for initial osdmap 2024-04-12T21:10:24.079 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 12 21:10:23 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-7[54359]: 2024-04-12T21:10:23.641+0000 7f6ea8897700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:10:24.254 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-12T21:10:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-12T21:10:24.255 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-12T21:10:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-12T21:10:24.255 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115 s, 912 MB/s 2024-04-12T21:10:24.255 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-12T21:10:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:24 smithi104 ceph-mon[36624]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:10:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:24 smithi104 ceph-mon[36624]: osdmap e50: 8 total, 7 up, 8 in 2024-04-12T21:10:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:24 smithi104 ceph-mon[36624]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:24 smithi104 ceph-mon[36624]: osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] boot 2024-04-12T21:10:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:24 smithi104 ceph-mon[36624]: osdmap e51: 8 total, 8 up, 8 in 2024-04-12T21:10:24.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:24 smithi081 ceph-mon[31453]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:10:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:24 smithi081 ceph-mon[31453]: osdmap e50: 8 total, 7 up, 8 in 2024-04-12T21:10:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:24 smithi081 ceph-mon[31453]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:24 smithi081 ceph-mon[31453]: osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] boot 2024-04-12T21:10:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:24 smithi081 ceph-mon[31453]: osdmap e51: 8 total, 8 up, 8 in 2024-04-12T21:10:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:24.954 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:24 smithi153 ceph-mon[36569]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:10:24.955 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:24 smithi153 ceph-mon[36569]: osdmap e50: 8 total, 7 up, 8 in 2024-04-12T21:10:24.955 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:24.955 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:24 smithi153 ceph-mon[36569]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-04-12T21:10:24.955 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:24 smithi153 ceph-mon[36569]: osd.7 [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] boot 2024-04-12T21:10:24.955 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:24 smithi153 ceph-mon[36569]: osdmap e51: 8 total, 8 up, 8 in 2024-04-12T21:10:24.955 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:24.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-04-12T21:10:25.226 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:10:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:10:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:10:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi104 2024-04-12T21:10:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 5693M 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:25.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:25 smithi081 ceph-mon[31453]: osdmap e52: 8 total, 8 up, 8 in 2024-04-12T21:10:26.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:10:26.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi104 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 5693M 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:26.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:26.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:25 smithi153 ceph-mon[36569]: osdmap e52: 8 total, 8 up, 8 in 2024-04-12T21:10:26.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:10:26.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:10:26.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-12T21:10:26.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi104 2024-04-12T21:10:26.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 5693M 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:25 smithi104 ceph-mon[36624]: osdmap e52: 8 total, 8 up, 8 in 2024-04-12T21:10:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 -- 172.21.15.153:0/3808058835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 msgr2=0x7ff628109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 --2- 172.21.15.153:0/3808058835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 0x7ff628109e60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff62000b0a0 tx=0x7ff62000b3b0 comp rx=0 tx=0).stop 2024-04-12T21:10:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 -- 172.21.15.153:0/3808058835 shutdown_connections 2024-04-12T21:10:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 --2- 172.21.15.153:0/3808058835 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 0x7ff628109e60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 --2- 172.21.15.153:0/3808058835 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff628101a10 0x7ff628101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:26.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 --2- 172.21.15.153:0/3808058835 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff628102c10 0x7ff628103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 -- 172.21.15.153:0/3808058835 >> 172.21.15.153:0/3808058835 conn(0x7ff6280fcfd0 msgr2=0x7ff6280ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 -- 172.21.15.153:0/3808058835 shutdown_connections 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.896+0000 7ff62f647700 1 -- 172.21.15.153:0/3808058835 wait complete. 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62f647700 1 Processor -- start 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62f647700 1 -- start start 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62f647700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff628101a10 0x7ff628198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62f647700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff628102c10 0x7ff628199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62f647700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 0x7ff6281a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62f647700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff62810d670 con 0x7ff628102c10 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62f647700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff62810d4f0 con 0x7ff628101a10 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62f647700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff62810d7f0 con 0x7ff6281035b0 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62dbe4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 0x7ff6281a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62dbe4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 0x7ff6281a0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33700/0 (socket says 172.21.15.153:33700) 2024-04-12T21:10:26.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62dbe4700 1 -- 172.21.15.153:0/4011784467 learned_addr learned my addr 172.21.15.153:0/4011784467 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62d3e3700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff628101a10 0x7ff628198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62dbe4700 1 -- 172.21.15.153:0/4011784467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff628101a10 msgr2=0x7ff628198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62dbe4700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff628101a10 0x7ff628198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62dbe4700 1 -- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff628102c10 msgr2=0x7ff628199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62dbe4700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff628102c10 0x7ff628199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.897+0000 7ff62dbe4700 1 -- 172.21.15.153:0/4011784467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6281a09a0 con 0x7ff6281035b0 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.898+0000 7ff62dbe4700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 0x7ff6281a0300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff620009800 tx=0x7ff62000bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.898+0000 7ff61e7fc700 1 -- 172.21.15.153:0/4011784467 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff620021070 con 0x7ff6281035b0 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.898+0000 7ff61e7fc700 1 -- 172.21.15.153:0/4011784467 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff620005520 con 0x7ff6281035b0 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.898+0000 7ff61e7fc700 1 -- 172.21.15.153:0/4011784467 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62001b8d0 con 0x7ff6281035b0 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.898+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6281a0c30 con 0x7ff6281035b0 2024-04-12T21:10:26.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.898+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6281a1100 con 0x7ff6281035b0 2024-04-12T21:10:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.899+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6281a12d0 con 0x7ff6281035b0 2024-04-12T21:10:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.900+0000 7ff61e7fc700 1 -- 172.21.15.153:0/4011784467 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff620024070 con 0x7ff6281035b0 2024-04-12T21:10:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.900+0000 7ff61e7fc700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff6140a3c40 0x7ff6140a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.901+0000 7ff61e7fc700 1 -- 172.21.15.153:0/4011784467 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7ff62000f020 con 0x7ff6281035b0 2024-04-12T21:10:26.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.901+0000 7ff62d3e3700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff6140a3c40 0x7ff6140a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:26.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.902+0000 7ff62d3e3700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff6140a3c40 0x7ff6140a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff628102a70 tx=0x7ff62400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:10:26.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:26.904+0000 7ff61e7fc700 1 -- 172.21.15.153:0/4011784467 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff62002b020 con 0x7ff6281035b0 2024-04-12T21:10:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:26 smithi081 ceph-mon[31453]: pgmap v194: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:26.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:26 smithi081 ceph-mon[31453]: osdmap e53: 8 total, 8 up, 8 in 2024-04-12T21:10:27.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:26 smithi153 ceph-mon[36569]: pgmap v194: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:27.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:26 smithi153 ceph-mon[36569]: osdmap e53: 8 total, 8 up, 8 in 2024-04-12T21:10:27.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:26 smithi104 ceph-mon[36624]: pgmap v194: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:26 smithi104 ceph-mon[36624]: osdmap e53: 8 total, 8 up, 8 in 2024-04-12T21:10:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:27.111+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff6280ff120 con 0x7ff6140a3c40 2024-04-12T21:10:27.917 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:27 smithi153 ceph-mon[36569]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-12T21:10:27.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:27 smithi153 ceph-mon[36569]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:27.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:27.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:27.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:27 smithi104 ceph-mon[36624]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-12T21:10:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:27 smithi104 ceph-mon[36624]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:28.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:27 smithi081 ceph-mon[31453]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-12T21:10:28.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:27 smithi081 ceph-mon[31453]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:28.863 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:28 smithi153 ceph-mon[36569]: osdmap e54: 8 total, 8 up, 8 in 2024-04-12T21:10:28.863 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:28 smithi153 ceph-mon[36569]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:29.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:28 smithi104 ceph-mon[36624]: osdmap e54: 8 total, 8 up, 8 in 2024-04-12T21:10:29.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:28 smithi104 ceph-mon[36624]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:29.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:28 smithi081 ceph-mon[31453]: osdmap e54: 8 total, 8 up, 8 in 2024-04-12T21:10:29.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:28 smithi081 ceph-mon[31453]: pgmap v197: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:30.016 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:29 smithi153 ceph-mon[36569]: osdmap e55: 8 total, 8 up, 8 in 2024-04-12T21:10:30.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:29 smithi104 ceph-mon[36624]: osdmap e55: 8 total, 8 up, 8 in 2024-04-12T21:10:30.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:29 smithi081 ceph-mon[31453]: osdmap e55: 8 total, 8 up, 8 in 2024-04-12T21:10:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:30 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/744932763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]: dispatch 2024-04-12T21:10:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:30 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]: dispatch 2024-04-12T21:10:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:30 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]': finished 2024-04-12T21:10:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:30 smithi153 ceph-mon[36569]: osdmap e56: 9 total, 8 up, 9 in 2024-04-12T21:10:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:30 smithi153 ceph-mon[36569]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:30 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/3423660502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:10:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:30 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/744932763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]: dispatch 2024-04-12T21:10:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:30 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]: dispatch 2024-04-12T21:10:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:30 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]': finished 2024-04-12T21:10:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:30 smithi104 ceph-mon[36624]: osdmap e56: 9 total, 8 up, 9 in 2024-04-12T21:10:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:30 smithi104 ceph-mon[36624]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:30 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/3423660502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:10:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:30 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/744932763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]: dispatch 2024-04-12T21:10:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:30 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]: dispatch 2024-04-12T21:10:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:30 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "799f954d-0c19-4fb6-a80d-5df6f65588c5"}]': finished 2024-04-12T21:10:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:30 smithi081 ceph-mon[31453]: osdmap e56: 9 total, 8 up, 9 in 2024-04-12T21:10:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:30 smithi081 ceph-mon[31453]: pgmap v200: 1 pgs: 1 remapped; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:30 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/3423660502' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:10:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:32 smithi081 ceph-mon[31453]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-12T21:10:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:32 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:10:32.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:32 smithi153 ceph-mon[36569]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-12T21:10:32.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:32 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:10:32.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:32 smithi104 ceph-mon[36624]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-12T21:10:32.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:32 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:10:33.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:33 smithi153 ceph-mon[36569]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:33.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:33 smithi081 ceph-mon[31453]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:33 smithi104 ceph-mon[36624]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:35.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:35 smithi153 ceph-mon[36569]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:35.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:35 smithi081 ceph-mon[31453]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:35.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:35 smithi104 ceph-mon[36624]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:37.636 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:37 smithi153 ceph-mon[36569]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:37.636 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-12T21:10:37.637 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:37 smithi081 ceph-mon[31453]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-12T21:10:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:37 smithi104 ceph-mon[36624]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-12T21:10:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:38.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:38 smithi081 ceph-mon[31453]: Deploying daemon osd.8 on smithi153 2024-04-12T21:10:38.713 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:38 smithi153 ceph-mon[36569]: Deploying daemon osd.8 on smithi153 2024-04-12T21:10:38.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:38 smithi104 ceph-mon[36624]: Deploying daemon osd.8 on smithi153 2024-04-12T21:10:39.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:39 smithi081 ceph-mon[31453]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:39.733 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:39 smithi153 ceph-mon[36569]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:39.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:39 smithi104 ceph-mon[36624]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:41.524 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:41 smithi153 ceph-mon[36569]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:41.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:41 smithi081 ceph-mon[31453]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:41 smithi104 ceph-mon[36624]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:43 smithi153 ceph-mon[36569]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:43 smithi104 ceph-mon[36624]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:43.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:43 smithi081 ceph-mon[31453]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:43.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:43.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:43.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:45 smithi153 ceph-mon[36569]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:45.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:45 smithi104 ceph-mon[36624]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:45.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:45 smithi081 ceph-mon[31453]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:45.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.947+0000 7ff61e7fc700 1 -- 172.21.15.153:0/4011784467 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff6280ff120 con 0x7ff6140a3c40 2024-04-12T21:10:45.944 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 8 on host 'smithi153' 2024-04-12T21:10:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff6140a3c40 msgr2=0x7ff6140a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff6140a3c40 0x7ff6140a6100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff628102a70 tx=0x7ff62400b410 comp rx=0 tx=0).stop 2024-04-12T21:10:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 msgr2=0x7ff6281a0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 0x7ff6281a0300 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff620009800 tx=0x7ff62000bd70 comp rx=0 tx=0).stop 2024-04-12T21:10:45.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 shutdown_connections 2024-04-12T21:10:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff6140a3c40 0x7ff6140a6100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6281035b0 0x7ff6281a0300 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff628101a10 0x7ff628198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 --2- 172.21.15.153:0/4011784467 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff628102c10 0x7ff628199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 >> 172.21.15.153:0/4011784467 conn(0x7ff6280fcfd0 msgr2=0x7ff6280fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:10:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 shutdown_connections 2024-04-12T21:10:45.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:10:45.950+0000 7ff62f647700 1 -- 172.21.15.153:0/4011784467 wait complete. 2024-04-12T21:10:46.591 DEBUG:teuthology.orchestra.run.smithi153:osd.8> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.8.service 2024-04-12T21:10:46.594 INFO:tasks.cephadm:Deploying osd.9 on smithi081 with /dev/vg_nvme/lv_1... 2024-04-12T21:10:46.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-12T21:10:47.054 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:10:47.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:46 smithi081 ceph-mon[31453]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:46 smithi081 ceph-mon[31453]: from='osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-12T21:10:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:46 smithi081 ceph-mon[31453]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-12T21:10:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:46 smithi153 ceph-mon[36569]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:47.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:46 smithi153 ceph-mon[36569]: from='osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-12T21:10:47.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:46 smithi153 ceph-mon[36569]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-12T21:10:47.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:46 smithi104 ceph-mon[36624]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:47.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:46 smithi104 ceph-mon[36624]: from='osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-12T21:10:47.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:46 smithi104 ceph-mon[36624]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-12T21:10:47.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:47.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:47 smithi153 ceph-mon[36569]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-12T21:10:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:47 smithi153 ceph-mon[36569]: from='osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:10:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:47 smithi153 ceph-mon[36569]: osdmap e57: 9 total, 8 up, 9 in 2024-04-12T21:10:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:48.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:47 smithi153 ceph-mon[36569]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:10:48.250 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:47 smithi081 ceph-mon[31453]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-12T21:10:48.251 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:47 smithi081 ceph-mon[31453]: from='osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:10:48.251 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:47 smithi081 ceph-mon[31453]: osdmap e57: 9 total, 8 up, 9 in 2024-04-12T21:10:48.251 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:48.251 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:47 smithi081 ceph-mon[31453]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:10:48.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:47 smithi104 ceph-mon[36624]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-12T21:10:48.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:47 smithi104 ceph-mon[36624]: from='osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:10:48.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:47 smithi104 ceph-mon[36624]: osdmap e57: 9 total, 8 up, 9 in 2024-04-12T21:10:48.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:48.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:47 smithi104 ceph-mon[36624]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:10:48.931 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 12 21:10:48 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-8[55739]: 2024-04-12T21:10:48.672+0000 7f1da624f700 -1 osd.8 0 waiting for initial osdmap 2024-04-12T21:10:48.931 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 12 21:10:48 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-8[55739]: 2024-04-12T21:10:48.683+0000 7f1d9e03a700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:10:49.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:48 smithi081 ceph-mon[31453]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:10:49.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:48 smithi081 ceph-mon[31453]: osdmap e58: 9 total, 8 up, 9 in 2024-04-12T21:10:49.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:49.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:49.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:48 smithi081 ceph-mon[31453]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:49.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:48 smithi081 ceph-mon[31453]: from='osd.8 ' entity='osd.8' 2024-04-12T21:10:49.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:48 smithi153 ceph-mon[36569]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:10:49.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:48 smithi153 ceph-mon[36569]: osdmap e58: 9 total, 8 up, 9 in 2024-04-12T21:10:49.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:49.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:49.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:48 smithi153 ceph-mon[36569]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:49.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:48 smithi153 ceph-mon[36569]: from='osd.8 ' entity='osd.8' 2024-04-12T21:10:49.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:48 smithi104 ceph-mon[36624]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:10:49.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:48 smithi104 ceph-mon[36624]: osdmap e58: 9 total, 8 up, 9 in 2024-04-12T21:10:49.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:49.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:49.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:48 smithi104 ceph-mon[36624]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 237 MiB used, 715 GiB / 715 GiB avail 2024-04-12T21:10:49.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:48 smithi104 ceph-mon[36624]: from='osd.8 ' entity='osd.8' 2024-04-12T21:10:50.039 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-12T21:10:50.039 INFO:teuthology.orchestra.run.smithi081.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-12T21:10:50.039 INFO:teuthology.orchestra.run.smithi081.stderr: stderr: 10+0 records in 2024-04-12T21:10:50.039 INFO:teuthology.orchestra.run.smithi081.stderr:10+0 records out 2024-04-12T21:10:50.039 INFO:teuthology.orchestra.run.smithi081.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0316371 s, 331 MB/s 2024-04-12T21:10:50.039 INFO:teuthology.orchestra.run.smithi081.stderr:--> Zapping successful for: 2024-04-12T21:10:50.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:49 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:10:50.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:49 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:10:50.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:49 smithi081 ceph-mon[31453]: osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] boot 2024-04-12T21:10:50.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:49 smithi081 ceph-mon[31453]: osdmap e59: 9 total, 9 up, 9 in 2024-04-12T21:10:50.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:50.040 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:49 smithi081 ceph-mon[31453]: osdmap e60: 9 total, 9 up, 9 in 2024-04-12T21:10:50.166 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:49 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:10:50.166 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:49 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:10:50.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:49 smithi153 ceph-mon[36569]: osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] boot 2024-04-12T21:10:50.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:49 smithi153 ceph-mon[36569]: osdmap e59: 9 total, 9 up, 9 in 2024-04-12T21:10:50.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:50.167 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:49 smithi153 ceph-mon[36569]: osdmap e60: 9 total, 9 up, 9 in 2024-04-12T21:10:50.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:49 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:10:50.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:49 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:10:50.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:49 smithi104 ceph-mon[36624]: osd.8 [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] boot 2024-04-12T21:10:50.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:49 smithi104 ceph-mon[36624]: osdmap e59: 9 total, 9 up, 9 in 2024-04-12T21:10:50.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-12T21:10:50.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:49 smithi104 ceph-mon[36624]: osdmap e60: 9 total, 9 up, 9 in 2024-04-12T21:10:50.989 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi081:vg_nvme/lv_1 2024-04-12T21:10:51.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi153 2024-04-12T21:10:51.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:51.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 5693M 2024-04-12T21:10:51.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:51.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:51.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:51.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:51.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:51.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:51 smithi081 ceph-mon[31453]: osdmap e61: 9 total, 9 up, 9 in 2024-04-12T21:10:51.433 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:10:51.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi153 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 5693M 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:51.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:51 smithi153 ceph-mon[36569]: osdmap e61: 9 total, 9 up, 9 in 2024-04-12T21:10:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi153 2024-04-12T21:10:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 632 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 5693M 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:10:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:51 smithi104 ceph-mon[36624]: osdmap e61: 9 total, 9 up, 9 in 2024-04-12T21:10:53.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:52 smithi081 ceph-mon[31453]: osdmap e62: 9 total, 9 up, 9 in 2024-04-12T21:10:53.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:52 smithi081 ceph-mon[31453]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 632 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:53.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:52 smithi153 ceph-mon[36569]: osdmap e62: 9 total, 9 up, 9 in 2024-04-12T21:10:53.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:52 smithi153 ceph-mon[36569]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 632 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:53.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:52 smithi104 ceph-mon[36624]: osdmap e62: 9 total, 9 up, 9 in 2024-04-12T21:10:53.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:52 smithi104 ceph-mon[36624]: pgmap v217: 1 pgs: 1 remapped+peering; 577 KiB data, 632 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:53.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.677+0000 7f59fca00700 1 -- 172.21.15.81:0/2593418680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f8105f50 msgr2=0x7f59f81063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:53.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.677+0000 7f59fca00700 1 --2- 172.21.15.81:0/2593418680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f8105f50 0x7f59f81063b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f59e40098f0 tx=0x7f59e4009c00 comp rx=0 tx=0).stop 2024-04-12T21:10:53.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.678+0000 7f59fca00700 1 -- 172.21.15.81:0/2593418680 shutdown_connections 2024-04-12T21:10:53.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.678+0000 7f59fca00700 1 --2- 172.21.15.81:0/2593418680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59f81068f0 0x7f59f810d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:53.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.678+0000 7f59fca00700 1 --2- 172.21.15.81:0/2593418680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59f8104d50 0x7f59f8105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:53.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.678+0000 7f59fca00700 1 --2- 172.21.15.81:0/2593418680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f8105f50 0x7f59f81063b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:53.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.678+0000 7f59fca00700 1 -- 172.21.15.81:0/2593418680 >> 172.21.15.81:0/2593418680 conn(0x7f59f8100310 msgr2=0x7f59f8102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:10:53.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.678+0000 7f59fca00700 1 -- 172.21.15.81:0/2593418680 shutdown_connections 2024-04-12T21:10:53.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.678+0000 7f59fca00700 1 -- 172.21.15.81:0/2593418680 wait complete. 2024-04-12T21:10:53.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.680+0000 7f59fca00700 1 Processor -- start 2024-04-12T21:10:53.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.680+0000 7f59fca00700 1 -- start start 2024-04-12T21:10:53.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.680+0000 7f59fca00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59f8104d50 0x7f59f819af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:53.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.680+0000 7f59fca00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59f8105f50 0x7f59f819b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.680+0000 7f59fca00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f81068f0 0x7f59f81a2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.680+0000 7f59fca00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f59f8110960 con 0x7f59f8104d50 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.680+0000 7f59fca00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f59f81107e0 con 0x7f59f8105f50 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.680+0000 7f59fca00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f59f8110ae0 con 0x7f59f81068f0 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59f8104d50 0x7f59f819af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59f8104d50 0x7f59f819af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:60300/0 (socket says 172.21.15.81:60300) 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5e00700 1 -- 172.21.15.81:0/2193014186 learned_addr learned my addr 172.21.15.81:0/2193014186 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f6800700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f81068f0 0x7f59f81a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:53.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5400700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59f8105f50 0x7f59f819b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:53.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5e00700 1 -- 172.21.15.81:0/2193014186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f81068f0 msgr2=0x7f59f81a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:53.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5e00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f81068f0 0x7f59f81a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5e00700 1 -- 172.21.15.81:0/2193014186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59f8105f50 msgr2=0x7f59f819b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5e00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59f8105f50 0x7f59f819b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f5e00700 1 -- 172.21.15.81:0/2193014186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f81a2bd0 con 0x7f59f8104d50 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.681+0000 7f59f6800700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f81068f0 0x7f59f81a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.682+0000 7f59f5e00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59f8104d50 0x7f59f819af70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f59e800e9a0 tx=0x7f59e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.682+0000 7f59eea00700 1 -- 172.21.15.81:0/2193014186 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8004220 con 0x7f59f8104d50 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.682+0000 7f59eea00700 1 -- 172.21.15.81:0/2193014186 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59e80043c0 con 0x7f59f8104d50 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.682+0000 7f59eea00700 1 -- 172.21.15.81:0/2193014186 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e8003df0 con 0x7f59f8104d50 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.682+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f81a2ec0 con 0x7f59f8104d50 2024-04-12T21:10:53.680 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.682+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f81a32c0 con 0x7f59f8104d50 2024-04-12T21:10:53.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.684+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f802dfa0 con 0x7f59f8104d50 2024-04-12T21:10:53.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.685+0000 7f59eea00700 1 -- 172.21.15.81:0/2193014186 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f59e800c810 con 0x7f59f8104d50 2024-04-12T21:10:53.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.687+0000 7f59eea00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:10:53.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.687+0000 7f59eea00700 1 -- 172.21.15.81:0/2193014186 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f59e80cfc10 con 0x7f59f8104d50 2024-04-12T21:10:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.687+0000 7f59f5400700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:10:53.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.688+0000 7f59f5400700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f59e40098c0 tx=0x7f59e401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:10:53.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.692+0000 7f59eea00700 1 -- 172.21.15.81:0/2193014186 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f59e809cf00 con 0x7f59f8104d50 2024-04-12T21:10:53.924 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:10:53.927+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f59f8062ef0 con 0x7f59dc0a3cc0 2024-04-12T21:10:54.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:53 smithi153 ceph-mon[36569]: osdmap e63: 9 total, 9 up, 9 in 2024-04-12T21:10:54.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:54.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:54.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:54.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:53 smithi104 ceph-mon[36624]: osdmap e63: 9 total, 9 up, 9 in 2024-04-12T21:10:54.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:54.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:54.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:54.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:53 smithi081 ceph-mon[31453]: osdmap e63: 9 total, 9 up, 9 in 2024-04-12T21:10:54.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:10:54.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:10:54.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:10:55.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:54 smithi081 ceph-mon[31453]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:55.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:54 smithi081 ceph-mon[31453]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:55.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:54 smithi153 ceph-mon[36569]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:55.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:54 smithi153 ceph-mon[36569]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:55.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:54 smithi104 ceph-mon[36624]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi081:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:10:55.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:54 smithi104 ceph-mon[36624]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:10:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:57 smithi104 ceph-mon[36624]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-12T21:10:57.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:57 smithi081 ceph-mon[31453]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-12T21:10:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:57 smithi153 ceph-mon[36569]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-12T21:10:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:58 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2169558525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]: dispatch 2024-04-12T21:10:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:58 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]: dispatch 2024-04-12T21:10:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:58 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]': finished 2024-04-12T21:10:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:58 smithi081 ceph-mon[31453]: osdmap e64: 10 total, 9 up, 10 in 2024-04-12T21:10:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:10:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:58 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2169558525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]: dispatch 2024-04-12T21:10:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:58 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]: dispatch 2024-04-12T21:10:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:58 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]': finished 2024-04-12T21:10:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:58 smithi153 ceph-mon[36569]: osdmap e64: 10 total, 9 up, 10 in 2024-04-12T21:10:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:10:58.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:58 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2169558525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]: dispatch 2024-04-12T21:10:58.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:58 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]: dispatch 2024-04-12T21:10:58.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:58 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbb02774-5697-4bcb-ac44-8b01cef5f3fb"}]': finished 2024-04-12T21:10:58.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:58 smithi104 ceph-mon[36624]: osdmap e64: 10 total, 9 up, 10 in 2024-04-12T21:10:58.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:10:59.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:59 smithi081 ceph-mon[31453]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-12T21:10:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:10:59 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2055786632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:10:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:59 smithi153 ceph-mon[36569]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-12T21:10:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:10:59 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2055786632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:10:59.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:59 smithi104 ceph-mon[36624]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-12T21:10:59.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:10:59 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2055786632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:11:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:01 smithi081 ceph-mon[31453]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:11:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:01 smithi153 ceph-mon[36569]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:11:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:01 smithi104 ceph-mon[36624]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:11:03.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:03 smithi081 ceph-mon[31453]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-12T21:11:03.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:03 smithi153 ceph-mon[36569]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-12T21:11:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:03 smithi104 ceph-mon[36624]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-12T21:11:05.604 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:05 smithi153 ceph-mon[36569]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:11:05.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:05 smithi081 ceph-mon[31453]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:11:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:05 smithi104 ceph-mon[36624]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:11:07.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:07 smithi081 ceph-mon[31453]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:07.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:07 smithi153 ceph-mon[36569]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:07 smithi104 ceph-mon[36624]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:08.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-12T21:11:08.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:08.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-12T21:11:08.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:08.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-12T21:11:08.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:09 smithi081 ceph-mon[31453]: Deploying daemon osd.9 on smithi081 2024-04-12T21:11:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:09 smithi081 ceph-mon[31453]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:09 smithi153 ceph-mon[36569]: Deploying daemon osd.9 on smithi081 2024-04-12T21:11:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:09 smithi153 ceph-mon[36569]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:09 smithi104 ceph-mon[36624]: Deploying daemon osd.9 on smithi081 2024-04-12T21:11:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:09 smithi104 ceph-mon[36624]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:11.532 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:11 smithi081 ceph-mon[31453]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:11 smithi153 ceph-mon[36569]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:11 smithi104 ceph-mon[36624]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:13.630 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:13 smithi104 ceph-mon[36624]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:13.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:13 smithi081 ceph-mon[31453]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:13 smithi153 ceph-mon[36569]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:15.555 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:15 smithi081 ceph-mon[31453]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:15 smithi153 ceph-mon[36569]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:15 smithi104 ceph-mon[36624]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:17.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:17.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:17.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:17.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:16 smithi153 ceph-mon[36569]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:17.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:17.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:17.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:17.062 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:16 smithi081 ceph-mon[31453]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:17.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:17.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:17.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:17.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:16 smithi104 ceph-mon[36624]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:19.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:19 smithi081 ceph-mon[31453]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:19 smithi153 ceph-mon[36569]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:19 smithi104 ceph-mon[36624]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:20.871 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.875+0000 7f59eea00700 1 -- 172.21.15.81:0/2193014186 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f59f8062ef0 con 0x7f59dc0a3cc0 2024-04-12T21:11:20.871 INFO:teuthology.orchestra.run.smithi081.stdout:Created osd(s) 9 on host 'smithi081' 2024-04-12T21:11:20.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f59dc0a3cc0 msgr2=0x7f59dc0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:20.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f59e40098c0 tx=0x7f59e401a040 comp rx=0 tx=0).stop 2024-04-12T21:11:20.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59f8104d50 msgr2=0x7f59f819af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:20.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59f8104d50 0x7f59f819af70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f59e800e9a0 tx=0x7f59e800ecb0 comp rx=0 tx=0).stop 2024-04-12T21:11:20.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 shutdown_connections 2024-04-12T21:11:20.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f59dc0a3cc0 0x7f59dc0a6180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:20.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59f81068f0 0x7f59f81a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:20.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f59f8104d50 0x7f59f819af70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:20.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 --2- 172.21.15.81:0/2193014186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f59f8105f50 0x7f59f819b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:20.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 >> 172.21.15.81:0/2193014186 conn(0x7f59f8100310 msgr2=0x7f59f8109250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:11:20.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 shutdown_connections 2024-04-12T21:11:20.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:11:20.878+0000 7f59fca00700 1 -- 172.21.15.81:0/2193014186 wait complete. 2024-04-12T21:11:20.939 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:20.939 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:20.939 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:20 smithi081 ceph-mon[31453]: from='osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-12T21:11:20.939 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:20 smithi081 ceph-mon[31453]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-12T21:11:20.939 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:20 smithi081 ceph-mon[31453]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:20 smithi153 ceph-mon[36569]: from='osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-12T21:11:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:20 smithi153 ceph-mon[36569]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-12T21:11:21.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:20 smithi153 ceph-mon[36569]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:21.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:21.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:21.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:20 smithi104 ceph-mon[36624]: from='osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-12T21:11:21.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:20 smithi104 ceph-mon[36624]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-12T21:11:21.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:20 smithi104 ceph-mon[36624]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:21.685 DEBUG:teuthology.orchestra.run.smithi081:osd.9> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.9.service 2024-04-12T21:11:21.689 INFO:tasks.cephadm:Deploying osd.10 on smithi104 with /dev/vg_nvme/lv_1... 2024-04-12T21:11:21.689 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-12T21:11:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-12T21:11:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: osdmap e65: 10 total, 9 up, 10 in 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:21.936 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.b/config 2024-04-12T21:11:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-12T21:11:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:11:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: osdmap e65: 10 total, 9 up, 10 in 2024-04-12T21:11:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:11:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:22.057 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: osdmap e65: 10 total, 9 up, 10 in 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]: dispatch 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:22.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:22.878 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:22 smithi081 ceph-mon[31453]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:11:22.878 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:22 smithi081 ceph-mon[31453]: osdmap e66: 10 total, 9 up, 10 in 2024-04-12T21:11:22.878 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:22.878 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:22.878 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:22 smithi081 ceph-mon[31453]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:22.878 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:22 smithi081 ceph-mon[31453]: from='osd.9 ' entity='osd.9' 2024-04-12T21:11:22.879 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 12 21:11:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-9[66309]: 2024-04-12T21:11:22.613+0000 7f0596600700 -1 osd.9 0 waiting for initial osdmap 2024-04-12T21:11:22.879 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 12 21:11:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-9[66309]: 2024-04-12T21:11:22.648+0000 7f058c400700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:11:23.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:22 smithi153 ceph-mon[36569]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:11:23.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:22 smithi153 ceph-mon[36569]: osdmap e66: 10 total, 9 up, 10 in 2024-04-12T21:11:23.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:23.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:23.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:22 smithi153 ceph-mon[36569]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:23.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:22 smithi153 ceph-mon[36569]: from='osd.9 ' entity='osd.9' 2024-04-12T21:11:23.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:22 smithi104 ceph-mon[36624]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi081", "root=default"]}]': finished 2024-04-12T21:11:23.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:22 smithi104 ceph-mon[36624]: osdmap e66: 10 total, 9 up, 10 in 2024-04-12T21:11:23.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:23.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:23.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:22 smithi104 ceph-mon[36624]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-04-12T21:11:23.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:22 smithi104 ceph-mon[36624]: from='osd.9 ' entity='osd.9' 2024-04-12T21:11:23.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:23 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:11:23.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:23 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:11:23.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:23.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:23 smithi081 ceph-mon[31453]: osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] boot 2024-04-12T21:11:23.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:23 smithi081 ceph-mon[31453]: osdmap e67: 10 total, 10 up, 10 in 2024-04-12T21:11:23.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:24.022 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-12T21:11:24.023 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-12T21:11:24.023 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-04-12T21:11:24.023 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-04-12T21:11:24.023 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114234 s, 918 MB/s 2024-04-12T21:11:24.023 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-04-12T21:11:24.023 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:23 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:11:24.023 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:23 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:11:24.023 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:24.023 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:23 smithi104 ceph-mon[36624]: osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] boot 2024-04-12T21:11:24.024 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:23 smithi104 ceph-mon[36624]: osdmap e67: 10 total, 10 up, 10 in 2024-04-12T21:11:24.024 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:23 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:11:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:23 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:11:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:23 smithi153 ceph-mon[36569]: osd.9 [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] boot 2024-04-12T21:11:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:23 smithi153 ceph-mon[36569]: osdmap e67: 10 total, 10 up, 10 in 2024-04-12T21:11:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-12T21:11:24.680 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2024-04-12T21:11:24.924 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.b/config 2024-04-12T21:11:25.046 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:24 smithi104 ceph-mon[36624]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:25.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:24 smithi153 ceph-mon[36569]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:25.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:24 smithi081 ceph-mon[31453]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:25 smithi081 ceph-mon[31453]: osdmap e68: 10 total, 10 up, 10 in 2024-04-12T21:11:26.026 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:25 smithi104 ceph-mon[36624]: osdmap e68: 10 total, 10 up, 10 in 2024-04-12T21:11:26.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:25 smithi153 ceph-mon[36569]: osdmap e68: 10 total, 10 up, 10 in 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 -- 172.21.15.104:0/118705579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98801035b0 msgr2=0x7f9880109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 --2- 172.21.15.104:0/118705579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98801035b0 0x7f9880109e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f987800b0d0 tx=0x7f987800b3e0 comp rx=0 tx=0).stop 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 -- 172.21.15.104:0/118705579 shutdown_connections 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 --2- 172.21.15.104:0/118705579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98801035b0 0x7f9880109e60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 --2- 172.21.15.104:0/118705579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9880101a10 0x7f9880101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 --2- 172.21.15.104:0/118705579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9880102c10 0x7f9880103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 -- 172.21.15.104:0/118705579 >> 172.21.15.104:0/118705579 conn(0x7f98800fcfb0 msgr2=0x7f98800ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 -- 172.21.15.104:0/118705579 shutdown_connections 2024-04-12T21:11:26.533 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.535+0000 7f9887956700 1 -- 172.21.15.104:0/118705579 wait complete. 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9887956700 1 Processor -- start 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9887956700 1 -- start start 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9887956700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9880101a10 0x7f988006bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9887956700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9880102c10 0x7f988019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9887956700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98801035b0 0x7f98801a0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9887956700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f988010d670 con 0x7f9880101a10 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9887956700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f988010d4f0 con 0x7f9880102c10 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9887956700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f988010d7f0 con 0x7f98801035b0 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9880102c10 0x7f988019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9880102c10 0x7f988019d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:50480/0 (socket says 172.21.15.104:50480) 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 -- 172.21.15.104:0/2010317868 learned_addr learned my addr 172.21.15.104:0/2010317868 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9885ef3700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98801035b0 0x7f98801a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 -- 172.21.15.104:0/2010317868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98801035b0 msgr2=0x7f98801a0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:26.534 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98801035b0 0x7f98801a0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 -- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9880101a10 msgr2=0x7f988006bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9880101a10 0x7f988006bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 -- 172.21.15.104:0/2010317868 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98801a08d0 con 0x7f9880102c10 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9885ef3700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98801035b0 0x7f98801a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.536+0000 7f9884ef1700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9880102c10 0x7f988019d970 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f987000c9b0 tx=0x7f987000cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.537+0000 7f98767fc700 1 -- 172.21.15.104:0/2010317868 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870007b00 con 0x7f9880102c10 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.537+0000 7f98767fc700 1 -- 172.21.15.104:0/2010317868 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9870007ca0 con 0x7f9880102c10 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.537+0000 7f98767fc700 1 -- 172.21.15.104:0/2010317868 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870018780 con 0x7f9880102c10 2024-04-12T21:11:26.535 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.537+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98801a0bc0 con 0x7f9880102c10 2024-04-12T21:11:26.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.537+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98801a1100 con 0x7f9880102c10 2024-04-12T21:11:26.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.538+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98800697a0 con 0x7f9880102c10 2024-04-12T21:11:26.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.539+0000 7f98767fc700 1 -- 172.21.15.104:0/2010317868 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f987000f450 con 0x7f9880102c10 2024-04-12T21:11:26.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.539+0000 7f98767fc700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f986c0a3c40 0x7f986c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:11:26.536 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.539+0000 7f98767fc700 1 -- 172.21.15.104:0/2010317868 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f98700cfca0 con 0x7f9880102c10 2024-04-12T21:11:26.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.540+0000 7f98856f2700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f986c0a3c40 0x7f986c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:11:26.537 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.541+0000 7f98856f2700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f986c0a3c40 0x7f986c0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9880102a70 tx=0x7f987c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:11:26.540 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.544+0000 7f98767fc700 1 -- 172.21.15.104:0/2010317868 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9870014030 con 0x7f9880102c10 2024-04-12T21:11:26.754 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:26.758+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9880062ef0 con 0x7f986c0a3c40 2024-04-12T21:11:27.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:27.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:27.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:27.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:27.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:27.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:27 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi081 2024-04-12T21:11:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:27 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 4272M 2024-04-12T21:11:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:27 smithi153 ceph-mon[36569]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:11:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:11:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:11:28.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:27 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi081 2024-04-12T21:11:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:27 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 4272M 2024-04-12T21:11:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:27 smithi104 ceph-mon[36624]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:11:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:11:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:11:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:27 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi081 2024-04-12T21:11:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:27 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 4272M 2024-04-12T21:11:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:27 smithi081 ceph-mon[31453]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:11:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:11:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:11:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:29.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:28 smithi153 ceph-mon[36569]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:29.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:28 smithi104 ceph-mon[36624]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:29.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:28 smithi081 ceph-mon[31453]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:30.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:29 smithi153 ceph-mon[36569]: from='client.? 172.21.15.104:0/499019968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]: dispatch 2024-04-12T21:11:30.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:29 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]: dispatch 2024-04-12T21:11:30.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:29 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]': finished 2024-04-12T21:11:30.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:29 smithi153 ceph-mon[36569]: osdmap e69: 11 total, 10 up, 11 in 2024-04-12T21:11:30.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:30.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:29 smithi104 ceph-mon[36624]: from='client.? 172.21.15.104:0/499019968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]: dispatch 2024-04-12T21:11:30.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:29 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]: dispatch 2024-04-12T21:11:30.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:29 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]': finished 2024-04-12T21:11:30.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:29 smithi104 ceph-mon[36624]: osdmap e69: 11 total, 10 up, 11 in 2024-04-12T21:11:30.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:30.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:29 smithi081 ceph-mon[31453]: from='client.? 172.21.15.104:0/499019968' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]: dispatch 2024-04-12T21:11:30.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:29 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]: dispatch 2024-04-12T21:11:30.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:29 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30c8758d-3e32-4d59-9819-a13a38c1745d"}]': finished 2024-04-12T21:11:30.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:29 smithi081 ceph-mon[31453]: osdmap e69: 11 total, 10 up, 11 in 2024-04-12T21:11:30.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:30 smithi153 ceph-mon[36569]: from='client.? 172.21.15.104:0/640996090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:11:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:30 smithi153 ceph-mon[36569]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:31.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:30 smithi104 ceph-mon[36624]: from='client.? 172.21.15.104:0/640996090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:11:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:30 smithi104 ceph-mon[36624]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:31.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:30 smithi081 ceph-mon[31453]: from='client.? 172.21.15.104:0/640996090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:11:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:30 smithi081 ceph-mon[31453]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:33 smithi104 ceph-mon[36624]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:33.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:33 smithi081 ceph-mon[31453]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:33 smithi153 ceph-mon[36569]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:35.604 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:35 smithi153 ceph-mon[36569]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:35.608 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:35 smithi104 ceph-mon[36624]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:35.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:35 smithi081 ceph-mon[31453]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:37.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:37 smithi104 ceph-mon[36624]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:37.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-12T21:11:37.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:37 smithi081 ceph-mon[31453]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-12T21:11:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:37 smithi153 ceph-mon[36569]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-12T21:11:37.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:38.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:38 smithi104 ceph-mon[36624]: Deploying daemon osd.10 on smithi104 2024-04-12T21:11:38.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:38 smithi081 ceph-mon[31453]: Deploying daemon osd.10 on smithi104 2024-04-12T21:11:38.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:38 smithi153 ceph-mon[36569]: Deploying daemon osd.10 on smithi104 2024-04-12T21:11:39.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:39 smithi104 ceph-mon[36624]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:39.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:39 smithi081 ceph-mon[31453]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:39 smithi153 ceph-mon[36569]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:41.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:41 smithi104 ceph-mon[36624]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:41.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:41 smithi081 ceph-mon[31453]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:41 smithi153 ceph-mon[36569]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:43.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:43 smithi081 ceph-mon[31453]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:43.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:43.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:43.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:43.702 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:43 smithi104 ceph-mon[36624]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:43.702 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:43.702 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:43.702 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:43 smithi153 ceph-mon[36569]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:45.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:45 smithi081 ceph-mon[31453]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:45.706 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:45 smithi104 ceph-mon[36624]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:45 smithi153 ceph-mon[36569]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:46.738 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.742+0000 7f98767fc700 1 -- 172.21.15.104:0/2010317868 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9880062ef0 con 0x7f986c0a3c40 2024-04-12T21:11:46.738 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 10 on host 'smithi104' 2024-04-12T21:11:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f986c0a3c40 msgr2=0x7f986c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f986c0a3c40 0x7f986c0a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9880102a70 tx=0x7f987c006d20 comp rx=0 tx=0).stop 2024-04-12T21:11:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9880102c10 msgr2=0x7f988019d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:46.740 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9880102c10 0x7f988019d970 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f987000c9b0 tx=0x7f987000cd70 comp rx=0 tx=0).stop 2024-04-12T21:11:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 shutdown_connections 2024-04-12T21:11:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f986c0a3c40 0x7f986c0a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98801035b0 0x7f98801a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9880101a10 0x7f988006bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 --2- 172.21.15.104:0/2010317868 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9880102c10 0x7f988019d970 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 >> 172.21.15.104:0/2010317868 conn(0x7f98800fcfb0 msgr2=0x7f9880105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:11:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 shutdown_connections 2024-04-12T21:11:46.741 INFO:teuthology.orchestra.run.smithi104.stderr:2024-04-12T21:11:46.744+0000 7f9887956700 1 -- 172.21.15.104:0/2010317868 wait complete. 2024-04-12T21:11:47.447 DEBUG:teuthology.orchestra.run.smithi104:osd.10> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.10.service 2024-04-12T21:11:47.449 INFO:tasks.cephadm:Deploying osd.11 on smithi153 with /dev/vg_nvme/lv_1... 2024-04-12T21:11:47.449 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-12T21:11:47.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:47 smithi081 ceph-mon[31453]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:47.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:47.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:47.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:47 smithi081 ceph-mon[31453]: from='osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-12T21:11:47.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:47 smithi081 ceph-mon[31453]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-12T21:11:47.701 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:11:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:47 smithi153 ceph-mon[36569]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:47 smithi153 ceph-mon[36569]: from='osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-12T21:11:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:47 smithi153 ceph-mon[36569]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-12T21:11:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:47 smithi104 ceph-mon[36624]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:47 smithi104 ceph-mon[36624]: from='osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-12T21:11:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:47 smithi104 ceph-mon[36624]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-12T21:11:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:48 smithi153 ceph-mon[36569]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-12T21:11:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:48 smithi153 ceph-mon[36569]: osdmap e70: 11 total, 10 up, 11 in 2024-04-12T21:11:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:48 smithi153 ceph-mon[36569]: from='osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:11:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:48 smithi153 ceph-mon[36569]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:11:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:48 smithi153 ceph-mon[36569]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-mon[36624]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-12T21:11:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-mon[36624]: osdmap e70: 11 total, 10 up, 11 in 2024-04-12T21:11:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-mon[36624]: from='osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:11:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-mon[36624]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:11:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-mon[36624]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:48.828 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-10[60985]: 2024-04-12T21:11:48.707+0000 7f0d22ef4700 -1 osd.10 0 waiting for initial osdmap 2024-04-12T21:11:48.828 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 12 21:11:48 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-10[60985]: 2024-04-12T21:11:48.719+0000 7f0d1ece7700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:11:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:48 smithi081 ceph-mon[31453]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-12T21:11:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:48 smithi081 ceph-mon[31453]: osdmap e70: 11 total, 10 up, 11 in 2024-04-12T21:11:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:48 smithi081 ceph-mon[31453]: from='osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:11:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:48 smithi081 ceph-mon[31453]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-04-12T21:11:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:48 smithi081 ceph-mon[31453]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:49.801 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-12T21:11:49.802 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-12T21:11:49.802 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-04-12T21:11:49.802 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-04-12T21:11:49.802 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0117579 s, 892 MB/s 2024-04-12T21:11:49.802 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-04-12T21:11:49.825 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:49 smithi153 ceph-mon[36569]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:11:49.826 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:49 smithi153 ceph-mon[36569]: osdmap e71: 11 total, 10 up, 11 in 2024-04-12T21:11:49.826 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:49 smithi104 ceph-mon[36624]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:11:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:49 smithi104 ceph-mon[36624]: osdmap e71: 11 total, 10 up, 11 in 2024-04-12T21:11:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:50.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:49 smithi081 ceph-mon[31453]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-04-12T21:11:50.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:49 smithi081 ceph-mon[31453]: osdmap e71: 11 total, 10 up, 11 in 2024-04-12T21:11:50.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:50.455 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-04-12T21:11:50.708 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:11:50.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:50 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:11:50.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:50 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:11:50.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:50.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:50 smithi104 ceph-mon[36624]: osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] boot 2024-04-12T21:11:50.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:50 smithi104 ceph-mon[36624]: osdmap e72: 11 total, 11 up, 11 in 2024-04-12T21:11:50.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:50.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:50 smithi104 ceph-mon[36624]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:51.022 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:50 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:11:51.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:50 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:11:51.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:51.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:50 smithi153 ceph-mon[36569]: osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] boot 2024-04-12T21:11:51.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:50 smithi153 ceph-mon[36569]: osdmap e72: 11 total, 11 up, 11 in 2024-04-12T21:11:51.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:51.023 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:50 smithi153 ceph-mon[36569]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:51.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:50 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:11:51.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:50 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:11:51.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:51.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:50 smithi081 ceph-mon[31453]: osd.10 [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] boot 2024-04-12T21:11:51.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:50 smithi081 ceph-mon[31453]: osdmap e72: 11 total, 11 up, 11 in 2024-04-12T21:11:51.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-12T21:11:51.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:50 smithi081 ceph-mon[31453]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-04-12T21:11:52.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: osdmap e73: 11 total, 11 up, 11 in 2024-04-12T21:11:52.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi104 2024-04-12T21:11:52.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 4270M 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: osdmap e73: 11 total, 11 up, 11 in 2024-04-12T21:11:52.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi104 2024-04-12T21:11:52.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 4270M 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: osdmap e73: 11 total, 11 up, 11 in 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi104 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 4270M 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:52.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:11:52.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:52.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:11:52.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:11:52.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 -- 172.21.15.153:0/247314552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2928103d20 msgr2=0x7f2928104180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 --2- 172.21.15.153:0/247314552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2928103d20 0x7f2928104180 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2918009990 tx=0x7f2918009ca0 comp rx=0 tx=0).stop 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 -- 172.21.15.153:0/247314552 shutdown_connections 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 --2- 172.21.15.153:0/247314552 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29281046c0 0x7f292810af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 --2- 172.21.15.153:0/247314552 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2928102b20 0x7f2928102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 --2- 172.21.15.153:0/247314552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2928103d20 0x7f2928104180 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 -- 172.21.15.153:0/247314552 >> 172.21.15.153:0/247314552 conn(0x7f29280fe0c0 msgr2=0x7f2928100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 -- 172.21.15.153:0/247314552 shutdown_connections 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.329+0000 7f292ca5c700 1 -- 172.21.15.153:0/247314552 wait complete. 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292ca5c700 1 Processor -- start 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292ca5c700 1 -- start start 2024-04-12T21:11:52.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292ca5c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2928102b20 0x7f2928198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292ca5c700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2928103d20 0x7f29281992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292ca5c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29281046c0 0x7f29281a0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292ca5c700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f292810e7a0 con 0x7f2928103d20 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292ca5c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f292810e620 con 0x7f2928102b20 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292ca5c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f292810e920 con 0x7f29281046c0 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f2926d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29281046c0 0x7f29281a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f2926d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29281046c0 0x7f29281a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34206/0 (socket says 172.21.15.153:34206) 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f2926d9d700 1 -- 172.21.15.153:0/2558570177 learned_addr learned my addr 172.21.15.153:0/2558570177 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f292659c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2928102b20 0x7f2928198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f2926d9d700 1 -- 172.21.15.153:0/2558570177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2928102b20 msgr2=0x7f2928198da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f2926d9d700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2928102b20 0x7f2928198da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f2926d9d700 1 -- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2928103d20 msgr2=0x7f29281992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:11:52.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f2926d9d700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2928103d20 0x7f29281992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.330+0000 7f2926d9d700 1 -- 172.21.15.153:0/2558570177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29281a09f0 con 0x7f29281046c0 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.331+0000 7f292659c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2928102b20 0x7f2928198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.331+0000 7f2926d9d700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29281046c0 0x7f29281a0320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f292000ada0 tx=0x7f292000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.331+0000 7f291f7fe700 1 -- 172.21.15.153:0/2558570177 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920017020 con 0x7f29281046c0 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.331+0000 7f291f7fe700 1 -- 172.21.15.153:0/2558570177 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2920004bb0 con 0x7f29281046c0 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.331+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29281a0ce0 con 0x7f29281046c0 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.331+0000 7f291f7fe700 1 -- 172.21.15.153:0/2558570177 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2920005520 con 0x7f29281046c0 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.331+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29281a1290 con 0x7f29281046c0 2024-04-12T21:11:52.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.332+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f292802dfa0 con 0x7f29281046c0 2024-04-12T21:11:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.333+0000 7f291f7fe700 1 -- 172.21.15.153:0/2558570177 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f29200056c0 con 0x7f29281046c0 2024-04-12T21:11:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.334+0000 7f291f7fe700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29140a3cc0 0x7f29140a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:11:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.334+0000 7f291f7fe700 1 -- 172.21.15.153:0/2558570177 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f29200cfaa0 con 0x7f29281046c0 2024-04-12T21:11:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.334+0000 7f292659c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29140a3cc0 0x7f29140a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:11:52.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.335+0000 7f292659c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29140a3cc0 0x7f29140a6180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f292819a280 tx=0x7f2910008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:11:52.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.338+0000 7f291f7fe700 1 -- 172.21.15.153:0/2558570177 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2920021020 con 0x7f29281046c0 2024-04-12T21:11:52.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:11:52.547+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2928062ef0 con 0x7f29140a3cc0 2024-04-12T21:11:53.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:52 smithi153 ceph-mon[36569]: osdmap e74: 11 total, 11 up, 11 in 2024-04-12T21:11:53.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:52 smithi153 ceph-mon[36569]: osdmap e75: 11 total, 11 up, 11 in 2024-04-12T21:11:53.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:52 smithi153 ceph-mon[36569]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:11:53.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:11:53.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:11:53.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:52 smithi081 ceph-mon[31453]: osdmap e74: 11 total, 11 up, 11 in 2024-04-12T21:11:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:52 smithi081 ceph-mon[31453]: osdmap e75: 11 total, 11 up, 11 in 2024-04-12T21:11:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:52 smithi081 ceph-mon[31453]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:11:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:11:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:11:53.054 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:52 smithi104 ceph-mon[36624]: osdmap e74: 11 total, 11 up, 11 in 2024-04-12T21:11:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:52 smithi104 ceph-mon[36624]: osdmap e75: 11 total, 11 up, 11 in 2024-04-12T21:11:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:52 smithi104 ceph-mon[36624]: pgmap v260: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:11:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-12T21:11:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-12T21:11:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:11:54.128 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:54 smithi153 ceph-mon[36569]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:11:54.128 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:54 smithi153 ceph-mon[36569]: osdmap e76: 11 total, 11 up, 11 in 2024-04-12T21:11:54.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:54 smithi081 ceph-mon[31453]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:11:54.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:54 smithi081 ceph-mon[31453]: osdmap e76: 11 total, 11 up, 11 in 2024-04-12T21:11:54.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:54 smithi104 ceph-mon[36624]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:11:54.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:54 smithi104 ceph-mon[36624]: osdmap e76: 11 total, 11 up, 11 in 2024-04-12T21:11:55.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:55 smithi081 ceph-mon[31453]: pgmap v262: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:11:55.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:55 smithi153 ceph-mon[36569]: pgmap v262: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:11:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:55 smithi104 ceph-mon[36624]: pgmap v262: 1 pgs: 1 remapped+peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:11:56.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:56 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/4144345345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]: dispatch 2024-04-12T21:11:56.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:56 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]: dispatch 2024-04-12T21:11:56.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:56 smithi081 ceph-mon[31453]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]': finished 2024-04-12T21:11:56.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:56 smithi081 ceph-mon[31453]: osdmap e77: 12 total, 11 up, 12 in 2024-04-12T21:11:56.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:11:56.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:56 smithi081 ceph-mon[31453]: from='client.? 172.21.15.153:0/841579812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:11:56.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:56 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/4144345345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]: dispatch 2024-04-12T21:11:56.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:56 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]: dispatch 2024-04-12T21:11:56.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:56 smithi153 ceph-mon[36569]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]': finished 2024-04-12T21:11:56.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:56 smithi153 ceph-mon[36569]: osdmap e77: 12 total, 11 up, 12 in 2024-04-12T21:11:56.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:11:56.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:56 smithi153 ceph-mon[36569]: from='client.? 172.21.15.153:0/841579812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:11:56.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:56 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/4144345345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]: dispatch 2024-04-12T21:11:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:56 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]: dispatch 2024-04-12T21:11:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:56 smithi104 ceph-mon[36624]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2"}]': finished 2024-04-12T21:11:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:56 smithi104 ceph-mon[36624]: osdmap e77: 12 total, 11 up, 12 in 2024-04-12T21:11:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:11:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:56 smithi104 ceph-mon[36624]: from='client.? 172.21.15.153:0/841579812' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-12T21:11:57.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:57 smithi081 ceph-mon[31453]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-12T21:11:57.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:57 smithi153 ceph-mon[36569]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-12T21:11:57.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:57 smithi104 ceph-mon[36624]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-04-12T21:11:59.529 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:11:59 smithi153 ceph-mon[36569]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-12T21:11:59.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:11:59 smithi081 ceph-mon[31453]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-12T21:11:59.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:11:59 smithi104 ceph-mon[36624]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-04-12T21:12:01.504 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:01 smithi153 ceph-mon[36569]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:12:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:01 smithi081 ceph-mon[31453]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:12:01.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:01 smithi104 ceph-mon[36624]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-12T21:12:03.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:03 smithi153 ceph-mon[36569]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-12T21:12:03.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-12T21:12:03.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:03 smithi081 ceph-mon[31453]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-12T21:12:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-12T21:12:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:03 smithi104 ceph-mon[36624]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-12T21:12:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-12T21:12:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:04.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:04 smithi153 ceph-mon[36569]: Deploying daemon osd.11 on smithi153 2024-04-12T21:12:04.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:04 smithi081 ceph-mon[31453]: Deploying daemon osd.11 on smithi153 2024-04-12T21:12:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:04 smithi104 ceph-mon[36624]: Deploying daemon osd.11 on smithi153 2024-04-12T21:12:05.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:05 smithi081 ceph-mon[31453]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:12:05.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:05 smithi153 ceph-mon[36569]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:12:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:05 smithi104 ceph-mon[36624]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-12T21:12:07.625 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:07 smithi153 ceph-mon[36569]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-12T21:12:07.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:07 smithi081 ceph-mon[31453]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-12T21:12:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:07 smithi104 ceph-mon[36624]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-12T21:12:09.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:09.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:09.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:09.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:09 smithi153 ceph-mon[36569]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:09.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:09.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:09.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:09.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:09 smithi104 ceph-mon[36624]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:09.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:09 smithi081 ceph-mon[31453]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:11.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:11 smithi081 ceph-mon[31453]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:11.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:11.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:11.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:11.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:11.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:11.738 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:11 smithi153 ceph-mon[36569]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:11.739 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:11.739 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:11.739 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:11.739 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:11.739 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:11.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:11 smithi104 ceph-mon[36624]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:12.127 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 11 on host 'smithi153' 2024-04-12T21:12:12.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.130+0000 7f291f7fe700 1 -- 172.21.15.153:0/2558570177 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2928062ef0 con 0x7f29140a3cc0 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.132+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29140a3cc0 msgr2=0x7f29140a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.132+0000 7f292ca5c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29140a3cc0 0x7f29140a6180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f292819a280 tx=0x7f2910008040 comp rx=0 tx=0).stop 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.132+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29281046c0 msgr2=0x7f29281a0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.132+0000 7f292ca5c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29281046c0 0x7f29281a0320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f292000ada0 tx=0x7f292000c330 comp rx=0 tx=0).stop 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.133+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 shutdown_connections 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.133+0000 7f292ca5c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29140a3cc0 0x7f29140a6180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.133+0000 7f292ca5c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29281046c0 0x7f29281a0320 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.133+0000 7f292ca5c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2928102b20 0x7f2928198da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.133+0000 7f292ca5c700 1 --2- 172.21.15.153:0/2558570177 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2928103d20 0x7f29281992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.133+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 >> 172.21.15.153:0/2558570177 conn(0x7f29280fe0c0 msgr2=0x7f2928106fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.133+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 shutdown_connections 2024-04-12T21:12:12.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:12.133+0000 7f292ca5c700 1 -- 172.21.15.153:0/2558570177 wait complete. 2024-04-12T21:12:12.833 DEBUG:teuthology.orchestra.run.smithi153:osd.11> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@osd.11.service 2024-04-12T21:12:12.835 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-12T21:12:12.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd stat -f json 2024-04-12T21:12:13.294 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:12:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:13 smithi081 ceph-mon[31453]: from='osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-12T21:12:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:13 smithi081 ceph-mon[31453]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-12T21:12:13.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:13 smithi081 ceph-mon[31453]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:13 smithi153 ceph-mon[36569]: from='osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-12T21:12:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:13 smithi153 ceph-mon[36569]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-12T21:12:13.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:13 smithi153 ceph-mon[36569]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:13.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:13.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:13 smithi104 ceph-mon[36624]: from='osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-12T21:12:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:13 smithi104 ceph-mon[36624]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-12T21:12:13.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:13 smithi104 ceph-mon[36624]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:14.226 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:14 smithi153 ceph-mon[36569]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-12T21:12:14.226 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:14 smithi153 ceph-mon[36569]: from='osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:12:14.226 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:14 smithi153 ceph-mon[36569]: osdmap e78: 12 total, 11 up, 12 in 2024-04-12T21:12:14.227 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:14.227 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:14 smithi153 ceph-mon[36569]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:12:14.227 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 12 21:12:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-11[62488]: 2024-04-12T21:12:14.134+0000 7f2137915700 -1 osd.11 0 waiting for initial osdmap 2024-04-12T21:12:14.227 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 12 21:12:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-osd-11[62488]: 2024-04-12T21:12:14.148+0000 7f2133708700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-12T21:12:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:14 smithi081 ceph-mon[31453]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-12T21:12:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:14 smithi081 ceph-mon[31453]: from='osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:12:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:14 smithi081 ceph-mon[31453]: osdmap e78: 12 total, 11 up, 12 in 2024-04-12T21:12:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:14 smithi081 ceph-mon[31453]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:12:14.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:14 smithi104 ceph-mon[36624]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-12T21:12:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:14 smithi104 ceph-mon[36624]: from='osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:12:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:14 smithi104 ceph-mon[36624]: osdmap e78: 12 total, 11 up, 12 in 2024-04-12T21:12:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:14 smithi104 ceph-mon[36624]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-04-12T21:12:15.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:15 smithi081 ceph-mon[31453]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:12:15.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:15 smithi081 ceph-mon[31453]: osdmap e79: 12 total, 11 up, 12 in 2024-04-12T21:12:15.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:15.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:15 smithi081 ceph-mon[31453]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:15.236 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:15.375 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:15 smithi153 ceph-mon[36569]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:12:15.375 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:15 smithi153 ceph-mon[36569]: osdmap e79: 12 total, 11 up, 12 in 2024-04-12T21:12:15.375 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:15.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:15 smithi153 ceph-mon[36569]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:15.376 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 -- 172.21.15.81:0/3820520027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 msgr2=0x7fc56010ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 --2- 172.21.15.81:0/3820520027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 0x7fc56010ac70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc558009f90 tx=0x7fc55800b330 comp rx=0 tx=0).stop 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 -- 172.21.15.81:0/3820520027 shutdown_connections 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 --2- 172.21.15.81:0/3820520027 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 0x7fc56010ac70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 --2- 172.21.15.81:0/3820520027 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5601035a0 0x7fc560105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 --2- 172.21.15.81:0/3820520027 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc560105ed0 0x7fc5601082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 -- 172.21.15.81:0/3820520027 >> 172.21.15.81:0/3820520027 conn(0x7fc5600fcfb0 msgr2=0x7fc5600ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 -- 172.21.15.81:0/3820520027 shutdown_connections 2024-04-12T21:12:15.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.398+0000 7fc565800700 1 -- 172.21.15.81:0/3820520027 wait complete. 2024-04-12T21:12:15.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.399+0000 7fc565800700 1 Processor -- start 2024-04-12T21:12:15.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.399+0000 7fc565800700 1 -- start start 2024-04-12T21:12:15.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.399+0000 7fc565800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5601035a0 0x7fc56019af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc565800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc560105ed0 0x7fc56019b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc565800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 0x7fc5601a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc565800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010e740 con 0x7fc5601035a0 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc565800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010e5c0 con 0x7fc560108800 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc565800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010e8c0 con 0x7fc560105ed0 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5601035a0 0x7fc56019af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55e000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc560105ed0 0x7fc56019b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5601035a0 0x7fc56019af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:47558/0 (socket says 172.21.15.81:47558) 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55e000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc560105ed0 0x7fc56019b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:34126/0 (socket says 172.21.15.81:34126) 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55ea00700 1 -- 172.21.15.81:0/806245975 learned_addr learned my addr 172.21.15.81:0/806245975 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:15.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55f400700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 0x7fc5601a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:15.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55ea00700 1 -- 172.21.15.81:0/806245975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc560105ed0 msgr2=0x7fc56019b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:15.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55ea00700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc560105ed0 0x7fc56019b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55ea00700 1 -- 172.21.15.81:0/806245975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 msgr2=0x7fc5601a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55ea00700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 0x7fc5601a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55ea00700 1 -- 172.21.15.81:0/806245975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5601a2be0 con 0x7fc5601035a0 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55e000700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc560105ed0 0x7fc56019b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.400+0000 7fc55f400700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 0x7fc5601a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.401+0000 7fc55ea00700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5601035a0 0x7fc56019af80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e9f0 tx=0x7fc55000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.401+0000 7fc557400700 1 -- 172.21.15.81:0/806245975 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc550004a30 con 0x7fc5601035a0 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.401+0000 7fc557400700 1 -- 172.21.15.81:0/806245975 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc550005500 con 0x7fc5601035a0 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.401+0000 7fc557400700 1 -- 172.21.15.81:0/806245975 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55000cd60 con 0x7fc5601035a0 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.401+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5601a2ed0 con 0x7fc5601035a0 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.401+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5601a3320 con 0x7fc5601035a0 2024-04-12T21:12:15.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.403+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56002dfa0 con 0x7fc5601035a0 2024-04-12T21:12:15.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.405+0000 7fc557400700 1 -- 172.21.15.81:0/806245975 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc550004c60 con 0x7fc5601035a0 2024-04-12T21:12:15.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.406+0000 7fc557400700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5440a3c40 0x7fc5440a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:15.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.406+0000 7fc557400700 1 -- 172.21.15.81:0/806245975 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7fc5500d0160 con 0x7fc5601035a0 2024-04-12T21:12:15.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.406+0000 7fc55e000700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5440a3c40 0x7fc5440a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:15.403 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.407+0000 7fc55e000700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5440a3c40 0x7fc5440a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc54c011490 tx=0x7fc54c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:15.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.411+0000 7fc557400700 1 -- 172.21.15.81:0/806245975 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc55009cf40 con 0x7fc5601035a0 2024-04-12T21:12:15.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:15 smithi104 ceph-mon[36624]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-04-12T21:12:15.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:15 smithi104 ceph-mon[36624]: osdmap e79: 12 total, 11 up, 12 in 2024-04-12T21:12:15.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:15.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:15 smithi104 ceph-mon[36624]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-04-12T21:12:15.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:15.640 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.644+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fc5600637c0 con 0x7fc5601035a0 2024-04-12T21:12:15.641 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.645+0000 7fc557400700 1 -- 172.21.15.81:0/806245975 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7fc55009c840 con 0x7fc5601035a0 2024-04-12T21:12:15.641 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:15.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.647+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5440a3c40 msgr2=0x7fc5440a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:15.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.647+0000 7fc565800700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5440a3c40 0x7fc5440a6100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc54c011490 tx=0x7fc54c009260 comp rx=0 tx=0).stop 2024-04-12T21:12:15.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.647+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5601035a0 msgr2=0x7fc56019af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:15.643 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.647+0000 7fc565800700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5601035a0 0x7fc56019af80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e9f0 tx=0x7fc55000ed00 comp rx=0 tx=0).stop 2024-04-12T21:12:15.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.648+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 shutdown_connections 2024-04-12T21:12:15.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.648+0000 7fc565800700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5440a3c40 0x7fc5440a6100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.648+0000 7fc565800700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc560108800 0x7fc5601a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.648+0000 7fc565800700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc5601035a0 0x7fc56019af80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.648+0000 7fc565800700 1 --2- 172.21.15.81:0/806245975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc560105ed0 0x7fc56019b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:15.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.648+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 >> 172.21.15.81:0/806245975 conn(0x7fc5600fcfb0 msgr2=0x7fc560102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:15.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.648+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 shutdown_connections 2024-04-12T21:12:15.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:15.648+0000 7fc565800700 1 -- 172.21.15.81:0/806245975 wait complete. 2024-04-12T21:12:16.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: purged_snaps scrub starts 2024-04-12T21:12:16.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: purged_snaps scrub ok 2024-04-12T21:12:16.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] boot 2024-04-12T21:12:16.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: osdmap e80: 12 total, 12 up, 12 in 2024-04-12T21:12:16.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/806245975' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.504 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1712956335,"num_in_osds":12,"osd_in_since":1712956315,"num_remapped_pgs":0} 2024-04-12T21:12:16.505 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-12T21:12:16.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: purged_snaps scrub starts 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: purged_snaps scrub ok 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] boot 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: osdmap e80: 12 total, 12 up, 12 in 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/806245975' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:16.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:16.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: purged_snaps scrub starts 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: purged_snaps scrub ok 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: osd.11 [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] boot 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: osdmap e80: 12 total, 12 up, 12 in 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/806245975' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:12:16.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:16.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:16.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:16.938 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:12:17.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:17 smithi081 ceph-mon[31453]: Detected new or changed devices on smithi153 2024-04-12T21:12:17.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:17 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 4270M 2024-04-12T21:12:17.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:17 smithi081 ceph-mon[31453]: osdmap e81: 12 total, 12 up, 12 in 2024-04-12T21:12:17.340 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:17 smithi081 ceph-mon[31453]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:17 smithi153 ceph-mon[36569]: Detected new or changed devices on smithi153 2024-04-12T21:12:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:17 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 4270M 2024-04-12T21:12:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:17 smithi153 ceph-mon[36569]: osdmap e81: 12 total, 12 up, 12 in 2024-04-12T21:12:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:17 smithi153 ceph-mon[36569]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:17 smithi104 ceph-mon[36624]: Detected new or changed devices on smithi153 2024-04-12T21:12:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:17 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 4270M 2024-04-12T21:12:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:17 smithi104 ceph-mon[36624]: osdmap e81: 12 total, 12 up, 12 in 2024-04-12T21:12:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:17 smithi104 ceph-mon[36624]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.009+0000 7f07b6600700 1 -- 172.21.15.81:0/1356175429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0109f00 msgr2=0x7f07b0111ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.009+0000 7f07b6600700 1 --2- 172.21.15.81:0/1356175429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0109f00 0x7f07b0111ad0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f07a800b0d0 tx=0x7f07a800b3e0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.009+0000 7f07b6600700 1 -- 172.21.15.81:0/1356175429 shutdown_connections 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.009+0000 7f07b6600700 1 --2- 172.21.15.81:0/1356175429 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0109f00 0x7f07b0111ad0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.009+0000 7f07b6600700 1 --2- 172.21.15.81:0/1356175429 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07b0104e20 0x7f07b0105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.009+0000 7f07b6600700 1 --2- 172.21.15.81:0/1356175429 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b01057d0 0x7f07b0109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.009+0000 7f07b6600700 1 -- 172.21.15.81:0/1356175429 >> 172.21.15.81:0/1356175429 conn(0x7f07b01006f0 msgr2=0x7f07b0102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.010+0000 7f07b6600700 1 -- 172.21.15.81:0/1356175429 shutdown_connections 2024-04-12T21:12:19.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.010+0000 7f07b6600700 1 -- 172.21.15.81:0/1356175429 wait complete. 2024-04-12T21:12:19.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.011+0000 7f07b6600700 1 Processor -- start 2024-04-12T21:12:19.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.011+0000 7f07b6600700 1 -- start start 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07b6600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0104e20 0x7f07b019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07b6600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07b01057d0 0x7f07b019b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07b6600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b0109f00 0x7f07b019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07b6600700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0114f00 con 0x7f07b0104e20 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07b6600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0114d80 con 0x7f07b01057d0 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07b6600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0115080 con 0x7f07b0109f00 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07b5200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0104e20 0x7f07b019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07afe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07b01057d0 0x7f07b019b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07afe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07b01057d0 0x7f07b019b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:39124/0 (socket says 172.21.15.81:39124) 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07afe00700 1 -- 172.21.15.81:0/476368149 learned_addr learned my addr 172.21.15.81:0/476368149 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:19.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07b5c00700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b0109f00 0x7f07b019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:19.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07afe00700 1 -- 172.21.15.81:0/476368149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b0109f00 msgr2=0x7f07b019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07afe00700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b0109f00 0x7f07b019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07afe00700 1 -- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0104e20 msgr2=0x7f07b019b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07afe00700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0104e20 0x7f07b019b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.012+0000 7f07afe00700 1 -- 172.21.15.81:0/476368149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b01a0310 con 0x7f07b01057d0 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.013+0000 7f07b5200700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0104e20 0x7f07b019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.013+0000 7f07afe00700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07b01057d0 0x7f07b019b890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f079c00e9f0 tx=0x7f079c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.013+0000 7f07ad600700 1 -- 172.21.15.81:0/476368149 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c004a30 con 0x7f07b01057d0 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.013+0000 7f07ad600700 1 -- 172.21.15.81:0/476368149 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f079c00cca0 con 0x7f07b01057d0 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.013+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b01a0600 con 0x7f07b01057d0 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.013+0000 7f07ad600700 1 -- 172.21.15.81:0/476368149 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079c0056c0 con 0x7f07b01057d0 2024-04-12T21:12:19.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.013+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b01a7e80 con 0x7f07b01057d0 2024-04-12T21:12:19.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.014+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b00697a0 con 0x7f07b01057d0 2024-04-12T21:12:19.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.016+0000 7f07ad600700 1 -- 172.21.15.81:0/476368149 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f079c00c7c0 con 0x7f07b01057d0 2024-04-12T21:12:19.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.018+0000 7f07ad600700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f07980a3c40 0x7f07980a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:19.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.018+0000 7f07ad600700 1 -- 172.21.15.81:0/476368149 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f079c0d0100 con 0x7f07b01057d0 2024-04-12T21:12:19.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.018+0000 7f07b5200700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f07980a3c40 0x7f07980a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:19.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.019+0000 7f07b5200700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f07980a3c40 0x7f07980a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f07a4011490 tx=0x7f07a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:19.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.020+0000 7f07ad600700 1 -- 172.21.15.81:0/476368149 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f079c09cee0 con 0x7f07b01057d0 2024-04-12T21:12:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.244+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07b019c660 con 0x7f07b01057d0 2024-04-12T21:12:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.245+0000 7f07ad600700 1 -- 172.21.15.81:0/476368149 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18480 (secure 0 0 0) 0x7f079c09c7e0 con 0x7f07b01057d0 2024-04-12T21:12:19.242 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:19.242 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":81,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:12:16.153549+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.248+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f07980a3c40 msgr2=0x7f07980a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.248+0000 7f07b6600700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f07980a3c40 0x7f07980a6100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f07a4011490 tx=0x7f07a4009260 comp rx=0 tx=0).stop 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07b01057d0 msgr2=0x7f07b019b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07b01057d0 0x7f07b019b890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f079c00e9f0 tx=0x7f079c00ed00 comp rx=0 tx=0).stop 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 shutdown_connections 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f07980a3c40 0x7f07980a6100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07b0109f00 0x7f07b019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f07b0104e20 0x7f07b019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 --2- 172.21.15.81:0/476368149 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f07b01057d0 0x7f07b019b890 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 >> 172.21.15.81:0/476368149 conn(0x7f07b01006f0 msgr2=0x7f07b0101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:19.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 shutdown_connections 2024-04-12T21:12:19.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:19.249+0000 7f07b6600700 1 -- 172.21.15.81:0/476368149 wait complete. 2024-04-12T21:12:19.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:19 smithi081 ceph-mon[31453]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:19.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:19 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/476368149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:12:19.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:19 smithi153 ceph-mon[36569]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:19.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:19 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/476368149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:12:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:19 smithi104 ceph-mon[36624]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:19.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:19 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/476368149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:12:20.009 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T21:12:20.009 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-12T21:12:20.451 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:12:21.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:21 smithi081 ceph-mon[31453]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:21 smithi153 ceph-mon[36569]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:21 smithi104 ceph-mon[36624]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 -- 172.21.15.81:0/1059405368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b6410f550 msgr2=0x7f8b64111940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 --2- 172.21.15.81:0/1059405368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b6410f550 0x7f8b64111940 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c00b0a0 tx=0x7f8b5c00b3b0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 -- 172.21.15.81:0/1059405368 shutdown_connections 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 --2- 172.21.15.81:0/1059405368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b6410f550 0x7f8b64111940 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 --2- 172.21.15.81:0/1059405368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6406b050 0x7f8b6406b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 --2- 172.21.15.81:0/1059405368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b6406b970 0x7f8b6410f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 -- 172.21.15.81:0/1059405368 >> 172.21.15.81:0/1059405368 conn(0x7f8b64078920 msgr2=0x7f8b64078d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 -- 172.21.15.81:0/1059405368 shutdown_connections 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.601+0000 7f8b6a600700 1 -- 172.21.15.81:0/1059405368 wait complete. 2024-04-12T21:12:22.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.602+0000 7f8b6a600700 1 Processor -- start 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.602+0000 7f8b6a600700 1 -- start start 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b6a600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b6406b050 0x7f8b6419b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b6a600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b6406b970 0x7f8b6419b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b6a600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6410f550 0x7f8b6419fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b6a600700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64114fa0 con 0x7f8b6406b970 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b6a600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64114e20 con 0x7f8b6406b050 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b6a600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b64115120 con 0x7f8b6410f550 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b63400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b6406b050 0x7f8b6419b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:22.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b63400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b6406b050 0x7f8b6419b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:39138/0 (socket says 172.21.15.81:39138) 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b63400700 1 -- 172.21.15.81:0/4105080565 learned_addr learned my addr 172.21.15.81:0/4105080565 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b63e00700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6410f550 0x7f8b6419fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b62a00700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b6406b970 0x7f8b6419b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.603+0000 7f8b63e00700 1 -- 172.21.15.81:0/4105080565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b6406b050 msgr2=0x7f8b6419b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b63e00700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b6406b050 0x7f8b6419b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b63e00700 1 -- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b6406b970 msgr2=0x7f8b6419b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b63e00700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b6406b970 0x7f8b6419b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b63e00700 1 -- 172.21.15.81:0/4105080565 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b641a0440 con 0x7f8b6410f550 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b63400700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b6406b050 0x7f8b6419b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b63e00700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6410f550 0x7f8b6419fd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8b6406a550 tx=0x7f8b5c004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b5be00700 1 -- 172.21.15.81:0/4105080565 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c021070 con 0x7f8b6410f550 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b5be00700 1 -- 172.21.15.81:0/4105080565 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b5c007b80 con 0x7f8b6410f550 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b5be00700 1 -- 172.21.15.81:0/4105080565 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c005040 con 0x7f8b6410f550 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b641a06d0 con 0x7f8b6410f550 2024-04-12T21:12:22.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.604+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b641a7fb0 con 0x7f8b6410f550 2024-04-12T21:12:22.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.605+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b64101fc0 con 0x7f8b6410f550 2024-04-12T21:12:22.603 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.608+0000 7f8b5be00700 1 -- 172.21.15.81:0/4105080565 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8b5c024070 con 0x7f8b6410f550 2024-04-12T21:12:22.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.609+0000 7f8b5be00700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8b480a3c40 0x7f8b480a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:22.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.609+0000 7f8b5be00700 1 -- 172.21.15.81:0/4105080565 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f8b5c00f020 con 0x7f8b6410f550 2024-04-12T21:12:22.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.609+0000 7f8b63400700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8b480a3c40 0x7f8b480a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:22.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.610+0000 7f8b63400700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8b480a3c40 0x7f8b480a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b54011550 tx=0x7f8b54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:22.607 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.612+0000 7f8b5be00700 1 -- 172.21.15.81:0/4105080565 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b5c02b020 con 0x7f8b6410f550 2024-04-12T21:12:22.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.839+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8b6402dfa0 con 0x7f8b6410f550 2024-04-12T21:12:22.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.839+0000 7f8b5be00700 1 -- 172.21.15.81:0/4105080565 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f8b5c007d30 con 0x7f8b6410f550 2024-04-12T21:12:22.836 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:12:22.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.843+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8b480a3c40 msgr2=0x7f8b480a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.843+0000 7f8b6a600700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8b480a3c40 0x7f8b480a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b54011550 tx=0x7f8b54009260 comp rx=0 tx=0).stop 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.843+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6410f550 msgr2=0x7f8b6419fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.843+0000 7f8b6a600700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6410f550 0x7f8b6419fd00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8b6406a550 tx=0x7f8b5c004d00 comp rx=0 tx=0).stop 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.844+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 shutdown_connections 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.844+0000 7f8b6a600700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8b480a3c40 0x7f8b480a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.844+0000 7f8b6a600700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b6410f550 0x7f8b6419fd00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.844+0000 7f8b6a600700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b6406b050 0x7f8b6419b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.844+0000 7f8b6a600700 1 --2- 172.21.15.81:0/4105080565 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b6406b970 0x7f8b6419b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.844+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 >> 172.21.15.81:0/4105080565 conn(0x7f8b64078920 msgr2=0x7f8b64110a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.844+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 shutdown_connections 2024-04-12T21:12:22.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:22.844+0000 7f8b6a600700 1 -- 172.21.15.81:0/4105080565 wait complete. 2024-04-12T21:12:23.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:23 smithi081 ceph-mon[31453]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:23.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:23 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4105080565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:12:23.665 INFO:tasks.cephadm:Adding mds.a on smithi081 2024-04-12T21:12:23.665 INFO:tasks.cephadm:Adding mds.d on smithi081 2024-04-12T21:12:23.665 INFO:tasks.cephadm:Adding mds.g on smithi081 2024-04-12T21:12:23.665 INFO:tasks.cephadm:Adding mds.j on smithi081 2024-04-12T21:12:23.665 INFO:tasks.cephadm:Adding mds.b on smithi104 2024-04-12T21:12:23.665 INFO:tasks.cephadm:Adding mds.e on smithi104 2024-04-12T21:12:23.666 INFO:tasks.cephadm:Adding mds.h on smithi104 2024-04-12T21:12:23.666 INFO:tasks.cephadm:Adding mds.k on smithi104 2024-04-12T21:12:23.666 INFO:tasks.cephadm:Adding mds.c on smithi153 2024-04-12T21:12:23.666 INFO:tasks.cephadm:Adding mds.f on smithi153 2024-04-12T21:12:23.666 INFO:tasks.cephadm:Adding mds.i on smithi153 2024-04-12T21:12:23.666 INFO:tasks.cephadm:Adding mds.l on smithi153 2024-04-12T21:12:23.666 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph orch apply mds all '12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l' 2024-04-12T21:12:23.696 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:23 smithi153 ceph-mon[36569]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:23.697 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:23 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4105080565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:12:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:23 smithi104 ceph-mon[36624]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:23 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4105080565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:12:23.917 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.c/config 2024-04-12T21:12:25.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 -- 172.21.15.153:0/2148222599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0103d40 msgr2=0x7f3fb01041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 --2- 172.21.15.153:0/2148222599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0103d40 0x7f3fb01041a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3fac009990 tx=0x7f3fac009ca0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 -- 172.21.15.153:0/2148222599 shutdown_connections 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 --2- 172.21.15.153:0/2148222599 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb01046e0 0x7f3fb010af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 --2- 172.21.15.153:0/2148222599 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb0102b40 0x7f3fb0102f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 --2- 172.21.15.153:0/2148222599 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0103d40 0x7f3fb01041a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 -- 172.21.15.153:0/2148222599 >> 172.21.15.153:0/2148222599 conn(0x7f3fb00fe100 msgr2=0x7f3fb0100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 -- 172.21.15.153:0/2148222599 shutdown_connections 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.571+0000 7f3fb7417700 1 -- 172.21.15.153:0/2148222599 wait complete. 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb7417700 1 Processor -- start 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb7417700 1 -- start start 2024-04-12T21:12:25.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb7417700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0102b40 0x7f3fb0198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb7417700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb0103d40 0x7f3fb0199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb7417700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb01046e0 0x7f3fb01a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb7417700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb010e820 con 0x7f3fb0103d40 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb7417700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb010e6a0 con 0x7f3fb01046e0 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb7417700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb010e9a0 con 0x7f3fb0102b40 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb51b3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0102b40 0x7f3fb0198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb51b3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0102b40 0x7f3fb0198ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33736/0 (socket says 172.21.15.153:33736) 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb51b3700 1 -- 172.21.15.153:0/1228590608 learned_addr learned my addr 172.21.15.153:0/1228590608 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb59b4700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb01046e0 0x7f3fb01a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb51b3700 1 -- 172.21.15.153:0/1228590608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb01046e0 msgr2=0x7f3fb01a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb51b3700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb01046e0 0x7f3fb01a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb51b3700 1 -- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb0103d40 msgr2=0x7f3fb0199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb51b3700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb0103d40 0x7f3fb0199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.572+0000 7f3fb51b3700 1 -- 172.21.15.153:0/1228590608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb01a0ad0 con 0x7f3fb0102b40 2024-04-12T21:12:25.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.573+0000 7f3fb51b3700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0102b40 0x7f3fb0198ec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000b530 tx=0x7f3fa000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:25.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.573+0000 7f3fa67fc700 1 -- 172.21.15.153:0/1228590608 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa0015020 con 0x7f3fb0102b40 2024-04-12T21:12:25.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.573+0000 7f3fa67fc700 1 -- 172.21.15.153:0/1228590608 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fa0011d10 con 0x7f3fb0102b40 2024-04-12T21:12:25.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.573+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fb01a0dc0 con 0x7f3fb0102b40 2024-04-12T21:12:25.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.573+0000 7f3fa67fc700 1 -- 172.21.15.153:0/1228590608 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa000f2e0 con 0x7f3fb0102b40 2024-04-12T21:12:25.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.573+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fb01a12f0 con 0x7f3fb0102b40 2024-04-12T21:12:25.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.574+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fb002dfa0 con 0x7f3fb0102b40 2024-04-12T21:12:25.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.575+0000 7f3fa67fc700 1 -- 172.21.15.153:0/1228590608 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3fa0018020 con 0x7f3fb0102b40 2024-04-12T21:12:25.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.575+0000 7f3fa67fc700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3f9c0a3cc0 0x7f3f9c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:25.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.575+0000 7f3fa67fc700 1 -- 172.21.15.153:0/1228590608 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3fa00cf6c0 con 0x7f3fb0102b40 2024-04-12T21:12:25.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.576+0000 7f3fb49b2700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3f9c0a3cc0 0x7f3f9c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:25.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.577+0000 7f3fb49b2700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3f9c0a3cc0 0x7f3f9c0a6180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3fac005f50 tx=0x7f3fac005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:25.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.580+0000 7f3fa67fc700 1 -- 172.21.15.153:0/1228590608 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3fa001f020 con 0x7f3fb0102b40 2024-04-12T21:12:25.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:25 smithi081 ceph-mon[31453]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:25.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.785+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}) -- 0x7f3fb0062ff0 con 0x7f3f9c0a3cc0 2024-04-12T21:12:25.781 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:25 smithi153 ceph-mon[36569]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:25.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.802+0000 7f3fa67fc700 1 -- 172.21.15.153:0/1228590608 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f3fb0062ff0 con 0x7f3f9c0a3cc0 2024-04-12T21:12:25.798 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mds.all update... 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3f9c0a3cc0 msgr2=0x7f3f9c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3f9c0a3cc0 0x7f3f9c0a6180 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3fac005f50 tx=0x7f3fac005e80 comp rx=0 tx=0).stop 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0102b40 msgr2=0x7f3fb0198ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0102b40 0x7f3fb0198ec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000b530 tx=0x7f3fa000b8f0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 shutdown_connections 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3f9c0a3cc0 0x7f3f9c0a6180 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3fb01046e0 0x7f3fb01a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3fb0102b40 0x7f3fb0198ec0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 --2- 172.21.15.153:0/1228590608 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3fb0103d40 0x7f3fb0199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 >> 172.21.15.153:0/1228590608 conn(0x7f3fb00fe100 msgr2=0x7f3fb0107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:25.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 shutdown_connections 2024-04-12T21:12:25.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-04-12T21:12:25.805+0000 7f3fb7417700 1 -- 172.21.15.153:0/1228590608 wait complete. 2024-04-12T21:12:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:25 smithi104 ceph-mon[36624]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:26.610 DEBUG:teuthology.orchestra.run.smithi081:mds.a> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service 2024-04-12T21:12:26.617 DEBUG:teuthology.orchestra.run.smithi081:mds.d> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service 2024-04-12T21:12:26.663 DEBUG:teuthology.orchestra.run.smithi081:mds.g> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service 2024-04-12T21:12:26.707 DEBUG:teuthology.orchestra.run.smithi081:mds.j> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service 2024-04-12T21:12:26.755 DEBUG:teuthology.orchestra.run.smithi104:mds.b> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service 2024-04-12T21:12:26.757 DEBUG:teuthology.orchestra.run.smithi104:mds.e> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service 2024-04-12T21:12:26.800 DEBUG:teuthology.orchestra.run.smithi104:mds.h> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service 2024-04-12T21:12:26.843 DEBUG:teuthology.orchestra.run.smithi104:mds.k> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service 2024-04-12T21:12:26.868 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:26 smithi153 ceph-mon[36569]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:12:26.868 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:26 smithi153 ceph-mon[36569]: Saving service mds.all spec with placement smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-12T21:12:26.868 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:26.868 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:26.868 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:26 smithi153 ceph-mon[36569]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:26.886 DEBUG:teuthology.orchestra.run.smithi153:mds.c> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service 2024-04-12T21:12:26.888 DEBUG:teuthology.orchestra.run.smithi153:mds.f> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service 2024-04-12T21:12:26.930 DEBUG:teuthology.orchestra.run.smithi153:mds.i> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service 2024-04-12T21:12:26.973 DEBUG:teuthology.orchestra.run.smithi153:mds.l> sudo journalctl -f -n 0 -u ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service 2024-04-12T21:12:27.018 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-12T21:12:27.018 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:12:27.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:26 smithi104 ceph-mon[36624]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:12:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:26 smithi104 ceph-mon[36624]: Saving service mds.all spec with placement smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-12T21:12:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:26 smithi104 ceph-mon[36624]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:27.103 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:26 smithi081 ceph-mon[31453]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:12:27.103 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:26 smithi081 ceph-mon[31453]: Saving service mds.all spec with placement smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-12T21:12:27.103 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:27.103 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:12:27.103 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:26 smithi081 ceph-mon[31453]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:27.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 -- 172.21.15.81:0/3175111788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d8c101070 msgr2=0x7f6d8c1014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 --2- 172.21.15.81:0/3175111788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d8c101070 0x7f6d8c1014d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009900 tx=0x7f6d7402bc30 comp rx=0 tx=0).stop 2024-04-12T21:12:27.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 -- 172.21.15.81:0/3175111788 shutdown_connections 2024-04-12T21:12:27.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 --2- 172.21.15.81:0/3175111788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d8c101070 0x7f6d8c1014d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 --2- 172.21.15.81:0/3175111788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d8c101a10 0x7f6d8c101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 --2- 172.21.15.81:0/3175111788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d8c100710 0x7f6d8c100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 -- 172.21.15.81:0/3175111788 >> 172.21.15.81:0/3175111788 conn(0x7f6d8c0fc2d0 msgr2=0x7f6d8c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:27.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 -- 172.21.15.81:0/3175111788 shutdown_connections 2024-04-12T21:12:27.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.256+0000 7f6d90a00640 1 -- 172.21.15.81:0/3175111788 wait complete. 2024-04-12T21:12:27.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.257+0000 7f6d90a00640 1 Processor -- start 2024-04-12T21:12:27.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.257+0000 7f6d90a00640 1 -- start start 2024-04-12T21:12:27.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d90a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d8c100710 0x7f6d8c19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:27.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d90a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d8c101070 0x7f6d8c19a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:27.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d90a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d8c101a10 0x7f6d8c1a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:27.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d90a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8c10fce0 con 0x7f6d8c100710 2024-04-12T21:12:27.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d90a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8c10fb60 con 0x7f6d8c101070 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d90a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8c10fe60 con 0x7f6d8c101a10 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d83e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d8c101070 0x7f6d8c19a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d8a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d8c100710 0x7f6d8c19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d83e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d8c101070 0x7f6d8c19a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:37348/0 (socket says 172.21.15.81:37348) 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d83e00640 1 -- 172.21.15.81:0/1142773682 learned_addr learned my addr 172.21.15.81:0/1142773682 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d8b200640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d8c101a10 0x7f6d8c1a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d83e00640 1 -- 172.21.15.81:0/1142773682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d8c101a10 msgr2=0x7f6d8c1a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d83e00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d8c101a10 0x7f6d8c1a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d83e00640 1 -- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d8c100710 msgr2=0x7f6d8c19a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d83e00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d8c100710 0x7f6d8c19a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.258+0000 7f6d83e00640 1 -- 172.21.15.81:0/1142773682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d8c1a1f30 con 0x7f6d8c101070 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.259+0000 7f6d83e00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d8c101070 0x7f6d8c19a850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009230 tx=0x7f6d7402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.259+0000 7f6d83400640 1 -- 172.21.15.81:0/1142773682 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74041070 con 0x7f6d8c101070 2024-04-12T21:12:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.259+0000 7f6d83400640 1 -- 172.21.15.81:0/1142773682 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d74004450 con 0x7f6d8c101070 2024-04-12T21:12:27.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.259+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d8c1a21c0 con 0x7f6d8c101070 2024-04-12T21:12:27.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.259+0000 7f6d83400640 1 -- 172.21.15.81:0/1142773682 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74003e20 con 0x7f6d8c101070 2024-04-12T21:12:27.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.259+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d8c1a2700 con 0x7f6d8c101070 2024-04-12T21:12:27.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.261+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d8c10b5f0 con 0x7f6d8c101070 2024-04-12T21:12:27.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.262+0000 7f6d83400640 1 -- 172.21.15.81:0/1142773682 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6d74044070 con 0x7f6d8c101070 2024-04-12T21:12:27.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.264+0000 7f6d83400640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d700a3c30 0x7f6d700a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:27.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.264+0000 7f6d83400640 1 -- 172.21.15.81:0/1142773682 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d740f1af0 con 0x7f6d8c101070 2024-04-12T21:12:27.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.264+0000 7f6d8a800640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d700a3c30 0x7f6d700a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:27.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.265+0000 7f6d8a800640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d700a3c30 0x7f6d700a60f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009bd0 tx=0x7f6d7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:27.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.268+0000 7f6d83400640 1 -- 172.21.15.81:0/1142773682 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d740be280 con 0x7f6d8c101070 2024-04-12T21:12:27.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.472+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d8c062bc0 con 0x7f6d8c101070 2024-04-12T21:12:27.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.473+0000 7f6d83400640 1 -- 172.21.15.81:0/1142773682 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18480 (secure 0 0 0) 0x7f6d740c2580 con 0x7f6d8c101070 2024-04-12T21:12:27.470 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:27.470 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":81,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:12:16.153549+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:12:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d700a3c30 msgr2=0x7f6d700a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d700a3c30 0x7f6d700a60f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009bd0 tx=0x7f6d7c005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d8c101070 msgr2=0x7f6d8c19a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d8c101070 0x7f6d8c19a850 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6d74009230 tx=0x7f6d7402c650 comp rx=0 tx=0).stop 2024-04-12T21:12:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 shutdown_connections 2024-04-12T21:12:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d700a3c30 0x7f6d700a60f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.472 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d8c101a10 0x7f6d8c1a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d8c101070 0x7f6d8c19a850 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 --2- 172.21.15.81:0/1142773682 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d8c100710 0x7f6d8c19a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 >> 172.21.15.81:0/1142773682 conn(0x7f6d8c0fc2d0 msgr2=0x7f6d8c0fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 shutdown_connections 2024-04-12T21:12:27.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.476+0000 7f6d90a00640 1 -- 172.21.15.81:0/1142773682 wait complete. 2024-04-12T21:12:27.491 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T21:12:27.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T21:12:27.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.708+0000 7f1285000640 1 -- 172.21.15.81:0/4145635321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 msgr2=0x7f1280078020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.708+0000 7f1285000640 1 --2- 172.21.15.81:0/4145635321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 0x7f1280078020 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f126c009900 tx=0x7f126c02bc30 comp rx=0 tx=0).stop 2024-04-12T21:12:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.709+0000 7f1285000640 1 -- 172.21.15.81:0/4145635321 shutdown_connections 2024-04-12T21:12:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.709+0000 7f1285000640 1 --2- 172.21.15.81:0/4145635321 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1280078560 0x7f1280078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.709+0000 7f1285000640 1 --2- 172.21.15.81:0/4145635321 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 0x7f1280078020 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.709+0000 7f1285000640 1 --2- 172.21.15.81:0/4145635321 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1280076980 0x7f1280076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.709+0000 7f1285000640 1 -- 172.21.15.81:0/4145635321 >> 172.21.15.81:0/4145635321 conn(0x7f12800ff690 msgr2=0x7f1280101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.709+0000 7f1285000640 1 -- 172.21.15.81:0/4145635321 shutdown_connections 2024-04-12T21:12:27.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.709+0000 7f1285000640 1 -- 172.21.15.81:0/4145635321 wait complete. 2024-04-12T21:12:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f1285000640 1 Processor -- start 2024-04-12T21:12:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f1285000640 1 -- start start 2024-04-12T21:12:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f1285000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1280076980 0x7f128019e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f1285000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 0x7f128019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:27.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f1285000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1280078560 0x7f12801a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f1285000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f1280114170 con 0x7f1280077bc0 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f1285000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1280113ff0 con 0x7f1280078560 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f1285000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12801142f0 con 0x7f1280076980 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f127f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1280076980 0x7f128019e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f127ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 0x7f128019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f127f400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1280076980 0x7f128019e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:35366/0 (socket says 172.21.15.81:35366) 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f127f400640 1 -- 172.21.15.81:0/821353272 learned_addr learned my addr 172.21.15.81:0/821353272 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.710+0000 7f127fe00640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1280078560 0x7f12801a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f127f400640 1 -- 172.21.15.81:0/821353272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1280078560 msgr2=0x7f12801a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f127f400640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1280078560 0x7f12801a5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f127f400640 1 -- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 msgr2=0x7f128019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f127f400640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 0x7f128019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f127f400640 1 -- 172.21.15.81:0/821353272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12801a6430 con 0x7f1280076980 2024-04-12T21:12:27.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f127ea00640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 0x7f128019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f127f400640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1280076980 0x7f128019e810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f127000e9f0 tx=0x7f127000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f1277e00640 1 -- 172.21.15.81:0/821353272 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127000cd80 con 0x7f1280076980 2024-04-12T21:12:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12801a6720 con 0x7f1280076980 2024-04-12T21:12:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f1277e00640 1 -- 172.21.15.81:0/821353272 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1270010910 con 0x7f1280076980 2024-04-12T21:12:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f1277e00640 1 -- 172.21.15.81:0/821353272 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1270018990 con 0x7f1280076980 2024-04-12T21:12:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.711+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12801a6c40 con 0x7f1280076980 2024-04-12T21:12:27.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.713+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1280110a90 con 0x7f1280076980 2024-04-12T21:12:27.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.715+0000 7f1277e00640 1 -- 172.21.15.81:0/821353272 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1270010ac0 con 0x7f1280076980 2024-04-12T21:12:27.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.716+0000 7f1277e00640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f12640a3bb0 0x7f12640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:27.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.716+0000 7f1277e00640 1 -- 172.21.15.81:0/821353272 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f12700d0460 con 0x7f1280076980 2024-04-12T21:12:27.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.716+0000 7f127ea00640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f12640a3bb0 0x7f12640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:27.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.717+0000 7f127ea00640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f12640a3bb0 0x7f12640a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f126c02c7b0 tx=0x7f126c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:27.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.719+0000 7f1277e00640 1 -- 172.21.15.81:0/821353272 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f127009cc90 con 0x7f1280076980 2024-04-12T21:12:27.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:27.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:27.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:27.913 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:27.914 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:27.914 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:27.914 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:27 smithi153 ceph-mon[36569]: Deploying daemon mds.l on smithi153 2024-04-12T21:12:27.914 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:27 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1142773682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:12:27.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.921+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1280062bc0 con 0x7f1280076980 2024-04-12T21:12:27.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.922+0000 7f1277e00640 1 -- 172.21.15.81:0/821353272 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f12700a0f90 con 0x7f1280076980 2024-04-12T21:12:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f12640a3bb0 msgr2=0x7f12640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f12640a3bb0 0x7f12640a6070 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f126c02c7b0 tx=0x7f126c005fd0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1280076980 msgr2=0x7f128019e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1280076980 0x7f128019e810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f127000e9f0 tx=0x7f127000eeb0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 shutdown_connections 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f12640a3bb0 0x7f12640a6070 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1280078560 0x7f12801a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1280077bc0 0x7f128019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 --2- 172.21.15.81:0/821353272 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1280076980 0x7f128019e810 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 >> 172.21.15.81:0/821353272 conn(0x7f12800ff690 msgr2=0x7f1280101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 shutdown_connections 2024-04-12T21:12:27.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:27.925+0000 7f1285000640 1 -- 172.21.15.81:0/821353272 wait complete. 2024-04-12T21:12:27.939 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:27 smithi104 ceph-mon[36624]: Deploying daemon mds.l on smithi153 2024-04-12T21:12:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:27 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1142773682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:12:28.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:28.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:12:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:27 smithi081 ceph-mon[31453]: Deploying daemon mds.l on smithi153 2024-04-12T21:12:28.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:27 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1142773682' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:12:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.159+0000 7fd6d5e00640 1 -- 172.21.15.81:0/1903898358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d010a1d0 msgr2=0x7fd6d0110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.159+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/1903898358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d010a1d0 0x7fd6d0110a60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8009ca0 tx=0x7fd6c802c0f0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.160+0000 7fd6d5e00640 1 -- 172.21.15.81:0/1903898358 shutdown_connections 2024-04-12T21:12:28.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.160+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/1903898358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d010a1d0 0x7fd6d0110a60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.160+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/1903898358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6d0109810 0x7fd6d0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.160+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/1903898358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6d0108620 0x7fd6d0108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.160+0000 7fd6d5e00640 1 -- 172.21.15.81:0/1903898358 >> 172.21.15.81:0/1903898358 conn(0x7fd6d00775e0 msgr2=0x7fd6d0079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:28.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.160+0000 7fd6d5e00640 1 -- 172.21.15.81:0/1903898358 shutdown_connections 2024-04-12T21:12:28.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.160+0000 7fd6d5e00640 1 -- 172.21.15.81:0/1903898358 wait complete. 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6d5e00640 1 Processor -- start 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6d5e00640 1 -- start start 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6d5e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6d0108620 0x7fd6d019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6d5e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6d0109810 0x7fd6d019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6d5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d010a1d0 0x7fd6d01a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6d5e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d0114280 con 0x7fd6d0108620 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6d5e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d0114100 con 0x7fd6d0109810 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6d5e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d0114400 con 0x7fd6d010a1d0 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.161+0000 7fd6cea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6d0108620 0x7fd6d019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6d0108620 0x7fd6d019e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41046/0 (socket says 172.21.15.81:41046) 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cea00640 1 -- 172.21.15.81:0/4056821861 learned_addr learned my addr 172.21.15.81:0/4056821861 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:28.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cf400640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d010a1d0 0x7fd6d01a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6c6000640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6d0109810 0x7fd6d019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cea00640 1 -- 172.21.15.81:0/4056821861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d010a1d0 msgr2=0x7fd6d01a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cea00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d010a1d0 0x7fd6d01a5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cea00640 1 -- 172.21.15.81:0/4056821861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6d0109810 msgr2=0x7fd6d019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cea00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6d0109810 0x7fd6d019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cea00640 1 -- 172.21.15.81:0/4056821861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d01a64e0 con 0x7fd6d0108620 2024-04-12T21:12:28.159 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6cea00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6d0108620 0x7fd6d019e8c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000e9c0 tx=0x7fd6c000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.162+0000 7fd6d4a00640 1 -- 172.21.15.81:0/4056821861 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c000cd80 con 0x7fd6d0108620 2024-04-12T21:12:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.163+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d01a67d0 con 0x7fd6d0108620 2024-04-12T21:12:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.163+0000 7fd6d4a00640 1 -- 172.21.15.81:0/4056821861 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c00035f0 con 0x7fd6d0108620 2024-04-12T21:12:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.163+0000 7fd6d4a00640 1 -- 172.21.15.81:0/4056821861 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0010690 con 0x7fd6d0108620 2024-04-12T21:12:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.163+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d01a6d70 con 0x7fd6d0108620 2024-04-12T21:12:28.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.164+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d010f8e0 con 0x7fd6d0108620 2024-04-12T21:12:28.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.166+0000 7fd6d4a00640 1 -- 172.21.15.81:0/4056821861 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd6c00037a0 con 0x7fd6d0108620 2024-04-12T21:12:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.167+0000 7fd6d4a00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd6b40a3bb0 0x7fd6b40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.167+0000 7fd6d4a00640 1 -- 172.21.15.81:0/4056821861 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd6c00d0470 con 0x7fd6d0108620 2024-04-12T21:12:28.163 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.168+0000 7fd6c6000640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd6b40a3bb0 0x7fd6b40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:28.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.168+0000 7fd6c6000640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd6b40a3bb0 0x7fd6b40a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd6d019fdd0 tx=0x7fd6b800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:28.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.170+0000 7fd6d4a00640 1 -- 172.21.15.81:0/4056821861 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd6c009d2e0 con 0x7fd6d0108620 2024-04-12T21:12:28.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.411+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6d002d5e0 con 0x7fd6d0108620 2024-04-12T21:12:28.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.412+0000 7fd6d4a00640 1 -- 172.21.15.81:0/4056821861 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fd6c009cbe0 con 0x7fd6d0108620 2024-04-12T21:12:28.408 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:28.408 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-12T21:12:28.409 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 1 2024-04-12T21:12:28.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd6b40a3bb0 msgr2=0x7fd6b40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:28.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd6b40a3bb0 0x7fd6b40a6070 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd6d019fdd0 tx=0x7fd6b800b410 comp rx=0 tx=0).stop 2024-04-12T21:12:28.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6d0108620 msgr2=0x7fd6d019e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:28.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6d0108620 0x7fd6d019e8c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000e9c0 tx=0x7fd6c000ee80 comp rx=0 tx=0).stop 2024-04-12T21:12:28.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 shutdown_connections 2024-04-12T21:12:28.411 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd6b40a3bb0 0x7fd6b40a6070 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6d010a1d0 0x7fd6d01a5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6d0109810 0x7fd6d019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 --2- 172.21.15.81:0/4056821861 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6d0108620 0x7fd6d019e8c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:28.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.415+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 >> 172.21.15.81:0/4056821861 conn(0x7fd6d00775e0 msgr2=0x7fd6d010cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:28.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.416+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 shutdown_connections 2024-04-12T21:12:28.412 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:28.416+0000 7fd6d5e00640 1 -- 172.21.15.81:0/4056821861 wait complete. 2024-04-12T21:12:28.969 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:28 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/821353272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:12:28.969 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:28 smithi153 ceph-mon[36569]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:28.969 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:28 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4056821861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:29.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:28 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/821353272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:12:29.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:28 smithi104 ceph-mon[36624]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:29.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:28 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4056821861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:29.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:28 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/821353272' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:12:29.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:28 smithi081 ceph-mon[31453]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:29.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:28 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4056821861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:29.302 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 systemd[1]: Starting Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:29.631 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 podman[67283]: 2024-04-12 21:12:29.277494522 +0000 UTC m=+0.018056620 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:29.632 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 podman[67283]: 2024-04-12 21:12:29.400063396 +0000 UTC m=+0.140625528 container create 2811ba92030226bf07edc1dc711f8ad196c0c5fe83846d7906618ec0439ac606 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-12T21:12:29.632 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 podman[67283]: 2024-04-12 21:12:29.610971965 +0000 UTC m=+0.351534063 container init 2811ba92030226bf07edc1dc711f8ad196c0c5fe83846d7906618ec0439ac606 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-12T21:12:29.632 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 podman[67283]: 2024-04-12 21:12:29.6136231 +0000 UTC m=+0.354185190 container start 2811ba92030226bf07edc1dc711f8ad196c0c5fe83846d7906618ec0439ac606 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:12:30.052 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 ceph-mds[67302]: -- 172.21.15.153:0/597531822 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d22c01a40 con 0x562d22bd4c00 2024-04-12T21:12:30.052 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: starting mds.l at 2024-04-12T21:12:30.052 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 bash[67283]: 2811ba92030226bf07edc1dc711f8ad196c0c5fe83846d7906618ec0439ac606 2024-04-12T21:12:30.052 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:12:29 smithi153 systemd[1]: Started Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:30.433 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:30.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2250851837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dcc105ba0 msgr2=0x7f2dcc106020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:30.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2250851837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dcc105ba0 0x7f2dcc106020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009ca0 tx=0x7f2dc402c100 comp rx=0 tx=0).stop 2024-04-12T21:12:30.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2250851837 shutdown_connections 2024-04-12T21:12:30.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2250851837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dcc105ba0 0x7f2dcc106020 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2250851837 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dcc1031f0 0x7f2dcc105660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2250851837 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dcc100880 0x7f2dcc102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2250851837 >> 172.21.15.81:0/2250851837 conn(0x7f2dcc0fc480 msgr2=0x7f2dcc0fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:30.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2250851837 shutdown_connections 2024-04-12T21:12:30.654 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.657+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2250851837 wait complete. 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.658+0000 7f2dd0a00640 1 Processor -- start 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dd0a00640 1 -- start start 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dd0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dcc100880 0x7f2dcc19e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dd0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dcc1031f0 0x7f2dcc19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dd0a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dcc105ba0 0x7f2dcc1a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dd0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2dcc114290 con 0x7f2dcc100880 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dd0a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2dcc114110 con 0x7f2dcc105ba0 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dd0a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2dcc114410 con 0x7f2dcc1031f0 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dc9e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dcc100880 0x7f2dcc19e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dc9e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dcc100880 0x7f2dcc19e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41064/0 (socket says 172.21.15.81:41064) 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dc9e00640 1 -- 172.21.15.81:0/2144169591 learned_addr learned my addr 172.21.15.81:0/2144169591 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:30.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dca800640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dcc105ba0 0x7f2dcc1a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:30.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.659+0000 7f2dc9400640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dcc1031f0 0x7f2dcc19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:30.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.660+0000 7f2dc9e00640 1 -- 172.21.15.81:0/2144169591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dcc1031f0 msgr2=0x7f2dcc19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:30.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.660+0000 7f2dc9e00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dcc1031f0 0x7f2dcc19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.660+0000 7f2dc9e00640 1 -- 172.21.15.81:0/2144169591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dcc105ba0 msgr2=0x7f2dcc1a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:30.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.660+0000 7f2dc9e00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dcc105ba0 0x7f2dcc1a5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.660+0000 7f2dc9e00640 1 -- 172.21.15.81:0/2144169591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dcc1a6500 con 0x7f2dcc100880 2024-04-12T21:12:30.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.660+0000 7f2dc9e00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dcc100880 0x7f2dcc19e8e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00ea60 tx=0x7f2dbc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:30.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.661+0000 7f2dc2a00640 1 -- 172.21.15.81:0/2144169591 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc00cde0 con 0x7f2dcc100880 2024-04-12T21:12:30.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.661+0000 7f2dc2a00640 1 -- 172.21.15.81:0/2144169591 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dbc0035f0 con 0x7f2dcc100880 2024-04-12T21:12:30.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.661+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dcc1a67f0 con 0x7f2dcc100880 2024-04-12T21:12:30.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.661+0000 7f2dc2a00640 1 -- 172.21.15.81:0/2144169591 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc010670 con 0x7f2dcc100880 2024-04-12T21:12:30.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.662+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dcc1a6d40 con 0x7f2dcc100880 2024-04-12T21:12:30.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.663+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dcc0fdf70 con 0x7f2dcc100880 2024-04-12T21:12:30.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.664+0000 7f2dc2a00640 1 -- 172.21.15.81:0/2144169591 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2dbc0037a0 con 0x7f2dcc100880 2024-04-12T21:12:30.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.666+0000 7f2dc2a00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2db00a3bb0 0x7f2db00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:30.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.666+0000 7f2dc2a00640 1 -- 172.21.15.81:0/2144169591 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2dbc0d0360 con 0x7f2dcc100880 2024-04-12T21:12:30.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.666+0000 7f2dc9400640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2db00a3bb0 0x7f2db00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:30.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.667+0000 7f2dc9400640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2db00a3bb0 0x7f2db00a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc19fdf0 tx=0x7f2db4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:30.665 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.670+0000 7f2dc2a00640 1 -- 172.21.15.81:0/2144169591 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2dbc09d140 con 0x7f2dcc100880 2024-04-12T21:12:30.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:boot 2024-04-12T21:12:30.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: fsmap 1 up:standby 2024-04-12T21:12:30.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:12:30.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:30.753 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:30.754 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:30.754 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:30.754 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:30.754 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:30.754 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: Deploying daemon mds.k on smithi104 2024-04-12T21:12:30.754 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:30 smithi104 ceph-mon[36624]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:boot 2024-04-12T21:12:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: fsmap 1 up:standby 2024-04-12T21:12:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:12:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: Deploying daemon mds.k on smithi104 2024-04-12T21:12:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:30 smithi081 ceph-mon[31453]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:30.912 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.916+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dcc02d5e0 con 0x7f2dcc100880 2024-04-12T21:12:30.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.917+0000 7f2dc2a00640 1 -- 172.21.15.81:0/2144169591 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f2dbc09ca40 con 0x7f2dcc100880 2024-04-12T21:12:30.913 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:30.913 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-12T21:12:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 2 2024-04-12T21:12:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2db00a3bb0 msgr2=0x7f2db00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2db00a3bb0 0x7f2db00a6070 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc19fdf0 tx=0x7f2db4005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:30.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dcc100880 msgr2=0x7f2dcc19e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:30.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dcc100880 0x7f2dcc19e8e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc00ea60 tx=0x7f2dbc00ef20 comp rx=0 tx=0).stop 2024-04-12T21:12:30.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 shutdown_connections 2024-04-12T21:12:30.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2db00a3bb0 0x7f2db00a6070 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2dcc1031f0 0x7f2dcc19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2dcc105ba0 0x7f2dcc1a5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 --2- 172.21.15.81:0/2144169591 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2dcc100880 0x7f2dcc19e8e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:30.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 >> 172.21.15.81:0/2144169591 conn(0x7f2dcc0fc480 msgr2=0x7f2dcc101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:30.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 shutdown_connections 2024-04-12T21:12:30.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:30.920+0000 7f2dd0a00640 1 -- 172.21.15.81:0/2144169591 wait complete. 2024-04-12T21:12:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:boot 2024-04-12T21:12:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: fsmap 1 up:standby 2024-04-12T21:12:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:12:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: Deploying daemon mds.k on smithi104 2024-04-12T21:12:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:30 smithi153 ceph-mon[36569]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:31 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2144169591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:31 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2144169591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:32.078 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:31 smithi104 systemd[1]: Starting Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:31 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2144169591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:32.462 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:32 smithi104 podman[65276]: 2024-04-12 21:12:32.102771851 +0000 UTC m=+0.017800816 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:32.462 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:32 smithi104 podman[65276]: 2024-04-12 21:12:32.225359827 +0000 UTC m=+0.140388783 container create 2e08d3c1310f9c38e9f7c06ec4ed37b502a9cb6ed80255ed61c6312877f417e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:12:32.462 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:32 smithi104 podman[65276]: 2024-04-12 21:12:32.467361078 +0000 UTC m=+0.382390032 container init 2e08d3c1310f9c38e9f7c06ec4ed37b502a9cb6ed80255ed61c6312877f417e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-12T21:12:32.789 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:32 smithi104 podman[65276]: 2024-04-12 21:12:32.470249518 +0000 UTC m=+0.385278485 container start 2e08d3c1310f9c38e9f7c06ec4ed37b502a9cb6ed80255ed61c6312877f417e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T21:12:32.789 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-mds[65295]: -- 172.21.15.104:0/1707365425 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56339973fa40 con 0x563399712800 2024-04-12T21:12:32.789 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: starting mds.k at 2024-04-12T21:12:32.789 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:32 smithi104 bash[65276]: 2e08d3c1310f9c38e9f7c06ec4ed37b502a9cb6ed80255ed61c6312877f417e3 2024-04-12T21:12:32.789 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:12:32 smithi104 systemd[1]: Started Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:32.935 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:32 smithi153 ceph-mon[36569]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:33.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-mon[36624]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:33.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.148+0000 7fb046c00640 1 -- 172.21.15.81:0/3415366134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb040105410 msgr2=0x7fb040105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:33.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.148+0000 7fb046c00640 1 --2- 172.21.15.81:0/3415366134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb040105410 0x7fb040105890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009900 tx=0x7fb02c02bc30 comp rx=0 tx=0).stop 2024-04-12T21:12:33.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.148+0000 7fb046c00640 1 -- 172.21.15.81:0/3415366134 shutdown_connections 2024-04-12T21:12:33.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.148+0000 7fb046c00640 1 --2- 172.21.15.81:0/3415366134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb040105410 0x7fb040105890 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.148+0000 7fb046c00640 1 --2- 172.21.15.81:0/3415366134 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb040105dd0 0x7fb04010c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.148+0000 7fb046c00640 1 --2- 172.21.15.81:0/3415366134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb040104220 0x7fb040104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.148+0000 7fb046c00640 1 -- 172.21.15.81:0/3415366134 >> 172.21.15.81:0/3415366134 conn(0x7fb0400ff7b0 msgr2=0x7fb040101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:33.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.148+0000 7fb046c00640 1 -- 172.21.15.81:0/3415366134 shutdown_connections 2024-04-12T21:12:33.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.149+0000 7fb046c00640 1 -- 172.21.15.81:0/3415366134 wait complete. 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.149+0000 7fb046c00640 1 Processor -- start 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb046c00640 1 -- start start 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb046c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb040104220 0x7fb04019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb046c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb040105410 0x7fb04019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb046c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb040105dd0 0x7fb0401a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb046c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb04010ff00 con 0x7fb040104220 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb046c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb04010fd80 con 0x7fb040105dd0 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb046c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb040110080 con 0x7fb040105410 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb03fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb040104220 0x7fb04019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:33.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb03fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb040104220 0x7fb04019a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:41076/0 (socket says 172.21.15.81:41076) 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb03fe00640 1 -- 172.21.15.81:0/1109391168 learned_addr learned my addr 172.21.15.81:0/1109391168 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb044c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb040105dd0 0x7fb0401a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.150+0000 7fb03f400640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb040105410 0x7fb04019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb044c00640 1 -- 172.21.15.81:0/1109391168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb040105410 msgr2=0x7fb04019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb044c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb040105410 0x7fb04019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb044c00640 1 -- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb040104220 msgr2=0x7fb04019a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb044c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb040104220 0x7fb04019a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb044c00640 1 -- 172.21.15.81:0/1109391168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0401a2180 con 0x7fb040105dd0 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb044c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb040105dd0 0x7fb0401a1b10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb038009fd0 tx=0x7fb03800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb03cc00640 1 -- 172.21.15.81:0/1109391168 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03800ce20 con 0x7fb040105dd0 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb03cc00640 1 -- 172.21.15.81:0/1109391168 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb038005020 con 0x7fb040105dd0 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb03cc00640 1 -- 172.21.15.81:0/1109391168 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038010720 con 0x7fb040105dd0 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0401a2470 con 0x7fb040105dd0 2024-04-12T21:12:33.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.151+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0401a2940 con 0x7fb040105dd0 2024-04-12T21:12:33.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.153+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb04010b4e0 con 0x7fb040105dd0 2024-04-12T21:12:33.151 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.154+0000 7fb03cc00640 1 -- 172.21.15.81:0/1109391168 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb038005540 con 0x7fb040105dd0 2024-04-12T21:12:33.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:32 smithi081 ceph-mon[31453]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:33.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:33.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:33.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:33.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:33.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.156+0000 7fb03cc00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb0240a3bb0 0x7fb0240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:33.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.156+0000 7fb03cc00640 1 -- 172.21.15.81:0/1109391168 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb0380d06b0 con 0x7fb040105dd0 2024-04-12T21:12:33.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.156+0000 7fb03fe00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb0240a3bb0 0x7fb0240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:33.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.156+0000 7fb03fe00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb0240a3bb0 0x7fb0240a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb0300058c0 tx=0x7fb030005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:33.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.160+0000 7fb03cc00640 1 -- 172.21.15.81:0/1109391168 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb03809d490 con 0x7fb040105dd0 2024-04-12T21:12:33.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.428+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb04002d5e0 con 0x7fb040105dd0 2024-04-12T21:12:33.426 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.429+0000 7fb03cc00640 1 -- 172.21.15.81:0/1109391168 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fb03809cd90 con 0x7fb040105dd0 2024-04-12T21:12:33.426 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:33.426 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-12T21:12:33.427 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 3 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb0240a3bb0 msgr2=0x7fb0240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb0240a3bb0 0x7fb0240a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb0300058c0 tx=0x7fb030005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb040105dd0 msgr2=0x7fb0401a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb040105dd0 0x7fb0401a1b10 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb038009fd0 tx=0x7fb03800ef10 comp rx=0 tx=0).stop 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 shutdown_connections 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb0240a3bb0 0x7fb0240a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb040105410 0x7fb04019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb040105dd0 0x7fb0401a1b10 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 --2- 172.21.15.81:0/1109391168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb040104220 0x7fb04019a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:33.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 >> 172.21.15.81:0/1109391168 conn(0x7fb0400ff7b0 msgr2=0x7fb0401086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:33.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 shutdown_connections 2024-04-12T21:12:33.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:33.432+0000 7fb046c00640 1 -- 172.21.15.81:0/1109391168 wait complete. 2024-04-12T21:12:33.889 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:33 smithi104 ceph-mon[36624]: Deploying daemon mds.j on smithi081 2024-04-12T21:12:33.889 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:33 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:boot 2024-04-12T21:12:33.890 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:33 smithi104 ceph-mon[36624]: fsmap 2 up:standby 2024-04-12T21:12:33.890 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:12:33.890 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:33 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1109391168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:34.025 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:33 smithi081 ceph-mon[31453]: Deploying daemon mds.j on smithi081 2024-04-12T21:12:34.025 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:33 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:boot 2024-04-12T21:12:34.025 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:33 smithi081 ceph-mon[31453]: fsmap 2 up:standby 2024-04-12T21:12:34.025 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:12:34.025 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:33 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1109391168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:33 smithi153 ceph-mon[36569]: Deploying daemon mds.j on smithi081 2024-04-12T21:12:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:33 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:boot 2024-04-12T21:12:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:33 smithi153 ceph-mon[36569]: fsmap 2 up:standby 2024-04-12T21:12:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:12:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:33 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1109391168' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:35.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:34 smithi104 ceph-mon[36624]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:34 smithi081 ceph-mon[31453]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:35.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:34 smithi153 ceph-mon[36569]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:35.450 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:35.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 -- 172.21.15.81:0/3017923622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 msgr2=0x7f746c074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:35.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 --2- 172.21.15.81:0/3017923622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 0x7f746c074450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f745c009900 tx=0x7f745c02bc30 comp rx=0 tx=0).stop 2024-04-12T21:12:35.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 -- 172.21.15.81:0/3017923622 shutdown_connections 2024-04-12T21:12:35.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 --2- 172.21.15.81:0/3017923622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f746c109d20 0x7f746c10c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:35.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 --2- 172.21.15.81:0/3017923622 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f746c074a20 0x7f746c074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:35.723 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 --2- 172.21.15.81:0/3017923622 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 0x7f746c074450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:35.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 -- 172.21.15.81:0/3017923622 >> 172.21.15.81:0/3017923622 conn(0x7f746c0fd3d0 msgr2=0x7f746c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:35.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 -- 172.21.15.81:0/3017923622 shutdown_connections 2024-04-12T21:12:35.724 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.726+0000 7f746be00640 1 -- 172.21.15.81:0/3017923622 wait complete. 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.727+0000 7f746be00640 1 Processor -- start 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.727+0000 7f746be00640 1 -- start start 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f746c074a20 0x7f746c19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 0x7f746c19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f746c109d20 0x7f746c1a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f746c10fca0 con 0x7f746c074a20 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746be00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f746c10fb20 con 0x7f746c076000 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746be00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f746c10fe20 con 0x7f746c109d20 2024-04-12T21:12:35.726 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f746c074a20 0x7f746c19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f746c074a20 0x7f746c19a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56614/0 (socket says 172.21.15.81:56614) 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746aa00640 1 -- 172.21.15.81:0/462924313 learned_addr learned my addr 172.21.15.81:0/462924313 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746a000640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 0x7f746c19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746b400640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f746c109d20 0x7f746c1a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746aa00640 1 -- 172.21.15.81:0/462924313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f746c109d20 msgr2=0x7f746c1a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746aa00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f746c109d20 0x7f746c1a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746aa00640 1 -- 172.21.15.81:0/462924313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 msgr2=0x7f746c19a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746aa00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 0x7f746c19a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746aa00640 1 -- 172.21.15.81:0/462924313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c1a1f80 con 0x7f746c074a20 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.728+0000 7f746a000640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 0x7f746c19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.729+0000 7f746aa00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f746c074a20 0x7f746c19a360 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f745c009230 tx=0x7f745c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.729+0000 7f7470a00640 1 -- 172.21.15.81:0/462924313 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c041070 con 0x7f746c074a20 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.729+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746c1a2210 con 0x7f746c074a20 2024-04-12T21:12:35.727 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.729+0000 7f7470a00640 1 -- 172.21.15.81:0/462924313 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f745c004450 con 0x7f746c074a20 2024-04-12T21:12:35.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.729+0000 7f7470a00640 1 -- 172.21.15.81:0/462924313 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c003e20 con 0x7f746c074a20 2024-04-12T21:12:35.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.729+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746c1a2700 con 0x7f746c074a20 2024-04-12T21:12:35.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.731+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746c108a50 con 0x7f746c074a20 2024-04-12T21:12:35.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.732+0000 7f7470a00640 1 -- 172.21.15.81:0/462924313 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f745c044070 con 0x7f746c074a20 2024-04-12T21:12:35.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.735+0000 7f7470a00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f74500a3bb0 0x7f74500a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:35.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.735+0000 7f7470a00640 1 -- 172.21.15.81:0/462924313 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f745c0f1b20 con 0x7f746c074a20 2024-04-12T21:12:35.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.735+0000 7f746a000640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f74500a3bb0 0x7f74500a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:35.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.735+0000 7f746a000640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f74500a3bb0 0x7f74500a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f746c19b870 tx=0x7f7458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:35.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:35.740+0000 7f7470a00640 1 -- 172.21.15.81:0/462924313 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f745c0be2b0 con 0x7f746c074a20 2024-04-12T21:12:35.901 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:35 smithi081 systemd[1]: Starting Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:36.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.007+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f746c02d800 con 0x7f746c074a20 2024-04-12T21:12:36.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.008+0000 7f7470a00640 1 -- 172.21.15.81:0/462924313 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f745c0b8e20 con 0x7f746c074a20 2024-04-12T21:12:36.005 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:36.005 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-12T21:12:36.006 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 3 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f74500a3bb0 msgr2=0x7f74500a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f74500a3bb0 0x7f74500a6070 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f746c19b870 tx=0x7f7458005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f746c074a20 msgr2=0x7f746c19a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f746c074a20 0x7f746c19a360 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f745c009230 tx=0x7f745c02c650 comp rx=0 tx=0).stop 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 shutdown_connections 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f74500a3bb0 0x7f74500a6070 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f746c109d20 0x7f746c1a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f746c076000 0x7f746c19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 --2- 172.21.15.81:0/462924313 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f746c074a20 0x7f746c19a360 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 >> 172.21.15.81:0/462924313 conn(0x7f746c0fd3d0 msgr2=0x7f746c105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:36.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 shutdown_connections 2024-04-12T21:12:36.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:36.011+0000 7f746be00640 1 -- 172.21.15.81:0/462924313 wait complete. 2024-04-12T21:12:36.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:36 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/462924313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:36.288 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:35 smithi081 podman[71755]: 2024-04-12 21:12:35.892506732 +0000 UTC m=+0.036904568 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:36.288 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:36 smithi081 podman[71755]: 2024-04-12 21:12:36.011054303 +0000 UTC m=+0.155452154 container create f82f7deb6ea97f3aac70ee05122ff72964f8fd69dc4762d0bd33885d6ca57c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-12T21:12:36.288 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:36 smithi081 podman[71755]: 2024-04-12 21:12:36.290401498 +0000 UTC m=+0.434799358 container init f82f7deb6ea97f3aac70ee05122ff72964f8fd69dc4762d0bd33885d6ca57c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux ) 2024-04-12T21:12:36.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:36 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/462924313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:36.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:36 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/462924313' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:36.651 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:36 smithi081 podman[71755]: 2024-04-12 21:12:36.30221844 +0000 UTC m=+0.446616269 container start f82f7deb6ea97f3aac70ee05122ff72964f8fd69dc4762d0bd33885d6ca57c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:12:36.652 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:36 smithi081 ceph-mds[71775]: -- 172.21.15.81:0/2724564426 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55907c0cba40 con 0x55907c09e800 2024-04-12T21:12:36.652 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:36 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: starting mds.j at 2024-04-12T21:12:36.652 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:36 smithi081 bash[71755]: f82f7deb6ea97f3aac70ee05122ff72964f8fd69dc4762d0bd33885d6ca57c4c 2024-04-12T21:12:36.652 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:12:36 smithi081 systemd[1]: Started Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:37.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:37 smithi104 ceph-mon[36624]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:37.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:37.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:37.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:37.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:37 smithi081 ceph-mon[31453]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:37.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:37.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:37.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:37.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:37 smithi153 ceph-mon[36569]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:37.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:37.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:37.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:37.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:38.028 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 -- 172.21.15.81:0/4291059438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 msgr2=0x7fc544105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 --2- 172.21.15.81:0/4291059438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 0x7fc544105890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc530009900 tx=0x7fc53002bc50 comp rx=0 tx=0).stop 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 -- 172.21.15.81:0/4291059438 shutdown_connections 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 --2- 172.21.15.81:0/4291059438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc544105dd0 0x7fc54410c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 --2- 172.21.15.81:0/4291059438 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 0x7fc544105890 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 --2- 172.21.15.81:0/4291059438 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc544104220 0x7fc544104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 -- 172.21.15.81:0/4291059438 >> 172.21.15.81:0/4291059438 conn(0x7fc5440ff7b0 msgr2=0x7fc544101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 -- 172.21.15.81:0/4291059438 shutdown_connections 2024-04-12T21:12:38.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.232+0000 7fc549a00640 1 -- 172.21.15.81:0/4291059438 wait complete. 2024-04-12T21:12:38.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.233+0000 7fc549a00640 1 Processor -- start 2024-04-12T21:12:38.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.233+0000 7fc549a00640 1 -- start start 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.233+0000 7fc549a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc544104220 0x7fc54419a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.233+0000 7fc549a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 0x7fc54419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.233+0000 7fc549a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc544105dd0 0x7fc5441a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.233+0000 7fc549a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc54410ff00 con 0x7fc544105dd0 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc549a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc54410fd80 con 0x7fc544105410 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc549a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc544110080 con 0x7fc544104220 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 0x7fc54419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc543400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc544105dd0 0x7fc5441a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc544104220 0x7fc54419a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 0x7fc54419aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:50136/0 (socket says 172.21.15.81:50136) 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542000640 1 -- 172.21.15.81:0/302289124 learned_addr learned my addr 172.21.15.81:0/302289124 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542000640 1 -- 172.21.15.81:0/302289124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc544104220 msgr2=0x7fc54419a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:38.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542000640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc544104220 0x7fc54419a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542000640 1 -- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc544105dd0 msgr2=0x7fc5441a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542000640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc544105dd0 0x7fc5441a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542000640 1 -- 172.21.15.81:0/302289124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5441a2120 con 0x7fc544105410 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.234+0000 7fc542a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc544104220 0x7fc54419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.235+0000 7fc542000640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 0x7fc54419aa40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc53002c260 tx=0x7fc53002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.235+0000 7fc53b400640 1 -- 172.21.15.81:0/302289124 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc530041070 con 0x7fc544105410 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.235+0000 7fc53b400640 1 -- 172.21.15.81:0/302289124 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc530004450 con 0x7fc544105410 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.235+0000 7fc53b400640 1 -- 172.21.15.81:0/302289124 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc530003e50 con 0x7fc544105410 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.235+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5441a23b0 con 0x7fc544105410 2024-04-12T21:12:38.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.235+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5441a2950 con 0x7fc544105410 2024-04-12T21:12:38.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.236+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc54410b4e0 con 0x7fc544105410 2024-04-12T21:12:38.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.238+0000 7fc53b400640 1 -- 172.21.15.81:0/302289124 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc530044070 con 0x7fc544105410 2024-04-12T21:12:38.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.240+0000 7fc53b400640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5280a3dd0 0x7fc5280a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:38.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.240+0000 7fc53b400640 1 -- 172.21.15.81:0/302289124 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc5300f1d20 con 0x7fc544105410 2024-04-12T21:12:38.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.240+0000 7fc542a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5280a3dd0 0x7fc5280a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:38.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.241+0000 7fc542a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5280a3dd0 0x7fc5280a6290 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc544105270 tx=0x7fc534005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:38.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.244+0000 7fc53b400640 1 -- 172.21.15.81:0/302289124 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5300f6070 con 0x7fc544105410 2024-04-12T21:12:38.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:38 smithi104 ceph-mon[36624]: Deploying daemon mds.d on smithi081 2024-04-12T21:12:38.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:38 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:boot 2024-04-12T21:12:38.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:38 smithi104 ceph-mon[36624]: fsmap 3 up:standby 2024-04-12T21:12:38.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:12:38.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:38 smithi081 ceph-mon[31453]: Deploying daemon mds.d on smithi081 2024-04-12T21:12:38.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:38 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:boot 2024-04-12T21:12:38.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:38 smithi081 ceph-mon[31453]: fsmap 3 up:standby 2024-04-12T21:12:38.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:12:38.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.501+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc54402d830 con 0x7fc544105410 2024-04-12T21:12:38.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.502+0000 7fc53b400640 1 -- 172.21.15.81:0/302289124 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7fc5300beb80 con 0x7fc544105410 2024-04-12T21:12:38.499 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:38.499 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-12T21:12:38.499 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 4 2024-04-12T21:12:38.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5280a3dd0 msgr2=0x7fc5280a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:38.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5280a3dd0 0x7fc5280a6290 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc544105270 tx=0x7fc534005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:38.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 msgr2=0x7fc54419aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:38.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 0x7fc54419aa40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc53002c260 tx=0x7fc53002c670 comp rx=0 tx=0).stop 2024-04-12T21:12:38.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 shutdown_connections 2024-04-12T21:12:38.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc5280a3dd0 0x7fc5280a6290 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc544105dd0 0x7fc5441a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc544105410 0x7fc54419aa40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 --2- 172.21.15.81:0/302289124 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc544104220 0x7fc54419a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 >> 172.21.15.81:0/302289124 conn(0x7fc5440ff7b0 msgr2=0x7fc5441086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 shutdown_connections 2024-04-12T21:12:38.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:38.505+0000 7fc549a00640 1 -- 172.21.15.81:0/302289124 wait complete. 2024-04-12T21:12:38.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:38 smithi153 ceph-mon[36569]: Deploying daemon mds.d on smithi081 2024-04-12T21:12:38.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:38 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:boot 2024-04-12T21:12:38.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:38 smithi153 ceph-mon[36569]: fsmap 3 up:standby 2024-04-12T21:12:38.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:12:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:39 smithi081 ceph-mon[31453]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:39 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/302289124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:39.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:39 smithi153 ceph-mon[36569]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:39.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:39 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/302289124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:39.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:39 smithi104 ceph-mon[36624]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:39.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:39 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/302289124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:40.022 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:39 smithi081 systemd[1]: Starting Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:40.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:40 smithi081 podman[72099]: 2024-04-12 21:12:39.929639175 +0000 UTC m=+0.034120724 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:40.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:40 smithi081 podman[72099]: 2024-04-12 21:12:40.042139801 +0000 UTC m=+0.146621385 container create fa4bc41faa469be6dae2c2f6df19c18f720b3dbe9e6f15e3a73b4f0bfb1e7b26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:12:40.310 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:40 smithi081 podman[72099]: 2024-04-12 21:12:40.313116547 +0000 UTC m=+0.417598177 container init fa4bc41faa469be6dae2c2f6df19c18f720b3dbe9e6f15e3a73b4f0bfb1e7b26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-12T21:12:40.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:40.603 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:40 smithi081 podman[72099]: 2024-04-12 21:12:40.324046469 +0000 UTC m=+0.428528026 container start fa4bc41faa469be6dae2c2f6df19c18f720b3dbe9e6f15e3a73b4f0bfb1e7b26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-12T21:12:40.603 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:40 smithi081 ceph-mds[72119]: -- 172.21.15.81:0/2654110401 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a62c51da40 con 0x55a62c4f0c00 2024-04-12T21:12:40.603 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: starting mds.d at 2024-04-12T21:12:40.603 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:40 smithi081 bash[72099]: fa4bc41faa469be6dae2c2f6df19c18f720b3dbe9e6f15e3a73b4f0bfb1e7b26 2024-04-12T21:12:40.603 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:12:40 smithi081 systemd[1]: Started Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 -- 172.21.15.81:0/1787677907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105430 msgr2=0x7f0de41058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 --2- 172.21.15.81:0/1787677907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105430 0x7f0de41058b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009860 tx=0x7f0dcc02bb70 comp rx=0 tx=0).stop 2024-04-12T21:12:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 -- 172.21.15.81:0/1787677907 shutdown_connections 2024-04-12T21:12:40.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 --2- 172.21.15.81:0/1787677907 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0de4105df0 0x7f0de410c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:40.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 --2- 172.21.15.81:0/1787677907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105430 0x7f0de41058b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:40.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 --2- 172.21.15.81:0/1787677907 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de4104240 0x7f0de4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:40.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 -- 172.21.15.81:0/1787677907 >> 172.21.15.81:0/1787677907 conn(0x7f0de40ff7b0 msgr2=0x7f0de4101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:40.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 -- 172.21.15.81:0/1787677907 shutdown_connections 2024-04-12T21:12:40.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.741+0000 7f0dea400640 1 -- 172.21.15.81:0/1787677907 wait complete. 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.742+0000 7f0dea400640 1 Processor -- start 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0dea400640 1 -- start start 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0dea400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0de4104240 0x7f0de419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0dea400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de4105430 0x7f0de419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0dea400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105df0 0x7f0de41a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0dea400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0de410fed0 con 0x7f0de4104240 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0dea400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0de410fd50 con 0x7f0de4105df0 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0dea400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0de4110050 con 0x7f0de4105430 2024-04-12T21:12:40.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0de3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0de4104240 0x7f0de419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:40.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0de3e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105df0 0x7f0de41a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:40.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0ddaa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de4105430 0x7f0de419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:40.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0de3400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0de4104240 0x7f0de419a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56668/0 (socket says 172.21.15.81:56668) 2024-04-12T21:12:40.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.743+0000 7f0de3400640 1 -- 172.21.15.81:0/834226537 learned_addr learned my addr 172.21.15.81:0/834226537 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de3400640 1 -- 172.21.15.81:0/834226537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de4105430 msgr2=0x7f0de419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de3400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de4105430 0x7f0de419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de3400640 1 -- 172.21.15.81:0/834226537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105df0 msgr2=0x7f0de41a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de3400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105df0 0x7f0de41a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de3400640 1 -- 172.21.15.81:0/834226537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de41a2180 con 0x7f0de4104240 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de3e00640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105df0 0x7f0de41a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de3400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0de4104240 0x7f0de419a560 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400eb70 tx=0x7f0dd400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de0c00640 1 -- 172.21.15.81:0/834226537 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4003b70 con 0x7f0de4104240 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de41a2470 con 0x7f0de4104240 2024-04-12T21:12:40.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de0c00640 1 -- 172.21.15.81:0/834226537 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0dd400ee70 con 0x7f0de4104240 2024-04-12T21:12:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0de0c00640 1 -- 172.21.15.81:0/834226537 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4010830 con 0x7f0de4104240 2024-04-12T21:12:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.744+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de41a2990 con 0x7f0de4104240 2024-04-12T21:12:40.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.746+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de410b5e0 con 0x7f0de4104240 2024-04-12T21:12:40.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.748+0000 7f0de0c00640 1 -- 172.21.15.81:0/834226537 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0dd4004020 con 0x7f0de4104240 2024-04-12T21:12:40.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.749+0000 7f0de0c00640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0dc80a3bb0 0x7f0dc80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:40.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.749+0000 7f0de0c00640 1 -- 172.21.15.81:0/834226537 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0dd40d05e0 con 0x7f0de4104240 2024-04-12T21:12:40.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.749+0000 7f0ddaa00640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0dc80a3bb0 0x7f0dc80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:40.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.750+0000 7f0ddaa00640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0dc80a3bb0 0x7f0dc80a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009830 tx=0x7f0dcc034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:40.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.754+0000 7f0de0c00640 1 -- 172.21.15.81:0/834226537 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dd409d460 con 0x7f0de4104240 2024-04-12T21:12:40.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:40.998+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0de402d5e0 con 0x7f0de4104240 2024-04-12T21:12:40.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.000+0000 7f0de0c00640 1 -- 172.21.15.81:0/834226537 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f0de402d5e0 con 0x7f0de4104240 2024-04-12T21:12:40.996 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:40.997 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-12T21:12:40.997 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 4 2024-04-12T21:12:40.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0dc80a3bb0 msgr2=0x7f0dc80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:40.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0dc80a3bb0 0x7f0dc80a6070 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc009830 tx=0x7f0dcc034ab0 comp rx=0 tx=0).stop 2024-04-12T21:12:40.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0de4104240 msgr2=0x7f0de419a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:40.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0de4104240 0x7f0de419a560 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0dd400eb70 tx=0x7f0dd400c6a0 comp rx=0 tx=0).stop 2024-04-12T21:12:40.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 shutdown_connections 2024-04-12T21:12:41.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0dc80a3bb0 0x7f0dc80a6070 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:41.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0de4105430 0x7f0de419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:41.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0de4105df0 0x7f0de41a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:41.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 --2- 172.21.15.81:0/834226537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0de4104240 0x7f0de419a560 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:41.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 >> 172.21.15.81:0/834226537 conn(0x7f0de40ff7b0 msgr2=0x7f0de41087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:41.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 shutdown_connections 2024-04-12T21:12:41.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:41.003+0000 7f0dea400640 1 -- 172.21.15.81:0/834226537 wait complete. 2024-04-12T21:12:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:41 smithi104 ceph-mon[36624]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:41.507 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:41 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/834226537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:41.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:41 smithi081 ceph-mon[31453]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:41.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:41.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:41.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:41.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:41 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/834226537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:41 smithi153 ceph-mon[36569]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:41.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:41.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:41 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/834226537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:42.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:42 smithi104 ceph-mon[36624]: Deploying daemon mds.e on smithi104 2024-04-12T21:12:42.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:42 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:boot 2024-04-12T21:12:42.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:42 smithi104 ceph-mon[36624]: fsmap 4 up:standby 2024-04-12T21:12:42.495 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:12:42.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:42 smithi081 ceph-mon[31453]: Deploying daemon mds.e on smithi104 2024-04-12T21:12:42.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:42 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:boot 2024-04-12T21:12:42.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:42 smithi081 ceph-mon[31453]: fsmap 4 up:standby 2024-04-12T21:12:42.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:12:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:42 smithi153 ceph-mon[36569]: Deploying daemon mds.e on smithi104 2024-04-12T21:12:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:42 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:boot 2024-04-12T21:12:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:42 smithi153 ceph-mon[36569]: fsmap 4 up:standby 2024-04-12T21:12:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:12:43.020 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:43.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 -- 172.21.15.81:0/480987308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3690105430 msgr2=0x7f36901058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:43.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 --2- 172.21.15.81:0/480987308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3690105430 0x7f36901058b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3678009900 tx=0x7f367802bc30 comp rx=0 tx=0).stop 2024-04-12T21:12:43.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 -- 172.21.15.81:0/480987308 shutdown_connections 2024-04-12T21:12:43.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 --2- 172.21.15.81:0/480987308 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3690105df0 0x7f369010c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 --2- 172.21.15.81:0/480987308 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3690105430 0x7f36901058b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 --2- 172.21.15.81:0/480987308 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3690104240 0x7f3690104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 -- 172.21.15.81:0/480987308 >> 172.21.15.81:0/480987308 conn(0x7f36900ff7b0 msgr2=0x7f3690101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:43.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 -- 172.21.15.81:0/480987308 shutdown_connections 2024-04-12T21:12:43.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.251+0000 7f3695400640 1 -- 172.21.15.81:0/480987308 wait complete. 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.252+0000 7f3695400640 1 Processor -- start 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.252+0000 7f3695400640 1 -- start start 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f3695400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3690104240 0x7f369019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f3695400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3690105430 0x7f369019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f3695400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3690105df0 0x7f36901a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f3695400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f369010fed0 con 0x7f3690105df0 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f3695400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f369010fd50 con 0x7f3690105430 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f3695400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3690110050 con 0x7f3690104240 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368de00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3690105430 0x7f369019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3690105df0 0x7f36901a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:43.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368de00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3690105430 0x7f369019aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:50186/0 (socket says 172.21.15.81:50186) 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368de00640 1 -- 172.21.15.81:0/691169844 learned_addr learned my addr 172.21.15.81:0/691169844 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368e800640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3690104240 0x7f369019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368de00640 1 -- 172.21.15.81:0/691169844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3690104240 msgr2=0x7f369019a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368de00640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3690104240 0x7f369019a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368de00640 1 -- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3690105df0 msgr2=0x7f36901a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368de00640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3690105df0 0x7f36901a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.253+0000 7f368de00640 1 -- 172.21.15.81:0/691169844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36901a21a0 con 0x7f3690105430 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.254+0000 7f368de00640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3690105430 0x7f369019aac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3678009230 tx=0x7f367802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.254+0000 7f3687400640 1 -- 172.21.15.81:0/691169844 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678041070 con 0x7f3690105430 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.254+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36901a2430 con 0x7f3690105430 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.254+0000 7f3687400640 1 -- 172.21.15.81:0/691169844 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3678004450 con 0x7f3690105430 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.254+0000 7f3687400640 1 -- 172.21.15.81:0/691169844 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3678003db0 con 0x7f3690105430 2024-04-12T21:12:43.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.254+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36901a2970 con 0x7f3690105430 2024-04-12T21:12:43.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.255+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f369010b5b0 con 0x7f3690105430 2024-04-12T21:12:43.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.257+0000 7f3687400640 1 -- 172.21.15.81:0/691169844 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3678044070 con 0x7f3690105430 2024-04-12T21:12:43.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.258+0000 7f3687400640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36740a3bb0 0x7f36740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:43.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.259+0000 7f3687400640 1 -- 172.21.15.81:0/691169844 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f36780f1cf0 con 0x7f3690105430 2024-04-12T21:12:43.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.259+0000 7f368e800640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36740a3bb0 0x7f36740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:43.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.259+0000 7f368e800640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36740a3bb0 0x7f36740a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3690105290 tx=0x7f3680005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:43.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.263+0000 7f3687400640 1 -- 172.21.15.81:0/691169844 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36780bead0 con 0x7f3690105430 2024-04-12T21:12:43.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.507+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f369002d5e0 con 0x7f3690105430 2024-04-12T21:12:43.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.511+0000 7f3687400640 1 -- 172.21.15.81:0/691169844 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f36780be3d0 con 0x7f3690105430 2024-04-12T21:12:43.507 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:43.507 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-12T21:12:43.508 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 5 2024-04-12T21:12:43.510 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.514+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36740a3bb0 msgr2=0x7f36740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.514+0000 7f3695400640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36740a3bb0 0x7f36740a6070 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3690105290 tx=0x7f3680005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.514+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3690105430 msgr2=0x7f369019aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.514+0000 7f3695400640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3690105430 0x7f369019aac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3678009230 tx=0x7f367802c650 comp rx=0 tx=0).stop 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.515+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 shutdown_connections 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.515+0000 7f3695400640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36740a3bb0 0x7f36740a6070 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.515+0000 7f3695400640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3690105df0 0x7f36901a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.515+0000 7f3695400640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3690105430 0x7f369019aac0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.515+0000 7f3695400640 1 --2- 172.21.15.81:0/691169844 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3690104240 0x7f369019a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.515+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 >> 172.21.15.81:0/691169844 conn(0x7f36900ff7b0 msgr2=0x7f3690108780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.515+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 shutdown_connections 2024-04-12T21:12:43.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:43.515+0000 7f3695400640 1 -- 172.21.15.81:0/691169844 wait complete. 2024-04-12T21:12:43.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:43 smithi104 ceph-mon[36624]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:43.578 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:12:43 smithi104 systemd[1]: Started Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:43.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:43 smithi081 ceph-mon[31453]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:43 smithi153 ceph-mon[36569]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:44.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/691169844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:44.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:44.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: Deploying daemon mds.f on smithi153 2024-04-12T21:12:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:boot 2024-04-12T21:12:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: fsmap 5 up:standby 2024-04-12T21:12:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:12:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:12:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/691169844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: Deploying daemon mds.f on smithi153 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:boot 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: fsmap 5 up:standby 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:12:44.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:12:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/691169844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: Deploying daemon mds.f on smithi153 2024-04-12T21:12:44.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:44.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:boot 2024-04-12T21:12:44.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: fsmap 5 up:standby 2024-04-12T21:12:44.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:12:44.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:12:45.529 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:45.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.755+0000 7f75cfe00640 1 -- 172.21.15.81:0/2689057129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 msgr2=0x7f75d0078020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.755+0000 7f75cfe00640 1 --2- 172.21.15.81:0/2689057129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 0x7f75d0078020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f75bc009900 tx=0x7f75bc02bc50 comp rx=0 tx=0).stop 2024-04-12T21:12:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.755+0000 7f75cfe00640 1 -- 172.21.15.81:0/2689057129 shutdown_connections 2024-04-12T21:12:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.755+0000 7f75cfe00640 1 --2- 172.21.15.81:0/2689057129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 0x7f75d0078020 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.755+0000 7f75cfe00640 1 --2- 172.21.15.81:0/2689057129 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75d0078560 0x7f75d0078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.755+0000 7f75cfe00640 1 --2- 172.21.15.81:0/2689057129 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d0076980 0x7f75d0076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.755+0000 7f75cfe00640 1 -- 172.21.15.81:0/2689057129 >> 172.21.15.81:0/2689057129 conn(0x7f75d00ff5d0 msgr2=0x7f75d01019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.756+0000 7f75cfe00640 1 -- 172.21.15.81:0/2689057129 shutdown_connections 2024-04-12T21:12:45.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.756+0000 7f75cfe00640 1 -- 172.21.15.81:0/2689057129 wait complete. 2024-04-12T21:12:45.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.756+0000 7f75cfe00640 1 Processor -- start 2024-04-12T21:12:45.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cfe00640 1 -- start start 2024-04-12T21:12:45.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75d0076980 0x7f75d019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:45.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cfe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 0x7f75d019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cfe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d0078560 0x7f75d01a5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cfe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f75d0114110 con 0x7f75d0076980 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cfe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f75d0113f90 con 0x7f75d0078560 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cfe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f75d0114290 con 0x7f75d0077bc0 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75ce000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 0x7f75d019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75d0076980 0x7f75d019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75ce000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 0x7f75d019ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:51960/0 (socket says 172.21.15.81:51960) 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75ce000640 1 -- 172.21.15.81:0/1675942448 learned_addr learned my addr 172.21.15.81:0/1675942448 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.757+0000 7f75cf400640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d0078560 0x7f75d01a5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75ce000640 1 -- 172.21.15.81:0/1675942448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d0078560 msgr2=0x7f75d01a5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75ce000640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d0078560 0x7f75d01a5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75ce000640 1 -- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75d0076980 msgr2=0x7f75d019e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75ce000640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75d0076980 0x7f75d019e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:45.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75ce000640 1 -- 172.21.15.81:0/1675942448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d01a6300 con 0x7f75d0077bc0 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75cf400640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d0078560 0x7f75d01a5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75cea00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75d0076980 0x7f75d019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75ce000640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 0x7f75d019ec20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75bc02c260 tx=0x7f75bc02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75c7400640 1 -- 172.21.15.81:0/1675942448 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc041070 con 0x7f75d0077bc0 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75c7400640 1 -- 172.21.15.81:0/1675942448 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75bc004450 con 0x7f75d0077bc0 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75c7400640 1 -- 172.21.15.81:0/1675942448 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc034440 con 0x7f75d0077bc0 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.758+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d01a6590 con 0x7f75d0077bc0 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.759+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d01a6a90 con 0x7f75d0077bc0 2024-04-12T21:12:45.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.760+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d0110af0 con 0x7f75d0077bc0 2024-04-12T21:12:45.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.762+0000 7f75c7400640 1 -- 172.21.15.81:0/1675942448 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f75bc044070 con 0x7f75d0077bc0 2024-04-12T21:12:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.763+0000 7f75c7400640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75b40a3c30 0x7f75b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.763+0000 7f75c7400640 1 -- 172.21.15.81:0/1675942448 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f75bc0f1ca0 con 0x7f75d0077bc0 2024-04-12T21:12:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.763+0000 7f75cea00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75b40a3c30 0x7f75b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:45.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.764+0000 7f75cea00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75b40a3c30 0x7f75b40a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f75c00095f0 tx=0x7f75c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:45.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:45.765+0000 7f75c7400640 1 -- 172.21.15.81:0/1675942448 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f75bc0bea80 con 0x7f75d0077bc0 2024-04-12T21:12:46.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.012+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75d002d7a0 con 0x7f75d0077bc0 2024-04-12T21:12:46.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.013+0000 7f75c7400640 1 -- 172.21.15.81:0/1675942448 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7f75bc0be380 con 0x7f75d0077bc0 2024-04-12T21:12:46.009 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:46.009 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-12T21:12:46.010 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 6 2024-04-12T21:12:46.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.016+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75b40a3c30 msgr2=0x7f75b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:46.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.016+0000 7f75cfe00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75b40a3c30 0x7f75b40a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f75c00095f0 tx=0x7f75c0005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:46.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.016+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 msgr2=0x7f75d019ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.016+0000 7f75cfe00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 0x7f75d019ec20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75bc02c260 tx=0x7f75bc02c670 comp rx=0 tx=0).stop 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.017+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 shutdown_connections 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.017+0000 7f75cfe00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75b40a3c30 0x7f75b40a60f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.017+0000 7f75cfe00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75d0077bc0 0x7f75d019ec20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.017+0000 7f75cfe00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f75d0078560 0x7f75d01a5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.017+0000 7f75cfe00640 1 --2- 172.21.15.81:0/1675942448 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f75d0076980 0x7f75d019e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.017+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 >> 172.21.15.81:0/1675942448 conn(0x7f75d00ff5d0 msgr2=0x7f75d0100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.017+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 shutdown_connections 2024-04-12T21:12:46.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:46.017+0000 7f75cfe00640 1 -- 172.21.15.81:0/1675942448 wait complete. 2024-04-12T21:12:46.052 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:45 smithi153 systemd[1]: Starting Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:46.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:46 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1675942448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:46.318 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:46 smithi153 podman[67590]: 2024-04-12 21:12:45.970977156 +0000 UTC m=+0.018071013 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:46.318 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:46 smithi153 podman[67590]: 2024-04-12 21:12:46.086963479 +0000 UTC m=+0.134057319 container create bfc8f4961e5874992c88dbea567ca192a4975f7483d4283ce9cb8721f9e9d886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T21:12:46.318 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:46 smithi153 podman[67590]: 2024-04-12 21:12:46.323319342 +0000 UTC m=+0.370413183 container init bfc8f4961e5874992c88dbea567ca192a4975f7483d4283ce9cb8721f9e9d886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:12:46.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:46 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1675942448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:46.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:46 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1675942448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:46.802 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:46 smithi153 podman[67590]: 2024-04-12 21:12:46.326140279 +0000 UTC m=+0.373234120 container start bfc8f4961e5874992c88dbea567ca192a4975f7483d4283ce9cb8721f9e9d886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-12T21:12:46.802 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:46 smithi153 ceph-mds[67609]: -- 172.21.15.153:0/1014439521 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555a1dd21a40 con 0x555a1dcf4c00 2024-04-12T21:12:46.802 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:46 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: starting mds.f at 2024-04-12T21:12:46.803 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:46 smithi153 bash[67590]: bfc8f4961e5874992c88dbea567ca192a4975f7483d4283ce9cb8721f9e9d886 2024-04-12T21:12:46.803 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:12:46 smithi153 systemd[1]: Started Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:boot 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: fsmap 6 up:standby 2024-04-12T21:12:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:boot 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: fsmap 6 up:standby 2024-04-12T21:12:47.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:12:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:boot 2024-04-12T21:12:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: fsmap 6 up:standby 2024-04-12T21:12:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:12:48.032 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.257+0000 7f4ea0a00640 1 -- 172.21.15.81:0/350388653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 msgr2=0x7f4e9c110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:48.253 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.257+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/350388653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 0x7f4e9c110880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4e94009900 tx=0x7f4e9402bc30 comp rx=0 tx=0).stop 2024-04-12T21:12:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.258+0000 7f4ea0a00640 1 -- 172.21.15.81:0/350388653 shutdown_connections 2024-04-12T21:12:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.258+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/350388653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 0x7f4e9c110880 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.258+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/350388653 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c109630 0x7f4e9c109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.258+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/350388653 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c108440 0x7f4e9c108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.258+0000 7f4ea0a00640 1 -- 172.21.15.81:0/350388653 >> 172.21.15.81:0/350388653 conn(0x7f4e9c077400 msgr2=0x7f4e9c079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.258+0000 7f4ea0a00640 1 -- 172.21.15.81:0/350388653 shutdown_connections 2024-04-12T21:12:48.254 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.258+0000 7f4ea0a00640 1 -- 172.21.15.81:0/350388653 wait complete. 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4ea0a00640 1 Processor -- start 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4ea0a00640 1 -- start start 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4ea0a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c108440 0x7f4e9c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4ea0a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c109630 0x7f4e9c19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4ea0a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 0x7f4e9c1a5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4ea0a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c114120 con 0x7f4e9c109630 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4ea0a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c113fa0 con 0x7f4e9c108440 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4ea0a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c1142a0 con 0x7f4e9c109ff0 2024-04-12T21:12:48.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4e9aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c109630 0x7f4e9c19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4e9aa00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c109630 0x7f4e9c19ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57182/0 (socket says 172.21.15.81:57182) 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4e9be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 0x7f4e9c1a5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4e9aa00640 1 -- 172.21.15.81:0/1318225760 learned_addr learned my addr 172.21.15.81:0/1318225760 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.259+0000 7f4e9b400640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c108440 0x7f4e9c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e9be00640 1 -- 172.21.15.81:0/1318225760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c108440 msgr2=0x7f4e9c19e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e9be00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c108440 0x7f4e9c19e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e9be00640 1 -- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c109630 msgr2=0x7f4e9c19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e9be00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c109630 0x7f4e9c19ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e9be00640 1 -- 172.21.15.81:0/1318225760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c1a6360 con 0x7f4e9c109ff0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e9be00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 0x7f4e9c1a5cf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e9402c170 tx=0x7f4e9402ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e93e00640 1 -- 172.21.15.81:0/1318225760 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94043020 con 0x7f4e9c109ff0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9c1a65f0 con 0x7f4e9c109ff0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e93e00640 1 -- 172.21.15.81:0/1318225760 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e940054e0 con 0x7f4e9c109ff0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.260+0000 7f4e93e00640 1 -- 172.21.15.81:0/1318225760 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94003be0 con 0x7f4e9c109ff0 2024-04-12T21:12:48.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.261+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e9c1a6b60 con 0x7f4e9c109ff0 2024-04-12T21:12:48.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.262+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9c10f700 con 0x7f4e9c109ff0 2024-04-12T21:12:48.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.264+0000 7f4e93e00640 1 -- 172.21.15.81:0/1318225760 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4e94046020 con 0x7f4e9c109ff0 2024-04-12T21:12:48.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.265+0000 7f4e93e00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4e800a3bb0 0x7f4e800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:48.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.265+0000 7f4e93e00640 1 -- 172.21.15.81:0/1318225760 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4e940f2070 con 0x7f4e9c109ff0 2024-04-12T21:12:48.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.265+0000 7f4e9b400640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4e800a3bb0 0x7f4e800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:48.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.266+0000 7f4e9b400640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4e800a3bb0 0x7f4e800a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c109490 tx=0x7f4e8c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:48.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.269+0000 7f4e93e00640 1 -- 172.21.15.81:0/1318225760 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e9404d020 con 0x7f4e9c109ff0 2024-04-12T21:12:48.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.513+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e9c062bc0 con 0x7f4e9c109ff0 2024-04-12T21:12:48.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.514+0000 7f4e93e00640 1 -- 172.21.15.81:0/1318225760 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f4e940bee50 con 0x7f4e9c109ff0 2024-04-12T21:12:48.510 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:48.510 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-12T21:12:48.510 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 7 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4e800a3bb0 msgr2=0x7f4e800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4e800a3bb0 0x7f4e800a6070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c109490 tx=0x7f4e8c00b390 comp rx=0 tx=0).stop 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 msgr2=0x7f4e9c1a5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 0x7f4e9c1a5cf0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e9402c170 tx=0x7f4e9402ce60 comp rx=0 tx=0).stop 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 shutdown_connections 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4e800a3bb0 0x7f4e800a6070 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e9c109ff0 0x7f4e9c1a5cf0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4e9c109630 0x7f4e9c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 --2- 172.21.15.81:0/1318225760 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4e9c108440 0x7f4e9c19e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 >> 172.21.15.81:0/1318225760 conn(0x7f4e9c077400 msgr2=0x7f4e9c10c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 shutdown_connections 2024-04-12T21:12:48.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:48.517+0000 7f4ea0a00640 1 -- 172.21.15.81:0/1318225760 wait complete. 2024-04-12T21:12:48.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:48 smithi153 ceph-mon[36569]: Deploying daemon mds.c on smithi153 2024-04-12T21:12:48.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:48 smithi104 ceph-mon[36624]: Deploying daemon mds.c on smithi153 2024-04-12T21:12:48.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:48 smithi081 ceph-mon[31453]: Deploying daemon mds.c on smithi153 2024-04-12T21:12:48.958 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:48 smithi153 systemd[1]: Starting Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:48.958 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:48 smithi153 podman[67900]: 2024-04-12 21:12:48.865384584 +0000 UTC m=+0.017608085 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:49.214 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:48 smithi153 podman[67900]: 2024-04-12 21:12:48.992343703 +0000 UTC m=+0.144567194 container create 7c24e15e697f354995ba2d4198783d9ba82b7f821d527d2358f49969ba55bddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS) 2024-04-12T21:12:49.490 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:49 smithi153 podman[67900]: 2024-04-12 21:12:49.219892743 +0000 UTC m=+0.372116239 container init 7c24e15e697f354995ba2d4198783d9ba82b7f821d527d2358f49969ba55bddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-12T21:12:49.490 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:49 smithi153 podman[67900]: 2024-04-12 21:12:49.222677272 +0000 UTC m=+0.374900772 container start 7c24e15e697f354995ba2d4198783d9ba82b7f821d527d2358f49969ba55bddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-04-12T21:12:49.490 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mds[67919]: -- 172.21.15.153:0/507559364 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560759b61a40 con 0x560759b34800 2024-04-12T21:12:49.490 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: starting mds.c at 2024-04-12T21:12:49.490 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:49 smithi153 bash[67900]: 7c24e15e697f354995ba2d4198783d9ba82b7f821d527d2358f49969ba55bddc 2024-04-12T21:12:49.490 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:12:49 smithi153 systemd[1]: Started Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1318225760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:49 smithi153 ceph-mon[36569]: Deploying daemon mds.h on smithi104 2024-04-12T21:12:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1318225760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:49 smithi104 ceph-mon[36624]: Deploying daemon mds.h on smithi104 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1318225760' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:49 smithi081 ceph-mon[31453]: Deploying daemon mds.h on smithi104 2024-04-12T21:12:50.534 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 -- 172.21.15.81:0/1553771715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105df0 msgr2=0x7f948410c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 --2- 172.21.15.81:0/1553771715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105df0 0x7f948410c680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f947c009ca0 tx=0x7f947c02c100 comp rx=0 tx=0).stop 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 -- 172.21.15.81:0/1553771715 shutdown_connections 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 --2- 172.21.15.81:0/1553771715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9484105430 0x7f94841058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 --2- 172.21.15.81:0/1553771715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105df0 0x7f948410c680 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 --2- 172.21.15.81:0/1553771715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9484104240 0x7f9484104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 -- 172.21.15.81:0/1553771715 >> 172.21.15.81:0/1553771715 conn(0x7f94840ff7b0 msgr2=0x7f9484101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 -- 172.21.15.81:0/1553771715 shutdown_connections 2024-04-12T21:12:50.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.758+0000 7f948be00640 1 -- 172.21.15.81:0/1553771715 wait complete. 2024-04-12T21:12:50.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f948be00640 1 Processor -- start 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f948be00640 1 -- start start 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f948be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9484104240 0x7f948419a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f948be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105430 0x7f948419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f948be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9484105df0 0x7f94841a1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f948be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f948410fea0 con 0x7f9484105df0 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f948be00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f948410fd20 con 0x7f9484105430 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f948be00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9484110020 con 0x7f9484104240 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f9489e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9484105df0 0x7f94841a1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f9488a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105430 0x7f948419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f9488a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105430 0x7f948419aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:41120/0 (socket says 172.21.15.81:41120) 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f9488a00640 1 -- 172.21.15.81:0/1614112777 learned_addr learned my addr 172.21.15.81:0/1614112777 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:50.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.759+0000 7f9489400640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9484104240 0x7f948419a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9488a00640 1 -- 172.21.15.81:0/1614112777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9484104240 msgr2=0x7f948419a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9488a00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9484104240 0x7f948419a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9488a00640 1 -- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9484105df0 msgr2=0x7f94841a1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9488a00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9484105df0 0x7f94841a1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9488a00640 1 -- 172.21.15.81:0/1614112777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94841a2100 con 0x7f9484105430 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9488a00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105430 0x7f948419aa20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f947000ea60 tx=0x7f947000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9482000640 1 -- 172.21.15.81:0/1614112777 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947000cde0 con 0x7f9484105430 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9482000640 1 -- 172.21.15.81:0/1614112777 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94700035f0 con 0x7f9484105430 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94841a23f0 con 0x7f9484105430 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f9482000640 1 -- 172.21.15.81:0/1614112777 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470010670 con 0x7f9484105430 2024-04-12T21:12:50.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.760+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94841a2990 con 0x7f9484105430 2024-04-12T21:12:50.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.761+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f948410b560 con 0x7f9484105430 2024-04-12T21:12:50.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.764+0000 7f9482000640 1 -- 172.21.15.81:0/1614112777 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f94700037a0 con 0x7f9484105430 2024-04-12T21:12:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.765+0000 7f9482000640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94680a3c30 0x7f94680a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.765+0000 7f9482000640 1 -- 172.21.15.81:0/1614112777 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f94700d0430 con 0x7f9484105430 2024-04-12T21:12:50.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.766+0000 7f9489400640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94680a3c30 0x7f94680a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.766+0000 7f9489400640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94680a3c30 0x7f94680a60f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f94740095f0 tx=0x7f9474005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:50.762 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:50.767+0000 7f9482000640 1 -- 172.21.15.81:0/1614112777 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f947009d210 con 0x7f9484105430 2024-04-12T21:12:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:50 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:boot 2024-04-12T21:12:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:50 smithi153 ceph-mon[36569]: fsmap 7 up:standby 2024-04-12T21:12:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:12:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:50 smithi153 ceph-mon[36569]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:50.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:50 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:boot 2024-04-12T21:12:50.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:50 smithi104 ceph-mon[36624]: fsmap 7 up:standby 2024-04-12T21:12:50.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:12:50.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:50 smithi104 ceph-mon[36624]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:50.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:50 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:boot 2024-04-12T21:12:50.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:50 smithi081 ceph-mon[31453]: fsmap 7 up:standby 2024-04-12T21:12:50.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:12:50.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:50 smithi081 ceph-mon[31453]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:51.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.013+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f948402d5e0 con 0x7f9484105430 2024-04-12T21:12:51.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.014+0000 7f9482000640 1 -- 172.21.15.81:0/1614112777 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7f947009cb10 con 0x7f9484105430 2024-04-12T21:12:51.010 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:51.010 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-12T21:12:51.011 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 8 2024-04-12T21:12:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94680a3c30 msgr2=0x7f94680a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94680a3c30 0x7f94680a60f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f94740095f0 tx=0x7f9474005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105430 msgr2=0x7f948419aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105430 0x7f948419aa20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f947000ea60 tx=0x7f947000ef20 comp rx=0 tx=0).stop 2024-04-12T21:12:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 shutdown_connections 2024-04-12T21:12:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94680a3c30 0x7f94680a60f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9484105df0 0x7f94841a1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:51.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9484105430 0x7f948419aa20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 --2- 172.21.15.81:0/1614112777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9484104240 0x7f948419a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 >> 172.21.15.81:0/1614112777 conn(0x7f94840ff7b0 msgr2=0x7f9484108730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 shutdown_connections 2024-04-12T21:12:51.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:51.017+0000 7f948be00640 1 -- 172.21.15.81:0/1614112777 wait complete. 2024-04-12T21:12:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:51 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1614112777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:51.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:51 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1614112777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:51.812 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:51 smithi104 systemd[1]: Starting Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:51.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:51 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1614112777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:52.071 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:51 smithi104 podman[65901]: 2024-04-12 21:12:51.719490019 +0000 UTC m=+0.017249931 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:52.071 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:51 smithi104 podman[65901]: 2024-04-12 21:12:51.848098715 +0000 UTC m=+0.145858644 container create a442b0fc5a8921cb4a3be3b7c82830c8035020e9d03b3f79dcaccc0228d48ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-12T21:12:52.071 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:52 smithi104 podman[65901]: 2024-04-12 21:12:52.076223732 +0000 UTC m=+0.373983596 container init a442b0fc5a8921cb4a3be3b7c82830c8035020e9d03b3f79dcaccc0228d48ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:12:52.328 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:52 smithi104 podman[65901]: 2024-04-12 21:12:52.07886672 +0000 UTC m=+0.376626604 container start a442b0fc5a8921cb4a3be3b7c82830c8035020e9d03b3f79dcaccc0228d48ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:12:52.328 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:52 smithi104 ceph-mds[65921]: -- 172.21.15.104:0/2676494493 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558e212bba40 con 0x558e2128f000 2024-04-12T21:12:52.328 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: starting mds.h at 2024-04-12T21:12:52.328 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:52 smithi104 bash[65901]: a442b0fc5a8921cb4a3be3b7c82830c8035020e9d03b3f79dcaccc0228d48ead 2024-04-12T21:12:52.328 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:12:52 smithi104 systemd[1]: Started Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:53.033 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:53.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.264+0000 7f79fd000640 1 -- 172.21.15.81:0/1509607761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8076980 msgr2=0x7f79f8076da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:53.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.264+0000 7f79fd000640 1 --2- 172.21.15.81:0/1509607761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8076980 0x7f79f8076da0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009900 tx=0x7f79e802bc30 comp rx=0 tx=0).stop 2024-04-12T21:12:53.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.265+0000 7f79fd000640 1 -- 172.21.15.81:0/1509607761 shutdown_connections 2024-04-12T21:12:53.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.265+0000 7f79fd000640 1 --2- 172.21.15.81:0/1509607761 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f8077bc0 0x7f79f8078020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.265+0000 7f79fd000640 1 --2- 172.21.15.81:0/1509607761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79f8078560 0x7f79f8078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.265+0000 7f79fd000640 1 --2- 172.21.15.81:0/1509607761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8076980 0x7f79f8076da0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.265+0000 7f79fd000640 1 -- 172.21.15.81:0/1509607761 >> 172.21.15.81:0/1509607761 conn(0x7f79f80ff7b0 msgr2=0x7f79f8101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:53.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.265+0000 7f79fd000640 1 -- 172.21.15.81:0/1509607761 shutdown_connections 2024-04-12T21:12:53.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.265+0000 7f79fd000640 1 -- 172.21.15.81:0/1509607761 wait complete. 2024-04-12T21:12:53.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79fd000640 1 Processor -- start 2024-04-12T21:12:53.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79fd000640 1 -- start start 2024-04-12T21:12:53.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79fd000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f8076980 0x7f79f819e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:53.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79fd000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8077bc0 0x7f79f819eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:53.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79fd000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79f8078560 0x7f79f81a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:53.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79fd000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8114310 con 0x7f79f8077bc0 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79fd000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8114190 con 0x7f79f8078560 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79fd000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f79f8114490 con 0x7f79f8076980 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79f5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8077bc0 0x7f79f819eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79f5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f8076980 0x7f79f819e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79f5e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f8076980 0x7f79f819e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:52002/0 (socket says 172.21.15.81:52002) 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79f5e00640 1 -- 172.21.15.81:0/653796859 learned_addr learned my addr 172.21.15.81:0/653796859 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.266+0000 7f79f6800640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79f8078560 0x7f79f81a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79f5e00640 1 -- 172.21.15.81:0/653796859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79f8078560 msgr2=0x7f79f81a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79f5e00640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79f8078560 0x7f79f81a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79f5e00640 1 -- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8077bc0 msgr2=0x7f79f819eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79f5e00640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8077bc0 0x7f79f819eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79f5e00640 1 -- 172.21.15.81:0/653796859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f81a6580 con 0x7f79f8076980 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79f5400640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8077bc0 0x7f79f819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:53.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79f5e00640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f8076980 0x7f79f819e960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f79e8005ab0 tx=0x7f79e802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:53.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79eea00640 1 -- 172.21.15.81:0/653796859 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8043020 con 0x7f79f8076980 2024-04-12T21:12:53.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79eea00640 1 -- 172.21.15.81:0/653796859 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79e80054e0 con 0x7f79f8076980 2024-04-12T21:12:53.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79eea00640 1 -- 172.21.15.81:0/653796859 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8003ce0 con 0x7f79f8076980 2024-04-12T21:12:53.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.267+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f81a6810 con 0x7f79f8076980 2024-04-12T21:12:53.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.268+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f81a6d10 con 0x7f79f8076980 2024-04-12T21:12:53.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.269+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f8110d40 con 0x7f79f8076980 2024-04-12T21:12:53.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.270+0000 7f79eea00640 1 -- 172.21.15.81:0/653796859 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f79e8046020 con 0x7f79f8076980 2024-04-12T21:12:53.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.271+0000 7f79eea00640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f79dc0a3c30 0x7f79dc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:53.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.272+0000 7f79eea00640 1 -- 172.21.15.81:0/653796859 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f79e80f1fe0 con 0x7f79f8076980 2024-04-12T21:12:53.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.272+0000 7f79f5400640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f79dc0a3c30 0x7f79dc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:53.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.272+0000 7f79f5400640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f79dc0a3c30 0x7f79dc0a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f79f819fe70 tx=0x7f79e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:53.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.276+0000 7f79eea00640 1 -- 172.21.15.81:0/653796859 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79e804d020 con 0x7f79f8076980 2024-04-12T21:12:53.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: Deploying daemon mds.i on smithi153 2024-04-12T21:12:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:boot 2024-04-12T21:12:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: fsmap 8 up:standby 2024-04-12T21:12:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:12:53.518 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.523+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79f8062bc0 con 0x7f79f8076980 2024-04-12T21:12:53.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.523+0000 7f79eea00640 1 -- 172.21.15.81:0/653796859 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7f79e80bee20 con 0x7f79f8076980 2024-04-12T21:12:53.519 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:53.520 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-12T21:12:53.520 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 9 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f79dc0a3c30 msgr2=0x7f79dc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f79dc0a3c30 0x7f79dc0a60f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f79f819fe70 tx=0x7f79e4005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f8076980 msgr2=0x7f79f819e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f8076980 0x7f79f819e960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f79e8005ab0 tx=0x7f79e802c6c0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 shutdown_connections 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f79dc0a3c30 0x7f79dc0a60f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f79f8078560 0x7f79f81a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f79f8077bc0 0x7f79f819eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 --2- 172.21.15.81:0/653796859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f79f8076980 0x7f79f819e960 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 >> 172.21.15.81:0/653796859 conn(0x7f79f80ff7b0 msgr2=0x7f79f8100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 shutdown_connections 2024-04-12T21:12:53.524 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:53.528+0000 7f79fd000640 1 -- 172.21.15.81:0/653796859 wait complete. 2024-04-12T21:12:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:53.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: Deploying daemon mds.i on smithi153 2024-04-12T21:12:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:boot 2024-04-12T21:12:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: fsmap 8 up:standby 2024-04-12T21:12:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:12:53.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: Deploying daemon mds.i on smithi153 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:boot 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: fsmap 8 up:standby 2024-04-12T21:12:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:12:54.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:54 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/653796859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:54.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:54 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/653796859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:54.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:54 smithi153 systemd[1]: Starting Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:54.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:54 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/653796859' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:55.020 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:54 smithi153 podman[68209]: 2024-04-12 21:12:54.66197078 +0000 UTC m=+0.017649910 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:55.020 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:54 smithi153 podman[68209]: 2024-04-12 21:12:54.779201233 +0000 UTC m=+0.134880349 container create 06cc72315b98bc82b88a8146ba82705212198b952adc553d53b253798561350f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-12T21:12:55.020 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:54 smithi153 podman[68209]: 2024-04-12 21:12:54.99877754 +0000 UTC m=+0.354456657 container init 06cc72315b98bc82b88a8146ba82705212198b952adc553d53b253798561350f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:12:55.020 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:55 smithi153 podman[68209]: 2024-04-12 21:12:55.001389656 +0000 UTC m=+0.357068773 container start 06cc72315b98bc82b88a8146ba82705212198b952adc553d53b253798561350f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-12T21:12:55.297 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-mds[68229]: -- 172.21.15.153:0/1146228744 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564fb4db9a40 con 0x564fb4d8c800 2024-04-12T21:12:55.298 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: starting mds.i at 2024-04-12T21:12:55.298 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:55 smithi153 bash[68209]: 06cc72315b98bc82b88a8146ba82705212198b952adc553d53b253798561350f 2024-04-12T21:12:55.298 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:12:55 smithi153 systemd[1]: Started Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:55.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:55.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-mon[36569]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:55.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:55.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:55.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:55 smithi104 ceph-mon[36624]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:55.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:55 smithi081 ceph-mon[31453]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:55.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:55.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:12:55.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:12:55.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:12:55.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.771+0000 7fb233c00640 1 -- 172.21.15.81:0/3781467737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c105df0 msgr2=0x7fb22c10c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:55.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.771+0000 7fb233c00640 1 --2- 172.21.15.81:0/3781467737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c105df0 0x7fb22c10c680 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb224009ca0 tx=0x7fb22402c100 comp rx=0 tx=0).stop 2024-04-12T21:12:55.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.771+0000 7fb233c00640 1 -- 172.21.15.81:0/3781467737 shutdown_connections 2024-04-12T21:12:55.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.771+0000 7fb233c00640 1 --2- 172.21.15.81:0/3781467737 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c105df0 0x7fb22c10c680 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:55.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.771+0000 7fb233c00640 1 --2- 172.21.15.81:0/3781467737 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22c105430 0x7fb22c1058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:55.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.771+0000 7fb233c00640 1 --2- 172.21.15.81:0/3781467737 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c104240 0x7fb22c104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:55.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.771+0000 7fb233c00640 1 -- 172.21.15.81:0/3781467737 >> 172.21.15.81:0/3781467737 conn(0x7fb22c0ff7b0 msgr2=0x7fb22c101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:55.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.772+0000 7fb233c00640 1 -- 172.21.15.81:0/3781467737 shutdown_connections 2024-04-12T21:12:55.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.772+0000 7fb233c00640 1 -- 172.21.15.81:0/3781467737 wait complete. 2024-04-12T21:12:55.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb233c00640 1 Processor -- start 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb233c00640 1 -- start start 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb233c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c104240 0x7fb22c19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb233c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22c105430 0x7fb22c19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb233c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c105df0 0x7fb22c1a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb233c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c10fed0 con 0x7fb22c104240 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb233c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c10fd50 con 0x7fb22c105430 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb233c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c110050 con 0x7fb22c105df0 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.773+0000 7fb231200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c104240 0x7fb22c19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c104240 0x7fb22c19a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42478/0 (socket says 172.21.15.81:42478) 2024-04-12T21:12:55.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231200640 1 -- 172.21.15.81:0/1114956777 learned_addr learned my addr 172.21.15.81:0/1114956777 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c105df0 0x7fb22c1a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb22be00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22c105430 0x7fb22c19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231c00640 1 -- 172.21.15.81:0/1114956777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22c105430 msgr2=0x7fb22c19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22c105430 0x7fb22c19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231c00640 1 -- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c104240 msgr2=0x7fb22c19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c104240 0x7fb22c19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231c00640 1 -- 172.21.15.81:0/1114956777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c1a20d0 con 0x7fb22c105df0 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231200640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c104240 0x7fb22c19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.774+0000 7fb231c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c105df0 0x7fb22c1a1a60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb224037040 tx=0x7fb224035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.775+0000 7fb229600640 1 -- 172.21.15.81:0/1114956777 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224039040 con 0x7fb22c105df0 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.775+0000 7fb229600640 1 -- 172.21.15.81:0/1114956777 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb224044070 con 0x7fb22c105df0 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.775+0000 7fb229600640 1 -- 172.21.15.81:0/1114956777 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224004350 con 0x7fb22c105df0 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.775+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22c1a2360 con 0x7fb22c105df0 2024-04-12T21:12:55.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.775+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22c1a2870 con 0x7fb22c105df0 2024-04-12T21:12:55.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.776+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb22c10b560 con 0x7fb22c105df0 2024-04-12T21:12:55.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.778+0000 7fb229600640 1 -- 172.21.15.81:0/1114956777 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb2240044f0 con 0x7fb22c105df0 2024-04-12T21:12:55.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.779+0000 7fb229600640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2140a3c30 0x7fb2140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:55.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.779+0000 7fb229600640 1 -- 172.21.15.81:0/1114956777 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb2240f2f60 con 0x7fb22c105df0 2024-04-12T21:12:55.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.779+0000 7fb231200640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2140a3c30 0x7fb2140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:55.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.780+0000 7fb231200640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2140a3c30 0x7fb2140a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb2200095f0 tx=0x7fb220005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:55.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:55.781+0000 7fb229600640 1 -- 172.21.15.81:0/1114956777 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2240bfe60 con 0x7fb22c105df0 2024-04-12T21:12:56.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.047+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb22c02d5e0 con 0x7fb22c105df0 2024-04-12T21:12:56.044 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.048+0000 7fb229600640 1 -- 172.21.15.81:0/1114956777 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7fb2240bf760 con 0x7fb22c105df0 2024-04-12T21:12:56.044 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:56.044 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-12T21:12:56.044 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 10 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2140a3c30 msgr2=0x7fb2140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2140a3c30 0x7fb2140a60f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb2200095f0 tx=0x7fb220005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c105df0 msgr2=0x7fb22c1a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c105df0 0x7fb22c1a1a60 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb224037040 tx=0x7fb224035d10 comp rx=0 tx=0).stop 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 shutdown_connections 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2140a3c30 0x7fb2140a60f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb22c105df0 0x7fb22c1a1a60 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb22c105430 0x7fb22c19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:56.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 --2- 172.21.15.81:0/1114956777 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb22c104240 0x7fb22c19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:56.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 >> 172.21.15.81:0/1114956777 conn(0x7fb22c0ff7b0 msgr2=0x7fb22c108730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:56.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 shutdown_connections 2024-04-12T21:12:56.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:56.051+0000 7fb233c00640 1 -- 172.21.15.81:0/1114956777 wait complete. 2024-04-12T21:12:56.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:56 smithi081 ceph-mon[31453]: Deploying daemon mds.g on smithi081 2024-04-12T21:12:56.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:56 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:boot 2024-04-12T21:12:56.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:56 smithi081 ceph-mon[31453]: fsmap 9 up:standby 2024-04-12T21:12:56.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:12:56.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:12:56.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:56 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1114956777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:56.415 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:56 smithi153 ceph-mon[36569]: Deploying daemon mds.g on smithi081 2024-04-12T21:12:56.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:56 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:boot 2024-04-12T21:12:56.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:56 smithi153 ceph-mon[36569]: fsmap 9 up:standby 2024-04-12T21:12:56.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:12:56.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:12:56.416 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:56 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1114956777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:56 smithi104 ceph-mon[36624]: Deploying daemon mds.g on smithi081 2024-04-12T21:12:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:56 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:boot 2024-04-12T21:12:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:56 smithi104 ceph-mon[36624]: fsmap 9 up:standby 2024-04-12T21:12:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:12:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:12:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:56 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1114956777' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:57.575 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:57 smithi081 ceph-mon[31453]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:57.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:57 smithi104 ceph-mon[36624]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:57 smithi153 ceph-mon[36569]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:58.069 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:12:58.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 -- 172.21.15.81:0/2257880029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60078560 msgr2=0x7f2d60078e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:58.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 --2- 172.21.15.81:0/2257880029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60078560 0x7f2d60078e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f2d58009ca0 tx=0x7f2d5802c0f0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 -- 172.21.15.81:0/2257880029 shutdown_connections 2024-04-12T21:12:58.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 --2- 172.21.15.81:0/2257880029 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60078560 0x7f2d60078e00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 --2- 172.21.15.81:0/2257880029 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d60077bc0 0x7f2d60078020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 --2- 172.21.15.81:0/2257880029 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d60076980 0x7f2d60076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.473 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 -- 172.21.15.81:0/2257880029 >> 172.21.15.81:0/2257880029 conn(0x7f2d600ff7b0 msgr2=0x7f2d60101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:58.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 -- 172.21.15.81:0/2257880029 shutdown_connections 2024-04-12T21:12:58.474 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.477+0000 7f2d65000640 1 -- 172.21.15.81:0/2257880029 wait complete. 2024-04-12T21:12:58.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d65000640 1 Processor -- start 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d65000640 1 -- start start 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d65000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60076980 0x7f2d6019e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d65000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d60077bc0 0x7f2d6019edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d65000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d60078560 0x7f2d601a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d65000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f2d601142b0 con 0x7f2d60076980 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d65000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f2d60114130 con 0x7f2d60078560 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d65000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d60114430 con 0x7f2d60077bc0 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d5de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60076980 0x7f2d6019e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d5d400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d60077bc0 0x7f2d6019edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d5de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60076980 0x7f2d6019e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42498/0 (socket says 172.21.15.81:42498) 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.479+0000 7f2d5de00640 1 -- 172.21.15.81:0/1075972442 learned_addr learned my addr 172.21.15.81:0/1075972442 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5e800640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d60078560 0x7f2d601a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5d400640 1 -- 172.21.15.81:0/1075972442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d60078560 msgr2=0x7f2d601a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:58.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5d400640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d60078560 0x7f2d601a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5d400640 1 -- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60076980 msgr2=0x7f2d6019e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:58.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5d400640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60076980 0x7f2d6019e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5d400640 1 -- 172.21.15.81:0/1075972442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d601a64a0 con 0x7f2d60077bc0 2024-04-12T21:12:58.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5e800640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d60078560 0x7f2d601a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:12:58.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5de00640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60076980 0x7f2d6019e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:12:58.477 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d5d400640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d60077bc0 0x7f2d6019edc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00e9f0 tx=0x7f2d4c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:58.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d56a00640 1 -- 172.21.15.81:0/1075972442 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c00ccc0 con 0x7f2d60077bc0 2024-04-12T21:12:58.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d601a6790 con 0x7f2d60077bc0 2024-04-12T21:12:58.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d56a00640 1 -- 172.21.15.81:0/1075972442 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d4c010910 con 0x7f2d60077bc0 2024-04-12T21:12:58.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.480+0000 7f2d56a00640 1 -- 172.21.15.81:0/1075972442 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4c018960 con 0x7f2d60077bc0 2024-04-12T21:12:58.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.481+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d601a6d30 con 0x7f2d60077bc0 2024-04-12T21:12:58.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.482+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d60110cd0 con 0x7f2d60077bc0 2024-04-12T21:12:58.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.484+0000 7f2d56a00640 1 -- 172.21.15.81:0/1075972442 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f2d4c010ac0 con 0x7f2d60077bc0 2024-04-12T21:12:58.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.486+0000 7f2d56a00640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2d440a3c30 0x7f2d440a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:12:58.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.486+0000 7f2d56a00640 1 -- 172.21.15.81:0/1075972442 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2d4c0d0450 con 0x7f2d60077bc0 2024-04-12T21:12:58.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.486+0000 7f2d5de00640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2d440a3c30 0x7f2d440a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:12:58.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.487+0000 7f2d5de00640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2d440a3c30 0x7f2d440a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d500095f0 tx=0x7f2d50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:12:58.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.487+0000 7f2d56a00640 1 -- 172.21.15.81:0/1075972442 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d4c09d230 con 0x7f2d60077bc0 2024-04-12T21:12:58.652 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:58 smithi081 systemd[1]: Starting Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:12:58.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.745+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d6002d5e0 con 0x7f2d60077bc0 2024-04-12T21:12:58.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.746+0000 7f2d56a00640 1 -- 172.21.15.81:0/1075972442 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f2d4c09cb30 con 0x7f2d60077bc0 2024-04-12T21:12:58.742 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:12:58.742 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-12T21:12:58.743 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 10 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2d440a3c30 msgr2=0x7f2d440a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2d440a3c30 0x7f2d440a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2d500095f0 tx=0x7f2d50005f90 comp rx=0 tx=0).stop 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d60077bc0 msgr2=0x7f2d6019edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d60077bc0 0x7f2d6019edc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c00e9f0 tx=0x7f2d4c00eeb0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 shutdown_connections 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f2d440a3c30 0x7f2d440a60f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d60077bc0 0x7f2d6019edc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2d60078560 0x7f2d601a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 --2- 172.21.15.81:0/1075972442 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f2d60076980 0x7f2d6019e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 >> 172.21.15.81:0/1075972442 conn(0x7f2d600ff7b0 msgr2=0x7f2d60100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 shutdown_connections 2024-04-12T21:12:58.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:12:58.749+0000 7f2d65000640 1 -- 172.21.15.81:0/1075972442 wait complete. 2024-04-12T21:12:59.049 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:58 smithi081 podman[72738]: 2024-04-12 21:12:58.651316892 +0000 UTC m=+0.032796666 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:12:59.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:58 smithi081 podman[72738]: 2024-04-12 21:12:58.782309039 +0000 UTC m=+0.163788857 container create 2a629bf61cef78527afcdb7eaf83f62eec80c3d33cb94d66d00a01e37ad7bfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T21:12:59.050 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:59 smithi081 podman[72738]: 2024-04-12 21:12:59.053223945 +0000 UTC m=+0.434703762 container init 2a629bf61cef78527afcdb7eaf83f62eec80c3d33cb94d66d00a01e37ad7bfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-12T21:12:59.325 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:59 smithi081 podman[72738]: 2024-04-12 21:12:59.065195006 +0000 UTC m=+0.446674827 container start 2a629bf61cef78527afcdb7eaf83f62eec80c3d33cb94d66d00a01e37ad7bfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, GIT_CLEAN=True, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0) 2024-04-12T21:12:59.325 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:59 smithi081 ceph-mds[72758]: -- 172.21.15.81:0/3292533734 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b628eda40 con 0x559b628c1000 2024-04-12T21:12:59.325 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: starting mds.g at 2024-04-12T21:12:59.325 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:59 smithi081 bash[72738]: 2a629bf61cef78527afcdb7eaf83f62eec80c3d33cb94d66d00a01e37ad7bfc9 2024-04-12T21:12:59.325 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:12:59 smithi081 systemd[1]: Started Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:12:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:59 smithi104 ceph-mon[36624]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:59 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1075972442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:12:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:59.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:59 smithi081 ceph-mon[31453]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:59 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1075972442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:59.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:12:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:59 smithi153 ceph-mon[36569]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:12:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:59 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1075972442' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:12:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:12:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:12:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:00.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:00 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:boot 2024-04-12T21:13:00.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:00 smithi081 ceph-mon[31453]: fsmap 10 up:standby 2024-04-12T21:13:00.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:13:00.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:00.565 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:13:00.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:13:00.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:00.566 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:00 smithi081 ceph-mon[31453]: Deploying daemon mds.a on smithi081 2024-04-12T21:13:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:00 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:boot 2024-04-12T21:13:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:00 smithi104 ceph-mon[36624]: fsmap 10 up:standby 2024-04-12T21:13:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:13:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:13:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:13:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:00 smithi104 ceph-mon[36624]: Deploying daemon mds.a on smithi081 2024-04-12T21:13:00.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:13:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:00 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:boot 2024-04-12T21:13:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:00 smithi153 ceph-mon[36569]: fsmap 10 up:standby 2024-04-12T21:13:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:13:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:13:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:13:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:00.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:00 smithi153 ceph-mon[36569]: Deploying daemon mds.a on smithi081 2024-04-12T21:13:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 -- 172.21.15.81:0/2716013439 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a8100890 msgr2=0x7f29a810a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 --2- 172.21.15.81:0/2716013439 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a8100890 0x7f29a810a440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f29a0009be0 tx=0x7f29a002bef0 comp rx=0 tx=0).stop 2024-04-12T21:13:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 -- 172.21.15.81:0/2716013439 shutdown_connections 2024-04-12T21:13:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 --2- 172.21.15.81:0/2716013439 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a8100890 0x7f29a810a440 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 --2- 172.21.15.81:0/2716013439 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29a80ffed0 0x7f29a8100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 --2- 172.21.15.81:0/2716013439 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a8103460 0x7f29a8103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 -- 172.21.15.81:0/2716013439 >> 172.21.15.81:0/2716013439 conn(0x7f29a80fba50 msgr2=0x7f29a80fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:00.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 -- 172.21.15.81:0/2716013439 shutdown_connections 2024-04-12T21:13:00.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.969+0000 7f29aea00640 1 -- 172.21.15.81:0/2716013439 wait complete. 2024-04-12T21:13:00.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.970+0000 7f29aea00640 1 Processor -- start 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.970+0000 7f29aea00640 1 -- start start 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.970+0000 7f29aea00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a80ffed0 0x7f29a819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.970+0000 7f29aea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a8100890 0x7f29a819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.970+0000 7f29aea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29a8103460 0x7f29a81a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.970+0000 7f29aea00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f29a810db00 con 0x7f29a8103460 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.970+0000 7f29aea00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f29a810d980 con 0x7f29a80ffed0 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.970+0000 7f29aea00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29a810dc80 con 0x7f29a8100890 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ae000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29a8103460 0x7f29a81a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ae000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29a8103460 0x7f29a81a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42546/0 (socket says 172.21.15.81:42546) 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ae000640 1 -- 172.21.15.81:0/2899052858 learned_addr learned my addr 172.21.15.81:0/2899052858 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ad600640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a80ffed0 0x7f29a819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29acc00640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a8100890 0x7f29a819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:00.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ae000640 1 -- 172.21.15.81:0/2899052858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a8100890 msgr2=0x7f29a819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ae000640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a8100890 0x7f29a819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ae000640 1 -- 172.21.15.81:0/2899052858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a80ffed0 msgr2=0x7f29a819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ae000640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a80ffed0 0x7f29a819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.971+0000 7f29ae000640 1 -- 172.21.15.81:0/2899052858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a81a1ef0 con 0x7f29a8103460 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.972+0000 7f29ae000640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29a8103460 0x7f29a81a1880 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f29a0036040 tx=0x7f29a00337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.972+0000 7f29a6000640 1 -- 172.21.15.81:0/2899052858 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a0043020 con 0x7f29a8103460 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.972+0000 7f29a6000640 1 -- 172.21.15.81:0/2899052858 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29a002ca60 con 0x7f29a8103460 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.972+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29a81a2180 con 0x7f29a8103460 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.972+0000 7f29a6000640 1 -- 172.21.15.81:0/2899052858 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a0034690 con 0x7f29a8103460 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.972+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29a81a2720 con 0x7f29a8103460 2024-04-12T21:13:00.969 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.973+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29a81a2920 con 0x7f29a8103460 2024-04-12T21:13:00.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.975+0000 7f29a6000640 1 -- 172.21.15.81:0/2899052858 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f29a0046020 con 0x7f29a8103460 2024-04-12T21:13:00.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.976+0000 7f29a6000640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29900a3c30 0x7f29900a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:00.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.977+0000 7f29a6000640 1 -- 172.21.15.81:0/2899052858 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29a00f2580 con 0x7f29a8103460 2024-04-12T21:13:00.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.977+0000 7f29ad600640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29900a3c30 0x7f29900a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:00.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.977+0000 7f29ad600640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29900a3c30 0x7f29900a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29a806a200 tx=0x7f299c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:00.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:00.981+0000 7f29a6000640 1 -- 172.21.15.81:0/2899052858 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29a004d020 con 0x7f29a8103460 2024-04-12T21:13:01.221 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.225+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29a81a2af0 con 0x7f29a8103460 2024-04-12T21:13:01.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.227+0000 7f29a6000640 1 -- 172.21.15.81:0/2899052858 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f29a00dde20 con 0x7f29a8103460 2024-04-12T21:13:01.223 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:01.223 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-12T21:13:01.224 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 11 2024-04-12T21:13:01.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29900a3c30 msgr2=0x7f29900a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:01.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29900a3c30 0x7f29900a60f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f29a806a200 tx=0x7f299c005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:01.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29a8103460 msgr2=0x7f29a81a1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29a8103460 0x7f29a81a1880 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f29a0036040 tx=0x7f29a00337b0 comp rx=0 tx=0).stop 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 shutdown_connections 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29900a3c30 0x7f29900a60f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29a8100890 0x7f29a819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29a8103460 0x7f29a81a1880 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 --2- 172.21.15.81:0/2899052858 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29a80ffed0 0x7f29a819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.230+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 >> 172.21.15.81:0/2899052858 conn(0x7f29a80fba50 msgr2=0x7f29a8100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.231+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 shutdown_connections 2024-04-12T21:13:01.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:01.231+0000 7f29aea00640 1 -- 172.21.15.81:0/2899052858 wait complete. 2024-04-12T21:13:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:01 smithi081 ceph-mon[31453]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:01 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2899052858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:01 smithi153 ceph-mon[36569]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:01 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2899052858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:01.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:01 smithi104 ceph-mon[36624]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:01 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2899052858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:03.040 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:02 smithi081 systemd[1]: Starting Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:13:03.246 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:13:03.314 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 podman[73084]: 2024-04-12 21:13:02.950023641 +0000 UTC m=+0.034593607 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:13:03.314 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 podman[73084]: 2024-04-12 21:13:03.082164167 +0000 UTC m=+0.166734164 container create dfcc5c53b4f2b296dc3e2295c408fc86f0ca0efbbf441d85b081929657b2c02a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 -- 172.21.15.81:0/3860395741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7da4078560 msgr2=0x7f7da4078e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 --2- 172.21.15.81:0/3860395741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7da4078560 0x7f7da4078e00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c009be0 tx=0x7f7d9c02bef0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 -- 172.21.15.81:0/3860395741 shutdown_connections 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 --2- 172.21.15.81:0/3860395741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7da4078560 0x7f7da4078e00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 --2- 172.21.15.81:0/3860395741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4077bc0 0x7f7da4078020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 --2- 172.21.15.81:0/3860395741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7da4076980 0x7f7da4076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 -- 172.21.15.81:0/3860395741 >> 172.21.15.81:0/3860395741 conn(0x7f7da40ff7b0 msgr2=0x7f7da4101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 -- 172.21.15.81:0/3860395741 shutdown_connections 2024-04-12T21:13:03.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.453+0000 7f7da8e00640 1 -- 172.21.15.81:0/3860395741 wait complete. 2024-04-12T21:13:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.454+0000 7f7da8e00640 1 Processor -- start 2024-04-12T21:13:03.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.454+0000 7f7da8e00640 1 -- start start 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.454+0000 7f7da8e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7da4076980 0x7f7da419e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.454+0000 7f7da8e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7da4077bc0 0x7f7da419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.454+0000 7f7da8e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4078560 0x7f7da41a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.454+0000 7f7da8e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7da41142d0 con 0x7f7da4077bc0 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.454+0000 7f7da8e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7da4114150 con 0x7f7da4078560 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.454+0000 7f7da8e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7da4114450 con 0x7f7da4076980 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7da4077bc0 0x7f7da419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7da4077bc0 0x7f7da419ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:42570/0 (socket says 172.21.15.81:42570) 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 -- 172.21.15.81:0/215101701 learned_addr learned my addr 172.21.15.81:0/215101701 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7da4076980 0x7f7da419e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da2800640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4078560 0x7f7da41a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 -- 172.21.15.81:0/215101701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7da4076980 msgr2=0x7f7da419e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:03.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7da4076980 0x7f7da419e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 -- 172.21.15.81:0/215101701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4078560 msgr2=0x7f7da41a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4078560 0x7f7da41a5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 -- 172.21.15.81:0/215101701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da41a64f0 con 0x7f7da4077bc0 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.455+0000 7f7da1400640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7da4077bc0 0x7f7da419ee10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e9f0 tx=0x7f7d9000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.456+0000 7f7d9aa00640 1 -- 172.21.15.81:0/215101701 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000ccc0 con 0x7f7da4077bc0 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.456+0000 7f7d9aa00640 1 -- 172.21.15.81:0/215101701 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d9000ce60 con 0x7f7da4077bc0 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.456+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da41a67e0 con 0x7f7da4077bc0 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.456+0000 7f7d9aa00640 1 -- 172.21.15.81:0/215101701 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90010690 con 0x7f7da4077bc0 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.456+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da41a6ce0 con 0x7f7da4077bc0 2024-04-12T21:13:03.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.457+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da4110d00 con 0x7f7da4077bc0 2024-04-12T21:13:03.455 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.459+0000 7f7d9aa00640 1 -- 172.21.15.81:0/215101701 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7d900046c0 con 0x7f7da4077bc0 2024-04-12T21:13:03.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.461+0000 7f7d9aa00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7d880a3bb0 0x7f7d880a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:03.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.461+0000 7f7d9aa00640 1 -- 172.21.15.81:0/215101701 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f7d900d05f0 con 0x7f7da4077bc0 2024-04-12T21:13:03.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.461+0000 7f7da1e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7d880a3bb0 0x7f7d880a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:03.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.462+0000 7f7da1e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7d880a3bb0 0x7f7d880a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7d94005fd0 tx=0x7f7d94009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:03.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.464+0000 7f7d9aa00640 1 -- 172.21.15.81:0/215101701 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d90053ff0 con 0x7f7da4077bc0 2024-04-12T21:13:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:03 smithi081 ceph-mon[31453]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:03.652 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 podman[73084]: 2024-04-12 21:13:03.378064517 +0000 UTC m=+0.462634554 container init dfcc5c53b4f2b296dc3e2295c408fc86f0ca0efbbf441d85b081929657b2c02a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-12T21:13:03.652 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 podman[73084]: 2024-04-12 21:13:03.386621288 +0000 UTC m=+0.471191316 container start dfcc5c53b4f2b296dc3e2295c408fc86f0ca0efbbf441d85b081929657b2c02a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-12T21:13:03.652 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 ceph-mds[73123]: -- 172.21.15.81:0/2460111954 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564fff1b64e0 con 0x564fff18cc00 2024-04-12T21:13:03.652 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 ceph-mds[73123]: -- 172.21.15.81:0/2460111954 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564fff1b9a40 con 0x564fff18cc00 2024-04-12T21:13:03.653 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: starting mds.a at 2024-04-12T21:13:03.653 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 bash[73084]: dfcc5c53b4f2b296dc3e2295c408fc86f0ca0efbbf441d85b081929657b2c02a 2024-04-12T21:13:03.653 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:13:03 smithi081 systemd[1]: Started Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:13:03.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.743+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7da402d830 con 0x7f7da4077bc0 2024-04-12T21:13:03.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.744+0000 7f7d9aa00640 1 -- 172.21.15.81:0/215101701 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f7d9009d3d0 con 0x7f7da4077bc0 2024-04-12T21:13:03.740 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:03.740 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-12T21:13:03.741 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 11 2024-04-12T21:13:03.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7d880a3bb0 msgr2=0x7f7d880a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:03.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7d880a3bb0 0x7f7d880a6070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7d94005fd0 tx=0x7f7d94009260 comp rx=0 tx=0).stop 2024-04-12T21:13:03.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7da4077bc0 msgr2=0x7f7da419ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:03.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7da4077bc0 0x7f7da419ee10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000e9f0 tx=0x7f7d9000edf0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 shutdown_connections 2024-04-12T21:13:03.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7d880a3bb0 0x7f7d880a6070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7da4078560 0x7f7da41a5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7da4077bc0 0x7f7da419ee10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 --2- 172.21.15.81:0/215101701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7da4076980 0x7f7da419e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:03.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 >> 172.21.15.81:0/215101701 conn(0x7f7da40ff7b0 msgr2=0x7f7da4100e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:03.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 shutdown_connections 2024-04-12T21:13:03.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:03.747+0000 7f7da8e00640 1 -- 172.21.15.81:0/215101701 wait complete. 2024-04-12T21:13:03.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:03 smithi153 ceph-mon[36569]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:03.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:03 smithi104 ceph-mon[36624]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: Deploying daemon mds.b on smithi104 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/215101701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:boot 2024-04-12T21:13:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: fsmap 11 up:standby 2024-04-12T21:13:04.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: Deploying daemon mds.b on smithi104 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/215101701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:boot 2024-04-12T21:13:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: fsmap 11 up:standby 2024-04-12T21:13:04.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:13:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: Deploying daemon mds.b on smithi104 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/215101701' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:boot 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: fsmap 11 up:standby 2024-04-12T21:13:05.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:13:05.763 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.986+0000 7efded000640 1 -- 172.21.15.81:0/1323211952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efde8101010 msgr2=0x7efde8101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.986+0000 7efded000640 1 --2- 172.21.15.81:0/1323211952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efde8101010 0x7efde8101470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009900 tx=0x7efdd402bc40 comp rx=0 tx=0).stop 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.987+0000 7efded000640 1 -- 172.21.15.81:0/1323211952 shutdown_connections 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.987+0000 7efded000640 1 --2- 172.21.15.81:0/1323211952 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efde81019b0 0x7efde8101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.987+0000 7efded000640 1 --2- 172.21.15.81:0/1323211952 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efde8101010 0x7efde8101470 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.987+0000 7efded000640 1 --2- 172.21.15.81:0/1323211952 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde81006b0 0x7efde8100ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.987+0000 7efded000640 1 -- 172.21.15.81:0/1323211952 >> 172.21.15.81:0/1323211952 conn(0x7efde80fc290 msgr2=0x7efde80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.987+0000 7efded000640 1 -- 172.21.15.81:0/1323211952 shutdown_connections 2024-04-12T21:13:05.983 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.987+0000 7efded000640 1 -- 172.21.15.81:0/1323211952 wait complete. 2024-04-12T21:13:05.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.987+0000 7efded000640 1 Processor -- start 2024-04-12T21:13:05.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efded000640 1 -- start start 2024-04-12T21:13:05.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efded000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde81006b0 0x7efde819a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efded000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efde8101010 0x7efde819a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efded000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efde81019b0 0x7efde81a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efded000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efde810fd30 con 0x7efde81019b0 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efded000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efde810fbb0 con 0x7efde8101010 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efded000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efde810feb0 con 0x7efde81006b0 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efde7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde81006b0 0x7efde819a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efde7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde81006b0 0x7efde819a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:60062/0 (socket says 172.21.15.81:60062) 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efde7400640 1 -- 172.21.15.81:0/963002980 learned_addr learned my addr 172.21.15.81:0/963002980 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efde7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efde81019b0 0x7efde81a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.988+0000 7efde6a00640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efde8101010 0x7efde819a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efde7400640 1 -- 172.21.15.81:0/963002980 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efde8101010 msgr2=0x7efde819a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efde7400640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efde8101010 0x7efde819a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efde7400640 1 -- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efde81019b0 msgr2=0x7efde81a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:05.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efde7400640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efde81019b0 0x7efde81a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:05.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efde7400640 1 -- 172.21.15.81:0/963002980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde81a1f60 con 0x7efde81006b0 2024-04-12T21:13:05.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efde7400640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde81006b0 0x7efde819a340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efdd800e9f0 tx=0x7efdd800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:05.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efddfe00640 1 -- 172.21.15.81:0/963002980 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd800cd80 con 0x7efde81006b0 2024-04-12T21:13:05.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efded000640 1 -- 172.21.15.81:0/963002980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde81a2250 con 0x7efde81006b0 2024-04-12T21:13:05.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efddfe00640 1 -- 172.21.15.81:0/963002980 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdd8010910 con 0x7efde81006b0 2024-04-12T21:13:05.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.989+0000 7efddfe00640 1 -- 172.21.15.81:0/963002980 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd80188f0 con 0x7efde81006b0 2024-04-12T21:13:05.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.990+0000 7efded000640 1 -- 172.21.15.81:0/963002980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde81a2770 con 0x7efde81006b0 2024-04-12T21:13:05.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.991+0000 7efded000640 1 -- 172.21.15.81:0/963002980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde810b400 con 0x7efde81006b0 2024-04-12T21:13:05.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.993+0000 7efddfe00640 1 -- 172.21.15.81:0/963002980 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efdd8010ac0 con 0x7efde81006b0 2024-04-12T21:13:05.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.994+0000 7efddfe00640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efdcc0a3bb0 0x7efdcc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.994+0000 7efddfe00640 1 -- 172.21.15.81:0/963002980 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efdd80d0730 con 0x7efde81006b0 2024-04-12T21:13:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.994+0000 7efde6a00640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efdcc0a3bb0 0x7efdcc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:05.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.995+0000 7efde6a00640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efdcc0a3bb0 0x7efdcc0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efdd402c250 tx=0x7efdd402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:05.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:05.997+0000 7efddfe00640 1 -- 172.21.15.81:0/963002980 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efdd809cec0 con 0x7efde81006b0 2024-04-12T21:13:06.037 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:05 smithi104 systemd[1]: Starting Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:13:06.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.238+0000 7efded000640 1 -- 172.21.15.81:0/963002980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efde802d5e0 con 0x7efde81006b0 2024-04-12T21:13:06.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.239+0000 7efddfe00640 1 -- 172.21.15.81:0/963002980 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9659 (secure 0 0 0) 0x7efdd80a11c0 con 0x7efde81006b0 2024-04-12T21:13:06.235 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:06.235 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-12T21:13:06.236 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 12 2024-04-12T21:13:06.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.242+0000 7efded000640 1 -- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efdcc0a3bb0 msgr2=0x7efdcc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:06.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.242+0000 7efded000640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efdcc0a3bb0 0x7efdcc0a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efdd402c250 tx=0x7efdd402cf90 comp rx=0 tx=0).stop 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.242+0000 7efded000640 1 -- 172.21.15.81:0/963002980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde81006b0 msgr2=0x7efde819a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.242+0000 7efded000640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde81006b0 0x7efde819a340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efdd800e9f0 tx=0x7efdd800eeb0 comp rx=0 tx=0).stop 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.243+0000 7efded000640 1 -- 172.21.15.81:0/963002980 shutdown_connections 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.243+0000 7efded000640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efdcc0a3bb0 0x7efdcc0a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.243+0000 7efded000640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efde81019b0 0x7efde81a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.243+0000 7efded000640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efde8101010 0x7efde819a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.243+0000 7efded000640 1 --2- 172.21.15.81:0/963002980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efde81006b0 0x7efde819a340 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.243+0000 7efded000640 1 -- 172.21.15.81:0/963002980 >> 172.21.15.81:0/963002980 conn(0x7efde80fc290 msgr2=0x7efde80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.243+0000 7efded000640 1 -- 172.21.15.81:0/963002980 shutdown_connections 2024-04-12T21:13:06.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:06.243+0000 7efded000640 1 -- 172.21.15.81:0/963002980 wait complete. 2024-04-12T21:13:06.288 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:06 smithi104 podman[66212]: 2024-04-12 21:13:05.944960624 +0000 UTC m=+0.017308926 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:13:06.289 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:06 smithi104 podman[66212]: 2024-04-12 21:13:06.059629554 +0000 UTC m=+0.131977891 container create ddefba9658f604ca54adb31278347c1d0555779c01f437f0ddd12a318aaebe98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-12T21:13:06.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:06 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/963002980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:06.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:06 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/963002980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:06.578 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:06 smithi104 podman[66212]: 2024-04-12 21:13:06.295365569 +0000 UTC m=+0.367713858 container init ddefba9658f604ca54adb31278347c1d0555779c01f437f0ddd12a318aaebe98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS) 2024-04-12T21:13:06.578 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:06 smithi104 podman[66212]: 2024-04-12 21:13:06.297931021 +0000 UTC m=+0.370279318 container start ddefba9658f604ca54adb31278347c1d0555779c01f437f0ddd12a318aaebe98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-12T21:13:06.578 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:06 smithi104 ceph-mds[66231]: -- 172.21.15.104:0/1822789905 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5637704cfa40 con 0x5637704a2c00 2024-04-12T21:13:06.578 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: starting mds.b at 2024-04-12T21:13:06.578 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:06 smithi104 bash[66212]: ddefba9658f604ca54adb31278347c1d0555779c01f437f0ddd12a318aaebe98 2024-04-12T21:13:06.578 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:06 smithi104 systemd[1]: Started Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:13:06.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:06 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/963002980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:07.711 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:07.711 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.711 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.711 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.711 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:13:07.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] up:boot 2024-04-12T21:13:07.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: fsmap 12 up:standby 2024-04-12T21:13:07.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:13:07.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:13:07.712 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.718 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] up:boot 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: fsmap 12 up:standby 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:13:07.719 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] up:boot 2024-04-12T21:13:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: fsmap 12 up:standby 2024-04-12T21:13:07.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:13:07.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:13:07.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:08.259 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:13:08.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 -- 172.21.15.81:0/4274698213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5d8101e60 msgr2=0x7fb5d8102280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:08.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 --2- 172.21.15.81:0/4274698213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5d8101e60 0x7fb5d8102280 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009900 tx=0x7fb5c802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:08.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 -- 172.21.15.81:0/4274698213 shutdown_connections 2024-04-12T21:13:08.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 --2- 172.21.15.81:0/4274698213 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d8103970 0x7fb5d810a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 --2- 172.21.15.81:0/4274698213 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5d8102fb0 0x7fb5d8103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 --2- 172.21.15.81:0/4274698213 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5d8101e60 0x7fb5d8102280 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.481 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 -- 172.21.15.81:0/4274698213 >> 172.21.15.81:0/4274698213 conn(0x7fb5d80fd3d0 msgr2=0x7fb5d80ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:08.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 -- 172.21.15.81:0/4274698213 shutdown_connections 2024-04-12T21:13:08.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.485+0000 7fb5dd200640 1 -- 172.21.15.81:0/4274698213 wait complete. 2024-04-12T21:13:08.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.486+0000 7fb5dd200640 1 Processor -- start 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.486+0000 7fb5dd200640 1 -- start start 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5dd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5d8101e60 0x7fb5d8195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5dd200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d8102fb0 0x7fb5d8196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5dd200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5d8103970 0x7fb5d819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5dd200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d810b980 con 0x7fb5d8101e60 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5dd200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d810b800 con 0x7fb5d8103970 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5dd200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d810bb00 con 0x7fb5d8102fb0 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5d8103970 0x7fb5d819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d6a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d8102fb0 0x7fb5d8196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5d8101e60 0x7fb5d8195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5d8103970 0x7fb5d819d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52436/0 (socket says 172.21.15.81:52436) 2024-04-12T21:13:08.483 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7e00640 1 -- 172.21.15.81:0/973097666 learned_addr learned my addr 172.21.15.81:0/973097666 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:08.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7e00640 1 -- 172.21.15.81:0/973097666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d8102fb0 msgr2=0x7fb5d8196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:08.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7e00640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d8102fb0 0x7fb5d8196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7e00640 1 -- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5d8101e60 msgr2=0x7fb5d8195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7e00640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5d8101e60 0x7fb5d8195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.487+0000 7fb5d7e00640 1 -- 172.21.15.81:0/973097666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d819db10 con 0x7fb5d8103970 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.488+0000 7fb5d7e00640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5d8103970 0x7fb5d819d4a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000eb10 tx=0x7fb5d000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.488+0000 7fb5cfe00640 1 -- 172.21.15.81:0/973097666 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0004b20 con 0x7fb5d8103970 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.488+0000 7fb5cfe00640 1 -- 172.21.15.81:0/973097666 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5d0004cc0 con 0x7fb5d8103970 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.488+0000 7fb5cfe00640 1 -- 172.21.15.81:0/973097666 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0010760 con 0x7fb5d8103970 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.488+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d819de00 con 0x7fb5d8103970 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.488+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d819e270 con 0x7fb5d8103970 2024-04-12T21:13:08.485 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.490+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d8109110 con 0x7fb5d8103970 2024-04-12T21:13:08.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.491+0000 7fb5cfe00640 1 -- 172.21.15.81:0/973097666 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb5d0005020 con 0x7fb5d8103970 2024-04-12T21:13:08.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.492+0000 7fb5cfe00640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb5bc0a3bb0 0x7fb5bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:08.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.493+0000 7fb5cfe00640 1 -- 172.21.15.81:0/973097666 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb5d00d06a0 con 0x7fb5d8103970 2024-04-12T21:13:08.488 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.493+0000 7fb5d7400640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb5bc0a3bb0 0x7fb5bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:08.489 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.493+0000 7fb5d7400640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb5bc0a3bb0 0x7fb5bc0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb5c802c240 tx=0x7fb5c802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:08.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.497+0000 7fb5cfe00640 1 -- 172.21.15.81:0/973097666 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5d009d480 con 0x7fb5d8103970 2024-04-12T21:13:08.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.745+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5d8062c60 con 0x7fb5d8103970 2024-04-12T21:13:08.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.746+0000 7fb5cfe00640 1 -- 172.21.15.81:0/973097666 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10488 (secure 0 0 0) 0x7fb5d009cd80 con 0x7fb5d8103970 2024-04-12T21:13:08.742 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:08.742 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24593,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-12T21:13:08.743 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 13 2024-04-12T21:13:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb5bc0a3bb0 msgr2=0x7fb5bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb5bc0a3bb0 0x7fb5bc0a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb5c802c240 tx=0x7fb5c802df90 comp rx=0 tx=0).stop 2024-04-12T21:13:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5d8103970 msgr2=0x7fb5d819d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:08.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5d8103970 0x7fb5d819d4a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000eb10 tx=0x7fb5d000c2d0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 shutdown_connections 2024-04-12T21:13:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb5bc0a3bb0 0x7fb5bc0a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5d8102fb0 0x7fb5d8196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5d8103970 0x7fb5d819d4a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 --2- 172.21.15.81:0/973097666 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb5d8101e60 0x7fb5d8195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 >> 172.21.15.81:0/973097666 conn(0x7fb5d80fd3d0 msgr2=0x7fb5d81062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.749+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 shutdown_connections 2024-04-12T21:13:08.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.750+0000 7fb5dd200640 1 -- 172.21.15.81:0/973097666 wait complete. 2024-04-12T21:13:08.764 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:08.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.956+0000 7fe01ca00640 1 -- 172.21.15.81:0/731923946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 msgr2=0x7fe018100d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.956+0000 7fe01ca00640 1 --2- 172.21.15.81:0/731923946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 0x7fe018100d30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe008009900 tx=0x7fe00802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.957+0000 7fe01ca00640 1 -- 172.21.15.81:0/731923946 shutdown_connections 2024-04-12T21:13:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.957+0000 7fe01ca00640 1 --2- 172.21.15.81:0/731923946 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe018101270 0x7fe0181016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.957+0000 7fe01ca00640 1 --2- 172.21.15.81:0/731923946 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe018101c10 0x7fe018102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.957+0000 7fe01ca00640 1 --2- 172.21.15.81:0/731923946 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 0x7fe018100d30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.957+0000 7fe01ca00640 1 -- 172.21.15.81:0/731923946 >> 172.21.15.81:0/731923946 conn(0x7fe0180fc4b0 msgr2=0x7fe0180fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.957+0000 7fe01ca00640 1 -- 172.21.15.81:0/731923946 shutdown_connections 2024-04-12T21:13:08.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.957+0000 7fe01ca00640 1 -- 172.21.15.81:0/731923946 wait complete. 2024-04-12T21:13:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.958+0000 7fe01ca00640 1 Processor -- start 2024-04-12T21:13:08.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.958+0000 7fe01ca00640 1 -- start start 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe01ca00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 0x7fe01819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe01ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe018101270 0x7fe01819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe01ca00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe018101c10 0x7fe0181a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe01ca00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fe01810ff30 con 0x7fe018101270 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe01ca00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fe01810fdb0 con 0x7fe018100910 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe01ca00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe0181100b0 con 0x7fe018101c10 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe015e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 0x7fe01819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe015e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 0x7fe01819a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52458/0 (socket says 172.21.15.81:52458) 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe00fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe018101270 0x7fe01819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe015e00640 1 -- 172.21.15.81:0/1134686257 learned_addr learned my addr 172.21.15.81:0/1134686257 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:08.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe016800640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe018101c10 0x7fe0181a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe015e00640 1 -- 172.21.15.81:0/1134686257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe018101c10 msgr2=0x7fe0181a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe015e00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe018101c10 0x7fe0181a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe015e00640 1 -- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe018101270 msgr2=0x7fe01819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe015e00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe018101270 0x7fe01819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.959+0000 7fe015e00640 1 -- 172.21.15.81:0/1134686257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0181a21e0 con 0x7fe018100910 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.960+0000 7fe015e00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 0x7fe01819a5c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe00802c8e0 tx=0x7fe0080049c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.960+0000 7fe00ea00640 1 -- 172.21.15.81:0/1134686257 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008043020 con 0x7fe018100910 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.960+0000 7fe00ea00640 1 -- 172.21.15.81:0/1134686257 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0080054e0 con 0x7fe018100910 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.960+0000 7fe00ea00640 1 -- 172.21.15.81:0/1134686257 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008003c70 con 0x7fe018100910 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.960+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0181a2470 con 0x7fe018100910 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.960+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0181a2970 con 0x7fe018100910 2024-04-12T21:13:08.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.961+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe01810b860 con 0x7fe018100910 2024-04-12T21:13:08.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.963+0000 7fe00ea00640 1 -- 172.21.15.81:0/1134686257 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe008046020 con 0x7fe018100910 2024-04-12T21:13:08.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.965+0000 7fe00ea00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdffc0a3c30 0x7fdffc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:08.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.965+0000 7fe00ea00640 1 -- 172.21.15.81:0/1134686257 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe0080f21f0 con 0x7fe018100910 2024-04-12T21:13:08.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.965+0000 7fe00fe00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdffc0a3c30 0x7fdffc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:08.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.966+0000 7fe00fe00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdffc0a3c30 0x7fdffc0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe01819bad0 tx=0x7fe000005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:08.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:08.969+0000 7fe00ea00640 1 -- 172.21.15.81:0/1134686257 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe00804d020 con 0x7fe018100910 2024-04-12T21:13:09.184 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.188+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe01802d870 con 0x7fe018100910 2024-04-12T21:13:09.185 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.189+0000 7fe00ea00640 1 -- 172.21.15.81:0/1134686257 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18480 (secure 0 0 0) 0x7fe0080bf070 con 0x7fe018100910 2024-04-12T21:13:09.186 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:09.186 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":81,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:12:16.153549+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:09.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdffc0a3c30 msgr2=0x7fdffc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdffc0a3c30 0x7fdffc0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe01819bad0 tx=0x7fe000005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:09.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 msgr2=0x7fe01819a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 0x7fe01819a5c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe00802c8e0 tx=0x7fe0080049c0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 shutdown_connections 2024-04-12T21:13:09.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdffc0a3c30 0x7fdffc0a60f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe018101c10 0x7fe0181a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe018101270 0x7fe01819ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 --2- 172.21.15.81:0/1134686257 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe018100910 0x7fe01819a5c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 >> 172.21.15.81:0/1134686257 conn(0x7fe0180fc4b0 msgr2=0x7fe0180fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:09.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 shutdown_connections 2024-04-12T21:13:09.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.192+0000 7fe01ca00640 1 -- 172.21.15.81:0/1134686257 wait complete. 2024-04-12T21:13:09.206 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-12T21:13:09.206 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 -- 172.21.15.81:0/2499150136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 msgr2=0x7faac4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 --2- 172.21.15.81:0/2499150136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 0x7faac4100d20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faab4009900 tx=0x7faab402bcc0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 -- 172.21.15.81:0/2499150136 shutdown_connections 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 --2- 172.21.15.81:0/2499150136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faac4101260 0x7faac41016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 --2- 172.21.15.81:0/2499150136 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faac4101c00 0x7faac4102080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 --2- 172.21.15.81:0/2499150136 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 0x7faac4100d20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 -- 172.21.15.81:0/2499150136 >> 172.21.15.81:0/2499150136 conn(0x7faac40fc4a0 msgr2=0x7faac40fe8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 -- 172.21.15.81:0/2499150136 shutdown_connections 2024-04-12T21:13:09.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.410+0000 7faacb200640 1 -- 172.21.15.81:0/2499150136 wait complete. 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.411+0000 7faacb200640 1 Processor -- start 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.411+0000 7faacb200640 1 -- start start 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.411+0000 7faacb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 0x7faac419a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.411+0000 7faacb200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faac4101260 0x7faac419aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faacb200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faac4101c00 0x7faac41a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faacb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faac410fef0 con 0x7faac4100900 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faacb200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7faac410fd70 con 0x7faac4101c00 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faacb200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faac4110070 con 0x7faac4101260 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 0x7faac419a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 0x7faac419a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58608/0 (socket says 172.21.15.81:58608) 2024-04-12T21:13:09.408 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3e00640 1 -- 172.21.15.81:0/1366564010 learned_addr learned my addr 172.21.15.81:0/1366564010 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:09.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac9200640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faac4101c00 0x7faac41a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:09.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3400640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faac4101260 0x7faac419aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:09.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3e00640 1 -- 172.21.15.81:0/1366564010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faac4101260 msgr2=0x7faac419aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3e00640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faac4101260 0x7faac419aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3e00640 1 -- 172.21.15.81:0/1366564010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faac4101c00 msgr2=0x7faac41a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3e00640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faac4101c00 0x7faac41a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3e00640 1 -- 172.21.15.81:0/1366564010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac41a20e0 con 0x7faac4100900 2024-04-12T21:13:09.409 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.412+0000 7faac3400640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faac4101260 0x7faac419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:13:09.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.413+0000 7faac3e00640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 0x7faac419a4c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faab4005ab0 tx=0x7faab402c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:09.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.413+0000 7faac0c00640 1 -- 172.21.15.81:0/1366564010 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4043020 con 0x7faac4100900 2024-04-12T21:13:09.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.413+0000 7faac0c00640 1 -- 172.21.15.81:0/1366564010 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faab40054e0 con 0x7faac4100900 2024-04-12T21:13:09.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.413+0000 7faac0c00640 1 -- 172.21.15.81:0/1366564010 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab4003ca0 con 0x7faac4100900 2024-04-12T21:13:09.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.413+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac41a2370 con 0x7faac4100900 2024-04-12T21:13:09.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.413+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac41a2870 con 0x7faac4100900 2024-04-12T21:13:09.410 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.414+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac410b790 con 0x7faac4100900 2024-04-12T21:13:09.415 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.419+0000 7faac0c00640 1 -- 172.21.15.81:0/1366564010 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7faab4046020 con 0x7faac4100900 2024-04-12T21:13:09.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.421+0000 7faac0c00640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faaa80a3c30 0x7faaa80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:09.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.421+0000 7faac0c00640 1 -- 172.21.15.81:0/1366564010 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7faab40f2490 con 0x7faac4100900 2024-04-12T21:13:09.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.421+0000 7faac3400640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faaa80a3c30 0x7faaa80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:09.417 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.422+0000 7faac3400640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faaa80a3c30 0x7faaa80a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faac419b9d0 tx=0x7faab0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:09.418 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.422+0000 7faac0c00640 1 -- 172.21.15.81:0/1366564010 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faab404d020 con 0x7faac4100900 2024-04-12T21:13:09.646 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.650+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faac4062bc0 con 0x7faac4100900 2024-04-12T21:13:09.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.651+0000 7faac0c00640 1 -- 172.21.15.81:0/1366564010 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7faab40bf310 con 0x7faac4100900 2024-04-12T21:13:09.647 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:13:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faaa80a3c30 msgr2=0x7faaa80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faaa80a3c30 0x7faaa80a60f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faac419b9d0 tx=0x7faab0005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 msgr2=0x7faac419a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 0x7faac419a4c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7faab4005ab0 tx=0x7faab402c750 comp rx=0 tx=0).stop 2024-04-12T21:13:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 shutdown_connections 2024-04-12T21:13:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faaa80a3c30 0x7faaa80a60f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faac4101260 0x7faac419aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faac4101c00 0x7faac41a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 --2- 172.21.15.81:0/1366564010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faac4100900 0x7faac419a4c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 >> 172.21.15.81:0/1366564010 conn(0x7faac40fc4a0 msgr2=0x7faac40fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:09.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 shutdown_connections 2024-04-12T21:13:09.651 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.654+0000 7faacb200640 1 -- 172.21.15.81:0/1366564010 wait complete. 2024-04-12T21:13:09.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:09 smithi081 ceph-mon[31453]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:09.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:09 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/973097666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:09 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1134686257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:09.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-12T21:13:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:09 smithi153 ceph-mon[36569]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:09 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/973097666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:09 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1134686257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:09 smithi104 ceph-mon[36624]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:09 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/973097666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:09 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1134686257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 -- 172.21.15.81:0/1445892242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e81028c0 msgr2=0x7fb1e8104cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1445892242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e81028c0 0x7fb1e8104cb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc009860 tx=0x7fb1dc02bb90 comp rx=0 tx=0).stop 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 -- 172.21.15.81:0/1445892242 shutdown_connections 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1445892242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e8107b20 0x7fb1e8109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1445892242 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e81051f0 0x7fb1e81075e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1445892242 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e81028c0 0x7fb1e8104cb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 -- 172.21.15.81:0/1445892242 >> 172.21.15.81:0/1445892242 conn(0x7fb1e80fc290 msgr2=0x7fb1e80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 -- 172.21.15.81:0/1445892242 shutdown_connections 2024-04-12T21:13:09.863 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.867+0000 7fb1ee200640 1 -- 172.21.15.81:0/1445892242 wait complete. 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.868+0000 7fb1ee200640 1 Processor -- start 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.868+0000 7fb1ee200640 1 -- start start 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.868+0000 7fb1ee200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81028c0 0x7fb1e819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.868+0000 7fb1ee200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e81051f0 0x7fb1e819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.868+0000 7fb1ee200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e8107b20 0x7fb1e81a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.868+0000 7fb1ee200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e810dac0 con 0x7fb1e81051f0 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.868+0000 7fb1ee200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e810d940 con 0x7fb1e8107b20 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.868+0000 7fb1ee200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e810dc40 con 0x7fb1e81028c0 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81028c0 0x7fb1e819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1e7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e81051f0 0x7fb1e819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81028c0 0x7fb1e819a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:60140/0 (socket says 172.21.15.81:60140) 2024-04-12T21:13:09.865 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 -- 172.21.15.81:0/1507944660 learned_addr learned my addr 172.21.15.81:0/1507944660 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:09.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ed800640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e8107b20 0x7fb1e81a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:09.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 -- 172.21.15.81:0/1507944660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e8107b20 msgr2=0x7fb1e81a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e8107b20 0x7fb1e81a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 -- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e81051f0 msgr2=0x7fb1e819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:09.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e81051f0 0x7fb1e819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:09.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 -- 172.21.15.81:0/1507944660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e81a1fc0 con 0x7fb1e81028c0 2024-04-12T21:13:09.866 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.869+0000 7fb1ece00640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81028c0 0x7fb1e819a3a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0349e0 tx=0x7fb1dc032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:09.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.870+0000 7fb1e5600640 1 -- 172.21.15.81:0/1507944660 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc041030 con 0x7fb1e81028c0 2024-04-12T21:13:09.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.870+0000 7fb1e5600640 1 -- 172.21.15.81:0/1507944660 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1dc02c530 con 0x7fb1e81028c0 2024-04-12T21:13:09.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.870+0000 7fb1e5600640 1 -- 172.21.15.81:0/1507944660 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc039040 con 0x7fb1e81028c0 2024-04-12T21:13:09.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.870+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e81a2250 con 0x7fb1e81028c0 2024-04-12T21:13:09.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.870+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e81a26f0 con 0x7fb1e81028c0 2024-04-12T21:13:09.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.871+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1e8109670 con 0x7fb1e81028c0 2024-04-12T21:13:09.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.873+0000 7fb1e5600640 1 -- 172.21.15.81:0/1507944660 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb1dc044070 con 0x7fb1e81028c0 2024-04-12T21:13:09.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.874+0000 7fb1e5600640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb1d00a3bb0 0x7fb1d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:09.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.874+0000 7fb1e5600640 1 -- 172.21.15.81:0/1507944660 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb1dc0f17f0 con 0x7fb1e81028c0 2024-04-12T21:13:09.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.874+0000 7fb1e7e00640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb1d00a3bb0 0x7fb1d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:09.870 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.875+0000 7fb1e7e00640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb1d00a3bb0 0x7fb1d00a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb1e819b8b0 tx=0x7fb1d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:09.873 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:09.877+0000 7fb1e5600640 1 -- 172.21.15.81:0/1507944660 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb1dc0be5d0 con 0x7fb1e81028c0 2024-04-12T21:13:10.078 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.082+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb1e802d850 con 0x7fb1e81028c0 2024-04-12T21:13:10.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.083+0000 7fb1e5600640 1 -- 172.21.15.81:0/1507944660 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fb1dc0bded0 con 0x7fb1e81028c0 2024-04-12T21:13:10.079 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:10.079 INFO:teuthology.orchestra.run.smithi081.stdout:[ 2024-04-12T21:13:10.079 INFO:teuthology.orchestra.run.smithi081.stdout: { 2024-04-12T21:13:10.079 INFO:teuthology.orchestra.run.smithi081.stdout: "poolnum": 1, 2024-04-12T21:13:10.079 INFO:teuthology.orchestra.run.smithi081.stdout: "poolname": ".mgr" 2024-04-12T21:13:10.079 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-12T21:13:10.079 INFO:teuthology.orchestra.run.smithi081.stdout:] 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb1d00a3bb0 msgr2=0x7fb1d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb1d00a3bb0 0x7fb1d00a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb1e819b8b0 tx=0x7fb1d4009260 comp rx=0 tx=0).stop 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81028c0 msgr2=0x7fb1e819a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81028c0 0x7fb1e819a3a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0349e0 tx=0x7fb1dc032450 comp rx=0 tx=0).stop 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 shutdown_connections 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb1d00a3bb0 0x7fb1d00a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb1e8107b20 0x7fb1e81a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb1e81051f0 0x7fb1e819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 --2- 172.21.15.81:0/1507944660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb1e81028c0 0x7fb1e819a3a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.085+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 >> 172.21.15.81:0/1507944660 conn(0x7fb1e80fc290 msgr2=0x7fb1e8102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:10.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.086+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 shutdown_connections 2024-04-12T21:13:10.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.086+0000 7fb1ee200640 1 -- 172.21.15.81:0/1507944660 wait complete. 2024-04-12T21:13:10.101 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-12T21:13:10.101 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.313+0000 7f4424a00640 1 -- 172.21.15.81:0/2312881451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420101a40 msgr2=0x7f4420101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.313+0000 7f4424a00640 1 --2- 172.21.15.81:0/2312881451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420101a40 0x7f4420101ec0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4418009be0 tx=0x7f441802bef0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.314+0000 7f4424a00640 1 -- 172.21.15.81:0/2312881451 shutdown_connections 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.314+0000 7f4424a00640 1 --2- 172.21.15.81:0/2312881451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420101a40 0x7f4420101ec0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.314+0000 7f4424a00640 1 --2- 172.21.15.81:0/2312881451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44201010a0 0x7f4420101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.314+0000 7f4424a00640 1 --2- 172.21.15.81:0/2312881451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4420100740 0x7f4420100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.314+0000 7f4424a00640 1 -- 172.21.15.81:0/2312881451 >> 172.21.15.81:0/2312881451 conn(0x7f44200fc360 msgr2=0x7f44200fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.314+0000 7f4424a00640 1 -- 172.21.15.81:0/2312881451 shutdown_connections 2024-04-12T21:13:10.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.314+0000 7f4424a00640 1 -- 172.21.15.81:0/2312881451 wait complete. 2024-04-12T21:13:10.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.315+0000 7f4424a00640 1 Processor -- start 2024-04-12T21:13:10.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.315+0000 7f4424a00640 1 -- start start 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.315+0000 7f4424a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420100740 0x7f442019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.315+0000 7f4424a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44201010a0 0x7f442019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.315+0000 7f4424a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4420101a40 0x7f44201a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.315+0000 7f4424a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f442010fd70 con 0x7f44201010a0 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.315+0000 7f4424a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f442010fbf0 con 0x7f4420100740 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.315+0000 7f4424a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f442010fef0 con 0x7f4420101a40 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420100740 0x7f442019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44201010a0 0x7f442019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420100740 0x7f442019a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:52504/0 (socket says 172.21.15.81:52504) 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 -- 172.21.15.81:0/3158004227 learned_addr learned my addr 172.21.15.81:0/3158004227 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441f200640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4420101a40 0x7f44201a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 -- 172.21.15.81:0/3158004227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4420101a40 msgr2=0x7f44201a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:10.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4420101a40 0x7f44201a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 -- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44201010a0 msgr2=0x7f442019a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44201010a0 0x7f442019a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 -- 172.21.15.81:0/3158004227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44201a1f50 con 0x7f4420100740 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.316+0000 7f441e800640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420100740 0x7f442019a330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f441000e9c0 tx=0x7f441000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.317+0000 7f4417400640 1 -- 172.21.15.81:0/3158004227 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441000cde0 con 0x7f4420100740 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.317+0000 7f4417400640 1 -- 172.21.15.81:0/3158004227 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44100035f0 con 0x7f4420100740 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.317+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44201a2240 con 0x7f4420100740 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.317+0000 7f4417400640 1 -- 172.21.15.81:0/3158004227 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4410010690 con 0x7f4420100740 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.317+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44201a2790 con 0x7f4420100740 2024-04-12T21:13:10.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.318+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f442010b490 con 0x7f4420100740 2024-04-12T21:13:10.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.320+0000 7f4417400640 1 -- 172.21.15.81:0/3158004227 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f44100037a0 con 0x7f4420100740 2024-04-12T21:13:10.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.321+0000 7f4417400640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44040a3bb0 0x7f44040a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:10.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.321+0000 7f4417400640 1 -- 172.21.15.81:0/3158004227 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f44100d0310 con 0x7f4420100740 2024-04-12T21:13:10.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.321+0000 7f441de00640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44040a3bb0 0x7f44040a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:10.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.322+0000 7f441de00640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44040a3bb0 0x7f44040a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f442019b840 tx=0x7f440c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:10.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.329+0000 7f4417400640 1 -- 172.21.15.81:0/3158004227 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f441009d0f0 con 0x7f4420100740 2024-04-12T21:13:10.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:10.550+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f442002d5e0 con 0x7f4420100740 2024-04-12T21:13:10.723 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.723 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.723 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:10.723 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:13:10.723 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.724 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:10 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1366564010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:10.724 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:13:10.724 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:10 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1507944660' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-12T21:13:10.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:10.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:13:10.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:10 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1366564010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:13:10.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:10 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1507944660' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-12T21:13:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:13:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:10 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1366564010' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:13:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:10 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1507944660' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-12T21:13:11.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.514+0000 7f4417400640 1 -- 172.21.15.81:0/3158004227 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f441009c9f0 con 0x7f4420100740 2024-04-12T21:13:11.510 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_metadata' created 2024-04-12T21:13:11.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.518+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44040a3bb0 msgr2=0x7f44040a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:11.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.518+0000 7f4424a00640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44040a3bb0 0x7f44040a6070 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f442019b840 tx=0x7f440c005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:11.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.518+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420100740 msgr2=0x7f442019a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:11.514 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.518+0000 7f4424a00640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420100740 0x7f442019a330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f441000e9c0 tx=0x7f441000ee80 comp rx=0 tx=0).stop 2024-04-12T21:13:11.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.519+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 shutdown_connections 2024-04-12T21:13:11.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.519+0000 7f4424a00640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44040a3bb0 0x7f44040a6070 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.519+0000 7f4424a00640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4420101a40 0x7f44201a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.519+0000 7f4424a00640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f44201010a0 0x7f442019a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.519+0000 7f4424a00640 1 --2- 172.21.15.81:0/3158004227 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4420100740 0x7f442019a330 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.519+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 >> 172.21.15.81:0/3158004227 conn(0x7f44200fc360 msgr2=0x7f44200fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:11.515 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.520+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 shutdown_connections 2024-04-12T21:13:11.516 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.520+0000 7f4424a00640 1 -- 172.21.15.81:0/3158004227 wait complete. 2024-04-12T21:13:11.540 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-12T21:13:11.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 -- 172.21.15.81:0/82918263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4078560 msgr2=0x7f64f4078e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:11.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 --2- 172.21.15.81:0/82918263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4078560 0x7f64f4078e00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009900 tx=0x7f64ec02bc50 comp rx=0 tx=0).stop 2024-04-12T21:13:11.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 -- 172.21.15.81:0/82918263 shutdown_connections 2024-04-12T21:13:11.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 --2- 172.21.15.81:0/82918263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4078560 0x7f64f4078e00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 --2- 172.21.15.81:0/82918263 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4077bc0 0x7f64f4078020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 --2- 172.21.15.81:0/82918263 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f4076980 0x7f64f4076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 -- 172.21.15.81:0/82918263 >> 172.21.15.81:0/82918263 conn(0x7f64f40ff7b0 msgr2=0x7f64f4101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:11.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 -- 172.21.15.81:0/82918263 shutdown_connections 2024-04-12T21:13:11.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.740+0000 7f64fbe00640 1 -- 172.21.15.81:0/82918263 wait complete. 2024-04-12T21:13:11.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.741+0000 7f64fbe00640 1 Processor -- start 2024-04-12T21:13:11.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.741+0000 7f64fbe00640 1 -- start start 2024-04-12T21:13:11.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64fbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f4076980 0x7f64f419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:11.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64fbe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4077bc0 0x7f64f419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:11.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64fbe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4078560 0x7f64f41a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:11.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64fbe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41142f0 con 0x7f64f4078560 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64fbe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4114170 con 0x7f64f4077bc0 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64fbe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4114470 con 0x7f64f4076980 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f9400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f4076980 0x7f64f419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f8a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4077bc0 0x7f64f419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f9400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f4076980 0x7f64f419e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:60182/0 (socket says 172.21.15.81:60182) 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f9400640 1 -- 172.21.15.81:0/540077089 learned_addr learned my addr 172.21.15.81:0/540077089 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f9e00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4078560 0x7f64f41a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f8a00640 1 -- 172.21.15.81:0/540077089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f4076980 msgr2=0x7f64f419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f8a00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f4076980 0x7f64f419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f8a00640 1 -- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4078560 msgr2=0x7f64f41a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f8a00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4078560 0x7f64f41a5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f8a00640 1 -- 172.21.15.81:0/540077089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f41a6510 con 0x7f64f4077bc0 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.742+0000 7f64f9400640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f4076980 0x7f64f419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:11.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.743+0000 7f64f8a00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4077bc0 0x7f64f419ee80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f64e000e9f0 tx=0x7f64e000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.743+0000 7f64f2000640 1 -- 172.21.15.81:0/540077089 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e000ccc0 con 0x7f64f4077bc0 2024-04-12T21:13:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.743+0000 7f64f2000640 1 -- 172.21.15.81:0/540077089 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64e0010910 con 0x7f64f4077bc0 2024-04-12T21:13:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.743+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f41a6800 con 0x7f64f4077bc0 2024-04-12T21:13:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.743+0000 7f64f2000640 1 -- 172.21.15.81:0/540077089 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0018960 con 0x7f64f4077bc0 2024-04-12T21:13:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.743+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f41a6d50 con 0x7f64f4077bc0 2024-04-12T21:13:11.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.744+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f4110b90 con 0x7f64f4077bc0 2024-04-12T21:13:11.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.746+0000 7f64f2000640 1 -- 172.21.15.81:0/540077089 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f64e0010ac0 con 0x7f64f4077bc0 2024-04-12T21:13:11.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.747+0000 7f64f2000640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f64dc0a3bb0 0x7f64dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:11.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.748+0000 7f64f2000640 1 -- 172.21.15.81:0/540077089 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f64e00d04b0 con 0x7f64f4077bc0 2024-04-12T21:13:11.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.748+0000 7f64f9400640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f64dc0a3bb0 0x7f64dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:11.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.748+0000 7f64f9400640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f64dc0a3bb0 0x7f64dc0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f64e80049d0 tx=0x7f64e8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:11.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.752+0000 7f64f2000640 1 -- 172.21.15.81:0/540077089 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64e009cad0 con 0x7f64f4077bc0 2024-04-12T21:13:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:11 smithi153 ceph-mon[36569]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:11 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3158004227' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-12T21:13:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:11 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-12T21:13:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:11.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:11 smithi104 ceph-mon[36624]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:11 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3158004227' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-12T21:13:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:11 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-12T21:13:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:11 smithi081 ceph-mon[31453]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:11 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3158004227' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-12T21:13:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:11 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-12T21:13:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:11.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:11.973+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f64f4062bc0 con 0x7f64f4077bc0 2024-04-12T21:13:12.513 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.517+0000 7f64f2000640 1 -- 172.21.15.81:0/540077089 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7f64e00a0dd0 con 0x7f64f4077bc0 2024-04-12T21:13:12.514 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data' created 2024-04-12T21:13:12.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f64dc0a3bb0 msgr2=0x7f64dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:12.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f64dc0a3bb0 0x7f64dc0a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f64e80049d0 tx=0x7f64e8009500 comp rx=0 tx=0).stop 2024-04-12T21:13:12.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4077bc0 msgr2=0x7f64f419ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:12.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4077bc0 0x7f64f419ee80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f64e000e9f0 tx=0x7f64e000eeb0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 shutdown_connections 2024-04-12T21:13:12.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f64dc0a3bb0 0x7f64dc0a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f64f4078560 0x7f64f41a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f64f4077bc0 0x7f64f419ee80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 --2- 172.21.15.81:0/540077089 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64f4076980 0x7f64f419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.524+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 >> 172.21.15.81:0/540077089 conn(0x7f64f40ff7b0 msgr2=0x7f64f41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:12.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.526+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 shutdown_connections 2024-04-12T21:13:12.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.526+0000 7f64fbe00640 1 -- 172.21.15.81:0/540077089 wait complete. 2024-04-12T21:13:12.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 -- 172.21.15.81:0/235755949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 msgr2=0x7f5e2810c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 --2- 172.21.15.81:0/235755949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 0x7f5e2810c640 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5e18009900 tx=0x7f5e1802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 -- 172.21.15.81:0/235755949 shutdown_connections 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 --2- 172.21.15.81:0/235755949 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 0x7f5e2810c640 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 --2- 172.21.15.81:0/235755949 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e281053f0 0x7f5e28105870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 --2- 172.21.15.81:0/235755949 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e28104200 0x7f5e28104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 -- 172.21.15.81:0/235755949 >> 172.21.15.81:0/235755949 conn(0x7f5e280ff7b0 msgr2=0x7f5e28101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 -- 172.21.15.81:0/235755949 shutdown_connections 2024-04-12T21:13:12.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.747+0000 7f5e2f600640 1 -- 172.21.15.81:0/235755949 wait complete. 2024-04-12T21:13:12.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2f600640 1 Processor -- start 2024-04-12T21:13:12.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2f600640 1 -- start start 2024-04-12T21:13:12.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2f600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e28104200 0x7f5e2819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:12.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2f600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e281053f0 0x7f5e2819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:12.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2f600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 0x7f5e281a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2f600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2810fe90 con 0x7f5e28105db0 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2f600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2810fd10 con 0x7f5e281053f0 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2f600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5e28110010 con 0x7f5e28104200 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.748+0000 7f5e2d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 0x7f5e281a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e27e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e281053f0 0x7f5e2819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e2d600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 0x7f5e281a1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58682/0 (socket says 172.21.15.81:58682) 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e2d600640 1 -- 172.21.15.81:0/54791364 learned_addr learned my addr 172.21.15.81:0/54791364 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e2cc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e28104200 0x7f5e2819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e27e00640 1 -- 172.21.15.81:0/54791364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e28104200 msgr2=0x7f5e2819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e27e00640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e28104200 0x7f5e2819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e27e00640 1 -- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 msgr2=0x7f5e281a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e27e00640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 0x7f5e281a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e27e00640 1 -- 172.21.15.81:0/54791364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e281a20f0 con 0x7f5e281053f0 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e2d600640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 0x7f5e281a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:12.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e27e00640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e281053f0 0x7f5e2819aa10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000e9f0 tx=0x7f5e1000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:12.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.749+0000 7f5e25600640 1 -- 172.21.15.81:0/54791364 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e1000ccc0 con 0x7f5e281053f0 2024-04-12T21:13:12.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.750+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e281a23e0 con 0x7f5e281053f0 2024-04-12T21:13:12.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.750+0000 7f5e25600640 1 -- 172.21.15.81:0/54791364 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e10010910 con 0x7f5e281053f0 2024-04-12T21:13:12.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.750+0000 7f5e25600640 1 -- 172.21.15.81:0/54791364 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e10018960 con 0x7f5e281053f0 2024-04-12T21:13:12.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.750+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e281a2980 con 0x7f5e281053f0 2024-04-12T21:13:12.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.751+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2810b4c0 con 0x7f5e281053f0 2024-04-12T21:13:12.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.753+0000 7f5e25600640 1 -- 172.21.15.81:0/54791364 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5e10010ac0 con 0x7f5e281053f0 2024-04-12T21:13:12.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.754+0000 7f5e25600640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5e0c0a3bb0 0x7f5e0c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.754+0000 7f5e25600640 1 -- 172.21.15.81:0/54791364 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f5e100d0630 con 0x7f5e281053f0 2024-04-12T21:13:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.754+0000 7f5e2cc00640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5e0c0a3bb0 0x7f5e0c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:12.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.755+0000 7f5e2cc00640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5e0c0a3bb0 0x7f5e0c0a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5e28105250 tx=0x7f5e20009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:12.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:12.758+0000 7f5e25600640 1 -- 172.21.15.81:0/54791364 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e1009cad0 con 0x7f5e281053f0 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: osdmap e82: 12 total, 12 up, 12 in 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/540077089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: pgmap v307: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:12.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:13:12.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: osdmap e82: 12 total, 12 up, 12 in 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/540077089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: pgmap v307: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:13:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: osdmap e82: 12 total, 12 up, 12 in 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/540077089' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: pgmap v307: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:13:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:13:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:13:12.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.000+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f5e28062c30 con 0x7f5e281053f0 2024-04-12T21:13:13.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.530+0000 7f5e25600640 1 -- 172.21.15.81:0/54791364 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f5e100a0dd0 con 0x7f5e281053f0 2024-04-12T21:13:13.526 INFO:teuthology.orchestra.run.smithi081.stderr:new fs with metadata pool 2 and data pool 3 2024-04-12T21:13:13.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.534+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5e0c0a3bb0 msgr2=0x7f5e0c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:13.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.534+0000 7f5e2f600640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5e0c0a3bb0 0x7f5e0c0a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5e28105250 tx=0x7f5e20009680 comp rx=0 tx=0).stop 2024-04-12T21:13:13.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.534+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e281053f0 msgr2=0x7f5e2819aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:13.530 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.534+0000 7f5e2f600640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e281053f0 0x7f5e2819aa10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5e1000e9f0 tx=0x7f5e1000eeb0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.537+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 shutdown_connections 2024-04-12T21:13:13.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.537+0000 7f5e2f600640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5e0c0a3bb0 0x7f5e0c0a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.537+0000 7f5e2f600640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5e28105db0 0x7f5e281a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.537+0000 7f5e2f600640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5e281053f0 0x7f5e2819aa10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.537+0000 7f5e2f600640 1 --2- 172.21.15.81:0/54791364 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5e28104200 0x7f5e2819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.537+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 >> 172.21.15.81:0/54791364 conn(0x7f5e280ff7b0 msgr2=0x7f5e28108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:13.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.537+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 shutdown_connections 2024-04-12T21:13:13.533 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.537+0000 7f5e2f600640 1 -- 172.21.15.81:0/54791364 wait complete. 2024-04-12T21:13:13.556 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-12T21:13:13.556 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-12T21:13:13.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 -- 172.21.15.81:0/1283687127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f48105410 msgr2=0x7f6f48105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:13.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/1283687127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f48105410 0x7f6f48105890 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6f30009900 tx=0x7f6f3002bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:13.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 -- 172.21.15.81:0/1283687127 shutdown_connections 2024-04-12T21:13:13.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/1283687127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f48105dd0 0x7f6f4810c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/1283687127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f48105410 0x7f6f48105890 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/1283687127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48104220 0x7f6f48104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 -- 172.21.15.81:0/1283687127 >> 172.21.15.81:0/1283687127 conn(0x7f6f480ff770 msgr2=0x7f6f48101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:13.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 -- 172.21.15.81:0/1283687127 shutdown_connections 2024-04-12T21:13:13.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.768+0000 7f6f4dc00640 1 -- 172.21.15.81:0/1283687127 wait complete. 2024-04-12T21:13:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.769+0000 7f6f4dc00640 1 Processor -- start 2024-04-12T21:13:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.769+0000 7f6f4dc00640 1 -- start start 2024-04-12T21:13:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f4dc00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f48104220 0x7f6f4819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f4dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f48105410 0x7f6f4819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f4dc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48105dd0 0x7f6f481a1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f4dc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4810ff00 con 0x7f6f48105410 2024-04-12T21:13:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f4dc00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4810fd80 con 0x7f6f48104220 2024-04-12T21:13:13.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f4dc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6f48110080 con 0x7f6f48105dd0 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f3e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f48105410 0x7f6f4819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f46a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f48104220 0x7f6f4819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f3e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f48105410 0x7f6f4819aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58694/0 (socket says 172.21.15.81:58694) 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f3e000640 1 -- 172.21.15.81:0/3518433596 learned_addr learned my addr 172.21.15.81:0/3518433596 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f47400640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48105dd0 0x7f6f481a1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f3e000640 1 -- 172.21.15.81:0/3518433596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48105dd0 msgr2=0x7f6f481a1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f3e000640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48105dd0 0x7f6f481a1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f3e000640 1 -- 172.21.15.81:0/3518433596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f48104220 msgr2=0x7f6f4819a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f3e000640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f48104220 0x7f6f4819a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f3e000640 1 -- 172.21.15.81:0/3518433596 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f481a2170 con 0x7f6f48105410 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.770+0000 7f6f47400640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48105dd0 0x7f6f481a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.771+0000 7f6f3e000640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f48105410 0x7f6f4819aa90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6f30000c00 tx=0x7f6f3002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.771+0000 7f6f3fe00640 1 -- 172.21.15.81:0/3518433596 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30043020 con 0x7f6f48105410 2024-04-12T21:13:13.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.771+0000 7f6f3fe00640 1 -- 172.21.15.81:0/3518433596 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f300054e0 con 0x7f6f48105410 2024-04-12T21:13:13.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.771+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f481a2400 con 0x7f6f48105410 2024-04-12T21:13:13.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.771+0000 7f6f3fe00640 1 -- 172.21.15.81:0/3518433596 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f30003c40 con 0x7f6f48105410 2024-04-12T21:13:13.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.771+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f481a2950 con 0x7f6f48105410 2024-04-12T21:13:13.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.772+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f4810b540 con 0x7f6f48105410 2024-04-12T21:13:13.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.774+0000 7f6f3fe00640 1 -- 172.21.15.81:0/3518433596 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6f30046020 con 0x7f6f48105410 2024-04-12T21:13:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.776+0000 7f6f3fe00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6f2c0a3bb0 0x7f6f2c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.776+0000 7f6f3fe00640 1 -- 172.21.15.81:0/3518433596 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f6f300f23d0 con 0x7f6f48105410 2024-04-12T21:13:13.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.776+0000 7f6f46a00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6f2c0a3bb0 0x7f6f2c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:13.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.777+0000 7f6f46a00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6f2c0a3bb0 0x7f6f2c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6f48105270 tx=0x7f6f38009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:13.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.781+0000 7f6f3fe00640 1 -- 172.21.15.81:0/3518433596 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f3004d020 con 0x7f6f48105410 2024-04-12T21:13:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:13 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-12T21:13:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:13 smithi153 ceph-mon[36569]: osdmap e83: 12 total, 12 up, 12 in 2024-04-12T21:13:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:13 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/54791364' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-12T21:13:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:13 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-12T21:13:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:13 smithi153 ceph-mon[36569]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:13:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:13 smithi153 ceph-mon[36569]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-12T21:13:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:13 smithi153 ceph-mon[36569]: osdmap e84: 12 total, 12 up, 12 in 2024-04-12T21:13:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:13 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-12T21:13:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:13 smithi104 ceph-mon[36624]: osdmap e83: 12 total, 12 up, 12 in 2024-04-12T21:13:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:13 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/54791364' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-12T21:13:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:13 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-12T21:13:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:13 smithi104 ceph-mon[36624]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:13:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:13 smithi104 ceph-mon[36624]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-12T21:13:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:13 smithi104 ceph-mon[36624]: osdmap e84: 12 total, 12 up, 12 in 2024-04-12T21:13:13.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:13 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-12T21:13:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:13 smithi081 ceph-mon[31453]: osdmap e83: 12 total, 12 up, 12 in 2024-04-12T21:13:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:13 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/54791364' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-12T21:13:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:13 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-12T21:13:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:13 smithi081 ceph-mon[31453]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:13:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:13 smithi081 ceph-mon[31453]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-12T21:13:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:13 smithi081 ceph-mon[31453]: osdmap e84: 12 total, 12 up, 12 in 2024-04-12T21:13:13.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:13.994+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f6f48062bc0 con 0x7f6f48105410 2024-04-12T21:13:14.106 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.110+0000 7f6f3fe00640 1 -- 172.21.15.81:0/3518433596 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f6f300bee60 con 0x7f6f48105410 2024-04-12T21:13:14.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.114+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6f2c0a3bb0 msgr2=0x7f6f2c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:14.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.114+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6f2c0a3bb0 0x7f6f2c0a6070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6f48105270 tx=0x7f6f38009500 comp rx=0 tx=0).stop 2024-04-12T21:13:14.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.114+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f48105410 msgr2=0x7f6f4819aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:14.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.114+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f48105410 0x7f6f4819aa90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6f30000c00 tx=0x7f6f3002c650 comp rx=0 tx=0).stop 2024-04-12T21:13:14.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.116+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 shutdown_connections 2024-04-12T21:13:14.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.116+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6f2c0a3bb0 0x7f6f2c0a6070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.116+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f48105dd0 0x7f6f481a1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.116+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6f48105410 0x7f6f4819aa90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.116+0000 7f6f4dc00640 1 --2- 172.21.15.81:0/3518433596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f48104220 0x7f6f4819a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.116+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 >> 172.21.15.81:0/3518433596 conn(0x7f6f480ff770 msgr2=0x7f6f48108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:14.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.116+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 shutdown_connections 2024-04-12T21:13:14.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.117+0000 7f6f4dc00640 1 -- 172.21.15.81:0/3518433596 wait complete. 2024-04-12T21:13:14.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-12T21:13:14.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 -- 172.21.15.81:0/1759457793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920102fb0 msgr2=0x7fc920103430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:14.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 --2- 172.21.15.81:0/1759457793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920102fb0 0x7fc920103430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc908009900 tx=0x7fc90802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:14.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 -- 172.21.15.81:0/1759457793 shutdown_connections 2024-04-12T21:13:14.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 --2- 172.21.15.81:0/1759457793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc920103970 0x7fc92010a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 --2- 172.21.15.81:0/1759457793 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920102fb0 0x7fc920103430 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 --2- 172.21.15.81:0/1759457793 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc920101e60 0x7fc920102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 -- 172.21.15.81:0/1759457793 >> 172.21.15.81:0/1759457793 conn(0x7fc9200fd3d0 msgr2=0x7fc9200ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 -- 172.21.15.81:0/1759457793 shutdown_connections 2024-04-12T21:13:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.357+0000 7fc925800640 1 -- 172.21.15.81:0/1759457793 wait complete. 2024-04-12T21:13:14.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.358+0000 7fc925800640 1 Processor -- start 2024-04-12T21:13:14.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.358+0000 7fc925800640 1 -- start start 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc925800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920101e60 0x7fc92019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc925800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc920102fb0 0x7fc92019a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc925800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc920103970 0x7fc9201a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc925800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc92010db50 con 0x7fc920101e60 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc925800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc92010d9d0 con 0x7fc920103970 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc925800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc92010dcd0 con 0x7fc920102fb0 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920101e60 0x7fc92019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920101e60 0x7fc92019a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58710/0 (socket says 172.21.15.81:58710) 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91fe00640 1 -- 172.21.15.81:0/647490186 learned_addr learned my addr 172.21.15.81:0/647490186 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc924e00640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc920103970 0x7fc9201a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:14.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91f400640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc920102fb0 0x7fc92019a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91fe00640 1 -- 172.21.15.81:0/647490186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc920102fb0 msgr2=0x7fc92019a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91fe00640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc920102fb0 0x7fc92019a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91fe00640 1 -- 172.21.15.81:0/647490186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc920103970 msgr2=0x7fc9201a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91fe00640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc920103970 0x7fc9201a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91fe00640 1 -- 172.21.15.81:0/647490186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9201a1fa0 con 0x7fc920101e60 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.359+0000 7fc91f400640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc920102fb0 0x7fc92019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.360+0000 7fc91fe00640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920101e60 0x7fc92019a380 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc91000ea80 tx=0x7fc91000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.360+0000 7fc91cc00640 1 -- 172.21.15.81:0/647490186 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91000ce40 con 0x7fc920101e60 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.360+0000 7fc91cc00640 1 -- 172.21.15.81:0/647490186 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc910005020 con 0x7fc920101e60 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.360+0000 7fc91cc00640 1 -- 172.21.15.81:0/647490186 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc910011760 con 0x7fc920101e60 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.360+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9201a2290 con 0x7fc920101e60 2024-04-12T21:13:14.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.360+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9201a2760 con 0x7fc920101e60 2024-04-12T21:13:14.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.361+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9201a2a20 con 0x7fc920101e60 2024-04-12T21:13:14.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.363+0000 7fc91cc00640 1 -- 172.21.15.81:0/647490186 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc9100051d0 con 0x7fc920101e60 2024-04-12T21:13:14.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.365+0000 7fc91cc00640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc9040a3c30 0x7fc9040a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:14.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.365+0000 7fc91cc00640 1 -- 172.21.15.81:0/647490186 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fc9100d0a40 con 0x7fc920101e60 2024-04-12T21:13:14.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.365+0000 7fc91f400640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc9040a3c30 0x7fc9040a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:14.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.366+0000 7fc91f400640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc9040a3c30 0x7fc9040a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc90802c170 tx=0x7fc90802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:14.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.369+0000 7fc91cc00640 1 -- 172.21.15.81:0/647490186 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc910016030 con 0x7fc920101e60 2024-04-12T21:13:14.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:14.584+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fc9201a2bf0 con 0x7fc920101e60 2024-04-12T21:13:14.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:14 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-12T21:13:14.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:14 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-12T21:13:14.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:14 smithi153 ceph-mon[36569]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:13:14.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:14 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3518433596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-12T21:13:14.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:14 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3518433596' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-12T21:13:14.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:14 smithi153 ceph-mon[36569]: osdmap e85: 12 total, 12 up, 12 in 2024-04-12T21:13:14.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:14 smithi153 ceph-mon[36569]: pgmap v311: 97 pgs: 74 unknown, 23 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 5 op/s 2024-04-12T21:13:14.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:14 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-12T21:13:14.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:14 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-12T21:13:14.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:14 smithi104 ceph-mon[36624]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:13:14.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:14 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3518433596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-12T21:13:14.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:14 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3518433596' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-12T21:13:14.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:14 smithi104 ceph-mon[36624]: osdmap e85: 12 total, 12 up, 12 in 2024-04-12T21:13:14.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:14 smithi104 ceph-mon[36624]: pgmap v311: 97 pgs: 74 unknown, 23 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 5 op/s 2024-04-12T21:13:14.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:14 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-12T21:13:14.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:14 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-12T21:13:14.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:14 smithi081 ceph-mon[31453]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:13:14.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:14 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3518433596' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-12T21:13:14.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:14 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3518433596' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-12T21:13:14.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:14 smithi081 ceph-mon[31453]: osdmap e85: 12 total, 12 up, 12 in 2024-04-12T21:13:14.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:14 smithi081 ceph-mon[31453]: pgmap v311: 97 pgs: 74 unknown, 23 active+clean; 577 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 5 op/s 2024-04-12T21:13:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] up:active 2024-04-12T21:13:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:15 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:15 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:15 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/647490186' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-12T21:13:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:15 smithi153 ceph-mon[36569]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-12T21:13:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:15 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:13:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] up:active 2024-04-12T21:13:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:15 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:15 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:15 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/647490186' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-12T21:13:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:15 smithi104 ceph-mon[36624]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-12T21:13:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:15 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:13:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] up:active 2024-04-12T21:13:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:15 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:15 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:15 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/647490186' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-12T21:13:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:15 smithi081 ceph-mon[31453]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-12T21:13:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:15 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:13:16.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.570+0000 7fc91cc00640 1 -- 172.21.15.81:0/647490186 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fc91009d390 con 0x7fc920101e60 2024-04-12T21:13:16.566 INFO:teuthology.orchestra.run.smithi081.stderr:pool 'cephfs_data_ec' created 2024-04-12T21:13:16.570 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc9040a3c30 msgr2=0x7fc9040a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc9040a3c30 0x7fc9040a60f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc90802c170 tx=0x7fc90802df90 comp rx=0 tx=0).stop 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920101e60 msgr2=0x7fc92019a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920101e60 0x7fc92019a380 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc91000ea80 tx=0x7fc91000ef40 comp rx=0 tx=0).stop 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 shutdown_connections 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc9040a3c30 0x7fc9040a60f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc920102fb0 0x7fc92019a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc920103970 0x7fc9201a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 --2- 172.21.15.81:0/647490186 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc920101e60 0x7fc92019a380 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.571 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.575+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 >> 172.21.15.81:0/647490186 conn(0x7fc9200fd3d0 msgr2=0x7fc9200741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:16.572 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.576+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 shutdown_connections 2024-04-12T21:13:16.573 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.578+0000 7fc925800640 1 -- 172.21.15.81:0/647490186 wait complete. 2024-04-12T21:13:16.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-12T21:13:16.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.794+0000 7f87c9400640 1 -- 172.21.15.81:0/886161150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87c4108620 msgr2=0x7f87c4108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.794+0000 7f87c9400640 1 --2- 172.21.15.81:0/886161150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87c4108620 0x7f87c4108a40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f87b4009900 tx=0x7f87b402bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.794+0000 7f87c9400640 1 -- 172.21.15.81:0/886161150 shutdown_connections 2024-04-12T21:13:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.794+0000 7f87c9400640 1 --2- 172.21.15.81:0/886161150 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c410a1d0 0x7f87c4110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.794+0000 7f87c9400640 1 --2- 172.21.15.81:0/886161150 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c4109810 0x7f87c4109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.794+0000 7f87c9400640 1 --2- 172.21.15.81:0/886161150 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87c4108620 0x7f87c4108a40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.794+0000 7f87c9400640 1 -- 172.21.15.81:0/886161150 >> 172.21.15.81:0/886161150 conn(0x7f87c40775c0 msgr2=0x7f87c4079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.795+0000 7f87c9400640 1 -- 172.21.15.81:0/886161150 shutdown_connections 2024-04-12T21:13:16.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.795+0000 7f87c9400640 1 -- 172.21.15.81:0/886161150 wait complete. 2024-04-12T21:13:16.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.795+0000 7f87c9400640 1 Processor -- start 2024-04-12T21:13:16.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c9400640 1 -- start start 2024-04-12T21:13:16.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c9400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c4108620 0x7f87c419e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:16.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c9400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c4109810 0x7f87c419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:16.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87c410a1d0 0x7f87c41a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:16.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c9400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f87c41142f0 con 0x7f87c410a1d0 2024-04-12T21:13:16.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c9400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4114170 con 0x7f87c4109810 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c9400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f87c4114470 con 0x7f87c4108620 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c4108620 0x7f87c419e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c4108620 0x7f87c419e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:44812/0 (socket says 172.21.15.81:44812) 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c2800640 1 -- 172.21.15.81:0/3134936419 learned_addr learned my addr 172.21.15.81:0/3134936419 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c3200640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87c410a1d0 0x7f87c41a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.796+0000 7f87c1e00640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c4109810 0x7f87c419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87c2800640 1 -- 172.21.15.81:0/3134936419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c4109810 msgr2=0x7f87c419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87c2800640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c4109810 0x7f87c419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87c2800640 1 -- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87c410a1d0 msgr2=0x7f87c41a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87c2800640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87c410a1d0 0x7f87c41a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87c2800640 1 -- 172.21.15.81:0/3134936419 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87c41a6580 con 0x7f87c4108620 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87c2800640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c4108620 0x7f87c419e9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87b4000c00 tx=0x7f87b402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87bb400640 1 -- 172.21.15.81:0/3134936419 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4043020 con 0x7f87c4108620 2024-04-12T21:13:16.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87bb400640 1 -- 172.21.15.81:0/3134936419 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87b40054e0 con 0x7f87c4108620 2024-04-12T21:13:16.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87bb400640 1 -- 172.21.15.81:0/3134936419 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4003c40 con 0x7f87c4108620 2024-04-12T21:13:16.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87c41a6810 con 0x7f87c4108620 2024-04-12T21:13:16.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.797+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87c41a6db0 con 0x7f87c4108620 2024-04-12T21:13:16.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.798+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87c410fa10 con 0x7f87c4108620 2024-04-12T21:13:16.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.801+0000 7f87bb400640 1 -- 172.21.15.81:0/3134936419 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f87b4046020 con 0x7f87c4108620 2024-04-12T21:13:16.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.803+0000 7f87bb400640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f87a80a3c30 0x7f87a80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:16.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.803+0000 7f87bb400640 1 -- 172.21.15.81:0/3134936419 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f87b40f29c0 con 0x7f87c4108620 2024-04-12T21:13:16.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.803+0000 7f87c1e00640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f87a80a3c30 0x7f87a80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:16.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.804+0000 7f87c1e00640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f87a80a3c30 0x7f87a80a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f87c419fe70 tx=0x7f87b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:16.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:16.805+0000 7f87bb400640 1 -- 172.21.15.81:0/3134936419 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87b404d020 con 0x7f87c4108620 2024-04-12T21:13:16.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:16 smithi104 ceph-mon[36624]: osdmap e86: 12 total, 12 up, 12 in 2024-04-12T21:13:16.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:16 smithi104 ceph-mon[36624]: pgmap v313: 97 pgs: 5 unknown, 92 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-04-12T21:13:16.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:16 smithi081 ceph-mon[31453]: osdmap e86: 12 total, 12 up, 12 in 2024-04-12T21:13:17.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:16 smithi081 ceph-mon[31453]: pgmap v313: 97 pgs: 5 unknown, 92 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-04-12T21:13:17.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:16 smithi153 ceph-mon[36569]: osdmap e86: 12 total, 12 up, 12 in 2024-04-12T21:13:17.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:16 smithi153 ceph-mon[36569]: pgmap v313: 97 pgs: 5 unknown, 92 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 9 op/s 2024-04-12T21:13:17.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.002+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f87c402d5e0 con 0x7f87c4108620 2024-04-12T21:13:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.578+0000 7f87bb400640 1 -- 172.21.15.81:0/3134936419 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f87b40bf120 con 0x7f87c4108620 2024-04-12T21:13:17.575 INFO:teuthology.orchestra.run.smithi081.stderr:set pool 4 allow_ec_overwrites to true 2024-04-12T21:13:17.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.584+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f87a80a3c30 msgr2=0x7f87a80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.584+0000 7f87c9400640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f87a80a3c30 0x7f87a80a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f87c419fe70 tx=0x7f87b0005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.584+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c4108620 msgr2=0x7f87c419e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:17.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.584+0000 7f87c9400640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c4108620 0x7f87c419e9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f87b4000c00 tx=0x7f87b402c650 comp rx=0 tx=0).stop 2024-04-12T21:13:17.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.585+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 shutdown_connections 2024-04-12T21:13:17.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.585+0000 7f87c9400640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f87a80a3c30 0x7f87a80a60f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.585+0000 7f87c9400640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f87c410a1d0 0x7f87c41a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.585+0000 7f87c9400640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87c4109810 0x7f87c419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.585+0000 7f87c9400640 1 --2- 172.21.15.81:0/3134936419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f87c4108620 0x7f87c419e9b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.585+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 >> 172.21.15.81:0/3134936419 conn(0x7f87c40775c0 msgr2=0x7f87c410cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:17.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.586+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 shutdown_connections 2024-04-12T21:13:17.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.586+0000 7f87c9400640 1 -- 172.21.15.81:0/3134936419 wait complete. 2024-04-12T21:13:17.616 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-12T21:13:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 -- 172.21.15.81:0/4229773430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308104220 msgr2=0x7fb308104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 --2- 172.21.15.81:0/4229773430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308104220 0x7fb308104640 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009900 tx=0x7fb2f802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 -- 172.21.15.81:0/4229773430 shutdown_connections 2024-04-12T21:13:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 --2- 172.21.15.81:0/4229773430 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308105dd0 0x7fb30810c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 --2- 172.21.15.81:0/4229773430 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb308105410 0x7fb308105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 --2- 172.21.15.81:0/4229773430 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308104220 0x7fb308104640 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 -- 172.21.15.81:0/4229773430 >> 172.21.15.81:0/4229773430 conn(0x7fb3080ff7b0 msgr2=0x7fb308101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:17.796 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 -- 172.21.15.81:0/4229773430 shutdown_connections 2024-04-12T21:13:17.797 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.800+0000 7fb30fc00640 1 -- 172.21.15.81:0/4229773430 wait complete. 2024-04-12T21:13:17.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30fc00640 1 Processor -- start 2024-04-12T21:13:17.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30fc00640 1 -- start start 2024-04-12T21:13:17.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30fc00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308104220 0x7fb30819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:17.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30fc00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb308105410 0x7fb30819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:17.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30fc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308105dd0 0x7fb3081a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:17.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30fc00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb30810ff00 con 0x7fb308105dd0 2024-04-12T21:13:17.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30fc00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb30810fd80 con 0x7fb308105410 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30fc00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb308110080 con 0x7fb308104220 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.801+0000 7fb30dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308105dd0 0x7fb3081a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308105dd0 0x7fb3081a1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52430/0 (socket says 172.21.15.81:52430) 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30dc00640 1 -- 172.21.15.81:0/2357729966 learned_addr learned my addr 172.21.15.81:0/2357729966 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb2ffe00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb308105410 0x7fb30819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30d200640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308104220 0x7fb30819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30dc00640 1 -- 172.21.15.81:0/2357729966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308104220 msgr2=0x7fb30819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30dc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308104220 0x7fb30819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30dc00640 1 -- 172.21.15.81:0/2357729966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb308105410 msgr2=0x7fb30819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30dc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb308105410 0x7fb30819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30dc00640 1 -- 172.21.15.81:0/2357729966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3081a20f0 con 0x7fb308105dd0 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30dc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308105dd0 0x7fb3081a1a80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb30000eaa0 tx=0x7fb30000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb306000640 1 -- 172.21.15.81:0/2357729966 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30000cdb0 con 0x7fb308105dd0 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.802+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3081a23e0 con 0x7fb308105dd0 2024-04-12T21:13:17.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.803+0000 7fb306000640 1 -- 172.21.15.81:0/2357729966 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb300005020 con 0x7fb308105dd0 2024-04-12T21:13:17.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.803+0000 7fb306000640 1 -- 172.21.15.81:0/2357729966 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3000116b0 con 0x7fb308105dd0 2024-04-12T21:13:17.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.803+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3081a2930 con 0x7fb308105dd0 2024-04-12T21:13:17.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.804+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb30810b540 con 0x7fb308105dd0 2024-04-12T21:13:17.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.806+0000 7fb306000640 1 -- 172.21.15.81:0/2357729966 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb300005560 con 0x7fb308105dd0 2024-04-12T21:13:17.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.807+0000 7fb306000640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2ec0a3bb0 0x7fb2ec0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:17.802 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.807+0000 7fb306000640 1 -- 172.21.15.81:0/2357729966 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fb3000d0c50 con 0x7fb308105dd0 2024-04-12T21:13:17.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.807+0000 7fb30d200640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2ec0a3bb0 0x7fb2ec0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:17.803 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.808+0000 7fb30d200640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2ec0a3bb0 0x7fb2ec0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb2f802c170 tx=0x7fb2f802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:17.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:17.810+0000 7fb306000640 1 -- 172.21.15.81:0/2357729966 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb300016030 con 0x7fb308105dd0 2024-04-12T21:13:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:17 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/647490186' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-12T21:13:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:17 smithi081 ceph-mon[31453]: osdmap e87: 12 total, 12 up, 12 in 2024-04-12T21:13:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:17 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3134936419' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-12T21:13:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:17 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-12T21:13:18.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:17 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/647490186' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-12T21:13:18.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:17 smithi153 ceph-mon[36569]: osdmap e87: 12 total, 12 up, 12 in 2024-04-12T21:13:18.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:17 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3134936419' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-12T21:13:18.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:17 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-12T21:13:18.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.060+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fb308062bc0 con 0x7fb308105dd0 2024-04-12T21:13:18.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:17 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/647490186' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-12T21:13:18.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:17 smithi104 ceph-mon[36624]: osdmap e87: 12 total, 12 up, 12 in 2024-04-12T21:13:18.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:17 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3134936419' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-12T21:13:18.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:17 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-12T21:13:18.577 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:13:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2024-04-12T21:13:18.312+0000 7f1808457700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:13:18.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.597+0000 7fb306000640 1 -- 172.21.15.81:0/2357729966 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fb30009cd60 con 0x7fb308105dd0 2024-04-12T21:13:18.593 INFO:teuthology.orchestra.run.smithi081.stderr:added data pool 4 to fsmap 2024-04-12T21:13:18.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2ec0a3bb0 msgr2=0x7fb2ec0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:18.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2ec0a3bb0 0x7fb2ec0a6070 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb2f802c170 tx=0x7fb2f802df90 comp rx=0 tx=0).stop 2024-04-12T21:13:18.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308105dd0 msgr2=0x7fb3081a1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:18.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308105dd0 0x7fb3081a1a80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb30000eaa0 tx=0x7fb30000ef60 comp rx=0 tx=0).stop 2024-04-12T21:13:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 shutdown_connections 2024-04-12T21:13:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb2ec0a3bb0 0x7fb2ec0a6070 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb308105dd0 0x7fb3081a1a80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb308105410 0x7fb30819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 --2- 172.21.15.81:0/2357729966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb308104220 0x7fb30819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.601+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 >> 172.21.15.81:0/2357729966 conn(0x7fb3080ff7b0 msgr2=0x7fb308108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:18.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.602+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 shutdown_connections 2024-04-12T21:13:18.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.602+0000 7fb30fc00640 1 -- 172.21.15.81:0/2357729966 wait complete. 2024-04-12T21:13:18.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:13:18.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 -- 172.21.15.81:0/2474273631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c109ff0 msgr2=0x7f990c110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:18.829 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 --2- 172.21.15.81:0/2474273631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c109ff0 0x7f990c110880 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9904009ca0 tx=0x7f990402c100 comp rx=0 tx=0).stop 2024-04-12T21:13:18.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 -- 172.21.15.81:0/2474273631 shutdown_connections 2024-04-12T21:13:18.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 --2- 172.21.15.81:0/2474273631 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c109ff0 0x7f990c110880 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 --2- 172.21.15.81:0/2474273631 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f990c109630 0x7f990c109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 --2- 172.21.15.81:0/2474273631 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990c108440 0x7f990c108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 -- 172.21.15.81:0/2474273631 >> 172.21.15.81:0/2474273631 conn(0x7f990c077400 msgr2=0x7f990c079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:18.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 -- 172.21.15.81:0/2474273631 shutdown_connections 2024-04-12T21:13:18.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.833+0000 7f9911a00640 1 -- 172.21.15.81:0/2474273631 wait complete. 2024-04-12T21:13:18.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.834+0000 7f9911a00640 1 Processor -- start 2024-04-12T21:13:18.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9911a00640 1 -- start start 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9911a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c108440 0x7f990c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9911a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990c109630 0x7f990c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9911a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f990c109ff0 0x7f990c1a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9911a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f990c1140d0 con 0x7f990c108440 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9911a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f990c113f50 con 0x7f990c109ff0 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9911a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f990c114250 con 0x7f990c109630 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f990be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c108440 0x7f990c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f990be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c108440 0x7f990c19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52448/0 (socket says 172.21.15.81:52448) 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f990be00640 1 -- 172.21.15.81:0/3900581755 learned_addr learned my addr 172.21.15.81:0/3900581755 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9903e00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990c109630 0x7f990c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.835+0000 7f9911000640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f990c109ff0 0x7f990c1a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f990be00640 1 -- 172.21.15.81:0/3900581755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990c109630 msgr2=0x7f990c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:18.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f990be00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990c109630 0x7f990c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f990be00640 1 -- 172.21.15.81:0/3900581755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f990c109ff0 msgr2=0x7f990c1a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f990be00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f990c109ff0 0x7f990c1a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f990be00640 1 -- 172.21.15.81:0/3900581755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c1a6320 con 0x7f990c108440 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f9903e00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990c109630 0x7f990c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f990be00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c108440 0x7f990c19e700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00e9c0 tx=0x7f98fc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f9909600640 1 -- 172.21.15.81:0/3900581755 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00cd80 con 0x7f990c108440 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f9909600640 1 -- 172.21.15.81:0/3900581755 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98fc0035f0 con 0x7f990c108440 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990c1a6610 con 0x7f990c108440 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f9909600640 1 -- 172.21.15.81:0/3900581755 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc010690 con 0x7f990c108440 2024-04-12T21:13:18.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.836+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990c1a6b60 con 0x7f990c108440 2024-04-12T21:13:18.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.838+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f990c10f760 con 0x7f990c108440 2024-04-12T21:13:18.835 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.840+0000 7f9909600640 1 -- 172.21.15.81:0/3900581755 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f98fc0037a0 con 0x7f990c108440 2024-04-12T21:13:18.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.841+0000 7f9909600640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98ec0a3c30 0x7f98ec0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:18.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.841+0000 7f9909600640 1 -- 172.21.15.81:0/3900581755 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98fc0d0980 con 0x7f990c108440 2024-04-12T21:13:18.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.841+0000 7f9903e00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98ec0a3c30 0x7f98ec0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:18.837 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.842+0000 7f9903e00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98ec0a3c30 0x7f98ec0a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f990c19fc10 tx=0x7f98f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:18.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:18.846+0000 7f9909600640 1 -- 172.21.15.81:0/3900581755 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98fc09d0c0 con 0x7f990c108440 2024-04-12T21:13:18.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:18 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-12T21:13:18.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:18 smithi081 ceph-mon[31453]: osdmap e88: 12 total, 12 up, 12 in 2024-04-12T21:13:18.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:18 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2357729966' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-12T21:13:18.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:18 smithi081 ceph-mon[31453]: pgmap v316: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-12T21:13:19.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:18 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-12T21:13:19.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:18 smithi153 ceph-mon[36569]: osdmap e88: 12 total, 12 up, 12 in 2024-04-12T21:13:19.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:18 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2357729966' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-12T21:13:19.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:18 smithi153 ceph-mon[36569]: pgmap v316: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-12T21:13:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:18 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-12T21:13:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:18 smithi104 ceph-mon[36624]: osdmap e88: 12 total, 12 up, 12 in 2024-04-12T21:13:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:18 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2357729966' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-12T21:13:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:18 smithi104 ceph-mon[36624]: pgmap v316: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-12T21:13:19.086 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.090+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f990c02d5e0 con 0x7f990c108440 2024-04-12T21:13:19.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.092+0000 7f9909600640 1 -- 172.21.15.81:0/3900581755 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11806 (secure 0 0 0) 0x7f98fc09c9c0 con 0x7f990c108440 2024-04-12T21:13:19.088 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:19.088 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T21:13:13.003153+0000","modified":"2024-04-12T21:13:18.062435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-12T21:13:19.088 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-04-12T21:13:19.090 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.094+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98ec0a3c30 msgr2=0x7f98ec0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.094+0000 7f9911a00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98ec0a3c30 0x7f98ec0a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f990c19fc10 tx=0x7f98f4005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.094+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c108440 msgr2=0x7f990c19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.094+0000 7f9911a00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c108440 0x7f990c19e700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00e9c0 tx=0x7f98fc00ee80 comp rx=0 tx=0).stop 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.095+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 shutdown_connections 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.095+0000 7f9911a00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98ec0a3c30 0x7f98ec0a60f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.095+0000 7f9911a00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f990c109630 0x7f990c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.095+0000 7f9911a00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f990c109ff0 0x7f990c1a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.095+0000 7f9911a00640 1 --2- 172.21.15.81:0/3900581755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f990c108440 0x7f990c19e700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.095+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 >> 172.21.15.81:0/3900581755 conn(0x7f990c077400 msgr2=0x7f990c10c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.095+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 shutdown_connections 2024-04-12T21:13:19.091 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.095+0000 7f9911a00640 1 -- 172.21.15.81:0/3900581755 wait complete. 2024-04-12T21:13:19.111 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:19.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 -- 172.21.15.81:0/230510112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0108440 msgr2=0x7fc1c0108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 --2- 172.21.15.81:0/230510112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0108440 0x7fc1c0108860 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0009900 tx=0x7fc1b002bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:19.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 -- 172.21.15.81:0/230510112 shutdown_connections 2024-04-12T21:13:19.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 --2- 172.21.15.81:0/230510112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1c0109ff0 0x7fc1c0110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 --2- 172.21.15.81:0/230510112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1c0109630 0x7fc1c0109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 --2- 172.21.15.81:0/230510112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0108440 0x7fc1c0108860 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 -- 172.21.15.81:0/230510112 >> 172.21.15.81:0/230510112 conn(0x7fc1c0077400 msgr2=0x7fc1c0079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:19.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 -- 172.21.15.81:0/230510112 shutdown_connections 2024-04-12T21:13:19.327 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.330+0000 7fc1c5200640 1 -- 172.21.15.81:0/230510112 wait complete. 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.331+0000 7fc1c5200640 1 Processor -- start 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1c5200640 1 -- start start 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1c5200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1c0108440 0x7fc1c019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1c5200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1c0109630 0x7fc1c019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1c5200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0109ff0 0x7fc1c01a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1c5200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c0114120 con 0x7fc1c0109ff0 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1c5200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c0113fa0 con 0x7fc1c0109630 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1c5200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c01142a0 con 0x7fc1c0108440 2024-04-12T21:13:19.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1bfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0109ff0 0x7fc1c01a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1bfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0109ff0 0x7fc1c01a5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52464/0 (socket says 172.21.15.81:52464) 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1bfe00640 1 -- 172.21.15.81:0/4084457060 learned_addr learned my addr 172.21.15.81:0/4084457060 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1bf400640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1c0108440 0x7fc1c019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.332+0000 7fc1bea00640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1c0109630 0x7fc1c019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1bfe00640 1 -- 172.21.15.81:0/4084457060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1c0108440 msgr2=0x7fc1c019e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1bfe00640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1c0108440 0x7fc1c019e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1bfe00640 1 -- 172.21.15.81:0/4084457060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1c0109630 msgr2=0x7fc1c019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1bfe00640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1c0109630 0x7fc1c019eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1bfe00640 1 -- 172.21.15.81:0/4084457060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c01a6380 con 0x7fc1c0109ff0 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1bfe00640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0109ff0 0x7fc1c01a5d10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800efd0 tx=0x7fc1b800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1b7e00640 1 -- 172.21.15.81:0/4084457060 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8003b70 con 0x7fc1c0109ff0 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1b7e00640 1 -- 172.21.15.81:0/4084457060 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1b8003d10 con 0x7fc1c0109ff0 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1b7e00640 1 -- 172.21.15.81:0/4084457060 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80109f0 con 0x7fc1c0109ff0 2024-04-12T21:13:19.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c01a6670 con 0x7fc1c0109ff0 2024-04-12T21:13:19.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.333+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c01a6b70 con 0x7fc1c0109ff0 2024-04-12T21:13:19.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.334+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c010f830 con 0x7fc1c0109ff0 2024-04-12T21:13:19.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.336+0000 7fc1b7e00640 1 -- 172.21.15.81:0/4084457060 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc1b8004020 con 0x7fc1c0109ff0 2024-04-12T21:13:19.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.339+0000 7fc1b7e00640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc1a40a3c30 0x7fc1a40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:19.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.339+0000 7fc1b7e00640 1 -- 172.21.15.81:0/4084457060 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc1b80d0a40 con 0x7fc1c0109ff0 2024-04-12T21:13:19.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.339+0000 7fc1bf400640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc1a40a3c30 0x7fc1a40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:19.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.340+0000 7fc1bf400640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc1a40a3c30 0x7fc1a40a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc1b002c170 tx=0x7fc1b002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:19.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.343+0000 7fc1b7e00640 1 -- 172.21.15.81:0/4084457060 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc1b809d180 con 0x7fc1c0109ff0 2024-04-12T21:13:19.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.552+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1c0062bc0 con 0x7fc1c0109ff0 2024-04-12T21:13:19.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.557+0000 7fc1b7e00640 1 -- 172.21.15.81:0/4084457060 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7fc1b809ca80 con 0x7fc1c0109ff0 2024-04-12T21:13:19.553 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:19.553 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc1a40a3c30 msgr2=0x7fc1a40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc1a40a3c30 0x7fc1a40a60f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc1b002c170 tx=0x7fc1b002df90 comp rx=0 tx=0).stop 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0109ff0 msgr2=0x7fc1c01a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0109ff0 0x7fc1c01a5d10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc1b800efd0 tx=0x7fc1b800c6a0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 shutdown_connections 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc1a40a3c30 0x7fc1a40a60f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc1c0109ff0 0x7fc1c01a5d10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc1c0109630 0x7fc1c019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 --2- 172.21.15.81:0/4084457060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc1c0108440 0x7fc1c019e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 >> 172.21.15.81:0/4084457060 conn(0x7fc1c0077400 msgr2=0x7fc1c010ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:19.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 shutdown_connections 2024-04-12T21:13:19.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.560+0000 7fc1c5200640 1 -- 172.21.15.81:0/4084457060 wait complete. 2024-04-12T21:13:19.574 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:19.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 -- 172.21.15.81:0/3145334223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109630 msgr2=0x7f8b14109ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3145334223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109630 0x7f8b14109ab0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b00009900 tx=0x7f8b0002bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:19.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 -- 172.21.15.81:0/3145334223 shutdown_connections 2024-04-12T21:13:19.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3145334223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109630 0x7f8b14109ab0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3145334223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14109ff0 0x7f8b14110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3145334223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b14108440 0x7f8b14108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 -- 172.21.15.81:0/3145334223 >> 172.21.15.81:0/3145334223 conn(0x7f8b14077400 msgr2=0x7f8b14079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:19.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 -- 172.21.15.81:0/3145334223 shutdown_connections 2024-04-12T21:13:19.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.779+0000 7f8b1a800640 1 -- 172.21.15.81:0/3145334223 wait complete. 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.780+0000 7f8b1a800640 1 Processor -- start 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.780+0000 7f8b1a800640 1 -- start start 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b1a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b14108440 0x7f8b1419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b1a800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14109630 0x7f8b1419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b1a800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109ff0 0x7f8b141a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b1a800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8b141140d0 con 0x7f8b14108440 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b1a800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8b14113f50 con 0x7f8b14109630 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b1a800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8b14114250 con 0x7f8b14109ff0 2024-04-12T21:13:19.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b14108440 0x7f8b1419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b14108440 0x7f8b1419e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52470/0 (socket says 172.21.15.81:52470) 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19400640 1 -- 172.21.15.81:0/3156304567 learned_addr learned my addr 172.21.15.81:0/3156304567 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b18a00640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14109630 0x7f8b1419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19e00640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109ff0 0x7f8b141a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19e00640 1 -- 172.21.15.81:0/3156304567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14109630 msgr2=0x7f8b1419ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19e00640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14109630 0x7f8b1419ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19e00640 1 -- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b14108440 msgr2=0x7f8b1419e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19e00640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b14108440 0x7f8b1419e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19e00640 1 -- 172.21.15.81:0/3156304567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b141a6320 con 0x7f8b14109ff0 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.781+0000 7f8b19400640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b14108440 0x7f8b1419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.782+0000 7f8b19e00640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109ff0 0x7f8b141a5cb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009fd0 tx=0x7f8b0c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.782+0000 7f8b12000640 1 -- 172.21.15.81:0/3156304567 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c00ce20 con 0x7f8b14109ff0 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.782+0000 7f8b12000640 1 -- 172.21.15.81:0/3156304567 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b0c005020 con 0x7f8b14109ff0 2024-04-12T21:13:19.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.782+0000 7f8b12000640 1 -- 172.21.15.81:0/3156304567 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c010720 con 0x7f8b14109ff0 2024-04-12T21:13:19.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.782+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b141a6610 con 0x7f8b14109ff0 2024-04-12T21:13:19.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.782+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b141a6b30 con 0x7f8b14109ff0 2024-04-12T21:13:19.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.783+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b1410f760 con 0x7f8b14109ff0 2024-04-12T21:13:19.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.786+0000 7f8b12000640 1 -- 172.21.15.81:0/3156304567 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8b0c005540 con 0x7f8b14109ff0 2024-04-12T21:13:19.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.787+0000 7f8b12000640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8afc0a3bb0 0x7f8afc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:19.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.787+0000 7f8b12000640 1 -- 172.21.15.81:0/3156304567 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8b0c0d0c70 con 0x7f8b14109ff0 2024-04-12T21:13:19.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.787+0000 7f8b19400640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8afc0a3bb0 0x7f8afc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:19.783 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.788+0000 7f8b19400640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8afc0a3bb0 0x7f8afc0a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8b080058c0 tx=0x7f8b08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:19.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.791+0000 7f8b12000640 1 -- 172.21.15.81:0/3156304567 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b0c09d3b0 con 0x7f8b14109ff0 2024-04-12T21:13:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:19 smithi081 ceph-mon[31453]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-12T21:13:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:19 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2357729966' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-12T21:13:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:19 smithi081 ceph-mon[31453]: osdmap e89: 12 total, 12 up, 12 in 2024-04-12T21:13:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:19 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:19 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3900581755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:19 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4084457060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:19.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.989+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b14062bc0 con 0x7f8b14109ff0 2024-04-12T21:13:19.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.992+0000 7f8b12000640 1 -- 172.21.15.81:0/3156304567 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7f8b0c09ccb0 con 0x7f8b14109ff0 2024-04-12T21:13:19.987 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:19.988 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:19.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8afc0a3bb0 msgr2=0x7f8afc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8afc0a3bb0 0x7f8afc0a6070 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8b080058c0 tx=0x7f8b08005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109ff0 msgr2=0x7f8b141a5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109ff0 0x7f8b141a5cb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009fd0 tx=0x7f8b0c00ef10 comp rx=0 tx=0).stop 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 shutdown_connections 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8afc0a3bb0 0x7f8afc0a6070 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8b14109ff0 0x7f8b141a5cb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8b14109630 0x7f8b1419ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 --2- 172.21.15.81:0/3156304567 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8b14108440 0x7f8b1419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 >> 172.21.15.81:0/3156304567 conn(0x7f8b14077400 msgr2=0x7f8b1410c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 shutdown_connections 2024-04-12T21:13:19.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:19.994+0000 7f8b1a800640 1 -- 172.21.15.81:0/3156304567 wait complete. 2024-04-12T21:13:20.023 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:13:20.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:19 smithi153 ceph-mon[36569]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-12T21:13:20.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:19 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2357729966' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-12T21:13:20.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:19 smithi153 ceph-mon[36569]: osdmap e89: 12 total, 12 up, 12 in 2024-04-12T21:13:20.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:19 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:20.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:19 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3900581755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:20.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:19 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4084457060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:20.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:19 smithi104 ceph-mon[36624]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-12T21:13:20.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:19 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2357729966' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-12T21:13:20.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:19 smithi104 ceph-mon[36624]: osdmap e89: 12 total, 12 up, 12 in 2024-04-12T21:13:20.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:19 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:20.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:19 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3900581755' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:20.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:19 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4084457060' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 -- 172.21.15.81:0/3839835280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 msgr2=0x7f7a78108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3839835280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 0x7f7a78108860 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009900 tx=0x7f7a6802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 -- 172.21.15.81:0/3839835280 shutdown_connections 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3839835280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a78109ff0 0x7f7a78110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3839835280 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78109630 0x7f7a78109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3839835280 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 0x7f7a78108860 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 -- 172.21.15.81:0/3839835280 >> 172.21.15.81:0/3839835280 conn(0x7f7a78077400 msgr2=0x7f7a78079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 -- 172.21.15.81:0/3839835280 shutdown_connections 2024-04-12T21:13:20.226 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.230+0000 7f7a7de00640 1 -- 172.21.15.81:0/3839835280 wait complete. 2024-04-12T21:13:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7de00640 1 Processor -- start 2024-04-12T21:13:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7de00640 1 -- start start 2024-04-12T21:13:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 0x7f7a7819e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7de00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78109630 0x7f7a7819ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7de00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a78109ff0 0x7f7a781a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7de00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7a78114120 con 0x7f7a78108440 2024-04-12T21:13:20.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7de00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7a78113fa0 con 0x7f7a78109630 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7de00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a781142a0 con 0x7f7a78109ff0 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.231+0000 7f7a7ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 0x7f7a7819e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7ca00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 0x7f7a7819e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52486/0 (socket says 172.21.15.81:52486) 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7ca00640 1 -- 172.21.15.81:0/3502585292 learned_addr learned my addr 172.21.15.81:0/3502585292 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7d400640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a78109ff0 0x7f7a781a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a6fe00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78109630 0x7f7a7819ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7ca00640 1 -- 172.21.15.81:0/3502585292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a78109ff0 msgr2=0x7f7a781a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7ca00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a78109ff0 0x7f7a781a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7ca00640 1 -- 172.21.15.81:0/3502585292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78109630 msgr2=0x7f7a7819ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7ca00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78109630 0x7f7a7819ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7ca00640 1 -- 172.21.15.81:0/3502585292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a781a6370 con 0x7f7a78108440 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a7ca00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 0x7f7a7819e7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7a68000c00 tx=0x7f7a6802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.232+0000 7f7a76000640 1 -- 172.21.15.81:0/3502585292 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68043020 con 0x7f7a78108440 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.233+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a781a6600 con 0x7f7a78108440 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.233+0000 7f7a76000640 1 -- 172.21.15.81:0/3502585292 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a680054e0 con 0x7f7a78108440 2024-04-12T21:13:20.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.233+0000 7f7a76000640 1 -- 172.21.15.81:0/3502585292 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68003c40 con 0x7f7a78108440 2024-04-12T21:13:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.233+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a781a6b50 con 0x7f7a78108440 2024-04-12T21:13:20.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.234+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a7810f760 con 0x7f7a78108440 2024-04-12T21:13:20.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.236+0000 7f7a76000640 1 -- 172.21.15.81:0/3502585292 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7a68046020 con 0x7f7a78108440 2024-04-12T21:13:20.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.237+0000 7f7a76000640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a580a3bb0 0x7f7a580a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:20.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.237+0000 7f7a76000640 1 -- 172.21.15.81:0/3502585292 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7a680f2be0 con 0x7f7a78108440 2024-04-12T21:13:20.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.237+0000 7f7a6fe00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a580a3bb0 0x7f7a580a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:20.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.238+0000 7f7a6fe00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a580a3bb0 0x7f7a580a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7a600095f0 tx=0x7f7a60005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:20.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.239+0000 7f7a76000640 1 -- 172.21.15.81:0/3502585292 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a6804d020 con 0x7f7a78108440 2024-04-12T21:13:20.475 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.479+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a7802d5e0 con 0x7f7a78108440 2024-04-12T21:13:20.476 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.481+0000 7f7a76000640 1 -- 172.21.15.81:0/3502585292 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11806 (secure 0 0 0) 0x7f7a680a9020 con 0x7f7a78108440 2024-04-12T21:13:20.476 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:20.477 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24547,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-12T21:13:13.003153+0000","modified":"2024-04-12T21:13:18.062435+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-12T21:13:20.477 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 17 2024-04-12T21:13:20.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a580a3bb0 msgr2=0x7f7a580a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a580a3bb0 0x7f7a580a6070 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7a600095f0 tx=0x7f7a60005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:20.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 msgr2=0x7f7a7819e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 0x7f7a7819e7a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7a68000c00 tx=0x7f7a6802c650 comp rx=0 tx=0).stop 2024-04-12T21:13:20.479 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 shutdown_connections 2024-04-12T21:13:20.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a580a3bb0 0x7f7a580a6070 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a78109ff0 0x7f7a781a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a78109630 0x7f7a7819ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 --2- 172.21.15.81:0/3502585292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a78108440 0x7f7a7819e7a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.483+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 >> 172.21.15.81:0/3502585292 conn(0x7f7a78077400 msgr2=0x7f7a7810c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:20.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.484+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 shutdown_connections 2024-04-12T21:13:20.480 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.484+0000 7f7a7de00640 1 -- 172.21.15.81:0/3502585292 wait complete. 2024-04-12T21:13:20.498 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:20.700 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 -- 172.21.15.81:0/2094022794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 msgr2=0x7fcd581056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 --2- 172.21.15.81:0/2094022794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 0x7fcd581056b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcd48009900 tx=0x7fcd4802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 -- 172.21.15.81:0/2094022794 shutdown_connections 2024-04-12T21:13:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 --2- 172.21.15.81:0/2094022794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd58105bf0 0x7fcd5810c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 --2- 172.21.15.81:0/2094022794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 0x7fcd581056b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 --2- 172.21.15.81:0/2094022794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd58104040 0x7fcd58104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 -- 172.21.15.81:0/2094022794 >> 172.21.15.81:0/2094022794 conn(0x7fcd580ff5d0 msgr2=0x7fcd581019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 -- 172.21.15.81:0/2094022794 shutdown_connections 2024-04-12T21:13:20.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.704+0000 7fcd5e400640 1 -- 172.21.15.81:0/2094022794 wait complete. 2024-04-12T21:13:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.705+0000 7fcd5e400640 1 Processor -- start 2024-04-12T21:13:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5e400640 1 -- start start 2024-04-12T21:13:20.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd58104040 0x7fcd5819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 0x7fcd5819a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5e400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd58105bf0 0x7fcd581a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5810fd20 con 0x7fcd58105230 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5e400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5810fba0 con 0x7fcd58105bf0 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5e400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5810fea0 con 0x7fcd58104040 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd4fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 0x7fcd5819a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd4fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 0x7fcd5819a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52506/0 (socket says 172.21.15.81:52506) 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd4fe00640 1 -- 172.21.15.81:0/1310037741 learned_addr learned my addr 172.21.15.81:0/1310037741 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5d000640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd58104040 0x7fcd5819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.706+0000 7fcd5da00640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd58105bf0 0x7fcd581a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.707+0000 7fcd4fe00640 1 -- 172.21.15.81:0/1310037741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd58104040 msgr2=0x7fcd5819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.707+0000 7fcd4fe00640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd58104040 0x7fcd5819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.707+0000 7fcd4fe00640 1 -- 172.21.15.81:0/1310037741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd58105bf0 msgr2=0x7fcd581a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.707+0000 7fcd4fe00640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd58105bf0 0x7fcd581a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.707+0000 7fcd4fe00640 1 -- 172.21.15.81:0/1310037741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd581a1f10 con 0x7fcd58105230 2024-04-12T21:13:20.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.707+0000 7fcd4fe00640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 0x7fcd5819a830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcd48000c00 tx=0x7fcd4802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:20.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.707+0000 7fcd56000640 1 -- 172.21.15.81:0/1310037741 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48043020 con 0x7fcd58105230 2024-04-12T21:13:20.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.707+0000 7fcd56000640 1 -- 172.21.15.81:0/1310037741 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd480054e0 con 0x7fcd58105230 2024-04-12T21:13:20.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.708+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd581a21a0 con 0x7fcd58105230 2024-04-12T21:13:20.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.708+0000 7fcd56000640 1 -- 172.21.15.81:0/1310037741 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48003c40 con 0x7fcd58105230 2024-04-12T21:13:20.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.708+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd581a26f0 con 0x7fcd58105230 2024-04-12T21:13:20.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.709+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd5810b300 con 0x7fcd58105230 2024-04-12T21:13:20.706 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.710+0000 7fcd56000640 1 -- 172.21.15.81:0/1310037741 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcd48046020 con 0x7fcd58105230 2024-04-12T21:13:20.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.712+0000 7fcd56000640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcd380a3bb0 0x7fcd380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:20.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.712+0000 7fcd56000640 1 -- 172.21.15.81:0/1310037741 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcd480f2670 con 0x7fcd58105230 2024-04-12T21:13:20.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.712+0000 7fcd5d000640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcd380a3bb0 0x7fcd380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:20.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.713+0000 7fcd5d000640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcd380a3bb0 0x7fcd380a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcd58105090 tx=0x7fcd40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:20.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.716+0000 7fcd56000640 1 -- 172.21.15.81:0/1310037741 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd4804d020 con 0x7fcd58105230 2024-04-12T21:13:20.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:20 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3156304567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:20.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:20 smithi081 ceph-mon[31453]: pgmap v318: 129 pgs: 19 unknown, 110 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 2 op/s 2024-04-12T21:13:20.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:20 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3502585292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:20.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.919+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd58062bc0 con 0x7fcd58105230 2024-04-12T21:13:20.917 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.922+0000 7fcd56000640 1 -- 172.21.15.81:0/1310037741 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7fcd480bedb0 con 0x7fcd58105230 2024-04-12T21:13:20.918 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:20.918 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcd380a3bb0 msgr2=0x7fcd380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcd380a3bb0 0x7fcd380a6070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fcd58105090 tx=0x7fcd40005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 msgr2=0x7fcd5819a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 0x7fcd5819a830 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcd48000c00 tx=0x7fcd4802c650 comp rx=0 tx=0).stop 2024-04-12T21:13:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 shutdown_connections 2024-04-12T21:13:20.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcd380a3bb0 0x7fcd380a6070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcd58105bf0 0x7fcd581a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcd58105230 0x7fcd5819a830 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 --2- 172.21.15.81:0/1310037741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd58104040 0x7fcd5819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:20.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.924+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 >> 172.21.15.81:0/1310037741 conn(0x7fcd580ff5d0 msgr2=0x7fcd581084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:20.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.925+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 shutdown_connections 2024-04-12T21:13:20.921 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:20.925+0000 7fcd5e400640 1 -- 172.21.15.81:0/1310037741 wait complete. 2024-04-12T21:13:20.940 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-12T21:13:13.003153+0000', 'modified': '2024-04-12T21:13:18.062435+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24593': {'gid': 24593, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.104:6839/1944172329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 1944172329}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 1944172329}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24593, 'qdb_cluster': [24593]} 2024-04-12T21:13:20.940 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-12T21:13:20.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-12T21:13:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:20 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3156304567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:20 smithi153 ceph-mon[36569]: pgmap v318: 129 pgs: 19 unknown, 110 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 2 op/s 2024-04-12T21:13:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:20 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3502585292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:21.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:20 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3156304567' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:21.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:20 smithi104 ceph-mon[36624]: pgmap v318: 129 pgs: 19 unknown, 110 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s wr, 2 op/s 2024-04-12T21:13:21.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:20 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3502585292' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 -- 172.21.15.81:0/1231163368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f850c104220 msgr2=0x7f850c104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 --2- 172.21.15.81:0/1231163368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f850c104220 0x7f850c104640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8500009900 tx=0x7f850002bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 -- 172.21.15.81:0/1231163368 shutdown_connections 2024-04-12T21:13:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 --2- 172.21.15.81:0/1231163368 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f850c105dd0 0x7f850c10c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 --2- 172.21.15.81:0/1231163368 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f850c105410 0x7f850c105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 --2- 172.21.15.81:0/1231163368 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f850c104220 0x7f850c104640 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 -- 172.21.15.81:0/1231163368 >> 172.21.15.81:0/1231163368 conn(0x7f850c0ff7b0 msgr2=0x7f850c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:21.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 -- 172.21.15.81:0/1231163368 shutdown_connections 2024-04-12T21:13:21.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.156+0000 7f8513a00640 1 -- 172.21.15.81:0/1231163368 wait complete. 2024-04-12T21:13:21.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.157+0000 7f8513a00640 1 Processor -- start 2024-04-12T21:13:21.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8513a00640 1 -- start start 2024-04-12T21:13:21.155 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8513a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f850c104220 0x7f850c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8513a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f850c105410 0x7f850c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8513a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f850c105dd0 0x7f850c1a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8513a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f850c10fed0 con 0x7f850c105410 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8513a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f850c10fd50 con 0x7f850c105dd0 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8513a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f850c110050 con 0x7f850c104220 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f850be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f850c105410 0x7f850c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f850be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f850c105410 0x7f850c19aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52524/0 (socket says 172.21.15.81:52524) 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f850be00640 1 -- 172.21.15.81:0/87410576 learned_addr learned my addr 172.21.15.81:0/87410576 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8511000640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f850c104220 0x7f850c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.158+0000 7f8511a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f850c105dd0 0x7f850c1a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f850be00640 1 -- 172.21.15.81:0/87410576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f850c104220 msgr2=0x7f850c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f850be00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f850c104220 0x7f850c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f850be00640 1 -- 172.21.15.81:0/87410576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f850c105dd0 msgr2=0x7f850c1a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.156 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f850be00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f850c105dd0 0x7f850c1a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f850be00640 1 -- 172.21.15.81:0/87410576 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c1a21b0 con 0x7f850c105410 2024-04-12T21:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f8511a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f850c105dd0 0x7f850c1a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f850be00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f850c105410 0x7f850c19aad0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f84f800e9f0 tx=0x7f84f800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f8509600640 1 -- 172.21.15.81:0/87410576 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f800ccc0 con 0x7f850c105410 2024-04-12T21:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f8513a00640 1 -- 172.21.15.81:0/87410576 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850c1a24a0 con 0x7f850c105410 2024-04-12T21:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f8509600640 1 -- 172.21.15.81:0/87410576 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84f8010910 con 0x7f850c105410 2024-04-12T21:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f8509600640 1 -- 172.21.15.81:0/87410576 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f80188a0 con 0x7f850c105410 2024-04-12T21:13:21.157 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.159+0000 7f8513a00640 1 -- 172.21.15.81:0/87410576 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850c1a29f0 con 0x7f850c105410 2024-04-12T21:13:21.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.162+0000 7f85167fd740 1 Processor -- start 2024-04-12T21:13:21.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.162+0000 7f8509600640 1 -- 172.21.15.81:0/87410576 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f84f8010ac0 con 0x7f850c105410 2024-04-12T21:13:21.158 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.162+0000 7f85167fd740 1 -- start start 2024-04-12T21:13:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f85167fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 0x5653c1854990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f85167fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c1859a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f85167fd740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c185c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f85167fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5653c17ba990 con 0x5653c1854f60 2024-04-12T21:13:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f85167fd740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5653c1761530 con 0x5653c185a0d0 2024-04-12T21:13:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f85167fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5653c17b6950 con 0x5653c1854570 2024-04-12T21:13:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f850be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c185c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.160 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f8511000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c1859a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f8511000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c1859a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52526/0 (socket says 172.21.15.81:52526) 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f8511000640 1 -- 172.21.15.81:0/3321049250 learned_addr learned my addr 172.21.15.81:0/3321049250 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f8511a00640 1 --2- 172.21.15.81:0/3321049250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 0x5653c1854990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f850be00640 1 -- 172.21.15.81:0/3321049250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 msgr2=0x5653c1854990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f850be00640 1 --2- 172.21.15.81:0/3321049250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 0x5653c1854990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f850be00640 1 -- 172.21.15.81:0/3321049250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 msgr2=0x5653c1859a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f850be00640 1 --2- 172.21.15.81:0/3321049250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c1859a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.163+0000 7f850be00640 1 -- 172.21.15.81:0/3321049250 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c14cd2d0 con 0x5653c185a0d0 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.164+0000 7f850be00640 1 --2- 172.21.15.81:0/3321049250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c185c560 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f84f80ce2d0 tx=0x7f84f80d9150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b52aee29d64b1d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.164+0000 7f84fe000640 1 -- 172.21.15.81:0/3321049250 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f80e4070 con 0x5653c185a0d0 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.164+0000 7f8509600640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84f40a3bb0 0x7f84f40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.164+0000 7f84fe000640 1 -- 172.21.15.81:0/3321049250 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84f809d210 con 0x5653c185a0d0 2024-04-12T21:13:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.164+0000 7f84fe000640 1 -- 172.21.15.81:0/3321049250 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8018bd0 con 0x5653c185a0d0 2024-04-12T21:13:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.164+0000 7f8509600640 1 -- 172.21.15.81:0/87410576 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f84f80d0ad0 con 0x7f850c105410 2024-04-12T21:13:21.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.164+0000 7f8511a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84f40a3bb0 0x7f84f40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 -- 172.21.15.81:0/3321049250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 msgr2=0x5653c185c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f8511a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84f40a3bb0 0x7f84f40a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x5653c18557c0 tx=0x7f8504009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 --2- 172.21.15.81:0/3321049250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c185c560 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f84f80ce2d0 tx=0x7f84f80d9150 comp rx=0 tx=0).stop 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 -- 172.21.15.81:0/3321049250 shutdown_connections 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 --2- 172.21.15.81:0/3321049250 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c185c560 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 --2- 172.21.15.81:0/3321049250 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c1859a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 --2- 172.21.15.81:0/3321049250 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 0x5653c1854990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 -- 172.21.15.81:0/3321049250 >> 172.21.15.81:0/3321049250 conn(0x5653c184fae0 msgr2=0x5653c184fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 -- 172.21.15.81:0/3321049250 shutdown_connections 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.165+0000 7f85167fd740 1 -- 172.21.15.81:0/3321049250 wait complete. 2024-04-12T21:13:21.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 1 -- start start 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 1 Processor -- start 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 0x5653c185ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c185ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1865fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5653c185d080 con 0x5653c1854f60 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5653c17748f0 con 0x5653c185a0d0 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5653c185d200 con 0x5653c1854570 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.166+0000 7f85167fd740 10 client.0 _ll_register_callbacks cb 0x5653c17742e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1865fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f850be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c185ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1865fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:38332/0 (socket says 172.21.15.81:38332) 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 -- 172.21.15.81:0/606849243 learned_addr learned my addr 172.21.15.81:0/606849243 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511000640 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 0x5653c185ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.167 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 -- 172.21.15.81:0/606849243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 msgr2=0x5653c185ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 0x5653c185ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 -- 172.21.15.81:0/606849243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 msgr2=0x5653c185ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c185ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 -- 172.21.15.81:0/606849243 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c15976b0 con 0x5653c185a0d0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f850be00640 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c185ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f8511a00640 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1865fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f850400fb50 tx=0x7f8504004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54c40050ccd5788 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.167+0000 7f84f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f84f2a00640 1 -- 172.21.15.81:0/606849243 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8504011d50 con 0x5653c185a0d0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f85167fd740 1 -- 172.21.15.81:0/606849243 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84f809d210 con 0x5653c185a0d0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f84f2a00640 1 -- 172.21.15.81:0/606849243 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f850401f420 con 0x5653c185a0d0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f84f2a00640 1 -- 172.21.15.81:0/606849243 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f850401f620 con 0x5653c185a0d0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f85167fd740 10 client.24701 fetch_fsmap learned FSMap version 17 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f85167fd740 1 -- 172.21.15.81:0/606849243 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5653c18693f0 con 0x5653c185a0d0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f84f2a00640 1 -- 172.21.15.81:0/606849243 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84f809d210 con 0x5653c185a0d0 2024-04-12T21:13:21.168 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f84f2a00640 1 -- 172.21.15.81:0/606849243 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f8504027070 con 0x5653c185a0d0 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.168+0000 7f85167fd740 10 client.24701 fetch_fsmap finished waiting for FSMap version 17 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.169+0000 7f85167fd740 10 client.24701 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.169+0000 7f85167fd740 20 client.24701 populate_metadata read hostname 'smithi081' 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.169+0000 7f85167fd740 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x5653c186bac0 0x5653c186cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.169+0000 7f85167fd740 4 client.24701 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.169+0000 7f85167fd740 1 -- 172.21.15.81:0/606849243 --> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653c1760b40 con 0x5653c186bac0 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.169+0000 7f8511000640 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x5653c186bac0 0x5653c186cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.169+0000 7f8511000640 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x5653c186bac0 0x5653c186cf30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38929825de3c2a50 server_cookie=f63015a5c4fe02a6 in_seq=0 out_seq=0 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.169+0000 7f84f2a00640 10 client.24701 ms_handle_connect on v2:172.21.15.104:6838/1944172329 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.170+0000 7f84f2a00640 1 -- 172.21.15.81:0/606849243 <== mds.0 v2:172.21.15.104:6838/1944172329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653c1760b40 con 0x5653c186bac0 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.170+0000 7f84f2a00640 10 client.24701 handle_command_reply: tid=0 2024-04-12T21:13:21.169 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 client.24701 shutdown 2024-04-12T21:13:21.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 -- 172.21.15.81:0/606849243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 msgr2=0x5653c1865fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1865fc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f850400fb50 tx=0x7f8504004ab0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 -- 172.21.15.81:0/606849243 shutdown_connections 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x5653c186bac0 0x5653c186cf30 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1865fc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c185ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f84f2a00640 0 client.24701 ms_handle_reset on v2:172.21.15.104:6838/1944172329 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 --2- 172.21.15.81:0/606849243 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854570 0x5653c185ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 -- 172.21.15.81:0/606849243 >> 172.21.15.81:0/606849243 conn(0x5653c184fae0 msgr2=0x5653c184ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 -- 172.21.15.81:0/606849243 shutdown_connections 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 1 -- 172.21.15.81:0/606849243 wait complete. 2024-04-12T21:13:21.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.171+0000 7f85167fd740 20 client.24701 trim_cache size 0 max 16384 2024-04-12T21:13:21.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.190+0000 7f85167fd740 1 Processor -- start 2024-04-12T21:13:21.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f85167fd740 1 -- start start 2024-04-12T21:13:21.187 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f85167fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854570 0x5653c186a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f85167fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854f60 0x5653c1865f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f85167fd740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1856460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f85167fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5653c186b270 con 0x5653c1854570 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f85167fd740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5653c186b940 con 0x5653c185a0d0 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f85167fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5653c186c010 con 0x5653c1854f60 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f8511000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854570 0x5653c186a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f850be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854f60 0x5653c1865f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f8511000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854570 0x5653c186a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52550/0 (socket says 172.21.15.81:52550) 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f8511000640 1 -- 172.21.15.81:0/2265466373 learned_addr learned my addr 172.21.15.81:0/2265466373 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f8511a00640 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1856460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f8511000640 1 -- 172.21.15.81:0/2265466373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854f60 msgr2=0x5653c1865f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.191+0000 7f8511000640 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854f60 0x5653c1865f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f8511000640 1 -- 172.21.15.81:0/2265466373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 msgr2=0x5653c1856460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f8511000640 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1856460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.188 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f8511000640 1 -- 172.21.15.81:0/2265466373 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653c186b720 con 0x5653c1854570 2024-04-12T21:13:21.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f8511a00640 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1856460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:13:21.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f850be00640 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854f60 0x5653c1865f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:21.189 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f8511000640 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854570 0x5653c186a860 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f850002c7b0 tx=0x7f850002df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9652d4cbd6ff88f server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f84fd600640 1 -- 172.21.15.81:0/2265466373 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850004e030 con 0x5653c1854570 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f84fd600640 1 -- 172.21.15.81:0/2265466373 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f850002cb40 con 0x5653c1854570 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.192+0000 7f84fd600640 1 -- 172.21.15.81:0/2265466373 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850004a810 con 0x5653c1854570 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 -- 172.21.15.81:0/2265466373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854570 msgr2=0x5653c186a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854570 0x5653c186a860 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f850002c7b0 tx=0x7f850002df90 comp rx=0 tx=0).stop 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 -- 172.21.15.81:0/2265466373 shutdown_connections 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c1854f60 0x5653c1865f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c185a0d0 0x5653c1856460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 --2- 172.21.15.81:0/2265466373 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854570 0x5653c186a860 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.194 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 -- 172.21.15.81:0/2265466373 >> 172.21.15.81:0/2265466373 conn(0x5653c184fae0 msgr2=0x5653c184fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 -- 172.21.15.81:0/2265466373 shutdown_connections 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.193+0000 7f85167fd740 1 -- 172.21.15.81:0/2265466373 wait complete. 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 1 -- start start 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 1 Processor -- start 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c18621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c185a0d0 0x5653c18626f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c1865300 0x5653c1865780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x5653c186cdb0 con 0x5653c1854f60 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x5653c186c6e0 con 0x5653c1865300 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x5653c186d480 con 0x5653c185a0d0 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f85167fd740 10 client.0 _ll_register_callbacks cb 0x7f850400f9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f8511000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c18621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f8511000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c18621b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:52552/0 (socket says 172.21.15.81:52552) 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.194+0000 7f8511000640 1 -- 172.21.15.81:0/2440883822 learned_addr learned my addr 172.21.15.81:0/2440883822 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:21.195 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f8511a00640 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c1865300 0x5653c1865780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f850be00640 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c185a0d0 0x5653c18626f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f8511000640 1 -- 172.21.15.81:0/2440883822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c185a0d0 msgr2=0x5653c18626f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f8511000640 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c185a0d0 0x5653c18626f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f8511000640 1 -- 172.21.15.81:0/2440883822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c1865300 msgr2=0x5653c1865780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f8511000640 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c1865300 0x5653c1865780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f8511000640 1 -- 172.21.15.81:0/2440883822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653c1865cc0 con 0x5653c1854f60 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f8511000640 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c18621b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8500000c00 tx=0x7f850004a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c2814a9a657c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f84f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f84f2a00640 1 -- 172.21.15.81:0/2440883822 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500057060 con 0x5653c1854f60 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f85167fd740 1 -- 172.21.15.81:0/2440883822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f850002cb40 con 0x5653c1854f60 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.195+0000 7f84f2a00640 1 -- 172.21.15.81:0/2440883822 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8500005030 con 0x5653c1854f60 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f84f2a00640 1 -- 172.21.15.81:0/2440883822 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f850005e050 con 0x5653c1854f60 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f85167fd740 10 client.14583 fetch_fsmap learned FSMap version 17 2024-04-12T21:13:21.196 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f85167fd740 1 -- 172.21.15.81:0/2440883822 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x5653c18d2430 con 0x5653c1854f60 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f84f2a00640 1 -- 172.21.15.81:0/2440883822 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f850002cb40 con 0x5653c1854f60 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f84f2a00640 1 -- 172.21.15.81:0/2440883822 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f8500004c90 con 0x5653c1854f60 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f85167fd740 10 client.14583 fetch_fsmap finished waiting for FSMap version 17 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f85167fd740 10 client.14583 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f85167fd740 20 client.14583 populate_metadata read hostname 'smithi081' 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f85167fd740 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x5653c18d6c50 0x5653c18d5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f85167fd740 4 client.14583 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f85167fd740 1 -- 172.21.15.81:0/2440883822 --> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] -- command(tid 0: {"prefix": "status"}) -- 0x5653c17c83b0 con 0x5653c18d6c50 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.196+0000 7f850be00640 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x5653c18d6c50 0x5653c18d5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f850be00640 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x5653c18d6c50 0x5653c18d5960 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f75d70cc51e2f75 server_cookie=84ef6dcdf2687b73 in_seq=0 out_seq=0 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f84f2a00640 10 client.14583 ms_handle_connect on v2:172.21.15.104:6838/1944172329 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f84f2a00640 1 -- 172.21.15.81:0/2440883822 <== mds.0 v2:172.21.15.104:6838/1944172329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5653c17c83b0 con 0x5653c18d6c50 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f84f2a00640 10 client.14583 handle_command_reply: tid=0 2024-04-12T21:13:21.197 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 client.14583 shutdown 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 -- 172.21.15.81:0/2440883822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 msgr2=0x5653c18621b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c18621b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8500000c00 tx=0x7f850004a460 comp rx=0 tx=0).stop 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 -- 172.21.15.81:0/2440883822 shutdown_connections 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x5653c18d6c50 0x5653c18d5960 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x5653c185a0d0 0x5653c18626f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x5653c1865300 0x5653c1865780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f84f2a00640 0 client.14583 ms_handle_reset on v2:172.21.15.104:6838/1944172329 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 --2- 172.21.15.81:0/2440883822 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x5653c1854f60 0x5653c18621b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 -- 172.21.15.81:0/2440883822 >> 172.21.15.81:0/2440883822 conn(0x5653c184fae0 msgr2=0x5653c186a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.197+0000 7f85167fd740 1 -- 172.21.15.81:0/2440883822 shutdown_connections 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f85167fd740 1 -- 172.21.15.81:0/2440883822 wait complete. 2024-04-12T21:13:21.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f85167fd740 20 client.14583 trim_cache size 0 max 16384 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "11b8cc88-f910-11ee-bc8d-c7b262605968", 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24593, 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 7.6689175440000001, 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 17, 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 89, 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 88, 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 14.874474384999999 2024-04-12T21:13:21.199 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-12T21:13:21.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 -- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84f40a3bb0 msgr2=0x7f84f40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84f40a3bb0 0x7f84f40a6070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x5653c18557c0 tx=0x7f8504009500 comp rx=0 tx=0).stop 2024-04-12T21:13:21.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 -- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f850c105410 msgr2=0x7f850c19aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f850c105410 0x7f850c19aad0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f84f800e9f0 tx=0x7f84f800edf0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 -- 172.21.15.81:0/87410576 shutdown_connections 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84f40a3bb0 0x7f84f40a6070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f850c105dd0 0x7f850c1a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f850c105410 0x7f850c19aad0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 --2- 172.21.15.81:0/87410576 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f850c104220 0x7f850c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 -- 172.21.15.81:0/87410576 >> 172.21.15.81:0/87410576 conn(0x7f850c0ff7b0 msgr2=0x7f850c1087e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 -- 172.21.15.81:0/87410576 shutdown_connections 2024-04-12T21:13:21.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:21.198+0000 7f8513a00640 1 -- 172.21.15.81:0/87410576 wait complete. 2024-04-12T21:13:21.210 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-12T21:13:21.210 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-12T21:13:21.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph fs ls 2024-04-12T21:13:21.665 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:21 smithi081 ceph-mon[31453]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-12T21:13:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:21 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:13:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:21 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1310037741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:21 smithi153 ceph-mon[36569]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-12T21:13:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:21 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:13:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:21 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1310037741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:22.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:21 smithi104 ceph-mon[36624]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-12T21:13:22.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:21 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:13:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:21 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1310037741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:22 smithi081 ceph-mon[31453]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-12T21:13:23.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:22 smithi153 ceph-mon[36569]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-12T21:13:23.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:22 smithi104 ceph-mon[36624]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-12T21:13:24.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 -- 172.21.15.81:0/3356131901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c104e40 msgr2=0x7f9f0c105220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:24.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 --2- 172.21.15.81:0/3356131901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c104e40 0x7f9f0c105220 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0099c0 tx=0x7f9efc009cd0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 -- 172.21.15.81:0/3356131901 shutdown_connections 2024-04-12T21:13:24.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 --2- 172.21.15.81:0/3356131901 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f0c109f20 0x7f9f0c111af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 --2- 172.21.15.81:0/3356131901 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c104e40 0x7f9f0c105220 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 --2- 172.21.15.81:0/3356131901 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c1057f0 0x7f9f0c109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.064 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 -- 172.21.15.81:0/3356131901 >> 172.21.15.81:0/3356131901 conn(0x7f9f0c1006d0 msgr2=0x7f9f0c102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:24.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 -- 172.21.15.81:0/3356131901 shutdown_connections 2024-04-12T21:13:24.065 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.068+0000 7f9f0be00700 1 -- 172.21.15.81:0/3356131901 wait complete. 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.069+0000 7f9f0be00700 1 Processor -- start 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.069+0000 7f9f0be00700 1 -- start start 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0be00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f0c104e40 0x7f9f0c076fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0be00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c1057f0 0x7f9f0c077500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0be00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c109f20 0x7f9f0c077a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0be00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c114f20 con 0x7f9f0c109f20 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0be00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c114da0 con 0x7f9f0c104e40 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0be00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c1150a0 con 0x7f9f0c1057f0 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0aa00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f0c104e40 0x7f9f0c076fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0b400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c109f20 0x7f9f0c077a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0aa00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f0c104e40 0x7f9f0c076fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:38358/0 (socket says 172.21.15.81:38358) 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0aa00700 1 -- 172.21.15.81:0/2059328240 learned_addr learned my addr 172.21.15.81:0/2059328240 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:24.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0a000700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c1057f0 0x7f9f0c077500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:24.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0aa00700 1 -- 172.21.15.81:0/2059328240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c1057f0 msgr2=0x7f9f0c077500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:24.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0aa00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c1057f0 0x7f9f0c077500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0aa00700 1 -- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c109f20 msgr2=0x7f9f0c077a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0aa00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c109f20 0x7f9f0c077a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0aa00700 1 -- 172.21.15.81:0/2059328240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c1aba00 con 0x7f9f0c104e40 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.070+0000 7f9f0b400700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c109f20 0x7f9f0c077a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.071+0000 7f9f0aa00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f0c104e40 0x7f9f0c076fc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0098e0 tx=0x7f9efc003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.071+0000 7f9f03400700 1 -- 172.21.15.81:0/2059328240 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc009e60 con 0x7f9f0c104e40 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.071+0000 7f9f03400700 1 -- 172.21.15.81:0/2059328240 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9efc003680 con 0x7f9f0c104e40 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.071+0000 7f9f03400700 1 -- 172.21.15.81:0/2059328240 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc00fdf0 con 0x7f9f0c104e40 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.071+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0c1abc90 con 0x7f9f0c104e40 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.071+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0c1ac0e0 con 0x7f9f0c104e40 2024-04-12T21:13:24.068 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.072+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f0c106320 con 0x7f9f0c104e40 2024-04-12T21:13:24.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.074+0000 7f9f03400700 1 -- 172.21.15.81:0/2059328240 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9efc00f4f0 con 0x7f9f0c104e40 2024-04-12T21:13:24.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.076+0000 7f9f03400700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9ef00a3c40 0x7f9ef00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:24.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.076+0000 7f9f03400700 1 -- 172.21.15.81:0/2059328240 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9efc0d1380 con 0x7f9f0c104e40 2024-04-12T21:13:24.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.076+0000 7f9f0a000700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9ef00a3c40 0x7f9ef00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:24.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.077+0000 7f9f0a000700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9ef00a3c40 0x7f9ef00a6100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c078780 tx=0x7f9ef800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:24.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.080+0000 7f9f03400700 1 -- 172.21.15.81:0/2059328240 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9efc013070 con 0x7f9f0c104e40 2024-04-12T21:13:24.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.358+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9f0c02e160 con 0x7f9f0c104e40 2024-04-12T21:13:24.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.359+0000 7f9f03400700 1 -- 172.21.15.81:0/2059328240 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f9efc09db50 con 0x7f9f0c104e40 2024-04-12T21:13:24.355 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-12T21:13:24.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9ef00a3c40 msgr2=0x7f9ef00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:24.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9ef00a3c40 0x7f9ef00a6100 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c078780 tx=0x7f9ef800b3c0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f0c104e40 msgr2=0x7f9f0c076fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:24.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f0c104e40 0x7f9f0c076fc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0098e0 tx=0x7f9efc003e00 comp rx=0 tx=0).stop 2024-04-12T21:13:24.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 shutdown_connections 2024-04-12T21:13:24.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9ef00a3c40 0x7f9ef00a6100 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9f0c109f20 0x7f9f0c077a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9f0c104e40 0x7f9f0c076fc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 --2- 172.21.15.81:0/2059328240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f0c1057f0 0x7f9f0c077500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:24.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.361+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 >> 172.21.15.81:0/2059328240 conn(0x7f9f0c1006d0 msgr2=0x7f9f0c1023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:24.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.362+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 shutdown_connections 2024-04-12T21:13:24.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:24.362+0000 7f9f0be00700 1 -- 172.21.15.81:0/2059328240 wait complete. 2024-04-12T21:13:24.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:24 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2059328240' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T21:13:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:24 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2059328240' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T21:13:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:24 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2059328240' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T21:13:25.165 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-12T21:13:25.165 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-12T21:13:25.165 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-04-12T21:13:25.165 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-12T21:13:25.165 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-12T21:13:25.165 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-12T21:13:25.165 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-12T21:13:25.165 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-12T21:13:25.166 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-12T21:13:25.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:25.166 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-04-12T21:13:25.218 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-12T21:13:25.218 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-12T21:13:25.218 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-04-12T21:13:25.218 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 42058sec preferred_lft 42058sec 2024-04-12T21:13:25.219 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-04-12T21:13:25.220 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-04-12T21:13:25.220 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-12T21:13:25.220 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:13:25.220 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:13:25.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add name ceph-brx type bridge 2024-04-12T21:13:25.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr flush dev ceph-brx 2024-04-12T21:13:25.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set ceph-brx up 2024-04-12T21:13:25.221 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-12T21:13:25.221 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:13:25.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:13:25.628 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:25 smithi081 ceph-mon[31453]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 925 B/s wr, 0 op/s 2024-04-12T21:13:25.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:13:25.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:25.722 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-12T21:13:25.794 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-04-12T21:13:25.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:25.798 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-04-12T21:13:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:25 smithi153 ceph-mon[36569]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 925 B/s wr, 0 op/s 2024-04-12T21:13:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:25 smithi104 ceph-mon[36624]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 925 B/s wr, 0 op/s 2024-04-12T21:13:25.849 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-04-12T21:13:25.849 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-12T21:13:25.849 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-12T21:13:25.849 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-12T21:13:25.849 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-12T21:13:25.850 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:13:25.850 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:13:25.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-12T21:13:25.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-12T21:13:25.850 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-12T21:13:25.850 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:13:25.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:13:26.599 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:13:26.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:26.609 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-04-12T21:13:26.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:26.664 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list-id 2024-04-12T21:13:26.713 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:13:26.714 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:13:26.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-12T21:13:26.714 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-12T21:13:26.714 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:13:26.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:13:26.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:13:26.916 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-12T21:13:26.917 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:13:26.917 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:13:26.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-12T21:13:26.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-12T21:13:26.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-12T21:13:26.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-12T21:13:26.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-12T21:13:26.917 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:13:27.019 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:13:27.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:13:27.264 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:13:27.264 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:13:27.264 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-04-12T21:13:27.265 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set dev brx.0 master ceph-brx 2024-04-12T21:13:27.265 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:13:27.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:13:27.484 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:13:27.493 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-12T21:13:27.493 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T21:13:27.493 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-12T21:13:27.545 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-12T21:13:27.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T21:13:27.545 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-12T21:13:27.594 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe fuse 2024-04-12T21:13:27.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:27 smithi081 ceph-mon[31453]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 819 B/s wr, 0 op/s 2024-04-12T21:13:27.679 DEBUG:teuthology.orchestra.run.smithi081:> cat /proc/self/mounts | awk '{print $2}' 2024-04-12T21:13:27.730 INFO:teuthology.orchestra.run.smithi081.stdout:/proc 2024-04-12T21:13:27.730 INFO:teuthology.orchestra.run.smithi081.stdout:/sys 2024-04-12T21:13:27.730 INFO:teuthology.orchestra.run.smithi081.stdout:/dev 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/security 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/shm 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/pts 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/run 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/cgroup 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/pstore 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/bpf 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/ 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/selinux 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/sys/fs/binfmt_misc 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/hugepages 2024-04-12T21:13:27.731 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/mqueue 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/debug 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/tracing 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/fuse/connections 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/config 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-sysctl.service 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/nfs/rpc_pipefs 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/run/user/1000 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/a2e1aabc070654511889f2571648a2bdb3ea423c3911f40b7db21ffdd021a282/merged 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/905071e601f5c1c5d8827903616aaca66bbc08558222d9bc9452b3eb77fe704f/merged 2024-04-12T21:13:27.732 INFO:teuthology.orchestra.run.smithi081.stdout:/run/user/0 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/sys/fs/binfmt_misc 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/e1636e41ebf18ec7b1214dc287712944cf4c91b83e3401e0f5dc018e5458f7a5/merged 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/998c1ad46ca0791975fe0a7f126c4954386100a75957fe4aaf4f46973e8d84e1/merged 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/454b8e1af4978446b07af5af3ad56c793a272265be2b7c0fb8dc5944a4afb26b/merged 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/2431f632cb9b8440b7f997da50134d4bfd78d90379e411fc0d7a0496575e462f/merged 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/6056b0a48c24bf206144336c5d9b8f40689f816eb57ba2a8a7e3d74bde58ef97/merged 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/c25ddf906f36b2546bb447b4ec5c37c7bde7c54dd4ed6c16b7ec248f592dd7ed/merged 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/67a00eddeebf5661d88a84e57dae6a105375583bdc674c49fca3e1a70df84fd6/merged 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/168586b962855483d79a53eed2e54ea0ef306af9cb8f6e57f126deaddc16a08c/merged 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-12T21:13:27.733 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-12T21:13:27.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:27.734 DEBUG:teuthology.orchestra.run.smithi081:> ls /sys/fs/fuse/connections 2024-04-12T21:13:27.783 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-12T21:13:27.783 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-12T21:13:27.788 DEBUG:teuthology.orchestra.run.smithi081:> sudo modprobe fuse 2024-04-12T21:13:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:27 smithi153 ceph-mon[36569]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 819 B/s wr, 0 op/s 2024-04-12T21:13:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:27 smithi104 ceph-mon[36624]: pgmap v321: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 819 B/s wr, 0 op/s 2024-04-12T21:13:27.872 DEBUG:teuthology.orchestra.run.smithi081:> cat /proc/self/mounts | awk '{print $2}' 2024-04-12T21:13:27.916 INFO:teuthology.orchestra.run.smithi081.stdout:/proc 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/sys 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/dev 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/security 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/shm 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/pts 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/run 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/cgroup 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/pstore 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/bpf 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/ 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/selinux 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/sys/fs/binfmt_misc 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/hugepages 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/dev/mqueue 2024-04-12T21:13:27.917 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/debug 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/tracing 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/fs/fuse/connections 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/sys/kernel/config 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-sysctl.service 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/nfs/rpc_pipefs 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/run/user/1000 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/ceph 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/a2e1aabc070654511889f2571648a2bdb3ea423c3911f40b7db21ffdd021a282/merged 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/905071e601f5c1c5d8827903616aaca66bbc08558222d9bc9452b3eb77fe704f/merged 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/run/user/0 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/proc/sys/fs/binfmt_misc 2024-04-12T21:13:27.918 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/e1636e41ebf18ec7b1214dc287712944cf4c91b83e3401e0f5dc018e5458f7a5/merged 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/998c1ad46ca0791975fe0a7f126c4954386100a75957fe4aaf4f46973e8d84e1/merged 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/454b8e1af4978446b07af5af3ad56c793a272265be2b7c0fb8dc5944a4afb26b/merged 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/2431f632cb9b8440b7f997da50134d4bfd78d90379e411fc0d7a0496575e462f/merged 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/6056b0a48c24bf206144336c5d9b8f40689f816eb57ba2a8a7e3d74bde58ef97/merged 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/c25ddf906f36b2546bb447b4ec5c37c7bde7c54dd4ed6c16b7ec248f592dd7ed/merged 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/67a00eddeebf5661d88a84e57dae6a105375583bdc674c49fca3e1a70df84fd6/merged 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/var/lib/containers/storage/overlay/168586b962855483d79a53eed2e54ea0ef306af9cb8f6e57f126deaddc16a08c/merged 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-12T21:13:27.919 INFO:teuthology.orchestra.run.smithi081.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-12T21:13:27.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:27.920 DEBUG:teuthology.orchestra.run.smithi081:> ls /sys/fs/fuse/connections 2024-04-12T21:13:27.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi081.stderr:2024-04-12T21:13:27.997+0000 7feb30440480 -1 init, newargv = 0x557dcf4883c0 newargc=15 2024-04-12T21:13:27.994 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi081.stderr:ceph-fuse[75262]: starting ceph client 2024-04-12T21:13:28.005 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi081.stderr:ceph-fuse[75262]: starting fuse 2024-04-12T21:13:28.011 INFO:teuthology.orchestra.run.smithi081.stdout:119 2024-04-12T21:13:28.011 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-04-12T21:13:28.011 DEBUG:teuthology.orchestra.run.smithi081:> sudo stdin-killer -- python3 -c ' 2024-04-12T21:13:28.011 DEBUG:teuthology.orchestra.run.smithi081:> import glob 2024-04-12T21:13:28.011 DEBUG:teuthology.orchestra.run.smithi081:> import re 2024-04-12T21:13:28.011 DEBUG:teuthology.orchestra.run.smithi081:> import os 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> import subprocess 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> def _find_admin_socket(client_name): 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> files = glob.glob(asok_path) 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> # Given a non-glob path, it better be there 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> if "*" not in asok_path: 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> assert(len(files) == 1) 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> return files[0] 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> for f in files: 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-12T21:13:28.012 DEBUG:teuthology.orchestra.run.smithi081:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-12T21:13:28.013 DEBUG:teuthology.orchestra.run.smithi081:> contents = proc_f.read() 2024-04-12T21:13:28.013 DEBUG:teuthology.orchestra.run.smithi081:> if mountpoint in contents: 2024-04-12T21:13:28.013 DEBUG:teuthology.orchestra.run.smithi081:> return f 2024-04-12T21:13:28.013 DEBUG:teuthology.orchestra.run.smithi081:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-12T21:13:28.013 DEBUG:teuthology.orchestra.run.smithi081:> 2024-04-12T21:13:28.013 DEBUG:teuthology.orchestra.run.smithi081:> print(_find_admin_socket("client.admin")) 2024-04-12T21:13:28.013 DEBUG:teuthology.orchestra.run.smithi081:> ' 2024-04-12T21:13:28.209 INFO:teuthology.orchestra.run.smithi081.stdout:/var/run/ceph/ceph-client.admin.75262.asok 2024-04-12T21:13:28.213 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:13:28.226 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75262.asok 2024-04-12T21:13:28.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:28.226 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75262.asok status 2024-04-12T21:13:28.467 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-12T21:13:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "metadata": { 2024-04-12T21:13:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_sha1": "324baf8ddb91dea46cac50cb535c306115791355", 2024-04-12T21:13:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "ceph_version": "ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev)", 2024-04-12T21:13:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "entity_id": "admin", 2024-04-12T21:13:28.467 INFO:teuthology.orchestra.run.smithi081.stdout: "hostname": "smithi081", 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "pid": "75262", 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "root": "/" 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "dentry_count": 0, 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "dentry_pinned_count": 0, 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 14595, 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "inst": { 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "name": { 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "client", 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "num": 14595 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": { 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "192.168.0.1:0", 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2927649524 2024-04-12T21:13:28.468 INFO:teuthology.orchestra.run.smithi081.stdout: } 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": { 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "type": "v1", 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "addr": "192.168.0.1:0", 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "nonce": 2927649524 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: }, 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "inst_str": "client.14595 192.168.0.1:0/2927649524", 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "addr_str": "192.168.0.1:0/2927649524", 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "inode_count": 1, 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "mds_epoch": 17, 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_epoch": 89, 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "osd_epoch_barrier": 0, 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "blocklisted": false, 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs" 2024-04-12T21:13:28.469 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-12T21:13:28.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:28.483 DEBUG:teuthology.orchestra.run.smithi081:> stat --file-system '--printf=%T 2024-04-12T21:13:28.483 DEBUG:teuthology.orchestra.run.smithi081:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-12T21:13:28.538 INFO:teuthology.orchestra.run.smithi081.stdout:fuseblk 2024-04-12T21:13:28.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-12T21:13:28.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:28.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-12T21:13:28.628 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-12T21:13:28.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:28 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-12T21:13:28.775 INFO:teuthology.orchestra.run.smithi081.stdout:# file: . 2024-04-12T21:13:28.775 INFO:teuthology.orchestra.run.smithi081.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-12T21:13:28.775 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:28.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:13:28.785 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:13:28.785 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T21:13:28.835 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 -- 172.21.15.81:0/3825751735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140102f80 msgr2=0x7f6140103400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 --2- 172.21.15.81:0/3825751735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140102f80 0x7f6140103400 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f612c009900 tx=0x7f612c02bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 -- 172.21.15.81:0/3825751735 shutdown_connections 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 --2- 172.21.15.81:0/3825751735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140102f80 0x7f6140103400 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 --2- 172.21.15.81:0/3825751735 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6140103940 0x7f614010a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 --2- 172.21.15.81:0/3825751735 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6140101e30 0x7f6140102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 -- 172.21.15.81:0/3825751735 >> 172.21.15.81:0/3825751735 conn(0x7f61400fd3a0 msgr2=0x7f61400ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 -- 172.21.15.81:0/3825751735 shutdown_connections 2024-04-12T21:13:29.093 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.096+0000 7f6146000640 1 -- 172.21.15.81:0/3825751735 wait complete. 2024-04-12T21:13:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.097+0000 7f6146000640 1 Processor -- start 2024-04-12T21:13:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.097+0000 7f6146000640 1 -- start start 2024-04-12T21:13:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.097+0000 7f6146000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140101e30 0x7f61401981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.097+0000 7f6146000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6140102f80 0x7f61401986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.097+0000 7f6146000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6140103940 0x7f614019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.097+0000 7f6146000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f614010db30 con 0x7f6140103940 2024-04-12T21:13:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.097+0000 7f6146000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f614010d9b0 con 0x7f6140102f80 2024-04-12T21:13:29.096 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.097+0000 7f6146000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f614010dcb0 con 0x7f6140101e30 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140101e30 0x7f61401981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140101e30 0x7f61401981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:58192/0 (socket says 172.21.15.81:58192) 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 -- 172.21.15.81:0/634835923 learned_addr learned my addr 172.21.15.81:0/634835923 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6145600640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6140103940 0x7f614019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f613fe00640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6140102f80 0x7f61401986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 -- 172.21.15.81:0/634835923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6140102f80 msgr2=0x7f61401986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6140102f80 0x7f61401986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 -- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6140103940 msgr2=0x7f614019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6140103940 0x7f614019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 -- 172.21.15.81:0/634835923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f614019fdc0 con 0x7f6140101e30 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.098+0000 7f6144c00640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140101e30 0x7f61401981a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f613400ea60 tx=0x7f613400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.099+0000 7f613d600640 1 -- 172.21.15.81:0/634835923 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f613400cde0 con 0x7f6140101e30 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.099+0000 7f613d600640 1 -- 172.21.15.81:0/634835923 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61340035f0 con 0x7f6140101e30 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.099+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61401a00b0 con 0x7f6140101e30 2024-04-12T21:13:29.097 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.099+0000 7f613d600640 1 -- 172.21.15.81:0/634835923 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6134010670 con 0x7f6140101e30 2024-04-12T21:13:29.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.099+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61401a05d0 con 0x7f6140101e30 2024-04-12T21:13:29.098 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.100+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6140109140 con 0x7f6140101e30 2024-04-12T21:13:29.099 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.102+0000 7f613d600640 1 -- 172.21.15.81:0/634835923 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f61340037a0 con 0x7f6140101e30 2024-04-12T21:13:29.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.104+0000 7f613d600640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61280a3bb0 0x7f61280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:29.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.104+0000 7f613d600640 1 -- 172.21.15.81:0/634835923 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f61340d09c0 con 0x7f6140101e30 2024-04-12T21:13:29.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.104+0000 7f613fe00640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61280a3bb0 0x7f61280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:29.101 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.105+0000 7f613fe00640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61280a3bb0 0x7f61280a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f612c02c170 tx=0x7f612c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:29.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.106+0000 7f613d600640 1 -- 172.21.15.81:0/634835923 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f613409d1d0 con 0x7f6140101e30 2024-04-12T21:13:29.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.316+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6140062bc0 con 0x7f6140101e30 2024-04-12T21:13:29.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.319+0000 7f613d600640 1 -- 172.21.15.81:0/634835923 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7f613409cad0 con 0x7f6140101e30 2024-04-12T21:13:29.316 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:29.316 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:29.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.321+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61280a3bb0 msgr2=0x7f61280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.321+0000 7f6146000640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61280a3bb0 0x7f61280a6070 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f612c02c170 tx=0x7f612c02df90 comp rx=0 tx=0).stop 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.321+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140101e30 msgr2=0x7f61401981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.321+0000 7f6146000640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140101e30 0x7f61401981a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f613400ea60 tx=0x7f613400ef20 comp rx=0 tx=0).stop 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.322+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 shutdown_connections 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.322+0000 7f6146000640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f61280a3bb0 0x7f61280a6070 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.322+0000 7f6146000640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6140103940 0x7f614019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.322+0000 7f6146000640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6140102f80 0x7f61401986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.322+0000 7f6146000640 1 --2- 172.21.15.81:0/634835923 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6140101e30 0x7f61401981a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.322+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 >> 172.21.15.81:0/634835923 conn(0x7f61400fd3a0 msgr2=0x7f6140106310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.322+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 shutdown_connections 2024-04-12T21:13:29.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.322+0000 7f6146000640 1 -- 172.21.15.81:0/634835923 wait complete. 2024-04-12T21:13:29.338 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-12T21:13:10.561238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-12T21:13:11.985544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-12T21:13:15.579438+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-12T21:13:29.338 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T21:13:29.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.555+0000 7fc406400640 1 -- 172.21.15.81:0/2771749408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc400108620 msgr2=0x7fc400108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.555+0000 7fc406400640 1 --2- 172.21.15.81:0/2771749408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc400108620 0x7fc400108a40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009900 tx=0x7fc3f002bc50 comp rx=0 tx=0).stop 2024-04-12T21:13:29.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.555+0000 7fc406400640 1 -- 172.21.15.81:0/2771749408 shutdown_connections 2024-04-12T21:13:29.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.555+0000 7fc406400640 1 --2- 172.21.15.81:0/2771749408 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc400109810 0x7fc400109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.555+0000 7fc406400640 1 --2- 172.21.15.81:0/2771749408 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc40010a1d0 0x7fc400110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.555+0000 7fc406400640 1 --2- 172.21.15.81:0/2771749408 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc400108620 0x7fc400108a40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.555+0000 7fc406400640 1 -- 172.21.15.81:0/2771749408 >> 172.21.15.81:0/2771749408 conn(0x7fc4000775c0 msgr2=0x7fc400079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:29.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.556+0000 7fc406400640 1 -- 172.21.15.81:0/2771749408 shutdown_connections 2024-04-12T21:13:29.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.556+0000 7fc406400640 1 -- 172.21.15.81:0/2771749408 wait complete. 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.557+0000 7fc406400640 1 Processor -- start 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.557+0000 7fc406400640 1 -- start start 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.557+0000 7fc406400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc400108620 0x7fc40019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.557+0000 7fc406400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc400109810 0x7fc40019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.557+0000 7fc406400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc40010a1d0 0x7fc4001a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.557+0000 7fc406400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc400114290 con 0x7fc400108620 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.557+0000 7fc406400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc400114110 con 0x7fc400109810 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.557+0000 7fc406400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc400114410 con 0x7fc40010a1d0 2024-04-12T21:13:29.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ff400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc400108620 0x7fc40019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ff400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc400108620 0x7fc40019e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38690/0 (socket says 172.21.15.81:38690) 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ff400640 1 -- 172.21.15.81:0/400495722 learned_addr learned my addr 172.21.15.81:0/400495722 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ffe00640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc40010a1d0 0x7fc4001a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3fea00640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc400109810 0x7fc40019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ff400640 1 -- 172.21.15.81:0/400495722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc40010a1d0 msgr2=0x7fc4001a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ff400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc40010a1d0 0x7fc4001a5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ff400640 1 -- 172.21.15.81:0/400495722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc400109810 msgr2=0x7fc40019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ff400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc400109810 0x7fc40019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ff400640 1 -- 172.21.15.81:0/400495722 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4001a65d0 con 0x7fc400108620 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.558+0000 7fc3ffe00640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc40010a1d0 0x7fc4001a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:29.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.559+0000 7fc3ff400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc400108620 0x7fc40019e9b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0000c00 tx=0x7fc3f002c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:29.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.559+0000 7fc3f7e00640 1 -- 172.21.15.81:0/400495722 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0042020 con 0x7fc400108620 2024-04-12T21:13:29.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.559+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4001a6860 con 0x7fc400108620 2024-04-12T21:13:29.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.559+0000 7fc3f7e00640 1 -- 172.21.15.81:0/400495722 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3f0004450 con 0x7fc400108620 2024-04-12T21:13:29.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.559+0000 7fc3f7e00640 1 -- 172.21.15.81:0/400495722 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f003e460 con 0x7fc400108620 2024-04-12T21:13:29.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.559+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4001a6db0 con 0x7fc400108620 2024-04-12T21:13:29.557 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.560+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc40010f940 con 0x7fc400108620 2024-04-12T21:13:29.559 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.562+0000 7fc3f7e00640 1 -- 172.21.15.81:0/400495722 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc3f0045020 con 0x7fc400108620 2024-04-12T21:13:29.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.565+0000 7fc3f7e00640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc3e40a3bb0 0x7fc3e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:29.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.565+0000 7fc3f7e00640 1 -- 172.21.15.81:0/400495722 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc3f00f25c0 con 0x7fc400108620 2024-04-12T21:13:29.561 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.565+0000 7fc3fea00640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc3e40a3bb0 0x7fc3e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:29.562 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.565+0000 7fc3fea00640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc3e40a3bb0 0x7fc3e40a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc40019fec0 tx=0x7fc3ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:29.566 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.570+0000 7fc3f7e00640 1 -- 172.21.15.81:0/400495722 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3f004c020 con 0x7fc400108620 2024-04-12T21:13:29.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:29 smithi081 ceph-mon[31453]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s wr, 0 op/s 2024-04-12T21:13:29.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:29 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/634835923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:29.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.772+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc400062bc0 con 0x7fc400108620 2024-04-12T21:13:29.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.773+0000 7fc3f7e00640 1 -- 172.21.15.81:0/400495722 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fc3f00bed90 con 0x7fc400108620 2024-04-12T21:13:29.770 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:13:29.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.775+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc3e40a3bb0 msgr2=0x7fc3e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.775+0000 7fc406400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc3e40a3bb0 0x7fc3e40a6070 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fc40019fec0 tx=0x7fc3ec005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.775+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc400108620 msgr2=0x7fc40019e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.775+0000 7fc406400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc400108620 0x7fc40019e9b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0000c00 tx=0x7fc3f002c660 comp rx=0 tx=0).stop 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.776+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 shutdown_connections 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.776+0000 7fc406400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc3e40a3bb0 0x7fc3e40a6070 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.776+0000 7fc406400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc40010a1d0 0x7fc4001a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.776+0000 7fc406400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc400109810 0x7fc40019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.776+0000 7fc406400640 1 --2- 172.21.15.81:0/400495722 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc400108620 0x7fc40019e9b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.776+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 >> 172.21.15.81:0/400495722 conn(0x7fc4000775c0 msgr2=0x7fc40010cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.776+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 shutdown_connections 2024-04-12T21:13:29.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.776+0000 7fc406400640 1 -- 172.21.15.81:0/400495722 wait complete. 2024-04-12T21:13:29.791 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-12T21:13:29.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:29 smithi153 ceph-mon[36569]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s wr, 0 op/s 2024-04-12T21:13:29.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:29 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/634835923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:29.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:29 smithi104 ceph-mon[36624]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 716 B/s wr, 0 op/s 2024-04-12T21:13:29.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:29 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/634835923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:29.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.999+0000 7f43a2600640 1 -- 172.21.15.81:0/2893857564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c076980 msgr2=0x7f439c076da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:29.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.999+0000 7f43a2600640 1 --2- 172.21.15.81:0/2893857564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c076980 0x7f439c076da0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f438c009900 tx=0x7f438c02bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:29.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.999+0000 7f43a2600640 1 -- 172.21.15.81:0/2893857564 shutdown_connections 2024-04-12T21:13:29.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.999+0000 7f43a2600640 1 --2- 172.21.15.81:0/2893857564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439c078560 0x7f439c078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.999+0000 7f43a2600640 1 --2- 172.21.15.81:0/2893857564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f439c077bc0 0x7f439c078020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.999+0000 7f43a2600640 1 --2- 172.21.15.81:0/2893857564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c076980 0x7f439c076da0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:29.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.999+0000 7f43a2600640 1 -- 172.21.15.81:0/2893857564 >> 172.21.15.81:0/2893857564 conn(0x7f439c0ff7b0 msgr2=0x7f439c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:29.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:29.999+0000 7f43a2600640 1 -- 172.21.15.81:0/2893857564 shutdown_connections 2024-04-12T21:13:29.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.000+0000 7f43a2600640 1 -- 172.21.15.81:0/2893857564 wait complete. 2024-04-12T21:13:29.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.000+0000 7f43a2600640 1 Processor -- start 2024-04-12T21:13:29.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f43a2600640 1 -- start start 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f43a2600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439c076980 0x7f439c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f43a2600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c077bc0 0x7f439c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f43a2600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f439c078560 0x7f439c1a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f43a2600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f439c1142b0 con 0x7f439c077bc0 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f43a2600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f439c114130 con 0x7f439c078560 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f43a2600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f439c114430 con 0x7f439c076980 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f4393e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c077bc0 0x7f439c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f439b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439c076980 0x7f439c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f4393e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c077bc0 0x7f439c19eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38712/0 (socket says 172.21.15.81:38712) 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f439b400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439c076980 0x7f439c19e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:58234/0 (socket says 172.21.15.81:58234) 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f439be00640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f439c078560 0x7f439c1a5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.001+0000 7f4393e00640 1 -- 172.21.15.81:0/2169672960 learned_addr learned my addr 172.21.15.81:0/2169672960 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:30.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f439be00640 1 -- 172.21.15.81:0/2169672960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439c076980 msgr2=0x7f439c19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f439be00640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439c076980 0x7f439c19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f439be00640 1 -- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c077bc0 msgr2=0x7f439c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f439be00640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c077bc0 0x7f439c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f439be00640 1 -- 172.21.15.81:0/2169672960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439c1a65a0 con 0x7f439c078560 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f439be00640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f439c078560 0x7f439c1a5f30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4394009fd0 tx=0x7f439400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f4398c00640 1 -- 172.21.15.81:0/2169672960 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439400ce20 con 0x7f439c078560 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f4398c00640 1 -- 172.21.15.81:0/2169672960 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4394005020 con 0x7f439c078560 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f4398c00640 1 -- 172.21.15.81:0/2169672960 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394010720 con 0x7f439c078560 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f439c1a6890 con 0x7f439c078560 2024-04-12T21:13:30.001 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.002+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f439c1a6d90 con 0x7f439c078560 2024-04-12T21:13:30.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.004+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f439c110b90 con 0x7f439c078560 2024-04-12T21:13:30.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.006+0000 7f4398c00640 1 -- 172.21.15.81:0/2169672960 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4394005540 con 0x7f439c078560 2024-04-12T21:13:30.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.007+0000 7f4398c00640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43800a3c30 0x7f43800a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.007+0000 7f4398c00640 1 -- 172.21.15.81:0/2169672960 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f43940d0ba0 con 0x7f439c078560 2024-04-12T21:13:30.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.007+0000 7f439b400640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43800a3c30 0x7f43800a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.008+0000 7f439b400640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43800a3c30 0x7f43800a60f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f438c02c7b0 tx=0x7f438c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:30.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.011+0000 7f4398c00640 1 -- 172.21.15.81:0/2169672960 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f439409d2e0 con 0x7f439c078560 2024-04-12T21:13:30.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.213+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f439c062c60 con 0x7f439c078560 2024-04-12T21:13:30.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.214+0000 7f4398c00640 1 -- 172.21.15.81:0/2169672960 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f439409cbe0 con 0x7f439c078560 2024-04-12T21:13:30.211 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:13:30.214 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43800a3c30 msgr2=0x7f43800a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43800a3c30 0x7f43800a60f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f438c02c7b0 tx=0x7f438c005fd0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f439c078560 msgr2=0x7f439c1a5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f439c078560 0x7f439c1a5f30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4394009fd0 tx=0x7f439400ef10 comp rx=0 tx=0).stop 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 shutdown_connections 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f43800a3c30 0x7f43800a60f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f439c078560 0x7f439c1a5f30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f439c077bc0 0x7f439c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 --2- 172.21.15.81:0/2169672960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f439c076980 0x7f439c19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.217+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 >> 172.21.15.81:0/2169672960 conn(0x7f439c0ff7b0 msgr2=0x7f439c1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.218+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 shutdown_connections 2024-04-12T21:13:30.215 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.218+0000 7f43a2600640 1 -- 172.21.15.81:0/2169672960 wait complete. 2024-04-12T21:13:30.234 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-12T21:13:30.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 -- 172.21.15.81:0/3717545443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c104240 msgr2=0x7f0a4c104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 --2- 172.21.15.81:0/3717545443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c104240 0x7f0a4c104660 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009900 tx=0x7f0a3c02bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:30.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 -- 172.21.15.81:0/3717545443 shutdown_connections 2024-04-12T21:13:30.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 --2- 172.21.15.81:0/3717545443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a4c105df0 0x7f0a4c10c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 --2- 172.21.15.81:0/3717545443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a4c105430 0x7f0a4c1058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 --2- 172.21.15.81:0/3717545443 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c104240 0x7f0a4c104660 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 -- 172.21.15.81:0/3717545443 >> 172.21.15.81:0/3717545443 conn(0x7f0a4c0ff7b0 msgr2=0x7f0a4c101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:30.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 -- 172.21.15.81:0/3717545443 shutdown_connections 2024-04-12T21:13:30.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.461+0000 7f0a51000640 1 -- 172.21.15.81:0/3717545443 wait complete. 2024-04-12T21:13:30.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.462+0000 7f0a51000640 1 Processor -- start 2024-04-12T21:13:30.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.462+0000 7f0a51000640 1 -- start start 2024-04-12T21:13:30.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a51000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a4c104240 0x7f0a4c19a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a51000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c105430 0x7f0a4c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a51000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a4c105df0 0x7f0a4c1a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a51000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c10fea0 con 0x7f0a4c105430 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a51000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c10fd20 con 0x7f0a4c105df0 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a51000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c110020 con 0x7f0a4c104240 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c105430 0x7f0a4c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c105430 0x7f0a4c19aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38724/0 (socket says 172.21.15.81:38724) 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49400640 1 -- 172.21.15.81:0/403404708 learned_addr learned my addr 172.21.15.81:0/403404708 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49e00640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a4c104240 0x7f0a4c19a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a4a800640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a4c105df0 0x7f0a4c1a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49400640 1 -- 172.21.15.81:0/403404708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a4c104240 msgr2=0x7f0a4c19a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49400640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a4c104240 0x7f0a4c19a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49400640 1 -- 172.21.15.81:0/403404708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a4c105df0 msgr2=0x7f0a4c1a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49400640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a4c105df0 0x7f0a4c1a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.463+0000 7f0a49400640 1 -- 172.21.15.81:0/403404708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c1a2180 con 0x7f0a4c105430 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.464+0000 7f0a4a800640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a4c105df0 0x7f0a4c1a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.464+0000 7f0a49e00640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a4c104240 0x7f0a4c19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.464+0000 7f0a49400640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c105430 0x7f0a4c19aaa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800e9f0 tx=0x7f0a3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.464+0000 7f0a42a00640 1 -- 172.21.15.81:0/403404708 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3800cd80 con 0x7f0a4c105430 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.464+0000 7f0a42a00640 1 -- 172.21.15.81:0/403404708 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a38010910 con 0x7f0a4c105430 2024-04-12T21:13:30.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.464+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4c1a2470 con 0x7f0a4c105430 2024-04-12T21:13:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.464+0000 7f0a42a00640 1 -- 172.21.15.81:0/403404708 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a38018990 con 0x7f0a4c105430 2024-04-12T21:13:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.464+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4c1a29c0 con 0x7f0a4c105430 2024-04-12T21:13:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.466+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a4c10b630 con 0x7f0a4c105430 2024-04-12T21:13:30.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.468+0000 7f0a42a00640 1 -- 172.21.15.81:0/403404708 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0a38010ac0 con 0x7f0a4c105430 2024-04-12T21:13:30.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.469+0000 7f0a42a00640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0a300a3bb0 0x7f0a300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.469+0000 7f0a42a00640 1 -- 172.21.15.81:0/403404708 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0a380d09c0 con 0x7f0a4c105430 2024-04-12T21:13:30.466 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.470+0000 7f0a49e00640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0a300a3bb0 0x7f0a300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.467 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.470+0000 7f0a49e00640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0a300a3bb0 0x7f0a300a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c000c00 tx=0x7f0a3c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:30.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.474+0000 7f0a42a00640 1 -- 172.21.15.81:0/403404708 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a3809cab0 con 0x7f0a4c105430 2024-04-12T21:13:30.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:30 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/400495722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:30.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:30 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2169672960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:13:30.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.676+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a4c062bc0 con 0x7f0a4c105430 2024-04-12T21:13:30.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.677+0000 7f0a42a00640 1 -- 172.21.15.81:0/403404708 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f0a380a0db0 con 0x7f0a4c105430 2024-04-12T21:13:30.673 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-04-12T21:13:30.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0a300a3bb0 msgr2=0x7f0a300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0a300a3bb0 0x7f0a300a6070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c000c00 tx=0x7f0a3c005fd0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c105430 msgr2=0x7f0a4c19aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c105430 0x7f0a4c19aaa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800e9f0 tx=0x7f0a3800eeb0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 shutdown_connections 2024-04-12T21:13:30.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0a300a3bb0 0x7f0a300a6070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0a4c105df0 0x7f0a4c1a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0a4c105430 0x7f0a4c19aaa0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 --2- 172.21.15.81:0/403404708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0a4c104240 0x7f0a4c19a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.679+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 >> 172.21.15.81:0/403404708 conn(0x7f0a4c0ff7b0 msgr2=0x7f0a4c108800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:30.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.680+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 shutdown_connections 2024-04-12T21:13:30.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.680+0000 7f0a51000640 1 -- 172.21.15.81:0/403404708 wait complete. 2024-04-12T21:13:30.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-12T21:13:30.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:30 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/400495722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:30.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:30 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2169672960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:13:30.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:30 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/400495722' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:30.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:30 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2169672960' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:13:30.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.913+0000 7f7b43e00640 1 -- 172.21.15.81:0/2632010958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 msgr2=0x7f7b44109b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.910 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.913+0000 7f7b43e00640 1 --2- 172.21.15.81:0/2632010958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 0x7f7b44109b70 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009860 tx=0x7f7b2c02bb70 comp rx=0 tx=0).stop 2024-04-12T21:13:30.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.913+0000 7f7b43e00640 1 -- 172.21.15.81:0/2632010958 shutdown_connections 2024-04-12T21:13:30.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.913+0000 7f7b43e00640 1 --2- 172.21.15.81:0/2632010958 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 0x7f7b44109b70 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.913+0000 7f7b43e00640 1 --2- 172.21.15.81:0/2632010958 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b4410a0b0 0x7f7b44110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.913+0000 7f7b43e00640 1 --2- 172.21.15.81:0/2632010958 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b44108500 0x7f7b44108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.914+0000 7f7b43e00640 1 -- 172.21.15.81:0/2632010958 >> 172.21.15.81:0/2632010958 conn(0x7f7b44077490 msgr2=0x7f7b440798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:30.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.914+0000 7f7b43e00640 1 -- 172.21.15.81:0/2632010958 shutdown_connections 2024-04-12T21:13:30.911 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.914+0000 7f7b43e00640 1 -- 172.21.15.81:0/2632010958 wait complete. 2024-04-12T21:13:30.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43e00640 1 Processor -- start 2024-04-12T21:13:30.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43e00640 1 -- start start 2024-04-12T21:13:30.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b44108500 0x7f7b4419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 0x7f7b4419ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b4410a0b0 0x7f7b441a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44114190 con 0x7f7b4410a0b0 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44114010 con 0x7f7b44108500 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7b44114310 con 0x7f7b441096f0 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b42a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b44108500 0x7f7b4419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b4410a0b0 0x7f7b441a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b42a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b44108500 0x7f7b4419e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49370/0 (socket says 172.21.15.81:49370) 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.915+0000 7f7b42a00640 1 -- 172.21.15.81:0/825481659 learned_addr learned my addr 172.21.15.81:0/825481659 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b3be00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 0x7f7b4419ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b43400640 1 -- 172.21.15.81:0/825481659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 msgr2=0x7f7b4419ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b43400640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 0x7f7b4419ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b43400640 1 -- 172.21.15.81:0/825481659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b44108500 msgr2=0x7f7b4419e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b43400640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b44108500 0x7f7b4419e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:30.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b43400640 1 -- 172.21.15.81:0/825481659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b441a6410 con 0x7f7b4410a0b0 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b42a00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b44108500 0x7f7b4419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b3be00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 0x7f7b4419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b43400640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b4410a0b0 0x7f7b441a5da0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00efc0 tx=0x7f7b3c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.916+0000 7f7b3b400640 1 -- 172.21.15.81:0/825481659 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c003970 con 0x7f7b4410a0b0 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.917+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b441a6700 con 0x7f7b4410a0b0 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.917+0000 7f7b3b400640 1 -- 172.21.15.81:0/825481659 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b3c004020 con 0x7f7b4410a0b0 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.917+0000 7f7b3b400640 1 -- 172.21.15.81:0/825481659 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00b9f0 con 0x7f7b4410a0b0 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.917+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b441a6c20 con 0x7f7b4410a0b0 2024-04-12T21:13:30.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.918+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b4410f8a0 con 0x7f7b4410a0b0 2024-04-12T21:13:30.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.920+0000 7f7b3b400640 1 -- 172.21.15.81:0/825481659 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7b3c01d020 con 0x7f7b4410a0b0 2024-04-12T21:13:30.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.921+0000 7f7b3b400640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7b240a3bb0 0x7f7b240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:30.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.921+0000 7f7b3b400640 1 -- 172.21.15.81:0/825481659 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7b3c0d05d0 con 0x7f7b4410a0b0 2024-04-12T21:13:30.918 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.921+0000 7f7b42a00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7b240a3bb0 0x7f7b240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:30.920 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.922+0000 7f7b42a00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7b240a3bb0 0x7f7b240a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7b340058c0 tx=0x7f7b34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:30.922 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:30.925+0000 7f7b3b400640 1 -- 172.21.15.81:0/825481659 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b3c09cd10 con 0x7f7b4410a0b0 2024-04-12T21:13:31.125 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.128+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7b44062bc0 con 0x7f7b4410a0b0 2024-04-12T21:13:31.126 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.129+0000 7f7b3b400640 1 -- 172.21.15.81:0/825481659 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f7b3c09c610 con 0x7f7b4410a0b0 2024-04-12T21:13:31.126 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7b240a3bb0 msgr2=0x7f7b240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7b240a3bb0 0x7f7b240a6070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7b340058c0 tx=0x7f7b34005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b4410a0b0 msgr2=0x7f7b441a5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b4410a0b0 0x7f7b441a5da0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c00efc0 tx=0x7f7b3c00c800 comp rx=0 tx=0).stop 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 shutdown_connections 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7b240a3bb0 0x7f7b240a6070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b441096f0 0x7f7b4419ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7b4410a0b0 0x7f7b441a5da0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 --2- 172.21.15.81:0/825481659 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7b44108500 0x7f7b4419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.129 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 >> 172.21.15.81:0/825481659 conn(0x7f7b44077490 msgr2=0x7f7b4410ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:31.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 shutdown_connections 2024-04-12T21:13:31.130 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.132+0000 7f7b43e00640 1 -- 172.21.15.81:0/825481659 wait complete. 2024-04-12T21:13:31.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 -- 172.21.15.81:0/2744962749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c074a20 msgr2=0x7fed1c074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 --2- 172.21.15.81:0/2744962749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c074a20 0x7fed1c074ea0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fed04009860 tx=0x7fed0402bb70 comp rx=0 tx=0).stop 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 -- 172.21.15.81:0/2744962749 shutdown_connections 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 --2- 172.21.15.81:0/2744962749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c074a20 0x7fed1c074ea0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 --2- 172.21.15.81:0/2744962749 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed1c10e4a0 0x7fed1c110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 --2- 172.21.15.81:0/2744962749 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed1c076000 0x7fed1c074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 -- 172.21.15.81:0/2744962749 >> 172.21.15.81:0/2744962749 conn(0x7fed1c0fd460 msgr2=0x7fed1c0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 -- 172.21.15.81:0/2744962749 shutdown_connections 2024-04-12T21:13:31.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.368+0000 7fed21600640 1 -- 172.21.15.81:0/2744962749 wait complete. 2024-04-12T21:13:31.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.369+0000 7fed21600640 1 Processor -- start 2024-04-12T21:13:31.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.369+0000 7fed21600640 1 -- start start 2024-04-12T21:13:31.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.369+0000 7fed21600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed1c074a20 0x7fed1c19e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:31.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.369+0000 7fed21600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed1c076000 0x7fed1c19ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.369+0000 7fed21600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c10e4a0 0x7fed1c1a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.369+0000 7fed21600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c114120 con 0x7fed1c076000 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.369+0000 7fed21600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c113fa0 con 0x7fed1c074a20 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.369+0000 7fed21600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1142a0 con 0x7fed1c10e4a0 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed1c076000 0x7fed1c19ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed1c076000 0x7fed1c19ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38768/0 (socket says 172.21.15.81:38768) 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 -- 172.21.15.81:0/80378787 learned_addr learned my addr 172.21.15.81:0/80378787 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed20c00640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c10e4a0 0x7fed1c1a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1be00640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed1c074a20 0x7fed1c19e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 -- 172.21.15.81:0/80378787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c10e4a0 msgr2=0x7fed1c1a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c10e4a0 0x7fed1c1a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 -- 172.21.15.81:0/80378787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed1c074a20 msgr2=0x7fed1c19e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed1c074a20 0x7fed1c19e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 -- 172.21.15.81:0/80378787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c1a63d0 con 0x7fed1c076000 2024-04-12T21:13:31.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.370+0000 7fed1b400640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed1c076000 0x7fed1c19ecf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fed04005940 tx=0x7fed04034930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:31.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.371+0000 7fed18c00640 1 -- 172.21.15.81:0/80378787 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04041020 con 0x7fed1c076000 2024-04-12T21:13:31.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.371+0000 7fed18c00640 1 -- 172.21.15.81:0/80378787 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed0402c9f0 con 0x7fed1c076000 2024-04-12T21:13:31.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.371+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c1a6660 con 0x7fed1c076000 2024-04-12T21:13:31.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.371+0000 7fed18c00640 1 -- 172.21.15.81:0/80378787 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04039040 con 0x7fed1c076000 2024-04-12T21:13:31.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.371+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c1a6b50 con 0x7fed1c076000 2024-04-12T21:13:31.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.372+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed1c104670 con 0x7fed1c076000 2024-04-12T21:13:31.370 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.374+0000 7fed18c00640 1 -- 172.21.15.81:0/80378787 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fed04044020 con 0x7fed1c076000 2024-04-12T21:13:31.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.375+0000 7fed18c00640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed000a3bb0 0x7fed000a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:31.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.375+0000 7fed18c00640 1 -- 172.21.15.81:0/80378787 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fed040f1fe0 con 0x7fed1c076000 2024-04-12T21:13:31.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.376+0000 7fed1be00640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed000a3bb0 0x7fed000a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:31.372 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.376+0000 7fed1be00640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed000a3bb0 0x7fed000a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fed14005fd0 tx=0x7fed14010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:31.376 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.379+0000 7fed18c00640 1 -- 172.21.15.81:0/80378787 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed0404b020 con 0x7fed1c076000 2024-04-12T21:13:31.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.600+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fed1c02d5e0 con 0x7fed1c076000 2024-04-12T21:13:31.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.601+0000 7fed18c00640 1 -- 172.21.15.81:0/80378787 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7fed040be780 con 0x7fed1c076000 2024-04-12T21:13:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/1836548478 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6801/1636479292 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6800/3647696686 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/60786815 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:31.598 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/3894299177 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:31.599 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2659089739 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:31.599 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2550894701 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:31.599 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6800/1636479292 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:31.599 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6801/3647696686 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:31.599 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/3673746318 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:31.599 INFO:teuthology.orchestra.run.smithi081.stderr:listed 10 entries 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed000a3bb0 msgr2=0x7fed000a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed000a3bb0 0x7fed000a6070 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fed14005fd0 tx=0x7fed14010040 comp rx=0 tx=0).stop 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed1c076000 msgr2=0x7fed1c19ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed1c076000 0x7fed1c19ecf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fed04005940 tx=0x7fed04034930 comp rx=0 tx=0).stop 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 shutdown_connections 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed000a3bb0 0x7fed000a6070 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed1c10e4a0 0x7fed1c1a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed1c076000 0x7fed1c19ecf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 --2- 172.21.15.81:0/80378787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed1c074a20 0x7fed1c19e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 >> 172.21.15.81:0/80378787 conn(0x7fed1c0fd460 msgr2=0x7fed1c101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:31.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 shutdown_connections 2024-04-12T21:13:31.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.604+0000 7fed21600640 1 -- 172.21.15.81:0/80378787 wait complete. 2024-04-12T21:13:31.620 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:13:31.620 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T21:13:31.669 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:31 smithi081 ceph-mon[31453]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-12T21:13:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:31 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/403404708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:13:31.789 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:31 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/825481659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:13:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:31 smithi153 ceph-mon[36569]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-12T21:13:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:31 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/403404708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:13:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:31 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/825481659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:13:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:31 smithi104 ceph-mon[36624]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-12T21:13:31.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:31 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/403404708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:13:31.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:31 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/825481659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:13:31.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 -- 172.21.15.81:0/2188586041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c0749c0 msgr2=0x7f6d4c074e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2188586041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c0749c0 0x7f6d4c074e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009900 tx=0x7f6d3402bcc0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 -- 172.21.15.81:0/2188586041 shutdown_connections 2024-04-12T21:13:31.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2188586041 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d4c109cf0 0x7f6d4c10c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2188586041 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c0749c0 0x7f6d4c074e20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2188586041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c075fa0 0x7f6d4c0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 -- 172.21.15.81:0/2188586041 >> 172.21.15.81:0/2188586041 conn(0x7f6d4c0fd3d0 msgr2=0x7f6d4c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:31.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 -- 172.21.15.81:0/2188586041 shutdown_connections 2024-04-12T21:13:31.926 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.928+0000 7f6d4be00640 1 -- 172.21.15.81:0/2188586041 wait complete. 2024-04-12T21:13:31.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.929+0000 7f6d4be00640 1 Processor -- start 2024-04-12T21:13:31.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.929+0000 7f6d4be00640 1 -- start start 2024-04-12T21:13:31.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.929+0000 7f6d4be00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d4c0749c0 0x7f6d4c19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:31.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.929+0000 7f6d4be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c075fa0 0x7f6d4c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:31.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4be00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c109cf0 0x7f6d4c1a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:31.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4be00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c10fd40 con 0x7f6d4c075fa0 2024-04-12T21:13:31.928 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4be00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c10fbc0 con 0x7f6d4c0749c0 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4be00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4c10fec0 con 0x7f6d4c109cf0 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c075fa0 0x7f6d4c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4a000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c075fa0 0x7f6d4c19a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38792/0 (socket says 172.21.15.81:38792) 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4a000640 1 -- 172.21.15.81:0/2994754499 learned_addr learned my addr 172.21.15.81:0/2994754499 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4aa00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d4c0749c0 0x7f6d4c19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4b400640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c109cf0 0x7f6d4c1a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4a000640 1 -- 172.21.15.81:0/2994754499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c109cf0 msgr2=0x7f6d4c1a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4a000640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c109cf0 0x7f6d4c1a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4a000640 1 -- 172.21.15.81:0/2994754499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d4c0749c0 msgr2=0x7f6d4c19a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4a000640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d4c0749c0 0x7f6d4c19a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4a000640 1 -- 172.21.15.81:0/2994754499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4c1a1f60 con 0x7f6d4c075fa0 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.930+0000 7f6d4aa00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d4c0749c0 0x7f6d4c19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.931+0000 7f6d4a000640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c075fa0 0x7f6d4c19a8d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d34005ab0 tx=0x7f6d3402c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:31.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.931+0000 7f6d43400640 1 -- 172.21.15.81:0/2994754499 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d34043020 con 0x7f6d4c075fa0 2024-04-12T21:13:31.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.931+0000 7f6d43400640 1 -- 172.21.15.81:0/2994754499 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d340054e0 con 0x7f6d4c075fa0 2024-04-12T21:13:31.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.931+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4c1a21f0 con 0x7f6d4c075fa0 2024-04-12T21:13:31.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.931+0000 7f6d43400640 1 -- 172.21.15.81:0/2994754499 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d34003cc0 con 0x7f6d4c075fa0 2024-04-12T21:13:31.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.931+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d4c1a2740 con 0x7f6d4c075fa0 2024-04-12T21:13:31.930 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.932+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d4c108ab0 con 0x7f6d4c075fa0 2024-04-12T21:13:31.931 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.934+0000 7f6d43400640 1 -- 172.21.15.81:0/2994754499 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6d34046020 con 0x7f6d4c075fa0 2024-04-12T21:13:31.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.936+0000 7f6d43400640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d300a3bb0 0x7f6d300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:31.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.936+0000 7f6d43400640 1 -- 172.21.15.81:0/2994754499 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f6d340f27c0 con 0x7f6d4c075fa0 2024-04-12T21:13:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.936+0000 7f6d4aa00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d300a3bb0 0x7f6d300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:31.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.937+0000 7f6d4aa00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d300a3bb0 0x7f6d300a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c005fd0 tx=0x7f6d3c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:31.936 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:31.940+0000 7f6d43400640 1 -- 172.21.15.81:0/2994754499 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d3404d020 con 0x7f6d4c075fa0 2024-04-12T21:13:32.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.145+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d4c062bc0 con 0x7f6d4c075fa0 2024-04-12T21:13:32.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.149+0000 7f6d43400640 1 -- 172.21.15.81:0/2994754499 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7f6d340bef00 con 0x7f6d4c075fa0 2024-04-12T21:13:32.146 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:32.147 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d300a3bb0 msgr2=0x7f6d300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d300a3bb0 0x7f6d300a6070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c005fd0 tx=0x7f6d3c009500 comp rx=0 tx=0).stop 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c075fa0 msgr2=0x7f6d4c19a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c075fa0 0x7f6d4c19a8d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d34005ab0 tx=0x7f6d3402c750 comp rx=0 tx=0).stop 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 shutdown_connections 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f6d300a3bb0 0x7f6d300a6070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6d4c109cf0 0x7f6d4c1a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6d4c075fa0 0x7f6d4c19a8d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 --2- 172.21.15.81:0/2994754499 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6d4c0749c0 0x7f6d4c19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 >> 172.21.15.81:0/2994754499 conn(0x7f6d4c0fd3d0 msgr2=0x7f6d4c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:32.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 shutdown_connections 2024-04-12T21:13:32.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.152+0000 7f6d4be00640 1 -- 172.21.15.81:0/2994754499 wait complete. 2024-04-12T21:13:32.169 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-12T21:13:10.561238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-12T21:13:11.985544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-12T21:13:15.579438+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-12T21:13:32.169 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T21:13:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 -- 172.21.15.81:0/3434123087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e30075fa0 msgr2=0x7f0e300743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 --2- 172.21.15.81:0/3434123087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e30075fa0 0x7f0e300743f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0e20009900 tx=0x7f0e2002bcd0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 -- 172.21.15.81:0/3434123087 shutdown_connections 2024-04-12T21:13:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 --2- 172.21.15.81:0/3434123087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e300749c0 0x7f0e30074e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 --2- 172.21.15.81:0/3434123087 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e30109d60 0x7f0e3010c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.388 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 --2- 172.21.15.81:0/3434123087 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e30075fa0 0x7f0e300743f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 -- 172.21.15.81:0/3434123087 >> 172.21.15.81:0/3434123087 conn(0x7f0e300fd460 msgr2=0x7f0e300ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 -- 172.21.15.81:0/3434123087 shutdown_connections 2024-04-12T21:13:32.389 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.391+0000 7f0e34a00640 1 -- 172.21.15.81:0/3434123087 wait complete. 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.392+0000 7f0e34a00640 1 Processor -- start 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.392+0000 7f0e34a00640 1 -- start start 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e34a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e300749c0 0x7f0e3019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e34a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e30075fa0 0x7f0e3019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e34a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e30109d60 0x7f0e301a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e34a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3010fdb0 con 0x7f0e30109d60 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e34a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3010fc30 con 0x7f0e30075fa0 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e34a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0e3010ff30 con 0x7f0e300749c0 2024-04-12T21:13:32.391 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e27e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e30075fa0 0x7f0e3019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e27e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e30075fa0 0x7f0e3019a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:49418/0 (socket says 172.21.15.81:49418) 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e27e00640 1 -- 172.21.15.81:0/2359711904 learned_addr learned my addr 172.21.15.81:0/2359711904 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e2fe00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e30109d60 0x7f0e301a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e2f400640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e300749c0 0x7f0e3019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e27e00640 1 -- 172.21.15.81:0/2359711904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e300749c0 msgr2=0x7f0e3019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e27e00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e300749c0 0x7f0e3019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e27e00640 1 -- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e30109d60 msgr2=0x7f0e301a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e27e00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e30109d60 0x7f0e301a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.393+0000 7f0e27e00640 1 -- 172.21.15.81:0/2359711904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e301a2050 con 0x7f0e30075fa0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.394+0000 7f0e27e00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e30075fa0 0x7f0e3019a970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800e9f0 tx=0x7f0e1800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.394+0000 7f0e2cc00640 1 -- 172.21.15.81:0/2359711904 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1800cd80 con 0x7f0e30075fa0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.394+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e301a2340 con 0x7f0e30075fa0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.394+0000 7f0e2cc00640 1 -- 172.21.15.81:0/2359711904 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e18005500 con 0x7f0e30075fa0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.394+0000 7f0e2cc00640 1 -- 172.21.15.81:0/2359711904 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e18010690 con 0x7f0e30075fa0 2024-04-12T21:13:32.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.394+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e301a2890 con 0x7f0e30075fa0 2024-04-12T21:13:32.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.395+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e30108b20 con 0x7f0e30075fa0 2024-04-12T21:13:32.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.398+0000 7f0e2cc00640 1 -- 172.21.15.81:0/2359711904 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0e180046c0 con 0x7f0e30075fa0 2024-04-12T21:13:32.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.399+0000 7f0e2cc00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0e140a3bb0 0x7f0e140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:32.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.399+0000 7f0e2cc00640 1 -- 172.21.15.81:0/2359711904 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0e180d0ca0 con 0x7f0e30075fa0 2024-04-12T21:13:32.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.399+0000 7f0e2f400640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0e140a3bb0 0x7f0e140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:32.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.400+0000 7f0e2f400640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0e140a3bb0 0x7f0e140a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0e2002c2e0 tx=0x7f0e2002c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:32.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.401+0000 7f0e2cc00640 1 -- 172.21.15.81:0/2359711904 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e18053ff0 con 0x7f0e30075fa0 2024-04-12T21:13:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.605+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0e3002d7e0 con 0x7f0e30075fa0 2024-04-12T21:13:32.602 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.606+0000 7f0e2cc00640 1 -- 172.21.15.81:0/2359711904 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f0e1809d3e0 con 0x7f0e30075fa0 2024-04-12T21:13:32.603 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.608+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0e140a3bb0 msgr2=0x7f0e140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.608+0000 7f0e34a00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0e140a3bb0 0x7f0e140a6070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0e2002c2e0 tx=0x7f0e2002c920 comp rx=0 tx=0).stop 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.608+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e30075fa0 msgr2=0x7f0e3019a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.608+0000 7f0e34a00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e30075fa0 0x7f0e3019a970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800e9f0 tx=0x7f0e1800edf0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.609+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 shutdown_connections 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.609+0000 7f0e34a00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0e140a3bb0 0x7f0e140a6070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.609+0000 7f0e34a00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0e30109d60 0x7f0e301a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.609+0000 7f0e34a00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e30075fa0 0x7f0e3019a970 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.609+0000 7f0e34a00640 1 --2- 172.21.15.81:0/2359711904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e300749c0 0x7f0e3019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.609+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 >> 172.21.15.81:0/2359711904 conn(0x7f0e300fd460 msgr2=0x7f0e30105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.609+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 shutdown_connections 2024-04-12T21:13:32.606 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.609+0000 7f0e34a00640 1 -- 172.21.15.81:0/2359711904 wait complete. 2024-04-12T21:13:32.623 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-12T21:13:32.701 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:32 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/80378787' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:13:32.701 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:32 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2994754499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:32.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:32 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/80378787' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:13:32.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:32 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2994754499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:32.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:32 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/80378787' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:13:32.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:32 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2994754499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:32.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.843+0000 7f719f000640 1 -- 172.21.15.81:0/3011929966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 msgr2=0x7f7198108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.843+0000 7f719f000640 1 --2- 172.21.15.81:0/3011929966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 0x7f7198108920 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f718c009900 tx=0x7f718c02bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:32.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.844+0000 7f719f000640 1 -- 172.21.15.81:0/3011929966 shutdown_connections 2024-04-12T21:13:32.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.844+0000 7f719f000640 1 --2- 172.21.15.81:0/3011929966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f719810a0b0 0x7f7198110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.844+0000 7f719f000640 1 --2- 172.21.15.81:0/3011929966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f71981096f0 0x7f7198109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.844+0000 7f719f000640 1 --2- 172.21.15.81:0/3011929966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 0x7f7198108920 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.844+0000 7f719f000640 1 -- 172.21.15.81:0/3011929966 >> 172.21.15.81:0/3011929966 conn(0x7f7198077490 msgr2=0x7f71980798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:32.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.844+0000 7f719f000640 1 -- 172.21.15.81:0/3011929966 shutdown_connections 2024-04-12T21:13:32.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.844+0000 7f719f000640 1 -- 172.21.15.81:0/3011929966 wait complete. 2024-04-12T21:13:32.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719f000640 1 Processor -- start 2024-04-12T21:13:32.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719f000640 1 -- start start 2024-04-12T21:13:32.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719f000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 0x7f719819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719f000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71981096f0 0x7f719819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719f000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719810a0b0 0x7f71981a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719f000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f71981141e0 con 0x7f7198108500 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719f000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7198114060 con 0x7f719810a0b0 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719f000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7198114360 con 0x7f71981096f0 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.845+0000 7f719dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 0x7f719819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719dc00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 0x7f719819e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38830/0 (socket says 172.21.15.81:38830) 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719dc00640 1 -- 172.21.15.81:0/1739098755 learned_addr learned my addr 172.21.15.81:0/1739098755 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719e600640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719810a0b0 0x7f71981a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719d200640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71981096f0 0x7f719819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719dc00640 1 -- 172.21.15.81:0/1739098755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71981096f0 msgr2=0x7f719819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719dc00640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71981096f0 0x7f719819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719dc00640 1 -- 172.21.15.81:0/1739098755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719810a0b0 msgr2=0x7f71981a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719dc00640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719810a0b0 0x7f71981a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:32.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719dc00640 1 -- 172.21.15.81:0/1739098755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71981a63d0 con 0x7f7198108500 2024-04-12T21:13:32.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719e600640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719810a0b0 0x7f71981a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:32.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.846+0000 7f719dc00640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 0x7f719819e7b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f718c000c00 tx=0x7f718c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:32.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.847+0000 7f7196000640 1 -- 172.21.15.81:0/1739098755 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c043020 con 0x7f7198108500 2024-04-12T21:13:32.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.847+0000 7f7196000640 1 -- 172.21.15.81:0/1739098755 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f718c0054e0 con 0x7f7198108500 2024-04-12T21:13:32.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.847+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71981a6660 con 0x7f7198108500 2024-04-12T21:13:32.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.847+0000 7f7196000640 1 -- 172.21.15.81:0/1739098755 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718c003c40 con 0x7f7198108500 2024-04-12T21:13:32.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.847+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71981a6bb0 con 0x7f7198108500 2024-04-12T21:13:32.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.848+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719810f7c0 con 0x7f7198108500 2024-04-12T21:13:32.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.850+0000 7f7196000640 1 -- 172.21.15.81:0/1739098755 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f718c046020 con 0x7f7198108500 2024-04-12T21:13:32.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.851+0000 7f7196000640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f71800a3bb0 0x7f71800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:32.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.851+0000 7f7196000640 1 -- 172.21.15.81:0/1739098755 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f718c0f2640 con 0x7f7198108500 2024-04-12T21:13:32.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.851+0000 7f719d200640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f71800a3bb0 0x7f71800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:32.848 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.852+0000 7f719d200640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f71800a3bb0 0x7f71800a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f719819fcc0 tx=0x7f7184005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:32.852 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:32.856+0000 7f7196000640 1 -- 172.21.15.81:0/1739098755 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f718c04d020 con 0x7f7198108500 2024-04-12T21:13:33.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.059+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f719802d800 con 0x7f7198108500 2024-04-12T21:13:33.057 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.060+0000 7f7196000640 1 -- 172.21.15.81:0/1739098755 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f718c0bed80 con 0x7f7198108500 2024-04-12T21:13:33.057 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:13:33.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.062+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f71800a3bb0 msgr2=0x7f71800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.062+0000 7f719f000640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f71800a3bb0 0x7f71800a6070 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f719819fcc0 tx=0x7f7184005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:33.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.062+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 msgr2=0x7f719819e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.062+0000 7f719f000640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 0x7f719819e7b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f718c000c00 tx=0x7f718c02c650 comp rx=0 tx=0).stop 2024-04-12T21:13:33.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.063+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 shutdown_connections 2024-04-12T21:13:33.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.063+0000 7f719f000640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f71800a3bb0 0x7f71800a6070 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.063+0000 7f719f000640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71981096f0 0x7f719819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.063+0000 7f719f000640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f719810a0b0 0x7f71981a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.063+0000 7f719f000640 1 --2- 172.21.15.81:0/1739098755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7198108500 0x7f719819e7b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.063+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 >> 172.21.15.81:0/1739098755 conn(0x7f7198077490 msgr2=0x7f719810c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:33.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.063+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 shutdown_connections 2024-04-12T21:13:33.060 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.063+0000 7f719f000640 1 -- 172.21.15.81:0/1739098755 wait complete. 2024-04-12T21:13:33.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 -- 172.21.15.81:0/2396518568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 msgr2=0x7fa5f81034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 --2- 172.21.15.81:0/2396518568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 0x7fa5f81034e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009900 tx=0x7fa5e402bc40 comp rx=0 tx=0).stop 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 -- 172.21.15.81:0/2396518568 shutdown_connections 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 --2- 172.21.15.81:0/2396518568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5f8103a20 0x7fa5f810a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 --2- 172.21.15.81:0/2396518568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 0x7fa5f81034e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 --2- 172.21.15.81:0/2396518568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8101e80 0x7fa5f81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 -- 172.21.15.81:0/2396518568 >> 172.21.15.81:0/2396518568 conn(0x7fa5f80fd460 msgr2=0x7fa5f80ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 -- 172.21.15.81:0/2396518568 shutdown_connections 2024-04-12T21:13:33.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.295+0000 7fa5f7200640 1 -- 172.21.15.81:0/2396518568 wait complete. 2024-04-12T21:13:33.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.296+0000 7fa5f7200640 1 Processor -- start 2024-04-12T21:13:33.293 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f7200640 1 -- start start 2024-04-12T21:13:33.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f7200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5f8101e80 0x7fa5f819a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f7200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 0x7fa5f819a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f7200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8103a20 0x7fa5f81a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f7200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f810db50 con 0x7fa5f8103060 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f7200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f810d9d0 con 0x7fa5f8101e80 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f7200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f810dcd0 con 0x7fa5f8103a20 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 0x7fa5f819a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f5400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 0x7fa5f819a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38838/0 (socket says 172.21.15.81:38838) 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.297+0000 7fa5f5400640 1 -- 172.21.15.81:0/973247493 learned_addr learned my addr 172.21.15.81:0/973247493 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5f6800640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8103a20 0x7fa5f81a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5f5e00640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5f8101e80 0x7fa5f819a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5f5400640 1 -- 172.21.15.81:0/973247493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8103a20 msgr2=0x7fa5f81a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5f5400640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8103a20 0x7fa5f81a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5f5400640 1 -- 172.21.15.81:0/973247493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5f8101e80 msgr2=0x7fa5f819a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5f5400640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5f8101e80 0x7fa5f819a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5f5400640 1 -- 172.21.15.81:0/973247493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f81a2000 con 0x7fa5f8103060 2024-04-12T21:13:33.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5f5400640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 0x7fa5f819a920 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009230 tx=0x7fa5e402c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:33.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.298+0000 7fa5eea00640 1 -- 172.21.15.81:0/973247493 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4041070 con 0x7fa5f8103060 2024-04-12T21:13:33.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.299+0000 7fa5eea00640 1 -- 172.21.15.81:0/973247493 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5e4004450 con 0x7fa5f8103060 2024-04-12T21:13:33.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.299+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f81a2290 con 0x7fa5f8103060 2024-04-12T21:13:33.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.299+0000 7fa5eea00640 1 -- 172.21.15.81:0/973247493 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4003e00 con 0x7fa5f8103060 2024-04-12T21:13:33.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.299+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f81a2780 con 0x7fa5f8103060 2024-04-12T21:13:33.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.300+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f81a2aa0 con 0x7fa5f8103060 2024-04-12T21:13:33.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.302+0000 7fa5eea00640 1 -- 172.21.15.81:0/973247493 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa5e4044070 con 0x7fa5f8103060 2024-04-12T21:13:33.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.303+0000 7fa5eea00640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa5dc0a3bb0 0x7fa5dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:33.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.303+0000 7fa5eea00640 1 -- 172.21.15.81:0/973247493 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa5e40f25e0 con 0x7fa5f8103060 2024-04-12T21:13:33.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.304+0000 7fa5f5e00640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa5dc0a3bb0 0x7fa5dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:33.301 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.304+0000 7fa5f5e00640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa5dc0a3bb0 0x7fa5dc0a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8102bd0 tx=0x7fa5e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:33.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.306+0000 7fa5eea00640 1 -- 172.21.15.81:0/973247493 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5e40be6d0 con 0x7fa5f8103060 2024-04-12T21:13:33.508 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.511+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa5f81a2c70 con 0x7fa5f8103060 2024-04-12T21:13:33.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.512+0000 7fa5eea00640 1 -- 172.21.15.81:0/973247493 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fa5e40c29d0 con 0x7fa5f8103060 2024-04-12T21:13:33.509 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa5dc0a3bb0 msgr2=0x7fa5dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa5dc0a3bb0 0x7fa5dc0a6070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8102bd0 tx=0x7fa5e8005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 msgr2=0x7fa5f819a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 0x7fa5f819a920 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009230 tx=0x7fa5e402c660 comp rx=0 tx=0).stop 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 shutdown_connections 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa5dc0a3bb0 0x7fa5dc0a6070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa5f8103a20 0x7fa5f81a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa5f8103060 0x7fa5f819a920 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 --2- 172.21.15.81:0/973247493 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa5f8101e80 0x7fa5f819a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 >> 172.21.15.81:0/973247493 conn(0x7fa5f80fd460 msgr2=0x7fa5f80741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 shutdown_connections 2024-04-12T21:13:33.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.515+0000 7fa5f7200640 1 -- 172.21.15.81:0/973247493 wait complete. 2024-04-12T21:13:33.530 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-12T21:13:33.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 -- 172.21.15.81:0/2618311494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c103a20 msgr2=0x7f658c10a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 --2- 172.21.15.81:0/2618311494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c103a20 0x7f658c10a340 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6584009ca0 tx=0x7f658402c100 comp rx=0 tx=0).stop 2024-04-12T21:13:33.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 -- 172.21.15.81:0/2618311494 shutdown_connections 2024-04-12T21:13:33.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 --2- 172.21.15.81:0/2618311494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f658c103060 0x7f658c1034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 --2- 172.21.15.81:0/2618311494 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c103a20 0x7f658c10a340 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 --2- 172.21.15.81:0/2618311494 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f658c101e80 0x7f658c1022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 -- 172.21.15.81:0/2618311494 >> 172.21.15.81:0/2618311494 conn(0x7f658c0fd460 msgr2=0x7f658c0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:33.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 -- 172.21.15.81:0/2618311494 shutdown_connections 2024-04-12T21:13:33.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.751+0000 7f6592000640 1 -- 172.21.15.81:0/2618311494 wait complete. 2024-04-12T21:13:33.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.752+0000 7f6592000640 1 Processor -- start 2024-04-12T21:13:33.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6592000640 1 -- start start 2024-04-12T21:13:33.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6592000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c101e80 0x7f658c075d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:33.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6592000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f658c103060 0x7f658c0743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:33.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6592000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f658c103a20 0x7f658c0748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:33.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6592000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f658c10db80 con 0x7f658c101e80 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6592000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f658c10da00 con 0x7f658c103060 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6592000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f658c10dd00 con 0x7f658c103a20 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6590c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c101e80 0x7f658c075d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6590c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c101e80 0x7f658c075d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38854/0 (socket says 172.21.15.81:38854) 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6590c00640 1 -- 172.21.15.81:0/1036096172 learned_addr learned my addr 172.21.15.81:0/1036096172 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.753+0000 7f6591600640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f658c103a20 0x7f658c0748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f658be00640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f658c103060 0x7f658c0743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f6590c00640 1 -- 172.21.15.81:0/1036096172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f658c103a20 msgr2=0x7f658c0748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f6590c00640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f658c103a20 0x7f658c0748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f6590c00640 1 -- 172.21.15.81:0/1036096172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f658c103060 msgr2=0x7f658c0743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f6590c00640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f658c103060 0x7f658c0743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f6590c00640 1 -- 172.21.15.81:0/1036096172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f658c075180 con 0x7f658c101e80 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f6591600640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f658c103a20 0x7f658c0748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f658be00640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f658c103060 0x7f658c0743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:33.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f6590c00640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c101e80 0x7f658c075d70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f658000e9c0 tx=0x7f658000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:33.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.754+0000 7f6589600640 1 -- 172.21.15.81:0/1036096172 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658000cde0 con 0x7f658c101e80 2024-04-12T21:13:33.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.755+0000 7f6589600640 1 -- 172.21.15.81:0/1036096172 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6580010910 con 0x7f658c101e80 2024-04-12T21:13:33.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.755+0000 7f6589600640 1 -- 172.21.15.81:0/1036096172 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65800189c0 con 0x7f658c101e80 2024-04-12T21:13:33.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.755+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f658c1a2310 con 0x7f658c101e80 2024-04-12T21:13:33.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.755+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f658c1a2810 con 0x7f658c101e80 2024-04-12T21:13:33.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.756+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f658c109150 con 0x7f658c101e80 2024-04-12T21:13:33.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.759+0000 7f6589600640 1 -- 172.21.15.81:0/1036096172 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f6580010ac0 con 0x7f658c101e80 2024-04-12T21:13:33.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.760+0000 7f6589600640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f65740a3c30 0x7f65740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:33.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.760+0000 7f6589600640 1 -- 172.21.15.81:0/1036096172 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f65800d0b50 con 0x7f658c101e80 2024-04-12T21:13:33.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.760+0000 7f658be00640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f65740a3c30 0x7f65740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:33.757 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.761+0000 7f658be00640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f65740a3c30 0x7f65740a60f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f658c075ae0 tx=0x7f6578005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:33.761 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.764+0000 7f6589600640 1 -- 172.21.15.81:0/1036096172 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6580053ff0 con 0x7f658c101e80 2024-04-12T21:13:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:33 smithi153 ceph-mon[36569]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-12T21:13:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:33 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2359711904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:33 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1739098755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:13:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:33 smithi104 ceph-mon[36624]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-12T21:13:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:33 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2359711904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:33 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1739098755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:13:33.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:33 smithi081 ceph-mon[31453]: pgmap v324: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-12T21:13:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:33 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2359711904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:13:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:33 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1739098755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:13:33.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.966+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f658c062bc0 con 0x7f658c101e80 2024-04-12T21:13:33.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.967+0000 7f6589600640 1 -- 172.21.15.81:0/1036096172 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f658009d290 con 0x7f658c101e80 2024-04-12T21:13:33.964 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:13:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f65740a3c30 msgr2=0x7f65740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f65740a3c30 0x7f65740a60f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f658c075ae0 tx=0x7f6578005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c101e80 msgr2=0x7f658c075d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c101e80 0x7f658c075d70 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f658000e9c0 tx=0x7f658000ee80 comp rx=0 tx=0).stop 2024-04-12T21:13:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 shutdown_connections 2024-04-12T21:13:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f65740a3c30 0x7f65740a60f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f658c103a20 0x7f658c0748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f658c103060 0x7f658c0743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 --2- 172.21.15.81:0/1036096172 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f658c101e80 0x7f658c075d70 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:33.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 >> 172.21.15.81:0/1036096172 conn(0x7f658c0fd460 msgr2=0x7f658c106320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:33.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 shutdown_connections 2024-04-12T21:13:33.968 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:33.970+0000 7f6592000640 1 -- 172.21.15.81:0/1036096172 wait complete. 2024-04-12T21:13:33.988 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 -- 172.21.15.81:0/1988636719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 msgr2=0x7fb49410c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 --2- 172.21.15.81:0/1988636719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 0x7fb49410c220 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb484009900 tx=0x7fb48402bcc0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 -- 172.21.15.81:0/1988636719 shutdown_connections 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 --2- 172.21.15.81:0/1988636719 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 0x7fb49410c220 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 --2- 172.21.15.81:0/1988636719 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4940749c0 0x7fb494074e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 --2- 172.21.15.81:0/1988636719 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb494075fa0 0x7fb4940743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 -- 172.21.15.81:0/1988636719 >> 172.21.15.81:0/1988636719 conn(0x7fb4940fd460 msgr2=0x7fb4940ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 -- 172.21.15.81:0/1988636719 shutdown_connections 2024-04-12T21:13:34.198 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.201+0000 7fb493200640 1 -- 172.21.15.81:0/1988636719 wait complete. 2024-04-12T21:13:34.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.202+0000 7fb493200640 1 Processor -- start 2024-04-12T21:13:34.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.202+0000 7fb493200640 1 -- start start 2024-04-12T21:13:34.200 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.202+0000 7fb493200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4940749c0 0x7fb49419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.202+0000 7fb493200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb494075fa0 0x7fb49419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.202+0000 7fb493200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 0x7fb4941a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.202+0000 7fb493200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb49410fd80 con 0x7fb494109d80 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.202+0000 7fb493200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb49410fc00 con 0x7fb494075fa0 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.202+0000 7fb493200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb49410ff00 con 0x7fb4940749c0 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 0x7fb4941a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 0x7fb4941a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38868/0 (socket says 172.21.15.81:38868) 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 -- 172.21.15.81:0/2647451207 learned_addr learned my addr 172.21.15.81:0/2647451207 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb491400640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb494075fa0 0x7fb49419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb491e00640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4940749c0 0x7fb49419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 -- 172.21.15.81:0/2647451207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4940749c0 msgr2=0x7fb49419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4940749c0 0x7fb49419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 -- 172.21.15.81:0/2647451207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb494075fa0 msgr2=0x7fb49419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb494075fa0 0x7fb49419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 -- 172.21.15.81:0/2647451207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4941a1f70 con 0x7fb494109d80 2024-04-12T21:13:34.201 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb491400640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb494075fa0 0x7fb49419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.203+0000 7fb492800640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 0x7fb4941a1900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb484005ab0 tx=0x7fb48402c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.204+0000 7fb48aa00640 1 -- 172.21.15.81:0/2647451207 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484043020 con 0x7fb494109d80 2024-04-12T21:13:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.204+0000 7fb48aa00640 1 -- 172.21.15.81:0/2647451207 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4840054e0 con 0x7fb494109d80 2024-04-12T21:13:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.204+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4941a2200 con 0x7fb494109d80 2024-04-12T21:13:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.204+0000 7fb48aa00640 1 -- 172.21.15.81:0/2647451207 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb484003c40 con 0x7fb494109d80 2024-04-12T21:13:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.204+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4941a27a0 con 0x7fb494109d80 2024-04-12T21:13:34.202 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.205+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb494108b40 con 0x7fb494109d80 2024-04-12T21:13:34.204 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.207+0000 7fb48aa00640 1 -- 172.21.15.81:0/2647451207 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb484046020 con 0x7fb494109d80 2024-04-12T21:13:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.208+0000 7fb48aa00640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4780a3c30 0x7fb4780a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.209+0000 7fb48aa00640 1 -- 172.21.15.81:0/2647451207 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb4840f26f0 con 0x7fb494109d80 2024-04-12T21:13:34.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.209+0000 7fb491e00640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4780a3c30 0x7fb4780a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:34.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.209+0000 7fb491e00640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4780a3c30 0x7fb4780a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0099a0 tx=0x7fb48c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:34.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.212+0000 7fb48aa00640 1 -- 172.21.15.81:0/2647451207 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb48404d020 con 0x7fb494109d80 2024-04-12T21:13:34.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.428+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb494062bc0 con 0x7fb494109d80 2024-04-12T21:13:34.425 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.429+0000 7fb48aa00640 1 -- 172.21.15.81:0/2647451207 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7fb4840bee30 con 0x7fb494109d80 2024-04-12T21:13:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/1836548478 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6801/1636479292 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6800/3647696686 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/60786815 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/3894299177 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2659089739 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/2550894701 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:34.426 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6800/1636479292 2024-04-13T21:05:22.179703+0000 2024-04-12T21:13:34.427 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:6801/3647696686 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:34.427 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81:0/3673746318 2024-04-13T21:04:48.054528+0000 2024-04-12T21:13:34.427 INFO:teuthology.orchestra.run.smithi081.stderr:listed 10 entries 2024-04-12T21:13:34.428 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.431+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4780a3c30 msgr2=0x7fb4780a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.431+0000 7fb493200640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4780a3c30 0x7fb4780a60f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0099a0 tx=0x7fb48c009500 comp rx=0 tx=0).stop 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.431+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 msgr2=0x7fb4941a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.431+0000 7fb493200640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 0x7fb4941a1900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb484005ab0 tx=0x7fb48402c6e0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.432+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 shutdown_connections 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.432+0000 7fb493200640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4780a3c30 0x7fb4780a60f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.432+0000 7fb493200640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb494109d80 0x7fb4941a1900 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.432+0000 7fb493200640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb494075fa0 0x7fb49419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.432+0000 7fb493200640 1 --2- 172.21.15.81:0/2647451207 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb4940749c0 0x7fb49419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.432+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 >> 172.21.15.81:0/2647451207 conn(0x7fb4940fd460 msgr2=0x7fb494105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.432+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 shutdown_connections 2024-04-12T21:13:34.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.432+0000 7fb493200640 1 -- 172.21.15.81:0/2647451207 wait complete. 2024-04-12T21:13:34.448 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi081.front.sepia.ceph.com... 2024-04-12T21:13:34.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:34.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-12T21:13:34.528 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi081.stderr:ceph-fuse[75262]: fuse finished with error 0 and tester_r 0 2024-04-12T21:13:34.533 INFO:teuthology.orchestra.run:waiting for 300 2024-04-12T21:13:34.571 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi081.front.sepia.ceph.com 2024-04-12T21:13:34.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:13:34.571 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-12T21:13:34.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:34.757 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:34 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/973247493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:13:34.757 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:34 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1036096172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:13:34.757 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:34 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2647451207' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:13:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:34 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/973247493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:13:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:34 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1036096172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:13:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:34 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2647451207' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:13:34.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:34 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/973247493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:13:34.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:34 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1036096172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:13:34.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:34 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2647451207' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-12T21:13:34.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 -- 172.21.15.81:0/55698673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8101e30 msgr2=0x7f3be8102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:34.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 --2- 172.21.15.81:0/55698673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8101e30 0x7f3be8102250 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8009900 tx=0x7f3bd802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:34.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 -- 172.21.15.81:0/55698673 shutdown_connections 2024-04-12T21:13:34.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 --2- 172.21.15.81:0/55698673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3be8103940 0x7f3be810a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 --2- 172.21.15.81:0/55698673 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3be8102f80 0x7f3be8103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 --2- 172.21.15.81:0/55698673 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8101e30 0x7f3be8102250 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 -- 172.21.15.81:0/55698673 >> 172.21.15.81:0/55698673 conn(0x7f3be80fd3a0 msgr2=0x7f3be80ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:34.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 -- 172.21.15.81:0/55698673 shutdown_connections 2024-04-12T21:13:34.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.894+0000 7f3bece00640 1 -- 172.21.15.81:0/55698673 wait complete. 2024-04-12T21:13:34.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.895+0000 7f3bece00640 1 Processor -- start 2024-04-12T21:13:34.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.895+0000 7f3bece00640 1 -- start start 2024-04-12T21:13:34.892 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3bece00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3be8101e30 0x7f3be819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:34.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3bece00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8102f80 0x7f3be819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:34.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3bece00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3be8103940 0x7f3be81a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:34.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3bece00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3be810dab0 con 0x7f3be8102f80 2024-04-12T21:13:34.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3bece00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3be810d930 con 0x7f3be8103940 2024-04-12T21:13:34.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3bece00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3be810dc30 con 0x7f3be8101e30 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3be7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3be8101e30 0x7f3be819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3be6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8102f80 0x7f3be819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3be7e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3be8103940 0x7f3be81a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3be7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3be8101e30 0x7f3be819a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:52070/0 (socket says 172.21.15.81:52070) 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.896+0000 7f3be7400640 1 -- 172.21.15.81:0/4155663490 learned_addr learned my addr 172.21.15.81:0/4155663490 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3be7e00640 1 -- 172.21.15.81:0/4155663490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3be8101e30 msgr2=0x7f3be819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3be7e00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3be8101e30 0x7f3be819a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3be7e00640 1 -- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8102f80 msgr2=0x7f3be819a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:34.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3be7e00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8102f80 0x7f3be819a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3be7e00640 1 -- 172.21.15.81:0/4155663490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be81a1fb0 con 0x7f3be8103940 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3be6a00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8102f80 0x7f3be819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3be7e00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3be8103940 0x7f3be81a1940 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3be000ec50 tx=0x7f3be000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3bdfe00640 1 -- 172.21.15.81:0/4155663490 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be0003b70 con 0x7f3be8103940 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be81a22a0 con 0x7f3be8103940 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3bdfe00640 1 -- 172.21.15.81:0/4155663490 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3be0003d10 con 0x7f3be8103940 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3bdfe00640 1 -- 172.21.15.81:0/4155663490 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be0010890 con 0x7f3be8103940 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.897+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be81a27f0 con 0x7f3be8103940 2024-04-12T21:13:34.895 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.898+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be81a2ab0 con 0x7f3be8103940 2024-04-12T21:13:34.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.901+0000 7f3bdfe00640 1 -- 172.21.15.81:0/4155663490 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3be0004b40 con 0x7f3be8103940 2024-04-12T21:13:34.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.902+0000 7f3bdfe00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3bcc0a3bb0 0x7f3bcc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:34.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.902+0000 7f3bdfe00640 1 -- 172.21.15.81:0/4155663490 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3be00d0df0 con 0x7f3be8103940 2024-04-12T21:13:34.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.902+0000 7f3be7400640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3bcc0a3bb0 0x7f3bcc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:34.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.903+0000 7f3be7400640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3bcc0a3bb0 0x7f3bcc0a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8000c00 tx=0x7f3bd802c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:34.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:34.904+0000 7f3bdfe00640 1 -- 172.21.15.81:0/4155663490 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3be009d530 con 0x7f3be8103940 2024-04-12T21:13:35.103 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.107+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3be81a2c80 con 0x7f3be8103940 2024-04-12T21:13:35.105 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.110+0000 7f3bdfe00640 1 -- 172.21.15.81:0/4155663490 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7f3be009ce30 con 0x7f3be8103940 2024-04-12T21:13:35.106 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:35.107 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:35.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.111+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3bcc0a3bb0 msgr2=0x7f3bcc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.108 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.111+0000 7f3bece00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3bcc0a3bb0 0x7f3bcc0a6070 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8000c00 tx=0x7f3bd802c880 comp rx=0 tx=0).stop 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.111+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3be8103940 msgr2=0x7f3be81a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.111+0000 7f3bece00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3be8103940 0x7f3be81a1940 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3be000ec50 tx=0x7f3be000c6a0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.113+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 shutdown_connections 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.113+0000 7f3bece00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3bcc0a3bb0 0x7f3bcc0a6070 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.113+0000 7f3bece00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3be8103940 0x7f3be81a1940 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.113+0000 7f3bece00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3be8102f80 0x7f3be819a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.113+0000 7f3bece00640 1 --2- 172.21.15.81:0/4155663490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3be8101e30 0x7f3be819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.113+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 >> 172.21.15.81:0/4155663490 conn(0x7f3be80fd3a0 msgr2=0x7f3be80741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.113+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 shutdown_connections 2024-04-12T21:13:35.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.113+0000 7f3bece00640 1 -- 172.21.15.81:0/4155663490 wait complete. 2024-04-12T21:13:35.127 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:35.325 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.329+0000 7f5b43600640 1 -- 172.21.15.81:0/391952854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b3c10a1d0 msgr2=0x7f5b3c110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.329+0000 7f5b43600640 1 --2- 172.21.15.81:0/391952854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b3c10a1d0 0x7f5b3c110a60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5b34009ca0 tx=0x7f5b3402c100 comp rx=0 tx=0).stop 2024-04-12T21:13:35.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.330+0000 7f5b43600640 1 -- 172.21.15.81:0/391952854 shutdown_connections 2024-04-12T21:13:35.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.330+0000 7f5b43600640 1 --2- 172.21.15.81:0/391952854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b3c10a1d0 0x7f5b3c110a60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.330+0000 7f5b43600640 1 --2- 172.21.15.81:0/391952854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b3c109810 0x7f5b3c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.330+0000 7f5b43600640 1 --2- 172.21.15.81:0/391952854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b3c108620 0x7f5b3c108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.330+0000 7f5b43600640 1 -- 172.21.15.81:0/391952854 >> 172.21.15.81:0/391952854 conn(0x7f5b3c0775e0 msgr2=0x7f5b3c079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:35.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.330+0000 7f5b43600640 1 -- 172.21.15.81:0/391952854 shutdown_connections 2024-04-12T21:13:35.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.330+0000 7f5b43600640 1 -- 172.21.15.81:0/391952854 wait complete. 2024-04-12T21:13:35.328 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.331+0000 7f5b43600640 1 Processor -- start 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.331+0000 7f5b43600640 1 -- start start 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.331+0000 7f5b43600640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b3c108620 0x7f5b3c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.331+0000 7f5b43600640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b3c109810 0x7f5b3c19ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.331+0000 7f5b43600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b3c10a1d0 0x7f5b3c1a5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.331+0000 7f5b43600640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c114280 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.331+0000 7f5b43600640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c114100 con 0x7f5b3c109810 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.331+0000 7f5b43600640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b3c114400 con 0x7f5b3c108620 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b3c10a1d0 0x7f5b3c1a5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b40c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b3c108620 0x7f5b3c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b3c10a1d0 0x7f5b3c1a5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46344/0 (socket says 172.21.15.81:46344) 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 -- 172.21.15.81:0/3670186200 learned_addr learned my addr 172.21.15.81:0/3670186200 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:35.329 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b3be00640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b3c109810 0x7f5b3c19ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 -- 172.21.15.81:0/3670186200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b3c108620 msgr2=0x7f5b3c19e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b3c108620 0x7f5b3c19e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 -- 172.21.15.81:0/3670186200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b3c109810 msgr2=0x7f5b3c19ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b3c109810 0x7f5b3c19ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 -- 172.21.15.81:0/3670186200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3c1a64c0 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.332+0000 7f5b41600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b3c10a1d0 0x7f5b3c1a5e50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5b3402c640 tx=0x7f5b34035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.333+0000 7f5b39600640 1 -- 172.21.15.81:0/3670186200 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34039040 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.333+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b3c1a6750 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.333+0000 7f5b39600640 1 -- 172.21.15.81:0/3670186200 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b34044070 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.333+0000 7f5b39600640 1 -- 172.21.15.81:0/3670186200 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34004360 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.333+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b3c1a6c40 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.334+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b3c10f8e0 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.336+0000 7f5b39600640 1 -- 172.21.15.81:0/3670186200 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5b34004500 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.337+0000 7f5b39600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5b240a3bb0 0x7f5b240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:35.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.337+0000 7f5b39600640 1 -- 172.21.15.81:0/3670186200 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5b340f2c60 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.337+0000 7f5b40c00640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5b240a3bb0 0x7f5b240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:35.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.338+0000 7f5b40c00640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5b240a3bb0 0x7f5b240a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c109670 tx=0x7f5b30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:35.337 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.341+0000 7f5b39600640 1 -- 172.21.15.81:0/3670186200 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b340bf3a0 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.544+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5b3c062bc0 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.546+0000 7f5b39600640 1 -- 172.21.15.81:0/3670186200 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7f5b340098b0 con 0x7f5b3c10a1d0 2024-04-12T21:13:35.542 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:35.542 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:35.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5b240a3bb0 msgr2=0x7f5b240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5b240a3bb0 0x7f5b240a6070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c109670 tx=0x7f5b30005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:35.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b3c10a1d0 msgr2=0x7f5b3c1a5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b3c10a1d0 0x7f5b3c1a5e50 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5b3402c640 tx=0x7f5b34035d10 comp rx=0 tx=0).stop 2024-04-12T21:13:35.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 shutdown_connections 2024-04-12T21:13:35.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5b240a3bb0 0x7f5b240a6070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b3c10a1d0 0x7f5b3c1a5e50 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b3c109810 0x7f5b3c19ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 --2- 172.21.15.81:0/3670186200 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b3c108620 0x7f5b3c19e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 >> 172.21.15.81:0/3670186200 conn(0x7f5b3c0775e0 msgr2=0x7f5b3c10cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:35.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 shutdown_connections 2024-04-12T21:13:35.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.549+0000 7f5b43600640 1 -- 172.21.15.81:0/3670186200 wait complete. 2024-04-12T21:13:35.563 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-12T21:13:35.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 -- 172.21.15.81:0/1135285814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc1053b0 msgr2=0x7feafc1077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 --2- 172.21.15.81:0/1135285814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc1053b0 0x7feafc1077a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feae4009ca0 tx=0x7feae402c0f0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 -- 172.21.15.81:0/1135285814 shutdown_connections 2024-04-12T21:13:35.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 --2- 172.21.15.81:0/1135285814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc1053b0 0x7feafc1077a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 --2- 172.21.15.81:0/1135285814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feafc107ce0 0x7feafc10a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 --2- 172.21.15.81:0/1135285814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feafc102a80 0x7feafc104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 -- 172.21.15.81:0/1135285814 >> 172.21.15.81:0/1135285814 conn(0x7feafc0fc470 msgr2=0x7feafc0fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:35.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 -- 172.21.15.81:0/1135285814 shutdown_connections 2024-04-12T21:13:35.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.778+0000 7feb01e00640 1 -- 172.21.15.81:0/1135285814 wait complete. 2024-04-12T21:13:35.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.779+0000 7feb01e00640 1 Processor -- start 2024-04-12T21:13:35.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feb01e00640 1 -- start start 2024-04-12T21:13:35.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feb01e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc102a80 0x7feafc1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:35.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feb01e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feafc1053b0 0x7feafc1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:35.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feb01e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feafc107ce0 0x7feafc19f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:35.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feb01e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10dc90 con 0x7feafc1053b0 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feb01e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10db10 con 0x7feafc107ce0 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feb01e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feafc10de10 con 0x7feafc102a80 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feafa000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feafc1053b0 0x7feafc1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feafaa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc102a80 0x7feafc1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feafa000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feafc1053b0 0x7feafc1988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46360/0 (socket says 172.21.15.81:46360) 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feafa000640 1 -- 172.21.15.81:0/3157516966 learned_addr learned my addr 172.21.15.81:0/3157516966 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.780+0000 7feafb400640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feafc107ce0 0x7feafc19f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feafa000640 1 -- 172.21.15.81:0/3157516966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc102a80 msgr2=0x7feafc1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feafa000640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc102a80 0x7feafc1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feafa000640 1 -- 172.21.15.81:0/3157516966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feafc107ce0 msgr2=0x7feafc19f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feafa000640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feafc107ce0 0x7feafc19f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feafa000640 1 -- 172.21.15.81:0/3157516966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feafc19ffc0 con 0x7feafc1053b0 2024-04-12T21:13:35.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feafaa00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc102a80 0x7feafc1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:35.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feafa000640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feafc1053b0 0x7feafc1988e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feae40324c0 tx=0x7feae4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:35.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feb00a00640 1 -- 172.21.15.81:0/3157516966 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4041020 con 0x7feafc1053b0 2024-04-12T21:13:35.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feafc1a0250 con 0x7feafc1053b0 2024-04-12T21:13:35.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feb00a00640 1 -- 172.21.15.81:0/3157516966 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feae4034cd0 con 0x7feafc1053b0 2024-04-12T21:13:35.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feb00a00640 1 -- 172.21.15.81:0/3157516966 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4039040 con 0x7feafc1053b0 2024-04-12T21:13:35.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.781+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feafc1a0740 con 0x7feafc1053b0 2024-04-12T21:13:35.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.783+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feafc0fdf60 con 0x7feafc1053b0 2024-04-12T21:13:35.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.785+0000 7feb00a00640 1 -- 172.21.15.81:0/3157516966 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7feae4044020 con 0x7feafc1053b0 2024-04-12T21:13:35.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.786+0000 7feb00a00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feadc0a3bb0 0x7feadc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:35.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.786+0000 7feb00a00640 1 -- 172.21.15.81:0/3157516966 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7feae40f24f0 con 0x7feafc1053b0 2024-04-12T21:13:35.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.786+0000 7feafaa00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feadc0a3bb0 0x7feadc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:35.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.787+0000 7feafaa00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feadc0a3bb0 0x7feadc0a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feaec009fc0 tx=0x7feaec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:35.785 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:35 smithi081 ceph-mon[31453]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-12T21:13:35.785 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:35 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4155663490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:35.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:35.791+0000 7feb00a00640 1 -- 172.21.15.81:0/3157516966 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feae404b020 con 0x7feafc1053b0 2024-04-12T21:13:35.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:35 smithi153 ceph-mon[36569]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-12T21:13:35.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:35 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4155663490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:35 smithi104 ceph-mon[36624]: pgmap v325: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-12T21:13:35.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:35 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4155663490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:36.026 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.030+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7feafc062bc0 con 0x7feafc1053b0 2024-04-12T21:13:36.465 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.469+0000 7feb00a00640 1 -- 172.21.15.81:0/3157516966 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7feae40becc0 con 0x7feafc1053b0 2024-04-12T21:13:36.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.472+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feadc0a3bb0 msgr2=0x7feadc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:36.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.472+0000 7feb01e00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feadc0a3bb0 0x7feadc0a6070 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feaec009fc0 tx=0x7feaec005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:36.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.472+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feafc1053b0 msgr2=0x7feafc1988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.472+0000 7feb01e00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feafc1053b0 0x7feafc1988e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feae40324c0 tx=0x7feae4032f40 comp rx=0 tx=0).stop 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.472+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 shutdown_connections 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.473+0000 7feb01e00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7feadc0a3bb0 0x7feadc0a6070 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.473+0000 7feb01e00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feafc107ce0 0x7feafc19f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.473+0000 7feb01e00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feafc1053b0 0x7feafc1988e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.473+0000 7feb01e00640 1 --2- 172.21.15.81:0/3157516966 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feafc102a80 0x7feafc1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.473+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 >> 172.21.15.81:0/3157516966 conn(0x7feafc0fc470 msgr2=0x7feafc1048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.473+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 shutdown_connections 2024-04-12T21:13:36.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.473+0000 7feb01e00640 1 -- 172.21.15.81:0/3157516966 wait complete. 2024-04-12T21:13:36.489 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-12T21:13:36.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-12T21:13:36.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:36 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3670186200' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:36.704 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:36 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3157516966' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-12T21:13:36.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.712+0000 7f93f2800640 1 -- 172.21.15.81:0/2649452166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 msgr2=0x7f93ec10c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:36.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.712+0000 7f93f2800640 1 --2- 172.21.15.81:0/2649452166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 0x7f93ec10c220 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009ca0 tx=0x7f93e402c100 comp rx=0 tx=0).stop 2024-04-12T21:13:36.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.712+0000 7f93f2800640 1 -- 172.21.15.81:0/2649452166 shutdown_connections 2024-04-12T21:13:36.708 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.712+0000 7f93f2800640 1 --2- 172.21.15.81:0/2649452166 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 0x7f93ec10c220 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.712+0000 7f93f2800640 1 --2- 172.21.15.81:0/2649452166 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93ec074a20 0x7f93ec074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.712+0000 7f93f2800640 1 --2- 172.21.15.81:0/2649452166 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ec076000 0x7f93ec074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.713+0000 7f93f2800640 1 -- 172.21.15.81:0/2649452166 >> 172.21.15.81:0/2649452166 conn(0x7f93ec0fd460 msgr2=0x7f93ec0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.713+0000 7f93f2800640 1 -- 172.21.15.81:0/2649452166 shutdown_connections 2024-04-12T21:13:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.713+0000 7f93f2800640 1 -- 172.21.15.81:0/2649452166 wait complete. 2024-04-12T21:13:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.713+0000 7f93f2800640 1 Processor -- start 2024-04-12T21:13:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f2800640 1 -- start start 2024-04-12T21:13:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f2800640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ec074a20 0x7f93ec19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f2800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93ec076000 0x7f93ec19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f2800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 0x7f93ec1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f2800640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f93ec10fd50 con 0x7f93ec109d90 2024-04-12T21:13:36.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f2800640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f93ec10fbd0 con 0x7f93ec076000 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f2800640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f93ec10fed0 con 0x7f93ec074a20 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f1e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 0x7f93ec1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f1400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ec074a20 0x7f93ec19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f1e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 0x7f93ec1a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46366/0 (socket says 172.21.15.81:46366) 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f1e00640 1 -- 172.21.15.81:0/3741563187 learned_addr learned my addr 172.21.15.81:0/3741563187 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.714+0000 7f93f0a00640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93ec076000 0x7f93ec19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93f1e00640 1 -- 172.21.15.81:0/3741563187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ec074a20 msgr2=0x7f93ec19a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93f1e00640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ec074a20 0x7f93ec19a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93f1e00640 1 -- 172.21.15.81:0/3741563187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93ec076000 msgr2=0x7f93ec19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93f1e00640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93ec076000 0x7f93ec19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93f1e00640 1 -- 172.21.15.81:0/3741563187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ec1a1fa0 con 0x7f93ec109d90 2024-04-12T21:13:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93f1e00640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 0x7f93ec1a1930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f93e402c640 tx=0x7f93e4035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93ea000640 1 -- 172.21.15.81:0/3741563187 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e4039040 con 0x7f93ec109d90 2024-04-12T21:13:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93ea000640 1 -- 172.21.15.81:0/3741563187 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93e4044070 con 0x7f93ec109d90 2024-04-12T21:13:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93ec1a2230 con 0x7f93ec109d90 2024-04-12T21:13:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93ea000640 1 -- 172.21.15.81:0/3741563187 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e40043c0 con 0x7f93ec109d90 2024-04-12T21:13:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.715+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93ec1a2720 con 0x7f93ec109d90 2024-04-12T21:13:36.712 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.717+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93ec108ac0 con 0x7f93ec109d90 2024-04-12T21:13:36.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.718+0000 7f93ea000640 1 -- 172.21.15.81:0/3741563187 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f93e4004560 con 0x7f93ec109d90 2024-04-12T21:13:36.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.720+0000 7f93ea000640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f93d40a3bb0 0x7f93d40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:36.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.720+0000 7f93ea000640 1 -- 172.21.15.81:0/3741563187 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f93e40f2a40 con 0x7f93ec109d90 2024-04-12T21:13:36.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.720+0000 7f93f1400640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f93d40a3bb0 0x7f93d40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:36.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.721+0000 7f93f1400640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f93d40a3bb0 0x7f93d40a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009bd0 tx=0x7f93e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:36.720 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.724+0000 7f93ea000640 1 -- 172.21.15.81:0/3741563187 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93e404e020 con 0x7f93ec109d90 2024-04-12T21:13:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:36 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3670186200' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:36 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3157516966' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-12T21:13:36.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:36 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3670186200' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:36 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3157516966' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-12T21:13:36.964 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:36.969+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f93ec062bc0 con 0x7f93ec109d90 2024-04-12T21:13:37.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.113+0000 7f93ea000640 1 -- 172.21.15.81:0/3741563187 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f93e40bf180 con 0x7f93ec109d90 2024-04-12T21:13:37.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f93d40a3bb0 msgr2=0x7f93d40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:37.112 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f93d40a3bb0 0x7f93d40a6070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f93e0009bd0 tx=0x7f93e0005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:37.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 msgr2=0x7f93ec1a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:37.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 0x7f93ec1a1930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f93e402c640 tx=0x7f93e4035d10 comp rx=0 tx=0).stop 2024-04-12T21:13:37.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 shutdown_connections 2024-04-12T21:13:37.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f93d40a3bb0 0x7f93d40a6070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f93ec109d90 0x7f93ec1a1930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f93ec076000 0x7f93ec19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 --2- 172.21.15.81:0/3741563187 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93ec074a20 0x7f93ec19a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.117+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 >> 172.21.15.81:0/3741563187 conn(0x7f93ec0fd460 msgr2=0x7f93ec105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:37.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.118+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 shutdown_connections 2024-04-12T21:13:37.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.118+0000 7f93f2800640 1 -- 172.21.15.81:0/3741563187 wait complete. 2024-04-12T21:13:37.140 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-12T21:13:37.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 -- 172.21.15.81:0/3903342162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed64108500 msgr2=0x7fed64108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:37.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 --2- 172.21.15.81:0/3903342162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed64108500 0x7fed64108920 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009900 tx=0x7fed4c02bc40 comp rx=0 tx=0).stop 2024-04-12T21:13:37.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 -- 172.21.15.81:0/3903342162 shutdown_connections 2024-04-12T21:13:37.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 --2- 172.21.15.81:0/3903342162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed6410a0b0 0x7fed64110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 --2- 172.21.15.81:0/3903342162 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed641096f0 0x7fed64109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 --2- 172.21.15.81:0/3903342162 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed64108500 0x7fed64108920 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 -- 172.21.15.81:0/3903342162 >> 172.21.15.81:0/3903342162 conn(0x7fed64077490 msgr2=0x7fed640798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:37.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 -- 172.21.15.81:0/3903342162 shutdown_connections 2024-04-12T21:13:37.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.356+0000 7fed69200640 1 -- 172.21.15.81:0/3903342162 wait complete. 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.357+0000 7fed69200640 1 Processor -- start 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.357+0000 7fed69200640 1 -- start start 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.357+0000 7fed69200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed64108500 0x7fed6419e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.357+0000 7fed69200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed641096f0 0x7fed6419ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.357+0000 7fed69200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed6410a0b0 0x7fed641a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.357+0000 7fed69200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fed64114190 con 0x7fed641096f0 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.357+0000 7fed69200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fed64114010 con 0x7fed6410a0b0 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.357+0000 7fed69200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed64114310 con 0x7fed64108500 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed64108500 0x7fed6419e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed62a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed641096f0 0x7fed6419ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed64108500 0x7fed6419e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:52140/0 (socket says 172.21.15.81:52140) 2024-04-12T21:13:37.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 -- 172.21.15.81:0/1334420045 learned_addr learned my addr 172.21.15.81:0/1334420045 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63e00640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed6410a0b0 0x7fed641a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 -- 172.21.15.81:0/1334420045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed6410a0b0 msgr2=0x7fed641a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed6410a0b0 0x7fed641a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 -- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed641096f0 msgr2=0x7fed6419ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed641096f0 0x7fed6419ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 -- 172.21.15.81:0/1334420045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed641a63e0 con 0x7fed64108500 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed62a00640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed641096f0 0x7fed6419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.358+0000 7fed63400640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed64108500 0x7fed6419e7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed4c000c00 tx=0x7fed4c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.359+0000 7fed5be00640 1 -- 172.21.15.81:0/1334420045 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c041070 con 0x7fed64108500 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.359+0000 7fed5be00640 1 -- 172.21.15.81:0/1334420045 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed4c004450 con 0x7fed64108500 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.359+0000 7fed5be00640 1 -- 172.21.15.81:0/1334420045 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4c034440 con 0x7fed64108500 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.359+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed641a6670 con 0x7fed64108500 2024-04-12T21:13:37.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.359+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed641a6b10 con 0x7fed64108500 2024-04-12T21:13:37.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.360+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed6410f820 con 0x7fed64108500 2024-04-12T21:13:37.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.362+0000 7fed5be00640 1 -- 172.21.15.81:0/1334420045 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fed4c044070 con 0x7fed64108500 2024-04-12T21:13:37.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.363+0000 7fed5be00640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed480a3c30 0x7fed480a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:37.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.363+0000 7fed5be00640 1 -- 172.21.15.81:0/1334420045 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fed4c0f20f0 con 0x7fed64108500 2024-04-12T21:13:37.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.363+0000 7fed62a00640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed480a3c30 0x7fed480a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:37.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.364+0000 7fed62a00640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed480a3c30 0x7fed480a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fed6419fcd0 tx=0x7fed54005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:37.363 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.367+0000 7fed5be00640 1 -- 172.21.15.81:0/1334420045 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed4c0be270 con 0x7fed64108500 2024-04-12T21:13:37.610 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:37.615+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fed6402d5e0 con 0x7fed64108500 2024-04-12T21:13:37.719 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:37 smithi081 ceph-mon[31453]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-12T21:13:37.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:37 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3157516966' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-12T21:13:37.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:37 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:37.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:37 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3741563187' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-12T21:13:37.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:37 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3741563187' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-12T21:13:37.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:37 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:13:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:37 smithi153 ceph-mon[36569]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-12T21:13:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:37 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3157516966' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-12T21:13:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:37 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:37 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3741563187' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-12T21:13:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:37 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3741563187' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-12T21:13:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:37 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:13:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:37 smithi104 ceph-mon[36624]: pgmap v326: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-12T21:13:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:37 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3157516966' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-12T21:13:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:37 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:13:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:37 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3741563187' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-12T21:13:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:37 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3741563187' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-12T21:13:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:37 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:13:38.482 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.486+0000 7fed5be00640 1 -- 172.21.15.81:0/1334420045 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) ==== 115+0+0 (secure 0 0 0) 0x7fed4c0c2570 con 0x7fed64108500 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed480a3c30 msgr2=0x7fed480a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed480a3c30 0x7fed480a60f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fed6419fcd0 tx=0x7fed54005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed64108500 msgr2=0x7fed6419e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed64108500 0x7fed6419e7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fed4c000c00 tx=0x7fed4c02c650 comp rx=0 tx=0).stop 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 shutdown_connections 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fed480a3c30 0x7fed480a60f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fed6410a0b0 0x7fed641a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fed641096f0 0x7fed6419ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 --2- 172.21.15.81:0/1334420045 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed64108500 0x7fed6419e7c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.486 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 >> 172.21.15.81:0/1334420045 conn(0x7fed64077490 msgr2=0x7fed6410c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:38.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 shutdown_connections 2024-04-12T21:13:38.487 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.490+0000 7fed69200640 1 -- 172.21.15.81:0/1334420045 wait complete. 2024-04-12T21:13:38.511 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-12T21:13:38.511 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-12T21:13:38.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 -- 172.21.15.81:0/3238983139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcac8108500 msgr2=0x7fcac8108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:38.728 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 --2- 172.21.15.81:0/3238983139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcac8108500 0x7fcac8108920 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009900 tx=0x7fcabc02bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:38.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 -- 172.21.15.81:0/3238983139 shutdown_connections 2024-04-12T21:13:38.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 --2- 172.21.15.81:0/3238983139 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcac810a0b0 0x7fcac8110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 --2- 172.21.15.81:0/3238983139 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac81096f0 0x7fcac8109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 --2- 172.21.15.81:0/3238983139 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcac8108500 0x7fcac8108920 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 -- 172.21.15.81:0/3238983139 >> 172.21.15.81:0/3238983139 conn(0x7fcac8077490 msgr2=0x7fcac80798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:38.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 -- 172.21.15.81:0/3238983139 shutdown_connections 2024-04-12T21:13:38.729 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.732+0000 7fcacec00640 1 -- 172.21.15.81:0/3238983139 wait complete. 2024-04-12T21:13:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.733+0000 7fcacec00640 1 Processor -- start 2024-04-12T21:13:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.733+0000 7fcacec00640 1 -- start start 2024-04-12T21:13:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcacec00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcac8108500 0x7fcac819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcacec00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcac81096f0 0x7fcac819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcacec00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac810a0b0 0x7fcac81a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcacec00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcac81141e0 con 0x7fcac810a0b0 2024-04-12T21:13:38.731 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcacec00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcac8114060 con 0x7fcac8108500 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcacec00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcac8114360 con 0x7fcac81096f0 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcace200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac810a0b0 0x7fcac81a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcace200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac810a0b0 0x7fcac81a5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46394/0 (socket says 172.21.15.81:46394) 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcacce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcac81096f0 0x7fcac819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcacd800640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcac8108500 0x7fcac819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcace200640 1 -- 172.21.15.81:0/2660066395 learned_addr learned my addr 172.21.15.81:0/2660066395 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.734+0000 7fcace200640 1 -- 172.21.15.81:0/2660066395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcac81096f0 msgr2=0x7fcac819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.735+0000 7fcace200640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcac81096f0 0x7fcac819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.735+0000 7fcace200640 1 -- 172.21.15.81:0/2660066395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcac8108500 msgr2=0x7fcac819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.735+0000 7fcace200640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcac8108500 0x7fcac819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.735+0000 7fcace200640 1 -- 172.21.15.81:0/2660066395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac81a63d0 con 0x7fcac810a0b0 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.735+0000 7fcacd800640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcac8108500 0x7fcac819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.735+0000 7fcace200640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac810a0b0 0x7fcac81a5d60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcac000eb10 tx=0x7fcac000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:38.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.735+0000 7fcac6000640 1 -- 172.21.15.81:0/2660066395 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0004b20 con 0x7fcac810a0b0 2024-04-12T21:13:38.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.736+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac81a66c0 con 0x7fcac810a0b0 2024-04-12T21:13:38.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.736+0000 7fcac6000640 1 -- 172.21.15.81:0/2660066395 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcac0004cc0 con 0x7fcac810a0b0 2024-04-12T21:13:38.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.736+0000 7fcac6000640 1 -- 172.21.15.81:0/2660066395 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac0010760 con 0x7fcac810a0b0 2024-04-12T21:13:38.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.736+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac81a6c60 con 0x7fcac810a0b0 2024-04-12T21:13:38.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.737+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac810f7c0 con 0x7fcac810a0b0 2024-04-12T21:13:38.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.739+0000 7fcac6000640 1 -- 172.21.15.81:0/2660066395 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fcac0005020 con 0x7fcac810a0b0 2024-04-12T21:13:38.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.740+0000 7fcac6000640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcab00a3c30 0x7fcab00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:38.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.740+0000 7fcac6000640 1 -- 172.21.15.81:0/2660066395 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fcac00d0cb0 con 0x7fcac810a0b0 2024-04-12T21:13:38.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.740+0000 7fcacd800640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcab00a3c30 0x7fcab00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:38.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.741+0000 7fcacd800640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcab00a3c30 0x7fcab00a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcabc02c170 tx=0x7fcabc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:38.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.745+0000 7fcac6000640 1 -- 172.21.15.81:0/2660066395 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcac009d3f0 con 0x7fcac810a0b0 2024-04-12T21:13:38.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:38 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1334420045' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-12T21:13:38.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:38 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-12T21:13:38.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:38 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1334420045' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-12T21:13:38.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:38 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-12T21:13:38.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:38 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1334420045' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-12T21:13:38.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:38 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-12T21:13:38.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:38.975+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fcac8062400 con 0x7fcab00a3c30 2024-04-12T21:13:39.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.010+0000 7fcac6000640 1 -- 172.21.15.81:0/2660066395 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcac8062400 con 0x7fcab00a3c30 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcab00a3c30 msgr2=0x7fcab00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcab00a3c30 0x7fcab00a60f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fcabc02c170 tx=0x7fcabc02df90 comp rx=0 tx=0).stop 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac810a0b0 msgr2=0x7fcac81a5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac810a0b0 0x7fcac81a5d60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcac000eb10 tx=0x7fcac000c2d0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 shutdown_connections 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcab00a3c30 0x7fcab00a60f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcac81096f0 0x7fcac819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcac810a0b0 0x7fcac81a5d60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 --2- 172.21.15.81:0/2660066395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcac8108500 0x7fcac819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 >> 172.21.15.81:0/2660066395 conn(0x7fcac8077490 msgr2=0x7fcac810c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:39.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 shutdown_connections 2024-04-12T21:13:39.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.013+0000 7fcacec00640 1 -- 172.21.15.81:0/2660066395 wait complete. 2024-04-12T21:13:39.029 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-12T21:13:39.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 -- 172.21.15.81:0/300379284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 msgr2=0x7f870c074450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 --2- 172.21.15.81:0/300379284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 0x7f870c074450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8704009be0 tx=0x7f870402bef0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 -- 172.21.15.81:0/300379284 shutdown_connections 2024-04-12T21:13:39.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 --2- 172.21.15.81:0/300379284 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f870c109e10 0x7f870c10c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 --2- 172.21.15.81:0/300379284 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f870c074a20 0x7f870c074ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 --2- 172.21.15.81:0/300379284 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 0x7f870c074450 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 -- 172.21.15.81:0/300379284 >> 172.21.15.81:0/300379284 conn(0x7f870c0fd4c0 msgr2=0x7f870c0ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:39.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 -- 172.21.15.81:0/300379284 shutdown_connections 2024-04-12T21:13:39.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.239+0000 7f8710e00640 1 -- 172.21.15.81:0/300379284 wait complete. 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.240+0000 7f8710e00640 1 Processor -- start 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f8710e00640 1 -- start start 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f8710e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f870c074a20 0x7f870c19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f8710e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 0x7f870c19ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f8710e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f870c109e10 0x7f870c1a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f8710e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f870c10fd60 con 0x7f870c109e10 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f8710e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f870c10fbe0 con 0x7f870c074a20 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f8710e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f870c10fee0 con 0x7f870c076000 2024-04-12T21:13:39.238 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f870aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 0x7f870c19ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f870be00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f870c109e10 0x7f870c1a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f870aa00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 0x7f870c19ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:52206/0 (socket says 172.21.15.81:52206) 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f870aa00640 1 -- 172.21.15.81:0/1813902564 learned_addr learned my addr 172.21.15.81:0/1813902564 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.241+0000 7f870b400640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f870c074a20 0x7f870c19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f870b400640 1 -- 172.21.15.81:0/1813902564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 msgr2=0x7f870c19ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f870b400640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 0x7f870c19ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f870b400640 1 -- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f870c109e10 msgr2=0x7f870c1a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f870b400640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f870c109e10 0x7f870c1a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f870b400640 1 -- 172.21.15.81:0/1813902564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c1a63e0 con 0x7f870c074a20 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f870be00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f870c109e10 0x7f870c1a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f870b400640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f870c074a20 0x7f870c19e810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8704036040 tx=0x7f8704033580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f8703e00640 1 -- 172.21.15.81:0/1813902564 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704043020 con 0x7f870c074a20 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f870c1a6670 con 0x7f870c074a20 2024-04-12T21:13:39.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f8703e00640 1 -- 172.21.15.81:0/1813902564 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8704003e50 con 0x7f870c074a20 2024-04-12T21:13:39.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f8703e00640 1 -- 172.21.15.81:0/1813902564 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704034690 con 0x7f870c074a20 2024-04-12T21:13:39.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.242+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f870c1a6c10 con 0x7f870c074a20 2024-04-12T21:13:39.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.244+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f870c104730 con 0x7f870c074a20 2024-04-12T21:13:39.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.246+0000 7f8703e00640 1 -- 172.21.15.81:0/1813902564 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8704046020 con 0x7f870c074a20 2024-04-12T21:13:39.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.247+0000 7f8703e00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f86f00a3c30 0x7f86f00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:39.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.247+0000 7f8703e00640 1 -- 172.21.15.81:0/1813902564 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f87040f32a0 con 0x7f870c074a20 2024-04-12T21:13:39.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.247+0000 7f870aa00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f86f00a3c30 0x7f86f00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:39.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.248+0000 7f870aa00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f86f00a3c30 0x7f86f00a60f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f870c19fcd0 tx=0x7f86f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:39.245 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.250+0000 7f8703e00640 1 -- 172.21.15.81:0/1813902564 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f870404d020 con 0x7f870c074a20 2024-04-12T21:13:39.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:13:38 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:13:38.990+0000 7f3273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-12T21:13:39.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:13:38 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:13:38.990+0000 7f3273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-12T21:13:39.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:13:38 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:13:38.990+0000 7f3273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-12T21:13:39.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:13:38 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:13:38.990+0000 7f3273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-12T21:13:39.402 INFO:journalctl@ceph.mgr.x.smithi081.stdout:Apr 12 21:13:38 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mgr-x[31683]: 2024-04-12T21:13:38.990+0000 7f3273000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-12T21:13:39.478 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.483+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f870c062490 con 0x7f86f00a3c30 2024-04-12T21:13:39.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.526+0000 7f8703e00640 1 -- 172.21.15.81:0/1813902564 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f870c062490 con 0x7f86f00a3c30 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f86f00a3c30 msgr2=0x7f86f00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f86f00a3c30 0x7f86f00a60f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f870c19fcd0 tx=0x7f86f4009260 comp rx=0 tx=0).stop 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f870c074a20 msgr2=0x7f870c19e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f870c074a20 0x7f870c19e810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8704036040 tx=0x7f8704033580 comp rx=0 tx=0).stop 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 shutdown_connections 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f86f00a3c30 0x7f86f00a60f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f870c076000 0x7f870c19ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f870c109e10 0x7f870c1a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 --2- 172.21.15.81:0/1813902564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f870c074a20 0x7f870c19e810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 >> 172.21.15.81:0/1813902564 conn(0x7f870c0fd4c0 msgr2=0x7f870c101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 shutdown_connections 2024-04-12T21:13:39.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.529+0000 7f8710e00640 1 -- 172.21.15.81:0/1813902564 wait complete. 2024-04-12T21:13:39.546 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-12T21:13:39.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 -- 172.21.15.81:0/2699633048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105410 msgr2=0x7fc22c105890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 --2- 172.21.15.81:0/2699633048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105410 0x7fc22c105890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc218009900 tx=0x7fc21802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:39.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 -- 172.21.15.81:0/2699633048 shutdown_connections 2024-04-12T21:13:39.764 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 --2- 172.21.15.81:0/2699633048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc22c105dd0 0x7fc22c10c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 --2- 172.21.15.81:0/2699633048 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105410 0x7fc22c105890 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 --2- 172.21.15.81:0/2699633048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c104220 0x7fc22c104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 -- 172.21.15.81:0/2699633048 >> 172.21.15.81:0/2699633048 conn(0x7fc22c0ff7b0 msgr2=0x7fc22c101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:39.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 -- 172.21.15.81:0/2699633048 shutdown_connections 2024-04-12T21:13:39.765 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.768+0000 7fc233c00640 1 -- 172.21.15.81:0/2699633048 wait complete. 2024-04-12T21:13:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.769+0000 7fc233c00640 1 Processor -- start 2024-04-12T21:13:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.769+0000 7fc233c00640 1 -- start start 2024-04-12T21:13:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc233c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc22c104220 0x7fc22c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc233c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c105410 0x7fc22c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc233c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105dd0 0x7fc22c1a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc233c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c10feb0 con 0x7fc22c105dd0 2024-04-12T21:13:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc233c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c10fd30 con 0x7fc22c105410 2024-04-12T21:13:39.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc233c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c110030 con 0x7fc22c104220 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc231200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc22c104220 0x7fc22c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc231200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc22c104220 0x7fc22c19a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:52222/0 (socket says 172.21.15.81:52222) 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc231200640 1 -- 172.21.15.81:0/320220767 learned_addr learned my addr 172.21.15.81:0/320220767 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc231c00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105dd0 0x7fc22c1a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.770+0000 7fc22be00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c105410 0x7fc22c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc231200640 1 -- 172.21.15.81:0/320220767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c105410 msgr2=0x7fc22c19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc231200640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c105410 0x7fc22c19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc231200640 1 -- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105dd0 msgr2=0x7fc22c1a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc231200640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105dd0 0x7fc22c1a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc231200640 1 -- 172.21.15.81:0/320220767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c1a2160 con 0x7fc22c104220 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc231c00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105dd0 0x7fc22c1a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc231200640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc22c104220 0x7fc22c19a540 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc22000e9c0 tx=0x7fc22000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc229600640 1 -- 172.21.15.81:0/320220767 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22000cd80 con 0x7fc22c104220 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc229600640 1 -- 172.21.15.81:0/320220767 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2200035f0 con 0x7fc22c104220 2024-04-12T21:13:39.768 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22c1a2450 con 0x7fc22c104220 2024-04-12T21:13:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc229600640 1 -- 172.21.15.81:0/320220767 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220010690 con 0x7fc22c104220 2024-04-12T21:13:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.771+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22c1a29c0 con 0x7fc22c104220 2024-04-12T21:13:39.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.772+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc22c10b560 con 0x7fc22c104220 2024-04-12T21:13:39.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.774+0000 7fc229600640 1 -- 172.21.15.81:0/320220767 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fc2200037a0 con 0x7fc22c104220 2024-04-12T21:13:39.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.776+0000 7fc229600640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc2140a3c30 0x7fc2140a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:39.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.776+0000 7fc229600640 1 -- 172.21.15.81:0/320220767 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc2200d0b80 con 0x7fc22c104220 2024-04-12T21:13:39.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.776+0000 7fc22be00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc2140a3c30 0x7fc2140a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:39.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.777+0000 7fc22be00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc2140a3c30 0x7fc2140a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc21802c7b0 tx=0x7fc218005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:39.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:39.778+0000 7fc229600640 1 -- 172.21.15.81:0/320220767 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc22009d2c0 con 0x7fc22c104220 2024-04-12T21:13:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:39 smithi153 ceph-mon[36569]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-12T21:13:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:39 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-12T21:13:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:39 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:13:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:39 smithi104 ceph-mon[36624]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-12T21:13:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:39 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-12T21:13:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:39 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:13:39.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:39 smithi081 ceph-mon[31453]: pgmap v327: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-12T21:13:39.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:39 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-12T21:13:39.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:39 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:13:40.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.014+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fc22c062400 con 0x7fc2140a3c30 2024-04-12T21:13:40.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.023+0000 7fc229600640 1 -- 172.21.15.81:0/320220767 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fc22c062400 con 0x7fc2140a3c30 2024-04-12T21:13:40.018 INFO:teuthology.orchestra.run.smithi081.stdout:/volumes/qa/sv_0/a35797a2-b544-40d4-8ebb-8c6b17e65397 2024-04-12T21:13:40.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc2140a3c30 msgr2=0x7fc2140a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc2140a3c30 0x7fc2140a60f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc21802c7b0 tx=0x7fc218005fd0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc22c104220 msgr2=0x7fc22c19a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc22c104220 0x7fc22c19a540 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc22000e9c0 tx=0x7fc22000ee80 comp rx=0 tx=0).stop 2024-04-12T21:13:40.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 shutdown_connections 2024-04-12T21:13:40.021 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fc2140a3c30 0x7fc2140a60f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fc22c105dd0 0x7fc22c1a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc22c105410 0x7fc22c19aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 --2- 172.21.15.81:0/320220767 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc22c104220 0x7fc22c19a540 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.025+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 >> 172.21.15.81:0/320220767 conn(0x7fc22c0ff7b0 msgr2=0x7fc22c108730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:40.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.026+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 shutdown_connections 2024-04-12T21:13:40.022 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.026+0000 7fc233c00640 1 -- 172.21.15.81:0/320220767 wait complete. 2024-04-12T21:13:40.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-12T21:13:40.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 -- 172.21.15.81:0/288408483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba30105290 msgr2=0x7fba30107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 --2- 172.21.15.81:0/288408483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba30105290 0x7fba30107680 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fba18009900 tx=0x7fba1802bc40 comp rx=0 tx=0).stop 2024-04-12T21:13:40.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 -- 172.21.15.81:0/288408483 shutdown_connections 2024-04-12T21:13:40.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 --2- 172.21.15.81:0/288408483 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba30105290 0x7fba30107680 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 --2- 172.21.15.81:0/288408483 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba30107bc0 0x7fba3010a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 --2- 172.21.15.81:0/288408483 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba30102960 0x7fba30104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 -- 172.21.15.81:0/288408483 >> 172.21.15.81:0/288408483 conn(0x7fba300fc370 msgr2=0x7fba300fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:40.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 -- 172.21.15.81:0/288408483 shutdown_connections 2024-04-12T21:13:40.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.259+0000 7fba2fe00640 1 -- 172.21.15.81:0/288408483 wait complete. 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.260+0000 7fba2fe00640 1 Processor -- start 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.260+0000 7fba2fe00640 1 -- start start 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba30102960 0x7fba301022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba30105290 0x7fba30100910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba30107bc0 0x7fba30100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fba3010db40 con 0x7fba30105290 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2fe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fba3010d9c0 con 0x7fba30107bc0 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2fe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba3010dcc0 con 0x7fba30102960 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2f400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba30107bc0 0x7fba30100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba27e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba30105290 0x7fba30100910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:40.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2f400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba30107bc0 0x7fba30100e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:37028/0 (socket says 172.21.15.81:37028) 2024-04-12T21:13:40.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2f400640 1 -- 172.21.15.81:0/432454532 learned_addr learned my addr 172.21.15.81:0/432454532 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:40.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.261+0000 7fba2ea00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba30102960 0x7fba301022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba2f400640 1 -- 172.21.15.81:0/432454532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba30102960 msgr2=0x7fba301022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba2f400640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba30102960 0x7fba301022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba2f400640 1 -- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba30105290 msgr2=0x7fba30100910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba2f400640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba30105290 0x7fba30100910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba2f400640 1 -- 172.21.15.81:0/432454532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba301016e0 con 0x7fba30107bc0 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba2f400640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba30107bc0 0x7fba30100e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fba28009cf0 tx=0x7fba2800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba27400640 1 -- 172.21.15.81:0/432454532 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28004d90 con 0x7fba30107bc0 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba301a2330 con 0x7fba30107bc0 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba27400640 1 -- 172.21.15.81:0/432454532 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba28005510 con 0x7fba30107bc0 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba27400640 1 -- 172.21.15.81:0/432454532 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba28010a60 con 0x7fba30107bc0 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.262+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba301a2850 con 0x7fba30107bc0 2024-04-12T21:13:40.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.263+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba300fde60 con 0x7fba30107bc0 2024-04-12T21:13:40.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.266+0000 7fba27400640 1 -- 172.21.15.81:0/432454532 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fba280056c0 con 0x7fba30107bc0 2024-04-12T21:13:40.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.267+0000 7fba27400640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fba140a3bb0 0x7fba140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:40.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.267+0000 7fba27400640 1 -- 172.21.15.81:0/432454532 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fba280d16d0 con 0x7fba30107bc0 2024-04-12T21:13:40.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.267+0000 7fba2ea00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fba140a3bb0 0x7fba140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:40.263 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.268+0000 7fba2ea00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fba140a3bb0 0x7fba140a6070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fba200095f0 tx=0x7fba20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:40.265 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.270+0000 7fba27400640 1 -- 172.21.15.81:0/432454532 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba2809de10 con 0x7fba30107bc0 2024-04-12T21:13:40.502 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.506+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fba30062400 con 0x7fba140a3bb0 2024-04-12T21:13:40.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.516+0000 7fba27400640 1 -- 172.21.15.81:0/432454532 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fba28099d80 con 0x7fba30107bc0 2024-04-12T21:13:40.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.549+0000 7fba27400640 1 -- 172.21.15.81:0/432454532 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fba30062400 con 0x7fba140a3bb0 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fba140a3bb0 msgr2=0x7fba140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fba140a3bb0 0x7fba140a6070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fba200095f0 tx=0x7fba20005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba30107bc0 msgr2=0x7fba30100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba30107bc0 0x7fba30100e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fba28009cf0 tx=0x7fba2800c6a0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 shutdown_connections 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fba140a3bb0 0x7fba140a6070 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fba30107bc0 0x7fba30100e50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fba30105290 0x7fba30100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 --2- 172.21.15.81:0/432454532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba30102960 0x7fba301022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 >> 172.21.15.81:0/432454532 conn(0x7fba300fc370 msgr2=0x7fba301032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:40.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.552+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 shutdown_connections 2024-04-12T21:13:40.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.553+0000 7fba2fe00640 1 -- 172.21.15.81:0/432454532 wait complete. 2024-04-12T21:13:40.567 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-12T21:13:40.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 -- 172.21.15.81:0/4141370010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa914074a20 msgr2=0x7fa914074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 --2- 172.21.15.81:0/4141370010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa914074a20 0x7fa914074ea0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa900009930 tx=0x7fa90002bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:40.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 -- 172.21.15.81:0/4141370010 shutdown_connections 2024-04-12T21:13:40.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 --2- 172.21.15.81:0/4141370010 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa91410e4a0 0x7fa914110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 --2- 172.21.15.81:0/4141370010 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa914074a20 0x7fa914074ea0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 --2- 172.21.15.81:0/4141370010 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914076000 0x7fa914074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 -- 172.21.15.81:0/4141370010 >> 172.21.15.81:0/4141370010 conn(0x7fa9140fd460 msgr2=0x7fa9140ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:40.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 -- 172.21.15.81:0/4141370010 shutdown_connections 2024-04-12T21:13:40.778 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.781+0000 7fa918a00640 1 -- 172.21.15.81:0/4141370010 wait complete. 2024-04-12T21:13:40.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.782+0000 7fa918a00640 1 Processor -- start 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.782+0000 7fa918a00640 1 -- start start 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.782+0000 7fa918a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa914074a20 0x7fa91419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.782+0000 7fa918a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914076000 0x7fa91419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.782+0000 7fa918a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91410e4a0 0x7fa9141a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.782+0000 7fa918a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa914114160 con 0x7fa91410e4a0 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.782+0000 7fa918a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa914113fe0 con 0x7fa914074a20 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.782+0000 7fa918a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa9141142e0 con 0x7fa914076000 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91410e4a0 0x7fa9141a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91410e4a0 0x7fa9141a5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46506/0 (socket says 172.21.15.81:46506) 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 -- 172.21.15.81:0/3104532771 learned_addr learned my addr 172.21.15.81:0/3104532771 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913400640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa914074a20 0x7fa91419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa912a00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914076000 0x7fa91419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 -- 172.21.15.81:0/3104532771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914076000 msgr2=0x7fa91419ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914076000 0x7fa91419ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 -- 172.21.15.81:0/3104532771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa914074a20 msgr2=0x7fa91419e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa914074a20 0x7fa91419e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 -- 172.21.15.81:0/3104532771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9141a6420 con 0x7fa91410e4a0 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.783+0000 7fa913e00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91410e4a0 0x7fa9141a5db0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00eb10 tx=0x7fa90c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.784+0000 7fa90be00640 1 -- 172.21.15.81:0/3104532771 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c004b20 con 0x7fa91410e4a0 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.784+0000 7fa90be00640 1 -- 172.21.15.81:0/3104532771 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c004cc0 con 0x7fa91410e4a0 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.784+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9141a6710 con 0x7fa91410e4a0 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.784+0000 7fa90be00640 1 -- 172.21.15.81:0/3104532771 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c010760 con 0x7fa91410e4a0 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.784+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9141a6be0 con 0x7fa91410e4a0 2024-04-12T21:13:40.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.785+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa914104560 con 0x7fa91410e4a0 2024-04-12T21:13:40.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.787+0000 7fa90be00640 1 -- 172.21.15.81:0/3104532771 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa90c005020 con 0x7fa91410e4a0 2024-04-12T21:13:40.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.788+0000 7fa90be00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa8f80a3c40 0x7fa8f80a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:40.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.789+0000 7fa90be00640 1 -- 172.21.15.81:0/3104532771 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa90c0d0da0 con 0x7fa91410e4a0 2024-04-12T21:13:40.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.789+0000 7fa913400640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa8f80a3c40 0x7fa8f80a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:40.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.790+0000 7fa913400640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa8f80a3c40 0x7fa8f80a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa904009bd0 tx=0x7fa904005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:40.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:40.793+0000 7fa90be00640 1 -- 172.21.15.81:0/3104532771 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa90c09d380 con 0x7fa91410e4a0 2024-04-12T21:13:40.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:40 smithi153 ceph-mon[36569]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:40.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:40 smithi153 ceph-mon[36569]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:40.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:40 smithi104 ceph-mon[36624]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:40.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:40 smithi104 ceph-mon[36624]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:40.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:40 smithi081 ceph-mon[31453]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:40.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:40 smithi081 ceph-mon[31453]: from='client.24839 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.022+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fa914062400 con 0x7fa8f80a3c40 2024-04-12T21:13:41.027 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.031+0000 7fa90be00640 1 -- 172.21.15.81:0/3104532771 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa914062400 con 0x7fa8f80a3c40 2024-04-12T21:13:41.027 INFO:teuthology.orchestra.run.smithi081.stdout:/volumes/qa/sv_1/8dc75769-56c4-40a9-9818-3b2b61bf5c2b 2024-04-12T21:13:41.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa8f80a3c40 msgr2=0x7fa8f80a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa8f80a3c40 0x7fa8f80a6100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa904009bd0 tx=0x7fa904005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:41.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91410e4a0 msgr2=0x7fa9141a5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91410e4a0 0x7fa9141a5db0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00eb10 tx=0x7fa90c00c2d0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 shutdown_connections 2024-04-12T21:13:41.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa8f80a3c40 0x7fa8f80a6100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa914076000 0x7fa91419ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa91410e4a0 0x7fa9141a5db0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 --2- 172.21.15.81:0/3104532771 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa914074a20 0x7fa91419e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 >> 172.21.15.81:0/3104532771 conn(0x7fa9140fd460 msgr2=0x7fa914101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:41.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 shutdown_connections 2024-04-12T21:13:41.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.034+0000 7fa918a00640 1 -- 172.21.15.81:0/3104532771 wait complete. 2024-04-12T21:13:41.049 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:13:41.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.260+0000 7fdf5b200640 1 -- 172.21.15.81:0/2433446839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c074a20 msgr2=0x7fdf5c074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.260+0000 7fdf5b200640 1 --2- 172.21.15.81:0/2433446839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c074a20 0x7fdf5c074ea0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009930 tx=0x7fdf4802bc30 comp rx=0 tx=0).stop 2024-04-12T21:13:41.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.260+0000 7fdf5b200640 1 -- 172.21.15.81:0/2433446839 shutdown_connections 2024-04-12T21:13:41.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.260+0000 7fdf5b200640 1 --2- 172.21.15.81:0/2433446839 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf5c10e400 0x7fdf5c1107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.260+0000 7fdf5b200640 1 --2- 172.21.15.81:0/2433446839 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c074a20 0x7fdf5c074ea0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.260+0000 7fdf5b200640 1 --2- 172.21.15.81:0/2433446839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf5c076000 0x7fdf5c074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.261+0000 7fdf5b200640 1 -- 172.21.15.81:0/2433446839 >> 172.21.15.81:0/2433446839 conn(0x7fdf5c0fd3a0 msgr2=0x7fdf5c0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:41.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.261+0000 7fdf5b200640 1 -- 172.21.15.81:0/2433446839 shutdown_connections 2024-04-12T21:13:41.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.261+0000 7fdf5b200640 1 -- 172.21.15.81:0/2433446839 wait complete. 2024-04-12T21:13:41.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.262+0000 7fdf5b200640 1 Processor -- start 2024-04-12T21:13:41.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.262+0000 7fdf5b200640 1 -- start start 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.262+0000 7fdf5b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf5c074a20 0x7fdf5c19e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.262+0000 7fdf5b200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf5c076000 0x7fdf5c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.262+0000 7fdf5b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c10e400 0x7fdf5c1a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.262+0000 7fdf5b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c114110 con 0x7fdf5c10e400 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5b200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c113f90 con 0x7fdf5c076000 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c114290 con 0x7fdf5c074a20 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c10e400 0x7fdf5c1a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5a800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c10e400 0x7fdf5c1a5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46526/0 (socket says 172.21.15.81:46526) 2024-04-12T21:13:41.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5a800640 1 -- 172.21.15.81:0/1631693042 learned_addr learned my addr 172.21.15.81:0/1631693042 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:41.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf59e00640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf5c074a20 0x7fdf5c19e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:41.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf59400640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf5c076000 0x7fdf5c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:41.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5a800640 1 -- 172.21.15.81:0/1631693042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf5c074a20 msgr2=0x7fdf5c19e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5a800640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf5c074a20 0x7fdf5c19e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5a800640 1 -- 172.21.15.81:0/1631693042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf5c076000 msgr2=0x7fdf5c19ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5a800640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf5c076000 0x7fdf5c19ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.263+0000 7fdf5a800640 1 -- 172.21.15.81:0/1631693042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c1a63a0 con 0x7fdf5c10e400 2024-04-12T21:13:41.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.264+0000 7fdf5a800640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c10e400 0x7fdf5c1a5d30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400eb10 tx=0x7fdf5400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:41.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.264+0000 7fdf52a00640 1 -- 172.21.15.81:0/1631693042 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54004b20 con 0x7fdf5c10e400 2024-04-12T21:13:41.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.264+0000 7fdf52a00640 1 -- 172.21.15.81:0/1631693042 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf54004cc0 con 0x7fdf5c10e400 2024-04-12T21:13:41.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.264+0000 7fdf52a00640 1 -- 172.21.15.81:0/1631693042 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54010760 con 0x7fdf5c10e400 2024-04-12T21:13:41.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.264+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5c1a6690 con 0x7fdf5c10e400 2024-04-12T21:13:41.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.264+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5c1a6b60 con 0x7fdf5c10e400 2024-04-12T21:13:41.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.265+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf5c1044c0 con 0x7fdf5c10e400 2024-04-12T21:13:41.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.267+0000 7fdf52a00640 1 -- 172.21.15.81:0/1631693042 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdf54005020 con 0x7fdf5c10e400 2024-04-12T21:13:41.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.268+0000 7fdf52a00640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdf400a3cc0 0x7fdf400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:41.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.268+0000 7fdf52a00640 1 -- 172.21.15.81:0/1631693042 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdf540d0d20 con 0x7fdf5c10e400 2024-04-12T21:13:41.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.269+0000 7fdf59e00640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdf400a3cc0 0x7fdf400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:41.264 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.269+0000 7fdf59e00640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdf400a3cc0 0x7fdf400a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c005fd0 tx=0x7fdf4c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:41.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.273+0000 7fdf52a00640 1 -- 172.21.15.81:0/1631693042 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdf5409d300 con 0x7fdf5c10e400 2024-04-12T21:13:41.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.501+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf5c02d5e0 con 0x7fdf5c10e400 2024-04-12T21:13:41.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.503+0000 7fdf52a00640 1 -- 172.21.15.81:0/1631693042 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11815 (secure 0 0 0) 0x7fdf5409cc00 con 0x7fdf5c10e400 2024-04-12T21:13:41.499 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:41.499 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T21:13:13.003153+0000","modified":"2024-04-12T21:13:37.617228+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-12T21:13:41.500 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-04-12T21:13:41.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.507+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdf400a3cc0 msgr2=0x7fdf400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.507+0000 7fdf5b200640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdf400a3cc0 0x7fdf400a6180 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c005fd0 tx=0x7fdf4c00b390 comp rx=0 tx=0).stop 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.507+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c10e400 msgr2=0x7fdf5c1a5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.507+0000 7fdf5b200640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c10e400 0x7fdf5c1a5d30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdf5400eb10 tx=0x7fdf5400c2d0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.508+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 shutdown_connections 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.508+0000 7fdf5b200640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fdf400a3cc0 0x7fdf400a6180 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.508+0000 7fdf5b200640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fdf5c10e400 0x7fdf5c1a5d30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.508+0000 7fdf5b200640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf5c076000 0x7fdf5c19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.508+0000 7fdf5b200640 1 --2- 172.21.15.81:0/1631693042 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf5c074a20 0x7fdf5c19e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.508+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 >> 172.21.15.81:0/1631693042 conn(0x7fdf5c0fd3a0 msgr2=0x7fdf5c101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.508+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 shutdown_connections 2024-04-12T21:13:41.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.508+0000 7fdf5b200640 1 -- 172.21.15.81:0/1631693042 wait complete. 2024-04-12T21:13:41.520 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:13:41.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 -- 172.21.15.81:0/3772387772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103bc0 msgr2=0x7f90d010a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 --2- 172.21.15.81:0/3772387772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103bc0 0x7f90d010a450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009ca0 tx=0x7f90c802c100 comp rx=0 tx=0).stop 2024-04-12T21:13:41.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 -- 172.21.15.81:0/3772387772 shutdown_connections 2024-04-12T21:13:41.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 --2- 172.21.15.81:0/3772387772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90d0103200 0x7f90d0103680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 --2- 172.21.15.81:0/3772387772 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103bc0 0x7f90d010a450 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 --2- 172.21.15.81:0/3772387772 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d0102010 0x7f90d0102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 -- 172.21.15.81:0/3772387772 >> 172.21.15.81:0/3772387772 conn(0x7f90d00fd5a0 msgr2=0x7f90d00ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:41.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 -- 172.21.15.81:0/3772387772 shutdown_connections 2024-04-12T21:13:41.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.714+0000 7f90d6200640 1 -- 172.21.15.81:0/3772387772 wait complete. 2024-04-12T21:13:41.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.715+0000 7f90d6200640 1 Processor -- start 2024-04-12T21:13:41.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.715+0000 7f90d6200640 1 -- start start 2024-04-12T21:13:41.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.715+0000 7f90d6200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90d0102010 0x7f90d019a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:41.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.715+0000 7f90d6200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103200 0x7f90d019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:41.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.715+0000 7f90d6200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d0103bc0 0x7f90d01a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:41.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.715+0000 7f90d6200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f90d010dca0 con 0x7f90d0103200 2024-04-12T21:13:41.713 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.715+0000 7f90d6200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f90d010db20 con 0x7f90d0103bc0 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.715+0000 7f90d6200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f90d010de20 con 0x7f90d0102010 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90c6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103200 0x7f90d019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90c6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103200 0x7f90d019aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46544/0 (socket says 172.21.15.81:46544) 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90c6a00640 1 -- 172.21.15.81:0/3901931386 learned_addr learned my addr 172.21.15.81:0/3901931386 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90cf400640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90d0102010 0x7f90d019a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90cfe00640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d0103bc0 0x7f90d01a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90c6a00640 1 -- 172.21.15.81:0/3901931386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90d0102010 msgr2=0x7f90d019a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90c6a00640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90d0102010 0x7f90d019a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90c6a00640 1 -- 172.21.15.81:0/3901931386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d0103bc0 msgr2=0x7f90d01a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90c6a00640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d0103bc0 0x7f90d01a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.716+0000 7f90c6a00640 1 -- 172.21.15.81:0/3901931386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d01a2150 con 0x7f90d0103200 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.717+0000 7f90c6a00640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103200 0x7f90d019aa70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f90b800e9f0 tx=0x7f90b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.717+0000 7f90ccc00640 1 -- 172.21.15.81:0/3901931386 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b800cd80 con 0x7f90d0103200 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.717+0000 7f90ccc00640 1 -- 172.21.15.81:0/3901931386 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90b8005500 con 0x7f90d0103200 2024-04-12T21:13:41.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.717+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d01a2440 con 0x7f90d0103200 2024-04-12T21:13:41.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.717+0000 7f90ccc00640 1 -- 172.21.15.81:0/3901931386 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8010690 con 0x7f90d0103200 2024-04-12T21:13:41.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.717+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d01a2990 con 0x7f90d0103200 2024-04-12T21:13:41.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.718+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90d01a2c50 con 0x7f90d0103200 2024-04-12T21:13:41.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.720+0000 7f90ccc00640 1 -- 172.21.15.81:0/3901931386 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f90b80046c0 con 0x7f90d0103200 2024-04-12T21:13:41.716 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.721+0000 7f90ccc00640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f90b40a3cc0 0x7f90b40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:41.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.721+0000 7f90ccc00640 1 -- 172.21.15.81:0/3901931386 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f90b80d0b50 con 0x7f90d0103200 2024-04-12T21:13:41.717 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.722+0000 7f90cf400640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f90b40a3cc0 0x7f90b40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:41.718 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.722+0000 7f90cf400640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f90b40a3cc0 0x7f90b40a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f90d0103060 tx=0x7f90c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:41.721 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.726+0000 7f90ccc00640 1 -- 172.21.15.81:0/3901931386 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90b809d190 con 0x7f90d0103200 2024-04-12T21:13:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:41 smithi153 ceph-mon[36569]: from='client.24844 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:41 smithi153 ceph-mon[36569]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 426 B/s wr, 0 op/s 2024-04-12T21:13:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:41 smithi153 ceph-mon[36569]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:41 smithi153 ceph-mon[36569]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-12T21:13:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:41 smithi153 ceph-mon[36569]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:41 smithi104 ceph-mon[36624]: from='client.24844 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:41 smithi104 ceph-mon[36624]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 426 B/s wr, 0 op/s 2024-04-12T21:13:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:41 smithi104 ceph-mon[36624]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:41 smithi104 ceph-mon[36624]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-12T21:13:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:41 smithi104 ceph-mon[36624]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:41 smithi081 ceph-mon[31453]: from='client.24844 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:41 smithi081 ceph-mon[31453]: pgmap v328: 129 pgs: 129 active+clean; 581 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 426 B/s wr, 0 op/s 2024-04-12T21:13:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:41 smithi081 ceph-mon[31453]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:41 smithi081 ceph-mon[31453]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-12T21:13:41.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:41 smithi081 ceph-mon[31453]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:13:41.925 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.930+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f90d01a2e20 con 0x7f90d0103200 2024-04-12T21:13:41.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.934+0000 7f90ccc00640 1 -- 172.21.15.81:0/3901931386 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7f90b80bd0d0 con 0x7f90d0103200 2024-04-12T21:13:41.930 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:41.930 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:41.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f90b40a3cc0 msgr2=0x7f90b40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f90b40a3cc0 0x7f90b40a6180 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f90d0103060 tx=0x7f90c0005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:41.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103200 msgr2=0x7f90d019aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:41.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103200 0x7f90d019aa70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f90b800e9f0 tx=0x7f90b800edf0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 shutdown_connections 2024-04-12T21:13:41.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f90b40a3cc0 0x7f90b40a6180 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f90d0103bc0 0x7f90d01a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.934 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f90d0103200 0x7f90d019aa70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 --2- 172.21.15.81:0/3901931386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90d0102010 0x7f90d019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:41.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 >> 172.21.15.81:0/3901931386 conn(0x7f90d00fd5a0 msgr2=0x7f90d00741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:41.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.937+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 shutdown_connections 2024-04-12T21:13:41.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:41.938+0000 7f90d6200640 1 -- 172.21.15.81:0/3901931386 wait complete. 2024-04-12T21:13:41.952 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-12T21:13:41.952 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-12T21:13:42.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 -- 172.21.15.81:0/2389215794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105bf0 msgr2=0x7f535410c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.161 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 --2- 172.21.15.81:0/2389215794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105bf0 0x7f535410c480 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f534c009ca0 tx=0x7f534c02c100 comp rx=0 tx=0).stop 2024-04-12T21:13:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 -- 172.21.15.81:0/2389215794 shutdown_connections 2024-04-12T21:13:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 --2- 172.21.15.81:0/2389215794 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105bf0 0x7f535410c480 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 --2- 172.21.15.81:0/2389215794 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5354105230 0x7f53541056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 --2- 172.21.15.81:0/2389215794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5354104040 0x7f5354104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 -- 172.21.15.81:0/2389215794 >> 172.21.15.81:0/2389215794 conn(0x7f53540ff5d0 msgr2=0x7f53541019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 -- 172.21.15.81:0/2389215794 shutdown_connections 2024-04-12T21:13:42.162 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.165+0000 7f535ae00640 1 -- 172.21.15.81:0/2389215794 wait complete. 2024-04-12T21:13:42.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.166+0000 7f535ae00640 1 Processor -- start 2024-04-12T21:13:42.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.166+0000 7f535ae00640 1 -- start start 2024-04-12T21:13:42.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.166+0000 7f535ae00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5354104040 0x7f535419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:42.164 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.166+0000 7f535ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105230 0x7f535419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.166+0000 7f535ae00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5354105bf0 0x7f53541a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.166+0000 7f535ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f535410fcd0 con 0x7f5354105230 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.166+0000 7f535ae00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f535410fb50 con 0x7f5354104040 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.166+0000 7f535ae00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f535410fe50 con 0x7f5354105bf0 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105230 0x7f535419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5354104040 0x7f535419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105230 0x7f535419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46554/0 (socket says 172.21.15.81:46554) 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359000640 1 -- 172.21.15.81:0/1137296357 learned_addr learned my addr 172.21.15.81:0/1137296357 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f535a400640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5354105bf0 0x7f53541a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359000640 1 -- 172.21.15.81:0/1137296357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5354105bf0 msgr2=0x7f53541a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359000640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5354105bf0 0x7f53541a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359000640 1 -- 172.21.15.81:0/1137296357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5354104040 msgr2=0x7f535419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.165 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359000640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5354104040 0x7f535419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359000640 1 -- 172.21.15.81:0/1137296357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53541a1ee0 con 0x7f5354105230 2024-04-12T21:13:42.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.167+0000 7f5359a00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5354104040 0x7f535419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:42.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.168+0000 7f5359000640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105230 0x7f535419a800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f534400e9f0 tx=0x7f534400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:42.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.168+0000 7f5352000640 1 -- 172.21.15.81:0/1137296357 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534400cd80 con 0x7f5354105230 2024-04-12T21:13:42.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.168+0000 7f5352000640 1 -- 172.21.15.81:0/1137296357 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5344010910 con 0x7f5354105230 2024-04-12T21:13:42.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.168+0000 7f5352000640 1 -- 172.21.15.81:0/1137296357 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344018990 con 0x7f5354105230 2024-04-12T21:13:42.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.168+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53541a21d0 con 0x7f5354105230 2024-04-12T21:13:42.166 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.168+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53541a2720 con 0x7f5354105230 2024-04-12T21:13:42.170 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.169+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f535410b360 con 0x7f5354105230 2024-04-12T21:13:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.171+0000 7f5352000640 1 -- 172.21.15.81:0/1137296357 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5344010ac0 con 0x7f5354105230 2024-04-12T21:13:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.172+0000 7f5352000640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f533c0a3c40 0x7f533c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.173+0000 7f5352000640 1 -- 172.21.15.81:0/1137296357 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f53440d0b50 con 0x7f5354105230 2024-04-12T21:13:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.173+0000 7f5359a00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f533c0a3c40 0x7f533c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:42.171 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.173+0000 7f5359a00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f533c0a3c40 0x7f533c0a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5348005fd0 tx=0x7f534800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:42.173 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.178+0000 7f5352000640 1 -- 172.21.15.81:0/1137296357 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f534409cae0 con 0x7f5354105230 2024-04-12T21:13:42.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.374+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5354062400 con 0x7f533c0a3c40 2024-04-12T21:13:42.383 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.387+0000 7f5352000640 1 -- 172.21.15.81:0/1137296357 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340238 (secure 0 0 0) 0x7f5354062400 con 0x7f533c0a3c40 2024-04-12T21:13:42.388 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:42.390 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-04-12T21:13:42.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f533c0a3c40 msgr2=0x7f533c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f533c0a3c40 0x7f533c0a6100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5348005fd0 tx=0x7f534800b390 comp rx=0 tx=0).stop 2024-04-12T21:13:42.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105230 msgr2=0x7f535419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.392 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105230 0x7f535419a800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f534400e9f0 tx=0x7f534400eeb0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 shutdown_connections 2024-04-12T21:13:42.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f533c0a3c40 0x7f533c0a6100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5354105bf0 0x7f53541a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5354105230 0x7f535419a800 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 --2- 172.21.15.81:0/1137296357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5354104040 0x7f535419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 >> 172.21.15.81:0/1137296357 conn(0x7f53540ff5d0 msgr2=0x7f5354108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:42.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 shutdown_connections 2024-04-12T21:13:42.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.393+0000 7f535ae00640 1 -- 172.21.15.81:0/1137296357 wait complete. 2024-04-12T21:13:42.405 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-12T21:13:42.329933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631871,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":243,"num_write_kb":2999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375068,"kb_used_data":13400,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566756,"statfs":{"total":1151940427776,"available":1151556358144,"internally_reserved":0,"allocated":13721600,"data_stored":7553887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39093,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":35,"num_write":17,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006258"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.686261+0000","last_change":"2024-04-12T21:13:13.157179+0000","last_active":"2024-04-12T21:13:18.686261+0000","last_peered":"2024-04-12T21:13:18.686261+0000","last_clean":"2024-04-12T21:13:18.686261+0000","last_became_active":"2024-04-12T21:13:13.156826+0000","last_became_peered":"2024-04-12T21:13:13.156826+0000","last_unstale":"2024-04-12T21:13:18.686261+0000","last_undegraded":"2024-04-12T21:13:18.686261+0000","last_fullsized":"2024-04-12T21:13:18.686261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:51:25.310758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536786+0000","last_change":"2024-04-12T21:13:13.150917+0000","last_active":"2024-04-12T21:13:19.536786+0000","last_peered":"2024-04-12T21:13:19.536786+0000","last_clean":"2024-04-12T21:13:19.536786+0000","last_became_active":"2024-04-12T21:13:13.150491+0000","last_became_peered":"2024-04-12T21:13:13.150491+0000","last_unstale":"2024-04-12T21:13:19.536786+0000","last_undegraded":"2024-04-12T21:13:19.536786+0000","last_fullsized":"2024-04-12T21:13:19.536786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:08:04.486913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536320+0000","last_change":"2024-04-12T21:13:13.150355+0000","last_active":"2024-04-12T21:13:19.536320+0000","last_peered":"2024-04-12T21:13:19.536320+0000","last_clean":"2024-04-12T21:13:19.536320+0000","last_became_active":"2024-04-12T21:13:13.149979+0000","last_became_peered":"2024-04-12T21:13:13.149979+0000","last_unstale":"2024-04-12T21:13:19.536320+0000","last_undegraded":"2024-04-12T21:13:19.536320+0000","last_fullsized":"2024-04-12T21:13:19.536320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:02:56.175842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533647+0000","last_change":"2024-04-12T21:13:13.132074+0000","last_active":"2024-04-12T21:13:19.533647+0000","last_peered":"2024-04-12T21:13:19.533647+0000","last_clean":"2024-04-12T21:13:19.533647+0000","last_became_active":"2024-04-12T21:13:13.130234+0000","last_became_peered":"2024-04-12T21:13:13.130234+0000","last_unstale":"2024-04-12T21:13:19.533647+0000","last_undegraded":"2024-04-12T21:13:19.533647+0000","last_fullsized":"2024-04-12T21:13:19.533647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:18:51.706215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533958+0000","last_change":"2024-04-12T21:13:13.146577+0000","last_active":"2024-04-12T21:13:19.533958+0000","last_peered":"2024-04-12T21:13:19.533958+0000","last_clean":"2024-04-12T21:13:19.533958+0000","last_became_active":"2024-04-12T21:13:13.146162+0000","last_became_peered":"2024-04-12T21:13:13.146162+0000","last_unstale":"2024-04-12T21:13:19.533958+0000","last_undegraded":"2024-04-12T21:13:19.533958+0000","last_fullsized":"2024-04-12T21:13:19.533958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:43:10.267515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247478+0000","last_change":"2024-04-12T21:13:13.137103+0000","last_active":"2024-04-12T21:13:19.247478+0000","last_peered":"2024-04-12T21:13:19.247478+0000","last_clean":"2024-04-12T21:13:19.247478+0000","last_became_active":"2024-04-12T21:13:13.136750+0000","last_became_peered":"2024-04-12T21:13:13.136750+0000","last_unstale":"2024-04-12T21:13:19.247478+0000","last_undegraded":"2024-04-12T21:13:19.247478+0000","last_fullsized":"2024-04-12T21:13:19.247478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:11:58.538179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534916+0000","last_change":"2024-04-12T21:13:13.135440+0000","last_active":"2024-04-12T21:13:19.534916+0000","last_peered":"2024-04-12T21:13:19.534916+0000","last_clean":"2024-04-12T21:13:19.534916+0000","last_became_active":"2024-04-12T21:13:13.135027+0000","last_became_peered":"2024-04-12T21:13:13.135027+0000","last_unstale":"2024-04-12T21:13:19.534916+0000","last_undegraded":"2024-04-12T21:13:19.534916+0000","last_fullsized":"2024-04-12T21:13:19.534916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:37:50.875158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535051+0000","last_change":"2024-04-12T21:13:13.143059+0000","last_active":"2024-04-12T21:13:19.535051+0000","last_peered":"2024-04-12T21:13:19.535051+0000","last_clean":"2024-04-12T21:13:19.535051+0000","last_became_active":"2024-04-12T21:13:13.142698+0000","last_became_peered":"2024-04-12T21:13:13.142698+0000","last_unstale":"2024-04-12T21:13:19.535051+0000","last_undegraded":"2024-04-12T21:13:19.535051+0000","last_fullsized":"2024-04-12T21:13:19.535051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:30:05.170007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011613+0000","last_change":"2024-04-12T21:13:13.128917+0000","last_active":"2024-04-12T21:13:19.011613+0000","last_peered":"2024-04-12T21:13:19.011613+0000","last_clean":"2024-04-12T21:13:19.011613+0000","last_became_active":"2024-04-12T21:13:13.128166+0000","last_became_peered":"2024-04-12T21:13:13.128166+0000","last_unstale":"2024-04-12T21:13:19.011613+0000","last_undegraded":"2024-04-12T21:13:19.011613+0000","last_fullsized":"2024-04-12T21:13:19.011613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:49:41.282798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247930+0000","last_change":"2024-04-12T21:13:13.130833+0000","last_active":"2024-04-12T21:13:19.247930+0000","last_peered":"2024-04-12T21:13:19.247930+0000","last_clean":"2024-04-12T21:13:19.247930+0000","last_became_active":"2024-04-12T21:13:13.130506+0000","last_became_peered":"2024-04-12T21:13:13.130506+0000","last_unstale":"2024-04-12T21:13:19.247930+0000","last_undegraded":"2024-04-12T21:13:19.247930+0000","last_fullsized":"2024-04-12T21:13:19.247930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:08:30.527381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535373+0000","last_change":"2024-04-12T21:13:13.129682+0000","last_active":"2024-04-12T21:13:19.535373+0000","last_peered":"2024-04-12T21:13:19.535373+0000","last_clean":"2024-04-12T21:13:19.535373+0000","last_became_active":"2024-04-12T21:13:13.129364+0000","last_became_peered":"2024-04-12T21:13:13.129364+0000","last_unstale":"2024-04-12T21:13:19.535373+0000","last_undegraded":"2024-04-12T21:13:19.535373+0000","last_fullsized":"2024-04-12T21:13:19.535373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:19:48.442418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533584+0000","last_change":"2024-04-12T21:13:13.133083+0000","last_active":"2024-04-12T21:13:19.533584+0000","last_peered":"2024-04-12T21:13:19.533584+0000","last_clean":"2024-04-12T21:13:19.533584+0000","last_became_active":"2024-04-12T21:13:13.132679+0000","last_became_peered":"2024-04-12T21:13:13.132679+0000","last_unstale":"2024-04-12T21:13:19.533584+0000","last_undegraded":"2024-04-12T21:13:19.533584+0000","last_fullsized":"2024-04-12T21:13:19.533584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:44:46.344475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.605631+0000","last_change":"2024-04-12T21:13:13.137692+0000","last_active":"2024-04-12T21:13:18.605631+0000","last_peered":"2024-04-12T21:13:18.605631+0000","last_clean":"2024-04-12T21:13:18.605631+0000","last_became_active":"2024-04-12T21:13:13.128523+0000","last_became_peered":"2024-04-12T21:13:13.128523+0000","last_unstale":"2024-04-12T21:13:18.605631+0000","last_undegraded":"2024-04-12T21:13:18.605631+0000","last_fullsized":"2024-04-12T21:13:18.605631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:52:30.847637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535951+0000","last_change":"2024-04-12T21:13:13.149749+0000","last_active":"2024-04-12T21:13:19.535951+0000","last_peered":"2024-04-12T21:13:19.535951+0000","last_clean":"2024-04-12T21:13:19.535951+0000","last_became_active":"2024-04-12T21:13:13.149369+0000","last_became_peered":"2024-04-12T21:13:13.149369+0000","last_unstale":"2024-04-12T21:13:19.535951+0000","last_undegraded":"2024-04-12T21:13:19.535951+0000","last_fullsized":"2024-04-12T21:13:19.535951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:20.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535600+0000","last_change":"2024-04-12T21:13:13.151510+0000","last_active":"2024-04-12T21:13:19.535600+0000","last_peered":"2024-04-12T21:13:19.535600+0000","last_clean":"2024-04-12T21:13:19.535600+0000","last_became_active":"2024-04-12T21:13:13.150620+0000","last_became_peered":"2024-04-12T21:13:13.150620+0000","last_unstale":"2024-04-12T21:13:19.535600+0000","last_undegraded":"2024-04-12T21:13:19.535600+0000","last_fullsized":"2024-04-12T21:13:19.535600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:59:14.772221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.010994+0000","last_change":"2024-04-12T21:13:13.156421+0000","last_active":"2024-04-12T21:13:19.010994+0000","last_peered":"2024-04-12T21:13:19.010994+0000","last_clean":"2024-04-12T21:13:19.010994+0000","last_became_active":"2024-04-12T21:13:13.156042+0000","last_became_peered":"2024-04-12T21:13:13.156042+0000","last_unstale":"2024-04-12T21:13:19.010994+0000","last_undegraded":"2024-04-12T21:13:19.010994+0000","last_fullsized":"2024-04-12T21:13:19.010994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:53:38.234230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.686130+0000","last_change":"2024-04-12T21:13:13.156986+0000","last_active":"2024-04-12T21:13:18.686130+0000","last_peered":"2024-04-12T21:13:18.686130+0000","last_clean":"2024-04-12T21:13:18.686130+0000","last_became_active":"2024-04-12T21:13:13.156705+0000","last_became_peered":"2024-04-12T21:13:13.156705+0000","last_unstale":"2024-04-12T21:13:18.686130+0000","last_undegraded":"2024-04-12T21:13:18.686130+0000","last_fullsized":"2024-04-12T21:13:18.686130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:49:42.442696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.607922+0000","last_change":"2024-04-12T21:13:13.136259+0000","last_active":"2024-04-12T21:13:18.607922+0000","last_peered":"2024-04-12T21:13:18.607922+0000","last_clean":"2024-04-12T21:13:18.607922+0000","last_became_active":"2024-04-12T21:13:13.135865+0000","last_became_peered":"2024-04-12T21:13:13.135865+0000","last_unstale":"2024-04-12T21:13:18.607922+0000","last_undegraded":"2024-04-12T21:13:18.607922+0000","last_fullsized":"2024-04-12T21:13:18.607922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:30:35.934682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601358+0000","last_change":"2024-04-12T21:13:13.140460+0000","last_active":"2024-04-12T21:13:18.601358+0000","last_peered":"2024-04-12T21:13:18.601358+0000","last_clean":"2024-04-12T21:13:18.601358+0000","last_became_active":"2024-04-12T21:13:13.140115+0000","last_became_peered":"2024-04-12T21:13:13.140115+0000","last_unstale":"2024-04-12T21:13:18.601358+0000","last_undegraded":"2024-04-12T21:13:18.601358+0000","last_fullsized":"2024-04-12T21:13:18.601358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:56:08.880106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601079+0000","last_change":"2024-04-12T21:13:17.606904+0000","last_active":"2024-04-12T21:13:18.601079+0000","last_peered":"2024-04-12T21:13:18.601079+0000","last_clean":"2024-04-12T21:13:18.601079+0000","last_became_active":"2024-04-12T21:13:17.606390+0000","last_became_peered":"2024-04-12T21:13:17.606390+0000","last_unstale":"2024-04-12T21:13:18.601079+0000","last_undegraded":"2024-04-12T21:13:18.601079+0000","last_fullsized":"2024-04-12T21:13:18.601079+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:56:13.440022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.608318+0000","last_change":"2024-04-12T21:13:12.535260+0000","last_active":"2024-04-12T21:13:18.608318+0000","last_peered":"2024-04-12T21:13:18.608318+0000","last_clean":"2024-04-12T21:13:18.608318+0000","last_became_active":"2024-04-12T21:13:12.534798+0000","last_became_peered":"2024-04-12T21:13:12.534798+0000","last_unstale":"2024-04-12T21:13:18.608318+0000","last_undegraded":"2024-04-12T21:13:18.608318+0000","last_fullsized":"2024-04-12T21:13:18.608318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:33:13.954242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.538074+0000","last_change":"2024-04-12T21:13:13.142701+0000","last_active":"2024-04-12T21:13:19.538074+0000","last_peered":"2024-04-12T21:13:19.538074+0000","last_clean":"2024-04-12T21:13:19.538074+0000","last_became_active":"2024-04-12T21:13:13.142329+0000","last_became_peered":"2024-04-12T21:13:13.142329+0000","last_unstale":"2024-04-12T21:13:19.538074+0000","last_undegraded":"2024-04-12T21:13:19.538074+0000","last_fullsized":"2024-04-12T21:13:19.538074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:59:02.356770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533177+0000","last_change":"2024-04-12T21:13:17.604050+0000","last_active":"2024-04-12T21:13:19.533177+0000","last_peered":"2024-04-12T21:13:19.533177+0000","last_clean":"2024-04-12T21:13:19.533177+0000","last_became_active":"2024-04-12T21:13:17.603628+0000","last_became_peered":"2024-04-12T21:13:17.603628+0000","last_unstale":"2024-04-12T21:13:19.533177+0000","last_undegraded":"2024-04-12T21:13:19.533177+0000","last_fullsized":"2024-04-12T21:13:19.533177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:24:24.986986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.538890+0000","last_change":"2024-04-12T21:13:12.546263+0000","last_active":"2024-04-12T21:13:19.538890+0000","last_peered":"2024-04-12T21:13:19.538890+0000","last_clean":"2024-04-12T21:13:19.538890+0000","last_became_active":"2024-04-12T21:13:12.545710+0000","last_became_peered":"2024-04-12T21:13:12.545710+0000","last_unstale":"2024-04-12T21:13:19.538890+0000","last_undegraded":"2024-04-12T21:13:19.538890+0000","last_fullsized":"2024-04-12T21:13:19.538890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:18:33.732571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536292+0000","last_change":"2024-04-12T21:13:13.156478+0000","last_active":"2024-04-12T21:13:19.536292+0000","last_peered":"2024-04-12T21:13:19.536292+0000","last_clean":"2024-04-12T21:13:19.536292+0000","last_became_active":"2024-04-12T21:13:13.155485+0000","last_became_peered":"2024-04-12T21:13:13.155485+0000","last_unstale":"2024-04-12T21:13:19.536292+0000","last_undegraded":"2024-04-12T21:13:19.536292+0000","last_fullsized":"2024-04-12T21:13:19.536292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:11:24.161377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533356+0000","last_change":"2024-04-12T21:13:17.607327+0000","last_active":"2024-04-12T21:13:19.533356+0000","last_peered":"2024-04-12T21:13:19.533356+0000","last_clean":"2024-04-12T21:13:19.533356+0000","last_became_active":"2024-04-12T21:13:17.606949+0000","last_became_peered":"2024-04-12T21:13:17.606949+0000","last_unstale":"2024-04-12T21:13:19.533356+0000","last_undegraded":"2024-04-12T21:13:19.533356+0000","last_fullsized":"2024-04-12T21:13:19.533356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:43:27.855461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534426+0000","last_change":"2024-04-12T21:13:12.533082+0000","last_active":"2024-04-12T21:13:19.534426+0000","last_peered":"2024-04-12T21:13:19.534426+0000","last_clean":"2024-04-12T21:13:19.534426+0000","last_became_active":"2024-04-12T21:13:12.532739+0000","last_became_peered":"2024-04-12T21:13:12.532739+0000","last_unstale":"2024-04-12T21:13:19.534426+0000","last_undegraded":"2024-04-12T21:13:19.534426+0000","last_fullsized":"2024-04-12T21:13:19.534426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:40:01.885223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011288+0000","last_change":"2024-04-12T21:13:13.136718+0000","last_active":"2024-04-12T21:13:19.011288+0000","last_peered":"2024-04-12T21:13:19.011288+0000","last_clean":"2024-04-12T21:13:19.011288+0000","last_became_active":"2024-04-12T21:13:13.136357+0000","last_became_peered":"2024-04-12T21:13:13.136357+0000","last_unstale":"2024-04-12T21:13:19.011288+0000","last_undegraded":"2024-04-12T21:13:19.011288+0000","last_fullsized":"2024-04-12T21:13:19.011288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:08:57.380556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533366+0000","last_change":"2024-04-12T21:13:17.601381+0000","last_active":"2024-04-12T21:13:19.533366+0000","last_peered":"2024-04-12T21:13:19.533366+0000","last_clean":"2024-04-12T21:13:19.533366+0000","last_became_active":"2024-04-12T21:13:17.600868+0000","last_became_peered":"2024-04-12T21:13:17.600868+0000","last_unstale":"2024-04-12T21:13:19.533366+0000","last_undegraded":"2024-04-12T21:13:19.533366+0000","last_fullsized":"2024-04-12T21:13:19.533366+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:53:36.234779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534532+0000","last_change":"2024-04-12T21:13:12.528285+0000","last_active":"2024-04-12T21:13:19.534532+0000","last_peered":"2024-04-12T21:13:19.534532+0000","last_clean":"2024-04-12T21:13:19.534532+0000","last_became_active":"2024-04-12T21:13:12.527915+0000","last_became_peered":"2024-04-12T21:13:12.527915+0000","last_unstale":"2024-04-12T21:13:19.534532+0000","last_undegraded":"2024-04-12T21:13:19.534532+0000","last_fullsized":"2024-04-12T21:13:19.534532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:44.119836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534987+0000","last_change":"2024-04-12T21:13:13.128654+0000","last_active":"2024-04-12T21:13:19.534987+0000","last_peered":"2024-04-12T21:13:19.534987+0000","last_clean":"2024-04-12T21:13:19.534987+0000","last_became_active":"2024-04-12T21:13:13.128274+0000","last_became_peered":"2024-04-12T21:13:13.128274+0000","last_unstale":"2024-04-12T21:13:19.534987+0000","last_undegraded":"2024-04-12T21:13:19.534987+0000","last_fullsized":"2024-04-12T21:13:19.534987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:48:08.827157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601427+0000","last_change":"2024-04-12T21:13:17.594360+0000","last_active":"2024-04-12T21:13:18.601427+0000","last_peered":"2024-04-12T21:13:18.601427+0000","last_clean":"2024-04-12T21:13:18.601427+0000","last_became_active":"2024-04-12T21:13:17.593924+0000","last_became_peered":"2024-04-12T21:13:17.593924+0000","last_unstale":"2024-04-12T21:13:18.601427+0000","last_undegraded":"2024-04-12T21:13:18.601427+0000","last_fullsized":"2024-04-12T21:13:18.601427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:42:14.269657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534575+0000","last_change":"2024-04-12T21:13:12.532481+0000","last_active":"2024-04-12T21:13:19.534575+0000","last_peered":"2024-04-12T21:13:19.534575+0000","last_clean":"2024-04-12T21:13:19.534575+0000","last_became_active":"2024-04-12T21:13:12.532126+0000","last_became_peered":"2024-04-12T21:13:12.532126+0000","last_unstale":"2024-04-12T21:13:19.534575+0000","last_undegraded":"2024-04-12T21:13:19.534575+0000","last_fullsized":"2024-04-12T21:13:19.534575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:19:52.510076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534475+0000","last_change":"2024-04-12T21:13:13.138290+0000","last_active":"2024-04-12T21:13:19.534475+0000","last_peered":"2024-04-12T21:13:19.534475+0000","last_clean":"2024-04-12T21:13:19.534475+0000","last_became_active":"2024-04-12T21:13:13.137854+0000","last_became_peered":"2024-04-12T21:13:13.137854+0000","last_unstale":"2024-04-12T21:13:19.534475+0000","last_undegraded":"2024-04-12T21:13:19.534475+0000","last_fullsized":"2024-04-12T21:13:19.534475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:27:45.455704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601280+0000","last_change":"2024-04-12T21:13:17.605453+0000","last_active":"2024-04-12T21:13:18.601280+0000","last_peered":"2024-04-12T21:13:18.601280+0000","last_clean":"2024-04-12T21:13:18.601280+0000","last_became_active":"2024-04-12T21:13:17.604918+0000","last_became_peered":"2024-04-12T21:13:17.604918+0000","last_unstale":"2024-04-12T21:13:18.601280+0000","last_undegraded":"2024-04-12T21:13:18.601280+0000","last_fullsized":"2024-04-12T21:13:18.601280+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:14:40.800326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247423+0000","last_change":"2024-04-12T21:13:12.539083+0000","last_active":"2024-04-12T21:13:19.247423+0000","last_peered":"2024-04-12T21:13:19.247423+0000","last_clean":"2024-04-12T21:13:19.247423+0000","last_became_active":"2024-04-12T21:13:12.538771+0000","last_became_peered":"2024-04-12T21:13:12.538771+0000","last_unstale":"2024-04-12T21:13:19.247423+0000","last_undegraded":"2024-04-12T21:13:19.247423+0000","last_fullsized":"2024-04-12T21:13:19.247423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:31.560665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534689+0000","last_change":"2024-04-12T21:13:13.146087+0000","last_active":"2024-04-12T21:13:19.534689+0000","last_peered":"2024-04-12T21:13:19.534689+0000","last_clean":"2024-04-12T21:13:19.534689+0000","last_became_active":"2024-04-12T21:13:13.145450+0000","last_became_peered":"2024-04-12T21:13:13.145450+0000","last_unstale":"2024-04-12T21:13:19.534689+0000","last_undegraded":"2024-04-12T21:13:19.534689+0000","last_fullsized":"2024-04-12T21:13:19.534689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:03:07.727779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.686197+0000","last_change":"2024-04-12T21:13:17.607603+0000","last_active":"2024-04-12T21:13:18.686197+0000","last_peered":"2024-04-12T21:13:18.686197+0000","last_clean":"2024-04-12T21:13:18.686197+0000","last_became_active":"2024-04-12T21:13:17.606972+0000","last_became_peered":"2024-04-12T21:13:17.606972+0000","last_unstale":"2024-04-12T21:13:18.686197+0000","last_undegraded":"2024-04-12T21:13:18.686197+0000","last_fullsized":"2024-04-12T21:13:18.686197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:16:37.593158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604108+0000","last_change":"2024-04-12T21:13:12.541405+0000","last_active":"2024-04-12T21:13:18.604108+0000","last_peered":"2024-04-12T21:13:18.604108+0000","last_clean":"2024-04-12T21:13:18.604108+0000","last_became_active":"2024-04-12T21:13:12.540612+0000","last_became_peered":"2024-04-12T21:13:12.540612+0000","last_unstale":"2024-04-12T21:13:18.604108+0000","last_undegraded":"2024-04-12T21:13:18.604108+0000","last_fullsized":"2024-04-12T21:13:18.604108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:38:10.749857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535348+0000","last_change":"2024-04-12T21:13:13.138447+0000","last_active":"2024-04-12T21:13:19.535348+0000","last_peered":"2024-04-12T21:13:19.535348+0000","last_clean":"2024-04-12T21:13:19.535348+0000","last_became_active":"2024-04-12T21:13:13.137990+0000","last_became_peered":"2024-04-12T21:13:13.137990+0000","last_unstale":"2024-04-12T21:13:19.535348+0000","last_undegraded":"2024-04-12T21:13:19.535348+0000","last_fullsized":"2024-04-12T21:13:19.535348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:58:50.381257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536769+0000","last_change":"2024-04-12T21:13:17.614003+0000","last_active":"2024-04-12T21:13:19.536769+0000","last_peered":"2024-04-12T21:13:19.536769+0000","last_clean":"2024-04-12T21:13:19.536769+0000","last_became_active":"2024-04-12T21:13:17.613517+0000","last_became_peered":"2024-04-12T21:13:17.613517+0000","last_unstale":"2024-04-12T21:13:19.536769+0000","last_undegraded":"2024-04-12T21:13:19.536769+0000","last_fullsized":"2024-04-12T21:13:19.536769+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:33.590816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533995+0000","last_change":"2024-04-12T21:13:12.530503+0000","last_active":"2024-04-12T21:13:19.533995+0000","last_peered":"2024-04-12T21:13:19.533995+0000","last_clean":"2024-04-12T21:13:19.533995+0000","last_became_active":"2024-04-12T21:13:12.530161+0000","last_became_peered":"2024-04-12T21:13:12.530161+0000","last_unstale":"2024-04-12T21:13:19.533995+0000","last_undegraded":"2024-04-12T21:13:19.533995+0000","last_fullsized":"2024-04-12T21:13:19.533995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:06:18.363283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604050+0000","last_change":"2024-04-12T21:13:13.146401+0000","last_active":"2024-04-12T21:13:18.604050+0000","last_peered":"2024-04-12T21:13:18.604050+0000","last_clean":"2024-04-12T21:13:18.604050+0000","last_became_active":"2024-04-12T21:13:13.145527+0000","last_became_peered":"2024-04-12T21:13:13.145527+0000","last_unstale":"2024-04-12T21:13:18.604050+0000","last_undegraded":"2024-04-12T21:13:18.604050+0000","last_fullsized":"2024-04-12T21:13:18.604050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:40:41.302305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535689+0000","last_change":"2024-04-12T21:13:17.609141+0000","last_active":"2024-04-12T21:13:19.535689+0000","last_peered":"2024-04-12T21:13:19.535689+0000","last_clean":"2024-04-12T21:13:19.535689+0000","last_became_active":"2024-04-12T21:13:17.603368+0000","last_became_peered":"2024-04-12T21:13:17.603368+0000","last_unstale":"2024-04-12T21:13:19.535689+0000","last_undegraded":"2024-04-12T21:13:19.535689+0000","last_fullsized":"2024-04-12T21:13:19.535689+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:07:54.235284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247866+0000","last_change":"2024-04-12T21:13:12.540388+0000","last_active":"2024-04-12T21:13:19.247866+0000","last_peered":"2024-04-12T21:13:19.247866+0000","last_clean":"2024-04-12T21:13:19.247866+0000","last_became_active":"2024-04-12T21:13:12.540033+0000","last_became_peered":"2024-04-12T21:13:12.540033+0000","last_unstale":"2024-04-12T21:13:19.247866+0000","last_undegraded":"2024-04-12T21:13:19.247866+0000","last_fullsized":"2024-04-12T21:13:19.247866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:46:53.785074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011582+0000","last_change":"2024-04-12T21:13:13.137208+0000","last_active":"2024-04-12T21:13:19.011582+0000","last_peered":"2024-04-12T21:13:19.011582+0000","last_clean":"2024-04-12T21:13:19.011582+0000","last_became_active":"2024-04-12T21:13:13.136866+0000","last_became_peered":"2024-04-12T21:13:13.136866+0000","last_unstale":"2024-04-12T21:13:19.011582+0000","last_undegraded":"2024-04-12T21:13:19.011582+0000","last_fullsized":"2024-04-12T21:13:19.011582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:48:36.655808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604760+0000","last_change":"2024-04-12T21:13:17.617564+0000","last_active":"2024-04-12T21:13:18.604760+0000","last_peered":"2024-04-12T21:13:18.604760+0000","last_clean":"2024-04-12T21:13:18.604760+0000","last_became_active":"2024-04-12T21:13:17.615980+0000","last_became_peered":"2024-04-12T21:13:17.615980+0000","last_unstale":"2024-04-12T21:13:18.604760+0000","last_undegraded":"2024-04-12T21:13:18.604760+0000","last_fullsized":"2024-04-12T21:13:18.604760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:22:38.370357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535118+0000","last_change":"2024-04-12T21:13:12.532732+0000","last_active":"2024-04-12T21:13:19.535118+0000","last_peered":"2024-04-12T21:13:19.535118+0000","last_clean":"2024-04-12T21:13:19.535118+0000","last_became_active":"2024-04-12T21:13:12.532336+0000","last_became_peered":"2024-04-12T21:13:12.532336+0000","last_unstale":"2024-04-12T21:13:19.535118+0000","last_undegraded":"2024-04-12T21:13:19.535118+0000","last_fullsized":"2024-04-12T21:13:19.535118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:43:20.472987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.537029+0000","last_change":"2024-04-12T21:13:13.152737+0000","last_active":"2024-04-12T21:13:19.537029+0000","last_peered":"2024-04-12T21:13:19.537029+0000","last_clean":"2024-04-12T21:13:19.537029+0000","last_became_active":"2024-04-12T21:13:13.150914+0000","last_became_peered":"2024-04-12T21:13:13.150914+0000","last_unstale":"2024-04-12T21:13:19.537029+0000","last_undegraded":"2024-04-12T21:13:19.537029+0000","last_fullsized":"2024-04-12T21:13:19.537029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:51:29.020746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533641+0000","last_change":"2024-04-12T21:13:17.604826+0000","last_active":"2024-04-12T21:13:19.533641+0000","last_peered":"2024-04-12T21:13:19.533641+0000","last_clean":"2024-04-12T21:13:19.533641+0000","last_became_active":"2024-04-12T21:13:17.604427+0000","last_became_peered":"2024-04-12T21:13:17.604427+0000","last_unstale":"2024-04-12T21:13:19.533641+0000","last_undegraded":"2024-04-12T21:13:19.533641+0000","last_fullsized":"2024-04-12T21:13:19.533641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:04:55.602539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534864+0000","last_change":"2024-04-12T21:13:12.542641+0000","last_active":"2024-04-12T21:13:19.534864+0000","last_peered":"2024-04-12T21:13:19.534864+0000","last_clean":"2024-04-12T21:13:19.534864+0000","last_became_active":"2024-04-12T21:13:12.542291+0000","last_became_peered":"2024-04-12T21:13:12.542291+0000","last_unstale":"2024-04-12T21:13:19.534864+0000","last_undegraded":"2024-04-12T21:13:19.534864+0000","last_fullsized":"2024-04-12T21:13:19.534864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:37:00.186153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011383+0000","last_change":"2024-04-12T21:13:13.129280+0000","last_active":"2024-04-12T21:13:19.011383+0000","last_peered":"2024-04-12T21:13:19.011383+0000","last_clean":"2024-04-12T21:13:19.011383+0000","last_became_active":"2024-04-12T21:13:13.128881+0000","last_became_peered":"2024-04-12T21:13:13.128881+0000","last_unstale":"2024-04-12T21:13:19.011383+0000","last_undegraded":"2024-04-12T21:13:19.011383+0000","last_fullsized":"2024-04-12T21:13:19.011383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:00:23.629024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604875+0000","last_change":"2024-04-12T21:13:17.606533+0000","last_active":"2024-04-12T21:13:18.604875+0000","last_peered":"2024-04-12T21:13:18.604875+0000","last_clean":"2024-04-12T21:13:18.604875+0000","last_became_active":"2024-04-12T21:13:17.606092+0000","last_became_peered":"2024-04-12T21:13:17.606092+0000","last_unstale":"2024-04-12T21:13:18.604875+0000","last_undegraded":"2024-04-12T21:13:18.604875+0000","last_fullsized":"2024-04-12T21:13:18.604875+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:55:56.080659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'21","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:40.534202+0000","last_change":"2024-04-12T21:13:12.531797+0000","last_active":"2024-04-12T21:13:40.534202+0000","last_peered":"2024-04-12T21:13:40.534202+0000","last_clean":"2024-04-12T21:13:40.534202+0000","last_became_active":"2024-04-12T21:13:12.531346+0000","last_became_peered":"2024-04-12T21:13:12.531346+0000","last_unstale":"2024-04-12T21:13:40.534202+0000","last_undegraded":"2024-04-12T21:13:40.534202+0000","last_fullsized":"2024-04-12T21:13:40.534202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:59:32.429211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40471,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":29,"num_write":21,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534281+0000","last_change":"2024-04-12T21:13:13.131501+0000","last_active":"2024-04-12T21:13:19.534281+0000","last_peered":"2024-04-12T21:13:19.534281+0000","last_clean":"2024-04-12T21:13:19.534281+0000","last_became_active":"2024-04-12T21:13:13.131161+0000","last_became_peered":"2024-04-12T21:13:13.131161+0000","last_unstale":"2024-04-12T21:13:19.534281+0000","last_undegraded":"2024-04-12T21:13:19.534281+0000","last_fullsized":"2024-04-12T21:13:19.534281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:45:27.046250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535286+0000","last_change":"2024-04-12T21:13:17.610917+0000","last_active":"2024-04-12T21:13:19.535286+0000","last_peered":"2024-04-12T21:13:19.535286+0000","last_clean":"2024-04-12T21:13:19.535286+0000","last_became_active":"2024-04-12T21:13:17.610418+0000","last_became_peered":"2024-04-12T21:13:17.610418+0000","last_unstale":"2024-04-12T21:13:19.535286+0000","last_undegraded":"2024-04-12T21:13:19.535286+0000","last_fullsized":"2024-04-12T21:13:19.535286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:15:48.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601587+0000","last_change":"2024-04-12T21:13:12.529571+0000","last_active":"2024-04-12T21:13:18.601587+0000","last_peered":"2024-04-12T21:13:18.601587+0000","last_clean":"2024-04-12T21:13:18.601587+0000","last_became_active":"2024-04-12T21:13:12.529250+0000","last_became_peered":"2024-04-12T21:13:12.529250+0000","last_unstale":"2024-04-12T21:13:18.601587+0000","last_undegraded":"2024-04-12T21:13:18.601587+0000","last_fullsized":"2024-04-12T21:13:18.601587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:35:53.476770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.605570+0000","last_change":"2024-04-12T21:13:13.150100+0000","last_active":"2024-04-12T21:13:18.605570+0000","last_peered":"2024-04-12T21:13:18.605570+0000","last_clean":"2024-04-12T21:13:18.605570+0000","last_became_active":"2024-04-12T21:13:13.149711+0000","last_became_peered":"2024-04-12T21:13:13.149711+0000","last_unstale":"2024-04-12T21:13:18.605570+0000","last_undegraded":"2024-04-12T21:13:18.605570+0000","last_fullsized":"2024-04-12T21:13:18.605570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:33:27.121682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685975+0000","last_change":"2024-04-12T21:13:17.605440+0000","last_active":"2024-04-12T21:13:18.685975+0000","last_peered":"2024-04-12T21:13:18.685975+0000","last_clean":"2024-04-12T21:13:18.685975+0000","last_became_active":"2024-04-12T21:13:17.604899+0000","last_became_peered":"2024-04-12T21:13:17.604899+0000","last_unstale":"2024-04-12T21:13:18.685975+0000","last_undegraded":"2024-04-12T21:13:18.685975+0000","last_fullsized":"2024-04-12T21:13:18.685975+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:53:01.678608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534489+0000","last_change":"2024-04-12T21:13:12.545077+0000","last_active":"2024-04-12T21:13:19.534489+0000","last_peered":"2024-04-12T21:13:19.534489+0000","last_clean":"2024-04-12T21:13:19.534489+0000","last_became_active":"2024-04-12T21:13:12.544755+0000","last_became_peered":"2024-04-12T21:13:12.544755+0000","last_unstale":"2024-04-12T21:13:19.534489+0000","last_undegraded":"2024-04-12T21:13:19.534489+0000","last_fullsized":"2024-04-12T21:13:19.534489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:31:28.487336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535204+0000","last_change":"2024-04-12T21:13:13.128047+0000","last_active":"2024-04-12T21:13:19.535204+0000","last_peered":"2024-04-12T21:13:19.535204+0000","last_clean":"2024-04-12T21:13:19.535204+0000","last_became_active":"2024-04-12T21:13:13.127692+0000","last_became_peered":"2024-04-12T21:13:13.127692+0000","last_unstale":"2024-04-12T21:13:19.535204+0000","last_undegraded":"2024-04-12T21:13:19.535204+0000","last_fullsized":"2024-04-12T21:13:19.535204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:43:29.552438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.608039+0000","last_change":"2024-04-12T21:13:17.614240+0000","last_active":"2024-04-12T21:13:18.608039+0000","last_peered":"2024-04-12T21:13:18.608039+0000","last_clean":"2024-04-12T21:13:18.608039+0000","last_became_active":"2024-04-12T21:13:17.613834+0000","last_became_peered":"2024-04-12T21:13:17.613834+0000","last_unstale":"2024-04-12T21:13:18.608039+0000","last_undegraded":"2024-04-12T21:13:18.608039+0000","last_fullsized":"2024-04-12T21:13:18.608039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:29:23.949101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533431+0000","last_change":"2024-04-12T21:13:12.527868+0000","last_active":"2024-04-12T21:13:19.533431+0000","last_peered":"2024-04-12T21:13:19.533431+0000","last_clean":"2024-04-12T21:13:19.533431+0000","last_became_active":"2024-04-12T21:13:12.527521+0000","last_became_peered":"2024-04-12T21:13:12.527521+0000","last_unstale":"2024-04-12T21:13:19.533431+0000","last_undegraded":"2024-04-12T21:13:19.533431+0000","last_fullsized":"2024-04-12T21:13:19.533431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:46:23.649552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534851+0000","last_change":"2024-04-12T21:13:13.124207+0000","last_active":"2024-04-12T21:13:19.534851+0000","last_peered":"2024-04-12T21:13:19.534851+0000","last_clean":"2024-04-12T21:13:19.534851+0000","last_became_active":"2024-04-12T21:13:13.123866+0000","last_became_peered":"2024-04-12T21:13:13.123866+0000","last_unstale":"2024-04-12T21:13:19.534851+0000","last_undegraded":"2024-04-12T21:13:19.534851+0000","last_fullsized":"2024-04-12T21:13:19.534851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:04.263260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606612+0000","last_change":"2024-04-12T21:13:17.613064+0000","last_active":"2024-04-12T21:13:18.606612+0000","last_peered":"2024-04-12T21:13:18.606612+0000","last_clean":"2024-04-12T21:13:18.606612+0000","last_became_active":"2024-04-12T21:13:17.612569+0000","last_became_peered":"2024-04-12T21:13:17.612569+0000","last_unstale":"2024-04-12T21:13:18.606612+0000","last_undegraded":"2024-04-12T21:13:18.606612+0000","last_fullsized":"2024-04-12T21:13:18.606612+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:35:27.274086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533625+0000","last_change":"2024-04-12T21:13:12.543193+0000","last_active":"2024-04-12T21:13:19.533625+0000","last_peered":"2024-04-12T21:13:19.533625+0000","last_clean":"2024-04-12T21:13:19.533625+0000","last_became_active":"2024-04-12T21:13:12.542885+0000","last_became_peered":"2024-04-12T21:13:12.542885+0000","last_unstale":"2024-04-12T21:13:19.533625+0000","last_undegraded":"2024-04-12T21:13:19.533625+0000","last_fullsized":"2024-04-12T21:13:19.533625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:02:11.100366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534264+0000","last_change":"2024-04-12T21:13:13.130682+0000","last_active":"2024-04-12T21:13:19.534264+0000","last_peered":"2024-04-12T21:13:19.534264+0000","last_clean":"2024-04-12T21:13:19.534264+0000","last_became_active":"2024-04-12T21:13:13.130299+0000","last_became_peered":"2024-04-12T21:13:13.130299+0000","last_unstale":"2024-04-12T21:13:19.534264+0000","last_undegraded":"2024-04-12T21:13:19.534264+0000","last_fullsized":"2024-04-12T21:13:19.534264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:52:02.094959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534309+0000","last_change":"2024-04-12T21:13:17.609472+0000","last_active":"2024-04-12T21:13:19.534309+0000","last_peered":"2024-04-12T21:13:19.534309+0000","last_clean":"2024-04-12T21:13:19.534309+0000","last_became_active":"2024-04-12T21:13:17.608991+0000","last_became_peered":"2024-04-12T21:13:17.608991+0000","last_unstale":"2024-04-12T21:13:19.534309+0000","last_undegraded":"2024-04-12T21:13:19.534309+0000","last_fullsized":"2024-04-12T21:13:19.534309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:17:57.962701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534436+0000","last_change":"2024-04-12T21:13:12.530058+0000","last_active":"2024-04-12T21:13:19.534436+0000","last_peered":"2024-04-12T21:13:19.534436+0000","last_clean":"2024-04-12T21:13:19.534436+0000","last_became_active":"2024-04-12T21:13:12.529682+0000","last_became_peered":"2024-04-12T21:13:12.529682+0000","last_unstale":"2024-04-12T21:13:19.534436+0000","last_undegraded":"2024-04-12T21:13:19.534436+0000","last_fullsized":"2024-04-12T21:13:19.534436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:17:25.621593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685972+0000","last_change":"2024-04-12T21:13:13.129975+0000","last_active":"2024-04-12T21:13:18.685972+0000","last_peered":"2024-04-12T21:13:18.685972+0000","last_clean":"2024-04-12T21:13:18.685972+0000","last_became_active":"2024-04-12T21:13:13.129621+0000","last_became_peered":"2024-04-12T21:13:13.129621+0000","last_unstale":"2024-04-12T21:13:18.685972+0000","last_undegraded":"2024-04-12T21:13:18.685972+0000","last_fullsized":"2024-04-12T21:13:18.685972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:49:48.267685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533966+0000","last_change":"2024-04-12T21:13:17.738019+0000","last_active":"2024-04-12T21:13:19.533966+0000","last_peered":"2024-04-12T21:13:19.533966+0000","last_clean":"2024-04-12T21:13:19.533966+0000","last_became_active":"2024-04-12T21:13:17.737553+0000","last_became_peered":"2024-04-12T21:13:17.737553+0000","last_unstale":"2024-04-12T21:13:19.533966+0000","last_undegraded":"2024-04-12T21:13:19.533966+0000","last_fullsized":"2024-04-12T21:13:19.533966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:29:07.288025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606241+0000","last_change":"2024-04-12T21:13:12.544661+0000","last_active":"2024-04-12T21:13:18.606241+0000","last_peered":"2024-04-12T21:13:18.606241+0000","last_clean":"2024-04-12T21:13:18.606241+0000","last_became_active":"2024-04-12T21:13:12.544321+0000","last_became_peered":"2024-04-12T21:13:12.544321+0000","last_unstale":"2024-04-12T21:13:18.606241+0000","last_undegraded":"2024-04-12T21:13:18.606241+0000","last_fullsized":"2024-04-12T21:13:18.606241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:55:43.517970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247739+0000","last_change":"2024-04-12T21:13:13.140480+0000","last_active":"2024-04-12T21:13:19.247739+0000","last_peered":"2024-04-12T21:13:19.247739+0000","last_clean":"2024-04-12T21:13:19.247739+0000","last_became_active":"2024-04-12T21:13:13.140128+0000","last_became_peered":"2024-04-12T21:13:13.140128+0000","last_unstale":"2024-04-12T21:13:19.247739+0000","last_undegraded":"2024-04-12T21:13:19.247739+0000","last_fullsized":"2024-04-12T21:13:19.247739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:40:32.066929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606642+0000","last_change":"2024-04-12T21:13:17.609235+0000","last_active":"2024-04-12T21:13:18.606642+0000","last_peered":"2024-04-12T21:13:18.606642+0000","last_clean":"2024-04-12T21:13:18.606642+0000","last_became_active":"2024-04-12T21:13:17.608814+0000","last_became_peered":"2024-04-12T21:13:17.608814+0000","last_unstale":"2024-04-12T21:13:18.606642+0000","last_undegraded":"2024-04-12T21:13:18.606642+0000","last_fullsized":"2024-04-12T21:13:18.606642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:15:35.287239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535019+0000","last_change":"2024-04-12T21:13:12.539638+0000","last_active":"2024-04-12T21:13:19.535019+0000","last_peered":"2024-04-12T21:13:19.535019+0000","last_clean":"2024-04-12T21:13:19.535019+0000","last_became_active":"2024-04-12T21:13:12.539272+0000","last_became_peered":"2024-04-12T21:13:12.539272+0000","last_unstale":"2024-04-12T21:13:19.535019+0000","last_undegraded":"2024-04-12T21:13:19.535019+0000","last_fullsized":"2024-04-12T21:13:19.535019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:11:42.992489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247490+0000","last_change":"2024-04-12T21:13:13.149326+0000","last_active":"2024-04-12T21:13:19.247490+0000","last_peered":"2024-04-12T21:13:19.247490+0000","last_clean":"2024-04-12T21:13:19.247490+0000","last_became_active":"2024-04-12T21:13:13.149023+0000","last_became_peered":"2024-04-12T21:13:13.149023+0000","last_unstale":"2024-04-12T21:13:19.247490+0000","last_undegraded":"2024-04-12T21:13:19.247490+0000","last_fullsized":"2024-04-12T21:13:19.247490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:15:58.479778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606944+0000","last_change":"2024-04-12T21:13:17.740067+0000","last_active":"2024-04-12T21:13:18.606944+0000","last_peered":"2024-04-12T21:13:18.606944+0000","last_clean":"2024-04-12T21:13:18.606944+0000","last_became_active":"2024-04-12T21:13:17.738186+0000","last_became_peered":"2024-04-12T21:13:17.738186+0000","last_unstale":"2024-04-12T21:13:18.606944+0000","last_undegraded":"2024-04-12T21:13:18.606944+0000","last_fullsized":"2024-04-12T21:13:18.606944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:15:05.241855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534206+0000","last_change":"2024-04-12T21:13:13.129755+0000","last_active":"2024-04-12T21:13:19.534206+0000","last_peered":"2024-04-12T21:13:19.534206+0000","last_clean":"2024-04-12T21:13:19.534206+0000","last_became_active":"2024-04-12T21:13:13.129422+0000","last_became_peered":"2024-04-12T21:13:13.129422+0000","last_unstale":"2024-04-12T21:13:19.534206+0000","last_undegraded":"2024-04-12T21:13:19.534206+0000","last_fullsized":"2024-04-12T21:13:19.534206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:59:08.675086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604081+0000","last_change":"2024-04-12T21:13:12.551549+0000","last_active":"2024-04-12T21:13:18.604081+0000","last_peered":"2024-04-12T21:13:18.604081+0000","last_clean":"2024-04-12T21:13:18.604081+0000","last_became_active":"2024-04-12T21:13:12.551204+0000","last_became_peered":"2024-04-12T21:13:12.551204+0000","last_unstale":"2024-04-12T21:13:18.604081+0000","last_undegraded":"2024-04-12T21:13:18.604081+0000","last_fullsized":"2024-04-12T21:13:18.604081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:33:06.721971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:20.649632+0000","last_change":"2024-04-12T21:13:17.603828+0000","last_active":"2024-04-12T21:13:20.649632+0000","last_peered":"2024-04-12T21:13:20.649632+0000","last_clean":"2024-04-12T21:13:20.649632+0000","last_became_active":"2024-04-12T21:13:17.603399+0000","last_became_peered":"2024-04-12T21:13:17.603399+0000","last_unstale":"2024-04-12T21:13:20.649632+0000","last_undegraded":"2024-04-12T21:13:20.649632+0000","last_fullsized":"2024-04-12T21:13:20.649632+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:22:28.126722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.537753+0000","last_change":"2024-04-12T21:13:13.137828+0000","last_active":"2024-04-12T21:13:19.537753+0000","last_peered":"2024-04-12T21:13:19.537753+0000","last_clean":"2024-04-12T21:13:19.537753+0000","last_became_active":"2024-04-12T21:13:13.137534+0000","last_became_peered":"2024-04-12T21:13:13.137534+0000","last_unstale":"2024-04-12T21:13:19.537753+0000","last_undegraded":"2024-04-12T21:13:19.537753+0000","last_fullsized":"2024-04-12T21:13:19.537753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:24:20.238210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533345+0000","last_change":"2024-04-12T21:13:12.533215+0000","last_active":"2024-04-12T21:13:19.533345+0000","last_peered":"2024-04-12T21:13:19.533345+0000","last_clean":"2024-04-12T21:13:19.533345+0000","last_became_active":"2024-04-12T21:13:12.532852+0000","last_became_peered":"2024-04-12T21:13:12.532852+0000","last_unstale":"2024-04-12T21:13:19.533345+0000","last_undegraded":"2024-04-12T21:13:19.533345+0000","last_fullsized":"2024-04-12T21:13:19.533345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:41:17.861406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534687+0000","last_change":"2024-04-12T21:11:53.092756+0000","last_active":"2024-04-12T21:13:19.534687+0000","last_peered":"2024-04-12T21:13:19.534687+0000","last_clean":"2024-04-12T21:13:19.534687+0000","last_became_active":"2024-04-12T21:11:53.092364+0000","last_became_peered":"2024-04-12T21:11:53.092364+0000","last_unstale":"2024-04-12T21:13:19.534687+0000","last_undegraded":"2024-04-12T21:13:19.534687+0000","last_fullsized":"2024-04-12T21:13:19.534687+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:08:04.402338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:08:04.402338+0000","last_clean_scrub_stamp":"2024-04-12T21:08:04.402338+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:26:11.910015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534993+0000","last_change":"2024-04-12T21:13:17.738168+0000","last_active":"2024-04-12T21:13:19.534993+0000","last_peered":"2024-04-12T21:13:19.534993+0000","last_clean":"2024-04-12T21:13:19.534993+0000","last_became_active":"2024-04-12T21:13:17.737697+0000","last_became_peered":"2024-04-12T21:13:17.737697+0000","last_unstale":"2024-04-12T21:13:19.534993+0000","last_undegraded":"2024-04-12T21:13:19.534993+0000","last_fullsized":"2024-04-12T21:13:19.534993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:11:31.025636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.603869+0000","last_change":"2024-04-12T21:13:13.130398+0000","last_active":"2024-04-12T21:13:18.603869+0000","last_peered":"2024-04-12T21:13:18.603869+0000","last_clean":"2024-04-12T21:13:18.603869+0000","last_became_active":"2024-04-12T21:13:13.129488+0000","last_became_peered":"2024-04-12T21:13:13.129488+0000","last_unstale":"2024-04-12T21:13:18.603869+0000","last_undegraded":"2024-04-12T21:13:18.603869+0000","last_fullsized":"2024-04-12T21:13:18.603869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:21:03.737140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534789+0000","last_change":"2024-04-12T21:13:12.533367+0000","last_active":"2024-04-12T21:13:19.534789+0000","last_peered":"2024-04-12T21:13:19.534789+0000","last_clean":"2024-04-12T21:13:19.534789+0000","last_became_active":"2024-04-12T21:13:12.533042+0000","last_became_peered":"2024-04-12T21:13:12.533042+0000","last_unstale":"2024-04-12T21:13:19.534789+0000","last_undegraded":"2024-04-12T21:13:19.534789+0000","last_fullsized":"2024-04-12T21:13:19.534789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:56:07.671693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685862+0000","last_change":"2024-04-12T21:13:17.593951+0000","last_active":"2024-04-12T21:13:18.685862+0000","last_peered":"2024-04-12T21:13:18.685862+0000","last_clean":"2024-04-12T21:13:18.685862+0000","last_became_active":"2024-04-12T21:13:17.593528+0000","last_became_peered":"2024-04-12T21:13:17.593528+0000","last_unstale":"2024-04-12T21:13:18.685862+0000","last_undegraded":"2024-04-12T21:13:18.685862+0000","last_fullsized":"2024-04-12T21:13:18.685862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:24:25.081109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534323+0000","last_change":"2024-04-12T21:13:12.543040+0000","last_active":"2024-04-12T21:13:19.534323+0000","last_peered":"2024-04-12T21:13:19.534323+0000","last_clean":"2024-04-12T21:13:19.534323+0000","last_became_active":"2024-04-12T21:13:12.542698+0000","last_became_peered":"2024-04-12T21:13:12.542698+0000","last_unstale":"2024-04-12T21:13:19.534323+0000","last_undegraded":"2024-04-12T21:13:19.534323+0000","last_fullsized":"2024-04-12T21:13:19.534323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:57:55.712648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.607050+0000","last_change":"2024-04-12T21:13:13.138975+0000","last_active":"2024-04-12T21:13:18.607050+0000","last_peered":"2024-04-12T21:13:18.607050+0000","last_clean":"2024-04-12T21:13:18.607050+0000","last_became_active":"2024-04-12T21:13:13.138538+0000","last_became_peered":"2024-04-12T21:13:13.138538+0000","last_unstale":"2024-04-12T21:13:18.607050+0000","last_undegraded":"2024-04-12T21:13:18.607050+0000","last_fullsized":"2024-04-12T21:13:18.607050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:12:10.488914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535679+0000","last_change":"2024-04-12T21:13:17.739156+0000","last_active":"2024-04-12T21:13:19.535679+0000","last_peered":"2024-04-12T21:13:19.535679+0000","last_clean":"2024-04-12T21:13:19.535679+0000","last_became_active":"2024-04-12T21:13:17.738153+0000","last_became_peered":"2024-04-12T21:13:17.738153+0000","last_unstale":"2024-04-12T21:13:19.535679+0000","last_undegraded":"2024-04-12T21:13:19.535679+0000","last_fullsized":"2024-04-12T21:13:19.535679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:18:18.930005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.607810+0000","last_change":"2024-04-12T21:13:12.547658+0000","last_active":"2024-04-12T21:13:18.607810+0000","last_peered":"2024-04-12T21:13:18.607810+0000","last_clean":"2024-04-12T21:13:18.607810+0000","last_became_active":"2024-04-12T21:13:12.547197+0000","last_became_peered":"2024-04-12T21:13:12.547197+0000","last_unstale":"2024-04-12T21:13:18.607810+0000","last_undegraded":"2024-04-12T21:13:18.607810+0000","last_fullsized":"2024-04-12T21:13:18.607810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:57:18.066730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.537968+0000","last_change":"2024-04-12T21:13:13.143876+0000","last_active":"2024-04-12T21:13:19.537968+0000","last_peered":"2024-04-12T21:13:19.537968+0000","last_clean":"2024-04-12T21:13:19.537968+0000","last_became_active":"2024-04-12T21:13:13.140389+0000","last_became_peered":"2024-04-12T21:13:13.140389+0000","last_unstale":"2024-04-12T21:13:19.537968+0000","last_undegraded":"2024-04-12T21:13:19.537968+0000","last_fullsized":"2024-04-12T21:13:19.537968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:36:13.910978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601169+0000","last_change":"2024-04-12T21:13:17.592755+0000","last_active":"2024-04-12T21:13:18.601169+0000","last_peered":"2024-04-12T21:13:18.601169+0000","last_clean":"2024-04-12T21:13:18.601169+0000","last_became_active":"2024-04-12T21:13:17.592316+0000","last_became_peered":"2024-04-12T21:13:17.592316+0000","last_unstale":"2024-04-12T21:13:18.601169+0000","last_undegraded":"2024-04-12T21:13:18.601169+0000","last_fullsized":"2024-04-12T21:13:18.601169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:16:41.986623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533964+0000","last_change":"2024-04-12T21:13:12.538168+0000","last_active":"2024-04-12T21:13:19.533964+0000","last_peered":"2024-04-12T21:13:19.533964+0000","last_clean":"2024-04-12T21:13:19.533964+0000","last_became_active":"2024-04-12T21:13:12.537784+0000","last_became_peered":"2024-04-12T21:13:12.537784+0000","last_unstale":"2024-04-12T21:13:19.533964+0000","last_undegraded":"2024-04-12T21:13:19.533964+0000","last_fullsized":"2024-04-12T21:13:19.533964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:57:58.333171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536843+0000","last_change":"2024-04-12T21:13:13.153230+0000","last_active":"2024-04-12T21:13:19.536843+0000","last_peered":"2024-04-12T21:13:19.536843+0000","last_clean":"2024-04-12T21:13:19.536843+0000","last_became_active":"2024-04-12T21:13:13.152795+0000","last_became_peered":"2024-04-12T21:13:13.152795+0000","last_unstale":"2024-04-12T21:13:19.536843+0000","last_undegraded":"2024-04-12T21:13:19.536843+0000","last_fullsized":"2024-04-12T21:13:19.536843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:11:21.058591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533701+0000","last_change":"2024-04-12T21:13:17.738428+0000","last_active":"2024-04-12T21:13:19.533701+0000","last_peered":"2024-04-12T21:13:19.533701+0000","last_clean":"2024-04-12T21:13:19.533701+0000","last_became_active":"2024-04-12T21:13:17.737932+0000","last_became_peered":"2024-04-12T21:13:17.737932+0000","last_unstale":"2024-04-12T21:13:19.533701+0000","last_undegraded":"2024-04-12T21:13:19.533701+0000","last_fullsized":"2024-04-12T21:13:19.533701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:22:26.310841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.010984+0000","last_change":"2024-04-12T21:13:12.542689+0000","last_active":"2024-04-12T21:13:19.010984+0000","last_peered":"2024-04-12T21:13:19.010984+0000","last_clean":"2024-04-12T21:13:19.010984+0000","last_became_active":"2024-04-12T21:13:12.542371+0000","last_became_peered":"2024-04-12T21:13:12.542371+0000","last_unstale":"2024-04-12T21:13:19.010984+0000","last_undegraded":"2024-04-12T21:13:19.010984+0000","last_fullsized":"2024-04-12T21:13:19.010984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:08:48.439015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533625+0000","last_change":"2024-04-12T21:13:13.130341+0000","last_active":"2024-04-12T21:13:19.533625+0000","last_peered":"2024-04-12T21:13:19.533625+0000","last_clean":"2024-04-12T21:13:19.533625+0000","last_became_active":"2024-04-12T21:13:13.129886+0000","last_became_peered":"2024-04-12T21:13:13.129886+0000","last_unstale":"2024-04-12T21:13:19.533625+0000","last_undegraded":"2024-04-12T21:13:19.533625+0000","last_fullsized":"2024-04-12T21:13:19.533625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:43:18.384216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.608208+0000","last_change":"2024-04-12T21:13:17.615138+0000","last_active":"2024-04-12T21:13:18.608208+0000","last_peered":"2024-04-12T21:13:18.608208+0000","last_clean":"2024-04-12T21:13:18.608208+0000","last_became_active":"2024-04-12T21:13:17.614710+0000","last_became_peered":"2024-04-12T21:13:17.614710+0000","last_unstale":"2024-04-12T21:13:18.608208+0000","last_undegraded":"2024-04-12T21:13:18.608208+0000","last_fullsized":"2024-04-12T21:13:18.608208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:13:47.956600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536688+0000","last_change":"2024-04-12T21:13:12.546160+0000","last_active":"2024-04-12T21:13:19.536688+0000","last_peered":"2024-04-12T21:13:19.536688+0000","last_clean":"2024-04-12T21:13:19.536688+0000","last_became_active":"2024-04-12T21:13:12.544193+0000","last_became_peered":"2024-04-12T21:13:12.544193+0000","last_unstale":"2024-04-12T21:13:19.536688+0000","last_undegraded":"2024-04-12T21:13:19.536688+0000","last_fullsized":"2024-04-12T21:13:19.536688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:19:39.640814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685618+0000","last_change":"2024-04-12T21:13:13.154604+0000","last_active":"2024-04-12T21:13:18.685618+0000","last_peered":"2024-04-12T21:13:18.685618+0000","last_clean":"2024-04-12T21:13:18.685618+0000","last_became_active":"2024-04-12T21:13:13.154257+0000","last_became_peered":"2024-04-12T21:13:13.154257+0000","last_unstale":"2024-04-12T21:13:18.685618+0000","last_undegraded":"2024-04-12T21:13:18.685618+0000","last_fullsized":"2024-04-12T21:13:18.685618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:31:45.550602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:21.407872+0000","last_change":"2024-04-12T21:13:17.605698+0000","last_active":"2024-04-12T21:13:21.407872+0000","last_peered":"2024-04-12T21:13:21.407872+0000","last_clean":"2024-04-12T21:13:21.407872+0000","last_became_active":"2024-04-12T21:13:17.605241+0000","last_became_peered":"2024-04-12T21:13:17.605241+0000","last_unstale":"2024-04-12T21:13:21.407872+0000","last_undegraded":"2024-04-12T21:13:21.407872+0000","last_fullsized":"2024-04-12T21:13:21.407872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:14:32.785778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.600944+0000","last_change":"2024-04-12T21:13:12.536798+0000","last_active":"2024-04-12T21:13:18.600944+0000","last_peered":"2024-04-12T21:13:18.600944+0000","last_clean":"2024-04-12T21:13:18.600944+0000","last_became_active":"2024-04-12T21:13:12.536451+0000","last_became_peered":"2024-04-12T21:13:12.536451+0000","last_unstale":"2024-04-12T21:13:18.600944+0000","last_undegraded":"2024-04-12T21:13:18.600944+0000","last_fullsized":"2024-04-12T21:13:18.600944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:07:07.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.537718+0000","last_change":"2024-04-12T21:13:13.154644+0000","last_active":"2024-04-12T21:13:19.537718+0000","last_peered":"2024-04-12T21:13:19.537718+0000","last_clean":"2024-04-12T21:13:19.537718+0000","last_became_active":"2024-04-12T21:13:13.154310+0000","last_became_peered":"2024-04-12T21:13:13.154310+0000","last_unstale":"2024-04-12T21:13:19.537718+0000","last_undegraded":"2024-04-12T21:13:19.537718+0000","last_fullsized":"2024-04-12T21:13:19.537718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:51:36.426506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533731+0000","last_change":"2024-04-12T21:13:17.737547+0000","last_active":"2024-04-12T21:13:19.533731+0000","last_peered":"2024-04-12T21:13:19.533731+0000","last_clean":"2024-04-12T21:13:19.533731+0000","last_became_active":"2024-04-12T21:13:17.737190+0000","last_became_peered":"2024-04-12T21:13:17.737190+0000","last_unstale":"2024-04-12T21:13:19.533731+0000","last_undegraded":"2024-04-12T21:13:19.533731+0000","last_fullsized":"2024-04-12T21:13:19.533731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:23:52.221788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534162+0000","last_change":"2024-04-12T21:13:12.534349+0000","last_active":"2024-04-12T21:13:19.534162+0000","last_peered":"2024-04-12T21:13:19.534162+0000","last_clean":"2024-04-12T21:13:19.534162+0000","last_became_active":"2024-04-12T21:13:12.533933+0000","last_became_peered":"2024-04-12T21:13:12.533933+0000","last_unstale":"2024-04-12T21:13:19.534162+0000","last_undegraded":"2024-04-12T21:13:19.534162+0000","last_fullsized":"2024-04-12T21:13:19.534162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:24:34.110260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535128+0000","last_change":"2024-04-12T21:13:13.128079+0000","last_active":"2024-04-12T21:13:19.535128+0000","last_peered":"2024-04-12T21:13:19.535128+0000","last_clean":"2024-04-12T21:13:19.535128+0000","last_became_active":"2024-04-12T21:13:13.127744+0000","last_became_peered":"2024-04-12T21:13:13.127744+0000","last_unstale":"2024-04-12T21:13:19.535128+0000","last_undegraded":"2024-04-12T21:13:19.535128+0000","last_fullsized":"2024-04-12T21:13:19.535128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:39:41.851323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011200+0000","last_change":"2024-04-12T21:13:17.596088+0000","last_active":"2024-04-12T21:13:19.011200+0000","last_peered":"2024-04-12T21:13:19.011200+0000","last_clean":"2024-04-12T21:13:19.011200+0000","last_became_active":"2024-04-12T21:13:17.595639+0000","last_became_peered":"2024-04-12T21:13:17.595639+0000","last_unstale":"2024-04-12T21:13:19.011200+0000","last_undegraded":"2024-04-12T21:13:19.011200+0000","last_fullsized":"2024-04-12T21:13:19.011200+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:25:09.763945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601529+0000","last_change":"2024-04-12T21:13:12.543947+0000","last_active":"2024-04-12T21:13:18.601529+0000","last_peered":"2024-04-12T21:13:18.601529+0000","last_clean":"2024-04-12T21:13:18.601529+0000","last_became_active":"2024-04-12T21:13:12.543511+0000","last_became_peered":"2024-04-12T21:13:12.543511+0000","last_unstale":"2024-04-12T21:13:18.601529+0000","last_undegraded":"2024-04-12T21:13:18.601529+0000","last_fullsized":"2024-04-12T21:13:18.601529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:43:23.354835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535450+0000","last_change":"2024-04-12T21:13:13.127178+0000","last_active":"2024-04-12T21:13:19.535450+0000","last_peered":"2024-04-12T21:13:19.535450+0000","last_clean":"2024-04-12T21:13:19.535450+0000","last_became_active":"2024-04-12T21:13:13.126856+0000","last_became_peered":"2024-04-12T21:13:13.126856+0000","last_unstale":"2024-04-12T21:13:19.535450+0000","last_undegraded":"2024-04-12T21:13:19.535450+0000","last_fullsized":"2024-04-12T21:13:19.535450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:09:34.068946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536004+0000","last_change":"2024-04-12T21:13:17.619628+0000","last_active":"2024-04-12T21:13:19.536004+0000","last_peered":"2024-04-12T21:13:19.536004+0000","last_clean":"2024-04-12T21:13:19.536004+0000","last_became_active":"2024-04-12T21:13:17.619015+0000","last_became_peered":"2024-04-12T21:13:17.619015+0000","last_unstale":"2024-04-12T21:13:19.536004+0000","last_undegraded":"2024-04-12T21:13:19.536004+0000","last_fullsized":"2024-04-12T21:13:19.536004+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:39:54.297165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535118+0000","last_change":"2024-04-12T21:13:12.538982+0000","last_active":"2024-04-12T21:13:19.535118+0000","last_peered":"2024-04-12T21:13:19.535118+0000","last_clean":"2024-04-12T21:13:19.535118+0000","last_became_active":"2024-04-12T21:13:12.538604+0000","last_became_peered":"2024-04-12T21:13:12.538604+0000","last_unstale":"2024-04-12T21:13:19.535118+0000","last_undegraded":"2024-04-12T21:13:19.535118+0000","last_fullsized":"2024-04-12T21:13:19.535118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:40:48.181188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535030+0000","last_change":"2024-04-12T21:13:13.145537+0000","last_active":"2024-04-12T21:13:19.535030+0000","last_peered":"2024-04-12T21:13:19.535030+0000","last_clean":"2024-04-12T21:13:19.535030+0000","last_became_active":"2024-04-12T21:13:13.145283+0000","last_became_peered":"2024-04-12T21:13:13.145283+0000","last_unstale":"2024-04-12T21:13:19.535030+0000","last_undegraded":"2024-04-12T21:13:19.535030+0000","last_fullsized":"2024-04-12T21:13:19.535030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:26:29.339674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533830+0000","last_change":"2024-04-12T21:13:17.594394+0000","last_active":"2024-04-12T21:13:19.533830+0000","last_peered":"2024-04-12T21:13:19.533830+0000","last_clean":"2024-04-12T21:13:19.533830+0000","last_became_active":"2024-04-12T21:13:17.593960+0000","last_became_peered":"2024-04-12T21:13:17.593960+0000","last_unstale":"2024-04-12T21:13:19.533830+0000","last_undegraded":"2024-04-12T21:13:19.533830+0000","last_fullsized":"2024-04-12T21:13:19.533830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:00:53.462145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606325+0000","last_change":"2024-04-12T21:13:12.546385+0000","last_active":"2024-04-12T21:13:18.606325+0000","last_peered":"2024-04-12T21:13:18.606325+0000","last_clean":"2024-04-12T21:13:18.606325+0000","last_became_active":"2024-04-12T21:13:12.545902+0000","last_became_peered":"2024-04-12T21:13:12.545902+0000","last_unstale":"2024-04-12T21:13:18.606325+0000","last_undegraded":"2024-04-12T21:13:18.606325+0000","last_fullsized":"2024-04-12T21:13:18.606325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:57:16.419641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535297+0000","last_change":"2024-04-12T21:13:13.130478+0000","last_active":"2024-04-12T21:13:19.535297+0000","last_peered":"2024-04-12T21:13:19.535297+0000","last_clean":"2024-04-12T21:13:19.535297+0000","last_became_active":"2024-04-12T21:13:13.130164+0000","last_became_peered":"2024-04-12T21:13:13.130164+0000","last_unstale":"2024-04-12T21:13:19.535297+0000","last_undegraded":"2024-04-12T21:13:19.535297+0000","last_fullsized":"2024-04-12T21:13:19.535297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:22:29.186629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685678+0000","last_change":"2024-04-12T21:13:13.135406+0000","last_active":"2024-04-12T21:13:18.685678+0000","last_peered":"2024-04-12T21:13:18.685678+0000","last_clean":"2024-04-12T21:13:18.685678+0000","last_became_active":"2024-04-12T21:13:13.134840+0000","last_became_peered":"2024-04-12T21:13:13.134840+0000","last_unstale":"2024-04-12T21:13:18.685678+0000","last_undegraded":"2024-04-12T21:13:18.685678+0000","last_fullsized":"2024-04-12T21:13:18.685678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:22:22.315016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601615+0000","last_change":"2024-04-12T21:13:13.127249+0000","last_active":"2024-04-12T21:13:18.601615+0000","last_peered":"2024-04-12T21:13:18.601615+0000","last_clean":"2024-04-12T21:13:18.601615+0000","last_became_active":"2024-04-12T21:13:13.126901+0000","last_became_peered":"2024-04-12T21:13:13.126901+0000","last_unstale":"2024-04-12T21:13:18.601615+0000","last_undegraded":"2024-04-12T21:13:18.601615+0000","last_fullsized":"2024-04-12T21:13:18.601615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:32:18.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.600970+0000","last_change":"2024-04-12T21:13:13.151728+0000","last_active":"2024-04-12T21:13:18.600970+0000","last_peered":"2024-04-12T21:13:18.600970+0000","last_clean":"2024-04-12T21:13:18.600970+0000","last_became_active":"2024-04-12T21:13:13.151457+0000","last_became_peered":"2024-04-12T21:13:13.151457+0000","last_unstale":"2024-04-12T21:13:18.600970+0000","last_undegraded":"2024-04-12T21:13:18.600970+0000","last_fullsized":"2024-04-12T21:13:18.600970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:48:32.641671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536071+0000","last_change":"2024-04-12T21:13:13.150109+0000","last_active":"2024-04-12T21:13:19.536071+0000","last_peered":"2024-04-12T21:13:19.536071+0000","last_clean":"2024-04-12T21:13:19.536071+0000","last_became_active":"2024-04-12T21:13:13.149132+0000","last_became_peered":"2024-04-12T21:13:13.149132+0000","last_unstale":"2024-04-12T21:13:19.536071+0000","last_undegraded":"2024-04-12T21:13:19.536071+0000","last_fullsized":"2024-04-12T21:13:19.536071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:36:00.062993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606883+0000","last_change":"2024-04-12T21:13:13.143405+0000","last_active":"2024-04-12T21:13:18.606883+0000","last_peered":"2024-04-12T21:13:18.606883+0000","last_clean":"2024-04-12T21:13:18.606883+0000","last_became_active":"2024-04-12T21:13:13.142973+0000","last_became_peered":"2024-04-12T21:13:13.142973+0000","last_unstale":"2024-04-12T21:13:18.606883+0000","last_undegraded":"2024-04-12T21:13:18.606883+0000","last_fullsized":"2024-04-12T21:13:18.606883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:10:03.106739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.686058+0000","last_change":"2024-04-12T21:13:13.129864+0000","last_active":"2024-04-12T21:13:18.686058+0000","last_peered":"2024-04-12T21:13:18.686058+0000","last_clean":"2024-04-12T21:13:18.686058+0000","last_became_active":"2024-04-12T21:13:13.129518+0000","last_became_peered":"2024-04-12T21:13:13.129518+0000","last_unstale":"2024-04-12T21:13:18.686058+0000","last_undegraded":"2024-04-12T21:13:18.686058+0000","last_fullsized":"2024-04-12T21:13:18.686058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:44:38.370002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601520+0000","last_change":"2024-04-12T21:13:13.129153+0000","last_active":"2024-04-12T21:13:18.601520+0000","last_peered":"2024-04-12T21:13:18.601520+0000","last_clean":"2024-04-12T21:13:18.601520+0000","last_became_active":"2024-04-12T21:13:13.128799+0000","last_became_peered":"2024-04-12T21:13:13.128799+0000","last_unstale":"2024-04-12T21:13:18.601520+0000","last_undegraded":"2024-04-12T21:13:18.601520+0000","last_fullsized":"2024-04-12T21:13:18.601520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:37:51.675110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.538172+0000","last_change":"2024-04-12T21:13:13.144638+0000","last_active":"2024-04-12T21:13:19.538172+0000","last_peered":"2024-04-12T21:13:19.538172+0000","last_clean":"2024-04-12T21:13:19.538172+0000","last_became_active":"2024-04-12T21:13:13.144275+0000","last_became_peered":"2024-04-12T21:13:13.144275+0000","last_unstale":"2024-04-12T21:13:19.538172+0000","last_undegraded":"2024-04-12T21:13:19.538172+0000","last_fullsized":"2024-04-12T21:13:19.538172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:38:09.225551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534298+0000","last_change":"2024-04-12T21:13:13.145845+0000","last_active":"2024-04-12T21:13:19.534298+0000","last_peered":"2024-04-12T21:13:19.534298+0000","last_clean":"2024-04-12T21:13:19.534298+0000","last_became_active":"2024-04-12T21:13:13.145399+0000","last_became_peered":"2024-04-12T21:13:13.145399+0000","last_unstale":"2024-04-12T21:13:19.534298+0000","last_undegraded":"2024-04-12T21:13:19.534298+0000","last_fullsized":"2024-04-12T21:13:19.534298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:07:13.075365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536956+0000","last_change":"2024-04-12T21:13:13.138373+0000","last_active":"2024-04-12T21:13:19.536956+0000","last_peered":"2024-04-12T21:13:19.536956+0000","last_clean":"2024-04-12T21:13:19.536956+0000","last_became_active":"2024-04-12T21:13:13.137366+0000","last_became_peered":"2024-04-12T21:13:13.137366+0000","last_unstale":"2024-04-12T21:13:19.536956+0000","last_undegraded":"2024-04-12T21:13:19.536956+0000","last_fullsized":"2024-04-12T21:13:19.536956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:05:22.070677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011282+0000","last_change":"2024-04-12T21:13:13.147902+0000","last_active":"2024-04-12T21:13:19.011282+0000","last_peered":"2024-04-12T21:13:19.011282+0000","last_clean":"2024-04-12T21:13:19.011282+0000","last_became_active":"2024-04-12T21:13:13.147513+0000","last_became_peered":"2024-04-12T21:13:13.147513+0000","last_unstale":"2024-04-12T21:13:19.011282+0000","last_undegraded":"2024-04-12T21:13:19.011282+0000","last_fullsized":"2024-04-12T21:13:19.011282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:43:34.228432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.605734+0000","last_change":"2024-04-12T21:13:13.154703+0000","last_active":"2024-04-12T21:13:18.605734+0000","last_peered":"2024-04-12T21:13:18.605734+0000","last_clean":"2024-04-12T21:13:18.605734+0000","last_became_active":"2024-04-12T21:13:13.154340+0000","last_became_peered":"2024-04-12T21:13:13.154340+0000","last_unstale":"2024-04-12T21:13:18.605734+0000","last_undegraded":"2024-04-12T21:13:18.605734+0000","last_fullsized":"2024-04-12T21:13:18.605734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:29:05.500481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685922+0000","last_change":"2024-04-12T21:13:13.133441+0000","last_active":"2024-04-12T21:13:18.685922+0000","last_peered":"2024-04-12T21:13:18.685922+0000","last_clean":"2024-04-12T21:13:18.685922+0000","last_became_active":"2024-04-12T21:13:13.133113+0000","last_became_peered":"2024-04-12T21:13:13.133113+0000","last_unstale":"2024-04-12T21:13:18.685922+0000","last_undegraded":"2024-04-12T21:13:18.685922+0000","last_fullsized":"2024-04-12T21:13:18.685922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:05:36.924405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":41269,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":18,"num_read_kb":35,"num_write":60,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":69848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1090918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":991232,"data_stored":477819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1585152,"data_stored":1072261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":487657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":477319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":481333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":1593344,"data_stored":1071861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":40665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T21:13:42.413 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-12T21:13:42.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 -- 172.21.15.81:0/261145265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 msgr2=0x7f826810a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 --2- 172.21.15.81:0/261145265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 0x7f826810a260 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8260009ca0 tx=0x7f826002c100 comp rx=0 tx=0).stop 2024-04-12T21:13:42.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 -- 172.21.15.81:0/261145265 shutdown_connections 2024-04-12T21:13:42.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 --2- 172.21.15.81:0/261145265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 0x7f826810a260 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 --2- 172.21.15.81:0/261145265 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8268102f80 0x7f8268103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 --2- 172.21.15.81:0/261145265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8268101e30 0x7f8268102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 -- 172.21.15.81:0/261145265 >> 172.21.15.81:0/261145265 conn(0x7f82680fd3a0 msgr2=0x7f82680ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:42.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 -- 172.21.15.81:0/261145265 shutdown_connections 2024-04-12T21:13:42.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.661+0000 7f826e400640 1 -- 172.21.15.81:0/261145265 wait complete. 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.662+0000 7f826e400640 1 Processor -- start 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.662+0000 7f826e400640 1 -- start start 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.662+0000 7f826e400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8268101e30 0x7f8268198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.662+0000 7f826e400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8268102f80 0x7f8268198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.662+0000 7f826e400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 0x7f826819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.662+0000 7f826e400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f826810dad0 con 0x7f8268102f80 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.662+0000 7f826e400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f826810d950 con 0x7f8268103940 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.662+0000 7f826e400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f826810dc50 con 0x7f8268101e30 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f8267e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8268102f80 0x7f8268198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f826da00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 0x7f826819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:42.660 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f8267e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8268102f80 0x7f8268198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:46574/0 (socket says 172.21.15.81:46574) 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f8267e00640 1 -- 172.21.15.81:0/1168186249 learned_addr learned my addr 172.21.15.81:0/1168186249 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f826d000640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8268101e30 0x7f8268198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f826da00640 1 -- 172.21.15.81:0/1168186249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8268101e30 msgr2=0x7f8268198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f826da00640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8268101e30 0x7f8268198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f826da00640 1 -- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8268102f80 msgr2=0x7f8268198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f826da00640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8268102f80 0x7f8268198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f826da00640 1 -- 172.21.15.81:0/1168186249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826819fd50 con 0x7f8268103940 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.663+0000 7f826da00640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 0x7f826819f6e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8260009bf0 tx=0x7f8260035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.664+0000 7f8265600640 1 -- 172.21.15.81:0/1168186249 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260039040 con 0x7f8268103940 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.664+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826819ffe0 con 0x7f8268103940 2024-04-12T21:13:42.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.664+0000 7f8265600640 1 -- 172.21.15.81:0/1168186249 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8260044070 con 0x7f8268103940 2024-04-12T21:13:42.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.664+0000 7f8265600640 1 -- 172.21.15.81:0/1168186249 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8260004460 con 0x7f8268103940 2024-04-12T21:13:42.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.664+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82681a04a0 con 0x7f8268103940 2024-04-12T21:13:42.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.665+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82681090e0 con 0x7f8268103940 2024-04-12T21:13:42.662 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.667+0000 7f8265600640 1 -- 172.21.15.81:0/1168186249 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8260004600 con 0x7f8268103940 2024-04-12T21:13:42.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.668+0000 7f8265600640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f82500a3c40 0x7f82500a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:42.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.668+0000 7f8265600640 1 -- 172.21.15.81:0/1168186249 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f82600f2a40 con 0x7f8268103940 2024-04-12T21:13:42.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.668+0000 7f826d000640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f82500a3c40 0x7f82500a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:42.664 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.669+0000 7f826d000640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f82500a3c40 0x7f82500a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f825c009bd0 tx=0x7f825c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:42.668 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.673+0000 7f8265600640 1 -- 172.21.15.81:0/1168186249 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f826004e020 con 0x7f8268103940 2024-04-12T21:13:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:42 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1631693042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:42 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3901931386' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:42 smithi153 ceph-mon[36569]: pgmap v329: 129 pgs: 129 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.9 KiB/s wr, 2 op/s 2024-04-12T21:13:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:42 smithi153 ceph-mon[36569]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:13:42.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:42 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1631693042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:42.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:42 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3901931386' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:42.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:42 smithi104 ceph-mon[36624]: pgmap v329: 129 pgs: 129 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.9 KiB/s wr, 2 op/s 2024-04-12T21:13:42.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:42 smithi104 ceph-mon[36624]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:13:42.867 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.871+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8268062400 con 0x7f82500a3c40 2024-04-12T21:13:42.891 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.896+0000 7f8265600640 1 -- 172.21.15.81:0/1168186249 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340238 (secure 0 0 0) 0x7f8268062400 con 0x7f82500a3c40 2024-04-12T21:13:42.894 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:42.898 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f82500a3c40 msgr2=0x7f82500a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f82500a3c40 0x7f82500a6100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f825c009bd0 tx=0x7f825c005f90 comp rx=0 tx=0).stop 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 msgr2=0x7f826819f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 0x7f826819f6e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8260009bf0 tx=0x7f8260035d10 comp rx=0 tx=0).stop 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 shutdown_connections 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f82500a3c40 0x7f82500a6100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8268103940 0x7f826819f6e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8268102f80 0x7f8268198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 --2- 172.21.15.81:0/1168186249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8268101e30 0x7f8268198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 >> 172.21.15.81:0/1168186249 conn(0x7f82680fd3a0 msgr2=0x7f82681062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 shutdown_connections 2024-04-12T21:13:42.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:42.904+0000 7f826e400640 1 -- 172.21.15.81:0/1168186249 wait complete. 2024-04-12T21:13:42.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:42 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1631693042' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:13:42.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:42 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3901931386' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:42.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:42 smithi081 ceph-mon[31453]: pgmap v329: 129 pgs: 129 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.9 KiB/s wr, 2 op/s 2024-04-12T21:13:42.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:42 smithi081 ceph-mon[31453]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:13:42.921 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-04-12T21:13:42.329933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631871,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":243,"num_write_kb":2999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375068,"kb_used_data":13400,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124566756,"statfs":{"total":1151940427776,"available":1151556358144,"internally_reserved":0,"allocated":13721600,"data_stored":7553887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39093,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":35,"num_write":17,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006258"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.686261+0000","last_change":"2024-04-12T21:13:13.157179+0000","last_active":"2024-04-12T21:13:18.686261+0000","last_peered":"2024-04-12T21:13:18.686261+0000","last_clean":"2024-04-12T21:13:18.686261+0000","last_became_active":"2024-04-12T21:13:13.156826+0000","last_became_peered":"2024-04-12T21:13:13.156826+0000","last_unstale":"2024-04-12T21:13:18.686261+0000","last_undegraded":"2024-04-12T21:13:18.686261+0000","last_fullsized":"2024-04-12T21:13:18.686261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:51:25.310758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536786+0000","last_change":"2024-04-12T21:13:13.150917+0000","last_active":"2024-04-12T21:13:19.536786+0000","last_peered":"2024-04-12T21:13:19.536786+0000","last_clean":"2024-04-12T21:13:19.536786+0000","last_became_active":"2024-04-12T21:13:13.150491+0000","last_became_peered":"2024-04-12T21:13:13.150491+0000","last_unstale":"2024-04-12T21:13:19.536786+0000","last_undegraded":"2024-04-12T21:13:19.536786+0000","last_fullsized":"2024-04-12T21:13:19.536786+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:08:04.486913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536320+0000","last_change":"2024-04-12T21:13:13.150355+0000","last_active":"2024-04-12T21:13:19.536320+0000","last_peered":"2024-04-12T21:13:19.536320+0000","last_clean":"2024-04-12T21:13:19.536320+0000","last_became_active":"2024-04-12T21:13:13.149979+0000","last_became_peered":"2024-04-12T21:13:13.149979+0000","last_unstale":"2024-04-12T21:13:19.536320+0000","last_undegraded":"2024-04-12T21:13:19.536320+0000","last_fullsized":"2024-04-12T21:13:19.536320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:02:56.175842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533647+0000","last_change":"2024-04-12T21:13:13.132074+0000","last_active":"2024-04-12T21:13:19.533647+0000","last_peered":"2024-04-12T21:13:19.533647+0000","last_clean":"2024-04-12T21:13:19.533647+0000","last_became_active":"2024-04-12T21:13:13.130234+0000","last_became_peered":"2024-04-12T21:13:13.130234+0000","last_unstale":"2024-04-12T21:13:19.533647+0000","last_undegraded":"2024-04-12T21:13:19.533647+0000","last_fullsized":"2024-04-12T21:13:19.533647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:18:51.706215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533958+0000","last_change":"2024-04-12T21:13:13.146577+0000","last_active":"2024-04-12T21:13:19.533958+0000","last_peered":"2024-04-12T21:13:19.533958+0000","last_clean":"2024-04-12T21:13:19.533958+0000","last_became_active":"2024-04-12T21:13:13.146162+0000","last_became_peered":"2024-04-12T21:13:13.146162+0000","last_unstale":"2024-04-12T21:13:19.533958+0000","last_undegraded":"2024-04-12T21:13:19.533958+0000","last_fullsized":"2024-04-12T21:13:19.533958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:43:10.267515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247478+0000","last_change":"2024-04-12T21:13:13.137103+0000","last_active":"2024-04-12T21:13:19.247478+0000","last_peered":"2024-04-12T21:13:19.247478+0000","last_clean":"2024-04-12T21:13:19.247478+0000","last_became_active":"2024-04-12T21:13:13.136750+0000","last_became_peered":"2024-04-12T21:13:13.136750+0000","last_unstale":"2024-04-12T21:13:19.247478+0000","last_undegraded":"2024-04-12T21:13:19.247478+0000","last_fullsized":"2024-04-12T21:13:19.247478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:11:58.538179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534916+0000","last_change":"2024-04-12T21:13:13.135440+0000","last_active":"2024-04-12T21:13:19.534916+0000","last_peered":"2024-04-12T21:13:19.534916+0000","last_clean":"2024-04-12T21:13:19.534916+0000","last_became_active":"2024-04-12T21:13:13.135027+0000","last_became_peered":"2024-04-12T21:13:13.135027+0000","last_unstale":"2024-04-12T21:13:19.534916+0000","last_undegraded":"2024-04-12T21:13:19.534916+0000","last_fullsized":"2024-04-12T21:13:19.534916+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:37:50.875158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535051+0000","last_change":"2024-04-12T21:13:13.143059+0000","last_active":"2024-04-12T21:13:19.535051+0000","last_peered":"2024-04-12T21:13:19.535051+0000","last_clean":"2024-04-12T21:13:19.535051+0000","last_became_active":"2024-04-12T21:13:13.142698+0000","last_became_peered":"2024-04-12T21:13:13.142698+0000","last_unstale":"2024-04-12T21:13:19.535051+0000","last_undegraded":"2024-04-12T21:13:19.535051+0000","last_fullsized":"2024-04-12T21:13:19.535051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:30:05.170007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011613+0000","last_change":"2024-04-12T21:13:13.128917+0000","last_active":"2024-04-12T21:13:19.011613+0000","last_peered":"2024-04-12T21:13:19.011613+0000","last_clean":"2024-04-12T21:13:19.011613+0000","last_became_active":"2024-04-12T21:13:13.128166+0000","last_became_peered":"2024-04-12T21:13:13.128166+0000","last_unstale":"2024-04-12T21:13:19.011613+0000","last_undegraded":"2024-04-12T21:13:19.011613+0000","last_fullsized":"2024-04-12T21:13:19.011613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:49:41.282798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247930+0000","last_change":"2024-04-12T21:13:13.130833+0000","last_active":"2024-04-12T21:13:19.247930+0000","last_peered":"2024-04-12T21:13:19.247930+0000","last_clean":"2024-04-12T21:13:19.247930+0000","last_became_active":"2024-04-12T21:13:13.130506+0000","last_became_peered":"2024-04-12T21:13:13.130506+0000","last_unstale":"2024-04-12T21:13:19.247930+0000","last_undegraded":"2024-04-12T21:13:19.247930+0000","last_fullsized":"2024-04-12T21:13:19.247930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:08:30.527381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535373+0000","last_change":"2024-04-12T21:13:13.129682+0000","last_active":"2024-04-12T21:13:19.535373+0000","last_peered":"2024-04-12T21:13:19.535373+0000","last_clean":"2024-04-12T21:13:19.535373+0000","last_became_active":"2024-04-12T21:13:13.129364+0000","last_became_peered":"2024-04-12T21:13:13.129364+0000","last_unstale":"2024-04-12T21:13:19.535373+0000","last_undegraded":"2024-04-12T21:13:19.535373+0000","last_fullsized":"2024-04-12T21:13:19.535373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:19:48.442418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533584+0000","last_change":"2024-04-12T21:13:13.133083+0000","last_active":"2024-04-12T21:13:19.533584+0000","last_peered":"2024-04-12T21:13:19.533584+0000","last_clean":"2024-04-12T21:13:19.533584+0000","last_became_active":"2024-04-12T21:13:13.132679+0000","last_became_peered":"2024-04-12T21:13:13.132679+0000","last_unstale":"2024-04-12T21:13:19.533584+0000","last_undegraded":"2024-04-12T21:13:19.533584+0000","last_fullsized":"2024-04-12T21:13:19.533584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:44:46.344475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.605631+0000","last_change":"2024-04-12T21:13:13.137692+0000","last_active":"2024-04-12T21:13:18.605631+0000","last_peered":"2024-04-12T21:13:18.605631+0000","last_clean":"2024-04-12T21:13:18.605631+0000","last_became_active":"2024-04-12T21:13:13.128523+0000","last_became_peered":"2024-04-12T21:13:13.128523+0000","last_unstale":"2024-04-12T21:13:18.605631+0000","last_undegraded":"2024-04-12T21:13:18.605631+0000","last_fullsized":"2024-04-12T21:13:18.605631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:52:30.847637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535951+0000","last_change":"2024-04-12T21:13:13.149749+0000","last_active":"2024-04-12T21:13:19.535951+0000","last_peered":"2024-04-12T21:13:19.535951+0000","last_clean":"2024-04-12T21:13:19.535951+0000","last_became_active":"2024-04-12T21:13:13.149369+0000","last_became_peered":"2024-04-12T21:13:13.149369+0000","last_unstale":"2024-04-12T21:13:19.535951+0000","last_undegraded":"2024-04-12T21:13:19.535951+0000","last_fullsized":"2024-04-12T21:13:19.535951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:20.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535600+0000","last_change":"2024-04-12T21:13:13.151510+0000","last_active":"2024-04-12T21:13:19.535600+0000","last_peered":"2024-04-12T21:13:19.535600+0000","last_clean":"2024-04-12T21:13:19.535600+0000","last_became_active":"2024-04-12T21:13:13.150620+0000","last_became_peered":"2024-04-12T21:13:13.150620+0000","last_unstale":"2024-04-12T21:13:19.535600+0000","last_undegraded":"2024-04-12T21:13:19.535600+0000","last_fullsized":"2024-04-12T21:13:19.535600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:59:14.772221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.010994+0000","last_change":"2024-04-12T21:13:13.156421+0000","last_active":"2024-04-12T21:13:19.010994+0000","last_peered":"2024-04-12T21:13:19.010994+0000","last_clean":"2024-04-12T21:13:19.010994+0000","last_became_active":"2024-04-12T21:13:13.156042+0000","last_became_peered":"2024-04-12T21:13:13.156042+0000","last_unstale":"2024-04-12T21:13:19.010994+0000","last_undegraded":"2024-04-12T21:13:19.010994+0000","last_fullsized":"2024-04-12T21:13:19.010994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:53:38.234230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.686130+0000","last_change":"2024-04-12T21:13:13.156986+0000","last_active":"2024-04-12T21:13:18.686130+0000","last_peered":"2024-04-12T21:13:18.686130+0000","last_clean":"2024-04-12T21:13:18.686130+0000","last_became_active":"2024-04-12T21:13:13.156705+0000","last_became_peered":"2024-04-12T21:13:13.156705+0000","last_unstale":"2024-04-12T21:13:18.686130+0000","last_undegraded":"2024-04-12T21:13:18.686130+0000","last_fullsized":"2024-04-12T21:13:18.686130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:49:42.442696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.607922+0000","last_change":"2024-04-12T21:13:13.136259+0000","last_active":"2024-04-12T21:13:18.607922+0000","last_peered":"2024-04-12T21:13:18.607922+0000","last_clean":"2024-04-12T21:13:18.607922+0000","last_became_active":"2024-04-12T21:13:13.135865+0000","last_became_peered":"2024-04-12T21:13:13.135865+0000","last_unstale":"2024-04-12T21:13:18.607922+0000","last_undegraded":"2024-04-12T21:13:18.607922+0000","last_fullsized":"2024-04-12T21:13:18.607922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:30:35.934682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601358+0000","last_change":"2024-04-12T21:13:13.140460+0000","last_active":"2024-04-12T21:13:18.601358+0000","last_peered":"2024-04-12T21:13:18.601358+0000","last_clean":"2024-04-12T21:13:18.601358+0000","last_became_active":"2024-04-12T21:13:13.140115+0000","last_became_peered":"2024-04-12T21:13:13.140115+0000","last_unstale":"2024-04-12T21:13:18.601358+0000","last_undegraded":"2024-04-12T21:13:18.601358+0000","last_fullsized":"2024-04-12T21:13:18.601358+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:56:08.880106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601079+0000","last_change":"2024-04-12T21:13:17.606904+0000","last_active":"2024-04-12T21:13:18.601079+0000","last_peered":"2024-04-12T21:13:18.601079+0000","last_clean":"2024-04-12T21:13:18.601079+0000","last_became_active":"2024-04-12T21:13:17.606390+0000","last_became_peered":"2024-04-12T21:13:17.606390+0000","last_unstale":"2024-04-12T21:13:18.601079+0000","last_undegraded":"2024-04-12T21:13:18.601079+0000","last_fullsized":"2024-04-12T21:13:18.601079+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:56:13.440022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.608318+0000","last_change":"2024-04-12T21:13:12.535260+0000","last_active":"2024-04-12T21:13:18.608318+0000","last_peered":"2024-04-12T21:13:18.608318+0000","last_clean":"2024-04-12T21:13:18.608318+0000","last_became_active":"2024-04-12T21:13:12.534798+0000","last_became_peered":"2024-04-12T21:13:12.534798+0000","last_unstale":"2024-04-12T21:13:18.608318+0000","last_undegraded":"2024-04-12T21:13:18.608318+0000","last_fullsized":"2024-04-12T21:13:18.608318+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:33:13.954242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.538074+0000","last_change":"2024-04-12T21:13:13.142701+0000","last_active":"2024-04-12T21:13:19.538074+0000","last_peered":"2024-04-12T21:13:19.538074+0000","last_clean":"2024-04-12T21:13:19.538074+0000","last_became_active":"2024-04-12T21:13:13.142329+0000","last_became_peered":"2024-04-12T21:13:13.142329+0000","last_unstale":"2024-04-12T21:13:19.538074+0000","last_undegraded":"2024-04-12T21:13:19.538074+0000","last_fullsized":"2024-04-12T21:13:19.538074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:59:02.356770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533177+0000","last_change":"2024-04-12T21:13:17.604050+0000","last_active":"2024-04-12T21:13:19.533177+0000","last_peered":"2024-04-12T21:13:19.533177+0000","last_clean":"2024-04-12T21:13:19.533177+0000","last_became_active":"2024-04-12T21:13:17.603628+0000","last_became_peered":"2024-04-12T21:13:17.603628+0000","last_unstale":"2024-04-12T21:13:19.533177+0000","last_undegraded":"2024-04-12T21:13:19.533177+0000","last_fullsized":"2024-04-12T21:13:19.533177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:24:24.986986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.538890+0000","last_change":"2024-04-12T21:13:12.546263+0000","last_active":"2024-04-12T21:13:19.538890+0000","last_peered":"2024-04-12T21:13:19.538890+0000","last_clean":"2024-04-12T21:13:19.538890+0000","last_became_active":"2024-04-12T21:13:12.545710+0000","last_became_peered":"2024-04-12T21:13:12.545710+0000","last_unstale":"2024-04-12T21:13:19.538890+0000","last_undegraded":"2024-04-12T21:13:19.538890+0000","last_fullsized":"2024-04-12T21:13:19.538890+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:18:33.732571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536292+0000","last_change":"2024-04-12T21:13:13.156478+0000","last_active":"2024-04-12T21:13:19.536292+0000","last_peered":"2024-04-12T21:13:19.536292+0000","last_clean":"2024-04-12T21:13:19.536292+0000","last_became_active":"2024-04-12T21:13:13.155485+0000","last_became_peered":"2024-04-12T21:13:13.155485+0000","last_unstale":"2024-04-12T21:13:19.536292+0000","last_undegraded":"2024-04-12T21:13:19.536292+0000","last_fullsized":"2024-04-12T21:13:19.536292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:11:24.161377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533356+0000","last_change":"2024-04-12T21:13:17.607327+0000","last_active":"2024-04-12T21:13:19.533356+0000","last_peered":"2024-04-12T21:13:19.533356+0000","last_clean":"2024-04-12T21:13:19.533356+0000","last_became_active":"2024-04-12T21:13:17.606949+0000","last_became_peered":"2024-04-12T21:13:17.606949+0000","last_unstale":"2024-04-12T21:13:19.533356+0000","last_undegraded":"2024-04-12T21:13:19.533356+0000","last_fullsized":"2024-04-12T21:13:19.533356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:43:27.855461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534426+0000","last_change":"2024-04-12T21:13:12.533082+0000","last_active":"2024-04-12T21:13:19.534426+0000","last_peered":"2024-04-12T21:13:19.534426+0000","last_clean":"2024-04-12T21:13:19.534426+0000","last_became_active":"2024-04-12T21:13:12.532739+0000","last_became_peered":"2024-04-12T21:13:12.532739+0000","last_unstale":"2024-04-12T21:13:19.534426+0000","last_undegraded":"2024-04-12T21:13:19.534426+0000","last_fullsized":"2024-04-12T21:13:19.534426+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:40:01.885223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011288+0000","last_change":"2024-04-12T21:13:13.136718+0000","last_active":"2024-04-12T21:13:19.011288+0000","last_peered":"2024-04-12T21:13:19.011288+0000","last_clean":"2024-04-12T21:13:19.011288+0000","last_became_active":"2024-04-12T21:13:13.136357+0000","last_became_peered":"2024-04-12T21:13:13.136357+0000","last_unstale":"2024-04-12T21:13:19.011288+0000","last_undegraded":"2024-04-12T21:13:19.011288+0000","last_fullsized":"2024-04-12T21:13:19.011288+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:08:57.380556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533366+0000","last_change":"2024-04-12T21:13:17.601381+0000","last_active":"2024-04-12T21:13:19.533366+0000","last_peered":"2024-04-12T21:13:19.533366+0000","last_clean":"2024-04-12T21:13:19.533366+0000","last_became_active":"2024-04-12T21:13:17.600868+0000","last_became_peered":"2024-04-12T21:13:17.600868+0000","last_unstale":"2024-04-12T21:13:19.533366+0000","last_undegraded":"2024-04-12T21:13:19.533366+0000","last_fullsized":"2024-04-12T21:13:19.533366+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:53:36.234779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534532+0000","last_change":"2024-04-12T21:13:12.528285+0000","last_active":"2024-04-12T21:13:19.534532+0000","last_peered":"2024-04-12T21:13:19.534532+0000","last_clean":"2024-04-12T21:13:19.534532+0000","last_became_active":"2024-04-12T21:13:12.527915+0000","last_became_peered":"2024-04-12T21:13:12.527915+0000","last_unstale":"2024-04-12T21:13:19.534532+0000","last_undegraded":"2024-04-12T21:13:19.534532+0000","last_fullsized":"2024-04-12T21:13:19.534532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:44.119836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534987+0000","last_change":"2024-04-12T21:13:13.128654+0000","last_active":"2024-04-12T21:13:19.534987+0000","last_peered":"2024-04-12T21:13:19.534987+0000","last_clean":"2024-04-12T21:13:19.534987+0000","last_became_active":"2024-04-12T21:13:13.128274+0000","last_became_peered":"2024-04-12T21:13:13.128274+0000","last_unstale":"2024-04-12T21:13:19.534987+0000","last_undegraded":"2024-04-12T21:13:19.534987+0000","last_fullsized":"2024-04-12T21:13:19.534987+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:48:08.827157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601427+0000","last_change":"2024-04-12T21:13:17.594360+0000","last_active":"2024-04-12T21:13:18.601427+0000","last_peered":"2024-04-12T21:13:18.601427+0000","last_clean":"2024-04-12T21:13:18.601427+0000","last_became_active":"2024-04-12T21:13:17.593924+0000","last_became_peered":"2024-04-12T21:13:17.593924+0000","last_unstale":"2024-04-12T21:13:18.601427+0000","last_undegraded":"2024-04-12T21:13:18.601427+0000","last_fullsized":"2024-04-12T21:13:18.601427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:42:14.269657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534575+0000","last_change":"2024-04-12T21:13:12.532481+0000","last_active":"2024-04-12T21:13:19.534575+0000","last_peered":"2024-04-12T21:13:19.534575+0000","last_clean":"2024-04-12T21:13:19.534575+0000","last_became_active":"2024-04-12T21:13:12.532126+0000","last_became_peered":"2024-04-12T21:13:12.532126+0000","last_unstale":"2024-04-12T21:13:19.534575+0000","last_undegraded":"2024-04-12T21:13:19.534575+0000","last_fullsized":"2024-04-12T21:13:19.534575+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:19:52.510076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534475+0000","last_change":"2024-04-12T21:13:13.138290+0000","last_active":"2024-04-12T21:13:19.534475+0000","last_peered":"2024-04-12T21:13:19.534475+0000","last_clean":"2024-04-12T21:13:19.534475+0000","last_became_active":"2024-04-12T21:13:13.137854+0000","last_became_peered":"2024-04-12T21:13:13.137854+0000","last_unstale":"2024-04-12T21:13:19.534475+0000","last_undegraded":"2024-04-12T21:13:19.534475+0000","last_fullsized":"2024-04-12T21:13:19.534475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:27:45.455704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601280+0000","last_change":"2024-04-12T21:13:17.605453+0000","last_active":"2024-04-12T21:13:18.601280+0000","last_peered":"2024-04-12T21:13:18.601280+0000","last_clean":"2024-04-12T21:13:18.601280+0000","last_became_active":"2024-04-12T21:13:17.604918+0000","last_became_peered":"2024-04-12T21:13:17.604918+0000","last_unstale":"2024-04-12T21:13:18.601280+0000","last_undegraded":"2024-04-12T21:13:18.601280+0000","last_fullsized":"2024-04-12T21:13:18.601280+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:14:40.800326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247423+0000","last_change":"2024-04-12T21:13:12.539083+0000","last_active":"2024-04-12T21:13:19.247423+0000","last_peered":"2024-04-12T21:13:19.247423+0000","last_clean":"2024-04-12T21:13:19.247423+0000","last_became_active":"2024-04-12T21:13:12.538771+0000","last_became_peered":"2024-04-12T21:13:12.538771+0000","last_unstale":"2024-04-12T21:13:19.247423+0000","last_undegraded":"2024-04-12T21:13:19.247423+0000","last_fullsized":"2024-04-12T21:13:19.247423+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:31.560665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534689+0000","last_change":"2024-04-12T21:13:13.146087+0000","last_active":"2024-04-12T21:13:19.534689+0000","last_peered":"2024-04-12T21:13:19.534689+0000","last_clean":"2024-04-12T21:13:19.534689+0000","last_became_active":"2024-04-12T21:13:13.145450+0000","last_became_peered":"2024-04-12T21:13:13.145450+0000","last_unstale":"2024-04-12T21:13:19.534689+0000","last_undegraded":"2024-04-12T21:13:19.534689+0000","last_fullsized":"2024-04-12T21:13:19.534689+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:03:07.727779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.686197+0000","last_change":"2024-04-12T21:13:17.607603+0000","last_active":"2024-04-12T21:13:18.686197+0000","last_peered":"2024-04-12T21:13:18.686197+0000","last_clean":"2024-04-12T21:13:18.686197+0000","last_became_active":"2024-04-12T21:13:17.606972+0000","last_became_peered":"2024-04-12T21:13:17.606972+0000","last_unstale":"2024-04-12T21:13:18.686197+0000","last_undegraded":"2024-04-12T21:13:18.686197+0000","last_fullsized":"2024-04-12T21:13:18.686197+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:16:37.593158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604108+0000","last_change":"2024-04-12T21:13:12.541405+0000","last_active":"2024-04-12T21:13:18.604108+0000","last_peered":"2024-04-12T21:13:18.604108+0000","last_clean":"2024-04-12T21:13:18.604108+0000","last_became_active":"2024-04-12T21:13:12.540612+0000","last_became_peered":"2024-04-12T21:13:12.540612+0000","last_unstale":"2024-04-12T21:13:18.604108+0000","last_undegraded":"2024-04-12T21:13:18.604108+0000","last_fullsized":"2024-04-12T21:13:18.604108+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:38:10.749857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535348+0000","last_change":"2024-04-12T21:13:13.138447+0000","last_active":"2024-04-12T21:13:19.535348+0000","last_peered":"2024-04-12T21:13:19.535348+0000","last_clean":"2024-04-12T21:13:19.535348+0000","last_became_active":"2024-04-12T21:13:13.137990+0000","last_became_peered":"2024-04-12T21:13:13.137990+0000","last_unstale":"2024-04-12T21:13:19.535348+0000","last_undegraded":"2024-04-12T21:13:19.535348+0000","last_fullsized":"2024-04-12T21:13:19.535348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:58:50.381257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536769+0000","last_change":"2024-04-12T21:13:17.614003+0000","last_active":"2024-04-12T21:13:19.536769+0000","last_peered":"2024-04-12T21:13:19.536769+0000","last_clean":"2024-04-12T21:13:19.536769+0000","last_became_active":"2024-04-12T21:13:17.613517+0000","last_became_peered":"2024-04-12T21:13:17.613517+0000","last_unstale":"2024-04-12T21:13:19.536769+0000","last_undegraded":"2024-04-12T21:13:19.536769+0000","last_fullsized":"2024-04-12T21:13:19.536769+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:33.590816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533995+0000","last_change":"2024-04-12T21:13:12.530503+0000","last_active":"2024-04-12T21:13:19.533995+0000","last_peered":"2024-04-12T21:13:19.533995+0000","last_clean":"2024-04-12T21:13:19.533995+0000","last_became_active":"2024-04-12T21:13:12.530161+0000","last_became_peered":"2024-04-12T21:13:12.530161+0000","last_unstale":"2024-04-12T21:13:19.533995+0000","last_undegraded":"2024-04-12T21:13:19.533995+0000","last_fullsized":"2024-04-12T21:13:19.533995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:06:18.363283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604050+0000","last_change":"2024-04-12T21:13:13.146401+0000","last_active":"2024-04-12T21:13:18.604050+0000","last_peered":"2024-04-12T21:13:18.604050+0000","last_clean":"2024-04-12T21:13:18.604050+0000","last_became_active":"2024-04-12T21:13:13.145527+0000","last_became_peered":"2024-04-12T21:13:13.145527+0000","last_unstale":"2024-04-12T21:13:18.604050+0000","last_undegraded":"2024-04-12T21:13:18.604050+0000","last_fullsized":"2024-04-12T21:13:18.604050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:40:41.302305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535689+0000","last_change":"2024-04-12T21:13:17.609141+0000","last_active":"2024-04-12T21:13:19.535689+0000","last_peered":"2024-04-12T21:13:19.535689+0000","last_clean":"2024-04-12T21:13:19.535689+0000","last_became_active":"2024-04-12T21:13:17.603368+0000","last_became_peered":"2024-04-12T21:13:17.603368+0000","last_unstale":"2024-04-12T21:13:19.535689+0000","last_undegraded":"2024-04-12T21:13:19.535689+0000","last_fullsized":"2024-04-12T21:13:19.535689+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:07:54.235284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247866+0000","last_change":"2024-04-12T21:13:12.540388+0000","last_active":"2024-04-12T21:13:19.247866+0000","last_peered":"2024-04-12T21:13:19.247866+0000","last_clean":"2024-04-12T21:13:19.247866+0000","last_became_active":"2024-04-12T21:13:12.540033+0000","last_became_peered":"2024-04-12T21:13:12.540033+0000","last_unstale":"2024-04-12T21:13:19.247866+0000","last_undegraded":"2024-04-12T21:13:19.247866+0000","last_fullsized":"2024-04-12T21:13:19.247866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:46:53.785074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011582+0000","last_change":"2024-04-12T21:13:13.137208+0000","last_active":"2024-04-12T21:13:19.011582+0000","last_peered":"2024-04-12T21:13:19.011582+0000","last_clean":"2024-04-12T21:13:19.011582+0000","last_became_active":"2024-04-12T21:13:13.136866+0000","last_became_peered":"2024-04-12T21:13:13.136866+0000","last_unstale":"2024-04-12T21:13:19.011582+0000","last_undegraded":"2024-04-12T21:13:19.011582+0000","last_fullsized":"2024-04-12T21:13:19.011582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:48:36.655808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604760+0000","last_change":"2024-04-12T21:13:17.617564+0000","last_active":"2024-04-12T21:13:18.604760+0000","last_peered":"2024-04-12T21:13:18.604760+0000","last_clean":"2024-04-12T21:13:18.604760+0000","last_became_active":"2024-04-12T21:13:17.615980+0000","last_became_peered":"2024-04-12T21:13:17.615980+0000","last_unstale":"2024-04-12T21:13:18.604760+0000","last_undegraded":"2024-04-12T21:13:18.604760+0000","last_fullsized":"2024-04-12T21:13:18.604760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:22:38.370357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535118+0000","last_change":"2024-04-12T21:13:12.532732+0000","last_active":"2024-04-12T21:13:19.535118+0000","last_peered":"2024-04-12T21:13:19.535118+0000","last_clean":"2024-04-12T21:13:19.535118+0000","last_became_active":"2024-04-12T21:13:12.532336+0000","last_became_peered":"2024-04-12T21:13:12.532336+0000","last_unstale":"2024-04-12T21:13:19.535118+0000","last_undegraded":"2024-04-12T21:13:19.535118+0000","last_fullsized":"2024-04-12T21:13:19.535118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:43:20.472987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.537029+0000","last_change":"2024-04-12T21:13:13.152737+0000","last_active":"2024-04-12T21:13:19.537029+0000","last_peered":"2024-04-12T21:13:19.537029+0000","last_clean":"2024-04-12T21:13:19.537029+0000","last_became_active":"2024-04-12T21:13:13.150914+0000","last_became_peered":"2024-04-12T21:13:13.150914+0000","last_unstale":"2024-04-12T21:13:19.537029+0000","last_undegraded":"2024-04-12T21:13:19.537029+0000","last_fullsized":"2024-04-12T21:13:19.537029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:51:29.020746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533641+0000","last_change":"2024-04-12T21:13:17.604826+0000","last_active":"2024-04-12T21:13:19.533641+0000","last_peered":"2024-04-12T21:13:19.533641+0000","last_clean":"2024-04-12T21:13:19.533641+0000","last_became_active":"2024-04-12T21:13:17.604427+0000","last_became_peered":"2024-04-12T21:13:17.604427+0000","last_unstale":"2024-04-12T21:13:19.533641+0000","last_undegraded":"2024-04-12T21:13:19.533641+0000","last_fullsized":"2024-04-12T21:13:19.533641+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:04:55.602539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534864+0000","last_change":"2024-04-12T21:13:12.542641+0000","last_active":"2024-04-12T21:13:19.534864+0000","last_peered":"2024-04-12T21:13:19.534864+0000","last_clean":"2024-04-12T21:13:19.534864+0000","last_became_active":"2024-04-12T21:13:12.542291+0000","last_became_peered":"2024-04-12T21:13:12.542291+0000","last_unstale":"2024-04-12T21:13:19.534864+0000","last_undegraded":"2024-04-12T21:13:19.534864+0000","last_fullsized":"2024-04-12T21:13:19.534864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:37:00.186153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011383+0000","last_change":"2024-04-12T21:13:13.129280+0000","last_active":"2024-04-12T21:13:19.011383+0000","last_peered":"2024-04-12T21:13:19.011383+0000","last_clean":"2024-04-12T21:13:19.011383+0000","last_became_active":"2024-04-12T21:13:13.128881+0000","last_became_peered":"2024-04-12T21:13:13.128881+0000","last_unstale":"2024-04-12T21:13:19.011383+0000","last_undegraded":"2024-04-12T21:13:19.011383+0000","last_fullsized":"2024-04-12T21:13:19.011383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:00:23.629024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604875+0000","last_change":"2024-04-12T21:13:17.606533+0000","last_active":"2024-04-12T21:13:18.604875+0000","last_peered":"2024-04-12T21:13:18.604875+0000","last_clean":"2024-04-12T21:13:18.604875+0000","last_became_active":"2024-04-12T21:13:17.606092+0000","last_became_peered":"2024-04-12T21:13:17.606092+0000","last_unstale":"2024-04-12T21:13:18.604875+0000","last_undegraded":"2024-04-12T21:13:18.604875+0000","last_fullsized":"2024-04-12T21:13:18.604875+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:55:56.080659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'21","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:40.534202+0000","last_change":"2024-04-12T21:13:12.531797+0000","last_active":"2024-04-12T21:13:40.534202+0000","last_peered":"2024-04-12T21:13:40.534202+0000","last_clean":"2024-04-12T21:13:40.534202+0000","last_became_active":"2024-04-12T21:13:12.531346+0000","last_became_peered":"2024-04-12T21:13:12.531346+0000","last_unstale":"2024-04-12T21:13:40.534202+0000","last_undegraded":"2024-04-12T21:13:40.534202+0000","last_fullsized":"2024-04-12T21:13:40.534202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:59:32.429211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40471,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":29,"num_write":21,"num_write_kb":48,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534281+0000","last_change":"2024-04-12T21:13:13.131501+0000","last_active":"2024-04-12T21:13:19.534281+0000","last_peered":"2024-04-12T21:13:19.534281+0000","last_clean":"2024-04-12T21:13:19.534281+0000","last_became_active":"2024-04-12T21:13:13.131161+0000","last_became_peered":"2024-04-12T21:13:13.131161+0000","last_unstale":"2024-04-12T21:13:19.534281+0000","last_undegraded":"2024-04-12T21:13:19.534281+0000","last_fullsized":"2024-04-12T21:13:19.534281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:45:27.046250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535286+0000","last_change":"2024-04-12T21:13:17.610917+0000","last_active":"2024-04-12T21:13:19.535286+0000","last_peered":"2024-04-12T21:13:19.535286+0000","last_clean":"2024-04-12T21:13:19.535286+0000","last_became_active":"2024-04-12T21:13:17.610418+0000","last_became_peered":"2024-04-12T21:13:17.610418+0000","last_unstale":"2024-04-12T21:13:19.535286+0000","last_undegraded":"2024-04-12T21:13:19.535286+0000","last_fullsized":"2024-04-12T21:13:19.535286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:15:48.032310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601587+0000","last_change":"2024-04-12T21:13:12.529571+0000","last_active":"2024-04-12T21:13:18.601587+0000","last_peered":"2024-04-12T21:13:18.601587+0000","last_clean":"2024-04-12T21:13:18.601587+0000","last_became_active":"2024-04-12T21:13:12.529250+0000","last_became_peered":"2024-04-12T21:13:12.529250+0000","last_unstale":"2024-04-12T21:13:18.601587+0000","last_undegraded":"2024-04-12T21:13:18.601587+0000","last_fullsized":"2024-04-12T21:13:18.601587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:35:53.476770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.605570+0000","last_change":"2024-04-12T21:13:13.150100+0000","last_active":"2024-04-12T21:13:18.605570+0000","last_peered":"2024-04-12T21:13:18.605570+0000","last_clean":"2024-04-12T21:13:18.605570+0000","last_became_active":"2024-04-12T21:13:13.149711+0000","last_became_peered":"2024-04-12T21:13:13.149711+0000","last_unstale":"2024-04-12T21:13:18.605570+0000","last_undegraded":"2024-04-12T21:13:18.605570+0000","last_fullsized":"2024-04-12T21:13:18.605570+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:33:27.121682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685975+0000","last_change":"2024-04-12T21:13:17.605440+0000","last_active":"2024-04-12T21:13:18.685975+0000","last_peered":"2024-04-12T21:13:18.685975+0000","last_clean":"2024-04-12T21:13:18.685975+0000","last_became_active":"2024-04-12T21:13:17.604899+0000","last_became_peered":"2024-04-12T21:13:17.604899+0000","last_unstale":"2024-04-12T21:13:18.685975+0000","last_undegraded":"2024-04-12T21:13:18.685975+0000","last_fullsized":"2024-04-12T21:13:18.685975+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:53:01.678608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534489+0000","last_change":"2024-04-12T21:13:12.545077+0000","last_active":"2024-04-12T21:13:19.534489+0000","last_peered":"2024-04-12T21:13:19.534489+0000","last_clean":"2024-04-12T21:13:19.534489+0000","last_became_active":"2024-04-12T21:13:12.544755+0000","last_became_peered":"2024-04-12T21:13:12.544755+0000","last_unstale":"2024-04-12T21:13:19.534489+0000","last_undegraded":"2024-04-12T21:13:19.534489+0000","last_fullsized":"2024-04-12T21:13:19.534489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:31:28.487336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535204+0000","last_change":"2024-04-12T21:13:13.128047+0000","last_active":"2024-04-12T21:13:19.535204+0000","last_peered":"2024-04-12T21:13:19.535204+0000","last_clean":"2024-04-12T21:13:19.535204+0000","last_became_active":"2024-04-12T21:13:13.127692+0000","last_became_peered":"2024-04-12T21:13:13.127692+0000","last_unstale":"2024-04-12T21:13:19.535204+0000","last_undegraded":"2024-04-12T21:13:19.535204+0000","last_fullsized":"2024-04-12T21:13:19.535204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:43:29.552438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.608039+0000","last_change":"2024-04-12T21:13:17.614240+0000","last_active":"2024-04-12T21:13:18.608039+0000","last_peered":"2024-04-12T21:13:18.608039+0000","last_clean":"2024-04-12T21:13:18.608039+0000","last_became_active":"2024-04-12T21:13:17.613834+0000","last_became_peered":"2024-04-12T21:13:17.613834+0000","last_unstale":"2024-04-12T21:13:18.608039+0000","last_undegraded":"2024-04-12T21:13:18.608039+0000","last_fullsized":"2024-04-12T21:13:18.608039+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:29:23.949101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533431+0000","last_change":"2024-04-12T21:13:12.527868+0000","last_active":"2024-04-12T21:13:19.533431+0000","last_peered":"2024-04-12T21:13:19.533431+0000","last_clean":"2024-04-12T21:13:19.533431+0000","last_became_active":"2024-04-12T21:13:12.527521+0000","last_became_peered":"2024-04-12T21:13:12.527521+0000","last_unstale":"2024-04-12T21:13:19.533431+0000","last_undegraded":"2024-04-12T21:13:19.533431+0000","last_fullsized":"2024-04-12T21:13:19.533431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:46:23.649552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534851+0000","last_change":"2024-04-12T21:13:13.124207+0000","last_active":"2024-04-12T21:13:19.534851+0000","last_peered":"2024-04-12T21:13:19.534851+0000","last_clean":"2024-04-12T21:13:19.534851+0000","last_became_active":"2024-04-12T21:13:13.123866+0000","last_became_peered":"2024-04-12T21:13:13.123866+0000","last_unstale":"2024-04-12T21:13:19.534851+0000","last_undegraded":"2024-04-12T21:13:19.534851+0000","last_fullsized":"2024-04-12T21:13:19.534851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:04.263260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606612+0000","last_change":"2024-04-12T21:13:17.613064+0000","last_active":"2024-04-12T21:13:18.606612+0000","last_peered":"2024-04-12T21:13:18.606612+0000","last_clean":"2024-04-12T21:13:18.606612+0000","last_became_active":"2024-04-12T21:13:17.612569+0000","last_became_peered":"2024-04-12T21:13:17.612569+0000","last_unstale":"2024-04-12T21:13:18.606612+0000","last_undegraded":"2024-04-12T21:13:18.606612+0000","last_fullsized":"2024-04-12T21:13:18.606612+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:35:27.274086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533625+0000","last_change":"2024-04-12T21:13:12.543193+0000","last_active":"2024-04-12T21:13:19.533625+0000","last_peered":"2024-04-12T21:13:19.533625+0000","last_clean":"2024-04-12T21:13:19.533625+0000","last_became_active":"2024-04-12T21:13:12.542885+0000","last_became_peered":"2024-04-12T21:13:12.542885+0000","last_unstale":"2024-04-12T21:13:19.533625+0000","last_undegraded":"2024-04-12T21:13:19.533625+0000","last_fullsized":"2024-04-12T21:13:19.533625+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:02:11.100366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534264+0000","last_change":"2024-04-12T21:13:13.130682+0000","last_active":"2024-04-12T21:13:19.534264+0000","last_peered":"2024-04-12T21:13:19.534264+0000","last_clean":"2024-04-12T21:13:19.534264+0000","last_became_active":"2024-04-12T21:13:13.130299+0000","last_became_peered":"2024-04-12T21:13:13.130299+0000","last_unstale":"2024-04-12T21:13:19.534264+0000","last_undegraded":"2024-04-12T21:13:19.534264+0000","last_fullsized":"2024-04-12T21:13:19.534264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:52:02.094959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534309+0000","last_change":"2024-04-12T21:13:17.609472+0000","last_active":"2024-04-12T21:13:19.534309+0000","last_peered":"2024-04-12T21:13:19.534309+0000","last_clean":"2024-04-12T21:13:19.534309+0000","last_became_active":"2024-04-12T21:13:17.608991+0000","last_became_peered":"2024-04-12T21:13:17.608991+0000","last_unstale":"2024-04-12T21:13:19.534309+0000","last_undegraded":"2024-04-12T21:13:19.534309+0000","last_fullsized":"2024-04-12T21:13:19.534309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:17:57.962701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534436+0000","last_change":"2024-04-12T21:13:12.530058+0000","last_active":"2024-04-12T21:13:19.534436+0000","last_peered":"2024-04-12T21:13:19.534436+0000","last_clean":"2024-04-12T21:13:19.534436+0000","last_became_active":"2024-04-12T21:13:12.529682+0000","last_became_peered":"2024-04-12T21:13:12.529682+0000","last_unstale":"2024-04-12T21:13:19.534436+0000","last_undegraded":"2024-04-12T21:13:19.534436+0000","last_fullsized":"2024-04-12T21:13:19.534436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:17:25.621593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685972+0000","last_change":"2024-04-12T21:13:13.129975+0000","last_active":"2024-04-12T21:13:18.685972+0000","last_peered":"2024-04-12T21:13:18.685972+0000","last_clean":"2024-04-12T21:13:18.685972+0000","last_became_active":"2024-04-12T21:13:13.129621+0000","last_became_peered":"2024-04-12T21:13:13.129621+0000","last_unstale":"2024-04-12T21:13:18.685972+0000","last_undegraded":"2024-04-12T21:13:18.685972+0000","last_fullsized":"2024-04-12T21:13:18.685972+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:49:48.267685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533966+0000","last_change":"2024-04-12T21:13:17.738019+0000","last_active":"2024-04-12T21:13:19.533966+0000","last_peered":"2024-04-12T21:13:19.533966+0000","last_clean":"2024-04-12T21:13:19.533966+0000","last_became_active":"2024-04-12T21:13:17.737553+0000","last_became_peered":"2024-04-12T21:13:17.737553+0000","last_unstale":"2024-04-12T21:13:19.533966+0000","last_undegraded":"2024-04-12T21:13:19.533966+0000","last_fullsized":"2024-04-12T21:13:19.533966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:29:07.288025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606241+0000","last_change":"2024-04-12T21:13:12.544661+0000","last_active":"2024-04-12T21:13:18.606241+0000","last_peered":"2024-04-12T21:13:18.606241+0000","last_clean":"2024-04-12T21:13:18.606241+0000","last_became_active":"2024-04-12T21:13:12.544321+0000","last_became_peered":"2024-04-12T21:13:12.544321+0000","last_unstale":"2024-04-12T21:13:18.606241+0000","last_undegraded":"2024-04-12T21:13:18.606241+0000","last_fullsized":"2024-04-12T21:13:18.606241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:55:43.517970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247739+0000","last_change":"2024-04-12T21:13:13.140480+0000","last_active":"2024-04-12T21:13:19.247739+0000","last_peered":"2024-04-12T21:13:19.247739+0000","last_clean":"2024-04-12T21:13:19.247739+0000","last_became_active":"2024-04-12T21:13:13.140128+0000","last_became_peered":"2024-04-12T21:13:13.140128+0000","last_unstale":"2024-04-12T21:13:19.247739+0000","last_undegraded":"2024-04-12T21:13:19.247739+0000","last_fullsized":"2024-04-12T21:13:19.247739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:40:32.066929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606642+0000","last_change":"2024-04-12T21:13:17.609235+0000","last_active":"2024-04-12T21:13:18.606642+0000","last_peered":"2024-04-12T21:13:18.606642+0000","last_clean":"2024-04-12T21:13:18.606642+0000","last_became_active":"2024-04-12T21:13:17.608814+0000","last_became_peered":"2024-04-12T21:13:17.608814+0000","last_unstale":"2024-04-12T21:13:18.606642+0000","last_undegraded":"2024-04-12T21:13:18.606642+0000","last_fullsized":"2024-04-12T21:13:18.606642+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:15:35.287239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535019+0000","last_change":"2024-04-12T21:13:12.539638+0000","last_active":"2024-04-12T21:13:19.535019+0000","last_peered":"2024-04-12T21:13:19.535019+0000","last_clean":"2024-04-12T21:13:19.535019+0000","last_became_active":"2024-04-12T21:13:12.539272+0000","last_became_peered":"2024-04-12T21:13:12.539272+0000","last_unstale":"2024-04-12T21:13:19.535019+0000","last_undegraded":"2024-04-12T21:13:19.535019+0000","last_fullsized":"2024-04-12T21:13:19.535019+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:11:42.992489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.247490+0000","last_change":"2024-04-12T21:13:13.149326+0000","last_active":"2024-04-12T21:13:19.247490+0000","last_peered":"2024-04-12T21:13:19.247490+0000","last_clean":"2024-04-12T21:13:19.247490+0000","last_became_active":"2024-04-12T21:13:13.149023+0000","last_became_peered":"2024-04-12T21:13:13.149023+0000","last_unstale":"2024-04-12T21:13:19.247490+0000","last_undegraded":"2024-04-12T21:13:19.247490+0000","last_fullsized":"2024-04-12T21:13:19.247490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:15:58.479778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606944+0000","last_change":"2024-04-12T21:13:17.740067+0000","last_active":"2024-04-12T21:13:18.606944+0000","last_peered":"2024-04-12T21:13:18.606944+0000","last_clean":"2024-04-12T21:13:18.606944+0000","last_became_active":"2024-04-12T21:13:17.738186+0000","last_became_peered":"2024-04-12T21:13:17.738186+0000","last_unstale":"2024-04-12T21:13:18.606944+0000","last_undegraded":"2024-04-12T21:13:18.606944+0000","last_fullsized":"2024-04-12T21:13:18.606944+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:15:05.241855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534206+0000","last_change":"2024-04-12T21:13:13.129755+0000","last_active":"2024-04-12T21:13:19.534206+0000","last_peered":"2024-04-12T21:13:19.534206+0000","last_clean":"2024-04-12T21:13:19.534206+0000","last_became_active":"2024-04-12T21:13:13.129422+0000","last_became_peered":"2024-04-12T21:13:13.129422+0000","last_unstale":"2024-04-12T21:13:19.534206+0000","last_undegraded":"2024-04-12T21:13:19.534206+0000","last_fullsized":"2024-04-12T21:13:19.534206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:59:08.675086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.604081+0000","last_change":"2024-04-12T21:13:12.551549+0000","last_active":"2024-04-12T21:13:18.604081+0000","last_peered":"2024-04-12T21:13:18.604081+0000","last_clean":"2024-04-12T21:13:18.604081+0000","last_became_active":"2024-04-12T21:13:12.551204+0000","last_became_peered":"2024-04-12T21:13:12.551204+0000","last_unstale":"2024-04-12T21:13:18.604081+0000","last_undegraded":"2024-04-12T21:13:18.604081+0000","last_fullsized":"2024-04-12T21:13:18.604081+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:33:06.721971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:20.649632+0000","last_change":"2024-04-12T21:13:17.603828+0000","last_active":"2024-04-12T21:13:20.649632+0000","last_peered":"2024-04-12T21:13:20.649632+0000","last_clean":"2024-04-12T21:13:20.649632+0000","last_became_active":"2024-04-12T21:13:17.603399+0000","last_became_peered":"2024-04-12T21:13:17.603399+0000","last_unstale":"2024-04-12T21:13:20.649632+0000","last_undegraded":"2024-04-12T21:13:20.649632+0000","last_fullsized":"2024-04-12T21:13:20.649632+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:22:28.126722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.537753+0000","last_change":"2024-04-12T21:13:13.137828+0000","last_active":"2024-04-12T21:13:19.537753+0000","last_peered":"2024-04-12T21:13:19.537753+0000","last_clean":"2024-04-12T21:13:19.537753+0000","last_became_active":"2024-04-12T21:13:13.137534+0000","last_became_peered":"2024-04-12T21:13:13.137534+0000","last_unstale":"2024-04-12T21:13:19.537753+0000","last_undegraded":"2024-04-12T21:13:19.537753+0000","last_fullsized":"2024-04-12T21:13:19.537753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:24:20.238210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533345+0000","last_change":"2024-04-12T21:13:12.533215+0000","last_active":"2024-04-12T21:13:19.533345+0000","last_peered":"2024-04-12T21:13:19.533345+0000","last_clean":"2024-04-12T21:13:19.533345+0000","last_became_active":"2024-04-12T21:13:12.532852+0000","last_became_peered":"2024-04-12T21:13:12.532852+0000","last_unstale":"2024-04-12T21:13:19.533345+0000","last_undegraded":"2024-04-12T21:13:19.533345+0000","last_fullsized":"2024-04-12T21:13:19.533345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:41:17.861406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":346,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534687+0000","last_change":"2024-04-12T21:11:53.092756+0000","last_active":"2024-04-12T21:13:19.534687+0000","last_peered":"2024-04-12T21:13:19.534687+0000","last_clean":"2024-04-12T21:13:19.534687+0000","last_became_active":"2024-04-12T21:11:53.092364+0000","last_became_peered":"2024-04-12T21:11:53.092364+0000","last_unstale":"2024-04-12T21:13:19.534687+0000","last_undegraded":"2024-04-12T21:13:19.534687+0000","last_fullsized":"2024-04-12T21:13:19.534687+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:08:04.402338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:08:04.402338+0000","last_clean_scrub_stamp":"2024-04-12T21:08:04.402338+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:26:11.910015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534993+0000","last_change":"2024-04-12T21:13:17.738168+0000","last_active":"2024-04-12T21:13:19.534993+0000","last_peered":"2024-04-12T21:13:19.534993+0000","last_clean":"2024-04-12T21:13:19.534993+0000","last_became_active":"2024-04-12T21:13:17.737697+0000","last_became_peered":"2024-04-12T21:13:17.737697+0000","last_unstale":"2024-04-12T21:13:19.534993+0000","last_undegraded":"2024-04-12T21:13:19.534993+0000","last_fullsized":"2024-04-12T21:13:19.534993+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:11:31.025636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.603869+0000","last_change":"2024-04-12T21:13:13.130398+0000","last_active":"2024-04-12T21:13:18.603869+0000","last_peered":"2024-04-12T21:13:18.603869+0000","last_clean":"2024-04-12T21:13:18.603869+0000","last_became_active":"2024-04-12T21:13:13.129488+0000","last_became_peered":"2024-04-12T21:13:13.129488+0000","last_unstale":"2024-04-12T21:13:18.603869+0000","last_undegraded":"2024-04-12T21:13:18.603869+0000","last_fullsized":"2024-04-12T21:13:18.603869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:21:03.737140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534789+0000","last_change":"2024-04-12T21:13:12.533367+0000","last_active":"2024-04-12T21:13:19.534789+0000","last_peered":"2024-04-12T21:13:19.534789+0000","last_clean":"2024-04-12T21:13:19.534789+0000","last_became_active":"2024-04-12T21:13:12.533042+0000","last_became_peered":"2024-04-12T21:13:12.533042+0000","last_unstale":"2024-04-12T21:13:19.534789+0000","last_undegraded":"2024-04-12T21:13:19.534789+0000","last_fullsized":"2024-04-12T21:13:19.534789+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:56:07.671693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685862+0000","last_change":"2024-04-12T21:13:17.593951+0000","last_active":"2024-04-12T21:13:18.685862+0000","last_peered":"2024-04-12T21:13:18.685862+0000","last_clean":"2024-04-12T21:13:18.685862+0000","last_became_active":"2024-04-12T21:13:17.593528+0000","last_became_peered":"2024-04-12T21:13:17.593528+0000","last_unstale":"2024-04-12T21:13:18.685862+0000","last_undegraded":"2024-04-12T21:13:18.685862+0000","last_fullsized":"2024-04-12T21:13:18.685862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:24:25.081109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534323+0000","last_change":"2024-04-12T21:13:12.543040+0000","last_active":"2024-04-12T21:13:19.534323+0000","last_peered":"2024-04-12T21:13:19.534323+0000","last_clean":"2024-04-12T21:13:19.534323+0000","last_became_active":"2024-04-12T21:13:12.542698+0000","last_became_peered":"2024-04-12T21:13:12.542698+0000","last_unstale":"2024-04-12T21:13:19.534323+0000","last_undegraded":"2024-04-12T21:13:19.534323+0000","last_fullsized":"2024-04-12T21:13:19.534323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:57:55.712648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.607050+0000","last_change":"2024-04-12T21:13:13.138975+0000","last_active":"2024-04-12T21:13:18.607050+0000","last_peered":"2024-04-12T21:13:18.607050+0000","last_clean":"2024-04-12T21:13:18.607050+0000","last_became_active":"2024-04-12T21:13:13.138538+0000","last_became_peered":"2024-04-12T21:13:13.138538+0000","last_unstale":"2024-04-12T21:13:18.607050+0000","last_undegraded":"2024-04-12T21:13:18.607050+0000","last_fullsized":"2024-04-12T21:13:18.607050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:12:10.488914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535679+0000","last_change":"2024-04-12T21:13:17.739156+0000","last_active":"2024-04-12T21:13:19.535679+0000","last_peered":"2024-04-12T21:13:19.535679+0000","last_clean":"2024-04-12T21:13:19.535679+0000","last_became_active":"2024-04-12T21:13:17.738153+0000","last_became_peered":"2024-04-12T21:13:17.738153+0000","last_unstale":"2024-04-12T21:13:19.535679+0000","last_undegraded":"2024-04-12T21:13:19.535679+0000","last_fullsized":"2024-04-12T21:13:19.535679+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:18:18.930005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.607810+0000","last_change":"2024-04-12T21:13:12.547658+0000","last_active":"2024-04-12T21:13:18.607810+0000","last_peered":"2024-04-12T21:13:18.607810+0000","last_clean":"2024-04-12T21:13:18.607810+0000","last_became_active":"2024-04-12T21:13:12.547197+0000","last_became_peered":"2024-04-12T21:13:12.547197+0000","last_unstale":"2024-04-12T21:13:18.607810+0000","last_undegraded":"2024-04-12T21:13:18.607810+0000","last_fullsized":"2024-04-12T21:13:18.607810+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:57:18.066730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.537968+0000","last_change":"2024-04-12T21:13:13.143876+0000","last_active":"2024-04-12T21:13:19.537968+0000","last_peered":"2024-04-12T21:13:19.537968+0000","last_clean":"2024-04-12T21:13:19.537968+0000","last_became_active":"2024-04-12T21:13:13.140389+0000","last_became_peered":"2024-04-12T21:13:13.140389+0000","last_unstale":"2024-04-12T21:13:19.537968+0000","last_undegraded":"2024-04-12T21:13:19.537968+0000","last_fullsized":"2024-04-12T21:13:19.537968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:36:13.910978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601169+0000","last_change":"2024-04-12T21:13:17.592755+0000","last_active":"2024-04-12T21:13:18.601169+0000","last_peered":"2024-04-12T21:13:18.601169+0000","last_clean":"2024-04-12T21:13:18.601169+0000","last_became_active":"2024-04-12T21:13:17.592316+0000","last_became_peered":"2024-04-12T21:13:17.592316+0000","last_unstale":"2024-04-12T21:13:18.601169+0000","last_undegraded":"2024-04-12T21:13:18.601169+0000","last_fullsized":"2024-04-12T21:13:18.601169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:16:41.986623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533964+0000","last_change":"2024-04-12T21:13:12.538168+0000","last_active":"2024-04-12T21:13:19.533964+0000","last_peered":"2024-04-12T21:13:19.533964+0000","last_clean":"2024-04-12T21:13:19.533964+0000","last_became_active":"2024-04-12T21:13:12.537784+0000","last_became_peered":"2024-04-12T21:13:12.537784+0000","last_unstale":"2024-04-12T21:13:19.533964+0000","last_undegraded":"2024-04-12T21:13:19.533964+0000","last_fullsized":"2024-04-12T21:13:19.533964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:57:58.333171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536843+0000","last_change":"2024-04-12T21:13:13.153230+0000","last_active":"2024-04-12T21:13:19.536843+0000","last_peered":"2024-04-12T21:13:19.536843+0000","last_clean":"2024-04-12T21:13:19.536843+0000","last_became_active":"2024-04-12T21:13:13.152795+0000","last_became_peered":"2024-04-12T21:13:13.152795+0000","last_unstale":"2024-04-12T21:13:19.536843+0000","last_undegraded":"2024-04-12T21:13:19.536843+0000","last_fullsized":"2024-04-12T21:13:19.536843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:11:21.058591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533701+0000","last_change":"2024-04-12T21:13:17.738428+0000","last_active":"2024-04-12T21:13:19.533701+0000","last_peered":"2024-04-12T21:13:19.533701+0000","last_clean":"2024-04-12T21:13:19.533701+0000","last_became_active":"2024-04-12T21:13:17.737932+0000","last_became_peered":"2024-04-12T21:13:17.737932+0000","last_unstale":"2024-04-12T21:13:19.533701+0000","last_undegraded":"2024-04-12T21:13:19.533701+0000","last_fullsized":"2024-04-12T21:13:19.533701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:22:26.310841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.010984+0000","last_change":"2024-04-12T21:13:12.542689+0000","last_active":"2024-04-12T21:13:19.010984+0000","last_peered":"2024-04-12T21:13:19.010984+0000","last_clean":"2024-04-12T21:13:19.010984+0000","last_became_active":"2024-04-12T21:13:12.542371+0000","last_became_peered":"2024-04-12T21:13:12.542371+0000","last_unstale":"2024-04-12T21:13:19.010984+0000","last_undegraded":"2024-04-12T21:13:19.010984+0000","last_fullsized":"2024-04-12T21:13:19.010984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:08:48.439015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533625+0000","last_change":"2024-04-12T21:13:13.130341+0000","last_active":"2024-04-12T21:13:19.533625+0000","last_peered":"2024-04-12T21:13:19.533625+0000","last_clean":"2024-04-12T21:13:19.533625+0000","last_became_active":"2024-04-12T21:13:13.129886+0000","last_became_peered":"2024-04-12T21:13:13.129886+0000","last_unstale":"2024-04-12T21:13:19.533625+0000","last_undegraded":"2024-04-12T21:13:19.533625+0000","last_fullsized":"2024-04-12T21:13:19.533625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:43:18.384216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.608208+0000","last_change":"2024-04-12T21:13:17.615138+0000","last_active":"2024-04-12T21:13:18.608208+0000","last_peered":"2024-04-12T21:13:18.608208+0000","last_clean":"2024-04-12T21:13:18.608208+0000","last_became_active":"2024-04-12T21:13:17.614710+0000","last_became_peered":"2024-04-12T21:13:17.614710+0000","last_unstale":"2024-04-12T21:13:18.608208+0000","last_undegraded":"2024-04-12T21:13:18.608208+0000","last_fullsized":"2024-04-12T21:13:18.608208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:13:47.956600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536688+0000","last_change":"2024-04-12T21:13:12.546160+0000","last_active":"2024-04-12T21:13:19.536688+0000","last_peered":"2024-04-12T21:13:19.536688+0000","last_clean":"2024-04-12T21:13:19.536688+0000","last_became_active":"2024-04-12T21:13:12.544193+0000","last_became_peered":"2024-04-12T21:13:12.544193+0000","last_unstale":"2024-04-12T21:13:19.536688+0000","last_undegraded":"2024-04-12T21:13:19.536688+0000","last_fullsized":"2024-04-12T21:13:19.536688+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:19:39.640814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685618+0000","last_change":"2024-04-12T21:13:13.154604+0000","last_active":"2024-04-12T21:13:18.685618+0000","last_peered":"2024-04-12T21:13:18.685618+0000","last_clean":"2024-04-12T21:13:18.685618+0000","last_became_active":"2024-04-12T21:13:13.154257+0000","last_became_peered":"2024-04-12T21:13:13.154257+0000","last_unstale":"2024-04-12T21:13:18.685618+0000","last_undegraded":"2024-04-12T21:13:18.685618+0000","last_fullsized":"2024-04-12T21:13:18.685618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:31:45.550602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:21.407872+0000","last_change":"2024-04-12T21:13:17.605698+0000","last_active":"2024-04-12T21:13:21.407872+0000","last_peered":"2024-04-12T21:13:21.407872+0000","last_clean":"2024-04-12T21:13:21.407872+0000","last_became_active":"2024-04-12T21:13:17.605241+0000","last_became_peered":"2024-04-12T21:13:17.605241+0000","last_unstale":"2024-04-12T21:13:21.407872+0000","last_undegraded":"2024-04-12T21:13:21.407872+0000","last_fullsized":"2024-04-12T21:13:21.407872+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:14:32.785778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.600944+0000","last_change":"2024-04-12T21:13:12.536798+0000","last_active":"2024-04-12T21:13:18.600944+0000","last_peered":"2024-04-12T21:13:18.600944+0000","last_clean":"2024-04-12T21:13:18.600944+0000","last_became_active":"2024-04-12T21:13:12.536451+0000","last_became_peered":"2024-04-12T21:13:12.536451+0000","last_unstale":"2024-04-12T21:13:18.600944+0000","last_undegraded":"2024-04-12T21:13:18.600944+0000","last_fullsized":"2024-04-12T21:13:18.600944+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:07:07.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.537718+0000","last_change":"2024-04-12T21:13:13.154644+0000","last_active":"2024-04-12T21:13:19.537718+0000","last_peered":"2024-04-12T21:13:19.537718+0000","last_clean":"2024-04-12T21:13:19.537718+0000","last_became_active":"2024-04-12T21:13:13.154310+0000","last_became_peered":"2024-04-12T21:13:13.154310+0000","last_unstale":"2024-04-12T21:13:19.537718+0000","last_undegraded":"2024-04-12T21:13:19.537718+0000","last_fullsized":"2024-04-12T21:13:19.537718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:51:36.426506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533731+0000","last_change":"2024-04-12T21:13:17.737547+0000","last_active":"2024-04-12T21:13:19.533731+0000","last_peered":"2024-04-12T21:13:19.533731+0000","last_clean":"2024-04-12T21:13:19.533731+0000","last_became_active":"2024-04-12T21:13:17.737190+0000","last_became_peered":"2024-04-12T21:13:17.737190+0000","last_unstale":"2024-04-12T21:13:19.533731+0000","last_undegraded":"2024-04-12T21:13:19.533731+0000","last_fullsized":"2024-04-12T21:13:19.533731+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:23:52.221788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534162+0000","last_change":"2024-04-12T21:13:12.534349+0000","last_active":"2024-04-12T21:13:19.534162+0000","last_peered":"2024-04-12T21:13:19.534162+0000","last_clean":"2024-04-12T21:13:19.534162+0000","last_became_active":"2024-04-12T21:13:12.533933+0000","last_became_peered":"2024-04-12T21:13:12.533933+0000","last_unstale":"2024-04-12T21:13:19.534162+0000","last_undegraded":"2024-04-12T21:13:19.534162+0000","last_fullsized":"2024-04-12T21:13:19.534162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:24:34.110260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535128+0000","last_change":"2024-04-12T21:13:13.128079+0000","last_active":"2024-04-12T21:13:19.535128+0000","last_peered":"2024-04-12T21:13:19.535128+0000","last_clean":"2024-04-12T21:13:19.535128+0000","last_became_active":"2024-04-12T21:13:13.127744+0000","last_became_peered":"2024-04-12T21:13:13.127744+0000","last_unstale":"2024-04-12T21:13:19.535128+0000","last_undegraded":"2024-04-12T21:13:19.535128+0000","last_fullsized":"2024-04-12T21:13:19.535128+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:39:41.851323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011200+0000","last_change":"2024-04-12T21:13:17.596088+0000","last_active":"2024-04-12T21:13:19.011200+0000","last_peered":"2024-04-12T21:13:19.011200+0000","last_clean":"2024-04-12T21:13:19.011200+0000","last_became_active":"2024-04-12T21:13:17.595639+0000","last_became_peered":"2024-04-12T21:13:17.595639+0000","last_unstale":"2024-04-12T21:13:19.011200+0000","last_undegraded":"2024-04-12T21:13:19.011200+0000","last_fullsized":"2024-04-12T21:13:19.011200+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:25:09.763945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601529+0000","last_change":"2024-04-12T21:13:12.543947+0000","last_active":"2024-04-12T21:13:18.601529+0000","last_peered":"2024-04-12T21:13:18.601529+0000","last_clean":"2024-04-12T21:13:18.601529+0000","last_became_active":"2024-04-12T21:13:12.543511+0000","last_became_peered":"2024-04-12T21:13:12.543511+0000","last_unstale":"2024-04-12T21:13:18.601529+0000","last_undegraded":"2024-04-12T21:13:18.601529+0000","last_fullsized":"2024-04-12T21:13:18.601529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:43:23.354835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535450+0000","last_change":"2024-04-12T21:13:13.127178+0000","last_active":"2024-04-12T21:13:19.535450+0000","last_peered":"2024-04-12T21:13:19.535450+0000","last_clean":"2024-04-12T21:13:19.535450+0000","last_became_active":"2024-04-12T21:13:13.126856+0000","last_became_peered":"2024-04-12T21:13:13.126856+0000","last_unstale":"2024-04-12T21:13:19.535450+0000","last_undegraded":"2024-04-12T21:13:19.535450+0000","last_fullsized":"2024-04-12T21:13:19.535450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:09:34.068946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536004+0000","last_change":"2024-04-12T21:13:17.619628+0000","last_active":"2024-04-12T21:13:19.536004+0000","last_peered":"2024-04-12T21:13:19.536004+0000","last_clean":"2024-04-12T21:13:19.536004+0000","last_became_active":"2024-04-12T21:13:17.619015+0000","last_became_peered":"2024-04-12T21:13:17.619015+0000","last_unstale":"2024-04-12T21:13:19.536004+0000","last_undegraded":"2024-04-12T21:13:19.536004+0000","last_fullsized":"2024-04-12T21:13:19.536004+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:39:54.297165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535118+0000","last_change":"2024-04-12T21:13:12.538982+0000","last_active":"2024-04-12T21:13:19.535118+0000","last_peered":"2024-04-12T21:13:19.535118+0000","last_clean":"2024-04-12T21:13:19.535118+0000","last_became_active":"2024-04-12T21:13:12.538604+0000","last_became_peered":"2024-04-12T21:13:12.538604+0000","last_unstale":"2024-04-12T21:13:19.535118+0000","last_undegraded":"2024-04-12T21:13:19.535118+0000","last_fullsized":"2024-04-12T21:13:19.535118+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:40:48.181188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535030+0000","last_change":"2024-04-12T21:13:13.145537+0000","last_active":"2024-04-12T21:13:19.535030+0000","last_peered":"2024-04-12T21:13:19.535030+0000","last_clean":"2024-04-12T21:13:19.535030+0000","last_became_active":"2024-04-12T21:13:13.145283+0000","last_became_peered":"2024-04-12T21:13:13.145283+0000","last_unstale":"2024-04-12T21:13:19.535030+0000","last_undegraded":"2024-04-12T21:13:19.535030+0000","last_fullsized":"2024-04-12T21:13:19.535030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:26:29.339674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.533830+0000","last_change":"2024-04-12T21:13:17.594394+0000","last_active":"2024-04-12T21:13:19.533830+0000","last_peered":"2024-04-12T21:13:19.533830+0000","last_clean":"2024-04-12T21:13:19.533830+0000","last_became_active":"2024-04-12T21:13:17.593960+0000","last_became_peered":"2024-04-12T21:13:17.593960+0000","last_unstale":"2024-04-12T21:13:19.533830+0000","last_undegraded":"2024-04-12T21:13:19.533830+0000","last_fullsized":"2024-04-12T21:13:19.533830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:00:53.462145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606325+0000","last_change":"2024-04-12T21:13:12.546385+0000","last_active":"2024-04-12T21:13:18.606325+0000","last_peered":"2024-04-12T21:13:18.606325+0000","last_clean":"2024-04-12T21:13:18.606325+0000","last_became_active":"2024-04-12T21:13:12.545902+0000","last_became_peered":"2024-04-12T21:13:12.545902+0000","last_unstale":"2024-04-12T21:13:18.606325+0000","last_undegraded":"2024-04-12T21:13:18.606325+0000","last_fullsized":"2024-04-12T21:13:18.606325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:57:16.419641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.535297+0000","last_change":"2024-04-12T21:13:13.130478+0000","last_active":"2024-04-12T21:13:19.535297+0000","last_peered":"2024-04-12T21:13:19.535297+0000","last_clean":"2024-04-12T21:13:19.535297+0000","last_became_active":"2024-04-12T21:13:13.130164+0000","last_became_peered":"2024-04-12T21:13:13.130164+0000","last_unstale":"2024-04-12T21:13:19.535297+0000","last_undegraded":"2024-04-12T21:13:19.535297+0000","last_fullsized":"2024-04-12T21:13:19.535297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:22:29.186629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685678+0000","last_change":"2024-04-12T21:13:13.135406+0000","last_active":"2024-04-12T21:13:18.685678+0000","last_peered":"2024-04-12T21:13:18.685678+0000","last_clean":"2024-04-12T21:13:18.685678+0000","last_became_active":"2024-04-12T21:13:13.134840+0000","last_became_peered":"2024-04-12T21:13:13.134840+0000","last_unstale":"2024-04-12T21:13:18.685678+0000","last_undegraded":"2024-04-12T21:13:18.685678+0000","last_fullsized":"2024-04-12T21:13:18.685678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:22:22.315016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601615+0000","last_change":"2024-04-12T21:13:13.127249+0000","last_active":"2024-04-12T21:13:18.601615+0000","last_peered":"2024-04-12T21:13:18.601615+0000","last_clean":"2024-04-12T21:13:18.601615+0000","last_became_active":"2024-04-12T21:13:13.126901+0000","last_became_peered":"2024-04-12T21:13:13.126901+0000","last_unstale":"2024-04-12T21:13:18.601615+0000","last_undegraded":"2024-04-12T21:13:18.601615+0000","last_fullsized":"2024-04-12T21:13:18.601615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:32:18.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.600970+0000","last_change":"2024-04-12T21:13:13.151728+0000","last_active":"2024-04-12T21:13:18.600970+0000","last_peered":"2024-04-12T21:13:18.600970+0000","last_clean":"2024-04-12T21:13:18.600970+0000","last_became_active":"2024-04-12T21:13:13.151457+0000","last_became_peered":"2024-04-12T21:13:13.151457+0000","last_unstale":"2024-04-12T21:13:18.600970+0000","last_undegraded":"2024-04-12T21:13:18.600970+0000","last_fullsized":"2024-04-12T21:13:18.600970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:48:32.641671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536071+0000","last_change":"2024-04-12T21:13:13.150109+0000","last_active":"2024-04-12T21:13:19.536071+0000","last_peered":"2024-04-12T21:13:19.536071+0000","last_clean":"2024-04-12T21:13:19.536071+0000","last_became_active":"2024-04-12T21:13:13.149132+0000","last_became_peered":"2024-04-12T21:13:13.149132+0000","last_unstale":"2024-04-12T21:13:19.536071+0000","last_undegraded":"2024-04-12T21:13:19.536071+0000","last_fullsized":"2024-04-12T21:13:19.536071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:36:00.062993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.606883+0000","last_change":"2024-04-12T21:13:13.143405+0000","last_active":"2024-04-12T21:13:18.606883+0000","last_peered":"2024-04-12T21:13:18.606883+0000","last_clean":"2024-04-12T21:13:18.606883+0000","last_became_active":"2024-04-12T21:13:13.142973+0000","last_became_peered":"2024-04-12T21:13:13.142973+0000","last_unstale":"2024-04-12T21:13:18.606883+0000","last_undegraded":"2024-04-12T21:13:18.606883+0000","last_fullsized":"2024-04-12T21:13:18.606883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:10:03.106739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.686058+0000","last_change":"2024-04-12T21:13:13.129864+0000","last_active":"2024-04-12T21:13:18.686058+0000","last_peered":"2024-04-12T21:13:18.686058+0000","last_clean":"2024-04-12T21:13:18.686058+0000","last_became_active":"2024-04-12T21:13:13.129518+0000","last_became_peered":"2024-04-12T21:13:13.129518+0000","last_unstale":"2024-04-12T21:13:18.686058+0000","last_undegraded":"2024-04-12T21:13:18.686058+0000","last_fullsized":"2024-04-12T21:13:18.686058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:44:38.370002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.601520+0000","last_change":"2024-04-12T21:13:13.129153+0000","last_active":"2024-04-12T21:13:18.601520+0000","last_peered":"2024-04-12T21:13:18.601520+0000","last_clean":"2024-04-12T21:13:18.601520+0000","last_became_active":"2024-04-12T21:13:13.128799+0000","last_became_peered":"2024-04-12T21:13:13.128799+0000","last_unstale":"2024-04-12T21:13:18.601520+0000","last_undegraded":"2024-04-12T21:13:18.601520+0000","last_fullsized":"2024-04-12T21:13:18.601520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:37:51.675110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.538172+0000","last_change":"2024-04-12T21:13:13.144638+0000","last_active":"2024-04-12T21:13:19.538172+0000","last_peered":"2024-04-12T21:13:19.538172+0000","last_clean":"2024-04-12T21:13:19.538172+0000","last_became_active":"2024-04-12T21:13:13.144275+0000","last_became_peered":"2024-04-12T21:13:13.144275+0000","last_unstale":"2024-04-12T21:13:19.538172+0000","last_undegraded":"2024-04-12T21:13:19.538172+0000","last_fullsized":"2024-04-12T21:13:19.538172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:38:09.225551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.534298+0000","last_change":"2024-04-12T21:13:13.145845+0000","last_active":"2024-04-12T21:13:19.534298+0000","last_peered":"2024-04-12T21:13:19.534298+0000","last_clean":"2024-04-12T21:13:19.534298+0000","last_became_active":"2024-04-12T21:13:13.145399+0000","last_became_peered":"2024-04-12T21:13:13.145399+0000","last_unstale":"2024-04-12T21:13:19.534298+0000","last_undegraded":"2024-04-12T21:13:19.534298+0000","last_fullsized":"2024-04-12T21:13:19.534298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:07:13.075365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.536956+0000","last_change":"2024-04-12T21:13:13.138373+0000","last_active":"2024-04-12T21:13:19.536956+0000","last_peered":"2024-04-12T21:13:19.536956+0000","last_clean":"2024-04-12T21:13:19.536956+0000","last_became_active":"2024-04-12T21:13:13.137366+0000","last_became_peered":"2024-04-12T21:13:13.137366+0000","last_unstale":"2024-04-12T21:13:19.536956+0000","last_undegraded":"2024-04-12T21:13:19.536956+0000","last_fullsized":"2024-04-12T21:13:19.536956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:05:22.070677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:19.011282+0000","last_change":"2024-04-12T21:13:13.147902+0000","last_active":"2024-04-12T21:13:19.011282+0000","last_peered":"2024-04-12T21:13:19.011282+0000","last_clean":"2024-04-12T21:13:19.011282+0000","last_became_active":"2024-04-12T21:13:13.147513+0000","last_became_peered":"2024-04-12T21:13:13.147513+0000","last_unstale":"2024-04-12T21:13:19.011282+0000","last_undegraded":"2024-04-12T21:13:19.011282+0000","last_fullsized":"2024-04-12T21:13:19.011282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:43:34.228432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.605734+0000","last_change":"2024-04-12T21:13:13.154703+0000","last_active":"2024-04-12T21:13:18.605734+0000","last_peered":"2024-04-12T21:13:18.605734+0000","last_clean":"2024-04-12T21:13:18.605734+0000","last_became_active":"2024-04-12T21:13:13.154340+0000","last_became_peered":"2024-04-12T21:13:13.154340+0000","last_unstale":"2024-04-12T21:13:18.605734+0000","last_undegraded":"2024-04-12T21:13:18.605734+0000","last_fullsized":"2024-04-12T21:13:18.605734+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:29:05.500481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-04-12T21:13:18.685922+0000","last_change":"2024-04-12T21:13:13.133441+0000","last_active":"2024-04-12T21:13:18.685922+0000","last_peered":"2024-04-12T21:13:18.685922+0000","last_clean":"2024-04-12T21:13:18.685922+0000","last_became_active":"2024-04-12T21:13:13.133113+0000","last_became_peered":"2024-04-12T21:13:13.133113+0000","last_unstale":"2024-04-12T21:13:18.685922+0000","last_undegraded":"2024-04-12T21:13:18.685922+0000","last_fullsized":"2024-04-12T21:13:18.685922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:05:36.924405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":41269,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":18,"num_read_kb":35,"num_write":60,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":69848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383700,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1090918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":991232,"data_stored":477819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1585152,"data_stored":1072261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":487657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":477319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":481333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":1593344,"data_stored":1071861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":40665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T21:13:42.926 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-12T21:13:42.927 INFO:tasks.cephadm:Setting up client nodes... 2024-04-12T21:13:42.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-12T21:13:43.412 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:43.784 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:43 smithi081 ceph-mon[31453]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:13:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:43 smithi153 ceph-mon[36569]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:13:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:43 smithi104 ceph-mon[36624]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:13:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:44 smithi153 ceph-mon[36569]: pgmap v330: 129 pgs: 129 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-12T21:13:44.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:44 smithi104 ceph-mon[36624]: pgmap v330: 129 pgs: 129 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-12T21:13:44.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:44 smithi081 ceph-mon[31453]: pgmap v330: 129 pgs: 129 active+clean; 617 KiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.8 KiB/s wr, 3 op/s 2024-04-12T21:13:45.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 -- 172.21.15.81:0/2505089564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 msgr2=0x7f9898101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:45.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 --2- 172.21.15.81:0/2505089564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 0x7f9898101850 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f988c0098f0 tx=0x7f988c009c00 comp rx=0 tx=0).stop 2024-04-12T21:13:45.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 -- 172.21.15.81:0/2505089564 shutdown_connections 2024-04-12T21:13:45.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 --2- 172.21.15.81:0/2505089564 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9898102750 0x7f9898102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:45.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 --2- 172.21.15.81:0/2505089564 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 0x7f9898101850 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:45.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 --2- 172.21.15.81:0/2505089564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9898101d90 0x7f9898102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:45.732 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 -- 172.21.15.81:0/2505089564 >> 172.21.15.81:0/2505089564 conn(0x7f98980fcfd0 msgr2=0x7f98980ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 -- 172.21.15.81:0/2505089564 shutdown_connections 2024-04-12T21:13:45.733 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.736+0000 7f98a0800700 1 -- 172.21.15.81:0/2505089564 wait complete. 2024-04-12T21:13:45.734 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.737+0000 7f98a0800700 1 Processor -- start 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.737+0000 7f98a0800700 1 -- start start 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.737+0000 7f98a0800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 0x7f989819aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.737+0000 7f98a0800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9898101d90 0x7f989819b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.737+0000 7f98a0800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9898102750 0x7f98981a2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.737+0000 7f98a0800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9898110940 con 0x7f9898101430 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.737+0000 7f98a0800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f98981107c0 con 0x7f9898101d90 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.737+0000 7f98a0800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9898110ac0 con 0x7f9898102750 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 0x7f989819aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989de00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 0x7f989819aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59720/0 (socket says 172.21.15.81:59720) 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989de00700 1 -- 172.21.15.81:0/4008964015 learned_addr learned my addr 172.21.15.81:0/4008964015 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989e800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9898102750 0x7f98981a2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:45.735 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989d400700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9898101d90 0x7f989819b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989e800700 1 -- 172.21.15.81:0/4008964015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9898101d90 msgr2=0x7f989819b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989e800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9898101d90 0x7f989819b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989e800700 1 -- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 msgr2=0x7f989819aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989e800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 0x7f989819aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989e800700 1 -- 172.21.15.81:0/4008964015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98981a2b30 con 0x7f9898102750 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.738+0000 7f989de00700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 0x7f989819aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.739+0000 7f989e800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9898102750 0x7f98981a2490 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f989400c0c0 tx=0x7f989400c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.739+0000 7f9892a00700 1 -- 172.21.15.81:0/4008964015 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9894003a00 con 0x7f9898102750 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.739+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98981a2e20 con 0x7f9898102750 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.739+0000 7f9892a00700 1 -- 172.21.15.81:0/4008964015 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9894003ba0 con 0x7f9898102750 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.739+0000 7f9892a00700 1 -- 172.21.15.81:0/4008964015 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9894004880 con 0x7f9898102750 2024-04-12T21:13:45.736 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.739+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98981a3280 con 0x7f9898102750 2024-04-12T21:13:45.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.740+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9898050ff0 con 0x7f9898102750 2024-04-12T21:13:45.738 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.742+0000 7f9892a00700 1 -- 172.21.15.81:0/4008964015 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9894012050 con 0x7f9898102750 2024-04-12T21:13:45.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.744+0000 7f9892a00700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98800a3d80 0x7f98800a6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:45.739 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.744+0000 7f9892a00700 1 -- 172.21.15.81:0/4008964015 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98940cfff0 con 0x7f9898102750 2024-04-12T21:13:45.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.744+0000 7f989de00700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98800a3d80 0x7f98800a6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:45.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.745+0000 7f989de00700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98800a3d80 0x7f98800a6240 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f988c00fb70 tx=0x7f988c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:45.742 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:45.746+0000 7f9892a00700 1 -- 172.21.15.81:0/4008964015 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9894022060 con 0x7f9898102750 2024-04-12T21:13:46.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.060+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f989802dfa0 con 0x7f9898102750 2024-04-12T21:13:46.063 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.067+0000 7f9892a00700 1 -- 172.21.15.81:0/4008964015 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f989802dfa0 con 0x7f9898102750 2024-04-12T21:13:46.063 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-04-12T21:13:46.063 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQAKpBlmy3y7AxAABOtuFM5u3tk5X3lf80NVdg== 2024-04-12T21:13:46.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98800a3d80 msgr2=0x7f98800a6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:46.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98800a3d80 0x7f98800a6240 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f988c00fb70 tx=0x7f988c019040 comp rx=0 tx=0).stop 2024-04-12T21:13:46.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9898102750 msgr2=0x7f98981a2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:46.066 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9898102750 0x7f98981a2490 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f989400c0c0 tx=0x7f989400c3d0 comp rx=0 tx=0).stop 2024-04-12T21:13:46.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 shutdown_connections 2024-04-12T21:13:46.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98800a3d80 0x7f98800a6240 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:46.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9898102750 0x7f98981a2490 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:46.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9898101430 0x7f989819aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:46.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 --2- 172.21.15.81:0/4008964015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9898101d90 0x7f989819b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:46.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.070+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 >> 172.21.15.81:0/4008964015 conn(0x7f98980fcfd0 msgr2=0x7f98980fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:46.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.071+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 shutdown_connections 2024-04-12T21:13:46.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:46.071+0000 7f98a0800700 1 -- 172.21.15.81:0/4008964015 wait complete. 2024-04-12T21:13:46.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:46 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4008964015' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:13:46.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:46 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:13:46.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:46 smithi081 ceph-mon[31453]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:13:46.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:46 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4008964015' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:13:46.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:46 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:13:46.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:46 smithi153 ceph-mon[36569]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:13:46.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:46 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4008964015' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:13:46.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:46 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-12T21:13:46.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:46 smithi104 ceph-mon[36624]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-12T21:13:46.904 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:13:46.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-12T21:13:46.904 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-12T21:13:47.057 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-12T21:13:47.057 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-12T21:13:47.057 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph mgr dump --format=json 2024-04-12T21:13:47.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:47 smithi081 ceph-mon[31453]: pgmap v331: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.3 KiB/s wr, 4 op/s 2024-04-12T21:13:47.417 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:47 smithi153 ceph-mon[36569]: pgmap v331: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.3 KiB/s wr, 4 op/s 2024-04-12T21:13:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:47.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:47 smithi104 ceph-mon[36624]: pgmap v331: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.3 KiB/s wr, 4 op/s 2024-04-12T21:13:49.537 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:49 smithi081 ceph-mon[31453]: pgmap v332: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-12T21:13:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:49 smithi153 ceph-mon[36569]: pgmap v332: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.818+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1279135433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c98075600 msgr2=0x7f3c9810efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.818+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1279135433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c98075600 0x7f3c9810efe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3c800098f0 tx=0x7f3c80009c00 comp rx=0 tx=0).stop 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.819+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1279135433 shutdown_connections 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.819+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1279135433 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c9810f520 0x7f3c98111910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.819+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1279135433 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c98074ce0 0x7f3c980750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.819+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1279135433 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c98075600 0x7f3c9810efe0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.819+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1279135433 >> 172.21.15.81:0/1279135433 conn(0x7f3c98079c70 msgr2=0x7f3c9807a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.819+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1279135433 shutdown_connections 2024-04-12T21:13:49.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.819+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1279135433 wait complete. 2024-04-12T21:13:49.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.820+0000 7f3c9dc00700 1 Processor -- start 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.820+0000 7f3c9dc00700 1 -- start start 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.820+0000 7f3c9dc00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c98074ce0 0x7f3c9819b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.820+0000 7f3c9dc00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c98075600 0x7f3c9819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.820+0000 7f3c9dc00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9810f520 0x7f3c9819fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.820+0000 7f3c9dc00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3c98114fe0 con 0x7f3c9810f520 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.820+0000 7f3c9dc00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3c98114e60 con 0x7f3c98075600 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.820+0000 7f3c9dc00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c98115160 con 0x7f3c98074ce0 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c97400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9810f520 0x7f3c9819fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c97400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9810f520 0x7f3c9819fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59740/0 (socket says 172.21.15.81:59740) 2024-04-12T21:13:49.817 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c97400700 1 -- 172.21.15.81:0/1538097314 learned_addr learned my addr 172.21.15.81:0/1538097314 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c8fe00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c98075600 0x7f3c9819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c96a00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c98074ce0 0x7f3c9819b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c97400700 1 -- 172.21.15.81:0/1538097314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c98074ce0 msgr2=0x7f3c9819b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c97400700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c98074ce0 0x7f3c9819b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c97400700 1 -- 172.21.15.81:0/1538097314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c98075600 msgr2=0x7f3c9819ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c97400700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c98075600 0x7f3c9819ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c97400700 1 -- 172.21.15.81:0/1538097314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c981a0540 con 0x7f3c9810f520 2024-04-12T21:13:49.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.821+0000 7f3c8fe00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c98075600 0x7f3c9819ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.822+0000 7f3c97400700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9810f520 0x7f3c9819fe00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000e3c0 tx=0x7f3c9000e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.822+0000 7f3c8f400700 1 -- 172.21.15.81:0/1538097314 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c900090d0 con 0x7f3c9810f520 2024-04-12T21:13:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.822+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c981a0830 con 0x7f3c9810f520 2024-04-12T21:13:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.822+0000 7f3c8f400700 1 -- 172.21.15.81:0/1538097314 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c90009270 con 0x7f3c9810f520 2024-04-12T21:13:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.822+0000 7f3c8f400700 1 -- 172.21.15.81:0/1538097314 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90004030 con 0x7f3c9810f520 2024-04-12T21:13:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.822+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c981a80b0 con 0x7f3c9810f520 2024-04-12T21:13:49.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.823+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c980697a0 con 0x7f3c9810f520 2024-04-12T21:13:49.821 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.825+0000 7f3c8f400700 1 -- 172.21.15.81:0/1538097314 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3c90012050 con 0x7f3c9810f520 2024-04-12T21:13:49.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.827+0000 7f3c8f400700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c7c0a3d50 0x7f3c7c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:49.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.827+0000 7f3c8f400700 1 -- 172.21.15.81:0/1538097314 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3c900d0410 con 0x7f3c9810f520 2024-04-12T21:13:49.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.827+0000 7f3c96a00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c7c0a3d50 0x7f3c7c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:49.823 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.828+0000 7f3c96a00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c7c0a3d50 0x7f3c7c0a6210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009c60 tx=0x7f3c88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:49.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:49.830+0000 7f3c8f400700 1 -- 172.21.15.81:0/1538097314 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c90022060 con 0x7f3c9810f520 2024-04-12T21:13:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:49 smithi104 ceph-mon[36624]: pgmap v332: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-12T21:13:50.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.118+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3c9802dfa0 con 0x7f3c9810f520 2024-04-12T21:13:50.120 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.124+0000 7f3c8f400700 1 -- 172.21.15.81:0/1538097314 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264997 (secure 0 0 0) 0x7f3c90010040 con 0x7f3c9810f520 2024-04-12T21:13:50.126 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.130+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c7c0a3d50 msgr2=0x7f3c7c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.130+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c7c0a3d50 0x7f3c7c0a6210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3c88009c60 tx=0x7f3c88009260 comp rx=0 tx=0).stop 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.130+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9810f520 msgr2=0x7f3c9819fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.130+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9810f520 0x7f3c9819fe00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000e3c0 tx=0x7f3c9000e6d0 comp rx=0 tx=0).stop 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.131+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 shutdown_connections 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.131+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c7c0a3d50 0x7f3c7c0a6210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.131+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c9810f520 0x7f3c9819fe00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.131+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c98074ce0 0x7f3c9819b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.131+0000 7f3c9dc00700 1 --2- 172.21.15.81:0/1538097314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c98075600 0x7f3c9819ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.131+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 >> 172.21.15.81:0/1538097314 conn(0x7f3c98079c70 msgr2=0x7f3c980ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.131+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 shutdown_connections 2024-04-12T21:13:50.128 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:50.131+0000 7f3c9dc00700 1 -- 172.21.15.81:0/1538097314 wait complete. 2024-04-12T21:13:50.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:50 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1538097314' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T21:13:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:50 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1538097314' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T21:13:50.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:50 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1538097314' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-12T21:13:50.998 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":3972789130},{"type":"v1","addr":"172.21.15.81:6801","nonce":3972789130}]},"active_addr":"172.21.15.81:6801/3972789130","active_change":"2024-04-12T21:05:22.180064+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.81:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":1492603400}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":1505395468}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":2871951000}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":940793971}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.81:0","nonce":3244566895}]}]} 2024-04-12T21:13:51.003 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-12T21:13:51.003 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-12T21:13:51.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-12T21:13:51.454 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:51.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:51 smithi081 ceph-mon[31453]: pgmap v333: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-12T21:13:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:51 smithi153 ceph-mon[36569]: pgmap v333: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-12T21:13:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:51 smithi104 ceph-mon[36624]: pgmap v333: 129 pgs: 129 active+clean; 622 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.2 KiB/s wr, 4 op/s 2024-04-12T21:13:53.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:53 smithi081 ceph-mon[31453]: pgmap v334: 129 pgs: 129 active+clean; 634 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 -- 172.21.15.81:0/757013169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608107a00 msgr2=0x7f8608107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 --2- 172.21.15.81:0/757013169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608107a00 0x7f8608107de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009990 tx=0x7f85f8009ca0 comp rx=0 tx=0).stop 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 -- 172.21.15.81:0/757013169 shutdown_connections 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 --2- 172.21.15.81:0/757013169 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8608102370 0x7f860810e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 --2- 172.21.15.81:0/757013169 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8608107a00 0x7f8608107de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 --2- 172.21.15.81:0/757013169 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f86081019b0 0x7f8608101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 -- 172.21.15.81:0/757013169 >> 172.21.15.81:0/757013169 conn(0x7f8608078960 msgr2=0x7f8608078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 -- 172.21.15.81:0/757013169 shutdown_connections 2024-04-12T21:13:53.747 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.751+0000 7f860f200700 1 -- 172.21.15.81:0/757013169 wait complete. 2024-04-12T21:13:53.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.752+0000 7f860f200700 1 Processor -- start 2024-04-12T21:13:53.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.752+0000 7f860f200700 1 -- start start 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f860f200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86081019b0 0x7f860819b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f860f200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8608102370 0x7f860819b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f860f200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8608107a00 0x7f860819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f860f200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8608112db0 con 0x7f86081019b0 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f860f200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8608112c30 con 0x7f8608107a00 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f860f200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8608112f30 con 0x7f8608102370 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f8607400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8608102370 0x7f860819b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f8607e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86081019b0 0x7f860819b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f860d200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8608107a00 0x7f860819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f8607400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8608102370 0x7f860819b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:44624/0 (socket says 172.21.15.81:44624) 2024-04-12T21:13:53.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.753+0000 7f8607400700 1 -- 172.21.15.81:0/3880046613 learned_addr learned my addr 172.21.15.81:0/3880046613 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:53.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8607400700 1 -- 172.21.15.81:0/3880046613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8608107a00 msgr2=0x7f860819fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:53.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8607400700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8608107a00 0x7f860819fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:53.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8607400700 1 -- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86081019b0 msgr2=0x7f860819b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:53.750 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8607400700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86081019b0 0x7f860819b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:53.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8607400700 1 -- 172.21.15.81:0/3880046613 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86081a0460 con 0x7f8608102370 2024-04-12T21:13:53.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f860d200700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8608107a00 0x7f860819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:53.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8607400700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8608102370 0x7f860819b990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f85f400e9a0 tx=0x7f85f400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:53.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8604c00700 1 -- 172.21.15.81:0/3880046613 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f400ca10 con 0x7f8608102370 2024-04-12T21:13:53.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86081a0750 con 0x7f8608102370 2024-04-12T21:13:53.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8604c00700 1 -- 172.21.15.81:0/3880046613 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85f400cbb0 con 0x7f8608102370 2024-04-12T21:13:53.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f8604c00700 1 -- 172.21.15.81:0/3880046613 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4018680 con 0x7f8608102370 2024-04-12T21:13:53.751 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.754+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86081a7fd0 con 0x7f8608102370 2024-04-12T21:13:53.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.756+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86080697a0 con 0x7f8608102370 2024-04-12T21:13:53.753 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.757+0000 7f8604c00700 1 -- 172.21.15.81:0/3880046613 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f85f4010430 con 0x7f8608102370 2024-04-12T21:13:53.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.759+0000 7f8604c00700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f85ec0a3cd0 0x7f85ec0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:53.754 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.759+0000 7f8604c00700 1 -- 172.21.15.81:0/3880046613 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f85f40d0a30 con 0x7f8608102370 2024-04-12T21:13:53.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.759+0000 7f8607e00700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f85ec0a3cd0 0x7f85ec0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:53.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.760+0000 7f8607e00700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f85ec0a3cd0 0x7f85ec0a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f85f800b2e0 tx=0x7f85f8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:53.758 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:53.763+0000 7f8604c00700 1 -- 172.21.15.81:0/3880046613 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f85f409d010 con 0x7f8608102370 2024-04-12T21:13:53.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:53 smithi153 ceph-mon[36569]: pgmap v334: 129 pgs: 129 active+clean; 634 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-04-12T21:13:53.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:53 smithi104 ceph-mon[36624]: pgmap v334: 129 pgs: 129 active+clean; 634 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2024-04-12T21:13:54.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.015+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86081a81a0 con 0x7f8608102370 2024-04-12T21:13:54.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.017+0000 7f8604c00700 1 -- 172.21.15.81:0/3880046613 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7f85f409c910 con 0x7f8608102370 2024-04-12T21:13:54.014 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:54.014 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:54.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.021+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f85ec0a3cd0 msgr2=0x7f85ec0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:54.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.021+0000 7f860f200700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f85ec0a3cd0 0x7f85ec0a6190 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f85f800b2e0 tx=0x7f85f8005960 comp rx=0 tx=0).stop 2024-04-12T21:13:54.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.021+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8608102370 msgr2=0x7f860819b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:54.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.021+0000 7f860f200700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8608102370 0x7f860819b990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f85f400e9a0 tx=0x7f85f400ecb0 comp rx=0 tx=0).stop 2024-04-12T21:13:54.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.022+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 shutdown_connections 2024-04-12T21:13:54.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.022+0000 7f860f200700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f85ec0a3cd0 0x7f85ec0a6190 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:54.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.022+0000 7f860f200700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8608107a00 0x7f860819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:54.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.022+0000 7f860f200700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f86081019b0 0x7f860819b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:54.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.022+0000 7f860f200700 1 --2- 172.21.15.81:0/3880046613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8608102370 0x7f860819b990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:54.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.022+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 >> 172.21.15.81:0/3880046613 conn(0x7f8608078960 msgr2=0x7f860810ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:54.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.022+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 shutdown_connections 2024-04-12T21:13:54.018 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:54.022+0000 7f860f200700 1 -- 172.21.15.81:0/3880046613 wait complete. 2024-04-12T21:13:54.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:54 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3880046613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:54.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:54 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3880046613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:54.831 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:54 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3880046613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:54.897 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-12T21:13:54.897 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd dump --format=json 2024-04-12T21:13:55.329 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:55.696 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:55 smithi081 ceph-mon[31453]: pgmap v335: 129 pgs: 129 active+clean; 634 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-04-12T21:13:55.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:55 smithi153 ceph-mon[36569]: pgmap v335: 129 pgs: 129 active+clean; 634 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-04-12T21:13:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:55 smithi104 ceph-mon[36624]: pgmap v335: 129 pgs: 129 active+clean; 634 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 3 op/s 2024-04-12T21:13:57.579 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 -- 172.21.15.81:0/1506373229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f761810e420 msgr2=0x7f7618115ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 --2- 172.21.15.81:0/1506373229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f761810e420 0x7f7618115ff0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7610009fc0 tx=0x7f761000b330 comp rx=0 tx=0).stop 2024-04-12T21:13:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 -- 172.21.15.81:0/1506373229 shutdown_connections 2024-04-12T21:13:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 --2- 172.21.15.81:0/1506373229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f761810e420 0x7f7618115ff0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 --2- 172.21.15.81:0/1506373229 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7618109340 0x7f7618109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 --2- 172.21.15.81:0/1506373229 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7618109cf0 0x7f761810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 -- 172.21.15.81:0/1506373229 >> 172.21.15.81:0/1506373229 conn(0x7f7618076280 msgr2=0x7f76181078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 -- 172.21.15.81:0/1506373229 shutdown_connections 2024-04-12T21:13:57.580 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.583+0000 7f761da00700 1 -- 172.21.15.81:0/1506373229 wait complete. 2024-04-12T21:13:57.581 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.585+0000 7f761da00700 1 Processor -- start 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.585+0000 7f761da00700 1 -- start start 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.585+0000 7f761da00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7618109340 0x7f76181a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.585+0000 7f761da00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7618109cf0 0x7f76181a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.585+0000 7f761da00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f761810e420 0x7f761819f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.585+0000 7f761da00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7618119460 con 0x7f761810e420 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.585+0000 7f761da00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f76181192e0 con 0x7f7618109cf0 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.585+0000 7f761da00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f76181195e0 con 0x7f7618109340 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7617400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f761810e420 0x7f761819f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7617400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f761810e420 0x7f761819f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58238/0 (socket says 172.21.15.81:58238) 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7616000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7618109cf0 0x7f76181a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7617400700 1 -- 172.21.15.81:0/1048969205 learned_addr learned my addr 172.21.15.81:0/1048969205 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:13:57.582 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7616a00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7618109340 0x7f76181a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:57.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7616000700 1 -- 172.21.15.81:0/1048969205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7618109340 msgr2=0x7f76181a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:57.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7616000700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7618109340 0x7f76181a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.583 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7616000700 1 -- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f761810e420 msgr2=0x7f761819f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7616000700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f761810e420 0x7f761819f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7616000700 1 -- 172.21.15.81:0/1048969205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761819ff20 con 0x7f7618109cf0 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.586+0000 7f7617400700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f761810e420 0x7f761819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.587+0000 7f7616000700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7618109cf0 0x7f76181a5ab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7604005fd0 tx=0x7f760400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.587+0000 7f760f400700 1 -- 172.21.15.81:0/1048969205 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7604010900 con 0x7f7618109cf0 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.587+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76181a0210 con 0x7f7618109cf0 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.587+0000 7f760f400700 1 -- 172.21.15.81:0/1048969205 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7604018420 con 0x7f7618109cf0 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.587+0000 7f760f400700 1 -- 172.21.15.81:0/1048969205 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760400f680 con 0x7f7618109cf0 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.587+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76181ac4f0 con 0x7f7618109cf0 2024-04-12T21:13:57.584 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.588+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f761810a820 con 0x7f7618109cf0 2024-04-12T21:13:57.585 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.590+0000 7f760f400700 1 -- 172.21.15.81:0/1048969205 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f76040185d0 con 0x7f7618109cf0 2024-04-12T21:13:57.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.591+0000 7f760f400700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75fc0a3cd0 0x7f75fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:13:57.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.591+0000 7f760f400700 1 -- 172.21.15.81:0/1048969205 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f76040d0850 con 0x7f7618109cf0 2024-04-12T21:13:57.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.592+0000 7f7616a00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75fc0a3cd0 0x7f75fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:13:57.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.592+0000 7f7616a00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75fc0a3cd0 0x7f75fc0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7608009c60 tx=0x7f7608009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:13:57.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.596+0000 7f760f400700 1 -- 172.21.15.81:0/1048969205 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f760409ce30 con 0x7f7618109cf0 2024-04-12T21:13:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:57 smithi153 ceph-mon[36569]: pgmap v336: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-04-12T21:13:57.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.829+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76181ac6c0 con 0x7f7618109cf0 2024-04-12T21:13:57.826 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:57 smithi081 ceph-mon[31453]: pgmap v336: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-04-12T21:13:57.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.831+0000 7f760f400700 1 -- 172.21.15.81:0/1048969205 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24152 (secure 0 0 0) 0x7f760409c730 con 0x7f7618109cf0 2024-04-12T21:13:57.828 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:13:57.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:57 smithi104 ceph-mon[36624]: pgmap v336: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-04-12T21:13:57.829 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":89,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:13:18.062246+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:13:57.830 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75fc0a3cd0 msgr2=0x7f75fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:57.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75fc0a3cd0 0x7f75fc0a6190 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7608009c60 tx=0x7f7608009260 comp rx=0 tx=0).stop 2024-04-12T21:13:57.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7618109cf0 msgr2=0x7f76181a5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:13:57.831 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7618109cf0 0x7f76181a5ab0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7604005fd0 tx=0x7f760400eca0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 shutdown_connections 2024-04-12T21:13:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f75fc0a3cd0 0x7f75fc0a6190 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f761810e420 0x7f761819f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7618109340 0x7f76181a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 --2- 172.21.15.81:0/1048969205 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7618109cf0 0x7f76181a5ab0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:13:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 >> 172.21.15.81:0/1048969205 conn(0x7f7618076280 msgr2=0x7f76180767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:13:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 shutdown_connections 2024-04-12T21:13:57.832 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:13:57.835+0000 7f761da00700 1 -- 172.21.15.81:0/1048969205 wait complete. 2024-04-12T21:13:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:58 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1048969205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:58.775 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-12T21:13:58.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-12T21:13:58.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-12T21:13:58.776 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-12T21:13:58.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-12T21:13:58.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-12T21:13:58.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-12T21:13:58.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-12T21:13:58.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-12T21:13:58.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-12T21:13:58.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-12T21:13:58.779 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-12T21:13:58.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:58 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1048969205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:58.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:58 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1048969205' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:13:59.705 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:13:59 smithi153 ceph-mon[36569]: pgmap v337: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-12T21:13:59.802 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:13:59 smithi104 ceph-mon[36624]: pgmap v337: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-12T21:13:59.854 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:59.856 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:13:59.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:13:59 smithi081 ceph-mon[31453]: pgmap v337: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-12T21:14:01.607 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:01.607 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:01.800 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:01 smithi081 ceph-mon[31453]: pgmap v338: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-12T21:14:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:01 smithi153 ceph-mon[36569]: pgmap v338: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-12T21:14:01.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:01 smithi104 ceph-mon[36624]: pgmap v338: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-12T21:14:02.171 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:02.179 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:02.182 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:02.183 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:02.203 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:02.755 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:03.599 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:03 smithi081 ceph-mon[31453]: pgmap v339: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-12T21:14:03.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:03 smithi153 ceph-mon[36569]: pgmap v339: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-12T21:14:03.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:03 smithi104 ceph-mon[36624]: pgmap v339: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-12T21:14:05.556 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:05 smithi081 ceph-mon[31453]: pgmap v340: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 1 op/s 2024-04-12T21:14:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:05 smithi153 ceph-mon[36569]: pgmap v340: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 1 op/s 2024-04-12T21:14:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:05 smithi104 ceph-mon[36624]: pgmap v340: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 1 op/s 2024-04-12T21:14:06.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 -- 172.21.15.81:0/1483056609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c104f20 msgr2=0x7f141c105300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 --2- 172.21.15.81:0/1483056609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c104f20 0x7f141c105300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f14100099c0 tx=0x7f1410009cd0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 -- 172.21.15.81:0/1483056609 shutdown_connections 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 --2- 172.21.15.81:0/1483056609 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c10a000 0x7f141c111bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 --2- 172.21.15.81:0/1483056609 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c104f20 0x7f141c105300 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 --2- 172.21.15.81:0/1483056609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f141c1058d0 0x7f141c109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 -- 172.21.15.81:0/1483056609 >> 172.21.15.81:0/1483056609 conn(0x7f141c100790 msgr2=0x7f141c102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 -- 172.21.15.81:0/1483056609 shutdown_connections 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.885+0000 7f1424800700 1 -- 172.21.15.81:0/1483056609 wait complete. 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.887+0000 7f1424800700 1 Processor -- start 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.887+0000 7f1424800700 1 -- start start 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.887+0000 7f1424800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c104f20 0x7f141c19b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.887+0000 7f1424800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f141c1058d0 0x7f141c19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:06.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.887+0000 7f1424800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c10a000 0x7f141c19fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.887+0000 7f1424800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f141c115040 con 0x7f141c10a000 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.887+0000 7f1424800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f141c114ec0 con 0x7f141c1058d0 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.887+0000 7f1424800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f141c1151c0 con 0x7f141c104f20 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1422800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c10a000 0x7f141c19fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1422800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c10a000 0x7f141c19fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:57986/0 (socket says 172.21.15.81:57986) 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1422800700 1 -- 172.21.15.81:0/1213886330 learned_addr learned my addr 172.21.15.81:0/1213886330 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1421400700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f141c1058d0 0x7f141c19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1421e00700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c104f20 0x7f141c19b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:06.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1422800700 1 -- 172.21.15.81:0/1213886330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c104f20 msgr2=0x7f141c19b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1422800700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c104f20 0x7f141c19b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1422800700 1 -- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f141c1058d0 msgr2=0x7f141c19ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1422800700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f141c1058d0 0x7f141c19ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1422800700 1 -- 172.21.15.81:0/1213886330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c1a04e0 con 0x7f141c10a000 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1421e00700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f141c104f20 0x7f141c19b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.888+0000 7f1421400700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f141c1058d0 0x7f141c19ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.889+0000 7f1422800700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c10a000 0x7f141c19fda0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f141800ef30 tx=0x7f141800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.889+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14180092e0 con 0x7f141c10a000 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.889+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1418004bb0 con 0x7f141c10a000 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.889+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141c1a0770 con 0x7f141c10a000 2024-04-12T21:14:06.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.889+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418004280 con 0x7f141c10a000 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.889+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141c1a8050 con 0x7f141c10a000 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.890+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f141c063240 con 0x7f141c10a000 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.892+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f141801b030 con 0x7f141c10a000 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.894+0000 7f1416a00700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f14040a3d50 0x7f14040a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.894+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f14180d0b40 con 0x7f141c10a000 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.894+0000 7f1421e00700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f14040a3d50 0x7f14040a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.894+0000 7f1416a00700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] conn(0x7f14040aea10 0x7f14040b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.894+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 --> [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14040b1520 con 0x7f14040aea10 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.894+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f14180d0ef0 con 0x7f141c10a000 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.895+0000 7f1421e00700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f14040a3d50 0x7f14040a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1410000c00 tx=0x7f1410005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.895+0000 7f1421400700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] conn(0x7f14040aea10 0x7f14040b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.895+0000 7f1421400700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] conn(0x7f14040aea10 0x7f14040b0e50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.896+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 <== osd.1 v2:172.21.15.104:6800/3106352403 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14040b1520 con 0x7f14040aea10 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.914+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 --> [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f141c0636c0 con 0x7f14040aea10 2024-04-12T21:14:06.955 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.915+0000 7f1416a00700 1 -- 172.21.15.81:0/1213886330 <== osd.1 v2:172.21.15.104:6800/3106352403 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f141c0636c0 con 0x7f14040aea10 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] conn(0x7f14040aea10 msgr2=0x7f14040b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] conn(0x7f14040aea10 0x7f14040b0e50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f14040a3d50 msgr2=0x7f14040a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f14040a3d50 0x7f14040a6210 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1410000c00 tx=0x7f1410005fb0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c10a000 msgr2=0x7f141c19fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 --2- 172.21.15.81:0/1213886330 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f141c10a000 0x7f141c19fda0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f141800ef30 tx=0x7f141800c3a0 comp rx=0 tx=0).stop 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1422800700 1 -- 172.21.15.81:0/1213886330 reap_dead start 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 shutdown_connections 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 >> 172.21.15.81:0/1213886330 conn(0x7f141c100790 msgr2=0x7f141c102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 shutdown_connections 2024-04-12T21:14:06.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:06.916+0000 7f1424800700 1 -- 172.21.15.81:0/1213886330 wait complete. 2024-04-12T21:14:07.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:07 smithi081 ceph-mon[31453]: pgmap v341: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 2 op/s 2024-04-12T21:14:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:07 smithi153 ceph-mon[36569]: pgmap v341: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 2 op/s 2024-04-12T21:14:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:07 smithi104 ceph-mon[36624]: pgmap v341: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 426 B/s wr, 2 op/s 2024-04-12T21:14:09.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:09 smithi081 ceph-mon[31453]: pgmap v342: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:09 smithi153 ceph-mon[36569]: pgmap v342: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:09 smithi104 ceph-mon[36624]: pgmap v342: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:11.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 -- 172.21.15.81:0/2883509725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bf81034c0 msgr2=0x7f9bf810f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 --2- 172.21.15.81:0/2883509725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bf81034c0 0x7f9bf810f920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9bf000b0a0 tx=0x7f9bf000b3b0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.491 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 -- 172.21.15.81:0/2883509725 shutdown_connections 2024-04-12T21:14:11.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 --2- 172.21.15.81:0/2883509725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bf81034c0 0x7f9bf810f920 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 --2- 172.21.15.81:0/2883509725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf8108ac0 0x7f9bf8108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 --2- 172.21.15.81:0/2883509725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bf8102b00 0x7f9bf8102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 -- 172.21.15.81:0/2883509725 >> 172.21.15.81:0/2883509725 conn(0x7f9bf8079c10 msgr2=0x7f9bf807a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:11.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 -- 172.21.15.81:0/2883509725 shutdown_connections 2024-04-12T21:14:11.492 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.495+0000 7f9c00400700 1 -- 172.21.15.81:0/2883509725 wait complete. 2024-04-12T21:14:11.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.496+0000 7f9c00400700 1 Processor -- start 2024-04-12T21:14:11.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9c00400700 1 -- start start 2024-04-12T21:14:11.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9c00400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf8102b00 0x7f9bf8199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9c00400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bf81034c0 0x7f9bf8199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9c00400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bf8108ac0 0x7f9bf819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9c00400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf8112e70 con 0x7f9bf81034c0 2024-04-12T21:14:11.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9c00400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf8112cf0 con 0x7f9bf8108ac0 2024-04-12T21:14:11.493 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9c00400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9bf8112ff0 con 0x7f9bf8102b00 2024-04-12T21:14:11.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9bfd000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bf81034c0 0x7f9bf8199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9bfd000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bf81034c0 0x7f9bf8199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58006/0 (socket says 172.21.15.81:58006) 2024-04-12T21:14:11.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.497+0000 7f9bfd000700 1 -- 172.21.15.81:0/2647121813 learned_addr learned my addr 172.21.15.81:0/2647121813 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:11.494 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.498+0000 7f9bfe400700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bf8108ac0 0x7f9bf819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.498+0000 7f9bfd000700 1 -- 172.21.15.81:0/2647121813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf8102b00 msgr2=0x7f9bf8199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.498+0000 7f9bfd000700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9bf8102b00 0x7f9bf8199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.498+0000 7f9bfd000700 1 -- 172.21.15.81:0/2647121813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bf8108ac0 msgr2=0x7f9bf819dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.498+0000 7f9bfd000700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bf8108ac0 0x7f9bf819dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.498+0000 7f9bfd000700 1 -- 172.21.15.81:0/2647121813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bf819e300 con 0x7f9bf81034c0 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.498+0000 7f9bfe400700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9bf8108ac0 0x7f9bf819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.498+0000 7f9bfd000700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bf81034c0 0x7f9bf8199880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9be400e9a0 tx=0x7f9be400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.499+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be400cbd0 con 0x7f9bf81034c0 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.499+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9be400cd70 con 0x7f9bf81034c0 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.499+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bf819e5f0 con 0x7f9bf81034c0 2024-04-12T21:14:11.495 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.499+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9be4018750 con 0x7f9bf81034c0 2024-04-12T21:14:11.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.499+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bf81a5e70 con 0x7f9bf81034c0 2024-04-12T21:14:11.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.500+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9bf8063200 con 0x7f9bf81034c0 2024-04-12T21:14:11.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.502+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9be4010430 con 0x7f9bf81034c0 2024-04-12T21:14:11.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.503+0000 7f9bf6000700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9be00a3d50 0x7f9be00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.504+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9be40d1270 con 0x7f9bf81034c0 2024-04-12T21:14:11.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.504+0000 7f9bfda00700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9be00a3d50 0x7f9be00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.504+0000 7f9bf6000700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] conn(0x7f9be00aea10 0x7f9be00b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.504+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 --> [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9be00b1520 con 0x7f9be00aea10 2024-04-12T21:14:11.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.504+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f9be40d1620 con 0x7f9bf81034c0 2024-04-12T21:14:11.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.504+0000 7f9bfe400700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] conn(0x7f9be00aea10 0x7f9be00b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.504+0000 7f9bfda00700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9be00a3d50 0x7f9be00a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0097b0 tx=0x7f9bec006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:11.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.505+0000 7f9bfe400700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] conn(0x7f9be00aea10 0x7f9be00b0e50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:11.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.506+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 <== osd.3 v2:172.21.15.81:6810/4225327027 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9be00b1520 con 0x7f9be00aea10 2024-04-12T21:14:11.519 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.524+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 --> [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9bf8063600 con 0x7f9be00aea10 2024-04-12T21:14:11.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.524+0000 7f9bf6000700 1 -- 172.21.15.81:0/2647121813 <== osd.3 v2:172.21.15.81:6810/4225327027 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9bf8063600 con 0x7f9be00aea10 2024-04-12T21:14:11.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] conn(0x7f9be00aea10 msgr2=0x7f9be00b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] conn(0x7f9be00aea10 0x7f9be00b0e50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9be00a3d50 msgr2=0x7f9be00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9be00a3d50 0x7f9be00a6210 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9bec0097b0 tx=0x7f9bec006d20 comp rx=0 tx=0).stop 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bf81034c0 msgr2=0x7f9bf8199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 --2- 172.21.15.81:0/2647121813 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9bf81034c0 0x7f9bf8199880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9be400e9a0 tx=0x7f9be400ecb0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9bfe400700 1 -- 172.21.15.81:0/2647121813 reap_dead start 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 shutdown_connections 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 >> 172.21.15.81:0/2647121813 conn(0x7f9bf8079c10 msgr2=0x7f9bf8101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 shutdown_connections 2024-04-12T21:14:11.521 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.525+0000 7f9c00400700 1 -- 172.21.15.81:0/2647121813 wait complete. 2024-04-12T21:14:11.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:11 smithi081 ceph-mon[31453]: pgmap v343: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:11 smithi153 ceph-mon[36569]: pgmap v343: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:11.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:11 smithi104 ceph-mon[36624]: pgmap v343: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:11.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 -- 172.21.15.81:0/3759127754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8394074ce0 msgr2=0x7f83940750c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 --2- 172.21.15.81:0/3759127754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8394074ce0 0x7f83940750c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f838800b0a0 tx=0x7f838800b3b0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 -- 172.21.15.81:0/3759127754 shutdown_connections 2024-04-12T21:14:11.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 --2- 172.21.15.81:0/3759127754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8394113c80 0x7f8394116070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 --2- 172.21.15.81:0/3759127754 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8394074ce0 0x7f83940750c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 --2- 172.21.15.81:0/3759127754 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394075600 0x7f8394113740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 -- 172.21.15.81:0/3759127754 >> 172.21.15.81:0/3759127754 conn(0x7f8394079c10 msgr2=0x7f839407a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:11.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 -- 172.21.15.81:0/3759127754 shutdown_connections 2024-04-12T21:14:11.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.974+0000 7f839c000700 1 -- 172.21.15.81:0/3759127754 wait complete. 2024-04-12T21:14:11.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.975+0000 7f839c000700 1 Processor -- start 2024-04-12T21:14:11.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.976+0000 7f839c000700 1 -- start start 2024-04-12T21:14:11.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.976+0000 7f839c000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8394074ce0 0x7f83941a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.976+0000 7f839c000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8394075600 0x7f83941a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.976+0000 7f839c000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394113c80 0x7f839419f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.976+0000 7f839c000700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8394119400 con 0x7f8394113c80 2024-04-12T21:14:11.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.976+0000 7f839c000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8394119280 con 0x7f8394074ce0 2024-04-12T21:14:11.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.976+0000 7f839c000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8394119580 con 0x7f8394075600 2024-04-12T21:14:11.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.976+0000 7f839a000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394113c80 0x7f839419f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f8399600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8394074ce0 0x7f83941a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f839a000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394113c80 0x7f839419f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:58020/0 (socket says 172.21.15.81:58020) 2024-04-12T21:14:11.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f839a000700 1 -- 172.21.15.81:0/3149597539 learned_addr learned my addr 172.21.15.81:0/3149597539 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:11.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f8398c00700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8394075600 0x7f83941a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f8399600700 1 -- 172.21.15.81:0/3149597539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8394075600 msgr2=0x7f83941a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f8399600700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8394075600 0x7f83941a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f8399600700 1 -- 172.21.15.81:0/3149597539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394113c80 msgr2=0x7f839419f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:11.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f8399600700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8394113c80 0x7f839419f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:11.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f8399600700 1 -- 172.21.15.81:0/3149597539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839419ff60 con 0x7f8394074ce0 2024-04-12T21:14:11.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.977+0000 7f8399600700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8394074ce0 0x7f83941a5600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f838800b780 tx=0x7f8388004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:11.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.978+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388021070 con 0x7f8394074ce0 2024-04-12T21:14:11.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.978+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8388007b80 con 0x7f8394074ce0 2024-04-12T21:14:11.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.978+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83941a01f0 con 0x7f8394074ce0 2024-04-12T21:14:11.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.978+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83880052a0 con 0x7f8394074ce0 2024-04-12T21:14:11.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.978+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83941ac460 con 0x7f8394074ce0 2024-04-12T21:14:11.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.979+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f83941134a0 con 0x7f8394074ce0 2024-04-12T21:14:11.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.981+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8388024070 con 0x7f8394074ce0 2024-04-12T21:14:11.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.982+0000 7f8392000700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f837c0a3d50 0x7f837c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.983+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f838800f020 con 0x7f8394074ce0 2024-04-12T21:14:11.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.983+0000 7f8398c00700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f837c0a3d50 0x7f837c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.983+0000 7f8392000700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] conn(0x7f837c0aea10 0x7f837c0b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:11.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.983+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 --> [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f837c0b1520 con 0x7f837c0aea10 2024-04-12T21:14:11.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.983+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f83880d2210 con 0x7f8394074ce0 2024-04-12T21:14:11.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.983+0000 7f839a000700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] conn(0x7f837c0aea10 0x7f837c0b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:11.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.983+0000 7f8398c00700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f837c0a3d50 0x7f837c0a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8394110630 tx=0x7f8380009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:11.979 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.984+0000 7f839a000700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] conn(0x7f837c0aea10 0x7f837c0b0e50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:11.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:11.985+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 <== osd.4 v2:172.21.15.104:6808/742573974 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f837c0b1520 con 0x7f837c0aea10 2024-04-12T21:14:12.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.003+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 --> [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f83940763d0 con 0x7f837c0aea10 2024-04-12T21:14:12.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.004+0000 7f8392000700 1 -- 172.21.15.81:0/3149597539 <== osd.4 v2:172.21.15.104:6808/742573974 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f83940763d0 con 0x7f837c0aea10 2024-04-12T21:14:12.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.004+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 >> [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] conn(0x7f837c0aea10 msgr2=0x7f837c0b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:12.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.004+0000 7f839c000700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] conn(0x7f837c0aea10 0x7f837c0b0e50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:12.003 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f837c0a3d50 msgr2=0x7f837c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:12.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839c000700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f837c0a3d50 0x7f837c0a6210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8394110630 tx=0x7f8380009260 comp rx=0 tx=0).stop 2024-04-12T21:14:12.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8394074ce0 msgr2=0x7f83941a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:12.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839c000700 1 --2- 172.21.15.81:0/3149597539 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8394074ce0 0x7f83941a5600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f838800b780 tx=0x7f8388004d00 comp rx=0 tx=0).stop 2024-04-12T21:14:12.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839a000700 1 -- 172.21.15.81:0/3149597539 reap_dead start 2024-04-12T21:14:12.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 shutdown_connections 2024-04-12T21:14:12.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 >> 172.21.15.81:0/3149597539 conn(0x7f8394079c10 msgr2=0x7f8394112ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:12.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 shutdown_connections 2024-04-12T21:14:12.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:12.005+0000 7f839c000700 1 -- 172.21.15.81:0/3149597539 wait complete. 2024-04-12T21:14:13.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:13 smithi081 ceph-mon[31453]: pgmap v344: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:13.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:14:13.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:13 smithi153 ceph-mon[36569]: pgmap v344: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:13.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:14:13.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:13 smithi104 ceph-mon[36624]: pgmap v344: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:13.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:14:13.810 INFO:teuthology.orchestra.run.smithi081.stdout:141733920834 2024-04-12T21:14:13.810 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-12T21:14:14.311 INFO:teuthology.orchestra.run.smithi081.stdout:107374182470 2024-04-12T21:14:14.311 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-12T21:14:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:15 smithi153 ceph-mon[36569]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:15 smithi104 ceph-mon[36624]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.879 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:15 smithi081 ceph-mon[31453]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:15.879 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.879 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.879 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.879 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:15.940 INFO:teuthology.orchestra.run.smithi081.stdout:55834574929 2024-04-12T21:14:15.940 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.499+0000 7fee0fc00700 1 -- 172.21.15.81:0/3225661645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee08074ce0 msgr2=0x7fee080750c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.499+0000 7fee0fc00700 1 --2- 172.21.15.81:0/3225661645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee08074ce0 0x7fee080750c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009990 tx=0x7fedfc009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.499+0000 7fee0fc00700 1 -- 172.21.15.81:0/3225661645 shutdown_connections 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.499+0000 7fee0fc00700 1 --2- 172.21.15.81:0/3225661645 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee0810f510 0x7fee08111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.499+0000 7fee0fc00700 1 --2- 172.21.15.81:0/3225661645 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee08074ce0 0x7fee080750c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.499+0000 7fee0fc00700 1 --2- 172.21.15.81:0/3225661645 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee08075600 0x7fee0810efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.499+0000 7fee0fc00700 1 -- 172.21.15.81:0/3225661645 >> 172.21.15.81:0/3225661645 conn(0x7fee08079c10 msgr2=0x7fee0807a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.500+0000 7fee0fc00700 1 -- 172.21.15.81:0/3225661645 shutdown_connections 2024-04-12T21:14:17.496 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.500+0000 7fee0fc00700 1 -- 172.21.15.81:0/3225661645 wait complete. 2024-04-12T21:14:17.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.501+0000 7fee0fc00700 1 Processor -- start 2024-04-12T21:14:17.497 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.501+0000 7fee0fc00700 1 -- start start 2024-04-12T21:14:17.498 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee0fc00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee08074ce0 0x7fee0819b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee0fc00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08075600 0x7fee0819b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee0fc00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee0810f510 0x7fee0819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee0fc00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fee08115020 con 0x7fee0810f510 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee0fc00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fee08114ea0 con 0x7fee08074ce0 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee0fc00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fee081151a0 con 0x7fee08075600 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee07e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08075600 0x7fee0819b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee07e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08075600 0x7fee0819b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:39290/0 (socket says 172.21.15.81:39290) 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.502+0000 7fee07e00700 1 -- 172.21.15.81:0/648485017 learned_addr learned my addr 172.21.15.81:0/648485017 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee0d200700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee08074ce0 0x7fee0819b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee0dc00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee0810f510 0x7fee0819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee07e00700 1 -- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee08074ce0 msgr2=0x7fee0819b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee07e00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee08074ce0 0x7fee0819b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee07e00700 1 -- 172.21.15.81:0/648485017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee0810f510 msgr2=0x7fee0819fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:17.499 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee07e00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fee0810f510 0x7fee0819fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:17.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee07e00700 1 -- 172.21.15.81:0/648485017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee081a0380 con 0x7fee08075600 2024-04-12T21:14:17.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee0d200700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fee08074ce0 0x7fee0819b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:17.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.503+0000 7fee07e00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08075600 0x7fee0819b8b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedf400eb30 tx=0x7fedf400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:17.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.504+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf400cbd0 con 0x7fee08075600 2024-04-12T21:14:17.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.504+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf4004500 con 0x7fee08075600 2024-04-12T21:14:17.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.504+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4010430 con 0x7fee08075600 2024-04-12T21:14:17.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.504+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee081a0670 con 0x7fee08075600 2024-04-12T21:14:17.500 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.504+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee081a7ef0 con 0x7fee08075600 2024-04-12T21:14:17.501 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.505+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fee08100660 con 0x7fee08075600 2024-04-12T21:14:17.503 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.507+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fedf4004020 con 0x7fee08075600 2024-04-12T21:14:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.509+0000 7fee05600700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fedf00a3d50 0x7fedf00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.509+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fedf40d0b10 con 0x7fee08075600 2024-04-12T21:14:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.509+0000 7fee0d200700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fedf00a3d50 0x7fedf00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.509+0000 7fee05600700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] conn(0x7fedf00aea10 0x7fedf00b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.509+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 --> [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fedf401ee50 con 0x7fedf00aea10 2024-04-12T21:14:17.505 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.509+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fedf40d52a0 con 0x7fee08075600 2024-04-12T21:14:17.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.510+0000 7fee0d200700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fedf00a3d50 0x7fedf00a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00b2e0 tx=0x7fedfc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:17.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.510+0000 7fee0dc00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] conn(0x7fedf00aea10 0x7fedf00b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:17.506 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.510+0000 7fee0dc00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] conn(0x7fedf00aea10 0x7fedf00b0e50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:17.507 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.511+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 <== osd.10 v2:172.21.15.104:6824/1019144761 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fedf401ee50 con 0x7fedf00aea10 2024-04-12T21:14:17.525 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.530+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 --> [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fee080763a0 con 0x7fedf00aea10 2024-04-12T21:14:17.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.530+0000 7fee05600700 1 -- 172.21.15.81:0/648485017 <== osd.10 v2:172.21.15.104:6824/1019144761 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fee080763a0 con 0x7fedf00aea10 2024-04-12T21:14:17.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] conn(0x7fedf00aea10 msgr2=0x7fedf00b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:17.527 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] conn(0x7fedf00aea10 0x7fedf00b0e50 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fedf00a3d50 msgr2=0x7fedf00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fedf00a3d50 0x7fedf00a6210 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00b2e0 tx=0x7fedfc005960 comp rx=0 tx=0).stop 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08075600 msgr2=0x7fee0819b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 --2- 172.21.15.81:0/648485017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee08075600 0x7fee0819b8b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fedf400eb30 tx=0x7fedf400ee40 comp rx=0 tx=0).stop 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0dc00700 1 -- 172.21.15.81:0/648485017 reap_dead start 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 shutdown_connections 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 >> 172.21.15.81:0/648485017 conn(0x7fee08079c10 msgr2=0x7fee080ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.531+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 shutdown_connections 2024-04-12T21:14:17.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.532+0000 7fee0fc00700 1 -- 172.21.15.81:0/648485017 wait complete. 2024-04-12T21:14:17.750 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:17 smithi081 ceph-mon[31453]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:17.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:17 smithi153 ceph-mon[36569]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:17.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:17 smithi104 ceph-mon[36624]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:18.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.754+0000 7f3a63400700 1 -- 172.21.15.81:0/3001838926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c10e420 msgr2=0x7f3a5c115ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.754+0000 7f3a63400700 1 --2- 172.21.15.81:0/3001838926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c10e420 0x7f3a5c115ff0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400b0d0 tx=0x7f3a5400b3e0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.754+0000 7f3a63400700 1 -- 172.21.15.81:0/3001838926 shutdown_connections 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.754+0000 7f3a63400700 1 --2- 172.21.15.81:0/3001838926 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c10e420 0x7f3a5c115ff0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.754+0000 7f3a63400700 1 --2- 172.21.15.81:0/3001838926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c109340 0x7f3a5c109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.754+0000 7f3a63400700 1 --2- 172.21.15.81:0/3001838926 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c109cf0 0x7f3a5c10dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.754+0000 7f3a63400700 1 -- 172.21.15.81:0/3001838926 >> 172.21.15.81:0/3001838926 conn(0x7f3a5c076280 msgr2=0x7f3a5c1078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.754+0000 7f3a63400700 1 -- 172.21.15.81:0/3001838926 shutdown_connections 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.755+0000 7f3a63400700 1 -- 172.21.15.81:0/3001838926 wait complete. 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.756+0000 7f3a63400700 1 Processor -- start 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.756+0000 7f3a63400700 1 -- start start 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a63400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c109340 0x7f3a5c1a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a63400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c109cf0 0x7f3a5c1a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a63400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c10e420 0x7f3a5c19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a63400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c119480 con 0x7f3a5c109cf0 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a63400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c119300 con 0x7f3a5c109340 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a63400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c119600 con 0x7f3a5c10e420 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a5be00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c109cf0 0x7f3a5c1a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a60a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c109340 0x7f3a5c1a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a60a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c109340 0x7f3a5c1a55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:57578/0 (socket says 172.21.15.81:57578) 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a60a00700 1 -- 172.21.15.81:0/3368511184 learned_addr learned my addr 172.21.15.81:0/3368511184 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a61400700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c10e420 0x7f3a5c19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a5be00700 1 -- 172.21.15.81:0/3368511184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c10e420 msgr2=0x7f3a5c19f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a5be00700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c10e420 0x7f3a5c19f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.757+0000 7f3a5be00700 1 -- 172.21.15.81:0/3368511184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c109340 msgr2=0x7f3a5c1a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a5be00700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3a5c109340 0x7f3a5c1a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a5be00700 1 -- 172.21.15.81:0/3368511184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5c19ff60 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a61400700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a5c10e420 0x7f3a5c19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a5be00700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c109cf0 0x7f3a5c1a5af0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e9c0 tx=0x7f3a4800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4800c9c0 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a4800cb60 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48018680 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a5c1a0250 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.758+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a5c1ac530 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.759+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3a5c077180 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.761+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3a48010430 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.763+0000 7f3a59600700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3a440a3d50 0x7f3a440a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.763+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3a480d0c60 con 0x7f3a5c109cf0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.763+0000 7f3a60a00700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3a440a3d50 0x7f3a440a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.763+0000 7f3a59600700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] conn(0x7f3a440aea10 0x7f3a440b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.763+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 --> [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a48019ec0 con 0x7f3a440aea10 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.763+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f3a480d1010 con 0x7f3a5c109cf0 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.764+0000 7f3a61400700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] conn(0x7f3a440aea10 0x7f3a440b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.764+0000 7f3a60a00700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3a440a3d50 0x7f3a440a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000bd30 tx=0x7f3a5000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.764+0000 7f3a61400700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] conn(0x7f3a440aea10 0x7f3a440b0e50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.765+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 <== osd.0 v2:172.21.15.81:6802/4059792347 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3a48019ec0 con 0x7f3a440aea10 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.783+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 --> [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3a5c10a820 con 0x7f3a440aea10 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.783+0000 7f3a59600700 1 -- 172.21.15.81:0/3368511184 <== osd.0 v2:172.21.15.81:6802/4059792347 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3a5c10a820 con 0x7f3a440aea10 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] conn(0x7f3a440aea10 msgr2=0x7f3a440b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] conn(0x7f3a440aea10 0x7f3a440b0e50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3a440a3d50 msgr2=0x7f3a440a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3a440a3d50 0x7f3a440a6210 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000bd30 tx=0x7f3a5000b480 comp rx=0 tx=0).stop 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c109cf0 msgr2=0x7f3a5c1a5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 --2- 172.21.15.81:0/3368511184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3a5c109cf0 0x7f3a5c1a5af0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800e9c0 tx=0x7f3a4800ecd0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a61400700 1 -- 172.21.15.81:0/3368511184 reap_dead start 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 shutdown_connections 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 >> 172.21.15.81:0/3368511184 conn(0x7f3a5c076280 msgr2=0x7f3a5c076760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 shutdown_connections 2024-04-12T21:14:18.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.784+0000 7f3a63400700 1 -- 172.21.15.81:0/3368511184 wait complete. 2024-04-12T21:14:18.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.406+0000 7f622e600700 1 -- 172.21.15.81:0/2323839271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62281034c0 msgr2=0x7f622810f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.406+0000 7f622e600700 1 --2- 172.21.15.81:0/2323839271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62281034c0 0x7f622810f920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f622000b0a0 tx=0x7f622000b3b0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.406+0000 7f622e600700 1 -- 172.21.15.81:0/2323839271 shutdown_connections 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.406+0000 7f622e600700 1 --2- 172.21.15.81:0/2323839271 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f62281034c0 0x7f622810f920 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.406+0000 7f622e600700 1 --2- 172.21.15.81:0/2323839271 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6228108ac0 0x7f6228108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.406+0000 7f622e600700 1 --2- 172.21.15.81:0/2323839271 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6228102b00 0x7f6228102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.406+0000 7f622e600700 1 -- 172.21.15.81:0/2323839271 >> 172.21.15.81:0/2323839271 conn(0x7f6228079c10 msgr2=0x7f622807a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.407+0000 7f622e600700 1 -- 172.21.15.81:0/2323839271 shutdown_connections 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.407+0000 7f622e600700 1 -- 172.21.15.81:0/2323839271 wait complete. 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.408+0000 7f622e600700 1 Processor -- start 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.408+0000 7f622e600700 1 -- start start 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f622e600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6228102b00 0x7f6228199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f622e600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62281034c0 0x7f62281997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f622e600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6228108ac0 0x7f622819db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f622e600700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6228112e30 con 0x7f6228102b00 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f622e600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6228112cb0 con 0x7f62281034c0 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f622e600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6228112fb0 con 0x7f6228108ac0 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f6227400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6228102b00 0x7f6228199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f6227e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6228108ac0 0x7f622819db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f6226a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62281034c0 0x7f62281997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f6227e00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6228108ac0 0x7f622819db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:39266/0 (socket says 172.21.15.81:39266) 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f6227e00700 1 -- 172.21.15.81:0/2024383390 learned_addr learned my addr 172.21.15.81:0/2024383390 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.409+0000 7f6226a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62281034c0 0x7f62281997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:57528/0 (socket says 172.21.15.81:57528) 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f6227e00700 1 -- 172.21.15.81:0/2024383390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62281034c0 msgr2=0x7f62281997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f6227e00700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62281034c0 0x7f62281997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f6227e00700 1 -- 172.21.15.81:0/2024383390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6228102b00 msgr2=0x7f6228199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f6227e00700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6228102b00 0x7f6228199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f6227e00700 1 -- 172.21.15.81:0/2024383390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622819e290 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f6226a00700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62281034c0 0x7f62281997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f6227e00700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6228108ac0 0x7f622819db50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f622000b070 tx=0x7f6220004c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220021070 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6220003c00 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622819e520 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220004740 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.410+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62281a5e00 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.411+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6228063240 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.414+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6220007980 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.415+0000 7f621fe00700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f620c0a3cd0 0x7f620c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.415+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f622000f020 con 0x7f6228108ac0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.415+0000 7f6227400700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f620c0a3cd0 0x7f620c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.415+0000 7f621fe00700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] conn(0x7f620c0ae990 0x7f620c0b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:18.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.415+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 --> [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f620c0b14a0 con 0x7f620c0ae990 2024-04-12T21:14:18.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.415+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f62200d20d0 con 0x7f6228108ac0 2024-04-12T21:14:18.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.416+0000 7f6226a00700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] conn(0x7f620c0ae990 0x7f620c0b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:18.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.416+0000 7f6227400700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f620c0a3cd0 0x7f620c0a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6218009be0 tx=0x7f6218009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:18.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.416+0000 7f6226a00700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] conn(0x7f620c0ae990 0x7f620c0b0dd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:18.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.417+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 <== osd.7 v2:172.21.15.104:6816/1651003262 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f620c0b14a0 con 0x7f620c0ae990 2024-04-12T21:14:18.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.436+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 --> [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f62280636c0 con 0x7f620c0ae990 2024-04-12T21:14:18.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.437+0000 7f621fe00700 1 -- 172.21.15.81:0/2024383390 <== osd.7 v2:172.21.15.104:6816/1651003262 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f62280636c0 con 0x7f620c0ae990 2024-04-12T21:14:18.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 >> [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] conn(0x7f620c0ae990 msgr2=0x7f620c0b0dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] conn(0x7f620c0ae990 0x7f620c0b0dd0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f620c0a3cd0 msgr2=0x7f620c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f620c0a3cd0 0x7f620c0a6190 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6218009be0 tx=0x7f6218009260 comp rx=0 tx=0).stop 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6228108ac0 msgr2=0x7f622819db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 --2- 172.21.15.81:0/2024383390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6228108ac0 0x7f622819db50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f622000b070 tx=0x7f6220004c60 comp rx=0 tx=0).stop 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f6227e00700 1 -- 172.21.15.81:0/2024383390 reap_dead start 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 shutdown_connections 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 >> 172.21.15.81:0/2024383390 conn(0x7f6228079c10 msgr2=0x7f6228101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 shutdown_connections 2024-04-12T21:14:18.369 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:17.438+0000 7f622e600700 1 -- 172.21.15.81:0/2024383390 wait complete. 2024-04-12T21:14:19.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:19 smithi081 ceph-mon[31453]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:14:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:19 smithi153 ceph-mon[36569]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:14:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:19 smithi104 ceph-mon[36624]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.888+0000 7f38e7a00700 1 -- 172.21.15.81:0/1436720819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0108b70 msgr2=0x7f38e0108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.888+0000 7f38e7a00700 1 --2- 172.21.15.81:0/1436720819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0108b70 0x7f38e0108f50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f38d40098f0 tx=0x7f38d4009c00 comp rx=0 tx=0).stop 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.888+0000 7f38e7a00700 1 -- 172.21.15.81:0/1436720819 shutdown_connections 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.889+0000 7f38e7a00700 1 --2- 172.21.15.81:0/1436720819 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38e01034d0 0x7f38e010f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.889+0000 7f38e7a00700 1 --2- 172.21.15.81:0/1436720819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0108b70 0x7f38e0108f50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.889+0000 7f38e7a00700 1 --2- 172.21.15.81:0/1436720819 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e0102b30 0x7f38e0102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.889+0000 7f38e7a00700 1 -- 172.21.15.81:0/1436720819 >> 172.21.15.81:0/1436720819 conn(0x7f38e0079c60 msgr2=0x7f38e007a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.889+0000 7f38e7a00700 1 -- 172.21.15.81:0/1436720819 shutdown_connections 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.889+0000 7f38e7a00700 1 -- 172.21.15.81:0/1436720819 wait complete. 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.890+0000 7f38e7a00700 1 Processor -- start 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.890+0000 7f38e7a00700 1 -- start start 2024-04-12T21:14:20.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e7a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0102b30 0x7f38e0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e7a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e01034d0 0x7f38e0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e7a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38e0108b70 0x7f38e019dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e7a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0112e20 con 0x7f38e0108b70 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e7a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0112ca0 con 0x7f38e0102b30 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e7a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f38e0112fa0 con 0x7f38e01034d0 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e5000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0102b30 0x7f38e0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38dfe00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e01034d0 0x7f38e0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e5000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0102b30 0x7f38e0199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:57602/0 (socket says 172.21.15.81:57602) 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e5000700 1 -- 172.21.15.81:0/2266818801 learned_addr learned my addr 172.21.15.81:0/2266818801 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.891+0000 7f38e5a00700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38e0108b70 0x7f38e019dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38e5000700 1 -- 172.21.15.81:0/2266818801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e01034d0 msgr2=0x7f38e0199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38e5000700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f38e01034d0 0x7f38e0199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38e5000700 1 -- 172.21.15.81:0/2266818801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38e0108b70 msgr2=0x7f38e019dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:20.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38e5000700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f38e0108b70 0x7f38e019dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38e5000700 1 -- 172.21.15.81:0/2266818801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e019e340 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38e5000700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0102b30 0x7f38e0199330 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f38d4005950 tx=0x7f38d4005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4005230 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38d400ba40 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e019e5d0 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.892+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38d4003b60 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.893+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e01a5eb0 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.893+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f38e00631c0 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.896+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f38d4003d00 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.897+0000 7f38dd600700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f38c80a3cd0 0x7f38c80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.897+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f38d40d1c50 con 0x7f38e0102b30 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.897+0000 7f38dfe00700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f38c80a3cd0 0x7f38c80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.898+0000 7f38dd600700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] conn(0x7f38c80ae860 0x7f38c80b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:20.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.898+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 --> [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38d400b890 con 0x7f38c80ae860 2024-04-12T21:14:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.898+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f38d40d2000 con 0x7f38e0102b30 2024-04-12T21:14:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.898+0000 7f38e5a00700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] conn(0x7f38c80ae860 0x7f38c80b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.898+0000 7f38dfe00700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f38c80a3cd0 0x7f38c80a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f38e007a340 tx=0x7f38cc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.899+0000 7f38e5a00700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] conn(0x7f38c80ae860 0x7f38c80b0ca0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.900+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 <== osd.2 v2:172.21.15.153:6800/1512806537 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f38d400b890 con 0x7f38c80ae860 2024-04-12T21:14:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.917+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 --> [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f38e0063640 con 0x7f38c80ae860 2024-04-12T21:14:20.358 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.918+0000 7f38dd600700 1 -- 172.21.15.81:0/2266818801 <== osd.2 v2:172.21.15.153:6800/1512806537 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f38e0063640 con 0x7f38c80ae860 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 >> [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] conn(0x7f38c80ae860 msgr2=0x7f38c80b0ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] conn(0x7f38c80ae860 0x7f38c80b0ca0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f38c80a3cd0 msgr2=0x7f38c80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f38c80a3cd0 0x7f38c80a6190 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f38e007a340 tx=0x7f38cc00b410 comp rx=0 tx=0).stop 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0102b30 msgr2=0x7f38e0199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 --2- 172.21.15.81:0/2266818801 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f38e0102b30 0x7f38e0199330 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f38d4005950 tx=0x7f38d4005040 comp rx=0 tx=0).stop 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e5a00700 1 -- 172.21.15.81:0/2266818801 reap_dead start 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 shutdown_connections 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 >> 172.21.15.81:0/2266818801 conn(0x7f38e0079c60 msgr2=0x7f38e0100ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 shutdown_connections 2024-04-12T21:14:20.359 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:19.919+0000 7f38e7a00700 1 -- 172.21.15.81:0/2266818801 wait complete. 2024-04-12T21:14:21.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:21 smithi081 ceph-mon[31453]: pgmap v348: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:14:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:21 smithi153 ceph-mon[36569]: pgmap v348: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:14:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:21 smithi104 ceph-mon[36624]: pgmap v348: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:14:21.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 -- 172.21.15.81:0/3605161854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7010a030 msgr2=0x7f9d70111c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:21.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 --2- 172.21.15.81:0/3605161854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7010a030 0x7f9d70111c00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800b0a0 tx=0x7f9d6800b3b0 comp rx=0 tx=0).stop 2024-04-12T21:14:21.993 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 -- 172.21.15.81:0/3605161854 shutdown_connections 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 --2- 172.21.15.81:0/3605161854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d7010a030 0x7f9d70111c00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 --2- 172.21.15.81:0/3605161854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d70104f50 0x7f9d70105330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 --2- 172.21.15.81:0/3605161854 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d70105900 0x7f9d70109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 -- 172.21.15.81:0/3605161854 >> 172.21.15.81:0/3605161854 conn(0x7f9d701007e0 msgr2=0x7f9d70102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 -- 172.21.15.81:0/3605161854 shutdown_connections 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.694+0000 7f9d75a00700 1 -- 172.21.15.81:0/3605161854 wait complete. 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.695+0000 7f9d75a00700 1 Processor -- start 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.695+0000 7f9d75a00700 1 -- start start 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d75a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d70104f50 0x7f9d7019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d75a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d70105900 0x7f9d7019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d75a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7010a030 0x7f9d7019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d75a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9d70115010 con 0x7f9d70104f50 2024-04-12T21:14:21.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d75a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9d70114e90 con 0x7f9d70105900 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d75a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9d70115190 con 0x7f9d7010a030 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d6ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d70104f50 0x7f9d7019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d6ea00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d70104f50 0x7f9d7019b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:39964/0 (socket says 172.21.15.81:39964) 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d6ea00700 1 -- 172.21.15.81:0/3604419940 learned_addr learned my addr 172.21.15.81:0/3604419940 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d6e000700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d70105900 0x7f9d7019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d6f400700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7010a030 0x7f9d7019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d6ea00700 1 -- 172.21.15.81:0/3604419940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7010a030 msgr2=0x7f9d7019fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.696+0000 7f9d6ea00700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9d7010a030 0x7f9d7019fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d6ea00700 1 -- 172.21.15.81:0/3604419940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d70105900 msgr2=0x7f9d7019b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d6ea00700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d70105900 0x7f9d7019b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d6ea00700 1 -- 172.21.15.81:0/3604419940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d701a0410 con 0x7f9d70104f50 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d6e000700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9d70105900 0x7f9d7019b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d6ea00700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d70104f50 0x7f9d7019b400 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000e9a0 tx=0x7f9d6000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:21.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6000ca10 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6000cbb0 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d701a0700 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60018680 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.697+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d701a7f80 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.698+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9d700631c0 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.700+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9d60010430 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.702+0000 7f9d67400700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9d540a3d50 0x7f9d540a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.702+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9d600d11e0 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.702+0000 7f9d6e000700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9d540a3d50 0x7f9d540a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.702+0000 7f9d67400700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] conn(0x7f9d540aea10 0x7f9d540b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.702+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 --> [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d60027e50 con 0x7f9d540aea10 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.702+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f9d600d1590 con 0x7f9d70104f50 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.702+0000 7f9d6f400700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] conn(0x7f9d540aea10 0x7f9d540b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.703+0000 7f9d6e000700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9d540a3d50 0x7f9d540a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9d70103ee0 tx=0x7f9d5c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:21.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.703+0000 7f9d6f400700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] conn(0x7f9d540aea10 0x7f9d540b0e50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.705+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 <== osd.6 v2:172.21.15.81:6818/3825683804 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d60027e50 con 0x7f9d540aea10 2024-04-12T21:14:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.722+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 --> [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9d700635c0 con 0x7f9d540aea10 2024-04-12T21:14:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.723+0000 7f9d67400700 1 -- 172.21.15.81:0/3604419940 <== osd.6 v2:172.21.15.81:6818/3825683804 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9d700635c0 con 0x7f9d540aea10 2024-04-12T21:14:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] conn(0x7f9d540aea10 msgr2=0x7f9d540b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] conn(0x7f9d540aea10 0x7f9d540b0e50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9d540a3d50 msgr2=0x7f9d540a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:21.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9d540a3d50 0x7f9d540a6210 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9d70103ee0 tx=0x7f9d5c009260 comp rx=0 tx=0).stop 2024-04-12T21:14:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d70104f50 msgr2=0x7f9d7019b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 --2- 172.21.15.81:0/3604419940 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9d70104f50 0x7f9d7019b400 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000e9a0 tx=0x7f9d6000ecb0 comp rx=0 tx=0).stop 2024-04-12T21:14:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d6f400700 1 -- 172.21.15.81:0/3604419940 reap_dead start 2024-04-12T21:14:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 shutdown_connections 2024-04-12T21:14:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 >> 172.21.15.81:0/3604419940 conn(0x7f9d701007e0 msgr2=0x7f9d70101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 shutdown_connections 2024-04-12T21:14:21.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.724+0000 7f9d75a00700 1 -- 172.21.15.81:0/3604419940 wait complete. 2024-04-12T21:14:22.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.794+0000 7ff084200700 1 -- 172.21.15.81:0/407750574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff07c074ce0 msgr2=0x7ff07c0750c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:22.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.794+0000 7ff084200700 1 --2- 172.21.15.81:0/407750574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff07c074ce0 0x7ff07c0750c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff070009990 tx=0x7ff070009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:22.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.794+0000 7ff084200700 1 -- 172.21.15.81:0/407750574 shutdown_connections 2024-04-12T21:14:22.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.794+0000 7ff084200700 1 --2- 172.21.15.81:0/407750574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff07c10f4b0 0x7ff07c1118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:22.456 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.794+0000 7ff084200700 1 --2- 172.21.15.81:0/407750574 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff07c074ce0 0x7ff07c0750c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.794+0000 7ff084200700 1 --2- 172.21.15.81:0/407750574 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07c075600 0x7ff07c10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.794+0000 7ff084200700 1 -- 172.21.15.81:0/407750574 >> 172.21.15.81:0/407750574 conn(0x7ff07c079c60 msgr2=0x7ff07c07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.795+0000 7ff084200700 1 -- 172.21.15.81:0/407750574 shutdown_connections 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.795+0000 7ff084200700 1 -- 172.21.15.81:0/407750574 wait complete. 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.796+0000 7ff084200700 1 Processor -- start 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.796+0000 7ff084200700 1 -- start start 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.796+0000 7ff084200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff07c074ce0 0x7ff07c19b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.796+0000 7ff084200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07c075600 0x7ff07c19ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.796+0000 7ff084200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff07c10f4b0 0x7ff07c19fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.796+0000 7ff084200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c115080 con 0x7ff07c10f4b0 2024-04-12T21:14:22.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.796+0000 7ff084200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c114f00 con 0x7ff07c075600 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.796+0000 7ff084200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c115200 con 0x7ff07c074ce0 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff082200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff07c10f4b0 0x7ff07c19fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff081800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff07c074ce0 0x7ff07c19b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff081800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff07c074ce0 0x7ff07c19b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:39360/0 (socket says 172.21.15.81:39360) 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff081800700 1 -- 172.21.15.81:0/42113345 learned_addr learned my addr 172.21.15.81:0/42113345 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff080e00700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07c075600 0x7ff07c19ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff082200700 1 -- 172.21.15.81:0/42113345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff07c074ce0 msgr2=0x7ff07c19b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff082200700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff07c074ce0 0x7ff07c19b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff082200700 1 -- 172.21.15.81:0/42113345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07c075600 msgr2=0x7ff07c19ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff082200700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07c075600 0x7ff07c19ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:22.458 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff082200700 1 -- 172.21.15.81:0/42113345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07c1a04d0 con 0x7ff07c10f4b0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff081800700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff07c074ce0 0x7ff07c19b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff080e00700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff07c075600 0x7ff07c19ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.797+0000 7ff082200700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff07c10f4b0 0x7ff07c19fd90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff07c06b2e0 tx=0x7ff07400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.798+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0740092e0 con 0x7ff07c10f4b0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.798+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff074004bb0 con 0x7ff07c10f4b0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.798+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff07c1a0760 con 0x7ff07c10f4b0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.798+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074004020 con 0x7ff07c10f4b0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.798+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff07c1a8040 con 0x7ff07c10f4b0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.799+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff07c063240 con 0x7ff07c10f4b0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.801+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff07401b030 con 0x7ff07c10f4b0 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.802+0000 7ff07a000700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff0640a3d50 0x7ff0640a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:22.459 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.803+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff0740d0b40 con 0x7ff07c10f4b0 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.803+0000 7ff081800700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff0640a3d50 0x7ff0640a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.803+0000 7ff07a000700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] conn(0x7ff0640aea10 0x7ff0640b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.803+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 --> [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff074010ec0 con 0x7ff0640aea10 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.803+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7ff0740d0ef0 con 0x7ff07c10f4b0 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.803+0000 7ff081800700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff0640a3d50 0x7ff0640a6210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff070009750 tx=0x7ff07000b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.803+0000 7ff080e00700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] conn(0x7ff0640aea10 0x7ff0640b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.804+0000 7ff080e00700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] conn(0x7ff0640aea10 0x7ff0640b0e50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.805+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 <== osd.5 v2:172.21.15.153:6808/2854081701 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff074010ec0 con 0x7ff0640aea10 2024-04-12T21:14:22.460 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.823+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 --> [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff07c0636c0 con 0x7ff0640aea10 2024-04-12T21:14:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.823+0000 7ff07a000700 1 -- 172.21.15.81:0/42113345 <== osd.5 v2:172.21.15.153:6808/2854081701 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff07c0636c0 con 0x7ff0640aea10 2024-04-12T21:14:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 >> [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] conn(0x7ff0640aea10 msgr2=0x7ff0640b0e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] conn(0x7ff0640aea10 0x7ff0640b0e50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff0640a3d50 msgr2=0x7ff0640a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:22.461 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff0640a3d50 0x7ff0640a6210 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff070009750 tx=0x7ff07000b280 comp rx=0 tx=0).stop 2024-04-12T21:14:22.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff07c10f4b0 msgr2=0x7ff07c19fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:22.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 --2- 172.21.15.81:0/42113345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff07c10f4b0 0x7ff07c19fd90 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff07c06b2e0 tx=0x7ff07400ef90 comp rx=0 tx=0).stop 2024-04-12T21:14:22.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff082200700 1 -- 172.21.15.81:0/42113345 reap_dead start 2024-04-12T21:14:22.462 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 shutdown_connections 2024-04-12T21:14:22.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 >> 172.21.15.81:0/42113345 conn(0x7ff07c079c60 msgr2=0x7ff07c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:22.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 shutdown_connections 2024-04-12T21:14:22.463 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:21.824+0000 7ff084200700 1 -- 172.21.15.81:0/42113345 wait complete. 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-04-12T21:14:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-12T21:14:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-12T21:14:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-12T21:14:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-12T21:14:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-12T21:14:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-12T21:14:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-12T21:14:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-12T21:14:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-04-12T21:14:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-04-12T21:14:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-12T21:14:22.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-04-12T21:14:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2024-04-12T21:14:23.370 INFO:teuthology.orchestra.run.smithi081.stdout:309237645344 2024-04-12T21:14:23.370 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-12T21:14:23.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: pgmap v349: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:23.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-12T21:14:23.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-12T21:14:23.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-12T21:14:23.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-12T21:14:23.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-12T21:14:23.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-12T21:14:23.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-12T21:14:23.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-04-12T21:14:23.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-04-12T21:14:23.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-12T21:14:23.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: osdmap e90: 12 total, 12 up, 12 in 2024-04-12T21:14:23.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:23 smithi081 ceph-mon[31453]: osdmap e91: 12 total, 12 up, 12 in 2024-04-12T21:14:23.709 INFO:teuthology.orchestra.run.smithi081.stdout:34359738455 2024-04-12T21:14:23.709 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-12T21:14:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: pgmap v349: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: osdmap e90: 12 total, 12 up, 12 in 2024-04-12T21:14:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:23 smithi153 ceph-mon[36569]: osdmap e91: 12 total, 12 up, 12 in 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: pgmap v349: 129 pgs: 129 active+clean; 638 KiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-04-12T21:14:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2024-04-12T21:14:23.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: osdmap e90: 12 total, 12 up, 12 in 2024-04-12T21:14:23.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:23 smithi104 ceph-mon[36624]: osdmap e91: 12 total, 12 up, 12 in 2024-04-12T21:14:23.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.944+0000 7fa302a00700 1 -- 172.21.15.81:0/4208354972 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc012630 con 0x7fa304103520 2024-04-12T21:14:23.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 -- 172.21.15.81:0/4208354972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa304103520 msgr2=0x7fa30410f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:23.941 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 --2- 172.21.15.81:0/4208354972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa304103520 0x7fa30410f990 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009f90 tx=0x7fa2fc00b330 comp rx=0 tx=0).stop 2024-04-12T21:14:23.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 -- 172.21.15.81:0/4208354972 shutdown_connections 2024-04-12T21:14:23.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 --2- 172.21.15.81:0/4208354972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa304103520 0x7fa30410f990 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:23.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 --2- 172.21.15.81:0/4208354972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa304107a40 0x7fa304107e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:23.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 --2- 172.21.15.81:0/4208354972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa304102b60 0x7fa304102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:23.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 -- 172.21.15.81:0/4208354972 >> 172.21.15.81:0/4208354972 conn(0x7fa304079c10 msgr2=0x7fa30407a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:23.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 -- 172.21.15.81:0/4208354972 shutdown_connections 2024-04-12T21:14:23.942 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.945+0000 7fa30ac00700 1 -- 172.21.15.81:0/4208354972 wait complete. 2024-04-12T21:14:23.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.946+0000 7fa30ac00700 1 Processor -- start 2024-04-12T21:14:23.943 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.947+0000 7fa30ac00700 1 -- start start 2024-04-12T21:14:23.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.947+0000 7fa30ac00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa304102b60 0x7fa304197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:23.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.947+0000 7fa30ac00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa304103520 0x7fa304197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:23.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.947+0000 7fa30ac00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa304107a40 0x7fa30419b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:23.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.947+0000 7fa30ac00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa304110d20 con 0x7fa304103520 2024-04-12T21:14:23.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.947+0000 7fa30ac00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa304110ba0 con 0x7fa304102b60 2024-04-12T21:14:23.944 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.947+0000 7fa30ac00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa304110ea0 con 0x7fa304107a40 2024-04-12T21:14:23.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.947+0000 7fa303e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa304102b60 0x7fa304197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:23.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa303e00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa304102b60 0x7fa304197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:57656/0 (socket says 172.21.15.81:57656) 2024-04-12T21:14:23.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa303e00700 1 -- 172.21.15.81:0/3091184345 learned_addr learned my addr 172.21.15.81:0/3091184345 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:23.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa303400700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa304103520 0x7fa304197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:23.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa308c00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa304107a40 0x7fa30419b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:23.945 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa303e00700 1 -- 172.21.15.81:0/3091184345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa304107a40 msgr2=0x7fa30419b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:23.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa303e00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa304107a40 0x7fa30419b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:23.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa303e00700 1 -- 172.21.15.81:0/3091184345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa304103520 msgr2=0x7fa304197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:23.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa303e00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa304103520 0x7fa304197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:23.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.948+0000 7fa303e00700 1 -- 172.21.15.81:0/3091184345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30419c040 con 0x7fa304102b60 2024-04-12T21:14:23.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.949+0000 7fa303e00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa304102b60 0x7fa304197030 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400b590 tx=0x7fa2f400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:23.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.949+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4004020 con 0x7fa304102b60 2024-04-12T21:14:23.946 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.949+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f4003680 con 0x7fa304102b60 2024-04-12T21:14:23.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.949+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30419c330 con 0x7fa304102b60 2024-04-12T21:14:23.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.949+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f400c740 con 0x7fa304102b60 2024-04-12T21:14:23.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.949+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3041a3bb0 con 0x7fa304102b60 2024-04-12T21:14:23.947 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.950+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa3040631c0 con 0x7fa304102b60 2024-04-12T21:14:23.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.953+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa2f40041c0 con 0x7fa304102b60 2024-04-12T21:14:23.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.955+0000 7fa300c00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa2e80a3d50 0x7fa2e80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:23.952 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.955+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fa2f40d94b0 con 0x7fa304102b60 2024-04-12T21:14:23.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.956+0000 7fa300c00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] conn(0x7fa2e80aeeb0 0x7fa2e80b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:23.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.956+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 --> [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2f402fea0 con 0x7fa2e80aeeb0 2024-04-12T21:14:23.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.956+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fa2f40d9860 con 0x7fa304102b60 2024-04-12T21:14:23.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.956+0000 7fa308c00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] conn(0x7fa2e80aeeb0 0x7fa2e80b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:23.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.956+0000 7fa303400700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa2e80a3d50 0x7fa2e80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:23.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.957+0000 7fa308c00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] conn(0x7fa2e80aeeb0 0x7fa2e80b1350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:23.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.957+0000 7fa303400700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa2e80a3d50 0x7fa2e80a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa30407a180 tx=0x7fa2f000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:23.954 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.958+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 <== osd.8 v2:172.21.15.153:6816/2565156870 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa2f402fea0 con 0x7fa2e80aeeb0 2024-04-12T21:14:23.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.977+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 --> [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa3040635f0 con 0x7fa2e80aeeb0 2024-04-12T21:14:23.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.977+0000 7fa300c00700 1 -- 172.21.15.81:0/3091184345 <== osd.8 v2:172.21.15.153:6816/2565156870 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa3040635f0 con 0x7fa2e80aeeb0 2024-04-12T21:14:23.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 >> [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] conn(0x7fa2e80aeeb0 msgr2=0x7fa2e80b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:23.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa30ac00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] conn(0x7fa2e80aeeb0 0x7fa2e80b1350 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:23.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa2e80a3d50 msgr2=0x7fa2e80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:23.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa30ac00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa2e80a3d50 0x7fa2e80a6210 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa30407a180 tx=0x7fa2f000b410 comp rx=0 tx=0).stop 2024-04-12T21:14:23.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa304102b60 msgr2=0x7fa304197030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:23.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa30ac00700 1 --2- 172.21.15.81:0/3091184345 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa304102b60 0x7fa304197030 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400b590 tx=0x7fa2f400b8a0 comp rx=0 tx=0).stop 2024-04-12T21:14:23.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa308c00700 1 -- 172.21.15.81:0/3091184345 reap_dead start 2024-04-12T21:14:23.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 shutdown_connections 2024-04-12T21:14:23.977 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.980+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 >> 172.21.15.81:0/3091184345 conn(0x7fa304079c10 msgr2=0x7fa304100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:23.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.981+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 shutdown_connections 2024-04-12T21:14:23.978 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:23.981+0000 7fa30ac00700 1 -- 172.21.15.81:0/3091184345 wait complete. 2024-04-12T21:14:24.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:24 smithi081 ceph-mon[31453]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-12T21:14:24.689 INFO:teuthology.orchestra.run.smithi081.stdout:219043332145 2024-04-12T21:14:24.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-12T21:14:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:24 smithi153 ceph-mon[36569]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-12T21:14:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:24 smithi104 ceph-mon[36624]: pgmap v352: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-12T21:14:25.722 INFO:teuthology.orchestra.run.smithi081.stdout:77309411407 2024-04-12T21:14:25.722 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-12T21:14:25.771 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:25 smithi081 ceph-mon[31453]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.739+0000 7f421c000700 1 -- 172.21.15.81:0/2258356120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 msgr2=0x7f4214105330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.739+0000 7f421c000700 1 --2- 172.21.15.81:0/2258356120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 0x7f4214105330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f42080099c0 tx=0x7f4208009cd0 comp rx=0 tx=0).stop 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.740+0000 7f421c000700 1 -- 172.21.15.81:0/2258356120 shutdown_connections 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.740+0000 7f421c000700 1 --2- 172.21.15.81:0/2258356120 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421410a030 0x7f4214111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.740+0000 7f421c000700 1 --2- 172.21.15.81:0/2258356120 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 0x7f4214105330 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.740+0000 7f421c000700 1 --2- 172.21.15.81:0/2258356120 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4214105900 0x7f4214109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.740+0000 7f421c000700 1 -- 172.21.15.81:0/2258356120 >> 172.21.15.81:0/2258356120 conn(0x7f42141007e0 msgr2=0x7f4214102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.740+0000 7f421c000700 1 -- 172.21.15.81:0/2258356120 shutdown_connections 2024-04-12T21:14:25.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.740+0000 7f421c000700 1 -- 172.21.15.81:0/2258356120 wait complete. 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.741+0000 7f421c000700 1 Processor -- start 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.741+0000 7f421c000700 1 -- start start 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.741+0000 7f421c000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 0x7f421419b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.741+0000 7f421c000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4214105900 0x7f421419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.741+0000 7f421c000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421410a030 0x7f421419fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.741+0000 7f421c000700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4214115080 con 0x7f4214104f50 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.741+0000 7f421c000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4214114f00 con 0x7f421410a030 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.741+0000 7f421c000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4214115200 con 0x7f4214105900 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4219600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 0x7f421419b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4219600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 0x7f421419b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35814/0 (socket says 172.21.15.81:35814) 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4219600700 1 -- 172.21.15.81:0/2994639897 learned_addr learned my addr 172.21.15.81:0/2994639897 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4218c00700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4214105900 0x7f421419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f421a000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421410a030 0x7f421419fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:25.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4219600700 1 -- 172.21.15.81:0/2994639897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4214105900 msgr2=0x7f421419ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4219600700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4214105900 0x7f421419ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4219600700 1 -- 172.21.15.81:0/2994639897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421410a030 msgr2=0x7f421419fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4219600700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421410a030 0x7f421419fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4219600700 1 -- 172.21.15.81:0/2994639897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42141a0500 con 0x7f4214104f50 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f4218c00700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4214105900 0x7f421419ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.742+0000 7f421a000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f421410a030 0x7f421419fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.743+0000 7f4219600700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 0x7f421419b4f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4208006010 tx=0x7f4208003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.743+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42080050c0 con 0x7f4214104f50 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.743+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f420800f460 con 0x7f4214104f50 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.743+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42141a0790 con 0x7f4214104f50 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.743+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4208017650 con 0x7f4214104f50 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.743+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42141a8070 con 0x7f4214104f50 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.744+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4214063240 con 0x7f4214104f50 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.746+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f420800f610 con 0x7f4214104f50 2024-04-12T21:14:25.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.748+0000 7f4212000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3cd0 0x7f41fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.748+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f42080d18d0 con 0x7f4214104f50 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.748+0000 7f4218c00700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3cd0 0x7f41fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.748+0000 7f4212000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] conn(0x7f41fc0aee30 0x7f41fc0b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.748+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 --> [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4208001d30 con 0x7f41fc0aee30 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.748+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f42080d1c80 con 0x7f4214104f50 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.748+0000 7f421a000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] conn(0x7f41fc0aee30 0x7f41fc0b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.749+0000 7f4218c00700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3cd0 0x7f41fc0a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f421410b640 tx=0x7f4200009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.749+0000 7f421a000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] conn(0x7f41fc0aee30 0x7f41fc0b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.750+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 <== osd.9 v2:172.21.15.81:6826/1653329023 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4208001d30 con 0x7f41fc0aee30 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.768+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 --> [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f42140636c0 con 0x7f41fc0aee30 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.768+0000 7f4212000700 1 -- 172.21.15.81:0/2994639897 <== osd.9 v2:172.21.15.81:6826/1653329023 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f42140636c0 con 0x7f41fc0aee30 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] conn(0x7f41fc0aee30 msgr2=0x7f41fc0b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] conn(0x7f41fc0aee30 0x7f41fc0b12d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:25.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3cd0 msgr2=0x7f41fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:25.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3cd0 0x7f41fc0a6190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f421410b640 tx=0x7f4200009260 comp rx=0 tx=0).stop 2024-04-12T21:14:25.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 msgr2=0x7f421419b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:25.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 --2- 172.21.15.81:0/2994639897 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214104f50 0x7f421419b4f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4208006010 tx=0x7f4208003a90 comp rx=0 tx=0).stop 2024-04-12T21:14:25.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421a000700 1 -- 172.21.15.81:0/2994639897 reap_dead start 2024-04-12T21:14:25.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 shutdown_connections 2024-04-12T21:14:25.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 >> 172.21.15.81:0/2994639897 conn(0x7f42141007e0 msgr2=0x7f4214102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:25.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 shutdown_connections 2024-04-12T21:14:25.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:25.769+0000 7f421c000700 1 -- 172.21.15.81:0/2994639897 wait complete. 2024-04-12T21:14:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:25 smithi153 ceph-mon[36569]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T21:14:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:25 smithi104 ceph-mon[36624]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-12T21:14:26.682 INFO:teuthology.orchestra.run.smithi081.stdout:287762808871 2024-04-12T21:14:26.683 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-12T21:14:26.728 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:26 smithi081 ceph-mon[31453]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:26 smithi153 ceph-mon[36569]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:26.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:26 smithi104 ceph-mon[36624]: pgmap v353: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:27.489 INFO:teuthology.orchestra.run.smithi081.stdout:163208757310 2024-04-12T21:14:27.489 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-12T21:14:27.930 INFO:teuthology.orchestra.run.smithi081.stdout:253403070510 2024-04-12T21:14:27.931 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-12T21:14:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 -- 172.21.15.81:0/4187322058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc1058e0 msgr2=0x7f80bc109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 --2- 172.21.15.81:0/4187322058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc1058e0 0x7f80bc109930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009920 tx=0x7f80a8009c30 comp rx=0 tx=0).stop 2024-04-12T21:14:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 -- 172.21.15.81:0/4187322058 shutdown_connections 2024-04-12T21:14:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 --2- 172.21.15.81:0/4187322058 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80bc10a010 0x7f80bc111be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 --2- 172.21.15.81:0/4187322058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80bc104f30 0x7f80bc105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 --2- 172.21.15.81:0/4187322058 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc1058e0 0x7f80bc109930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:28.004 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 -- 172.21.15.81:0/4187322058 >> 172.21.15.81:0/4187322058 conn(0x7f80bc1007c0 msgr2=0x7f80bc102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 -- 172.21.15.81:0/4187322058 shutdown_connections 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.241+0000 7f80c2600700 1 -- 172.21.15.81:0/4187322058 wait complete. 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.242+0000 7f80c2600700 1 Processor -- start 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.243+0000 7f80c2600700 1 -- start start 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.243+0000 7f80c2600700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc104f30 0x7f80bc19b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.243+0000 7f80c2600700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80bc1058e0 0x7f80bc19bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.243+0000 7f80c2600700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80bc10a010 0x7f80bc19fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.243+0000 7f80c2600700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc115040 con 0x7f80bc104f30 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.243+0000 7f80c2600700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc114ec0 con 0x7f80bc10a010 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.243+0000 7f80c2600700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc1151c0 con 0x7f80bc1058e0 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc104f30 0x7f80bc19b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc104f30 0x7f80bc19b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35818/0 (socket says 172.21.15.81:35818) 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 -- 172.21.15.81:0/1459804532 learned_addr learned my addr 172.21.15.81:0/1459804532 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:28.005 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80baa00700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80bc1058e0 0x7f80bc19bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bbe00700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80bc10a010 0x7f80bc19fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 -- 172.21.15.81:0/1459804532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80bc1058e0 msgr2=0x7f80bc19bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80bc1058e0 0x7f80bc19bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 -- 172.21.15.81:0/1459804532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80bc10a010 msgr2=0x7f80bc19fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f80bc10a010 0x7f80bc19fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 -- 172.21.15.81:0/1459804532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80bc1a0530 con 0x7f80bc104f30 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.244+0000 7f80bb400700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc104f30 0x7f80bc19b570 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00e9c0 tx=0x7f80ac00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.245+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac004a30 con 0x7f80bc104f30 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.245+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80ac00cca0 con 0x7f80bc104f30 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.245+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80bc1a0820 con 0x7f80bc104f30 2024-04-12T21:14:28.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.245+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac005610 con 0x7f80bc104f30 2024-04-12T21:14:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.245+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80bc1a80a0 con 0x7f80bc104f30 2024-04-12T21:14:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.246+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f80bc063240 con 0x7f80bc104f30 2024-04-12T21:14:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.248+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f80ac004cf0 con 0x7f80bc104f30 2024-04-12T21:14:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.249+0000 7f80b3e00700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80a00a3d50 0x7f80a00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.250+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f80ac0d0d60 con 0x7f80bc104f30 2024-04-12T21:14:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.250+0000 7f80baa00700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80a00a3d50 0x7f80a00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.250+0000 7f80b3e00700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] conn(0x7f80a00aeeb0 0x7f80a00b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:28.007 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.250+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 --> [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80ac00ce50 con 0x7f80a00aeeb0 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.250+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f80ac0d11b0 con 0x7f80bc104f30 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.250+0000 7f80bbe00700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] conn(0x7f80a00aeeb0 0x7f80a00b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.250+0000 7f80baa00700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80a00a3d50 0x7f80a00a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f80a800c010 tx=0x7f80a8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.251+0000 7f80bbe00700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] conn(0x7f80a00aeeb0 0x7f80a00b1350 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.252+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 <== osd.11 v2:172.21.15.153:6824/2242325561 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f80ac00ce50 con 0x7f80a00aeeb0 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.269+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 --> [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f80bc063640 con 0x7f80a00aeeb0 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.270+0000 7f80b3e00700 1 -- 172.21.15.81:0/1459804532 <== osd.11 v2:172.21.15.153:6824/2242325561 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f80bc063640 con 0x7f80a00aeeb0 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 >> [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] conn(0x7f80a00aeeb0 msgr2=0x7f80a00b1350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80c2600700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] conn(0x7f80a00aeeb0 0x7f80a00b1350 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80a00a3d50 msgr2=0x7f80a00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80c2600700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80a00a3d50 0x7f80a00a6210 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f80a800c010 tx=0x7f80a8019040 comp rx=0 tx=0).stop 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc104f30 msgr2=0x7f80bc19b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:28.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80c2600700 1 --2- 172.21.15.81:0/1459804532 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f80bc104f30 0x7f80bc19b570 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00e9c0 tx=0x7f80ac00ecd0 comp rx=0 tx=0).stop 2024-04-12T21:14:28.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80bbe00700 1 -- 172.21.15.81:0/1459804532 reap_dead start 2024-04-12T21:14:28.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 shutdown_connections 2024-04-12T21:14:28.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.271+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 >> 172.21.15.81:0/1459804532 conn(0x7f80bc1007c0 msgr2=0x7f80bc101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:28.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.272+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 shutdown_connections 2024-04-12T21:14:28.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:27.272+0000 7f80c2600700 1 -- 172.21.15.81:0/1459804532 wait complete. 2024-04-12T21:14:28.939 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:28.940 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:28.940 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:28.998 INFO:teuthology.orchestra.run.smithi081.stdout:197568495670 2024-04-12T21:14:28.999 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-12T21:14:29.396 INFO:teuthology.orchestra.run.smithi081.stdout:343597383710 2024-04-12T21:14:29.397 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-12T21:14:29.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:29 smithi081 ceph-mon[31453]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:29.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:29 smithi153 ceph-mon[36569]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:29.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:29 smithi104 ceph-mon[36624]: pgmap v354: 129 pgs: 1 peering, 128 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:30.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:30 smithi081 ceph-mon[31453]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T21:14:30.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:30 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:14:30.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:30 smithi153 ceph-mon[36569]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T21:14:30.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:30 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:14:30.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:30 smithi104 ceph-mon[36624]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-12T21:14:30.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:30 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:14:31.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:31 smithi081 ceph-mon[31453]: pgmap v355: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 0 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:31 smithi153 ceph-mon[36569]: pgmap v355: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 0 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:31.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:31 smithi104 ceph-mon[36624]: pgmap v355: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 0 op/s; 18 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:33.603 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:33.603 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:33.617 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:33.625 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:33.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:33 smithi081 ceph-mon[31453]: pgmap v356: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 830 B/s rd, 1 op/s; 15 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:33 smithi153 ceph-mon[36569]: pgmap v356: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 830 B/s rd, 1 op/s; 15 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:33 smithi104 ceph-mon[36624]: pgmap v356: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 830 B/s rd, 1 op/s; 15 B/s, 1 keys/s, 0 objects/s recovering 2024-04-12T21:14:35.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 -- 172.21.15.81:0/1693189955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c103520 msgr2=0x7f806c10f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:35.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 --2- 172.21.15.81:0/1693189955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c103520 0x7f806c10f990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8064009fc0 tx=0x7f806400b330 comp rx=0 tx=0).stop 2024-04-12T21:14:35.283 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 -- 172.21.15.81:0/1693189955 shutdown_connections 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 --2- 172.21.15.81:0/1693189955 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c103520 0x7f806c10f990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 --2- 172.21.15.81:0/1693189955 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f806c107a40 0x7f806c107e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 --2- 172.21.15.81:0/1693189955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f806c102b60 0x7f806c102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 -- 172.21.15.81:0/1693189955 >> 172.21.15.81:0/1693189955 conn(0x7f806c079c10 msgr2=0x7f806c07a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 -- 172.21.15.81:0/1693189955 shutdown_connections 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.708+0000 7f8071c00700 1 -- 172.21.15.81:0/1693189955 wait complete. 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.709+0000 7f8071c00700 1 Processor -- start 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.709+0000 7f8071c00700 1 -- start start 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f8071c00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c102b60 0x7f806c074db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f8071c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f806c103520 0x7f806c0752f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f8071c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f806c107a40 0x7f806c075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f8071c00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112e70 con 0x7f806c102b60 2024-04-12T21:14:35.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f8071c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112cf0 con 0x7f806c107a40 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f8071c00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f806c112ff0 con 0x7f806c103520 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f806aa00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c102b60 0x7f806c074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f806b400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f806c107a40 0x7f806c075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f806aa00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c102b60 0x7f806c074db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:35832/0 (socket says 172.21.15.81:35832) 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f806aa00700 1 -- 172.21.15.81:0/420759814 learned_addr learned my addr 172.21.15.81:0/420759814 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.710+0000 7f8063e00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f806c103520 0x7f806c0752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f806aa00700 1 -- 172.21.15.81:0/420759814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f806c103520 msgr2=0x7f806c0752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f806aa00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f806c103520 0x7f806c0752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f806aa00700 1 -- 172.21.15.81:0/420759814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f806c107a40 msgr2=0x7f806c075830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f806aa00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f806c107a40 0x7f806c075830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f806aa00700 1 -- 172.21.15.81:0/420759814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c1a7740 con 0x7f806c102b60 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f806aa00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c102b60 0x7f806c074db0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f805c00ec10 tx=0x7f805c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f8063400700 1 -- 172.21.15.81:0/420759814 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c00cc90 con 0x7f806c102b60 2024-04-12T21:14:35.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f8063400700 1 -- 172.21.15.81:0/420759814 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f805c00ce30 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c1a7a30 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f8063400700 1 -- 172.21.15.81:0/420759814 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805c010690 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.711+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c1a7fc0 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.712+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f806c0697a0 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.715+0000 7f8063400700 1 -- 172.21.15.81:0/420759814 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f805c003680 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.716+0000 7f8063400700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80500a3d50 0x7f80500a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.716+0000 7f8063400700 1 -- 172.21.15.81:0/420759814 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f805c0d0c90 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.716+0000 7f8063e00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80500a3d50 0x7f80500a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.717+0000 7f8063e00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80500a3d50 0x7f80500a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f806c07a1a0 tx=0x7f8054009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.719+0000 7f8063400700 1 -- 172.21.15.81:0/420759814 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f805c09d1e0 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.962+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f806c02dfa0 con 0x7f806c102b60 2024-04-12T21:14:35.286 INFO:teuthology.orchestra.run.smithi081.stdout:141733920838 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.963+0000 7f8063400700 1 -- 172.21.15.81:0/420759814 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f805c09cae0 con 0x7f806c102b60 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80500a3d50 msgr2=0x7f80500a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80500a3d50 0x7f80500a6210 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f806c07a1a0 tx=0x7f8054009260 comp rx=0 tx=0).stop 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c102b60 msgr2=0x7f806c074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c102b60 0x7f806c074db0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f805c00ec10 tx=0x7f805c00ef20 comp rx=0 tx=0).stop 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 shutdown_connections 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f80500a3d50 0x7f80500a6210 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f806c107a40 0x7f806c075830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f806c102b60 0x7f806c074db0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 --2- 172.21.15.81:0/420759814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f806c103520 0x7f806c0752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 >> 172.21.15.81:0/420759814 conn(0x7f806c079c10 msgr2=0x7f806c100b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:35.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.966+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 shutdown_connections 2024-04-12T21:14:35.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:34.967+0000 7f8071c00700 1 -- 172.21.15.81:0/420759814 wait complete. 2024-04-12T21:14:35.778 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:35 smithi081 ceph-mon[31453]: pgmap v357: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 753 B/s rd, 1 op/s; 13 B/s, 0 keys/s, 0 objects/s recovering 2024-04-12T21:14:35.778 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:35 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/420759814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T21:14:35.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:35 smithi153 ceph-mon[36569]: pgmap v357: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 753 B/s rd, 1 op/s; 13 B/s, 0 keys/s, 0 objects/s recovering 2024-04-12T21:14:35.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:35 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/420759814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T21:14:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:35 smithi104 ceph-mon[36624]: pgmap v357: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 753 B/s rd, 1 op/s; 13 B/s, 0 keys/s, 0 objects/s recovering 2024-04-12T21:14:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:35 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/420759814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-12T21:14:36.939 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:36.940 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:36.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.963+0000 7f73f3200700 1 -- 172.21.15.81:0/310824203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 msgr2=0x7f73ec0750c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:36.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.963+0000 7f73f3200700 1 --2- 172.21.15.81:0/310824203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 0x7f73ec0750c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009990 tx=0x7f73dc009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:36.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.963+0000 7f73f3200700 1 -- 172.21.15.81:0/310824203 shutdown_connections 2024-04-12T21:14:36.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.963+0000 7f73f3200700 1 --2- 172.21.15.81:0/310824203 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ec113c80 0x7f73ec116070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:36.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.963+0000 7f73f3200700 1 --2- 172.21.15.81:0/310824203 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 0x7f73ec0750c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:36.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.963+0000 7f73f3200700 1 --2- 172.21.15.81:0/310824203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec075600 0x7f73ec113740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:36.959 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.963+0000 7f73f3200700 1 -- 172.21.15.81:0/310824203 >> 172.21.15.81:0/310824203 conn(0x7f73ec079c10 msgr2=0x7f73ec07a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:36.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.964+0000 7f73f3200700 1 -- 172.21.15.81:0/310824203 shutdown_connections 2024-04-12T21:14:36.960 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.964+0000 7f73f3200700 1 -- 172.21.15.81:0/310824203 wait complete. 2024-04-12T21:14:36.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.965+0000 7f73f3200700 1 Processor -- start 2024-04-12T21:14:36.961 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.966+0000 7f73f3200700 1 -- start start 2024-04-12T21:14:36.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.966+0000 7f73f3200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 0x7f73ec1a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:36.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.966+0000 7f73f3200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ec075600 0x7f73ec1a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:36.962 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.966+0000 7f73f3200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec113c80 0x7f73ec19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:36.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.966+0000 7f73f3200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec119400 con 0x7f73ec075600 2024-04-12T21:14:36.963 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.966+0000 7f73f3200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec119280 con 0x7f73ec074ce0 2024-04-12T21:14:36.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73f3200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f73ec119580 con 0x7f73ec113c80 2024-04-12T21:14:36.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73ebe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 0x7f73ec1a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:36.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73ebe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 0x7f73ec1a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:40666/0 (socket says 172.21.15.81:40666) 2024-04-12T21:14:36.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73ebe00700 1 -- 172.21.15.81:0/2739765658 learned_addr learned my addr 172.21.15.81:0/2739765658 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:36.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73f1200700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec113c80 0x7f73ec19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:36.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73eb400700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ec075600 0x7f73ec1a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:36.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73ebe00700 1 -- 172.21.15.81:0/2739765658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec113c80 msgr2=0x7f73ec19f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:36.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73ebe00700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec113c80 0x7f73ec19f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:36.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73ebe00700 1 -- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ec075600 msgr2=0x7f73ec1a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:36.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73ebe00700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ec075600 0x7f73ec1a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73ebe00700 1 -- 172.21.15.81:0/2739765658 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73ec19ffe0 con 0x7f73ec074ce0 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.967+0000 7f73eb400700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ec075600 0x7f73ec1a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.968+0000 7f73ebe00700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 0x7f73ec1a5630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73dc000c00 tx=0x7f73dc00bb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.968+0000 7f73e8c00700 1 -- 172.21.15.81:0/2739765658 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc027070 con 0x7f73ec074ce0 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.968+0000 7f73e8c00700 1 -- 172.21.15.81:0/2739765658 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73dc003d70 con 0x7f73ec074ce0 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.968+0000 7f73e8c00700 1 -- 172.21.15.81:0/2739765658 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc00f460 con 0x7f73ec074ce0 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.969+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73ec1a0270 con 0x7f73ec074ce0 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.969+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73ec1ac3e0 con 0x7f73ec074ce0 2024-04-12T21:14:36.967 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.971+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73ec0763d0 con 0x7f73ec074ce0 2024-04-12T21:14:36.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.973+0000 7f73e8c00700 1 -- 172.21.15.81:0/2739765658 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f73dc009d10 con 0x7f73ec074ce0 2024-04-12T21:14:36.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.974+0000 7f73e8c00700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f73d00a3cd0 0x7f73d00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:36.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.974+0000 7f73e8c00700 1 -- 172.21.15.81:0/2739765658 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f73dc0d1ce0 con 0x7f73ec074ce0 2024-04-12T21:14:36.970 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.975+0000 7f73eb400700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f73d00a3cd0 0x7f73d00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:36.971 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.975+0000 7f73eb400700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f73d00a3cd0 0x7f73d00a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f73ec1a0ec0 tx=0x7f73d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:36.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.979+0000 7f73e8c00700 1 -- 172.21.15.81:0/2739765658 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f73dc013070 con 0x7f73ec074ce0 2024-04-12T21:14:37.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.867+0000 7f633f200700 1 -- 172.21.15.81:0/319549537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 msgr2=0x7f6338102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:37.047 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.867+0000 7f633f200700 1 --2- 172.21.15.81:0/319549537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 0x7f6338102f80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6324009990 tx=0x7f6324009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.867+0000 7f633f200700 1 -- 172.21.15.81:0/319549537 shutdown_connections 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.867+0000 7f633f200700 1 --2- 172.21.15.81:0/319549537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63381034c0 0x7f633810f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.867+0000 7f633f200700 1 --2- 172.21.15.81:0/319549537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6338108ac0 0x7f6338108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.867+0000 7f633f200700 1 --2- 172.21.15.81:0/319549537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 0x7f6338102f80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.867+0000 7f633f200700 1 -- 172.21.15.81:0/319549537 >> 172.21.15.81:0/319549537 conn(0x7f6338079c10 msgr2=0x7f633807a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.868+0000 7f633f200700 1 -- 172.21.15.81:0/319549537 shutdown_connections 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.868+0000 7f633f200700 1 -- 172.21.15.81:0/319549537 wait complete. 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.869+0000 7f633f200700 1 Processor -- start 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.869+0000 7f633f200700 1 -- start start 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.869+0000 7f633f200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 0x7f6338074db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.869+0000 7f633f200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63381034c0 0x7f63380752f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.869+0000 7f633f200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6338108ac0 0x7f6338075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.869+0000 7f633f200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f6338112df0 con 0x7f6338102b00 2024-04-12T21:14:37.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.869+0000 7f633f200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f6338112c70 con 0x7f63381034c0 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.869+0000 7f633f200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6338112f70 con 0x7f6338108ac0 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 0x7f6338074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 0x7f6338074db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50318/0 (socket says 172.21.15.81:50318) 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 -- 172.21.15.81:0/2852989491 learned_addr learned my addr 172.21.15.81:0/2852989491 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f633d200700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6338108ac0 0x7f6338075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337400700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63381034c0 0x7f63380752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 -- 172.21.15.81:0/2852989491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6338108ac0 msgr2=0x7f6338075830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6338108ac0 0x7f6338075830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 -- 172.21.15.81:0/2852989491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63381034c0 msgr2=0x7f63380752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63381034c0 0x7f63380752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 -- 172.21.15.81:0/2852989491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63381a78f0 con 0x7f6338102b00 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337400700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63381034c0 0x7f63380752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:14:37.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f633d200700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6338108ac0 0x7f6338075830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.870+0000 7f6337e00700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 0x7f6338074db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f632800eb30 tx=0x7f632800eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.871+0000 7f6334c00700 1 -- 172.21.15.81:0/2852989491 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632800cc80 con 0x7f6338102b00 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.871+0000 7f6334c00700 1 -- 172.21.15.81:0/2852989491 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6328004500 con 0x7f6338102b00 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.871+0000 7f6334c00700 1 -- 172.21.15.81:0/2852989491 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328010430 con 0x7f6338102b00 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.871+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63381a7be0 con 0x7f6338102b00 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.871+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63381a8120 con 0x7f6338102b00 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.872+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63380697a0 con 0x7f6338102b00 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.876+0000 7f6334c00700 1 -- 172.21.15.81:0/2852989491 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f6328003f50 con 0x7f6338102b00 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.877+0000 7f6334c00700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f631c0a3cd0 0x7f631c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.877+0000 7f6334c00700 1 -- 172.21.15.81:0/2852989491 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f63280d0b90 con 0x7f6338102b00 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.878+0000 7f6337400700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f631c0a3cd0 0x7f631c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.878+0000 7f6337400700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f631c0a3cd0 0x7f631c0a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6324006010 tx=0x7f632401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:37.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:36.882+0000 7f6334c00700 1 -- 172.21.15.81:0/2852989491 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f632809d050 con 0x7f6338102b00 2024-04-12T21:14:37.148 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.152+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f633802dfa0 con 0x7f6338102b00 2024-04-12T21:14:37.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.153+0000 7f6334c00700 1 -- 172.21.15.81:0/2852989491 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f632809c950 con 0x7f6338102b00 2024-04-12T21:14:37.149 INFO:teuthology.orchestra.run.smithi081.stdout:107374182475 2024-04-12T21:14:37.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f631c0a3cd0 msgr2=0x7f631c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:37.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f631c0a3cd0 0x7f631c0a6190 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6324006010 tx=0x7f632401a040 comp rx=0 tx=0).stop 2024-04-12T21:14:37.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 msgr2=0x7f6338074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:37.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 0x7f6338074db0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f632800eb30 tx=0x7f632800eef0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 shutdown_connections 2024-04-12T21:14:37.152 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f631c0a3cd0 0x7f631c0a6190 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6338108ac0 0x7f6338075830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f6338102b00 0x7f6338074db0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 --2- 172.21.15.81:0/2852989491 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f63381034c0 0x7f63380752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.156+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 >> 172.21.15.81:0/2852989491 conn(0x7f6338079c10 msgr2=0x7f63381018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:37.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.157+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 shutdown_connections 2024-04-12T21:14:37.153 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.157+0000 7f633f200700 1 -- 172.21.15.81:0/2852989491 wait complete. 2024-04-12T21:14:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.233+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f73ec1ac5b0 con 0x7f73ec074ce0 2024-04-12T21:14:37.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.234+0000 7f73e8c00700 1 -- 172.21.15.81:0/2739765658 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f73dc09e230 con 0x7f73ec074ce0 2024-04-12T21:14:37.229 INFO:teuthology.orchestra.run.smithi081.stdout:55834574935 2024-04-12T21:14:37.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f73d00a3cd0 msgr2=0x7f73d00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:37.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f73d00a3cd0 0x7f73d00a6190 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f73ec1a0ec0 tx=0x7f73d8009260 comp rx=0 tx=0).stop 2024-04-12T21:14:37.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 msgr2=0x7f73ec1a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:37.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 0x7f73ec1a5630 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73dc000c00 tx=0x7f73dc00bb40 comp rx=0 tx=0).stop 2024-04-12T21:14:37.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 shutdown_connections 2024-04-12T21:14:37.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f73d00a3cd0 0x7f73d00a6190 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73ec113c80 0x7f73ec19f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f73ec074ce0 0x7f73ec1a5630 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.236+0000 7f73f3200700 1 --2- 172.21.15.81:0/2739765658 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f73ec075600 0x7f73ec1a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:37.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.237+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 >> 172.21.15.81:0/2739765658 conn(0x7f73ec079c10 msgr2=0x7f73ec112ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:37.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.237+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 shutdown_connections 2024-04-12T21:14:37.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:37.237+0000 7f73f3200700 1 -- 172.21.15.81:0/2739765658 wait complete. 2024-04-12T21:14:37.738 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:37 smithi081 ceph-mon[31453]: pgmap v358: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2024-04-12T21:14:37.738 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:37 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2852989491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T21:14:37.738 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:37 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2739765658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T21:14:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:37 smithi153 ceph-mon[36569]: pgmap v358: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2024-04-12T21:14:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:37 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2852989491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T21:14:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:37 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2739765658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T21:14:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:37 smithi104 ceph-mon[36624]: pgmap v358: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s; 12 B/s, 0 keys/s, 0 objects/s recovering 2024-04-12T21:14:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:37 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2852989491' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-12T21:14:37.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:37 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2739765658' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-12T21:14:39.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:39 smithi081 ceph-mon[31453]: pgmap v359: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-04-12T21:14:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:39 smithi153 ceph-mon[36569]: pgmap v359: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-04-12T21:14:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:39 smithi104 ceph-mon[36624]: pgmap v359: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s; 9 B/s, 0 objects/s recovering 2024-04-12T21:14:39.916 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182470 got 107374182475 for osd.3 2024-04-12T21:14:39.916 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:41.310 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574929 got 55834574935 for osd.1 2024-04-12T21:14:41.311 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:41.525 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:41 smithi081 ceph-mon[31453]: pgmap v360: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:41 smithi153 ceph-mon[36569]: pgmap v360: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:41 smithi104 ceph-mon[36624]: pgmap v360: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:43.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29baa00700 1 -- 172.21.15.81:0/2017756697 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a800bc40 con 0x7f29bc102b00 2024-04-12T21:14:43.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 -- 172.21.15.81:0/2017756697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 msgr2=0x7f29bc102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:43.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 --2- 172.21.15.81:0/2017756697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 0x7f29bc102f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f29a8009990 tx=0x7f29a8009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 -- 172.21.15.81:0/2017756697 shutdown_connections 2024-04-12T21:14:43.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 --2- 172.21.15.81:0/2017756697 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29bc1034c0 0x7f29bc10f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 --2- 172.21.15.81:0/2017756697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29bc108b80 0x7f29bc108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 --2- 172.21.15.81:0/2017756697 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 0x7f29bc102f80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 -- 172.21.15.81:0/2017756697 >> 172.21.15.81:0/2017756697 conn(0x7f29bc079c60 msgr2=0x7f29bc07a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 -- 172.21.15.81:0/2017756697 shutdown_connections 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.670+0000 7f29c2a00700 1 -- 172.21.15.81:0/2017756697 wait complete. 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.671+0000 7f29c2a00700 1 Processor -- start 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29c2a00700 1 -- start start 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29c2a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 0x7f29bc197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29c2a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29bc1034c0 0x7f29bc197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29c2a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29bc108b80 0x7f29bc19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29c2a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc110d20 con 0x7f29bc102b00 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29c2a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc110ba0 con 0x7f29bc108b80 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29c2a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f29bc110ea0 con 0x7f29bc1034c0 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29bbe00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 0x7f29bc197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29bbe00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 0x7f29bc197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50342/0 (socket says 172.21.15.81:50342) 2024-04-12T21:14:43.040 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.672+0000 7f29bbe00700 1 -- 172.21.15.81:0/4039535004 learned_addr learned my addr 172.21.15.81:0/4039535004 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29c0a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29bc108b80 0x7f29bc19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29bb400700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29bc1034c0 0x7f29bc197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29c0a00700 1 -- 172.21.15.81:0/4039535004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29bc1034c0 msgr2=0x7f29bc197570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29c0a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29bc1034c0 0x7f29bc197570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29c0a00700 1 -- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 msgr2=0x7f29bc197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29c0a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 0x7f29bc197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29c0a00700 1 -- 172.21.15.81:0/4039535004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29bc19c040 con 0x7f29bc108b80 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29bb400700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29bc1034c0 0x7f29bc197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.673+0000 7f29c0a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29bc108b80 0x7f29bc19b900 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f29b4012320 tx=0x7f29b4012630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.674+0000 7f29b8c00700 1 -- 172.21.15.81:0/4039535004 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b4004bb0 con 0x7f29bc108b80 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.674+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29bc19c2d0 con 0x7f29bc108b80 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.674+0000 7f29b8c00700 1 -- 172.21.15.81:0/4039535004 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29b4004d50 con 0x7f29bc108b80 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.674+0000 7f29b8c00700 1 -- 172.21.15.81:0/4039535004 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b4004400 con 0x7f29bc108b80 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.674+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29bc1a3bb0 con 0x7f29bc108b80 2024-04-12T21:14:43.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.675+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29bc0697a0 con 0x7f29bc108b80 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.678+0000 7f29b8c00700 1 -- 172.21.15.81:0/4039535004 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f29b401f030 con 0x7f29bc108b80 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.679+0000 7f29b8c00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29a00a3d50 0x7f29a00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.679+0000 7f29b8c00700 1 -- 172.21.15.81:0/4039535004 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f29b40d4350 con 0x7f29bc108b80 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.679+0000 7f29bbe00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29a00a3d50 0x7f29a00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.680+0000 7f29bbe00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29a00a3d50 0x7f29a00a6210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009ce0 tx=0x7f29ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.684+0000 7f29b8c00700 1 -- 172.21.15.81:0/4039535004 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29b4026020 con 0x7f29bc108b80 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.911+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f29bc198390 con 0x7f29bc108b80 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.912+0000 7f29b8c00700 1 -- 172.21.15.81:0/4039535004 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f29b40a08a0 con 0x7f29bc108b80 2024-04-12T21:14:43.042 INFO:teuthology.orchestra.run.smithi081.stdout:77309411411 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29a00a3d50 msgr2=0x7f29a00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29a00a3d50 0x7f29a00a6210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f29ac009ce0 tx=0x7f29ac009260 comp rx=0 tx=0).stop 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29bc108b80 msgr2=0x7f29bc19b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29bc108b80 0x7f29bc19b900 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f29b4012320 tx=0x7f29b4012630 comp rx=0 tx=0).stop 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 shutdown_connections 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f29a00a3d50 0x7f29a00a6210 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f29bc108b80 0x7f29bc19b900 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f29bc102b00 0x7f29bc197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 --2- 172.21.15.81:0/4039535004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29bc1034c0 0x7f29bc197570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 >> 172.21.15.81:0/4039535004 conn(0x7f29bc079c60 msgr2=0x7f29bc1015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 shutdown_connections 2024-04-12T21:14:43.043 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:42.915+0000 7f29c2a00700 1 -- 172.21.15.81:0/4039535004 wait complete. 2024-04-12T21:14:43.297 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:43.298 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:43.299 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:14:43.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:43 smithi081 ceph-mon[31453]: pgmap v361: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:43.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:43 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4039535004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T21:14:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:43 smithi153 ceph-mon[36569]: pgmap v361: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:43 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4039535004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T21:14:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:43 smithi104 ceph-mon[36624]: pgmap v361: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:43 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4039535004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-12T21:14:44.047 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920834 got 141733920838 for osd.4 2024-04-12T21:14:44.047 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:44.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.835+0000 7faadc400700 1 -- 172.21.15.81:0/3913300625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad41058e0 msgr2=0x7faad4109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:44.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.835+0000 7faadc400700 1 --2- 172.21.15.81:0/3913300625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad41058e0 0x7faad4109930 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faac0009990 tx=0x7faac0009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:44.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.835+0000 7faadc400700 1 -- 172.21.15.81:0/3913300625 shutdown_connections 2024-04-12T21:14:44.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.835+0000 7faadc400700 1 --2- 172.21.15.81:0/3913300625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faad410a010 0x7faad4111be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:44.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.835+0000 7faadc400700 1 --2- 172.21.15.81:0/3913300625 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad4104f30 0x7faad4105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:44.913 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.835+0000 7faadc400700 1 --2- 172.21.15.81:0/3913300625 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad41058e0 0x7faad4109930 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.835+0000 7faadc400700 1 -- 172.21.15.81:0/3913300625 >> 172.21.15.81:0/3913300625 conn(0x7faad41007c0 msgr2=0x7faad4102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.836+0000 7faadc400700 1 -- 172.21.15.81:0/3913300625 shutdown_connections 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.836+0000 7faadc400700 1 -- 172.21.15.81:0/3913300625 wait complete. 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.837+0000 7faadc400700 1 Processor -- start 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.837+0000 7faadc400700 1 -- start start 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.837+0000 7faadc400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4104f30 0x7faad419b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.837+0000 7faadc400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad41058e0 0x7faad419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.837+0000 7faadc400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faad410a010 0x7faad419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.837+0000 7faadc400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7faad4114ff0 con 0x7faad41058e0 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.837+0000 7faadc400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7faad4114e70 con 0x7faad4104f30 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.837+0000 7faadc400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7faad4115170 con 0x7faad410a010 2024-04-12T21:14:44.914 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad41058e0 0x7faad419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad41058e0 0x7faad419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:50366/0 (socket says 172.21.15.81:50366) 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 -- 172.21.15.81:0/188848127 learned_addr learned my addr 172.21.15.81:0/188848127 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faada400700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faad410a010 0x7faad419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9a00700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4104f30 0x7faad419b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 -- 172.21.15.81:0/188848127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faad410a010 msgr2=0x7faad419fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faad410a010 0x7faad419fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 -- 172.21.15.81:0/188848127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4104f30 msgr2=0x7faad419b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4104f30 0x7faad419b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 -- 172.21.15.81:0/188848127 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad41a0470 con 0x7faad41058e0 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9a00700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4104f30 0x7faad419b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.838+0000 7faad9000700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad41058e0 0x7faad419b9a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faac0000c00 tx=0x7faac0005610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.839+0000 7faad2000700 1 -- 172.21.15.81:0/188848127 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0026070 con 0x7faad41058e0 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.839+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad41a0700 con 0x7faad41058e0 2024-04-12T21:14:44.915 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.839+0000 7faad2000700 1 -- 172.21.15.81:0/188848127 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faac0021410 con 0x7faad41058e0 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.839+0000 7faad2000700 1 -- 172.21.15.81:0/188848127 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0017650 con 0x7faad41058e0 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.839+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad41a7fe0 con 0x7faad41058e0 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.840+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faad40697a0 con 0x7faad41058e0 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.842+0000 7faad2000700 1 -- 172.21.15.81:0/188848127 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7faac0029070 con 0x7faad41058e0 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.843+0000 7faad2000700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faabc0a3cd0 0x7faabc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.844+0000 7faad2000700 1 -- 172.21.15.81:0/188848127 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7faac00d14c0 con 0x7faad41058e0 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.844+0000 7faad9a00700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faabc0a3cd0 0x7faabc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.845+0000 7faad9a00700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faabc0a3cd0 0x7faabc0a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faac8009cc0 tx=0x7faac8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:44.916 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:44.848+0000 7faad2000700 1 -- 172.21.15.81:0/188848127 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faac0013070 con 0x7faad41058e0 2024-04-12T21:14:45.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.076+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7faad402dfa0 con 0x7faad41058e0 2024-04-12T21:14:45.072 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.076+0000 7faad2000700 1 -- 172.21.15.81:0/188848127 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7faac009da10 con 0x7faad41058e0 2024-04-12T21:14:45.072 INFO:teuthology.orchestra.run.smithi081.stdout:287762808875 2024-04-12T21:14:45.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faabc0a3cd0 msgr2=0x7faabc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:45.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faabc0a3cd0 0x7faabc0a6190 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faac8009cc0 tx=0x7faac8009260 comp rx=0 tx=0).stop 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad41058e0 msgr2=0x7faad419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad41058e0 0x7faad419b9a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faac0000c00 tx=0x7faac0005610 comp rx=0 tx=0).stop 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 shutdown_connections 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7faabc0a3cd0 0x7faabc0a6190 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faad410a010 0x7faad419fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faad4104f30 0x7faad419b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 --2- 172.21.15.81:0/188848127 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7faad41058e0 0x7faad419b9a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.079+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 >> 172.21.15.81:0/188848127 conn(0x7faad41007c0 msgr2=0x7faad41012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.080+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 shutdown_connections 2024-04-12T21:14:45.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:45.080+0000 7faadc400700 1 -- 172.21.15.81:0/188848127 wait complete. 2024-04-12T21:14:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:45 smithi153 ceph-mon[36569]: pgmap v362: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:45 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/188848127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-12T21:14:45.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:45 smithi104 ceph-mon[36624]: pgmap v362: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:45.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:45 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/188848127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-12T21:14:45.862 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:45 smithi081 ceph-mon[31453]: pgmap v362: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:45.862 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:45 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/188848127' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-12T21:14:47.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 -- 172.21.15.81:0/129187075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 msgr2=0x7feab0102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:47.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 --2- 172.21.15.81:0/129187075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 0x7feab0102f80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fea98009990 tx=0x7fea98009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 -- 172.21.15.81:0/129187075 shutdown_connections 2024-04-12T21:14:47.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 --2- 172.21.15.81:0/129187075 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab01034c0 0x7feab010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 --2- 172.21.15.81:0/129187075 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab0108ac0 0x7feab0108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 --2- 172.21.15.81:0/129187075 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 0x7feab0102f80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 -- 172.21.15.81:0/129187075 >> 172.21.15.81:0/129187075 conn(0x7feab0079c10 msgr2=0x7feab007a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:47.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 -- 172.21.15.81:0/129187075 shutdown_connections 2024-04-12T21:14:47.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.299+0000 7feab4a00700 1 -- 172.21.15.81:0/129187075 wait complete. 2024-04-12T21:14:47.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feab4a00700 1 Processor -- start 2024-04-12T21:14:47.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feab4a00700 1 -- start start 2024-04-12T21:14:47.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feab4a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 0x7feab0199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:47.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feab4a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab01034c0 0x7feab0199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:47.297 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feab4a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab0108ac0 0x7feab019dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:47.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feab4a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7feab0112df0 con 0x7feab0102b00 2024-04-12T21:14:47.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feab4a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7feab0112c70 con 0x7feab01034c0 2024-04-12T21:14:47.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feab4a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feab0112f70 con 0x7feab0108ac0 2024-04-12T21:14:47.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.301+0000 7feaade00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 0x7feab0199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:47.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaade00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 0x7feab0199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51526/0 (socket says 172.21.15.81:51526) 2024-04-12T21:14:47.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaade00700 1 -- 172.21.15.81:0/4021712253 learned_addr learned my addr 172.21.15.81:0/4021712253 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:47.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaae800700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab0108ac0 0x7feab019dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:47.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaa5400700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab01034c0 0x7feab0199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaade00700 1 -- 172.21.15.81:0/4021712253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab0108ac0 msgr2=0x7feab019dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaade00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab0108ac0 0x7feab019dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaade00700 1 -- 172.21.15.81:0/4021712253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab01034c0 msgr2=0x7feab0199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaade00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab01034c0 0x7feab0199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaade00700 1 -- 172.21.15.81:0/4021712253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab019e310 con 0x7feab0102b00 2024-04-12T21:14:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.302+0000 7feaade00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 0x7feab0199300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feaa000eb30 tx=0x7feaa000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.303+0000 7feaa7400700 1 -- 172.21.15.81:0/4021712253 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa000cbd0 con 0x7feab0102b00 2024-04-12T21:14:47.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.303+0000 7feaa7400700 1 -- 172.21.15.81:0/4021712253 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feaa0004500 con 0x7feab0102b00 2024-04-12T21:14:47.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.303+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab019e600 con 0x7feab0102b00 2024-04-12T21:14:47.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.303+0000 7feaa7400700 1 -- 172.21.15.81:0/4021712253 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0010430 con 0x7feab0102b00 2024-04-12T21:14:47.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.303+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab01a5e80 con 0x7feab0102b00 2024-04-12T21:14:47.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.304+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab00697a0 con 0x7feab0102b00 2024-04-12T21:14:47.302 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.306+0000 7feaa7400700 1 -- 172.21.15.81:0/4021712253 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7feaa0003f00 con 0x7feab0102b00 2024-04-12T21:14:47.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.308+0000 7feaa7400700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fea940a3cd0 0x7fea940a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:47.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.308+0000 7feaa7400700 1 -- 172.21.15.81:0/4021712253 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7feaa00d0cd0 con 0x7feab0102b00 2024-04-12T21:14:47.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.308+0000 7feaa5400700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fea940a3cd0 0x7fea940a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:47.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.309+0000 7feaa5400700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fea940a3cd0 0x7fea940a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feab007a2c0 tx=0x7fea9800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:47.306 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.311+0000 7feaa7400700 1 -- 172.21.15.81:0/4021712253 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feaa009d220 con 0x7feab0102b00 2024-04-12T21:14:47.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.547+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7feab002dfa0 con 0x7feab0102b00 2024-04-12T21:14:47.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.548+0000 7feaa7400700 1 -- 172.21.15.81:0/4021712253 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feaa009cb20 con 0x7feab0102b00 2024-04-12T21:14:47.544 INFO:teuthology.orchestra.run.smithi081.stdout:34359738461 2024-04-12T21:14:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fea940a3cd0 msgr2=0x7fea940a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fea940a3cd0 0x7fea940a6190 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feab007a2c0 tx=0x7fea9800b260 comp rx=0 tx=0).stop 2024-04-12T21:14:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 msgr2=0x7feab0199300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 0x7feab0199300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7feaa000eb30 tx=0x7feaa000ee40 comp rx=0 tx=0).stop 2024-04-12T21:14:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 shutdown_connections 2024-04-12T21:14:47.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fea940a3cd0 0x7fea940a6190 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feab0108ac0 0x7feab019dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7feab0102b00 0x7feab0199300 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 --2- 172.21.15.81:0/4021712253 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7feab01034c0 0x7feab0199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.551+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 >> 172.21.15.81:0/4021712253 conn(0x7feab0079c10 msgr2=0x7feab0101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.552+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 shutdown_connections 2024-04-12T21:14:47.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:47.552+0000 7feab4a00700 1 -- 172.21.15.81:0/4021712253 wait complete. 2024-04-12T21:14:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:47 smithi153 ceph-mon[36569]: pgmap v363: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:47 smithi104 ceph-mon[36624]: pgmap v363: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:47 smithi081 ceph-mon[31453]: pgmap v363: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:48.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411411 for osd.2 2024-04-12T21:14:48.479 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:48.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:48 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4021712253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T21:14:48.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:48 smithi153 ceph-mon[36569]: pgmap v364: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:48.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.690+0000 7f5b08400700 1 -- 172.21.15.81:0/2289154278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00074ce0 msgr2=0x7f5b000750c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:48.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.690+0000 7f5b08400700 1 --2- 172.21.15.81:0/2289154278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00074ce0 0x7f5b000750c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5af4009990 tx=0x7f5af4009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.690+0000 7f5b08400700 1 -- 172.21.15.81:0/2289154278 shutdown_connections 2024-04-12T21:14:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.690+0000 7f5b08400700 1 --2- 172.21.15.81:0/2289154278 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b0010f4b0 0x7f5b001118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.690+0000 7f5b08400700 1 --2- 172.21.15.81:0/2289154278 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00074ce0 0x7f5b000750c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.690+0000 7f5b08400700 1 --2- 172.21.15.81:0/2289154278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b00075600 0x7f5b0010ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.690+0000 7f5b08400700 1 -- 172.21.15.81:0/2289154278 >> 172.21.15.81:0/2289154278 conn(0x7f5b00079c10 msgr2=0x7f5b0007a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:48.687 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:48 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4021712253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T21:14:48.687 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:48 smithi081 ceph-mon[31453]: pgmap v364: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.691+0000 7f5b08400700 1 -- 172.21.15.81:0/2289154278 shutdown_connections 2024-04-12T21:14:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.691+0000 7f5b08400700 1 -- 172.21.15.81:0/2289154278 wait complete. 2024-04-12T21:14:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.692+0000 7f5b08400700 1 Processor -- start 2024-04-12T21:14:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.692+0000 7f5b08400700 1 -- start start 2024-04-12T21:14:48.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b08400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b00074ce0 0x7f5b0019b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b08400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00075600 0x7f5b0019b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b08400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0010f4b0 0x7f5b0019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b08400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5b00114fd0 con 0x7f5b00075600 2024-04-12T21:14:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b08400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5b00114e50 con 0x7f5b0010f4b0 2024-04-12T21:14:48.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b08400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5b00115150 con 0x7f5b00074ce0 2024-04-12T21:14:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b05000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00075600 0x7f5b0019b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b05000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00075600 0x7f5b0019b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:51544/0 (socket says 172.21.15.81:51544) 2024-04-12T21:14:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b05000700 1 -- 172.21.15.81:0/3951089047 learned_addr learned my addr 172.21.15.81:0/3951089047 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b06400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0010f4b0 0x7f5b0019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b05a00700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b00074ce0 0x7f5b0019b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b06400700 1 -- 172.21.15.81:0/3951089047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b00074ce0 msgr2=0x7f5b0019b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:48.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b06400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b00074ce0 0x7f5b0019b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b06400700 1 -- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00075600 msgr2=0x7f5b0019b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.693+0000 7f5b06400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00075600 0x7f5b0019b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.694+0000 7f5b06400700 1 -- 172.21.15.81:0/3951089047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b001a0440 con 0x7f5b0010f4b0 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.694+0000 7f5b05000700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00075600 0x7f5b0019b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.694+0000 7f5b06400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0010f4b0 0x7f5b0019fd00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5af800ef90 tx=0x7f5af800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.694+0000 7f5afe000700 1 -- 172.21.15.81:0/3951089047 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af80092e0 con 0x7f5b0010f4b0 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.694+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b001a06d0 con 0x7f5b0010f4b0 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.694+0000 7f5afe000700 1 -- 172.21.15.81:0/3951089047 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5af8004bb0 con 0x7f5b0010f4b0 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.694+0000 7f5afe000700 1 -- 172.21.15.81:0/3951089047 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8004280 con 0x7f5b0010f4b0 2024-04-12T21:14:48.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.694+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b001a7fb0 con 0x7f5b0010f4b0 2024-04-12T21:14:48.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.695+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b000697a0 con 0x7f5b0010f4b0 2024-04-12T21:14:48.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.697+0000 7f5afe000700 1 -- 172.21.15.81:0/3951089047 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5af801b030 con 0x7f5b0010f4b0 2024-04-12T21:14:48.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.699+0000 7f5afe000700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5ae80a3cd0 0x7f5ae80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:48.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.699+0000 7f5afe000700 1 -- 172.21.15.81:0/3951089047 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f5af80d0870 con 0x7f5b0010f4b0 2024-04-12T21:14:48.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.699+0000 7f5b05a00700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5ae80a3cd0 0x7f5ae80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:48.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.700+0000 7f5b05a00700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5ae80a3cd0 0x7f5ae80a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5af400b2e0 tx=0x7f5af4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:48.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.702+0000 7f5afe000700 1 -- 172.21.15.81:0/3951089047 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5af8022020 con 0x7f5b0010f4b0 2024-04-12T21:14:48.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:48 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4021712253' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-12T21:14:48.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:48 smithi104 ceph-mon[36624]: pgmap v364: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:48.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.933+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f5b001a8180 con 0x7f5b0010f4b0 2024-04-12T21:14:48.929 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.934+0000 7f5afe000700 1 -- 172.21.15.81:0/3951089047 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5af8010040 con 0x7f5b0010f4b0 2024-04-12T21:14:48.929 INFO:teuthology.orchestra.run.smithi081.stdout:219043332151 2024-04-12T21:14:48.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5ae80a3cd0 msgr2=0x7f5ae80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:48.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5ae80a3cd0 0x7f5ae80a6190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5af400b2e0 tx=0x7f5af4005960 comp rx=0 tx=0).stop 2024-04-12T21:14:48.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0010f4b0 msgr2=0x7f5b0019fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:48.932 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0010f4b0 0x7f5b0019fd00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5af800ef90 tx=0x7f5af800c3a0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 shutdown_connections 2024-04-12T21:14:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f5ae80a3cd0 0x7f5ae80a6190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b0010f4b0 0x7f5b0019fd00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5b00074ce0 0x7f5b0019b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 --2- 172.21.15.81:0/3951089047 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f5b00075600 0x7f5b0019b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.936+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 >> 172.21.15.81:0/3951089047 conn(0x7f5b00079c10 msgr2=0x7f5b000ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.937+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 shutdown_connections 2024-04-12T21:14:48.933 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:48.937+0000 7f5b08400700 1 -- 172.21.15.81:0/3951089047 wait complete. 2024-04-12T21:14:49.383 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738455 got 34359738461 for osd.0 2024-04-12T21:14:49.384 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:49.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:49 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3951089047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-12T21:14:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:49 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3951089047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-12T21:14:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:49 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3951089047' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-12T21:14:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:50 smithi153 ceph-mon[36569]: pgmap v365: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:50.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:50 smithi104 ceph-mon[36624]: pgmap v365: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:50.829 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:50 smithi081 ceph-mon[31453]: pgmap v365: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 -- 172.21.15.81:0/3674894043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2dc075600 msgr2=0x7ff2dc10efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 --2- 172.21.15.81:0/3674894043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2dc075600 0x7ff2dc10efd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009990 tx=0x7ff2cc009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 -- 172.21.15.81:0/3674894043 shutdown_connections 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 --2- 172.21.15.81:0/3674894043 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc10f510 0x7ff2dc111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 --2- 172.21.15.81:0/3674894043 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2dc074ce0 0x7ff2dc0750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 --2- 172.21.15.81:0/3674894043 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2dc075600 0x7ff2dc10efd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 -- 172.21.15.81:0/3674894043 >> 172.21.15.81:0/3674894043 conn(0x7ff2dc079c10 msgr2=0x7ff2dc07a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 -- 172.21.15.81:0/3674894043 shutdown_connections 2024-04-12T21:14:51.539 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.543+0000 7ff2e4800700 1 -- 172.21.15.81:0/3674894043 wait complete. 2024-04-12T21:14:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.544+0000 7ff2e4800700 1 Processor -- start 2024-04-12T21:14:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.544+0000 7ff2e4800700 1 -- start start 2024-04-12T21:14:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e4800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2dc074ce0 0x7ff2dc19b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e4800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2dc075600 0x7ff2dc19b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e4800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc10f510 0x7ff2dc19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e4800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc115050 con 0x7ff2dc074ce0 2024-04-12T21:14:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e4800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc114ed0 con 0x7ff2dc10f510 2024-04-12T21:14:51.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e4800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff2dc1151d0 con 0x7ff2dc075600 2024-04-12T21:14:51.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e1400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2dc075600 0x7ff2dc19b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:51.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e2800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc10f510 0x7ff2dc19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:51.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e1e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2dc074ce0 0x7ff2dc19b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:51.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e1400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2dc075600 0x7ff2dc19b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:45278/0 (socket says 172.21.15.81:45278) 2024-04-12T21:14:51.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.545+0000 7ff2e1400700 1 -- 172.21.15.81:0/224167305 learned_addr learned my addr 172.21.15.81:0/224167305 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2e2800700 1 -- 172.21.15.81:0/224167305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2dc075600 msgr2=0x7ff2dc19b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2e2800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2dc075600 0x7ff2dc19b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2e2800700 1 -- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2dc074ce0 msgr2=0x7ff2dc19b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2e2800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2dc074ce0 0x7ff2dc19b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2e2800700 1 -- 172.21.15.81:0/224167305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2dc1a0450 con 0x7ff2dc10f510 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2e1400700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2dc075600 0x7ff2dc19b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2e2800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc10f510 0x7ff2dc19fd10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800ef90 tx=0x7ff2d800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2d6a00700 1 -- 172.21.15.81:0/224167305 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d80092e0 con 0x7ff2dc10f510 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2d6a00700 1 -- 172.21.15.81:0/224167305 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2d8004bb0 con 0x7ff2dc10f510 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2d6a00700 1 -- 172.21.15.81:0/224167305 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8004020 con 0x7ff2dc10f510 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.546+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2dc1a06e0 con 0x7ff2dc10f510 2024-04-12T21:14:51.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.547+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2dc1a7fc0 con 0x7ff2dc10f510 2024-04-12T21:14:51.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.548+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2dc0697a0 con 0x7ff2dc10f510 2024-04-12T21:14:51.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.550+0000 7ff2d6a00700 1 -- 172.21.15.81:0/224167305 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff2d801b030 con 0x7ff2dc10f510 2024-04-12T21:14:51.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.551+0000 7ff2d6a00700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff2c40a3cd0 0x7ff2c40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:51.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.551+0000 7ff2d6a00700 1 -- 172.21.15.81:0/224167305 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff2d80d0470 con 0x7ff2dc10f510 2024-04-12T21:14:51.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.551+0000 7ff2e1e00700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff2c40a3cd0 0x7ff2c40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:51.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.552+0000 7ff2e1e00700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff2c40a3cd0 0x7ff2c40a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0009c60 tx=0x7ff2d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:51.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.556+0000 7ff2d6a00700 1 -- 172.21.15.81:0/224167305 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff2d8022020 con 0x7ff2dc10f510 2024-04-12T21:14:51.781 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.786+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff2dc1a8190 con 0x7ff2dc10f510 2024-04-12T21:14:51.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.786+0000 7ff2d6a00700 1 -- 172.21.15.81:0/224167305 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff2d8010040 con 0x7ff2dc10f510 2024-04-12T21:14:51.782 INFO:teuthology.orchestra.run.smithi081.stdout:309237645350 2024-04-12T21:14:51.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.789+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff2c40a3cd0 msgr2=0x7ff2c40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:51.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.789+0000 7ff2e4800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff2c40a3cd0 0x7ff2c40a6190 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0009c60 tx=0x7ff2d0009260 comp rx=0 tx=0).stop 2024-04-12T21:14:51.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.789+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc10f510 msgr2=0x7ff2dc19fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.789+0000 7ff2e4800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc10f510 0x7ff2dc19fd10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff2d800ef90 tx=0x7ff2d800c3a0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.789+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 shutdown_connections 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.789+0000 7ff2e4800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff2c40a3cd0 0x7ff2c40a6190 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.790+0000 7ff2e4800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff2dc10f510 0x7ff2dc19fd10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.790+0000 7ff2e4800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff2dc074ce0 0x7ff2dc19b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.790+0000 7ff2e4800700 1 --2- 172.21.15.81:0/224167305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2dc075600 0x7ff2dc19b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.790+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 >> 172.21.15.81:0/224167305 conn(0x7ff2dc079c10 msgr2=0x7ff2dc0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.790+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 shutdown_connections 2024-04-12T21:14:51.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:51.790+0000 7ff2e4800700 1 -- 172.21.15.81:0/224167305 wait complete. 2024-04-12T21:14:52.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:51 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/224167305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-12T21:14:52.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:51 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/224167305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-12T21:14:52.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:51 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/224167305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-12T21:14:53.123 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:52 smithi081 ceph-mon[31453]: pgmap v366: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:53.299 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332145 got 219043332151 for osd.7 2024-04-12T21:14:53.299 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:53.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:52 smithi153 ceph-mon[36569]: pgmap v366: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:53.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:52 smithi104 ceph-mon[36624]: pgmap v366: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:14:54.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:53 smithi081 ceph-mon[31453]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-12T21:14:54.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:53 smithi153 ceph-mon[36569]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-12T21:14:54.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:53 smithi104 ceph-mon[36624]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-12T21:14:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 -- 172.21.15.81:0/1195902577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 msgr2=0x7fbf04109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:55.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 --2- 172.21.15.81:0/1195902577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 0x7fbf04109950 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009990 tx=0x7fbef0009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 -- 172.21.15.81:0/1195902577 shutdown_connections 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 --2- 172.21.15.81:0/1195902577 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0410a030 0x7fbf04111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 --2- 172.21.15.81:0/1195902577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf04104f50 0x7fbf04105330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 --2- 172.21.15.81:0/1195902577 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 0x7fbf04109950 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 -- 172.21.15.81:0/1195902577 >> 172.21.15.81:0/1195902577 conn(0x7fbf041007e0 msgr2=0x7fbf04102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 -- 172.21.15.81:0/1195902577 shutdown_connections 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.352+0000 7fbf09a00700 1 -- 172.21.15.81:0/1195902577 wait complete. 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.353+0000 7fbf09a00700 1 Processor -- start 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.353+0000 7fbf09a00700 1 -- start start 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.353+0000 7fbf09a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf04104f50 0x7fbf0419b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.353+0000 7fbf09a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 0x7fbf0419ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.353+0000 7fbf09a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0410a030 0x7fbf0419fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:55.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.353+0000 7fbf09a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04115060 con 0x7fbf0410a030 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.353+0000 7fbf09a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04114ee0 con 0x7fbf04105900 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.353+0000 7fbf09a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbf041151e0 con 0x7fbf04104f50 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf04104f50 0x7fbf0419b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf03400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0410a030 0x7fbf0419fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 0x7fbf0419ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf04104f50 0x7fbf0419b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:45286/0 (socket says 172.21.15.81:45286) 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02a00700 1 -- 172.21.15.81:0/541917961 learned_addr learned my addr 172.21.15.81:0/541917961 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02000700 1 -- 172.21.15.81:0/541917961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf04104f50 msgr2=0x7fbf0419b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02000700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf04104f50 0x7fbf0419b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02000700 1 -- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0410a030 msgr2=0x7fbf0419fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02000700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0410a030 0x7fbf0419fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.354+0000 7fbf02000700 1 -- 172.21.15.81:0/541917961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf041a04e0 con 0x7fbf04105900 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.355+0000 7fbf02000700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 0x7fbf0419ba10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbef0000c00 tx=0x7fbef00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:55.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.355+0000 7fbefb400700 1 -- 172.21.15.81:0/541917961 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef0026070 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.355+0000 7fbefb400700 1 -- 172.21.15.81:0/541917961 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbef000b6c0 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.355+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf041a0770 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.355+0000 7fbefb400700 1 -- 172.21.15.81:0/541917961 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef00173f0 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.355+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf041a8050 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.356+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf040697a0 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.358+0000 7fbefb400700 1 -- 172.21.15.81:0/541917961 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbef0029070 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.359+0000 7fbefb400700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbee80a3cc0 0x7fbee80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.360+0000 7fbefb400700 1 -- 172.21.15.81:0/541917961 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fbef00d1700 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.360+0000 7fbf02a00700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbee80a3cc0 0x7fbee80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.360+0000 7fbf02a00700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbee80a3cc0 0x7fbee80a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009d60 tx=0x7fbef4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.362+0000 7fbefb400700 1 -- 172.21.15.81:0/541917961 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbef0013070 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.596+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbf0402dfa0 con 0x7fbf04105900 2024-04-12T21:14:55.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.596+0000 7fbefb400700 1 -- 172.21.15.81:0/541917961 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbef009ddb0 con 0x7fbf04105900 2024-04-12T21:14:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbee80a3cc0 msgr2=0x7fbee80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbee80a3cc0 0x7fbee80a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009d60 tx=0x7fbef4009260 comp rx=0 tx=0).stop 2024-04-12T21:14:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 msgr2=0x7fbf0419ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 0x7fbf0419ba10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbef0000c00 tx=0x7fbef00047b0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.037 INFO:teuthology.orchestra.run.smithi081.stdout:163208757316 2024-04-12T21:14:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 shutdown_connections 2024-04-12T21:14:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbee80a3cc0 0x7fbee80a6180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbf0410a030 0x7fbf0419fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf04104f50 0x7fbf0419b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 --2- 172.21.15.81:0/541917961 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbf04105900 0x7fbf0419ba10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.599+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 >> 172.21.15.81:0/541917961 conn(0x7fbf041007e0 msgr2=0x7fbf04101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.600+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 shutdown_connections 2024-04-12T21:14:55.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:54.600+0000 7fbf09a00700 1 -- 172.21.15.81:0/541917961 wait complete. 2024-04-12T21:14:55.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:54 smithi081 ceph-mon[31453]: pgmap v367: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-04-12T21:14:55.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:54 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/541917961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T21:14:55.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:54 smithi153 ceph-mon[36569]: pgmap v367: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-04-12T21:14:55.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:54 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/541917961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T21:14:55.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:54 smithi104 ceph-mon[36624]: pgmap v367: 129 pgs: 129 active+clean; 638 KiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 0 B/s wr, 1 op/s 2024-04-12T21:14:55.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:54 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/541917961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-12T21:14:55.783 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808875 for osd.9 2024-04-12T21:14:55.783 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:56.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:56 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1459501455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-12T21:14:56.596 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:56 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1459501455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-12T21:14:56.653 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757310 got 163208757316 for osd.5 2024-04-12T21:14:56.653 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:56.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:56 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1459501455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-12T21:14:57.025 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645344 got 309237645350 for osd.10 2024-04-12T21:14:57.026 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:57.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 -- 172.21.15.81:0/2617385038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981089e0 msgr2=0x7fd698108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:57.132 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 --2- 172.21.15.81:0/2617385038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981089e0 0x7fd698108dc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd688009990 tx=0x7fd688009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 -- 172.21.15.81:0/2617385038 shutdown_connections 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 --2- 172.21.15.81:0/2617385038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6981033a0 0x7fd69810f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 --2- 172.21.15.81:0/2617385038 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981089e0 0x7fd698108dc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 --2- 172.21.15.81:0/2617385038 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6981029e0 0x7fd698102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 -- 172.21.15.81:0/2617385038 >> 172.21.15.81:0/2617385038 conn(0x7fd6980fe530 msgr2=0x7fd698100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 -- 172.21.15.81:0/2617385038 shutdown_connections 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.017+0000 7fd69d000700 1 -- 172.21.15.81:0/2617385038 wait complete. 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.018+0000 7fd69d000700 1 Processor -- start 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.018+0000 7fd69d000700 1 -- start start 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd69d000700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981029e0 0x7fd698074db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd69d000700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6981033a0 0x7fd6980752f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:57.133 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd69d000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6981089e0 0x7fd698075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd69d000700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd698112d60 con 0x7fd6981029e0 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd69d000700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd698112be0 con 0x7fd6981033a0 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd69d000700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd698112ee0 con 0x7fd6981089e0 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981029e0 0x7fd698074db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981029e0 0x7fd698074db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32810/0 (socket says 172.21.15.81:32810) 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697400700 1 -- 172.21.15.81:0/1459501455 learned_addr learned my addr 172.21.15.81:0/1459501455 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd696a00700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6981033a0 0x7fd6980752f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697e00700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6981089e0 0x7fd698075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697400700 1 -- 172.21.15.81:0/1459501455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6981089e0 msgr2=0x7fd698075830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697400700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6981089e0 0x7fd698075830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697400700 1 -- 172.21.15.81:0/1459501455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6981033a0 msgr2=0x7fd6980752f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697400700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6981033a0 0x7fd6980752f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697400700 1 -- 172.21.15.81:0/1459501455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6981a7700 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd696a00700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6981033a0 0x7fd6980752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.019+0000 7fd697e00700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6981089e0 0x7fd698075830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.020+0000 7fd697400700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981029e0 0x7fd698074db0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd688005910 tx=0x7fd68800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.020+0000 7fd68fe00700 1 -- 172.21.15.81:0/1459501455 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688027070 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.020+0000 7fd68fe00700 1 -- 172.21.15.81:0/1459501455 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd688003d70 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.020+0000 7fd68fe00700 1 -- 172.21.15.81:0/1459501455 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68800f460 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.020+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6981a7990 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.020+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6981a7d90 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.021+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6980697a0 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.023+0000 7fd68fe00700 1 -- 172.21.15.81:0/1459501455 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd68802a070 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.024+0000 7fd68fe00700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd67c0a3c40 0x7fd67c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.024+0000 7fd68fe00700 1 -- 172.21.15.81:0/1459501455 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fd6880d1540 con 0x7fd6981029e0 2024-04-12T21:14:57.135 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.025+0000 7fd696a00700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd67c0a3c40 0x7fd67c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:57.136 INFO:teuthology.orchestra.run.smithi081.stdout:253403070516 2024-04-12T21:14:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.025+0000 7fd696a00700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd67c0a3c40 0x7fd67c0a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd698108080 tx=0x7fd684009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.029+0000 7fd68fe00700 1 -- 172.21.15.81:0/1459501455 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd688013070 con 0x7fd6981029e0 2024-04-12T21:14:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.257+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fd69802e160 con 0x7fd6981029e0 2024-04-12T21:14:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.258+0000 7fd68fe00700 1 -- 172.21.15.81:0/1459501455 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd68809dbf0 con 0x7fd6981029e0 2024-04-12T21:14:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd67c0a3c40 msgr2=0x7fd67c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd67c0a3c40 0x7fd67c0a6100 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd698108080 tx=0x7fd684009260 comp rx=0 tx=0).stop 2024-04-12T21:14:57.136 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981029e0 msgr2=0x7fd698074db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981029e0 0x7fd698074db0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd688005910 tx=0x7fd68800bab0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 shutdown_connections 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd67c0a3c40 0x7fd67c0a6100 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6981089e0 0x7fd698075830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd6981029e0 0x7fd698074db0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 --2- 172.21.15.81:0/1459501455 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6981033a0 0x7fd6980752f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 >> 172.21.15.81:0/1459501455 conn(0x7fd6980fe530 msgr2=0x7fd6980feb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 shutdown_connections 2024-04-12T21:14:57.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:56.261+0000 7fd69d000700 1 -- 172.21.15.81:0/1459501455 wait complete. 2024-04-12T21:14:57.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:57 smithi104 ceph-mon[36624]: pgmap v368: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:14:57.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:57 smithi081 ceph-mon[31453]: pgmap v368: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:14:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:57 smithi153 ceph-mon[36569]: pgmap v368: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:14:57.920 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070510 got 253403070516 for osd.8 2024-04-12T21:14:57.920 DEBUG:teuthology.parallel:result is None 2024-04-12T21:14:58.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:58.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:58.686 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:58 smithi081 ceph-mon[31453]: pgmap v369: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:14:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.810+0000 7f4c23400700 1 -- 172.21.15.81:0/1566067598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c075600 msgr2=0x7f4c1c10ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.810+0000 7f4c23400700 1 --2- 172.21.15.81:0/1566067598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c075600 0x7f4c1c10ef70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c009990 tx=0x7f4c0c009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.811+0000 7f4c23400700 1 -- 172.21.15.81:0/1566067598 shutdown_connections 2024-04-12T21:14:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.811+0000 7f4c23400700 1 --2- 172.21.15.81:0/1566067598 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c1c10f4b0 0x7f4c1c1118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.811+0000 7f4c23400700 1 --2- 172.21.15.81:0/1566067598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c1c074ce0 0x7f4c1c0750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.811+0000 7f4c23400700 1 --2- 172.21.15.81:0/1566067598 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c075600 0x7f4c1c10ef70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.811+0000 7f4c23400700 1 -- 172.21.15.81:0/1566067598 >> 172.21.15.81:0/1566067598 conn(0x7f4c1c079c10 msgr2=0x7f4c1c07a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:58.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.811+0000 7f4c23400700 1 -- 172.21.15.81:0/1566067598 shutdown_connections 2024-04-12T21:14:58.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.811+0000 7f4c23400700 1 -- 172.21.15.81:0/1566067598 wait complete. 2024-04-12T21:14:58.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.812+0000 7f4c23400700 1 Processor -- start 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.812+0000 7f4c23400700 1 -- start start 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.812+0000 7f4c23400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c1c074ce0 0x7f4c1c1a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.812+0000 7f4c23400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c1c075600 0x7f4c1c1a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.812+0000 7f4c23400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c10f4b0 0x7f4c1c10dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.812+0000 7f4c23400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c115020 con 0x7f4c1c074ce0 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.812+0000 7f4c23400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c114ea0 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.812+0000 7f4c23400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c1151a0 con 0x7f4c1c075600 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.813+0000 7f4c20a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c1c074ce0 0x7f4c1c1a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.813+0000 7f4c20a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c1c074ce0 0x7f4c1c1a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:32836/0 (socket says 172.21.15.81:32836) 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.813+0000 7f4c20a00700 1 -- 172.21.15.81:0/3027819990 learned_addr learned my addr 172.21.15.81:0/3027819990 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.813+0000 7f4c1be00700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c1c075600 0x7f4c1c1a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.813+0000 7f4c21400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c10f4b0 0x7f4c1c10dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c21400700 1 -- 172.21.15.81:0/3027819990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c1c075600 msgr2=0x7f4c1c1a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:58.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c21400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c1c075600 0x7f4c1c1a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c21400700 1 -- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c1c074ce0 msgr2=0x7f4c1c1a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c21400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c1c074ce0 0x7f4c1c1a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c21400700 1 -- 172.21.15.81:0/3027819990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c10e560 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c1be00700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c1c075600 0x7f4c1c1a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c20a00700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c1c074ce0 0x7f4c1c1a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c21400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c10f4b0 0x7f4c1c10dfa0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400ef90 tx=0x7f4c1400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c19600700 1 -- 172.21.15.81:0/3027819990 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140092e0 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1c10e7f0 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c19600700 1 -- 172.21.15.81:0/3027819990 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c14004bb0 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.814+0000 7f4c19600700 1 -- 172.21.15.81:0/3027819990 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14004280 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.815+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c1c10ecb0 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.817+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c1c0697a0 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.818+0000 7f4c19600700 1 -- 172.21.15.81:0/3027819990 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4c1401b030 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.819+0000 7f4c19600700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c040a3c40 0x7f4c040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.819+0000 7f4c19600700 1 -- 172.21.15.81:0/3027819990 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4c140d0330 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.815 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.820+0000 7f4c20a00700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c040a3c40 0x7f4c040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:58.816 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.820+0000 7f4c20a00700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c040a3c40 0x7f4c040a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4c10011590 tx=0x7f4c10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:58.819 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.824+0000 7f4c19600700 1 -- 172.21.15.81:0/3027819990 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c14022020 con 0x7f4c1c10f4b0 2024-04-12T21:14:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.689+0000 7f9615800700 1 -- 172.21.15.81:0/1246353112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f961006a7c0 msgr2=0x7f961006ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.689+0000 7f9615800700 1 --2- 172.21.15.81:0/1246353112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f961006a7c0 0x7f961006ac20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009990 tx=0x7f95f8009ca0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.689+0000 7f9615800700 1 -- 172.21.15.81:0/1246353112 shutdown_connections 2024-04-12T21:14:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.689+0000 7f9615800700 1 --2- 172.21.15.81:0/1246353112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961006b160 0x7f961010f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.689+0000 7f9615800700 1 --2- 172.21.15.81:0/1246353112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9610108aa0 0x7f9610108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.689+0000 7f9615800700 1 --2- 172.21.15.81:0/1246353112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f961006a7c0 0x7f961006ac20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.690+0000 7f9615800700 1 -- 172.21.15.81:0/1246353112 >> 172.21.15.81:0/1246353112 conn(0x7f9610078960 msgr2=0x7f9610078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:58.972 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.690+0000 7f9615800700 1 -- 172.21.15.81:0/1246353112 shutdown_connections 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.690+0000 7f9615800700 1 -- 172.21.15.81:0/1246353112 wait complete. 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.691+0000 7f9615800700 1 Processor -- start 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.691+0000 7f9615800700 1 -- start start 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.691+0000 7f9615800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961006a7c0 0x7f961019b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f9615800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f961006b160 0x7f961019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f9615800700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9610108aa0 0x7f961019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f9615800700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9610112e20 con 0x7f961006b160 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f9615800700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9610112ca0 con 0x7f9610108aa0 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f9615800700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9610112fa0 con 0x7f961006a7c0 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f960ea00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961006a7c0 0x7f961019b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f960ea00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961006a7c0 0x7f961019b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:51374/0 (socket says 172.21.15.81:51374) 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f960ea00700 1 -- 172.21.15.81:0/537583223 learned_addr learned my addr 172.21.15.81:0/537583223 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.692+0000 7f960f400700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9610108aa0 0x7f961019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f960ea00700 1 -- 172.21.15.81:0/537583223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9610108aa0 msgr2=0x7f961019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:58.973 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f960ea00700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9610108aa0 0x7f961019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f960ea00700 1 -- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f961006b160 msgr2=0x7f961019b9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f960ea00700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f961006b160 0x7f961019b9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f960ea00700 1 -- 172.21.15.81:0/537583223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96101a04c0 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f960ea00700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961006a7c0 0x7f961019b4b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f960000eb30 tx=0x7f960000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f9607400700 1 -- 172.21.15.81:0/537583223 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960000cc90 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96101a07b0 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f9607400700 1 -- 172.21.15.81:0/537583223 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9600004500 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f9607400700 1 -- 172.21.15.81:0/537583223 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600010430 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.693+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96101a8030 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.695+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96100697a0 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.697+0000 7f9607400700 1 -- 172.21.15.81:0/537583223 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9600003f70 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.698+0000 7f9607400700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f95f40a3cc0 0x7f95f40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.698+0000 7f9607400700 1 -- 172.21.15.81:0/537583223 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f96000d0a10 con 0x7f961006a7c0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.698+0000 7f9607e00700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f95f40a3cc0 0x7f95f40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:14:58.974 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.699+0000 7f9607e00700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f95f40a3cc0 0x7f95f40a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f95f80098e0 tx=0x7f95f8005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:14:58.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.702+0000 7f9607400700 1 -- 172.21.15.81:0/537583223 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f960009d030 con 0x7f961006a7c0 2024-04-12T21:14:58.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.947+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f961002dfa0 con 0x7f961006a7c0 2024-04-12T21:14:58.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.947+0000 7f9607400700 1 -- 172.21.15.81:0/537583223 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f960009c930 con 0x7f961006a7c0 2024-04-12T21:14:58.975 INFO:teuthology.orchestra.run.smithi081.stdout:197568495677 2024-04-12T21:14:58.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f95f40a3cc0 msgr2=0x7f95f40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:58.975 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f95f40a3cc0 0x7f95f40a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f95f80098e0 tx=0x7f95f8005e80 comp rx=0 tx=0).stop 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961006a7c0 msgr2=0x7f961019b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961006a7c0 0x7f961019b4b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f960000eb30 tx=0x7f960000ee40 comp rx=0 tx=0).stop 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 shutdown_connections 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f95f40a3cc0 0x7f95f40a6180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9610108aa0 0x7f961019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f961006a7c0 0x7f961019b4b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 --2- 172.21.15.81:0/537583223 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f961006b160 0x7f961019b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.950+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 >> 172.21.15.81:0/537583223 conn(0x7f9610078960 msgr2=0x7f961010df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.951+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 shutdown_connections 2024-04-12T21:14:58.976 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:58.951+0000 7f9615800700 1 -- 172.21.15.81:0/537583223 wait complete. 2024-04-12T21:14:59.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:59.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:59.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:58 smithi153 ceph-mon[36569]: pgmap v369: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:14:59.070 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.074+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f4c1c10ef10 con 0x7f4c1c10f4b0 2024-04-12T21:14:59.071 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.075+0000 7f4c19600700 1 -- 172.21.15.81:0/3027819990 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4c14010040 con 0x7f4c1c10f4b0 2024-04-12T21:14:59.071 INFO:teuthology.orchestra.run.smithi081.stdout:343597383716 2024-04-12T21:14:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c040a3c40 msgr2=0x7f4c040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c040a3c40 0x7f4c040a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4c10011590 tx=0x7f4c10009260 comp rx=0 tx=0).stop 2024-04-12T21:14:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c10f4b0 msgr2=0x7f4c1c10dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:14:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c10f4b0 0x7f4c1c10dfa0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400ef90 tx=0x7f4c1400c3a0 comp rx=0 tx=0).stop 2024-04-12T21:14:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 shutdown_connections 2024-04-12T21:14:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c040a3c40 0x7f4c040a6100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:59.074 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c1c10f4b0 0x7f4c1c10dfa0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:59.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c1c074ce0 0x7f4c1c1a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:59.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 --2- 172.21.15.81:0/3027819990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c1c075600 0x7f4c1c1a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:14:59.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 >> 172.21.15.81:0/3027819990 conn(0x7f4c1c079c10 msgr2=0x7f4c1c0ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:14:59.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 shutdown_connections 2024-04-12T21:14:59.075 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:14:59.078+0000 7f4c23400700 1 -- 172.21.15.81:0/3027819990 wait complete. 2024-04-12T21:14:59.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:59.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:59.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:58 smithi104 ceph-mon[36624]: pgmap v369: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:14:59.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:14:59.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:14:59.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:14:59.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:59 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/537583223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-12T21:14:59.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:14:59 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3027819990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-12T21:15:00.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:15:00.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:15:00.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:15:00.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:59 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/537583223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-12T21:15:00.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:14:59 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3027819990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-12T21:15:00.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:15:00.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:15:00.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:15:00.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:59 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/537583223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-12T21:15:00.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:14:59 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3027819990' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-12T21:15:00.101 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495670 got 197568495677 for osd.6 2024-04-12T21:15:00.101 DEBUG:teuthology.parallel:result is None 2024-04-12T21:15:00.676 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383710 got 343597383716 for osd.11 2024-04-12T21:15:00.676 DEBUG:teuthology.parallel:result is None 2024-04-12T21:15:00.676 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-12T21:15:00.676 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph pg dump --format=json 2024-04-12T21:15:00.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:00 smithi081 ceph-mon[31453]: pgmap v370: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:00 smithi153 ceph-mon[36569]: pgmap v370: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:01.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:00 smithi104 ceph-mon[36624]: pgmap v370: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:01.134 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:03 smithi081 ceph-mon[31453]: pgmap v371: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:03.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 -- 172.21.15.81:0/108075759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cd0102b00 msgr2=0x7f8cd0102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:03.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 --2- 172.21.15.81:0/108075759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cd0102b00 0x7f8cd0102f80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8009990 tx=0x7f8cb8009ca0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 -- 172.21.15.81:0/108075759 shutdown_connections 2024-04-12T21:15:03.671 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 --2- 172.21.15.81:0/108075759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd01034c0 0x7f8cd010f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 --2- 172.21.15.81:0/108075759 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd0108ac0 0x7f8cd0108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 --2- 172.21.15.81:0/108075759 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cd0102b00 0x7f8cd0102f80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 -- 172.21.15.81:0/108075759 >> 172.21.15.81:0/108075759 conn(0x7f8cd0079c10 msgr2=0x7f8cd007a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:03.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 -- 172.21.15.81:0/108075759 shutdown_connections 2024-04-12T21:15:03.672 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.675+0000 7f8cd6400700 1 -- 172.21.15.81:0/108075759 wait complete. 2024-04-12T21:15:03.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8cd6400700 1 Processor -- start 2024-04-12T21:15:03.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8cd6400700 1 -- start start 2024-04-12T21:15:03.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8cd6400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd0102b00 0x7f8cd01992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:03.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8cd6400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cd01034c0 0x7f8cd0199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:03.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8cd6400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd0108ac0 0x7f8cd019dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:03.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8cd6400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd0112dc0 con 0x7f8cd01034c0 2024-04-12T21:15:03.673 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8cd6400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd0112c40 con 0x7f8cd0102b00 2024-04-12T21:15:03.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8cd6400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8cd0112f40 con 0x7f8cd0108ac0 2024-04-12T21:15:03.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.677+0000 7f8ccf400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd0102b00 0x7f8cd01992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:03.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccf400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd0102b00 0x7f8cd01992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:51344/0 (socket says 172.21.15.81:51344) 2024-04-12T21:15:03.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccf400700 1 -- 172.21.15.81:0/358706716 learned_addr learned my addr 172.21.15.81:0/358706716 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:03.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8cc6a00700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cd01034c0 0x7f8cd0199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:03.674 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccfe00700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd0108ac0 0x7f8cd019dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:03.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccf400700 1 -- 172.21.15.81:0/358706716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd0108ac0 msgr2=0x7f8cd019dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:03.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccf400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd0108ac0 0x7f8cd019dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccf400700 1 -- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cd01034c0 msgr2=0x7f8cd0199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:03.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccf400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cd01034c0 0x7f8cd0199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.675 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccf400700 1 -- 172.21.15.81:0/358706716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cd019e2e0 con 0x7f8cd0102b00 2024-04-12T21:15:03.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.678+0000 7f8ccf400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd0102b00 0x7f8cd01992d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000ef10 tx=0x7f8cc000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:03.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.679+0000 7f8cccc00700 1 -- 172.21.15.81:0/358706716 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc000ca70 con 0x7f8cd0102b00 2024-04-12T21:15:03.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.679+0000 7f8cccc00700 1 -- 172.21.15.81:0/358706716 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8cc0019070 con 0x7f8cd0102b00 2024-04-12T21:15:03.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.679+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cd019e570 con 0x7f8cd0102b00 2024-04-12T21:15:03.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.679+0000 7f8cccc00700 1 -- 172.21.15.81:0/358706716 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00146e0 con 0x7f8cd0102b00 2024-04-12T21:15:03.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.679+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cd0074ce0 con 0x7f8cd0102b00 2024-04-12T21:15:03.676 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.680+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cd00697a0 con 0x7f8cd0102b00 2024-04-12T21:15:03.677 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.682+0000 7f8cccc00700 1 -- 172.21.15.81:0/358706716 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8cc0012050 con 0x7f8cd0102b00 2024-04-12T21:15:03.678 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.683+0000 7f8cccc00700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8cb40a3cc0 0x7f8cb40a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:03.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.683+0000 7f8cccc00700 1 -- 172.21.15.81:0/358706716 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8cc00d0750 con 0x7f8cd0102b00 2024-04-12T21:15:03.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.683+0000 7f8cc6a00700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8cb40a3cc0 0x7f8cb40a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:03.679 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.684+0000 7f8cc6a00700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8cb40a3cc0 0x7f8cb40a6180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800b2e0 tx=0x7f8cb8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:03.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.688+0000 7f8cccc00700 1 -- 172.21.15.81:0/358706716 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8cc0010040 con 0x7f8cd0102b00 2024-04-12T21:15:03.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:03 smithi153 ceph-mon[36569]: pgmap v371: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:03.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:03 smithi104 ceph-mon[36624]: pgmap v371: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:03.903 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.908+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8cd0062ef0 con 0x7f8cb40a3cc0 2024-04-12T21:15:03.935 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.940+0000 7f8cccc00700 1 -- 172.21.15.81:0/358706716 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340409 (secure 0 0 0) 0x7f8cd0062ef0 con 0x7f8cb40a3cc0 2024-04-12T21:15:03.946 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:03.949 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8cb40a3cc0 msgr2=0x7f8cb40a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8cb40a3cc0 0x7f8cb40a6180 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800b2e0 tx=0x7f8cb8005960 comp rx=0 tx=0).stop 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd0102b00 msgr2=0x7f8cd01992d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd0102b00 0x7f8cd01992d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000ef10 tx=0x7f8cc000c3a0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 shutdown_connections 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8cb40a3cc0 0x7f8cb40a6180 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8cd0108ac0 0x7f8cd019dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8cd0102b00 0x7f8cd01992d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 --2- 172.21.15.81:0/358706716 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8cd01034c0 0x7f8cd0199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 >> 172.21.15.81:0/358706716 conn(0x7f8cd0079c10 msgr2=0x7f8cd0101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 shutdown_connections 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:03.947+0000 7f8cd6400700 1 -- 172.21.15.81:0/358706716 wait complete. 2024-04-12T21:15:03.950 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-04-12T21:15:04.818 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-04-12T21:15:02.373883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653867,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":307,"num_read_kb":263,"num_write":269,"num_write_kb":3032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180887,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383904,"kb_used_data":13952,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557920,"statfs":{"total":1151940427776,"available":1151547310080,"internally_reserved":0,"allocated":14286848,"data_stored":7947503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":16,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005416"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.347757+0000","last_change":"2024-04-12T21:13:13.157179+0000","last_active":"2024-04-12T21:14:52.347757+0000","last_peered":"2024-04-12T21:14:52.347757+0000","last_clean":"2024-04-12T21:14:52.347757+0000","last_became_active":"2024-04-12T21:13:13.156826+0000","last_became_peered":"2024-04-12T21:13:13.156826+0000","last_unstale":"2024-04-12T21:14:52.347757+0000","last_undegraded":"2024-04-12T21:14:52.347757+0000","last_fullsized":"2024-04-12T21:14:52.347757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:51:25.310758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.355515+0000","last_change":"2024-04-12T21:13:13.150917+0000","last_active":"2024-04-12T21:14:52.355515+0000","last_peered":"2024-04-12T21:14:52.355515+0000","last_clean":"2024-04-12T21:14:52.355515+0000","last_became_active":"2024-04-12T21:13:13.150491+0000","last_became_peered":"2024-04-12T21:13:13.150491+0000","last_unstale":"2024-04-12T21:14:52.355515+0000","last_undegraded":"2024-04-12T21:14:52.355515+0000","last_fullsized":"2024-04-12T21:14:52.355515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:08:04.486913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.354347+0000","last_change":"2024-04-12T21:13:13.150355+0000","last_active":"2024-04-12T21:14:52.354347+0000","last_peered":"2024-04-12T21:14:52.354347+0000","last_clean":"2024-04-12T21:14:52.354347+0000","last_became_active":"2024-04-12T21:13:13.149979+0000","last_became_peered":"2024-04-12T21:13:13.149979+0000","last_unstale":"2024-04-12T21:14:52.354347+0000","last_undegraded":"2024-04-12T21:14:52.354347+0000","last_fullsized":"2024-04-12T21:14:52.354347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:02:56.175842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.325955+0000","last_change":"2024-04-12T21:13:13.132074+0000","last_active":"2024-04-12T21:14:52.325955+0000","last_peered":"2024-04-12T21:14:52.325955+0000","last_clean":"2024-04-12T21:14:52.325955+0000","last_became_active":"2024-04-12T21:13:13.130234+0000","last_became_peered":"2024-04-12T21:13:13.130234+0000","last_unstale":"2024-04-12T21:14:52.325955+0000","last_undegraded":"2024-04-12T21:14:52.325955+0000","last_fullsized":"2024-04-12T21:14:52.325955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:18:51.706215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.343269+0000","last_change":"2024-04-12T21:13:13.146577+0000","last_active":"2024-04-12T21:14:52.343269+0000","last_peered":"2024-04-12T21:14:52.343269+0000","last_clean":"2024-04-12T21:14:52.343269+0000","last_became_active":"2024-04-12T21:13:13.146162+0000","last_became_peered":"2024-04-12T21:13:13.146162+0000","last_unstale":"2024-04-12T21:14:52.343269+0000","last_undegraded":"2024-04-12T21:14:52.343269+0000","last_fullsized":"2024-04-12T21:14:52.343269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:43:10.267515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.336247+0000","last_change":"2024-04-12T21:13:13.137103+0000","last_active":"2024-04-12T21:14:52.336247+0000","last_peered":"2024-04-12T21:14:52.336247+0000","last_clean":"2024-04-12T21:14:52.336247+0000","last_became_active":"2024-04-12T21:13:13.136750+0000","last_became_peered":"2024-04-12T21:13:13.136750+0000","last_unstale":"2024-04-12T21:14:52.336247+0000","last_undegraded":"2024-04-12T21:14:52.336247+0000","last_fullsized":"2024-04-12T21:14:52.336247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:11:58.538179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.331493+0000","last_change":"2024-04-12T21:13:13.135440+0000","last_active":"2024-04-12T21:14:52.331493+0000","last_peered":"2024-04-12T21:14:52.331493+0000","last_clean":"2024-04-12T21:14:52.331493+0000","last_became_active":"2024-04-12T21:13:13.135027+0000","last_became_peered":"2024-04-12T21:13:13.135027+0000","last_unstale":"2024-04-12T21:14:52.331493+0000","last_undegraded":"2024-04-12T21:14:52.331493+0000","last_fullsized":"2024-04-12T21:14:52.331493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:37:50.875158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.361131+0000","last_change":"2024-04-12T21:13:13.143059+0000","last_active":"2024-04-12T21:14:52.361131+0000","last_peered":"2024-04-12T21:14:52.361131+0000","last_clean":"2024-04-12T21:14:52.361131+0000","last_became_active":"2024-04-12T21:13:13.142698+0000","last_became_peered":"2024-04-12T21:13:13.142698+0000","last_unstale":"2024-04-12T21:14:52.361131+0000","last_undegraded":"2024-04-12T21:14:52.361131+0000","last_fullsized":"2024-04-12T21:14:52.361131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:30:05.170007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.342514+0000","last_change":"2024-04-12T21:13:13.128917+0000","last_active":"2024-04-12T21:14:52.342514+0000","last_peered":"2024-04-12T21:14:52.342514+0000","last_clean":"2024-04-12T21:14:52.342514+0000","last_became_active":"2024-04-12T21:13:13.128166+0000","last_became_peered":"2024-04-12T21:13:13.128166+0000","last_unstale":"2024-04-12T21:14:52.342514+0000","last_undegraded":"2024-04-12T21:14:52.342514+0000","last_fullsized":"2024-04-12T21:14:52.342514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:49:41.282798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.329082+0000","last_change":"2024-04-12T21:13:13.130833+0000","last_active":"2024-04-12T21:14:52.329082+0000","last_peered":"2024-04-12T21:14:52.329082+0000","last_clean":"2024-04-12T21:14:52.329082+0000","last_became_active":"2024-04-12T21:13:13.130506+0000","last_became_peered":"2024-04-12T21:13:13.130506+0000","last_unstale":"2024-04-12T21:14:52.329082+0000","last_undegraded":"2024-04-12T21:14:52.329082+0000","last_fullsized":"2024-04-12T21:14:52.329082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:08:30.527381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.346297+0000","last_change":"2024-04-12T21:13:13.129682+0000","last_active":"2024-04-12T21:14:52.346297+0000","last_peered":"2024-04-12T21:14:52.346297+0000","last_clean":"2024-04-12T21:14:52.346297+0000","last_became_active":"2024-04-12T21:13:13.129364+0000","last_became_peered":"2024-04-12T21:13:13.129364+0000","last_unstale":"2024-04-12T21:14:52.346297+0000","last_undegraded":"2024-04-12T21:14:52.346297+0000","last_fullsized":"2024-04-12T21:14:52.346297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:19:48.442418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.322729+0000","last_change":"2024-04-12T21:13:13.133083+0000","last_active":"2024-04-12T21:14:52.322729+0000","last_peered":"2024-04-12T21:14:52.322729+0000","last_clean":"2024-04-12T21:14:52.322729+0000","last_became_active":"2024-04-12T21:13:13.132679+0000","last_became_peered":"2024-04-12T21:13:13.132679+0000","last_unstale":"2024-04-12T21:14:52.322729+0000","last_undegraded":"2024-04-12T21:14:52.322729+0000","last_fullsized":"2024-04-12T21:14:52.322729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:44:46.344475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.342870+0000","last_change":"2024-04-12T21:13:13.137692+0000","last_active":"2024-04-12T21:14:52.342870+0000","last_peered":"2024-04-12T21:14:52.342870+0000","last_clean":"2024-04-12T21:14:52.342870+0000","last_became_active":"2024-04-12T21:13:13.128523+0000","last_became_peered":"2024-04-12T21:13:13.128523+0000","last_unstale":"2024-04-12T21:14:52.342870+0000","last_undegraded":"2024-04-12T21:14:52.342870+0000","last_fullsized":"2024-04-12T21:14:52.342870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:52:30.847637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.363315+0000","last_change":"2024-04-12T21:13:13.149749+0000","last_active":"2024-04-12T21:14:52.363315+0000","last_peered":"2024-04-12T21:14:52.363315+0000","last_clean":"2024-04-12T21:14:52.363315+0000","last_became_active":"2024-04-12T21:13:13.149369+0000","last_became_peered":"2024-04-12T21:13:13.149369+0000","last_unstale":"2024-04-12T21:14:52.363315+0000","last_undegraded":"2024-04-12T21:14:52.363315+0000","last_fullsized":"2024-04-12T21:14:52.363315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:20.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.353980+0000","last_change":"2024-04-12T21:13:13.151510+0000","last_active":"2024-04-12T21:14:52.353980+0000","last_peered":"2024-04-12T21:14:52.353980+0000","last_clean":"2024-04-12T21:14:52.353980+0000","last_became_active":"2024-04-12T21:13:13.150620+0000","last_became_peered":"2024-04-12T21:13:13.150620+0000","last_unstale":"2024-04-12T21:14:52.353980+0000","last_undegraded":"2024-04-12T21:14:52.353980+0000","last_fullsized":"2024-04-12T21:14:52.353980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:59:14.772221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.347149+0000","last_change":"2024-04-12T21:13:13.156421+0000","last_active":"2024-04-12T21:14:52.347149+0000","last_peered":"2024-04-12T21:14:52.347149+0000","last_clean":"2024-04-12T21:14:52.347149+0000","last_became_active":"2024-04-12T21:13:13.156042+0000","last_became_peered":"2024-04-12T21:13:13.156042+0000","last_unstale":"2024-04-12T21:14:52.347149+0000","last_undegraded":"2024-04-12T21:14:52.347149+0000","last_fullsized":"2024-04-12T21:14:52.347149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:53:38.234230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.348047+0000","last_change":"2024-04-12T21:13:13.156986+0000","last_active":"2024-04-12T21:14:52.348047+0000","last_peered":"2024-04-12T21:14:52.348047+0000","last_clean":"2024-04-12T21:14:52.348047+0000","last_became_active":"2024-04-12T21:13:13.156705+0000","last_became_peered":"2024-04-12T21:13:13.156705+0000","last_unstale":"2024-04-12T21:14:52.348047+0000","last_undegraded":"2024-04-12T21:14:52.348047+0000","last_fullsized":"2024-04-12T21:14:52.348047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:49:42.442696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.342212+0000","last_change":"2024-04-12T21:13:13.136259+0000","last_active":"2024-04-12T21:14:52.342212+0000","last_peered":"2024-04-12T21:14:52.342212+0000","last_clean":"2024-04-12T21:14:52.342212+0000","last_became_active":"2024-04-12T21:13:13.135865+0000","last_became_peered":"2024-04-12T21:13:13.135865+0000","last_unstale":"2024-04-12T21:14:52.342212+0000","last_undegraded":"2024-04-12T21:14:52.342212+0000","last_fullsized":"2024-04-12T21:14:52.342212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:30:35.934682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.332926+0000","last_change":"2024-04-12T21:13:13.140460+0000","last_active":"2024-04-12T21:14:52.332926+0000","last_peered":"2024-04-12T21:14:52.332926+0000","last_clean":"2024-04-12T21:14:52.332926+0000","last_became_active":"2024-04-12T21:13:13.140115+0000","last_became_peered":"2024-04-12T21:13:13.140115+0000","last_unstale":"2024-04-12T21:14:52.332926+0000","last_undegraded":"2024-04-12T21:14:52.332926+0000","last_fullsized":"2024-04-12T21:14:52.332926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:56:08.880106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.497683+0000","last_change":"2024-04-12T21:13:17.606904+0000","last_active":"2024-04-12T21:14:23.497683+0000","last_peered":"2024-04-12T21:14:23.497683+0000","last_clean":"2024-04-12T21:14:23.497683+0000","last_became_active":"2024-04-12T21:13:17.606390+0000","last_became_peered":"2024-04-12T21:13:17.606390+0000","last_unstale":"2024-04-12T21:14:23.497683+0000","last_undegraded":"2024-04-12T21:14:23.497683+0000","last_fullsized":"2024-04-12T21:14:23.497683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:56:13.440022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506687+0000","last_change":"2024-04-12T21:13:12.535260+0000","last_active":"2024-04-12T21:14:23.506687+0000","last_peered":"2024-04-12T21:14:23.506687+0000","last_clean":"2024-04-12T21:14:23.506687+0000","last_became_active":"2024-04-12T21:13:12.534798+0000","last_became_peered":"2024-04-12T21:13:12.534798+0000","last_unstale":"2024-04-12T21:14:23.506687+0000","last_undegraded":"2024-04-12T21:14:23.506687+0000","last_fullsized":"2024-04-12T21:14:23.506687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:33:13.954242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.356476+0000","last_change":"2024-04-12T21:13:13.142701+0000","last_active":"2024-04-12T21:14:52.356476+0000","last_peered":"2024-04-12T21:14:52.356476+0000","last_clean":"2024-04-12T21:14:52.356476+0000","last_became_active":"2024-04-12T21:13:13.142329+0000","last_became_peered":"2024-04-12T21:13:13.142329+0000","last_unstale":"2024-04-12T21:14:52.356476+0000","last_undegraded":"2024-04-12T21:14:52.356476+0000","last_fullsized":"2024-04-12T21:14:52.356476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:59:02.356770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503791+0000","last_change":"2024-04-12T21:13:17.604050+0000","last_active":"2024-04-12T21:14:23.503791+0000","last_peered":"2024-04-12T21:14:23.503791+0000","last_clean":"2024-04-12T21:14:23.503791+0000","last_became_active":"2024-04-12T21:13:17.603628+0000","last_became_peered":"2024-04-12T21:13:17.603628+0000","last_unstale":"2024-04-12T21:14:23.503791+0000","last_undegraded":"2024-04-12T21:14:23.503791+0000","last_fullsized":"2024-04-12T21:14:23.503791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:24:24.986986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500866+0000","last_change":"2024-04-12T21:13:12.546263+0000","last_active":"2024-04-12T21:14:23.500866+0000","last_peered":"2024-04-12T21:14:23.500866+0000","last_clean":"2024-04-12T21:14:23.500866+0000","last_became_active":"2024-04-12T21:13:12.545710+0000","last_became_peered":"2024-04-12T21:13:12.545710+0000","last_unstale":"2024-04-12T21:14:23.500866+0000","last_undegraded":"2024-04-12T21:14:23.500866+0000","last_fullsized":"2024-04-12T21:14:23.500866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:18:33.732571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364281+0000","last_change":"2024-04-12T21:13:13.156478+0000","last_active":"2024-04-12T21:14:52.364281+0000","last_peered":"2024-04-12T21:14:52.364281+0000","last_clean":"2024-04-12T21:14:52.364281+0000","last_became_active":"2024-04-12T21:13:13.155485+0000","last_became_peered":"2024-04-12T21:13:13.155485+0000","last_unstale":"2024-04-12T21:14:52.364281+0000","last_undegraded":"2024-04-12T21:14:52.364281+0000","last_fullsized":"2024-04-12T21:14:52.364281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:11:24.161377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503495+0000","last_change":"2024-04-12T21:13:17.607327+0000","last_active":"2024-04-12T21:14:23.503495+0000","last_peered":"2024-04-12T21:14:23.503495+0000","last_clean":"2024-04-12T21:14:23.503495+0000","last_became_active":"2024-04-12T21:13:17.606949+0000","last_became_peered":"2024-04-12T21:13:17.606949+0000","last_unstale":"2024-04-12T21:14:23.503495+0000","last_undegraded":"2024-04-12T21:14:23.503495+0000","last_fullsized":"2024-04-12T21:14:23.503495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:43:27.855461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504116+0000","last_change":"2024-04-12T21:13:12.533082+0000","last_active":"2024-04-12T21:14:23.504116+0000","last_peered":"2024-04-12T21:14:23.504116+0000","last_clean":"2024-04-12T21:14:23.504116+0000","last_became_active":"2024-04-12T21:13:12.532739+0000","last_became_peered":"2024-04-12T21:13:12.532739+0000","last_unstale":"2024-04-12T21:14:23.504116+0000","last_undegraded":"2024-04-12T21:14:23.504116+0000","last_fullsized":"2024-04-12T21:14:23.504116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:40:01.885223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.349310+0000","last_change":"2024-04-12T21:13:13.136718+0000","last_active":"2024-04-12T21:14:52.349310+0000","last_peered":"2024-04-12T21:14:52.349310+0000","last_clean":"2024-04-12T21:14:52.349310+0000","last_became_active":"2024-04-12T21:13:13.136357+0000","last_became_peered":"2024-04-12T21:13:13.136357+0000","last_unstale":"2024-04-12T21:14:52.349310+0000","last_undegraded":"2024-04-12T21:14:52.349310+0000","last_fullsized":"2024-04-12T21:14:52.349310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:08:57.380556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503252+0000","last_change":"2024-04-12T21:13:17.601381+0000","last_active":"2024-04-12T21:14:23.503252+0000","last_peered":"2024-04-12T21:14:23.503252+0000","last_clean":"2024-04-12T21:14:23.503252+0000","last_became_active":"2024-04-12T21:13:17.600868+0000","last_became_peered":"2024-04-12T21:13:17.600868+0000","last_unstale":"2024-04-12T21:14:23.503252+0000","last_undegraded":"2024-04-12T21:14:23.503252+0000","last_fullsized":"2024-04-12T21:14:23.503252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:53:36.234779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.505045+0000","last_change":"2024-04-12T21:13:12.528285+0000","last_active":"2024-04-12T21:14:23.505045+0000","last_peered":"2024-04-12T21:14:23.505045+0000","last_clean":"2024-04-12T21:14:23.505045+0000","last_became_active":"2024-04-12T21:13:12.527915+0000","last_became_peered":"2024-04-12T21:13:12.527915+0000","last_unstale":"2024-04-12T21:14:23.505045+0000","last_undegraded":"2024-04-12T21:14:23.505045+0000","last_fullsized":"2024-04-12T21:14:23.505045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:44.119836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.338764+0000","last_change":"2024-04-12T21:13:13.128654+0000","last_active":"2024-04-12T21:14:52.338764+0000","last_peered":"2024-04-12T21:14:52.338764+0000","last_clean":"2024-04-12T21:14:52.338764+0000","last_became_active":"2024-04-12T21:13:13.128274+0000","last_became_peered":"2024-04-12T21:13:13.128274+0000","last_unstale":"2024-04-12T21:14:52.338764+0000","last_undegraded":"2024-04-12T21:14:52.338764+0000","last_fullsized":"2024-04-12T21:14:52.338764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:48:08.827157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498001+0000","last_change":"2024-04-12T21:13:17.594360+0000","last_active":"2024-04-12T21:14:23.498001+0000","last_peered":"2024-04-12T21:14:23.498001+0000","last_clean":"2024-04-12T21:14:23.498001+0000","last_became_active":"2024-04-12T21:13:17.593924+0000","last_became_peered":"2024-04-12T21:13:17.593924+0000","last_unstale":"2024-04-12T21:14:23.498001+0000","last_undegraded":"2024-04-12T21:14:23.498001+0000","last_fullsized":"2024-04-12T21:14:23.498001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:42:14.269657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.502843+0000","last_change":"2024-04-12T21:13:12.532481+0000","last_active":"2024-04-12T21:14:23.502843+0000","last_peered":"2024-04-12T21:14:23.502843+0000","last_clean":"2024-04-12T21:14:23.502843+0000","last_became_active":"2024-04-12T21:13:12.532126+0000","last_became_peered":"2024-04-12T21:13:12.532126+0000","last_unstale":"2024-04-12T21:14:23.502843+0000","last_undegraded":"2024-04-12T21:14:23.502843+0000","last_fullsized":"2024-04-12T21:14:23.502843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:19:52.510076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.339625+0000","last_change":"2024-04-12T21:13:13.138290+0000","last_active":"2024-04-12T21:14:52.339625+0000","last_peered":"2024-04-12T21:14:52.339625+0000","last_clean":"2024-04-12T21:14:52.339625+0000","last_became_active":"2024-04-12T21:13:13.137854+0000","last_became_peered":"2024-04-12T21:13:13.137854+0000","last_unstale":"2024-04-12T21:14:52.339625+0000","last_undegraded":"2024-04-12T21:14:52.339625+0000","last_fullsized":"2024-04-12T21:14:52.339625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:27:45.455704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.941407+0000","last_change":"2024-04-12T21:14:23.941407+0000","last_active":"2024-04-12T21:14:23.941407+0000","last_peered":"2024-04-12T21:14:23.941407+0000","last_clean":"2024-04-12T21:14:23.941407+0000","last_became_active":"2024-04-12T21:14:23.940906+0000","last_became_peered":"2024-04-12T21:14:23.940906+0000","last_unstale":"2024-04-12T21:14:23.941407+0000","last_undegraded":"2024-04-12T21:14:23.941407+0000","last_fullsized":"2024-04-12T21:14:23.941407+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:49:30.433604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.502757+0000","last_change":"2024-04-12T21:13:12.539083+0000","last_active":"2024-04-12T21:14:23.502757+0000","last_peered":"2024-04-12T21:14:23.502757+0000","last_clean":"2024-04-12T21:14:23.502757+0000","last_became_active":"2024-04-12T21:13:12.538771+0000","last_became_peered":"2024-04-12T21:13:12.538771+0000","last_unstale":"2024-04-12T21:14:23.502757+0000","last_undegraded":"2024-04-12T21:14:23.502757+0000","last_fullsized":"2024-04-12T21:14:23.502757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:31.560665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.344062+0000","last_change":"2024-04-12T21:14:23.507246+0000","last_active":"2024-04-12T21:14:52.344062+0000","last_peered":"2024-04-12T21:14:52.344062+0000","last_clean":"2024-04-12T21:14:52.344062+0000","last_became_active":"2024-04-12T21:14:23.506815+0000","last_became_peered":"2024-04-12T21:14:23.506815+0000","last_unstale":"2024-04-12T21:14:52.344062+0000","last_undegraded":"2024-04-12T21:14:52.344062+0000","last_fullsized":"2024-04-12T21:14:52.344062+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:02:55.930713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500131+0000","last_change":"2024-04-12T21:13:17.607603+0000","last_active":"2024-04-12T21:14:23.500131+0000","last_peered":"2024-04-12T21:14:23.500131+0000","last_clean":"2024-04-12T21:14:23.500131+0000","last_became_active":"2024-04-12T21:13:17.606972+0000","last_became_peered":"2024-04-12T21:13:17.606972+0000","last_unstale":"2024-04-12T21:14:23.500131+0000","last_undegraded":"2024-04-12T21:14:23.500131+0000","last_fullsized":"2024-04-12T21:14:23.500131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:16:37.593158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938475+0000","last_change":"2024-04-12T21:13:12.541405+0000","last_active":"2024-04-12T21:14:23.938475+0000","last_peered":"2024-04-12T21:14:23.938475+0000","last_clean":"2024-04-12T21:14:23.938475+0000","last_became_active":"2024-04-12T21:13:12.540612+0000","last_became_peered":"2024-04-12T21:13:12.540612+0000","last_unstale":"2024-04-12T21:14:23.938475+0000","last_undegraded":"2024-04-12T21:14:23.938475+0000","last_fullsized":"2024-04-12T21:14:23.938475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:38:10.749857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.361409+0000","last_change":"2024-04-12T21:13:13.138447+0000","last_active":"2024-04-12T21:14:52.361409+0000","last_peered":"2024-04-12T21:14:52.361409+0000","last_clean":"2024-04-12T21:14:52.361409+0000","last_became_active":"2024-04-12T21:13:13.137990+0000","last_became_peered":"2024-04-12T21:13:13.137990+0000","last_unstale":"2024-04-12T21:14:52.361409+0000","last_undegraded":"2024-04-12T21:14:52.361409+0000","last_fullsized":"2024-04-12T21:14:52.361409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:58:50.381257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.653459+0000","last_change":"2024-04-12T21:13:17.614003+0000","last_active":"2024-04-12T21:14:23.653459+0000","last_peered":"2024-04-12T21:14:23.653459+0000","last_clean":"2024-04-12T21:14:23.653459+0000","last_became_active":"2024-04-12T21:13:17.613517+0000","last_became_peered":"2024-04-12T21:13:17.613517+0000","last_unstale":"2024-04-12T21:14:23.653459+0000","last_undegraded":"2024-04-12T21:14:23.653459+0000","last_fullsized":"2024-04-12T21:14:23.653459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:33.590816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501831+0000","last_change":"2024-04-12T21:13:12.530503+0000","last_active":"2024-04-12T21:14:23.501831+0000","last_peered":"2024-04-12T21:14:23.501831+0000","last_clean":"2024-04-12T21:14:23.501831+0000","last_became_active":"2024-04-12T21:13:12.530161+0000","last_became_peered":"2024-04-12T21:13:12.530161+0000","last_unstale":"2024-04-12T21:14:23.501831+0000","last_undegraded":"2024-04-12T21:14:23.501831+0000","last_fullsized":"2024-04-12T21:14:23.501831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:06:18.363283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.357391+0000","last_change":"2024-04-12T21:13:13.146401+0000","last_active":"2024-04-12T21:14:52.357391+0000","last_peered":"2024-04-12T21:14:52.357391+0000","last_clean":"2024-04-12T21:14:52.357391+0000","last_became_active":"2024-04-12T21:13:13.145527+0000","last_became_peered":"2024-04-12T21:13:13.145527+0000","last_unstale":"2024-04-12T21:14:52.357391+0000","last_undegraded":"2024-04-12T21:14:52.357391+0000","last_fullsized":"2024-04-12T21:14:52.357391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:40:41.302305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498492+0000","last_change":"2024-04-12T21:13:17.609141+0000","last_active":"2024-04-12T21:14:23.498492+0000","last_peered":"2024-04-12T21:14:23.498492+0000","last_clean":"2024-04-12T21:14:23.498492+0000","last_became_active":"2024-04-12T21:13:17.603368+0000","last_became_peered":"2024-04-12T21:13:17.603368+0000","last_unstale":"2024-04-12T21:14:23.498492+0000","last_undegraded":"2024-04-12T21:14:23.498492+0000","last_fullsized":"2024-04-12T21:14:23.498492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:07:54.235284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504591+0000","last_change":"2024-04-12T21:13:12.540388+0000","last_active":"2024-04-12T21:14:23.504591+0000","last_peered":"2024-04-12T21:14:23.504591+0000","last_clean":"2024-04-12T21:14:23.504591+0000","last_became_active":"2024-04-12T21:13:12.540033+0000","last_became_peered":"2024-04-12T21:13:12.540033+0000","last_unstale":"2024-04-12T21:14:23.504591+0000","last_undegraded":"2024-04-12T21:14:23.504591+0000","last_fullsized":"2024-04-12T21:14:23.504591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:46:53.785074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.349620+0000","last_change":"2024-04-12T21:13:13.137208+0000","last_active":"2024-04-12T21:14:52.349620+0000","last_peered":"2024-04-12T21:14:52.349620+0000","last_clean":"2024-04-12T21:14:52.349620+0000","last_became_active":"2024-04-12T21:13:13.136866+0000","last_became_peered":"2024-04-12T21:13:13.136866+0000","last_unstale":"2024-04-12T21:14:52.349620+0000","last_undegraded":"2024-04-12T21:14:52.349620+0000","last_fullsized":"2024-04-12T21:14:52.349620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:48:36.655808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501219+0000","last_change":"2024-04-12T21:13:17.617564+0000","last_active":"2024-04-12T21:14:23.501219+0000","last_peered":"2024-04-12T21:14:23.501219+0000","last_clean":"2024-04-12T21:14:23.501219+0000","last_became_active":"2024-04-12T21:13:17.615980+0000","last_became_peered":"2024-04-12T21:13:17.615980+0000","last_unstale":"2024-04-12T21:14:23.501219+0000","last_undegraded":"2024-04-12T21:14:23.501219+0000","last_fullsized":"2024-04-12T21:14:23.501219+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:22:38.370357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.509734+0000","last_change":"2024-04-12T21:14:23.509734+0000","last_active":"2024-04-12T21:14:23.509734+0000","last_peered":"2024-04-12T21:14:23.509734+0000","last_clean":"2024-04-12T21:14:23.509734+0000","last_became_active":"2024-04-12T21:14:23.509186+0000","last_became_peered":"2024-04-12T21:14:23.509186+0000","last_unstale":"2024-04-12T21:14:23.509734+0000","last_undegraded":"2024-04-12T21:14:23.509734+0000","last_fullsized":"2024-04-12T21:14:23.509734+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:51:37.972583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364072+0000","last_change":"2024-04-12T21:13:13.152737+0000","last_active":"2024-04-12T21:14:52.364072+0000","last_peered":"2024-04-12T21:14:52.364072+0000","last_clean":"2024-04-12T21:14:52.364072+0000","last_became_active":"2024-04-12T21:13:13.150914+0000","last_became_peered":"2024-04-12T21:13:13.150914+0000","last_unstale":"2024-04-12T21:14:52.364072+0000","last_undegraded":"2024-04-12T21:14:52.364072+0000","last_fullsized":"2024-04-12T21:14:52.364072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:51:29.020746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503683+0000","last_change":"2024-04-12T21:13:17.604826+0000","last_active":"2024-04-12T21:14:23.503683+0000","last_peered":"2024-04-12T21:14:23.503683+0000","last_clean":"2024-04-12T21:14:23.503683+0000","last_became_active":"2024-04-12T21:13:17.604427+0000","last_became_peered":"2024-04-12T21:13:17.604427+0000","last_unstale":"2024-04-12T21:14:23.503683+0000","last_undegraded":"2024-04-12T21:14:23.503683+0000","last_fullsized":"2024-04-12T21:14:23.503683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:04:55.602539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503221+0000","last_change":"2024-04-12T21:13:12.542641+0000","last_active":"2024-04-12T21:14:23.503221+0000","last_peered":"2024-04-12T21:14:23.503221+0000","last_clean":"2024-04-12T21:14:23.503221+0000","last_became_active":"2024-04-12T21:13:12.542291+0000","last_became_peered":"2024-04-12T21:13:12.542291+0000","last_unstale":"2024-04-12T21:14:23.503221+0000","last_undegraded":"2024-04-12T21:14:23.503221+0000","last_fullsized":"2024-04-12T21:14:23.503221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:37:00.186153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.348538+0000","last_change":"2024-04-12T21:14:23.941262+0000","last_active":"2024-04-12T21:14:52.348538+0000","last_peered":"2024-04-12T21:14:52.348538+0000","last_clean":"2024-04-12T21:14:52.348538+0000","last_became_active":"2024-04-12T21:14:23.940721+0000","last_became_peered":"2024-04-12T21:14:23.940721+0000","last_unstale":"2024-04-12T21:14:52.348538+0000","last_undegraded":"2024-04-12T21:14:52.348538+0000","last_fullsized":"2024-04-12T21:14:52.348538+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:07:13.835188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503151+0000","last_change":"2024-04-12T21:13:17.606533+0000","last_active":"2024-04-12T21:14:23.503151+0000","last_peered":"2024-04-12T21:14:23.503151+0000","last_clean":"2024-04-12T21:14:23.503151+0000","last_became_active":"2024-04-12T21:13:17.606092+0000","last_became_peered":"2024-04-12T21:13:17.606092+0000","last_unstale":"2024-04-12T21:14:23.503151+0000","last_undegraded":"2024-04-12T21:14:23.503151+0000","last_fullsized":"2024-04-12T21:14:23.503151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:55:56.080659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'33","reported_seq":237,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:15:00.261482+0000","last_change":"2024-04-12T21:13:12.531797+0000","last_active":"2024-04-12T21:15:00.261482+0000","last_peered":"2024-04-12T21:15:00.261482+0000","last_clean":"2024-04-12T21:15:00.261482+0000","last_became_active":"2024-04-12T21:13:12.531346+0000","last_became_peered":"2024-04-12T21:13:12.531346+0000","last_unstale":"2024-04-12T21:15:00.261482+0000","last_undegraded":"2024-04-12T21:15:00.261482+0000","last_fullsized":"2024-04-12T21:15:00.261482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:59:32.429211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62467,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":177,"num_read_kb":150,"num_write":33,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.333798+0000","last_change":"2024-04-12T21:13:13.131501+0000","last_active":"2024-04-12T21:14:52.333798+0000","last_peered":"2024-04-12T21:14:52.333798+0000","last_clean":"2024-04-12T21:14:52.333798+0000","last_became_active":"2024-04-12T21:13:13.131161+0000","last_became_peered":"2024-04-12T21:13:13.131161+0000","last_unstale":"2024-04-12T21:14:52.333798+0000","last_undegraded":"2024-04-12T21:14:52.333798+0000","last_fullsized":"2024-04-12T21:14:52.333798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:45:27.046250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.942111+0000","last_change":"2024-04-12T21:14:23.941552+0000","last_active":"2024-04-12T21:14:23.942111+0000","last_peered":"2024-04-12T21:14:23.942111+0000","last_clean":"2024-04-12T21:14:23.942111+0000","last_became_active":"2024-04-12T21:14:23.940998+0000","last_became_peered":"2024-04-12T21:14:23.940998+0000","last_unstale":"2024-04-12T21:14:23.942111+0000","last_undegraded":"2024-04-12T21:14:23.942111+0000","last_fullsized":"2024-04-12T21:14:23.942111+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:34:04.055211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.514794+0000","last_change":"2024-04-12T21:14:23.514343+0000","last_active":"2024-04-12T21:14:23.514794+0000","last_peered":"2024-04-12T21:14:23.514794+0000","last_clean":"2024-04-12T21:14:23.514794+0000","last_became_active":"2024-04-12T21:14:23.505690+0000","last_became_peered":"2024-04-12T21:14:23.505690+0000","last_unstale":"2024-04-12T21:14:23.514794+0000","last_undegraded":"2024-04-12T21:14:23.514794+0000","last_fullsized":"2024-04-12T21:14:23.514794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:24:12.746487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.341334+0000","last_change":"2024-04-12T21:13:13.150100+0000","last_active":"2024-04-12T21:14:52.341334+0000","last_peered":"2024-04-12T21:14:52.341334+0000","last_clean":"2024-04-12T21:14:52.341334+0000","last_became_active":"2024-04-12T21:13:13.149711+0000","last_became_peered":"2024-04-12T21:13:13.149711+0000","last_unstale":"2024-04-12T21:14:52.341334+0000","last_undegraded":"2024-04-12T21:14:52.341334+0000","last_fullsized":"2024-04-12T21:14:52.341334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:33:27.121682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.941361+0000","last_change":"2024-04-12T21:14:23.941361+0000","last_active":"2024-04-12T21:14:23.941361+0000","last_peered":"2024-04-12T21:14:23.941361+0000","last_clean":"2024-04-12T21:14:23.941361+0000","last_became_active":"2024-04-12T21:14:23.940817+0000","last_became_peered":"2024-04-12T21:14:23.940817+0000","last_unstale":"2024-04-12T21:14:23.941361+0000","last_undegraded":"2024-04-12T21:14:23.941361+0000","last_fullsized":"2024-04-12T21:14:23.941361+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:52:18.459215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.502514+0000","last_change":"2024-04-12T21:13:12.545077+0000","last_active":"2024-04-12T21:14:23.502514+0000","last_peered":"2024-04-12T21:14:23.502514+0000","last_clean":"2024-04-12T21:14:23.502514+0000","last_became_active":"2024-04-12T21:13:12.544755+0000","last_became_peered":"2024-04-12T21:13:12.544755+0000","last_unstale":"2024-04-12T21:14:23.502514+0000","last_undegraded":"2024-04-12T21:14:23.502514+0000","last_fullsized":"2024-04-12T21:14:23.502514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:31:28.487336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.347339+0000","last_change":"2024-04-12T21:13:13.128047+0000","last_active":"2024-04-12T21:14:52.347339+0000","last_peered":"2024-04-12T21:14:52.347339+0000","last_clean":"2024-04-12T21:14:52.347339+0000","last_became_active":"2024-04-12T21:13:13.127692+0000","last_became_peered":"2024-04-12T21:13:13.127692+0000","last_unstale":"2024-04-12T21:14:52.347339+0000","last_undegraded":"2024-04-12T21:14:52.347339+0000","last_fullsized":"2024-04-12T21:14:52.347339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:43:29.552438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506489+0000","last_change":"2024-04-12T21:13:17.614240+0000","last_active":"2024-04-12T21:14:23.506489+0000","last_peered":"2024-04-12T21:14:23.506489+0000","last_clean":"2024-04-12T21:14:23.506489+0000","last_became_active":"2024-04-12T21:13:17.613834+0000","last_became_peered":"2024-04-12T21:13:17.613834+0000","last_unstale":"2024-04-12T21:14:23.506489+0000","last_undegraded":"2024-04-12T21:14:23.506489+0000","last_fullsized":"2024-04-12T21:14:23.506489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:29:23.949101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503750+0000","last_change":"2024-04-12T21:13:12.527868+0000","last_active":"2024-04-12T21:14:23.503750+0000","last_peered":"2024-04-12T21:14:23.503750+0000","last_clean":"2024-04-12T21:14:23.503750+0000","last_became_active":"2024-04-12T21:13:12.527521+0000","last_became_peered":"2024-04-12T21:13:12.527521+0000","last_unstale":"2024-04-12T21:14:23.503750+0000","last_undegraded":"2024-04-12T21:14:23.503750+0000","last_fullsized":"2024-04-12T21:14:23.503750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:46:23.649552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.339504+0000","last_change":"2024-04-12T21:13:13.124207+0000","last_active":"2024-04-12T21:14:52.339504+0000","last_peered":"2024-04-12T21:14:52.339504+0000","last_clean":"2024-04-12T21:14:52.339504+0000","last_became_active":"2024-04-12T21:13:13.123866+0000","last_became_peered":"2024-04-12T21:13:13.123866+0000","last_unstale":"2024-04-12T21:14:52.339504+0000","last_undegraded":"2024-04-12T21:14:52.339504+0000","last_fullsized":"2024-04-12T21:14:52.339504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:04.263260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938143+0000","last_change":"2024-04-12T21:13:17.613064+0000","last_active":"2024-04-12T21:14:23.938143+0000","last_peered":"2024-04-12T21:14:23.938143+0000","last_clean":"2024-04-12T21:14:23.938143+0000","last_became_active":"2024-04-12T21:13:17.612569+0000","last_became_peered":"2024-04-12T21:13:17.612569+0000","last_unstale":"2024-04-12T21:14:23.938143+0000","last_undegraded":"2024-04-12T21:14:23.938143+0000","last_fullsized":"2024-04-12T21:14:23.938143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:35:27.274086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503107+0000","last_change":"2024-04-12T21:13:12.543193+0000","last_active":"2024-04-12T21:14:23.503107+0000","last_peered":"2024-04-12T21:14:23.503107+0000","last_clean":"2024-04-12T21:14:23.503107+0000","last_became_active":"2024-04-12T21:13:12.542885+0000","last_became_peered":"2024-04-12T21:13:12.542885+0000","last_unstale":"2024-04-12T21:14:23.503107+0000","last_undegraded":"2024-04-12T21:14:23.503107+0000","last_fullsized":"2024-04-12T21:14:23.503107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:02:11.100366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337710+0000","last_change":"2024-04-12T21:13:13.130682+0000","last_active":"2024-04-12T21:14:52.337710+0000","last_peered":"2024-04-12T21:14:52.337710+0000","last_clean":"2024-04-12T21:14:52.337710+0000","last_became_active":"2024-04-12T21:13:13.130299+0000","last_became_peered":"2024-04-12T21:13:13.130299+0000","last_unstale":"2024-04-12T21:14:52.337710+0000","last_undegraded":"2024-04-12T21:14:52.337710+0000","last_fullsized":"2024-04-12T21:14:52.337710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:52:02.094959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501865+0000","last_change":"2024-04-12T21:13:17.609472+0000","last_active":"2024-04-12T21:14:23.501865+0000","last_peered":"2024-04-12T21:14:23.501865+0000","last_clean":"2024-04-12T21:14:23.501865+0000","last_became_active":"2024-04-12T21:13:17.608991+0000","last_became_peered":"2024-04-12T21:13:17.608991+0000","last_unstale":"2024-04-12T21:14:23.501865+0000","last_undegraded":"2024-04-12T21:14:23.501865+0000","last_fullsized":"2024-04-12T21:14:23.501865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:17:57.962701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503725+0000","last_change":"2024-04-12T21:13:12.530058+0000","last_active":"2024-04-12T21:14:23.503725+0000","last_peered":"2024-04-12T21:14:23.503725+0000","last_clean":"2024-04-12T21:14:23.503725+0000","last_became_active":"2024-04-12T21:13:12.529682+0000","last_became_peered":"2024-04-12T21:13:12.529682+0000","last_unstale":"2024-04-12T21:14:23.503725+0000","last_undegraded":"2024-04-12T21:14:23.503725+0000","last_fullsized":"2024-04-12T21:14:23.503725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:17:25.621593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337223+0000","last_change":"2024-04-12T21:13:13.129975+0000","last_active":"2024-04-12T21:14:52.337223+0000","last_peered":"2024-04-12T21:14:52.337223+0000","last_clean":"2024-04-12T21:14:52.337223+0000","last_became_active":"2024-04-12T21:13:13.129621+0000","last_became_peered":"2024-04-12T21:13:13.129621+0000","last_unstale":"2024-04-12T21:14:52.337223+0000","last_undegraded":"2024-04-12T21:14:52.337223+0000","last_fullsized":"2024-04-12T21:14:52.337223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:49:48.267685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504102+0000","last_change":"2024-04-12T21:13:17.738019+0000","last_active":"2024-04-12T21:14:23.504102+0000","last_peered":"2024-04-12T21:14:23.504102+0000","last_clean":"2024-04-12T21:14:23.504102+0000","last_became_active":"2024-04-12T21:13:17.737553+0000","last_became_peered":"2024-04-12T21:13:17.737553+0000","last_unstale":"2024-04-12T21:14:23.504102+0000","last_undegraded":"2024-04-12T21:14:23.504102+0000","last_fullsized":"2024-04-12T21:14:23.504102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:29:07.288025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.505748+0000","last_change":"2024-04-12T21:13:12.544661+0000","last_active":"2024-04-12T21:14:23.505748+0000","last_peered":"2024-04-12T21:14:23.505748+0000","last_clean":"2024-04-12T21:14:23.505748+0000","last_became_active":"2024-04-12T21:13:12.544321+0000","last_became_peered":"2024-04-12T21:13:12.544321+0000","last_unstale":"2024-04-12T21:14:23.505748+0000","last_undegraded":"2024-04-12T21:14:23.505748+0000","last_fullsized":"2024-04-12T21:14:23.505748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:55:43.517970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.358349+0000","last_change":"2024-04-12T21:13:13.140480+0000","last_active":"2024-04-12T21:14:52.358349+0000","last_peered":"2024-04-12T21:14:52.358349+0000","last_clean":"2024-04-12T21:14:52.358349+0000","last_became_active":"2024-04-12T21:13:13.140128+0000","last_became_peered":"2024-04-12T21:13:13.140128+0000","last_unstale":"2024-04-12T21:14:52.358349+0000","last_undegraded":"2024-04-12T21:14:52.358349+0000","last_fullsized":"2024-04-12T21:14:52.358349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:40:32.066929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938876+0000","last_change":"2024-04-12T21:13:17.609235+0000","last_active":"2024-04-12T21:14:23.938876+0000","last_peered":"2024-04-12T21:14:23.938876+0000","last_clean":"2024-04-12T21:14:23.938876+0000","last_became_active":"2024-04-12T21:13:17.608814+0000","last_became_peered":"2024-04-12T21:13:17.608814+0000","last_unstale":"2024-04-12T21:14:23.938876+0000","last_undegraded":"2024-04-12T21:14:23.938876+0000","last_fullsized":"2024-04-12T21:14:23.938876+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:15:35.287239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500807+0000","last_change":"2024-04-12T21:13:12.539638+0000","last_active":"2024-04-12T21:14:23.500807+0000","last_peered":"2024-04-12T21:14:23.500807+0000","last_clean":"2024-04-12T21:14:23.500807+0000","last_became_active":"2024-04-12T21:13:12.539272+0000","last_became_peered":"2024-04-12T21:13:12.539272+0000","last_unstale":"2024-04-12T21:14:23.500807+0000","last_undegraded":"2024-04-12T21:14:23.500807+0000","last_fullsized":"2024-04-12T21:14:23.500807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:11:42.992489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.352009+0000","last_change":"2024-04-12T21:13:13.149326+0000","last_active":"2024-04-12T21:14:52.352009+0000","last_peered":"2024-04-12T21:14:52.352009+0000","last_clean":"2024-04-12T21:14:52.352009+0000","last_became_active":"2024-04-12T21:13:13.149023+0000","last_became_peered":"2024-04-12T21:13:13.149023+0000","last_unstale":"2024-04-12T21:14:52.352009+0000","last_undegraded":"2024-04-12T21:14:52.352009+0000","last_fullsized":"2024-04-12T21:14:52.352009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:15:58.479778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938430+0000","last_change":"2024-04-12T21:13:17.740067+0000","last_active":"2024-04-12T21:14:23.938430+0000","last_peered":"2024-04-12T21:14:23.938430+0000","last_clean":"2024-04-12T21:14:23.938430+0000","last_became_active":"2024-04-12T21:13:17.738186+0000","last_became_peered":"2024-04-12T21:13:17.738186+0000","last_unstale":"2024-04-12T21:14:23.938430+0000","last_undegraded":"2024-04-12T21:14:23.938430+0000","last_fullsized":"2024-04-12T21:14:23.938430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:15:05.241855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.333729+0000","last_change":"2024-04-12T21:14:23.509373+0000","last_active":"2024-04-12T21:14:52.333729+0000","last_peered":"2024-04-12T21:14:52.333729+0000","last_clean":"2024-04-12T21:14:52.333729+0000","last_became_active":"2024-04-12T21:14:23.508595+0000","last_became_peered":"2024-04-12T21:14:23.508595+0000","last_unstale":"2024-04-12T21:14:52.333729+0000","last_undegraded":"2024-04-12T21:14:52.333729+0000","last_fullsized":"2024-04-12T21:14:52.333729+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:45:49.388383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938366+0000","last_change":"2024-04-12T21:13:12.551549+0000","last_active":"2024-04-12T21:14:23.938366+0000","last_peered":"2024-04-12T21:14:23.938366+0000","last_clean":"2024-04-12T21:14:23.938366+0000","last_became_active":"2024-04-12T21:13:12.551204+0000","last_became_peered":"2024-04-12T21:13:12.551204+0000","last_unstale":"2024-04-12T21:14:23.938366+0000","last_undegraded":"2024-04-12T21:14:23.938366+0000","last_fullsized":"2024-04-12T21:14:23.938366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:33:06.721971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501481+0000","last_change":"2024-04-12T21:13:17.603828+0000","last_active":"2024-04-12T21:14:23.501481+0000","last_peered":"2024-04-12T21:14:23.501481+0000","last_clean":"2024-04-12T21:14:23.501481+0000","last_became_active":"2024-04-12T21:13:17.603399+0000","last_became_peered":"2024-04-12T21:13:17.603399+0000","last_unstale":"2024-04-12T21:14:23.501481+0000","last_undegraded":"2024-04-12T21:14:23.501481+0000","last_fullsized":"2024-04-12T21:14:23.501481+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:22:28.126722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.357541+0000","last_change":"2024-04-12T21:13:13.137828+0000","last_active":"2024-04-12T21:14:52.357541+0000","last_peered":"2024-04-12T21:14:52.357541+0000","last_clean":"2024-04-12T21:14:52.357541+0000","last_became_active":"2024-04-12T21:13:13.137534+0000","last_became_peered":"2024-04-12T21:13:13.137534+0000","last_unstale":"2024-04-12T21:14:52.357541+0000","last_undegraded":"2024-04-12T21:14:52.357541+0000","last_fullsized":"2024-04-12T21:14:52.357541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:24:20.238210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503628+0000","last_change":"2024-04-12T21:13:12.533215+0000","last_active":"2024-04-12T21:14:23.503628+0000","last_peered":"2024-04-12T21:14:23.503628+0000","last_clean":"2024-04-12T21:14:23.503628+0000","last_became_active":"2024-04-12T21:13:12.532852+0000","last_became_peered":"2024-04-12T21:13:12.532852+0000","last_unstale":"2024-04-12T21:14:23.503628+0000","last_undegraded":"2024-04-12T21:14:23.503628+0000","last_fullsized":"2024-04-12T21:14:23.503628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:41:17.861406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500416+0000","last_change":"2024-04-12T21:11:53.092756+0000","last_active":"2024-04-12T21:14:23.500416+0000","last_peered":"2024-04-12T21:14:23.500416+0000","last_clean":"2024-04-12T21:14:23.500416+0000","last_became_active":"2024-04-12T21:11:53.092364+0000","last_became_peered":"2024-04-12T21:11:53.092364+0000","last_unstale":"2024-04-12T21:14:23.500416+0000","last_undegraded":"2024-04-12T21:14:23.500416+0000","last_fullsized":"2024-04-12T21:14:23.500416+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:08:04.402338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:08:04.402338+0000","last_clean_scrub_stamp":"2024-04-12T21:08:04.402338+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:26:11.910015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.502584+0000","last_change":"2024-04-12T21:13:17.738168+0000","last_active":"2024-04-12T21:14:23.502584+0000","last_peered":"2024-04-12T21:14:23.502584+0000","last_clean":"2024-04-12T21:14:23.502584+0000","last_became_active":"2024-04-12T21:13:17.737697+0000","last_became_peered":"2024-04-12T21:13:17.737697+0000","last_unstale":"2024-04-12T21:14:23.502584+0000","last_undegraded":"2024-04-12T21:14:23.502584+0000","last_fullsized":"2024-04-12T21:14:23.502584+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:11:31.025636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.323212+0000","last_change":"2024-04-12T21:13:13.130398+0000","last_active":"2024-04-12T21:14:52.323212+0000","last_peered":"2024-04-12T21:14:52.323212+0000","last_clean":"2024-04-12T21:14:52.323212+0000","last_became_active":"2024-04-12T21:13:13.129488+0000","last_became_peered":"2024-04-12T21:13:13.129488+0000","last_unstale":"2024-04-12T21:14:52.323212+0000","last_undegraded":"2024-04-12T21:14:52.323212+0000","last_fullsized":"2024-04-12T21:14:52.323212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:21:03.737140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500410+0000","last_change":"2024-04-12T21:13:12.533367+0000","last_active":"2024-04-12T21:14:23.500410+0000","last_peered":"2024-04-12T21:14:23.500410+0000","last_clean":"2024-04-12T21:14:23.500410+0000","last_became_active":"2024-04-12T21:13:12.533042+0000","last_became_peered":"2024-04-12T21:13:12.533042+0000","last_unstale":"2024-04-12T21:14:23.500410+0000","last_undegraded":"2024-04-12T21:14:23.500410+0000","last_fullsized":"2024-04-12T21:14:23.500410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:56:07.671693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500591+0000","last_change":"2024-04-12T21:13:17.593951+0000","last_active":"2024-04-12T21:14:23.500591+0000","last_peered":"2024-04-12T21:14:23.500591+0000","last_clean":"2024-04-12T21:14:23.500591+0000","last_became_active":"2024-04-12T21:13:17.593528+0000","last_became_peered":"2024-04-12T21:13:17.593528+0000","last_unstale":"2024-04-12T21:14:23.500591+0000","last_undegraded":"2024-04-12T21:14:23.500591+0000","last_fullsized":"2024-04-12T21:14:23.500591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:24:25.081109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504157+0000","last_change":"2024-04-12T21:13:12.543040+0000","last_active":"2024-04-12T21:14:23.504157+0000","last_peered":"2024-04-12T21:14:23.504157+0000","last_clean":"2024-04-12T21:14:23.504157+0000","last_became_active":"2024-04-12T21:13:12.542698+0000","last_became_peered":"2024-04-12T21:13:12.542698+0000","last_unstale":"2024-04-12T21:14:23.504157+0000","last_undegraded":"2024-04-12T21:14:23.504157+0000","last_fullsized":"2024-04-12T21:14:23.504157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:57:55.712648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.335923+0000","last_change":"2024-04-12T21:13:13.138975+0000","last_active":"2024-04-12T21:14:52.335923+0000","last_peered":"2024-04-12T21:14:52.335923+0000","last_clean":"2024-04-12T21:14:52.335923+0000","last_became_active":"2024-04-12T21:13:13.138538+0000","last_became_peered":"2024-04-12T21:13:13.138538+0000","last_unstale":"2024-04-12T21:14:52.335923+0000","last_undegraded":"2024-04-12T21:14:52.335923+0000","last_fullsized":"2024-04-12T21:14:52.335923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:12:10.488914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498341+0000","last_change":"2024-04-12T21:13:17.739156+0000","last_active":"2024-04-12T21:14:23.498341+0000","last_peered":"2024-04-12T21:14:23.498341+0000","last_clean":"2024-04-12T21:14:23.498341+0000","last_became_active":"2024-04-12T21:13:17.738153+0000","last_became_peered":"2024-04-12T21:13:17.738153+0000","last_unstale":"2024-04-12T21:14:23.498341+0000","last_undegraded":"2024-04-12T21:14:23.498341+0000","last_fullsized":"2024-04-12T21:14:23.498341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:18:18.930005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506504+0000","last_change":"2024-04-12T21:13:12.547658+0000","last_active":"2024-04-12T21:14:23.506504+0000","last_peered":"2024-04-12T21:14:23.506504+0000","last_clean":"2024-04-12T21:14:23.506504+0000","last_became_active":"2024-04-12T21:13:12.547197+0000","last_became_peered":"2024-04-12T21:13:12.547197+0000","last_unstale":"2024-04-12T21:14:23.506504+0000","last_undegraded":"2024-04-12T21:14:23.506504+0000","last_fullsized":"2024-04-12T21:14:23.506504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:57:18.066730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.356304+0000","last_change":"2024-04-12T21:13:13.143876+0000","last_active":"2024-04-12T21:14:52.356304+0000","last_peered":"2024-04-12T21:14:52.356304+0000","last_clean":"2024-04-12T21:14:52.356304+0000","last_became_active":"2024-04-12T21:13:13.140389+0000","last_became_peered":"2024-04-12T21:13:13.140389+0000","last_unstale":"2024-04-12T21:14:52.356304+0000","last_undegraded":"2024-04-12T21:14:52.356304+0000","last_fullsized":"2024-04-12T21:14:52.356304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:36:13.910978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.497556+0000","last_change":"2024-04-12T21:13:17.592755+0000","last_active":"2024-04-12T21:14:23.497556+0000","last_peered":"2024-04-12T21:14:23.497556+0000","last_clean":"2024-04-12T21:14:23.497556+0000","last_became_active":"2024-04-12T21:13:17.592316+0000","last_became_peered":"2024-04-12T21:13:17.592316+0000","last_unstale":"2024-04-12T21:14:23.497556+0000","last_undegraded":"2024-04-12T21:14:23.497556+0000","last_fullsized":"2024-04-12T21:14:23.497556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:16:41.986623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504560+0000","last_change":"2024-04-12T21:13:12.538168+0000","last_active":"2024-04-12T21:14:23.504560+0000","last_peered":"2024-04-12T21:14:23.504560+0000","last_clean":"2024-04-12T21:14:23.504560+0000","last_became_active":"2024-04-12T21:13:12.537784+0000","last_became_peered":"2024-04-12T21:13:12.537784+0000","last_unstale":"2024-04-12T21:14:23.504560+0000","last_undegraded":"2024-04-12T21:14:23.504560+0000","last_fullsized":"2024-04-12T21:14:23.504560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:57:58.333171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.363050+0000","last_change":"2024-04-12T21:13:13.153230+0000","last_active":"2024-04-12T21:14:52.363050+0000","last_peered":"2024-04-12T21:14:52.363050+0000","last_clean":"2024-04-12T21:14:52.363050+0000","last_became_active":"2024-04-12T21:13:13.152795+0000","last_became_peered":"2024-04-12T21:13:13.152795+0000","last_unstale":"2024-04-12T21:14:52.363050+0000","last_undegraded":"2024-04-12T21:14:52.363050+0000","last_fullsized":"2024-04-12T21:14:52.363050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:11:21.058591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504718+0000","last_change":"2024-04-12T21:13:17.738428+0000","last_active":"2024-04-12T21:14:23.504718+0000","last_peered":"2024-04-12T21:14:23.504718+0000","last_clean":"2024-04-12T21:14:23.504718+0000","last_became_active":"2024-04-12T21:13:17.737932+0000","last_became_peered":"2024-04-12T21:13:17.737932+0000","last_unstale":"2024-04-12T21:14:23.504718+0000","last_undegraded":"2024-04-12T21:14:23.504718+0000","last_fullsized":"2024-04-12T21:14:23.504718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:22:26.310841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506739+0000","last_change":"2024-04-12T21:13:12.542689+0000","last_active":"2024-04-12T21:14:23.506739+0000","last_peered":"2024-04-12T21:14:23.506739+0000","last_clean":"2024-04-12T21:14:23.506739+0000","last_became_active":"2024-04-12T21:13:12.542371+0000","last_became_peered":"2024-04-12T21:13:12.542371+0000","last_unstale":"2024-04-12T21:14:23.506739+0000","last_undegraded":"2024-04-12T21:14:23.506739+0000","last_fullsized":"2024-04-12T21:14:23.506739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:08:48.439015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.302630+0000","last_change":"2024-04-12T21:13:13.130341+0000","last_active":"2024-04-12T21:14:52.302630+0000","last_peered":"2024-04-12T21:14:52.302630+0000","last_clean":"2024-04-12T21:14:52.302630+0000","last_became_active":"2024-04-12T21:13:13.129886+0000","last_became_peered":"2024-04-12T21:13:13.129886+0000","last_unstale":"2024-04-12T21:14:52.302630+0000","last_undegraded":"2024-04-12T21:14:52.302630+0000","last_fullsized":"2024-04-12T21:14:52.302630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:43:18.384216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506600+0000","last_change":"2024-04-12T21:13:17.615138+0000","last_active":"2024-04-12T21:14:23.506600+0000","last_peered":"2024-04-12T21:14:23.506600+0000","last_clean":"2024-04-12T21:14:23.506600+0000","last_became_active":"2024-04-12T21:13:17.614710+0000","last_became_peered":"2024-04-12T21:13:17.614710+0000","last_unstale":"2024-04-12T21:14:23.506600+0000","last_undegraded":"2024-04-12T21:14:23.506600+0000","last_fullsized":"2024-04-12T21:14:23.506600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:13:47.956600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498707+0000","last_change":"2024-04-12T21:13:12.546160+0000","last_active":"2024-04-12T21:14:23.498707+0000","last_peered":"2024-04-12T21:14:23.498707+0000","last_clean":"2024-04-12T21:14:23.498707+0000","last_became_active":"2024-04-12T21:13:12.544193+0000","last_became_peered":"2024-04-12T21:13:12.544193+0000","last_unstale":"2024-04-12T21:14:23.498707+0000","last_undegraded":"2024-04-12T21:14:23.498707+0000","last_fullsized":"2024-04-12T21:14:23.498707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:19:39.640814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.344337+0000","last_change":"2024-04-12T21:13:13.154604+0000","last_active":"2024-04-12T21:14:52.344337+0000","last_peered":"2024-04-12T21:14:52.344337+0000","last_clean":"2024-04-12T21:14:52.344337+0000","last_became_active":"2024-04-12T21:13:13.154257+0000","last_became_peered":"2024-04-12T21:13:13.154257+0000","last_unstale":"2024-04-12T21:14:52.344337+0000","last_undegraded":"2024-04-12T21:14:52.344337+0000","last_fullsized":"2024-04-12T21:14:52.344337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:31:45.550602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.517587+0000","last_change":"2024-04-12T21:14:23.517587+0000","last_active":"2024-04-12T21:14:23.517587+0000","last_peered":"2024-04-12T21:14:23.517587+0000","last_clean":"2024-04-12T21:14:23.517587+0000","last_became_active":"2024-04-12T21:14:23.507668+0000","last_became_peered":"2024-04-12T21:14:23.507668+0000","last_unstale":"2024-04-12T21:14:23.517587+0000","last_undegraded":"2024-04-12T21:14:23.517587+0000","last_fullsized":"2024-04-12T21:14:23.517587+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:31:05.648702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.497721+0000","last_change":"2024-04-12T21:13:12.536798+0000","last_active":"2024-04-12T21:14:23.497721+0000","last_peered":"2024-04-12T21:14:23.497721+0000","last_clean":"2024-04-12T21:14:23.497721+0000","last_became_active":"2024-04-12T21:13:12.536451+0000","last_became_peered":"2024-04-12T21:13:12.536451+0000","last_unstale":"2024-04-12T21:14:23.497721+0000","last_undegraded":"2024-04-12T21:14:23.497721+0000","last_fullsized":"2024-04-12T21:14:23.497721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:07:07.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364253+0000","last_change":"2024-04-12T21:13:13.154644+0000","last_active":"2024-04-12T21:14:52.364253+0000","last_peered":"2024-04-12T21:14:52.364253+0000","last_clean":"2024-04-12T21:14:52.364253+0000","last_became_active":"2024-04-12T21:13:13.154310+0000","last_became_peered":"2024-04-12T21:13:13.154310+0000","last_unstale":"2024-04-12T21:14:52.364253+0000","last_undegraded":"2024-04-12T21:14:52.364253+0000","last_fullsized":"2024-04-12T21:14:52.364253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:51:36.426506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503466+0000","last_change":"2024-04-12T21:13:17.737547+0000","last_active":"2024-04-12T21:14:23.503466+0000","last_peered":"2024-04-12T21:14:23.503466+0000","last_clean":"2024-04-12T21:14:23.503466+0000","last_became_active":"2024-04-12T21:13:17.737190+0000","last_became_peered":"2024-04-12T21:13:17.737190+0000","last_unstale":"2024-04-12T21:14:23.503466+0000","last_undegraded":"2024-04-12T21:14:23.503466+0000","last_fullsized":"2024-04-12T21:14:23.503466+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:23:52.221788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.510271+0000","last_change":"2024-04-12T21:14:23.509809+0000","last_active":"2024-04-12T21:14:23.510271+0000","last_peered":"2024-04-12T21:14:23.510271+0000","last_clean":"2024-04-12T21:14:23.510271+0000","last_became_active":"2024-04-12T21:14:23.505368+0000","last_became_peered":"2024-04-12T21:14:23.505368+0000","last_unstale":"2024-04-12T21:14:23.510271+0000","last_undegraded":"2024-04-12T21:14:23.510271+0000","last_fullsized":"2024-04-12T21:14:23.510271+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:49.529205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.348933+0000","last_change":"2024-04-12T21:13:13.128079+0000","last_active":"2024-04-12T21:14:52.348933+0000","last_peered":"2024-04-12T21:14:52.348933+0000","last_clean":"2024-04-12T21:14:52.348933+0000","last_became_active":"2024-04-12T21:13:13.127744+0000","last_became_peered":"2024-04-12T21:13:13.127744+0000","last_unstale":"2024-04-12T21:14:52.348933+0000","last_undegraded":"2024-04-12T21:14:52.348933+0000","last_fullsized":"2024-04-12T21:14:52.348933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:39:41.851323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506557+0000","last_change":"2024-04-12T21:13:17.596088+0000","last_active":"2024-04-12T21:14:23.506557+0000","last_peered":"2024-04-12T21:14:23.506557+0000","last_clean":"2024-04-12T21:14:23.506557+0000","last_became_active":"2024-04-12T21:13:17.595639+0000","last_became_peered":"2024-04-12T21:13:17.595639+0000","last_unstale":"2024-04-12T21:14:23.506557+0000","last_undegraded":"2024-04-12T21:14:23.506557+0000","last_fullsized":"2024-04-12T21:14:23.506557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:25:09.763945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498196+0000","last_change":"2024-04-12T21:13:12.543947+0000","last_active":"2024-04-12T21:14:23.498196+0000","last_peered":"2024-04-12T21:14:23.498196+0000","last_clean":"2024-04-12T21:14:23.498196+0000","last_became_active":"2024-04-12T21:13:12.543511+0000","last_became_peered":"2024-04-12T21:13:12.543511+0000","last_unstale":"2024-04-12T21:14:23.498196+0000","last_undegraded":"2024-04-12T21:14:23.498196+0000","last_fullsized":"2024-04-12T21:14:23.498196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:43:23.354835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.346426+0000","last_change":"2024-04-12T21:13:13.127178+0000","last_active":"2024-04-12T21:14:52.346426+0000","last_peered":"2024-04-12T21:14:52.346426+0000","last_clean":"2024-04-12T21:14:52.346426+0000","last_became_active":"2024-04-12T21:13:13.126856+0000","last_became_peered":"2024-04-12T21:13:13.126856+0000","last_unstale":"2024-04-12T21:14:52.346426+0000","last_undegraded":"2024-04-12T21:14:52.346426+0000","last_fullsized":"2024-04-12T21:14:52.346426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:09:34.068946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.652983+0000","last_change":"2024-04-12T21:13:17.619628+0000","last_active":"2024-04-12T21:14:23.652983+0000","last_peered":"2024-04-12T21:14:23.652983+0000","last_clean":"2024-04-12T21:14:23.652983+0000","last_became_active":"2024-04-12T21:13:17.619015+0000","last_became_peered":"2024-04-12T21:13:17.619015+0000","last_unstale":"2024-04-12T21:14:23.652983+0000","last_undegraded":"2024-04-12T21:14:23.652983+0000","last_fullsized":"2024-04-12T21:14:23.652983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:39:54.297165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503448+0000","last_change":"2024-04-12T21:13:12.538982+0000","last_active":"2024-04-12T21:14:23.503448+0000","last_peered":"2024-04-12T21:14:23.503448+0000","last_clean":"2024-04-12T21:14:23.503448+0000","last_became_active":"2024-04-12T21:13:12.538604+0000","last_became_peered":"2024-04-12T21:13:12.538604+0000","last_unstale":"2024-04-12T21:14:23.503448+0000","last_undegraded":"2024-04-12T21:14:23.503448+0000","last_fullsized":"2024-04-12T21:14:23.503448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:40:48.181188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.341305+0000","last_change":"2024-04-12T21:13:13.145537+0000","last_active":"2024-04-12T21:14:52.341305+0000","last_peered":"2024-04-12T21:14:52.341305+0000","last_clean":"2024-04-12T21:14:52.341305+0000","last_became_active":"2024-04-12T21:13:13.145283+0000","last_became_peered":"2024-04-12T21:13:13.145283+0000","last_unstale":"2024-04-12T21:14:52.341305+0000","last_undegraded":"2024-04-12T21:14:52.341305+0000","last_fullsized":"2024-04-12T21:14:52.341305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:26:29.339674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501477+0000","last_change":"2024-04-12T21:13:17.594394+0000","last_active":"2024-04-12T21:14:23.501477+0000","last_peered":"2024-04-12T21:14:23.501477+0000","last_clean":"2024-04-12T21:14:23.501477+0000","last_became_active":"2024-04-12T21:13:17.593960+0000","last_became_peered":"2024-04-12T21:13:17.593960+0000","last_unstale":"2024-04-12T21:14:23.501477+0000","last_undegraded":"2024-04-12T21:14:23.501477+0000","last_fullsized":"2024-04-12T21:14:23.501477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:00:53.462145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938259+0000","last_change":"2024-04-12T21:13:12.546385+0000","last_active":"2024-04-12T21:14:23.938259+0000","last_peered":"2024-04-12T21:14:23.938259+0000","last_clean":"2024-04-12T21:14:23.938259+0000","last_became_active":"2024-04-12T21:13:12.545902+0000","last_became_peered":"2024-04-12T21:13:12.545902+0000","last_unstale":"2024-04-12T21:14:23.938259+0000","last_undegraded":"2024-04-12T21:14:23.938259+0000","last_fullsized":"2024-04-12T21:14:23.938259+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:57:16.419641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.341095+0000","last_change":"2024-04-12T21:13:13.130478+0000","last_active":"2024-04-12T21:14:52.341095+0000","last_peered":"2024-04-12T21:14:52.341095+0000","last_clean":"2024-04-12T21:14:52.341095+0000","last_became_active":"2024-04-12T21:13:13.130164+0000","last_became_peered":"2024-04-12T21:13:13.130164+0000","last_unstale":"2024-04-12T21:14:52.341095+0000","last_undegraded":"2024-04-12T21:14:52.341095+0000","last_fullsized":"2024-04-12T21:14:52.341095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:22:29.186629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337126+0000","last_change":"2024-04-12T21:13:13.135406+0000","last_active":"2024-04-12T21:14:52.337126+0000","last_peered":"2024-04-12T21:14:52.337126+0000","last_clean":"2024-04-12T21:14:52.337126+0000","last_became_active":"2024-04-12T21:13:13.134840+0000","last_became_peered":"2024-04-12T21:13:13.134840+0000","last_unstale":"2024-04-12T21:14:52.337126+0000","last_undegraded":"2024-04-12T21:14:52.337126+0000","last_fullsized":"2024-04-12T21:14:52.337126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:22:22.315016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337956+0000","last_change":"2024-04-12T21:13:13.127249+0000","last_active":"2024-04-12T21:14:52.337956+0000","last_peered":"2024-04-12T21:14:52.337956+0000","last_clean":"2024-04-12T21:14:52.337956+0000","last_became_active":"2024-04-12T21:13:13.126901+0000","last_became_peered":"2024-04-12T21:13:13.126901+0000","last_unstale":"2024-04-12T21:14:52.337956+0000","last_undegraded":"2024-04-12T21:14:52.337956+0000","last_fullsized":"2024-04-12T21:14:52.337956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:32:18.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.352059+0000","last_change":"2024-04-12T21:13:13.151728+0000","last_active":"2024-04-12T21:14:52.352059+0000","last_peered":"2024-04-12T21:14:52.352059+0000","last_clean":"2024-04-12T21:14:52.352059+0000","last_became_active":"2024-04-12T21:13:13.151457+0000","last_became_peered":"2024-04-12T21:13:13.151457+0000","last_unstale":"2024-04-12T21:14:52.352059+0000","last_undegraded":"2024-04-12T21:14:52.352059+0000","last_fullsized":"2024-04-12T21:14:52.352059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:48:32.641671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.363440+0000","last_change":"2024-04-12T21:13:13.150109+0000","last_active":"2024-04-12T21:14:52.363440+0000","last_peered":"2024-04-12T21:14:52.363440+0000","last_clean":"2024-04-12T21:14:52.363440+0000","last_became_active":"2024-04-12T21:13:13.149132+0000","last_became_peered":"2024-04-12T21:13:13.149132+0000","last_unstale":"2024-04-12T21:14:52.363440+0000","last_undegraded":"2024-04-12T21:14:52.363440+0000","last_fullsized":"2024-04-12T21:14:52.363440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:36:00.062993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364228+0000","last_change":"2024-04-12T21:13:13.143405+0000","last_active":"2024-04-12T21:14:52.364228+0000","last_peered":"2024-04-12T21:14:52.364228+0000","last_clean":"2024-04-12T21:14:52.364228+0000","last_became_active":"2024-04-12T21:13:13.142973+0000","last_became_peered":"2024-04-12T21:13:13.142973+0000","last_unstale":"2024-04-12T21:14:52.364228+0000","last_undegraded":"2024-04-12T21:14:52.364228+0000","last_fullsized":"2024-04-12T21:14:52.364228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:10:03.106739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.334093+0000","last_change":"2024-04-12T21:13:13.129864+0000","last_active":"2024-04-12T21:14:52.334093+0000","last_peered":"2024-04-12T21:14:52.334093+0000","last_clean":"2024-04-12T21:14:52.334093+0000","last_became_active":"2024-04-12T21:13:13.129518+0000","last_became_peered":"2024-04-12T21:13:13.129518+0000","last_unstale":"2024-04-12T21:14:52.334093+0000","last_undegraded":"2024-04-12T21:14:52.334093+0000","last_fullsized":"2024-04-12T21:14:52.334093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:44:38.370002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.346620+0000","last_change":"2024-04-12T21:13:13.129153+0000","last_active":"2024-04-12T21:14:52.346620+0000","last_peered":"2024-04-12T21:14:52.346620+0000","last_clean":"2024-04-12T21:14:52.346620+0000","last_became_active":"2024-04-12T21:13:13.128799+0000","last_became_peered":"2024-04-12T21:13:13.128799+0000","last_unstale":"2024-04-12T21:14:52.346620+0000","last_undegraded":"2024-04-12T21:14:52.346620+0000","last_fullsized":"2024-04-12T21:14:52.346620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:37:51.675110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.354279+0000","last_change":"2024-04-12T21:13:13.144638+0000","last_active":"2024-04-12T21:14:52.354279+0000","last_peered":"2024-04-12T21:14:52.354279+0000","last_clean":"2024-04-12T21:14:52.354279+0000","last_became_active":"2024-04-12T21:13:13.144275+0000","last_became_peered":"2024-04-12T21:13:13.144275+0000","last_unstale":"2024-04-12T21:14:52.354279+0000","last_undegraded":"2024-04-12T21:14:52.354279+0000","last_fullsized":"2024-04-12T21:14:52.354279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:38:09.225551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.353660+0000","last_change":"2024-04-12T21:13:13.145845+0000","last_active":"2024-04-12T21:14:52.353660+0000","last_peered":"2024-04-12T21:14:52.353660+0000","last_clean":"2024-04-12T21:14:52.353660+0000","last_became_active":"2024-04-12T21:13:13.145399+0000","last_became_peered":"2024-04-12T21:13:13.145399+0000","last_unstale":"2024-04-12T21:14:52.353660+0000","last_undegraded":"2024-04-12T21:14:52.353660+0000","last_fullsized":"2024-04-12T21:14:52.353660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:07:13.075365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.354775+0000","last_change":"2024-04-12T21:13:13.138373+0000","last_active":"2024-04-12T21:14:52.354775+0000","last_peered":"2024-04-12T21:14:52.354775+0000","last_clean":"2024-04-12T21:14:52.354775+0000","last_became_active":"2024-04-12T21:13:13.137366+0000","last_became_peered":"2024-04-12T21:13:13.137366+0000","last_unstale":"2024-04-12T21:14:52.354775+0000","last_undegraded":"2024-04-12T21:14:52.354775+0000","last_fullsized":"2024-04-12T21:14:52.354775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:05:22.070677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.359799+0000","last_change":"2024-04-12T21:13:13.147902+0000","last_active":"2024-04-12T21:14:52.359799+0000","last_peered":"2024-04-12T21:14:52.359799+0000","last_clean":"2024-04-12T21:14:52.359799+0000","last_became_active":"2024-04-12T21:13:13.147513+0000","last_became_peered":"2024-04-12T21:13:13.147513+0000","last_unstale":"2024-04-12T21:14:52.359799+0000","last_undegraded":"2024-04-12T21:14:52.359799+0000","last_fullsized":"2024-04-12T21:14:52.359799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:43:34.228432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364585+0000","last_change":"2024-04-12T21:13:13.154703+0000","last_active":"2024-04-12T21:14:52.364585+0000","last_peered":"2024-04-12T21:14:52.364585+0000","last_clean":"2024-04-12T21:14:52.364585+0000","last_became_active":"2024-04-12T21:13:13.154340+0000","last_became_peered":"2024-04-12T21:13:13.154340+0000","last_unstale":"2024-04-12T21:14:52.364585+0000","last_undegraded":"2024-04-12T21:14:52.364585+0000","last_fullsized":"2024-04-12T21:14:52.364585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:29:05.500481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337426+0000","last_change":"2024-04-12T21:13:13.133441+0000","last_active":"2024-04-12T21:14:52.337426+0000","last_peered":"2024-04-12T21:14:52.337426+0000","last_clean":"2024-04-12T21:14:52.337426+0000","last_became_active":"2024-04-12T21:13:13.133113+0000","last_became_peered":"2024-04-12T21:13:13.133113+0000","last_unstale":"2024-04-12T21:14:52.337426+0000","last_undegraded":"2024-04-12T21:14:52.337426+0000","last_fullsized":"2024-04-12T21:14:52.337426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:05:36.924405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63265,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":181,"num_read_kb":154,"num_write":86,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":189795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":500283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645353,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1699840,"data_stored":1153278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":500831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070517,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":500853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1095295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":496153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":558620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":496243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":558620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":496257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":500271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1626112,"data_stored":1090799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T21:15:04.825 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph pg dump --format=json 2024-04-12T21:15:05.291 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:05.791 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:05 smithi081 ceph-mon[31453]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:15:05.791 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:05 smithi081 ceph-mon[31453]: pgmap v372: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:05 smithi153 ceph-mon[36569]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:15:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:05 smithi153 ceph-mon[36569]: pgmap v372: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:05 smithi104 ceph-mon[36624]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:15:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:05 smithi104 ceph-mon[36624]: pgmap v372: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.423+0000 7f853c200700 1 -- 172.21.15.81:0/292369715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f853410a010 msgr2=0x7f8534111be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.423+0000 7f853c200700 1 --2- 172.21.15.81:0/292369715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f853410a010 0x7f8534111be0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f852c00b0d0 tx=0x7f852c00b3e0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.423+0000 7f853c200700 1 -- 172.21.15.81:0/292369715 shutdown_connections 2024-04-12T21:15:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.423+0000 7f853c200700 1 --2- 172.21.15.81:0/292369715 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f853410a010 0x7f8534111be0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.423+0000 7f853c200700 1 --2- 172.21.15.81:0/292369715 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8534104f30 0x7f8534105310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.423+0000 7f853c200700 1 --2- 172.21.15.81:0/292369715 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85341058e0 0x7f8534109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.423+0000 7f853c200700 1 -- 172.21.15.81:0/292369715 >> 172.21.15.81:0/292369715 conn(0x7f85341007c0 msgr2=0x7f8534102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:07.420 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.423+0000 7f853c200700 1 -- 172.21.15.81:0/292369715 shutdown_connections 2024-04-12T21:15:07.421 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.424+0000 7f853c200700 1 -- 172.21.15.81:0/292369715 wait complete. 2024-04-12T21:15:07.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f853c200700 1 Processor -- start 2024-04-12T21:15:07.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f853c200700 1 -- start start 2024-04-12T21:15:07.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f853c200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8534104f30 0x7f853419b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:07.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f853c200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85341058e0 0x7f853419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f853c200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853410a010 0x7f853419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f853c200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8534115000 con 0x7f85341058e0 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f853c200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8534114e80 con 0x7f853410a010 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f853c200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8534115180 con 0x7f8534104f30 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f8538e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85341058e0 0x7f853419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f8539800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8534104f30 0x7f853419b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.425+0000 7f8538e00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85341058e0 0x7f853419b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38196/0 (socket says 172.21.15.81:38196) 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f8538e00700 1 -- 172.21.15.81:0/1889304254 learned_addr learned my addr 172.21.15.81:0/1889304254 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f853a200700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853410a010 0x7f853419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f8538e00700 1 -- 172.21.15.81:0/1889304254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8534104f30 msgr2=0x7f853419b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f8538e00700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8534104f30 0x7f853419b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f8538e00700 1 -- 172.21.15.81:0/1889304254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853410a010 msgr2=0x7f853419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:07.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f8538e00700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853410a010 0x7f853419fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f8538e00700 1 -- 172.21.15.81:0/1889304254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85341a03e0 con 0x7f85341058e0 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f8539800700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8534104f30 0x7f853419b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.426+0000 7f8538e00700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85341058e0 0x7f853419b910 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f852000e970 tx=0x7f852000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.427+0000 7f8532000700 1 -- 172.21.15.81:0/1889304254 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520004a30 con 0x7f85341058e0 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.427+0000 7f8532000700 1 -- 172.21.15.81:0/1889304254 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f852000ccc0 con 0x7f85341058e0 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.427+0000 7f8532000700 1 -- 172.21.15.81:0/1889304254 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520005650 con 0x7f85341058e0 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.427+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85341a06d0 con 0x7f85341058e0 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.427+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85341a7f50 con 0x7f85341058e0 2024-04-12T21:15:07.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.428+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85340697a0 con 0x7f85341058e0 2024-04-12T21:15:07.427 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.432+0000 7f8532000700 1 -- 172.21.15.81:0/1889304254 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8520004bd0 con 0x7f85341058e0 2024-04-12T21:15:07.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.433+0000 7f8532000700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f851c0a3cc0 0x7f851c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:07.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.433+0000 7f8532000700 1 -- 172.21.15.81:0/1889304254 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f85200d0df0 con 0x7f85341058e0 2024-04-12T21:15:07.429 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.434+0000 7f8539800700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f851c0a3cc0 0x7f851c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:07.430 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.434+0000 7f8539800700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f851c0a3cc0 0x7f851c0a6180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8528011490 tx=0x7f8528009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:07.431 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.436+0000 7f8532000700 1 -- 172.21.15.81:0/1889304254 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f852009d410 con 0x7f85341058e0 2024-04-12T21:15:07.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:07 smithi081 ceph-mon[31453]: pgmap v373: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:07.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.659+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f85341018f0 con 0x7f851c0a3cc0 2024-04-12T21:15:07.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.689+0000 7f8532000700 1 -- 172.21.15.81:0/1889304254 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340409 (secure 0 0 0) 0x7f85341018f0 con 0x7f851c0a3cc0 2024-04-12T21:15:07.691 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:07.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f851c0a3cc0 msgr2=0x7f851c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:07.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f851c0a3cc0 0x7f851c0a6180 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8528011490 tx=0x7f8528009260 comp rx=0 tx=0).stop 2024-04-12T21:15:07.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85341058e0 msgr2=0x7f853419b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:07.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85341058e0 0x7f853419b910 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f852000e970 tx=0x7f852000ec80 comp rx=0 tx=0).stop 2024-04-12T21:15:07.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 shutdown_connections 2024-04-12T21:15:07.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f851c0a3cc0 0x7f851c0a6180 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f853410a010 0x7f853419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8534104f30 0x7f853419b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 --2- 172.21.15.81:0/1889304254 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f85341058e0 0x7f853419b910 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:07.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.697+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 >> 172.21.15.81:0/1889304254 conn(0x7f85341007c0 msgr2=0x7f85341011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:07.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.698+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 shutdown_connections 2024-04-12T21:15:07.694 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:07.698+0000 7f853c200700 1 -- 172.21.15.81:0/1889304254 wait complete. 2024-04-12T21:15:07.695 INFO:teuthology.orchestra.run.smithi081.stderr:dumped all 2024-04-12T21:15:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:07 smithi153 ceph-mon[36569]: pgmap v373: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:07 smithi104 ceph-mon[36624]: pgmap v373: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-12T21:15:08.533 INFO:teuthology.orchestra.run.smithi081.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-04-12T21:15:06.376013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653867,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":317,"num_read_kb":268,"num_write":269,"num_write_kb":3032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1180887,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":712,"ondisk_log_size":712,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":383904,"kb_used_data":13952,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124557920,"statfs":{"total":1151940427776,"available":1151547310080,"internally_reserved":0,"allocated":14286848,"data_stored":7947503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":378713402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":249,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":31,"num_read_kb":16,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005935"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.347757+0000","last_change":"2024-04-12T21:13:13.157179+0000","last_active":"2024-04-12T21:14:52.347757+0000","last_peered":"2024-04-12T21:14:52.347757+0000","last_clean":"2024-04-12T21:14:52.347757+0000","last_became_active":"2024-04-12T21:13:13.156826+0000","last_became_peered":"2024-04-12T21:13:13.156826+0000","last_unstale":"2024-04-12T21:14:52.347757+0000","last_undegraded":"2024-04-12T21:14:52.347757+0000","last_fullsized":"2024-04-12T21:14:52.347757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:51:25.310758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.355515+0000","last_change":"2024-04-12T21:13:13.150917+0000","last_active":"2024-04-12T21:14:52.355515+0000","last_peered":"2024-04-12T21:14:52.355515+0000","last_clean":"2024-04-12T21:14:52.355515+0000","last_became_active":"2024-04-12T21:13:13.150491+0000","last_became_peered":"2024-04-12T21:13:13.150491+0000","last_unstale":"2024-04-12T21:14:52.355515+0000","last_undegraded":"2024-04-12T21:14:52.355515+0000","last_fullsized":"2024-04-12T21:14:52.355515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:08:04.486913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.354347+0000","last_change":"2024-04-12T21:13:13.150355+0000","last_active":"2024-04-12T21:14:52.354347+0000","last_peered":"2024-04-12T21:14:52.354347+0000","last_clean":"2024-04-12T21:14:52.354347+0000","last_became_active":"2024-04-12T21:13:13.149979+0000","last_became_peered":"2024-04-12T21:13:13.149979+0000","last_unstale":"2024-04-12T21:14:52.354347+0000","last_undegraded":"2024-04-12T21:14:52.354347+0000","last_fullsized":"2024-04-12T21:14:52.354347+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:02:56.175842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.325955+0000","last_change":"2024-04-12T21:13:13.132074+0000","last_active":"2024-04-12T21:14:52.325955+0000","last_peered":"2024-04-12T21:14:52.325955+0000","last_clean":"2024-04-12T21:14:52.325955+0000","last_became_active":"2024-04-12T21:13:13.130234+0000","last_became_peered":"2024-04-12T21:13:13.130234+0000","last_unstale":"2024-04-12T21:14:52.325955+0000","last_undegraded":"2024-04-12T21:14:52.325955+0000","last_fullsized":"2024-04-12T21:14:52.325955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:18:51.706215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.343269+0000","last_change":"2024-04-12T21:13:13.146577+0000","last_active":"2024-04-12T21:14:52.343269+0000","last_peered":"2024-04-12T21:14:52.343269+0000","last_clean":"2024-04-12T21:14:52.343269+0000","last_became_active":"2024-04-12T21:13:13.146162+0000","last_became_peered":"2024-04-12T21:13:13.146162+0000","last_unstale":"2024-04-12T21:14:52.343269+0000","last_undegraded":"2024-04-12T21:14:52.343269+0000","last_fullsized":"2024-04-12T21:14:52.343269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:43:10.267515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.336247+0000","last_change":"2024-04-12T21:13:13.137103+0000","last_active":"2024-04-12T21:14:52.336247+0000","last_peered":"2024-04-12T21:14:52.336247+0000","last_clean":"2024-04-12T21:14:52.336247+0000","last_became_active":"2024-04-12T21:13:13.136750+0000","last_became_peered":"2024-04-12T21:13:13.136750+0000","last_unstale":"2024-04-12T21:14:52.336247+0000","last_undegraded":"2024-04-12T21:14:52.336247+0000","last_fullsized":"2024-04-12T21:14:52.336247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:11:58.538179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.331493+0000","last_change":"2024-04-12T21:13:13.135440+0000","last_active":"2024-04-12T21:14:52.331493+0000","last_peered":"2024-04-12T21:14:52.331493+0000","last_clean":"2024-04-12T21:14:52.331493+0000","last_became_active":"2024-04-12T21:13:13.135027+0000","last_became_peered":"2024-04-12T21:13:13.135027+0000","last_unstale":"2024-04-12T21:14:52.331493+0000","last_undegraded":"2024-04-12T21:14:52.331493+0000","last_fullsized":"2024-04-12T21:14:52.331493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:37:50.875158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.361131+0000","last_change":"2024-04-12T21:13:13.143059+0000","last_active":"2024-04-12T21:14:52.361131+0000","last_peered":"2024-04-12T21:14:52.361131+0000","last_clean":"2024-04-12T21:14:52.361131+0000","last_became_active":"2024-04-12T21:13:13.142698+0000","last_became_peered":"2024-04-12T21:13:13.142698+0000","last_unstale":"2024-04-12T21:14:52.361131+0000","last_undegraded":"2024-04-12T21:14:52.361131+0000","last_fullsized":"2024-04-12T21:14:52.361131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:30:05.170007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.342514+0000","last_change":"2024-04-12T21:13:13.128917+0000","last_active":"2024-04-12T21:14:52.342514+0000","last_peered":"2024-04-12T21:14:52.342514+0000","last_clean":"2024-04-12T21:14:52.342514+0000","last_became_active":"2024-04-12T21:13:13.128166+0000","last_became_peered":"2024-04-12T21:13:13.128166+0000","last_unstale":"2024-04-12T21:14:52.342514+0000","last_undegraded":"2024-04-12T21:14:52.342514+0000","last_fullsized":"2024-04-12T21:14:52.342514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:49:41.282798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.329082+0000","last_change":"2024-04-12T21:13:13.130833+0000","last_active":"2024-04-12T21:14:52.329082+0000","last_peered":"2024-04-12T21:14:52.329082+0000","last_clean":"2024-04-12T21:14:52.329082+0000","last_became_active":"2024-04-12T21:13:13.130506+0000","last_became_peered":"2024-04-12T21:13:13.130506+0000","last_unstale":"2024-04-12T21:14:52.329082+0000","last_undegraded":"2024-04-12T21:14:52.329082+0000","last_fullsized":"2024-04-12T21:14:52.329082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:08:30.527381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.346297+0000","last_change":"2024-04-12T21:13:13.129682+0000","last_active":"2024-04-12T21:14:52.346297+0000","last_peered":"2024-04-12T21:14:52.346297+0000","last_clean":"2024-04-12T21:14:52.346297+0000","last_became_active":"2024-04-12T21:13:13.129364+0000","last_became_peered":"2024-04-12T21:13:13.129364+0000","last_unstale":"2024-04-12T21:14:52.346297+0000","last_undegraded":"2024-04-12T21:14:52.346297+0000","last_fullsized":"2024-04-12T21:14:52.346297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:19:48.442418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.322729+0000","last_change":"2024-04-12T21:13:13.133083+0000","last_active":"2024-04-12T21:14:52.322729+0000","last_peered":"2024-04-12T21:14:52.322729+0000","last_clean":"2024-04-12T21:14:52.322729+0000","last_became_active":"2024-04-12T21:13:13.132679+0000","last_became_peered":"2024-04-12T21:13:13.132679+0000","last_unstale":"2024-04-12T21:14:52.322729+0000","last_undegraded":"2024-04-12T21:14:52.322729+0000","last_fullsized":"2024-04-12T21:14:52.322729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:44:46.344475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.342870+0000","last_change":"2024-04-12T21:13:13.137692+0000","last_active":"2024-04-12T21:14:52.342870+0000","last_peered":"2024-04-12T21:14:52.342870+0000","last_clean":"2024-04-12T21:14:52.342870+0000","last_became_active":"2024-04-12T21:13:13.128523+0000","last_became_peered":"2024-04-12T21:13:13.128523+0000","last_unstale":"2024-04-12T21:14:52.342870+0000","last_undegraded":"2024-04-12T21:14:52.342870+0000","last_fullsized":"2024-04-12T21:14:52.342870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:52:30.847637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.363315+0000","last_change":"2024-04-12T21:13:13.149749+0000","last_active":"2024-04-12T21:14:52.363315+0000","last_peered":"2024-04-12T21:14:52.363315+0000","last_clean":"2024-04-12T21:14:52.363315+0000","last_became_active":"2024-04-12T21:13:13.149369+0000","last_became_peered":"2024-04-12T21:13:13.149369+0000","last_unstale":"2024-04-12T21:14:52.363315+0000","last_undegraded":"2024-04-12T21:14:52.363315+0000","last_fullsized":"2024-04-12T21:14:52.363315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:39:20.057070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.353980+0000","last_change":"2024-04-12T21:13:13.151510+0000","last_active":"2024-04-12T21:14:52.353980+0000","last_peered":"2024-04-12T21:14:52.353980+0000","last_clean":"2024-04-12T21:14:52.353980+0000","last_became_active":"2024-04-12T21:13:13.150620+0000","last_became_peered":"2024-04-12T21:13:13.150620+0000","last_unstale":"2024-04-12T21:14:52.353980+0000","last_undegraded":"2024-04-12T21:14:52.353980+0000","last_fullsized":"2024-04-12T21:14:52.353980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:59:14.772221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.347149+0000","last_change":"2024-04-12T21:13:13.156421+0000","last_active":"2024-04-12T21:14:52.347149+0000","last_peered":"2024-04-12T21:14:52.347149+0000","last_clean":"2024-04-12T21:14:52.347149+0000","last_became_active":"2024-04-12T21:13:13.156042+0000","last_became_peered":"2024-04-12T21:13:13.156042+0000","last_unstale":"2024-04-12T21:14:52.347149+0000","last_undegraded":"2024-04-12T21:14:52.347149+0000","last_fullsized":"2024-04-12T21:14:52.347149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:53:38.234230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.348047+0000","last_change":"2024-04-12T21:13:13.156986+0000","last_active":"2024-04-12T21:14:52.348047+0000","last_peered":"2024-04-12T21:14:52.348047+0000","last_clean":"2024-04-12T21:14:52.348047+0000","last_became_active":"2024-04-12T21:13:13.156705+0000","last_became_peered":"2024-04-12T21:13:13.156705+0000","last_unstale":"2024-04-12T21:14:52.348047+0000","last_undegraded":"2024-04-12T21:14:52.348047+0000","last_fullsized":"2024-04-12T21:14:52.348047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:49:42.442696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.342212+0000","last_change":"2024-04-12T21:13:13.136259+0000","last_active":"2024-04-12T21:14:52.342212+0000","last_peered":"2024-04-12T21:14:52.342212+0000","last_clean":"2024-04-12T21:14:52.342212+0000","last_became_active":"2024-04-12T21:13:13.135865+0000","last_became_peered":"2024-04-12T21:13:13.135865+0000","last_unstale":"2024-04-12T21:14:52.342212+0000","last_undegraded":"2024-04-12T21:14:52.342212+0000","last_fullsized":"2024-04-12T21:14:52.342212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:30:35.934682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.332926+0000","last_change":"2024-04-12T21:13:13.140460+0000","last_active":"2024-04-12T21:14:52.332926+0000","last_peered":"2024-04-12T21:14:52.332926+0000","last_clean":"2024-04-12T21:14:52.332926+0000","last_became_active":"2024-04-12T21:13:13.140115+0000","last_became_peered":"2024-04-12T21:13:13.140115+0000","last_unstale":"2024-04-12T21:14:52.332926+0000","last_undegraded":"2024-04-12T21:14:52.332926+0000","last_fullsized":"2024-04-12T21:14:52.332926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:56:08.880106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.497683+0000","last_change":"2024-04-12T21:13:17.606904+0000","last_active":"2024-04-12T21:14:23.497683+0000","last_peered":"2024-04-12T21:14:23.497683+0000","last_clean":"2024-04-12T21:14:23.497683+0000","last_became_active":"2024-04-12T21:13:17.606390+0000","last_became_peered":"2024-04-12T21:13:17.606390+0000","last_unstale":"2024-04-12T21:14:23.497683+0000","last_undegraded":"2024-04-12T21:14:23.497683+0000","last_fullsized":"2024-04-12T21:14:23.497683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:56:13.440022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506687+0000","last_change":"2024-04-12T21:13:12.535260+0000","last_active":"2024-04-12T21:14:23.506687+0000","last_peered":"2024-04-12T21:14:23.506687+0000","last_clean":"2024-04-12T21:14:23.506687+0000","last_became_active":"2024-04-12T21:13:12.534798+0000","last_became_peered":"2024-04-12T21:13:12.534798+0000","last_unstale":"2024-04-12T21:14:23.506687+0000","last_undegraded":"2024-04-12T21:14:23.506687+0000","last_fullsized":"2024-04-12T21:14:23.506687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:33:13.954242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.356476+0000","last_change":"2024-04-12T21:13:13.142701+0000","last_active":"2024-04-12T21:14:52.356476+0000","last_peered":"2024-04-12T21:14:52.356476+0000","last_clean":"2024-04-12T21:14:52.356476+0000","last_became_active":"2024-04-12T21:13:13.142329+0000","last_became_peered":"2024-04-12T21:13:13.142329+0000","last_unstale":"2024-04-12T21:14:52.356476+0000","last_undegraded":"2024-04-12T21:14:52.356476+0000","last_fullsized":"2024-04-12T21:14:52.356476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:59:02.356770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503791+0000","last_change":"2024-04-12T21:13:17.604050+0000","last_active":"2024-04-12T21:14:23.503791+0000","last_peered":"2024-04-12T21:14:23.503791+0000","last_clean":"2024-04-12T21:14:23.503791+0000","last_became_active":"2024-04-12T21:13:17.603628+0000","last_became_peered":"2024-04-12T21:13:17.603628+0000","last_unstale":"2024-04-12T21:14:23.503791+0000","last_undegraded":"2024-04-12T21:14:23.503791+0000","last_fullsized":"2024-04-12T21:14:23.503791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:24:24.986986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500866+0000","last_change":"2024-04-12T21:13:12.546263+0000","last_active":"2024-04-12T21:14:23.500866+0000","last_peered":"2024-04-12T21:14:23.500866+0000","last_clean":"2024-04-12T21:14:23.500866+0000","last_became_active":"2024-04-12T21:13:12.545710+0000","last_became_peered":"2024-04-12T21:13:12.545710+0000","last_unstale":"2024-04-12T21:14:23.500866+0000","last_undegraded":"2024-04-12T21:14:23.500866+0000","last_fullsized":"2024-04-12T21:14:23.500866+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:18:33.732571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364281+0000","last_change":"2024-04-12T21:13:13.156478+0000","last_active":"2024-04-12T21:14:52.364281+0000","last_peered":"2024-04-12T21:14:52.364281+0000","last_clean":"2024-04-12T21:14:52.364281+0000","last_became_active":"2024-04-12T21:13:13.155485+0000","last_became_peered":"2024-04-12T21:13:13.155485+0000","last_unstale":"2024-04-12T21:14:52.364281+0000","last_undegraded":"2024-04-12T21:14:52.364281+0000","last_fullsized":"2024-04-12T21:14:52.364281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:11:24.161377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503495+0000","last_change":"2024-04-12T21:13:17.607327+0000","last_active":"2024-04-12T21:14:23.503495+0000","last_peered":"2024-04-12T21:14:23.503495+0000","last_clean":"2024-04-12T21:14:23.503495+0000","last_became_active":"2024-04-12T21:13:17.606949+0000","last_became_peered":"2024-04-12T21:13:17.606949+0000","last_unstale":"2024-04-12T21:14:23.503495+0000","last_undegraded":"2024-04-12T21:14:23.503495+0000","last_fullsized":"2024-04-12T21:14:23.503495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:43:27.855461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504116+0000","last_change":"2024-04-12T21:13:12.533082+0000","last_active":"2024-04-12T21:14:23.504116+0000","last_peered":"2024-04-12T21:14:23.504116+0000","last_clean":"2024-04-12T21:14:23.504116+0000","last_became_active":"2024-04-12T21:13:12.532739+0000","last_became_peered":"2024-04-12T21:13:12.532739+0000","last_unstale":"2024-04-12T21:14:23.504116+0000","last_undegraded":"2024-04-12T21:14:23.504116+0000","last_fullsized":"2024-04-12T21:14:23.504116+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:40:01.885223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.349310+0000","last_change":"2024-04-12T21:13:13.136718+0000","last_active":"2024-04-12T21:14:52.349310+0000","last_peered":"2024-04-12T21:14:52.349310+0000","last_clean":"2024-04-12T21:14:52.349310+0000","last_became_active":"2024-04-12T21:13:13.136357+0000","last_became_peered":"2024-04-12T21:13:13.136357+0000","last_unstale":"2024-04-12T21:14:52.349310+0000","last_undegraded":"2024-04-12T21:14:52.349310+0000","last_fullsized":"2024-04-12T21:14:52.349310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:08:57.380556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503252+0000","last_change":"2024-04-12T21:13:17.601381+0000","last_active":"2024-04-12T21:14:23.503252+0000","last_peered":"2024-04-12T21:14:23.503252+0000","last_clean":"2024-04-12T21:14:23.503252+0000","last_became_active":"2024-04-12T21:13:17.600868+0000","last_became_peered":"2024-04-12T21:13:17.600868+0000","last_unstale":"2024-04-12T21:14:23.503252+0000","last_undegraded":"2024-04-12T21:14:23.503252+0000","last_fullsized":"2024-04-12T21:14:23.503252+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:53:36.234779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.505045+0000","last_change":"2024-04-12T21:13:12.528285+0000","last_active":"2024-04-12T21:14:23.505045+0000","last_peered":"2024-04-12T21:14:23.505045+0000","last_clean":"2024-04-12T21:14:23.505045+0000","last_became_active":"2024-04-12T21:13:12.527915+0000","last_became_peered":"2024-04-12T21:13:12.527915+0000","last_unstale":"2024-04-12T21:14:23.505045+0000","last_undegraded":"2024-04-12T21:14:23.505045+0000","last_fullsized":"2024-04-12T21:14:23.505045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T09:04:44.119836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.338764+0000","last_change":"2024-04-12T21:13:13.128654+0000","last_active":"2024-04-12T21:14:52.338764+0000","last_peered":"2024-04-12T21:14:52.338764+0000","last_clean":"2024-04-12T21:14:52.338764+0000","last_became_active":"2024-04-12T21:13:13.128274+0000","last_became_peered":"2024-04-12T21:13:13.128274+0000","last_unstale":"2024-04-12T21:14:52.338764+0000","last_undegraded":"2024-04-12T21:14:52.338764+0000","last_fullsized":"2024-04-12T21:14:52.338764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:48:08.827157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498001+0000","last_change":"2024-04-12T21:13:17.594360+0000","last_active":"2024-04-12T21:14:23.498001+0000","last_peered":"2024-04-12T21:14:23.498001+0000","last_clean":"2024-04-12T21:14:23.498001+0000","last_became_active":"2024-04-12T21:13:17.593924+0000","last_became_peered":"2024-04-12T21:13:17.593924+0000","last_unstale":"2024-04-12T21:14:23.498001+0000","last_undegraded":"2024-04-12T21:14:23.498001+0000","last_fullsized":"2024-04-12T21:14:23.498001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:42:14.269657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.502843+0000","last_change":"2024-04-12T21:13:12.532481+0000","last_active":"2024-04-12T21:14:23.502843+0000","last_peered":"2024-04-12T21:14:23.502843+0000","last_clean":"2024-04-12T21:14:23.502843+0000","last_became_active":"2024-04-12T21:13:12.532126+0000","last_became_peered":"2024-04-12T21:13:12.532126+0000","last_unstale":"2024-04-12T21:14:23.502843+0000","last_undegraded":"2024-04-12T21:14:23.502843+0000","last_fullsized":"2024-04-12T21:14:23.502843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:19:52.510076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.339625+0000","last_change":"2024-04-12T21:13:13.138290+0000","last_active":"2024-04-12T21:14:52.339625+0000","last_peered":"2024-04-12T21:14:52.339625+0000","last_clean":"2024-04-12T21:14:52.339625+0000","last_became_active":"2024-04-12T21:13:13.137854+0000","last_became_peered":"2024-04-12T21:13:13.137854+0000","last_unstale":"2024-04-12T21:14:52.339625+0000","last_undegraded":"2024-04-12T21:14:52.339625+0000","last_fullsized":"2024-04-12T21:14:52.339625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:27:45.455704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.941407+0000","last_change":"2024-04-12T21:14:23.941407+0000","last_active":"2024-04-12T21:14:23.941407+0000","last_peered":"2024-04-12T21:14:23.941407+0000","last_clean":"2024-04-12T21:14:23.941407+0000","last_became_active":"2024-04-12T21:14:23.940906+0000","last_became_peered":"2024-04-12T21:14:23.940906+0000","last_unstale":"2024-04-12T21:14:23.941407+0000","last_undegraded":"2024-04-12T21:14:23.941407+0000","last_fullsized":"2024-04-12T21:14:23.941407+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:49:30.433604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.502757+0000","last_change":"2024-04-12T21:13:12.539083+0000","last_active":"2024-04-12T21:14:23.502757+0000","last_peered":"2024-04-12T21:14:23.502757+0000","last_clean":"2024-04-12T21:14:23.502757+0000","last_became_active":"2024-04-12T21:13:12.538771+0000","last_became_peered":"2024-04-12T21:13:12.538771+0000","last_unstale":"2024-04-12T21:14:23.502757+0000","last_undegraded":"2024-04-12T21:14:23.502757+0000","last_fullsized":"2024-04-12T21:14:23.502757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:31.560665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.344062+0000","last_change":"2024-04-12T21:14:23.507246+0000","last_active":"2024-04-12T21:14:52.344062+0000","last_peered":"2024-04-12T21:14:52.344062+0000","last_clean":"2024-04-12T21:14:52.344062+0000","last_became_active":"2024-04-12T21:14:23.506815+0000","last_became_peered":"2024-04-12T21:14:23.506815+0000","last_unstale":"2024-04-12T21:14:52.344062+0000","last_undegraded":"2024-04-12T21:14:52.344062+0000","last_fullsized":"2024-04-12T21:14:52.344062+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:02:55.930713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500131+0000","last_change":"2024-04-12T21:13:17.607603+0000","last_active":"2024-04-12T21:14:23.500131+0000","last_peered":"2024-04-12T21:14:23.500131+0000","last_clean":"2024-04-12T21:14:23.500131+0000","last_became_active":"2024-04-12T21:13:17.606972+0000","last_became_peered":"2024-04-12T21:13:17.606972+0000","last_unstale":"2024-04-12T21:14:23.500131+0000","last_undegraded":"2024-04-12T21:14:23.500131+0000","last_fullsized":"2024-04-12T21:14:23.500131+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:16:37.593158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938475+0000","last_change":"2024-04-12T21:13:12.541405+0000","last_active":"2024-04-12T21:14:23.938475+0000","last_peered":"2024-04-12T21:14:23.938475+0000","last_clean":"2024-04-12T21:14:23.938475+0000","last_became_active":"2024-04-12T21:13:12.540612+0000","last_became_peered":"2024-04-12T21:13:12.540612+0000","last_unstale":"2024-04-12T21:14:23.938475+0000","last_undegraded":"2024-04-12T21:14:23.938475+0000","last_fullsized":"2024-04-12T21:14:23.938475+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:38:10.749857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.361409+0000","last_change":"2024-04-12T21:13:13.138447+0000","last_active":"2024-04-12T21:14:52.361409+0000","last_peered":"2024-04-12T21:14:52.361409+0000","last_clean":"2024-04-12T21:14:52.361409+0000","last_became_active":"2024-04-12T21:13:13.137990+0000","last_became_peered":"2024-04-12T21:13:13.137990+0000","last_unstale":"2024-04-12T21:14:52.361409+0000","last_undegraded":"2024-04-12T21:14:52.361409+0000","last_fullsized":"2024-04-12T21:14:52.361409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:58:50.381257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.653459+0000","last_change":"2024-04-12T21:13:17.614003+0000","last_active":"2024-04-12T21:14:23.653459+0000","last_peered":"2024-04-12T21:14:23.653459+0000","last_clean":"2024-04-12T21:14:23.653459+0000","last_became_active":"2024-04-12T21:13:17.613517+0000","last_became_peered":"2024-04-12T21:13:17.613517+0000","last_unstale":"2024-04-12T21:14:23.653459+0000","last_undegraded":"2024-04-12T21:14:23.653459+0000","last_fullsized":"2024-04-12T21:14:23.653459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:33.590816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501831+0000","last_change":"2024-04-12T21:13:12.530503+0000","last_active":"2024-04-12T21:14:23.501831+0000","last_peered":"2024-04-12T21:14:23.501831+0000","last_clean":"2024-04-12T21:14:23.501831+0000","last_became_active":"2024-04-12T21:13:12.530161+0000","last_became_peered":"2024-04-12T21:13:12.530161+0000","last_unstale":"2024-04-12T21:14:23.501831+0000","last_undegraded":"2024-04-12T21:14:23.501831+0000","last_fullsized":"2024-04-12T21:14:23.501831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:06:18.363283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.357391+0000","last_change":"2024-04-12T21:13:13.146401+0000","last_active":"2024-04-12T21:14:52.357391+0000","last_peered":"2024-04-12T21:14:52.357391+0000","last_clean":"2024-04-12T21:14:52.357391+0000","last_became_active":"2024-04-12T21:13:13.145527+0000","last_became_peered":"2024-04-12T21:13:13.145527+0000","last_unstale":"2024-04-12T21:14:52.357391+0000","last_undegraded":"2024-04-12T21:14:52.357391+0000","last_fullsized":"2024-04-12T21:14:52.357391+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:40:41.302305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498492+0000","last_change":"2024-04-12T21:13:17.609141+0000","last_active":"2024-04-12T21:14:23.498492+0000","last_peered":"2024-04-12T21:14:23.498492+0000","last_clean":"2024-04-12T21:14:23.498492+0000","last_became_active":"2024-04-12T21:13:17.603368+0000","last_became_peered":"2024-04-12T21:13:17.603368+0000","last_unstale":"2024-04-12T21:14:23.498492+0000","last_undegraded":"2024-04-12T21:14:23.498492+0000","last_fullsized":"2024-04-12T21:14:23.498492+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:07:54.235284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504591+0000","last_change":"2024-04-12T21:13:12.540388+0000","last_active":"2024-04-12T21:14:23.504591+0000","last_peered":"2024-04-12T21:14:23.504591+0000","last_clean":"2024-04-12T21:14:23.504591+0000","last_became_active":"2024-04-12T21:13:12.540033+0000","last_became_peered":"2024-04-12T21:13:12.540033+0000","last_unstale":"2024-04-12T21:14:23.504591+0000","last_undegraded":"2024-04-12T21:14:23.504591+0000","last_fullsized":"2024-04-12T21:14:23.504591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:46:53.785074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.349620+0000","last_change":"2024-04-12T21:13:13.137208+0000","last_active":"2024-04-12T21:14:52.349620+0000","last_peered":"2024-04-12T21:14:52.349620+0000","last_clean":"2024-04-12T21:14:52.349620+0000","last_became_active":"2024-04-12T21:13:13.136866+0000","last_became_peered":"2024-04-12T21:13:13.136866+0000","last_unstale":"2024-04-12T21:14:52.349620+0000","last_undegraded":"2024-04-12T21:14:52.349620+0000","last_fullsized":"2024-04-12T21:14:52.349620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:48:36.655808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501219+0000","last_change":"2024-04-12T21:13:17.617564+0000","last_active":"2024-04-12T21:14:23.501219+0000","last_peered":"2024-04-12T21:14:23.501219+0000","last_clean":"2024-04-12T21:14:23.501219+0000","last_became_active":"2024-04-12T21:13:17.615980+0000","last_became_peered":"2024-04-12T21:13:17.615980+0000","last_unstale":"2024-04-12T21:14:23.501219+0000","last_undegraded":"2024-04-12T21:14:23.501219+0000","last_fullsized":"2024-04-12T21:14:23.501219+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:22:38.370357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.509734+0000","last_change":"2024-04-12T21:14:23.509734+0000","last_active":"2024-04-12T21:14:23.509734+0000","last_peered":"2024-04-12T21:14:23.509734+0000","last_clean":"2024-04-12T21:14:23.509734+0000","last_became_active":"2024-04-12T21:14:23.509186+0000","last_became_peered":"2024-04-12T21:14:23.509186+0000","last_unstale":"2024-04-12T21:14:23.509734+0000","last_undegraded":"2024-04-12T21:14:23.509734+0000","last_fullsized":"2024-04-12T21:14:23.509734+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:51:37.972583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364072+0000","last_change":"2024-04-12T21:13:13.152737+0000","last_active":"2024-04-12T21:14:52.364072+0000","last_peered":"2024-04-12T21:14:52.364072+0000","last_clean":"2024-04-12T21:14:52.364072+0000","last_became_active":"2024-04-12T21:13:13.150914+0000","last_became_peered":"2024-04-12T21:13:13.150914+0000","last_unstale":"2024-04-12T21:14:52.364072+0000","last_undegraded":"2024-04-12T21:14:52.364072+0000","last_fullsized":"2024-04-12T21:14:52.364072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:51:29.020746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503683+0000","last_change":"2024-04-12T21:13:17.604826+0000","last_active":"2024-04-12T21:14:23.503683+0000","last_peered":"2024-04-12T21:14:23.503683+0000","last_clean":"2024-04-12T21:14:23.503683+0000","last_became_active":"2024-04-12T21:13:17.604427+0000","last_became_peered":"2024-04-12T21:13:17.604427+0000","last_unstale":"2024-04-12T21:14:23.503683+0000","last_undegraded":"2024-04-12T21:14:23.503683+0000","last_fullsized":"2024-04-12T21:14:23.503683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:04:55.602539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503221+0000","last_change":"2024-04-12T21:13:12.542641+0000","last_active":"2024-04-12T21:14:23.503221+0000","last_peered":"2024-04-12T21:14:23.503221+0000","last_clean":"2024-04-12T21:14:23.503221+0000","last_became_active":"2024-04-12T21:13:12.542291+0000","last_became_peered":"2024-04-12T21:13:12.542291+0000","last_unstale":"2024-04-12T21:14:23.503221+0000","last_undegraded":"2024-04-12T21:14:23.503221+0000","last_fullsized":"2024-04-12T21:14:23.503221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:37:00.186153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.348538+0000","last_change":"2024-04-12T21:14:23.941262+0000","last_active":"2024-04-12T21:14:52.348538+0000","last_peered":"2024-04-12T21:14:52.348538+0000","last_clean":"2024-04-12T21:14:52.348538+0000","last_became_active":"2024-04-12T21:14:23.940721+0000","last_became_peered":"2024-04-12T21:14:23.940721+0000","last_unstale":"2024-04-12T21:14:52.348538+0000","last_undegraded":"2024-04-12T21:14:52.348538+0000","last_fullsized":"2024-04-12T21:14:52.348538+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:07:13.835188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503151+0000","last_change":"2024-04-12T21:13:17.606533+0000","last_active":"2024-04-12T21:14:23.503151+0000","last_peered":"2024-04-12T21:14:23.503151+0000","last_clean":"2024-04-12T21:14:23.503151+0000","last_became_active":"2024-04-12T21:13:17.606092+0000","last_became_peered":"2024-04-12T21:13:17.606092+0000","last_unstale":"2024-04-12T21:14:23.503151+0000","last_undegraded":"2024-04-12T21:14:23.503151+0000","last_fullsized":"2024-04-12T21:14:23.503151+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:55:56.080659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'33","reported_seq":247,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:15:05.269245+0000","last_change":"2024-04-12T21:13:12.531797+0000","last_active":"2024-04-12T21:15:05.269245+0000","last_peered":"2024-04-12T21:15:05.269245+0000","last_clean":"2024-04-12T21:15:05.269245+0000","last_became_active":"2024-04-12T21:13:12.531346+0000","last_became_peered":"2024-04-12T21:13:12.531346+0000","last_unstale":"2024-04-12T21:15:05.269245+0000","last_undegraded":"2024-04-12T21:15:05.269245+0000","last_fullsized":"2024-04-12T21:15:05.269245+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:59:32.429211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62467,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":187,"num_read_kb":155,"num_write":33,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.333798+0000","last_change":"2024-04-12T21:13:13.131501+0000","last_active":"2024-04-12T21:14:52.333798+0000","last_peered":"2024-04-12T21:14:52.333798+0000","last_clean":"2024-04-12T21:14:52.333798+0000","last_became_active":"2024-04-12T21:13:13.131161+0000","last_became_peered":"2024-04-12T21:13:13.131161+0000","last_unstale":"2024-04-12T21:14:52.333798+0000","last_undegraded":"2024-04-12T21:14:52.333798+0000","last_fullsized":"2024-04-12T21:14:52.333798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:45:27.046250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.942111+0000","last_change":"2024-04-12T21:14:23.941552+0000","last_active":"2024-04-12T21:14:23.942111+0000","last_peered":"2024-04-12T21:14:23.942111+0000","last_clean":"2024-04-12T21:14:23.942111+0000","last_became_active":"2024-04-12T21:14:23.940998+0000","last_became_peered":"2024-04-12T21:14:23.940998+0000","last_unstale":"2024-04-12T21:14:23.942111+0000","last_undegraded":"2024-04-12T21:14:23.942111+0000","last_fullsized":"2024-04-12T21:14:23.942111+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:34:04.055211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.514794+0000","last_change":"2024-04-12T21:14:23.514343+0000","last_active":"2024-04-12T21:14:23.514794+0000","last_peered":"2024-04-12T21:14:23.514794+0000","last_clean":"2024-04-12T21:14:23.514794+0000","last_became_active":"2024-04-12T21:14:23.505690+0000","last_became_peered":"2024-04-12T21:14:23.505690+0000","last_unstale":"2024-04-12T21:14:23.514794+0000","last_undegraded":"2024-04-12T21:14:23.514794+0000","last_fullsized":"2024-04-12T21:14:23.514794+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:24:12.746487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.341334+0000","last_change":"2024-04-12T21:13:13.150100+0000","last_active":"2024-04-12T21:14:52.341334+0000","last_peered":"2024-04-12T21:14:52.341334+0000","last_clean":"2024-04-12T21:14:52.341334+0000","last_became_active":"2024-04-12T21:13:13.149711+0000","last_became_peered":"2024-04-12T21:13:13.149711+0000","last_unstale":"2024-04-12T21:14:52.341334+0000","last_undegraded":"2024-04-12T21:14:52.341334+0000","last_fullsized":"2024-04-12T21:14:52.341334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:33:27.121682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.941361+0000","last_change":"2024-04-12T21:14:23.941361+0000","last_active":"2024-04-12T21:14:23.941361+0000","last_peered":"2024-04-12T21:14:23.941361+0000","last_clean":"2024-04-12T21:14:23.941361+0000","last_became_active":"2024-04-12T21:14:23.940817+0000","last_became_peered":"2024-04-12T21:14:23.940817+0000","last_unstale":"2024-04-12T21:14:23.941361+0000","last_undegraded":"2024-04-12T21:14:23.941361+0000","last_fullsized":"2024-04-12T21:14:23.941361+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:52:18.459215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.502514+0000","last_change":"2024-04-12T21:13:12.545077+0000","last_active":"2024-04-12T21:14:23.502514+0000","last_peered":"2024-04-12T21:14:23.502514+0000","last_clean":"2024-04-12T21:14:23.502514+0000","last_became_active":"2024-04-12T21:13:12.544755+0000","last_became_peered":"2024-04-12T21:13:12.544755+0000","last_unstale":"2024-04-12T21:14:23.502514+0000","last_undegraded":"2024-04-12T21:14:23.502514+0000","last_fullsized":"2024-04-12T21:14:23.502514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:31:28.487336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.347339+0000","last_change":"2024-04-12T21:13:13.128047+0000","last_active":"2024-04-12T21:14:52.347339+0000","last_peered":"2024-04-12T21:14:52.347339+0000","last_clean":"2024-04-12T21:14:52.347339+0000","last_became_active":"2024-04-12T21:13:13.127692+0000","last_became_peered":"2024-04-12T21:13:13.127692+0000","last_unstale":"2024-04-12T21:14:52.347339+0000","last_undegraded":"2024-04-12T21:14:52.347339+0000","last_fullsized":"2024-04-12T21:14:52.347339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:43:29.552438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506489+0000","last_change":"2024-04-12T21:13:17.614240+0000","last_active":"2024-04-12T21:14:23.506489+0000","last_peered":"2024-04-12T21:14:23.506489+0000","last_clean":"2024-04-12T21:14:23.506489+0000","last_became_active":"2024-04-12T21:13:17.613834+0000","last_became_peered":"2024-04-12T21:13:17.613834+0000","last_unstale":"2024-04-12T21:14:23.506489+0000","last_undegraded":"2024-04-12T21:14:23.506489+0000","last_fullsized":"2024-04-12T21:14:23.506489+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:29:23.949101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503750+0000","last_change":"2024-04-12T21:13:12.527868+0000","last_active":"2024-04-12T21:14:23.503750+0000","last_peered":"2024-04-12T21:14:23.503750+0000","last_clean":"2024-04-12T21:14:23.503750+0000","last_became_active":"2024-04-12T21:13:12.527521+0000","last_became_peered":"2024-04-12T21:13:12.527521+0000","last_unstale":"2024-04-12T21:14:23.503750+0000","last_undegraded":"2024-04-12T21:14:23.503750+0000","last_fullsized":"2024-04-12T21:14:23.503750+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:46:23.649552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.339504+0000","last_change":"2024-04-12T21:13:13.124207+0000","last_active":"2024-04-12T21:14:52.339504+0000","last_peered":"2024-04-12T21:14:52.339504+0000","last_clean":"2024-04-12T21:14:52.339504+0000","last_became_active":"2024-04-12T21:13:13.123866+0000","last_became_peered":"2024-04-12T21:13:13.123866+0000","last_unstale":"2024-04-12T21:14:52.339504+0000","last_undegraded":"2024-04-12T21:14:52.339504+0000","last_fullsized":"2024-04-12T21:14:52.339504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:38:04.263260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938143+0000","last_change":"2024-04-12T21:13:17.613064+0000","last_active":"2024-04-12T21:14:23.938143+0000","last_peered":"2024-04-12T21:14:23.938143+0000","last_clean":"2024-04-12T21:14:23.938143+0000","last_became_active":"2024-04-12T21:13:17.612569+0000","last_became_peered":"2024-04-12T21:13:17.612569+0000","last_unstale":"2024-04-12T21:14:23.938143+0000","last_undegraded":"2024-04-12T21:14:23.938143+0000","last_fullsized":"2024-04-12T21:14:23.938143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:35:27.274086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503107+0000","last_change":"2024-04-12T21:13:12.543193+0000","last_active":"2024-04-12T21:14:23.503107+0000","last_peered":"2024-04-12T21:14:23.503107+0000","last_clean":"2024-04-12T21:14:23.503107+0000","last_became_active":"2024-04-12T21:13:12.542885+0000","last_became_peered":"2024-04-12T21:13:12.542885+0000","last_unstale":"2024-04-12T21:14:23.503107+0000","last_undegraded":"2024-04-12T21:14:23.503107+0000","last_fullsized":"2024-04-12T21:14:23.503107+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:02:11.100366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337710+0000","last_change":"2024-04-12T21:13:13.130682+0000","last_active":"2024-04-12T21:14:52.337710+0000","last_peered":"2024-04-12T21:14:52.337710+0000","last_clean":"2024-04-12T21:14:52.337710+0000","last_became_active":"2024-04-12T21:13:13.130299+0000","last_became_peered":"2024-04-12T21:13:13.130299+0000","last_unstale":"2024-04-12T21:14:52.337710+0000","last_undegraded":"2024-04-12T21:14:52.337710+0000","last_fullsized":"2024-04-12T21:14:52.337710+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:52:02.094959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501865+0000","last_change":"2024-04-12T21:13:17.609472+0000","last_active":"2024-04-12T21:14:23.501865+0000","last_peered":"2024-04-12T21:14:23.501865+0000","last_clean":"2024-04-12T21:14:23.501865+0000","last_became_active":"2024-04-12T21:13:17.608991+0000","last_became_peered":"2024-04-12T21:13:17.608991+0000","last_unstale":"2024-04-12T21:14:23.501865+0000","last_undegraded":"2024-04-12T21:14:23.501865+0000","last_fullsized":"2024-04-12T21:14:23.501865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:17:57.962701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503725+0000","last_change":"2024-04-12T21:13:12.530058+0000","last_active":"2024-04-12T21:14:23.503725+0000","last_peered":"2024-04-12T21:14:23.503725+0000","last_clean":"2024-04-12T21:14:23.503725+0000","last_became_active":"2024-04-12T21:13:12.529682+0000","last_became_peered":"2024-04-12T21:13:12.529682+0000","last_unstale":"2024-04-12T21:14:23.503725+0000","last_undegraded":"2024-04-12T21:14:23.503725+0000","last_fullsized":"2024-04-12T21:14:23.503725+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:17:25.621593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337223+0000","last_change":"2024-04-12T21:13:13.129975+0000","last_active":"2024-04-12T21:14:52.337223+0000","last_peered":"2024-04-12T21:14:52.337223+0000","last_clean":"2024-04-12T21:14:52.337223+0000","last_became_active":"2024-04-12T21:13:13.129621+0000","last_became_peered":"2024-04-12T21:13:13.129621+0000","last_unstale":"2024-04-12T21:14:52.337223+0000","last_undegraded":"2024-04-12T21:14:52.337223+0000","last_fullsized":"2024-04-12T21:14:52.337223+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:49:48.267685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504102+0000","last_change":"2024-04-12T21:13:17.738019+0000","last_active":"2024-04-12T21:14:23.504102+0000","last_peered":"2024-04-12T21:14:23.504102+0000","last_clean":"2024-04-12T21:14:23.504102+0000","last_became_active":"2024-04-12T21:13:17.737553+0000","last_became_peered":"2024-04-12T21:13:17.737553+0000","last_unstale":"2024-04-12T21:14:23.504102+0000","last_undegraded":"2024-04-12T21:14:23.504102+0000","last_fullsized":"2024-04-12T21:14:23.504102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:29:07.288025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.505748+0000","last_change":"2024-04-12T21:13:12.544661+0000","last_active":"2024-04-12T21:14:23.505748+0000","last_peered":"2024-04-12T21:14:23.505748+0000","last_clean":"2024-04-12T21:14:23.505748+0000","last_became_active":"2024-04-12T21:13:12.544321+0000","last_became_peered":"2024-04-12T21:13:12.544321+0000","last_unstale":"2024-04-12T21:14:23.505748+0000","last_undegraded":"2024-04-12T21:14:23.505748+0000","last_fullsized":"2024-04-12T21:14:23.505748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:55:43.517970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.358349+0000","last_change":"2024-04-12T21:13:13.140480+0000","last_active":"2024-04-12T21:14:52.358349+0000","last_peered":"2024-04-12T21:14:52.358349+0000","last_clean":"2024-04-12T21:14:52.358349+0000","last_became_active":"2024-04-12T21:13:13.140128+0000","last_became_peered":"2024-04-12T21:13:13.140128+0000","last_unstale":"2024-04-12T21:14:52.358349+0000","last_undegraded":"2024-04-12T21:14:52.358349+0000","last_fullsized":"2024-04-12T21:14:52.358349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:40:32.066929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938876+0000","last_change":"2024-04-12T21:13:17.609235+0000","last_active":"2024-04-12T21:14:23.938876+0000","last_peered":"2024-04-12T21:14:23.938876+0000","last_clean":"2024-04-12T21:14:23.938876+0000","last_became_active":"2024-04-12T21:13:17.608814+0000","last_became_peered":"2024-04-12T21:13:17.608814+0000","last_unstale":"2024-04-12T21:14:23.938876+0000","last_undegraded":"2024-04-12T21:14:23.938876+0000","last_fullsized":"2024-04-12T21:14:23.938876+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:15:35.287239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500807+0000","last_change":"2024-04-12T21:13:12.539638+0000","last_active":"2024-04-12T21:14:23.500807+0000","last_peered":"2024-04-12T21:14:23.500807+0000","last_clean":"2024-04-12T21:14:23.500807+0000","last_became_active":"2024-04-12T21:13:12.539272+0000","last_became_peered":"2024-04-12T21:13:12.539272+0000","last_unstale":"2024-04-12T21:14:23.500807+0000","last_undegraded":"2024-04-12T21:14:23.500807+0000","last_fullsized":"2024-04-12T21:14:23.500807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:11:42.992489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.352009+0000","last_change":"2024-04-12T21:13:13.149326+0000","last_active":"2024-04-12T21:14:52.352009+0000","last_peered":"2024-04-12T21:14:52.352009+0000","last_clean":"2024-04-12T21:14:52.352009+0000","last_became_active":"2024-04-12T21:13:13.149023+0000","last_became_peered":"2024-04-12T21:13:13.149023+0000","last_unstale":"2024-04-12T21:14:52.352009+0000","last_undegraded":"2024-04-12T21:14:52.352009+0000","last_fullsized":"2024-04-12T21:14:52.352009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:15:58.479778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938430+0000","last_change":"2024-04-12T21:13:17.740067+0000","last_active":"2024-04-12T21:14:23.938430+0000","last_peered":"2024-04-12T21:14:23.938430+0000","last_clean":"2024-04-12T21:14:23.938430+0000","last_became_active":"2024-04-12T21:13:17.738186+0000","last_became_peered":"2024-04-12T21:13:17.738186+0000","last_unstale":"2024-04-12T21:14:23.938430+0000","last_undegraded":"2024-04-12T21:14:23.938430+0000","last_fullsized":"2024-04-12T21:14:23.938430+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:15:05.241855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.333729+0000","last_change":"2024-04-12T21:14:23.509373+0000","last_active":"2024-04-12T21:14:52.333729+0000","last_peered":"2024-04-12T21:14:52.333729+0000","last_clean":"2024-04-12T21:14:52.333729+0000","last_became_active":"2024-04-12T21:14:23.508595+0000","last_became_peered":"2024-04-12T21:14:23.508595+0000","last_unstale":"2024-04-12T21:14:52.333729+0000","last_undegraded":"2024-04-12T21:14:52.333729+0000","last_fullsized":"2024-04-12T21:14:52.333729+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:45:49.388383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938366+0000","last_change":"2024-04-12T21:13:12.551549+0000","last_active":"2024-04-12T21:14:23.938366+0000","last_peered":"2024-04-12T21:14:23.938366+0000","last_clean":"2024-04-12T21:14:23.938366+0000","last_became_active":"2024-04-12T21:13:12.551204+0000","last_became_peered":"2024-04-12T21:13:12.551204+0000","last_unstale":"2024-04-12T21:14:23.938366+0000","last_undegraded":"2024-04-12T21:14:23.938366+0000","last_fullsized":"2024-04-12T21:14:23.938366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:33:06.721971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501481+0000","last_change":"2024-04-12T21:13:17.603828+0000","last_active":"2024-04-12T21:14:23.501481+0000","last_peered":"2024-04-12T21:14:23.501481+0000","last_clean":"2024-04-12T21:14:23.501481+0000","last_became_active":"2024-04-12T21:13:17.603399+0000","last_became_peered":"2024-04-12T21:13:17.603399+0000","last_unstale":"2024-04-12T21:14:23.501481+0000","last_undegraded":"2024-04-12T21:14:23.501481+0000","last_fullsized":"2024-04-12T21:14:23.501481+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:22:28.126722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.357541+0000","last_change":"2024-04-12T21:13:13.137828+0000","last_active":"2024-04-12T21:14:52.357541+0000","last_peered":"2024-04-12T21:14:52.357541+0000","last_clean":"2024-04-12T21:14:52.357541+0000","last_became_active":"2024-04-12T21:13:13.137534+0000","last_became_peered":"2024-04-12T21:13:13.137534+0000","last_unstale":"2024-04-12T21:14:52.357541+0000","last_undegraded":"2024-04-12T21:14:52.357541+0000","last_fullsized":"2024-04-12T21:14:52.357541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:24:20.238210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503628+0000","last_change":"2024-04-12T21:13:12.533215+0000","last_active":"2024-04-12T21:14:23.503628+0000","last_peered":"2024-04-12T21:14:23.503628+0000","last_clean":"2024-04-12T21:14:23.503628+0000","last_became_active":"2024-04-12T21:13:12.532852+0000","last_became_peered":"2024-04-12T21:13:12.532852+0000","last_unstale":"2024-04-12T21:14:23.503628+0000","last_undegraded":"2024-04-12T21:14:23.503628+0000","last_fullsized":"2024-04-12T21:14:23.503628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:41:17.861406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500416+0000","last_change":"2024-04-12T21:11:53.092756+0000","last_active":"2024-04-12T21:14:23.500416+0000","last_peered":"2024-04-12T21:14:23.500416+0000","last_clean":"2024-04-12T21:14:23.500416+0000","last_became_active":"2024-04-12T21:11:53.092364+0000","last_became_peered":"2024-04-12T21:11:53.092364+0000","last_unstale":"2024-04-12T21:14:23.500416+0000","last_undegraded":"2024-04-12T21:14:23.500416+0000","last_fullsized":"2024-04-12T21:14:23.500416+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:08:04.402338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:08:04.402338+0000","last_clean_scrub_stamp":"2024-04-12T21:08:04.402338+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:26:11.910015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.502584+0000","last_change":"2024-04-12T21:13:17.738168+0000","last_active":"2024-04-12T21:14:23.502584+0000","last_peered":"2024-04-12T21:14:23.502584+0000","last_clean":"2024-04-12T21:14:23.502584+0000","last_became_active":"2024-04-12T21:13:17.737697+0000","last_became_peered":"2024-04-12T21:13:17.737697+0000","last_unstale":"2024-04-12T21:14:23.502584+0000","last_undegraded":"2024-04-12T21:14:23.502584+0000","last_fullsized":"2024-04-12T21:14:23.502584+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:11:31.025636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.323212+0000","last_change":"2024-04-12T21:13:13.130398+0000","last_active":"2024-04-12T21:14:52.323212+0000","last_peered":"2024-04-12T21:14:52.323212+0000","last_clean":"2024-04-12T21:14:52.323212+0000","last_became_active":"2024-04-12T21:13:13.129488+0000","last_became_peered":"2024-04-12T21:13:13.129488+0000","last_unstale":"2024-04-12T21:14:52.323212+0000","last_undegraded":"2024-04-12T21:14:52.323212+0000","last_fullsized":"2024-04-12T21:14:52.323212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:21:03.737140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500410+0000","last_change":"2024-04-12T21:13:12.533367+0000","last_active":"2024-04-12T21:14:23.500410+0000","last_peered":"2024-04-12T21:14:23.500410+0000","last_clean":"2024-04-12T21:14:23.500410+0000","last_became_active":"2024-04-12T21:13:12.533042+0000","last_became_peered":"2024-04-12T21:13:12.533042+0000","last_unstale":"2024-04-12T21:14:23.500410+0000","last_undegraded":"2024-04-12T21:14:23.500410+0000","last_fullsized":"2024-04-12T21:14:23.500410+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:56:07.671693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.500591+0000","last_change":"2024-04-12T21:13:17.593951+0000","last_active":"2024-04-12T21:14:23.500591+0000","last_peered":"2024-04-12T21:14:23.500591+0000","last_clean":"2024-04-12T21:14:23.500591+0000","last_became_active":"2024-04-12T21:13:17.593528+0000","last_became_peered":"2024-04-12T21:13:17.593528+0000","last_unstale":"2024-04-12T21:14:23.500591+0000","last_undegraded":"2024-04-12T21:14:23.500591+0000","last_fullsized":"2024-04-12T21:14:23.500591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:24:25.081109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504157+0000","last_change":"2024-04-12T21:13:12.543040+0000","last_active":"2024-04-12T21:14:23.504157+0000","last_peered":"2024-04-12T21:14:23.504157+0000","last_clean":"2024-04-12T21:14:23.504157+0000","last_became_active":"2024-04-12T21:13:12.542698+0000","last_became_peered":"2024-04-12T21:13:12.542698+0000","last_unstale":"2024-04-12T21:14:23.504157+0000","last_undegraded":"2024-04-12T21:14:23.504157+0000","last_fullsized":"2024-04-12T21:14:23.504157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:57:55.712648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.335923+0000","last_change":"2024-04-12T21:13:13.138975+0000","last_active":"2024-04-12T21:14:52.335923+0000","last_peered":"2024-04-12T21:14:52.335923+0000","last_clean":"2024-04-12T21:14:52.335923+0000","last_became_active":"2024-04-12T21:13:13.138538+0000","last_became_peered":"2024-04-12T21:13:13.138538+0000","last_unstale":"2024-04-12T21:14:52.335923+0000","last_undegraded":"2024-04-12T21:14:52.335923+0000","last_fullsized":"2024-04-12T21:14:52.335923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:12:10.488914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498341+0000","last_change":"2024-04-12T21:13:17.739156+0000","last_active":"2024-04-12T21:14:23.498341+0000","last_peered":"2024-04-12T21:14:23.498341+0000","last_clean":"2024-04-12T21:14:23.498341+0000","last_became_active":"2024-04-12T21:13:17.738153+0000","last_became_peered":"2024-04-12T21:13:17.738153+0000","last_unstale":"2024-04-12T21:14:23.498341+0000","last_undegraded":"2024-04-12T21:14:23.498341+0000","last_fullsized":"2024-04-12T21:14:23.498341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:18:18.930005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506504+0000","last_change":"2024-04-12T21:13:12.547658+0000","last_active":"2024-04-12T21:14:23.506504+0000","last_peered":"2024-04-12T21:14:23.506504+0000","last_clean":"2024-04-12T21:14:23.506504+0000","last_became_active":"2024-04-12T21:13:12.547197+0000","last_became_peered":"2024-04-12T21:13:12.547197+0000","last_unstale":"2024-04-12T21:14:23.506504+0000","last_undegraded":"2024-04-12T21:14:23.506504+0000","last_fullsized":"2024-04-12T21:14:23.506504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:57:18.066730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.356304+0000","last_change":"2024-04-12T21:13:13.143876+0000","last_active":"2024-04-12T21:14:52.356304+0000","last_peered":"2024-04-12T21:14:52.356304+0000","last_clean":"2024-04-12T21:14:52.356304+0000","last_became_active":"2024-04-12T21:13:13.140389+0000","last_became_peered":"2024-04-12T21:13:13.140389+0000","last_unstale":"2024-04-12T21:14:52.356304+0000","last_undegraded":"2024-04-12T21:14:52.356304+0000","last_fullsized":"2024-04-12T21:14:52.356304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:36:13.910978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.497556+0000","last_change":"2024-04-12T21:13:17.592755+0000","last_active":"2024-04-12T21:14:23.497556+0000","last_peered":"2024-04-12T21:14:23.497556+0000","last_clean":"2024-04-12T21:14:23.497556+0000","last_became_active":"2024-04-12T21:13:17.592316+0000","last_became_peered":"2024-04-12T21:13:17.592316+0000","last_unstale":"2024-04-12T21:14:23.497556+0000","last_undegraded":"2024-04-12T21:14:23.497556+0000","last_fullsized":"2024-04-12T21:14:23.497556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:16:41.986623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504560+0000","last_change":"2024-04-12T21:13:12.538168+0000","last_active":"2024-04-12T21:14:23.504560+0000","last_peered":"2024-04-12T21:14:23.504560+0000","last_clean":"2024-04-12T21:14:23.504560+0000","last_became_active":"2024-04-12T21:13:12.537784+0000","last_became_peered":"2024-04-12T21:13:12.537784+0000","last_unstale":"2024-04-12T21:14:23.504560+0000","last_undegraded":"2024-04-12T21:14:23.504560+0000","last_fullsized":"2024-04-12T21:14:23.504560+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:57:58.333171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.363050+0000","last_change":"2024-04-12T21:13:13.153230+0000","last_active":"2024-04-12T21:14:52.363050+0000","last_peered":"2024-04-12T21:14:52.363050+0000","last_clean":"2024-04-12T21:14:52.363050+0000","last_became_active":"2024-04-12T21:13:13.152795+0000","last_became_peered":"2024-04-12T21:13:13.152795+0000","last_unstale":"2024-04-12T21:14:52.363050+0000","last_undegraded":"2024-04-12T21:14:52.363050+0000","last_fullsized":"2024-04-12T21:14:52.363050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:11:21.058591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.504718+0000","last_change":"2024-04-12T21:13:17.738428+0000","last_active":"2024-04-12T21:14:23.504718+0000","last_peered":"2024-04-12T21:14:23.504718+0000","last_clean":"2024-04-12T21:14:23.504718+0000","last_became_active":"2024-04-12T21:13:17.737932+0000","last_became_peered":"2024-04-12T21:13:17.737932+0000","last_unstale":"2024-04-12T21:14:23.504718+0000","last_undegraded":"2024-04-12T21:14:23.504718+0000","last_fullsized":"2024-04-12T21:14:23.504718+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:22:26.310841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506739+0000","last_change":"2024-04-12T21:13:12.542689+0000","last_active":"2024-04-12T21:14:23.506739+0000","last_peered":"2024-04-12T21:14:23.506739+0000","last_clean":"2024-04-12T21:14:23.506739+0000","last_became_active":"2024-04-12T21:13:12.542371+0000","last_became_peered":"2024-04-12T21:13:12.542371+0000","last_unstale":"2024-04-12T21:14:23.506739+0000","last_undegraded":"2024-04-12T21:14:23.506739+0000","last_fullsized":"2024-04-12T21:14:23.506739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:08:48.439015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.302630+0000","last_change":"2024-04-12T21:13:13.130341+0000","last_active":"2024-04-12T21:14:52.302630+0000","last_peered":"2024-04-12T21:14:52.302630+0000","last_clean":"2024-04-12T21:14:52.302630+0000","last_became_active":"2024-04-12T21:13:13.129886+0000","last_became_peered":"2024-04-12T21:13:13.129886+0000","last_unstale":"2024-04-12T21:14:52.302630+0000","last_undegraded":"2024-04-12T21:14:52.302630+0000","last_fullsized":"2024-04-12T21:14:52.302630+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:43:18.384216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506600+0000","last_change":"2024-04-12T21:13:17.615138+0000","last_active":"2024-04-12T21:14:23.506600+0000","last_peered":"2024-04-12T21:14:23.506600+0000","last_clean":"2024-04-12T21:14:23.506600+0000","last_became_active":"2024-04-12T21:13:17.614710+0000","last_became_peered":"2024-04-12T21:13:17.614710+0000","last_unstale":"2024-04-12T21:14:23.506600+0000","last_undegraded":"2024-04-12T21:14:23.506600+0000","last_fullsized":"2024-04-12T21:14:23.506600+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:13:47.956600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498707+0000","last_change":"2024-04-12T21:13:12.546160+0000","last_active":"2024-04-12T21:14:23.498707+0000","last_peered":"2024-04-12T21:14:23.498707+0000","last_clean":"2024-04-12T21:14:23.498707+0000","last_became_active":"2024-04-12T21:13:12.544193+0000","last_became_peered":"2024-04-12T21:13:12.544193+0000","last_unstale":"2024-04-12T21:14:23.498707+0000","last_undegraded":"2024-04-12T21:14:23.498707+0000","last_fullsized":"2024-04-12T21:14:23.498707+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:19:39.640814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.344337+0000","last_change":"2024-04-12T21:13:13.154604+0000","last_active":"2024-04-12T21:14:52.344337+0000","last_peered":"2024-04-12T21:14:52.344337+0000","last_clean":"2024-04-12T21:14:52.344337+0000","last_became_active":"2024-04-12T21:13:13.154257+0000","last_became_peered":"2024-04-12T21:13:13.154257+0000","last_unstale":"2024-04-12T21:14:52.344337+0000","last_undegraded":"2024-04-12T21:14:52.344337+0000","last_fullsized":"2024-04-12T21:14:52.344337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:31:45.550602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.517587+0000","last_change":"2024-04-12T21:14:23.517587+0000","last_active":"2024-04-12T21:14:23.517587+0000","last_peered":"2024-04-12T21:14:23.517587+0000","last_clean":"2024-04-12T21:14:23.517587+0000","last_became_active":"2024-04-12T21:14:23.507668+0000","last_became_peered":"2024-04-12T21:14:23.507668+0000","last_unstale":"2024-04-12T21:14:23.517587+0000","last_undegraded":"2024-04-12T21:14:23.517587+0000","last_fullsized":"2024-04-12T21:14:23.517587+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:31:05.648702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.497721+0000","last_change":"2024-04-12T21:13:12.536798+0000","last_active":"2024-04-12T21:14:23.497721+0000","last_peered":"2024-04-12T21:14:23.497721+0000","last_clean":"2024-04-12T21:14:23.497721+0000","last_became_active":"2024-04-12T21:13:12.536451+0000","last_became_peered":"2024-04-12T21:13:12.536451+0000","last_unstale":"2024-04-12T21:14:23.497721+0000","last_undegraded":"2024-04-12T21:14:23.497721+0000","last_fullsized":"2024-04-12T21:14:23.497721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:07:07.935780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364253+0000","last_change":"2024-04-12T21:13:13.154644+0000","last_active":"2024-04-12T21:14:52.364253+0000","last_peered":"2024-04-12T21:14:52.364253+0000","last_clean":"2024-04-12T21:14:52.364253+0000","last_became_active":"2024-04-12T21:13:13.154310+0000","last_became_peered":"2024-04-12T21:13:13.154310+0000","last_unstale":"2024-04-12T21:14:52.364253+0000","last_undegraded":"2024-04-12T21:14:52.364253+0000","last_fullsized":"2024-04-12T21:14:52.364253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:51:36.426506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503466+0000","last_change":"2024-04-12T21:13:17.737547+0000","last_active":"2024-04-12T21:14:23.503466+0000","last_peered":"2024-04-12T21:14:23.503466+0000","last_clean":"2024-04-12T21:14:23.503466+0000","last_became_active":"2024-04-12T21:13:17.737190+0000","last_became_peered":"2024-04-12T21:13:17.737190+0000","last_unstale":"2024-04-12T21:14:23.503466+0000","last_undegraded":"2024-04-12T21:14:23.503466+0000","last_fullsized":"2024-04-12T21:14:23.503466+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:23:52.221788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.510271+0000","last_change":"2024-04-12T21:14:23.509809+0000","last_active":"2024-04-12T21:14:23.510271+0000","last_peered":"2024-04-12T21:14:23.510271+0000","last_clean":"2024-04-12T21:14:23.510271+0000","last_became_active":"2024-04-12T21:14:23.505368+0000","last_became_peered":"2024-04-12T21:14:23.505368+0000","last_unstale":"2024-04-12T21:14:23.510271+0000","last_undegraded":"2024-04-12T21:14:23.510271+0000","last_fullsized":"2024-04-12T21:14:23.510271+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T03:34:49.529205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.348933+0000","last_change":"2024-04-12T21:13:13.128079+0000","last_active":"2024-04-12T21:14:52.348933+0000","last_peered":"2024-04-12T21:14:52.348933+0000","last_clean":"2024-04-12T21:14:52.348933+0000","last_became_active":"2024-04-12T21:13:13.127744+0000","last_became_peered":"2024-04-12T21:13:13.127744+0000","last_unstale":"2024-04-12T21:14:52.348933+0000","last_undegraded":"2024-04-12T21:14:52.348933+0000","last_fullsized":"2024-04-12T21:14:52.348933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T01:39:41.851323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.506557+0000","last_change":"2024-04-12T21:13:17.596088+0000","last_active":"2024-04-12T21:14:23.506557+0000","last_peered":"2024-04-12T21:14:23.506557+0000","last_clean":"2024-04-12T21:14:23.506557+0000","last_became_active":"2024-04-12T21:13:17.595639+0000","last_became_peered":"2024-04-12T21:13:17.595639+0000","last_unstale":"2024-04-12T21:14:23.506557+0000","last_undegraded":"2024-04-12T21:14:23.506557+0000","last_fullsized":"2024-04-12T21:14:23.506557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:25:09.763945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.498196+0000","last_change":"2024-04-12T21:13:12.543947+0000","last_active":"2024-04-12T21:14:23.498196+0000","last_peered":"2024-04-12T21:14:23.498196+0000","last_clean":"2024-04-12T21:14:23.498196+0000","last_became_active":"2024-04-12T21:13:12.543511+0000","last_became_peered":"2024-04-12T21:13:12.543511+0000","last_unstale":"2024-04-12T21:14:23.498196+0000","last_undegraded":"2024-04-12T21:14:23.498196+0000","last_fullsized":"2024-04-12T21:14:23.498196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:43:23.354835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.346426+0000","last_change":"2024-04-12T21:13:13.127178+0000","last_active":"2024-04-12T21:14:52.346426+0000","last_peered":"2024-04-12T21:14:52.346426+0000","last_clean":"2024-04-12T21:14:52.346426+0000","last_became_active":"2024-04-12T21:13:13.126856+0000","last_became_peered":"2024-04-12T21:13:13.126856+0000","last_unstale":"2024-04-12T21:14:52.346426+0000","last_undegraded":"2024-04-12T21:14:52.346426+0000","last_fullsized":"2024-04-12T21:14:52.346426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:09:34.068946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.652983+0000","last_change":"2024-04-12T21:13:17.619628+0000","last_active":"2024-04-12T21:14:23.652983+0000","last_peered":"2024-04-12T21:14:23.652983+0000","last_clean":"2024-04-12T21:14:23.652983+0000","last_became_active":"2024-04-12T21:13:17.619015+0000","last_became_peered":"2024-04-12T21:13:17.619015+0000","last_unstale":"2024-04-12T21:14:23.652983+0000","last_undegraded":"2024-04-12T21:14:23.652983+0000","last_fullsized":"2024-04-12T21:14:23.652983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T02:39:54.297165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.503448+0000","last_change":"2024-04-12T21:13:12.538982+0000","last_active":"2024-04-12T21:14:23.503448+0000","last_peered":"2024-04-12T21:14:23.503448+0000","last_clean":"2024-04-12T21:14:23.503448+0000","last_became_active":"2024-04-12T21:13:12.538604+0000","last_became_peered":"2024-04-12T21:13:12.538604+0000","last_unstale":"2024-04-12T21:14:23.503448+0000","last_undegraded":"2024-04-12T21:14:23.503448+0000","last_fullsized":"2024-04-12T21:14:23.503448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:40:48.181188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.341305+0000","last_change":"2024-04-12T21:13:13.145537+0000","last_active":"2024-04-12T21:14:52.341305+0000","last_peered":"2024-04-12T21:14:52.341305+0000","last_clean":"2024-04-12T21:14:52.341305+0000","last_became_active":"2024-04-12T21:13:13.145283+0000","last_became_peered":"2024-04-12T21:13:13.145283+0000","last_unstale":"2024-04-12T21:14:52.341305+0000","last_undegraded":"2024-04-12T21:14:52.341305+0000","last_fullsized":"2024-04-12T21:14:52.341305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:26:29.339674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.501477+0000","last_change":"2024-04-12T21:13:17.594394+0000","last_active":"2024-04-12T21:14:23.501477+0000","last_peered":"2024-04-12T21:14:23.501477+0000","last_clean":"2024-04-12T21:14:23.501477+0000","last_became_active":"2024-04-12T21:13:17.593960+0000","last_became_peered":"2024-04-12T21:13:17.593960+0000","last_unstale":"2024-04-12T21:14:23.501477+0000","last_undegraded":"2024-04-12T21:14:23.501477+0000","last_fullsized":"2024-04-12T21:14:23.501477+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:16.555054+0000","last_clean_scrub_stamp":"2024-04-12T21:13:16.555054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:00:53.462145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:23.938259+0000","last_change":"2024-04-12T21:13:12.546385+0000","last_active":"2024-04-12T21:14:23.938259+0000","last_peered":"2024-04-12T21:14:23.938259+0000","last_clean":"2024-04-12T21:14:23.938259+0000","last_became_active":"2024-04-12T21:13:12.545902+0000","last_became_peered":"2024-04-12T21:13:12.545902+0000","last_unstale":"2024-04-12T21:14:23.938259+0000","last_undegraded":"2024-04-12T21:14:23.938259+0000","last_fullsized":"2024-04-12T21:14:23.938259+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:11.495781+0000","last_clean_scrub_stamp":"2024-04-12T21:13:11.495781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:57:16.419641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.341095+0000","last_change":"2024-04-12T21:13:13.130478+0000","last_active":"2024-04-12T21:14:52.341095+0000","last_peered":"2024-04-12T21:14:52.341095+0000","last_clean":"2024-04-12T21:14:52.341095+0000","last_became_active":"2024-04-12T21:13:13.130164+0000","last_became_peered":"2024-04-12T21:13:13.130164+0000","last_unstale":"2024-04-12T21:14:52.341095+0000","last_undegraded":"2024-04-12T21:14:52.341095+0000","last_fullsized":"2024-04-12T21:14:52.341095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T05:22:29.186629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337126+0000","last_change":"2024-04-12T21:13:13.135406+0000","last_active":"2024-04-12T21:14:52.337126+0000","last_peered":"2024-04-12T21:14:52.337126+0000","last_clean":"2024-04-12T21:14:52.337126+0000","last_became_active":"2024-04-12T21:13:13.134840+0000","last_became_peered":"2024-04-12T21:13:13.134840+0000","last_unstale":"2024-04-12T21:14:52.337126+0000","last_undegraded":"2024-04-12T21:14:52.337126+0000","last_fullsized":"2024-04-12T21:14:52.337126+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:22:22.315016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337956+0000","last_change":"2024-04-12T21:13:13.127249+0000","last_active":"2024-04-12T21:14:52.337956+0000","last_peered":"2024-04-12T21:14:52.337956+0000","last_clean":"2024-04-12T21:14:52.337956+0000","last_became_active":"2024-04-12T21:13:13.126901+0000","last_became_peered":"2024-04-12T21:13:13.126901+0000","last_unstale":"2024-04-12T21:14:52.337956+0000","last_undegraded":"2024-04-12T21:14:52.337956+0000","last_fullsized":"2024-04-12T21:14:52.337956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T04:32:18.857794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.352059+0000","last_change":"2024-04-12T21:13:13.151728+0000","last_active":"2024-04-12T21:14:52.352059+0000","last_peered":"2024-04-12T21:14:52.352059+0000","last_clean":"2024-04-12T21:14:52.352059+0000","last_became_active":"2024-04-12T21:13:13.151457+0000","last_became_peered":"2024-04-12T21:13:13.151457+0000","last_unstale":"2024-04-12T21:14:52.352059+0000","last_undegraded":"2024-04-12T21:14:52.352059+0000","last_fullsized":"2024-04-12T21:14:52.352059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:48:32.641671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.363440+0000","last_change":"2024-04-12T21:13:13.150109+0000","last_active":"2024-04-12T21:14:52.363440+0000","last_peered":"2024-04-12T21:14:52.363440+0000","last_clean":"2024-04-12T21:14:52.363440+0000","last_became_active":"2024-04-12T21:13:13.149132+0000","last_became_peered":"2024-04-12T21:13:13.149132+0000","last_unstale":"2024-04-12T21:14:52.363440+0000","last_undegraded":"2024-04-12T21:14:52.363440+0000","last_fullsized":"2024-04-12T21:14:52.363440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T21:36:00.062993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364228+0000","last_change":"2024-04-12T21:13:13.143405+0000","last_active":"2024-04-12T21:14:52.364228+0000","last_peered":"2024-04-12T21:14:52.364228+0000","last_clean":"2024-04-12T21:14:52.364228+0000","last_became_active":"2024-04-12T21:13:13.142973+0000","last_became_peered":"2024-04-12T21:13:13.142973+0000","last_unstale":"2024-04-12T21:14:52.364228+0000","last_undegraded":"2024-04-12T21:14:52.364228+0000","last_fullsized":"2024-04-12T21:14:52.364228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T23:10:03.106739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.334093+0000","last_change":"2024-04-12T21:13:13.129864+0000","last_active":"2024-04-12T21:14:52.334093+0000","last_peered":"2024-04-12T21:14:52.334093+0000","last_clean":"2024-04-12T21:14:52.334093+0000","last_became_active":"2024-04-12T21:13:13.129518+0000","last_became_peered":"2024-04-12T21:13:13.129518+0000","last_unstale":"2024-04-12T21:14:52.334093+0000","last_undegraded":"2024-04-12T21:14:52.334093+0000","last_fullsized":"2024-04-12T21:14:52.334093+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T07:44:38.370002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.346620+0000","last_change":"2024-04-12T21:13:13.129153+0000","last_active":"2024-04-12T21:14:52.346620+0000","last_peered":"2024-04-12T21:14:52.346620+0000","last_clean":"2024-04-12T21:14:52.346620+0000","last_became_active":"2024-04-12T21:13:13.128799+0000","last_became_peered":"2024-04-12T21:13:13.128799+0000","last_unstale":"2024-04-12T21:14:52.346620+0000","last_undegraded":"2024-04-12T21:14:52.346620+0000","last_fullsized":"2024-04-12T21:14:52.346620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:37:51.675110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.354279+0000","last_change":"2024-04-12T21:13:13.144638+0000","last_active":"2024-04-12T21:14:52.354279+0000","last_peered":"2024-04-12T21:14:52.354279+0000","last_clean":"2024-04-12T21:14:52.354279+0000","last_became_active":"2024-04-12T21:13:13.144275+0000","last_became_peered":"2024-04-12T21:13:13.144275+0000","last_unstale":"2024-04-12T21:14:52.354279+0000","last_undegraded":"2024-04-12T21:14:52.354279+0000","last_fullsized":"2024-04-12T21:14:52.354279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T08:38:09.225551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.353660+0000","last_change":"2024-04-12T21:13:13.145845+0000","last_active":"2024-04-12T21:14:52.353660+0000","last_peered":"2024-04-12T21:14:52.353660+0000","last_clean":"2024-04-12T21:14:52.353660+0000","last_became_active":"2024-04-12T21:13:13.145399+0000","last_became_peered":"2024-04-12T21:13:13.145399+0000","last_unstale":"2024-04-12T21:14:52.353660+0000","last_undegraded":"2024-04-12T21:14:52.353660+0000","last_fullsized":"2024-04-12T21:14:52.353660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:07:13.075365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.354775+0000","last_change":"2024-04-12T21:13:13.138373+0000","last_active":"2024-04-12T21:14:52.354775+0000","last_peered":"2024-04-12T21:14:52.354775+0000","last_clean":"2024-04-12T21:14:52.354775+0000","last_became_active":"2024-04-12T21:13:13.137366+0000","last_became_peered":"2024-04-12T21:13:13.137366+0000","last_unstale":"2024-04-12T21:14:52.354775+0000","last_undegraded":"2024-04-12T21:14:52.354775+0000","last_fullsized":"2024-04-12T21:14:52.354775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:05:22.070677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.359799+0000","last_change":"2024-04-12T21:13:13.147902+0000","last_active":"2024-04-12T21:14:52.359799+0000","last_peered":"2024-04-12T21:14:52.359799+0000","last_clean":"2024-04-12T21:14:52.359799+0000","last_became_active":"2024-04-12T21:13:13.147513+0000","last_became_peered":"2024-04-12T21:13:13.147513+0000","last_unstale":"2024-04-12T21:14:52.359799+0000","last_undegraded":"2024-04-12T21:14:52.359799+0000","last_fullsized":"2024-04-12T21:14:52.359799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-13T22:43:34.228432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.364585+0000","last_change":"2024-04-12T21:13:13.154703+0000","last_active":"2024-04-12T21:14:52.364585+0000","last_peered":"2024-04-12T21:14:52.364585+0000","last_clean":"2024-04-12T21:14:52.364585+0000","last_became_active":"2024-04-12T21:13:13.154340+0000","last_became_peered":"2024-04-12T21:13:13.154340+0000","last_unstale":"2024-04-12T21:14:52.364585+0000","last_undegraded":"2024-04-12T21:14:52.364585+0000","last_fullsized":"2024-04-12T21:14:52.364585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T06:29:05.500481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-12T21:14:52.337426+0000","last_change":"2024-04-12T21:13:13.133441+0000","last_active":"2024-04-12T21:14:52.337426+0000","last_peered":"2024-04-12T21:14:52.337426+0000","last_clean":"2024-04-12T21:14:52.337426+0000","last_became_active":"2024-04-12T21:13:13.133113+0000","last_became_peered":"2024-04-12T21:13:13.133113+0000","last_unstale":"2024-04-12T21:14:52.337426+0000","last_undegraded":"2024-04-12T21:14:52.337426+0000","last_fullsized":"2024-04-12T21:14:52.337426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-12T21:13:12.511569+0000","last_clean_scrub_stamp":"2024-04-12T21:13:12.511569+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-14T00:05:36.924405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63265,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":191,"num_read_kb":159,"num_write":86,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":189795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383718,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":500283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645354,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28676,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716476,"statfs":{"total":95995035648,"available":95965671424,"internally_reserved":0,"allocated":1699840,"data_stored":1153278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808879,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":500831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1028096,"data_stored":500853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332155,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1095295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495679,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":496153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":558620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":496243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36400,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708752,"statfs":{"total":95995035648,"available":95957762048,"internally_reserved":0,"allocated":1089536,"data_stored":558620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":496257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":500271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1626112,"data_stored":1090799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":62661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-12T21:15:08.538 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-12T21:15:08.538 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-12T21:15:08.538 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-12T21:15:08.538 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph health --format=json 2024-04-12T21:15:09.067 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:09.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:09 smithi081 ceph-mon[31453]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:15:09.482 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:09 smithi081 ceph-mon[31453]: pgmap v374: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:09 smithi153 ceph-mon[36569]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:15:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:09 smithi153 ceph-mon[36569]: pgmap v374: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:09 smithi104 ceph-mon[36624]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-12T21:15:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:09 smithi104 ceph-mon[36624]: pgmap v374: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.351+0000 7f3e33c00700 1 -- 172.21.15.81:0/1768846400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c075600 msgr2=0x7f3e2c10efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.351+0000 7f3e33c00700 1 --2- 172.21.15.81:0/1768846400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c075600 0x7f3e2c10efd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009990 tx=0x7f3e18009ca0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.351+0000 7f3e33c00700 1 -- 172.21.15.81:0/1768846400 shutdown_connections 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.351+0000 7f3e33c00700 1 --2- 172.21.15.81:0/1768846400 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e2c10f510 0x7f3e2c111900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.351+0000 7f3e33c00700 1 --2- 172.21.15.81:0/1768846400 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e2c074ce0 0x7f3e2c0750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.351+0000 7f3e33c00700 1 --2- 172.21.15.81:0/1768846400 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c075600 0x7f3e2c10efd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.351+0000 7f3e33c00700 1 -- 172.21.15.81:0/1768846400 >> 172.21.15.81:0/1768846400 conn(0x7f3e2c079c10 msgr2=0x7f3e2c07a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.352+0000 7f3e33c00700 1 -- 172.21.15.81:0/1768846400 shutdown_connections 2024-04-12T21:15:11.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.352+0000 7f3e33c00700 1 -- 172.21.15.81:0/1768846400 wait complete. 2024-04-12T21:15:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.353+0000 7f3e33c00700 1 Processor -- start 2024-04-12T21:15:11.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.353+0000 7f3e33c00700 1 -- start start 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.353+0000 7f3e33c00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c074ce0 0x7f3e2c19b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.353+0000 7f3e33c00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e2c075600 0x7f3e2c19b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.353+0000 7f3e33c00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e2c10f510 0x7f3e2c19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e33c00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2c115050 con 0x7f3e2c074ce0 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e33c00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2c114ed0 con 0x7f3e2c10f510 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e33c00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2c1151d0 con 0x7f3e2c075600 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c074ce0 0x7f3e2c19b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c074ce0 0x7f3e2c19b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38218/0 (socket says 172.21.15.81:38218) 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31200700 1 -- 172.21.15.81:0/2593082238 learned_addr learned my addr 172.21.15.81:0/2593082238 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e2c10f510 0x7f3e2c19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:11.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e2be00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e2c075600 0x7f3e2c19b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31c00700 1 -- 172.21.15.81:0/2593082238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e2c075600 msgr2=0x7f3e2c19b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e2c075600 0x7f3e2c19b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31c00700 1 -- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c074ce0 msgr2=0x7f3e2c19b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c074ce0 0x7f3e2c19b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31c00700 1 -- 172.21.15.81:0/2593082238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2c1a0430 con 0x7f3e2c10f510 2024-04-12T21:15:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.354+0000 7f3e31200700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c074ce0 0x7f3e2c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.355+0000 7f3e31c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e2c10f510 0x7f3e2c19fcf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3e24007ee0 tx=0x7f3e2400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.355+0000 7f3e29600700 1 -- 172.21.15.81:0/2593082238 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24004db0 con 0x7f3e2c10f510 2024-04-12T21:15:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.355+0000 7f3e29600700 1 -- 172.21.15.81:0/2593082238 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e24009e20 con 0x7f3e2c10f510 2024-04-12T21:15:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.355+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e2c1a0720 con 0x7f3e2c10f510 2024-04-12T21:15:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.355+0000 7f3e29600700 1 -- 172.21.15.81:0/2593082238 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e24015400 con 0x7f3e2c10f510 2024-04-12T21:15:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.355+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e2c1a7fa0 con 0x7f3e2c10f510 2024-04-12T21:15:11.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.356+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e2c0697a0 con 0x7f3e2c10f510 2024-04-12T21:15:11.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.359+0000 7f3e29600700 1 -- 172.21.15.81:0/2593082238 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3e2400db20 con 0x7f3e2c10f510 2024-04-12T21:15:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.360+0000 7f3e29600700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3e140a3cc0 0x7f3e140a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.360+0000 7f3e29600700 1 -- 172.21.15.81:0/2593082238 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e240cfff0 con 0x7f3e2c10f510 2024-04-12T21:15:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.361+0000 7f3e31200700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3e140a3cc0 0x7f3e140a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:11.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.361+0000 7f3e31200700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3e140a3cc0 0x7f3e140a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3e20011630 tx=0x7f3e20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:11.360 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.365+0000 7f3e29600700 1 -- 172.21.15.81:0/2593082238 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e24010070 con 0x7f3e2c10f510 2024-04-12T21:15:11.601 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:11 smithi081 ceph-mon[31453]: pgmap v375: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:11.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.684+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f3e2c1a81a0 con 0x7f3e2c10f510 2024-04-12T21:15:11.681 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.685+0000 7f3e29600700 1 -- 172.21.15.81:0/2593082238 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f3e2409c610 con 0x7f3e2c10f510 2024-04-12T21:15:11.681 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:11.682 INFO:teuthology.orchestra.run.smithi081.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-12T21:15:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.687+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3e140a3cc0 msgr2=0x7f3e140a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.687+0000 7f3e33c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3e140a3cc0 0x7f3e140a6180 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3e20011630 tx=0x7f3e20009260 comp rx=0 tx=0).stop 2024-04-12T21:15:11.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.687+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e2c10f510 msgr2=0x7f3e2c19fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.687+0000 7f3e33c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e2c10f510 0x7f3e2c19fcf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3e24007ee0 tx=0x7f3e2400d3b0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.688+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 shutdown_connections 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.688+0000 7f3e33c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3e140a3cc0 0x7f3e140a6180 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.688+0000 7f3e33c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3e2c10f510 0x7f3e2c19fcf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.688+0000 7f3e33c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3e2c074ce0 0x7f3e2c19b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.688+0000 7f3e33c00700 1 --2- 172.21.15.81:0/2593082238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3e2c075600 0x7f3e2c19b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.688+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 >> 172.21.15.81:0/2593082238 conn(0x7f3e2c079c10 msgr2=0x7f3e2c0ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.688+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 shutdown_connections 2024-04-12T21:15:11.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:11.688+0000 7f3e33c00700 1 -- 172.21.15.81:0/2593082238 wait complete. 2024-04-12T21:15:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:11 smithi153 ceph-mon[36569]: pgmap v375: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:11.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:11 smithi104 ceph-mon[36624]: pgmap v375: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:12.551 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-12T21:15:12.552 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 -- 172.21.15.81:0/3932879184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 msgr2=0x7f9010108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 --2- 172.21.15.81:0/3932879184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 0x7f9010108860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9004009900 tx=0x7f900402bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 -- 172.21.15.81:0/3932879184 shutdown_connections 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 --2- 172.21.15.81:0/3932879184 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9010109ff0 0x7f9010110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 --2- 172.21.15.81:0/3932879184 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9010109630 0x7f9010109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 --2- 172.21.15.81:0/3932879184 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 0x7f9010108860 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 -- 172.21.15.81:0/3932879184 >> 172.21.15.81:0/3932879184 conn(0x7f9010077400 msgr2=0x7f9010079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 -- 172.21.15.81:0/3932879184 shutdown_connections 2024-04-12T21:15:12.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.772+0000 7f9017200640 1 -- 172.21.15.81:0/3932879184 wait complete. 2024-04-12T21:15:12.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.773+0000 7f9017200640 1 Processor -- start 2024-04-12T21:15:12.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.773+0000 7f9017200640 1 -- start start 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.773+0000 7f9017200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 0x7f901019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.773+0000 7f9017200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9010109630 0x7f901019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.773+0000 7f9017200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9010109ff0 0x7f90101a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9017200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f90101140d0 con 0x7f9010108440 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9017200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9010113f50 con 0x7f9010109630 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9017200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9010114250 con 0x7f9010109ff0 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 0x7f901019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 0x7f901019e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38226/0 (socket says 172.21.15.81:38226) 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 -- 172.21.15.81:0/1912923065 learned_addr learned my addr 172.21.15.81:0/1912923065 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9016800640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9010109ff0 0x7f90101a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015400640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9010109630 0x7f901019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 -- 172.21.15.81:0/1912923065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9010109ff0 msgr2=0x7f90101a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9010109ff0 0x7f90101a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 -- 172.21.15.81:0/1912923065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9010109630 msgr2=0x7f901019ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9010109630 0x7f901019ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 -- 172.21.15.81:0/1912923065 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90101a6340 con 0x7f9010108440 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015400640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9010109630 0x7f901019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.774+0000 7f9015e00640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 0x7f901019e720 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9004000c00 tx=0x7f900402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.775+0000 7f900aa00640 1 -- 172.21.15.81:0/1912923065 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004041070 con 0x7f9010108440 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.775+0000 7f900aa00640 1 -- 172.21.15.81:0/1912923065 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9004004450 con 0x7f9010108440 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.775+0000 7f900aa00640 1 -- 172.21.15.81:0/1912923065 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9004034440 con 0x7f9010108440 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.775+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90101a65d0 con 0x7f9010108440 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.775+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90101a6a70 con 0x7f9010108440 2024-04-12T21:15:12.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.776+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f901010f7e0 con 0x7f9010108440 2024-04-12T21:15:12.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.778+0000 7f900aa00640 1 -- 172.21.15.81:0/1912923065 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9004044070 con 0x7f9010108440 2024-04-12T21:15:12.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.779+0000 7f900aa00640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ff80a3bb0 0x7f8ff80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:12.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.779+0000 7f900aa00640 1 -- 172.21.15.81:0/1912923065 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f90040f2350 con 0x7f9010108440 2024-04-12T21:15:12.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.779+0000 7f9015400640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ff80a3bb0 0x7f8ff80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:12.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.780+0000 7f9015400640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ff80a3bb0 0x7f8ff80a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f901019fc30 tx=0x7f9000005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:12.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.784+0000 7f900aa00640 1 -- 172.21.15.81:0/1912923065 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90040bea00 con 0x7f9010108440 2024-04-12T21:15:12.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:12 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2593082238' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T21:15:12.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:12 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2593082238' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T21:15:12.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:12 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2593082238' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-12T21:15:12.980 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.983+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9010062bf0 con 0x7f9010108440 2024-04-12T21:15:12.984 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.987+0000 7f900aa00640 1 -- 172.21.15.81:0/1912923065 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24681 (secure 0 0 0) 0x7f90040be300 con 0x7f9010108440 2024-04-12T21:15:12.985 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:12.985 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":91,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:14:23.492158+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ff80a3bb0 msgr2=0x7f8ff80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ff80a3bb0 0x7f8ff80a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f901019fc30 tx=0x7f9000005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 msgr2=0x7f901019e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 0x7f901019e720 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9004000c00 tx=0x7f900402c640 comp rx=0 tx=0).stop 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 shutdown_connections 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ff80a3bb0 0x7f8ff80a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9010109ff0 0x7f90101a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9010109630 0x7f901019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 --2- 172.21.15.81:0/1912923065 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9010108440 0x7f901019e720 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:12.987 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 >> 172.21.15.81:0/1912923065 conn(0x7f9010077400 msgr2=0x7f901010c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:12.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.990+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 shutdown_connections 2024-04-12T21:15:12.988 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:12.991+0000 7f9017200640 1 -- 172.21.15.81:0/1912923065 wait complete. 2024-04-12T21:15:13.007 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-12T21:13:10.561238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-12T21:13:11.985544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-12T21:13:15.579438+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-12T21:15:13.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 -- 172.21.15.81:0/3597568652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 msgr2=0x7f4c281023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 --2- 172.21.15.81:0/3597568652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 0x7f4c281023f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009900 tx=0x7f4c1002bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 -- 172.21.15.81:0/3597568652 shutdown_connections 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 --2- 172.21.15.81:0/3597568652 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c28102930 0x7f4c28109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 --2- 172.21.15.81:0/3597568652 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 0x7f4c281023f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 --2- 172.21.15.81:0/3597568652 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28100e20 0x7f4c28101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 -- 172.21.15.81:0/3597568652 >> 172.21.15.81:0/3597568652 conn(0x7f4c280fc390 msgr2=0x7f4c280fe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 -- 172.21.15.81:0/3597568652 shutdown_connections 2024-04-12T21:15:13.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.228+0000 7f4c27200640 1 -- 172.21.15.81:0/3597568652 wait complete. 2024-04-12T21:15:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.229+0000 7f4c27200640 1 Processor -- start 2024-04-12T21:15:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.229+0000 7f4c27200640 1 -- start start 2024-04-12T21:15:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c27200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28100e20 0x7f4c2810b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c27200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 0x7f4c28109e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c27200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c28102930 0x7f4c2810a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c27200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2810dbd0 con 0x7f4c28102930 2024-04-12T21:15:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c27200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2810da50 con 0x7f4c28101f70 2024-04-12T21:15:13.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c27200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2810dd50 con 0x7f4c28100e20 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c25400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 0x7f4c28109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c25400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 0x7f4c28109e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:58604/0 (socket says 172.21.15.81:58604) 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c25400640 1 -- 172.21.15.81:0/2461865390 learned_addr learned my addr 172.21.15.81:0/2461865390 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c26800640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c28102930 0x7f4c2810a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.230+0000 7f4c25e00640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28100e20 0x7f4c2810b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c26800640 1 -- 172.21.15.81:0/2461865390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28100e20 msgr2=0x7f4c2810b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c26800640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28100e20 0x7f4c2810b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c26800640 1 -- 172.21.15.81:0/2461865390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 msgr2=0x7f4c28109e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c26800640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 0x7f4c28109e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c26800640 1 -- 172.21.15.81:0/2461865390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2810abd0 con 0x7f4c28102930 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c25e00640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28100e20 0x7f4c2810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c25400640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 0x7f4c28109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:13.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c26800640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c28102930 0x7f4c2810a340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800e970 tx=0x7f4c1800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:13.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c1ea00640 1 -- 172.21.15.81:0/2461865390 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1800ccc0 con 0x7f4c28102930 2024-04-12T21:15:13.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c281a2320 con 0x7f4c28102930 2024-04-12T21:15:13.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c1ea00640 1 -- 172.21.15.81:0/2461865390 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c18005020 con 0x7f4c28102930 2024-04-12T21:15:13.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c281a2840 con 0x7f4c28102930 2024-04-12T21:15:13.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.231+0000 7f4c1ea00640 1 -- 172.21.15.81:0/2461865390 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1801c070 con 0x7f4c28102930 2024-04-12T21:15:13.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.232+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c28068d00 con 0x7f4c28102930 2024-04-12T21:15:13.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.234+0000 7f4c1ea00640 1 -- 172.21.15.81:0/2461865390 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4c180051d0 con 0x7f4c28102930 2024-04-12T21:15:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.236+0000 7f4c1ea00640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.236+0000 7f4c1ea00640 1 -- 172.21.15.81:0/2461865390 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4c180d0770 con 0x7f4c28102930 2024-04-12T21:15:13.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.236+0000 7f4c25e00640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:13.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.237+0000 7f4c25e00640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009fd0 tx=0x7f4c20009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:13.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.244+0000 7f4c1ea00640 1 -- 172.21.15.81:0/2461865390 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c1809cd90 con 0x7f4c28102930 2024-04-12T21:15:13.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.447+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4c28062bc0 con 0x7f4c28102930 2024-04-12T21:15:13.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.448+0000 7f4c1ea00640 1 -- 172.21.15.81:0/2461865390 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f4c1809c690 con 0x7f4c28102930 2024-04-12T21:15:13.445 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c0c0a3bb0 msgr2=0x7f4c0c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009fd0 tx=0x7f4c20009680 comp rx=0 tx=0).stop 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c28102930 msgr2=0x7f4c2810a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c28102930 0x7f4c2810a340 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800e970 tx=0x7f4c1800ee30 comp rx=0 tx=0).stop 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 shutdown_connections 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4c0c0a3bb0 0x7f4c0c0a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4c28102930 0x7f4c2810a340 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4c28101f70 0x7f4c28109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 --2- 172.21.15.81:0/2461865390 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4c28100e20 0x7f4c2810b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 >> 172.21.15.81:0/2461865390 conn(0x7f4c280fc390 msgr2=0x7f4c28105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 shutdown_connections 2024-04-12T21:15:13.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.451+0000 7f4c27200640 1 -- 172.21.15.81:0/2461865390 wait complete. 2024-04-12T21:15:13.468 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-12T21:15:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 -- 172.21.15.81:0/1102883218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c00104100 msgr2=0x7f3c00104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 --2- 172.21.15.81:0/1102883218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c00104100 0x7f3c00104520 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3bf0009860 tx=0x7f3bf002bb70 comp rx=0 tx=0).stop 2024-04-12T21:15:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 -- 172.21.15.81:0/1102883218 shutdown_connections 2024-04-12T21:15:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 --2- 172.21.15.81:0/1102883218 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c00105cb0 0x7f3c0010c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 --2- 172.21.15.81:0/1102883218 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c001052f0 0x7f3c00105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 --2- 172.21.15.81:0/1102883218 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c00104100 0x7f3c00104520 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 -- 172.21.15.81:0/1102883218 >> 172.21.15.81:0/1102883218 conn(0x7f3c000ff670 msgr2=0x7f3c00101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:13.682 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 -- 172.21.15.81:0/1102883218 shutdown_connections 2024-04-12T21:15:13.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.685+0000 7f3bff200640 1 -- 172.21.15.81:0/1102883218 wait complete. 2024-04-12T21:15:13.683 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.686+0000 7f3bff200640 1 Processor -- start 2024-04-12T21:15:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bff200640 1 -- start start 2024-04-12T21:15:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bff200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c00104100 0x7f3c0019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bff200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c001052f0 0x7f3c0019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bff200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c00105cb0 0x7f3c001a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bff200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0010fd60 con 0x7f3c00105cb0 2024-04-12T21:15:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bff200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0010fbe0 con 0x7f3c001052f0 2024-04-12T21:15:13.684 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bff200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0010fee0 con 0x7f3c00104100 2024-04-12T21:15:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c00104100 0x7f3c0019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c00105cb0 0x7f3c001a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bfde00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c00104100 0x7f3c0019a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:58100/0 (socket says 172.21.15.81:58100) 2024-04-12T21:15:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.687+0000 7f3bfde00640 1 -- 172.21.15.81:0/2279731734 learned_addr learned my addr 172.21.15.81:0/2279731734 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.688+0000 7f3bfd400640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c001052f0 0x7f3c0019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.688+0000 7f3bfde00640 1 -- 172.21.15.81:0/2279731734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c001052f0 msgr2=0x7f3c0019a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.685 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.688+0000 7f3bfde00640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c001052f0 0x7f3c0019a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.688+0000 7f3bfde00640 1 -- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c00105cb0 msgr2=0x7f3c001a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.688+0000 7f3bfde00640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c00105cb0 0x7f3c001a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.688+0000 7f3bfde00640 1 -- 172.21.15.81:0/2279731734 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c001a2070 con 0x7f3c00104100 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.688+0000 7f3bfde00640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c00104100 0x7f3c0019a450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00349e0 tx=0x7f3bf00042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.689+0000 7f3bf6a00640 1 -- 172.21.15.81:0/2279731734 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0041020 con 0x7f3c00104100 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.689+0000 7f3bf6a00640 1 -- 172.21.15.81:0/2279731734 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bf00323b0 con 0x7f3c00104100 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.689+0000 7f3bf6a00640 1 -- 172.21.15.81:0/2279731734 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0039040 con 0x7f3c00104100 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.689+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c001a2300 con 0x7f3c00104100 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.689+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c001a2840 con 0x7f3c00104100 2024-04-12T21:15:13.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.690+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c0010b4f0 con 0x7f3c00104100 2024-04-12T21:15:13.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.692+0000 7f3bf6a00640 1 -- 172.21.15.81:0/2279731734 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3bf0044020 con 0x7f3c00104100 2024-04-12T21:15:13.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.693+0000 7f3bf6a00640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3be40a3c30 0x7f3be40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.693+0000 7f3bf6a00640 1 -- 172.21.15.81:0/2279731734 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3bf00f2370 con 0x7f3c00104100 2024-04-12T21:15:13.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.694+0000 7f3bfd400640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3be40a3c30 0x7f3be40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:13.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.694+0000 7f3bfd400640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3be40a3c30 0x7f3be40a60f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c0019b960 tx=0x7f3bec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:13.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.696+0000 7f3bf6a00640 1 -- 172.21.15.81:0/2279731734 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3bf004b020 con 0x7f3c00104100 2024-04-12T21:15:13.729 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:13 smithi104 ceph-mon[36624]: pgmap v376: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:13.729 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:13 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1912923065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:13.729 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:13 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2461865390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:13 smithi153 ceph-mon[36569]: pgmap v376: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:13 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1912923065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:13 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2461865390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:13.893 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.897+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3c00062bc0 con 0x7f3c00104100 2024-04-12T21:15:13.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.898+0000 7f3bf6a00640 1 -- 172.21.15.81:0/2279731734 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f3bf00bea70 con 0x7f3c00104100 2024-04-12T21:15:13.894 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3be40a3c30 msgr2=0x7f3be40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3be40a3c30 0x7f3be40a60f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c0019b960 tx=0x7f3bec009260 comp rx=0 tx=0).stop 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c00104100 msgr2=0x7f3c0019a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c00104100 0x7f3c0019a450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3bf00349e0 tx=0x7f3bf00042f0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 shutdown_connections 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3be40a3c30 0x7f3be40a60f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c00105cb0 0x7f3c001a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c001052f0 0x7f3c0019a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 --2- 172.21.15.81:0/2279731734 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c00104100 0x7f3c0019a450 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:13.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.900+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 >> 172.21.15.81:0/2279731734 conn(0x7f3c000ff670 msgr2=0x7f3c001086c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:13.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.901+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 shutdown_connections 2024-04-12T21:15:13.898 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:13.901+0000 7f3bff200640 1 -- 172.21.15.81:0/2279731734 wait complete. 2024-04-12T21:15:13.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:13 smithi081 ceph-mon[31453]: pgmap v376: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:13.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:13 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1912923065' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:13.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:13 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2461865390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:13.917 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-12T21:15:14.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.142+0000 7f5657e00640 1 -- 172.21.15.81:0/659567568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f565810a0b0 msgr2=0x7f5658110940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.142+0000 7f5657e00640 1 --2- 172.21.15.81:0/659567568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f565810a0b0 0x7f5658110940 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5650009ca0 tx=0x7f565002c100 comp rx=0 tx=0).stop 2024-04-12T21:15:14.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.142+0000 7f5657e00640 1 -- 172.21.15.81:0/659567568 shutdown_connections 2024-04-12T21:15:14.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.142+0000 7f5657e00640 1 --2- 172.21.15.81:0/659567568 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f565810a0b0 0x7f5658110940 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.142+0000 7f5657e00640 1 --2- 172.21.15.81:0/659567568 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f56581096f0 0x7f5658109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.142+0000 7f5657e00640 1 --2- 172.21.15.81:0/659567568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5658108500 0x7f5658108920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.142+0000 7f5657e00640 1 -- 172.21.15.81:0/659567568 >> 172.21.15.81:0/659567568 conn(0x7f5658077490 msgr2=0x7f56580798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:14.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.143+0000 7f5657e00640 1 -- 172.21.15.81:0/659567568 shutdown_connections 2024-04-12T21:15:14.140 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.143+0000 7f5657e00640 1 -- 172.21.15.81:0/659567568 wait complete. 2024-04-12T21:15:14.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657e00640 1 Processor -- start 2024-04-12T21:15:14.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657e00640 1 -- start start 2024-04-12T21:15:14.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5658108500 0x7f565819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:14.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581096f0 0x7f565819ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:14.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f565810a0b0 0x7f56581a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:14.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f5658114160 con 0x7f565810a0b0 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f5658113fe0 con 0x7f5658108500 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f56581142e0 con 0x7f56581096f0 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5656a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5658108500 0x7f565819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5656000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581096f0 0x7f565819ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5656000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581096f0 0x7f565819ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:58120/0 (socket says 172.21.15.81:58120) 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5656000640 1 -- 172.21.15.81:0/4091311328 learned_addr learned my addr 172.21.15.81:0/4091311328 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.144+0000 7f5657400640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f565810a0b0 0x7f56581a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f5656000640 1 -- 172.21.15.81:0/4091311328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5658108500 msgr2=0x7f565819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f5656000640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5658108500 0x7f565819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f5656000640 1 -- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f565810a0b0 msgr2=0x7f56581a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f5656000640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f565810a0b0 0x7f56581a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f5656000640 1 -- 172.21.15.81:0/4091311328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56581a63a0 con 0x7f56581096f0 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f5656a00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5658108500 0x7f565819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f5656000640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581096f0 0x7f565819ecc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f564000e9f0 tx=0x7f564000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:14.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f564f400640 1 -- 172.21.15.81:0/4091311328 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564000cd80 con 0x7f56581096f0 2024-04-12T21:15:14.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56581a6690 con 0x7f56581096f0 2024-04-12T21:15:14.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f564f400640 1 -- 172.21.15.81:0/4091311328 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5640010910 con 0x7f56581096f0 2024-04-12T21:15:14.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.145+0000 7f564f400640 1 -- 172.21.15.81:0/4091311328 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56400188f0 con 0x7f56581096f0 2024-04-12T21:15:14.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.146+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56581a6be0 con 0x7f56581096f0 2024-04-12T21:15:14.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.147+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f565810f820 con 0x7f56581096f0 2024-04-12T21:15:14.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.149+0000 7f564f400640 1 -- 172.21.15.81:0/4091311328 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5640010ac0 con 0x7f56581096f0 2024-04-12T21:15:14.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.150+0000 7f564f400640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f563c0a3bb0 0x7f563c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:14.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.150+0000 7f564f400640 1 -- 172.21.15.81:0/4091311328 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f56400d0ef0 con 0x7f56581096f0 2024-04-12T21:15:14.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.150+0000 7f5656a00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f563c0a3bb0 0x7f563c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:14.147 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.151+0000 7f5656a00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f563c0a3bb0 0x7f563c0a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5658109550 tx=0x7f5648005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:14.149 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.152+0000 7f564f400640 1 -- 172.21.15.81:0/4091311328 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f564009cec0 con 0x7f56581096f0 2024-04-12T21:15:14.350 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.353+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f565802d770 con 0x7f56581096f0 2024-04-12T21:15:14.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.354+0000 7f564f400640 1 -- 172.21.15.81:0/4091311328 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f56400a11c0 con 0x7f56581096f0 2024-04-12T21:15:14.351 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-04-12T21:15:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f563c0a3bb0 msgr2=0x7f563c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f563c0a3bb0 0x7f563c0a6070 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5658109550 tx=0x7f5648005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581096f0 msgr2=0x7f565819ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581096f0 0x7f565819ecc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f564000e9f0 tx=0x7f564000eeb0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 shutdown_connections 2024-04-12T21:15:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f563c0a3bb0 0x7f563c0a6070 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f56581096f0 0x7f565819ecc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f565810a0b0 0x7f56581a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 --2- 172.21.15.81:0/4091311328 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5658108500 0x7f565819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 >> 172.21.15.81:0/4091311328 conn(0x7f5658077490 msgr2=0x7f565810c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:14.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 shutdown_connections 2024-04-12T21:15:14.355 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.357+0000 7f5657e00640 1 -- 172.21.15.81:0/4091311328 wait complete. 2024-04-12T21:15:14.374 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 -- 172.21.15.81:0/1766179014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 msgr2=0x7efe94108a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 --2- 172.21.15.81:0/1766179014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 0x7efe94108a40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efe84009900 tx=0x7efe8402bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 -- 172.21.15.81:0/1766179014 shutdown_connections 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 --2- 172.21.15.81:0/1766179014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94109810 0x7efe94109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 --2- 172.21.15.81:0/1766179014 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe9410a1d0 0x7efe94110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 --2- 172.21.15.81:0/1766179014 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 0x7efe94108a40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 -- 172.21.15.81:0/1766179014 >> 172.21.15.81:0/1766179014 conn(0x7efe940775c0 msgr2=0x7efe94079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 -- 172.21.15.81:0/1766179014 shutdown_connections 2024-04-12T21:15:14.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.595+0000 7efe99a00640 1 -- 172.21.15.81:0/1766179014 wait complete. 2024-04-12T21:15:14.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.596+0000 7efe99a00640 1 Processor -- start 2024-04-12T21:15:14.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe99a00640 1 -- start start 2024-04-12T21:15:14.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe99a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 0x7efe9419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe99a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94109810 0x7efe9419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe99a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe9410a1d0 0x7efe941a5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe99a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7efe941142a0 con 0x7efe9410a1d0 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe99a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7efe94114120 con 0x7efe94108620 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe99a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efe94114420 con 0x7efe94109810 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe93400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe9410a1d0 0x7efe941a5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe92a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 0x7efe9419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe92a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 0x7efe9419e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:58656/0 (socket says 172.21.15.81:58656) 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe92a00640 1 -- 172.21.15.81:0/224856391 learned_addr learned my addr 172.21.15.81:0/224856391 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe93400640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe9410a1d0 0x7efe941a5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38304/0 (socket says 172.21.15.81:38304) 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.597+0000 7efe8a000640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94109810 0x7efe9419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe93400640 1 -- 172.21.15.81:0/224856391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94109810 msgr2=0x7efe9419eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe93400640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94109810 0x7efe9419eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe93400640 1 -- 172.21.15.81:0/224856391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 msgr2=0x7efe9419e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe93400640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 0x7efe9419e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe93400640 1 -- 172.21.15.81:0/224856391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe941a65c0 con 0x7efe9410a1d0 2024-04-12T21:15:14.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe92a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 0x7efe9419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:14.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe8a000640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94109810 0x7efe9419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:14.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe93400640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe9410a1d0 0x7efe941a5f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009fd0 tx=0x7efe8c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:14.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe8be00640 1 -- 172.21.15.81:0/224856391 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c00ce20 con 0x7efe9410a1d0 2024-04-12T21:15:14.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe8be00640 1 -- 172.21.15.81:0/224856391 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe8c005020 con 0x7efe9410a1d0 2024-04-12T21:15:14.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.598+0000 7efe8be00640 1 -- 172.21.15.81:0/224856391 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c010720 con 0x7efe9410a1d0 2024-04-12T21:15:14.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.599+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe941a68b0 con 0x7efe9410a1d0 2024-04-12T21:15:14.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.599+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe941a6db0 con 0x7efe9410a1d0 2024-04-12T21:15:14.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.600+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe9410fa10 con 0x7efe9410a1d0 2024-04-12T21:15:14.598 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.602+0000 7efe8be00640 1 -- 172.21.15.81:0/224856391 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efe8c005540 con 0x7efe9410a1d0 2024-04-12T21:15:14.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.604+0000 7efe8be00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efe780a3c30 0x7efe780a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:14.600 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.604+0000 7efe8be00640 1 -- 172.21.15.81:0/224856391 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efe8c0d0de0 con 0x7efe9410a1d0 2024-04-12T21:15:14.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.604+0000 7efe92a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efe780a3c30 0x7efe780a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:14.601 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.605+0000 7efe92a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efe780a3c30 0x7efe780a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efe8402c8e0 tx=0x7efe84005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:14.605 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.609+0000 7efe8be00640 1 -- 172.21.15.81:0/224856391 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efe8c09d460 con 0x7efe9410a1d0 2024-04-12T21:15:14.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:14 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2279731734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:14.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:14 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4091311328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:14.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:14 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2279731734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:14.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:14 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4091311328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:14.806 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.810+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efe94062c60 con 0x7efe9410a1d0 2024-04-12T21:15:14.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.811+0000 7efe8be00640 1 -- 172.21.15.81:0/224856391 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7efe8c09cd60 con 0x7efe9410a1d0 2024-04-12T21:15:14.808 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efe780a3c30 msgr2=0x7efe780a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efe780a3c30 0x7efe780a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efe8402c8e0 tx=0x7efe84005b50 comp rx=0 tx=0).stop 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe9410a1d0 msgr2=0x7efe941a5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe9410a1d0 0x7efe941a5f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009fd0 tx=0x7efe8c00ef10 comp rx=0 tx=0).stop 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 shutdown_connections 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7efe780a3c30 0x7efe780a60f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efe94109810 0x7efe9419eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7efe9410a1d0 0x7efe941a5f50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 --2- 172.21.15.81:0/224856391 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7efe94108620 0x7efe9419e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 >> 172.21.15.81:0/224856391 conn(0x7efe940775c0 msgr2=0x7efe9410cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 shutdown_connections 2024-04-12T21:15:14.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:14.814+0000 7efe99a00640 1 -- 172.21.15.81:0/224856391 wait complete. 2024-04-12T21:15:14.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:14 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2279731734' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:14.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:14 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4091311328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:14.830 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 -- 172.21.15.81:0/3599858112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 msgr2=0x7f3700108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 --2- 172.21.15.81:0/3599858112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 0x7f3700108860 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f36f0009900 tx=0x7f36f002bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 -- 172.21.15.81:0/3599858112 shutdown_connections 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 --2- 172.21.15.81:0/3599858112 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3700109ff0 0x7f3700110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 --2- 172.21.15.81:0/3599858112 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3700109630 0x7f3700109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 --2- 172.21.15.81:0/3599858112 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 0x7f3700108860 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 -- 172.21.15.81:0/3599858112 >> 172.21.15.81:0/3599858112 conn(0x7f3700077400 msgr2=0x7f3700079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 -- 172.21.15.81:0/3599858112 shutdown_connections 2024-04-12T21:15:15.048 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.051+0000 7f3704a00640 1 -- 172.21.15.81:0/3599858112 wait complete. 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.052+0000 7f3704a00640 1 Processor -- start 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.052+0000 7f3704a00640 1 -- start start 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.052+0000 7f3704a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 0x7f370019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.052+0000 7f3704a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3700109630 0x7f370019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.052+0000 7f3704a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3700109ff0 0x7f37001a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.052+0000 7f3704a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3700114120 con 0x7f3700109630 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.052+0000 7f3704a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3700113fa0 con 0x7f3700109ff0 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.052+0000 7f3704a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f37001142a0 con 0x7f3700108440 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36f6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3700109630 0x7f370019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36ff400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 0x7f370019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36ff400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 0x7f370019e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:49266/0 (socket says 172.21.15.81:49266) 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36ff400640 1 -- 172.21.15.81:0/38892624 learned_addr learned my addr 172.21.15.81:0/38892624 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:15.050 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36ffe00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3700109ff0 0x7f37001a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:15.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36f6a00640 1 -- 172.21.15.81:0/38892624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 msgr2=0x7f370019e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36f6a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 0x7f370019e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36f6a00640 1 -- 172.21.15.81:0/38892624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3700109ff0 msgr2=0x7f37001a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36f6a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3700109ff0 0x7f37001a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36f6a00640 1 -- 172.21.15.81:0/38892624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37001a6370 con 0x7f3700109630 2024-04-12T21:15:15.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.053+0000 7f36f6a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3700109630 0x7f370019ec90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f36e800e9f0 tx=0x7f36e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:15.051 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.054+0000 7f36fcc00640 1 -- 172.21.15.81:0/38892624 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e800cd80 con 0x7f3700109630 2024-04-12T21:15:15.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.054+0000 7f36fcc00640 1 -- 172.21.15.81:0/38892624 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e80035f0 con 0x7f3700109630 2024-04-12T21:15:15.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.054+0000 7f36fcc00640 1 -- 172.21.15.81:0/38892624 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8010690 con 0x7f3700109630 2024-04-12T21:15:15.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.054+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37001a6660 con 0x7f3700109630 2024-04-12T21:15:15.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.054+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37001a6b60 con 0x7f3700109630 2024-04-12T21:15:15.052 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.055+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f370010f760 con 0x7f3700109630 2024-04-12T21:15:15.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.057+0000 7f36fcc00640 1 -- 172.21.15.81:0/38892624 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f36e8003b30 con 0x7f3700109630 2024-04-12T21:15:15.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.059+0000 7f36fcc00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36e40a3bb0 0x7f36e40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:15.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.059+0000 7f36fcc00640 1 -- 172.21.15.81:0/38892624 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f36e80d0a70 con 0x7f3700109630 2024-04-12T21:15:15.055 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.059+0000 7f36ff400640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36e40a3bb0 0x7f36e40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:15.056 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.060+0000 7f36ff400640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36e40a3bb0 0x7f36e40a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f36f0006010 tx=0x7f36f002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:15.059 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.063+0000 7f36fcc00640 1 -- 172.21.15.81:0/38892624 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36e809d090 con 0x7f3700109630 2024-04-12T21:15:15.303 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.306+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f370002d5e0 con 0x7f3700109630 2024-04-12T21:15:15.304 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.307+0000 7f36fcc00640 1 -- 172.21.15.81:0/38892624 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11815 (secure 0 0 0) 0x7f36e809c990 con 0x7f3700109630 2024-04-12T21:15:15.304 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:15.304 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T21:13:13.003153+0000","modified":"2024-04-12T21:13:37.617228+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-12T21:15:15.305 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36e40a3bb0 msgr2=0x7f36e40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36e40a3bb0 0x7f36e40a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f36f0006010 tx=0x7f36f002df90 comp rx=0 tx=0).stop 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3700109630 msgr2=0x7f370019ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3700109630 0x7f370019ec90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f36e800e9f0 tx=0x7f36e800eeb0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 shutdown_connections 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f36e40a3bb0 0x7f36e40a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3700109ff0 0x7f37001a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3700109630 0x7f370019ec90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.307 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 --2- 172.21.15.81:0/38892624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3700108440 0x7f370019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.310+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 >> 172.21.15.81:0/38892624 conn(0x7f3700077400 msgr2=0x7f370010c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:15.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.311+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 shutdown_connections 2024-04-12T21:15:15.308 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.311+0000 7f3704a00640 1 -- 172.21.15.81:0/38892624 wait complete. 2024-04-12T21:15:15.326 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:15:15.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 -- 172.21.15.81:0/2768402013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105230 msgr2=0x7f84d01056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 --2- 172.21.15.81:0/2768402013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105230 0x7f84d01056b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009900 tx=0x7f84bc02bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 -- 172.21.15.81:0/2768402013 shutdown_connections 2024-04-12T21:15:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 --2- 172.21.15.81:0/2768402013 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84d0105bf0 0x7f84d010c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 --2- 172.21.15.81:0/2768402013 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105230 0x7f84d01056b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 --2- 172.21.15.81:0/2768402013 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d0104040 0x7f84d0104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 -- 172.21.15.81:0/2768402013 >> 172.21.15.81:0/2768402013 conn(0x7f84d00ff5d0 msgr2=0x7f84d01019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 -- 172.21.15.81:0/2768402013 shutdown_connections 2024-04-12T21:15:15.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.550+0000 7f84d4e00640 1 -- 172.21.15.81:0/2768402013 wait complete. 2024-04-12T21:15:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.551+0000 7f84d4e00640 1 Processor -- start 2024-04-12T21:15:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.551+0000 7f84d4e00640 1 -- start start 2024-04-12T21:15:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.551+0000 7f84d4e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d0104040 0x7f84d019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.551+0000 7f84d4e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84d0105230 0x7f84d019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.551+0000 7f84d4e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105bf0 0x7f84d01a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.551+0000 7f84d4e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f84d010fd20 con 0x7f84d0105bf0 2024-04-12T21:15:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.551+0000 7f84d4e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f84d010fba0 con 0x7f84d0104040 2024-04-12T21:15:15.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.551+0000 7f84d4e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f84d010fea0 con 0x7f84d0105230 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105bf0 0x7f84d01a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84d0105230 0x7f84d019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cea00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84d0105230 0x7f84d019a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:49294/0 (socket says 172.21.15.81:49294) 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cfe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105bf0 0x7f84d01a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38478/0 (socket says 172.21.15.81:38478) 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cea00640 1 -- 172.21.15.81:0/3718212463 learned_addr learned my addr 172.21.15.81:0/3718212463 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cf400640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d0104040 0x7f84d019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cf400640 1 -- 172.21.15.81:0/3718212463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84d0105230 msgr2=0x7f84d019a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cf400640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84d0105230 0x7f84d019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cf400640 1 -- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105bf0 msgr2=0x7f84d01a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cf400640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105bf0 0x7f84d01a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cf400640 1 -- 172.21.15.81:0/3718212463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d01a1f40 con 0x7f84d0104040 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.552+0000 7f84cfe00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105bf0 0x7f84d01a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:15.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.553+0000 7f84cf400640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d0104040 0x7f84d019a320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84c000e9c0 tx=0x7f84c000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:15.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.553+0000 7f84c7e00640 1 -- 172.21.15.81:0/3718212463 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c000cd80 con 0x7f84d0104040 2024-04-12T21:15:15.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.553+0000 7f84c7e00640 1 -- 172.21.15.81:0/3718212463 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84c0010910 con 0x7f84d0104040 2024-04-12T21:15:15.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.553+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d01a2230 con 0x7f84d0104040 2024-04-12T21:15:15.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.553+0000 7f84c7e00640 1 -- 172.21.15.81:0/3718212463 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c0018980 con 0x7f84d0104040 2024-04-12T21:15:15.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.553+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d01a27a0 con 0x7f84d0104040 2024-04-12T21:15:15.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.554+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d010b360 con 0x7f84d0104040 2024-04-12T21:15:15.553 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.556+0000 7f84c7e00640 1 -- 172.21.15.81:0/3718212463 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f84c001f020 con 0x7f84d0104040 2024-04-12T21:15:15.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.558+0000 7f84c7e00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84b40a3c30 0x7f84b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:15.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.558+0000 7f84c7e00640 1 -- 172.21.15.81:0/3718212463 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f84c00d0980 con 0x7f84d0104040 2024-04-12T21:15:15.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.558+0000 7f84cea00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84b40a3c30 0x7f84b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:15.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.559+0000 7f84cea00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84b40a3c30 0x7f84b40a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f84bc02c8e0 tx=0x7f84bc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:15.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.560+0000 7f84c7e00640 1 -- 172.21.15.81:0/3718212463 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f84c009cfa0 con 0x7f84d0104040 2024-04-12T21:15:15.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:15 smithi081 ceph-mon[31453]: pgmap v377: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:15.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:15 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/224856391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:15.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:15 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/38892624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:15.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.777+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84d0062bc0 con 0x7f84d0104040 2024-04-12T21:15:15.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.780+0000 7f84c7e00640 1 -- 172.21.15.81:0/3718212463 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24681 (secure 0 0 0) 0x7f84c009c8a0 con 0x7f84d0104040 2024-04-12T21:15:15.777 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:15.777 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":91,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:14:23.492158+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:15:15.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.782+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84b40a3c30 msgr2=0x7f84b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.782+0000 7f84d4e00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84b40a3c30 0x7f84b40a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f84bc02c8e0 tx=0x7f84bc02df90 comp rx=0 tx=0).stop 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.782+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d0104040 msgr2=0x7f84d019a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.782+0000 7f84d4e00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d0104040 0x7f84d019a320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f84c000e9c0 tx=0x7f84c000ee80 comp rx=0 tx=0).stop 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.783+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 shutdown_connections 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.783+0000 7f84d4e00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f84b40a3c30 0x7f84b40a60f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.783+0000 7f84d4e00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f84d0105230 0x7f84d019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.783+0000 7f84d4e00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f84d0105bf0 0x7f84d01a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.783+0000 7f84d4e00640 1 --2- 172.21.15.81:0/3718212463 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84d0104040 0x7f84d019a320 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.783+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 >> 172.21.15.81:0/3718212463 conn(0x7f84d00ff5d0 msgr2=0x7f84d0108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.783+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 shutdown_connections 2024-04-12T21:15:15.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:15.783+0000 7f84d4e00640 1 -- 172.21.15.81:0/3718212463 wait complete. 2024-04-12T21:15:15.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-12T21:13:10.561238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-12T21:13:11.985544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-12T21:13:15.579438+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-12T21:15:15.800 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T21:15:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:15 smithi153 ceph-mon[36569]: pgmap v377: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:15 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/224856391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:15 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/38892624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:15 smithi104 ceph-mon[36624]: pgmap v377: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:15 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/224856391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:15 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/38892624' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 -- 172.21.15.81:0/429703473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103070 msgr2=0x7f4ec01034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/429703473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103070 0x7f4ec01034f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009900 tx=0x7f4eac02bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 -- 172.21.15.81:0/429703473 shutdown_connections 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/429703473 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ec0103a30 0x7f4ec010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/429703473 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103070 0x7f4ec01034f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/429703473 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ec0101f20 0x7f4ec0102340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 -- 172.21.15.81:0/429703473 >> 172.21.15.81:0/429703473 conn(0x7f4ec00fd490 msgr2=0x7f4ec00ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 -- 172.21.15.81:0/429703473 shutdown_connections 2024-04-12T21:15:16.006 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.009+0000 7f4ec6c00640 1 -- 172.21.15.81:0/429703473 wait complete. 2024-04-12T21:15:16.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.010+0000 7f4ec6c00640 1 Processor -- start 2024-04-12T21:15:16.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.010+0000 7f4ec6c00640 1 -- start start 2024-04-12T21:15:16.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.010+0000 7f4ec6c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ec0101f20 0x7f4ec019a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.010+0000 7f4ec6c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ec0103070 0x7f4ec019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.010+0000 7f4ec6c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103a30 0x7f4ec01a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.010+0000 7f4ec6c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec010dbb0 con 0x7f4ec0103a30 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.010+0000 7f4ec6c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec010da30 con 0x7f4ec0103070 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.010+0000 7f4ec6c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec010dd30 con 0x7f4ec0101f20 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103a30 0x7f4ec01a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103a30 0x7f4ec01a1a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38506/0 (socket says 172.21.15.81:38506) 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 -- 172.21.15.81:0/3405941817 learned_addr learned my addr 172.21.15.81:0/3405941817 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec4e00640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ec0103070 0x7f4ec019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec5800640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ec0101f20 0x7f4ec019a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 -- 172.21.15.81:0/3405941817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ec0101f20 msgr2=0x7f4ec019a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ec0101f20 0x7f4ec019a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 -- 172.21.15.81:0/3405941817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ec0103070 msgr2=0x7f4ec019a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ec0103070 0x7f4ec019a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 -- 172.21.15.81:0/3405941817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec01a2080 con 0x7f4ec0103a30 2024-04-12T21:15:16.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.011+0000 7f4ec6200640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103a30 0x7f4ec01a1a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800eb10 tx=0x7f4eb800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:16.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.012+0000 7f4ebe000640 1 -- 172.21.15.81:0/3405941817 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8004b20 con 0x7f4ec0103a30 2024-04-12T21:15:16.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.012+0000 7f4ebe000640 1 -- 172.21.15.81:0/3405941817 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb8004cc0 con 0x7f4ec0103a30 2024-04-12T21:15:16.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.012+0000 7f4ebe000640 1 -- 172.21.15.81:0/3405941817 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8010760 con 0x7f4ec0103a30 2024-04-12T21:15:16.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.012+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec01a2370 con 0x7f4ec0103a30 2024-04-12T21:15:16.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.012+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec01a2840 con 0x7f4ec0103a30 2024-04-12T21:15:16.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.013+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec01a2b00 con 0x7f4ec0103a30 2024-04-12T21:15:16.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.015+0000 7f4ebe000640 1 -- 172.21.15.81:0/3405941817 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4eb8005020 con 0x7f4ec0103a30 2024-04-12T21:15:16.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.016+0000 7f4ebe000640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4ea80a3bb0 0x7f4ea80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.016+0000 7f4ebe000640 1 -- 172.21.15.81:0/3405941817 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4eb80d0d30 con 0x7f4ec0103a30 2024-04-12T21:15:16.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.017+0000 7f4ec5800640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4ea80a3bb0 0x7f4ea80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.017+0000 7f4ec5800640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4ea80a3bb0 0x7f4ea80a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009bd0 tx=0x7f4eb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:16.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.021+0000 7f4ebe000640 1 -- 172.21.15.81:0/3405941817 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4eb809d350 con 0x7f4ec0103a30 2024-04-12T21:15:16.207 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.210+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4ec01a2cd0 con 0x7f4ec0103a30 2024-04-12T21:15:16.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.211+0000 7f4ebe000640 1 -- 172.21.15.81:0/3405941817 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f4eb809cc50 con 0x7f4ec0103a30 2024-04-12T21:15:16.208 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:15:16.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4ea80a3bb0 msgr2=0x7f4ea80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.210 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4ea80a3bb0 0x7f4ea80a6070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009bd0 tx=0x7f4eb4005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103a30 msgr2=0x7f4ec01a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103a30 0x7f4ec01a1a10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800eb10 tx=0x7f4eb800c2d0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 shutdown_connections 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f4ea80a3bb0 0x7f4ea80a6070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4ec0103a30 0x7f4ec01a1a10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4ec0103070 0x7f4ec019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 --2- 172.21.15.81:0/3405941817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ec0101f20 0x7f4ec019a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 >> 172.21.15.81:0/3405941817 conn(0x7f4ec00fd490 msgr2=0x7f4ec00741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 shutdown_connections 2024-04-12T21:15:16.211 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.214+0000 7f4ec6c00640 1 -- 172.21.15.81:0/3405941817 wait complete. 2024-04-12T21:15:16.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-12T21:15:16.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.444+0000 7f728f200640 1 -- 172.21.15.81:0/1551512510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 msgr2=0x7f72900ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.444+0000 7f728f200640 1 --2- 172.21.15.81:0/1551512510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 0x7f72900ffee0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f7280009900 tx=0x7f728002bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:16.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.444+0000 7f728f200640 1 -- 172.21.15.81:0/1551512510 shutdown_connections 2024-04-12T21:15:16.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.444+0000 7f728f200640 1 --2- 172.21.15.81:0/1551512510 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7290107bd0 0x7f7290109fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.445+0000 7f728f200640 1 --2- 172.21.15.81:0/1551512510 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7290100420 0x7f7290107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.445+0000 7f728f200640 1 --2- 172.21.15.81:0/1551512510 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 0x7f72900ffee0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.445+0000 7f728f200640 1 -- 172.21.15.81:0/1551512510 >> 172.21.15.81:0/1551512510 conn(0x7f72900fb680 msgr2=0x7f72900fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:16.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.445+0000 7f728f200640 1 -- 172.21.15.81:0/1551512510 shutdown_connections 2024-04-12T21:15:16.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.445+0000 7f728f200640 1 -- 172.21.15.81:0/1551512510 wait complete. 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728f200640 1 Processor -- start 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728f200640 1 -- start start 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728f200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 0x7f729019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728f200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7290100420 0x7f729019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728f200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7290107bd0 0x7f72901a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728f200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f729010db80 con 0x7f72900ffac0 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728f200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f729010da00 con 0x7f7290107bd0 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728f200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f729010dd00 con 0x7f7290100420 2024-04-12T21:15:16.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 0x7f729019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728de00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 0x7f729019a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38528/0 (socket says 172.21.15.81:38528) 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728de00640 1 -- 172.21.15.81:0/3364069964 learned_addr learned my addr 172.21.15.81:0/3364069964 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.446+0000 7f728e800640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7290107bd0 0x7f72901a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728d400640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7290100420 0x7f729019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728de00640 1 -- 172.21.15.81:0/3364069964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7290100420 msgr2=0x7f729019a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728de00640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7290100420 0x7f729019a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728de00640 1 -- 172.21.15.81:0/3364069964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7290107bd0 msgr2=0x7f72901a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728de00640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7290107bd0 0x7f72901a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728de00640 1 -- 172.21.15.81:0/3364069964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72901a2090 con 0x7f72900ffac0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728e800640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7290107bd0 0x7f72901a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728de00640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 0x7f729019a470 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f728002c8e0 tx=0x7f7280004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f7286a00640 1 -- 172.21.15.81:0/3364069964 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280043020 con 0x7f72900ffac0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f7286a00640 1 -- 172.21.15.81:0/3364069964 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72800054e0 con 0x7f72900ffac0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f7286a00640 1 -- 172.21.15.81:0/3364069964 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280003b50 con 0x7f72900ffac0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.447+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72901a2320 con 0x7f72900ffac0 2024-04-12T21:15:16.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.448+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72901a2820 con 0x7f72900ffac0 2024-04-12T21:15:16.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.449+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72901a2a20 con 0x7f72900ffac0 2024-04-12T21:15:16.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.451+0000 7f7286a00640 1 -- 172.21.15.81:0/3364069964 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7280046020 con 0x7f72900ffac0 2024-04-12T21:15:16.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.452+0000 7f7286a00640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f72740a3c30 0x7f72740a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.452+0000 7f7286a00640 1 -- 172.21.15.81:0/3364069964 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f72800f2720 con 0x7f72900ffac0 2024-04-12T21:15:16.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.453+0000 7f728d400640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f72740a3c30 0x7f72740a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.453+0000 7f728d400640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f72740a3c30 0x7f72740a60f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f727c0095f0 tx=0x7f727c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:16.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.457+0000 7f7286a00640 1 -- 172.21.15.81:0/3364069964 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f728004d020 con 0x7f72900ffac0 2024-04-12T21:15:16.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.660+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f72901a2bf0 con 0x7f72900ffac0 2024-04-12T21:15:16.658 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.661+0000 7f7286a00640 1 -- 172.21.15.81:0/3364069964 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f72800ba030 con 0x7f72900ffac0 2024-04-12T21:15:16.658 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f72740a3c30 msgr2=0x7f72740a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f72740a3c30 0x7f72740a60f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f727c0095f0 tx=0x7f727c005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 msgr2=0x7f729019a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 0x7f729019a470 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f728002c8e0 tx=0x7f7280004970 comp rx=0 tx=0).stop 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 shutdown_connections 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f72740a3c30 0x7f72740a60f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7290100420 0x7f729019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7290107bd0 0x7f72901a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 --2- 172.21.15.81:0/3364069964 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f72900ffac0 0x7f729019a470 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 >> 172.21.15.81:0/3364069964 conn(0x7f72900fb680 msgr2=0x7f72900fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 shutdown_connections 2024-04-12T21:15:16.661 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.664+0000 7f728f200640 1 -- 172.21.15.81:0/3364069964 wait complete. 2024-04-12T21:15:16.680 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-12T21:15:16.760 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:16 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3718212463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:16.760 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:16 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3405941817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:16.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:16 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3718212463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:16.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:16 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3405941817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:16.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:16 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3718212463' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:16.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:16 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3405941817' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 -- 172.21.15.81:0/1261406292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 msgr2=0x7f0f3810c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 --2- 172.21.15.81:0/1261406292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 0x7f0f3810c660 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009ca0 tx=0x7f0f3002c0f0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 -- 172.21.15.81:0/1261406292 shutdown_connections 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 --2- 172.21.15.81:0/1261406292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f38105410 0x7f0f38105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 --2- 172.21.15.81:0/1261406292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 0x7f0f3810c660 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 --2- 172.21.15.81:0/1261406292 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38104220 0x7f0f38104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 -- 172.21.15.81:0/1261406292 >> 172.21.15.81:0/1261406292 conn(0x7f0f380ff7b0 msgr2=0x7f0f38101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 -- 172.21.15.81:0/1261406292 shutdown_connections 2024-04-12T21:15:16.897 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.900+0000 7f0f3e000640 1 -- 172.21.15.81:0/1261406292 wait complete. 2024-04-12T21:15:16.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.901+0000 7f0f3e000640 1 Processor -- start 2024-04-12T21:15:16.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.901+0000 7f0f3e000640 1 -- start start 2024-04-12T21:15:16.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.901+0000 7f0f3e000640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f38104220 0x7f0f3819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.899 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.901+0000 7f0f3e000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38105410 0x7f0f3819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.901+0000 7f0f3e000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 0x7f0f381a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.901+0000 7f0f3e000640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810feb0 con 0x7f0f38105410 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.901+0000 7f0f3e000640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0f3810fd30 con 0x7f0f38105dd0 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.901+0000 7f0f3e000640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f38110030 con 0x7f0f38104220 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38105410 0x7f0f3819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f37e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 0x7f0f381a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f37400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f38104220 0x7f0f3819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f2fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38105410 0x7f0f3819aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38532/0 (socket says 172.21.15.81:38532) 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f2fe00640 1 -- 172.21.15.81:0/4102554521 learned_addr learned my addr 172.21.15.81:0/4102554521 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f37e00640 1 -- 172.21.15.81:0/4102554521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f38104220 msgr2=0x7f0f3819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f37e00640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f38104220 0x7f0f3819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f37e00640 1 -- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38105410 msgr2=0x7f0f3819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f37e00640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38105410 0x7f0f3819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:16.900 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f37e00640 1 -- 172.21.15.81:0/4102554521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f381a20f0 con 0x7f0f38105dd0 2024-04-12T21:15:16.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f2fe00640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38105410 0x7f0f3819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:16.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.902+0000 7f0f37e00640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 0x7f0f381a1a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0f30037040 tx=0x7f0f30032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:16.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.903+0000 7f0f34c00640 1 -- 172.21.15.81:0/4102554521 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f30041020 con 0x7f0f38105dd0 2024-04-12T21:15:16.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.903+0000 7f0f34c00640 1 -- 172.21.15.81:0/4102554521 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f30034cd0 con 0x7f0f38105dd0 2024-04-12T21:15:16.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.903+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f381a2380 con 0x7f0f38105dd0 2024-04-12T21:15:16.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.903+0000 7f0f34c00640 1 -- 172.21.15.81:0/4102554521 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f30039040 con 0x7f0f38105dd0 2024-04-12T21:15:16.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.903+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f381a2840 con 0x7f0f38105dd0 2024-04-12T21:15:16.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.904+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f3810b540 con 0x7f0f38105dd0 2024-04-12T21:15:16.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.906+0000 7f0f34c00640 1 -- 172.21.15.81:0/4102554521 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0f30044020 con 0x7f0f38105dd0 2024-04-12T21:15:16.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.907+0000 7f0f34c00640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0f1c0a3bb0 0x7f0f1c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:16.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.907+0000 7f0f34c00640 1 -- 172.21.15.81:0/4102554521 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0f300f29a0 con 0x7f0f38105dd0 2024-04-12T21:15:16.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.907+0000 7f0f37400640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0f1c0a3bb0 0x7f0f1c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:16.904 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.908+0000 7f0f37400640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0f1c0a3bb0 0x7f0f1c0a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0f38105270 tx=0x7f0f28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:16.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:16.909+0000 7f0f34c00640 1 -- 172.21.15.81:0/4102554521 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f3004b020 con 0x7f0f38105dd0 2024-04-12T21:15:17.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.113+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0f3802d5e0 con 0x7f0f38105dd0 2024-04-12T21:15:17.110 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.113+0000 7f0f34c00640 1 -- 172.21.15.81:0/4102554521 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f0f300bf050 con 0x7f0f38105dd0 2024-04-12T21:15:17.110 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-04-12T21:15:17.113 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0f1c0a3bb0 msgr2=0x7f0f1c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0f1c0a3bb0 0x7f0f1c0a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0f38105270 tx=0x7f0f28005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 msgr2=0x7f0f381a1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 0x7f0f381a1a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0f30037040 tx=0x7f0f30032ea0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 shutdown_connections 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0f1c0a3bb0 0x7f0f1c0a6070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f38105dd0 0x7f0f381a1a80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f38105410 0x7f0f3819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 --2- 172.21.15.81:0/4102554521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f38104220 0x7f0f3819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 >> 172.21.15.81:0/4102554521 conn(0x7f0f380ff7b0 msgr2=0x7f0f38108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 shutdown_connections 2024-04-12T21:15:17.114 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.117+0000 7f0f3e000640 1 -- 172.21.15.81:0/4102554521 wait complete. 2024-04-12T21:15:17.133 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-12T21:15:17.348 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.351+0000 7f421ae00640 1 -- 172.21.15.81:0/347361236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42141052d0 msgr2=0x7f4214105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.351+0000 7f421ae00640 1 --2- 172.21.15.81:0/347361236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42141052d0 0x7f4214105750 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4208009900 tx=0x7f420802bc40 comp rx=0 tx=0).stop 2024-04-12T21:15:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.352+0000 7f421ae00640 1 -- 172.21.15.81:0/347361236 shutdown_connections 2024-04-12T21:15:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.352+0000 7f421ae00640 1 --2- 172.21.15.81:0/347361236 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4214105c90 0x7f421410c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.352+0000 7f421ae00640 1 --2- 172.21.15.81:0/347361236 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f42141052d0 0x7f4214105750 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.352+0000 7f421ae00640 1 --2- 172.21.15.81:0/347361236 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42141040e0 0x7f4214104500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.352+0000 7f421ae00640 1 -- 172.21.15.81:0/347361236 >> 172.21.15.81:0/347361236 conn(0x7f42140ff670 msgr2=0x7f4214101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.352+0000 7f421ae00640 1 -- 172.21.15.81:0/347361236 shutdown_connections 2024-04-12T21:15:17.349 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.352+0000 7f421ae00640 1 -- 172.21.15.81:0/347361236 wait complete. 2024-04-12T21:15:17.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.353+0000 7f421ae00640 1 Processor -- start 2024-04-12T21:15:17.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.353+0000 7f421ae00640 1 -- start start 2024-04-12T21:15:17.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.353+0000 7f421ae00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42141040e0 0x7f421419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:17.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.353+0000 7f421ae00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42141052d0 0x7f421419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:17.351 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.353+0000 7f421ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214105c90 0x7f42141a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.353+0000 7f421ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f421410fdc0 con 0x7f4214105c90 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.353+0000 7f421ae00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f421410fc40 con 0x7f42141040e0 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.353+0000 7f421ae00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f421410ff40 con 0x7f42141052d0 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214105c90 0x7f42141a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214105c90 0x7f42141a19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38558/0 (socket says 172.21.15.81:38558) 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 -- 172.21.15.81:0/1863625755 learned_addr learned my addr 172.21.15.81:0/1863625755 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f4219000640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42141052d0 0x7f421419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f4219a00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42141040e0 0x7f421419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 -- 172.21.15.81:0/1863625755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42141052d0 msgr2=0x7f421419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42141052d0 0x7f421419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 -- 172.21.15.81:0/1863625755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42141040e0 msgr2=0x7f421419a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42141040e0 0x7f421419a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 -- 172.21.15.81:0/1863625755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42141a2050 con 0x7f4214105c90 2024-04-12T21:15:17.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421a400640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214105c90 0x7f42141a19e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f420c00eaa0 tx=0x7f420c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:17.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f4212000640 1 -- 172.21.15.81:0/1863625755 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c00cdb0 con 0x7f4214105c90 2024-04-12T21:15:17.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42141a2340 con 0x7f4214105c90 2024-04-12T21:15:17.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f4212000640 1 -- 172.21.15.81:0/1863625755 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f420c005020 con 0x7f4214105c90 2024-04-12T21:15:17.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.354+0000 7f4212000640 1 -- 172.21.15.81:0/1863625755 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420c0116b0 con 0x7f4214105c90 2024-04-12T21:15:17.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.355+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42141a2860 con 0x7f4214105c90 2024-04-12T21:15:17.353 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.356+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f421410b3a0 con 0x7f4214105c90 2024-04-12T21:15:17.354 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.358+0000 7f4212000640 1 -- 172.21.15.81:0/1863625755 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f420c005560 con 0x7f4214105c90 2024-04-12T21:15:17.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.360+0000 7f4212000640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3bb0 0x7f41fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:17.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.360+0000 7f4212000640 1 -- 172.21.15.81:0/1863625755 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f420c0d08f0 con 0x7f4214105c90 2024-04-12T21:15:17.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.360+0000 7f4219a00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3bb0 0x7f41fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:17.357 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.361+0000 7f4219a00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3bb0 0x7f41fc0a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4214105130 tx=0x7f4204005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:17.361 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.365+0000 7f4212000640 1 -- 172.21.15.81:0/1863625755 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f420c016030 con 0x7f4214105c90 2024-04-12T21:15:17.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.554+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4214062bc0 con 0x7f4214105c90 2024-04-12T21:15:17.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.555+0000 7f4212000640 1 -- 172.21.15.81:0/1863625755 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f420c09c920 con 0x7f4214105c90 2024-04-12T21:15:17.552 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:15:17.554 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.557+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3bb0 msgr2=0x7f41fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.557+0000 7f421ae00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3bb0 0x7f41fc0a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4214105130 tx=0x7f4204005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.557+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214105c90 msgr2=0x7f42141a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.557+0000 7f421ae00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214105c90 0x7f42141a19e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f420c00eaa0 tx=0x7f420c00ef60 comp rx=0 tx=0).stop 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.558+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 shutdown_connections 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.558+0000 7f421ae00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f41fc0a3bb0 0x7f41fc0a6070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.558+0000 7f421ae00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42141052d0 0x7f421419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.558+0000 7f421ae00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4214105c90 0x7f42141a19e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.558+0000 7f421ae00640 1 --2- 172.21.15.81:0/1863625755 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f42141040e0 0x7f421419a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.558+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 >> 172.21.15.81:0/1863625755 conn(0x7f42140ff670 msgr2=0x7f4214108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.558+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 shutdown_connections 2024-04-12T21:15:17.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.558+0000 7f421ae00640 1 -- 172.21.15.81:0/1863625755 wait complete. 2024-04-12T21:15:17.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:15:17.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 -- 172.21.15.81:0/1857463114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518078560 msgr2=0x7f3518078e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 --2- 172.21.15.81:0/1857463114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518078560 0x7f3518078e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3510009ca0 tx=0x7f351002c100 comp rx=0 tx=0).stop 2024-04-12T21:15:17.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 -- 172.21.15.81:0/1857463114 shutdown_connections 2024-04-12T21:15:17.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 --2- 172.21.15.81:0/1857463114 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518078560 0x7f3518078e00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 --2- 172.21.15.81:0/1857463114 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3518077bc0 0x7f3518078020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 --2- 172.21.15.81:0/1857463114 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518076980 0x7f3518076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 -- 172.21.15.81:0/1857463114 >> 172.21.15.81:0/1857463114 conn(0x7f35180ff5d0 msgr2=0x7f35181019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:17.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 -- 172.21.15.81:0/1857463114 shutdown_connections 2024-04-12T21:15:17.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.790+0000 7f351ce00640 1 -- 172.21.15.81:0/1857463114 wait complete. 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.791+0000 7f351ce00640 1 Processor -- start 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.791+0000 7f351ce00640 1 -- start start 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f351ce00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3518076980 0x7f351819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f351ce00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518077bc0 0x7f351819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f351ce00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518078560 0x7f35181a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f351ce00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f35181140d0 con 0x7f3518077bc0 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f351ce00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3518113f50 con 0x7f3518076980 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f351ce00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3518114250 con 0x7f3518078560 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f350ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518077bc0 0x7f351819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:17.789 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f350ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518077bc0 0x7f351819ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38580/0 (socket says 172.21.15.81:38580) 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f350ea00640 1 -- 172.21.15.81:0/302548725 learned_addr learned my addr 172.21.15.81:0/302548725 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f3517400640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3518076980 0x7f351819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f3517e00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518078560 0x7f35181a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f350ea00640 1 -- 172.21.15.81:0/302548725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518078560 msgr2=0x7f35181a5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f350ea00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518078560 0x7f35181a5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f350ea00640 1 -- 172.21.15.81:0/302548725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3518076980 msgr2=0x7f351819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f350ea00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3518076980 0x7f351819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.792+0000 7f350ea00640 1 -- 172.21.15.81:0/302548725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35181a62f0 con 0x7f3518077bc0 2024-04-12T21:15:17.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.793+0000 7f350ea00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518077bc0 0x7f351819ec10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f350000ea60 tx=0x7f350000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.793+0000 7f3514c00640 1 -- 172.21.15.81:0/302548725 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f350000cde0 con 0x7f3518077bc0 2024-04-12T21:15:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.793+0000 7f3514c00640 1 -- 172.21.15.81:0/302548725 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35000035f0 con 0x7f3518077bc0 2024-04-12T21:15:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.793+0000 7f3514c00640 1 -- 172.21.15.81:0/302548725 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3500010670 con 0x7f3518077bc0 2024-04-12T21:15:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.793+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35181a65e0 con 0x7f3518077bc0 2024-04-12T21:15:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.793+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35181a6ae0 con 0x7f3518077bc0 2024-04-12T21:15:17.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.794+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3518110b00 con 0x7f3518077bc0 2024-04-12T21:15:17.793 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.796+0000 7f3514c00640 1 -- 172.21.15.81:0/302548725 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f35000037a0 con 0x7f3518077bc0 2024-04-12T21:15:17.794 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.798+0000 7f3514c00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f34fc0a3bb0 0x7f34fc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.798+0000 7f3514c00640 1 -- 172.21.15.81:0/302548725 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f35000d0a50 con 0x7f3518077bc0 2024-04-12T21:15:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.798+0000 7f3517400640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f34fc0a3bb0 0x7f34fc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:17.795 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.799+0000 7f3517400640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f34fc0a3bb0 0x7f34fc0a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3508009bd0 tx=0x7f3508005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:17.798 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:17.802+0000 7f3514c00640 1 -- 172.21.15.81:0/302548725 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f350009d0d0 con 0x7f3518077bc0 2024-04-12T21:15:17.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:17 smithi153 ceph-mon[36569]: pgmap v378: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:17.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:17 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3364069964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:17.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:17 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/4102554521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:17 smithi104 ceph-mon[36624]: pgmap v378: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:17 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3364069964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:17 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/4102554521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:17 smithi081 ceph-mon[31453]: pgmap v378: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:17 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3364069964' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:17 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/4102554521' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:18.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.051+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f351802d5e0 con 0x7f3518077bc0 2024-04-12T21:15:18.049 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.053+0000 7f3514c00640 1 -- 172.21.15.81:0/302548725 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11815 (secure 0 0 0) 0x7f350009c9d0 con 0x7f3518077bc0 2024-04-12T21:15:18.050 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:18.050 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T21:13:13.003153+0000","modified":"2024-04-12T21:13:37.617228+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-12T21:15:18.050 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f34fc0a3bb0 msgr2=0x7f34fc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f34fc0a3bb0 0x7f34fc0a6070 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3508009bd0 tx=0x7f3508005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518077bc0 msgr2=0x7f351819ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518077bc0 0x7f351819ec10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f350000ea60 tx=0x7f350000ef20 comp rx=0 tx=0).stop 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 shutdown_connections 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f34fc0a3bb0 0x7f34fc0a6070 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3518078560 0x7f35181a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3518077bc0 0x7f351819ec10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 --2- 172.21.15.81:0/302548725 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3518076980 0x7f351819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.053 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 >> 172.21.15.81:0/302548725 conn(0x7f35180ff5d0 msgr2=0x7f3518100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:18.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 shutdown_connections 2024-04-12T21:15:18.054 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.056+0000 7f351ce00640 1 -- 172.21.15.81:0/302548725 wait complete. 2024-04-12T21:15:18.073 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:15:18.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 -- 172.21.15.81:0/2351989633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109ff0 msgr2=0x7f8efc110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 --2- 172.21.15.81:0/2351989633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109ff0 0x7f8efc110880 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4009900 tx=0x7f8ef402bc40 comp rx=0 tx=0).stop 2024-04-12T21:15:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 -- 172.21.15.81:0/2351989633 shutdown_connections 2024-04-12T21:15:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 --2- 172.21.15.81:0/2351989633 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109ff0 0x7f8efc110880 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 --2- 172.21.15.81:0/2351989633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8efc109630 0x7f8efc109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 --2- 172.21.15.81:0/2351989633 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8efc108440 0x7f8efc108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 -- 172.21.15.81:0/2351989633 >> 172.21.15.81:0/2351989633 conn(0x7f8efc077400 msgr2=0x7f8efc079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 -- 172.21.15.81:0/2351989633 shutdown_connections 2024-04-12T21:15:18.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.291+0000 7f8efb200640 1 -- 172.21.15.81:0/2351989633 wait complete. 2024-04-12T21:15:18.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.292+0000 7f8efb200640 1 Processor -- start 2024-04-12T21:15:18.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.292+0000 7f8efb200640 1 -- start start 2024-04-12T21:15:18.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.292+0000 7f8efb200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8efc108440 0x7f8efc19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.292+0000 7f8efb200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109630 0x7f8efc19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.292+0000 7f8efb200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8efc109ff0 0x7f8efc1a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.292+0000 7f8efb200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f8efc1140a0 con 0x7f8efc109630 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.292+0000 7f8efb200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f8efc113f20 con 0x7f8efc109ff0 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.292+0000 7f8efb200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8efc114220 con 0x7f8efc108440 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109630 0x7f8efc19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109630 0x7f8efc19ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38602/0 (socket says 172.21.15.81:38602) 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 -- 172.21.15.81:0/2428126295 learned_addr learned my addr 172.21.15.81:0/2428126295 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8efa800640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8efc109ff0 0x7f8efc1a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9e00640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8efc108440 0x7f8efc19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 -- 172.21.15.81:0/2428126295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8efc108440 msgr2=0x7f8efc19e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8efc108440 0x7f8efc19e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 -- 172.21.15.81:0/2428126295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8efc109ff0 msgr2=0x7f8efc1a5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8efc109ff0 0x7f8efc1a5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.290 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 -- 172.21.15.81:0/2428126295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8efc1a6330 con 0x7f8efc109630 2024-04-12T21:15:18.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.293+0000 7f8ef9400640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109630 0x7f8efc19ec50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800ea60 tx=0x7f8ee800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:18.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.294+0000 7f8ef2a00640 1 -- 172.21.15.81:0/2428126295 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee800cde0 con 0x7f8efc109630 2024-04-12T21:15:18.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.294+0000 7f8ef2a00640 1 -- 172.21.15.81:0/2428126295 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee80035f0 con 0x7f8efc109630 2024-04-12T21:15:18.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.294+0000 7f8ef2a00640 1 -- 172.21.15.81:0/2428126295 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8010670 con 0x7f8efc109630 2024-04-12T21:15:18.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.294+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8efc1a6620 con 0x7f8efc109630 2024-04-12T21:15:18.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.294+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8efc1a6b20 con 0x7f8efc109630 2024-04-12T21:15:18.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.295+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8efc10f760 con 0x7f8efc109630 2024-04-12T21:15:18.294 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.297+0000 7f8ef2a00640 1 -- 172.21.15.81:0/2428126295 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8ee80037a0 con 0x7f8efc109630 2024-04-12T21:15:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.299+0000 7f8ef2a00640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ee00a3c30 0x7f8ee00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.299+0000 7f8ef2a00640 1 -- 172.21.15.81:0/2428126295 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f8ee80d0a50 con 0x7f8efc109630 2024-04-12T21:15:18.295 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.299+0000 7f8ef9e00640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ee00a3c30 0x7f8ee00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:18.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.300+0000 7f8ef9e00640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ee00a3c30 0x7f8ee00a60f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8eec0099a0 tx=0x7f8eec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:18.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.303+0000 7f8ef2a00640 1 -- 172.21.15.81:0/2428126295 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ee809d070 con 0x7f8efc109630 2024-04-12T21:15:18.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.507+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8efc062c60 con 0x7f8efc109630 2024-04-12T21:15:18.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.512+0000 7f8ef2a00640 1 -- 172.21.15.81:0/2428126295 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24681 (secure 0 0 0) 0x7f8ee809c970 con 0x7f8efc109630 2024-04-12T21:15:18.509 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:18.509 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":91,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:14:23.492158+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:15:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ee00a3c30 msgr2=0x7f8ee00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ee00a3c30 0x7f8ee00a60f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8eec0099a0 tx=0x7f8eec009500 comp rx=0 tx=0).stop 2024-04-12T21:15:18.511 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109630 msgr2=0x7f8efc19ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109630 0x7f8efc19ec50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800ea60 tx=0x7f8ee800ef20 comp rx=0 tx=0).stop 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 shutdown_connections 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f8ee00a3c30 0x7f8ee00a60f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8efc109ff0 0x7f8efc1a5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8efc109630 0x7f8efc19ec50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 --2- 172.21.15.81:0/2428126295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8efc108440 0x7f8efc19e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 >> 172.21.15.81:0/2428126295 conn(0x7f8efc077400 msgr2=0x7f8efc10c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 shutdown_connections 2024-04-12T21:15:18.512 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.515+0000 7f8efb200640 1 -- 172.21.15.81:0/2428126295 wait complete. 2024-04-12T21:15:18.531 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:15:18.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.743+0000 7f01fd200640 1 -- 172.21.15.81:0/3181468706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109ff0 msgr2=0x7f01f8110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.740 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.743+0000 7f01fd200640 1 --2- 172.21.15.81:0/3181468706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109ff0 0x7f01f8110880 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009ca0 tx=0x7f01f002c100 comp rx=0 tx=0).stop 2024-04-12T21:15:18.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.744+0000 7f01fd200640 1 -- 172.21.15.81:0/3181468706 shutdown_connections 2024-04-12T21:15:18.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.744+0000 7f01fd200640 1 --2- 172.21.15.81:0/3181468706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01f8109630 0x7f01f8109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.744+0000 7f01fd200640 1 --2- 172.21.15.81:0/3181468706 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109ff0 0x7f01f8110880 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.744+0000 7f01fd200640 1 --2- 172.21.15.81:0/3181468706 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f8108440 0x7f01f8108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.744+0000 7f01fd200640 1 -- 172.21.15.81:0/3181468706 >> 172.21.15.81:0/3181468706 conn(0x7f01f8077400 msgr2=0x7f01f8079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:18.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.744+0000 7f01fd200640 1 -- 172.21.15.81:0/3181468706 shutdown_connections 2024-04-12T21:15:18.741 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.744+0000 7f01fd200640 1 -- 172.21.15.81:0/3181468706 wait complete. 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01fd200640 1 Processor -- start 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01fd200640 1 -- start start 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01fd200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f8108440 0x7f01f819e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01fd200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109630 0x7f01f819ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01fd200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01f8109ff0 0x7f01f81a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01fd200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f01f81140d0 con 0x7f01f8109630 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01fd200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8113f50 con 0x7f01f8108440 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01fd200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f01f8114250 con 0x7f01f8109ff0 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01eea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109630 0x7f01f819ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:18.743 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01eea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109630 0x7f01f819ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38630/0 (socket says 172.21.15.81:38630) 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01f7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f8108440 0x7f01f819e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01eea00640 1 -- 172.21.15.81:0/775903425 learned_addr learned my addr 172.21.15.81:0/775903425 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.745+0000 7f01f7e00640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01f8109ff0 0x7f01f81a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01eea00640 1 -- 172.21.15.81:0/775903425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01f8109ff0 msgr2=0x7f01f81a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01eea00640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01f8109ff0 0x7f01f81a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01eea00640 1 -- 172.21.15.81:0/775903425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f8108440 msgr2=0x7f01f819e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01eea00640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f8108440 0x7f01f819e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01eea00640 1 -- 172.21.15.81:0/775903425 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01f81a63e0 con 0x7f01f8109630 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01eea00640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109630 0x7f01f819ed00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f01e000e9f0 tx=0x7f01e000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01f4c00640 1 -- 172.21.15.81:0/775903425 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e000cd80 con 0x7f01f8109630 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01f4c00640 1 -- 172.21.15.81:0/775903425 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01e0005500 con 0x7f01f8109630 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01f81a66d0 con 0x7f01f8109630 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.746+0000 7f01f4c00640 1 -- 172.21.15.81:0/775903425 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01e0010690 con 0x7f01f8109630 2024-04-12T21:15:18.744 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.747+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01f81a6bd0 con 0x7f01f8109630 2024-04-12T21:15:18.745 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.748+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01f810f700 con 0x7f01f8109630 2024-04-12T21:15:18.746 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.750+0000 7f01f4c00640 1 -- 172.21.15.81:0/775903425 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f01e00046c0 con 0x7f01f8109630 2024-04-12T21:15:18.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.751+0000 7f01f4c00640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f01dc0a3bb0 0x7f01dc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:18.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.752+0000 7f01f4c00640 1 -- 172.21.15.81:0/775903425 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f01e00d0bc0 con 0x7f01f8109630 2024-04-12T21:15:18.748 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.752+0000 7f01f7400640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f01dc0a3bb0 0x7f01dc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:18.749 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.752+0000 7f01f7400640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f01dc0a3bb0 0x7f01dc0a6070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f01f8109490 tx=0x7f01e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:18.752 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.756+0000 7f01f4c00640 1 -- 172.21.15.81:0/775903425 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01e009d1e0 con 0x7f01f8109630 2024-04-12T21:15:18.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:18 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1863625755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:18.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:18 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/302548725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:18.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:18 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1863625755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:18.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:18 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/302548725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:18.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:18 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1863625755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:18.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:18 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/302548725' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:18.989 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.992+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01f802d5e0 con 0x7f01f8109630 2024-04-12T21:15:18.990 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.994+0000 7f01f4c00640 1 -- 172.21.15.81:0/775903425 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11815 (secure 0 0 0) 0x7f01e009cae0 con 0x7f01f8109630 2024-04-12T21:15:18.991 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:18.992 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T21:13:13.003153+0000","modified":"2024-04-12T21:13:37.617228+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-12T21:15:18.992 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-04-12T21:15:18.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f01dc0a3bb0 msgr2=0x7f01dc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f01dc0a3bb0 0x7f01dc0a6070 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f01f8109490 tx=0x7f01e8005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:18.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109630 msgr2=0x7f01f819ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109630 0x7f01f819ed00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f01e000e9f0 tx=0x7f01e000edf0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 shutdown_connections 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f01dc0a3bb0 0x7f01dc0a6070 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f01f8109ff0 0x7f01f81a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f01f8109630 0x7f01f819ed00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 --2- 172.21.15.81:0/775903425 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f01f8108440 0x7f01f819e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 >> 172.21.15.81:0/775903425 conn(0x7f01f8077400 msgr2=0x7f01f810c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 shutdown_connections 2024-04-12T21:15:18.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:18.997+0000 7f01fd200640 1 -- 172.21.15.81:0/775903425 wait complete. 2024-04-12T21:15:19.013 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-12T21:13:13.003153+0000', 'modified': '2024-04-12T21:13:37.617228+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6839/2426851956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 2426851956}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 2426851956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24593': {'gid': 24593, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.104:6839/1944172329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 1944172329}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 1944172329}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24593, 'qdb_cluster': [24593]} 2024-04-12T21:15:19.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-12T21:15:19.014 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 -- 172.21.15.81:0/2491590824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 msgr2=0x7ff734107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 --2- 172.21.15.81:0/2491590824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 0x7ff734107670 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff724009900 tx=0x7ff72402bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 -- 172.21.15.81:0/2491590824 shutdown_connections 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 --2- 172.21.15.81:0/2491590824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff734107bb0 0x7ff73410a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 --2- 172.21.15.81:0/2491590824 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 0x7ff734107670 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 --2- 172.21.15.81:0/2491590824 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734102950 0x7ff734104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 -- 172.21.15.81:0/2491590824 >> 172.21.15.81:0/2491590824 conn(0x7ff7340fc360 msgr2=0x7ff7340fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 -- 172.21.15.81:0/2491590824 shutdown_connections 2024-04-12T21:15:19.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.228+0000 7ff733e00640 1 -- 172.21.15.81:0/2491590824 wait complete. 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.229+0000 7ff733e00640 1 Processor -- start 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.229+0000 7ff733e00640 1 -- start start 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.229+0000 7ff733e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734102950 0x7ff7341022c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.229+0000 7ff733e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 0x7ff734100900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.229+0000 7ff733e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff734107bb0 0x7ff734100e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.229+0000 7ff733e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410db40 con 0x7ff734105280 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.229+0000 7ff733e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410d9c0 con 0x7ff734102950 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.229+0000 7ff733e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff73410dcc0 con 0x7ff734107bb0 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 0x7ff734100900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 0x7ff734100900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38642/0 (socket says 172.21.15.81:38642) 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 -- 172.21.15.81:0/1649769560 learned_addr learned my addr 172.21.15.81:0/1649769560 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff733400640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff734107bb0 0x7ff734100e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732a00640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734102950 0x7ff7341022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 -- 172.21.15.81:0/1649769560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff734107bb0 msgr2=0x7ff734100e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff734107bb0 0x7ff734100e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 -- 172.21.15.81:0/1649769560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734102950 msgr2=0x7ff7341022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734102950 0x7ff7341022c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 -- 172.21.15.81:0/1649769560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7341016d0 con 0x7ff734105280 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.230+0000 7ff732000640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 0x7ff734100900 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff724000c00 tx=0x7ff72402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.231+0000 7ff72b400640 1 -- 172.21.15.81:0/1649769560 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724041070 con 0x7ff734105280 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.231+0000 7ff72b400640 1 -- 172.21.15.81:0/1649769560 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff724004450 con 0x7ff734105280 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.231+0000 7ff72b400640 1 -- 172.21.15.81:0/1649769560 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff724034440 con 0x7ff734105280 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.231+0000 7ff733e00640 1 -- 172.21.15.81:0/1649769560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7341a2270 con 0x7ff734105280 2024-04-12T21:15:19.229 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.231+0000 7ff733e00640 1 -- 172.21.15.81:0/1649769560 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7341a2770 con 0x7ff734105280 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff73c7b3740 1 Processor -- start 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff72b400640 1 -- 172.21.15.81:0/1649769560 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff724044070 con 0x7ff734105280 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff73c7b3740 1 -- start start 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648377b0 0x555d64837bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648381a0 0x555d6483cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d6483f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555d6479d990 con 0x555d6483d310 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555d64744530 con 0x555d648381a0 2024-04-12T21:15:19.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.234+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555d64799950 con 0x555d648377b0 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d6483f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732000640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d6483f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38648/0 (socket says 172.21.15.81:38648) 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff733400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648377b0 0x555d64837bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648381a0 0x555d6483cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732000640 1 -- 172.21.15.81:0/3742883500 learned_addr learned my addr 172.21.15.81:0/3742883500 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732a00640 1 -- 172.21.15.81:0/3742883500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648377b0 msgr2=0x555d64837bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732a00640 1 --2- 172.21.15.81:0/3742883500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648377b0 0x555d64837bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732a00640 1 -- 172.21.15.81:0/3742883500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 msgr2=0x555d6483f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:19.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732a00640 1 --2- 172.21.15.81:0/3742883500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d6483f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff732a00640 1 -- 172.21.15.81:0/3742883500 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d644b02d0 con 0x555d648381a0 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.235+0000 7ff733400640 1 --2- 172.21.15.81:0/3742883500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648377b0 0x555d64837bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.236+0000 7ff732a00640 1 --2- 172.21.15.81:0/3742883500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648381a0 0x555d6483cca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009fa0 tx=0x7ff72c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=438840523e4fe6d server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.236+0000 7ff72b400640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7180a3c30 0x7ff7180a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.236+0000 7ff723e00640 1 -- 172.21.15.81:0/3742883500 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c003c30 con 0x555d648381a0 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.236+0000 7ff723e00640 1 -- 172.21.15.81:0/3742883500 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff72c003dd0 con 0x555d648381a0 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.236+0000 7ff723e00640 1 -- 172.21.15.81:0/3742883500 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c00fa50 con 0x555d648381a0 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.236+0000 7ff72b400640 1 -- 172.21.15.81:0/1649769560 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff7240f2140 con 0x7ff734105280 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.236+0000 7ff733400640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7180a3c30 0x7ff7180a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3742883500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648381a0 msgr2=0x555d6483cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3742883500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648381a0 0x555d6483cca0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009fa0 tx=0x7ff72c0118d0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff733400640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7180a3c30 0x7ff7180a60f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x555d64838a00 tx=0x7ff71c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3742883500 shutdown_connections 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3742883500 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d6483f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3742883500 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648381a0 0x555d6483cca0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3742883500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648377b0 0x555d64837bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3742883500 >> 172.21.15.81:0/3742883500 conn(0x555d64832d20 msgr2=0x555d64833120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3742883500 shutdown_connections 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3742883500 wait complete. 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.237+0000 7ff73c7b3740 1 -- start start 2024-04-12T21:15:19.236 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff73c7b3740 1 Processor -- start 2024-04-12T21:15:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d64841e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d64842360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64849330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555d648402c0 con 0x555d6483d310 2024-04-12T21:15:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555d647578f0 con 0x555d648377b0 2024-04-12T21:15:19.240 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555d64840440 con 0x555d648381a0 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff73c7b3740 10 client.0 _ll_register_callbacks cb 0x555d647572e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff733400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64849330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff732a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d64841e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff733400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64849330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38664/0 (socket says 172.21.15.81:38664) 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff733400640 1 -- 172.21.15.81:0/3548263963 learned_addr learned my addr 172.21.15.81:0/3548263963 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.238+0000 7ff732000640 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d64842360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff733400640 1 -- 172.21.15.81:0/3548263963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 msgr2=0x555d64842360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff733400640 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d64842360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff733400640 1 -- 172.21.15.81:0/3548263963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 msgr2=0x555d64841e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff733400640 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d64841e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff733400640 1 -- 172.21.15.81:0/3548263963 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d6457a6b0 con 0x555d6483d310 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff732a00640 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d64841e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff733400640 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64849330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff71c005880 tx=0x7ff71c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9a1fc940d21149 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:19.241 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff720c00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff720c00640 1 -- 172.21.15.81:0/3548263963 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c01ee40 con 0x555d6483d310 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff720c00640 1 -- 172.21.15.81:0/3548263963 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff71c018030 con 0x555d6483d310 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.239+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3548263963 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff72c003dd0 con 0x555d6483d310 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff720c00640 1 -- 172.21.15.81:0/3548263963 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff71c034050 con 0x555d6483d310 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff73c7b3740 10 client.15033 fetch_fsmap learned FSMap version 20 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3548263963 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555d6484c710 con 0x555d6483d310 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff720c00640 1 -- 172.21.15.81:0/3548263963 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff72c003dd0 con 0x555d6483d310 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff720c00640 1 -- 172.21.15.81:0/3548263963 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 20) ==== 8330+0+0 (secure 0 0 0) 0x7ff71c029420 con 0x555d6483d310 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff73c7b3740 10 client.15033 fetch_fsmap finished waiting for FSMap version 20 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff73c7b3740 10 client.15033 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff73c7b3740 20 client.15033 populate_metadata read hostname 'smithi081' 2024-04-12T21:15:19.242 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x555d6484ede0 0x555d64850250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff73c7b3740 4 client.15033 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.240+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3548263963 --> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d64743b40 con 0x555d6484ede0 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.241+0000 7ff732a00640 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x555d6484ede0 0x555d64850250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.241+0000 7ff732a00640 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x555d6484ede0 0x555d64850250 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a18035144a3b46a1 server_cookie=38fd21bc816ff3be in_seq=0 out_seq=0 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.241+0000 7ff720c00640 10 client.15033 ms_handle_connect on v2:172.21.15.104:6838/1944172329 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.242+0000 7ff720c00640 1 -- 172.21.15.81:0/3548263963 <== mds.0 v2:172.21.15.104:6838/1944172329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d64743b40 con 0x555d6484ede0 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.242+0000 7ff720c00640 10 client.15033 handle_command_reply: tid=0 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 client.15033 shutdown 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3548263963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 msgr2=0x555d64849330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64849330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff71c005880 tx=0x7ff71c004730 comp rx=0 tx=0).stop 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3548263963 shutdown_connections 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x555d6484ede0 0x555d64850250 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d64842360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64849330 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff720c00640 0 client.15033 ms_handle_reset on v2:172.21.15.104:6838/1944172329 2024-04-12T21:15:19.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3548263963 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d64841e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3548263963 >> 172.21.15.81:0/3548263963 conn(0x555d64832d20 msgr2=0x555d64831f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:19.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3548263963 shutdown_connections 2024-04-12T21:15:19.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3548263963 wait complete. 2024-04-12T21:15:19.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.243+0000 7ff73c7b3740 20 client.15033 trim_cache size 0 max 16384 2024-04-12T21:15:19.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.262+0000 7ff73c7b3740 1 Processor -- start 2024-04-12T21:15:19.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff73c7b3740 1 -- start start 2024-04-12T21:15:19.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d6484dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d648492f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64839880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555d6484e590 con 0x555d6483d310 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555d6484ec60 con 0x555d648377b0 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555d6484f330 con 0x555d648381a0 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff733400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64839880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff733400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64839880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38674/0 (socket says 172.21.15.81:38674) 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff733400640 1 -- 172.21.15.81:0/4252150774 learned_addr learned my addr 172.21.15.81:0/4252150774 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff732a00640 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d6484dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.263+0000 7ff732000640 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d648492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff732a00640 1 -- 172.21.15.81:0/4252150774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 msgr2=0x555d648492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff732a00640 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d648492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff732a00640 1 -- 172.21.15.81:0/4252150774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 msgr2=0x555d64839880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:19.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff732a00640 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64839880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff732a00640 1 -- 172.21.15.81:0/4252150774 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d6482e790 con 0x555d648377b0 2024-04-12T21:15:19.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff733400640 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64839880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:19.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff732a00640 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d6484dbd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x555d64833c90 tx=0x7ff72c009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b4c9c69982fd24c server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:19.266 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff723400640 1 -- 172.21.15.81:0/4252150774 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c031070 con 0x555d648377b0 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff723400640 1 -- 172.21.15.81:0/4252150774 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff72c02c8d0 con 0x555d648377b0 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.264+0000 7ff723400640 1 -- 172.21.15.81:0/4252150774 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c004280 con 0x555d648377b0 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 -- 172.21.15.81:0/4252150774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 msgr2=0x555d6484dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d6484dbd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x555d64833c90 tx=0x7ff72c009500 comp rx=0 tx=0).stop 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 -- 172.21.15.81:0/4252150774 shutdown_connections 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d648492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d6483d310 0x555d64839880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/4252150774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d648377b0 0x555d6484dbd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 -- 172.21.15.81:0/4252150774 >> 172.21.15.81:0/4252150774 conn(0x555d64832d20 msgr2=0x555d6482dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 -- 172.21.15.81:0/4252150774 shutdown_connections 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 -- 172.21.15.81:0/4252150774 wait complete. 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.265+0000 7ff73c7b3740 1 -- start start 2024-04-12T21:15:19.267 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff73c7b3740 1 Processor -- start 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d64847530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d6483d310 0x555d64847a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff73c7b3740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d64847fb0 0x555d64848430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x555d648500d0 con 0x555d64847fb0 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x555d6484fa00 con 0x555d6483d310 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff73c7b3740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x555d648507a0 con 0x555d648381a0 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff73c7b3740 10 client.0 _ll_register_callbacks cb 0x7ff71c01e800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff733400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d64847fb0 0x555d64848430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff732000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d6483d310 0x555d64847a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff733400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d64847fb0 0x555d64848430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38686/0 (socket says 172.21.15.81:38686) 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff733400640 1 -- 172.21.15.81:0/3287347051 learned_addr learned my addr 172.21.15.81:0/3287347051 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.266+0000 7ff732a00640 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d64847530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff733400640 1 -- 172.21.15.81:0/3287347051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 msgr2=0x555d64847530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:19.268 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff733400640 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d64847530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff733400640 1 -- 172.21.15.81:0/3287347051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d6483d310 msgr2=0x555d64847a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff733400640 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d6483d310 0x555d64847a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff733400640 1 -- 172.21.15.81:0/3287347051 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d64848b40 con 0x555d64847fb0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff732000640 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d6483d310 0x555d64847a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff733400640 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d64847fb0 0x555d64848430 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff71c005880 tx=0x7ff71c009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cae56b700c514f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff720c00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff720c00640 1 -- 172.21.15.81:0/3287347051 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c004040 con 0x555d64847fb0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff720c00640 1 -- 172.21.15.81:0/3287347051 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff71c0041e0 con 0x555d64847fb0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3287347051 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff72c02c8d0 con 0x555d64847fb0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff720c00640 1 -- 172.21.15.81:0/3287347051 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7ff71c046050 con 0x555d64847fb0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff73c7b3740 10 client.15039 fetch_fsmap learned FSMap version 20 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.267+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3287347051 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x555d648b5420 con 0x555d64847fb0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff720c00640 1 -- 172.21.15.81:0/3287347051 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff72c02c8d0 con 0x555d64847fb0 2024-04-12T21:15:19.269 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff720c00640 1 -- 172.21.15.81:0/3287347051 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 20) ==== 8330+0+0 (secure 0 0 0) 0x7ff71c038430 con 0x555d64847fb0 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff73c7b3740 10 client.15039 fetch_fsmap finished waiting for FSMap version 20 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff73c7b3740 10 client.15039 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff73c7b3740 20 client.15039 populate_metadata read hostname 'smithi081' 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x555d648ba040 0x555d648b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff73c7b3740 4 client.15039 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3287347051 --> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] -- command(tid 0: {"prefix": "status"}) -- 0x555d647ab3b0 con 0x555d648ba040 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.268+0000 7ff732a00640 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x555d648ba040 0x555d648b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff732a00640 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x555d648ba040 0x555d648b8d50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c4fc2715f8f9e2f server_cookie=7ac30bc74b6a4456 in_seq=0 out_seq=0 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff720c00640 10 client.15039 ms_handle_connect on v2:172.21.15.104:6838/1944172329 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff720c00640 1 -- 172.21.15.81:0/3287347051 <== mds.0 v2:172.21.15.104:6838/1944172329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x555d647ab3b0 con 0x555d648ba040 2024-04-12T21:15:19.270 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff720c00640 10 client.15039 handle_command_reply: tid=0 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff73c7b3740 1 client.15039 shutdown 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3287347051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d64847fb0 msgr2=0x555d64848430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d64847fb0 0x555d64848430 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff71c005880 tx=0x7ff71c009930 comp rx=0 tx=0).stop 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3287347051 shutdown_connections 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.269+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x555d648ba040 0x555d648b8d50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x555d64847fb0 0x555d64848430 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x555d6483d310 0x555d64847a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff73c7b3740 1 --2- 172.21.15.81:0/3287347051 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x555d648381a0 0x555d64847530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff720c00640 0 client.15039 ms_handle_reset on v2:172.21.15.104:6838/1944172329 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3287347051 >> 172.21.15.81:0/3287347051 conn(0x555d64832d20 msgr2=0x555d6484dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3287347051 shutdown_connections 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff73c7b3740 1 -- 172.21.15.81:0/3287347051 wait complete. 2024-04-12T21:15:19.271 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff73c7b3740 20 client.15039 trim_cache size 0 max 16384 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "11b8cc88-f910-11ee-bc8d-c7b262605968", 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24593, 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 125.742040168, 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 20, 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 91, 2024-04-12T21:15:19.272 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 89, 2024-04-12T21:15:19.273 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 132.94759687000001 2024-04-12T21:15:19.273 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-12T21:15:19.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff733e00640 1 -- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7180a3c30 msgr2=0x7ff7180a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff733e00640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7180a3c30 0x7ff7180a60f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x555d64838a00 tx=0x7ff71c005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:19.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff733e00640 1 -- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 msgr2=0x7ff734100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:19.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff733e00640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 0x7ff734100900 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff724000c00 tx=0x7ff72402c640 comp rx=0 tx=0).stop 2024-04-12T21:15:19.273 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff733e00640 1 -- 172.21.15.81:0/1649769560 shutdown_connections 2024-04-12T21:15:19.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff733e00640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7180a3c30 0x7ff7180a60f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff733e00640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff734107bb0 0x7ff734100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.270+0000 7ff733e00640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff734105280 0x7ff734100900 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.271+0000 7ff733e00640 1 --2- 172.21.15.81:0/1649769560 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff734102950 0x7ff7341022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:19.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.271+0000 7ff733e00640 1 -- 172.21.15.81:0/1649769560 >> 172.21.15.81:0/1649769560 conn(0x7ff7340fc360 msgr2=0x7ff734103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:19.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.271+0000 7ff733e00640 1 -- 172.21.15.81:0/1649769560 shutdown_connections 2024-04-12T21:15:19.274 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:19.271+0000 7ff733e00640 1 -- 172.21.15.81:0/1649769560 wait complete. 2024-04-12T21:15:19.284 INFO:tasks.cephadm:Setup complete, yielding 2024-04-12T21:15:19.284 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T21:15:19.293 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-12T21:15:19.293 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch status' 2024-04-12T21:15:19.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:19 smithi081 ceph-mon[31453]: pgmap v379: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:19.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:19 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2428126295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:19.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:19 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/775903425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:19.693 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:19 smithi153 ceph-mon[36569]: pgmap v379: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:19 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2428126295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:19 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/775903425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:19 smithi104 ceph-mon[36624]: pgmap v379: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:19 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2428126295' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:19.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:19 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/775903425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:21.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:21 smithi081 ceph-mon[31453]: pgmap v380: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:21 smithi153 ceph-mon[36569]: pgmap v380: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:21.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:21 smithi104 ceph-mon[36624]: pgmap v380: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:21.874 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.878+0000 7f98bfe00700 1 -- 172.21.15.81:0/4166605358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8076a60 msgr2=0x7f98b8074ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.878+0000 7f98bfe00700 1 --2- 172.21.15.81:0/4166605358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8076a60 0x7f98b8074ec0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009990 tx=0x7f98ac009ca0 comp rx=0 tx=0).stop 2024-04-12T21:15:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.879+0000 7f98bfe00700 1 -- 172.21.15.81:0/4166605358 shutdown_connections 2024-04-12T21:15:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.879+0000 7f98bfe00700 1 --2- 172.21.15.81:0/4166605358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b810aa10 0x7f98b810ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.879+0000 7f98bfe00700 1 --2- 172.21.15.81:0/4166605358 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8076a60 0x7f98b8074ec0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.879+0000 7f98bfe00700 1 --2- 172.21.15.81:0/4166605358 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98b8075490 0x7f98b8075910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.879+0000 7f98bfe00700 1 -- 172.21.15.81:0/4166605358 >> 172.21.15.81:0/4166605358 conn(0x7f98b80fe0b0 msgr2=0x7f98b81004f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.879+0000 7f98bfe00700 1 -- 172.21.15.81:0/4166605358 shutdown_connections 2024-04-12T21:15:21.875 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.879+0000 7f98bfe00700 1 -- 172.21.15.81:0/4166605358 wait complete. 2024-04-12T21:15:21.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.880+0000 7f98bfe00700 1 Processor -- start 2024-04-12T21:15:21.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.880+0000 7f98bfe00700 1 -- start start 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.880+0000 7f98bfe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8075490 0x7f98b819af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.880+0000 7f98bfe00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98b8076a60 0x7f98b819b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.880+0000 7f98bfe00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b810aa10 0x7f98b81a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bfe00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8110920 con 0x7f98b8076a60 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bfe00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f98b81107a0 con 0x7f98b8075490 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bfe00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f98b8110aa0 con 0x7f98b810aa10 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bca00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98b8076a60 0x7f98b819b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bca00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98b8076a60 0x7f98b819b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:38700/0 (socket says 172.21.15.81:38700) 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bca00700 1 -- 172.21.15.81:0/2660800249 learned_addr learned my addr 172.21.15.81:0/2660800249 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bd400700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8075490 0x7f98b819af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:21.877 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bde00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b810aa10 0x7f98b81a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:21.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bd400700 1 -- 172.21.15.81:0/2660800249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b810aa10 msgr2=0x7f98b81a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:21.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bd400700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b810aa10 0x7f98b81a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:21.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bd400700 1 -- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98b8076a60 msgr2=0x7f98b819b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:21.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bd400700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98b8076a60 0x7f98b819b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:21.878 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.881+0000 7f98bd400700 1 -- 172.21.15.81:0/2660800249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b81a2ba0 con 0x7f98b8075490 2024-04-12T21:15:21.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.882+0000 7f98bd400700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8075490 0x7f98b819af40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b8e0 tx=0x7f98ac00bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:21.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.882+0000 7f98b6000700 1 -- 172.21.15.81:0/2660800249 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac026070 con 0x7f98b8075490 2024-04-12T21:15:21.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.882+0000 7f98b6000700 1 -- 172.21.15.81:0/2660800249 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98ac00f460 con 0x7f98b8075490 2024-04-12T21:15:21.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.882+0000 7f98b6000700 1 -- 172.21.15.81:0/2660800249 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac021670 con 0x7f98b8075490 2024-04-12T21:15:21.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.882+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b81a2e30 con 0x7f98b8075490 2024-04-12T21:15:21.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.882+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b81a31d0 con 0x7f98b8075490 2024-04-12T21:15:21.879 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.884+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b802dfa0 con 0x7f98b8075490 2024-04-12T21:15:21.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.886+0000 7f98b6000700 1 -- 172.21.15.81:0/2660800249 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f98ac029070 con 0x7f98b8075490 2024-04-12T21:15:21.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.887+0000 7f98b6000700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98a00a3c40 0x7f98a00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:21.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.887+0000 7f98b6000700 1 -- 172.21.15.81:0/2660800249 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f98ac0d1ab0 con 0x7f98b8075490 2024-04-12T21:15:21.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.887+0000 7f98bca00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98a00a3c40 0x7f98a00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:21.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.888+0000 7f98bca00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98a00a3c40 0x7f98a00a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009d20 tx=0x7f98a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:21.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:21.889+0000 7f98b6000700 1 -- 172.21.15.81:0/2660800249 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98ac030080 con 0x7f98b8075490 2024-04-12T21:15:22.131 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.135+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f98b8062ef0 con 0x7f98a00a3c40 2024-04-12T21:15:22.134 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.138+0000 7f98b6000700 1 -- 172.21.15.81:0/2660800249 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f98b8062ef0 con 0x7f98a00a3c40 2024-04-12T21:15:22.134 INFO:teuthology.orchestra.run.smithi081.stdout:Backend: cephadm 2024-04-12T21:15:22.134 INFO:teuthology.orchestra.run.smithi081.stdout:Available: Yes 2024-04-12T21:15:22.135 INFO:teuthology.orchestra.run.smithi081.stdout:Paused: No 2024-04-12T21:15:22.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.141+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98a00a3c40 msgr2=0x7f98a00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:22.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.141+0000 7f98bfe00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98a00a3c40 0x7f98a00a6100 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f98a4009d20 tx=0x7f98a4009260 comp rx=0 tx=0).stop 2024-04-12T21:15:22.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.141+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8075490 msgr2=0x7f98b819af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:22.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.141+0000 7f98bfe00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8075490 0x7f98b819af40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b8e0 tx=0x7f98ac00bbb0 comp rx=0 tx=0).stop 2024-04-12T21:15:22.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.142+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 shutdown_connections 2024-04-12T21:15:22.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.142+0000 7f98bfe00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f98a00a3c40 0x7f98a00a6100 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:22.137 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.142+0000 7f98bfe00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f98b810aa10 0x7f98b81a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:22.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.142+0000 7f98bfe00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98b8075490 0x7f98b819af40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:22.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.142+0000 7f98bfe00700 1 --2- 172.21.15.81:0/2660800249 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f98b8076a60 0x7f98b819b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:22.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.142+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 >> 172.21.15.81:0/2660800249 conn(0x7f98b80fe0b0 msgr2=0x7f98b8106960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:22.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.142+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 shutdown_connections 2024-04-12T21:15:22.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:22.142+0000 7f98bfe00700 1 -- 172.21.15.81:0/2660800249 wait complete. 2024-04-12T21:15:22.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-12T21:15:23.427 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:23.724 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:23 smithi081 ceph-mon[31453]: from='client.25175 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:23.724 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:23 smithi081 ceph-mon[31453]: pgmap v381: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:23.724 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-12T21:15:23.724 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-12T21:15:23.724 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-04-12T21:15:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:23 smithi153 ceph-mon[36569]: from='client.25175 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:23 smithi153 ceph-mon[36569]: pgmap v381: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-12T21:15:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-12T21:15:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-04-12T21:15:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:23 smithi104 ceph-mon[36624]: from='client.25175 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:23 smithi104 ceph-mon[36624]: pgmap v381: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-04-12T21:15:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-04-12T21:15:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-04-12T21:15:24.793 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-12T21:15:24.793 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-12T21:15:24.793 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-04-12T21:15:24.793 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:24 smithi081 ceph-mon[31453]: osdmap e92: 12 total, 12 up, 12 in 2024-04-12T21:15:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-12T21:15:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-12T21:15:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-04-12T21:15:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:24 smithi153 ceph-mon[36569]: osdmap e92: 12 total, 12 up, 12 in 2024-04-12T21:15:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-04-12T21:15:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-04-12T21:15:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-04-12T21:15:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:24 smithi104 ceph-mon[36624]: osdmap e92: 12 total, 12 up, 12 in 2024-04-12T21:15:25.586 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.589+0000 7f22a1200700 1 -- 172.21.15.81:0/1335183365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 msgr2=0x7f229c108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.589+0000 7f22a1200700 1 --2- 172.21.15.81:0/1335183365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 0x7f229c108320 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2288009990 tx=0x7f2288009ca0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.590+0000 7f22a1200700 1 -- 172.21.15.81:0/1335183365 shutdown_connections 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.590+0000 7f22a1200700 1 --2- 172.21.15.81:0/1335183365 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f229c108860 0x7f229c10ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.590+0000 7f22a1200700 1 --2- 172.21.15.81:0/1335183365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f229c103600 0x7f229c1059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.590+0000 7f22a1200700 1 --2- 172.21.15.81:0/1335183365 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 0x7f229c108320 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.590+0000 7f22a1200700 1 -- 172.21.15.81:0/1335183365 >> 172.21.15.81:0/1335183365 conn(0x7f229c0fcfb0 msgr2=0x7f229c0ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.590+0000 7f22a1200700 1 -- 172.21.15.81:0/1335183365 shutdown_connections 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.590+0000 7f22a1200700 1 -- 172.21.15.81:0/1335183365 wait complete. 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.591+0000 7f22a1200700 1 Processor -- start 2024-04-12T21:15:25.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.591+0000 7f22a1200700 1 -- start start 2024-04-12T21:15:25.588 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.591+0000 7f22a1200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f229c103600 0x7f229c19af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.591+0000 7f22a1200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 0x7f229c19b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.591+0000 7f22a1200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f229c108860 0x7f229c1a2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.591+0000 7f22a1200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f229c10e710 con 0x7f229c105f30 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.591+0000 7f22a1200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f229c10e590 con 0x7f229c108860 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.591+0000 7f22a1200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f229c10e890 con 0x7f229c103600 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 0x7f229c19b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 0x7f229c19b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59174/0 (socket says 172.21.15.81:59174) 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299400700 1 -- 172.21.15.81:0/594781589 learned_addr learned my addr 172.21.15.81:0/594781589 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299e00700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f229c103600 0x7f229c19af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f229a800700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f229c108860 0x7f229c1a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299e00700 1 -- 172.21.15.81:0/594781589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f229c108860 msgr2=0x7f229c1a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299e00700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f229c108860 0x7f229c1a2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299e00700 1 -- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 msgr2=0x7f229c19b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:25.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299e00700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 0x7f229c19b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299e00700 1 -- 172.21.15.81:0/594781589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f229c1a2bd0 con 0x7f229c103600 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.592+0000 7f2299400700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 0x7f229c19b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.593+0000 7f2299e00700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f229c103600 0x7f229c19af70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f228c00ea20 tx=0x7f228c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.593+0000 7f2292a00700 1 -- 172.21.15.81:0/594781589 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c00cbd0 con 0x7f229c103600 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.593+0000 7f2292a00700 1 -- 172.21.15.81:0/594781589 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f228c00ee60 con 0x7f229c103600 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.593+0000 7f2292a00700 1 -- 172.21.15.81:0/594781589 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f228c018680 con 0x7f229c103600 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.593+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f229c1a2ec0 con 0x7f229c103600 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.593+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f229c1a3260 con 0x7f229c103600 2024-04-12T21:15:25.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.594+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f229c02dfa0 con 0x7f229c103600 2024-04-12T21:15:25.592 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.597+0000 7f2292a00700 1 -- 172.21.15.81:0/594781589 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f228c018820 con 0x7f229c103600 2024-04-12T21:15:25.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.599+0000 7f2292a00700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f22800a3cc0 0x7f22800a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:25.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.599+0000 7f2292a00700 1 -- 172.21.15.81:0/594781589 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f228c0d0cf0 con 0x7f229c103600 2024-04-12T21:15:25.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.599+0000 7f2299400700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f22800a3cc0 0x7f22800a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:25.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.600+0000 7f2299400700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f22800a3cc0 0x7f22800a6180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f228800b7b0 tx=0x7f228800b8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:25.596 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.601+0000 7f2292a00700 1 -- 172.21.15.81:0/594781589 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f228c09d2c0 con 0x7f229c103600 2024-04-12T21:15:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:25 smithi153 ceph-mon[36569]: pgmap v383: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:15:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:25 smithi153 ceph-mon[36569]: osdmap e93: 12 total, 12 up, 12 in 2024-04-12T21:15:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:25 smithi104 ceph-mon[36624]: pgmap v383: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:15:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:25 smithi104 ceph-mon[36624]: osdmap e93: 12 total, 12 up, 12 in 2024-04-12T21:15:25.836 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.840+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f229c062ef0 con 0x7f22800a3cc0 2024-04-12T21:15:25.836 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:25 smithi081 ceph-mon[31453]: pgmap v383: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:15:25.836 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:25 smithi081 ceph-mon[31453]: osdmap e93: 12 total, 12 up, 12 in 2024-04-12T21:15:25.876 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.880+0000 7f2292a00700 1 -- 172.21.15.81:0/594781589 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f229c062ef0 con 0x7f22800a3cc0 2024-04-12T21:15:25.877 INFO:teuthology.orchestra.run.smithi081.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-12T21:15:25.877 INFO:teuthology.orchestra.run.smithi081.stdout:mds.a smithi081 running (2m) 28s ago 2m 14.4M - 19.0.0-2879-g324baf8d 8717720dc5e5 dfcc5c53b4f2 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.b smithi104 running (2m) 71s ago 2m 25.1M - 19.0.0-2879-g324baf8d 8717720dc5e5 ddefba9658f6 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.c smithi153 running (2m) 71s ago 2m 14.0M - 19.0.0-2879-g324baf8d 8717720dc5e5 7c24e15e697f 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.d smithi081 running (2m) 28s ago 2m 12.6M - 19.0.0-2879-g324baf8d 8717720dc5e5 fa4bc41faa46 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.e smithi104 running (2m) 71s ago 2m 16.5M - 19.0.0-2879-g324baf8d 8717720dc5e5 32cd5d8c8a27 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.f smithi153 running (2m) 71s ago 2m 12.8M - 19.0.0-2879-g324baf8d 8717720dc5e5 bfc8f4961e58 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.g smithi081 running (2m) 28s ago 2m 12.4M - 19.0.0-2879-g324baf8d 8717720dc5e5 2a629bf61cef 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.h smithi104 running (2m) 71s ago 2m 15.2M - 19.0.0-2879-g324baf8d 8717720dc5e5 a442b0fc5a89 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.i smithi153 running (2m) 71s ago 2m 19.4M - 19.0.0-2879-g324baf8d 8717720dc5e5 06cc72315b98 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.j smithi081 running (2m) 28s ago 2m 12.4M - 19.0.0-2879-g324baf8d 8717720dc5e5 f82f7deb6ea9 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.k smithi104 running (2m) 71s ago 2m 14.0M - 19.0.0-2879-g324baf8d 8717720dc5e5 2e08d3c1310f 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mds.l smithi153 running (2m) 71s ago 2m 16.4M - 19.0.0-2879-g324baf8d 8717720dc5e5 2811ba920302 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.x smithi081 *:9283,8765 running (11m) 28s ago 11m 515M - 19.0.0-2879-g324baf8d 8717720dc5e5 eb2441a68443 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.y smithi104 *:8443,8765 running (8m) 71s ago 8m 454M - 19.0.0-2879-g324baf8d 8717720dc5e5 a2701ccbf323 2024-04-12T21:15:25.878 INFO:teuthology.orchestra.run.smithi081.stdout:mgr.z smithi153 *:8443,8765 running (8m) 71s ago 8m 454M - 19.0.0-2879-g324baf8d 8717720dc5e5 2c00aff4f604 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:mon.a smithi081 running (11m) 28s ago 11m 48.5M 2048M 19.0.0-2879-g324baf8d 8717720dc5e5 2d2703ba7ffb 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:mon.b smithi104 running (9m) 71s ago 9m 46.5M 2048M 19.0.0-2879-g324baf8d 8717720dc5e5 c71e9ccccdbc 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:mon.c smithi153 running (9m) 71s ago 9m 45.6M 2048M 19.0.0-2879-g324baf8d 8717720dc5e5 48ca4dc272f2 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 smithi081 running (8m) 28s ago 8m 52.2M 4272M 19.0.0-2879-g324baf8d 8717720dc5e5 f43af9020c48 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 smithi104 running (7m) 71s ago 7m 53.7M 4270M 19.0.0-2879-g324baf8d 8717720dc5e5 1b8ef1f10f23 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 smithi153 running (7m) 71s ago 7m 56.8M 4270M 19.0.0-2879-g324baf8d 8717720dc5e5 be1b33bce499 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 smithi081 running (6m) 28s ago 6m 43.4M 4272M 19.0.0-2879-g324baf8d 8717720dc5e5 83d6b3b5ce52 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 smithi104 running (6m) 71s ago 6m 51.2M 4270M 19.0.0-2879-g324baf8d 8717720dc5e5 13605918208d 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 smithi153 running (6m) 71s ago 6m 55.9M 4270M 19.0.0-2879-g324baf8d 8717720dc5e5 181685722b48 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 smithi081 running (5m) 28s ago 5m 41.8M 4272M 19.0.0-2879-g324baf8d 8717720dc5e5 f45f2df08c07 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 smithi104 running (5m) 71s ago 5m 46.9M 4270M 19.0.0-2879-g324baf8d 8717720dc5e5 8c802a59a3d8 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.8 smithi153 running (4m) 71s ago 4m 53.5M 4270M 19.0.0-2879-g324baf8d 8717720dc5e5 a8a09f0ad8e8 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.9 smithi081 running (4m) 28s ago 4m 49.1M 4272M 19.0.0-2879-g324baf8d 8717720dc5e5 a026172c350c 2024-04-12T21:15:25.879 INFO:teuthology.orchestra.run.smithi081.stdout:osd.10 smithi104 running (3m) 71s ago 3m 41.5M 4270M 19.0.0-2879-g324baf8d 8717720dc5e5 27e6a26d2263 2024-04-12T21:15:25.880 INFO:teuthology.orchestra.run.smithi081.stdout:osd.11 smithi153 running (3m) 71s ago 3m 43.6M 4270M 19.0.0-2879-g324baf8d 8717720dc5e5 7f5effe4e538 2024-04-12T21:15:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.883+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f22800a3cc0 msgr2=0x7f22800a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.883+0000 7f22a1200700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f22800a3cc0 0x7f22800a6180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f228800b7b0 tx=0x7f228800b8a0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.880 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.883+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f229c103600 msgr2=0x7f229c19af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.883+0000 7f22a1200700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f229c103600 0x7f229c19af70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f228c00ea20 tx=0x7f228c00ed30 comp rx=0 tx=0).stop 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.884+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 shutdown_connections 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.884+0000 7f22a1200700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f22800a3cc0 0x7f22800a6180 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.884+0000 7f22a1200700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f229c108860 0x7f229c1a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.884+0000 7f22a1200700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f229c103600 0x7f229c19af70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.884+0000 7f22a1200700 1 --2- 172.21.15.81:0/594781589 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f229c105f30 0x7f229c19b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.884+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 >> 172.21.15.81:0/594781589 conn(0x7f229c0fcfb0 msgr2=0x7f229c102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.884+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 shutdown_connections 2024-04-12T21:15:25.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:25.884+0000 7f22a1200700 1 -- 172.21.15.81:0/594781589 wait complete. 2024-04-12T21:15:26.685 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-12T21:15:26.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:26 smithi104 ceph-mon[36624]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:26.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:26 smithi104 ceph-mon[36624]: pgmap v385: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:26 smithi081 ceph-mon[31453]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:26.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:26 smithi081 ceph-mon[31453]: pgmap v385: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:27.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:26 smithi153 ceph-mon[36569]: from='client.25171 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:27.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:26 smithi153 ceph-mon[36569]: pgmap v385: 129 pgs: 129 active+clean; 639 KiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:27.120 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 -- 172.21.15.81:0/1130034540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 msgr2=0x7f60d8105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 --2- 172.21.15.81:0/1130034540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 0x7f60d8105990 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009990 tx=0x7f60cc009ca0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 -- 172.21.15.81:0/1130034540 shutdown_connections 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 --2- 172.21.15.81:0/1130034540 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60d8108800 0x7f60d810ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 --2- 172.21.15.81:0/1130034540 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 0x7f60d8105990 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 --2- 172.21.15.81:0/1130034540 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d8105ed0 0x7f60d81082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 -- 172.21.15.81:0/1130034540 >> 172.21.15.81:0/1130034540 conn(0x7f60d80fcfb0 msgr2=0x7f60d80ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 -- 172.21.15.81:0/1130034540 shutdown_connections 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.449+0000 7f60e0200700 1 -- 172.21.15.81:0/1130034540 wait complete. 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.450+0000 7f60e0200700 1 Processor -- start 2024-04-12T21:15:29.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.450+0000 7f60e0200700 1 -- start start 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60e0200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 0x7f60d819b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60e0200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60d8105ed0 0x7f60d819b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60e0200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d8108800 0x7f60d81a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60e0200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f60d810e730 con 0x7f60d81035a0 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60e0200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f60d810e5b0 con 0x7f60d8105ed0 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60e0200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f60d810e8b0 con 0x7f60d8108800 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dd800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 0x7f60d819b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dd800700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 0x7f60d819b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:59202/0 (socket says 172.21.15.81:59202) 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dd800700 1 -- 172.21.15.81:0/138776579 learned_addr learned my addr 172.21.15.81:0/138776579 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60de200700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d8108800 0x7f60d81a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:29.448 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dce00700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60d8105ed0 0x7f60d819b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dd800700 1 -- 172.21.15.81:0/138776579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d8108800 msgr2=0x7f60d81a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dd800700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d8108800 0x7f60d81a25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dd800700 1 -- 172.21.15.81:0/138776579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60d8105ed0 msgr2=0x7f60d819b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dd800700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60d8105ed0 0x7f60d819b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.451+0000 7f60dd800700 1 -- 172.21.15.81:0/138776579 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d81a2c90 con 0x7f60d81035a0 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.452+0000 7f60dd800700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 0x7f60d819b030 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0098e0 tx=0x7f60cc00bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.452+0000 7f60d6000700 1 -- 172.21.15.81:0/138776579 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc026070 con 0x7f60d81035a0 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.452+0000 7f60d6000700 1 -- 172.21.15.81:0/138776579 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60cc003d70 con 0x7f60d81035a0 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.452+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d81a2f20 con 0x7f60d81035a0 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.452+0000 7f60d6000700 1 -- 172.21.15.81:0/138776579 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc021410 con 0x7f60d81035a0 2024-04-12T21:15:29.449 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.452+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d81a3310 con 0x7f60d81035a0 2024-04-12T21:15:29.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.453+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d80697a0 con 0x7f60d81035a0 2024-04-12T21:15:29.451 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.455+0000 7f60d6000700 1 -- 172.21.15.81:0/138776579 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f60cc029070 con 0x7f60d81035a0 2024-04-12T21:15:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.457+0000 7f60d6000700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f60c00a3cc0 0x7f60c00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:29.452 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.457+0000 7f60d6000700 1 -- 172.21.15.81:0/138776579 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f60cc0d14e0 con 0x7f60d81035a0 2024-04-12T21:15:29.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.457+0000 7f60dce00700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f60c00a3cc0 0x7f60c00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:29.453 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.458+0000 7f60dce00700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f60c00a3cc0 0x7f60c00a6180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60c4009c60 tx=0x7f60c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:29.457 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.461+0000 7f60d6000700 1 -- 172.21.15.81:0/138776579 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60cc05dff0 con 0x7f60d81035a0 2024-04-12T21:15:29.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:29 smithi081 ceph-mon[31453]: pgmap v386: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:29.695 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.699+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f60d8062ef0 con 0x7f60c00a3cc0 2024-04-12T21:15:29.710 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.715+0000 7f60d6000700 1 -- 172.21.15.81:0/138776579 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f60d8062ef0 con 0x7f60c00a3cc0 2024-04-12T21:15:29.711 INFO:teuthology.orchestra.run.smithi081.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-12T21:15:29.711 INFO:teuthology.orchestra.run.smithi081.stdout:mds.all 12/12 75s ago 3m smithi081=a;smithi081=d;smithi081=g;smithi081=j;smithi104=b;smithi104=e;smithi104=h;smithi104=k;smithi153=c;smithi153=f;smithi153=i;smithi153=l;count:12 2024-04-12T21:15:29.711 INFO:teuthology.orchestra.run.smithi081.stdout:mgr 3/3 75s ago 8m smithi081=x;smithi104=y;smithi153=z;count:3 2024-04-12T21:15:29.711 INFO:teuthology.orchestra.run.smithi081.stdout:mon 3/3 75s ago 9m smithi081:172.21.15.81=a;smithi104:172.21.15.104=b;smithi153:172.21.15.153=c;count:3 2024-04-12T21:15:29.711 INFO:teuthology.orchestra.run.smithi081.stdout:osd 12 75s ago - 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f60c00a3cc0 msgr2=0x7f60c00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f60c00a3cc0 0x7f60c00a6180 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60c4009c60 tx=0x7f60c4009260 comp rx=0 tx=0).stop 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 msgr2=0x7f60d819b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 0x7f60d819b030 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0098e0 tx=0x7f60cc00bbb0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 shutdown_connections 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f60c00a3cc0 0x7f60c00a6180 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60d8108800 0x7f60d81a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f60d81035a0 0x7f60d819b030 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 --2- 172.21.15.81:0/138776579 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f60d8105ed0 0x7f60d819b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:29.714 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 >> 172.21.15.81:0/138776579 conn(0x7f60d80fcfb0 msgr2=0x7f60d8102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:29.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 shutdown_connections 2024-04-12T21:15:29.715 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:29.718+0000 7f60e0200700 1 -- 172.21.15.81:0/138776579 wait complete. 2024-04-12T21:15:29.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:29 smithi153 ceph-mon[36569]: pgmap v386: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:29.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:29 smithi104 ceph-mon[36624]: pgmap v386: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:30.537 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-12T21:15:30.997 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:31 smithi153 ceph-mon[36569]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:31 smithi153 ceph-mon[36569]: pgmap v387: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-12T21:15:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:31 smithi104 ceph-mon[36624]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:31 smithi104 ceph-mon[36624]: pgmap v387: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-12T21:15:31.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:31 smithi081 ceph-mon[31453]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:31.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:31 smithi081 ceph-mon[31453]: pgmap v387: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s 2024-04-12T21:15:33.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.034+0000 7fccf1400700 1 -- 172.21.15.81:0/4129343951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fccec109040 msgr2=0x7fccec109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:33.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.034+0000 7fccf1400700 1 --2- 172.21.15.81:0/4129343951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fccec109040 0x7fccec109460 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fccdc0099c0 tx=0x7fccdc009cd0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.035+0000 7fccf1400700 1 -- 172.21.15.81:0/4129343951 shutdown_connections 2024-04-12T21:15:33.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.035+0000 7fccf1400700 1 --2- 172.21.15.81:0/4129343951 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fccec10abe0 0x7fccec111490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.035+0000 7fccf1400700 1 --2- 172.21.15.81:0/4129343951 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fccec109040 0x7fccec109460 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.035+0000 7fccf1400700 1 --2- 172.21.15.81:0/4129343951 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccec10a240 0x7fccec10a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.031 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.035+0000 7fccf1400700 1 -- 172.21.15.81:0/4129343951 >> 172.21.15.81:0/4129343951 conn(0x7fccec078010 msgr2=0x7fccec07a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:33.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.035+0000 7fccf1400700 1 -- 172.21.15.81:0/4129343951 shutdown_connections 2024-04-12T21:15:33.032 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.035+0000 7fccf1400700 1 -- 172.21.15.81:0/4129343951 wait complete. 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.036+0000 7fccf1400700 1 Processor -- start 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.036+0000 7fccf1400700 1 -- start start 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccf1400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccec109040 0x7fccec19f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccf1400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fccec10a240 0x7fccec19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccf1400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fccec10abe0 0x7fccec1a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccf1400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fccec114c80 con 0x7fccec10a240 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccf1400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fccec114b00 con 0x7fccec109040 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccf1400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fccec114e00 con 0x7fccec10abe0 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccebe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccec109040 0x7fccec19f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fcceb400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fccec10a240 0x7fccec19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:33.033 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccebe00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccec109040 0x7fccec19f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:44906/0 (socket says 172.21.15.81:44906) 2024-04-12T21:15:33.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccebe00700 1 -- 172.21.15.81:0/558838059 learned_addr learned my addr 172.21.15.81:0/558838059 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:33.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.037+0000 7fccf0a00700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fccec10abe0 0x7fccec1a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:33.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fccebe00700 1 -- 172.21.15.81:0/558838059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fccec10abe0 msgr2=0x7fccec1a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:33.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fccebe00700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fccec10abe0 0x7fccec1a6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.034 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fccebe00700 1 -- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fccec10a240 msgr2=0x7fccec19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:33.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fccebe00700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fccec10a240 0x7fccec19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fccebe00700 1 -- 172.21.15.81:0/558838059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec1a6ec0 con 0x7fccec109040 2024-04-12T21:15:33.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fccebe00700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccec109040 0x7fccec19f260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00b580 tx=0x7fccdc00bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:33.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fcce8c00700 1 -- 172.21.15.81:0/558838059 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc009e60 con 0x7fccec109040 2024-04-12T21:15:33.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccec1a7150 con 0x7fccec109040 2024-04-12T21:15:33.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fcce8c00700 1 -- 172.21.15.81:0/558838059 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccdc003e30 con 0x7fccec109040 2024-04-12T21:15:33.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fcce8c00700 1 -- 172.21.15.81:0/558838059 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc0173f0 con 0x7fccec109040 2024-04-12T21:15:33.035 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.038+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccec1a75b0 con 0x7fccec109040 2024-04-12T21:15:33.036 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.039+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccec0697a0 con 0x7fccec109040 2024-04-12T21:15:33.037 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.042+0000 7fcce8c00700 1 -- 172.21.15.81:0/558838059 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fccdc00f680 con 0x7fccec109040 2024-04-12T21:15:33.038 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.043+0000 7fcce8c00700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fccd00a3c40 0x7fccd00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:33.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.043+0000 7fcce8c00700 1 -- 172.21.15.81:0/558838059 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fccdc0d1600 con 0x7fccec109040 2024-04-12T21:15:33.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.043+0000 7fcceb400700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fccd00a3c40 0x7fccd00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:33.039 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.044+0000 7fcceb400700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fccd00a3c40 0x7fccd00a6100 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009680 tx=0x7fccd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:33.041 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.045+0000 7fcce8c00700 1 -- 172.21.15.81:0/558838059 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fccdc013070 con 0x7fccec109040 2024-04-12T21:15:33.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.283+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fccec062ef0 con 0x7fccd00a3c40 2024-04-12T21:15:33.281 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.286+0000 7fcce8c00700 1 -- 172.21.15.81:0/558838059 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fccec062ef0 con 0x7fccd00a3c40 2024-04-12T21:15:33.282 INFO:teuthology.orchestra.run.smithi081.stdout:HOST ADDR LABELS STATUS 2024-04-12T21:15:33.282 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 172.21.15.81 2024-04-12T21:15:33.282 INFO:teuthology.orchestra.run.smithi081.stdout:smithi104 172.21.15.104 2024-04-12T21:15:33.282 INFO:teuthology.orchestra.run.smithi081.stdout:smithi153 172.21.15.153 2024-04-12T21:15:33.282 INFO:teuthology.orchestra.run.smithi081.stdout:3 hosts in cluster 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fccd00a3c40 msgr2=0x7fccd00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fccd00a3c40 0x7fccd00a6100 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fccd8009680 tx=0x7fccd8009260 comp rx=0 tx=0).stop 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccec109040 msgr2=0x7fccec19f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccec109040 0x7fccec19f260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00b580 tx=0x7fccdc00bee0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 shutdown_connections 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fccd00a3c40 0x7fccd00a6100 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fccec10abe0 0x7fccec1a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fccec109040 0x7fccec19f260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 --2- 172.21.15.81:0/558838059 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fccec10a240 0x7fccec19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 >> 172.21.15.81:0/558838059 conn(0x7fccec078010 msgr2=0x7fccec10d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:33.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 shutdown_connections 2024-04-12T21:15:33.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:33.289+0000 7fccf1400700 1 -- 172.21.15.81:0/558838059 wait complete. 2024-04-12T21:15:33.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:33 smithi081 ceph-mon[31453]: pgmap v388: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:15:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:33 smithi153 ceph-mon[36569]: pgmap v388: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:15:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:33 smithi104 ceph-mon[36624]: pgmap v388: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:15:34.140 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-12T21:15:34.599 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:34.751 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:34 smithi081 ceph-mon[31453]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:34.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:34 smithi153 ceph-mon[36569]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:34.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:34 smithi104 ceph-mon[36624]: from='client.25193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:35.795 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:35 smithi081 ceph-mon[31453]: pgmap v389: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:15:35.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:35 smithi153 ceph-mon[36569]: pgmap v389: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:15:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:35 smithi104 ceph-mon[36624]: pgmap v389: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 1 op/s 2024-04-12T21:15:36.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.704+0000 7f3c7c200700 1 -- 172.21.15.81:0/3982512168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c74102b20 msgr2=0x7f3c74102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:36.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.704+0000 7f3c7c200700 1 --2- 172.21.15.81:0/3982512168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c74102b20 0x7f3c74102f40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3c680099c0 tx=0x7f3c68009cd0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.701 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.705+0000 7f3c7c200700 1 -- 172.21.15.81:0/3982512168 shutdown_connections 2024-04-12T21:15:36.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.705+0000 7f3c7c200700 1 --2- 172.21.15.81:0/3982512168 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c741046c0 0x7f3c7410af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.705+0000 7f3c7c200700 1 --2- 172.21.15.81:0/3982512168 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c74102b20 0x7f3c74102f40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.705+0000 7f3c7c200700 1 --2- 172.21.15.81:0/3982512168 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c74103d20 0x7f3c74104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.705+0000 7f3c7c200700 1 -- 172.21.15.81:0/3982512168 >> 172.21.15.81:0/3982512168 conn(0x7f3c740fe080 msgr2=0x7f3c74100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:36.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.705+0000 7f3c7c200700 1 -- 172.21.15.81:0/3982512168 shutdown_connections 2024-04-12T21:15:36.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.705+0000 7f3c7c200700 1 -- 172.21.15.81:0/3982512168 wait complete. 2024-04-12T21:15:36.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.706+0000 7f3c7c200700 1 Processor -- start 2024-04-12T21:15:36.702 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.706+0000 7f3c7c200700 1 -- start start 2024-04-12T21:15:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.706+0000 7f3c7c200700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c74102b20 0x7f3c74076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:36.703 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.706+0000 7f3c7c200700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c74103d20 0x7f3c74074ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.706+0000 7f3c7c200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741046c0 0x7f3c740753e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.706+0000 7f3c7c200700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7410e750 con 0x7f3c741046c0 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.706+0000 7f3c7c200700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7410e5d0 con 0x7f3c74103d20 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.706+0000 7f3c7c200700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3c7410e8d0 con 0x7f3c74102b20 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c7a200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741046c0 0x7f3c740753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c79800700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c74102b20 0x7f3c74076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c7a200700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741046c0 0x7f3c740753e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40762/0 (socket says 172.21.15.81:40762) 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c7a200700 1 -- 172.21.15.81:0/798750638 learned_addr learned my addr 172.21.15.81:0/798750638 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c78e00700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c74103d20 0x7f3c74074ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c78e00700 1 -- 172.21.15.81:0/798750638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c74102b20 msgr2=0x7f3c74076850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c78e00700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c74102b20 0x7f3c74076850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c78e00700 1 -- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741046c0 msgr2=0x7f3c740753e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c78e00700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741046c0 0x7f3c740753e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.704 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c78e00700 1 -- 172.21.15.81:0/798750638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c74075c90 con 0x7f3c74103d20 2024-04-12T21:15:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.707+0000 7f3c7a200700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741046c0 0x7f3c740753e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.708+0000 7f3c78e00700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c74103d20 0x7f3c74074ea0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000e9f0 tx=0x7f3c6000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.708+0000 7f3c72000700 1 -- 172.21.15.81:0/798750638 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60004220 con 0x7f3c74103d20 2024-04-12T21:15:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.708+0000 7f3c72000700 1 -- 172.21.15.81:0/798750638 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c60004d10 con 0x7f3c74103d20 2024-04-12T21:15:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.708+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c741a2f50 con 0x7f3c74103d20 2024-04-12T21:15:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.708+0000 7f3c72000700 1 -- 172.21.15.81:0/798750638 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60005020 con 0x7f3c74103d20 2024-04-12T21:15:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.708+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c741a33b0 con 0x7f3c74103d20 2024-04-12T21:15:36.705 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.709+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c740697a0 con 0x7f3c74103d20 2024-04-12T21:15:36.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.712+0000 7f3c72000700 1 -- 172.21.15.81:0/798750638 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3c60004750 con 0x7f3c74103d20 2024-04-12T21:15:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.713+0000 7f3c72000700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c5c0a3c40 0x7f3c5c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.713+0000 7f3c72000700 1 -- 172.21.15.81:0/798750638 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f3c600d08b0 con 0x7f3c74103d20 2024-04-12T21:15:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.714+0000 7f3c79800700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c5c0a3c40 0x7f3c5c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:36.709 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.714+0000 7f3c79800700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c5c0a3c40 0x7f3c5c0a6100 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800b580 tx=0x7f3c68005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:36.711 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.715+0000 7f3c72000700 1 -- 172.21.15.81:0/798750638 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c6009ce80 con 0x7f3c74103d20 2024-04-12T21:15:36.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.955+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 --> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f3c74062ef0 con 0x7f3c5c0a3c40 2024-04-12T21:15:36.953 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.957+0000 7f3c72000700 1 -- 172.21.15.81:0/798750638 <== mgr.14152 v2:172.21.15.81:6800/3972789130 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f3c74062ef0 con 0x7f3c5c0a3c40 2024-04-12T21:15:36.954 INFO:teuthology.orchestra.run.smithi081.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-12T21:15:36.954 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400N7400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-12T21:15:36.954 INFO:teuthology.orchestra.run.smithi081.stdout:smithi104 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NW400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-12T21:15:36.954 INFO:teuthology.orchestra.run.smithi081.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-12T21:15:36.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.960+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c5c0a3c40 msgr2=0x7f3c5c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:36.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.960+0000 7f3c7c200700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c5c0a3c40 0x7f3c5c0a6100 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c6800b580 tx=0x7f3c68005fb0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.960+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c74103d20 msgr2=0x7f3c74074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:36.956 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.960+0000 7f3c7c200700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c74103d20 0x7f3c74074ea0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000e9f0 tx=0x7f3c6000ed00 comp rx=0 tx=0).stop 2024-04-12T21:15:36.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.961+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 shutdown_connections 2024-04-12T21:15:36.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.961+0000 7f3c7c200700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f3c5c0a3c40 0x7f3c5c0a6100 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.961+0000 7f3c7c200700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f3c741046c0 0x7f3c740753e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.961+0000 7f3c7c200700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c74102b20 0x7f3c74076850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.961+0000 7f3c7c200700 1 --2- 172.21.15.81:0/798750638 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3c74103d20 0x7f3c74074ea0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:36.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.961+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 >> 172.21.15.81:0/798750638 conn(0x7f3c740fe080 msgr2=0x7f3c74107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:36.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.961+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 shutdown_connections 2024-04-12T21:15:36.957 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:36.961+0000 7f3c7c200700 1 -- 172.21.15.81:0/798750638 wait complete. 2024-04-12T21:15:37.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:37 smithi081 ceph-mon[31453]: pgmap v390: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-12T21:15:37.791 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-12T21:15:37.800 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-12T21:15:37.800 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-12T21:15:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:37 smithi153 ceph-mon[36569]: pgmap v390: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-12T21:15:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:37 smithi104 ceph-mon[36624]: pgmap v390: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 2 op/s 2024-04-12T21:15:38.267 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:38.786 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:38 smithi081 ceph-mon[31453]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:38.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:38 smithi153 ceph-mon[36569]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:38.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:38 smithi104 ceph-mon[36624]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-12T21:15:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:39 smithi153 ceph-mon[36569]: pgmap v391: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:39 smithi104 ceph-mon[36624]: pgmap v391: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:39.836 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:39 smithi081 ceph-mon[31453]: pgmap v391: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:40.330 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.334+0000 7ff722a00700 1 -- 172.21.15.81:0/1763218788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c109150 msgr2=0x7ff71c109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:40.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.334+0000 7ff722a00700 1 --2- 172.21.15.81:0/1763218788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c109150 0x7ff71c109570 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff70c0099c0 tx=0x7ff70c009cd0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.335+0000 7ff722a00700 1 -- 172.21.15.81:0/1763218788 shutdown_connections 2024-04-12T21:15:40.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.335+0000 7ff722a00700 1 --2- 172.21.15.81:0/1763218788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c10acf0 0x7ff71c1115a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.335+0000 7ff722a00700 1 --2- 172.21.15.81:0/1763218788 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c109150 0x7ff71c109570 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.335+0000 7ff722a00700 1 --2- 172.21.15.81:0/1763218788 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c10a350 0x7ff71c10a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.335+0000 7ff722a00700 1 -- 172.21.15.81:0/1763218788 >> 172.21.15.81:0/1763218788 conn(0x7ff71c078120 msgr2=0x7ff71c07a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:40.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.335+0000 7ff722a00700 1 -- 172.21.15.81:0/1763218788 shutdown_connections 2024-04-12T21:15:40.331 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.335+0000 7ff722a00700 1 -- 172.21.15.81:0/1763218788 wait complete. 2024-04-12T21:15:40.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.336+0000 7ff722a00700 1 Processor -- start 2024-04-12T21:15:40.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.336+0000 7ff722a00700 1 -- start start 2024-04-12T21:15:40.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.336+0000 7ff722a00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c109150 0x7ff71c19f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:40.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.336+0000 7ff722a00700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c10a350 0x7ff71c19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:40.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.336+0000 7ff722a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c10acf0 0x7ff71c1a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:40.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.336+0000 7ff722a00700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c114d10 con 0x7ff71c10acf0 2024-04-12T21:15:40.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.336+0000 7ff722a00700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c114b90 con 0x7ff71c10a350 2024-04-12T21:15:40.332 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.336+0000 7ff722a00700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff71c114e90 con 0x7ff71c109150 2024-04-12T21:15:40.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff71be00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c109150 0x7ff71c19f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:40.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff720a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c10acf0 0x7ff71c1a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:40.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff71be00700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c109150 0x7ff71c19f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:52698/0 (socket says 172.21.15.81:52698) 2024-04-12T21:15:40.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff71be00700 1 -- 172.21.15.81:0/3812972459 learned_addr learned my addr 172.21.15.81:0/3812972459 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:40.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff71b400700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c10a350 0x7ff71c19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff720a00700 1 -- 172.21.15.81:0/3812972459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c109150 msgr2=0x7ff71c19f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff720a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c109150 0x7ff71c19f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff720a00700 1 -- 172.21.15.81:0/3812972459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c10a350 msgr2=0x7ff71c19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff720a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c10a350 0x7ff71c19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff720a00700 1 -- 172.21.15.81:0/3812972459 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff71c1a6f50 con 0x7ff71c10acf0 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff71b400700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c10a350 0x7ff71c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.337+0000 7ff71be00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c109150 0x7ff71c19f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.338+0000 7ff720a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c10acf0 0x7ff71c1a68b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff714009f60 tx=0x7ff714009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.338+0000 7ff718c00700 1 -- 172.21.15.81:0/3812972459 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7140092e0 con 0x7ff71c10acf0 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.338+0000 7ff718c00700 1 -- 172.21.15.81:0/3812972459 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff714004bb0 con 0x7ff71c10acf0 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.338+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff71c1a71e0 con 0x7ff71c10acf0 2024-04-12T21:15:40.334 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.338+0000 7ff718c00700 1 -- 172.21.15.81:0/3812972459 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714004280 con 0x7ff71c10acf0 2024-04-12T21:15:40.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.338+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff71c1a7720 con 0x7ff71c10acf0 2024-04-12T21:15:40.335 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.339+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff71c0697a0 con 0x7ff71c10acf0 2024-04-12T21:15:40.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.341+0000 7ff718c00700 1 -- 172.21.15.81:0/3812972459 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff71401b030 con 0x7ff71c10acf0 2024-04-12T21:15:40.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.342+0000 7ff718c00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7000a3cc0 0x7ff7000a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:40.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.343+0000 7ff718c00700 1 -- 172.21.15.81:0/3812972459 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7ff7140d01b0 con 0x7ff71c10acf0 2024-04-12T21:15:40.338 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.343+0000 7ff71be00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7000a3cc0 0x7ff7000a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:40.339 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.344+0000 7ff71be00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7000a3cc0 0x7ff7000a6180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff70c000c00 tx=0x7ff70c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:40.343 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.347+0000 7ff718c00700 1 -- 172.21.15.81:0/3812972459 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff714022020 con 0x7ff71c10acf0 2024-04-12T21:15:40.619 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.624+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7ff71c0636c0 con 0x7ff71c10acf0 2024-04-12T21:15:40.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.624+0000 7ff718c00700 1 -- 172.21.15.81:0/3812972459 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) ==== 76+0+3014 (secure 0 0 0) 0x7ff714010040 con 0x7ff71c10acf0 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:e20 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:legacy client fscid: 1 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:Filesystem 'cephfs' (1) 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:fs_name cephfs 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 20 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-12T21:15:40.621 INFO:teuthology.orchestra.run.smithi081.stdout:created 2024-04-12T21:13:13.003153+0000 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:modified 2024-04-12T21:13:37.617228+0000 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:tableserver 0 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:root 0 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:session_timeout 300 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:session_autoclose 300 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:max_file_size 1099511627776 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:max_xattr_size 65536 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:required_client_features {} 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:last_failure 0 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:last_failure_osd_epoch 0 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:max_mds 1 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:in 0 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:up {0=24593} 2024-04-12T21:15:40.622 INFO:teuthology.orchestra.run.smithi081.stdout:failed 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:damaged 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:stopped 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:data_pools [3,4] 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:metadata_pool 2 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:inline_data disabled 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:balancer 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:bal_rank_mask -1 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:standby_count_wanted 0 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:qdb_cluster leader: 24593 members: 24593 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.b{0:24593} state up:active seq 3 addr [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.i{0:24547} state up:standby-replay seq 1 addr [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:Standby daemons: 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.j{-1:14379} state up:standby seq 1 addr [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.623 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.d{-1:14391} state up:standby seq 1 addr [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.g{-1:14445} state up:standby seq 1 addr [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.a{-1:14457} state up:standby seq 1 addr [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.l{-1:24454} state up:standby seq 1 addr [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.k{-1:24458} state up:standby seq 1 addr [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.e{-1:24506} state up:standby seq 1 addr [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.f{-1:24517} state up:standby seq 1 addr [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.c{-1:24529} state up:standby seq 1 addr [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:[mds.h{-1:24536} state up:standby seq 1 addr [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] compat {c=[1],r=[1],i=[1fff]}] 2024-04-12T21:15:40.624 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7000a3cc0 msgr2=0x7ff7000a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7000a3cc0 0x7ff7000a6180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff70c000c00 tx=0x7ff70c005fb0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c10acf0 msgr2=0x7ff71c1a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c10acf0 0x7ff71c1a68b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff714009f60 tx=0x7ff714009f90 comp rx=0 tx=0).stop 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 shutdown_connections 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7ff7000a3cc0 0x7ff7000a6180 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7ff71c10acf0 0x7ff71c1a68b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff71c109150 0x7ff71c19f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 --2- 172.21.15.81:0/3812972459 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff71c10a350 0x7ff71c19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 >> 172.21.15.81:0/3812972459 conn(0x7ff71c078120 msgr2=0x7ff71c10d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 shutdown_connections 2024-04-12T21:15:40.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:40.628+0000 7ff722a00700 1 -- 172.21.15.81:0/3812972459 wait complete. 2024-04-12T21:15:40.626 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-04-12T21:15:41.419 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-12T21:15:41.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:41 smithi081 ceph-mon[31453]: pgmap v392: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:41.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:41 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3812972459' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T21:15:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:41 smithi153 ceph-mon[36569]: pgmap v392: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:41 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3812972459' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T21:15:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:41 smithi104 ceph-mon[36624]: pgmap v392: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:41 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3812972459' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-12T21:15:41.866 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:43.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:43 smithi081 ceph-mon[31453]: pgmap v393: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:43 smithi153 ceph-mon[36569]: pgmap v393: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:43 smithi104 ceph-mon[36624]: pgmap v393: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:44.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 -- 172.21.15.81:0/734033986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 msgr2=0x7f9458111490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:44.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 --2- 172.21.15.81:0/734033986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 0x7f9458111490 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f945000b0d0 tx=0x7f945000b3e0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.008 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 -- 172.21.15.81:0/734033986 shutdown_connections 2024-04-12T21:15:44.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 --2- 172.21.15.81:0/734033986 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 0x7f9458111490 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 --2- 172.21.15.81:0/734033986 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9458109040 0x7f9458109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 --2- 172.21.15.81:0/734033986 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f945810a240 0x7f945810a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 -- 172.21.15.81:0/734033986 >> 172.21.15.81:0/734033986 conn(0x7f9458078010 msgr2=0x7f945807a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:44.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 -- 172.21.15.81:0/734033986 shutdown_connections 2024-04-12T21:15:44.009 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.012+0000 7f945e400700 1 -- 172.21.15.81:0/734033986 wait complete. 2024-04-12T21:15:44.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.014+0000 7f945e400700 1 Processor -- start 2024-04-12T21:15:44.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.014+0000 7f945e400700 1 -- start start 2024-04-12T21:15:44.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.014+0000 7f945e400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9458109040 0x7f945819f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:44.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.014+0000 7f945e400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945810a240 0x7f945819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:44.010 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.014+0000 7f945e400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 0x7f94581a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.014+0000 7f945e400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9458114c70 con 0x7f945810abe0 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.014+0000 7f945e400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9458114af0 con 0x7f945810a240 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.014+0000 7f945e400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9458114df0 con 0x7f9458109040 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945da00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 0x7f94581a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945d000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9458109040 0x7f945819f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945d000700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9458109040 0x7f945819f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:52726/0 (socket says 172.21.15.81:52726) 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945d000700 1 -- 172.21.15.81:0/1536756269 learned_addr learned my addr 172.21.15.81:0/1536756269 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945da00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 0x7f94581a6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:40820/0 (socket says 172.21.15.81:40820) 2024-04-12T21:15:44.011 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f9457e00700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945810a240 0x7f945819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:44.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945da00700 1 -- 172.21.15.81:0/1536756269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9458109040 msgr2=0x7f945819f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:44.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945da00700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9458109040 0x7f945819f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945da00700 1 -- 172.21.15.81:0/1536756269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945810a240 msgr2=0x7f945819f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:44.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945da00700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945810a240 0x7f945819f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.015+0000 7f945da00700 1 -- 172.21.15.81:0/1536756269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94581a6ea0 con 0x7f945810abe0 2024-04-12T21:15:44.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.016+0000 7f945da00700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 0x7f94581a6800 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9450009fd0 tx=0x7f94500079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:44.012 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.016+0000 7f9455600700 1 -- 172.21.15.81:0/1536756269 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9450021070 con 0x7f945810abe0 2024-04-12T21:15:44.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.016+0000 7f9455600700 1 -- 172.21.15.81:0/1536756269 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f945001db30 con 0x7f945810abe0 2024-04-12T21:15:44.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.016+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94581a7130 con 0x7f945810abe0 2024-04-12T21:15:44.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.016+0000 7f9455600700 1 -- 172.21.15.81:0/1536756269 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945001bba0 con 0x7f945810abe0 2024-04-12T21:15:44.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.016+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94581a7630 con 0x7f945810abe0 2024-04-12T21:15:44.013 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.017+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94580697a0 con 0x7f945810abe0 2024-04-12T21:15:44.014 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.019+0000 7f9455600700 1 -- 172.21.15.81:0/1536756269 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9450024070 con 0x7f945810abe0 2024-04-12T21:15:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.021+0000 7f9455600700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94400a3cc0 0x7f94400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.021+0000 7f9455600700 1 -- 172.21.15.81:0/1536756269 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f945000f020 con 0x7f945810abe0 2024-04-12T21:15:44.016 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.021+0000 7f945d000700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94400a3cc0 0x7f94400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:44.017 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.022+0000 7f945d000700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94400a3cc0 0x7f94400a6180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f944c005950 tx=0x7f944c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:44.020 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.025+0000 7f9455600700 1 -- 172.21.15.81:0/1536756269 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f945002b020 con 0x7f945810abe0 2024-04-12T21:15:44.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.251+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f94580636c0 con 0x7f945810abe0 2024-04-12T21:15:44.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.253+0000 7f9455600700 1 -- 172.21.15.81:0/1536756269 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5689 (secure 0 0 0) 0x7f945009e2a0 con 0x7f945810abe0 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 93 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:created 2024-04-12T21:04:15.716001+0000 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:modified 2024-04-12T21:15:24.515674+0000 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:crush_version 27 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:full_ratio 0.95 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:backfillfull_ratio 0.9 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:nearfull_ratio 0.85 2024-04-12T21:15:44.251 INFO:teuthology.orchestra.run.smithi081.stdout:require_min_compat_client luminous 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:min_compat_client luminous 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:require_osd_release squid 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:stretch_mode_enabled false 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:max_osd 12 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6802/4059792347,v1:172.21.15.81:6803/4059792347] [v2:172.21.15.81:6804/4059792347,v1:172.21.15.81:6805/4059792347] exists,up 5891e774-9745-4511-a67d-b0d5b30f48ad 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6800/3106352403,v1:172.21.15.104:6801/3106352403] [v2:172.21.15.104:6802/3106352403,v1:172.21.15.104:6803/3106352403] exists,up 60afc2bd-b9e9-4489-9cbc-a756b944fdc4 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6800/1512806537,v1:172.21.15.153:6801/1512806537] [v2:172.21.15.153:6802/1512806537,v1:172.21.15.153:6803/1512806537] exists,up b4a8e96d-ddca-4406-977f-543283eadef9 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6810/4225327027,v1:172.21.15.81:6811/4225327027] [v2:172.21.15.81:6812/4225327027,v1:172.21.15.81:6813/4225327027] exists,up 5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6808/742573974,v1:172.21.15.104:6809/742573974] [v2:172.21.15.104:6810/742573974,v1:172.21.15.104:6811/742573974] exists,up 2eb029c7-091e-43bb-8085-4f736bcf6a62 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6808/2854081701,v1:172.21.15.153:6809/2854081701] [v2:172.21.15.153:6810/2854081701,v1:172.21.15.153:6811/2854081701] exists,up 0fe772d4-5800-4a88-a101-1dbf64b4abf6 2024-04-12T21:15:44.252 INFO:teuthology.orchestra.run.smithi081.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6818/3825683804,v1:172.21.15.81:6819/3825683804] [v2:172.21.15.81:6820/3825683804,v1:172.21.15.81:6821/3825683804] exists,up 77e17684-22e7-4e11-b27f-ffc4703182ba 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6816/1651003262,v1:172.21.15.104:6817/1651003262] [v2:172.21.15.104:6818/1651003262,v1:172.21.15.104:6819/1651003262] exists,up 535d287c-497d-47a4-99e0-0d72ace39cd4 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:osd.8 up in weight 1 up_from 59 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6816/2565156870,v1:172.21.15.153:6817/2565156870] [v2:172.21.15.153:6818/2565156870,v1:172.21.15.153:6819/2565156870] exists,up 799f954d-0c19-4fb6-a80d-5df6f65588c5 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.81:6826/1653329023,v1:172.21.15.81:6827/1653329023] [v2:172.21.15.81:6828/1653329023,v1:172.21.15.81:6829/1653329023] exists,up bbb02774-5697-4bcb-ac44-8b01cef5f3fb 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:osd.10 up in weight 1 up_from 72 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.104:6824/1019144761,v1:172.21.15.104:6825/1019144761] [v2:172.21.15.104:6826/1019144761,v1:172.21.15.104:6827/1019144761] exists,up 30c8758d-3e32-4d59-9819-a13a38c1745d 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.153:6824/2242325561,v1:172.21.15.153:6825/2242325561] [v2:172.21.15.153:6826/2242325561,v1:172.21.15.153:6827/2242325561] exists,up b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 2.b [8,11] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 2.13 [2,11] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 2.16 [2,4] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.0 [5,9] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.c [3,9] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.10 [5,9] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.14 [4,6] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.1b [4,8] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 3.3a [4,7] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 4.c [9,10] 2024-04-12T21:15:44.253 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 4.14 [9,8] 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 4.15 [5,10] 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:pg_upmap_items 4.1c [9,8] 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/1836548478 expires 2024-04-13T21:05:22.179703+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6801/1636479292 expires 2024-04-13T21:05:22.179703+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6800/3647696686 expires 2024-04-13T21:04:48.054528+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/60786815 expires 2024-04-13T21:05:22.179703+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/3894299177 expires 2024-04-13T21:04:48.054528+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/2659089739 expires 2024-04-13T21:04:48.054528+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/2550894701 expires 2024-04-13T21:05:22.179703+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6800/1636479292 expires 2024-04-13T21:05:22.179703+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:6801/3647696686 expires 2024-04-13T21:04:48.054528+0000 2024-04-12T21:15:44.254 INFO:teuthology.orchestra.run.smithi081.stdout:blocklist 172.21.15.81:0/3673746318 expires 2024-04-13T21:04:48.054528+0000 2024-04-12T21:15:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94400a3cc0 msgr2=0x7f94400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94400a3cc0 0x7f94400a6180 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f944c005950 tx=0x7f944c009260 comp rx=0 tx=0).stop 2024-04-12T21:15:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 msgr2=0x7f94581a6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 0x7f94581a6800 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9450009fd0 tx=0x7f94500079a0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 shutdown_connections 2024-04-12T21:15:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f94400a3cc0 0x7f94400a6180 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f945810abe0 0x7f94581a6800 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.255 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9458109040 0x7f945819f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 --2- 172.21.15.81:0/1536756269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f945810a240 0x7f945819f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:44.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.256+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 >> 172.21.15.81:0/1536756269 conn(0x7f9458078010 msgr2=0x7f945810d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:44.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.257+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 shutdown_connections 2024-04-12T21:15:44.256 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:44.257+0000 7f945e400700 1 -- 172.21.15.81:0/1536756269 wait complete. 2024-04-12T21:15:44.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:44 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1536756269' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-12T21:15:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:44 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1536756269' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-12T21:15:44.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:44 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1536756269' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-12T21:15:45.071 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-12T21:15:45.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:15:45.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 -- 172.21.15.81:0/321987657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44103940 msgr2=0x7f9e4410a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 --2- 172.21.15.81:0/321987657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44103940 0x7f9e4410a260 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9e3c009be0 tx=0x7f9e3c02bef0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 -- 172.21.15.81:0/321987657 shutdown_connections 2024-04-12T21:15:45.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 --2- 172.21.15.81:0/321987657 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44103940 0x7f9e4410a260 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 --2- 172.21.15.81:0/321987657 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e44102f80 0x7f9e44103400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.311 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 --2- 172.21.15.81:0/321987657 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e44101e30 0x7f9e44102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 -- 172.21.15.81:0/321987657 >> 172.21.15.81:0/321987657 conn(0x7f9e440fd3a0 msgr2=0x7f9e440ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:45.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 -- 172.21.15.81:0/321987657 shutdown_connections 2024-04-12T21:15:45.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.315+0000 7f9e43200640 1 -- 172.21.15.81:0/321987657 wait complete. 2024-04-12T21:15:45.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.317+0000 7f9e43200640 1 Processor -- start 2024-04-12T21:15:45.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.317+0000 7f9e43200640 1 -- start start 2024-04-12T21:15:45.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.317+0000 7f9e43200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44101e30 0x7f9e44075d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:45.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.317+0000 7f9e43200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e44102f80 0x7f9e440743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:45.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.317+0000 7f9e43200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e44103940 0x7f9e440748f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.317+0000 7f9e43200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4410db00 con 0x7f9e44103940 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.317+0000 7f9e43200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4410d980 con 0x7f9e44101e30 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.317+0000 7f9e43200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9e4410dc80 con 0x7f9e44102f80 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44101e30 0x7f9e44075d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e42800640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e44103940 0x7f9e440748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44101e30 0x7f9e44075d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56200/0 (socket says 172.21.15.81:56200) 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 -- 172.21.15.81:0/1716050327 learned_addr learned my addr 172.21.15.81:0/1716050327 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41400640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e44102f80 0x7f9e440743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 -- 172.21.15.81:0/1716050327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e44102f80 msgr2=0x7f9e440743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e44102f80 0x7f9e440743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 -- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e44103940 msgr2=0x7f9e440748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e44103940 0x7f9e440748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 -- 172.21.15.81:0/1716050327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e44075180 con 0x7f9e44101e30 2024-04-12T21:15:45.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e42800640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e44103940 0x7f9e440748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:45.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.318+0000 7f9e41e00640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44101e30 0x7f9e44075d70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400e9c0 tx=0x7f9e3400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:45.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.319+0000 7f9e3aa00640 1 -- 172.21.15.81:0/1716050327 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e3400ccc0 con 0x7f9e44101e30 2024-04-12T21:15:45.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.319+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e441a2200 con 0x7f9e44101e30 2024-04-12T21:15:45.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.319+0000 7f9e3aa00640 1 -- 172.21.15.81:0/1716050327 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e34005500 con 0x7f9e44101e30 2024-04-12T21:15:45.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.319+0000 7f9e3aa00640 1 -- 172.21.15.81:0/1716050327 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34010430 con 0x7f9e44101e30 2024-04-12T21:15:45.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.319+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e441a27a0 con 0x7f9e44101e30 2024-04-12T21:15:45.316 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.320+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e441090e0 con 0x7f9e44101e30 2024-04-12T21:15:45.318 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.322+0000 7f9e3aa00640 1 -- 172.21.15.81:0/1716050327 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f9e340046c0 con 0x7f9e44101e30 2024-04-12T21:15:45.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.323+0000 7f9e3aa00640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9e280a3c30 0x7f9e280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:45.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.324+0000 7f9e3aa00640 1 -- 172.21.15.81:0/1716050327 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f9e340d0ca0 con 0x7f9e44101e30 2024-04-12T21:15:45.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.324+0000 7f9e41400640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9e280a3c30 0x7f9e280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:45.320 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.324+0000 7f9e41400640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9e280a3c30 0x7f9e280a60f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e44075ae0 tx=0x7f9e2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:45.321 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.326+0000 7f9e3aa00640 1 -- 172.21.15.81:0/1716050327 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e34053ff0 con 0x7f9e44101e30 2024-04-12T21:15:45.526 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.530+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e44062bc0 con 0x7f9e44101e30 2024-04-12T21:15:45.528 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.533+0000 7f9e3aa00640 1 -- 172.21.15.81:0/1716050327 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24821 (secure 0 0 0) 0x7f9e3409d270 con 0x7f9e44101e30 2024-04-12T21:15:45.529 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:45.529 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":93,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:15:24.515674+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:15:45.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.535+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9e280a3c30 msgr2=0x7f9e280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.531 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.535+0000 7f9e43200640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9e280a3c30 0x7f9e280a60f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e44075ae0 tx=0x7f9e2c005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.535+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44101e30 msgr2=0x7f9e44075d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.535+0000 7f9e43200640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44101e30 0x7f9e44075d70 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400e9c0 tx=0x7f9e3400ee80 comp rx=0 tx=0).stop 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.536+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 shutdown_connections 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.536+0000 7f9e43200640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f9e280a3c30 0x7f9e280a60f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.536+0000 7f9e43200640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9e44102f80 0x7f9e440743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.536+0000 7f9e43200640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f9e44103940 0x7f9e440748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.536+0000 7f9e43200640 1 --2- 172.21.15.81:0/1716050327 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f9e44101e30 0x7f9e44075d70 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.536+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 >> 172.21.15.81:0/1716050327 conn(0x7f9e440fd3a0 msgr2=0x7f9e441062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.536+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 shutdown_connections 2024-04-12T21:15:45.532 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.536+0000 7f9e43200640 1 -- 172.21.15.81:0/1716050327 wait complete. 2024-04-12T21:15:45.551 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-12T21:13:10.561238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-12T21:13:11.985544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-12T21:13:15.579438+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-12T21:15:45.551 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T21:15:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1009942537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc104220 msgr2=0x7fbfcc104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1009942537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc104220 0x7fbfcc104640 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009900 tx=0x7fbfbc02bc40 comp rx=0 tx=0).stop 2024-04-12T21:15:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1009942537 shutdown_connections 2024-04-12T21:15:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1009942537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfcc105410 0x7fbfcc105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1009942537 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfcc105dd0 0x7fbfcc10c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1009942537 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc104220 0x7fbfcc104640 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1009942537 >> 172.21.15.81:0/1009942537 conn(0x7fbfcc0ff7b0 msgr2=0x7fbfcc101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:45.769 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1009942537 shutdown_connections 2024-04-12T21:15:45.770 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.773+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1009942537 wait complete. 2024-04-12T21:15:45.771 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.774+0000 7fbfd0e00640 1 Processor -- start 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.774+0000 7fbfd0e00640 1 -- start start 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.774+0000 7fbfd0e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfcc104220 0x7fbfcc19a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.774+0000 7fbfd0e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc105410 0x7fbfcc19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfd0e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfcc105dd0 0x7fbfcc1a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfd0e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbfcc10fed0 con 0x7fbfcc105410 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfd0e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbfcc10fd50 con 0x7fbfcc105dd0 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfd0e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbfcc110050 con 0x7fbfcc104220 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc105410 0x7fbfcc19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc9e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfcc104220 0x7fbfcc19a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:45.772 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc1400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc105410 0x7fbfcc19aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37616/0 (socket says 172.21.15.81:37616) 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc1400640 1 -- 172.21.15.81:0/1121175208 learned_addr learned my addr 172.21.15.81:0/1121175208 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfca800640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfcc105dd0 0x7fbfcc1a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc1400640 1 -- 172.21.15.81:0/1121175208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfcc104220 msgr2=0x7fbfcc19a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc1400640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfcc104220 0x7fbfcc19a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc1400640 1 -- 172.21.15.81:0/1121175208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfcc105dd0 msgr2=0x7fbfcc1a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc1400640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfcc105dd0 0x7fbfcc1a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfc1400640 1 -- 172.21.15.81:0/1121175208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfcc1a21a0 con 0x7fbfcc105410 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.775+0000 7fbfca800640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfcc105dd0 0x7fbfcc1a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:45.773 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.776+0000 7fbfc1400640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc105410 0x7fbfcc19aac0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400e9f0 tx=0x7fbfb400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:45.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.776+0000 7fbfc3400640 1 -- 172.21.15.81:0/1121175208 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb400cd80 con 0x7fbfcc105410 2024-04-12T21:15:45.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.776+0000 7fbfc3400640 1 -- 172.21.15.81:0/1121175208 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfb4010910 con 0x7fbfcc105410 2024-04-12T21:15:45.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.776+0000 7fbfc3400640 1 -- 172.21.15.81:0/1121175208 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb4018990 con 0x7fbfcc105410 2024-04-12T21:15:45.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.776+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfcc1a2490 con 0x7fbfcc105410 2024-04-12T21:15:45.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.776+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfcc1a2990 con 0x7fbfcc105410 2024-04-12T21:15:45.774 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.777+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfcc10b4e0 con 0x7fbfcc105410 2024-04-12T21:15:45.775 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.779+0000 7fbfc3400640 1 -- 172.21.15.81:0/1121175208 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbfb4010ac0 con 0x7fbfcc105410 2024-04-12T21:15:45.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.781+0000 7fbfc3400640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbfb00a3c30 0x7fbfb00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:45.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.781+0000 7fbfc3400640 1 -- 172.21.15.81:0/1121175208 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fbfb40d0e80 con 0x7fbfcc105410 2024-04-12T21:15:45.776 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.781+0000 7fbfc9e00640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbfb00a3c30 0x7fbfb00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:45.777 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.782+0000 7fbfc9e00640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbfb00a3c30 0x7fbfb00a60f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc02c250 tx=0x7fbfbc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:45.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.783+0000 7fbfc3400640 1 -- 172.21.15.81:0/1121175208 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbfb409ce00 con 0x7fbfcc105410 2024-04-12T21:15:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:45 smithi153 ceph-mon[36569]: pgmap v394: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:45.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:45 smithi104 ceph-mon[36624]: pgmap v394: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:45.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:45 smithi081 ceph-mon[31453]: pgmap v394: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:45.981 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.986+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbfcc02d770 con 0x7fbfcc105410 2024-04-12T21:15:45.982 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.987+0000 7fbfc3400640 1 -- 172.21.15.81:0/1121175208 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fbfb40a1100 con 0x7fbfcc105410 2024-04-12T21:15:45.983 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:15:45.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.989+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbfb00a3c30 msgr2=0x7fbfb00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.989+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbfb00a3c30 0x7fbfb00a60f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc02c250 tx=0x7fbfbc02cf90 comp rx=0 tx=0).stop 2024-04-12T21:15:45.985 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.989+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc105410 msgr2=0x7fbfcc19aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.989+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc105410 0x7fbfcc19aac0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbfb400e9f0 tx=0x7fbfb400eeb0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.990+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 shutdown_connections 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.990+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbfb00a3c30 0x7fbfb00a60f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.990+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbfcc105dd0 0x7fbfcc1a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.990+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbfcc105410 0x7fbfcc19aac0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.990+0000 7fbfd0e00640 1 --2- 172.21.15.81:0/1121175208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbfcc104220 0x7fbfcc19a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.990+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 >> 172.21.15.81:0/1121175208 conn(0x7fbfcc0ff7b0 msgr2=0x7fbfcc1086b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.990+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 shutdown_connections 2024-04-12T21:15:45.986 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:45.990+0000 7fbfd0e00640 1 -- 172.21.15.81:0/1121175208 wait complete. 2024-04-12T21:15:46.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-12T21:15:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 -- 172.21.15.81:0/2450877783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb510108500 msgr2=0x7fb510108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 --2- 172.21.15.81:0/2450877783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb510108500 0x7fb510108920 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb504009900 tx=0x7fb50402bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 -- 172.21.15.81:0/2450877783 shutdown_connections 2024-04-12T21:15:46.227 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 --2- 172.21.15.81:0/2450877783 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb51010a0b0 0x7fb510110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 --2- 172.21.15.81:0/2450877783 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb5101096f0 0x7fb510109b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 --2- 172.21.15.81:0/2450877783 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb510108500 0x7fb510108920 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 -- 172.21.15.81:0/2450877783 >> 172.21.15.81:0/2450877783 conn(0x7fb510077490 msgr2=0x7fb5100798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 -- 172.21.15.81:0/2450877783 shutdown_connections 2024-04-12T21:15:46.228 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.231+0000 7fb516a00640 1 -- 172.21.15.81:0/2450877783 wait complete. 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.232+0000 7fb516a00640 1 Processor -- start 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb516a00640 1 -- start start 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb516a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb510108500 0x7fb51019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb516a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5101096f0 0x7fb51019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb516a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51010a0b0 0x7fb5101a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb516a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb5101141b0 con 0x7fb510108500 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb516a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb510114030 con 0x7fb51010a0b0 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb516a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb510114330 con 0x7fb5101096f0 2024-04-12T21:15:46.230 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb515600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb510108500 0x7fb51019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb515600640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb510108500 0x7fb51019e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37630/0 (socket says 172.21.15.81:37630) 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb515600640 1 -- 172.21.15.81:0/2900674451 learned_addr learned my addr 172.21.15.81:0/2900674451 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb516000640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51010a0b0 0x7fb5101a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.233+0000 7fb514c00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5101096f0 0x7fb51019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb515600640 1 -- 172.21.15.81:0/2900674451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5101096f0 msgr2=0x7fb51019ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb515600640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5101096f0 0x7fb51019ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb515600640 1 -- 172.21.15.81:0/2900674451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51010a0b0 msgr2=0x7fb5101a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb515600640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51010a0b0 0x7fb5101a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb515600640 1 -- 172.21.15.81:0/2900674451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5101a63f0 con 0x7fb510108500 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb516000640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51010a0b0 0x7fb5101a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb515600640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb510108500 0x7fb51019e7d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb504000c00 tx=0x7fb50402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb50e000640 1 -- 172.21.15.81:0/2900674451 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb504043020 con 0x7fb510108500 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5101a6680 con 0x7fb510108500 2024-04-12T21:15:46.231 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb50e000640 1 -- 172.21.15.81:0/2900674451 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5040054e0 con 0x7fb510108500 2024-04-12T21:15:46.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.234+0000 7fb50e000640 1 -- 172.21.15.81:0/2900674451 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb504003c40 con 0x7fb510108500 2024-04-12T21:15:46.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.235+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5101a6bd0 con 0x7fb510108500 2024-04-12T21:15:46.232 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.236+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb51010f820 con 0x7fb510108500 2024-04-12T21:15:46.233 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.237+0000 7fb50e000640 1 -- 172.21.15.81:0/2900674451 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb504046020 con 0x7fb510108500 2024-04-12T21:15:46.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.239+0000 7fb50e000640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4f80a3bb0 0x7fb4f80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:46.234 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.239+0000 7fb50e000640 1 -- 172.21.15.81:0/2900674451 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fb5040f27b0 con 0x7fb510108500 2024-04-12T21:15:46.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.239+0000 7fb514c00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4f80a3bb0 0x7fb4f80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:46.235 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.240+0000 7fb514c00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4f80a3bb0 0x7fb4f80a6070 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb51019fce0 tx=0x7fb500009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:46.239 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.243+0000 7fb50e000640 1 -- 172.21.15.81:0/2900674451 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb50404d020 con 0x7fb510108500 2024-04-12T21:15:46.441 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.446+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb510062bc0 con 0x7fb510108500 2024-04-12T21:15:46.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.447+0000 7fb50e000640 1 -- 172.21.15.81:0/2900674451 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fb5040bed80 con 0x7fb510108500 2024-04-12T21:15:46.443 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:15:46.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4f80a3bb0 msgr2=0x7fb4f80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4f80a3bb0 0x7fb4f80a6070 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb51019fce0 tx=0x7fb500009500 comp rx=0 tx=0).stop 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb510108500 msgr2=0x7fb51019e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb510108500 0x7fb51019e7d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb504000c00 tx=0x7fb50402c650 comp rx=0 tx=0).stop 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 shutdown_connections 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb4f80a3bb0 0x7fb4f80a6070 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb5101096f0 0x7fb51019ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb51010a0b0 0x7fb5101a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 --2- 172.21.15.81:0/2900674451 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb510108500 0x7fb51019e7d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 >> 172.21.15.81:0/2900674451 conn(0x7fb510077490 msgr2=0x7fb51010c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 shutdown_connections 2024-04-12T21:15:46.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.450+0000 7fb516a00640 1 -- 172.21.15.81:0/2900674451 wait complete. 2024-04-12T21:15:46.463 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-12T21:15:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 -- 172.21.15.81:0/2554965015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 msgr2=0x7fb94c104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 --2- 172.21.15.81:0/2554965015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 0x7fb94c104d30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb93c009900 tx=0x7fb93c02bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 -- 172.21.15.81:0/2554965015 shutdown_connections 2024-04-12T21:15:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 --2- 172.21.15.81:0/2554965015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94c107ba0 0x7fb94c109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 --2- 172.21.15.81:0/2554965015 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c105270 0x7fb94c107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.686 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 --2- 172.21.15.81:0/2554965015 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 0x7fb94c104d30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 -- 172.21.15.81:0/2554965015 >> 172.21.15.81:0/2554965015 conn(0x7fb94c0fc370 msgr2=0x7fb94c0fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 -- 172.21.15.81:0/2554965015 shutdown_connections 2024-04-12T21:15:46.687 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.690+0000 7fb94b200640 1 -- 172.21.15.81:0/2554965015 wait complete. 2024-04-12T21:15:46.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.691+0000 7fb94b200640 1 Processor -- start 2024-04-12T21:15:46.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.691+0000 7fb94b200640 1 -- start start 2024-04-12T21:15:46.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.691+0000 7fb94b200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 0x7fb94c19a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:46.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.691+0000 7fb94b200640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c105270 0x7fb94c19a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:46.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.691+0000 7fb94b200640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94c107ba0 0x7fb94c1a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:46.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb94b200640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c10dbb0 con 0x7fb94c102940 2024-04-12T21:15:46.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb94b200640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c10da30 con 0x7fb94c105270 2024-04-12T21:15:46.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb94b200640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb94c10dd30 con 0x7fb94c107ba0 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 0x7fb94c19a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 0x7fb94c19a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37652/0 (socket says 172.21.15.81:37652) 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949e00640 1 -- 172.21.15.81:0/2242948660 learned_addr learned my addr 172.21.15.81:0/2242948660 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949400640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c105270 0x7fb94c19a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb94a800640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94c107ba0 0x7fb94c1a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949e00640 1 -- 172.21.15.81:0/2242948660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94c107ba0 msgr2=0x7fb94c1a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949e00640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94c107ba0 0x7fb94c1a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949e00640 1 -- 172.21.15.81:0/2242948660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c105270 msgr2=0x7fb94c19a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949e00640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c105270 0x7fb94c19a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb949e00640 1 -- 172.21.15.81:0/2242948660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94c1a2040 con 0x7fb94c102940 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.692+0000 7fb94a800640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94c107ba0 0x7fb94c1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:15:46.690 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.693+0000 7fb949e00640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 0x7fb94c19a420 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb93c000c00 tx=0x7fb93c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:46.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.693+0000 7fb942a00640 1 -- 172.21.15.81:0/2242948660 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c043020 con 0x7fb94c102940 2024-04-12T21:15:46.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.693+0000 7fb942a00640 1 -- 172.21.15.81:0/2242948660 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb93c0054e0 con 0x7fb94c102940 2024-04-12T21:15:46.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.693+0000 7fb942a00640 1 -- 172.21.15.81:0/2242948660 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c003c40 con 0x7fb94c102940 2024-04-12T21:15:46.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.693+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94c1a22d0 con 0x7fb94c102940 2024-04-12T21:15:46.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.693+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb94c1a27d0 con 0x7fb94c102940 2024-04-12T21:15:46.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.694+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb94c1096f0 con 0x7fb94c102940 2024-04-12T21:15:46.691 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.696+0000 7fb942a00640 1 -- 172.21.15.81:0/2242948660 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb93c046020 con 0x7fb94c102940 2024-04-12T21:15:46.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.697+0000 7fb942a00640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb9300a3bb0 0x7fb9300a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:46.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.697+0000 7fb942a00640 1 -- 172.21.15.81:0/2242948660 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fb93c0f28e0 con 0x7fb94c102940 2024-04-12T21:15:46.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.698+0000 7fb949400640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb9300a3bb0 0x7fb9300a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:46.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.698+0000 7fb949400640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb9300a3bb0 0x7fb9300a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb94c19b930 tx=0x7fb938005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:46.697 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.701+0000 7fb942a00640 1 -- 172.21.15.81:0/2242948660 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb93c04d020 con 0x7fb94c102940 2024-04-12T21:15:46.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:46 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1716050327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:46.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:46 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1121175208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:46.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:46 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2900674451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:46.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:46 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1716050327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:46.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:46 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1121175208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:46.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:46 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2900674451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:46.901 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.906+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb94c062bc0 con 0x7fb94c102940 2024-04-12T21:15:46.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:46 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1716050327' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:46.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:46 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1121175208' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:46.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:46 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2900674451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:46.902 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.907+0000 7fb942a00640 1 -- 172.21.15.81:0/2242948660 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fb93c0beeb0 con 0x7fb94c102940 2024-04-12T21:15:46.903 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-04-12T21:15:46.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.909+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb9300a3bb0 msgr2=0x7fb9300a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.909+0000 7fb94b200640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb9300a3bb0 0x7fb9300a6070 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb94c19b930 tx=0x7fb938005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:46.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.909+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 msgr2=0x7fb94c19a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:46.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.909+0000 7fb94b200640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 0x7fb94c19a420 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb93c000c00 tx=0x7fb93c02c650 comp rx=0 tx=0).stop 2024-04-12T21:15:46.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.910+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 shutdown_connections 2024-04-12T21:15:46.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.910+0000 7fb94b200640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fb9300a3bb0 0x7fb9300a6070 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.910+0000 7fb94b200640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb94c107ba0 0x7fb94c1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.905 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.910+0000 7fb94b200640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb94c105270 0x7fb94c19a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.910+0000 7fb94b200640 1 --2- 172.21.15.81:0/2242948660 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fb94c102940 0x7fb94c19a420 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:46.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.910+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 >> 172.21.15.81:0/2242948660 conn(0x7fb94c0fc370 msgr2=0x7fb94c1020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:46.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.910+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 shutdown_connections 2024-04-12T21:15:46.906 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:46.910+0000 7fb94b200640 1 -- 172.21.15.81:0/2242948660 wait complete. 2024-04-12T21:15:46.924 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-12T21:15:47.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.142+0000 7f55de400640 1 -- 172.21.15.81:0/3511408831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 msgr2=0x7f55d8109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.138 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.142+0000 7f55de400640 1 --2- 172.21.15.81:0/3511408831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 0x7f55d8109c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f55c4009900 tx=0x7f55c402bc50 comp rx=0 tx=0).stop 2024-04-12T21:15:47.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.142+0000 7f55de400640 1 -- 172.21.15.81:0/3511408831 shutdown_connections 2024-04-12T21:15:47.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.142+0000 7f55de400640 1 --2- 172.21.15.81:0/3511408831 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d810a1d0 0x7f55d8110a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.142+0000 7f55de400640 1 --2- 172.21.15.81:0/3511408831 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 0x7f55d8109c90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.142+0000 7f55de400640 1 --2- 172.21.15.81:0/3511408831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d8108620 0x7f55d8108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.142+0000 7f55de400640 1 -- 172.21.15.81:0/3511408831 >> 172.21.15.81:0/3511408831 conn(0x7f55d80775c0 msgr2=0x7f55d8079a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:47.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.143+0000 7f55de400640 1 -- 172.21.15.81:0/3511408831 shutdown_connections 2024-04-12T21:15:47.139 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.143+0000 7f55de400640 1 -- 172.21.15.81:0/3511408831 wait complete. 2024-04-12T21:15:47.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.143+0000 7f55de400640 1 Processor -- start 2024-04-12T21:15:47.141 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55de400640 1 -- start start 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55de400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d8108620 0x7f55d819ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55de400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 0x7f55d819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55de400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d810a1d0 0x7f55d81a5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55de400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f55d8114290 con 0x7f55d8109810 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55de400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f55d8114110 con 0x7f55d8108620 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55de400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55d8114410 con 0x7f55d810a1d0 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55d6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 0x7f55d819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55d7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d8108620 0x7f55d819ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55d7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d8108620 0x7f55d819ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56272/0 (socket says 172.21.15.81:56272) 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55d7400640 1 -- 172.21.15.81:0/3950132992 learned_addr learned my addr 172.21.15.81:0/3950132992 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55d6a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 0x7f55d819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37670/0 (socket says 172.21.15.81:37670) 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.144+0000 7f55d7e00640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d810a1d0 0x7f55d81a5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:47.142 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55d7400640 1 -- 172.21.15.81:0/3950132992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d810a1d0 msgr2=0x7f55d81a5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55d7400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d810a1d0 0x7f55d81a5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55d7400640 1 -- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 msgr2=0x7f55d819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55d7400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 0x7f55d819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55d7400640 1 -- 172.21.15.81:0/3950132992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d81a6620 con 0x7f55d8108620 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55d6a00640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 0x7f55d819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55d7400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d8108620 0x7f55d819ea00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55c800d770 tx=0x7f55c800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55cfe00640 1 -- 172.21.15.81:0/3950132992 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8017070 con 0x7f55d8108620 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d81a6910 con 0x7f55d8108620 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55cfe00640 1 -- 172.21.15.81:0/3950132992 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c8009c30 con 0x7f55d8108620 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.145+0000 7f55cfe00640 1 -- 172.21.15.81:0/3950132992 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8003dd0 con 0x7f55d8108620 2024-04-12T21:15:47.143 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.146+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d81a6e80 con 0x7f55d8108620 2024-04-12T21:15:47.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.147+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55d810fa10 con 0x7f55d8108620 2024-04-12T21:15:47.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.149+0000 7f55cfe00640 1 -- 172.21.15.81:0/3950132992 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f55c8003f70 con 0x7f55d8108620 2024-04-12T21:15:47.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.150+0000 7f55cfe00640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f55bc0a3bb0 0x7f55bc0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:47.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.150+0000 7f55cfe00640 1 -- 172.21.15.81:0/3950132992 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f55c80d0130 con 0x7f55d8108620 2024-04-12T21:15:47.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.150+0000 7f55d6a00640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f55bc0a3bb0 0x7f55bc0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:47.146 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.151+0000 7f55d6a00640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f55bc0a3bb0 0x7f55bc0a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f55c402c7d0 tx=0x7f55c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:47.150 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.155+0000 7f55cfe00640 1 -- 172.21.15.81:0/3950132992 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f55c8009de0 con 0x7f55d8108620 2024-04-12T21:15:47.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.356+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f55d8062bc0 con 0x7f55d8108620 2024-04-12T21:15:47.352 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.357+0000 7f55cfe00640 1 -- 172.21.15.81:0/3950132992 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f55c809c0b0 con 0x7f55d8108620 2024-04-12T21:15:47.353 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f55bc0a3bb0 msgr2=0x7f55bc0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f55bc0a3bb0 0x7f55bc0a6070 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f55c402c7d0 tx=0x7f55c4005fd0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d8108620 msgr2=0x7f55d819ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d8108620 0x7f55d819ea00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f55c800d770 tx=0x7f55c800dc30 comp rx=0 tx=0).stop 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 shutdown_connections 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f55bc0a3bb0 0x7f55bc0a6070 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d810a1d0 0x7f55d81a5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f55d8109810 0x7f55d819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 --2- 172.21.15.81:0/3950132992 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f55d8108620 0x7f55d819ea00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 >> 172.21.15.81:0/3950132992 conn(0x7f55d80775c0 msgr2=0x7f55d810cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 shutdown_connections 2024-04-12T21:15:47.356 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.360+0000 7f55de400640 1 -- 172.21.15.81:0/3950132992 wait complete. 2024-04-12T21:15:47.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 -- 172.21.15.81:0/635609655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 msgr2=0x7f28e010c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 --2- 172.21.15.81:0/635609655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 0x7f28e010c660 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009860 tx=0x7f28d802bb70 comp rx=0 tx=0).stop 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 -- 172.21.15.81:0/635609655 shutdown_connections 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 --2- 172.21.15.81:0/635609655 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 0x7f28e010c660 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 --2- 172.21.15.81:0/635609655 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0105410 0x7f28e0105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 --2- 172.21.15.81:0/635609655 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0104220 0x7f28e0104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 -- 172.21.15.81:0/635609655 >> 172.21.15.81:0/635609655 conn(0x7f28e00ff7b0 msgr2=0x7f28e0101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 -- 172.21.15.81:0/635609655 shutdown_connections 2024-04-12T21:15:47.587 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.591+0000 7f28e7400640 1 -- 172.21.15.81:0/635609655 wait complete. 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.592+0000 7f28e7400640 1 Processor -- start 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.592+0000 7f28e7400640 1 -- start start 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e7400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0104220 0x7f28e019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e7400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0105410 0x7f28e019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e7400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 0x7f28e01a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e7400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010feb0 con 0x7f28e0105410 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e7400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f28e010fd30 con 0x7f28e0104220 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e7400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28e0110030 con 0x7f28e0105dd0 2024-04-12T21:15:47.589 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28d7e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0105410 0x7f28e019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 0x7f28e01a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e5400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 0x7f28e01a1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:58890/0 (socket says 172.21.15.81:58890) 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e5400640 1 -- 172.21.15.81:0/503912017 learned_addr learned my addr 172.21.15.81:0/503912017 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e4a00640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0104220 0x7f28e019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e5400640 1 -- 172.21.15.81:0/503912017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0104220 msgr2=0x7f28e019a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e5400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0104220 0x7f28e019a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e5400640 1 -- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0105410 msgr2=0x7f28e019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e5400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0105410 0x7f28e019aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28e5400640 1 -- 172.21.15.81:0/503912017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e01a2120 con 0x7f28e0105dd0 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.593+0000 7f28d7e00640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0105410 0x7f28e019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:47.590 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.594+0000 7f28e5400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 0x7f28e01a1ab0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28d802c180 tx=0x7f28d8032460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:47.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.594+0000 7f28de000640 1 -- 172.21.15.81:0/503912017 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8041020 con 0x7f28e0105dd0 2024-04-12T21:15:47.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.594+0000 7f28de000640 1 -- 172.21.15.81:0/503912017 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28d802c9f0 con 0x7f28e0105dd0 2024-04-12T21:15:47.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.594+0000 7f28de000640 1 -- 172.21.15.81:0/503912017 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28d8039040 con 0x7f28e0105dd0 2024-04-12T21:15:47.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.594+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e01a23b0 con 0x7f28e0105dd0 2024-04-12T21:15:47.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.595+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e01a2850 con 0x7f28e0105dd0 2024-04-12T21:15:47.591 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.596+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e010b540 con 0x7f28e0105dd0 2024-04-12T21:15:47.593 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.598+0000 7f28de000640 1 -- 172.21.15.81:0/503912017 <== mon.1 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f28d8044020 con 0x7f28e0105dd0 2024-04-12T21:15:47.594 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.599+0000 7f28de000640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f28c40a3c30 0x7f28c40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:47.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.600+0000 7f28de000640 1 -- 172.21.15.81:0/503912017 <== mon.1 v2:172.21.15.153:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f28d80f2500 con 0x7f28e0105dd0 2024-04-12T21:15:47.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.600+0000 7f28e4a00640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f28c40a3c30 0x7f28c40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:47.595 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.600+0000 7f28e4a00640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f28c40a3c30 0x7f28c40a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f28e0105270 tx=0x7f28d0010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:47.597 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.602+0000 7f28de000640 1 -- 172.21.15.81:0/503912017 <== mon.1 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28d804b020 con 0x7f28e0105dd0 2024-04-12T21:15:47.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:47 smithi081 ceph-mon[31453]: pgmap v395: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:47.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:47 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2242948660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:47.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:47 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3950132992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:47 smithi153 ceph-mon[36569]: pgmap v395: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:47 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2242948660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:47 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3950132992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:47 smithi104 ceph-mon[36624]: pgmap v395: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:47 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2242948660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:47 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3950132992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:47.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.845+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28e002d5e0 con 0x7f28e0105dd0 2024-04-12T21:15:47.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.846+0000 7f28de000640 1 -- 172.21.15.81:0/503912017 <== mon.1 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11815 (secure 0 0 0) 0x7f28d80beb50 con 0x7f28e0105dd0 2024-04-12T21:15:47.842 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:47.842 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T21:13:13.003153+0000","modified":"2024-04-12T21:13:37.617228+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-12T21:15:47.842 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-04-12T21:15:47.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f28c40a3c30 msgr2=0x7f28c40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f28c40a3c30 0x7f28c40a60f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f28e0105270 tx=0x7f28d0010040 comp rx=0 tx=0).stop 2024-04-12T21:15:47.845 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 msgr2=0x7f28e01a1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 0x7f28e01a1ab0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28d802c180 tx=0x7f28d8032460 comp rx=0 tx=0).stop 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 shutdown_connections 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f28c40a3c30 0x7f28c40a60f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28e0105dd0 0x7f28e01a1ab0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f28e0105410 0x7f28e019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 --2- 172.21.15.81:0/503912017 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f28e0104220 0x7f28e019a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 >> 172.21.15.81:0/503912017 conn(0x7f28e00ff7b0 msgr2=0x7f28e0108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 shutdown_connections 2024-04-12T21:15:47.846 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:47.849+0000 7f28e7400640 1 -- 172.21.15.81:0/503912017 wait complete. 2024-04-12T21:15:47.864 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:15:48.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 -- 172.21.15.81:0/2785167036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 msgr2=0x7f8840104d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 --2- 172.21.15.81:0/2785167036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 0x7f8840104d50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8830009900 tx=0x7f883002bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:48.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 -- 172.21.15.81:0/2785167036 shutdown_connections 2024-04-12T21:15:48.076 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 --2- 172.21.15.81:0/2785167036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8840105290 0x7f8840107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 --2- 172.21.15.81:0/2785167036 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8840107bc0 0x7f8840109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 --2- 172.21.15.81:0/2785167036 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 0x7f8840104d50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 -- 172.21.15.81:0/2785167036 >> 172.21.15.81:0/2785167036 conn(0x7f88400fc290 msgr2=0x7f88400fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:48.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 -- 172.21.15.81:0/2785167036 shutdown_connections 2024-04-12T21:15:48.077 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.080+0000 7f883fe00640 1 -- 172.21.15.81:0/2785167036 wait complete. 2024-04-12T21:15:48.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.081+0000 7f883fe00640 1 Processor -- start 2024-04-12T21:15:48.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883fe00640 1 -- start start 2024-04-12T21:15:48.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883fe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 0x7f8840198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.079 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883fe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8840105290 0x7f8840198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883fe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8840107bc0 0x7f884019f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883fe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f884010db30 con 0x7f8840102960 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883fe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f884010d9b0 con 0x7f8840107bc0 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883fe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f884010dcb0 con 0x7f8840105290 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 0x7f8840198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883ea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 0x7f8840198200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37698/0 (socket says 172.21.15.81:37698) 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883ea00640 1 -- 172.21.15.81:0/1179277774 learned_addr learned my addr 172.21.15.81:0/1179277774 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883f400640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8840107bc0 0x7f884019f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.082+0000 7f883e000640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8840105290 0x7f8840198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f883ea00640 1 -- 172.21.15.81:0/1179277774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8840105290 msgr2=0x7f8840198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f883ea00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8840105290 0x7f8840198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f883ea00640 1 -- 172.21.15.81:0/1179277774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8840107bc0 msgr2=0x7f884019f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f883ea00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8840107bc0 0x7f884019f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f883ea00640 1 -- 172.21.15.81:0/1179277774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884019fe20 con 0x7f8840102960 2024-04-12T21:15:48.080 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f883f400640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8840107bc0 0x7f884019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:48.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f883ea00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 0x7f8840198200 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8830000c00 tx=0x7f883002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:48.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f8837400640 1 -- 172.21.15.81:0/1179277774 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8830043020 con 0x7f8840102960 2024-04-12T21:15:48.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f8837400640 1 -- 172.21.15.81:0/1179277774 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88300054e0 con 0x7f8840102960 2024-04-12T21:15:48.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88401a00b0 con 0x7f8840102960 2024-04-12T21:15:48.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.083+0000 7f8837400640 1 -- 172.21.15.81:0/1179277774 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8830003bd0 con 0x7f8840102960 2024-04-12T21:15:48.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.084+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88401a0600 con 0x7f8840102960 2024-04-12T21:15:48.081 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.085+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88400fdd80 con 0x7f8840102960 2024-04-12T21:15:48.082 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.086+0000 7f8837400640 1 -- 172.21.15.81:0/1179277774 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8830046020 con 0x7f8840102960 2024-04-12T21:15:48.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.088+0000 7f8837400640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f88200a3bb0 0x7f88200a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.088+0000 7f8837400640 1 -- 172.21.15.81:0/1179277774 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f88300f2810 con 0x7f8840102960 2024-04-12T21:15:48.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.088+0000 7f883e000640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f88200a3bb0 0x7f88200a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.089+0000 7f883e000640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f88200a3bb0 0x7f88200a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8840199710 tx=0x7f8828009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:48.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.093+0000 7f8837400640 1 -- 172.21.15.81:0/1179277774 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f883004d020 con 0x7f8840102960 2024-04-12T21:15:48.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.296+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8840062bc0 con 0x7f8840102960 2024-04-12T21:15:48.296 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.300+0000 7f8837400640 1 -- 172.21.15.81:0/1179277774 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24821 (secure 0 0 0) 0x7f88300bee40 con 0x7f8840102960 2024-04-12T21:15:48.296 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:48.297 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":93,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:15:24.515674+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:15:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f88200a3bb0 msgr2=0x7f88200a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f88200a3bb0 0x7f88200a6070 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8840199710 tx=0x7f8828009500 comp rx=0 tx=0).stop 2024-04-12T21:15:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 msgr2=0x7f8840198200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.298 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 0x7f8840198200 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8830000c00 tx=0x7f883002c650 comp rx=0 tx=0).stop 2024-04-12T21:15:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 shutdown_connections 2024-04-12T21:15:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f88200a3bb0 0x7f88200a6070 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8840105290 0x7f8840198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8840107bc0 0x7f884019f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 --2- 172.21.15.81:0/1179277774 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f8840102960 0x7f8840198200 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.299 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 >> 172.21.15.81:0/1179277774 conn(0x7f88400fc290 msgr2=0x7f88401034c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:48.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 shutdown_connections 2024-04-12T21:15:48.300 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.303+0000 7f883fe00640 1 -- 172.21.15.81:0/1179277774 wait complete. 2024-04-12T21:15:48.318 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-12T21:08:04.339581+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-12T21:13:10.561238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-12T21:13:11.985544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-12T21:13:15.579438+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-12T21:15:48.319 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-12T21:15:48.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 -- 172.21.15.81:0/3908899708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8104220 msgr2=0x7fafe8104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 --2- 172.21.15.81:0/3908899708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8104220 0x7fafe8104640 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009900 tx=0x7fafdc02bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:48.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 -- 172.21.15.81:0/3908899708 shutdown_connections 2024-04-12T21:15:48.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 --2- 172.21.15.81:0/3908899708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8105dd0 0x7fafe810c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.541 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 --2- 172.21.15.81:0/3908899708 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafe8105410 0x7fafe8105890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 --2- 172.21.15.81:0/3908899708 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8104220 0x7fafe8104640 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 -- 172.21.15.81:0/3908899708 >> 172.21.15.81:0/3908899708 conn(0x7fafe80ff7b0 msgr2=0x7fafe8101bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:48.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 -- 172.21.15.81:0/3908899708 shutdown_connections 2024-04-12T21:15:48.542 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.545+0000 7faff0400640 1 -- 172.21.15.81:0/3908899708 wait complete. 2024-04-12T21:15:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.546+0000 7faff0400640 1 Processor -- start 2024-04-12T21:15:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.546+0000 7faff0400640 1 -- start start 2024-04-12T21:15:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7faff0400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafe8104220 0x7fafe819a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7faff0400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8105410 0x7fafe819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7faff0400640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8105dd0 0x7fafe81a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7faff0400640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fafe810ff00 con 0x7fafe8104220 2024-04-12T21:15:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7faff0400640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fafe810fd80 con 0x7fafe8105410 2024-04-12T21:15:48.544 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7faff0400640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fafe8110080 con 0x7fafe8105dd0 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7fafeda00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafe8104220 0x7fafe819a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7fafed000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8105410 0x7fafe819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7fafed000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8105410 0x7fafe819aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56310/0 (socket says 172.21.15.81:56310) 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7fafed000640 1 -- 172.21.15.81:0/1675468158 learned_addr learned my addr 172.21.15.81:0/1675468158 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.547+0000 7fafee400640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8105dd0 0x7fafe81a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafed000640 1 -- 172.21.15.81:0/1675468158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8105dd0 msgr2=0x7fafe81a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafed000640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8105dd0 0x7fafe81a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafed000640 1 -- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafe8104220 msgr2=0x7fafe819a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafed000640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafe8104220 0x7fafe819a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafed000640 1 -- 172.21.15.81:0/1675468158 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe81a2140 con 0x7fafe8105410 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafee400640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8105dd0 0x7fafe81a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafeda00640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafe8104220 0x7fafe819a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:48.545 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafed000640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8105410 0x7fafe819aa60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fafd800e9f0 tx=0x7fafd800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafe6000640 1 -- 172.21.15.81:0/1675468158 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd800cd80 con 0x7fafe8105410 2024-04-12T21:15:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe81a2430 con 0x7fafe8105410 2024-04-12T21:15:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafe6000640 1 -- 172.21.15.81:0/1675468158 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd8010910 con 0x7fafe8105410 2024-04-12T21:15:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7fafe6000640 1 -- 172.21.15.81:0/1675468158 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd80188f0 con 0x7fafe8105410 2024-04-12T21:15:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.548+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe81a29d0 con 0x7fafe8105410 2024-04-12T21:15:48.546 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.550+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe810b540 con 0x7fafe8105410 2024-04-12T21:15:48.547 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.551+0000 7fafe6000640 1 -- 172.21.15.81:0/1675468158 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fafd8010ac0 con 0x7fafe8105410 2024-04-12T21:15:48.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.553+0000 7fafe6000640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fafd00a3c30 0x7fafd00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.553+0000 7fafe6000640 1 -- 172.21.15.81:0/1675468158 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fafd80d0b20 con 0x7fafe8105410 2024-04-12T21:15:48.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.553+0000 7fafeda00640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fafd00a3c30 0x7fafd00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.554+0000 7fafeda00640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fafd00a3c30 0x7fafd00a60f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fafdc02c170 tx=0x7fafdc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:48.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.557+0000 7fafe6000640 1 -- 172.21.15.81:0/1675468158 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafd809caa0 con 0x7fafe8105410 2024-04-12T21:15:48.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:48 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/503912017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:48.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:48 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1179277774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:48.755 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.760+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fafe8062bc0 con 0x7fafe8105410 2024-04-12T21:15:48.756 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.760+0000 7fafe6000640 1 -- 172.21.15.81:0/1675468158 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fafd80a0da0 con 0x7fafe8105410 2024-04-12T21:15:48.756 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 1 2024-04-12T21:15:48.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.763+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fafd00a3c30 msgr2=0x7fafd00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.759 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.763+0000 7faff0400640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fafd00a3c30 0x7fafd00a60f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fafdc02c170 tx=0x7fafdc02df90 comp rx=0 tx=0).stop 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.763+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8105410 msgr2=0x7fafe819aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.763+0000 7faff0400640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8105410 0x7fafe819aa60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fafd800e9f0 tx=0x7fafd800eeb0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.764+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 shutdown_connections 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.764+0000 7faff0400640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fafd00a3c30 0x7fafd00a60f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.764+0000 7faff0400640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fafe8105dd0 0x7fafe81a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.764+0000 7faff0400640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafe8105410 0x7fafe819aa60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.764+0000 7faff0400640 1 --2- 172.21.15.81:0/1675468158 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fafe8104220 0x7fafe819a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.764+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 >> 172.21.15.81:0/1675468158 conn(0x7fafe80ff7b0 msgr2=0x7fafe8108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.764+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 shutdown_connections 2024-04-12T21:15:48.760 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.764+0000 7faff0400640 1 -- 172.21.15.81:0/1675468158 wait complete. 2024-04-12T21:15:48.778 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-12T21:15:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:48 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/503912017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:48 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1179277774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:48.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:48 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/503912017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:48.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:48 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1179277774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:48.991 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.995+0000 7fbdffe00640 1 -- 172.21.15.81:0/3660426686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 msgr2=0x7fbe00104d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.995+0000 7fbdffe00640 1 --2- 172.21.15.81:0/3660426686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 0x7fbe00104d50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009900 tx=0x7fbdf002bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:48.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.995+0000 7fbdffe00640 1 -- 172.21.15.81:0/3660426686 shutdown_connections 2024-04-12T21:15:48.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.995+0000 7fbdffe00640 1 --2- 172.21.15.81:0/3660426686 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe00105290 0x7fbe00107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.995+0000 7fbdffe00640 1 --2- 172.21.15.81:0/3660426686 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe00107bc0 0x7fbe0010a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.995+0000 7fbdffe00640 1 --2- 172.21.15.81:0/3660426686 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 0x7fbe00104d50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.995+0000 7fbdffe00640 1 -- 172.21.15.81:0/3660426686 >> 172.21.15.81:0/3660426686 conn(0x7fbe000fc370 msgr2=0x7fbe000fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:48.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.996+0000 7fbdffe00640 1 -- 172.21.15.81:0/3660426686 shutdown_connections 2024-04-12T21:15:48.992 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.996+0000 7fbdffe00640 1 -- 172.21.15.81:0/3660426686 wait complete. 2024-04-12T21:15:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.996+0000 7fbdffe00640 1 Processor -- start 2024-04-12T21:15:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdffe00640 1 -- start start 2024-04-12T21:15:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdffe00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 0x7fbe0019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdffe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe00105290 0x7fbe0019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdffe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe00107bc0 0x7fbe001a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdffe00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010dbc0 con 0x7fbe00102960 2024-04-12T21:15:48.995 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdffe00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010da40 con 0x7fbe00105290 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdffe00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0010dd40 con 0x7fbe00107bc0 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdfea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 0x7fbe0019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdfea00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 0x7fbe0019a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37738/0 (socket says 172.21.15.81:37738) 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.997+0000 7fbdfea00640 1 -- 172.21.15.81:0/2402890001 learned_addr learned my addr 172.21.15.81:0/2402890001 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdf7e00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe00105290 0x7fbe0019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdff400640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe00107bc0 0x7fbe001a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdfea00640 1 -- 172.21.15.81:0/2402890001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe00107bc0 msgr2=0x7fbe001a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdfea00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe00107bc0 0x7fbe001a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdfea00640 1 -- 172.21.15.81:0/2402890001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe00105290 msgr2=0x7fbe0019a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdfea00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe00105290 0x7fbe0019a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdfea00640 1 -- 172.21.15.81:0/2402890001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe001a2010 con 0x7fbe00102960 2024-04-12T21:15:48.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdf7e00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe00105290 0x7fbe0019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:15:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdfea00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 0x7fbe0019a3f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0005ab0 tx=0x7fbdf002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdf7400640 1 -- 172.21.15.81:0/2402890001 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0043020 con 0x7fbe00102960 2024-04-12T21:15:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe001a22a0 con 0x7fbe00102960 2024-04-12T21:15:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdf7400640 1 -- 172.21.15.81:0/2402890001 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdf00054e0 con 0x7fbe00102960 2024-04-12T21:15:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.998+0000 7fbdf7400640 1 -- 172.21.15.81:0/2402890001 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0003ce0 con 0x7fbe00102960 2024-04-12T21:15:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:48.999+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe001a2840 con 0x7fbe00102960 2024-04-12T21:15:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.000+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe00109710 con 0x7fbe00102960 2024-04-12T21:15:48.997 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.002+0000 7fbdf7400640 1 -- 172.21.15.81:0/2402890001 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbdf0046020 con 0x7fbe00102960 2024-04-12T21:15:48.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.003+0000 7fbdf7400640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbde40a3c30 0x7fbde40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:48.998 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.003+0000 7fbdf7400640 1 -- 172.21.15.81:0/2402890001 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fbdf00f29c0 con 0x7fbe00102960 2024-04-12T21:15:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.003+0000 7fbdf7e00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbde40a3c30 0x7fbde40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:48.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.004+0000 7fbdf7e00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbde40a3c30 0x7fbde40a60f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbde80058c0 tx=0x7fbde8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:49.002 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.007+0000 7fbdf7400640 1 -- 172.21.15.81:0/2402890001 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbdf004d020 con 0x7fbe00102960 2024-04-12T21:15:49.205 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.209+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbe0002d800 con 0x7fbe00102960 2024-04-12T21:15:49.206 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.210+0000 7fbdf7400640 1 -- 172.21.15.81:0/2402890001 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fbdf00a7e20 con 0x7fbe00102960 2024-04-12T21:15:49.206 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:15:49.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbde40a3c30 msgr2=0x7fbde40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbde40a3c30 0x7fbde40a60f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbde80058c0 tx=0x7fbde8005850 comp rx=0 tx=0).stop 2024-04-12T21:15:49.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 msgr2=0x7fbe0019a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.208 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 0x7fbe0019a3f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0005ab0 tx=0x7fbdf002c6c0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 shutdown_connections 2024-04-12T21:15:49.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fbde40a3c30 0x7fbde40a60f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe00107bc0 0x7fbe001a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fbe00105290 0x7fbe0019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 --2- 172.21.15.81:0/2402890001 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fbe00102960 0x7fbe0019a3f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 >> 172.21.15.81:0/2402890001 conn(0x7fbe000fc370 msgr2=0x7fbe00101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:49.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 shutdown_connections 2024-04-12T21:15:49.209 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.213+0000 7fbdffe00640 1 -- 172.21.15.81:0/2402890001 wait complete. 2024-04-12T21:15:49.227 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-12T21:15:49.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 -- 172.21.15.81:0/2082661066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 msgr2=0x7f449410c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 --2- 172.21.15.81:0/2082661066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 0x7f449410c680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f448c009ca0 tx=0x7f448c02c0f0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 -- 172.21.15.81:0/2082661066 shutdown_connections 2024-04-12T21:15:49.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 --2- 172.21.15.81:0/2082661066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4494105430 0x7f44941058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 --2- 172.21.15.81:0/2082661066 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 0x7f449410c680 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 --2- 172.21.15.81:0/2082661066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4494104240 0x7f4494104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.439 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 -- 172.21.15.81:0/2082661066 >> 172.21.15.81:0/2082661066 conn(0x7f44940ff7b0 msgr2=0x7f4494101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:49.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 -- 172.21.15.81:0/2082661066 shutdown_connections 2024-04-12T21:15:49.440 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.443+0000 7f4499a00640 1 -- 172.21.15.81:0/2082661066 wait complete. 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.444+0000 7f4499a00640 1 Processor -- start 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.444+0000 7f4499a00640 1 -- start start 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.444+0000 7f4499a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4494104240 0x7f449419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.444+0000 7f4499a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4494105430 0x7f449419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.444+0000 7f4499a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 0x7f44941a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.444+0000 7f4499a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f449410fed0 con 0x7f4494105df0 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.444+0000 7f4499a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f449410fd50 con 0x7f4494105430 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.444+0000 7f4499a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4494110050 con 0x7f4494104240 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 0x7f44941a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 0x7f44941a1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37768/0 (socket says 172.21.15.81:37768) 2024-04-12T21:15:49.442 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 -- 172.21.15.81:0/1866877914 learned_addr learned my addr 172.21.15.81:0/1866877914 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4492a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4494104240 0x7f449419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f448be00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4494105430 0x7f449419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 -- 172.21.15.81:0/1866877914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4494104240 msgr2=0x7f449419a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4494104240 0x7f449419a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 -- 172.21.15.81:0/1866877914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4494105430 msgr2=0x7f449419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4494105430 0x7f449419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 -- 172.21.15.81:0/1866877914 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44941a2140 con 0x7f4494105df0 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.445+0000 7f4493400640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 0x7f44941a1ad0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f448c037040 tx=0x7f448c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.446+0000 7f448b400640 1 -- 172.21.15.81:0/1866877914 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c041020 con 0x7f4494105df0 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.446+0000 7f448b400640 1 -- 172.21.15.81:0/1866877914 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f448c034cd0 con 0x7f4494105df0 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.446+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44941a23d0 con 0x7f4494105df0 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.446+0000 7f448b400640 1 -- 172.21.15.81:0/1866877914 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c039040 con 0x7f4494105df0 2024-04-12T21:15:49.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.446+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44941a28c0 con 0x7f4494105df0 2024-04-12T21:15:49.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.447+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f449410b560 con 0x7f4494105df0 2024-04-12T21:15:49.444 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.449+0000 7f448b400640 1 -- 172.21.15.81:0/1866877914 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f448c044020 con 0x7f4494105df0 2024-04-12T21:15:49.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.450+0000 7f448b400640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44780a3bb0 0x7f44780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:49.445 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.450+0000 7f448b400640 1 -- 172.21.15.81:0/1866877914 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f448c0f25d0 con 0x7f4494105df0 2024-04-12T21:15:49.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.450+0000 7f4492a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44780a3bb0 0x7f44780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:49.446 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.451+0000 7f4492a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44780a3bb0 0x7f44780a6070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f44840095f0 tx=0x7f4484005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:49.450 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.455+0000 7f448b400640 1 -- 172.21.15.81:0/1866877914 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f448c04b020 con 0x7f4494105df0 2024-04-12T21:15:49.644 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.648+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4494062bc0 con 0x7f4494105df0 2024-04-12T21:15:49.645 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.649+0000 7f448b400640 1 -- 172.21.15.81:0/1866877914 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f448c0bec90 con 0x7f4494105df0 2024-04-12T21:15:49.645 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 64 2024-04-12T21:15:49.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44780a3bb0 msgr2=0x7f44780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44780a3bb0 0x7f44780a6070 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f44840095f0 tx=0x7f4484005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:49.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 msgr2=0x7f44941a1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.647 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 0x7f44941a1ad0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f448c037040 tx=0x7f448c032f40 comp rx=0 tx=0).stop 2024-04-12T21:15:49.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 shutdown_connections 2024-04-12T21:15:49.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f44780a3bb0 0x7f44780a6070 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f4494105df0 0x7f44941a1ad0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f4494105430 0x7f449419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 --2- 172.21.15.81:0/1866877914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4494104240 0x7f449419a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 >> 172.21.15.81:0/1866877914 conn(0x7f44940ff7b0 msgr2=0x7f4494108730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:49.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 shutdown_connections 2024-04-12T21:15:49.648 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.652+0000 7f4499a00640 1 -- 172.21.15.81:0/1866877914 wait complete. 2024-04-12T21:15:49.667 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-12T21:15:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:49 smithi153 ceph-mon[36569]: pgmap v396: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:49 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1675468158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:49 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/2402890001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:49 smithi104 ceph-mon[36624]: pgmap v396: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:49 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1675468158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:49 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/2402890001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 -- 172.21.15.81:0/3841895602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c074a20 msgr2=0x7f0f0c074ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 --2- 172.21.15.81:0/3841895602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c074a20 0x7f0f0c074ea0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009900 tx=0x7f0ef802bc30 comp rx=0 tx=0).stop 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 -- 172.21.15.81:0/3841895602 shutdown_connections 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 --2- 172.21.15.81:0/3841895602 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c109d90 0x7f0f0c10c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 --2- 172.21.15.81:0/3841895602 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c074a20 0x7f0f0c074ea0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 --2- 172.21.15.81:0/3841895602 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f0c076000 0x7f0f0c074450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 -- 172.21.15.81:0/3841895602 >> 172.21.15.81:0/3841895602 conn(0x7f0f0c0fd460 msgr2=0x7f0f0c0ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 -- 172.21.15.81:0/3841895602 shutdown_connections 2024-04-12T21:15:49.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.886+0000 7f0f11a00640 1 -- 172.21.15.81:0/3841895602 wait complete. 2024-04-12T21:15:49.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.887+0000 7f0f11a00640 1 Processor -- start 2024-04-12T21:15:49.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.887+0000 7f0f11a00640 1 -- start start 2024-04-12T21:15:49.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.887+0000 7f0f11a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f0c074a20 0x7f0f0c19a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.887+0000 7f0f11a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c076000 0x7f0f0c19a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f11a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c109d90 0x7f0f0c1a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f11a00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c10fd50 con 0x7f0f0c076000 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f11a00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c10fbd0 con 0x7f0f0c074a20 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f11a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0c10fed0 con 0x7f0f0c109d90 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c076000 0x7f0f0c19a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0b400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c076000 0x7f0f0c19a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37776/0 (socket says 172.21.15.81:37776) 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0b400640 1 -- 172.21.15.81:0/213043244 learned_addr learned my addr 172.21.15.81:0/213043244 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f11000640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c109d90 0x7f0f0c1a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0be00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f0c074a20 0x7f0f0c19a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0b400640 1 -- 172.21.15.81:0/213043244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c109d90 msgr2=0x7f0f0c1a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0b400640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c109d90 0x7f0f0c1a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0b400640 1 -- 172.21.15.81:0/213043244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f0c074a20 msgr2=0x7f0f0c19a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:49.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0b400640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f0c074a20 0x7f0f0c19a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:49.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.888+0000 7f0f0b400640 1 -- 172.21.15.81:0/213043244 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0c1a1fe0 con 0x7f0f0c076000 2024-04-12T21:15:49.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.889+0000 7f0f0b400640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c076000 0x7f0f0c19a900 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009230 tx=0x7f0ef8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:49.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.889+0000 7f0f08c00640 1 -- 172.21.15.81:0/213043244 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8041070 con 0x7f0f0c076000 2024-04-12T21:15:49.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.889+0000 7f0f08c00640 1 -- 172.21.15.81:0/213043244 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ef80044d0 con 0x7f0f0c076000 2024-04-12T21:15:49.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.889+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0c1a2270 con 0x7f0f0c076000 2024-04-12T21:15:49.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.889+0000 7f0f08c00640 1 -- 172.21.15.81:0/213043244 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8034440 con 0x7f0f0c076000 2024-04-12T21:15:49.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.889+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0c1a2760 con 0x7f0f0c076000 2024-04-12T21:15:49.886 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.890+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f0c108ac0 con 0x7f0f0c076000 2024-04-12T21:15:49.887 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.892+0000 7f0f08c00640 1 -- 172.21.15.81:0/213043244 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0ef8044070 con 0x7f0f0c076000 2024-04-12T21:15:49.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.894+0000 7f0f08c00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0ef00a3bb0 0x7f0ef00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:49.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.895+0000 7f0f08c00640 1 -- 172.21.15.81:0/213043244 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f0ef80f2350 con 0x7f0f0c076000 2024-04-12T21:15:49.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.895+0000 7f0f0be00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0ef00a3bb0 0x7f0ef00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:49.890 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.895+0000 7f0f0be00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0ef00a3bb0 0x7f0ef00a6070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009bd0 tx=0x7f0efc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:49.894 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:49.899+0000 7f0f08c00640 1 -- 172.21.15.81:0/213043244 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ef80be330 con 0x7f0f0c076000 2024-04-12T21:15:49.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:49 smithi081 ceph-mon[31453]: pgmap v396: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:49.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:49 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1675468158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-12T21:15:49.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:49 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/2402890001' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-12T21:15:50.083 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.087+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0f0c062bc0 con 0x7f0f0c076000 2024-04-12T21:15:50.084 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.089+0000 7f0f08c00640 1 -- 172.21.15.81:0/213043244 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f0ef80c2630 con 0x7f0f0c076000 2024-04-12T21:15:50.084 INFO:teuthology.orchestra.run.smithi081.stdout:pg_num: 32 2024-04-12T21:15:50.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0ef00a3bb0 msgr2=0x7f0ef00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0ef00a3bb0 0x7f0ef00a6070 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009bd0 tx=0x7f0efc005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:50.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c076000 msgr2=0x7f0f0c19a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c076000 0x7f0f0c19a900 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009230 tx=0x7f0ef8005d50 comp rx=0 tx=0).stop 2024-04-12T21:15:50.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 shutdown_connections 2024-04-12T21:15:50.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f0ef00a3bb0 0x7f0ef00a6070 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.087 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c109d90 0x7f0f0c1a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f0f0c076000 0x7f0f0c19a900 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 --2- 172.21.15.81:0/213043244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0f0c074a20 0x7f0f0c19a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 >> 172.21.15.81:0/213043244 conn(0x7f0f0c0fd460 msgr2=0x7f0f0c105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:50.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 shutdown_connections 2024-04-12T21:15:50.088 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.091+0000 7f0f11a00640 1 -- 172.21.15.81:0/213043244 wait complete. 2024-04-12T21:15:50.106 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-12T21:15:50.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.313+0000 7fa927e00640 1 -- 172.21.15.81:0/3932343073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 msgr2=0x7fa928108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.313+0000 7fa927e00640 1 --2- 172.21.15.81:0/3932343073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 0x7fa928108860 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa918009860 tx=0x7fa91802bb70 comp rx=0 tx=0).stop 2024-04-12T21:15:50.309 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.313+0000 7fa927e00640 1 -- 172.21.15.81:0/3932343073 shutdown_connections 2024-04-12T21:15:50.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.313+0000 7fa927e00640 1 --2- 172.21.15.81:0/3932343073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928109ff0 0x7fa928110880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.313+0000 7fa927e00640 1 --2- 172.21.15.81:0/3932343073 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa928109630 0x7fa928109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.313+0000 7fa927e00640 1 --2- 172.21.15.81:0/3932343073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 0x7fa928108860 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.313+0000 7fa927e00640 1 -- 172.21.15.81:0/3932343073 >> 172.21.15.81:0/3932343073 conn(0x7fa928077400 msgr2=0x7fa928079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:50.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.314+0000 7fa927e00640 1 -- 172.21.15.81:0/3932343073 shutdown_connections 2024-04-12T21:15:50.310 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.314+0000 7fa927e00640 1 -- 172.21.15.81:0/3932343073 wait complete. 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.315+0000 7fa927e00640 1 Processor -- start 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.315+0000 7fa927e00640 1 -- start start 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.315+0000 7fa927e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 0x7fa92819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.315+0000 7fa927e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa928109630 0x7fa92819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.315+0000 7fa927e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928109ff0 0x7fa9281a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.315+0000 7fa927e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fa9281140a0 con 0x7fa928109630 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.315+0000 7fa927e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fa928113f20 con 0x7fa928108440 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.315+0000 7fa927e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa928114220 con 0x7fa928109ff0 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa926a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 0x7fa92819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:50.312 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa926a00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 0x7fa92819e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56396/0 (socket says 172.21.15.81:56396) 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa926a00640 1 -- 172.21.15.81:0/3507164594 learned_addr learned my addr 172.21.15.81:0/3507164594 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa91fe00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa928109630 0x7fa92819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa927400640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928109ff0 0x7fa9281a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa926a00640 1 -- 172.21.15.81:0/3507164594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928109ff0 msgr2=0x7fa9281a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa926a00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928109ff0 0x7fa9281a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa926a00640 1 -- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa928109630 msgr2=0x7fa92819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa926a00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa928109630 0x7fa92819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.316+0000 7fa926a00640 1 -- 172.21.15.81:0/3507164594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9281a63d0 con 0x7fa928108440 2024-04-12T21:15:50.313 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.317+0000 7fa926a00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 0x7fa92819e7b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9180349e0 tx=0x7fa918032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:50.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.317+0000 7fa91f400640 1 -- 172.21.15.81:0/3507164594 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918041020 con 0x7fa928108440 2024-04-12T21:15:50.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.317+0000 7fa91f400640 1 -- 172.21.15.81:0/3507164594 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa91802c9f0 con 0x7fa928108440 2024-04-12T21:15:50.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.317+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9281a6660 con 0x7fa928108440 2024-04-12T21:15:50.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.317+0000 7fa91f400640 1 -- 172.21.15.81:0/3507164594 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa918039040 con 0x7fa928108440 2024-04-12T21:15:50.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.317+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9281a6b50 con 0x7fa928108440 2024-04-12T21:15:50.314 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.318+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa92810f7e0 con 0x7fa928108440 2024-04-12T21:15:50.315 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.320+0000 7fa91f400640 1 -- 172.21.15.81:0/3507164594 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fa918044020 con 0x7fa928108440 2024-04-12T21:15:50.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.321+0000 7fa91f400640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa90c0a3bb0 0x7fa90c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:50.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.322+0000 7fa91f400640 1 -- 172.21.15.81:0/3507164594 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fa9180f2390 con 0x7fa928108440 2024-04-12T21:15:50.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.322+0000 7fa91fe00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa90c0a3bb0 0x7fa90c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:50.317 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.322+0000 7fa91fe00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa90c0a3bb0 0x7fa90c0a6070 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa92819fcc0 tx=0x7fa910005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:50.319 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.324+0000 7fa91f400640 1 -- 172.21.15.81:0/3507164594 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa91804b020 con 0x7fa928108440 2024-04-12T21:15:50.551 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.556+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa92802d5e0 con 0x7fa928108440 2024-04-12T21:15:50.552 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.557+0000 7fa91f400640 1 -- 172.21.15.81:0/3507164594 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+11815 (secure 0 0 0) 0x7fa9180be9c0 con 0x7fa928108440 2024-04-12T21:15:50.553 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:50.554 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6835/1703364372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6834","nonce":1703364372},{"type":"v1","addr":"172.21.15.81:6835","nonce":1703364372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6837/809891880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6836","nonce":809891880},{"type":"v1","addr":"172.21.15.81:6837","nonce":809891880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6839/1348294553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6838","nonce":1348294553},{"type":"v1","addr":"172.21.15.81:6839","nonce":1348294553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.81:6841/2944644175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6840","nonce":2944644175},{"type":"v1","addr":"172.21.15.81:6841","nonce":2944644175}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24454,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6833/2327114952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6832","nonce":2327114952},{"type":"v1","addr":"172.21.15.153:6833","nonce":2327114952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6833/1562734606","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6832","nonce":1562734606},{"type":"v1","addr":"172.21.15.104:6833","nonce":1562734606}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6835/3576497721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6834","nonce":3576497721},{"type":"v1","addr":"172.21.15.104:6835","nonce":3576497721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6835/1712020888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6834","nonce":1712020888},{"type":"v1","addr":"172.21.15.153:6835","nonce":1712020888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24529,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.153:6837/455852644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6836","nonce":455852644},{"type":"v1","addr":"172.21.15.153:6837","nonce":455852644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24536,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.104:6837/435649651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6836","nonce":435649651},{"type":"v1","addr":"172.21.15.104:6837","nonce":435649651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-12T21:13:13.003153+0000","modified":"2024-04-12T21:13:37.617228+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24547":{"gid":24547,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.153:6839/2426851956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6838","nonce":2426851956},{"type":"v1","addr":"172.21.15.153:6839","nonce":2426851956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24593":{"gid":24593,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.104:6839/1944172329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6838","nonce":1944172329},{"type":"v1","addr":"172.21.15.104:6839","nonce":1944172329}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24593,"qdb_cluster":[24593]},"id":1}]} 2024-04-12T21:15:50.554 INFO:teuthology.orchestra.run.smithi081.stderr:dumped fsmap epoch 20 2024-04-12T21:15:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa90c0a3bb0 msgr2=0x7fa90c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa90c0a3bb0 0x7fa90c0a6070 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa92819fcc0 tx=0x7fa910005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:50.555 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 msgr2=0x7fa92819e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 0x7fa92819e7b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9180349e0 tx=0x7fa918032430 comp rx=0 tx=0).stop 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 shutdown_connections 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fa90c0a3bb0 0x7fa90c0a6070 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa928109ff0 0x7fa9281a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fa928109630 0x7fa92819ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 --2- 172.21.15.81:0/3507164594 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa928108440 0x7fa92819e7b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 >> 172.21.15.81:0/3507164594 conn(0x7fa928077400 msgr2=0x7fa92810c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 shutdown_connections 2024-04-12T21:15:50.556 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.560+0000 7fa927e00640 1 -- 172.21.15.81:0/3507164594 wait complete. 2024-04-12T21:15:50.573 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-12T21:15:50.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.783+0000 7fd495c00640 1 -- 172.21.15.81:0/30531108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd490109ff0 msgr2=0x7fd490110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.779 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.783+0000 7fd495c00640 1 --2- 172.21.15.81:0/30531108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd490109ff0 0x7fd490110880 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd488009ca0 tx=0x7fd48802c0f0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.784+0000 7fd495c00640 1 -- 172.21.15.81:0/30531108 shutdown_connections 2024-04-12T21:15:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.784+0000 7fd495c00640 1 --2- 172.21.15.81:0/30531108 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd490109ff0 0x7fd490110880 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.784+0000 7fd495c00640 1 --2- 172.21.15.81:0/30531108 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd490109630 0x7fd490109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.784+0000 7fd495c00640 1 --2- 172.21.15.81:0/30531108 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd490108440 0x7fd490108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.784+0000 7fd495c00640 1 -- 172.21.15.81:0/30531108 >> 172.21.15.81:0/30531108 conn(0x7fd490077400 msgr2=0x7fd490079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.784+0000 7fd495c00640 1 -- 172.21.15.81:0/30531108 shutdown_connections 2024-04-12T21:15:50.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.784+0000 7fd495c00640 1 -- 172.21.15.81:0/30531108 wait complete. 2024-04-12T21:15:50.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495c00640 1 Processor -- start 2024-04-12T21:15:50.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495c00640 1 -- start start 2024-04-12T21:15:50.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495c00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd490108440 0x7fd49019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:50.782 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495c00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd490109630 0x7fd49019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495c00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd490109ff0 0x7fd4901a5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495c00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fd4901140d0 con 0x7fd490109ff0 2024-04-12T21:15:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495c00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fd490113f50 con 0x7fd490109630 2024-04-12T21:15:50.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495c00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd490114250 con 0x7fd490108440 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd487400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd490109630 0x7fd49019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd487400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd490109630 0x7fd49019ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56410/0 (socket says 172.21.15.81:56410) 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd495200640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd490109ff0 0x7fd4901a5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.785+0000 7fd487400640 1 -- 172.21.15.81:0/255200188 learned_addr learned my addr 172.21.15.81:0/255200188 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd48fe00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd490108440 0x7fd49019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd487400640 1 -- 172.21.15.81:0/255200188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd490108440 msgr2=0x7fd49019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd487400640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd490108440 0x7fd49019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd487400640 1 -- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd490109ff0 msgr2=0x7fd4901a5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd487400640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd490109ff0 0x7fd4901a5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd487400640 1 -- 172.21.15.81:0/255200188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4901a62f0 con 0x7fd490109630 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd487400640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd490109630 0x7fd49019ec10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd47800ea60 tx=0x7fd47800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd48d600640 1 -- 172.21.15.81:0/255200188 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47800cde0 con 0x7fd490109630 2024-04-12T21:15:50.786 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd48d600640 1 -- 172.21.15.81:0/255200188 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4780035f0 con 0x7fd490109630 2024-04-12T21:15:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4901a65e0 con 0x7fd490109630 2024-04-12T21:15:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.786+0000 7fd48d600640 1 -- 172.21.15.81:0/255200188 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd478010670 con 0x7fd490109630 2024-04-12T21:15:50.787 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.787+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4901a6b30 con 0x7fd490109630 2024-04-12T21:15:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.788+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd49010f760 con 0x7fd490109630 2024-04-12T21:15:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.790+0000 7fd48d600640 1 -- 172.21.15.81:0/255200188 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd4780037a0 con 0x7fd490109630 2024-04-12T21:15:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.791+0000 7fd48d600640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd4740a3bb0 0x7fd4740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.791+0000 7fd48d600640 1 -- 172.21.15.81:0/255200188 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fd4780d0cb0 con 0x7fd490109630 2024-04-12T21:15:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.791+0000 7fd48fe00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd4740a3bb0 0x7fd4740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.792+0000 7fd48fe00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd4740a3bb0 0x7fd4740a6070 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd490109490 tx=0x7fd480005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:50.788 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.793+0000 7fd48d600640 1 -- 172.21.15.81:0/255200188 <== mon.2 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd47809d280 con 0x7fd490109630 2024-04-12T21:15:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:50 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1866877914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:50 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/213043244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:50 smithi153 ceph-mon[36569]: pgmap v397: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:50.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:50 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1866877914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:50.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:50 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/213043244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:50.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:50 smithi104 ceph-mon[36624]: pgmap v397: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:50.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:50 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1866877914' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-12T21:15:50.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:50 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/213043244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-12T21:15:50.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:50 smithi081 ceph-mon[31453]: pgmap v397: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:50.994 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:50.998+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd49002d830 con 0x7fd490109630 2024-04-12T21:15:50.996 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.001+0000 7fd48d600640 1 -- 172.21.15.81:0/255200188 <== mon.2 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24821 (secure 0 0 0) 0x7fd47809cb80 con 0x7fd490109630 2024-04-12T21:15:50.997 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-04-12T21:15:50.997 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":93,"fsid":"11b8cc88-f910-11ee-bc8d-c7b262605968","created":"2024-04-12T21:04:15.716001+0000","modified":"2024-04-12T21:15:24.515674+0000","last_up_change":"2024-04-12T21:12:15.138788+0000","last_in_change":"2024-04-12T21:11:55.198537+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-12T21:08:04.339581+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-12T21:13:10.561238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-12T21:13:11.985544+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-12T21:13:15.579438+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"5891e774-9745-4511-a67d-b0d5b30f48ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6803","nonce":4059792347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6805","nonce":4059792347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6809","nonce":4059792347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":4059792347},{"type":"v1","addr":"172.21.15.81:6807","nonce":4059792347}]},"public_addr":"172.21.15.81:6803/4059792347","cluster_addr":"172.21.15.81:6805/4059792347","heartbeat_back_addr":"172.21.15.81:6809/4059792347","heartbeat_front_addr":"172.21.15.81:6807/4059792347","state":["exists","up"]},{"osd":1,"uuid":"60afc2bd-b9e9-4489-9cbc-a756b944fdc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6801","nonce":3106352403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6803","nonce":3106352403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6807","nonce":3106352403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3106352403},{"type":"v1","addr":"172.21.15.104:6805","nonce":3106352403}]},"public_addr":"172.21.15.104:6801/3106352403","cluster_addr":"172.21.15.104:6803/3106352403","heartbeat_back_addr":"172.21.15.104:6807/3106352403","heartbeat_front_addr":"172.21.15.104:6805/3106352403","state":["exists","up"]},{"osd":2,"uuid":"b4a8e96d-ddca-4406-977f-543283eadef9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6801","nonce":1512806537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6803","nonce":1512806537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6807","nonce":1512806537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1512806537},{"type":"v1","addr":"172.21.15.153:6805","nonce":1512806537}]},"public_addr":"172.21.15.153:6801/1512806537","cluster_addr":"172.21.15.153:6803/1512806537","heartbeat_back_addr":"172.21.15.153:6807/1512806537","heartbeat_front_addr":"172.21.15.153:6805/1512806537","state":["exists","up"]},{"osd":3,"uuid":"5eb8c1b5-9a15-47f4-bb29-508ec6db0fdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6811","nonce":4225327027}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6813","nonce":4225327027}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6817","nonce":4225327027}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6814","nonce":4225327027},{"type":"v1","addr":"172.21.15.81:6815","nonce":4225327027}]},"public_addr":"172.21.15.81:6811/4225327027","cluster_addr":"172.21.15.81:6813/4225327027","heartbeat_back_addr":"172.21.15.81:6817/4225327027","heartbeat_front_addr":"172.21.15.81:6815/4225327027","state":["exists","up"]},{"osd":4,"uuid":"2eb029c7-091e-43bb-8085-4f736bcf6a62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6809","nonce":742573974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6811","nonce":742573974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6815","nonce":742573974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":742573974},{"type":"v1","addr":"172.21.15.104:6813","nonce":742573974}]},"public_addr":"172.21.15.104:6809/742573974","cluster_addr":"172.21.15.104:6811/742573974","heartbeat_back_addr":"172.21.15.104:6815/742573974","heartbeat_front_addr":"172.21.15.104:6813/742573974","state":["exists","up"]},{"osd":5,"uuid":"0fe772d4-5800-4a88-a101-1dbf64b4abf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6809","nonce":2854081701}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6811","nonce":2854081701}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6815","nonce":2854081701}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2854081701},{"type":"v1","addr":"172.21.15.153:6813","nonce":2854081701}]},"public_addr":"172.21.15.153:6809/2854081701","cluster_addr":"172.21.15.153:6811/2854081701","heartbeat_back_addr":"172.21.15.153:6815/2854081701","heartbeat_front_addr":"172.21.15.153:6813/2854081701","state":["exists","up"]},{"osd":6,"uuid":"77e17684-22e7-4e11-b27f-ffc4703182ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6818","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6819","nonce":3825683804}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6820","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6821","nonce":3825683804}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6824","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6825","nonce":3825683804}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":3825683804},{"type":"v1","addr":"172.21.15.81:6823","nonce":3825683804}]},"public_addr":"172.21.15.81:6819/3825683804","cluster_addr":"172.21.15.81:6821/3825683804","heartbeat_back_addr":"172.21.15.81:6825/3825683804","heartbeat_front_addr":"172.21.15.81:6823/3825683804","state":["exists","up"]},{"osd":7,"uuid":"535d287c-497d-47a4-99e0-0d72ace39cd4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6817","nonce":1651003262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6819","nonce":1651003262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6823","nonce":1651003262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":1651003262},{"type":"v1","addr":"172.21.15.104:6821","nonce":1651003262}]},"public_addr":"172.21.15.104:6817/1651003262","cluster_addr":"172.21.15.104:6819/1651003262","heartbeat_back_addr":"172.21.15.104:6823/1651003262","heartbeat_front_addr":"172.21.15.104:6821/1651003262","state":["exists","up"]},{"osd":8,"uuid":"799f954d-0c19-4fb6-a80d-5df6f65588c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6817","nonce":2565156870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6819","nonce":2565156870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6823","nonce":2565156870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":2565156870},{"type":"v1","addr":"172.21.15.153:6821","nonce":2565156870}]},"public_addr":"172.21.15.153:6817/2565156870","cluster_addr":"172.21.15.153:6819/2565156870","heartbeat_back_addr":"172.21.15.153:6823/2565156870","heartbeat_front_addr":"172.21.15.153:6821/2565156870","state":["exists","up"]},{"osd":9,"uuid":"bbb02774-5697-4bcb-ac44-8b01cef5f3fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6827","nonce":1653329023}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6829","nonce":1653329023}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6832","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6833","nonce":1653329023}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":1653329023},{"type":"v1","addr":"172.21.15.81:6831","nonce":1653329023}]},"public_addr":"172.21.15.81:6827/1653329023","cluster_addr":"172.21.15.81:6829/1653329023","heartbeat_back_addr":"172.21.15.81:6833/1653329023","heartbeat_front_addr":"172.21.15.81:6831/1653329023","state":["exists","up"]},{"osd":10,"uuid":"30c8758d-3e32-4d59-9819-a13a38c1745d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6825","nonce":1019144761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6827","nonce":1019144761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6831","nonce":1019144761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":1019144761},{"type":"v1","addr":"172.21.15.104:6829","nonce":1019144761}]},"public_addr":"172.21.15.104:6825/1019144761","cluster_addr":"172.21.15.104:6827/1019144761","heartbeat_back_addr":"172.21.15.104:6831/1019144761","heartbeat_front_addr":"172.21.15.104:6829/1019144761","state":["exists","up"]},{"osd":11,"uuid":"b0fe362c-d9e8-4272-a3d3-2fdf4f255ec2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6825","nonce":2242325561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6827","nonce":2242325561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6831","nonce":2242325561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2242325561},{"type":"v1","addr":"172.21.15.153:6829","nonce":2242325561}]},"public_addr":"172.21.15.153:6825/2242325561","cluster_addr":"172.21.15.153:6827/2242325561","heartbeat_back_addr":"172.21.15.153:6831/2242325561","heartbeat_front_addr":"172.21.15.153:6829/2242325561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:12.673200+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:07:37.328271+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:00.671352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:32.470927+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:08:58.697197+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:23.360500+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:09:56.442587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:22.654862+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:10:47.332892+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:20.785955+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:11:47.967144+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-12T21:12:13.129571+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.81:0/1836548478":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/60786815":"2024-04-13T21:05:22.179703+0000","172.21.15.81:0/3894299177":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2659089739":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/2550894701":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6800/1636479292":"2024-04-13T21:05:22.179703+0000","172.21.15.81:6801/3647696686":"2024-04-13T21:04:48.054528+0000","172.21.15.81:0/3673746318":"2024-04-13T21:04:48.054528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-12T21:15:50.999 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd4740a3bb0 msgr2=0x7fd4740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd4740a3bb0 0x7fd4740a6070 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd490109490 tx=0x7fd480005f90 comp rx=0 tx=0).stop 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd490109630 msgr2=0x7fd49019ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd490109630 0x7fd49019ec10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd47800ea60 tx=0x7fd47800ef20 comp rx=0 tx=0).stop 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 shutdown_connections 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fd4740a3bb0 0x7fd4740a6070 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fd490109ff0 0x7fd4901a5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd490109630 0x7fd49019ec10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 --2- 172.21.15.81:0/255200188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd490108440 0x7fd49019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 >> 172.21.15.81:0/255200188 conn(0x7fd490077400 msgr2=0x7fd49010c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 shutdown_connections 2024-04-12T21:15:51.000 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.004+0000 7fd495c00640 1 -- 172.21.15.81:0/255200188 wait complete. 2024-04-12T21:15:51.021 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-12T21:13:13.003153+0000', 'modified': '2024-04-12T21:13:37.617228+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24547': {'gid': 24547, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.153:6839/2426851956', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.153:6838', 'nonce': 2426851956}, {'type': 'v1', 'addr': '172.21.15.153:6839', 'nonce': 2426851956}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24593': {'gid': 24593, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.104:6839/1944172329', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.104:6838', 'nonce': 1944172329}, {'type': 'v1', 'addr': '172.21.15.104:6839', 'nonce': 1944172329}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24593, 'qdb_cluster': [24593]} 2024-04-12T21:15:51.021 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-12T21:15:51.021 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-12T21:15:51.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 -- 172.21.15.81:0/280176269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a44104100 msgr2=0x7f7a44104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 --2- 172.21.15.81:0/280176269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a44104100 0x7f7a44104520 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7a34009860 tx=0x7f7a3402bb70 comp rx=0 tx=0).stop 2024-04-12T21:15:51.243 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 -- 172.21.15.81:0/280176269 shutdown_connections 2024-04-12T21:15:51.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 --2- 172.21.15.81:0/280176269 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a441052f0 0x7f7a44105770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 --2- 172.21.15.81:0/280176269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a44105cb0 0x7f7a4410c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 --2- 172.21.15.81:0/280176269 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a44104100 0x7f7a44104520 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 -- 172.21.15.81:0/280176269 >> 172.21.15.81:0/280176269 conn(0x7f7a440ff670 msgr2=0x7f7a44101ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:51.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 -- 172.21.15.81:0/280176269 shutdown_connections 2024-04-12T21:15:51.244 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.247+0000 7f7a43e00640 1 -- 172.21.15.81:0/280176269 wait complete. 2024-04-12T21:15:51.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.248+0000 7f7a43e00640 1 Processor -- start 2024-04-12T21:15:51.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.248+0000 7f7a43e00640 1 -- start start 2024-04-12T21:15:51.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a43e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a44104100 0x7f7a4419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a43e00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a441052f0 0x7f7a4419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a43e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a44105cb0 0x7f7a441a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.246 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a43e00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4410fd90 con 0x7f7a44104100 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a43e00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4410fc10 con 0x7f7a44105cb0 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a43e00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4410ff10 con 0x7f7a441052f0 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a44104100 0x7f7a4419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a43400640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a44105cb0 0x7f7a441a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42a00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a44104100 0x7f7a4419a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37816/0 (socket says 172.21.15.81:37816) 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42a00640 1 -- 172.21.15.81:0/1021161485 learned_addr learned my addr 172.21.15.81:0/1021161485 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42000640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a441052f0 0x7f7a4419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42a00640 1 -- 172.21.15.81:0/1021161485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a441052f0 msgr2=0x7f7a4419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42a00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a441052f0 0x7f7a4419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42a00640 1 -- 172.21.15.81:0/1021161485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a44105cb0 msgr2=0x7f7a441a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42a00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a44105cb0 0x7f7a441a19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.249+0000 7f7a42a00640 1 -- 172.21.15.81:0/1021161485 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a441a2050 con 0x7f7a44104100 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.250+0000 7f7a42a00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a44104100 0x7f7a4419a430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7a3402c0b0 tx=0x7f7a34032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.250+0000 7f7a3b400640 1 -- 172.21.15.81:0/1021161485 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34041020 con 0x7f7a44104100 2024-04-12T21:15:51.247 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.250+0000 7f7a3b400640 1 -- 172.21.15.81:0/1021161485 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a3402c9f0 con 0x7f7a44104100 2024-04-12T21:15:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.250+0000 7f7a43e00640 1 -- 172.21.15.81:0/1021161485 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a441a22e0 con 0x7f7a44104100 2024-04-12T21:15:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.250+0000 7f7a3b400640 1 -- 172.21.15.81:0/1021161485 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34039040 con 0x7f7a44104100 2024-04-12T21:15:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.250+0000 7f7a43e00640 1 -- 172.21.15.81:0/1021161485 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a441a2820 con 0x7f7a44104100 2024-04-12T21:15:51.248 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.253+0000 7f7a3b400640 1 -- 172.21.15.81:0/1021161485 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7a34044020 con 0x7f7a44104100 2024-04-12T21:15:51.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.253+0000 7f7a4c7ed740 1 Processor -- start 2024-04-12T21:15:51.249 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.253+0000 7f7a4c7ed740 1 -- start start 2024-04-12T21:15:51.250 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de02bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 0x56019de07ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 0x56019de0a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56019dd68990 con 0x56019de027b0 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56019dd0f530 con 0x56019de08310 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56019dd64950 con 0x56019de031a0 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de02bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a42a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 0x56019de07ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de02bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37822/0 (socket says 172.21.15.81:37822) 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a43400640 1 -- 172.21.15.81:0/3452998323 learned_addr learned my addr 172.21.15.81:0/3452998323 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a42000640 1 --2- 172.21.15.81:0/3452998323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 0x56019de0a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.251 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a43400640 1 -- 172.21.15.81:0/3452998323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 msgr2=0x56019de07ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a43400640 1 --2- 172.21.15.81:0/3452998323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 0x56019de07ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a43400640 1 -- 172.21.15.81:0/3452998323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 msgr2=0x56019de0a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a43400640 1 --2- 172.21.15.81:0/3452998323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 0x56019de0a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.254+0000 7f7a43400640 1 -- 172.21.15.81:0/3452998323 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56019da7b2d0 con 0x56019de027b0 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.255+0000 7f7a3b400640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a280a3c30 0x7f7a280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.255+0000 7f7a43400640 1 --2- 172.21.15.81:0/3452998323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de02bd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00e450 tx=0x7f7a3c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c33dd96416225baa server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.255+0000 7f7a3b400640 1 -- 172.21.15.81:0/1021161485 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f7a340f2000 con 0x7f7a44104100 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.255+0000 7f7a42a00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a280a3c30 0x7f7a280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.255+0000 7f7a33e00640 1 -- 172.21.15.81:0/3452998323 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c00d910 con 0x56019de027b0 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.255+0000 7f7a33e00640 1 -- 172.21.15.81:0/3452998323 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a3c01c070 con 0x56019de027b0 2024-04-12T21:15:51.252 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.255+0000 7f7a33e00640 1 -- 172.21.15.81:0/3452998323 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c016380 con 0x56019de027b0 2024-04-12T21:15:51.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/3452998323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 msgr2=0x56019de02bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a42a00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a280a3c30 0x7f7a280a60f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x56019de035a0 tx=0x7f7a340f5ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:51.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/3452998323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de02bd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00e450 tx=0x7f7a3c0095c0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.257 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/3452998323 shutdown_connections 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/3452998323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 0x56019de07ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/3452998323 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 0x56019de0a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/3452998323 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de02bd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/3452998323 >> 172.21.15.81:0/3452998323 conn(0x56019ddfdd20 msgr2=0x56019ddfe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/3452998323 shutdown_connections 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.256+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/3452998323 wait complete. 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 1 -- start start 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 1 Processor -- start 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 0x56019de0d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 0x56019de142a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56019de0b2c0 con 0x56019de027b0 2024-04-12T21:15:51.258 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56019dd228f0 con 0x56019de08310 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56019de0b440 con 0x56019de031a0 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.257+0000 7f7a4c7ed740 10 client.0 _ll_register_callbacks cb 0x56019dd222e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37826/0 (socket says 172.21.15.81:37826) 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 -- 172.21.15.81:0/263084894 learned_addr learned my addr 172.21.15.81:0/263084894 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a42000640 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 0x56019de0d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 0x56019de142a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 -- 172.21.15.81:0/263084894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 msgr2=0x56019de0d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 0x56019de0d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 -- 172.21.15.81:0/263084894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 msgr2=0x56019de142a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 0x56019de142a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 -- 172.21.15.81:0/263084894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56019db456b0 con 0x56019de027b0 2024-04-12T21:15:51.259 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a43400640 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0cd90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009c20 tx=0x7f7a3c005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297a512514e2118a server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.258+0000 7f7a30c00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a30c00640 1 -- 172.21.15.81:0/263084894 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c003970 con 0x56019de027b0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a30c00640 1 -- 172.21.15.81:0/263084894 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f7a3c003b10 con 0x56019de027b0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/263084894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a3c01c070 con 0x56019de027b0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a30c00640 1 -- 172.21.15.81:0/263084894 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7a3c036050 con 0x56019de027b0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a4c7ed740 10 client.15171 fetch_fsmap learned FSMap version 20 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/263084894 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56019de17610 con 0x56019de027b0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a30c00640 1 -- 172.21.15.81:0/263084894 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a3c01c070 con 0x56019de027b0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a30c00640 1 -- 172.21.15.81:0/263084894 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 20) ==== 8330+0+0 (secure 0 0 0) 0x7f7a3c00b040 con 0x56019de027b0 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.259+0000 7f7a4c7ed740 10 client.15171 fetch_fsmap finished waiting for FSMap version 20 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.260+0000 7f7a4c7ed740 10 client.15171 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.260+0000 7f7a4c7ed740 20 client.15171 populate_metadata read hostname 'smithi081' 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.260+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x56019de198e0 0x56019de1ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.260 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.260+0000 7f7a4c7ed740 4 client.15171 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.260+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/263084894 --> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56019dd0eb40 con 0x56019de198e0 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.260+0000 7f7a42000640 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x56019de198e0 0x56019de1ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.260+0000 7f7a42000640 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x56019de198e0 0x56019de1ad50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78dcb0f3de5a8bc server_cookie=656927b01fb63924 in_seq=0 out_seq=0 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.260+0000 7f7a30c00640 10 client.15171 ms_handle_connect on v2:172.21.15.104:6838/1944172329 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.261+0000 7f7a30c00640 1 -- 172.21.15.81:0/263084894 <== mds.0 v2:172.21.15.104:6838/1944172329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56019dd0eb40 con 0x56019de198e0 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.261+0000 7f7a30c00640 10 client.15171 handle_command_reply: tid=0 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.261+0000 7f7a4c7ed740 1 client.15171 shutdown 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.261+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/263084894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 msgr2=0x56019de0cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.261+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0cd90 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009c20 tx=0x7f7a3c005a50 comp rx=0 tx=0).stop 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/263084894 shutdown_connections 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x56019de198e0 0x56019de1ad50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de031a0 0x56019de0d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de08310 0x56019de142a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.261 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a30c00640 0 client.15171 ms_handle_reset on v2:172.21.15.104:6838/1944172329 2024-04-12T21:15:51.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/263084894 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0cd90 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/263084894 >> 172.21.15.81:0/263084894 conn(0x56019ddfdd20 msgr2=0x56019ddfcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:51.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/263084894 shutdown_connections 2024-04-12T21:15:51.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/263084894 wait complete. 2024-04-12T21:15:51.262 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.262+0000 7f7a4c7ed740 20 client.15171 trim_cache size 0 max 16384 2024-04-12T21:15:51.277 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.281+0000 7f7a4c7ed740 1 Processor -- start 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.281+0000 7f7a4c7ed740 1 -- start start 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.281+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019ddfe100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.281+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de14230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 0x56019de04770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56019de19090 con 0x56019de027b0 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56019de19760 con 0x56019de031a0 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56019de19e30 con 0x56019de08310 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019ddfe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019ddfe100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37836/0 (socket says 172.21.15.81:37836) 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 -- 172.21.15.81:0/1158487856 learned_addr learned my addr 172.21.15.81:0/1158487856 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a42000640 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de14230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 0x56019de04770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.278 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 -- 172.21.15.81:0/1158487856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 msgr2=0x56019de04770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:51.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 0x56019de04770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 -- 172.21.15.81:0/1158487856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 msgr2=0x56019de14230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:51.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de14230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.279 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.282+0000 7f7a43400640 1 -- 172.21.15.81:0/1158487856 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56019de07a40 con 0x56019de027b0 2024-04-12T21:15:51.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.283+0000 7f7a43400640 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019ddfe100 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c021b10 tx=0x7f7a3c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5d4048c0db64d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:51.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.283+0000 7f7a33400640 1 -- 172.21.15.81:0/1158487856 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c022d90 con 0x56019de027b0 2024-04-12T21:15:51.284 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.283+0000 7f7a33400640 1 -- 172.21.15.81:0/1158487856 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a3c01c040 con 0x56019de027b0 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.283+0000 7f7a33400640 1 -- 172.21.15.81:0/1158487856 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c00db60 con 0x56019de027b0 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.283+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/1158487856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 msgr2=0x56019ddfe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.283+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019ddfe100 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c021b10 tx=0x7f7a3c009f90 comp rx=0 tx=0).stop 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/1158487856 shutdown_connections 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 0x56019de04770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de14230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/1158487856 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019ddfe100 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/1158487856 >> 172.21.15.81:0/1158487856 conn(0x56019ddfdd20 msgr2=0x56019de0b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/1158487856 shutdown_connections 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/1158487856 wait complete. 2024-04-12T21:15:51.285 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 -- start start 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.284+0000 7f7a4c7ed740 1 Processor -- start 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de0fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a4c7ed740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 0x56019de12cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x56019de1abd0 con 0x56019de027b0 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x56019de1a500 con 0x56019de031a0 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a4c7ed740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56019de1b2a0 con 0x56019de08310 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a4c7ed740 10 client.0 _ll_register_callbacks cb 0x7f7a3c009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a42000640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de0fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37844/0 (socket says 172.21.15.81:37844) 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 -- 172.21.15.81:0/2439240862 learned_addr learned my addr 172.21.15.81:0/2439240862 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:51.286 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 0x56019de12cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 -- 172.21.15.81:0/2439240862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 msgr2=0x56019de12cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 0x56019de12cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 -- 172.21.15.81:0/2439240862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 msgr2=0x56019de0fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de0fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a43400640 1 -- 172.21.15.81:0/2439240862 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56019de132c0 con 0x56019de027b0 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.285+0000 7f7a42000640 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de0fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a43400640 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0f770 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c006010 tx=0x7f7a3c005b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96de3e85563ec963 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a30c00640 10 client.0 ms_handle_connect on v2:172.21.15.81:3300/0 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a30c00640 1 -- 172.21.15.81:0/2439240862 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c021430 con 0x56019de027b0 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a30c00640 1 -- 172.21.15.81:0/2439240862 <== mon.0 v2:172.21.15.81:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f7a3c02c030 con 0x56019de027b0 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/2439240862 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a3c01c040 con 0x56019de027b0 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a30c00640 1 -- 172.21.15.81:0/2439240862 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7a3c0215d0 con 0x56019de027b0 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a4c7ed740 10 client.15177 fetch_fsmap learned FSMap version 20 2024-04-12T21:15:51.287 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/2439240862 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x56019de145c0 con 0x56019de027b0 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a30c00640 1 -- 172.21.15.81:0/2439240862 <== mon.0 v2:172.21.15.81:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a3c01c040 con 0x56019de027b0 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.286+0000 7f7a30c00640 1 -- 172.21.15.81:0/2439240862 <== mon.0 v2:172.21.15.81:3300/0 5 ==== fsmap(e 20) ==== 8330+0+0 (secure 0 0 0) 0x7f7a3c00b050 con 0x56019de027b0 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a4c7ed740 10 client.15177 fetch_fsmap finished waiting for FSMap version 20 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a4c7ed740 10 client.15177 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a4c7ed740 20 client.15177 populate_metadata read hostname 'smithi081' 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x56019de8c6e0 0x56019de8b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a4c7ed740 4 client.15177 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/2439240862 --> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] -- command(tid 0: {"prefix": "status"}) -- 0x56019dd763b0 con 0x56019de8c6e0 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a42000640 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x56019de8c6e0 0x56019de8b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a42000640 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x56019de8c6e0 0x56019de8b3f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bb9b40f0ad4249d server_cookie=fa0877551b8711d3 in_seq=0 out_seq=0 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.287+0000 7f7a30c00640 10 client.15177 ms_handle_connect on v2:172.21.15.104:6838/1944172329 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a30c00640 1 -- 172.21.15.81:0/2439240862 <== mds.0 v2:172.21.15.104:6838/1944172329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56019dd763b0 con 0x56019de8c6e0 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a30c00640 10 client.15177 handle_command_reply: tid=0 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 client.15177 shutdown 2024-04-12T21:15:51.288 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/2439240862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 msgr2=0x56019de0f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0f770 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c006010 tx=0x7f7a3c005b40 comp rx=0 tx=0).stop 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/2439240862 shutdown_connections 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x56019de8c6e0 0x56019de8b3f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56019de08310 0x56019de12cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x56019de031a0 0x56019de0fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 --2- 172.21.15.81:0/2439240862 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x56019de027b0 0x56019de0f770 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a30c00640 0 client.15177 ms_handle_reset on v2:172.21.15.104:6838/1944172329 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/2439240862 >> 172.21.15.81:0/2439240862 conn(0x56019ddfdd20 msgr2=0x56019ddfe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/2439240862 shutdown_connections 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 1 -- 172.21.15.81:0/2439240862 wait complete. 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.288+0000 7f7a4c7ed740 20 client.15177 trim_cache size 0 max 16384 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stdout:{ 2024-04-12T21:15:51.289 INFO:teuthology.orchestra.run.smithi081.stdout: "cluster_fsid": "11b8cc88-f910-11ee-bc8d-c7b262605968", 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "whoami": 0, 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "id": 24593, 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "want_state": "up:active", 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "state": "up:active", 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "fs_name": "cephfs", 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "rank_uptime": 157.75951171599999, 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "mdsmap_epoch": 20, 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch": 93, 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "osdmap_epoch_barrier": 89, 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout: "uptime": 164.96507082799999 2024-04-12T21:15:51.290 INFO:teuthology.orchestra.run.smithi081.stdout:} 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 -- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a280a3c30 msgr2=0x7f7a280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a280a3c30 0x7f7a280a60f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x56019de035a0 tx=0x7f7a340f5ad0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 -- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a44104100 msgr2=0x7f7a4419a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a44104100 0x7f7a4419a430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7a3402c0b0 tx=0x7f7a34032430 comp rx=0 tx=0).stop 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 -- 172.21.15.81:0/1021161485 shutdown_connections 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f7a280a3c30 0x7f7a280a60f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a441052f0 0x7f7a4419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7a44105cb0 0x7f7a441a19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 --2- 172.21.15.81:0/1021161485 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f7a44104100 0x7f7a4419a430 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 -- 172.21.15.81:0/1021161485 >> 172.21.15.81:0/1021161485 conn(0x7f7a440ff670 msgr2=0x7f7a44108610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:51.291 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 -- 172.21.15.81:0/1021161485 shutdown_connections 2024-04-12T21:15:51.292 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:51.289+0000 7f7a43e00640 1 -- 172.21.15.81:0/1021161485 wait complete. 2024-04-12T21:15:51.301 INFO:teuthology.run_tasks:Running task sequential... 2024-04-12T21:15:51.312 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-12T21:15:51.312 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-12T21:15:51.312 INFO:teuthology.task.print:Enabling mgr modules 2024-04-12T21:15:51.312 INFO:teuthology.run_tasks:Running task kclient... 2024-04-12T21:15:51.340 INFO:tasks.kclient:Mounting kernel clients... 2024-04-12T21:15:51.340 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-12T21:15:51.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:51.340 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-04-12T21:15:51.394 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-12T21:15:51.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:51.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-12T21:15:51.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:51.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link delete ceph-brx 2024-04-12T21:15:51.559 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:51 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/3507164594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:51.560 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:51 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/255200188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:51.601 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-12T21:15:51.601 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-12T21:15:51.601 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-12T21:15:51.601 DEBUG:teuthology.orchestra.run.smithi081:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:324baf8ddb91dea46cac50cb535c306115791355 shell --fsid 11b8cc88-f910-11ee-bc8d-c7b262605968 -- ceph fs ls 2024-04-12T21:15:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:51 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/3507164594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:51 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/255200188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:51 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/3507164594' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-12T21:15:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:51 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/255200188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-12T21:15:52.046 INFO:teuthology.orchestra.run.smithi081.stderr:Inferring config /var/lib/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968/mon.a/config 2024-04-12T21:15:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:52 smithi153 ceph-mon[36569]: pgmap v398: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:52.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:52 smithi104 ceph-mon[36624]: pgmap v398: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:52.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:52 smithi081 ceph-mon[31453]: pgmap v398: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:54.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 -- 172.21.15.81:0/316659972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 msgr2=0x7fcabc06ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:54.393 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 --2- 172.21.15.81:0/316659972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 0x7fcabc06ac20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009990 tx=0x7fcaa8009ca0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 -- 172.21.15.81:0/316659972 shutdown_connections 2024-04-12T21:15:54.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 --2- 172.21.15.81:0/316659972 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcabc06b160 0x7fcabc10f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 --2- 172.21.15.81:0/316659972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcabc108aa0 0x7fcabc108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 --2- 172.21.15.81:0/316659972 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 0x7fcabc06ac20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 -- 172.21.15.81:0/316659972 >> 172.21.15.81:0/316659972 conn(0x7fcabc078960 msgr2=0x7fcabc078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:54.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 -- 172.21.15.81:0/316659972 shutdown_connections 2024-04-12T21:15:54.394 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.397+0000 7fcac3400700 1 -- 172.21.15.81:0/316659972 wait complete. 2024-04-12T21:15:54.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.399+0000 7fcac3400700 1 Processor -- start 2024-04-12T21:15:54.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.399+0000 7fcac3400700 1 -- start start 2024-04-12T21:15:54.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.399+0000 7fcac3400700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 0x7fcabc19b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:54.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.399+0000 7fcac3400700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcabc06b160 0x7fcabc19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:54.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.399+0000 7fcac3400700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcabc108aa0 0x7fcabc19fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:54.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.399+0000 7fcac3400700 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc112de0 con 0x7fcabc06a7c0 2024-04-12T21:15:54.395 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.399+0000 7fcac3400700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc112c60 con 0x7fcabc108aa0 2024-04-12T21:15:54.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.399+0000 7fcac3400700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcabc112f60 con 0x7fcabc06b160 2024-04-12T21:15:54.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac0a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 0x7fcabc19b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:54.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac0a00700 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 0x7fcabc19b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:37850/0 (socket says 172.21.15.81:37850) 2024-04-12T21:15:54.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac0a00700 1 -- 172.21.15.81:0/1365151646 learned_addr learned my addr 172.21.15.81:0/1365151646 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:15:54.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac1400700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcabc108aa0 0x7fcabc19fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:54.396 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcabbe00700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcabc06b160 0x7fcabc19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac0a00700 1 -- 172.21.15.81:0/1365151646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcabc06b160 msgr2=0x7fcabc19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac0a00700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcabc06b160 0x7fcabc19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac0a00700 1 -- 172.21.15.81:0/1365151646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcabc108aa0 msgr2=0x7fcabc19fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac0a00700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcabc108aa0 0x7fcabc19fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.400+0000 7fcac0a00700 1 -- 172.21.15.81:0/1365151646 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcabc1a05a0 con 0x7fcabc06a7c0 2024-04-12T21:15:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.401+0000 7fcac0a00700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 0x7fcabc19b590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcab000e9c0 tx=0x7fcab000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.401+0000 7fcab9600700 1 -- 172.21.15.81:0/1365151646 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab000cbd0 con 0x7fcabc06a7c0 2024-04-12T21:15:54.397 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.401+0000 7fcab9600700 1 -- 172.21.15.81:0/1365151646 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcab000ee60 con 0x7fcabc06a7c0 2024-04-12T21:15:54.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.401+0000 7fcab9600700 1 -- 172.21.15.81:0/1365151646 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab0005630 con 0x7fcabc06a7c0 2024-04-12T21:15:54.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.401+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcabc1a0890 con 0x7fcabc06a7c0 2024-04-12T21:15:54.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.401+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcabc1a8110 con 0x7fcabc06a7c0 2024-04-12T21:15:54.398 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.402+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcabc0697a0 con 0x7fcabc06a7c0 2024-04-12T21:15:54.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.404+0000 7fcab9600700 1 -- 172.21.15.81:0/1365151646 <== mon.0 v2:172.21.15.81:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcab0004750 con 0x7fcabc06a7c0 2024-04-12T21:15:54.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.406+0000 7fcab9600700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcaa40a3c40 0x7fcaa40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:15:54.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.407+0000 7fcab9600700 1 -- 172.21.15.81:0/1365151646 <== mon.0 v2:172.21.15.81:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7fcab00d09a0 con 0x7fcabc06a7c0 2024-04-12T21:15:54.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.407+0000 7fcabbe00700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcaa40a3c40 0x7fcaa40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:15:54.404 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.408+0000 7fcabbe00700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcaa40a3c40 0x7fcaa40a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcabc078fb0 tx=0x7fcaa8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:15:54.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.412+0000 7fcab9600700 1 -- 172.21.15.81:0/1365151646 <== mon.0 v2:172.21.15.81:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcab009cfd0 con 0x7fcabc06a7c0 2024-04-12T21:15:54.688 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.693+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fcabc02dfa0 con 0x7fcabc06a7c0 2024-04-12T21:15:54.689 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.694+0000 7fcab9600700 1 -- 172.21.15.81:0/1365151646 <== mon.0 v2:172.21.15.81:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) ==== 53+0+88 (secure 0 0 0) 0x7fcab009c8d0 con 0x7fcabc06a7c0 2024-04-12T21:15:54.689 INFO:teuthology.orchestra.run.smithi081.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-12T21:15:54.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.696+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcaa40a3c40 msgr2=0x7fcaa40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:54.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.696+0000 7fcac3400700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcaa40a3c40 0x7fcaa40a6100 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcabc078fb0 tx=0x7fcaa8005fd0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.692 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.696+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 msgr2=0x7fcabc19b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.696+0000 7fcac3400700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 0x7fcabc19b590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcab000e9c0 tx=0x7fcab000ecd0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.696+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 shutdown_connections 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.697+0000 7fcac3400700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7fcaa40a3c40 0x7fcaa40a6100 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.697+0000 7fcac3400700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcabc108aa0 0x7fcabc19fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.697+0000 7fcac3400700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fcabc06a7c0 0x7fcabc19b590 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.697+0000 7fcac3400700 1 --2- 172.21.15.81:0/1365151646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcabc06b160 0x7fcabc19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.697+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 >> 172.21.15.81:0/1365151646 conn(0x7fcabc078960 msgr2=0x7fcabc10e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.697+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 shutdown_connections 2024-04-12T21:15:54.693 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:54.697+0000 7fcac3400700 1 -- 172.21.15.81:0/1365151646 wait complete. 2024-04-12T21:15:55.561 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-12T21:15:55.561 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-12T21:15:55.561 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi081.front.sepia.ceph.com 2024-04-12T21:15:55.561 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-12T21:15:55.561 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-12T21:15:55.561 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-12T21:15:55.561 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-12T21:15:55.561 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-12T21:15:55.562 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-12T21:15:55.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:55.562 DEBUG:teuthology.orchestra.run.smithi081:> ip addr 2024-04-12T21:15:55.612 INFO:teuthology.orchestra.run.smithi081.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: inet 127.0.0.1/8 scope host lo 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 ::1/128 scope host 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:30 brd ff:ff:ff:ff:ff:ff 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f0 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:d8:a1:31 brd ff:ff:ff:ff:ff:ff 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: altname enp7s0f1 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:52 brd ff:ff:ff:ff:ff:ff 2024-04-12T21:15:55.613 INFO:teuthology.orchestra.run.smithi081.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-12T21:15:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: link/ether 0c:c4:7a:8f:d7:53 brd ff:ff:ff:ff:ff:ff 2024-04-12T21:15:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: inet 172.21.15.81/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-12T21:15:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft 41908sec preferred_lft 41908sec 2024-04-12T21:15:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: inet6 fe80::ec4:7aff:fe8f:d753/64 scope link 2024-04-12T21:15:55.614 INFO:teuthology.orchestra.run.smithi081.stdout: valid_lft forever preferred_lft forever 2024-04-12T21:15:55.615 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-12T21:15:55.615 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:15:55.615 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:15:55.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add name ceph-brx type bridge 2024-04-12T21:15:55.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr flush dev ceph-brx 2024-04-12T21:15:55.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set ceph-brx up 2024-04-12T21:15:55.615 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-12T21:15:55.615 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:15:55.707 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:15:55.741 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:55 smithi081 ceph-mon[31453]: pgmap v399: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:55.741 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:55 smithi081 ceph-mon[31453]: from='client.? 172.21.15.81:0/1365151646' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T21:15:55.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:55 smithi153 ceph-mon[36569]: pgmap v399: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:55.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:55 smithi153 ceph-mon[36569]: from='client.? 172.21.15.81:0/1365151646' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T21:15:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:55 smithi104 ceph-mon[36624]: pgmap v399: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:55 smithi104 ceph-mon[36624]: from='client.? 172.21.15.81:0/1365151646' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-12T21:15:55.868 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:55.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:55.878 DEBUG:teuthology.orchestra.run.smithi081:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-12T21:15:55.951 INFO:teuthology.orchestra.run.smithi081.stdout:1 2024-04-12T21:15:55.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:55.955 DEBUG:teuthology.orchestra.run.smithi081:> route 2024-04-12T21:15:56.006 INFO:teuthology.orchestra.run.smithi081.stdout:Kernel IP routing table 2024-04-12T21:15:56.006 INFO:teuthology.orchestra.run.smithi081.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-12T21:15:56.006 INFO:teuthology.orchestra.run.smithi081.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-12T21:15:56.006 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-12T21:15:56.006 INFO:teuthology.orchestra.run.smithi081.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-12T21:15:56.007 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:15:56.007 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:15:56.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-12T21:15:56.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-12T21:15:56.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-12T21:15:56.007 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:15:56.109 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:15:56.237 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:56.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:56.247 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list 2024-04-12T21:15:56.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:56.299 DEBUG:teuthology.orchestra.run.smithi081:> ip netns list-id 2024-04-12T21:15:56.345 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:15:56.345 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:15:56.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-12T21:15:56.345 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-12T21:15:56.346 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:15:56.447 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:15:56.538 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:56.548 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-12T21:15:56.549 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:15:56.549 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:15:56.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-12T21:15:56.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-12T21:15:56.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-12T21:15:56.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-12T21:15:56.549 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-12T21:15:56.549 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:15:56.650 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:15:56.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:56.853 DEBUG:teuthology.orchestra.run.smithi081:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-12T21:15:56.853 DEBUG:teuthology.orchestra.run.smithi081:> set -e 2024-04-12T21:15:56.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set brx.0 up 2024-04-12T21:15:56.853 DEBUG:teuthology.orchestra.run.smithi081:> sudo ip link set dev brx.0 master ceph-brx 2024-04-12T21:15:56.853 DEBUG:teuthology.orchestra.run.smithi081:> ') 2024-04-12T21:15:56.948 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:56 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:15:57.067 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:57.076 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T21:15:57.076 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-12T21:15:57.122 INFO:teuthology.orchestra.run.smithi081.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-12T21:15:57.122 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-12T21:15:57.123 DEBUG:teuthology.orchestra.run.smithi081:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-12T21:15:57.218 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-12T21:15:57.218 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-12T21:15:57.218 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/8dc75769-56c4-40a9-9818-3b2b61bf5c2b 2024-04-12T21:15:57.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:57.218 DEBUG:teuthology.orchestra.run.smithi081:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/8dc75769-56c4-40a9-9818-3b2b61bf5c2b /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-12T21:15:57.351 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-12T21:15:57.351 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-04-12T21:15:57.619 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:57 smithi081 ceph-mon[31453]: pgmap v400: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:57 smithi153 ceph-mon[36569]: pgmap v400: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:57.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:57 smithi104 ceph-mon[36624]: pgmap v400: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:15:58.066 INFO:teuthology.orchestra.run.smithi081.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-04-12T21:15:58.066 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-04-12T21:15:58.066 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: resolved to: "172.21.15.81:6789,172.21.15.104:6789,172.21.15.153:6789" 2024-04-12T21:15:58.066 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: trying mount with new device syntax: 0@11b8cc88-f910-11ee-bc8d-c7b262605968.cephfs=/volumes/qa/sv_1/8dc75769-56c4-40a9-9818-3b2b61bf5c2b 2024-04-12T21:15:58.066 INFO:teuthology.orchestra.run.smithi081.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.81:6789/172.21.15.104:6789/172.21.15.153:6789" will pass to kernel 2024-04-12T21:15:58.072 INFO:teuthology.orchestra.run.smithi081.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-12T21:15:58.072 INFO:teuthology.orchestra.run.smithi081.stdout: You just mounted a file system that supports labels which does not 2024-04-12T21:15:58.072 INFO:teuthology.orchestra.run.smithi081.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-12T21:15:58.072 INFO:teuthology.orchestra.run.smithi081.stdout: applications will generate AVC messages and not be allowed access to 2024-04-12T21:15:58.072 INFO:teuthology.orchestra.run.smithi081.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-12T21:15:58.078 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-12T21:15:58.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-12T21:15:58.078 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-12T21:15:58.174 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-12T21:15:58.326 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-12T21:15:58.333 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:58.343 INFO:teuthology.orchestra.run.smithi081.stdout:client25316 2024-04-12T21:15:58.343 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:15:58.343 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-12T21:15:58.397 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-12T21:15:58.504 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-12T21:15:58.509 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:58.518 INFO:teuthology.orchestra.run.smithi081.stdout:client25316 2024-04-12T21:15:58.519 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-12T21:15:58.620 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-12T21:15:58.625 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:58.634 INFO:teuthology.orchestra.run.smithi081.stdout:client25316 2024-04-12T21:15:58.635 DEBUG:teuthology.orchestra.run.smithi081:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/11b8cc88-f910-11ee-bc8d-c7b262605968.client25316/status') 2024-04-12T21:15:58.737 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-12T21:15:58.777 INFO:teuthology.orchestra.run.smithi081.stdout:instance: client.25316 (3)172.21.15.81:0/3123326716 2024-04-12T21:15:58.777 INFO:teuthology.orchestra.run.smithi081.stdout:blocklisted: false 2024-04-12T21:15:58.778 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records in 2024-04-12T21:15:58.778 INFO:teuthology.orchestra.run.smithi081.stderr:0+1 records out 2024-04-12T21:15:58.778 INFO:teuthology.orchestra.run.smithi081.stderr:71 bytes copied, 6.4924e-05 s, 1.1 MB/s 2024-04-12T21:15:58.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-04-12T21:15:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-12T21:15:58.791 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-12T21:15:58.802 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-12T21:15:58.817 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi081.front.sepia.ceph.com 2024-04-12T21:15:58.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-12T21:15:58.824 INFO:teuthology.run_tasks:Running task workunit... 2024-04-12T21:15:58.838 INFO:tasks.workunit:Pulling workunits from ref 324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:15:58.839 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-12T21:15:58.839 INFO:tasks.workunit:timeout=3h 2024-04-12T21:15:58.839 INFO:tasks.workunit:cleanup=True 2024-04-12T21:15:58.839 DEBUG:teuthology.orchestra.run.smithi081:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-12T21:15:58.889 INFO:teuthology.orchestra.run.smithi081.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-12T21:15:58.889 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-12T21:15:58.889 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-04-12T21:15:58.889 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-12T21:15:58.889 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-12T21:15:58.890 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-04-12 21:13:40.518754433 +0000 2024-04-12T21:15:58.890 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-04-12 21:13:40.518754433 +0000 2024-04-12T21:15:58.890 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-04-12 21:15:58.171120437 +0000 2024-04-12T21:15:58.890 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-04-12 21:13:40.518754433 +0000 2024-04-12T21:15:58.890 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-12T21:15:58.890 DEBUG:teuthology.orchestra.run.smithi081:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-12T21:15:59.012 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 324baf8ddb91dea46cac50cb535c306115791355 2024-04-12T21:15:59.063 INFO:tasks.workunit.client.0.smithi081.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-12T21:15:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:59 smithi153 ceph-mon[36569]: pgmap v401: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:15:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:15:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:59 smithi104 ceph-mon[36624]: pgmap v401: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:15:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:15:59.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:59 smithi081 ceph-mon[31453]: pgmap v401: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:15:59.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:15:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:16:00.786 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:00.786 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:00.786 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:00.786 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:00.786 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:00 smithi104 ceph-mon[36624]: pgmap v402: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-04-12T21:16:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:00 smithi153 ceph-mon[36569]: pgmap v402: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-04-12T21:16:01.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:01.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:01.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:01.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:01.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:00 smithi081 ceph-mon[31453]: pgmap v402: 129 pgs: 129 active+clean; 639 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 85 B/s wr, 1 op/s 2024-04-12T21:16:02.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:02.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:03.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:02 smithi081 ceph-mon[31453]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 938 B/s wr, 3 op/s 2024-04-12T21:16:03.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:03.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:03.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:02 smithi153 ceph-mon[36569]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 938 B/s wr, 3 op/s 2024-04-12T21:16:03.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:03.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:03.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:02 smithi104 ceph-mon[36624]: pgmap v403: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 938 B/s wr, 3 op/s 2024-04-12T21:16:03.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:03.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:04.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:16:04.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:16:04.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:04.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:16:04.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:16:04.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:04.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:16:04.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:16:04.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:16:05.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:05 smithi081 ceph-mon[31453]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:05.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:05 smithi153 ceph-mon[36569]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:05.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:05 smithi104 ceph-mon[36624]: pgmap v404: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:07 smithi153 ceph-mon[36569]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-12T21:16:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:07 smithi104 ceph-mon[36624]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-12T21:16:07.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:07 smithi081 ceph-mon[31453]: pgmap v405: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-12T21:16:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:09 smithi153 ceph-mon[36569]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:09 smithi104 ceph-mon[36624]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:09.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:09 smithi081 ceph-mon[31453]: pgmap v406: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:11 smithi153 ceph-mon[36569]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:11.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:11 smithi104 ceph-mon[36624]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:11 smithi081 ceph-mon[31453]: pgmap v407: 129 pgs: 129 active+clean; 646 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 -- 172.21.15.81:0/2501662871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1704077bc0 msgr2=0x7f1704078020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 --2- 172.21.15.81:0/2501662871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1704077bc0 0x7f1704078020 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009900 tx=0x7f16ec02bc30 comp rx=0 tx=0).stop 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 -- 172.21.15.81:0/2501662871 shutdown_connections 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 --2- 172.21.15.81:0/2501662871 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704078560 0x7f1704078e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 --2- 172.21.15.81:0/2501662871 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1704077bc0 0x7f1704078020 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 --2- 172.21.15.81:0/2501662871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704076980 0x7f1704076da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 -- 172.21.15.81:0/2501662871 >> 172.21.15.81:0/2501662871 conn(0x7f17040ff7b0 msgr2=0x7f1704101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 -- 172.21.15.81:0/2501662871 shutdown_connections 2024-04-12T21:16:12.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.077+0000 7f170ae00640 1 -- 172.21.15.81:0/2501662871 wait complete. 2024-04-12T21:16:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.078+0000 7f170ae00640 1 Processor -- start 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.078+0000 7f170ae00640 1 -- start start 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f170ae00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704076980 0x7f170419e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f170ae00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704077bc0 0x7f170419edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f170ae00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1704078560 0x7f17041a5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f170ae00640 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f17041142d0 con 0x7f1704078560 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f170ae00640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f1704114150 con 0x7f1704076980 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f170ae00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1704114450 con 0x7f1704077bc0 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1703e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704076980 0x7f170419e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1703e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704076980 0x7f170419e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.81:56360/0 (socket says 172.21.15.81:56360) 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 learned_addr learned my addr 172.21.15.81:0/759453957 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1708e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1704078560 0x7f17041a5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f16fbe00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704077bc0 0x7f170419edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704077bc0 msgr2=0x7f170419edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1704077bc0 0x7f170419edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1704078560 msgr2=0x7f17041a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:16:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7f1704078560 0x7f17041a5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.079+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17041a64b0 con 0x7f1704076980 2024-04-12T21:16:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.080+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1704076980 0x7f170419e890 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e9f0 tx=0x7f16f400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:16:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.080+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f400cd80 con 0x7f1704076980 2024-04-12T21:16:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.080+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16f4005500 con 0x7f1704076980 2024-04-12T21:16:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.080+0000 7f170ae00640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17041a67a0 con 0x7f1704076980 2024-04-12T21:16:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.080+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4010690 con 0x7f1704076980 2024-04-12T21:16:12.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.080+0000 7f170ae00640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17041a6cc0 con 0x7f1704076980 2024-04-12T21:16:12.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.083+0000 7f170ddce740 1 Processor -- start 2024-04-12T21:16:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.083+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f16f40046c0 con 0x7f1704076980 2024-04-12T21:16:12.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.083+0000 7f170ddce740 1 -- start start 2024-04-12T21:16:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f170ddce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e245750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f170ddce740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e24a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f170ddce740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e24d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f170ddce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563c9e1ab680 con 0x563c9e245330 2024-04-12T21:16:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f170ddce740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x563c9e152270 con 0x563c9e24ae90 2024-04-12T21:16:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f170ddce740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563c9e1a7640 con 0x563c9e245d20 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f1708e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e245750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f1708e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e245750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56566/0 (socket says 172.21.15.81:56566) 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f1708e00640 1 -- 172.21.15.81:0/1190498880 learned_addr learned my addr 172.21.15.81:0/1190498880 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f1703e00640 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e24a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f16fbe00640 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e24d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f1708e00640 1 -- 172.21.15.81:0/1190498880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 msgr2=0x563c9e24a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f1708e00640 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e24a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f1708e00640 1 -- 172.21.15.81:0/1190498880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 msgr2=0x563c9e24d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.084+0000 7f1708e00640 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e24d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.085+0000 7f1708e00640 1 -- 172.21.15.81:0/1190498880 --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c9debe210 con 0x563c9e245330 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.085+0000 7f1708e00640 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e245750 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009fd0 tx=0x7f16fc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a167d7a4cb6b3cb server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:16:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.085+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 0x7f16e80a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.085+0000 7f16f9600640 1 -- 172.21.15.81:0/1190498880 <== mon.0 v2:172.21.15.81:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc004d20 con 0x563c9e245330 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.085+0000 7f16f9600640 1 -- 172.21.15.81:0/1190498880 <== mon.0 v2:172.21.15.81:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16fc004050 con 0x563c9e245330 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.085+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f16f40d0b30 con 0x7f1704076980 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.085+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 0x7f16e80a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.085+0000 7f16f9600640 1 -- 172.21.15.81:0/1190498880 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc005240 con 0x563c9e245330 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 -- 172.21.15.81:0/1190498880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 msgr2=0x563c9e245750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e245750 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f16fc009fd0 tx=0x7f16fc0118d0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 0x7f16e80a6190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ce5f0 tx=0x7f16f40d4610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:16:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 -- 172.21.15.81:0/1190498880 shutdown_connections 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e24a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e24d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 --2- 172.21.15.81:0/1190498880 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e245750 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 -- 172.21.15.81:0/1190498880 >> 172.21.15.81:0/1190498880 conn(0x563c9e20d4b0 msgr2=0x563c9e240ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 -- 172.21.15.81:0/1190498880 shutdown_connections 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.086+0000 7f170ddce740 1 -- 172.21.15.81:0/1190498880 wait complete. 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 1 -- start start 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 1 Processor -- start 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245330 0x563c9e24f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245d20 0x563c9e24fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e256bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x563c9e24de90 con 0x563c9e24ae90 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x563c9e165630 con 0x563c9e245d20 2024-04-12T21:16:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x563c9e24e010 con 0x563c9e245330 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.087+0000 7f170ddce740 10 client.0 _ll_register_callbacks cb 0x563c9e1a2330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f1708e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e256bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245d20 0x563c9e24fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f1708e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e256bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56576/0 (socket says 172.21.15.81:56576) 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f1708e00640 1 -- 172.21.15.81:0/962873680 learned_addr learned my addr 172.21.15.81:0/962873680 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f1708e00640 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245330 0x563c9e24f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f16fbe00640 1 -- 172.21.15.81:0/962873680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245330 msgr2=0x563c9e24f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f16fbe00640 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245330 0x563c9e24f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f16fbe00640 1 -- 172.21.15.81:0/962873680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 msgr2=0x563c9e256bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f16fbe00640 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e256bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.088+0000 7f16fbe00640 1 -- 172.21.15.81:0/962873680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c9df8a620 con 0x563c9e245d20 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.089+0000 7f16fbe00640 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245d20 0x563c9e24fb70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009230 tx=0x7f16ec004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=195117e7d0acfcaf server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:16:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.089+0000 7f16f2000640 10 client.0 ms_handle_connect on v2:172.21.15.104:3300/0 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.089+0000 7f16f2000640 1 -- 172.21.15.81:0/962873680 <== mon.2 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec005d50 con 0x563c9e245d20 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.089+0000 7f170ddce740 1 -- 172.21.15.81:0/962873680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16fc004050 con 0x563c9e245d20 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.089+0000 7f16f2000640 1 -- 172.21.15.81:0/962873680 <== mon.2 v2:172.21.15.104:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f16ec005230 con 0x563c9e245d20 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.089+0000 7f16f2000640 1 -- 172.21.15.81:0/962873680 <== mon.2 v2:172.21.15.104:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f16ec005430 con 0x563c9e245d20 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.089+0000 7f170ddce740 10 client.25328 fetch_fsmap learned FSMap version 20 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.089+0000 7f170ddce740 1 -- 172.21.15.81:0/962873680 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x563c9e259fe0 con 0x563c9e245d20 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.090+0000 7f16f2000640 1 -- 172.21.15.81:0/962873680 <== mon.2 v2:172.21.15.104:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16fc004050 con 0x563c9e245d20 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.090+0000 7f16f2000640 1 -- 172.21.15.81:0/962873680 <== mon.2 v2:172.21.15.104:3300/0 5 ==== fsmap(e 20) ==== 8330+0+0 (secure 0 0 0) 0x7f16ec003a00 con 0x563c9e245d20 2024-04-12T21:16:12.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.090+0000 7f170ddce740 10 client.25328 fetch_fsmap finished waiting for FSMap version 20 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.090+0000 7f170ddce740 10 client.25328 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.090+0000 7f170ddce740 20 client.25328 populate_metadata read hostname 'smithi081' 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.090+0000 7f170ddce740 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e25b650 0x563c9e25da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.090+0000 7f170ddce740 4 client.25328 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.090+0000 7f170ddce740 1 -- 172.21.15.81:0/962873680 --> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c9e1b90a0 con 0x563c9e25b650 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.091+0000 7f1708e00640 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e25b650 0x563c9e25da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.091+0000 7f1708e00640 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e25b650 0x563c9e25da60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=559c7c43d2d41612 server_cookie=a68eb65a2c937c0d in_seq=0 out_seq=0 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.091+0000 7f16f2000640 10 client.25328 ms_handle_connect on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f16f2000640 1 -- 172.21.15.81:0/962873680 <== mds.0 v2:172.21.15.104:6838/1944172329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563c9e1b90a0 con 0x563c9e25b650 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f16f2000640 10 client.25328 handle_command_reply: tid=0 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 client.25328 shutdown 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 -- 172.21.15.81:0/962873680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245d20 msgr2=0x563c9e24fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245d20 0x563c9e24fb70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009230 tx=0x7f16ec004270 comp rx=0 tx=0).stop 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 -- 172.21.15.81:0/962873680 shutdown_connections 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e25b650 0x563c9e25da60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e256bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f16f2000640 0 client.25328 ms_handle_reset on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245d20 0x563c9e24fb70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 --2- 172.21.15.81:0/962873680 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245330 0x563c9e24f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 -- 172.21.15.81:0/962873680 >> 172.21.15.81:0/962873680 conn(0x563c9e20d4b0 msgr2=0x563c9e23fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:16:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 -- 172.21.15.81:0/962873680 shutdown_connections 2024-04-12T21:16:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.092+0000 7f170ddce740 1 -- 172.21.15.81:0/962873680 wait complete. 2024-04-12T21:16:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.093+0000 7f170ddce740 20 client.25328 trim_cache size 0 max 16384 2024-04-12T21:16:12.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.109+0000 7f170ddce740 1 Processor -- start 2024-04-12T21:16:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.109+0000 7f170ddce740 1 -- start start 2024-04-12T21:16:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.109+0000 7f170ddce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e2445f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.109+0000 7f170ddce740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e256b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.109+0000 7f170ddce740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e262360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.109+0000 7f170ddce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f17041af220 con 0x563c9e245330 2024-04-12T21:16:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.109+0000 7f170ddce740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f17041b1aa0 con 0x563c9e24ae90 2024-04-12T21:16:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.109+0000 7f170ddce740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17041af890 con 0x563c9e245d20 2024-04-12T21:16:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f1708e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e2445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e256b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f1708e00640 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e2445f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.81:3300/0 says I am v2:172.21.15.81:56580/0 (socket says 172.21.15.81:56580) 2024-04-12T21:16:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f1708e00640 1 -- 172.21.15.81:0/4123770050 learned_addr learned my addr 172.21.15.81:0/4123770050 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f1708e00640 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e262360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f16fbe00640 1 -- 172.21.15.81:0/4123770050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 msgr2=0x563c9e262360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f16fbe00640 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e262360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f16fbe00640 1 -- 172.21.15.81:0/4123770050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 msgr2=0x563c9e2445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f16fbe00640 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e2445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f16fbe00640 1 -- 172.21.15.81:0/4123770050 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c9e23c500 con 0x563c9e245d20 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f1708e00640 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e262360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.110+0000 7f16fbe00640 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e256b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f16ec003d70 tx=0x7f16ec02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7772e4da5255436 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f16f8c00640 1 -- 172.21.15.81:0/4123770050 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec000e60 con 0x563c9e245d20 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f16f8c00640 1 -- 172.21.15.81:0/4123770050 <== mon.1 v2:172.21.15.153:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ec009520 con 0x563c9e245d20 2024-04-12T21:16:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f16f8c00640 1 -- 172.21.15.81:0/4123770050 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec034a90 con 0x563c9e245d20 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f170ddce740 1 -- 172.21.15.81:0/4123770050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 msgr2=0x563c9e256b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f170ddce740 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e256b70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f16ec003d70 tx=0x7f16ec02bc30 comp rx=0 tx=0).stop 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f170ddce740 1 -- 172.21.15.81:0/4123770050 shutdown_connections 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f170ddce740 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e256b70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f170ddce740 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e24ae90 0x563c9e262360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f170ddce740 1 --2- 172.21.15.81:0/4123770050 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e245330 0x563c9e2445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.111+0000 7f170ddce740 1 -- 172.21.15.81:0/4123770050 >> 172.21.15.81:0/4123770050 conn(0x563c9e20d4b0 msgr2=0x563c9e243ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.112+0000 7f170ddce740 1 -- 172.21.15.81:0/4123770050 shutdown_connections 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.112+0000 7f170ddce740 1 -- 172.21.15.81:0/4123770050 wait complete. 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.112+0000 7f170ddce740 1 -- start start 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.112+0000 7f170ddce740 1 Processor -- start 2024-04-12T21:16:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f170ddce740 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245330 0x563c9e254620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f170ddce740 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e254b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f170ddce740 1 --2- >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e255ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f170ddce740 1 -- --> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] -- mon_getmap magic: 0 -- 0x7f17041aff60 con 0x563c9e24ae90 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f170ddce740 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 -- 0x7f17041b13d0 con 0x563c9e245330 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f170ddce740 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f17041b0630 con 0x563c9e245d20 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f170ddce740 10 client.0 _ll_register_callbacks cb 0x7f16ec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e254b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f1708e00640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245330 0x563c9e254620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e254b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.81:56790/0 (socket says 172.21.15.81:56790) 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f16fbe00640 1 -- 172.21.15.81:0/861362842 learned_addr learned my addr 172.21.15.81:0/861362842 (peer_addr_for_me v2:172.21.15.81:0/0) 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e255ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f16fbe00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245330 msgr2=0x563c9e254620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f16fbe00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x563c9e245330 0x563c9e254620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f16fbe00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 msgr2=0x563c9e255ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-12T21:16:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f16fbe00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e255ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.113+0000 7f16fbe00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c9e2560a0 con 0x563c9e245d20 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x563c9e24ae90 0x563c9e255ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f16fbe00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x563c9e245d20 0x563c9e254b60 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f16ec02c7b0 tx=0x7f16ec005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7201116680383850 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f16f2000640 10 client.0 ms_handle_connect on v2:172.21.15.153:3300/0 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec000e60 con 0x563c9e245d20 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f170ddce740 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16ec009520 con 0x563c9e245d20 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9179+0+0 (secure 0 0 0) 0x7f16ec034540 con 0x563c9e245d20 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f16ec034780 con 0x563c9e245d20 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f170ddce740 10 client.25321 fetch_fsmap learned FSMap version 20 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.114+0000 7f170ddce740 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) -- 0x563c9e2c1e50 con 0x563c9e245d20 2024-04-12T21:16:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16ec009520 con 0x563c9e245d20 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 5 ==== fsmap(e 20) ==== 8330+0+0 (secure 0 0 0) 0x7f16ec004f50 con 0x563c9e245d20 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f170ddce740 10 client.25321 fetch_fsmap finished waiting for FSMap version 20 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f170ddce740 10 client.25321 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f170ddce740 20 client.25321 populate_metadata read hostname 'smithi081' 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f170ddce740 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f170ddce740 4 client.25321 mds_command: new command op to 24593 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f170ddce740 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563c9e1b2f10 con 0x563c9e2c2750 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.115+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.116+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=975f2a7cf442a5f2 server_cookie=2c8bc766e77209c4 in_seq=0 out_seq=0 2024-04-12T21:16:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.116+0000 7f16f2000640 10 client.25321 ms_handle_connect on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.154+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.154+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.154+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.154+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-04-12T21:16:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.154+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.154+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.154+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=28 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-12T21:16:12.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.355+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.355+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.355+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=28 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-12T21:16:12.474 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:16:12.474 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: *** Caught signal (Aborted) ** 2024-04-12T21:16:12.474 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: in thread 7f180444f700 thread_name:MR_Finisher 2024-04-12T21:16:12.474 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:16:12.474 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f181400dd20] 2024-04-12T21:16:12.474 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2: gsignal() 2024-04-12T21:16:12.474 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 3: abort() 2024-04-12T21:16:12.474 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56376dd16268] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56376ddb20fc] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56376e0e6f0a] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 7: (MDLog::trim_all()+0x610) [0x56376e0e8f10] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 8: (C_Flush_Journal::trim_mdlog()+0x154) [0x56376dd77a04] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 9: (Context::complete(int)+0xd) [0x56376dd2a39d] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 10: (MDSContext::complete(int)+0x5f) [0x56376e0dd13f] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 11: (C_IO_Wrapper::finish(int)+0x12) [0x56376e0ddbc2] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 12: (MDSContext::complete(int)+0x5f) [0x56376e0dd13f] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 13: (MDSIOContextBase::complete(int)+0x3b4) [0x56376e0dd644] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 14: (Finisher::finisher_thread_entry()+0x19d) [0x7f1815325cfd] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f18140031ca] 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 16: clone() 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2024-04-12T21:16:12.122+0000 7f180444f700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: in thread 7f180444f700 thread_name:MR_Finisher 2024-04-12T21:16:12.475 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f181400dd20] 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2: gsignal() 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 3: abort() 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56376dd16268] 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56376ddb20fc] 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56376e0e6f0a] 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 7: (MDLog::trim_all()+0x610) [0x56376e0e8f10] 2024-04-12T21:16:12.476 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 8: (C_Flush_Journal::trim_mdlog()+0x154) [0x56376dd77a04] 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 9: (Context::complete(int)+0xd) [0x56376dd2a39d] 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 10: (MDSContext::complete(int)+0x5f) [0x56376e0dd13f] 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 11: (C_IO_Wrapper::finish(int)+0x12) [0x56376e0ddbc2] 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 12: (MDSContext::complete(int)+0x5f) [0x56376e0dd13f] 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 13: (MDSIOContextBase::complete(int)+0x3b4) [0x56376e0dd644] 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 14: (Finisher::finisher_thread_entry()+0x19d) [0x7f1815325cfd] 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f18140031ca] 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 16: clone() 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 0> 2024-04-12T21:16:12.122+0000 7f180444f700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: in thread 7f180444f700 thread_name:MR_Finisher 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:16:12.477 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f181400dd20] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2: gsignal() 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 3: abort() 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56376dd16268] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56376ddb20fc] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56376e0e6f0a] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 7: (MDLog::trim_all()+0x610) [0x56376e0e8f10] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 8: (C_Flush_Journal::trim_mdlog()+0x154) [0x56376dd77a04] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 9: (Context::complete(int)+0xd) [0x56376dd2a39d] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 10: (MDSContext::complete(int)+0x5f) [0x56376e0dd13f] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 11: (C_IO_Wrapper::finish(int)+0x12) [0x56376e0ddbc2] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 12: (MDSContext::complete(int)+0x5f) [0x56376e0dd13f] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 13: (MDSIOContextBase::complete(int)+0x3b4) [0x56376e0dd644] 2024-04-12T21:16:12.478 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 14: (Finisher::finisher_thread_entry()+0x19d) [0x7f1815325cfd] 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f18140031ca] 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 16: clone() 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 0> 2024-04-12T21:16:12.122+0000 7f180444f700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: in thread 7f180444f700 thread_name:MR_Finisher 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f181400dd20] 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2: gsignal() 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 3: abort() 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56376dd16268] 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56376ddb20fc] 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56376e0e6f0a] 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 7: (MDLog::trim_all()+0x610) [0x56376e0e8f10] 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 8: (C_Flush_Journal::trim_mdlog()+0x154) [0x56376dd77a04] 2024-04-12T21:16:12.479 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 9: (Context::complete(int)+0xd) [0x56376dd2a39d] 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 10: (MDSContext::complete(int)+0x5f) [0x56376e0dd13f] 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 11: (C_IO_Wrapper::finish(int)+0x12) [0x56376e0ddbc2] 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 12: (MDSContext::complete(int)+0x5f) [0x56376e0dd13f] 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 13: (MDSIOContextBase::complete(int)+0x3b4) [0x56376e0dd644] 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 14: (Finisher::finisher_thread_entry()+0x19d) [0x7f1815325cfd] 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f18140031ca] 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 16: clone() 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[66227]: 2024-04-12T21:16:12.480 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 podman[68090]: 2024-04-12 21:16:12.183990082 +0000 UTC m=+0.020775362 container died ddefba9658f604ca54adb31278347c1d0555779c01f437f0ddd12a318aaebe98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-12T21:16:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.756+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.756+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:12.756+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=28 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-12T21:16:13.174 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 podman[68090]: 2024-04-12 21:16:12.929440033 +0000 UTC m=+0.766225333 container remove ddefba9658f604ca54adb31278347c1d0555779c01f437f0ddd12a318aaebe98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-12T21:16:13.174 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:12 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:16:13.174 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:13 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-12T21:16:13.175 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:13 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 1.005s CPU time. 2024-04-12T21:16:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:13.557+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:13.557+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:13.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:13.557+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=28 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-12T21:16:13.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:13 smithi081 ceph-mon[31453]: pgmap v408: 129 pgs: 129 active+clean; 650 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-12T21:16:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:13 smithi153 ceph-mon[36569]: pgmap v408: 129 pgs: 129 active+clean; 650 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-12T21:16:13.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:13 smithi104 ceph-mon[36624]: pgmap v408: 129 pgs: 129 active+clean; 650 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-12T21:16:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:15.160+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:15.160+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:15.160+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=28 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-12T21:16:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:15 smithi153 ceph-mon[36569]: pgmap v409: 129 pgs: 129 active+clean; 650 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-04-12T21:16:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:15 smithi104 ceph-mon[36624]: pgmap v409: 129 pgs: 129 active+clean; 650 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-04-12T21:16:15.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:15 smithi081 ceph-mon[31453]: pgmap v409: 129 pgs: 129 active+clean; 650 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-04-12T21:16:17.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:17 smithi153 ceph-mon[36569]: pgmap v410: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-12T21:16:17.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:17 smithi104 ceph-mon[36624]: pgmap v410: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-12T21:16:17.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:17 smithi081 ceph-mon[31453]: pgmap v410: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-12T21:16:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:18.364+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:18.364+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:18.364+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=28 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-12T21:16:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:19 smithi153 ceph-mon[36569]: pgmap v411: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-12T21:16:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:19 smithi104 ceph-mon[36624]: pgmap v411: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-12T21:16:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:19 smithi081 ceph-mon[31453]: pgmap v411: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-12T21:16:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:21 smithi153 ceph-mon[36569]: pgmap v412: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-12T21:16:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:21 smithi104 ceph-mon[36624]: pgmap v412: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-12T21:16:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:21 smithi081 ceph-mon[31453]: pgmap v412: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-04-12T21:16:23.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:23 smithi104 ceph-mon[36624]: pgmap v413: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-12T21:16:23.591 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:16:23.591 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 systemd[1]: Stopped Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:16:23.591 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 1.005s CPU time. 2024-04-12T21:16:23.591 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 systemd[1]: Starting Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:16:23.591 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 podman[68186]: 2024-04-12 21:16:23.468564889 +0000 UTC m=+0.017811070 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:16:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:23 smithi153 ceph-mon[36569]: pgmap v413: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-12T21:16:23.844 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 podman[68186]: 2024-04-12 21:16:23.595969793 +0000 UTC m=+0.145215964 container create bc5eb3014abcb56a092bbe2bea79825dd1d22040271ae828906728f43138691e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-12T21:16:23.844 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 podman[68186]: 2024-04-12 21:16:23.823826418 +0000 UTC m=+0.373072596 container init bc5eb3014abcb56a092bbe2bea79825dd1d22040271ae828906728f43138691e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-12T21:16:23.844 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 podman[68186]: 2024-04-12 21:16:23.826450242 +0000 UTC m=+0.375696422 container start bc5eb3014abcb56a092bbe2bea79825dd1d22040271ae828906728f43138691e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-12T21:16:23.865 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:23 smithi081 ceph-mon[31453]: pgmap v413: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-04-12T21:16:24.108 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 ceph-mds[68205]: -- 172.21.15.104:0/3632278310 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629d7857a40 con 0x5629d782ac00 2024-04-12T21:16:24.108 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: starting mds.b at 2024-04-12T21:16:24.108 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 bash[68186]: bc5eb3014abcb56a092bbe2bea79825dd1d22040271ae828906728f43138691e 2024-04-12T21:16:24.108 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:16:23 smithi104 systemd[1]: Started Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:16:24.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:16:23.867+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: daemon mds.b restarted 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: osdmap e94: 12 total, 12 up, 12 in 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:boot 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:16:24.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:24 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:16:24.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:24.770+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=28 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:24.771+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_RECONNECTING pgs=28 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2024-04-12T21:16:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:24.771+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_RECONNECTING pgs=28 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2024-04-12T21:16:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:24.771+0000 7f16f2000640 0 client.25321 ms_handle_remote_reset on v2:172.21.15.104:6838/1944172329 2024-04-12T21:16:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:24.771+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:16:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:24.771+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:16:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:24.771+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:16:24.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:24.771+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-12T21:16:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: daemon mds.b restarted 2024-04-12T21:16:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:16:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:16:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: osdmap e94: 12 total, 12 up, 12 in 2024-04-12T21:16:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:boot 2024-04-12T21:16:24.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:16:24.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:16:24.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:16:24.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:16:24.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:24 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: daemon mds.b restarted 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: osdmap e94: 12 total, 12 up, 12 in 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:boot 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:16:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:24 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:16:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:25 smithi153 ceph-mon[36569]: pgmap v415: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 409 B/s wr, 2 op/s 2024-04-12T21:16:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:reconnect 2024-04-12T21:16:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:25 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:16:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:25 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:16:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:25 smithi104 ceph-mon[36624]: pgmap v415: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 409 B/s wr, 2 op/s 2024-04-12T21:16:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:reconnect 2024-04-12T21:16:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:25 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:16:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:25 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:16:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:25 smithi081 ceph-mon[31453]: pgmap v415: 129 pgs: 129 active+clean; 653 KiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 409 B/s wr, 2 op/s 2024-04-12T21:16:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:reconnect 2024-04-12T21:16:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:25 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:16:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:25 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:16:27.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:26 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:rejoin 2024-04-12T21:16:27.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:26 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:16:27.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:26 smithi153 ceph-mon[36569]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:16:27.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:26 smithi153 ceph-mon[36569]: pgmap v416: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:16:27.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:26 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:rejoin 2024-04-12T21:16:27.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:26 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:16:27.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:26 smithi104 ceph-mon[36624]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:16:27.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:26 smithi104 ceph-mon[36624]: pgmap v416: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:16:27.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:26 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:rejoin 2024-04-12T21:16:27.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:26 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:16:27.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:26 smithi081 ceph-mon[31453]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:16:27.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:26 smithi081 ceph-mon[31453]: pgmap v416: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:16:28.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:27 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:16:28.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:27 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:16:28.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:27 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:active 2024-04-12T21:16:28.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:27 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:16:28.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:27 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:16:28.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:27 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:16:28.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:27 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:16:28.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:27 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:active 2024-04-12T21:16:28.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:27 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:16:28.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:27 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:16:28.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:27 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:16:28.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:27 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:16:28.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:27 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2426851956,v1:172.21.15.153:6839/2426851956] up:active 2024-04-12T21:16:28.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:27 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:16:28.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:27 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:16:29.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:28 smithi153 ceph-mon[36569]: pgmap v417: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:16:29.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:28 smithi104 ceph-mon[36624]: pgmap v417: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:16:29.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:28 smithi081 ceph-mon[31453]: pgmap v417: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2 op/s 2024-04-12T21:16:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:31 smithi153 ceph-mon[36569]: pgmap v418: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-12T21:16:31.802 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:16:31 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2024-04-12T21:16:31.322+0000 7f463d41c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:16:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:31 smithi104 ceph-mon[36624]: pgmap v418: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-12T21:16:31.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:31 smithi081 ceph-mon[31453]: pgmap v418: 129 pgs: 129 active+clean; 653 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 102 B/s wr, 7 op/s 2024-04-12T21:16:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:33 smithi153 ceph-mon[36569]: pgmap v419: 129 pgs: 129 active+clean; 668 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-12T21:16:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:33 smithi104 ceph-mon[36624]: pgmap v419: 129 pgs: 129 active+clean; 668 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-12T21:16:33.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:33 smithi081 ceph-mon[31453]: pgmap v419: 129 pgs: 129 active+clean; 668 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-12T21:16:35.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:35 smithi153 ceph-mon[36569]: pgmap v420: 129 pgs: 129 active+clean; 668 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-12T21:16:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:35 smithi104 ceph-mon[36624]: pgmap v420: 129 pgs: 129 active+clean; 668 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-12T21:16:35.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:35 smithi081 ceph-mon[31453]: pgmap v420: 129 pgs: 129 active+clean; 668 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-04-12T21:16:37.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:37.584+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:37.586+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:16:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:37.586+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:16:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:37.586+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:16:37.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:37.586+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:16:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:37 smithi153 ceph-mon[36569]: pgmap v421: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-04-12T21:16:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:37 smithi104 ceph-mon[36624]: pgmap v421: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-04-12T21:16:37.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:37 smithi081 ceph-mon[31453]: pgmap v421: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2024-04-12T21:16:39.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:39 smithi153 ceph-mon[36569]: pgmap v422: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T21:16:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:39 smithi104 ceph-mon[36624]: pgmap v422: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T21:16:39.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:39 smithi081 ceph-mon[31453]: pgmap v422: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T21:16:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:41 smithi153 ceph-mon[36569]: pgmap v423: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T21:16:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:41 smithi104 ceph-mon[36624]: pgmap v423: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T21:16:41.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:41 smithi081 ceph-mon[31453]: pgmap v423: 129 pgs: 129 active+clean; 673 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 9 op/s 2024-04-12T21:16:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:43 smithi153 ceph-mon[36569]: pgmap v424: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 5 op/s 2024-04-12T21:16:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:43 smithi104 ceph-mon[36624]: pgmap v424: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 5 op/s 2024-04-12T21:16:43.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:43 smithi081 ceph-mon[31453]: pgmap v424: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 2.8 KiB/s wr, 5 op/s 2024-04-12T21:16:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:45 smithi153 ceph-mon[36569]: pgmap v425: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-12T21:16:45.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:45 smithi104 ceph-mon[36624]: pgmap v425: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-12T21:16:45.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:45 smithi081 ceph-mon[31453]: pgmap v425: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-12T21:16:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:47 smithi153 ceph-mon[36569]: pgmap v426: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-12T21:16:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:47 smithi104 ceph-mon[36624]: pgmap v426: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-12T21:16:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:47 smithi081 ceph-mon[31453]: pgmap v426: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-12T21:16:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:49 smithi153 ceph-mon[36569]: pgmap v427: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:49 smithi104 ceph-mon[36624]: pgmap v427: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:49.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:49 smithi081 ceph-mon[31453]: pgmap v427: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:51 smithi153 ceph-mon[36569]: pgmap v428: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:51 smithi104 ceph-mon[36624]: pgmap v428: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:51.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:51 smithi081 ceph-mon[31453]: pgmap v428: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:52.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:52.592+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:16:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:52.594+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:16:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:52.594+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:16:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:52.594+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:16:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:16:52.594+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:16:53.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:53 smithi153 ceph-mon[36569]: pgmap v429: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:53.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:53 smithi104 ceph-mon[36624]: pgmap v429: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:53.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:53 smithi081 ceph-mon[31453]: pgmap v429: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-12T21:16:55.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:55 smithi153 ceph-mon[36569]: pgmap v430: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:16:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:55 smithi104 ceph-mon[36624]: pgmap v430: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:16:55.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:55 smithi081 ceph-mon[31453]: pgmap v430: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-12T21:16:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:57 smithi153 ceph-mon[36569]: pgmap v431: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-12T21:16:57.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:57 smithi104 ceph-mon[36624]: pgmap v431: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-12T21:16:57.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:57 smithi081 ceph-mon[31453]: pgmap v431: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s 2024-04-12T21:16:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:16:59 smithi153 ceph-mon[36569]: pgmap v432: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:16:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:16:59 smithi104 ceph-mon[36624]: pgmap v432: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:16:59.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:16:59 smithi081 ceph-mon[31453]: pgmap v432: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:17:00.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:00 smithi153 ceph-mon[36569]: pgmap v433: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:00.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:00 smithi104 ceph-mon[36624]: pgmap v433: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:00.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:00 smithi081 ceph-mon[31453]: pgmap v433: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:03.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:03 smithi153 ceph-mon[36569]: pgmap v434: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:03.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:03 smithi104 ceph-mon[36624]: pgmap v434: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:03.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:03 smithi081 ceph-mon[31453]: pgmap v434: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:04.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:17:04.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:17:04.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:17:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:05 smithi153 ceph-mon[36569]: pgmap v435: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:05 smithi104 ceph-mon[36624]: pgmap v435: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:05 smithi081 ceph-mon[31453]: pgmap v435: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:06.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:17:06.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:17:06.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:06.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:17:06.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:17:06.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:06.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:17:06.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:17:06.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:17:07.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:07.609+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:17:07.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:07.610+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:17:07.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:07.610+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:17:07.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:07.610+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:17:07.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:07.610+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:17:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:07 smithi153 ceph-mon[36569]: pgmap v436: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:07 smithi104 ceph-mon[36624]: pgmap v436: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:07.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:07 smithi081 ceph-mon[31453]: pgmap v436: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:09.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:09 smithi153 ceph-mon[36569]: pgmap v437: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:09 smithi104 ceph-mon[36624]: pgmap v437: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:09.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:09 smithi081 ceph-mon[31453]: pgmap v437: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:11 smithi153 ceph-mon[36569]: pgmap v438: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:11.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:11 smithi104 ceph-mon[36624]: pgmap v438: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:11 smithi081 ceph-mon[31453]: pgmap v438: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:13 smithi153 ceph-mon[36569]: pgmap v439: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:13.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:13 smithi104 ceph-mon[36624]: pgmap v439: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:13.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:13 smithi081 ceph-mon[31453]: pgmap v439: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:15 smithi153 ceph-mon[36569]: pgmap v440: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:15 smithi104 ceph-mon[36624]: pgmap v440: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:15.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:15 smithi081 ceph-mon[31453]: pgmap v440: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:17.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:17 smithi153 ceph-mon[36569]: pgmap v441: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:17.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:17 smithi104 ceph-mon[36624]: pgmap v441: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:17.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:17 smithi081 ceph-mon[31453]: pgmap v441: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:19 smithi153 ceph-mon[36569]: pgmap v442: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:19 smithi104 ceph-mon[36624]: pgmap v442: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:19 smithi081 ceph-mon[31453]: pgmap v442: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:21 smithi153 ceph-mon[36569]: pgmap v443: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:21 smithi104 ceph-mon[36624]: pgmap v443: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:21 smithi081 ceph-mon[31453]: pgmap v443: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:22.625+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:17:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:22.626+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:17:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:22.626+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:17:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:22.626+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:17:22.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:22.626+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:17:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:23 smithi153 ceph-mon[36569]: pgmap v444: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:23 smithi104 ceph-mon[36624]: pgmap v444: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:23 smithi081 ceph-mon[31453]: pgmap v444: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:25 smithi153 ceph-mon[36569]: pgmap v445: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:25 smithi104 ceph-mon[36624]: pgmap v445: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:25 smithi081 ceph-mon[31453]: pgmap v445: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:27.631 INFO:tasks.workunit.client.0.smithi081.stderr:Updating files: 62% (7669/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-12T21:17:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:27 smithi153 ceph-mon[36569]: pgmap v446: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:27 smithi104 ceph-mon[36624]: pgmap v446: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:27.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:27 smithi081 ceph-mon[31453]: pgmap v446: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr:Note: switching to '324baf8ddb91dea46cac50cb535c306115791355'. 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr:state without impacting any branches by switching back to a branch. 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:28.472 INFO:tasks.workunit.client.0.smithi081.stderr: git switch -c 2024-04-12T21:17:28.473 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:28.473 INFO:tasks.workunit.client.0.smithi081.stderr:Or undo this operation with: 2024-04-12T21:17:28.473 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:28.473 INFO:tasks.workunit.client.0.smithi081.stderr: git switch - 2024-04-12T21:17:28.473 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:28.473 INFO:tasks.workunit.client.0.smithi081.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-12T21:17:28.473 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:28.473 INFO:tasks.workunit.client.0.smithi081.stderr:HEAD is now at 324baf8ddb9 Merge PR #55421 into wip-vshankar-testing-20240411.061452 2024-04-12T21:17:28.485 DEBUG:teuthology.orchestra.run.smithi081:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-12T21:17:28.547 INFO:tasks.workunit.client.0.smithi081.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-12T21:17:28.556 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-12T21:17:28.556 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-12T21:17:29.527 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-12T21:17:29.654 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-12T21:17:29.737 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-12T21:17:29.743 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-12T21:17:29.743 INFO:tasks.workunit.client.0.smithi081.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-12T21:17:29.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:29 smithi153 ceph-mon[36569]: pgmap v447: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:17:29.827 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-12T21:17:29.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:29 smithi104 ceph-mon[36624]: pgmap v447: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:17:29.836 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-04-12T21:17:29.836 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-12T21:17:29.889 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-04-12T21:17:29.890 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-04-12T21:17:29.891 DEBUG:teuthology.orchestra.run.smithi081:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=324baf8ddb91dea46cac50cb535c306115791355 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-04-12T21:17:29.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:29 smithi081 ceph-mon[31453]: pgmap v447: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s rd, 1 op/s 2024-04-12T21:17:29.962 INFO:tasks.workunit.client.0.smithi081.stdout:getting blogbench 2024-04-12T21:17:29.962 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 'getting blogbench' 2024-04-12T21:17:29.962 INFO:tasks.workunit.client.0.smithi081.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-04-12T21:17:29.990 INFO:tasks.workunit.client.0.smithi081.stderr:--2024-04-12 21:17:29-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-04-12T21:17:29.991 INFO:tasks.workunit.client.0.smithi081.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-12T21:17:30.013 INFO:tasks.workunit.client.0.smithi081.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-12T21:17:30.033 INFO:tasks.workunit.client.0.smithi081.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-12T21:17:30.033 INFO:tasks.workunit.client.0.smithi081.stderr:Length: 103355 (101K) [application/octet-stream] 2024-04-12T21:17:30.036 INFO:tasks.workunit.client.0.smithi081.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-04-12T21:17:30.043 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:30.075 INFO:tasks.workunit.client.0.smithi081.stderr: 0K .......... .......... .......... .......... .......... 49% 1.27M 0s 2024-04-12T21:17:30.096 INFO:tasks.workunit.client.0.smithi081.stderr: 50K .......... .......... .......... .......... .......... 99% 2.40M 0s 2024-04-12T21:17:30.096 INFO:tasks.workunit.client.0.smithi081.stderr: 100K 100% 1.74T=0.06s 2024-04-12T21:17:30.096 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:30.096 INFO:tasks.workunit.client.0.smithi081.stderr:2024-04-12 21:17:30 (1.67 MB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-04-12T21:17:30.096 INFO:tasks.workunit.client.0.smithi081.stderr: 2024-04-12T21:17:30.098 INFO:tasks.workunit.client.0.smithi081.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-04-12T21:17:30.138 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/ 2024-04-12T21:17:30.139 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/m4/ 2024-04-12T21:17:30.140 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-04-12T21:17:30.142 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/man/ 2024-04-12T21:17:30.143 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/man/blogbench.8 2024-04-12T21:17:30.145 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/man/Makefile.am 2024-04-12T21:17:30.146 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/man/Makefile.in 2024-04-12T21:17:30.148 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/ 2024-04-12T21:17:30.149 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/mysnprintf.c 2024-04-12T21:17:30.151 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/mysnprintf.h 2024-04-12T21:17:30.152 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/blogbench.c 2024-04-12T21:17:30.154 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/blogbench.h 2024-04-12T21:17:30.156 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/fakesnprintf.c 2024-04-12T21:17:30.158 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/fakesnprintf.h 2024-04-12T21:17:30.159 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/blogbench_p.h 2024-04-12T21:17:30.162 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-04-12T21:17:30.163 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-04-12T21:17:30.165 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/Makefile.am 2024-04-12T21:17:30.167 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/Makefile.in 2024-04-12T21:17:30.168 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/process.c 2024-04-12T21:17:30.170 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/globals.h 2024-04-12T21:17:30.172 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/commenter.c 2024-04-12T21:17:30.173 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/rewriter.c 2024-04-12T21:17:30.175 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/reader.c 2024-04-12T21:17:30.177 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/writer.c 2024-04-12T21:17:30.178 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/src/helpers.c 2024-04-12T21:17:30.180 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/NEWS 2024-04-12T21:17:30.181 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/compile 2024-04-12T21:17:30.183 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/depcomp 2024-04-12T21:17:30.184 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/aclocal.m4 2024-04-12T21:17:30.186 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/README 2024-04-12T21:17:30.188 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/configure 2024-04-12T21:17:30.197 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/configure.ac 2024-04-12T21:17:30.199 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/config.guess 2024-04-12T21:17:30.201 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/install-sh 2024-04-12T21:17:30.202 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/config.sub 2024-04-12T21:17:30.204 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/missing 2024-04-12T21:17:30.206 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/mkinstalldirs 2024-04-12T21:17:30.207 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/Makefile.am 2024-04-12T21:17:30.209 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/Makefile.in 2024-04-12T21:17:30.211 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/config.h.in 2024-04-12T21:17:30.212 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/AUTHORS 2024-04-12T21:17:30.214 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/INSTALL 2024-04-12T21:17:30.216 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/ChangeLog 2024-04-12T21:17:30.217 INFO:tasks.workunit.client.0.smithi081.stdout:blogbench-1.0/COPYING 2024-04-12T21:17:30.220 INFO:tasks.workunit.client.0.smithi081.stderr:+ cd blogbench-1.0/ 2024-04-12T21:17:30.221 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 'making blogbench' 2024-04-12T21:17:30.221 INFO:tasks.workunit.client.0.smithi081.stderr:+ ./configure 2024-04-12T21:17:30.221 INFO:tasks.workunit.client.0.smithi081.stdout:making blogbench 2024-04-12T21:17:30.519 INFO:tasks.workunit.client.0.smithi081.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-12T21:17:31.538 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether build environment is sane... yes 2024-04-12T21:17:31.563 INFO:tasks.workunit.client.0.smithi081.stdout:checking for gawk... gawk 2024-04-12T21:17:31.595 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether make sets $(MAKE)... yes 2024-04-12T21:17:31.621 INFO:tasks.workunit.client.0.smithi081.stdout:checking for gcc... gcc 2024-04-12T21:17:31.745 INFO:tasks.workunit.client.0.smithi081.stdout:checking for C compiler default output file name... a.out 2024-04-12T21:17:31.752 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether the C compiler works... yes 2024-04-12T21:17:31.757 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether we are cross compiling... no 2024-04-12T21:17:31.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:31 smithi153 ceph-mon[36569]: pgmap v448: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:31 smithi104 ceph-mon[36624]: pgmap v448: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:31.837 INFO:tasks.workunit.client.0.smithi081.stdout:checking for suffix of executables... 2024-04-12T21:17:31.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:31 smithi081 ceph-mon[31453]: pgmap v448: 129 pgs: 129 active+clean; 677 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-04-12T21:17:31.919 INFO:tasks.workunit.client.0.smithi081.stdout:checking for suffix of object files... o 2024-04-12T21:17:32.007 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether we are using the GNU C compiler... yes 2024-04-12T21:17:32.103 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether gcc accepts -g... yes 2024-04-12T21:17:32.230 INFO:tasks.workunit.client.0.smithi081.stdout:checking for gcc option to accept ANSI C... none needed 2024-04-12T21:17:32.326 INFO:tasks.workunit.client.0.smithi081.stdout:checking for style of include used by make... GNU 2024-04-12T21:17:32.527 INFO:tasks.workunit.client.0.smithi081.stdout:checking dependency style of gcc... gcc3 2024-04-12T21:17:32.527 INFO:tasks.workunit.client.0.smithi081.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-12T21:17:32.676 INFO:tasks.workunit.client.0.smithi081.stdout:checking how to run the C preprocessor... gcc -E 2024-04-12T21:17:32.800 INFO:tasks.workunit.client.0.smithi081.stdout:checking for egrep... grep -E 2024-04-12T21:17:32.830 INFO:tasks.workunit.client.0.smithi081.stdout:checking for AIX... no 2024-04-12T21:17:32.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:32 smithi081 ceph-mon[31453]: pgmap v449: 129 pgs: 129 active+clean; 803 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-04-12T21:17:32.972 INFO:tasks.workunit.client.0.smithi081.stdout:checking for library containing strerror... none required 2024-04-12T21:17:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:32 smithi153 ceph-mon[36569]: pgmap v449: 129 pgs: 129 active+clean; 803 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-04-12T21:17:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:32 smithi104 ceph-mon[36624]: pgmap v449: 129 pgs: 129 active+clean; 803 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-04-12T21:17:33.322 INFO:tasks.workunit.client.0.smithi081.stdout:checking for ANSI C header files... yes 2024-04-12T21:17:33.442 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-12T21:17:33.577 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sys/types.h... yes 2024-04-12T21:17:33.711 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sys/stat.h... yes 2024-04-12T21:17:33.837 INFO:tasks.workunit.client.0.smithi081.stdout:checking for stdlib.h... yes 2024-04-12T21:17:33.966 INFO:tasks.workunit.client.0.smithi081.stdout:checking for string.h... yes 2024-04-12T21:17:34.097 INFO:tasks.workunit.client.0.smithi081.stdout:checking for memory.h... yes 2024-04-12T21:17:34.226 INFO:tasks.workunit.client.0.smithi081.stdout:checking for strings.h... yes 2024-04-12T21:17:34.350 INFO:tasks.workunit.client.0.smithi081.stdout:checking for inttypes.h... yes 2024-04-12T21:17:34.481 INFO:tasks.workunit.client.0.smithi081.stdout:checking for stdint.h... yes 2024-04-12T21:17:34.618 INFO:tasks.workunit.client.0.smithi081.stdout:checking for unistd.h... yes 2024-04-12T21:17:34.648 INFO:tasks.workunit.client.0.smithi081.stdout:checking for memory.h... (cached) yes 2024-04-12T21:17:34.668 INFO:tasks.workunit.client.0.smithi081.stdout:checking for strings.h... (cached) yes 2024-04-12T21:17:34.688 INFO:tasks.workunit.client.0.smithi081.stdout:checking for unistd.h... (cached) yes 2024-04-12T21:17:34.825 INFO:tasks.workunit.client.0.smithi081.stdout:checking locale.h usability... yes 2024-04-12T21:17:34.883 INFO:tasks.workunit.client.0.smithi081.stdout:checking locale.h presence... yes 2024-04-12T21:17:34.887 INFO:tasks.workunit.client.0.smithi081.stdout:checking for locale.h... yes 2024-04-12T21:17:34.922 INFO:tasks.workunit.client.0.smithi081.stdout:checking for stdlib.h... (cached) yes 2024-04-12T21:17:34.940 INFO:tasks.workunit.client.0.smithi081.stdout:checking for string.h... (cached) yes 2024-04-12T21:17:35.090 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/socket.h usability... yes 2024-04-12T21:17:35.149 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/socket.h presence... yes 2024-04-12T21:17:35.153 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sys/socket.h... yes 2024-04-12T21:17:35.306 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/time.h usability... yes 2024-04-12T21:17:35.346 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/time.h presence... yes 2024-04-12T21:17:35.348 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sys/time.h... yes 2024-04-12T21:17:35.463 INFO:tasks.workunit.client.0.smithi081.stdout:checking fcntl.h usability... yes 2024-04-12T21:17:35.512 INFO:tasks.workunit.client.0.smithi081.stdout:checking fcntl.h presence... yes 2024-04-12T21:17:35.515 INFO:tasks.workunit.client.0.smithi081.stdout:checking for fcntl.h... yes 2024-04-12T21:17:35.656 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/fcntl.h usability... yes 2024-04-12T21:17:35.707 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/fcntl.h presence... yes 2024-04-12T21:17:35.710 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sys/fcntl.h... yes 2024-04-12T21:17:35.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:35 smithi153 ceph-mon[36569]: pgmap v450: 129 pgs: 129 active+clean; 804 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 12 KiB/s wr, 2 op/s 2024-04-12T21:17:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:35 smithi104 ceph-mon[36624]: pgmap v450: 129 pgs: 129 active+clean; 804 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 12 KiB/s wr, 2 op/s 2024-04-12T21:17:35.885 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/param.h usability... yes 2024-04-12T21:17:35.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:35 smithi081 ceph-mon[31453]: pgmap v450: 129 pgs: 129 active+clean; 804 KiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 12 KiB/s wr, 2 op/s 2024-04-12T21:17:35.952 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/param.h presence... yes 2024-04-12T21:17:35.957 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sys/param.h... yes 2024-04-12T21:17:36.066 INFO:tasks.workunit.client.0.smithi081.stdout:checking ioctl.h usability... no 2024-04-12T21:17:36.126 INFO:tasks.workunit.client.0.smithi081.stdout:checking ioctl.h presence... no 2024-04-12T21:17:36.131 INFO:tasks.workunit.client.0.smithi081.stdout:checking for ioctl.h... no 2024-04-12T21:17:36.249 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/ioctl.h usability... yes 2024-04-12T21:17:36.302 INFO:tasks.workunit.client.0.smithi081.stdout:checking sys/ioctl.h presence... yes 2024-04-12T21:17:36.305 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sys/ioctl.h... yes 2024-04-12T21:17:36.436 INFO:tasks.workunit.client.0.smithi081.stdout:checking return type of signal handlers... void 2024-04-12T21:17:36.542 INFO:tasks.workunit.client.0.smithi081.stdout:checking for size_t... yes 2024-04-12T21:17:36.637 INFO:tasks.workunit.client.0.smithi081.stdout:checking for dev_t... yes 2024-04-12T21:17:36.747 INFO:tasks.workunit.client.0.smithi081.stdout:checking for short... yes 2024-04-12T21:17:36.920 INFO:tasks.workunit.client.0.smithi081.stdout:checking size of short... 2 2024-04-12T21:17:37.036 INFO:tasks.workunit.client.0.smithi081.stdout:checking for int... yes 2024-04-12T21:17:37.189 INFO:tasks.workunit.client.0.smithi081.stdout:checking size of int... 4 2024-04-12T21:17:37.302 INFO:tasks.workunit.client.0.smithi081.stdout:checking for long... yes 2024-04-12T21:17:37.450 INFO:tasks.workunit.client.0.smithi081.stdout:checking size of long... 8 2024-04-12T21:17:37.566 INFO:tasks.workunit.client.0.smithi081.stdout:checking for long long... yes 2024-04-12T21:17:37.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:37.642+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:17:37.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:37.643+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:17:37.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:37.643+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:17:37.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:37.643+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:17:37.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:37.643+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:17:37.719 INFO:tasks.workunit.client.0.smithi081.stdout:checking size of long long... 8 2024-04-12T21:17:37.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:37 smithi153 ceph-mon[36569]: pgmap v451: 129 pgs: 129 active+clean; 2.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 128 KiB/s wr, 12 op/s 2024-04-12T21:17:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:37 smithi104 ceph-mon[36624]: pgmap v451: 129 pgs: 129 active+clean; 2.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 128 KiB/s wr, 12 op/s 2024-04-12T21:17:37.842 INFO:tasks.workunit.client.0.smithi081.stdout:checking for ssize_t... yes 2024-04-12T21:17:37.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:37 smithi081 ceph-mon[31453]: pgmap v451: 129 pgs: 129 active+clean; 2.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 128 KiB/s wr, 12 op/s 2024-04-12T21:17:37.949 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sig_atomic_t... yes 2024-04-12T21:17:38.072 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether gcc needs -traditional... no 2024-04-12T21:17:38.175 INFO:tasks.workunit.client.0.smithi081.stdout:checking for an ANSI C-conforming const... yes 2024-04-12T21:17:38.271 INFO:tasks.workunit.client.0.smithi081.stdout:checking for inline... inline 2024-04-12T21:17:38.398 INFO:tasks.workunit.client.0.smithi081.stdout:checking for working alloca.h... yes 2024-04-12T21:17:38.499 INFO:tasks.workunit.client.0.smithi081.stdout:checking for alloca... yes 2024-04-12T21:17:38.634 INFO:tasks.workunit.client.0.smithi081.stdout:checking for working memcmp... yes 2024-04-12T21:17:38.768 INFO:tasks.workunit.client.0.smithi081.stdout:checking for strftime... yes 2024-04-12T21:17:38.924 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-04-12T21:17:39.058 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether stat accepts an empty string... no 2024-04-12T21:17:39.191 INFO:tasks.workunit.client.0.smithi081.stdout:checking for vprintf... yes 2024-04-12T21:17:39.324 INFO:tasks.workunit.client.0.smithi081.stdout:checking for _doprnt... no 2024-04-12T21:17:39.397 INFO:tasks.workunit.client.0.smithi081.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-12T21:17:39.436 INFO:tasks.workunit.client.0.smithi081.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-12T21:17:39.588 INFO:tasks.workunit.client.0.smithi081.stdout:checking for the pthreads library -lpthreads... no 2024-04-12T21:17:39.717 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether pthreads work without any flags... yes 2024-04-12T21:17:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:39 smithi153 ceph-mon[36569]: pgmap v452: 129 pgs: 129 active+clean; 2.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 128 KiB/s wr, 11 op/s 2024-04-12T21:17:39.825 INFO:tasks.workunit.client.0.smithi081.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-04-12T21:17:39.825 INFO:tasks.workunit.client.0.smithi081.stdout:checking if more special flags are required for pthreads... no 2024-04-12T21:17:39.825 INFO:tasks.workunit.client.0.smithi081.stdout:checking for cc_r... gcc 2024-04-12T21:17:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:39 smithi104 ceph-mon[36624]: pgmap v452: 129 pgs: 129 active+clean; 2.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 128 KiB/s wr, 11 op/s 2024-04-12T21:17:39.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:39 smithi081 ceph-mon[31453]: pgmap v452: 129 pgs: 129 active+clean; 2.1 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 128 KiB/s wr, 11 op/s 2024-04-12T21:17:39.978 INFO:tasks.workunit.client.0.smithi081.stdout:checking for setrlimit... yes 2024-04-12T21:17:40.119 INFO:tasks.workunit.client.0.smithi081.stdout:checking for setproctitle... no 2024-04-12T21:17:40.260 INFO:tasks.workunit.client.0.smithi081.stdout:checking for getopt_long... yes 2024-04-12T21:17:40.401 INFO:tasks.workunit.client.0.smithi081.stdout:checking for strtoull... yes 2024-04-12T21:17:40.552 INFO:tasks.workunit.client.0.smithi081.stdout:checking for strtoq... yes 2024-04-12T21:17:40.700 INFO:tasks.workunit.client.0.smithi081.stdout:checking for snprintf... yes 2024-04-12T21:17:40.840 INFO:tasks.workunit.client.0.smithi081.stdout:checking for vsnprintf... yes 2024-04-12T21:17:40.997 INFO:tasks.workunit.client.0.smithi081.stdout:checking for vfprintf... yes 2024-04-12T21:17:41.145 INFO:tasks.workunit.client.0.smithi081.stdout:checking for gethostname... yes 2024-04-12T21:17:41.300 INFO:tasks.workunit.client.0.smithi081.stdout:checking for setlocale... yes 2024-04-12T21:17:41.458 INFO:tasks.workunit.client.0.smithi081.stdout:checking for sync... yes 2024-04-12T21:17:41.610 INFO:tasks.workunit.client.0.smithi081.stdout:checking whether snprintf is C99 conformant... done 2024-04-12T21:17:41.645 INFO:tasks.workunit.client.0.smithi081.stdout:configure: creating ./config.status 2024-04-12T21:17:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:41 smithi153 ceph-mon[36569]: pgmap v453: 129 pgs: 129 active+clean; 2.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 181 KiB/s wr, 20 op/s 2024-04-12T21:17:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:41 smithi104 ceph-mon[36624]: pgmap v453: 129 pgs: 129 active+clean; 2.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 181 KiB/s wr, 20 op/s 2024-04-12T21:17:41.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:41 smithi081 ceph-mon[31453]: pgmap v453: 129 pgs: 129 active+clean; 2.6 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 181 KiB/s wr, 20 op/s 2024-04-12T21:17:42.066 INFO:tasks.workunit.client.0.smithi081.stdout:config.status: creating Makefile 2024-04-12T21:17:42.124 INFO:tasks.workunit.client.0.smithi081.stdout:config.status: creating src/Makefile 2024-04-12T21:17:42.180 INFO:tasks.workunit.client.0.smithi081.stdout:config.status: creating man/Makefile 2024-04-12T21:17:42.207 INFO:tasks.workunit.client.0.smithi081.stdout:config.status: creating config.h 2024-04-12T21:17:42.327 INFO:tasks.workunit.client.0.smithi081.stdout:config.status: executing depfiles commands 2024-04-12T21:17:42.728 INFO:tasks.workunit.client.0.smithi081.stderr:+ make 2024-04-12T21:17:42.743 INFO:tasks.workunit.client.0.smithi081.stdout:make all-recursive 2024-04-12T21:17:42.748 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-04-12T21:17:42.758 INFO:tasks.workunit.client.0.smithi081.stdout:Making all in src 2024-04-12T21:17:42.769 INFO:tasks.workunit.client.0.smithi081.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-04-12T21:17:42.769 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-04-12T21:17:42.769 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-04-12T21:17:42.924 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-04-12T21:17:42.924 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-04-12T21:17:43.039 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-04-12T21:17:43.039 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-04-12T21:17:43.135 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-04-12T21:17:43.135 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-04-12T21:17:43.230 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-04-12T21:17:43.231 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-04-12T21:17:43.325 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-04-12T21:17:43.325 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-04-12T21:17:43.407 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-04-12T21:17:43.407 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-04-12T21:17:43.532 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-04-12T21:17:43.532 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-04-12T21:17:43.605 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-04-12T21:17:43.605 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-04-12T21:17:43.679 INFO:tasks.workunit.client.0.smithi081.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-04-12T21:17:43.679 INFO:tasks.workunit.client.0.smithi081.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-04-12T21:17:43.759 INFO:tasks.workunit.client.0.smithi081.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-04-12T21:17:43.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:43 smithi153 ceph-mon[36569]: pgmap v454: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 273 KiB/s wr, 26 op/s 2024-04-12T21:17:43.803 INFO:tasks.workunit.client.0.smithi081.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-04-12T21:17:43.805 INFO:tasks.workunit.client.0.smithi081.stdout:Making all in man 2024-04-12T21:17:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:43 smithi104 ceph-mon[36624]: pgmap v454: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 273 KiB/s wr, 26 op/s 2024-04-12T21:17:43.836 INFO:tasks.workunit.client.0.smithi081.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-04-12T21:17:43.836 INFO:tasks.workunit.client.0.smithi081.stdout:make[2]: Nothing to be done for 'all'. 2024-04-12T21:17:43.836 INFO:tasks.workunit.client.0.smithi081.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-04-12T21:17:43.841 INFO:tasks.workunit.client.0.smithi081.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-04-12T21:17:43.844 INFO:tasks.workunit.client.0.smithi081.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-04-12T21:17:43.846 INFO:tasks.workunit.client.0.smithi081.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-04-12T21:17:43.847 INFO:tasks.workunit.client.0.smithi081.stderr:+ cd src 2024-04-12T21:17:43.847 INFO:tasks.workunit.client.0.smithi081.stderr:+ mkdir blogtest_in 2024-04-12T21:17:43.852 INFO:tasks.workunit.client.0.smithi081.stdout:running blogbench 2024-04-12T21:17:43.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ echo 'running blogbench' 2024-04-12T21:17:43.852 INFO:tasks.workunit.client.0.smithi081.stderr:+ ./blogbench -d blogtest_in 2024-04-12T21:17:43.854 INFO:tasks.workunit.client.0.smithi081.stdout: 2024-04-12T21:17:43.854 INFO:tasks.workunit.client.0.smithi081.stdout:Frequency = 10 secs 2024-04-12T21:17:43.854 INFO:tasks.workunit.client.0.smithi081.stdout:Scratch dir = [blogtest_in] 2024-04-12T21:17:43.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:43 smithi081 ceph-mon[31453]: pgmap v454: 129 pgs: 129 active+clean; 3.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 273 KiB/s wr, 26 op/s 2024-04-12T21:17:45.619 INFO:tasks.workunit.client.0.smithi081.stdout:Spawning 3 writers... 2024-04-12T21:17:45.619 INFO:tasks.workunit.client.0.smithi081.stdout:Spawning 1 rewriters... 2024-04-12T21:17:45.619 INFO:tasks.workunit.client.0.smithi081.stdout:Spawning 5 commenters... 2024-04-12T21:17:45.620 INFO:tasks.workunit.client.0.smithi081.stdout:Spawning 100 readers... 2024-04-12T21:17:45.627 INFO:tasks.workunit.client.0.smithi081.stdout:Benchmarking for 30 iterations. 2024-04-12T21:17:45.627 INFO:tasks.workunit.client.0.smithi081.stdout:The test will run during 5 minutes. 2024-04-12T21:17:45.627 INFO:tasks.workunit.client.0.smithi081.stdout: 2024-04-12T21:17:45.627 INFO:tasks.workunit.client.0.smithi081.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-04-12T21:17:45.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:45 smithi153 ceph-mon[36569]: pgmap v455: 129 pgs: 129 active+clean; 3.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 279 KiB/s wr, 30 op/s 2024-04-12T21:17:45.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:45 smithi104 ceph-mon[36624]: pgmap v455: 129 pgs: 129 active+clean; 3.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 279 KiB/s wr, 30 op/s 2024-04-12T21:17:45.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:45 smithi081 ceph-mon[31453]: pgmap v455: 129 pgs: 129 active+clean; 3.7 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 279 KiB/s wr, 30 op/s 2024-04-12T21:17:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:47 smithi153 ceph-mon[36569]: pgmap v456: 129 pgs: 129 active+clean; 4.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 413 KiB/s wr, 50 op/s 2024-04-12T21:17:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:47 smithi104 ceph-mon[36624]: pgmap v456: 129 pgs: 129 active+clean; 4.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 413 KiB/s wr, 50 op/s 2024-04-12T21:17:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:47 smithi081 ceph-mon[31453]: pgmap v456: 129 pgs: 129 active+clean; 4.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 413 KiB/s wr, 50 op/s 2024-04-12T21:17:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:49 smithi153 ceph-mon[36569]: pgmap v457: 129 pgs: 129 active+clean; 4.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 297 KiB/s wr, 40 op/s 2024-04-12T21:17:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:49 smithi104 ceph-mon[36624]: pgmap v457: 129 pgs: 129 active+clean; 4.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 297 KiB/s wr, 40 op/s 2024-04-12T21:17:49.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:49 smithi081 ceph-mon[31453]: pgmap v457: 129 pgs: 129 active+clean; 4.6 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 297 KiB/s wr, 40 op/s 2024-04-12T21:17:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:51 smithi153 ceph-mon[36569]: pgmap v458: 129 pgs: 129 active+clean; 8.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 699 KiB/s wr, 52 op/s 2024-04-12T21:17:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:51 smithi104 ceph-mon[36624]: pgmap v458: 129 pgs: 129 active+clean; 8.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 699 KiB/s wr, 52 op/s 2024-04-12T21:17:51.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:51 smithi081 ceph-mon[31453]: pgmap v458: 129 pgs: 129 active+clean; 8.7 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 699 KiB/s wr, 52 op/s 2024-04-12T21:17:52.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:52.657+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:17:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:52.658+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:17:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:52.658+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:17:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:52.658+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:17:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:17:52.658+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:17:53.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:53 smithi153 ceph-mon[36569]: pgmap v459: 129 pgs: 129 active+clean; 16 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-04-12T21:17:53.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:53 smithi104 ceph-mon[36624]: pgmap v459: 129 pgs: 129 active+clean; 16 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-04-12T21:17:53.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:53 smithi081 ceph-mon[31453]: pgmap v459: 129 pgs: 129 active+clean; 16 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 1.2 MiB/s wr, 56 op/s 2024-04-12T21:17:55.628 INFO:tasks.workunit.client.0.smithi081.stdout: 15 291599 1132 204519 957 39040 971 2024-04-12T21:17:55.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:55 smithi153 ceph-mon[36569]: pgmap v460: 129 pgs: 129 active+clean; 23 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 68 op/s 2024-04-12T21:17:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:55 smithi104 ceph-mon[36624]: pgmap v460: 129 pgs: 129 active+clean; 23 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 68 op/s 2024-04-12T21:17:55.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:55 smithi081 ceph-mon[31453]: pgmap v460: 129 pgs: 129 active+clean; 23 MiB data, 441 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 68 op/s 2024-04-12T21:17:56.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:56 smithi081 ceph-mon[31453]: pgmap v461: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 84 op/s 2024-04-12T21:17:57.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:56 smithi153 ceph-mon[36569]: pgmap v461: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 84 op/s 2024-04-12T21:17:57.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:56 smithi104 ceph-mon[36624]: pgmap v461: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 84 op/s 2024-04-12T21:17:59.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:17:59 smithi153 ceph-mon[36569]: pgmap v462: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 63 op/s 2024-04-12T21:17:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:17:59 smithi104 ceph-mon[36624]: pgmap v462: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 63 op/s 2024-04-12T21:17:59.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:17:59 smithi081 ceph-mon[31453]: pgmap v462: 129 pgs: 129 active+clean; 30 MiB data, 461 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 63 op/s 2024-04-12T21:18:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:01 smithi153 ceph-mon[36569]: pgmap v463: 129 pgs: 129 active+clean; 55 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 164 op/s 2024-04-12T21:18:01.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:01 smithi104 ceph-mon[36624]: pgmap v463: 129 pgs: 129 active+clean; 55 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 164 op/s 2024-04-12T21:18:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:01 smithi081 ceph-mon[31453]: pgmap v463: 129 pgs: 129 active+clean; 55 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.6 MiB/s wr, 164 op/s 2024-04-12T21:18:03.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:03 smithi153 ceph-mon[36569]: pgmap v464: 129 pgs: 129 active+clean; 65 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 211 op/s 2024-04-12T21:18:03.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:03 smithi104 ceph-mon[36624]: pgmap v464: 129 pgs: 129 active+clean; 65 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 211 op/s 2024-04-12T21:18:03.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:03 smithi081 ceph-mon[31453]: pgmap v464: 129 pgs: 129 active+clean; 65 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 211 op/s 2024-04-12T21:18:05.637 INFO:tasks.workunit.client.0.smithi081.stdout: 25 605059 492 431754 525 145733 1090 2024-04-12T21:18:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:05 smithi153 ceph-mon[36569]: pgmap v465: 129 pgs: 129 active+clean; 70 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.1 MiB/s wr, 213 op/s 2024-04-12T21:18:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:05 smithi104 ceph-mon[36624]: pgmap v465: 129 pgs: 129 active+clean; 70 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.1 MiB/s wr, 213 op/s 2024-04-12T21:18:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:05 smithi081 ceph-mon[31453]: pgmap v465: 129 pgs: 129 active+clean; 70 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.1 MiB/s wr, 213 op/s 2024-04-12T21:18:06.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:18:06.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:18:06.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:18:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:07.684+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:18:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:07.694+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:18:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:07.694+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:18:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:07.694+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:18:07.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:07.694+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:18:07.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:07 smithi153 ceph-mon[36569]: pgmap v466: 129 pgs: 129 active+clean; 75 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 204 op/s 2024-04-12T21:18:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:07 smithi104 ceph-mon[36624]: pgmap v466: 129 pgs: 129 active+clean; 75 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 204 op/s 2024-04-12T21:18:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:07 smithi081 ceph-mon[31453]: pgmap v466: 129 pgs: 129 active+clean; 75 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 204 op/s 2024-04-12T21:18:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:09 smithi104 ceph-mon[36624]: pgmap v467: 129 pgs: 129 active+clean; 75 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 184 op/s 2024-04-12T21:18:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:18:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:18:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:18:09.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:09 smithi081 ceph-mon[31453]: pgmap v467: 129 pgs: 129 active+clean; 75 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 184 op/s 2024-04-12T21:18:09.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:18:09.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:18:09.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:18:10.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:09 smithi153 ceph-mon[36569]: pgmap v467: 129 pgs: 129 active+clean; 75 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.2 MiB/s wr, 184 op/s 2024-04-12T21:18:10.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:18:10.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:18:10.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:18:10.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:10 smithi104 ceph-mon[36624]: pgmap v468: 129 pgs: 129 active+clean; 75 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 192 op/s 2024-04-12T21:18:10.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:10 smithi081 ceph-mon[31453]: pgmap v468: 129 pgs: 129 active+clean; 75 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 192 op/s 2024-04-12T21:18:11.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:10 smithi153 ceph-mon[36569]: pgmap v468: 129 pgs: 129 active+clean; 75 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 192 op/s 2024-04-12T21:18:13.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:13 smithi153 ceph-mon[36569]: pgmap v469: 129 pgs: 129 active+clean; 82 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 98 op/s 2024-04-12T21:18:13.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:13 smithi104 ceph-mon[36624]: pgmap v469: 129 pgs: 129 active+clean; 82 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 98 op/s 2024-04-12T21:18:13.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:13 smithi081 ceph-mon[31453]: pgmap v469: 129 pgs: 129 active+clean; 82 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.6 MiB/s wr, 98 op/s 2024-04-12T21:18:15.637 INFO:tasks.workunit.client.0.smithi081.stdout: 34 720806 623 530962 498 260092 1417 2024-04-12T21:18:15.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:15 smithi153 ceph-mon[36569]: pgmap v470: 129 pgs: 129 active+clean; 82 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-04-12T21:18:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:15 smithi104 ceph-mon[36624]: pgmap v470: 129 pgs: 129 active+clean; 82 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-04-12T21:18:15.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:15 smithi081 ceph-mon[31453]: pgmap v470: 129 pgs: 129 active+clean; 82 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 47 op/s 2024-04-12T21:18:17.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:17 smithi153 ceph-mon[36569]: pgmap v471: 129 pgs: 129 active+clean; 89 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 43 op/s 2024-04-12T21:18:17.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:17 smithi104 ceph-mon[36624]: pgmap v471: 129 pgs: 129 active+clean; 89 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 43 op/s 2024-04-12T21:18:17.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:17 smithi081 ceph-mon[31453]: pgmap v471: 129 pgs: 129 active+clean; 89 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.9 MiB/s wr, 43 op/s 2024-04-12T21:18:19.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:19 smithi153 ceph-mon[36569]: pgmap v472: 129 pgs: 129 active+clean; 89 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 35 op/s 2024-04-12T21:18:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:19 smithi104 ceph-mon[36624]: pgmap v472: 129 pgs: 129 active+clean; 89 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 35 op/s 2024-04-12T21:18:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:19 smithi081 ceph-mon[31453]: pgmap v472: 129 pgs: 129 active+clean; 89 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 35 op/s 2024-04-12T21:18:21.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:21 smithi153 ceph-mon[36569]: pgmap v473: 129 pgs: 129 active+clean; 95 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 49 op/s 2024-04-12T21:18:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:21 smithi104 ceph-mon[36624]: pgmap v473: 129 pgs: 129 active+clean; 95 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 49 op/s 2024-04-12T21:18:21.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:21 smithi081 ceph-mon[31453]: pgmap v473: 129 pgs: 129 active+clean; 95 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 49 op/s 2024-04-12T21:18:22.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:22.711+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:18:22.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:22.713+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:18:22.709 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:22.713+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:18:22.709 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:22.713+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:18:22.709 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:22.713+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:18:23.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:23 smithi153 ceph-mon[36569]: pgmap v474: 129 pgs: 129 active+clean; 95 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 48 op/s 2024-04-12T21:18:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:23 smithi104 ceph-mon[36624]: pgmap v474: 129 pgs: 129 active+clean; 95 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 48 op/s 2024-04-12T21:18:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:23 smithi081 ceph-mon[31453]: pgmap v474: 129 pgs: 129 active+clean; 95 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 48 op/s 2024-04-12T21:18:25.637 INFO:tasks.workunit.client.0.smithi081.stdout: 42 644190 482 485563 419 272031 1230 2024-04-12T21:18:25.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:25 smithi153 ceph-mon[36569]: pgmap v475: 129 pgs: 129 active+clean; 101 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-04-12T21:18:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:25 smithi104 ceph-mon[36624]: pgmap v475: 129 pgs: 129 active+clean; 101 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-04-12T21:18:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:25 smithi081 ceph-mon[31453]: pgmap v475: 129 pgs: 129 active+clean; 101 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 79 op/s 2024-04-12T21:18:27.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:27 smithi153 ceph-mon[36569]: pgmap v476: 129 pgs: 129 active+clean; 119 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 154 op/s 2024-04-12T21:18:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:27 smithi104 ceph-mon[36624]: pgmap v476: 129 pgs: 129 active+clean; 119 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 154 op/s 2024-04-12T21:18:27.933 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:27 smithi081 ceph-mon[31453]: pgmap v476: 129 pgs: 129 active+clean; 119 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 154 op/s 2024-04-12T21:18:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:28 smithi081 ceph-mon[31453]: pgmap v477: 129 pgs: 129 active+clean; 122 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 168 op/s 2024-04-12T21:18:29.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:28 smithi153 ceph-mon[36569]: pgmap v477: 129 pgs: 129 active+clean; 122 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 168 op/s 2024-04-12T21:18:29.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:28 smithi104 ceph-mon[36624]: pgmap v477: 129 pgs: 129 active+clean; 122 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 168 op/s 2024-04-12T21:18:31.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:31 smithi153 ceph-mon[36569]: pgmap v478: 129 pgs: 129 active+clean; 144 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 288 op/s 2024-04-12T21:18:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:31 smithi104 ceph-mon[36624]: pgmap v478: 129 pgs: 129 active+clean; 144 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 288 op/s 2024-04-12T21:18:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:31 smithi081 ceph-mon[31453]: pgmap v478: 129 pgs: 129 active+clean; 144 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.2 MiB/s wr, 288 op/s 2024-04-12T21:18:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:33 smithi153 ceph-mon[36569]: pgmap v479: 129 pgs: 129 active+clean; 189 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 8.5 MiB/s wr, 532 op/s 2024-04-12T21:18:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:33 smithi104 ceph-mon[36624]: pgmap v479: 129 pgs: 129 active+clean; 189 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 8.5 MiB/s wr, 532 op/s 2024-04-12T21:18:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:33 smithi081 ceph-mon[31453]: pgmap v479: 129 pgs: 129 active+clean; 189 MiB data, 877 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 8.5 MiB/s wr, 532 op/s 2024-04-12T21:18:35.637 INFO:tasks.workunit.client.0.smithi081.stdout: 45 540848 33 404067 164 247775 659 2024-04-12T21:18:35.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:35 smithi153 ceph-mon[36569]: pgmap v480: 129 pgs: 129 active+clean; 216 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 11 MiB/s wr, 654 op/s 2024-04-12T21:18:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:35 smithi104 ceph-mon[36624]: pgmap v480: 129 pgs: 129 active+clean; 216 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 11 MiB/s wr, 654 op/s 2024-04-12T21:18:35.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:35 smithi081 ceph-mon[31453]: pgmap v480: 129 pgs: 129 active+clean; 216 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 11 MiB/s wr, 654 op/s 2024-04-12T21:18:37.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:37.753+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:18:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:37.764+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:18:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:37.764+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:18:37.759 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:37.764+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:18:37.760 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:37.764+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:18:37.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:37 smithi153 ceph-mon[36569]: pgmap v481: 129 pgs: 129 active+clean; 219 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 757 op/s 2024-04-12T21:18:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:37 smithi104 ceph-mon[36624]: pgmap v481: 129 pgs: 129 active+clean; 219 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 757 op/s 2024-04-12T21:18:37.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:37 smithi081 ceph-mon[31453]: pgmap v481: 129 pgs: 129 active+clean; 219 MiB data, 1003 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 11 MiB/s wr, 757 op/s 2024-04-12T21:18:39.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:39 smithi153 ceph-mon[36569]: pgmap v482: 129 pgs: 129 active+clean; 222 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.9 MiB/s wr, 695 op/s 2024-04-12T21:18:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:39 smithi104 ceph-mon[36624]: pgmap v482: 129 pgs: 129 active+clean; 222 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.9 MiB/s wr, 695 op/s 2024-04-12T21:18:39.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:39 smithi081 ceph-mon[31453]: pgmap v482: 129 pgs: 129 active+clean; 222 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 9.9 MiB/s wr, 695 op/s 2024-04-12T21:18:41.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:41 smithi153 ceph-mon[36569]: pgmap v483: 129 pgs: 129 active+clean; 223 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.9 MiB/s wr, 721 op/s 2024-04-12T21:18:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:41 smithi104 ceph-mon[36624]: pgmap v483: 129 pgs: 129 active+clean; 223 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.9 MiB/s wr, 721 op/s 2024-04-12T21:18:41.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:41 smithi081 ceph-mon[31453]: pgmap v483: 129 pgs: 129 active+clean; 223 MiB data, 1024 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 9.9 MiB/s wr, 721 op/s 2024-04-12T21:18:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:43 smithi104 ceph-mon[36624]: pgmap v484: 129 pgs: 129 active+clean; 226 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 635 op/s 2024-04-12T21:18:43.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:43 smithi081 ceph-mon[31453]: pgmap v484: 129 pgs: 129 active+clean; 226 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 635 op/s 2024-04-12T21:18:44.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:43 smithi153 ceph-mon[36569]: pgmap v484: 129 pgs: 129 active+clean; 226 MiB data, 1019 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 8.4 MiB/s wr, 635 op/s 2024-04-12T21:18:45.647 INFO:tasks.workunit.client.0.smithi081.stdout: 53 620130 450 477434 333 331924 1477 2024-04-12T21:18:45.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:45 smithi104 ceph-mon[36624]: pgmap v485: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 387 op/s 2024-04-12T21:18:45.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:45 smithi081 ceph-mon[31453]: pgmap v485: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 387 op/s 2024-04-12T21:18:46.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:45 smithi153 ceph-mon[36569]: pgmap v485: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.5 MiB/s wr, 387 op/s 2024-04-12T21:18:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:47 smithi104 ceph-mon[36624]: pgmap v486: 129 pgs: 129 active+clean; 229 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 293 op/s 2024-04-12T21:18:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:47 smithi081 ceph-mon[31453]: pgmap v486: 129 pgs: 129 active+clean; 229 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 293 op/s 2024-04-12T21:18:48.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:47 smithi153 ceph-mon[36569]: pgmap v486: 129 pgs: 129 active+clean; 229 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 293 op/s 2024-04-12T21:18:48.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:48 smithi104 ceph-mon[36624]: pgmap v487: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 153 op/s 2024-04-12T21:18:48.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:48 smithi081 ceph-mon[31453]: pgmap v487: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 153 op/s 2024-04-12T21:18:49.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:48 smithi153 ceph-mon[36569]: pgmap v487: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 153 op/s 2024-04-12T21:18:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:51 smithi153 ceph-mon[36569]: pgmap v488: 129 pgs: 129 active+clean; 233 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-04-12T21:18:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:51 smithi104 ceph-mon[36624]: pgmap v488: 129 pgs: 129 active+clean; 233 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-04-12T21:18:51.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:51 smithi081 ceph-mon[31453]: pgmap v488: 129 pgs: 129 active+clean; 233 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 143 op/s 2024-04-12T21:18:52.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:52.786+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:18:52.798 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:52.802+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:18:52.798 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:52.802+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:18:52.798 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:52.802+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:18:52.798 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:18:52.802+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:18:53.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:53 smithi153 ceph-mon[36569]: pgmap v489: 129 pgs: 129 active+clean; 236 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-04-12T21:18:53.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:53 smithi104 ceph-mon[36624]: pgmap v489: 129 pgs: 129 active+clean; 236 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-04-12T21:18:53.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:53 smithi081 ceph-mon[31453]: pgmap v489: 129 pgs: 129 active+clean; 236 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 102 op/s 2024-04-12T21:18:55.648 INFO:tasks.workunit.client.0.smithi081.stdout: 63 691357 373 497821 421 379899 1445 2024-04-12T21:18:55.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:55 smithi153 ceph-mon[36569]: pgmap v490: 129 pgs: 129 active+clean; 240 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 73 op/s 2024-04-12T21:18:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:55 smithi104 ceph-mon[36624]: pgmap v490: 129 pgs: 129 active+clean; 240 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 73 op/s 2024-04-12T21:18:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:55 smithi081 ceph-mon[31453]: pgmap v490: 129 pgs: 129 active+clean; 240 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 73 op/s 2024-04-12T21:18:57.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:57 smithi153 ceph-mon[36569]: pgmap v491: 129 pgs: 129 active+clean; 246 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-04-12T21:18:57.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:57 smithi104 ceph-mon[36624]: pgmap v491: 129 pgs: 129 active+clean; 246 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-04-12T21:18:57.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:57 smithi081 ceph-mon[31453]: pgmap v491: 129 pgs: 129 active+clean; 246 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 97 op/s 2024-04-12T21:18:59.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:18:59 smithi153 ceph-mon[36569]: pgmap v492: 129 pgs: 129 active+clean; 247 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 69 op/s 2024-04-12T21:18:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:18:59 smithi104 ceph-mon[36624]: pgmap v492: 129 pgs: 129 active+clean; 247 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 69 op/s 2024-04-12T21:18:59.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:18:59 smithi081 ceph-mon[31453]: pgmap v492: 129 pgs: 129 active+clean; 247 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 69 op/s 2024-04-12T21:19:01.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:01 smithi153 ceph-mon[36569]: pgmap v493: 129 pgs: 129 active+clean; 248 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 97 op/s 2024-04-12T21:19:01.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:01 smithi104 ceph-mon[36624]: pgmap v493: 129 pgs: 129 active+clean; 248 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 97 op/s 2024-04-12T21:19:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:01 smithi081 ceph-mon[31453]: pgmap v493: 129 pgs: 129 active+clean; 248 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.0 MiB/s wr, 97 op/s 2024-04-12T21:19:03.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:03 smithi153 ceph-mon[36569]: pgmap v494: 129 pgs: 129 active+clean; 256 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 130 op/s 2024-04-12T21:19:03.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:03 smithi104 ceph-mon[36624]: pgmap v494: 129 pgs: 129 active+clean; 256 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 130 op/s 2024-04-12T21:19:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:03 smithi081 ceph-mon[31453]: pgmap v494: 129 pgs: 129 active+clean; 256 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 130 op/s 2024-04-12T21:19:05.648 INFO:tasks.workunit.client.0.smithi081.stdout: 71 669890 459 476109 339 362264 1424 2024-04-12T21:19:05.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:05 smithi153 ceph-mon[36569]: pgmap v495: 129 pgs: 129 active+clean; 262 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 149 op/s 2024-04-12T21:19:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:05 smithi104 ceph-mon[36624]: pgmap v495: 129 pgs: 129 active+clean; 262 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 149 op/s 2024-04-12T21:19:05.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:05 smithi081 ceph-mon[31453]: pgmap v495: 129 pgs: 129 active+clean; 262 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 149 op/s 2024-04-12T21:19:07.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:07.822+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:19:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:07 smithi104 ceph-mon[36624]: pgmap v496: 129 pgs: 129 active+clean; 264 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 178 op/s 2024-04-12T21:19:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:07.840+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:19:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:07.840+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:19:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:07.840+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:19:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:07.840+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:19:07.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:07 smithi081 ceph-mon[31453]: pgmap v496: 129 pgs: 129 active+clean; 264 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 178 op/s 2024-04-12T21:19:08.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:07 smithi153 ceph-mon[36569]: pgmap v496: 129 pgs: 129 active+clean; 264 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.6 MiB/s wr, 178 op/s 2024-04-12T21:19:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:09 smithi104 ceph-mon[36624]: pgmap v497: 129 pgs: 129 active+clean; 264 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 146 op/s 2024-04-12T21:19:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:19:09.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:09 smithi081 ceph-mon[31453]: pgmap v497: 129 pgs: 129 active+clean; 264 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 146 op/s 2024-04-12T21:19:09.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:19:10.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:09 smithi153 ceph-mon[36569]: pgmap v497: 129 pgs: 129 active+clean; 264 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 146 op/s 2024-04-12T21:19:10.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:19:10.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:10 smithi081 ceph-mon[31453]: pgmap v498: 129 pgs: 129 active+clean; 270 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 165 op/s 2024-04-12T21:19:11.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:10 smithi153 ceph-mon[36569]: pgmap v498: 129 pgs: 129 active+clean; 270 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 165 op/s 2024-04-12T21:19:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:10 smithi104 ceph-mon[36624]: pgmap v498: 129 pgs: 129 active+clean; 270 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.5 MiB/s wr, 165 op/s 2024-04-12T21:19:12.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:12.457+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40034f0 con 0x7f1704076980 2024-04-12T21:19:12.474 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:12.479+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40034f0 con 0x7f1704076980 2024-04-12T21:19:12.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:12.489+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00034c0 con 0x563c9e245d20 2024-04-12T21:19:12.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:12.499+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x563c9e2c1e50 con 0x563c9e245d20 2024-04-12T21:19:13.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:13 smithi153 ceph-mon[36569]: pgmap v499: 129 pgs: 129 active+clean; 286 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 228 op/s 2024-04-12T21:19:13.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:13 smithi104 ceph-mon[36624]: pgmap v499: 129 pgs: 129 active+clean; 286 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 228 op/s 2024-04-12T21:19:13.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:13 smithi081 ceph-mon[31453]: pgmap v499: 129 pgs: 129 active+clean; 286 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.7 MiB/s wr, 228 op/s 2024-04-12T21:19:15.665 INFO:tasks.workunit.client.0.smithi081.stdout: 74 596497 220 419314 223 338511 1210 2024-04-12T21:19:15.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:15 smithi153 ceph-mon[36569]: pgmap v500: 129 pgs: 129 active+clean; 291 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-04-12T21:19:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:15 smithi104 ceph-mon[36624]: pgmap v500: 129 pgs: 129 active+clean; 291 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-04-12T21:19:15.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:15 smithi081 ceph-mon[31453]: pgmap v500: 129 pgs: 129 active+clean; 291 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.5 MiB/s wr, 240 op/s 2024-04-12T21:19:17.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:17 smithi153 ceph-mon[36569]: pgmap v501: 129 pgs: 129 active+clean; 306 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 395 op/s 2024-04-12T21:19:17.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:17 smithi104 ceph-mon[36624]: pgmap v501: 129 pgs: 129 active+clean; 306 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 395 op/s 2024-04-12T21:19:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:17 smithi081 ceph-mon[31453]: pgmap v501: 129 pgs: 129 active+clean; 306 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.8 MiB/s wr, 395 op/s 2024-04-12T21:19:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:19 smithi153 ceph-mon[36569]: pgmap v502: 129 pgs: 129 active+clean; 316 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 436 op/s 2024-04-12T21:19:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:19 smithi104 ceph-mon[36624]: pgmap v502: 129 pgs: 129 active+clean; 316 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 436 op/s 2024-04-12T21:19:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:19 smithi081 ceph-mon[31453]: pgmap v502: 129 pgs: 129 active+clean; 316 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.2 MiB/s wr, 436 op/s 2024-04-12T21:19:21.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:21 smithi153 ceph-mon[36569]: pgmap v503: 129 pgs: 129 active+clean; 339 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.2 MiB/s wr, 554 op/s 2024-04-12T21:19:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:19:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:19:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:19:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:21 smithi104 ceph-mon[36624]: pgmap v503: 129 pgs: 129 active+clean; 339 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.2 MiB/s wr, 554 op/s 2024-04-12T21:19:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:19:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:19:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:19:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:21 smithi081 ceph-mon[31453]: pgmap v503: 129 pgs: 129 active+clean; 339 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.2 MiB/s wr, 554 op/s 2024-04-12T21:19:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:19:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:19:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:19:22.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:22.857+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:19:22.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:22.861+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:19:22.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:22.861+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:19:22.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:22.861+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:19:22.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:22.861+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:19:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:23 smithi104 ceph-mon[36624]: pgmap v504: 129 pgs: 129 active+clean; 374 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.8 MiB/s wr, 764 op/s 2024-04-12T21:19:23.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:23 smithi081 ceph-mon[31453]: pgmap v504: 129 pgs: 129 active+clean; 374 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.8 MiB/s wr, 764 op/s 2024-04-12T21:19:24.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:23 smithi153 ceph-mon[36569]: pgmap v504: 129 pgs: 129 active+clean; 374 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 9.8 MiB/s wr, 764 op/s 2024-04-12T21:19:25.668 INFO:tasks.workunit.client.0.smithi081.stdout: 77 440393 251 313017 101 255595 635 2024-04-12T21:19:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:25 smithi104 ceph-mon[36624]: pgmap v505: 129 pgs: 129 active+clean; 378 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 8.8 MiB/s wr, 684 op/s 2024-04-12T21:19:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:25 smithi081 ceph-mon[31453]: pgmap v505: 129 pgs: 129 active+clean; 378 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 8.8 MiB/s wr, 684 op/s 2024-04-12T21:19:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:25 smithi153 ceph-mon[36569]: pgmap v505: 129 pgs: 129 active+clean; 378 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 8.8 MiB/s wr, 684 op/s 2024-04-12T21:19:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:27 smithi104 ceph-mon[36624]: pgmap v506: 129 pgs: 129 active+clean; 380 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.2 MiB/s wr, 768 op/s 2024-04-12T21:19:27.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:27 smithi081 ceph-mon[31453]: pgmap v506: 129 pgs: 129 active+clean; 380 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.2 MiB/s wr, 768 op/s 2024-04-12T21:19:28.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:27 smithi153 ceph-mon[36569]: pgmap v506: 129 pgs: 129 active+clean; 380 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.2 MiB/s wr, 768 op/s 2024-04-12T21:19:29.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:29 smithi081 ceph-mon[31453]: pgmap v507: 129 pgs: 129 active+clean; 382 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.6 MiB/s wr, 596 op/s 2024-04-12T21:19:30.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:29 smithi153 ceph-mon[36569]: pgmap v507: 129 pgs: 129 active+clean; 382 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.6 MiB/s wr, 596 op/s 2024-04-12T21:19:30.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:29 smithi104 ceph-mon[36624]: pgmap v507: 129 pgs: 129 active+clean; 382 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.6 MiB/s wr, 596 op/s 2024-04-12T21:19:30.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:30 smithi081 ceph-mon[31453]: pgmap v508: 129 pgs: 129 active+clean; 381 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.8 MiB/s wr, 545 op/s 2024-04-12T21:19:31.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:30 smithi153 ceph-mon[36569]: pgmap v508: 129 pgs: 129 active+clean; 381 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.8 MiB/s wr, 545 op/s 2024-04-12T21:19:31.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:30 smithi104 ceph-mon[36624]: pgmap v508: 129 pgs: 129 active+clean; 381 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 6.8 MiB/s wr, 545 op/s 2024-04-12T21:19:33.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:33 smithi153 ceph-mon[36569]: pgmap v509: 129 pgs: 129 active+clean; 383 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 454 op/s 2024-04-12T21:19:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:33 smithi104 ceph-mon[36624]: pgmap v509: 129 pgs: 129 active+clean; 383 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 454 op/s 2024-04-12T21:19:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:33 smithi081 ceph-mon[31453]: pgmap v509: 129 pgs: 129 active+clean; 383 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 5.0 MiB/s wr, 454 op/s 2024-04-12T21:19:35.666 INFO:tasks.workunit.client.0.smithi081.stdout: 86 667389 525 466396 557 370564 1424 2024-04-12T21:19:35.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:35 smithi153 ceph-mon[36569]: pgmap v510: 129 pgs: 129 active+clean; 386 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 227 op/s 2024-04-12T21:19:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:35 smithi104 ceph-mon[36624]: pgmap v510: 129 pgs: 129 active+clean; 386 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 227 op/s 2024-04-12T21:19:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:35 smithi081 ceph-mon[31453]: pgmap v510: 129 pgs: 129 active+clean; 386 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 227 op/s 2024-04-12T21:19:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:37 smithi104 ceph-mon[36624]: pgmap v511: 129 pgs: 129 active+clean; 392 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 244 op/s 2024-04-12T21:19:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:37.875+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:19:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:37.876+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:19:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:37.876+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:19:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:37.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:19:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:37.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:19:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:37 smithi081 ceph-mon[31453]: pgmap v511: 129 pgs: 129 active+clean; 392 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 244 op/s 2024-04-12T21:19:38.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:37 smithi153 ceph-mon[36569]: pgmap v511: 129 pgs: 129 active+clean; 392 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.5 MiB/s wr, 244 op/s 2024-04-12T21:19:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:39 smithi104 ceph-mon[36624]: pgmap v512: 129 pgs: 129 active+clean; 395 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-04-12T21:19:39.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:39 smithi081 ceph-mon[31453]: pgmap v512: 129 pgs: 129 active+clean; 395 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-04-12T21:19:40.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:39 smithi153 ceph-mon[36569]: pgmap v512: 129 pgs: 129 active+clean; 395 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.0 MiB/s wr, 132 op/s 2024-04-12T21:19:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:41 smithi104 ceph-mon[36624]: pgmap v513: 129 pgs: 129 active+clean; 406 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 189 op/s 2024-04-12T21:19:41.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:41 smithi081 ceph-mon[31453]: pgmap v513: 129 pgs: 129 active+clean; 406 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 189 op/s 2024-04-12T21:19:42.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:41 smithi153 ceph-mon[36569]: pgmap v513: 129 pgs: 129 active+clean; 406 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 189 op/s 2024-04-12T21:19:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:43 smithi104 ceph-mon[36624]: pgmap v514: 129 pgs: 129 active+clean; 428 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 308 op/s 2024-04-12T21:19:43.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:43 smithi081 ceph-mon[31453]: pgmap v514: 129 pgs: 129 active+clean; 428 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 308 op/s 2024-04-12T21:19:44.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:43 smithi153 ceph-mon[36569]: pgmap v514: 129 pgs: 129 active+clean; 428 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.7 MiB/s wr, 308 op/s 2024-04-12T21:19:45.666 INFO:tasks.workunit.client.0.smithi081.stdout: 91 543448 150 389536 273 323340 856 2024-04-12T21:19:45.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:45 smithi081 ceph-mon[31453]: pgmap v515: 129 pgs: 129 active+clean; 434 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 314 op/s 2024-04-12T21:19:46.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:45 smithi153 ceph-mon[36569]: pgmap v515: 129 pgs: 129 active+clean; 434 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 314 op/s 2024-04-12T21:19:46.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:45 smithi104 ceph-mon[36624]: pgmap v515: 129 pgs: 129 active+clean; 434 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 5.0 MiB/s wr, 314 op/s 2024-04-12T21:19:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:47 smithi081 ceph-mon[31453]: pgmap v516: 129 pgs: 129 active+clean; 464 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.5 MiB/s wr, 502 op/s 2024-04-12T21:19:48.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:47 smithi153 ceph-mon[36569]: pgmap v516: 129 pgs: 129 active+clean; 464 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.5 MiB/s wr, 502 op/s 2024-04-12T21:19:48.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:47 smithi104 ceph-mon[36624]: pgmap v516: 129 pgs: 129 active+clean; 464 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 7.5 MiB/s wr, 502 op/s 2024-04-12T21:19:49.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:49 smithi081 ceph-mon[31453]: pgmap v517: 129 pgs: 129 active+clean; 463 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 478 op/s 2024-04-12T21:19:50.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:49 smithi153 ceph-mon[36569]: pgmap v517: 129 pgs: 129 active+clean; 463 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 478 op/s 2024-04-12T21:19:50.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:49 smithi104 ceph-mon[36624]: pgmap v517: 129 pgs: 129 active+clean; 463 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.8 MiB/s wr, 478 op/s 2024-04-12T21:19:50.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:50 smithi081 ceph-mon[31453]: pgmap v518: 129 pgs: 129 active+clean; 464 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 468 op/s 2024-04-12T21:19:51.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:50 smithi153 ceph-mon[36569]: pgmap v518: 129 pgs: 129 active+clean; 464 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 468 op/s 2024-04-12T21:19:51.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:50 smithi104 ceph-mon[36624]: pgmap v518: 129 pgs: 129 active+clean; 464 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 6.6 MiB/s wr, 468 op/s 2024-04-12T21:19:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:52.894+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:19:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:52.895+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:19:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:52.895+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:19:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:52.895+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:19:52.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:19:52.895+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:19:53.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:53 smithi153 ceph-mon[36569]: pgmap v519: 129 pgs: 129 active+clean; 469 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 423 op/s 2024-04-12T21:19:53.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:53 smithi104 ceph-mon[36624]: pgmap v519: 129 pgs: 129 active+clean; 469 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 423 op/s 2024-04-12T21:19:53.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:53 smithi081 ceph-mon[31453]: pgmap v519: 129 pgs: 129 active+clean; 469 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 6.2 MiB/s wr, 423 op/s 2024-04-12T21:19:55.687 INFO:tasks.workunit.client.0.smithi081.stdout: 98 661903 491 479017 442 389575 1676 2024-04-12T21:19:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:55 smithi104 ceph-mon[36624]: pgmap v520: 129 pgs: 129 active+clean; 470 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 286 op/s 2024-04-12T21:19:55.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:55 smithi081 ceph-mon[31453]: pgmap v520: 129 pgs: 129 active+clean; 470 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 286 op/s 2024-04-12T21:19:56.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:55 smithi153 ceph-mon[36569]: pgmap v520: 129 pgs: 129 active+clean; 470 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 286 op/s 2024-04-12T21:19:57.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:57 smithi104 ceph-mon[36624]: pgmap v521: 129 pgs: 129 active+clean; 472 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 288 op/s 2024-04-12T21:19:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:57 smithi081 ceph-mon[31453]: pgmap v521: 129 pgs: 129 active+clean; 472 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 288 op/s 2024-04-12T21:19:58.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:57 smithi153 ceph-mon[36569]: pgmap v521: 129 pgs: 129 active+clean; 472 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.1 MiB/s wr, 288 op/s 2024-04-12T21:19:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:19:59 smithi104 ceph-mon[36624]: pgmap v522: 129 pgs: 129 active+clean; 474 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-04-12T21:19:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:19:59 smithi081 ceph-mon[31453]: pgmap v522: 129 pgs: 129 active+clean; 474 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-04-12T21:20:00.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:19:59 smithi153 ceph-mon[36569]: pgmap v522: 129 pgs: 129 active+clean; 474 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 95 op/s 2024-04-12T21:20:00.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:00 smithi104 ceph-mon[36624]: overall HEALTH_OK 2024-04-12T21:20:00.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:00 smithi081 ceph-mon[31453]: overall HEALTH_OK 2024-04-12T21:20:01.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:00 smithi153 ceph-mon[36569]: overall HEALTH_OK 2024-04-12T21:20:01.708 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-mon[36569]: pgmap v523: 129 pgs: 129 active+clean; 477 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 99 op/s 2024-04-12T21:20:01.708 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:20:01.708 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: *** Caught signal (Aborted) ** 2024-04-12T21:20:01.708 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: in thread 7f463b418700 thread_name:ms_dispatch 2024-04-12T21:20:01.708 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:01.708 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4648fd2d20] 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2: gsignal() 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 3: abort() 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564fb2fbf268] 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564fb305b0fc] 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564fb338ff0a] 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 7: (MDLog::trim()+0xca9) [0x564fb33910c9] 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564fb33915a8] 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4647ebeb23] 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4648fc81ca] 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 11: clone() 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2024-04-12T21:20:01.408+0000 7f463b418700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: in thread 7f463b418700 thread_name:ms_dispatch 2024-04-12T21:20:01.709 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4648fd2d20] 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2: gsignal() 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 3: abort() 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564fb2fbf268] 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564fb305b0fc] 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564fb338ff0a] 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 7: (MDLog::trim()+0xca9) [0x564fb33910c9] 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564fb33915a8] 2024-04-12T21:20:01.710 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4647ebeb23] 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4648fc81ca] 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 11: clone() 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 0> 2024-04-12T21:20:01.408+0000 7f463b418700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: in thread 7f463b418700 thread_name:ms_dispatch 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:01.711 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4648fd2d20] 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2: gsignal() 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 3: abort() 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564fb2fbf268] 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564fb305b0fc] 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564fb338ff0a] 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 7: (MDLog::trim()+0xca9) [0x564fb33910c9] 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564fb33915a8] 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4647ebeb23] 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4648fc81ca] 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 11: clone() 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 0> 2024-04-12T21:20:01.408+0000 7f463b418700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: in thread 7f463b418700 thread_name:ms_dispatch 2024-04-12T21:20:01.712 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4648fd2d20] 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2: gsignal() 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 3: abort() 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564fb2fbf268] 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564fb305b0fc] 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564fb338ff0a] 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 7: (MDLog::trim()+0xca9) [0x564fb33910c9] 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564fb33915a8] 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4647ebeb23] 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4648fc81ca] 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 11: clone() 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[68225]: 2024-04-12T21:20:01.713 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:01 smithi153 podman[70518]: 2024-04-12 21:20:01.478173938 +0000 UTC m=+0.020251837 container died 06cc72315b98bc82b88a8146ba82705212198b952adc553d53b253798561350f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:20:01.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:01 smithi081 ceph-mon[31453]: pgmap v523: 129 pgs: 129 active+clean; 477 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 99 op/s 2024-04-12T21:20:02.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:01 smithi104 ceph-mon[36624]: pgmap v523: 129 pgs: 129 active+clean; 477 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 99 op/s 2024-04-12T21:20:02.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:02 smithi153 podman[70518]: 2024-04-12 21:20:02.268995146 +0000 UTC m=+0.811073048 container remove 06cc72315b98bc82b88a8146ba82705212198b952adc553d53b253798561350f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS) 2024-04-12T21:20:02.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:20:02.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-12T21:20:02.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 3min 58.375s CPU time. 2024-04-12T21:20:03.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:03 smithi081 ceph-mon[31453]: pgmap v524: 129 pgs: 129 active+clean; 481 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-04-12T21:20:04.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:03 smithi153 ceph-mon[36569]: pgmap v524: 129 pgs: 129 active+clean; 481 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-04-12T21:20:04.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:03 smithi104 ceph-mon[36624]: pgmap v524: 129 pgs: 129 active+clean; 481 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-04-12T21:20:05.689 INFO:tasks.workunit.client.0.smithi081.stdout: 104 692506 389 505795 331 421264 643 2024-04-12T21:20:05.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:05 smithi081 ceph-mon[31453]: pgmap v525: 129 pgs: 129 active+clean; 481 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-04-12T21:20:06.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:05 smithi153 ceph-mon[36569]: pgmap v525: 129 pgs: 129 active+clean; 481 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-04-12T21:20:06.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:05 smithi104 ceph-mon[36624]: pgmap v525: 129 pgs: 129 active+clean; 481 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-04-12T21:20:07.853 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:07 smithi153 ceph-mon[36569]: pgmap v526: 129 pgs: 129 active+clean; 482 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-04-12T21:20:07.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:07 smithi081 ceph-mon[31453]: pgmap v526: 129 pgs: 129 active+clean; 482 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-04-12T21:20:07.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:07.914+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:20:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:07.919+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:20:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:07.919+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:20:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:07.919+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:20:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:07.919+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:20:08.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:07 smithi104 ceph-mon[36624]: pgmap v526: 129 pgs: 129 active+clean; 482 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-04-12T21:20:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:08 smithi081 ceph-mon[31453]: pgmap v527: 129 pgs: 129 active+clean; 482 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 45 op/s 2024-04-12T21:20:09.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:08 smithi153 ceph-mon[36569]: pgmap v527: 129 pgs: 129 active+clean; 482 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 45 op/s 2024-04-12T21:20:09.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:08 smithi104 ceph-mon[36624]: pgmap v527: 129 pgs: 129 active+clean; 482 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 45 op/s 2024-04-12T21:20:11.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:11 smithi104 ceph-mon[36624]: pgmap v528: 129 pgs: 129 active+clean; 488 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-04-12T21:20:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:11 smithi081 ceph-mon[31453]: pgmap v528: 129 pgs: 129 active+clean; 488 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-04-12T21:20:12.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:11 smithi153 ceph-mon[36569]: pgmap v528: 129 pgs: 129 active+clean; 488 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 1.4 MiB/s wr, 82 op/s 2024-04-12T21:20:12.823 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:12 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:20:12.823 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:12 smithi153 systemd[1]: Stopped Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:20:12.823 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:12 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 3min 58.375s CPU time. 2024-04-12T21:20:12.823 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:12 smithi153 systemd[1]: Starting Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:20:12.823 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:12 smithi153 podman[70618]: 2024-04-12 21:20:12.689489858 +0000 UTC m=+0.018591492 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:20:12.823 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:12 smithi153 podman[70618]: 2024-04-12 21:20:12.828424753 +0000 UTC m=+0.157526388 container create 4f696b99bc68fd91034ccdaa66da2437817255fa006b07d221b4fa59d1ea7bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-12T21:20:13.093 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:13 smithi153 podman[70618]: 2024-04-12 21:20:13.073070602 +0000 UTC m=+0.402172252 container init 4f696b99bc68fd91034ccdaa66da2437817255fa006b07d221b4fa59d1ea7bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-12T21:20:13.093 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:13 smithi153 podman[70618]: 2024-04-12 21:20:13.075625332 +0000 UTC m=+0.404726979 container start 4f696b99bc68fd91034ccdaa66da2437817255fa006b07d221b4fa59d1ea7bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD) 2024-04-12T21:20:13.359 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mds[70637]: -- 172.21.15.153:0/2801853975 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557804883a40 con 0x557804856c00 2024-04-12T21:20:13.359 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: starting mds.i at 2024-04-12T21:20:13.360 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:13 smithi153 bash[70618]: 4f696b99bc68fd91034ccdaa66da2437817255fa006b07d221b4fa59d1ea7bd3 2024-04-12T21:20:13.360 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:13 smithi153 systemd[1]: Started Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:20:13.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:20:13.104+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: pgmap v529: 129 pgs: 129 active+clean; 507 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 2.8 MiB/s wr, 181 op/s 2024-04-12T21:20:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: daemon mds.i restarted 2024-04-12T21:20:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: osdmap e95: 12 total, 12 up, 12 in 2024-04-12T21:20:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:boot 2024-04-12T21:20:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:20:13.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:13.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:13.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:20:13.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:13 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:20:13.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: pgmap v529: 129 pgs: 129 active+clean; 507 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 2.8 MiB/s wr, 181 op/s 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: daemon mds.i restarted 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: osdmap e95: 12 total, 12 up, 12 in 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:boot 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:20:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:13 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:20:13.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: pgmap v529: 129 pgs: 129 active+clean; 507 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 2.8 MiB/s wr, 181 op/s 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: daemon mds.i restarted 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: osdmap e95: 12 total, 12 up, 12 in 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:boot 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:20:13.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:13 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:20:15.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:reconnect 2024-04-12T21:20:15.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:15 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:20:15.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:15 smithi081 ceph-mon[31453]: pgmap v531: 129 pgs: 129 active+clean; 511 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 3.1 MiB/s wr, 217 op/s 2024-04-12T21:20:15.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:15 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.497994 2024-04-12T21:20:15.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:reconnect 2024-04-12T21:20:15.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:15 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:20:15.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:15 smithi153 ceph-mon[36569]: pgmap v531: 129 pgs: 129 active+clean; 511 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 3.1 MiB/s wr, 217 op/s 2024-04-12T21:20:15.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:15 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.497994 2024-04-12T21:20:15.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:reconnect 2024-04-12T21:20:15.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:15 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:20:15.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:15 smithi104 ceph-mon[36624]: pgmap v531: 129 pgs: 129 active+clean; 511 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 3.1 MiB/s wr, 217 op/s 2024-04-12T21:20:15.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:15 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.497994 2024-04-12T21:20:15.689 INFO:tasks.workunit.client.0.smithi081.stdout: 104 471097 0 347527 0 289211 0 2024-04-12T21:20:16.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:16 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:rejoin 2024-04-12T21:20:16.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:16 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:20:16.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:16 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:rejoin 2024-04-12T21:20:16.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:16 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:20:16.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:16 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:rejoin 2024-04-12T21:20:16.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:16 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:20:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:17 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:clientreplay 2024-04-12T21:20:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:17 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:20:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:17 smithi153 ceph-mon[36569]: pgmap v532: 129 pgs: 129 active+clean; 543 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 6.4 MiB/s wr, 446 op/s 2024-04-12T21:20:17.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:17 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:clientreplay 2024-04-12T21:20:17.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:17 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:20:17.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:17 smithi081 ceph-mon[31453]: pgmap v532: 129 pgs: 129 active+clean; 543 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 6.4 MiB/s wr, 446 op/s 2024-04-12T21:20:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:17 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:clientreplay 2024-04-12T21:20:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:17 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:20:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:17 smithi104 ceph-mon[36624]: pgmap v532: 129 pgs: 129 active+clean; 543 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 6.4 MiB/s wr, 446 op/s 2024-04-12T21:20:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:19 smithi104 ceph-mon[36624]: pgmap v533: 129 pgs: 129 active+clean; 548 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 6.8 MiB/s wr, 479 op/s 2024-04-12T21:20:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:19 smithi104 ceph-mon[36624]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:20:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:19 smithi081 ceph-mon[31453]: pgmap v533: 129 pgs: 129 active+clean; 548 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 6.8 MiB/s wr, 479 op/s 2024-04-12T21:20:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:19 smithi081 ceph-mon[31453]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:20:20.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:19 smithi153 ceph-mon[36569]: pgmap v533: 129 pgs: 129 active+clean; 548 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 6.8 MiB/s wr, 479 op/s 2024-04-12T21:20:20.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:19 smithi153 ceph-mon[36569]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: *** Caught signal (Aborted) ** 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: in thread 7f31183d0700 thread_name:ms_dispatch 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3125f8ad20] 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2: gsignal() 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 3: abort() 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5629d55df268] 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5629d567b0fc] 2024-04-12T21:20:20.570 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5629d59aff0a] 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 7: (MDLog::trim()+0xca9) [0x5629d59b10c9] 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5629d59b15a8] 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3124e76b23] 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3125f801ca] 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 11: clone() 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2024-04-12T21:20:20.309+0000 7f31183d0700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: in thread 7f31183d0700 thread_name:ms_dispatch 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3125f8ad20] 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2: gsignal() 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 3: abort() 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5629d55df268] 2024-04-12T21:20:20.571 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5629d567b0fc] 2024-04-12T21:20:20.572 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5629d59aff0a] 2024-04-12T21:20:20.572 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 7: (MDLog::trim()+0xca9) [0x5629d59b10c9] 2024-04-12T21:20:20.572 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5629d59b15a8] 2024-04-12T21:20:20.572 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3124e76b23] 2024-04-12T21:20:20.572 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3125f801ca] 2024-04-12T21:20:20.572 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 11: clone() 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 0> 2024-04-12T21:20:20.309+0000 7f31183d0700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: in thread 7f31183d0700 thread_name:ms_dispatch 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3125f8ad20] 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2: gsignal() 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 3: abort() 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5629d55df268] 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5629d567b0fc] 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5629d59aff0a] 2024-04-12T21:20:20.573 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 7: (MDLog::trim()+0xca9) [0x5629d59b10c9] 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5629d59b15a8] 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3124e76b23] 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3125f801ca] 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 11: clone() 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 0> 2024-04-12T21:20:20.309+0000 7f31183d0700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: in thread 7f31183d0700 thread_name:ms_dispatch 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3125f8ad20] 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2: gsignal() 2024-04-12T21:20:20.574 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 3: abort() 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5629d55df268] 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5629d567b0fc] 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5629d59aff0a] 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 7: (MDLog::trim()+0xca9) [0x5629d59b10c9] 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5629d59b15a8] 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3124e76b23] 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3125f801ca] 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 11: clone() 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[68201]: 2024-04-12T21:20:20.575 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:20 smithi104 podman[68779]: 2024-04-12 21:20:20.370918084 +0000 UTC m=+0.019507448 container died bc5eb3014abcb56a092bbe2bea79825dd1d22040271ae828906728f43138691e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-12T21:20:20.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:20 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:20 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:20:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:20 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:active 2024-04-12T21:20:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:20 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:20:20.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:20 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:20.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:20.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:20:20.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:active 2024-04-12T21:20:20.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:20:20.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:20 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:20 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:20 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:20:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:20 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/3044964827,v1:172.21.15.104:6839/3044964827] up:active 2024-04-12T21:20:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:20 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:20:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:20 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:21.327 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:21 smithi104 podman[68779]: 2024-04-12 21:20:21.065208123 +0000 UTC m=+0.713797485 container remove bc5eb3014abcb56a092bbe2bea79825dd1d22040271ae828906728f43138691e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:20:21.327 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:21 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:20:21.327 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:21 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-12T21:20:21.328 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:21 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 28.207s CPU time. 2024-04-12T21:20:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:21 smithi081 ceph-mon[31453]: pgmap v534: 129 pgs: 129 active+clean; 555 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.9 MiB/s wr, 533 op/s 2024-04-12T21:20:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:20:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:21 smithi153 ceph-mon[36569]: pgmap v534: 129 pgs: 129 active+clean; 555 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.9 MiB/s wr, 533 op/s 2024-04-12T21:20:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:20:22.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:21 smithi104 ceph-mon[36624]: pgmap v534: 129 pgs: 129 active+clean; 555 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.9 MiB/s wr, 533 op/s 2024-04-12T21:20:22.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:20:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:22.921+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:20:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:22.921+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:20:22.916 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:22.921+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:20:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:23 smithi081 ceph-mon[31453]: pgmap v535: 129 pgs: 129 active+clean; 561 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 598 op/s 2024-04-12T21:20:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:20:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:20:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:20:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:23 smithi153 ceph-mon[36569]: pgmap v535: 129 pgs: 129 active+clean; 561 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 598 op/s 2024-04-12T21:20:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:20:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:20:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:20:24.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:23 smithi104 ceph-mon[36624]: pgmap v535: 129 pgs: 129 active+clean; 561 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 5.8 MiB/s wr, 598 op/s 2024-04-12T21:20:24.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:20:24.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:20:24.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:20:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:25 smithi081 ceph-mon[31453]: pgmap v536: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.1 MiB/s wr, 554 op/s 2024-04-12T21:20:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:25 smithi153 ceph-mon[36569]: pgmap v536: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.1 MiB/s wr, 554 op/s 2024-04-12T21:20:26.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:25 smithi104 ceph-mon[36624]: pgmap v536: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.1 MiB/s wr, 554 op/s 2024-04-12T21:20:27.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:27 smithi081 ceph-mon[31453]: pgmap v537: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 508 op/s 2024-04-12T21:20:28.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:27 smithi153 ceph-mon[36569]: pgmap v537: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 508 op/s 2024-04-12T21:20:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:27 smithi104 ceph-mon[36624]: pgmap v537: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 508 op/s 2024-04-12T21:20:28.868 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:28 smithi081 ceph-mon[31453]: pgmap v538: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 312 op/s 2024-04-12T21:20:29.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:28 smithi153 ceph-mon[36569]: pgmap v538: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 312 op/s 2024-04-12T21:20:29.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:28 smithi104 ceph-mon[36624]: pgmap v538: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 312 op/s 2024-04-12T21:20:31.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:31 smithi104 ceph-mon[36624]: pgmap v539: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.3 MiB/s wr, 280 op/s 2024-04-12T21:20:31.828 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:31 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:20:31.828 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:31 smithi104 systemd[1]: Stopped Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:20:31.828 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:31 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 28.207s CPU time. 2024-04-12T21:20:31.829 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:31 smithi104 systemd[1]: Starting Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:20:31.829 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:31 smithi104 podman[69005]: 2024-04-12 21:20:31.470867421 +0000 UTC m=+0.017055092 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:20:31.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:31 smithi081 ceph-mon[31453]: pgmap v539: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.3 MiB/s wr, 280 op/s 2024-04-12T21:20:32.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:31 smithi153 ceph-mon[36569]: pgmap v539: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.3 MiB/s wr, 280 op/s 2024-04-12T21:20:33.074 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:32 smithi104 podman[69005]: 2024-04-12 21:20:32.815978732 +0000 UTC m=+1.362166391 container create 8998707e32ccc51875106cd6da39996426343772e5ed5d455096146d56b88120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-12T21:20:33.074 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:33 smithi104 podman[69005]: 2024-04-12 21:20:33.052693705 +0000 UTC m=+1.598881373 container init 8998707e32ccc51875106cd6da39996426343772e5ed5d455096146d56b88120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-04-12T21:20:33.074 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:33 smithi104 podman[69005]: 2024-04-12 21:20:33.055794961 +0000 UTC m=+1.601982624 container start 8998707e32ccc51875106cd6da39996426343772e5ed5d455096146d56b88120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.vendor=CentOS) 2024-04-12T21:20:33.327 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mds[69025]: -- 172.21.15.104:0/1407414417 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564284db1a40 con 0x564284d85000 2024-04-12T21:20:33.327 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: starting mds.b at 2024-04-12T21:20:33.327 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:33 smithi104 bash[69005]: 8998707e32ccc51875106cd6da39996426343772e5ed5d455096146d56b88120 2024-04-12T21:20:33.327 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:33 smithi104 systemd[1]: Started Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:20:33.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:20:33.096+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: pgmap v540: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 692 KiB/s wr, 195 op/s 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: daemon mds.b restarted 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: osdmap e96: 12 total, 12 up, 12 in 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:boot 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:20:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:33 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: pgmap v540: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 692 KiB/s wr, 195 op/s 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: daemon mds.b restarted 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: osdmap e96: 12 total, 12 up, 12 in 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:boot 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:33.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:20:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:33 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: pgmap v540: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 692 KiB/s wr, 195 op/s 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: daemon mds.b restarted 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: osdmap e96: 12 total, 12 up, 12 in 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:boot 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:20:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:33 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:20:35.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:35 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:reconnect 2024-04-12T21:20:35.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:35 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:20:35.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:35 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.112998 2024-04-12T21:20:35.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:35 smithi081 ceph-mon[31453]: pgmap v542: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5 op/s 2024-04-12T21:20:35.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:35 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:reconnect 2024-04-12T21:20:35.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:35 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:20:35.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:35 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.112998 2024-04-12T21:20:35.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:35 smithi153 ceph-mon[36569]: pgmap v542: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5 op/s 2024-04-12T21:20:35.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:35 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:reconnect 2024-04-12T21:20:35.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:35 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:20:35.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:35 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.112998 2024-04-12T21:20:35.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:35 smithi104 ceph-mon[36624]: pgmap v542: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5 op/s 2024-04-12T21:20:36.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:36 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:rejoin 2024-04-12T21:20:36.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:36 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:20:36.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:36 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:rejoin 2024-04-12T21:20:36.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:36 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:20:36.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:36 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:rejoin 2024-04-12T21:20:36.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:36 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:20:37.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:37 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:clientreplay 2024-04-12T21:20:37.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:37 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:20:37.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:37 smithi081 ceph-mon[31453]: pgmap v543: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:20:37.452 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:37 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:clientreplay 2024-04-12T21:20:37.453 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:37 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:20:37.453 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:37 smithi153 ceph-mon[36569]: pgmap v543: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:20:37.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:37 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:clientreplay 2024-04-12T21:20:37.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:37 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:20:37.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:37 smithi104 ceph-mon[36624]: pgmap v543: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:20:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:37.925+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:20:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:37.925+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:20:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:37.925+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:20:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:37.925+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:20:37.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:37.925+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:20:38.182 INFO:tasks.workunit.client.0.smithi081.stdout: 104 1 0 1 0 0 0 2024-04-12T21:20:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:39 smithi104 ceph-mon[36624]: pgmap v544: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:20:39.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:39 smithi081 ceph-mon[31453]: pgmap v544: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:20:40.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:39 smithi153 ceph-mon[36569]: pgmap v544: 129 pgs: 129 active+clean; 560 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:20:40.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:40 smithi104 ceph-mon[36624]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:20:40.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:40 smithi081 ceph-mon[31453]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:20:41.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:40 smithi153 ceph-mon[36569]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:20:41.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:20:41.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: *** Caught signal (Aborted) ** 2024-04-12T21:20:41.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: in thread 7f5574692700 thread_name:ms_dispatch 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f558224cd20] 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2: gsignal() 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 3: abort() 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5578020d5268] 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5578021710fc] 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5578024a5f0a] 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 7: (MDLog::trim()+0xca9) [0x5578024a70c9] 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5578024a75a8] 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f5581138b23] 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f55822421ca] 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 11: clone() 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2024-04-12T21:20:41.104+0000 7f5574692700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:41.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: in thread 7f5574692700 thread_name:ms_dispatch 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f558224cd20] 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2: gsignal() 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 3: abort() 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5578020d5268] 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5578021710fc] 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5578024a5f0a] 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 7: (MDLog::trim()+0xca9) [0x5578024a70c9] 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5578024a75a8] 2024-04-12T21:20:41.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f5581138b23] 2024-04-12T21:20:41.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f55822421ca] 2024-04-12T21:20:41.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 11: clone() 2024-04-12T21:20:41.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:41.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2024-04-12T21:20:41.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 0> 2024-04-12T21:20:41.104+0000 7f5574692700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:41.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: in thread 7f5574692700 thread_name:ms_dispatch 2024-04-12T21:20:41.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2024-04-12T21:20:41.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f558224cd20] 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2: gsignal() 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 3: abort() 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5578020d5268] 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5578021710fc] 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5578024a5f0a] 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 7: (MDLog::trim()+0xca9) [0x5578024a70c9] 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5578024a75a8] 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f5581138b23] 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f55822421ca] 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 11: clone() 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2024-04-12T21:20:41.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 0> 2024-04-12T21:20:41.104+0000 7f5574692700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: in thread 7f5574692700 thread_name:ms_dispatch 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f558224cd20] 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2: gsignal() 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 3: abort() 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5578020d5268] 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5578021710fc] 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5578024a5f0a] 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 7: (MDLog::trim()+0xca9) [0x5578024a70c9] 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5578024a75a8] 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f5581138b23] 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f55822421ca] 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 11: clone() 2024-04-12T21:20:41.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:41.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70633]: 2024-04-12T21:20:41.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 podman[70818]: 2024-04-12 21:20:41.16897885 +0000 UTC m=+0.020109310 container died 4f696b99bc68fd91034ccdaa66da2437817255fa006b07d221b4fa59d1ea7bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-12T21:20:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:41 smithi104 ceph-mon[36624]: pgmap v545: 129 pgs: 129 active+clean; 567 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 889 KiB/s wr, 195 op/s 2024-04-12T21:20:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:41 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:41 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:20:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:41 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:active 2024-04-12T21:20:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:41 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:20:41.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:41 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:41.861 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:41 smithi081 ceph-mon[31453]: pgmap v545: 129 pgs: 129 active+clean; 567 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 889 KiB/s wr, 195 op/s 2024-04-12T21:20:41.861 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:41 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:41.861 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:41 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:20:41.861 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:41 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:active 2024-04-12T21:20:41.861 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:41 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:20:41.861 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:41 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:41.905 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-mon[36569]: pgmap v545: 129 pgs: 129 active+clean; 567 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 889 KiB/s wr, 195 op/s 2024-04-12T21:20:41.905 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:41.905 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:20:41.906 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/913575250,v1:172.21.15.153:6839/913575250] up:active 2024-04-12T21:20:41.906 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:20:41.906 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:41 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:42.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 podman[70818]: 2024-04-12 21:20:41.911037497 +0000 UTC m=+0.762167932 container remove 4f696b99bc68fd91034ccdaa66da2437817255fa006b07d221b4fa59d1ea7bd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True) 2024-04-12T21:20:42.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:41 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:20:42.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:42 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-12T21:20:42.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:42 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 28.354s CPU time. 2024-04-12T21:20:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:43 smithi104 ceph-mon[36624]: pgmap v546: 129 pgs: 129 active+clean; 573 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 348 op/s 2024-04-12T21:20:43.844 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:43 smithi081 ceph-mon[31453]: pgmap v546: 129 pgs: 129 active+clean; 573 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 348 op/s 2024-04-12T21:20:44.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:43 smithi153 ceph-mon[36569]: pgmap v546: 129 pgs: 129 active+clean; 573 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 1.6 MiB/s wr, 348 op/s 2024-04-12T21:20:45.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:45 smithi081 ceph-mon[31453]: pgmap v547: 129 pgs: 129 active+clean; 573 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 318 op/s 2024-04-12T21:20:46.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:45 smithi153 ceph-mon[36569]: pgmap v547: 129 pgs: 129 active+clean; 573 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 318 op/s 2024-04-12T21:20:46.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:45 smithi104 ceph-mon[36624]: pgmap v547: 129 pgs: 129 active+clean; 573 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 318 op/s 2024-04-12T21:20:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:47 smithi081 ceph-mon[31453]: pgmap v548: 129 pgs: 129 active+clean; 574 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 320 op/s 2024-04-12T21:20:48.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:47 smithi153 ceph-mon[36569]: pgmap v548: 129 pgs: 129 active+clean; 574 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 320 op/s 2024-04-12T21:20:48.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:47 smithi104 ceph-mon[36624]: pgmap v548: 129 pgs: 129 active+clean; 574 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 320 op/s 2024-04-12T21:20:48.182 INFO:tasks.workunit.client.0.smithi081.stdout: 105 25 17 9 13 1338 36 2024-04-12T21:20:49.861 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:49 smithi081 ceph-mon[31453]: pgmap v549: 129 pgs: 129 active+clean; 574 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 318 op/s 2024-04-12T21:20:50.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:49 smithi153 ceph-mon[36569]: pgmap v549: 129 pgs: 129 active+clean; 574 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 318 op/s 2024-04-12T21:20:50.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:49 smithi104 ceph-mon[36624]: pgmap v549: 129 pgs: 129 active+clean; 574 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 1.4 MiB/s wr, 318 op/s 2024-04-12T21:20:51.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:51 smithi081 ceph-mon[31453]: pgmap v550: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 320 op/s 2024-04-12T21:20:52.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:51 smithi153 ceph-mon[36569]: pgmap v550: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 320 op/s 2024-04-12T21:20:52.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:51 smithi104 ceph-mon[36624]: pgmap v550: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 320 op/s 2024-04-12T21:20:52.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:20:52.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 systemd[1]: Stopped Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:20:52.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 28.354s CPU time. 2024-04-12T21:20:52.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 systemd[1]: Starting Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:20:52.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 podman[70915]: 2024-04-12 21:20:52.442938628 +0000 UTC m=+0.018047571 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:20:52.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 podman[70915]: 2024-04-12 21:20:52.561690743 +0000 UTC m=+0.136799679 container create 8f8cf6c95a3afc2a1ba9f8aa517de3f6e287a628740e56258529eaae83d552a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-12T21:20:52.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:52.938+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:20:52.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:52.940+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:20:52.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:52.940+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:20:52.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:52.940+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:20:52.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:20:52.940+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:20:53.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:20:52.891+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:53.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 podman[70915]: 2024-04-12 21:20:52.848072369 +0000 UTC m=+0.423181313 container init 8f8cf6c95a3afc2a1ba9f8aa517de3f6e287a628740e56258529eaae83d552a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:20:53.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 podman[70915]: 2024-04-12 21:20:52.850619274 +0000 UTC m=+0.425728210 container start 8f8cf6c95a3afc2a1ba9f8aa517de3f6e287a628740e56258529eaae83d552a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:20:53.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 ceph-mds[70934]: -- 172.21.15.153:0/3214262493 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cfe2c51a40 con 0x55cfe2c25000 2024-04-12T21:20:53.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: starting mds.i at 2024-04-12T21:20:53.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 bash[70915]: 8f8cf6c95a3afc2a1ba9f8aa517de3f6e287a628740e56258529eaae83d552a0 2024-04-12T21:20:53.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:20:52 smithi153 systemd[1]: Started Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: pgmap v551: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 763 KiB/s wr, 166 op/s 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: daemon mds.i restarted 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: osdmap e97: 12 total, 12 up, 12 in 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:boot 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:20:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:53 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: pgmap v551: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 763 KiB/s wr, 166 op/s 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: daemon mds.i restarted 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: osdmap e97: 12 total, 12 up, 12 in 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:boot 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:20:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:53 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:20:54.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: pgmap v551: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 763 KiB/s wr, 166 op/s 2024-04-12T21:20:54.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: daemon mds.i restarted 2024-04-12T21:20:54.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:20:54.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:20:54.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: osdmap e97: 12 total, 12 up, 12 in 2024-04-12T21:20:54.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:boot 2024-04-12T21:20:54.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:20:54.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:20:54.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:20:54.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:20:54.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:53 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:20:55.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:54 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:reconnect 2024-04-12T21:20:55.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:54 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:20:55.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:54 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0259997 2024-04-12T21:20:55.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:54 smithi153 ceph-mon[36569]: pgmap v553: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 193 KiB/s wr, 31 op/s 2024-04-12T21:20:55.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:54 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:reconnect 2024-04-12T21:20:55.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:54 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:20:55.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:54 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0259997 2024-04-12T21:20:55.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:54 smithi104 ceph-mon[36624]: pgmap v553: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 193 KiB/s wr, 31 op/s 2024-04-12T21:20:55.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:54 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:reconnect 2024-04-12T21:20:55.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:54 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:20:55.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:54 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0259997 2024-04-12T21:20:55.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:54 smithi081 ceph-mon[31453]: pgmap v553: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 193 KiB/s wr, 31 op/s 2024-04-12T21:20:56.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:55 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:rejoin 2024-04-12T21:20:56.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:55 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:20:56.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:55 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:rejoin 2024-04-12T21:20:56.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:55 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:20:56.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:55 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:rejoin 2024-04-12T21:20:56.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:55 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:20:57.276 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:56 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:clientreplay 2024-04-12T21:20:57.276 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:56 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:20:57.276 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:56 smithi081 ceph-mon[31453]: pgmap v554: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 95 KiB/s wr, 30 op/s 2024-04-12T21:20:57.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:56 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:clientreplay 2024-04-12T21:20:57.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:56 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:20:57.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:56 smithi153 ceph-mon[36569]: pgmap v554: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 95 KiB/s wr, 30 op/s 2024-04-12T21:20:57.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:56 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:clientreplay 2024-04-12T21:20:57.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:56 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:20:57.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:56 smithi104 ceph-mon[36624]: pgmap v554: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 95 KiB/s wr, 30 op/s 2024-04-12T21:20:57.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:20:57.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: *** Caught signal (Aborted) ** 2024-04-12T21:20:57.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: in thread 7f0318564700 thread_name:ms_dispatch 2024-04-12T21:20:57.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:57.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f032611ed20] 2024-04-12T21:20:57.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2: gsignal() 2024-04-12T21:20:57.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 3: abort() 2024-04-12T21:20:57.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564283e4a268] 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564283ee60fc] 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56428421af0a] 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 7: (MDLog::trim()+0xca9) [0x56428421c0c9] 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56428421c5a8] 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f032500ab23] 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f03261141ca] 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 11: clone() 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2024-04-12T21:20:57.605+0000 7f0318564700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: in thread 7f0318564700 thread_name:ms_dispatch 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f032611ed20] 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2: gsignal() 2024-04-12T21:20:57.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 3: abort() 2024-04-12T21:20:57.949 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564283e4a268] 2024-04-12T21:20:57.949 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564283ee60fc] 2024-04-12T21:20:57.949 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56428421af0a] 2024-04-12T21:20:57.949 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 7: (MDLog::trim()+0xca9) [0x56428421c0c9] 2024-04-12T21:20:57.949 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56428421c5a8] 2024-04-12T21:20:57.949 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f032500ab23] 2024-04-12T21:20:57.949 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f03261141ca] 2024-04-12T21:20:57.949 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 11: clone() 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 0> 2024-04-12T21:20:57.605+0000 7f0318564700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: in thread 7f0318564700 thread_name:ms_dispatch 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f032611ed20] 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2: gsignal() 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 3: abort() 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564283e4a268] 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564283ee60fc] 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56428421af0a] 2024-04-12T21:20:57.950 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 7: (MDLog::trim()+0xca9) [0x56428421c0c9] 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56428421c5a8] 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f032500ab23] 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f03261141ca] 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 11: clone() 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: -1> 2024-04-12T21:20:57.605+0000 7f0318564700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: in thread 7f0318564700 thread_name:ms_dispatch 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f032611ed20] 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2: gsignal() 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 3: abort() 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564283e4a268] 2024-04-12T21:20:57.951 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564283ee60fc] 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56428421af0a] 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 7: (MDLog::trim()+0xca9) [0x56428421c0c9] 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56428421c5a8] 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f032500ab23] 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f03261141ca] 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 11: clone() 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69019]: 2024-04-12T21:20:57.952 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:57 smithi104 podman[69074]: 2024-04-12 21:20:57.665431737 +0000 UTC m=+0.019561652 container died 8998707e32ccc51875106cd6da39996426343772e5ed5d455096146d56b88120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T21:20:58.182 INFO:tasks.workunit.client.0.smithi081.stdout: 105 17 18 4 5 1382 29 2024-04-12T21:20:58.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:57 smithi153 ceph-mon[36569]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:20:58.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:57 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:58.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:57 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:20:58.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:57 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:active 2024-04-12T21:20:58.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:57 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:58.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-mon[36624]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:20:58.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:58.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:20:58.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:active 2024-04-12T21:20:58.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:57 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:58.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:57 smithi081 ceph-mon[31453]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:20:58.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:57 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:20:58.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:57 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:20:58.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:57 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/1090941463,v1:172.21.15.104:6839/1090941463] up:active 2024-04-12T21:20:58.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:57 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:20:59.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:20:58 smithi104 ceph-mon[36624]: pgmap v555: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 95 KiB/s wr, 58 op/s 2024-04-12T21:20:59.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:58 smithi104 podman[69074]: 2024-04-12 21:20:58.72091842 +0000 UTC m=+1.075048378 container remove 8998707e32ccc51875106cd6da39996426343772e5ed5d455096146d56b88120 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-12T21:20:59.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:58 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:20:59.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:58 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-12T21:20:59.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:20:58 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 22.323s CPU time. 2024-04-12T21:20:59.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:20:58 smithi153 ceph-mon[36569]: pgmap v555: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 95 KiB/s wr, 58 op/s 2024-04-12T21:20:59.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:20:58 smithi081 ceph-mon[31453]: pgmap v555: 129 pgs: 129 active+clean; 575 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 95 KiB/s wr, 58 op/s 2024-04-12T21:21:01.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:01 smithi081 ceph-mon[31453]: pgmap v556: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 631 KiB/s wr, 135 op/s 2024-04-12T21:21:02.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:01 smithi153 ceph-mon[36569]: pgmap v556: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 631 KiB/s wr, 135 op/s 2024-04-12T21:21:02.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:01 smithi104 ceph-mon[36624]: pgmap v556: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 631 KiB/s wr, 135 op/s 2024-04-12T21:21:03.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:03 smithi081 ceph-mon[31453]: pgmap v557: 129 pgs: 129 active+clean; 580 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 570 KiB/s wr, 240 op/s 2024-04-12T21:21:04.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:03 smithi153 ceph-mon[36569]: pgmap v557: 129 pgs: 129 active+clean; 580 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 570 KiB/s wr, 240 op/s 2024-04-12T21:21:04.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:03 smithi104 ceph-mon[36624]: pgmap v557: 129 pgs: 129 active+clean; 580 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 570 KiB/s wr, 240 op/s 2024-04-12T21:21:05.861 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:05 smithi081 ceph-mon[31453]: pgmap v558: 129 pgs: 129 active+clean; 580 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 498 KiB/s wr, 208 op/s 2024-04-12T21:21:06.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:05 smithi153 ceph-mon[36569]: pgmap v558: 129 pgs: 129 active+clean; 580 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 498 KiB/s wr, 208 op/s 2024-04-12T21:21:06.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:05 smithi104 ceph-mon[36624]: pgmap v558: 129 pgs: 129 active+clean; 580 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 498 KiB/s wr, 208 op/s 2024-04-12T21:21:07.853 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:07 smithi153 ceph-mon[36569]: pgmap v559: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 523 KiB/s wr, 207 op/s 2024-04-12T21:21:07.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:07 smithi081 ceph-mon[31453]: pgmap v559: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 523 KiB/s wr, 207 op/s 2024-04-12T21:21:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:07.941+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:21:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:07.941+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:21:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:07.941+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:21:08.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:07 smithi104 ceph-mon[36624]: pgmap v559: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 523 KiB/s wr, 207 op/s 2024-04-12T21:21:08.182 INFO:tasks.workunit.client.0.smithi081.stdout: 105 0 0 0 0 0 0 2024-04-12T21:21:09.366 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:21:09.366 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 systemd[1]: Stopped Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:21:09.366 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 22.323s CPU time. 2024-04-12T21:21:09.366 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 systemd[1]: Starting Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:21:09.366 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 podman[69171]: 2024-04-12 21:21:09.21723444 +0000 UTC m=+0.018086351 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:21:09.366 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 podman[69171]: 2024-04-12 21:21:09.371278571 +0000 UTC m=+0.172130501 container create e8185e4201b3306d6c930ce5d5503bd46aeb9d7619eeb992e5d4d5ef277c300f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-12T21:21:09.619 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:09 smithi104 ceph-mon[36624]: pgmap v560: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 523 KiB/s wr, 189 op/s 2024-04-12T21:21:09.619 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 podman[69171]: 2024-04-12 21:21:09.598796281 +0000 UTC m=+0.399648200 container init e8185e4201b3306d6c930ce5d5503bd46aeb9d7619eeb992e5d4d5ef277c300f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-12T21:21:09.619 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 podman[69171]: 2024-04-12 21:21:09.601310959 +0000 UTC m=+0.402162870 container start e8185e4201b3306d6c930ce5d5503bd46aeb9d7619eeb992e5d4d5ef277c300f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:21:09.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:09 smithi081 ceph-mon[31453]: pgmap v560: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 523 KiB/s wr, 189 op/s 2024-04-12T21:21:09.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:09 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:21:09.641+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:21:10.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:09 smithi153 ceph-mon[36569]: pgmap v560: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 523 KiB/s wr, 189 op/s 2024-04-12T21:21:10.077 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 ceph-mds[69190]: -- 172.21.15.104:0/996082457 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef98ec9a40 con 0x55ef98e9cc00 2024-04-12T21:21:10.077 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: starting mds.b at 2024-04-12T21:21:10.077 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 bash[69171]: e8185e4201b3306d6c930ce5d5503bd46aeb9d7619eeb992e5d4d5ef277c300f 2024-04-12T21:21:10.077 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:09 smithi104 systemd[1]: Started Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:21:10.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: daemon mds.b restarted 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: osdmap e98: 12 total, 12 up, 12 in 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:boot 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:21:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:10 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: daemon mds.b restarted 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: osdmap e98: 12 total, 12 up, 12 in 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:boot 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:21:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:21:11.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:10 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:21:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: daemon mds.b restarted 2024-04-12T21:21:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:21:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: osdmap e98: 12 total, 12 up, 12 in 2024-04-12T21:21:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:boot 2024-04-12T21:21:11.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: Standby daemon mds.i assigned to filesystem cephfs as rank 0 2024-04-12T21:21:11.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:21:11.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:21:11.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:21:11.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:10 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:replay} 11 up:standby 2024-04-12T21:21:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:11 smithi081 ceph-mon[31453]: pgmap v562: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 65 KiB/s wr, 120 op/s 2024-04-12T21:21:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:11 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:reconnect 2024-04-12T21:21:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:11 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:21:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:11 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0239996 2024-04-12T21:21:12.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:11 smithi153 ceph-mon[36569]: pgmap v562: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 65 KiB/s wr, 120 op/s 2024-04-12T21:21:12.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:11 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:reconnect 2024-04-12T21:21:12.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:11 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:21:12.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:11 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0239996 2024-04-12T21:21:12.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:11 smithi104 ceph-mon[36624]: pgmap v562: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 65 KiB/s wr, 120 op/s 2024-04-12T21:21:12.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:11 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:reconnect 2024-04-12T21:21:12.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:11 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:21:12.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:11 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0239996 2024-04-12T21:21:13.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:12 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:rejoin 2024-04-12T21:21:13.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:12 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:21:13.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:12 smithi153 ceph-mon[36569]: pgmap v563: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 KiB/s wr, 9 op/s 2024-04-12T21:21:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:12 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:rejoin 2024-04-12T21:21:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:12 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:21:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:12 smithi104 ceph-mon[36624]: pgmap v563: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 KiB/s wr, 9 op/s 2024-04-12T21:21:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:12 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:rejoin 2024-04-12T21:21:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:12 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:21:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:12 smithi081 ceph-mon[31453]: pgmap v563: 129 pgs: 129 active+clean; 581 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 62 KiB/s wr, 9 op/s 2024-04-12T21:21:13.978 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:13 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:clientreplay 2024-04-12T21:21:13.979 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:13 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:21:13.979 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:13 smithi104 ceph-mon[36624]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:21:14.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:13 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:clientreplay 2024-04-12T21:21:14.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:13 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:21:14.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:13 smithi153 ceph-mon[36569]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:21:14.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:13 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:clientreplay 2024-04-12T21:21:14.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:13 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:21:14.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:13 smithi081 ceph-mon[31453]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:21:14.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:21:14.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: *** Caught signal (Aborted) ** 2024-04-12T21:21:14.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: in thread 7fe269277700 thread_name:ms_dispatch 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe276e31d20] 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2: gsignal() 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 3: abort() 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55cfdfdfa268] 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55cfdfe960fc] 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55cfe01caf0a] 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 7: (MDLog::trim()+0xca9) [0x55cfe01cc0c9] 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55cfe01cc5a8] 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe275d1db23] 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe276e271ca] 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 11: clone() 2024-04-12T21:21:14.553 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2024-04-12T21:21:14.275+0000 7fe269277700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: in thread 7fe269277700 thread_name:ms_dispatch 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe276e31d20] 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2: gsignal() 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 3: abort() 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55cfdfdfa268] 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55cfdfe960fc] 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55cfe01caf0a] 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 7: (MDLog::trim()+0xca9) [0x55cfe01cc0c9] 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55cfe01cc5a8] 2024-04-12T21:21:14.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe275d1db23] 2024-04-12T21:21:14.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe276e271ca] 2024-04-12T21:21:14.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 11: clone() 2024-04-12T21:21:14.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:14.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2024-04-12T21:21:14.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 0> 2024-04-12T21:21:14.275+0000 7fe269277700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: in thread 7fe269277700 thread_name:ms_dispatch 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe276e31d20] 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2: gsignal() 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 3: abort() 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55cfdfdfa268] 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55cfdfe960fc] 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55cfe01caf0a] 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 7: (MDLog::trim()+0xca9) [0x55cfe01cc0c9] 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55cfe01cc5a8] 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe275d1db23] 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe276e271ca] 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 11: clone() 2024-04-12T21:21:14.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 0> 2024-04-12T21:21:14.275+0000 7fe269277700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: in thread 7fe269277700 thread_name:ms_dispatch 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe276e31d20] 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2: gsignal() 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 3: abort() 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55cfdfdfa268] 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55cfdfe960fc] 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55cfe01caf0a] 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 7: (MDLog::trim()+0xca9) [0x55cfe01cc0c9] 2024-04-12T21:21:14.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55cfe01cc5a8] 2024-04-12T21:21:14.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe275d1db23] 2024-04-12T21:21:14.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe276e271ca] 2024-04-12T21:21:14.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 11: clone() 2024-04-12T21:21:14.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:14.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[70930]: 2024-04-12T21:21:14.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:14 smithi153 podman[70970]: 2024-04-12 21:21:14.337837621 +0000 UTC m=+0.018435002 container died 8f8cf6c95a3afc2a1ba9f8aa517de3f6e287a628740e56258529eaae83d552a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:21:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:21:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:active 2024-04-12T21:21:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:21:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:14 smithi153 ceph-mon[36569]: pgmap v564: 129 pgs: 129 active+clean; 583 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 316 KiB/s wr, 47 op/s 2024-04-12T21:21:15.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:14 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:15.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:14 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:21:15.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:14 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:active 2024-04-12T21:21:15.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:14 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:21:15.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:14 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:15.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:14 smithi104 ceph-mon[36624]: pgmap v564: 129 pgs: 129 active+clean; 583 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 316 KiB/s wr, 47 op/s 2024-04-12T21:21:15.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:14 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:15.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:14 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:21:15.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:14 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2954636360,v1:172.21.15.153:6839/2954636360] up:active 2024-04-12T21:21:15.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:14 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:21:15.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:14 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:15.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:14 smithi081 ceph-mon[31453]: pgmap v564: 129 pgs: 129 active+clean; 583 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 316 KiB/s wr, 47 op/s 2024-04-12T21:21:16.301 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:15 smithi153 podman[70970]: 2024-04-12 21:21:15.827505472 +0000 UTC m=+1.508102844 container remove 8f8cf6c95a3afc2a1ba9f8aa517de3f6e287a628740e56258529eaae83d552a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-12T21:21:16.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:15 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:21:16.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:16 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-12T21:21:16.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:16 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 22.531s CPU time. 2024-04-12T21:21:17.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:17 smithi081 ceph-mon[31453]: pgmap v565: 129 pgs: 129 active+clean; 585 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 516 KiB/s wr, 237 op/s 2024-04-12T21:21:18.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:17 smithi153 ceph-mon[36569]: pgmap v565: 129 pgs: 129 active+clean; 585 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 516 KiB/s wr, 237 op/s 2024-04-12T21:21:18.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:17 smithi104 ceph-mon[36624]: pgmap v565: 129 pgs: 129 active+clean; 585 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 516 KiB/s wr, 237 op/s 2024-04-12T21:21:18.183 INFO:tasks.workunit.client.0.smithi081.stdout: 105 34 41 1 20 3986 43 2024-04-12T21:21:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:19 smithi081 ceph-mon[31453]: pgmap v566: 129 pgs: 129 active+clean; 585 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 516 KiB/s wr, 247 op/s 2024-04-12T21:21:20.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:19 smithi153 ceph-mon[36569]: pgmap v566: 129 pgs: 129 active+clean; 585 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 516 KiB/s wr, 247 op/s 2024-04-12T21:21:20.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:19 smithi104 ceph-mon[36624]: pgmap v566: 129 pgs: 129 active+clean; 585 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 516 KiB/s wr, 247 op/s 2024-04-12T21:21:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:21 smithi081 ceph-mon[31453]: pgmap v567: 129 pgs: 129 active+clean; 586 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 510 KiB/s wr, 230 op/s 2024-04-12T21:21:22.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:21 smithi153 ceph-mon[36569]: pgmap v567: 129 pgs: 129 active+clean; 586 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 510 KiB/s wr, 230 op/s 2024-04-12T21:21:22.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:21 smithi104 ceph-mon[36624]: pgmap v567: 129 pgs: 129 active+clean; 586 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 510 KiB/s wr, 230 op/s 2024-04-12T21:21:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:22.943+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:21:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:22.944+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:21:22.940 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:22.944+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:21:22.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:22.944+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:21:22.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:22.944+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:21:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:23 smithi081 ceph-mon[31453]: pgmap v568: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 551 KiB/s wr, 215 op/s 2024-04-12T21:21:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:21:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:23 smithi153 ceph-mon[36569]: pgmap v568: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 551 KiB/s wr, 215 op/s 2024-04-12T21:21:24.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:21:24.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:23 smithi104 ceph-mon[36624]: pgmap v568: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 551 KiB/s wr, 215 op/s 2024-04-12T21:21:24.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:21:24.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:21:24.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:21:24.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:21:25.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:21:25.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:21:25.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:21:25.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:21:25.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:21:25.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:21:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:25 smithi081 ceph-mon[31453]: pgmap v569: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 570 KiB/s wr, 217 op/s 2024-04-12T21:21:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:25 smithi153 ceph-mon[36569]: pgmap v569: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 570 KiB/s wr, 217 op/s 2024-04-12T21:21:26.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:25 smithi104 ceph-mon[36624]: pgmap v569: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 570 KiB/s wr, 217 op/s 2024-04-12T21:21:26.615 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:21:26.615 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 systemd[1]: Stopped Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:21:26.615 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 22.531s CPU time. 2024-04-12T21:21:26.615 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 systemd[1]: Starting Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:21:26.615 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 podman[71195]: 2024-04-12 21:21:26.445430931 +0000 UTC m=+0.018351344 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:21:26.892 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 podman[71195]: 2024-04-12 21:21:26.62029749 +0000 UTC m=+0.193217896 container create 2a79348821c94472bc23ebd5f12e94975b57e507a82c3f38014e8ad19df418e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:21:26.893 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:26 smithi153 ceph-mon[36569]: pgmap v570: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 351 KiB/s wr, 184 op/s 2024-04-12T21:21:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:26 smithi081 ceph-mon[31453]: pgmap v570: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 351 KiB/s wr, 184 op/s 2024-04-12T21:21:27.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:26 smithi104 ceph-mon[36624]: pgmap v570: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 351 KiB/s wr, 184 op/s 2024-04-12T21:21:27.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 podman[71195]: 2024-04-12 21:21:26.897830365 +0000 UTC m=+0.470750776 container init 2a79348821c94472bc23ebd5f12e94975b57e507a82c3f38014e8ad19df418e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409) 2024-04-12T21:21:27.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 podman[71195]: 2024-04-12 21:21:26.901332113 +0000 UTC m=+0.474252506 container start 2a79348821c94472bc23ebd5f12e94975b57e507a82c3f38014e8ad19df418e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-12T21:21:27.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 ceph-mds[71214]: -- 172.21.15.153:0/1968642875 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f3cdf19a40 con 0x55f3cdeec800 2024-04-12T21:21:27.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: starting mds.i at 2024-04-12T21:21:27.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:26 smithi153 bash[71195]: 2a79348821c94472bc23ebd5f12e94975b57e507a82c3f38014e8ad19df418e7 2024-04-12T21:21:27.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:27 smithi153 systemd[1]: Started Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:21:27.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:21:26.939+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:21:27.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: daemon mds.i restarted 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: osdmap e99: 12 total, 12 up, 12 in 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:boot 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:21:27.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:27 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: daemon mds.i restarted 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: osdmap e99: 12 total, 12 up, 12 in 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:boot 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:21:28.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:27 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:21:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: daemon mds.i restarted 2024-04-12T21:21:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:21:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: osdmap e99: 12 total, 12 up, 12 in 2024-04-12T21:21:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:boot 2024-04-12T21:21:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:21:28.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:21:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:21:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:21:28.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:27 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:21:28.183 INFO:tasks.workunit.client.0.smithi081.stdout: 105 0 0 0 0 0 0 2024-04-12T21:21:28.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:28 smithi081 ceph-mon[31453]: pgmap v572: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 168 KiB/s wr, 14 op/s 2024-04-12T21:21:29.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:28 smithi153 ceph-mon[36569]: pgmap v572: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 168 KiB/s wr, 14 op/s 2024-04-12T21:21:29.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:28 smithi104 ceph-mon[36624]: pgmap v572: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 168 KiB/s wr, 14 op/s 2024-04-12T21:21:29.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:29 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:reconnect 2024-04-12T21:21:29.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:29 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:21:29.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:29 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0409995 2024-04-12T21:21:30.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:29 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:reconnect 2024-04-12T21:21:30.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:29 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:21:30.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:29 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0409995 2024-04-12T21:21:30.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:29 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:reconnect 2024-04-12T21:21:30.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:29 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:21:30.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:29 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0409995 2024-04-12T21:21:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:30 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:rejoin 2024-04-12T21:21:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:30 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:21:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:30 smithi153 ceph-mon[36569]: pgmap v573: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 129 KiB/s wr, 12 op/s 2024-04-12T21:21:31.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:30 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:rejoin 2024-04-12T21:21:31.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:30 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:21:31.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:30 smithi104 ceph-mon[36624]: pgmap v573: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 129 KiB/s wr, 12 op/s 2024-04-12T21:21:31.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:30 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:rejoin 2024-04-12T21:21:31.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:30 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:21:31.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:30 smithi081 ceph-mon[31453]: pgmap v573: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 129 KiB/s wr, 12 op/s 2024-04-12T21:21:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:31 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:clientreplay 2024-04-12T21:21:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:31 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:21:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:31 smithi153 ceph-mon[36569]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:21:32.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:clientreplay 2024-04-12T21:21:32.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:21:32.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-mon[36624]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: *** Caught signal (Aborted) ** 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: in thread 7f581f217700 thread_name:ms_dispatch 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f582cdd1d20] 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2: gsignal() 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 3: abort() 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55ef9620b268] 2024-04-12T21:21:32.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55ef962a70fc] 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55ef965dbf0a] 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 7: (MDLog::trim()+0xca9) [0x55ef965dd0c9] 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55ef965dd5a8] 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f582bcbdb23] 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f582cdc71ca] 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 11: clone() 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2024-04-12T21:21:31.829+0000 7f581f217700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: in thread 7f581f217700 thread_name:ms_dispatch 2024-04-12T21:21:32.079 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f582cdd1d20] 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2: gsignal() 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 3: abort() 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55ef9620b268] 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55ef962a70fc] 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55ef965dbf0a] 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 7: (MDLog::trim()+0xca9) [0x55ef965dd0c9] 2024-04-12T21:21:32.080 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55ef965dd5a8] 2024-04-12T21:21:32.081 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f582bcbdb23] 2024-04-12T21:21:32.081 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f582cdc71ca] 2024-04-12T21:21:32.081 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 11: clone() 2024-04-12T21:21:32.081 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:32.081 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2024-04-12T21:21:32.081 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 0> 2024-04-12T21:21:31.829+0000 7f581f217700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:32.082 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: in thread 7f581f217700 thread_name:ms_dispatch 2024-04-12T21:21:32.082 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2024-04-12T21:21:32.082 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:32.082 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f582cdd1d20] 2024-04-12T21:21:32.082 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2: gsignal() 2024-04-12T21:21:32.082 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 3: abort() 2024-04-12T21:21:32.083 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55ef9620b268] 2024-04-12T21:21:32.083 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55ef962a70fc] 2024-04-12T21:21:32.083 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55ef965dbf0a] 2024-04-12T21:21:32.083 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 7: (MDLog::trim()+0xca9) [0x55ef965dd0c9] 2024-04-12T21:21:32.083 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55ef965dd5a8] 2024-04-12T21:21:32.083 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f582bcbdb23] 2024-04-12T21:21:32.084 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f582cdc71ca] 2024-04-12T21:21:32.084 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 11: clone() 2024-04-12T21:21:32.084 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:32.084 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2024-04-12T21:21:32.084 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: -18> 2024-04-12T21:21:31.829+0000 7f581f217700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:32.084 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: in thread 7f581f217700 thread_name:ms_dispatch 2024-04-12T21:21:32.084 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2024-04-12T21:21:32.085 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:32.085 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f582cdd1d20] 2024-04-12T21:21:32.085 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2: gsignal() 2024-04-12T21:21:32.085 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 3: abort() 2024-04-12T21:21:32.085 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55ef9620b268] 2024-04-12T21:21:32.085 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55ef962a70fc] 2024-04-12T21:21:32.085 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55ef965dbf0a] 2024-04-12T21:21:32.086 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 7: (MDLog::trim()+0xca9) [0x55ef965dd0c9] 2024-04-12T21:21:32.086 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55ef965dd5a8] 2024-04-12T21:21:32.086 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f582bcbdb23] 2024-04-12T21:21:32.086 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f582cdc71ca] 2024-04-12T21:21:32.086 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 11: clone() 2024-04-12T21:21:32.086 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:32.087 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69186]: 2024-04-12T21:21:32.087 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:31 smithi104 podman[69357]: 2024-04-12 21:21:31.887238087 +0000 UTC m=+0.019245643 container died e8185e4201b3306d6c930ce5d5503bd46aeb9d7619eeb992e5d4d5ef277c300f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, ceph=True, org.label-schema.license=GPLv2) 2024-04-12T21:21:32.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:31 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:clientreplay 2024-04-12T21:21:32.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:31 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:21:32.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:31 smithi081 ceph-mon[31453]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:21:33.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:32 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:32 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:21:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:32 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:active 2024-04-12T21:21:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:32 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:21:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:32 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:32 smithi153 ceph-mon[36569]: pgmap v574: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 24 KiB/s wr, 61 op/s 2024-04-12T21:21:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:32 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:32 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:21:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:32 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:active 2024-04-12T21:21:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:32 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:21:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:32 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:32 smithi104 ceph-mon[36624]: pgmap v574: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 24 KiB/s wr, 61 op/s 2024-04-12T21:21:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:32 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:32 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:21:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:32 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/115657957,v1:172.21.15.104:6839/115657957] up:active 2024-04-12T21:21:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:32 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-12T21:21:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:32 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:32 smithi081 ceph-mon[31453]: pgmap v574: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 24 KiB/s wr, 61 op/s 2024-04-12T21:21:35.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:35 smithi081 ceph-mon[31453]: pgmap v575: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 818 B/s wr, 100 op/s 2024-04-12T21:21:36.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:35 smithi153 ceph-mon[36569]: pgmap v575: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 818 B/s wr, 100 op/s 2024-04-12T21:21:36.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:35 smithi104 ceph-mon[36624]: pgmap v575: 129 pgs: 129 active+clean; 587 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 818 B/s wr, 100 op/s 2024-04-12T21:21:37.077 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:36 smithi104 podman[69357]: 2024-04-12 21:21:36.829128361 +0000 UTC m=+4.961135955 container remove e8185e4201b3306d6c930ce5d5503bd46aeb9d7619eeb992e5d4d5ef277c300f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-12T21:21:37.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:36 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:21:37.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:37 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-12T21:21:37.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:37 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 22.170s CPU time. 2024-04-12T21:21:37.853 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:37 smithi153 ceph-mon[36569]: pgmap v576: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 552 KiB/s wr, 268 op/s 2024-04-12T21:21:37.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:37 smithi081 ceph-mon[31453]: pgmap v576: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 552 KiB/s wr, 268 op/s 2024-04-12T21:21:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:37.950+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:21:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:37.950+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:21:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:37.950+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:21:38.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:37 smithi104 ceph-mon[36624]: pgmap v576: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 552 KiB/s wr, 268 op/s 2024-04-12T21:21:38.183 INFO:tasks.workunit.client.0.smithi081.stdout: 105 2 0 5 1 91 0 2024-04-12T21:21:39.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:39 smithi081 ceph-mon[31453]: pgmap v577: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 476 KiB/s wr, 231 op/s 2024-04-12T21:21:40.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:39 smithi153 ceph-mon[36569]: pgmap v577: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 476 KiB/s wr, 231 op/s 2024-04-12T21:21:40.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:39 smithi104 ceph-mon[36624]: pgmap v577: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 476 KiB/s wr, 231 op/s 2024-04-12T21:21:41.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:41 smithi081 ceph-mon[31453]: pgmap v578: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 460 KiB/s wr, 224 op/s 2024-04-12T21:21:42.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:41 smithi153 ceph-mon[36569]: pgmap v578: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 460 KiB/s wr, 224 op/s 2024-04-12T21:21:42.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:41 smithi104 ceph-mon[36624]: pgmap v578: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 460 KiB/s wr, 224 op/s 2024-04-12T21:21:43.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:42 smithi153 ceph-mon[36569]: pgmap v579: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 460 KiB/s wr, 224 op/s 2024-04-12T21:21:43.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:42 smithi104 ceph-mon[36624]: pgmap v579: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 460 KiB/s wr, 224 op/s 2024-04-12T21:21:43.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:42 smithi081 ceph-mon[31453]: pgmap v579: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 460 KiB/s wr, 224 op/s 2024-04-12T21:21:45.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:45 smithi081 ceph-mon[31453]: pgmap v580: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 460 KiB/s wr, 177 op/s 2024-04-12T21:21:46.051 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:45 smithi104 ceph-mon[36624]: pgmap v580: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 460 KiB/s wr, 177 op/s 2024-04-12T21:21:46.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:45 smithi153 ceph-mon[36569]: pgmap v580: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 460 KiB/s wr, 177 op/s 2024-04-12T21:21:47.299 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:21:47.299 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 systemd[1]: Stopped Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:21:47.299 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 22.170s CPU time. 2024-04-12T21:21:47.299 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 systemd[1]: Starting Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:21:47.577 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 podman[69465]: 2024-04-12 21:21:47.207310422 +0000 UTC m=+0.017658407 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:21:47.577 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 podman[69465]: 2024-04-12 21:21:47.387609531 +0000 UTC m=+0.197957467 container create 50ff63bae02b3808cb48b19929c7ebe8eef8fd573bb2119250fd1c428e4ddf0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409) 2024-04-12T21:21:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:47 smithi081 ceph-mon[31453]: pgmap v581: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 463 KiB/s wr, 144 op/s 2024-04-12T21:21:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:47 smithi081 ceph-mon[31453]: Replacing daemon mds.b as rank 0 with standby daemon mds.i 2024-04-12T21:21:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:47 smithi081 ceph-mon[31453]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-12T21:21:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:47 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:47 smithi081 ceph-mon[31453]: osdmap e100: 12 total, 12 up, 12 in 2024-04-12T21:21:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:47 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:replay} 10 up:standby 2024-04-12T21:21:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:47 smithi153 ceph-mon[36569]: pgmap v581: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 463 KiB/s wr, 144 op/s 2024-04-12T21:21:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:47 smithi153 ceph-mon[36569]: Replacing daemon mds.b as rank 0 with standby daemon mds.i 2024-04-12T21:21:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:47 smithi153 ceph-mon[36569]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-12T21:21:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:47 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:47 smithi153 ceph-mon[36569]: osdmap e100: 12 total, 12 up, 12 in 2024-04-12T21:21:48.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:47 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:replay} 10 up:standby 2024-04-12T21:21:48.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:47 smithi104 ceph-mon[36624]: pgmap v581: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 463 KiB/s wr, 144 op/s 2024-04-12T21:21:48.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:47 smithi104 ceph-mon[36624]: Replacing daemon mds.b as rank 0 with standby daemon mds.i 2024-04-12T21:21:48.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:47 smithi104 ceph-mon[36624]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-12T21:21:48.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:47 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:21:48.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:47 smithi104 ceph-mon[36624]: osdmap e100: 12 total, 12 up, 12 in 2024-04-12T21:21:48.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:47 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:replay} 10 up:standby 2024-04-12T21:21:48.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 podman[69465]: 2024-04-12 21:21:47.616276227 +0000 UTC m=+0.426624188 container init 50ff63bae02b3808cb48b19929c7ebe8eef8fd573bb2119250fd1c428e4ddf0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:21:48.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 podman[69465]: 2024-04-12 21:21:47.618964478 +0000 UTC m=+0.429312423 container start 50ff63bae02b3808cb48b19929c7ebe8eef8fd573bb2119250fd1c428e4ddf0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-12T21:21:48.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 ceph-mds[69485]: -- 172.21.15.104:0/3056809773 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5596647c1a40 con 0x559664794800 2024-04-12T21:21:48.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: starting mds.b at 2024-04-12T21:21:48.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 bash[69465]: 50ff63bae02b3808cb48b19929c7ebe8eef8fd573bb2119250fd1c428e4ddf0c 2024-04-12T21:21:48.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:21:47 smithi104 systemd[1]: Started Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:21:48.183 INFO:tasks.workunit.client.0.smithi081.stdout: 105 0 0 0 0 0 0 2024-04-12T21:21:49.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:49 smithi081 ceph-mon[31453]: pgmap v583: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-04-12T21:21:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:49 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:boot 2024-04-12T21:21:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:49 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:reconnect 2024-04-12T21:21:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:49 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:21:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:21:49.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:49 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0179997 2024-04-12T21:21:50.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:49 smithi153 ceph-mon[36569]: pgmap v583: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-04-12T21:21:50.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:49 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:boot 2024-04-12T21:21:50.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:49 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:reconnect 2024-04-12T21:21:50.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:49 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:21:50.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:21:50.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:49 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0179997 2024-04-12T21:21:50.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:49 smithi104 ceph-mon[36624]: pgmap v583: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4.0 KiB/s wr, 4 op/s 2024-04-12T21:21:50.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:49 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:boot 2024-04-12T21:21:50.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:49 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:reconnect 2024-04-12T21:21:50.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:49 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:reconnect} 11 up:standby 2024-04-12T21:21:50.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-12T21:21:50.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:49 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0179997 2024-04-12T21:21:50.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:50 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:rejoin 2024-04-12T21:21:50.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:50 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:21:50.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:50 smithi081 ceph-mon[31453]: pgmap v584: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-04-12T21:21:51.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:50 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:rejoin 2024-04-12T21:21:51.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:50 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:21:51.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:50 smithi153 ceph-mon[36569]: pgmap v584: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-04-12T21:21:51.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:50 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:rejoin 2024-04-12T21:21:51.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:50 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:rejoin} 11 up:standby 2024-04-12T21:21:51.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:50 smithi104 ceph-mon[36624]: pgmap v584: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.0 KiB/s wr, 3 op/s 2024-04-12T21:21:52.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:clientreplay 2024-04-12T21:21:52.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:21:52.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-mon[36569]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:21:52.054 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:21:52.054 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: *** Caught signal (Aborted) ** 2024-04-12T21:21:52.054 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: in thread 7f14c1c36700 thread_name:ms_dispatch 2024-04-12T21:21:52.054 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:52.054 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14cf7f0d20] 2024-04-12T21:21:52.055 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2: gsignal() 2024-04-12T21:21:52.055 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 3: abort() 2024-04-12T21:21:52.055 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f3cb59e268] 2024-04-12T21:21:52.055 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f3cb63a0fc] 2024-04-12T21:21:52.055 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f3cb96ef0a] 2024-04-12T21:21:52.055 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 7: (MDLog::trim()+0xca9) [0x55f3cb9700c9] 2024-04-12T21:21:52.056 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f3cb9705a8] 2024-04-12T21:21:52.056 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f14ce6dcb23] 2024-04-12T21:21:52.056 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f14cf7e61ca] 2024-04-12T21:21:52.056 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 11: clone() 2024-04-12T21:21:52.056 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2024-04-12T21:21:51.797+0000 7f14c1c36700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:52.056 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: in thread 7f14c1c36700 thread_name:ms_dispatch 2024-04-12T21:21:52.057 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2024-04-12T21:21:52.057 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:52.057 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14cf7f0d20] 2024-04-12T21:21:52.057 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2: gsignal() 2024-04-12T21:21:52.057 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 3: abort() 2024-04-12T21:21:52.057 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f3cb59e268] 2024-04-12T21:21:52.058 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f3cb63a0fc] 2024-04-12T21:21:52.058 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f3cb96ef0a] 2024-04-12T21:21:52.058 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 7: (MDLog::trim()+0xca9) [0x55f3cb9700c9] 2024-04-12T21:21:52.058 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f3cb9705a8] 2024-04-12T21:21:52.058 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f14ce6dcb23] 2024-04-12T21:21:52.058 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f14cf7e61ca] 2024-04-12T21:21:52.059 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 11: clone() 2024-04-12T21:21:52.059 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:52.059 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2024-04-12T21:21:52.059 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 0> 2024-04-12T21:21:51.797+0000 7f14c1c36700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:52.059 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: in thread 7f14c1c36700 thread_name:ms_dispatch 2024-04-12T21:21:52.059 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2024-04-12T21:21:52.059 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:52.060 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14cf7f0d20] 2024-04-12T21:21:52.060 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2: gsignal() 2024-04-12T21:21:52.060 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 3: abort() 2024-04-12T21:21:52.060 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f3cb59e268] 2024-04-12T21:21:52.060 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f3cb63a0fc] 2024-04-12T21:21:52.060 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f3cb96ef0a] 2024-04-12T21:21:52.061 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 7: (MDLog::trim()+0xca9) [0x55f3cb9700c9] 2024-04-12T21:21:52.061 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f3cb9705a8] 2024-04-12T21:21:52.061 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f14ce6dcb23] 2024-04-12T21:21:52.061 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f14cf7e61ca] 2024-04-12T21:21:52.061 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 11: clone() 2024-04-12T21:21:52.061 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:52.062 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2024-04-12T21:21:52.062 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: -1> 2024-04-12T21:21:51.797+0000 7f14c1c36700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:21:52.062 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: in thread 7f14c1c36700 thread_name:ms_dispatch 2024-04-12T21:21:52.062 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2024-04-12T21:21:52.062 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:21:52.062 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14cf7f0d20] 2024-04-12T21:21:52.063 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2: gsignal() 2024-04-12T21:21:52.063 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 3: abort() 2024-04-12T21:21:52.063 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f3cb59e268] 2024-04-12T21:21:52.063 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f3cb63a0fc] 2024-04-12T21:21:52.063 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f3cb96ef0a] 2024-04-12T21:21:52.063 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 7: (MDLog::trim()+0xca9) [0x55f3cb9700c9] 2024-04-12T21:21:52.064 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f3cb9705a8] 2024-04-12T21:21:52.064 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f14ce6dcb23] 2024-04-12T21:21:52.064 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f14cf7e61ca] 2024-04-12T21:21:52.064 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 11: clone() 2024-04-12T21:21:52.064 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:21:52.064 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71210]: 2024-04-12T21:21:52.064 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:51 smithi153 podman[71250]: 2024-04-12 21:21:51.856770564 +0000 UTC m=+0.018580537 container died 2a79348821c94472bc23ebd5f12e94975b57e507a82c3f38014e8ad19df418e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:21:52.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:51 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:clientreplay 2024-04-12T21:21:52.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:51 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:21:52.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:51 smithi104 ceph-mon[36624]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:21:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:51 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:clientreplay 2024-04-12T21:21:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:51 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:clientreplay} 11 up:standby 2024-04-12T21:21:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:51 smithi081 ceph-mon[31453]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:21:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:52 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:52 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:21:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:52 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:active 2024-04-12T21:21:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:52 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:21:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:52 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:52 smithi153 ceph-mon[36569]: pgmap v585: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 33 KiB/s wr, 93 op/s 2024-04-12T21:21:52.803 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:52 smithi153 podman[71250]: 2024-04-12 21:21:52.459975136 +0000 UTC m=+0.621785125 container remove 2a79348821c94472bc23ebd5f12e94975b57e507a82c3f38014e8ad19df418e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-12T21:21:52.803 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:52 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:21:52.803 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:52 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-12T21:21:52.803 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:21:52 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 22.234s CPU time. 2024-04-12T21:21:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:52.965+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:21:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:52.966+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-12T21:21:52.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:52.966+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-12T21:21:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:52.966+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:21:52.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:21:52.966+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:21:53.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:52 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:53.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:52 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:21:53.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:52 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:active 2024-04-12T21:21:53.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:52 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:21:53.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:52 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:53.044 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:52 smithi081 ceph-mon[31453]: pgmap v585: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 33 KiB/s wr, 93 op/s 2024-04-12T21:21:53.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:52 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:21:53.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:52 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:21:53.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:52 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2934748289,v1:172.21.15.153:6839/2934748289] up:active 2024-04-12T21:21:53.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:52 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 11 up:standby 2024-04-12T21:21:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:52 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:21:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:52 smithi104 ceph-mon[36624]: pgmap v585: 129 pgs: 129 active+clean; 592 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 33 KiB/s wr, 93 op/s 2024-04-12T21:21:55.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:55 smithi081 ceph-mon[31453]: pgmap v586: 129 pgs: 129 active+clean; 596 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 442 KiB/s wr, 133 op/s 2024-04-12T21:21:56.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:55 smithi153 ceph-mon[36569]: pgmap v586: 129 pgs: 129 active+clean; 596 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 442 KiB/s wr, 133 op/s 2024-04-12T21:21:56.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:55 smithi104 ceph-mon[36624]: pgmap v586: 129 pgs: 129 active+clean; 596 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 442 KiB/s wr, 133 op/s 2024-04-12T21:21:57.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:57 smithi081 ceph-mon[31453]: pgmap v587: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 520 KiB/s wr, 263 op/s 2024-04-12T21:21:58.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:57 smithi153 ceph-mon[36569]: pgmap v587: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 520 KiB/s wr, 263 op/s 2024-04-12T21:21:58.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:57 smithi104 ceph-mon[36624]: pgmap v587: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 520 KiB/s wr, 263 op/s 2024-04-12T21:21:58.183 INFO:tasks.workunit.client.0.smithi081.stdout: 105 12 1 9 1 257 1 2024-04-12T21:21:59.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:21:58 smithi153 ceph-mon[36569]: pgmap v588: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 462 KiB/s wr, 234 op/s 2024-04-12T21:21:59.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:21:58 smithi104 ceph-mon[36624]: pgmap v588: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 462 KiB/s wr, 234 op/s 2024-04-12T21:21:59.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:21:58 smithi081 ceph-mon[31453]: pgmap v588: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 462 KiB/s wr, 234 op/s 2024-04-12T21:22:01.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:01 smithi081 ceph-mon[31453]: pgmap v589: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 438 KiB/s wr, 221 op/s 2024-04-12T21:22:02.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:01 smithi153 ceph-mon[36569]: pgmap v589: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 438 KiB/s wr, 221 op/s 2024-04-12T21:22:02.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:01 smithi104 ceph-mon[36624]: pgmap v589: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 438 KiB/s wr, 221 op/s 2024-04-12T21:22:03.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:22:03.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:02 smithi153 systemd[1]: Stopped Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:22:03.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 22.234s CPU time. 2024-04-12T21:22:03.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:02 smithi153 systemd[1]: Starting Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:22:03.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:03 smithi153 podman[71349]: 2024-04-12 21:22:02.938981956 +0000 UTC m=+0.018694202 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:22:03.834 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:03 smithi153 ceph-mon[36569]: pgmap v590: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 438 KiB/s wr, 221 op/s 2024-04-12T21:22:03.835 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:03 smithi153 podman[71349]: 2024-04-12 21:22:03.569486047 +0000 UTC m=+0.649198316 container create 8810bdf2bd7aceeea91f749ae25e72d439ebd6b935788a508e999ab35eda366e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-12T21:22:03.880 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:03 smithi081 ceph-mon[31453]: pgmap v590: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 438 KiB/s wr, 221 op/s 2024-04-12T21:22:04.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:03 smithi104 ceph-mon[36624]: pgmap v590: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 438 KiB/s wr, 221 op/s 2024-04-12T21:22:04.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:22:03.883+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:22:04.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:03 smithi153 podman[71349]: 2024-04-12 21:22:03.840031783 +0000 UTC m=+0.919744018 container init 8810bdf2bd7aceeea91f749ae25e72d439ebd6b935788a508e999ab35eda366e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-12T21:22:04.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:03 smithi153 podman[71349]: 2024-04-12 21:22:03.843149455 +0000 UTC m=+0.922861685 container start 8810bdf2bd7aceeea91f749ae25e72d439ebd6b935788a508e999ab35eda366e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-12T21:22:04.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:03 smithi153 ceph-mds[71368]: -- 172.21.15.153:0/2515647751 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562c7e6e3a40 con 0x562c7e6b7000 2024-04-12T21:22:04.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:03 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: starting mds.i at 2024-04-12T21:22:04.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:03 smithi153 bash[71349]: 8810bdf2bd7aceeea91f749ae25e72d439ebd6b935788a508e999ab35eda366e 2024-04-12T21:22:04.302 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:03 smithi153 systemd[1]: Started Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:22:04.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: daemon mds.i restarted 2024-04-12T21:22:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:22:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: osdmap e101: 12 total, 12 up, 12 in 2024-04-12T21:22:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:boot 2024-04-12T21:22:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:22:04.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:22:04.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:22:04.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:22:04.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:22:04.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:04 smithi081 ceph-mon[31453]: pgmap v592: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 87 KiB/s wr, 135 op/s 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: daemon mds.i restarted 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: osdmap e101: 12 total, 12 up, 12 in 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:boot 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:22:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:04 smithi153 ceph-mon[36569]: pgmap v592: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 87 KiB/s wr, 135 op/s 2024-04-12T21:22:05.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: daemon mds.i restarted 2024-04-12T21:22:05.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:05.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:22:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: osdmap e101: 12 total, 12 up, 12 in 2024-04-12T21:22:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:boot 2024-04-12T21:22:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-12T21:22:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:22:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 11 up:standby, 1 failed 2024-04-12T21:22:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-12T21:22:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:replay} 11 up:standby 2024-04-12T21:22:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:04 smithi104 ceph-mon[36624]: pgmap v592: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 87 KiB/s wr, 135 op/s 2024-04-12T21:22:06.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:05 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:reconnect 2024-04-12T21:22:06.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:05 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:22:06.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:05 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139998 2024-04-12T21:22:06.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:05 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:reconnect 2024-04-12T21:22:06.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:05 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:22:06.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:05 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139998 2024-04-12T21:22:06.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:05 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:reconnect 2024-04-12T21:22:06.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:05 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:reconnect} 11 up:standby 2024-04-12T21:22:06.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:05 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139998 2024-04-12T21:22:07.202 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:06 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:rejoin 2024-04-12T21:22:07.203 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:06 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:22:07.203 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:06 smithi153 ceph-mon[36569]: pgmap v593: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.4 KiB/s wr, 5 op/s 2024-04-12T21:22:07.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:06 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:rejoin 2024-04-12T21:22:07.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:06 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:22:07.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:06 smithi081 ceph-mon[31453]: pgmap v593: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.4 KiB/s wr, 5 op/s 2024-04-12T21:22:07.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:06 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:rejoin 2024-04-12T21:22:07.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:06 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:rejoin} 11 up:standby 2024-04-12T21:22:07.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:06 smithi104 ceph-mon[36624]: pgmap v593: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 5.4 KiB/s wr, 5 op/s 2024-04-12T21:22:07.943 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:22:07.943 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: *** Caught signal (Aborted) ** 2024-04-12T21:22:07.943 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: in thread 7f79ac06e700 thread_name:ms_dispatch 2024-04-12T21:22:07.943 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:07.943 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79b9c28d20] 2024-04-12T21:22:07.943 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2: gsignal() 2024-04-12T21:22:07.943 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 3: abort() 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5596626ed268] 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5596627890fc] 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x559662abdf0a] 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 7: (MDLog::trim()+0xca9) [0x559662abf0c9] 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 8: (MDLog::log_trim_upkeep()+0x358) [0x559662abf5a8] 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f79b8b14b23] 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f79b9c1e1ca] 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 11: clone() 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2024-04-12T21:22:07.666+0000 7f79ac06e700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: in thread 7f79ac06e700 thread_name:ms_dispatch 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79b9c28d20] 2024-04-12T21:22:07.944 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2: gsignal() 2024-04-12T21:22:07.945 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 3: abort() 2024-04-12T21:22:07.945 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5596626ed268] 2024-04-12T21:22:07.945 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5596627890fc] 2024-04-12T21:22:07.945 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x559662abdf0a] 2024-04-12T21:22:07.945 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 7: (MDLog::trim()+0xca9) [0x559662abf0c9] 2024-04-12T21:22:07.945 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 8: (MDLog::log_trim_upkeep()+0x358) [0x559662abf5a8] 2024-04-12T21:22:07.945 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f79b8b14b23] 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f79b9c1e1ca] 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 11: clone() 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 0> 2024-04-12T21:22:07.666+0000 7f79ac06e700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: in thread 7f79ac06e700 thread_name:ms_dispatch 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79b9c28d20] 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2: gsignal() 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 3: abort() 2024-04-12T21:22:07.946 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5596626ed268] 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5596627890fc] 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x559662abdf0a] 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 7: (MDLog::trim()+0xca9) [0x559662abf0c9] 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 8: (MDLog::log_trim_upkeep()+0x358) [0x559662abf5a8] 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f79b8b14b23] 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f79b9c1e1ca] 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 11: clone() 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: -10> 2024-04-12T21:22:07.666+0000 7f79ac06e700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: in thread 7f79ac06e700 thread_name:ms_dispatch 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f79b9c28d20] 2024-04-12T21:22:07.947 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2: gsignal() 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 3: abort() 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5596626ed268] 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5596627890fc] 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x559662abdf0a] 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 7: (MDLog::trim()+0xca9) [0x559662abf0c9] 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 8: (MDLog::log_trim_upkeep()+0x358) [0x559662abf5a8] 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f79b8b14b23] 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f79b9c1e1ca] 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 11: clone() 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b[69481]: 2024-04-12T21:22:07.948 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:07 smithi104 podman[69522]: 2024-04-12 21:22:07.724648182 +0000 UTC m=+0.018385500 container died 50ff63bae02b3808cb48b19929c7ebe8eef8fd573bb2119250fd1c428e4ddf0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-12T21:22:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:07.982+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:22:07.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:07.982+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:22:07.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:07.982+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:22:08.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:clientreplay 2024-04-12T21:22:08.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:07 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:22:08.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:07 smithi081 ceph-mon[31453]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:22:08.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:07 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:08.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:07 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:22:08.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:active 2024-04-12T21:22:08.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:07 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:22:08.183 INFO:tasks.workunit.client.0.smithi081.stdout: 106 398 19 393 7 1036 15 2024-04-12T21:22:08.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:clientreplay 2024-04-12T21:22:08.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:07 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:22:08.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:07 smithi153 ceph-mon[36569]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:22:08.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:07 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:08.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:07 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:22:08.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:active 2024-04-12T21:22:08.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:07 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:22:08.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:clientreplay 2024-04-12T21:22:08.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=b=up:clientreplay} 11 up:standby 2024-04-12T21:22:08.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-mon[36624]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-12T21:22:08.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:08.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:22:08.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6838/2849509424,v1:172.21.15.104:6839/2849509424] up:active 2024-04-12T21:22:08.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:07 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-04-12T21:22:09.287 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:08 smithi104 ceph-mon[36624]: pgmap v594: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.4 KiB/s wr, 17 op/s 2024-04-12T21:22:09.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:08 smithi153 ceph-mon[36569]: pgmap v594: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.4 KiB/s wr, 17 op/s 2024-04-12T21:22:09.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:08 smithi081 ceph-mon[31453]: pgmap v594: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 5.4 KiB/s wr, 17 op/s 2024-04-12T21:22:10.077 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:09 smithi104 podman[69522]: 2024-04-12 21:22:09.593740818 +0000 UTC m=+1.887478156 container remove 50ff63bae02b3808cb48b19929c7ebe8eef8fd573bb2119250fd1c428e4ddf0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-12T21:22:10.077 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:09 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:22:10.077 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:09 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-12T21:22:10.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:09 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 21.213s CPU time. 2024-04-12T21:22:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:11 smithi081 ceph-mon[31453]: pgmap v595: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 818 B/s wr, 40 op/s 2024-04-12T21:22:12.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:11 smithi153 ceph-mon[36569]: pgmap v595: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 818 B/s wr, 40 op/s 2024-04-12T21:22:12.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:11 smithi104 ceph-mon[36624]: pgmap v595: 129 pgs: 129 active+clean; 597 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 818 B/s wr, 40 op/s 2024-04-12T21:22:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:12.753+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4005880 con 0x7f1704076980 2024-04-12T21:22:12.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:12.754+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4005880 con 0x7f1704076980 2024-04-12T21:22:12.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:12.797+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d0005850 con 0x563c9e245d20 2024-04-12T21:22:12.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:12.798+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d0005850 con 0x563c9e245d20 2024-04-12T21:22:13.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:12 smithi153 ceph-mon[36569]: pgmap v596: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 131 KiB/s wr, 93 op/s 2024-04-12T21:22:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:12 smithi104 ceph-mon[36624]: pgmap v596: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 131 KiB/s wr, 93 op/s 2024-04-12T21:22:13.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:12 smithi081 ceph-mon[31453]: pgmap v596: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 131 KiB/s wr, 93 op/s 2024-04-12T21:22:15.876 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:15 smithi081 ceph-mon[31453]: pgmap v597: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 135 KiB/s wr, 88 op/s 2024-04-12T21:22:16.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:15 smithi153 ceph-mon[36569]: pgmap v597: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 135 KiB/s wr, 88 op/s 2024-04-12T21:22:16.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:15 smithi104 ceph-mon[36624]: pgmap v597: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 135 KiB/s wr, 88 op/s 2024-04-12T21:22:17.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:16 smithi153 ceph-mon[36569]: pgmap v598: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 83 op/s 2024-04-12T21:22:17.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:16 smithi104 ceph-mon[36624]: pgmap v598: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 83 op/s 2024-04-12T21:22:17.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:16 smithi081 ceph-mon[31453]: pgmap v598: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 83 op/s 2024-04-12T21:22:18.184 INFO:tasks.workunit.client.0.smithi081.stdout: 106 0 0 0 0 0 0 2024-04-12T21:22:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:19 smithi081 ceph-mon[31453]: pgmap v599: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 81 op/s 2024-04-12T21:22:20.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:19 smithi153 ceph-mon[36569]: pgmap v599: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 81 op/s 2024-04-12T21:22:20.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:19 smithi104 ceph-mon[36624]: pgmap v599: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 81 op/s 2024-04-12T21:22:20.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:19 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:22:20.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:19 smithi104 systemd[1]: Stopped Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:22:20.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:19 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Consumed 21.213s CPU time. 2024-04-12T21:22:20.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:19 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Start request repeated too quickly. 2024-04-12T21:22:20.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:19 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-12T21:22:20.078 INFO:journalctl@ceph.mds.b.smithi104.stdout:Apr 12 21:22:19 smithi104 systemd[1]: Failed to start Ceph mds.b for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:22:20.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:20 smithi081 ceph-mon[31453]: pgmap v600: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 178 KiB/s wr, 72 op/s 2024-04-12T21:22:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:20 smithi153 ceph-mon[36569]: pgmap v600: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 178 KiB/s wr, 72 op/s 2024-04-12T21:22:21.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:20 smithi104 ceph-mon[36624]: pgmap v600: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 178 KiB/s wr, 72 op/s 2024-04-12T21:22:22.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:22 smithi081 ceph-mon[31453]: Replacing daemon mds.b as rank 0 with standby daemon mds.i 2024-04-12T21:22:22.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:22 smithi081 ceph-mon[31453]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-12T21:22:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:22 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:22 smithi081 ceph-mon[31453]: osdmap e102: 12 total, 12 up, 12 in 2024-04-12T21:22:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:22 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:replay} 10 up:standby 2024-04-12T21:22:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:22 smithi153 ceph-mon[36569]: Replacing daemon mds.b as rank 0 with standby daemon mds.i 2024-04-12T21:22:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:22 smithi153 ceph-mon[36569]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-12T21:22:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:22 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:22 smithi153 ceph-mon[36569]: osdmap e102: 12 total, 12 up, 12 in 2024-04-12T21:22:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:22 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:replay} 10 up:standby 2024-04-12T21:22:22.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:22 smithi104 ceph-mon[36624]: Replacing daemon mds.b as rank 0 with standby daemon mds.i 2024-04-12T21:22:22.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:22 smithi104 ceph-mon[36624]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-12T21:22:22.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:22 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:22.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:22 smithi104 ceph-mon[36624]: osdmap e102: 12 total, 12 up, 12 in 2024-04-12T21:22:22.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:22 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:replay} 10 up:standby 2024-04-12T21:22:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:22.997+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:22:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:22.997+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:22:22.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:22.997+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:22:23.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:23 smithi081 ceph-mon[31453]: pgmap v602: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 83 KiB/s wr, 8 op/s 2024-04-12T21:22:23.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:23 smithi153 ceph-mon[36569]: pgmap v602: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 83 KiB/s wr, 8 op/s 2024-04-12T21:22:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:23 smithi104 ceph-mon[36624]: pgmap v602: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 83 KiB/s wr, 8 op/s 2024-04-12T21:22:24.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:24 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:reconnect 2024-04-12T21:22:24.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:24 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:reconnect} 10 up:standby 2024-04-12T21:22:24.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:24 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0349994 2024-04-12T21:22:24.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:24 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:reconnect 2024-04-12T21:22:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:24 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:reconnect} 10 up:standby 2024-04-12T21:22:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:24 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0349994 2024-04-12T21:22:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:24 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:reconnect 2024-04-12T21:22:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:24 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:reconnect} 10 up:standby 2024-04-12T21:22:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:24 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0349994 2024-04-12T21:22:25.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:rejoin 2024-04-12T21:22:25.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:rejoin} 10 up:standby 2024-04-12T21:22:25.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: pgmap v603: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 70 KiB/s wr, 7 op/s 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:rejoin 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:rejoin} 10 up:standby 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: pgmap v603: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 70 KiB/s wr, 7 op/s 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:rejoin 2024-04-12T21:22:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:rejoin} 10 up:standby 2024-04-12T21:22:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:22:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: pgmap v603: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 70 KiB/s wr, 7 op/s 2024-04-12T21:22:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:clientreplay 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=i=up:clientreplay} 10 up:standby 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:22:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:22:26.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:22:26.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:22:26.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:22:26.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:clientreplay 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=i=up:clientreplay} 10 up:standby 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:clientreplay 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=i=up:clientreplay} 10 up:standby 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: daemon mds.i is now active in filesystem cephfs as rank 0 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:22:26.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:22:26.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:22:26.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:22:27.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:22:27.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:22:27.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:27.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:22:27.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:active 2024-04-12T21:22:27.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-04-12T21:22:27.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-04-12T21:22:27.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-mon[36569]: pgmap v604: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 KiB/s wr, 11 op/s 2024-04-12T21:22:27.554 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:22:27.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: *** Caught signal (Aborted) ** 2024-04-12T21:22:27.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: in thread 7fae7bba2700 thread_name:ms_dispatch 2024-04-12T21:22:27.555 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae8975cd20] 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2: gsignal() 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 3: abort() 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562c7b99b268] 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562c7ba370fc] 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562c7bd6bf0a] 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 7: (MDLog::trim()+0xca9) [0x562c7bd6d0c9] 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562c7bd6d5a8] 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fae88648b23] 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fae897521ca] 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 11: clone() 2024-04-12T21:22:27.556 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2024-04-12T21:22:27.302+0000 7fae7bba2700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: in thread 7fae7bba2700 thread_name:ms_dispatch 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae8975cd20] 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2: gsignal() 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 3: abort() 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562c7b99b268] 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562c7ba370fc] 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562c7bd6bf0a] 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 7: (MDLog::trim()+0xca9) [0x562c7bd6d0c9] 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562c7bd6d5a8] 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fae88648b23] 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fae897521ca] 2024-04-12T21:22:27.557 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 11: clone() 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 0> 2024-04-12T21:22:27.302+0000 7fae7bba2700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: in thread 7fae7bba2700 thread_name:ms_dispatch 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae8975cd20] 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2: gsignal() 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 3: abort() 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562c7b99b268] 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562c7ba370fc] 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562c7bd6bf0a] 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 7: (MDLog::trim()+0xca9) [0x562c7bd6d0c9] 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562c7bd6d5a8] 2024-04-12T21:22:27.558 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fae88648b23] 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fae897521ca] 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 11: clone() 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: -10> 2024-04-12T21:22:27.302+0000 7fae7bba2700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: in thread 7fae7bba2700 thread_name:ms_dispatch 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fae8975cd20] 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2: gsignal() 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 3: abort() 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562c7b99b268] 2024-04-12T21:22:27.559 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562c7ba370fc] 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562c7bd6bf0a] 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 7: (MDLog::trim()+0xca9) [0x562c7bd6d0c9] 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562c7bd6d5a8] 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fae88648b23] 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fae897521ca] 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 11: clone() 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i[71364]: 2024-04-12T21:22:27.560 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:27 smithi153 podman[71534]: 2024-04-12 21:22:27.364186739 +0000 UTC m=+0.018948653 container died 8810bdf2bd7aceeea91f749ae25e72d439ebd6b935788a508e999ab35eda366e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD) 2024-04-12T21:22:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:27 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:22:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:27 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:22:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:27 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:27 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:22:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:27 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:active 2024-04-12T21:22:27.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:27 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-04-12T21:22:27.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:27 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-04-12T21:22:27.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:27 smithi081 ceph-mon[31453]: pgmap v604: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 KiB/s wr, 11 op/s 2024-04-12T21:22:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:27 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:22:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:27 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:22:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:27 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:27 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:22:27.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:27 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6838/2387802483,v1:172.21.15.153:6839/2387802483] up:active 2024-04-12T21:22:27.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:27 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 10 up:standby 2024-04-12T21:22:27.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:27 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=i=up:active} 1 up:standby-replay 9 up:standby 2024-04-12T21:22:27.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:27 smithi104 ceph-mon[36624]: pgmap v604: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 16 KiB/s wr, 11 op/s 2024-04-12T21:22:28.184 INFO:tasks.workunit.client.0.smithi081.stdout: 106 2003 55 146 8 202 75 2024-04-12T21:22:28.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:28 smithi153 podman[71534]: 2024-04-12 21:22:28.12653802 +0000 UTC m=+0.781299922 container remove 8810bdf2bd7aceeea91f749ae25e72d439ebd6b935788a508e999ab35eda366e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-i, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-12T21:22:28.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:28 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:22:28.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:28 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-12T21:22:28.552 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:28 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 22.571s CPU time. 2024-04-12T21:22:29.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:28 smithi153 ceph-mon[36569]: pgmap v605: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 KiB/s wr, 18 op/s 2024-04-12T21:22:29.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:28 smithi104 ceph-mon[36624]: pgmap v605: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 KiB/s wr, 18 op/s 2024-04-12T21:22:29.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:28 smithi081 ceph-mon[31453]: pgmap v605: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 16 KiB/s wr, 18 op/s 2024-04-12T21:22:31.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:30 smithi153 ceph-mon[36569]: pgmap v606: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 17 KiB/s wr, 38 op/s 2024-04-12T21:22:31.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:30 smithi104 ceph-mon[36624]: pgmap v606: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 17 KiB/s wr, 38 op/s 2024-04-12T21:22:31.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:30 smithi081 ceph-mon[31453]: pgmap v606: 129 pgs: 129 active+clean; 598 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 17 KiB/s wr, 38 op/s 2024-04-12T21:22:33.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:33 smithi104 ceph-mon[36624]: pgmap v607: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 108 KiB/s wr, 67 op/s 2024-04-12T21:22:33.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:33 smithi081 ceph-mon[31453]: pgmap v607: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 108 KiB/s wr, 67 op/s 2024-04-12T21:22:33.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:33 smithi153 ceph-mon[36569]: pgmap v607: 129 pgs: 129 active+clean; 599 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 108 KiB/s wr, 67 op/s 2024-04-12T21:22:35.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:34 smithi153 ceph-mon[36569]: pgmap v608: 129 pgs: 129 active+clean; 600 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 115 KiB/s wr, 60 op/s 2024-04-12T21:22:35.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:34 smithi104 ceph-mon[36624]: pgmap v608: 129 pgs: 129 active+clean; 600 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 115 KiB/s wr, 60 op/s 2024-04-12T21:22:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:34 smithi081 ceph-mon[31453]: pgmap v608: 129 pgs: 129 active+clean; 600 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 115 KiB/s wr, 60 op/s 2024-04-12T21:22:37.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:36 smithi153 ceph-mon[36569]: pgmap v609: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 243 KiB/s wr, 72 op/s 2024-04-12T21:22:37.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:36 smithi104 ceph-mon[36624]: pgmap v609: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 243 KiB/s wr, 72 op/s 2024-04-12T21:22:37.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:36 smithi081 ceph-mon[31453]: pgmap v609: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 243 KiB/s wr, 72 op/s 2024-04-12T21:22:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:38.008+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:22:38.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:38.008+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:22:38.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:38.008+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:22:38.184 INFO:tasks.workunit.client.0.smithi081.stdout: 106 0 0 0 0 0 0 2024-04-12T21:22:38.854 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:38 smithi153 ceph-mon[36569]: pgmap v610: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 229 KiB/s wr, 64 op/s 2024-04-12T21:22:38.854 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:38 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:22:38.854 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:38 smithi153 systemd[1]: Stopped Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:22:38.854 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:38 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Consumed 22.571s CPU time. 2024-04-12T21:22:38.855 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:38 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Start request repeated too quickly. 2024-04-12T21:22:38.855 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:38 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-12T21:22:38.855 INFO:journalctl@ceph.mds.i.smithi153.stdout:Apr 12 21:22:38 smithi153 systemd[1]: Failed to start Ceph mds.i for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:22:39.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:38 smithi104 ceph-mon[36624]: pgmap v610: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 229 KiB/s wr, 64 op/s 2024-04-12T21:22:39.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:38 smithi081 ceph-mon[31453]: pgmap v610: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 229 KiB/s wr, 64 op/s 2024-04-12T21:22:41.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:40 smithi153 ceph-mon[36569]: pgmap v611: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 229 KiB/s wr, 58 op/s 2024-04-12T21:22:41.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:40 smithi104 ceph-mon[36624]: pgmap v611: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 229 KiB/s wr, 58 op/s 2024-04-12T21:22:41.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:40 smithi081 ceph-mon[31453]: pgmap v611: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 229 KiB/s wr, 58 op/s 2024-04-12T21:22:42.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:42 smithi081 ceph-mon[31453]: Replacing daemon mds.i as rank 0 with standby daemon mds.h 2024-04-12T21:22:42.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:42 smithi081 ceph-mon[31453]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-12T21:22:42.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:42 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:42.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:42 smithi081 ceph-mon[31453]: osdmap e103: 12 total, 12 up, 12 in 2024-04-12T21:22:42.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:42 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:22:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:42 smithi153 ceph-mon[36569]: Replacing daemon mds.i as rank 0 with standby daemon mds.h 2024-04-12T21:22:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:42 smithi153 ceph-mon[36569]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-12T21:22:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:42 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:42 smithi153 ceph-mon[36569]: osdmap e103: 12 total, 12 up, 12 in 2024-04-12T21:22:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:42 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:42 smithi104 ceph-mon[36624]: Replacing daemon mds.i as rank 0 with standby daemon mds.h 2024-04-12T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:42 smithi104 ceph-mon[36624]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-04-12T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:42 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:42 smithi104 ceph-mon[36624]: osdmap e103: 12 total, 12 up, 12 in 2024-04-12T21:22:42.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:42 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:22:43.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:43 smithi081 ceph-mon[31453]: pgmap v613: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 180 KiB/s wr, 18 op/s 2024-04-12T21:22:43.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:43 smithi153 ceph-mon[36569]: pgmap v613: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 180 KiB/s wr, 18 op/s 2024-04-12T21:22:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:43 smithi104 ceph-mon[36624]: pgmap v613: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 180 KiB/s wr, 18 op/s 2024-04-12T21:22:44.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:44 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:reconnect 2024-04-12T21:22:44.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:44 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:22:44.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:44 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0459995 2024-04-12T21:22:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:44 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:reconnect 2024-04-12T21:22:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:44 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:22:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:44 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0459995 2024-04-12T21:22:44.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:44 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:reconnect 2024-04-12T21:22:44.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:44 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:22:44.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:44 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0459995 2024-04-12T21:22:45.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:45 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:rejoin 2024-04-12T21:22:45.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:45 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:22:45.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:45 smithi081 ceph-mon[31453]: pgmap v614: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 154 KiB/s wr, 15 op/s 2024-04-12T21:22:45.728 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:45 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:rejoin 2024-04-12T21:22:45.728 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:45 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:22:45.728 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:45 smithi104 ceph-mon[36624]: pgmap v614: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 154 KiB/s wr, 15 op/s 2024-04-12T21:22:45.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:45 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:rejoin 2024-04-12T21:22:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:45 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:22:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:45 smithi153 ceph-mon[36569]: pgmap v614: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 154 KiB/s wr, 15 op/s 2024-04-12T21:22:46.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:46 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:clientreplay 2024-04-12T21:22:46.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:46 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:22:46.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:46 smithi153 ceph-mon[36569]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:22:46.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:clientreplay 2024-04-12T21:22:46.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:22:46.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-mon[36624]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:22:46.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:22:46.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: *** Caught signal (Aborted) ** 2024-04-12T21:22:46.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: in thread 7f93c49fa700 thread_name:ms_dispatch 2024-04-12T21:22:46.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:46.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93d25b4d20] 2024-04-12T21:22:46.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2: gsignal() 2024-04-12T21:22:46.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 3: abort() 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558e1f30b268] 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558e1f3a70fc] 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558e1f6dbf0a] 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 7: (MDLog::trim()+0xca9) [0x558e1f6dd0c9] 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558e1f6dd5a8] 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f93d14a0b23] 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f93d25aa1ca] 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 11: clone() 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2024-04-12T21:22:46.556+0000 7f93c49fa700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: in thread 7f93c49fa700 thread_name:ms_dispatch 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:46.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93d25b4d20] 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2: gsignal() 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 3: abort() 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558e1f30b268] 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558e1f3a70fc] 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558e1f6dbf0a] 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 7: (MDLog::trim()+0xca9) [0x558e1f6dd0c9] 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558e1f6dd5a8] 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f93d14a0b23] 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f93d25aa1ca] 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 11: clone() 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2024-04-12T21:22:46.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 0> 2024-04-12T21:22:46.556+0000 7f93c49fa700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: in thread 7f93c49fa700 thread_name:ms_dispatch 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93d25b4d20] 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2: gsignal() 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 3: abort() 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558e1f30b268] 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558e1f3a70fc] 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558e1f6dbf0a] 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 7: (MDLog::trim()+0xca9) [0x558e1f6dd0c9] 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558e1f6dd5a8] 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f93d14a0b23] 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f93d25aa1ca] 2024-04-12T21:22:46.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 11: clone() 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: -10> 2024-04-12T21:22:46.556+0000 7f93c49fa700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: in thread 7f93c49fa700 thread_name:ms_dispatch 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93d25b4d20] 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2: gsignal() 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 3: abort() 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558e1f30b268] 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558e1f3a70fc] 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558e1f6dbf0a] 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 7: (MDLog::trim()+0xca9) [0x558e1f6dd0c9] 2024-04-12T21:22:46.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558e1f6dd5a8] 2024-04-12T21:22:46.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f93d14a0b23] 2024-04-12T21:22:46.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f93d25aa1ca] 2024-04-12T21:22:46.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 11: clone() 2024-04-12T21:22:46.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:22:46.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[65917]: 2024-04-12T21:22:46.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:46 smithi104 podman[69712]: 2024-04-12 21:22:46.615660715 +0000 UTC m=+0.018877591 container died a442b0fc5a8921cb4a3be3b7c82830c8035020e9d03b3f79dcaccc0228d48ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:22:46.852 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:46 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:clientreplay 2024-04-12T21:22:46.853 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:46 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:22:46.853 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:46 smithi081 ceph-mon[31453]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:22:47.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:47 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:47 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:47 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:active 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:47 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:47 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:47 smithi104 ceph-mon[36624]: pgmap v615: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:47 smithi104 podman[69712]: 2024-04-12 21:22:47.236226262 +0000 UTC m=+0.639443137 container remove a442b0fc5a8921cb4a3be3b7c82830c8035020e9d03b3f79dcaccc0228d48ead (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:47 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:47 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-12T21:22:47.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:47 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 20.805s CPU time. 2024-04-12T21:22:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:47 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:47 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:22:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:47 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:active 2024-04-12T21:22:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:47 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:22:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:47 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:22:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:47 smithi153 ceph-mon[36569]: pgmap v615: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-12T21:22:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:47 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:22:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:47 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:22:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:47 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/435649651,v1:172.21.15.104:6837/435649651] up:active 2024-04-12T21:22:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:47 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:22:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:47 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:22:47.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:47 smithi081 ceph-mon[31453]: pgmap v615: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 19 KiB/s wr, 13 op/s 2024-04-12T21:22:48.184 INFO:tasks.workunit.client.0.smithi081.stdout: 106 4 1 3 1 4 2 2024-04-12T21:22:49.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:48 smithi153 ceph-mon[36569]: pgmap v616: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-12T21:22:49.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:48 smithi104 ceph-mon[36624]: pgmap v616: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-12T21:22:49.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:48 smithi081 ceph-mon[31453]: pgmap v616: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 19 KiB/s wr, 14 op/s 2024-04-12T21:22:51.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:50 smithi153 ceph-mon[36569]: pgmap v617: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 19 KiB/s wr, 22 op/s 2024-04-12T21:22:51.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:50 smithi104 ceph-mon[36624]: pgmap v617: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 19 KiB/s wr, 22 op/s 2024-04-12T21:22:51.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:50 smithi081 ceph-mon[31453]: pgmap v617: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 19 KiB/s wr, 22 op/s 2024-04-12T21:22:53.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:53.024+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:22:53.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:53.024+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:22:53.020 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:22:53.024+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:22:53.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:52 smithi153 ceph-mon[36569]: pgmap v618: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 35 KiB/s wr, 25 op/s 2024-04-12T21:22:53.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:52 smithi104 ceph-mon[36624]: pgmap v618: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 35 KiB/s wr, 25 op/s 2024-04-12T21:22:53.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:52 smithi081 ceph-mon[31453]: pgmap v618: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 35 KiB/s wr, 25 op/s 2024-04-12T21:22:54.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:54 smithi081 ceph-mon[31453]: pgmap v619: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 30 KiB/s wr, 22 op/s 2024-04-12T21:22:55.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:54 smithi153 ceph-mon[36569]: pgmap v619: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 30 KiB/s wr, 22 op/s 2024-04-12T21:22:55.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:54 smithi104 ceph-mon[36624]: pgmap v619: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 30 KiB/s wr, 22 op/s 2024-04-12T21:22:57.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:56 smithi153 ceph-mon[36569]: pgmap v620: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-12T21:22:57.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:56 smithi104 ceph-mon[36624]: pgmap v620: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-12T21:22:57.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:56 smithi081 ceph-mon[31453]: pgmap v620: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 30 KiB/s wr, 24 op/s 2024-04-12T21:22:57.855 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:57 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:22:57.855 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:57 smithi104 systemd[1]: Stopped Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:22:57.855 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:57 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 20.805s CPU time. 2024-04-12T21:22:57.855 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:57 smithi104 systemd[1]: Starting Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:22:57.856 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:57 smithi104 podman[69819]: 2024-04-12 21:22:57.723107389 +0000 UTC m=+0.018493189 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:22:58.109 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:57 smithi104 podman[69819]: 2024-04-12 21:22:57.8630891 +0000 UTC m=+0.158474887 container create e799184c8cefc7cde7a43ac69fbb1a17013ee3cfb7e0f7e86e2faedfed4127f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-12T21:22:58.184 INFO:tasks.workunit.client.0.smithi081.stdout: 106 0 0 0 0 0 0 2024-04-12T21:22:58.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: daemon mds.h restarted 2024-04-12T21:22:58.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:58.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:22:58.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: osdmap e104: 12 total, 12 up, 12 in 2024-04-12T21:22:58.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:boot 2024-04-12T21:22:58.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:22:58.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:22:58.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:22:58.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:22:58.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:58 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:22:58.577 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:58 smithi104 podman[69819]: 2024-04-12 21:22:58.114766742 +0000 UTC m=+0.410152544 container init e799184c8cefc7cde7a43ac69fbb1a17013ee3cfb7e0f7e86e2faedfed4127f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-12T21:22:58.577 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:58 smithi104 podman[69819]: 2024-04-12 21:22:58.117406925 +0000 UTC m=+0.412792728 container start e799184c8cefc7cde7a43ac69fbb1a17013ee3cfb7e0f7e86e2faedfed4127f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-12T21:22:58.577 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mds[69838]: -- 172.21.15.104:0/1175121187 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f486a3a40 con 0x563f48677000 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: starting mds.h at 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:58 smithi104 bash[69819]: e799184c8cefc7cde7a43ac69fbb1a17013ee3cfb7e0f7e86e2faedfed4127f7 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:22:58 smithi104 systemd[1]: Started Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: daemon mds.h restarted 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: osdmap e104: 12 total, 12 up, 12 in 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:boot 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:22:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:22:58.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:22:58.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:58 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:22:58.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:22:58.157+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: daemon mds.h restarted 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: osdmap e104: 12 total, 12 up, 12 in 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:boot 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:22:58.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:58 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:22:59.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:22:59 smithi153 ceph-mon[36569]: pgmap v622: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-12T21:22:59.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:22:59 smithi104 ceph-mon[36624]: pgmap v622: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-12T21:22:59.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:22:59 smithi081 ceph-mon[31453]: pgmap v622: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 17 KiB/s wr, 15 op/s 2024-04-12T21:23:00.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:00 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:reconnect 2024-04-12T21:23:00.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:00 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:23:00.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:00 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209996 2024-04-12T21:23:00.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:00 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:reconnect 2024-04-12T21:23:00.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:00 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:23:00.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:00 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209996 2024-04-12T21:23:00.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:00 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:reconnect 2024-04-12T21:23:00.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:00 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:23:00.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:00 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209996 2024-04-12T21:23:01.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:01 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:rejoin 2024-04-12T21:23:01.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:01 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:23:01.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:01 smithi153 ceph-mon[36569]: pgmap v623: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 16 KiB/s wr, 8 op/s 2024-04-12T21:23:01.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:01 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:rejoin 2024-04-12T21:23:01.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:01 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:23:01.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:01 smithi104 ceph-mon[36624]: pgmap v623: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 16 KiB/s wr, 8 op/s 2024-04-12T21:23:01.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:01 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:rejoin 2024-04-12T21:23:01.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:01 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:23:01.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:01 smithi081 ceph-mon[31453]: pgmap v623: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 16 KiB/s wr, 8 op/s 2024-04-12T21:23:02.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:clientreplay 2024-04-12T21:23:02.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:23:02.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-mon[36569]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:23:02.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:02 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:clientreplay 2024-04-12T21:23:02.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:02 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:23:02.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:02 smithi104 ceph-mon[36624]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:23:02.618 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:02 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:clientreplay 2024-04-12T21:23:02.619 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:02 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:23:02.619 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:02 smithi081 ceph-mon[31453]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:23:02.949 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: *** Caught signal (Aborted) ** 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: in thread 7f3bf8ee6700 thread_name:ms_dispatch 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3c06aa0d20] 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2: gsignal() 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 3: abort() 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560758050268] 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5607580ec0fc] 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560758420f0a] 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 7: (MDLog::trim()+0xca9) [0x5607584220c9] 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5607584225a8] 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3c0598cb23] 2024-04-12T21:23:02.950 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3c06a961ca] 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 11: clone() 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2024-04-12T21:23:02.578+0000 7f3bf8ee6700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: in thread 7f3bf8ee6700 thread_name:ms_dispatch 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3c06aa0d20] 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2: gsignal() 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 3: abort() 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560758050268] 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5607580ec0fc] 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560758420f0a] 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 7: (MDLog::trim()+0xca9) [0x5607584220c9] 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5607584225a8] 2024-04-12T21:23:02.951 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3c0598cb23] 2024-04-12T21:23:02.952 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3c06a961ca] 2024-04-12T21:23:02.952 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 11: clone() 2024-04-12T21:23:02.952 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:02.952 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2024-04-12T21:23:02.952 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 0> 2024-04-12T21:23:02.578+0000 7f3bf8ee6700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:02.952 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: in thread 7f3bf8ee6700 thread_name:ms_dispatch 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3c06aa0d20] 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2: gsignal() 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 3: abort() 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560758050268] 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5607580ec0fc] 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560758420f0a] 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 7: (MDLog::trim()+0xca9) [0x5607584220c9] 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5607584225a8] 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3c0598cb23] 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3c06a961ca] 2024-04-12T21:23:02.953 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 11: clone() 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: -1> 2024-04-12T21:23:02.578+0000 7f3bf8ee6700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: in thread 7f3bf8ee6700 thread_name:ms_dispatch 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3c06aa0d20] 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2: gsignal() 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 3: abort() 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560758050268] 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5607580ec0fc] 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560758420f0a] 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 7: (MDLog::trim()+0xca9) [0x5607584220c9] 2024-04-12T21:23:02.954 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5607584225a8] 2024-04-12T21:23:02.955 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3c0598cb23] 2024-04-12T21:23:02.955 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3c06a961ca] 2024-04-12T21:23:02.955 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 11: clone() 2024-04-12T21:23:02.955 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:02.955 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[67915]: 2024-04-12T21:23:02.955 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:02 smithi153 podman[71594]: 2024-04-12 21:23:02.640072604 +0000 UTC m=+0.018733759 container died 7c24e15e697f354995ba2d4198783d9ba82b7f821d527d2358f49969ba55bddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-12T21:23:03.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:03 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:03 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:23:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:03 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:active 2024-04-12T21:23:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:03 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:23:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:03 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:03 smithi153 ceph-mon[36569]: pgmap v624: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-12T21:23:03.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:03 smithi153 podman[71594]: 2024-04-12 21:23:03.255498554 +0000 UTC m=+0.634159715 container remove 7c24e15e697f354995ba2d4198783d9ba82b7f821d527d2358f49969ba55bddc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:23:03.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:03 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:23:03.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:03 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-12T21:23:03.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:03 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 21.854s CPU time. 2024-04-12T21:23:03.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:03 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:03.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:03 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:23:03.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:03 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:active 2024-04-12T21:23:03.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:03 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:23:03.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:03 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:03.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:03 smithi104 ceph-mon[36624]: pgmap v624: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-12T21:23:03.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:03 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:03.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:03 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:23:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:03 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/455852644,v1:172.21.15.153:6837/455852644] up:active 2024-04-12T21:23:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:03 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:23:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:03 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:03.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:03 smithi081 ceph-mon[31453]: pgmap v624: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.9 KiB/s wr, 7 op/s 2024-04-12T21:23:05.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:04 smithi153 ceph-mon[36569]: pgmap v625: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-04-12T21:23:05.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:04 smithi104 ceph-mon[36624]: pgmap v625: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-04-12T21:23:05.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:04 smithi081 ceph-mon[31453]: pgmap v625: 129 pgs: 129 active+clean; 601 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.9 KiB/s wr, 11 op/s 2024-04-12T21:23:06.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:06 smithi081 ceph-mon[31453]: pgmap v626: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 79 KiB/s wr, 32 op/s 2024-04-12T21:23:07.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:06 smithi153 ceph-mon[36569]: pgmap v626: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 79 KiB/s wr, 32 op/s 2024-04-12T21:23:07.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:06 smithi104 ceph-mon[36624]: pgmap v626: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 79 KiB/s wr, 32 op/s 2024-04-12T21:23:08.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:08.039+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:23:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:08.039+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:23:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:08.039+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:23:09.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:08 smithi153 ceph-mon[36569]: pgmap v627: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 88 KiB/s wr, 31 op/s 2024-04-12T21:23:09.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:08 smithi104 ceph-mon[36624]: pgmap v627: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 88 KiB/s wr, 31 op/s 2024-04-12T21:23:09.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:08 smithi081 ceph-mon[31453]: pgmap v627: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 88 KiB/s wr, 31 op/s 2024-04-12T21:23:11.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:10 smithi153 ceph-mon[36569]: pgmap v628: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 84 KiB/s wr, 29 op/s 2024-04-12T21:23:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:10 smithi104 ceph-mon[36624]: pgmap v628: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 84 KiB/s wr, 29 op/s 2024-04-12T21:23:11.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:10 smithi081 ceph-mon[31453]: pgmap v628: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 84 KiB/s wr, 29 op/s 2024-04-12T21:23:13.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:12 smithi153 ceph-mon[36569]: pgmap v629: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 110 KiB/s wr, 32 op/s 2024-04-12T21:23:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:12 smithi104 ceph-mon[36624]: pgmap v629: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 110 KiB/s wr, 32 op/s 2024-04-12T21:23:13.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:12 smithi081 ceph-mon[31453]: pgmap v629: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 110 KiB/s wr, 32 op/s 2024-04-12T21:23:13.809 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:13 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:23:13.809 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:13 smithi153 systemd[1]: Stopped Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:23:13.810 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:13 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 21.854s CPU time. 2024-04-12T21:23:13.810 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:13 smithi153 systemd[1]: Starting Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:23:13.810 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:13 smithi153 podman[71686]: 2024-04-12 21:23:13.689644738 +0000 UTC m=+0.018648465 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:23:13.810 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:13 smithi153 podman[71686]: 2024-04-12 21:23:13.81468175 +0000 UTC m=+0.143685517 container create 26add1764354ab996fe9c07766d288f6c306a2b7d457437700f75bc4f64f5ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-12T21:23:14.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:23:14.109+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: daemon mds.c restarted 2024-04-12T21:23:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: osdmap e105: 12 total, 12 up, 12 in 2024-04-12T21:23:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:boot 2024-04-12T21:23:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:23:14.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:14.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:14.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:23:14.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:14 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: daemon mds.c restarted 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: osdmap e105: 12 total, 12 up, 12 in 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:boot 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:23:14.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:23:14.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:14 smithi153 podman[71686]: 2024-04-12 21:23:14.067440151 +0000 UTC m=+0.396443869 container init 26add1764354ab996fe9c07766d288f6c306a2b7d457437700f75bc4f64f5ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-12T21:23:14.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:14 smithi153 podman[71686]: 2024-04-12 21:23:14.070535108 +0000 UTC m=+0.399538836 container start 26add1764354ab996fe9c07766d288f6c306a2b7d457437700f75bc4f64f5ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-12T21:23:14.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-mds[71706]: -- 172.21.15.153:0/4254404544 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564168bfba40 con 0x564168bce800 2024-04-12T21:23:14.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:14 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: starting mds.c at 2024-04-12T21:23:14.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:14 smithi153 bash[71686]: 26add1764354ab996fe9c07766d288f6c306a2b7d457437700f75bc4f64f5ff6 2024-04-12T21:23:14.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:14 smithi153 systemd[1]: Started Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:23:14.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: daemon mds.c restarted 2024-04-12T21:23:14.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:14.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:14.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: osdmap e105: 12 total, 12 up, 12 in 2024-04-12T21:23:14.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:boot 2024-04-12T21:23:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:23:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:23:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:14 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:23:15.479 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:15 smithi104 ceph-mon[36624]: pgmap v631: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 129 KiB/s wr, 31 op/s 2024-04-12T21:23:15.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:15 smithi153 ceph-mon[36569]: pgmap v631: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 129 KiB/s wr, 31 op/s 2024-04-12T21:23:15.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:15 smithi081 ceph-mon[31453]: pgmap v631: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 129 KiB/s wr, 31 op/s 2024-04-12T21:23:16.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:16 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:reconnect 2024-04-12T21:23:16.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:16 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:23:16.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:16 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0309996 2024-04-12T21:23:16.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:16 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:reconnect 2024-04-12T21:23:16.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:16 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:23:16.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:16 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0309996 2024-04-12T21:23:16.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:16 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:reconnect 2024-04-12T21:23:16.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:16 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:23:16.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:16 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0309996 2024-04-12T21:23:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:17 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:rejoin 2024-04-12T21:23:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:17 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:23:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:17 smithi153 ceph-mon[36569]: pgmap v632: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 53 KiB/s wr, 10 op/s 2024-04-12T21:23:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:17 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:rejoin 2024-04-12T21:23:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:17 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:23:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:17 smithi104 ceph-mon[36624]: pgmap v632: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 53 KiB/s wr, 10 op/s 2024-04-12T21:23:17.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:17 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:rejoin 2024-04-12T21:23:17.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:17 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:23:17.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:17 smithi081 ceph-mon[31453]: pgmap v632: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 53 KiB/s wr, 10 op/s 2024-04-12T21:23:18.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:18 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:clientreplay 2024-04-12T21:23:18.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:18 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:23:18.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:18 smithi153 ceph-mon[36569]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:23:18.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:clientreplay 2024-04-12T21:23:18.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:23:18.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-mon[36624]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:23:18.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:23:18.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: *** Caught signal (Aborted) ** 2024-04-12T21:23:18.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: in thread 7f3b18555700 thread_name:ms_dispatch 2024-04-12T21:23:18.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:18.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3b2610fd20] 2024-04-12T21:23:18.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2: gsignal() 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 3: abort() 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563f4717a268] 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563f472160fc] 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563f4754af0a] 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 7: (MDLog::trim()+0xca9) [0x563f4754c0c9] 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563f4754c5a8] 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3b24ffbb23] 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3b261051ca] 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 11: clone() 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2024-04-12T21:23:18.305+0000 7f3b18555700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: in thread 7f3b18555700 thread_name:ms_dispatch 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:18.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3b2610fd20] 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2: gsignal() 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 3: abort() 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563f4717a268] 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563f472160fc] 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563f4754af0a] 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 7: (MDLog::trim()+0xca9) [0x563f4754c0c9] 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563f4754c5a8] 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3b24ffbb23] 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3b261051ca] 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 11: clone() 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 0> 2024-04-12T21:23:18.305+0000 7f3b18555700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:18.580 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: in thread 7f3b18555700 thread_name:ms_dispatch 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3b2610fd20] 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2: gsignal() 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 3: abort() 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563f4717a268] 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563f472160fc] 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563f4754af0a] 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 7: (MDLog::trim()+0xca9) [0x563f4754c0c9] 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563f4754c5a8] 2024-04-12T21:23:18.581 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3b24ffbb23] 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3b261051ca] 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 11: clone() 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: -2> 2024-04-12T21:23:18.305+0000 7f3b18555700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: in thread 7f3b18555700 thread_name:ms_dispatch 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3b2610fd20] 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2: gsignal() 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 3: abort() 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563f4717a268] 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563f472160fc] 2024-04-12T21:23:18.582 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563f4754af0a] 2024-04-12T21:23:18.583 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 7: (MDLog::trim()+0xca9) [0x563f4754c0c9] 2024-04-12T21:23:18.583 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563f4754c5a8] 2024-04-12T21:23:18.583 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3b24ffbb23] 2024-04-12T21:23:18.583 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3b261051ca] 2024-04-12T21:23:18.583 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 11: clone() 2024-04-12T21:23:18.583 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:18.583 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[69834]: 2024-04-12T21:23:18.583 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:18 smithi104 podman[69874]: 2024-04-12 21:23:18.364496918 +0000 UTC m=+0.018304358 container died e799184c8cefc7cde7a43ac69fbb1a17013ee3cfb7e0f7e86e2faedfed4127f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-12T21:23:18.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:18 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:clientreplay 2024-04-12T21:23:18.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:18 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:23:18.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:18 smithi081 ceph-mon[31453]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:23:19.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:19 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:19.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:19 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:23:19.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:19 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:active 2024-04-12T21:23:19.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:19 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:23:19.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:19 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:19.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:19 smithi153 ceph-mon[36569]: pgmap v633: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 40 KiB/s wr, 9 op/s 2024-04-12T21:23:19.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:19 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:19.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:19 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:23:19.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:19 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:active 2024-04-12T21:23:19.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:19 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:23:19.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:19 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:19.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:19 smithi104 ceph-mon[36624]: pgmap v633: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 40 KiB/s wr, 9 op/s 2024-04-12T21:23:19.629 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:19 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:19.629 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:19 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:23:19.629 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:19 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1213954619,v1:172.21.15.104:6837/1213954619] up:active 2024-04-12T21:23:19.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:19 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:23:19.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:19 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:19.630 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:19 smithi081 ceph-mon[31453]: pgmap v633: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 40 KiB/s wr, 9 op/s 2024-04-12T21:23:20.577 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:20 smithi104 podman[69874]: 2024-04-12 21:23:20.118048097 +0000 UTC m=+1.771855524 container remove e799184c8cefc7cde7a43ac69fbb1a17013ee3cfb7e0f7e86e2faedfed4127f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-12T21:23:20.577 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:20 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:23:20.577 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:20 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-12T21:23:20.577 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:20 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 20.674s CPU time. 2024-04-12T21:23:21.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:20 smithi153 ceph-mon[36569]: pgmap v634: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 30 KiB/s wr, 13 op/s 2024-04-12T21:23:21.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:20 smithi104 ceph-mon[36624]: pgmap v634: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 30 KiB/s wr, 13 op/s 2024-04-12T21:23:21.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:20 smithi081 ceph-mon[31453]: pgmap v634: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 30 KiB/s wr, 13 op/s 2024-04-12T21:23:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:23.052+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:23:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:23.052+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:23:23.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:23.052+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:23:23.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:22 smithi153 ceph-mon[36569]: pgmap v635: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 37 KiB/s wr, 25 op/s 2024-04-12T21:23:23.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:22 smithi104 ceph-mon[36624]: pgmap v635: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 37 KiB/s wr, 25 op/s 2024-04-12T21:23:23.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:22 smithi081 ceph-mon[31453]: pgmap v635: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 37 KiB/s wr, 25 op/s 2024-04-12T21:23:24.955 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:24 smithi081 ceph-mon[31453]: pgmap v636: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 36 KiB/s wr, 24 op/s 2024-04-12T21:23:25.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:24 smithi153 ceph-mon[36569]: pgmap v636: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 36 KiB/s wr, 24 op/s 2024-04-12T21:23:25.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:24 smithi104 ceph-mon[36624]: pgmap v636: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 36 KiB/s wr, 24 op/s 2024-04-12T21:23:26.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:23:26.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:23:26.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:23:27.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:27 smithi081 ceph-mon[31453]: pgmap v637: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 33 KiB/s wr, 23 op/s 2024-04-12T21:23:27.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:27 smithi153 ceph-mon[36569]: pgmap v637: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 33 KiB/s wr, 23 op/s 2024-04-12T21:23:27.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:27 smithi104 ceph-mon[36624]: pgmap v637: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 33 KiB/s wr, 23 op/s 2024-04-12T21:23:28.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:23:28.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:23:28.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:23:28.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:23:28.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:23:28.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:23:28.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:23:28.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:23:28.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:23:29.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:29 smithi081 ceph-mon[31453]: pgmap v638: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 33 KiB/s wr, 21 op/s 2024-04-12T21:23:29.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:29 smithi153 ceph-mon[36569]: pgmap v638: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 33 KiB/s wr, 21 op/s 2024-04-12T21:23:29.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:29 smithi104 ceph-mon[36624]: pgmap v638: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 33 KiB/s wr, 21 op/s 2024-04-12T21:23:30.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:30 smithi104 ceph-mon[36624]: pgmap v639: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 33 KiB/s wr, 22 op/s 2024-04-12T21:23:30.837 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:30 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:23:30.837 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:30 smithi104 systemd[1]: Stopped Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:23:30.838 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:30 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 20.674s CPU time. 2024-04-12T21:23:30.838 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:30 smithi104 systemd[1]: Starting Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:23:30.838 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:30 smithi104 podman[70098]: 2024-04-12 21:23:30.722173937 +0000 UTC m=+0.018018364 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:23:31.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:30 smithi153 ceph-mon[36569]: pgmap v639: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 33 KiB/s wr, 22 op/s 2024-04-12T21:23:31.106 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:30 smithi104 podman[70098]: 2024-04-12 21:23:30.842885318 +0000 UTC m=+0.138729730 container create b6c6f8313add1192a1daccda23da93899826b83f8ecedd973dc2c9896c221a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5) 2024-04-12T21:23:31.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:30 smithi081 ceph-mon[31453]: pgmap v639: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 33 KiB/s wr, 22 op/s 2024-04-12T21:23:31.374 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:31 smithi104 podman[70098]: 2024-04-12 21:23:31.111930747 +0000 UTC m=+0.407775167 container init b6c6f8313add1192a1daccda23da93899826b83f8ecedd973dc2c9896c221a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-12T21:23:31.374 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:31 smithi104 podman[70098]: 2024-04-12 21:23:31.114750295 +0000 UTC m=+0.410594713 container start b6c6f8313add1192a1daccda23da93899826b83f8ecedd973dc2c9896c221a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:23:31.374 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mds[70117]: -- 172.21.15.104:0/1807383041 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d40a81a40 con 0x561d40a54800 2024-04-12T21:23:31.374 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: starting mds.h at 2024-04-12T21:23:31.374 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:31 smithi104 bash[70098]: b6c6f8313add1192a1daccda23da93899826b83f8ecedd973dc2c9896c221a3b 2024-04-12T21:23:31.374 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:31 smithi104 systemd[1]: Started Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:23:31.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:23:31.155+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: daemon mds.h restarted 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: osdmap e106: 12 total, 12 up, 12 in 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:boot 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:23:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:31 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:23:32.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: daemon mds.h restarted 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: osdmap e106: 12 total, 12 up, 12 in 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:boot 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:23:32.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:31 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:23:32.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: daemon mds.h restarted 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: osdmap e106: 12 total, 12 up, 12 in 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:boot 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:23:32.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:31 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:23:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:32 smithi153 ceph-mon[36569]: pgmap v641: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-12T21:23:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:32 smithi104 ceph-mon[36624]: pgmap v641: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-12T21:23:33.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:32 smithi081 ceph-mon[31453]: pgmap v641: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-12T21:23:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:33 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:reconnect 2024-04-12T21:23:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:33 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:23:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:33 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0219996 2024-04-12T21:23:34.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:33 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:reconnect 2024-04-12T21:23:34.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:33 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:23:34.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:33 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0219996 2024-04-12T21:23:34.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:33 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:reconnect 2024-04-12T21:23:34.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:33 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:23:34.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:33 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0219996 2024-04-12T21:23:35.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:34 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:rejoin 2024-04-12T21:23:35.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:34 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:23:35.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:34 smithi153 ceph-mon[36569]: pgmap v642: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-12T21:23:35.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:34 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:rejoin 2024-04-12T21:23:35.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:34 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:23:35.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:34 smithi104 ceph-mon[36624]: pgmap v642: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-12T21:23:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:34 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:rejoin 2024-04-12T21:23:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:34 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:23:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:34 smithi081 ceph-mon[31453]: pgmap v642: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.4 KiB/s wr, 4 op/s 2024-04-12T21:23:36.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:35 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:clientreplay 2024-04-12T21:23:36.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:35 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:23:36.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:35 smithi153 ceph-mon[36569]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:23:36.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:35 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:clientreplay 2024-04-12T21:23:36.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:35 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:23:36.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:35 smithi081 ceph-mon[31453]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:23:36.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:35 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:clientreplay 2024-04-12T21:23:36.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:35 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:23:36.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:35 smithi104 ceph-mon[36624]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:23:36.552 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: *** Caught signal (Aborted) ** 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: in thread 7f3df5b49700 thread_name:ms_dispatch 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3e03703d20] 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2: gsignal() 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 3: abort() 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564166eaa268] 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564166f460fc] 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56416727af0a] 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 7: (MDLog::trim()+0xca9) [0x56416727c0c9] 2024-04-12T21:23:36.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56416727c5a8] 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3e025efb23] 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3e036f91ca] 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 11: clone() 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2024-04-12T21:23:36.224+0000 7f3df5b49700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: in thread 7f3df5b49700 thread_name:ms_dispatch 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3e03703d20] 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2: gsignal() 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 3: abort() 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564166eaa268] 2024-04-12T21:23:36.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564166f460fc] 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56416727af0a] 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 7: (MDLog::trim()+0xca9) [0x56416727c0c9] 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56416727c5a8] 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3e025efb23] 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3e036f91ca] 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 11: clone() 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 0> 2024-04-12T21:23:36.224+0000 7f3df5b49700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: in thread 7f3df5b49700 thread_name:ms_dispatch 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3e03703d20] 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2: gsignal() 2024-04-12T21:23:36.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 3: abort() 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564166eaa268] 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564166f460fc] 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56416727af0a] 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 7: (MDLog::trim()+0xca9) [0x56416727c0c9] 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56416727c5a8] 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3e025efb23] 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3e036f91ca] 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 11: clone() 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: -4> 2024-04-12T21:23:36.224+0000 7f3df5b49700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: in thread 7f3df5b49700 thread_name:ms_dispatch 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2024-04-12T21:23:36.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3e03703d20] 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2: gsignal() 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 3: abort() 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564166eaa268] 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564166f460fc] 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56416727af0a] 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 7: (MDLog::trim()+0xca9) [0x56416727c0c9] 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56416727c5a8] 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3e025efb23] 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3e036f91ca] 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 11: clone() 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71702]: 2024-04-12T21:23:36.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:36 smithi153 podman[71871]: 2024-04-12 21:23:36.283106405 +0000 UTC m=+0.019959720 container died 26add1764354ab996fe9c07766d288f6c306a2b7d457437700f75bc4f64f5ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-12T21:23:37.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:37.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:23:37.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:active 2024-04-12T21:23:37.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:23:37.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:37.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:36 smithi153 ceph-mon[36569]: pgmap v643: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.8 KiB/s wr, 21 op/s 2024-04-12T21:23:37.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:36 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:37.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:36 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:23:37.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:36 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:active 2024-04-12T21:23:37.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:36 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:23:37.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:36 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:37.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:36 smithi104 ceph-mon[36624]: pgmap v643: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.8 KiB/s wr, 21 op/s 2024-04-12T21:23:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:36 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:36 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:23:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:36 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/4091976130,v1:172.21.15.153:6837/4091976130] up:active 2024-04-12T21:23:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:36 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:23:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:36 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:37.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:36 smithi081 ceph-mon[31453]: pgmap v643: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.8 KiB/s wr, 21 op/s 2024-04-12T21:23:37.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:37 smithi153 podman[71871]: 2024-04-12 21:23:37.047867936 +0000 UTC m=+0.784721250 container remove 26add1764354ab996fe9c07766d288f6c306a2b7d457437700f75bc4f64f5ff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:23:37.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:37 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:23:37.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:37 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-12T21:23:37.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:37 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 21.517s CPU time. 2024-04-12T21:23:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:38.066+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:23:38.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:38.066+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:23:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:38.066+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:23:39.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:38 smithi153 ceph-mon[36569]: pgmap v644: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.8 KiB/s wr, 24 op/s 2024-04-12T21:23:39.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:38 smithi104 ceph-mon[36624]: pgmap v644: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.8 KiB/s wr, 24 op/s 2024-04-12T21:23:39.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:38 smithi081 ceph-mon[31453]: pgmap v644: 129 pgs: 129 active+clean; 602 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.8 KiB/s wr, 24 op/s 2024-04-12T21:23:41.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:40 smithi153 ceph-mon[36569]: pgmap v645: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 52 KiB/s wr, 37 op/s 2024-04-12T21:23:41.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:40 smithi104 ceph-mon[36624]: pgmap v645: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 52 KiB/s wr, 37 op/s 2024-04-12T21:23:41.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:40 smithi081 ceph-mon[31453]: pgmap v645: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 52 KiB/s wr, 37 op/s 2024-04-12T21:23:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:42 smithi153 ceph-mon[36569]: pgmap v646: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 66 KiB/s wr, 36 op/s 2024-04-12T21:23:43.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:42 smithi104 ceph-mon[36624]: pgmap v646: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 66 KiB/s wr, 36 op/s 2024-04-12T21:23:43.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:42 smithi081 ceph-mon[31453]: pgmap v646: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 66 KiB/s wr, 36 op/s 2024-04-12T21:23:45.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:44 smithi153 ceph-mon[36569]: pgmap v647: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 74 KiB/s wr, 35 op/s 2024-04-12T21:23:45.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:44 smithi104 ceph-mon[36624]: pgmap v647: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 74 KiB/s wr, 35 op/s 2024-04-12T21:23:45.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:44 smithi081 ceph-mon[31453]: pgmap v647: 129 pgs: 129 active+clean; 603 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 74 KiB/s wr, 35 op/s 2024-04-12T21:23:47.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:46 smithi153 ceph-mon[36569]: pgmap v648: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 123 KiB/s wr, 41 op/s 2024-04-12T21:23:47.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:46 smithi104 ceph-mon[36624]: pgmap v648: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 123 KiB/s wr, 41 op/s 2024-04-12T21:23:47.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:46 smithi081 ceph-mon[31453]: pgmap v648: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 123 KiB/s wr, 41 op/s 2024-04-12T21:23:47.568 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:23:47.568 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 systemd[1]: Stopped Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:23:47.568 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 21.517s CPU time. 2024-04-12T21:23:47.568 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 systemd[1]: Starting Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:23:47.569 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 podman[71968]: 2024-04-12 21:23:47.445715213 +0000 UTC m=+0.018269168 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:23:47.569 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 podman[71968]: 2024-04-12 21:23:47.573721194 +0000 UTC m=+0.146275134 container create 9c8ee9f59cc2694cfc794f73c639ec233a619c437f6477383925a5eadfc5375e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:23:47.829 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 podman[71968]: 2024-04-12 21:23:47.809658292 +0000 UTC m=+0.382212231 container init 9c8ee9f59cc2694cfc794f73c639ec233a619c437f6477383925a5eadfc5375e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:23:47.829 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 podman[71968]: 2024-04-12 21:23:47.812258432 +0000 UTC m=+0.384812374 container start 9c8ee9f59cc2694cfc794f73c639ec233a619c437f6477383925a5eadfc5375e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-12T21:23:48.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:23:47.851+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:48.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 ceph-mds[71987]: -- 172.21.15.153:0/2814235561 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564e4a761a40 con 0x564e4a734c00 2024-04-12T21:23:48.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: starting mds.c at 2024-04-12T21:23:48.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 bash[71968]: 9c8ee9f59cc2694cfc794f73c639ec233a619c437f6477383925a5eadfc5375e 2024-04-12T21:23:48.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:23:47 smithi153 systemd[1]: Started Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: daemon mds.c restarted 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: osdmap e107: 12 total, 12 up, 12 in 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:boot 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:23:48.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:48 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: daemon mds.c restarted 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: osdmap e107: 12 total, 12 up, 12 in 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:boot 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:23:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:48 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:23:48.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: daemon mds.c restarted 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: osdmap e107: 12 total, 12 up, 12 in 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:boot 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:23:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:48 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:23:49.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:49 smithi081 ceph-mon[31453]: pgmap v650: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 144 KiB/s wr, 27 op/s 2024-04-12T21:23:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:49 smithi153 ceph-mon[36569]: pgmap v650: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 144 KiB/s wr, 27 op/s 2024-04-12T21:23:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:49 smithi104 ceph-mon[36624]: pgmap v650: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 144 KiB/s wr, 27 op/s 2024-04-12T21:23:50.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:50 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:reconnect 2024-04-12T21:23:50.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:50 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:23:50.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:50 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0269997 2024-04-12T21:23:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:50 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:reconnect 2024-04-12T21:23:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:50 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:23:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:50 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0269997 2024-04-12T21:23:50.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:50 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:reconnect 2024-04-12T21:23:50.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:50 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:23:50.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:50 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0269997 2024-04-12T21:23:51.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:51 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:rejoin 2024-04-12T21:23:51.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:51 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:23:51.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:51 smithi081 ceph-mon[31453]: pgmap v651: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 95 KiB/s wr, 14 op/s 2024-04-12T21:23:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:51 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:rejoin 2024-04-12T21:23:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:51 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:23:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:51 smithi153 ceph-mon[36569]: pgmap v651: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 95 KiB/s wr, 14 op/s 2024-04-12T21:23:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:51 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:rejoin 2024-04-12T21:23:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:51 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:23:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:51 smithi104 ceph-mon[36624]: pgmap v651: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 95 KiB/s wr, 14 op/s 2024-04-12T21:23:52.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:52 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:clientreplay 2024-04-12T21:23:52.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:52 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:23:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:52 smithi081 ceph-mon[31453]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:23:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:52 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:52 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:23:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:52 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:active 2024-04-12T21:23:52.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:52 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:52.756 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:clientreplay 2024-04-12T21:23:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:23:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-mon[36624]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:23:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:23:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:active 2024-04-12T21:23:52.757 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:52 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:clientreplay 2024-04-12T21:23:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:52 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:23:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:52 smithi153 ceph-mon[36569]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:23:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:52 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:23:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:52 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:23:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:52 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/1531124507,v1:172.21.15.104:6837/1531124507] up:active 2024-04-12T21:23:52.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:52 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:23:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:53.082+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:23:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:53.082+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:23:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:23:53.082+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:23:53.078 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:23:53.078 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: *** Caught signal (Aborted) ** 2024-04-12T21:23:53.078 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: in thread 7f844c189700 thread_name:ms_dispatch 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8459d43d20] 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2: gsignal() 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 3: abort() 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561d3f8de268] 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561d3f97a0fc] 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561d3fcaef0a] 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 7: (MDLog::trim()+0xca9) [0x561d3fcb00c9] 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561d3fcb05a8] 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f8458c2fb23] 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8459d391ca] 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 11: clone() 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2024-04-12T21:23:52.762+0000 7f844c189700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:53.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: in thread 7f844c189700 thread_name:ms_dispatch 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8459d43d20] 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2: gsignal() 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 3: abort() 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561d3f8de268] 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561d3f97a0fc] 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561d3fcaef0a] 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 7: (MDLog::trim()+0xca9) [0x561d3fcb00c9] 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561d3fcb05a8] 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f8458c2fb23] 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8459d391ca] 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 11: clone() 2024-04-12T21:23:53.080 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 0> 2024-04-12T21:23:52.762+0000 7f844c189700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: in thread 7f844c189700 thread_name:ms_dispatch 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8459d43d20] 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2: gsignal() 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 3: abort() 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561d3f8de268] 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561d3f97a0fc] 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561d3fcaef0a] 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 7: (MDLog::trim()+0xca9) [0x561d3fcb00c9] 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561d3fcb05a8] 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f8458c2fb23] 2024-04-12T21:23:53.081 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8459d391ca] 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 11: clone() 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: -10> 2024-04-12T21:23:52.762+0000 7f844c189700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: in thread 7f844c189700 thread_name:ms_dispatch 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8459d43d20] 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2: gsignal() 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 3: abort() 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561d3f8de268] 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561d3f97a0fc] 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561d3fcaef0a] 2024-04-12T21:23:53.082 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 7: (MDLog::trim()+0xca9) [0x561d3fcb00c9] 2024-04-12T21:23:53.083 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561d3fcb05a8] 2024-04-12T21:23:53.083 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f8458c2fb23] 2024-04-12T21:23:53.083 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8459d391ca] 2024-04-12T21:23:53.083 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 11: clone() 2024-04-12T21:23:53.083 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:23:53.083 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70113]: 2024-04-12T21:23:53.083 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:52 smithi104 podman[70165]: 2024-04-12 21:23:52.822976119 +0000 UTC m=+0.018723469 container died b6c6f8313add1192a1daccda23da93899826b83f8ecedd973dc2c9896c221a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-12T21:23:53.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:53 smithi081 ceph-mon[31453]: pgmap v652: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 72 KiB/s wr, 10 op/s 2024-04-12T21:23:53.730 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:53 smithi104 ceph-mon[36624]: pgmap v652: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 72 KiB/s wr, 10 op/s 2024-04-12T21:23:53.730 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:53 smithi104 podman[70165]: 2024-04-12 21:23:53.515347478 +0000 UTC m=+0.711094822 container remove b6c6f8313add1192a1daccda23da93899826b83f8ecedd973dc2c9896c221a3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:23:53.730 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:23:53.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:53 smithi153 ceph-mon[36569]: pgmap v652: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 72 KiB/s wr, 10 op/s 2024-04-12T21:23:54.077 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-12T21:23:54.077 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:23:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 21.709s CPU time. 2024-04-12T21:23:55.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:54 smithi153 ceph-mon[36569]: pgmap v653: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 59 KiB/s wr, 14 op/s 2024-04-12T21:23:55.053 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:54 smithi081 ceph-mon[31453]: pgmap v653: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 59 KiB/s wr, 14 op/s 2024-04-12T21:23:55.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:54 smithi104 ceph-mon[36624]: pgmap v653: 129 pgs: 129 active+clean; 604 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 59 KiB/s wr, 14 op/s 2024-04-12T21:23:57.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:56 smithi153 ceph-mon[36569]: pgmap v654: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 98 KiB/s wr, 36 op/s 2024-04-12T21:23:57.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:56 smithi104 ceph-mon[36624]: pgmap v654: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 98 KiB/s wr, 36 op/s 2024-04-12T21:23:57.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:56 smithi081 ceph-mon[31453]: pgmap v654: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 98 KiB/s wr, 36 op/s 2024-04-12T21:23:59.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:23:58 smithi153 ceph-mon[36569]: pgmap v655: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 103 KiB/s wr, 35 op/s 2024-04-12T21:23:59.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:23:58 smithi104 ceph-mon[36624]: pgmap v655: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 103 KiB/s wr, 35 op/s 2024-04-12T21:23:59.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:23:58 smithi081 ceph-mon[31453]: pgmap v655: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 103 KiB/s wr, 35 op/s 2024-04-12T21:24:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:00 smithi153 ceph-mon[36569]: pgmap v656: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 127 KiB/s wr, 35 op/s 2024-04-12T21:24:01.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:00 smithi104 ceph-mon[36624]: pgmap v656: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 127 KiB/s wr, 35 op/s 2024-04-12T21:24:01.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:00 smithi081 ceph-mon[31453]: pgmap v656: 129 pgs: 129 active+clean; 605 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 127 KiB/s wr, 35 op/s 2024-04-12T21:24:03.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:02 smithi153 ceph-mon[36569]: pgmap v657: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 38 op/s 2024-04-12T21:24:03.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:02 smithi104 ceph-mon[36624]: pgmap v657: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 38 op/s 2024-04-12T21:24:03.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:02 smithi081 ceph-mon[31453]: pgmap v657: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 178 KiB/s wr, 38 op/s 2024-04-12T21:24:04.197 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:03 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:24:04.197 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:03 smithi104 systemd[1]: Stopped Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:24:04.197 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:03 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 21.709s CPU time. 2024-04-12T21:24:04.197 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:03 smithi104 systemd[1]: Starting Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:24:04.197 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:04 smithi104 podman[70257]: 2024-04-12 21:24:03.968094029 +0000 UTC m=+0.018106950 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:24:04.197 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:04 smithi104 podman[70257]: 2024-04-12 21:24:04.081924755 +0000 UTC m=+0.131937687 container create a552c149ac838f1e83e93489037b1b79e305f6748b7f15ebbb6063eace707a27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:24:04.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:04 smithi104 podman[70257]: 2024-04-12 21:24:04.33682706 +0000 UTC m=+0.386839979 container init a552c149ac838f1e83e93489037b1b79e305f6748b7f15ebbb6063eace707a27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-12T21:24:04.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:04 smithi104 podman[70257]: 2024-04-12 21:24:04.339552893 +0000 UTC m=+0.389565813 container start a552c149ac838f1e83e93489037b1b79e305f6748b7f15ebbb6063eace707a27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:24:04.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mds[70276]: -- 172.21.15.104:0/2415840653 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55855a909a40 con 0x55855a8dd000 2024-04-12T21:24:04.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: starting mds.h at 2024-04-12T21:24:04.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:04 smithi104 bash[70257]: a552c149ac838f1e83e93489037b1b79e305f6748b7f15ebbb6063eace707a27 2024-04-12T21:24:04.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:04 smithi104 systemd[1]: Started Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:24:04.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: daemon mds.h restarted 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: osdmap e108: 12 total, 12 up, 12 in 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:boot 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:24:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:04 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:24:04.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:24:04.378+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:04.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: daemon mds.h restarted 2024-04-12T21:24:04.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: osdmap e108: 12 total, 12 up, 12 in 2024-04-12T21:24:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:boot 2024-04-12T21:24:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:24:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:24:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:04 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:24:04.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: daemon mds.h restarted 2024-04-12T21:24:04.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:04.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:04.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: osdmap e108: 12 total, 12 up, 12 in 2024-04-12T21:24:04.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:boot 2024-04-12T21:24:04.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:24:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:24:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:04 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:24:05.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:05 smithi153 ceph-mon[36569]: pgmap v659: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 214 KiB/s wr, 40 op/s 2024-04-12T21:24:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:05 smithi104 ceph-mon[36624]: pgmap v659: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 214 KiB/s wr, 40 op/s 2024-04-12T21:24:05.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:05 smithi081 ceph-mon[31453]: pgmap v659: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 214 KiB/s wr, 40 op/s 2024-04-12T21:24:06.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:06 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:reconnect 2024-04-12T21:24:06.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:06 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:24:06.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:06 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0419993 2024-04-12T21:24:06.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:06 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:reconnect 2024-04-12T21:24:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:06 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:24:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:06 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0419993 2024-04-12T21:24:06.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:06 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:reconnect 2024-04-12T21:24:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:06 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:24:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:06 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0419993 2024-04-12T21:24:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:rejoin 2024-04-12T21:24:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:07 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:24:07.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:07 smithi153 ceph-mon[36569]: pgmap v660: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 116 KiB/s wr, 13 op/s 2024-04-12T21:24:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:rejoin 2024-04-12T21:24:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:07 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:24:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:07 smithi104 ceph-mon[36624]: pgmap v660: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 116 KiB/s wr, 13 op/s 2024-04-12T21:24:07.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:rejoin 2024-04-12T21:24:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:07 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:24:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:07 smithi081 ceph-mon[31453]: pgmap v660: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 116 KiB/s wr, 13 op/s 2024-04-12T21:24:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:08.095+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:24:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:08.095+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:24:08.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:08.095+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:24:08.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:08 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:clientreplay 2024-04-12T21:24:08.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:08 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:24:08.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:08 smithi153 ceph-mon[36569]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:24:08.803 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:08 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:clientreplay 2024-04-12T21:24:08.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:08 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:24:08.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:08 smithi081 ceph-mon[31453]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:24:08.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:08 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:clientreplay 2024-04-12T21:24:08.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:08 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:24:08.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:08 smithi104 ceph-mon[36624]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:24:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:09 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:09 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:24:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:09 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:active 2024-04-12T21:24:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:09 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:24:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:09 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:09.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:09 smithi081 ceph-mon[31453]: pgmap v661: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 103 KiB/s wr, 13 op/s 2024-04-12T21:24:09.699 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:09.699 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:24:09.699 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:active 2024-04-12T21:24:09.699 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:24:09.699 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:09.699 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-mon[36569]: pgmap v661: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 103 KiB/s wr, 13 op/s 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: *** Caught signal (Aborted) ** 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: in thread 7f14a9778700 thread_name:ms_dispatch 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14b7332d20] 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2: gsignal() 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 3: abort() 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564e48f37268] 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564e48fd30fc] 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564e49307f0a] 2024-04-12T21:24:09.700 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 7: (MDLog::trim()+0xca9) [0x564e493090c9] 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564e493095a8] 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f14b621eb23] 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f14b73281ca] 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 11: clone() 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2024-04-12T21:24:09.356+0000 7f14a9778700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: in thread 7f14a9778700 thread_name:ms_dispatch 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14b7332d20] 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2: gsignal() 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 3: abort() 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564e48f37268] 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564e48fd30fc] 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564e49307f0a] 2024-04-12T21:24:09.701 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 7: (MDLog::trim()+0xca9) [0x564e493090c9] 2024-04-12T21:24:09.702 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564e493095a8] 2024-04-12T21:24:09.702 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f14b621eb23] 2024-04-12T21:24:09.702 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f14b73281ca] 2024-04-12T21:24:09.702 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 11: clone() 2024-04-12T21:24:09.702 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:09.702 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 0> 2024-04-12T21:24:09.356+0000 7f14a9778700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: in thread 7f14a9778700 thread_name:ms_dispatch 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14b7332d20] 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2: gsignal() 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 3: abort() 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564e48f37268] 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564e48fd30fc] 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564e49307f0a] 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 7: (MDLog::trim()+0xca9) [0x564e493090c9] 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564e493095a8] 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f14b621eb23] 2024-04-12T21:24:09.703 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f14b73281ca] 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 11: clone() 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 0> 2024-04-12T21:24:09.356+0000 7f14a9778700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: in thread 7f14a9778700 thread_name:ms_dispatch 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14b7332d20] 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2: gsignal() 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 3: abort() 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564e48f37268] 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564e48fd30fc] 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564e49307f0a] 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 7: (MDLog::trim()+0xca9) [0x564e493090c9] 2024-04-12T21:24:09.704 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564e493095a8] 2024-04-12T21:24:09.705 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f14b621eb23] 2024-04-12T21:24:09.705 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f14b73281ca] 2024-04-12T21:24:09.705 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 11: clone() 2024-04-12T21:24:09.705 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:09.705 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[71983]: 2024-04-12T21:24:09.705 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:09 smithi153 podman[72023]: 2024-04-12 21:24:09.420023102 +0000 UTC m=+0.018459271 container died 9c8ee9f59cc2694cfc794f73c639ec233a619c437f6477383925a5eadfc5375e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, ceph=True, CEPH_POINT_RELEASE=) 2024-04-12T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:09 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:09 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:09 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2004426592,v1:172.21.15.153:6837/2004426592] up:active 2024-04-12T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:09 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:09 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:09 smithi104 ceph-mon[36624]: pgmap v661: 129 pgs: 129 active+clean; 606 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 103 KiB/s wr, 13 op/s 2024-04-12T21:24:10.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:10 smithi153 podman[72023]: 2024-04-12 21:24:10.030902199 +0000 UTC m=+0.629338368 container remove 9c8ee9f59cc2694cfc794f73c639ec233a619c437f6477383925a5eadfc5375e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-12T21:24:10.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:10 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:24:10.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:10 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-12T21:24:10.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:10 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 22.723s CPU time. 2024-04-12T21:24:10.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:10 smithi081 ceph-mon[31453]: pgmap v662: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 262 KiB/s wr, 42 op/s 2024-04-12T21:24:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:10 smithi153 ceph-mon[36569]: pgmap v662: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 262 KiB/s wr, 42 op/s 2024-04-12T21:24:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:10 smithi104 ceph-mon[36624]: pgmap v662: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 262 KiB/s wr, 42 op/s 2024-04-12T21:24:13.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:12 smithi153 ceph-mon[36569]: pgmap v663: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 202 KiB/s wr, 61 op/s 2024-04-12T21:24:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:12 smithi104 ceph-mon[36624]: pgmap v663: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 202 KiB/s wr, 61 op/s 2024-04-12T21:24:13.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:12 smithi081 ceph-mon[31453]: pgmap v663: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 202 KiB/s wr, 61 op/s 2024-04-12T21:24:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:14 smithi153 ceph-mon[36569]: pgmap v664: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 223 KiB/s wr, 65 op/s 2024-04-12T21:24:15.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:14 smithi104 ceph-mon[36624]: pgmap v664: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 223 KiB/s wr, 65 op/s 2024-04-12T21:24:15.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:14 smithi081 ceph-mon[31453]: pgmap v664: 129 pgs: 129 active+clean; 607 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 223 KiB/s wr, 65 op/s 2024-04-12T21:24:17.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:16 smithi153 ceph-mon[36569]: pgmap v665: 129 pgs: 129 active+clean; 611 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 536 KiB/s wr, 74 op/s 2024-04-12T21:24:17.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:16 smithi104 ceph-mon[36624]: pgmap v665: 129 pgs: 129 active+clean; 611 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 536 KiB/s wr, 74 op/s 2024-04-12T21:24:17.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:16 smithi081 ceph-mon[31453]: pgmap v665: 129 pgs: 129 active+clean; 611 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 536 KiB/s wr, 74 op/s 2024-04-12T21:24:19.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:18 smithi153 ceph-mon[36569]: pgmap v666: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 569 KiB/s wr, 75 op/s 2024-04-12T21:24:19.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:18 smithi104 ceph-mon[36624]: pgmap v666: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 569 KiB/s wr, 75 op/s 2024-04-12T21:24:19.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:18 smithi081 ceph-mon[31453]: pgmap v666: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 569 KiB/s wr, 75 op/s 2024-04-12T21:24:20.613 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:24:20.613 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 systemd[1]: Stopped Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:24:20.613 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 22.723s CPU time. 2024-04-12T21:24:20.613 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 systemd[1]: Starting Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:24:20.613 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 podman[72124]: 2024-04-12 21:24:20.446043895 +0000 UTC m=+0.018077232 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:24:20.863 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:20 smithi153 ceph-mon[36569]: pgmap v667: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 569 KiB/s wr, 74 op/s 2024-04-12T21:24:20.863 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 podman[72124]: 2024-04-12 21:24:20.618055521 +0000 UTC m=+0.190088844 container create 6c42b687d23f3c32ac14b695d5eb220c01851b66aee05a8ebf1797f845b31624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-12T21:24:21.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:20 smithi104 ceph-mon[36624]: pgmap v667: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 569 KiB/s wr, 74 op/s 2024-04-12T21:24:21.140 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 podman[72124]: 2024-04-12 21:24:20.868423641 +0000 UTC m=+0.440456973 container init 6c42b687d23f3c32ac14b695d5eb220c01851b66aee05a8ebf1797f845b31624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-12T21:24:21.140 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 podman[72124]: 2024-04-12 21:24:20.871479016 +0000 UTC m=+0.443512359 container start 6c42b687d23f3c32ac14b695d5eb220c01851b66aee05a8ebf1797f845b31624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409) 2024-04-12T21:24:21.140 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 ceph-mds[72144]: -- 172.21.15.153:0/1661533585 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557f1d19fa40 con 0x557f1d172800 2024-04-12T21:24:21.140 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: starting mds.c at 2024-04-12T21:24:21.140 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 bash[72124]: 6c42b687d23f3c32ac14b695d5eb220c01851b66aee05a8ebf1797f845b31624 2024-04-12T21:24:21.140 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:20 smithi153 systemd[1]: Started Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:24:21.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:20 smithi081 ceph-mon[31453]: pgmap v667: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 569 KiB/s wr, 74 op/s 2024-04-12T21:24:21.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:20 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:24:20.911+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:21.997 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: daemon mds.c restarted 2024-04-12T21:24:21.997 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:21.997 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:21.997 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: osdmap e109: 12 total, 12 up, 12 in 2024-04-12T21:24:21.997 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:boot 2024-04-12T21:24:21.997 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:24:21.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:21.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:21.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:24:21.998 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:21 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:24:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: daemon mds.c restarted 2024-04-12T21:24:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: osdmap e109: 12 total, 12 up, 12 in 2024-04-12T21:24:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:boot 2024-04-12T21:24:22.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:24:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:24:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:21 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: daemon mds.c restarted 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: osdmap e109: 12 total, 12 up, 12 in 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:boot 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:22.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:24:22.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:21 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:24:23.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:23.110+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:24:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:23.111+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:24:23.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:23.111+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:24:23.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:22 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:reconnect 2024-04-12T21:24:23.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:22 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:24:23.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:22 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0589993 2024-04-12T21:24:23.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:22 smithi153 ceph-mon[36569]: pgmap v669: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 481 KiB/s wr, 32 op/s 2024-04-12T21:24:23.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:22 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:reconnect 2024-04-12T21:24:23.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:22 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:24:23.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:22 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0589993 2024-04-12T21:24:23.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:22 smithi104 ceph-mon[36624]: pgmap v669: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 481 KiB/s wr, 32 op/s 2024-04-12T21:24:23.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:22 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:reconnect 2024-04-12T21:24:23.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:22 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:24:23.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:22 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0589993 2024-04-12T21:24:23.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:22 smithi081 ceph-mon[31453]: pgmap v669: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 481 KiB/s wr, 32 op/s 2024-04-12T21:24:24.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:23 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:rejoin 2024-04-12T21:24:24.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:23 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:24:24.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:23 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:rejoin 2024-04-12T21:24:24.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:23 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:24:24.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:23 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:rejoin 2024-04-12T21:24:24.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:23 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:24:25.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:24 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:clientreplay 2024-04-12T21:24:25.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:24 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:24:25.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:24 smithi153 ceph-mon[36569]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:24:25.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:24 smithi153 ceph-mon[36569]: pgmap v670: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 454 KiB/s wr, 26 op/s 2024-04-12T21:24:25.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:24 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:clientreplay 2024-04-12T21:24:25.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:24 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:24:25.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:24 smithi081 ceph-mon[31453]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:24:25.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:24 smithi081 ceph-mon[31453]: pgmap v670: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 454 KiB/s wr, 26 op/s 2024-04-12T21:24:25.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:24 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:clientreplay 2024-04-12T21:24:25.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:24 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:24:25.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:24 smithi104 ceph-mon[36624]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:24:25.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:24 smithi104 ceph-mon[36624]: pgmap v670: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 454 KiB/s wr, 26 op/s 2024-04-12T21:24:25.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:24:25.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: *** Caught signal (Aborted) ** 2024-04-12T21:24:25.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: in thread 7fc1dca82700 thread_name:ms_dispatch 2024-04-12T21:24:25.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc1ea63cd20] 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2: gsignal() 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 3: abort() 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5585581ba268] 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5585582560fc] 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55855858af0a] 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 7: (MDLog::trim()+0xca9) [0x55855858c0c9] 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55855858c5a8] 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc1e9528b23] 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc1ea6321ca] 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 11: clone() 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2024-04-12T21:24:25.480+0000 7fc1dca82700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: in thread 7fc1dca82700 thread_name:ms_dispatch 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2024-04-12T21:24:25.829 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc1ea63cd20] 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2: gsignal() 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 3: abort() 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5585581ba268] 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5585582560fc] 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55855858af0a] 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 7: (MDLog::trim()+0xca9) [0x55855858c0c9] 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55855858c5a8] 2024-04-12T21:24:25.830 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc1e9528b23] 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc1ea6321ca] 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 11: clone() 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 0> 2024-04-12T21:24:25.480+0000 7fc1dca82700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: in thread 7fc1dca82700 thread_name:ms_dispatch 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc1ea63cd20] 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2: gsignal() 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 3: abort() 2024-04-12T21:24:25.831 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5585581ba268] 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5585582560fc] 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55855858af0a] 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 7: (MDLog::trim()+0xca9) [0x55855858c0c9] 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55855858c5a8] 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc1e9528b23] 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc1ea6321ca] 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 11: clone() 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: -9> 2024-04-12T21:24:25.480+0000 7fc1dca82700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: in thread 7fc1dca82700 thread_name:ms_dispatch 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:25.832 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc1ea63cd20] 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2: gsignal() 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 3: abort() 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5585581ba268] 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5585582560fc] 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55855858af0a] 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 7: (MDLog::trim()+0xca9) [0x55855858c0c9] 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55855858c5a8] 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc1e9528b23] 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc1ea6321ca] 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 11: clone() 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70272]: 2024-04-12T21:24:25.833 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:25 smithi104 podman[70312]: 2024-04-12 21:24:25.539624638 +0000 UTC m=+0.017897214 container died a552c149ac838f1e83e93489037b1b79e305f6748b7f15ebbb6063eace707a27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-12T21:24:26.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:25 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:26.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:25 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:24:26.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:active 2024-04-12T21:24:26.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:25 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:24:26.125 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:25 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:26.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:26.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:24:26.184 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:active 2024-04-12T21:24:26.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:24:26.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:25 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:26.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:25 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:26.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:25 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:24:26.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/3567585477,v1:172.21.15.104:6837/3567585477] up:active 2024-04-12T21:24:26.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:25 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:24:26.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:25 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:26.441 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:26 smithi104 podman[70312]: 2024-04-12 21:24:26.189669284 +0000 UTC m=+0.667941858 container remove a552c149ac838f1e83e93489037b1b79e305f6748b7f15ebbb6063eace707a27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T21:24:26.441 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:26 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:24:26.827 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:26 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-12T21:24:26.828 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:26 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 21.700s CPU time. 2024-04-12T21:24:27.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:26 smithi153 ceph-mon[36569]: pgmap v671: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 149 KiB/s wr, 55 op/s 2024-04-12T21:24:27.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:26 smithi104 ceph-mon[36624]: pgmap v671: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 149 KiB/s wr, 55 op/s 2024-04-12T21:24:27.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:26 smithi081 ceph-mon[31453]: pgmap v671: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 149 KiB/s wr, 55 op/s 2024-04-12T21:24:27.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:27.776+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4005b20 con 0x7f1704076980 2024-04-12T21:24:27.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:27.776+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4005b20 con 0x7f1704076980 2024-04-12T21:24:27.813 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:27.818+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d0005a20 con 0x563c9e245d20 2024-04-12T21:24:27.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:27.818+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d0005a20 con 0x563c9e245d20 2024-04-12T21:24:28.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:24:28.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:24:28.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:24:29.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:28 smithi153 ceph-mon[36569]: pgmap v672: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 118 KiB/s wr, 55 op/s 2024-04-12T21:24:29.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:24:29.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:24:29.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:24:29.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:28 smithi104 ceph-mon[36624]: pgmap v672: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 118 KiB/s wr, 55 op/s 2024-04-12T21:24:29.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:24:29.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:24:29.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:24:29.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:28 smithi081 ceph-mon[31453]: pgmap v672: 129 pgs: 129 active+clean; 612 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 118 KiB/s wr, 55 op/s 2024-04-12T21:24:29.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:24:29.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:24:29.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:24:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:30 smithi153 ceph-mon[36569]: pgmap v673: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 119 KiB/s wr, 59 op/s 2024-04-12T21:24:31.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:30 smithi104 ceph-mon[36624]: pgmap v673: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 119 KiB/s wr, 59 op/s 2024-04-12T21:24:31.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:30 smithi081 ceph-mon[31453]: pgmap v673: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 119 KiB/s wr, 59 op/s 2024-04-12T21:24:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:32 smithi153 ceph-mon[36569]: pgmap v674: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 102 KiB/s wr, 52 op/s 2024-04-12T21:24:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:32 smithi104 ceph-mon[36624]: pgmap v674: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 102 KiB/s wr, 52 op/s 2024-04-12T21:24:33.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:32 smithi081 ceph-mon[31453]: pgmap v674: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 102 KiB/s wr, 52 op/s 2024-04-12T21:24:35.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:34 smithi153 ceph-mon[36569]: pgmap v675: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 99 KiB/s wr, 51 op/s 2024-04-12T21:24:35.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:34 smithi104 ceph-mon[36624]: pgmap v675: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 99 KiB/s wr, 51 op/s 2024-04-12T21:24:35.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:34 smithi081 ceph-mon[31453]: pgmap v675: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 99 KiB/s wr, 51 op/s 2024-04-12T21:24:36.851 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:36 smithi104 ceph-mon[36624]: pgmap v676: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 111 KiB/s wr, 53 op/s 2024-04-12T21:24:36.852 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:36 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:24:36.852 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:36 smithi104 systemd[1]: Stopped Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:24:36.852 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:36 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 21.700s CPU time. 2024-04-12T21:24:36.852 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:36 smithi104 systemd[1]: Starting Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:24:36.852 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:36 smithi104 podman[70543]: 2024-04-12 21:24:36.722942917 +0000 UTC m=+0.018312838 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:24:36.852 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:36 smithi104 podman[70543]: 2024-04-12 21:24:36.856498558 +0000 UTC m=+0.151868469 container create 25735d17ca4c80eda856cb8d06ca77ad92050c1462489722a9e1986a98a93416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-04-12T21:24:37.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:36 smithi153 ceph-mon[36569]: pgmap v676: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 111 KiB/s wr, 53 op/s 2024-04-12T21:24:37.122 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:37 smithi104 podman[70543]: 2024-04-12 21:24:37.101044756 +0000 UTC m=+0.396414666 container init 25735d17ca4c80eda856cb8d06ca77ad92050c1462489722a9e1986a98a93416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , ceph=True) 2024-04-12T21:24:37.122 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:37 smithi104 podman[70543]: 2024-04-12 21:24:37.103750972 +0000 UTC m=+0.399120880 container start 25735d17ca4c80eda856cb8d06ca77ad92050c1462489722a9e1986a98a93416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-12T21:24:37.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:36 smithi081 ceph-mon[31453]: pgmap v676: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 111 KiB/s wr, 53 op/s 2024-04-12T21:24:37.384 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mds[70562]: -- 172.21.15.104:0/3297704066 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bcd13f3a40 con 0x55bcd13c6800 2024-04-12T21:24:37.384 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: starting mds.h at 2024-04-12T21:24:37.384 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:37 smithi104 bash[70543]: 25735d17ca4c80eda856cb8d06ca77ad92050c1462489722a9e1986a98a93416 2024-04-12T21:24:37.384 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:24:37 smithi104 systemd[1]: Started Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:24:37.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:24:37.144+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:38.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: daemon mds.h restarted 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: osdmap e110: 12 total, 12 up, 12 in 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:boot 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:24:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:37 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:24:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: daemon mds.h restarted 2024-04-12T21:24:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: osdmap e110: 12 total, 12 up, 12 in 2024-04-12T21:24:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:boot 2024-04-12T21:24:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:24:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:24:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:37 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:24:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:38.116+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:24:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:38.116+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:24:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:38.116+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:24:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: daemon mds.h restarted 2024-04-12T21:24:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: osdmap e110: 12 total, 12 up, 12 in 2024-04-12T21:24:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:boot 2024-04-12T21:24:38.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:24:38.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:38.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:38.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:24:38.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:37 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:24:39.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:38 smithi153 ceph-mon[36569]: pgmap v678: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 15 KiB/s wr, 10 op/s 2024-04-12T21:24:39.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:38 smithi081 ceph-mon[31453]: pgmap v678: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 15 KiB/s wr, 10 op/s 2024-04-12T21:24:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:38 smithi104 ceph-mon[36624]: pgmap v678: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 15 KiB/s wr, 10 op/s 2024-04-12T21:24:40.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:39 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:reconnect 2024-04-12T21:24:40.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:39 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:24:40.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:39 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:24:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:39 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:reconnect 2024-04-12T21:24:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:39 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:24:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:39 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:24:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:39 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:reconnect 2024-04-12T21:24:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:39 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:24:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:39 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:24:41.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:40 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:rejoin 2024-04-12T21:24:41.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:40 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:24:41.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:40 smithi153 ceph-mon[36569]: pgmap v679: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 14 KiB/s wr, 5 op/s 2024-04-12T21:24:41.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:40 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:rejoin 2024-04-12T21:24:41.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:40 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:24:41.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:40 smithi104 ceph-mon[36624]: pgmap v679: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 14 KiB/s wr, 5 op/s 2024-04-12T21:24:41.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:40 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:rejoin 2024-04-12T21:24:41.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:40 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:24:41.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:40 smithi081 ceph-mon[31453]: pgmap v679: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 14 KiB/s wr, 5 op/s 2024-04-12T21:24:41.941 INFO:tasks.workunit.client.0.smithi081.stdout: 2024-04-12T21:24:41.941 INFO:tasks.workunit.client.0.smithi081.stdout:Final score for writes: 106 2024-04-12T21:24:41.941 INFO:tasks.workunit.client.0.smithi081.stdout:Final score for reads : 179630 2024-04-12T21:24:41.941 INFO:tasks.workunit.client.0.smithi081.stdout: 2024-04-12T21:24:41.943 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-12T21:24:41.944 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-12T21:24:42.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:41 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:clientreplay 2024-04-12T21:24:42.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:41 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:24:42.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:41 smithi153 ceph-mon[36569]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:24:42.073 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:41 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:clientreplay 2024-04-12T21:24:42.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:41 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:24:42.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:41 smithi081 ceph-mon[31453]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:24:42.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:41 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:clientreplay 2024-04-12T21:24:42.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:41 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:24:42.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:41 smithi104 ceph-mon[36624]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:24:42.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:24:42.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: *** Caught signal (Aborted) ** 2024-04-12T21:24:42.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: in thread 7fda32a80700 thread_name:ms_dispatch 2024-04-12T21:24:42.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:42.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fda4063ad20] 2024-04-12T21:24:42.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2: gsignal() 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 3: abort() 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x557f1c0b0268] 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x557f1c14c0fc] 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x557f1c480f0a] 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 7: (MDLog::trim()+0xca9) [0x557f1c4820c9] 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 8: (MDLog::log_trim_upkeep()+0x358) [0x557f1c4825a8] 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fda3f526b23] 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fda406301ca] 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 11: clone() 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2024-04-12T21:24:42.124+0000 7fda32a80700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: in thread 7fda32a80700 thread_name:ms_dispatch 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:42.554 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fda4063ad20] 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2: gsignal() 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 3: abort() 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x557f1c0b0268] 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x557f1c14c0fc] 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x557f1c480f0a] 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 7: (MDLog::trim()+0xca9) [0x557f1c4820c9] 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 8: (MDLog::log_trim_upkeep()+0x358) [0x557f1c4825a8] 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fda3f526b23] 2024-04-12T21:24:42.555 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fda406301ca] 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 11: clone() 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 0> 2024-04-12T21:24:42.124+0000 7fda32a80700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: in thread 7fda32a80700 thread_name:ms_dispatch 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fda4063ad20] 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2: gsignal() 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 3: abort() 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x557f1c0b0268] 2024-04-12T21:24:42.556 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x557f1c14c0fc] 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x557f1c480f0a] 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 7: (MDLog::trim()+0xca9) [0x557f1c4820c9] 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 8: (MDLog::log_trim_upkeep()+0x358) [0x557f1c4825a8] 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fda3f526b23] 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fda406301ca] 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 11: clone() 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 0> 2024-04-12T21:24:42.124+0000 7fda32a80700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: in thread 7fda32a80700 thread_name:ms_dispatch 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fda4063ad20] 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2: gsignal() 2024-04-12T21:24:42.557 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 3: abort() 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x557f1c0b0268] 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x557f1c14c0fc] 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x557f1c480f0a] 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 7: (MDLog::trim()+0xca9) [0x557f1c4820c9] 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 8: (MDLog::log_trim_upkeep()+0x358) [0x557f1c4825a8] 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fda3f526b23] 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fda406301ca] 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 11: clone() 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72140]: 2024-04-12T21:24:42.558 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 podman[72323]: 2024-04-12 21:24:42.18404612 +0000 UTC m=+0.019384100 container died 6c42b687d23f3c32ac14b695d5eb220c01851b66aee05a8ebf1797f845b31624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-12T21:24:42.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:42.926 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:24:42.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:active 2024-04-12T21:24:42.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:24:42.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:42.927 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:42 smithi153 ceph-mon[36569]: pgmap v680: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 83 KiB/s wr, 21 op/s 2024-04-12T21:24:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:42 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:42 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:24:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:42 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:active 2024-04-12T21:24:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:42 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:24:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:42 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:42 smithi104 ceph-mon[36624]: pgmap v680: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 83 KiB/s wr, 21 op/s 2024-04-12T21:24:43.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:42 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:24:43.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:42 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:24:43.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:42 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/1050648490,v1:172.21.15.153:6837/1050648490] up:active 2024-04-12T21:24:43.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:42 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 9 up:standby 2024-04-12T21:24:43.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:42 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:24:43.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:42 smithi081 ceph-mon[31453]: pgmap v680: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 83 KiB/s wr, 21 op/s 2024-04-12T21:24:43.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 podman[72323]: 2024-04-12 21:24:42.931368374 +0000 UTC m=+0.766706352 container remove 6c42b687d23f3c32ac14b695d5eb220c01851b66aee05a8ebf1797f845b31624 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T21:24:43.302 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:42 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:24:43.303 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:43 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-12T21:24:43.303 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:43 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 21.971s CPU time. 2024-04-12T21:24:45.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:44 smithi153 ceph-mon[36569]: pgmap v681: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 96 KiB/s wr, 27 op/s 2024-04-12T21:24:45.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:44 smithi104 ceph-mon[36624]: pgmap v681: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 96 KiB/s wr, 27 op/s 2024-04-12T21:24:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:44 smithi081 ceph-mon[31453]: pgmap v681: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 96 KiB/s wr, 27 op/s 2024-04-12T21:24:47.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:46 smithi153 ceph-mon[36569]: pgmap v682: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 101 KiB/s wr, 40 op/s 2024-04-12T21:24:47.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:46 smithi104 ceph-mon[36624]: pgmap v682: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 101 KiB/s wr, 40 op/s 2024-04-12T21:24:47.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:46 smithi081 ceph-mon[31453]: pgmap v682: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 101 KiB/s wr, 40 op/s 2024-04-12T21:24:49.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:48 smithi153 ceph-mon[36569]: pgmap v683: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 100 KiB/s wr, 36 op/s 2024-04-12T21:24:49.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:48 smithi104 ceph-mon[36624]: pgmap v683: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 100 KiB/s wr, 36 op/s 2024-04-12T21:24:49.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:48 smithi081 ceph-mon[31453]: pgmap v683: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 100 KiB/s wr, 36 op/s 2024-04-12T21:24:51.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:50 smithi153 ceph-mon[36569]: pgmap v684: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 120 KiB/s wr, 36 op/s 2024-04-12T21:24:51.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:50 smithi104 ceph-mon[36624]: pgmap v684: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 120 KiB/s wr, 36 op/s 2024-04-12T21:24:51.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:50 smithi081 ceph-mon[31453]: pgmap v684: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 120 KiB/s wr, 36 op/s 2024-04-12T21:24:53.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:52 smithi153 ceph-mon[36569]: pgmap v685: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 154 KiB/s wr, 37 op/s 2024-04-12T21:24:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:52 smithi104 ceph-mon[36624]: pgmap v685: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 154 KiB/s wr, 37 op/s 2024-04-12T21:24:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:53.130+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:24:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:53.130+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:24:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:24:53.130+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:24:53.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:52 smithi081 ceph-mon[31453]: pgmap v685: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 154 KiB/s wr, 37 op/s 2024-04-12T21:24:53.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:53 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:24:53.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:53 smithi153 systemd[1]: Stopped Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:24:53.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:53 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 21.971s CPU time. 2024-04-12T21:24:53.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:53 smithi153 systemd[1]: Starting Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:24:53.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:53 smithi153 podman[72416]: 2024-04-12 21:24:53.43961599 +0000 UTC m=+0.018475664 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:24:54.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:54 smithi153 ceph-mon[36569]: pgmap v686: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 97 KiB/s wr, 23 op/s 2024-04-12T21:24:54.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:54 smithi153 podman[72416]: 2024-04-12 21:24:54.540956462 +0000 UTC m=+1.119816111 container create e39df45e30bec010300fccf0ca4036a1a9136ffe5fb0eb27da5ba5bcc41fa965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-12T21:24:55.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:54 smithi104 ceph-mon[36624]: pgmap v686: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 97 KiB/s wr, 23 op/s 2024-04-12T21:24:55.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:54 smithi081 ceph-mon[31453]: pgmap v686: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 97 KiB/s wr, 23 op/s 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: daemon mds.c restarted 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: osdmap e111: 12 total, 12 up, 12 in 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:boot 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:24:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:24:56.804 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:56 smithi153 podman[72416]: 2024-04-12 21:24:56.53780808 +0000 UTC m=+3.116667738 container init e39df45e30bec010300fccf0ca4036a1a9136ffe5fb0eb27da5ba5bcc41fa965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5) 2024-04-12T21:24:56.804 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:56 smithi153 podman[72416]: 2024-04-12 21:24:56.541329194 +0000 UTC m=+3.120188854 container start e39df45e30bec010300fccf0ca4036a1a9136ffe5fb0eb27da5ba5bcc41fa965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-12T21:24:56.804 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-mds[72435]: -- 172.21.15.153:0/2540700438 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559915e1fa40 con 0x559915df3000 2024-04-12T21:24:56.804 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: starting mds.c at 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:24:56.574+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: daemon mds.c restarted 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: osdmap e111: 12 total, 12 up, 12 in 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:boot 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:24:56.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:56 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:24:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: daemon mds.c restarted 2024-04-12T21:24:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:24:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:24:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: osdmap e111: 12 total, 12 up, 12 in 2024-04-12T21:24:57.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:boot 2024-04-12T21:24:57.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:24:57.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:24:57.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:24:57.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:24:57.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:56 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:24:57.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:57 smithi153 ceph-mon[36569]: pgmap v688: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 84 KiB/s wr, 6 op/s 2024-04-12T21:24:57.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:57 smithi153 bash[72416]: e39df45e30bec010300fccf0ca4036a1a9136ffe5fb0eb27da5ba5bcc41fa965 2024-04-12T21:24:57.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:24:57 smithi153 systemd[1]: Started Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:24:57.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:57 smithi081 ceph-mon[31453]: pgmap v688: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 84 KiB/s wr, 6 op/s 2024-04-12T21:24:58.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:57 smithi104 ceph-mon[36624]: pgmap v688: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 84 KiB/s wr, 6 op/s 2024-04-12T21:24:59.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:58 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:reconnect 2024-04-12T21:24:59.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:58 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:24:59.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:58 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199998 2024-04-12T21:24:59.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:58 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:reconnect 2024-04-12T21:24:59.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:58 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:24:59.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:58 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199998 2024-04-12T21:24:59.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:58 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:reconnect 2024-04-12T21:24:59.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:58 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:24:59.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:58 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199998 2024-04-12T21:25:00.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:59 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:rejoin 2024-04-12T21:25:00.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:59 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:25:00.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:24:59 smithi153 ceph-mon[36569]: pgmap v689: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 5 op/s 2024-04-12T21:25:00.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:59 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:rejoin 2024-04-12T21:25:00.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:59 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:25:00.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:24:59 smithi104 ceph-mon[36624]: pgmap v689: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 5 op/s 2024-04-12T21:25:00.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:59 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:rejoin 2024-04-12T21:25:00.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:59 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:25:00.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:24:59 smithi081 ceph-mon[31453]: pgmap v689: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 5 op/s 2024-04-12T21:25:00.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:clientreplay 2024-04-12T21:25:00.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:25:00.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-mon[36624]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:25:00.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-mon[36624]: pgmap v690: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:25:01.001 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:00 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:clientreplay 2024-04-12T21:25:01.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:00 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:25:01.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:00 smithi081 ceph-mon[31453]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:25:01.002 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:00 smithi081 ceph-mon[31453]: pgmap v690: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:25:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:00 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:clientreplay 2024-04-12T21:25:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:00 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:25:01.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:00 smithi153 ceph-mon[36569]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:25:01.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:00 smithi153 ceph-mon[36569]: pgmap v690: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:25:01.328 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: *** Caught signal (Aborted) ** 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: in thread 7fbffc24d700 thread_name:ms_dispatch 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc009e07d20] 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2: gsignal() 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 3: abort() 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55bccfe9d268] 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55bccff390fc] 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55bcd026df0a] 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 7: (MDLog::trim()+0xca9) [0x55bcd026f0c9] 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55bcd026f5a8] 2024-04-12T21:25:01.329 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc008cf3b23] 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc009dfd1ca] 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 11: clone() 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2024-04-12T21:25:00.963+0000 7fbffc24d700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: in thread 7fbffc24d700 thread_name:ms_dispatch 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc009e07d20] 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2: gsignal() 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 3: abort() 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55bccfe9d268] 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55bccff390fc] 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55bcd026df0a] 2024-04-12T21:25:01.330 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 7: (MDLog::trim()+0xca9) [0x55bcd026f0c9] 2024-04-12T21:25:01.331 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55bcd026f5a8] 2024-04-12T21:25:01.331 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc008cf3b23] 2024-04-12T21:25:01.331 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc009dfd1ca] 2024-04-12T21:25:01.331 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 11: clone() 2024-04-12T21:25:01.331 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:01.331 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 0> 2024-04-12T21:25:00.963+0000 7fbffc24d700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: in thread 7fbffc24d700 thread_name:ms_dispatch 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc009e07d20] 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2: gsignal() 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 3: abort() 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55bccfe9d268] 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55bccff390fc] 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55bcd026df0a] 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 7: (MDLog::trim()+0xca9) [0x55bcd026f0c9] 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55bcd026f5a8] 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc008cf3b23] 2024-04-12T21:25:01.332 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc009dfd1ca] 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 11: clone() 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: -10> 2024-04-12T21:25:00.963+0000 7fbffc24d700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: in thread 7fbffc24d700 thread_name:ms_dispatch 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc009e07d20] 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2: gsignal() 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 3: abort() 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55bccfe9d268] 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55bccff390fc] 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55bcd026df0a] 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 7: (MDLog::trim()+0xca9) [0x55bcd026f0c9] 2024-04-12T21:25:01.333 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55bcd026f5a8] 2024-04-12T21:25:01.334 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc008cf3b23] 2024-04-12T21:25:01.334 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc009dfd1ca] 2024-04-12T21:25:01.334 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 11: clone() 2024-04-12T21:25:01.334 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:01.334 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:00 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70558]: 2024-04-12T21:25:01.334 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:01 smithi104 podman[70610]: 2024-04-12 21:25:01.022434144 +0000 UTC m=+0.018525802 container died 25735d17ca4c80eda856cb8d06ca77ad92050c1462489722a9e1986a98a93416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:25:02.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:01 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:02.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:01 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:25:02.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:01 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:active 2024-04-12T21:25:02.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:01 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:25:02.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:01 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:02.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:01 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:02.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:01 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:25:02.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:01 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:active 2024-04-12T21:25:02.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:01 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:25:02.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:01 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:02.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:01 smithi104 podman[70610]: 2024-04-12 21:25:01.620242939 +0000 UTC m=+0.616334581 container remove 25735d17ca4c80eda856cb8d06ca77ad92050c1462489722a9e1986a98a93416 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:25:02.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:01 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:25:02.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:01 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-12T21:25:02.079 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:01 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 21.032s CPU time. 2024-04-12T21:25:02.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:01 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:02.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:01 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:25:02.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:01 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/4208469425,v1:172.21.15.104:6837/4208469425] up:active 2024-04-12T21:25:02.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:01 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:25:02.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:01 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:03.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:02 smithi153 ceph-mon[36569]: pgmap v691: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-12T21:25:03.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:02 smithi104 ceph-mon[36624]: pgmap v691: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-12T21:25:03.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:02 smithi081 ceph-mon[31453]: pgmap v691: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 32 KiB/s wr, 12 op/s 2024-04-12T21:25:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:04 smithi153 ceph-mon[36569]: pgmap v692: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 32 KiB/s wr, 15 op/s 2024-04-12T21:25:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:04 smithi104 ceph-mon[36624]: pgmap v692: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 32 KiB/s wr, 15 op/s 2024-04-12T21:25:05.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:04 smithi081 ceph-mon[31453]: pgmap v692: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 32 KiB/s wr, 15 op/s 2024-04-12T21:25:07.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:06 smithi153 ceph-mon[36569]: pgmap v693: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 32 KiB/s wr, 18 op/s 2024-04-12T21:25:07.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:06 smithi104 ceph-mon[36624]: pgmap v693: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 32 KiB/s wr, 18 op/s 2024-04-12T21:25:07.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:06 smithi081 ceph-mon[31453]: pgmap v693: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 32 KiB/s wr, 18 op/s 2024-04-12T21:25:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:08.146+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:25:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:08.146+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:25:08.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:08.146+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:25:10.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:09 smithi153 ceph-mon[36569]: pgmap v694: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 27 KiB/s wr, 15 op/s 2024-04-12T21:25:10.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:09 smithi104 ceph-mon[36624]: pgmap v694: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 27 KiB/s wr, 15 op/s 2024-04-12T21:25:10.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:09 smithi081 ceph-mon[31453]: pgmap v694: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 27 KiB/s wr, 15 op/s 2024-04-12T21:25:11.973 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:11 smithi153 ceph-mon[36569]: pgmap v695: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 27 KiB/s wr, 17 op/s 2024-04-12T21:25:12.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:11 smithi104 ceph-mon[36624]: pgmap v695: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 27 KiB/s wr, 17 op/s 2024-04-12T21:25:12.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:11 smithi081 ceph-mon[31453]: pgmap v695: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 27 KiB/s wr, 17 op/s 2024-04-12T21:25:12.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:12 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:25:12.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:12 smithi104 systemd[1]: Stopped Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:25:12.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:12 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 21.032s CPU time. 2024-04-12T21:25:12.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:12 smithi104 systemd[1]: Starting Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:25:12.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:12 smithi104 podman[70706]: 2024-04-12 21:25:12.21912403 +0000 UTC m=+0.018140866 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:25:12.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:12.783+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4005e80 con 0x7f1704076980 2024-04-12T21:25:12.780 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:12.784+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4005e80 con 0x7f1704076980 2024-04-12T21:25:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:12.824+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d0005c40 con 0x563c9e245d20 2024-04-12T21:25:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:12.825+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d0005c40 con 0x563c9e245d20 2024-04-12T21:25:13.667 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:13 smithi104 podman[70706]: 2024-04-12 21:25:13.362643575 +0000 UTC m=+1.161660410 container create 43e14763e62ab0d698aa584c0781fa85dcf160284cef888741853ac8f50b3a23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:25:13.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:13 smithi104 ceph-mon[36624]: pgmap v696: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 27 KiB/s wr, 15 op/s 2024-04-12T21:25:13.928 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:13 smithi104 podman[70706]: 2024-04-12 21:25:13.832459753 +0000 UTC m=+1.631476602 container init 43e14763e62ab0d698aa584c0781fa85dcf160284cef888741853ac8f50b3a23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:25:13.928 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:13 smithi104 podman[70706]: 2024-04-12 21:25:13.835455998 +0000 UTC m=+1.634472824 container start 43e14763e62ab0d698aa584c0781fa85dcf160284cef888741853ac8f50b3a23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409) 2024-04-12T21:25:13.928 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:13 smithi104 ceph-mds[70726]: -- 172.21.15.104:0/3615204915 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55571de9fa40 con 0x55571de73000 2024-04-12T21:25:13.928 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:13 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: starting mds.h at 2024-04-12T21:25:14.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:13 smithi153 ceph-mon[36569]: pgmap v696: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 27 KiB/s wr, 15 op/s 2024-04-12T21:25:14.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:13 smithi081 ceph-mon[31453]: pgmap v696: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 27 KiB/s wr, 15 op/s 2024-04-12T21:25:14.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:13 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:25:13.864+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:25:14.328 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:13 smithi104 bash[70706]: 43e14763e62ab0d698aa584c0781fa85dcf160284cef888741853ac8f50b3a23 2024-04-12T21:25:14.328 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:13 smithi104 systemd[1]: Started Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: daemon mds.h restarted 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: osdmap e112: 12 total, 12 up, 12 in 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:boot 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:25:15.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:14 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:25:15.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: daemon mds.h restarted 2024-04-12T21:25:15.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:15.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:25:15.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: osdmap e112: 12 total, 12 up, 12 in 2024-04-12T21:25:15.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:boot 2024-04-12T21:25:15.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:25:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:25:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:25:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:25:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:14 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:25:15.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: daemon mds.h restarted 2024-04-12T21:25:15.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:25:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: osdmap e112: 12 total, 12 up, 12 in 2024-04-12T21:25:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:boot 2024-04-12T21:25:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: Standby daemon mds.c assigned to filesystem cephfs as rank 0 2024-04-12T21:25:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:25:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:25:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-12T21:25:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:14 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:replay} 9 up:standby 2024-04-12T21:25:16.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:15 smithi081 ceph-mon[31453]: pgmap v698: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 614 B/s wr, 6 op/s 2024-04-12T21:25:16.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:reconnect 2024-04-12T21:25:16.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:15 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:25:16.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:15 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:25:16.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:15 smithi153 ceph-mon[36569]: pgmap v698: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 614 B/s wr, 6 op/s 2024-04-12T21:25:16.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:reconnect 2024-04-12T21:25:16.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:15 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:25:16.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:15 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:25:16.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:15 smithi104 ceph-mon[36624]: pgmap v698: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 614 B/s wr, 6 op/s 2024-04-12T21:25:16.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:reconnect 2024-04-12T21:25:16.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:15 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:reconnect} 9 up:standby 2024-04-12T21:25:16.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:15 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:25:17.229 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:16 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:rejoin 2024-04-12T21:25:17.229 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:16 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:25:17.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:16 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:rejoin 2024-04-12T21:25:17.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:16 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:25:17.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:16 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:rejoin 2024-04-12T21:25:17.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:16 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:rejoin} 9 up:standby 2024-04-12T21:25:18.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-mon[36569]: pgmap v699: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 B/s wr, 4 op/s 2024-04-12T21:25:18.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:clientreplay 2024-04-12T21:25:18.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:25:18.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-mon[36569]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:25:18.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:18.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:25:18.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:active 2024-04-12T21:25:18.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:18.047 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:25:18.047 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: *** Caught signal (Aborted) ** 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: in thread 7f2f43c27700 thread_name:ms_dispatch 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2f517e1d20] 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2: gsignal() 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 3: abort() 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55991400f268] 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5599140ab0fc] 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5599143dff0a] 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 7: (MDLog::trim()+0xca9) [0x5599143e10c9] 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5599143e15a8] 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f2f506cdb23] 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f2f517d71ca] 2024-04-12T21:25:18.048 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 11: clone() 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2024-04-12T21:25:17.700+0000 7f2f43c27700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: in thread 7f2f43c27700 thread_name:ms_dispatch 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2f517e1d20] 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2: gsignal() 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 3: abort() 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55991400f268] 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5599140ab0fc] 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5599143dff0a] 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 7: (MDLog::trim()+0xca9) [0x5599143e10c9] 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5599143e15a8] 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f2f506cdb23] 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f2f517d71ca] 2024-04-12T21:25:18.049 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 11: clone() 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 0> 2024-04-12T21:25:17.700+0000 7f2f43c27700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: in thread 7f2f43c27700 thread_name:ms_dispatch 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2f517e1d20] 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2: gsignal() 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 3: abort() 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55991400f268] 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5599140ab0fc] 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5599143dff0a] 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 7: (MDLog::trim()+0xca9) [0x5599143e10c9] 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5599143e15a8] 2024-04-12T21:25:18.050 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f2f506cdb23] 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f2f517d71ca] 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 11: clone() 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: -10> 2024-04-12T21:25:17.700+0000 7f2f43c27700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: in thread 7f2f43c27700 thread_name:ms_dispatch 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2f517e1d20] 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2: gsignal() 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 3: abort() 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55991400f268] 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5599140ab0fc] 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5599143dff0a] 2024-04-12T21:25:18.051 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 7: (MDLog::trim()+0xca9) [0x5599143e10c9] 2024-04-12T21:25:18.052 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5599143e15a8] 2024-04-12T21:25:18.052 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f2f506cdb23] 2024-04-12T21:25:18.052 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f2f517d71ca] 2024-04-12T21:25:18.052 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 11: clone() 2024-04-12T21:25:18.052 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:18.052 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72431]: 2024-04-12T21:25:18.052 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:17 smithi153 podman[72483]: 2024-04-12 21:25:17.757981083 +0000 UTC m=+0.018194135 container died e39df45e30bec010300fccf0ca4036a1a9136ffe5fb0eb27da5ba5bcc41fa965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-12T21:25:18.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:17 smithi081 ceph-mon[31453]: pgmap v699: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 B/s wr, 4 op/s 2024-04-12T21:25:18.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:17 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:clientreplay 2024-04-12T21:25:18.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:17 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:25:18.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:17 smithi081 ceph-mon[31453]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:25:18.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:17 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:18.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:17 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:25:18.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:17 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:active 2024-04-12T21:25:18.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:17 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:18.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:17 smithi104 ceph-mon[36624]: pgmap v699: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 B/s wr, 4 op/s 2024-04-12T21:25:18.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:17 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:clientreplay 2024-04-12T21:25:18.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:17 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:clientreplay} 9 up:standby 2024-04-12T21:25:18.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:17 smithi104 ceph-mon[36624]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:25:18.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:17 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:18.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:17 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:25:18.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:17 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/2171555802,v1:172.21.15.153:6837/2171555802] up:active 2024-04-12T21:25:18.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:17 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:18.802 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:18 smithi153 podman[72483]: 2024-04-12 21:25:18.462684796 +0000 UTC m=+0.722897865 container remove e39df45e30bec010300fccf0ca4036a1a9136ffe5fb0eb27da5ba5bcc41fa965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-12T21:25:18.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:18 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:25:18.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:18 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-12T21:25:18.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:18 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 21.477s CPU time. 2024-04-12T21:25:20.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:19 smithi081 ceph-mon[31453]: pgmap v700: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 102 B/s wr, 5 op/s 2024-04-12T21:25:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:19 smithi153 ceph-mon[36569]: pgmap v700: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 102 B/s wr, 5 op/s 2024-04-12T21:25:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:19 smithi104 ceph-mon[36624]: pgmap v700: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 102 B/s wr, 5 op/s 2024-04-12T21:25:22.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:21 smithi081 ceph-mon[31453]: pgmap v701: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-04-12T21:25:22.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:21 smithi153 ceph-mon[36569]: pgmap v701: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-04-12T21:25:22.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:21 smithi104 ceph-mon[36624]: pgmap v701: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-04-12T21:25:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:23.161+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:25:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:23.161+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:25:23.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:23.161+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:25:24.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:23 smithi153 ceph-mon[36569]: pgmap v702: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 KiB/s wr, 19 op/s 2024-04-12T21:25:24.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:23 smithi104 ceph-mon[36624]: pgmap v702: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 KiB/s wr, 19 op/s 2024-04-12T21:25:24.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:23 smithi081 ceph-mon[31453]: pgmap v702: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 KiB/s wr, 19 op/s 2024-04-12T21:25:26.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:25 smithi081 ceph-mon[31453]: pgmap v703: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 24 KiB/s wr, 18 op/s 2024-04-12T21:25:26.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:25 smithi153 ceph-mon[36569]: pgmap v703: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 24 KiB/s wr, 18 op/s 2024-04-12T21:25:26.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:25 smithi104 ceph-mon[36624]: pgmap v703: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 24 KiB/s wr, 18 op/s 2024-04-12T21:25:28.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:27 smithi153 ceph-mon[36569]: pgmap v704: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 21 KiB/s wr, 18 op/s 2024-04-12T21:25:28.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:27 smithi104 ceph-mon[36624]: pgmap v704: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 21 KiB/s wr, 18 op/s 2024-04-12T21:25:28.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:27 smithi081 ceph-mon[31453]: pgmap v704: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 21 KiB/s wr, 18 op/s 2024-04-12T21:25:29.067 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:28 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:25:29.067 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:28 smithi153 systemd[1]: Stopped Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:25:29.068 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:28 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 21.477s CPU time. 2024-04-12T21:25:29.068 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:28 smithi153 systemd[1]: Starting Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:25:29.068 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:29 smithi153 podman[72578]: 2024-04-12 21:25:28.946277362 +0000 UTC m=+0.017297848 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:25:29.068 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:29 smithi153 podman[72578]: 2024-04-12 21:25:29.071927923 +0000 UTC m=+0.142948393 container create 81fa5d8b6af7f8284d8067fd90ea2963b5a705f1272ceac7543b226ae8a2cae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:25:29.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:25:29.365+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:25:29.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:29 smithi153 podman[72578]: 2024-04-12 21:25:29.324832555 +0000 UTC m=+0.395853033 container init 81fa5d8b6af7f8284d8067fd90ea2963b5a705f1272ceac7543b226ae8a2cae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-12T21:25:29.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:29 smithi153 podman[72578]: 2024-04-12 21:25:29.327490647 +0000 UTC m=+0.398511137 container start 81fa5d8b6af7f8284d8067fd90ea2963b5a705f1272ceac7543b226ae8a2cae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:25:29.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mds[72650]: -- 172.21.15.153:0/3702104348 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a4f441ba40 con 0x55a4f43ee800 2024-04-12T21:25:29.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: starting mds.c at 2024-04-12T21:25:29.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:29 smithi153 bash[72578]: 81fa5d8b6af7f8284d8067fd90ea2963b5a705f1272ceac7543b226ae8a2cae3 2024-04-12T21:25:29.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:29 smithi153 systemd[1]: Started Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:25:30.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: pgmap v705: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-12T21:25:30.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:25:30.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: daemon mds.c restarted 2024-04-12T21:25:30.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:30.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:25:30.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: osdmap e113: 12 total, 12 up, 12 in 2024-04-12T21:25:30.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:boot 2024-04-12T21:25:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:25:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:25:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:25:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:25:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:29 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:25:30.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: pgmap v705: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-12T21:25:30.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:25:30.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: daemon mds.c restarted 2024-04-12T21:25:30.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:30.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:25:30.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: osdmap e113: 12 total, 12 up, 12 in 2024-04-12T21:25:30.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:boot 2024-04-12T21:25:30.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:25:30.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:25:30.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:25:30.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:25:30.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:29 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:25:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: pgmap v705: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 21 KiB/s wr, 17 op/s 2024-04-12T21:25:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:25:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: daemon mds.c restarted 2024-04-12T21:25:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:25:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: osdmap e113: 12 total, 12 up, 12 in 2024-04-12T21:25:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:boot 2024-04-12T21:25:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: Standby daemon mds.h assigned to filesystem cephfs as rank 0 2024-04-12T21:25:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:25:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 9 up:standby, 1 failed 2024-04-12T21:25:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-12T21:25:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:29 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:replay} 9 up:standby 2024-04-12T21:25:31.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:25:31.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:25:31.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:25:31.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:25:31.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:25:31.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:25:31.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:25:31.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:25:31.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:25:32.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:31 smithi153 ceph-mon[36569]: pgmap v707: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 24 KiB/s wr, 12 op/s 2024-04-12T21:25:32.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:31 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:reconnect 2024-04-12T21:25:32.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:31 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:25:32.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:31 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119999 2024-04-12T21:25:32.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:31 smithi104 ceph-mon[36624]: pgmap v707: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 24 KiB/s wr, 12 op/s 2024-04-12T21:25:32.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:31 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:reconnect 2024-04-12T21:25:32.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:31 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:25:32.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:31 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119999 2024-04-12T21:25:32.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:31 smithi081 ceph-mon[31453]: pgmap v707: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 24 KiB/s wr, 12 op/s 2024-04-12T21:25:32.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:31 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:reconnect 2024-04-12T21:25:32.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:31 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:reconnect} 9 up:standby 2024-04-12T21:25:32.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:31 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119999 2024-04-12T21:25:33.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:32 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:rejoin 2024-04-12T21:25:33.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:32 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:25:33.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:32 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:rejoin 2024-04-12T21:25:33.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:32 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:25:33.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:32 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:rejoin 2024-04-12T21:25:33.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:32 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:rejoin} 9 up:standby 2024-04-12T21:25:34.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:33 smithi153 ceph-mon[36569]: pgmap v708: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:25:34.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:33 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:clientreplay 2024-04-12T21:25:34.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:33 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:25:34.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:33 smithi153 ceph-mon[36569]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:25:34.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:33 smithi104 ceph-mon[36624]: pgmap v708: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:25:34.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:33 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:clientreplay 2024-04-12T21:25:34.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:33 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:25:34.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:33 smithi104 ceph-mon[36624]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:25:34.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:33 smithi081 ceph-mon[31453]: pgmap v708: 129 pgs: 129 active+clean; 614 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:25:34.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:33 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:clientreplay 2024-04-12T21:25:34.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:33 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=h=up:clientreplay} 9 up:standby 2024-04-12T21:25:34.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:33 smithi081 ceph-mon[31453]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-12T21:25:34.746 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: *** Caught signal (Aborted) ** 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: in thread 7f3406eb1700 thread_name:ms_dispatch 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3414a6bd20] 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2: gsignal() 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 3: abort() 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55571c606268] 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55571c6a20fc] 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55571c9d6f0a] 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 7: (MDLog::trim()+0xca9) [0x55571c9d80c9] 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55571c9d85a8] 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3413957b23] 2024-04-12T21:25:34.747 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3414a611ca] 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 11: clone() 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2024-04-12T21:25:34.394+0000 7f3406eb1700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: in thread 7f3406eb1700 thread_name:ms_dispatch 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3414a6bd20] 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2: gsignal() 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 3: abort() 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55571c606268] 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55571c6a20fc] 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55571c9d6f0a] 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 7: (MDLog::trim()+0xca9) [0x55571c9d80c9] 2024-04-12T21:25:34.748 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55571c9d85a8] 2024-04-12T21:25:34.749 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3413957b23] 2024-04-12T21:25:34.749 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3414a611ca] 2024-04-12T21:25:34.749 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 11: clone() 2024-04-12T21:25:34.749 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:34.749 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2024-04-12T21:25:34.749 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 0> 2024-04-12T21:25:34.394+0000 7f3406eb1700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: in thread 7f3406eb1700 thread_name:ms_dispatch 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3414a6bd20] 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2: gsignal() 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 3: abort() 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55571c606268] 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55571c6a20fc] 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55571c9d6f0a] 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 7: (MDLog::trim()+0xca9) [0x55571c9d80c9] 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55571c9d85a8] 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3413957b23] 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3414a611ca] 2024-04-12T21:25:34.750 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 11: clone() 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: -9> 2024-04-12T21:25:34.394+0000 7f3406eb1700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: in thread 7f3406eb1700 thread_name:ms_dispatch 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3414a6bd20] 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2: gsignal() 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 3: abort() 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55571c606268] 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55571c6a20fc] 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55571c9d6f0a] 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 7: (MDLog::trim()+0xca9) [0x55571c9d80c9] 2024-04-12T21:25:34.751 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55571c9d85a8] 2024-04-12T21:25:34.752 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f3413957b23] 2024-04-12T21:25:34.752 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f3414a611ca] 2024-04-12T21:25:34.752 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 11: clone() 2024-04-12T21:25:34.752 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:34.752 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h[70722]: 2024-04-12T21:25:34.752 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:34 smithi104 podman[70893]: 2024-04-12 21:25:34.457335915 +0000 UTC m=+0.019123187 container died 43e14763e62ab0d698aa584c0781fa85dcf160284cef888741853ac8f50b3a23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T21:25:35.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:35.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:25:35.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:active 2024-04-12T21:25:35.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:25:35.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:34 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:35.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:34 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:35.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:34 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:25:35.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:34 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:active 2024-04-12T21:25:35.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:34 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:25:35.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:34 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:35.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:34 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:35.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:34 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:25:35.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:34 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6836/93607722,v1:172.21.15.104:6837/93607722] up:active 2024-04-12T21:25:35.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:34 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 9 up:standby 2024-04-12T21:25:35.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:34 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=h=up:active} 1 up:standby-replay 8 up:standby 2024-04-12T21:25:35.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:35 smithi104 podman[70893]: 2024-04-12 21:25:35.168864211 +0000 UTC m=+0.730651473 container remove 43e14763e62ab0d698aa584c0781fa85dcf160284cef888741853ac8f50b3a23 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-h, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:25:35.578 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:35 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:25:35.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:35 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-12T21:25:35.579 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:35 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 21.449s CPU time. 2024-04-12T21:25:36.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:36 smithi153 ceph-mon[36569]: pgmap v709: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:25:36.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:36 smithi104 ceph-mon[36624]: pgmap v709: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:25:36.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:36 smithi081 ceph-mon[31453]: pgmap v709: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:25:38.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:38.177+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:25:38.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:38.177+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:25:38.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:38.177+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:25:38.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:38 smithi153 ceph-mon[36569]: pgmap v710: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 KiB/s wr, 25 op/s 2024-04-12T21:25:38.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:38 smithi104 ceph-mon[36624]: pgmap v710: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 KiB/s wr, 25 op/s 2024-04-12T21:25:38.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:38 smithi081 ceph-mon[31453]: pgmap v710: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 KiB/s wr, 25 op/s 2024-04-12T21:25:40.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:40 smithi153 ceph-mon[36569]: pgmap v711: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 KiB/s wr, 25 op/s 2024-04-12T21:25:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:40 smithi104 ceph-mon[36624]: pgmap v711: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 KiB/s wr, 25 op/s 2024-04-12T21:25:40.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:40 smithi081 ceph-mon[31453]: pgmap v711: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 KiB/s wr, 25 op/s 2024-04-12T21:25:42.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:42 smithi153 ceph-mon[36569]: pgmap v712: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 18 KiB/s wr, 24 op/s 2024-04-12T21:25:42.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:42 smithi104 ceph-mon[36624]: pgmap v712: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 18 KiB/s wr, 24 op/s 2024-04-12T21:25:42.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:42 smithi081 ceph-mon[31453]: pgmap v712: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 18 KiB/s wr, 24 op/s 2024-04-12T21:25:44.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:44 smithi153 ceph-mon[36569]: pgmap v713: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 17 KiB/s wr, 23 op/s 2024-04-12T21:25:44.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:44 smithi104 ceph-mon[36624]: pgmap v713: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 17 KiB/s wr, 23 op/s 2024-04-12T21:25:44.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:44 smithi081 ceph-mon[31453]: pgmap v713: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 17 KiB/s wr, 23 op/s 2024-04-12T21:25:46.039 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:45 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:25:46.040 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:45 smithi104 systemd[1]: Stopped Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:25:46.040 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:45 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Consumed 21.449s CPU time. 2024-04-12T21:25:46.040 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:45 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Start request repeated too quickly. 2024-04-12T21:25:46.040 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:45 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-12T21:25:46.040 INFO:journalctl@ceph.mds.h.smithi104.stdout:Apr 12 21:25:45 smithi104 systemd[1]: Failed to start Ceph mds.h for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:25:46.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:46 smithi153 ceph-mon[36569]: pgmap v714: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 17 KiB/s wr, 23 op/s 2024-04-12T21:25:46.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:46 smithi104 ceph-mon[36624]: pgmap v714: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 17 KiB/s wr, 23 op/s 2024-04-12T21:25:46.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:46 smithi081 ceph-mon[31453]: pgmap v714: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 17 KiB/s wr, 23 op/s 2024-04-12T21:25:48.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:48 smithi153 ceph-mon[36569]: pgmap v715: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 17 KiB/s wr, 21 op/s 2024-04-12T21:25:48.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:48 smithi104 ceph-mon[36624]: pgmap v715: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 17 KiB/s wr, 21 op/s 2024-04-12T21:25:48.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:48 smithi081 ceph-mon[31453]: pgmap v715: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 17 KiB/s wr, 21 op/s 2024-04-12T21:25:50.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:50 smithi153 ceph-mon[36569]: pgmap v716: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 170 B/s wr, 4 op/s 2024-04-12T21:25:50.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:50 smithi104 ceph-mon[36624]: pgmap v716: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 170 B/s wr, 4 op/s 2024-04-12T21:25:50.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:50 smithi081 ceph-mon[31453]: pgmap v716: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 170 B/s wr, 4 op/s 2024-04-12T21:25:52.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:52 smithi104 ceph-mon[36624]: pgmap v717: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 170 B/s wr, 4 op/s 2024-04-12T21:25:52.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:52 smithi081 ceph-mon[31453]: pgmap v717: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 170 B/s wr, 4 op/s 2024-04-12T21:25:52.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:52 smithi153 ceph-mon[36569]: pgmap v717: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 170 B/s wr, 4 op/s 2024-04-12T21:25:53.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:53.183+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:25:53.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:53.183+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:25:53.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:25:53.183+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:25:53.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:53 smithi104 ceph-mon[36624]: Replacing daemon mds.h as rank 0 with standby daemon mds.c 2024-04-12T21:25:53.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:53 smithi104 ceph-mon[36624]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-12T21:25:53.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:53 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:53.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:53 smithi104 ceph-mon[36624]: osdmap e114: 12 total, 12 up, 12 in 2024-04-12T21:25:53.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:53 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:replay} 8 up:standby 2024-04-12T21:25:53.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:53 smithi081 ceph-mon[31453]: Replacing daemon mds.h as rank 0 with standby daemon mds.c 2024-04-12T21:25:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:53 smithi081 ceph-mon[31453]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-12T21:25:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:53 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:53 smithi081 ceph-mon[31453]: osdmap e114: 12 total, 12 up, 12 in 2024-04-12T21:25:53.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:53 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:replay} 8 up:standby 2024-04-12T21:25:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:53 smithi153 ceph-mon[36569]: Replacing daemon mds.h as rank 0 with standby daemon mds.c 2024-04-12T21:25:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:53 smithi153 ceph-mon[36569]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-04-12T21:25:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:53 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:25:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:53 smithi153 ceph-mon[36569]: osdmap e114: 12 total, 12 up, 12 in 2024-04-12T21:25:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:53 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:replay} 8 up:standby 2024-04-12T21:25:54.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:54 smithi153 ceph-mon[36569]: pgmap v719: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3 op/s 2024-04-12T21:25:54.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:54 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:reconnect 2024-04-12T21:25:54.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:54 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:reconnect} 8 up:standby 2024-04-12T21:25:54.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:54 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199997 2024-04-12T21:25:54.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:54 smithi104 ceph-mon[36624]: pgmap v719: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3 op/s 2024-04-12T21:25:54.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:54 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:reconnect 2024-04-12T21:25:54.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:54 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:reconnect} 8 up:standby 2024-04-12T21:25:54.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:54 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199997 2024-04-12T21:25:54.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:54 smithi081 ceph-mon[31453]: pgmap v719: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3 op/s 2024-04-12T21:25:54.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:54 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:reconnect 2024-04-12T21:25:54.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:54 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:reconnect} 8 up:standby 2024-04-12T21:25:54.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:54 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199997 2024-04-12T21:25:55.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:55 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:rejoin 2024-04-12T21:25:55.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:55 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:rejoin} 8 up:standby 2024-04-12T21:25:55.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:55 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:rejoin 2024-04-12T21:25:55.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:55 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:rejoin} 8 up:standby 2024-04-12T21:25:55.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:55 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:rejoin 2024-04-12T21:25:55.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:55 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:rejoin} 8 up:standby 2024-04-12T21:25:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:56 smithi153 ceph-mon[36569]: pgmap v720: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2 op/s 2024-04-12T21:25:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:56 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:clientreplay 2024-04-12T21:25:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:56 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=c=up:clientreplay} 8 up:standby 2024-04-12T21:25:56.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:56 smithi153 ceph-mon[36569]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:25:56.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:56 smithi081 ceph-mon[31453]: pgmap v720: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2 op/s 2024-04-12T21:25:56.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:56 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:clientreplay 2024-04-12T21:25:56.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:56 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=c=up:clientreplay} 8 up:standby 2024-04-12T21:25:56.804 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:56 smithi081 ceph-mon[31453]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:25:56.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:56 smithi104 ceph-mon[36624]: pgmap v720: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2 op/s 2024-04-12T21:25:56.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:56 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:clientreplay 2024-04-12T21:25:56.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:56 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=c=up:clientreplay} 8 up:standby 2024-04-12T21:25:56.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:56 smithi104 ceph-mon[36624]: daemon mds.c is now active in filesystem cephfs as rank 0 2024-04-12T21:25:57.303 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: *** Caught signal (Aborted) ** 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: in thread 7fc18f07a700 thread_name:ms_dispatch 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc19cc34d20] 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2: gsignal() 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 3: abort() 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a4f1812268] 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a4f18ae0fc] 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a4f1be2f0a] 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 7: (MDLog::trim()+0xca9) [0x55a4f1be40c9] 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a4f1be45a8] 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc19bb20b23] 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc19cc2a1ca] 2024-04-12T21:25:57.304 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 11: clone() 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2024-04-12T21:25:57.019+0000 7fc18f07a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: in thread 7fc18f07a700 thread_name:ms_dispatch 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc19cc34d20] 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2: gsignal() 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 3: abort() 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a4f1812268] 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a4f18ae0fc] 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a4f1be2f0a] 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 7: (MDLog::trim()+0xca9) [0x55a4f1be40c9] 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a4f1be45a8] 2024-04-12T21:25:57.305 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc19bb20b23] 2024-04-12T21:25:57.306 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc19cc2a1ca] 2024-04-12T21:25:57.306 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 11: clone() 2024-04-12T21:25:57.306 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:57.306 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2024-04-12T21:25:57.306 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 0> 2024-04-12T21:25:57.019+0000 7fc18f07a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:57.306 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: in thread 7fc18f07a700 thread_name:ms_dispatch 2024-04-12T21:25:57.306 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2024-04-12T21:25:57.306 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc19cc34d20] 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2: gsignal() 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 3: abort() 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a4f1812268] 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a4f18ae0fc] 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a4f1be2f0a] 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 7: (MDLog::trim()+0xca9) [0x55a4f1be40c9] 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a4f1be45a8] 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc19bb20b23] 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc19cc2a1ca] 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 11: clone() 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2024-04-12T21:25:57.307 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: -10> 2024-04-12T21:25:57.019+0000 7fc18f07a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: in thread 7fc18f07a700 thread_name:ms_dispatch 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc19cc34d20] 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2: gsignal() 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 3: abort() 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a4f1812268] 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a4f18ae0fc] 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a4f1be2f0a] 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 7: (MDLog::trim()+0xca9) [0x55a4f1be40c9] 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a4f1be45a8] 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc19bb20b23] 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc19cc2a1ca] 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 11: clone() 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:25:57.308 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c[72593]: 2024-04-12T21:25:57.309 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:57 smithi153 podman[72775]: 2024-04-12 21:25:57.094574966 +0000 UTC m=+0.023930896 container died 81fa5d8b6af7f8284d8067fd90ea2963b5a705f1272ceac7543b226ae8a2cae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-12T21:25:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:25:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:active 2024-04-12T21:25:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 8 up:standby 2024-04-12T21:25:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:57 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 7 up:standby 2024-04-12T21:25:57.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:57 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:57.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:57 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:25:57.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:57 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:active 2024-04-12T21:25:57.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:57 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 8 up:standby 2024-04-12T21:25:57.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:57 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 7 up:standby 2024-04-12T21:25:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:57 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:25:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:57 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:25:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:57 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6836/3013687172,v1:172.21.15.153:6837/3013687172] up:active 2024-04-12T21:25:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:57 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 8 up:standby 2024-04-12T21:25:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:57 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=c=up:active} 1 up:standby-replay 7 up:standby 2024-04-12T21:25:58.264 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:58 smithi153 podman[72775]: 2024-04-12 21:25:58.015281837 +0000 UTC m=+0.944637739 container remove 81fa5d8b6af7f8284d8067fd90ea2963b5a705f1272ceac7543b226ae8a2cae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:25:58.264 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:58 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:25:58.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:58 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-12T21:25:58.553 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:25:58 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 22.168s CPU time. 2024-04-12T21:25:58.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:25:58 smithi153 ceph-mon[36569]: pgmap v721: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:25:58.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:25:58 smithi104 ceph-mon[36624]: pgmap v721: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:25:58.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:25:58 smithi081 ceph-mon[31453]: pgmap v721: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:26:00.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:00 smithi153 ceph-mon[36569]: pgmap v722: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 41 KiB/s wr, 6 op/s 2024-04-12T21:26:00.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:00 smithi104 ceph-mon[36624]: pgmap v722: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 41 KiB/s wr, 6 op/s 2024-04-12T21:26:00.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:00 smithi081 ceph-mon[31453]: pgmap v722: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 41 KiB/s wr, 6 op/s 2024-04-12T21:26:02.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:02 smithi153 ceph-mon[36569]: pgmap v723: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-12T21:26:02.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:02 smithi104 ceph-mon[36624]: pgmap v723: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-12T21:26:02.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:02 smithi081 ceph-mon[31453]: pgmap v723: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-12T21:26:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:04 smithi153 ceph-mon[36569]: pgmap v724: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 85 KiB/s wr, 15 op/s 2024-04-12T21:26:04.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:04 smithi104 ceph-mon[36624]: pgmap v724: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 85 KiB/s wr, 15 op/s 2024-04-12T21:26:04.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:04 smithi081 ceph-mon[31453]: pgmap v724: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 85 KiB/s wr, 15 op/s 2024-04-12T21:26:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:06 smithi153 ceph-mon[36569]: pgmap v725: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 73 KiB/s wr, 14 op/s 2024-04-12T21:26:06.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:06 smithi104 ceph-mon[36624]: pgmap v725: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 73 KiB/s wr, 14 op/s 2024-04-12T21:26:06.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:06 smithi081 ceph-mon[31453]: pgmap v725: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 73 KiB/s wr, 14 op/s 2024-04-12T21:26:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:08.198+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:26:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:08.198+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:26:08.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:08.198+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:26:08.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:08 smithi153 ceph-mon[36569]: pgmap v726: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 73 KiB/s wr, 16 op/s 2024-04-12T21:26:08.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:26:08 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:26:08.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:26:08 smithi153 systemd[1]: Stopped Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:26:08.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:26:08 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Consumed 22.168s CPU time. 2024-04-12T21:26:08.803 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:26:08 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Start request repeated too quickly. 2024-04-12T21:26:08.804 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:26:08 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-12T21:26:08.804 INFO:journalctl@ceph.mds.c.smithi153.stdout:Apr 12 21:26:08 smithi153 systemd[1]: Failed to start Ceph mds.c for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:26:08.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:08 smithi104 ceph-mon[36624]: pgmap v726: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 73 KiB/s wr, 16 op/s 2024-04-12T21:26:08.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:08 smithi081 ceph-mon[31453]: pgmap v726: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 73 KiB/s wr, 16 op/s 2024-04-12T21:26:10.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:10 smithi153 ceph-mon[36569]: pgmap v727: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 38 KiB/s wr, 12 op/s 2024-04-12T21:26:10.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:10 smithi104 ceph-mon[36624]: pgmap v727: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 38 KiB/s wr, 12 op/s 2024-04-12T21:26:10.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:10 smithi081 ceph-mon[31453]: pgmap v727: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 38 KiB/s wr, 12 op/s 2024-04-12T21:26:12.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:12 smithi153 ceph-mon[36569]: pgmap v728: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 38 KiB/s wr, 12 op/s 2024-04-12T21:26:12.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:12 smithi153 ceph-mon[36569]: Replacing daemon mds.c as rank 0 with standby daemon mds.f 2024-04-12T21:26:12.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:12 smithi153 ceph-mon[36569]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-12T21:26:12.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:12 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:12.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:12 smithi153 ceph-mon[36569]: osdmap e115: 12 total, 12 up, 12 in 2024-04-12T21:26:12.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:12 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:26:12.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:12 smithi104 ceph-mon[36624]: pgmap v728: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 38 KiB/s wr, 12 op/s 2024-04-12T21:26:12.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:12 smithi104 ceph-mon[36624]: Replacing daemon mds.c as rank 0 with standby daemon mds.f 2024-04-12T21:26:12.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:12 smithi104 ceph-mon[36624]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-12T21:26:12.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:12 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:12.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:12 smithi104 ceph-mon[36624]: osdmap e115: 12 total, 12 up, 12 in 2024-04-12T21:26:12.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:12 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:26:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:12 smithi081 ceph-mon[31453]: pgmap v728: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 38 KiB/s wr, 12 op/s 2024-04-12T21:26:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:12 smithi081 ceph-mon[31453]: Replacing daemon mds.c as rank 0 with standby daemon mds.f 2024-04-12T21:26:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:12 smithi081 ceph-mon[31453]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-12T21:26:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:12 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:12 smithi081 ceph-mon[31453]: osdmap e115: 12 total, 12 up, 12 in 2024-04-12T21:26:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:12 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:26:14.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:14 smithi153 ceph-mon[36569]: pgmap v730: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:26:14.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:14 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:reconnect 2024-04-12T21:26:14.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:14 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:26:14.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:14 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:26:14.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:14 smithi104 ceph-mon[36624]: pgmap v730: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:26:14.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:14 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:reconnect 2024-04-12T21:26:14.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:14 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:26:14.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:14 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:26:14.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:14 smithi081 ceph-mon[31453]: pgmap v730: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:26:14.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:14 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:reconnect 2024-04-12T21:26:14.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:14 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:26:14.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:14 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:26:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:rejoin 2024-04-12T21:26:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:15 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:26:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:rejoin 2024-04-12T21:26:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:15 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:26:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:rejoin 2024-04-12T21:26:15.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:15 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:26:16.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:16 smithi153 ceph-mon[36569]: pgmap v731: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3 op/s 2024-04-12T21:26:16.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:16 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:clientreplay 2024-04-12T21:26:16.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:16 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:26:16.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:16 smithi153 ceph-mon[36569]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:26:16.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:16 smithi104 ceph-mon[36624]: pgmap v731: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3 op/s 2024-04-12T21:26:16.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:16 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:clientreplay 2024-04-12T21:26:16.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:16 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:26:16.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:16 smithi104 ceph-mon[36624]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:26:16.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:16 smithi081 ceph-mon[31453]: pgmap v731: 129 pgs: 129 active+clean; 613 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3 op/s 2024-04-12T21:26:16.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:16 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:clientreplay 2024-04-12T21:26:16.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:16 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:26:16.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:16 smithi081 ceph-mon[31453]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:26:17.473 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2024-04-12T21:26:17.111+0000 7fe64ede5700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:26:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:26:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:active 2024-04-12T21:26:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:26:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:17.804 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:26:17.804 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: *** Caught signal (Aborted) ** 2024-04-12T21:26:17.804 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: in thread 7fe64cde1700 thread_name:ms_dispatch 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe65a99bd20] 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2: gsignal() 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 3: abort() 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x555a1b10e268] 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x555a1b1aa0fc] 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x555a1b4def0a] 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 7: (MDLog::trim()+0xca9) [0x555a1b4e00c9] 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 8: (MDLog::log_trim_upkeep()+0x358) [0x555a1b4e05a8] 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe659887b23] 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe65a9911ca] 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 11: clone() 2024-04-12T21:26:17.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2024-04-12T21:26:17.478+0000 7fe64cde1700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: in thread 7fe64cde1700 thread_name:ms_dispatch 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe65a99bd20] 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2: gsignal() 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 3: abort() 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x555a1b10e268] 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x555a1b1aa0fc] 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x555a1b4def0a] 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 7: (MDLog::trim()+0xca9) [0x555a1b4e00c9] 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 8: (MDLog::log_trim_upkeep()+0x358) [0x555a1b4e05a8] 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe659887b23] 2024-04-12T21:26:17.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe65a9911ca] 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 11: clone() 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 0> 2024-04-12T21:26:17.478+0000 7fe64cde1700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: in thread 7fe64cde1700 thread_name:ms_dispatch 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe65a99bd20] 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2: gsignal() 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 3: abort() 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x555a1b10e268] 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x555a1b1aa0fc] 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x555a1b4def0a] 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 7: (MDLog::trim()+0xca9) [0x555a1b4e00c9] 2024-04-12T21:26:17.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 8: (MDLog::log_trim_upkeep()+0x358) [0x555a1b4e05a8] 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe659887b23] 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe65a9911ca] 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 11: clone() 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: -9> 2024-04-12T21:26:17.478+0000 7fe64cde1700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: in thread 7fe64cde1700 thread_name:ms_dispatch 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe65a99bd20] 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2: gsignal() 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 3: abort() 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x555a1b10e268] 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x555a1b1aa0fc] 2024-04-12T21:26:17.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x555a1b4def0a] 2024-04-12T21:26:17.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 7: (MDLog::trim()+0xca9) [0x555a1b4e00c9] 2024-04-12T21:26:17.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 8: (MDLog::log_trim_upkeep()+0x358) [0x555a1b4e05a8] 2024-04-12T21:26:17.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe659887b23] 2024-04-12T21:26:17.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe65a9911ca] 2024-04-12T21:26:17.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 11: clone() 2024-04-12T21:26:17.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:17.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[67605]: 2024-04-12T21:26:17.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:17 smithi153 podman[72821]: 2024-04-12 21:26:17.538971327 +0000 UTC m=+0.018329034 container died bfc8f4961e5874992c88dbea567ca192a4975f7483d4283ce9cb8721f9e9d886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-12T21:26:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:17 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:17 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:26:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:17 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:active 2024-04-12T21:26:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:17 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:26:17.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:17 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:17.862 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:17 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:17.862 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:17 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:26:17.862 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:17 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/1712020888,v1:172.21.15.153:6835/1712020888] up:active 2024-04-12T21:26:17.862 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:17 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:26:17.862 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:17 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:18.553 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:18 smithi153 podman[72821]: 2024-04-12 21:26:18.26297464 +0000 UTC m=+0.742332346 container remove bfc8f4961e5874992c88dbea567ca192a4975f7483d4283ce9cb8721f9e9d886 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, io.buildah.version=1.33.5) 2024-04-12T21:26:18.553 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:18 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:26:18.553 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:18 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-12T21:26:18.553 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:18 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 22.720s CPU time. 2024-04-12T21:26:18.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:18 smithi104 ceph-mon[36624]: pgmap v732: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-04-12T21:26:18.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:18 smithi081 ceph-mon[31453]: pgmap v732: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-04-12T21:26:19.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:18 smithi153 ceph-mon[36569]: pgmap v732: 129 pgs: 129 active+clean; 612 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-04-12T21:26:20.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:20 smithi104 ceph-mon[36624]: pgmap v733: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 KiB/s wr, 16 op/s 2024-04-12T21:26:20.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:20 smithi081 ceph-mon[31453]: pgmap v733: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 KiB/s wr, 16 op/s 2024-04-12T21:26:21.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:20 smithi153 ceph-mon[36569]: pgmap v733: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.8 KiB/s wr, 16 op/s 2024-04-12T21:26:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:22 smithi081 ceph-mon[31453]: pgmap v734: 129 pgs: 129 active+clean; 610 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.8 KiB/s wr, 28 op/s 2024-04-12T21:26:23.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:22 smithi153 ceph-mon[36569]: pgmap v734: 129 pgs: 129 active+clean; 610 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.8 KiB/s wr, 28 op/s 2024-04-12T21:26:23.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:22 smithi104 ceph-mon[36624]: pgmap v734: 129 pgs: 129 active+clean; 610 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 2.8 KiB/s wr, 28 op/s 2024-04-12T21:26:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:23.209+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:26:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:23.209+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:26:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:23.209+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:26:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:24 smithi081 ceph-mon[31453]: pgmap v735: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 34 op/s 2024-04-12T21:26:25.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:24 smithi153 ceph-mon[36569]: pgmap v735: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 34 op/s 2024-04-12T21:26:25.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:24 smithi104 ceph-mon[36624]: pgmap v735: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 34 op/s 2024-04-12T21:26:26.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:26 smithi081 ceph-mon[31453]: pgmap v736: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 KiB/s wr, 29 op/s 2024-04-12T21:26:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:26 smithi153 ceph-mon[36569]: pgmap v736: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 KiB/s wr, 29 op/s 2024-04-12T21:26:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:26 smithi104 ceph-mon[36624]: pgmap v736: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 KiB/s wr, 29 op/s 2024-04-12T21:26:28.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:28 smithi153 ceph-mon[36569]: pgmap v737: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-12T21:26:28.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:28 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:26:28.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:28 smithi153 systemd[1]: Stopped Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:26:28.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:28 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 22.720s CPU time. 2024-04-12T21:26:28.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:28 smithi153 systemd[1]: Starting Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:26:28.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:28 smithi153 podman[72918]: 2024-04-12 21:26:28.689240218 +0000 UTC m=+0.017837407 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:26:28.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:28 smithi153 podman[72918]: 2024-04-12 21:26:28.813498699 +0000 UTC m=+0.142095893 container create 6fd7ff36a0b56880602b7a7efb3f05dd5a56c07aaa01a0b79213e969a54887c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-12T21:26:28.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:28 smithi081 ceph-mon[31453]: pgmap v737: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-12T21:26:29.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:28 smithi104 ceph-mon[36624]: pgmap v737: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 60 KiB/s wr, 31 op/s 2024-04-12T21:26:29.079 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:29 smithi153 podman[72918]: 2024-04-12 21:26:29.058505938 +0000 UTC m=+0.387103119 container init 6fd7ff36a0b56880602b7a7efb3f05dd5a56c07aaa01a0b79213e969a54887c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:26:29.079 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:29 smithi153 podman[72918]: 2024-04-12 21:26:29.061090291 +0000 UTC m=+0.389687458 container start 6fd7ff36a0b56880602b7a7efb3f05dd5a56c07aaa01a0b79213e969a54887c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True) 2024-04-12T21:26:29.082 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mds[72938]: -- 172.21.15.153:0/8082786 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5574b95bba40 con 0x5574b958e800 2024-04-12T21:26:29.348 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: starting mds.f at 2024-04-12T21:26:29.348 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:29 smithi153 bash[72918]: 6fd7ff36a0b56880602b7a7efb3f05dd5a56c07aaa01a0b79213e969a54887c4 2024-04-12T21:26:29.348 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:29 smithi153 systemd[1]: Started Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:26:29.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:26:29.100+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:26:29.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: daemon mds.f restarted 2024-04-12T21:26:29.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:26:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: osdmap e116: 12 total, 12 up, 12 in 2024-04-12T21:26:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:boot 2024-04-12T21:26:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:26:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:26:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:26:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:26:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:29 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:26:30.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: daemon mds.f restarted 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: osdmap e116: 12 total, 12 up, 12 in 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:boot 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:26:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:29 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:26:30.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: daemon mds.f restarted 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: osdmap e116: 12 total, 12 up, 12 in 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:boot 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:29 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:26:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:30 smithi081 ceph-mon[31453]: pgmap v738: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 59 KiB/s wr, 20 op/s 2024-04-12T21:26:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:30 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:reconnect 2024-04-12T21:26:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:30 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:26:30.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:30 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:26:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:30 smithi153 ceph-mon[36569]: pgmap v738: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 59 KiB/s wr, 20 op/s 2024-04-12T21:26:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:30 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:reconnect 2024-04-12T21:26:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:30 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:26:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:30 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:26:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:30 smithi104 ceph-mon[36624]: pgmap v738: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 59 KiB/s wr, 20 op/s 2024-04-12T21:26:31.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:30 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:reconnect 2024-04-12T21:26:31.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:30 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:26:31.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:30 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:26:31.899 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:26:31.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:31 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:rejoin 2024-04-12T21:26:31.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:31 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:26:31.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:31.900 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:31.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:26:31.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:31 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:rejoin 2024-04-12T21:26:31.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:31 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:26:31.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:31.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:26:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:31 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:rejoin 2024-04-12T21:26:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:31 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:26:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:32.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:32 smithi104 ceph-mon[36624]: pgmap v740: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 70 KiB/s wr, 11 op/s 2024-04-12T21:26:32.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:32.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:32.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:32 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:clientreplay 2024-04-12T21:26:32.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:32 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:26:32.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:32 smithi081 ceph-mon[31453]: pgmap v740: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 70 KiB/s wr, 11 op/s 2024-04-12T21:26:32.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:32.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:32.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:32 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:clientreplay 2024-04-12T21:26:32.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:32 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:26:33.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:32 smithi153 ceph-mon[36569]: pgmap v740: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 70 KiB/s wr, 11 op/s 2024-04-12T21:26:33.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:33.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:33.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:32 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:clientreplay 2024-04-12T21:26:33.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:32 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:26:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:33 smithi081 ceph-mon[31453]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:26:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:33 smithi153 ceph-mon[36569]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:26:34.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:33 smithi104 ceph-mon[36624]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:26:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:34 smithi081 ceph-mon[31453]: pgmap v741: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:26:34.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:34 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:34.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:34 smithi081 ceph-mon[31453]: Cluster is now healthy 2024-04-12T21:26:34.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:34 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:active 2024-04-12T21:26:34.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:34 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:26:34.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:34 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:34.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-mon[36624]: pgmap v741: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:26:34.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:34.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-mon[36624]: Cluster is now healthy 2024-04-12T21:26:34.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:active 2024-04-12T21:26:34.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:26:34.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:34.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.960 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:34.961 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:26:34.961 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: *** Caught signal (Aborted) ** 2024-04-12T21:26:34.961 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: in thread 7fb485ea4700 thread_name:ms_dispatch 2024-04-12T21:26:34.961 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:34.961 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb493a5ed20] 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2: gsignal() 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 3: abort() 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558c9ed19268] 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558c9edb50fc] 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558c9f0e9f0a] 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 7: (MDLog::trim()+0xca9) [0x558c9f0eb0c9] 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558c9f0eb5a8] 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb49294ab23] 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb493a541ca] 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 11: clone() 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2024-04-12T21:26:34.598+0000 7fb485ea4700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: in thread 7fb485ea4700 thread_name:ms_dispatch 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2024-04-12T21:26:34.962 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb493a5ed20] 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2: gsignal() 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 3: abort() 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558c9ed19268] 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558c9edb50fc] 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558c9f0e9f0a] 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 7: (MDLog::trim()+0xca9) [0x558c9f0eb0c9] 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558c9f0eb5a8] 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb49294ab23] 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb493a541ca] 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 11: clone() 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:34.963 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 0> 2024-04-12T21:26:34.598+0000 7fb485ea4700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: in thread 7fb485ea4700 thread_name:ms_dispatch 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb493a5ed20] 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2: gsignal() 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 3: abort() 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558c9ed19268] 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558c9edb50fc] 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558c9f0e9f0a] 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 7: (MDLog::trim()+0xca9) [0x558c9f0eb0c9] 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558c9f0eb5a8] 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb49294ab23] 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb493a541ca] 2024-04-12T21:26:34.964 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 11: clone() 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: -9> 2024-04-12T21:26:34.598+0000 7fb485ea4700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: in thread 7fb485ea4700 thread_name:ms_dispatch 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb493a5ed20] 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2: gsignal() 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 3: abort() 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558c9ed19268] 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558c9edb50fc] 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558c9f0e9f0a] 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 7: (MDLog::trim()+0xca9) [0x558c9f0eb0c9] 2024-04-12T21:26:34.965 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558c9f0eb5a8] 2024-04-12T21:26:34.966 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb49294ab23] 2024-04-12T21:26:34.966 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb493a541ca] 2024-04-12T21:26:34.966 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 11: clone() 2024-04-12T21:26:34.966 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:34.966 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[65604]: 2024-04-12T21:26:34.966 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:34 smithi104 podman[71513]: 2024-04-12 21:26:34.659334665 +0000 UTC m=+0.020083404 container died 32cd5d8c8a27e1ca53f92e18dd361ebbac8f21362175950333ed00a9aac7e4b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS) 2024-04-12T21:26:35.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:34 smithi153 ceph-mon[36569]: pgmap v741: 129 pgs: 129 active+clean; 611 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:26:35.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:34 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:35.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:34 smithi153 ceph-mon[36569]: Cluster is now healthy 2024-04-12T21:26:35.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:34 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3576497721,v1:172.21.15.104:6835/3576497721] up:active 2024-04-12T21:26:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:34 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:26:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:34 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:35.829 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:35 smithi104 podman[71513]: 2024-04-12 21:26:35.389533921 +0000 UTC m=+0.750282688 container remove 32cd5d8c8a27e1ca53f92e18dd361ebbac8f21362175950333ed00a9aac7e4b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-12T21:26:35.829 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:35 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:26:35.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:35 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-12T21:26:35.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:35 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 23.130s CPU time. 2024-04-12T21:26:36.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:36 smithi081 ceph-mon[31453]: pgmap v742: 129 pgs: 129 active+clean; 610 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:26:36.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:26:36.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:26:36.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:37.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:36 smithi153 ceph-mon[36569]: pgmap v742: 129 pgs: 129 active+clean; 610 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:26:37.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:26:37.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:26:37.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:37.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:36 smithi104 ceph-mon[36624]: pgmap v742: 129 pgs: 129 active+clean; 610 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:26:37.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:26:37.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:26:37.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:26:37.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:37 smithi081 ceph-mon[31453]: pgmap v743: 129 pgs: 129 active+clean; 609 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 29 KiB/s wr, 26 op/s 2024-04-12T21:26:37.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:37 smithi081 ceph-mon[31453]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:26:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:37 smithi153 ceph-mon[36569]: pgmap v743: 129 pgs: 129 active+clean; 609 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 29 KiB/s wr, 26 op/s 2024-04-12T21:26:38.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:37 smithi153 ceph-mon[36569]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:26:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:37 smithi104 ceph-mon[36624]: pgmap v743: 129 pgs: 129 active+clean; 609 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 29 KiB/s wr, 26 op/s 2024-04-12T21:26:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:37 smithi104 ceph-mon[36624]: Health check failed: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:26:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:38.220+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:26:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:38.220+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:26:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:38.220+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:26:40.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:39 smithi153 ceph-mon[36569]: pgmap v744: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 113 KiB/s wr, 50 op/s 2024-04-12T21:26:40.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:39 smithi104 ceph-mon[36624]: pgmap v744: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 113 KiB/s wr, 50 op/s 2024-04-12T21:26:40.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:39 smithi081 ceph-mon[31453]: pgmap v744: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 113 KiB/s wr, 50 op/s 2024-04-12T21:26:42.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:41 smithi153 ceph-mon[36569]: pgmap v745: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 96 KiB/s wr, 43 op/s 2024-04-12T21:26:42.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:41 smithi104 ceph-mon[36624]: pgmap v745: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 96 KiB/s wr, 43 op/s 2024-04-12T21:26:42.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:41 smithi081 ceph-mon[31453]: pgmap v745: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 96 KiB/s wr, 43 op/s 2024-04-12T21:26:43.760 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:43 smithi081 ceph-mon[31453]: pgmap v746: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-12T21:26:44.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:43 smithi153 ceph-mon[36569]: pgmap v746: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-12T21:26:44.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:43 smithi104 ceph-mon[36624]: pgmap v746: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 93 KiB/s wr, 43 op/s 2024-04-12T21:26:46.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:45 smithi153 ceph-mon[36569]: pgmap v747: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 93 KiB/s wr, 42 op/s 2024-04-12T21:26:46.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:45 smithi104 ceph-mon[36624]: pgmap v747: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 93 KiB/s wr, 42 op/s 2024-04-12T21:26:46.061 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:45 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:26:46.062 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:45 smithi104 systemd[1]: Stopped Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:26:46.062 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:45 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 23.130s CPU time. 2024-04-12T21:26:46.062 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:45 smithi104 systemd[1]: Starting Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:26:46.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:45 smithi081 ceph-mon[31453]: pgmap v747: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 93 KiB/s wr, 42 op/s 2024-04-12T21:26:46.313 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:46 smithi104 podman[71639]: 2024-04-12 21:26:45.968417942 +0000 UTC m=+0.018144348 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:26:46.313 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:46 smithi104 podman[71639]: 2024-04-12 21:26:46.089368472 +0000 UTC m=+0.139094899 container create bbcc8963f573b24920a459e335908848f851fea98618bb31fc3c3a1993d431f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-12T21:26:46.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:46 smithi104 podman[71639]: 2024-04-12 21:26:46.316919175 +0000 UTC m=+0.366645571 container init bbcc8963f573b24920a459e335908848f851fea98618bb31fc3c3a1993d431f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:26:46.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:46 smithi104 podman[71639]: 2024-04-12 21:26:46.319991633 +0000 UTC m=+0.369718026 container start bbcc8963f573b24920a459e335908848f851fea98618bb31fc3c3a1993d431f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-04-12T21:26:46.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mds[71658]: -- 172.21.15.104:0/1875863439 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a98db47a40 con 0x55a98db1a800 2024-04-12T21:26:46.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: starting mds.e at 2024-04-12T21:26:46.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:46 smithi104 bash[71639]: bbcc8963f573b24920a459e335908848f851fea98618bb31fc3c3a1993d431f2 2024-04-12T21:26:46.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:26:46 smithi104 systemd[1]: Started Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:26:46.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:26:46.359+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:26:47.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: daemon mds.e restarted 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: osdmap e117: 12 total, 12 up, 12 in 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:boot 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:26:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:46 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: daemon mds.e restarted 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: osdmap e117: 12 total, 12 up, 12 in 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:boot 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:26:47.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:46 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:26:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: daemon mds.e restarted 2024-04-12T21:26:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:26:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:26:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: osdmap e117: 12 total, 12 up, 12 in 2024-04-12T21:26:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:boot 2024-04-12T21:26:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:26:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:26:47.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:26:47.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:26:47.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:46 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:26:47.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:47 smithi104 ceph-mon[36624]: pgmap v748: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 93 KiB/s wr, 38 op/s 2024-04-12T21:26:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:47 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:reconnect 2024-04-12T21:26:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:47 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:26:48.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:47 smithi153 ceph-mon[36569]: pgmap v748: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 93 KiB/s wr, 38 op/s 2024-04-12T21:26:48.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:47 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:reconnect 2024-04-12T21:26:48.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:47 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:26:48.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:47 smithi081 ceph-mon[31453]: pgmap v748: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 93 KiB/s wr, 38 op/s 2024-04-12T21:26:48.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:47 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:reconnect 2024-04-12T21:26:48.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:47 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:26:49.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:48 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0219996 2024-04-12T21:26:49.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:48 smithi153 ceph-mon[36569]: pgmap v750: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:26:49.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:48 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:rejoin 2024-04-12T21:26:49.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:48 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:26:49.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:48 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0219996 2024-04-12T21:26:49.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:48 smithi104 ceph-mon[36624]: pgmap v750: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:26:49.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:48 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:rejoin 2024-04-12T21:26:49.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:48 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:26:49.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:48 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0219996 2024-04-12T21:26:49.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:48 smithi081 ceph-mon[31453]: pgmap v750: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:26:49.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:48 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:rejoin 2024-04-12T21:26:49.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:48 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:26:51.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:clientreplay 2024-04-12T21:26:51.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:26:51.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-mon[36569]: pgmap v751: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:26:51.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-mon[36569]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:26:51.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:26:51.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: *** Caught signal (Aborted) ** 2024-04-12T21:26:51.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: in thread 7ff03736d700 thread_name:ms_dispatch 2024-04-12T21:26:51.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff044f27d20] 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2: gsignal() 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 3: abort() 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5574b7c55268] 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5574b7cf10fc] 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5574b8025f0a] 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 7: (MDLog::trim()+0xca9) [0x5574b80270c9] 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5574b80275a8] 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff043e13b23] 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff044f1d1ca] 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 11: clone() 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2024-04-12T21:26:50.811+0000 7ff03736d700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:51.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: in thread 7ff03736d700 thread_name:ms_dispatch 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff044f27d20] 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2: gsignal() 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 3: abort() 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5574b7c55268] 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5574b7cf10fc] 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5574b8025f0a] 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 7: (MDLog::trim()+0xca9) [0x5574b80270c9] 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5574b80275a8] 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff043e13b23] 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff044f1d1ca] 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 11: clone() 2024-04-12T21:26:51.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 0> 2024-04-12T21:26:50.811+0000 7ff03736d700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: in thread 7ff03736d700 thread_name:ms_dispatch 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff044f27d20] 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2: gsignal() 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 3: abort() 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5574b7c55268] 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5574b7cf10fc] 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5574b8025f0a] 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 7: (MDLog::trim()+0xca9) [0x5574b80270c9] 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5574b80275a8] 2024-04-12T21:26:51.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff043e13b23] 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff044f1d1ca] 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 11: clone() 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: -10> 2024-04-12T21:26:50.811+0000 7ff03736d700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: in thread 7ff03736d700 thread_name:ms_dispatch 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff044f27d20] 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2: gsignal() 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 3: abort() 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5574b7c55268] 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5574b7cf10fc] 2024-04-12T21:26:51.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5574b8025f0a] 2024-04-12T21:26:51.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 7: (MDLog::trim()+0xca9) [0x5574b80270c9] 2024-04-12T21:26:51.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5574b80275a8] 2024-04-12T21:26:51.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff043e13b23] 2024-04-12T21:26:51.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff044f1d1ca] 2024-04-12T21:26:51.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 11: clone() 2024-04-12T21:26:51.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:26:51.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[72934]: 2024-04-12T21:26:51.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:50 smithi153 podman[73485]: 2024-04-12 21:26:50.872954405 +0000 UTC m=+0.020387204 container died 6fd7ff36a0b56880602b7a7efb3f05dd5a56c07aaa01a0b79213e969a54887c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-12T21:26:51.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:50 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:clientreplay 2024-04-12T21:26:51.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:50 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:26:51.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:50 smithi104 ceph-mon[36624]: pgmap v751: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:26:51.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:50 smithi104 ceph-mon[36624]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:26:51.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:50 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:clientreplay 2024-04-12T21:26:51.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:50 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:26:51.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:50 smithi081 ceph-mon[31453]: pgmap v751: 129 pgs: 129 active+clean; 608 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:26:51.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:50 smithi081 ceph-mon[31453]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:26:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:51 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:51 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:active 2024-04-12T21:26:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:51 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:26:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:51 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:52.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:51 smithi153 podman[73485]: 2024-04-12 21:26:51.568330815 +0000 UTC m=+0.715763595 container remove 6fd7ff36a0b56880602b7a7efb3f05dd5a56c07aaa01a0b79213e969a54887c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-12T21:26:52.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:51 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:26:52.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:51 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-12T21:26:52.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:26:51 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 22.392s CPU time. 2024-04-12T21:26:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:51 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:51 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:active 2024-04-12T21:26:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:51 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:26:52.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:51 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:51 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:26:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:51 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/3454340977,v1:172.21.15.153:6835/3454340977] up:active 2024-04-12T21:26:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:51 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:26:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:51 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:26:53.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:52 smithi153 ceph-mon[36569]: pgmap v752: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 93 KiB/s wr, 22 op/s 2024-04-12T21:26:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:52 smithi104 ceph-mon[36624]: pgmap v752: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 93 KiB/s wr, 22 op/s 2024-04-12T21:26:53.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:52 smithi081 ceph-mon[31453]: pgmap v752: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 93 KiB/s wr, 22 op/s 2024-04-12T21:26:53.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:53.234+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:26:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:53.235+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:26:53.232 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:26:53.235+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:26:55.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:54 smithi081 ceph-mon[31453]: pgmap v753: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 93 KiB/s wr, 26 op/s 2024-04-12T21:26:55.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:54 smithi153 ceph-mon[36569]: pgmap v753: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 93 KiB/s wr, 26 op/s 2024-04-12T21:26:55.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:54 smithi104 ceph-mon[36624]: pgmap v753: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 93 KiB/s wr, 26 op/s 2024-04-12T21:26:57.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:56 smithi153 ceph-mon[36569]: pgmap v754: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 96 KiB/s wr, 36 op/s 2024-04-12T21:26:57.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:56 smithi104 ceph-mon[36624]: pgmap v754: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 96 KiB/s wr, 36 op/s 2024-04-12T21:26:57.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:56 smithi081 ceph-mon[31453]: pgmap v754: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 96 KiB/s wr, 36 op/s 2024-04-12T21:26:59.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:26:58 smithi081 ceph-mon[31453]: pgmap v755: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 84 KiB/s wr, 32 op/s 2024-04-12T21:26:59.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:26:58 smithi153 ceph-mon[36569]: pgmap v755: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 84 KiB/s wr, 32 op/s 2024-04-12T21:26:59.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:26:58 smithi104 ceph-mon[36624]: pgmap v755: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 84 KiB/s wr, 32 op/s 2024-04-12T21:27:01.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:00 smithi153 ceph-mon[36569]: pgmap v756: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 80 KiB/s wr, 31 op/s 2024-04-12T21:27:01.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:00 smithi104 ceph-mon[36624]: pgmap v756: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 80 KiB/s wr, 31 op/s 2024-04-12T21:27:01.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:00 smithi081 ceph-mon[31453]: pgmap v756: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 80 KiB/s wr, 31 op/s 2024-04-12T21:27:02.173 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:01 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:27:02.174 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:01 smithi153 systemd[1]: Stopped Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:27:02.174 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:01 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 22.392s CPU time. 2024-04-12T21:27:02.174 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:01 smithi153 systemd[1]: Starting Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:27:02.174 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:02 smithi153 podman[73578]: 2024-04-12 21:27:01.941875946 +0000 UTC m=+0.018473329 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:27:02.174 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:02 smithi153 podman[73578]: 2024-04-12 21:27:02.052414517 +0000 UTC m=+0.129011910 container create 5e309fad80b7aba970e8133ab017c8af6657703c8c35e39e7aaa96fa8b754d52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS) 2024-04-12T21:27:02.435 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:02 smithi153 podman[73578]: 2024-04-12 21:27:02.29694152 +0000 UTC m=+0.373538907 container init 5e309fad80b7aba970e8133ab017c8af6657703c8c35e39e7aaa96fa8b754d52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:27:02.435 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:02 smithi153 podman[73578]: 2024-04-12 21:27:02.299899671 +0000 UTC m=+0.376497046 container start 5e309fad80b7aba970e8133ab017c8af6657703c8c35e39e7aaa96fa8b754d52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-04-12T21:27:02.435 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mds[73598]: -- 172.21.15.153:0/3625940644 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5625d1249a40 con 0x5625d121c800 2024-04-12T21:27:02.435 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: starting mds.f at 2024-04-12T21:27:02.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:27:02.339+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:02.804 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:02 smithi153 bash[73578]: 5e309fad80b7aba970e8133ab017c8af6657703c8c35e39e7aaa96fa8b754d52 2024-04-12T21:27:02.804 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:02 smithi153 systemd[1]: Started Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:27:03.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: pgmap v757: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 80 KiB/s wr, 33 op/s 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: daemon mds.f restarted 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: osdmap e118: 12 total, 12 up, 12 in 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:boot 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:27:03.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:02 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:27:03.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: pgmap v757: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 80 KiB/s wr, 33 op/s 2024-04-12T21:27:03.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: daemon mds.f restarted 2024-04-12T21:27:03.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:03.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:03.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: osdmap e118: 12 total, 12 up, 12 in 2024-04-12T21:27:03.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:boot 2024-04-12T21:27:03.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:27:03.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:03.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:03.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:27:03.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:02 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: pgmap v757: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 80 KiB/s wr, 33 op/s 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: daemon mds.f restarted 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: osdmap e118: 12 total, 12 up, 12 in 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:boot 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:27:03.367 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:02 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:27:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:04 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:reconnect 2024-04-12T21:27:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:04 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:27:04.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:04 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109999 2024-04-12T21:27:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:04 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:reconnect 2024-04-12T21:27:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:04 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:27:04.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:04 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109999 2024-04-12T21:27:04.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:04 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:reconnect 2024-04-12T21:27:04.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:04 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:27:04.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:04 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109999 2024-04-12T21:27:05.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:05 smithi081 ceph-mon[31453]: pgmap v759: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-04-12T21:27:05.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:05 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:rejoin 2024-04-12T21:27:05.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:05 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:27:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:05 smithi153 ceph-mon[36569]: pgmap v759: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-04-12T21:27:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:05 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:rejoin 2024-04-12T21:27:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:05 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:27:05.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:05 smithi104 ceph-mon[36624]: pgmap v759: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 2.6 KiB/s wr, 16 op/s 2024-04-12T21:27:05.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:05 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:rejoin 2024-04-12T21:27:05.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:05 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:27:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:06 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:clientreplay 2024-04-12T21:27:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:06 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:27:06.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:06 smithi081 ceph-mon[31453]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:27:06.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:clientreplay 2024-04-12T21:27:06.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:27:06.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-mon[36624]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:27:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:06 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:clientreplay 2024-04-12T21:27:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:06 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:27:06.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:06 smithi153 ceph-mon[36569]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:27:07.079 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: *** Caught signal (Aborted) ** 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: in thread 7fb9109b9700 thread_name:ms_dispatch 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb91e573d20] 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2: gsignal() 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 3: abort() 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a98b43c268] 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a98b4d80fc] 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a98b80cf0a] 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 7: (MDLog::trim()+0xca9) [0x55a98b80e0c9] 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a98b80e5a8] 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb91d45fb23] 2024-04-12T21:27:07.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb91e5691ca] 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 11: clone() 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2024-04-12T21:27:06.736+0000 7fb9109b9700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: in thread 7fb9109b9700 thread_name:ms_dispatch 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb91e573d20] 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2: gsignal() 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 3: abort() 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a98b43c268] 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a98b4d80fc] 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a98b80cf0a] 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 7: (MDLog::trim()+0xca9) [0x55a98b80e0c9] 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a98b80e5a8] 2024-04-12T21:27:07.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb91d45fb23] 2024-04-12T21:27:07.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb91e5691ca] 2024-04-12T21:27:07.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 11: clone() 2024-04-12T21:27:07.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:07.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2024-04-12T21:27:07.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 0> 2024-04-12T21:27:06.736+0000 7fb9109b9700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: in thread 7fb9109b9700 thread_name:ms_dispatch 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb91e573d20] 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2: gsignal() 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 3: abort() 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a98b43c268] 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a98b4d80fc] 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a98b80cf0a] 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 7: (MDLog::trim()+0xca9) [0x55a98b80e0c9] 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a98b80e5a8] 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb91d45fb23] 2024-04-12T21:27:07.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb91e5691ca] 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 11: clone() 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: -9> 2024-04-12T21:27:06.736+0000 7fb9109b9700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: in thread 7fb9109b9700 thread_name:ms_dispatch 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb91e573d20] 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2: gsignal() 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 3: abort() 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a98b43c268] 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a98b4d80fc] 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a98b80cf0a] 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 7: (MDLog::trim()+0xca9) [0x55a98b80e0c9] 2024-04-12T21:27:07.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a98b80e5a8] 2024-04-12T21:27:07.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb91d45fb23] 2024-04-12T21:27:07.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb91e5691ca] 2024-04-12T21:27:07.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 11: clone() 2024-04-12T21:27:07.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:07.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71654]: 2024-04-12T21:27:07.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:06 smithi104 podman[71708]: 2024-04-12 21:27:06.797336549 +0000 UTC m=+0.018504529 container died bbcc8963f573b24920a459e335908848f851fea98618bb31fc3c3a1993d431f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:27:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:07 smithi081 ceph-mon[31453]: pgmap v760: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:27:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:07 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:active 2024-04-12T21:27:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:07 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:27:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:07 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:07.733 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:07 smithi104 podman[71708]: 2024-04-12 21:27:07.512098963 +0000 UTC m=+0.733266943 container remove bbcc8963f573b24920a459e335908848f851fea98618bb31fc3c3a1993d431f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-12T21:27:07.734 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:07 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:27:07.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:07 smithi104 ceph-mon[36624]: pgmap v760: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:27:07.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:07 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:07.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:active 2024-04-12T21:27:07.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:07 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:27:07.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:07 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:07 smithi153 ceph-mon[36569]: pgmap v760: 129 pgs: 129 active+clean; 605 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:27:07.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:07 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:07.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3524028290,v1:172.21.15.104:6835/3524028290] up:active 2024-04-12T21:27:07.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:07 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:27:07.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:07 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:08.079 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:07 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-12T21:27:08.079 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:07 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 21.725s CPU time. 2024-04-12T21:27:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:08.249+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:27:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:08.249+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:27:08.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:08.249+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:27:09.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:09 smithi081 ceph-mon[31453]: pgmap v761: 129 pgs: 129 active+clean; 603 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.8 KiB/s wr, 21 op/s 2024-04-12T21:27:09.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:09 smithi153 ceph-mon[36569]: pgmap v761: 129 pgs: 129 active+clean; 603 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.8 KiB/s wr, 21 op/s 2024-04-12T21:27:09.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:09 smithi104 ceph-mon[36624]: pgmap v761: 129 pgs: 129 active+clean; 603 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.8 KiB/s wr, 21 op/s 2024-04-12T21:27:11.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:11 smithi153 ceph-mon[36569]: pgmap v762: 129 pgs: 129 active+clean; 602 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.0 KiB/s wr, 28 op/s 2024-04-12T21:27:11.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:11 smithi104 ceph-mon[36624]: pgmap v762: 129 pgs: 129 active+clean; 602 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.0 KiB/s wr, 28 op/s 2024-04-12T21:27:11.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:11 smithi081 ceph-mon[31453]: pgmap v762: 129 pgs: 129 active+clean; 602 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 3.0 KiB/s wr, 28 op/s 2024-04-12T21:27:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:13 smithi153 ceph-mon[36569]: pgmap v763: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 44 KiB/s wr, 44 op/s 2024-04-12T21:27:13.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:13 smithi104 ceph-mon[36624]: pgmap v763: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 44 KiB/s wr, 44 op/s 2024-04-12T21:27:13.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:13 smithi081 ceph-mon[31453]: pgmap v763: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 44 KiB/s wr, 44 op/s 2024-04-12T21:27:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:15 smithi153 ceph-mon[36569]: pgmap v764: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 38 KiB/s wr, 39 op/s 2024-04-12T21:27:15.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:15 smithi104 ceph-mon[36624]: pgmap v764: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 38 KiB/s wr, 39 op/s 2024-04-12T21:27:15.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:15 smithi081 ceph-mon[31453]: pgmap v764: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 38 KiB/s wr, 39 op/s 2024-04-12T21:27:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:17 smithi153 ceph-mon[36569]: pgmap v765: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 37 KiB/s wr, 39 op/s 2024-04-12T21:27:17.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:17 smithi104 ceph-mon[36624]: pgmap v765: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 37 KiB/s wr, 39 op/s 2024-04-12T21:27:17.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:17 smithi081 ceph-mon[31453]: pgmap v765: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 37 KiB/s wr, 39 op/s 2024-04-12T21:27:18.192 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:17 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:27:18.192 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:17 smithi104 systemd[1]: Stopped Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:27:18.192 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:17 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 21.725s CPU time. 2024-04-12T21:27:18.192 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:17 smithi104 systemd[1]: Starting Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:27:18.192 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:18 smithi104 podman[71800]: 2024-04-12 21:27:17.96333896 +0000 UTC m=+0.017842573 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:27:18.192 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:18 smithi104 podman[71800]: 2024-04-12 21:27:18.195704943 +0000 UTC m=+0.250208541 container create b66e15b8a79ea525c8c24c68e32c42a13e07b54eb9fbb89766e5904a6c61a165 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-12T21:27:18.444 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:18 smithi104 podman[71800]: 2024-04-12 21:27:18.423217243 +0000 UTC m=+0.477720849 container init b66e15b8a79ea525c8c24c68e32c42a13e07b54eb9fbb89766e5904a6c61a165 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T21:27:18.445 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:18 smithi104 podman[71800]: 2024-04-12 21:27:18.425824955 +0000 UTC m=+0.480328567 container start b66e15b8a79ea525c8c24c68e32c42a13e07b54eb9fbb89766e5904a6c61a165 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-12T21:27:18.829 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:18 smithi104 ceph-mds[71819]: -- 172.21.15.104:0/429632220 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ad42aada40 con 0x55ad42a80c00 2024-04-12T21:27:18.829 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:18 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: starting mds.e at 2024-04-12T21:27:18.829 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:18 smithi104 bash[71800]: b66e15b8a79ea525c8c24c68e32c42a13e07b54eb9fbb89766e5904a6c61a165 2024-04-12T21:27:18.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:18 smithi104 systemd[1]: Started Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:27:18.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:18 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:27:18.467+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:19.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: pgmap v766: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 37 KiB/s wr, 36 op/s 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: daemon mds.e restarted 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: osdmap e119: 12 total, 12 up, 12 in 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:boot 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:19 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: pgmap v766: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 37 KiB/s wr, 36 op/s 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: daemon mds.e restarted 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: osdmap e119: 12 total, 12 up, 12 in 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:boot 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:27:19.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:19 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:27:19.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: pgmap v766: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 37 KiB/s wr, 36 op/s 2024-04-12T21:27:19.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: daemon mds.e restarted 2024-04-12T21:27:19.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:19.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:19.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: osdmap e119: 12 total, 12 up, 12 in 2024-04-12T21:27:19.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:boot 2024-04-12T21:27:19.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:27:19.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:19.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:19.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:27:19.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:19 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:27:20.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:20 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:reconnect 2024-04-12T21:27:20.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:20 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:27:20.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:20 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:27:20.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:20 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:reconnect 2024-04-12T21:27:20.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:20 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:27:20.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:20 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:27:20.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:20 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:reconnect 2024-04-12T21:27:20.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:20 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:27:20.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:20 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:27:21.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:21 smithi153 ceph-mon[36569]: pgmap v768: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 41 KiB/s wr, 22 op/s 2024-04-12T21:27:21.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:21 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:rejoin 2024-04-12T21:27:21.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:21 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:27:21.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:21 smithi104 ceph-mon[36624]: pgmap v768: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 41 KiB/s wr, 22 op/s 2024-04-12T21:27:21.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:21 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:rejoin 2024-04-12T21:27:21.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:21 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:27:21.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:21 smithi081 ceph-mon[31453]: pgmap v768: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 41 KiB/s wr, 22 op/s 2024-04-12T21:27:21.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:21 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:rejoin 2024-04-12T21:27:21.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:21 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:27:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:22 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:clientreplay 2024-04-12T21:27:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:22 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:27:22.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:22 smithi153 ceph-mon[36569]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:27:22.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:22 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:22.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:22 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:active 2024-04-12T21:27:22.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:22 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:22 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:clientreplay 2024-04-12T21:27:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:22 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:27:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:22 smithi104 ceph-mon[36624]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:27:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:22 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:22 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:active 2024-04-12T21:27:22.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:22 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:22 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:clientreplay 2024-04-12T21:27:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:22 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:27:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:22 smithi081 ceph-mon[31453]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:27:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:22 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:22 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2797892682,v1:172.21.15.153:6835/2797892682] up:active 2024-04-12T21:27:22.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:22 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:23.264+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:27:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:23.264+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:27:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:23.264+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:27:23.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-mon[36569]: pgmap v769: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:27:23.792 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:27:23.792 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: *** Caught signal (Aborted) ** 2024-04-12T21:27:23.792 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: in thread 7f0270a5a700 thread_name:ms_dispatch 2024-04-12T21:27:23.792 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:23.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f027e614d20] 2024-04-12T21:27:23.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2: gsignal() 2024-04-12T21:27:23.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 3: abort() 2024-04-12T21:27:23.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5625ce3ca268] 2024-04-12T21:27:23.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5625ce4660fc] 2024-04-12T21:27:23.793 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5625ce79af0a] 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 7: (MDLog::trim()+0xca9) [0x5625ce79c0c9] 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5625ce79c5a8] 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f027d500b23] 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f027e60a1ca] 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 11: clone() 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2024-04-12T21:27:23.411+0000 7f0270a5a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: in thread 7f0270a5a700 thread_name:ms_dispatch 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f027e614d20] 2024-04-12T21:27:23.794 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2: gsignal() 2024-04-12T21:27:23.795 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 3: abort() 2024-04-12T21:27:23.795 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5625ce3ca268] 2024-04-12T21:27:23.795 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5625ce4660fc] 2024-04-12T21:27:23.795 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5625ce79af0a] 2024-04-12T21:27:23.795 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 7: (MDLog::trim()+0xca9) [0x5625ce79c0c9] 2024-04-12T21:27:23.795 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5625ce79c5a8] 2024-04-12T21:27:23.795 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f027d500b23] 2024-04-12T21:27:23.795 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f027e60a1ca] 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 11: clone() 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 0> 2024-04-12T21:27:23.411+0000 7f0270a5a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: in thread 7f0270a5a700 thread_name:ms_dispatch 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f027e614d20] 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2: gsignal() 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 3: abort() 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5625ce3ca268] 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5625ce4660fc] 2024-04-12T21:27:23.796 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5625ce79af0a] 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 7: (MDLog::trim()+0xca9) [0x5625ce79c0c9] 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5625ce79c5a8] 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f027d500b23] 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f027e60a1ca] 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 11: clone() 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: -9> 2024-04-12T21:27:23.411+0000 7f0270a5a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: in thread 7f0270a5a700 thread_name:ms_dispatch 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f027e614d20] 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2: gsignal() 2024-04-12T21:27:23.797 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 3: abort() 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5625ce3ca268] 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5625ce4660fc] 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5625ce79af0a] 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 7: (MDLog::trim()+0xca9) [0x5625ce79c0c9] 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5625ce79c5a8] 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f027d500b23] 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f027e60a1ca] 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 11: clone() 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73593]: 2024-04-12T21:27:23.798 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:23 smithi153 podman[73652]: 2024-04-12 21:27:23.473532858 +0000 UTC m=+0.019134892 container died 5e309fad80b7aba970e8133ab017c8af6657703c8c35e39e7aaa96fa8b754d52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:27:23.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:23 smithi104 ceph-mon[36624]: pgmap v769: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:27:23.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:23 smithi081 ceph-mon[31453]: pgmap v769: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:27:24.553 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:24 smithi153 podman[73652]: 2024-04-12 21:27:24.076657633 +0000 UTC m=+0.622259684 container remove 5e309fad80b7aba970e8133ab017c8af6657703c8c35e39e7aaa96fa8b754d52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5) 2024-04-12T21:27:24.554 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:24 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:27:24.554 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:24 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-12T21:27:24.554 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:24 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 23.038s CPU time. 2024-04-12T21:27:25.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:25 smithi153 ceph-mon[36569]: pgmap v770: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 204 B/s wr, 6 op/s 2024-04-12T21:27:25.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:25 smithi104 ceph-mon[36624]: pgmap v770: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 204 B/s wr, 6 op/s 2024-04-12T21:27:25.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:25 smithi081 ceph-mon[31453]: pgmap v770: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 204 B/s wr, 6 op/s 2024-04-12T21:27:27.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:27 smithi153 ceph-mon[36569]: pgmap v771: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 73 KiB/s wr, 20 op/s 2024-04-12T21:27:27.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:27 smithi104 ceph-mon[36624]: pgmap v771: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 73 KiB/s wr, 20 op/s 2024-04-12T21:27:27.878 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:27 smithi081 ceph-mon[31453]: pgmap v771: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 73 KiB/s wr, 20 op/s 2024-04-12T21:27:29.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:29 smithi153 ceph-mon[36569]: pgmap v772: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 73 KiB/s wr, 25 op/s 2024-04-12T21:27:29.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:29 smithi104 ceph-mon[36624]: pgmap v772: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 73 KiB/s wr, 25 op/s 2024-04-12T21:27:29.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:29 smithi081 ceph-mon[31453]: pgmap v772: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 73 KiB/s wr, 25 op/s 2024-04-12T21:27:31.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:31 smithi153 ceph-mon[36569]: pgmap v773: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 64 KiB/s wr, 23 op/s 2024-04-12T21:27:31.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:31 smithi104 ceph-mon[36624]: pgmap v773: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 64 KiB/s wr, 23 op/s 2024-04-12T21:27:31.846 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:31 smithi081 ceph-mon[31453]: pgmap v773: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 64 KiB/s wr, 23 op/s 2024-04-12T21:27:33.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:33 smithi153 ceph-mon[36569]: pgmap v774: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 23 op/s 2024-04-12T21:27:33.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:33 smithi104 ceph-mon[36624]: pgmap v774: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 23 op/s 2024-04-12T21:27:33.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:33 smithi081 ceph-mon[31453]: pgmap v774: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 23 op/s 2024-04-12T21:27:34.840 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:34 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:27:34.840 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:34 smithi153 systemd[1]: Stopped Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:27:34.840 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:34 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 23.038s CPU time. 2024-04-12T21:27:34.841 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:34 smithi153 systemd[1]: Starting Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:27:34.841 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:34 smithi153 podman[73749]: 2024-04-12 21:27:34.702899562 +0000 UTC m=+0.018253607 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:27:34.841 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:34 smithi153 podman[73749]: 2024-04-12 21:27:34.844154015 +0000 UTC m=+0.159508046 container create 65daf3132e6fc4b3635ce798cd8484744c5d5cd58d64045f57492467d0fbf068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux ) 2024-04-12T21:27:35.177 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:35 smithi153 podman[73749]: 2024-04-12 21:27:35.055163613 +0000 UTC m=+0.370517651 container init 65daf3132e6fc4b3635ce798cd8484744c5d5cd58d64045f57492467d0fbf068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-12T21:27:35.177 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:35 smithi153 podman[73749]: 2024-04-12 21:27:35.057646622 +0000 UTC m=+0.373000661 container start 65daf3132e6fc4b3635ce798cd8484744c5d5cd58d64045f57492467d0fbf068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-12T21:27:35.177 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mds[73768]: -- 172.21.15.153:0/727815573 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c52746fa40 con 0x55c527442800 2024-04-12T21:27:35.177 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: starting mds.f at 2024-04-12T21:27:35.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:27:35.095+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:35.553 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:35 smithi153 bash[73749]: 65daf3132e6fc4b3635ce798cd8484744c5d5cd58d64045f57492467d0fbf068 2024-04-12T21:27:35.553 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:35 smithi153 systemd[1]: Started Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:27:35.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: pgmap v775: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 61 KiB/s wr, 22 op/s 2024-04-12T21:27:35.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: daemon mds.f restarted 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: osdmap e120: 12 total, 12 up, 12 in 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:boot 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:35 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:27:35.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: pgmap v775: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 61 KiB/s wr, 22 op/s 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: daemon mds.f restarted 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: osdmap e120: 12 total, 12 up, 12 in 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:boot 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:35 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: pgmap v775: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 61 KiB/s wr, 22 op/s 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: daemon mds.f restarted 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: osdmap e120: 12 total, 12 up, 12 in 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:boot 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:27:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:35 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:27:37.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:37 smithi081 ceph-mon[31453]: pgmap v777: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 921 B/s wr, 11 op/s 2024-04-12T21:27:37.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:27:37.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:37 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:reconnect 2024-04-12T21:27:37.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:37 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:27:37.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:37 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199998 2024-04-12T21:27:37.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:37.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:37.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:37 smithi153 ceph-mon[36569]: pgmap v777: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 921 B/s wr, 11 op/s 2024-04-12T21:27:37.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:27:37.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:37 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:reconnect 2024-04-12T21:27:37.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:37 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:27:37.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:37 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199998 2024-04-12T21:27:37.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:37.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:37.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:37 smithi104 ceph-mon[36624]: pgmap v777: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 921 B/s wr, 11 op/s 2024-04-12T21:27:37.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:27:37.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:37 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:reconnect 2024-04-12T21:27:37.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:37 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:27:37.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:37 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0199998 2024-04-12T21:27:37.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:37.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:38.279+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:27:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:38.279+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:27:38.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:38.279+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:27:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:38 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:rejoin 2024-04-12T21:27:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:38 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:27:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:27:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:27:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:38 smithi081 ceph-mon[31453]: pgmap v778: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:27:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:38 smithi081 ceph-mon[31453]: pgmap v779: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:27:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:38.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:38 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:rejoin 2024-04-12T21:27:38.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:38 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:27:38.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:27:38.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:27:38.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:38 smithi153 ceph-mon[36569]: pgmap v778: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:27:38.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:38 smithi153 ceph-mon[36569]: pgmap v779: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:27:38.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:38.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:38 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:rejoin 2024-04-12T21:27:38.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:38 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:27:38.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:27:38.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:27:38.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:38 smithi104 ceph-mon[36624]: pgmap v778: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:27:38.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:38 smithi104 ceph-mon[36624]: pgmap v779: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:27:38.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:27:39.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:39 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:clientreplay 2024-04-12T21:27:39.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:39 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:27:39.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:39 smithi153 ceph-mon[36569]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:27:39.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:clientreplay 2024-04-12T21:27:39.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:27:39.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-mon[36624]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:27:39.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:27:39.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: *** Caught signal (Aborted) ** 2024-04-12T21:27:39.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: in thread 7f4712707700 thread_name:ms_dispatch 2024-04-12T21:27:39.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f47202c1d20] 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2: gsignal() 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 3: abort() 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55ad405dd268] 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55ad406790fc] 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55ad409adf0a] 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 7: (MDLog::trim()+0xca9) [0x55ad409af0c9] 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55ad409af5a8] 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f471f1adb23] 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f47202b71ca] 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 11: clone() 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2024-04-12T21:27:39.531+0000 7f4712707700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:39.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: in thread 7f4712707700 thread_name:ms_dispatch 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f47202c1d20] 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2: gsignal() 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 3: abort() 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55ad405dd268] 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55ad406790fc] 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55ad409adf0a] 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 7: (MDLog::trim()+0xca9) [0x55ad409af0c9] 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55ad409af5a8] 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f471f1adb23] 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f47202b71ca] 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 11: clone() 2024-04-12T21:27:39.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 0> 2024-04-12T21:27:39.531+0000 7f4712707700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: in thread 7f4712707700 thread_name:ms_dispatch 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f47202c1d20] 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2: gsignal() 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 3: abort() 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55ad405dd268] 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55ad406790fc] 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55ad409adf0a] 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 7: (MDLog::trim()+0xca9) [0x55ad409af0c9] 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55ad409af5a8] 2024-04-12T21:27:39.836 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f471f1adb23] 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f47202b71ca] 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 11: clone() 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: -9> 2024-04-12T21:27:39.531+0000 7f4712707700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: in thread 7f4712707700 thread_name:ms_dispatch 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f47202c1d20] 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2: gsignal() 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 3: abort() 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55ad405dd268] 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55ad406790fc] 2024-04-12T21:27:39.837 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55ad409adf0a] 2024-04-12T21:27:39.838 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 7: (MDLog::trim()+0xca9) [0x55ad409af0c9] 2024-04-12T21:27:39.838 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55ad409af5a8] 2024-04-12T21:27:39.838 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f471f1adb23] 2024-04-12T21:27:39.838 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f47202b71ca] 2024-04-12T21:27:39.838 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 11: clone() 2024-04-12T21:27:39.838 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:39.838 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[71815]: 2024-04-12T21:27:39.838 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:39 smithi104 podman[71986]: 2024-04-12 21:27:39.590836582 +0000 UTC m=+0.018376536 container died b66e15b8a79ea525c8c24c68e32c42a13e07b54eb9fbb89766e5904a6c61a165 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-12T21:27:39.846 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:39 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:clientreplay 2024-04-12T21:27:39.846 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:39 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:27:39.846 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:39 smithi081 ceph-mon[31453]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:27:40.482 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:40 smithi104 podman[71986]: 2024-04-12 21:27:40.234548585 +0000 UTC m=+0.662088548 container remove b66e15b8a79ea525c8c24c68e32c42a13e07b54eb9fbb89766e5904a6c61a165 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-12T21:27:40.482 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:40 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:27:40.482 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:40 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-12T21:27:40.482 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:40 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 22.140s CPU time. 2024-04-12T21:27:40.482 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:40 smithi104 ceph-mon[36624]: pgmap v780: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-12T21:27:40.482 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:40 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:40.482 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:40 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:active 2024-04-12T21:27:40.483 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:40 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:27:40.483 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:40 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:40 smithi153 ceph-mon[36569]: pgmap v780: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-12T21:27:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:40 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:40 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:active 2024-04-12T21:27:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:40 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:27:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:40 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:40.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:40 smithi081 ceph-mon[31453]: pgmap v780: 129 pgs: 129 active+clean; 601 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 0 B/s wr, 6 op/s 2024-04-12T21:27:40.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:40 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:40.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:40 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/993026884,v1:172.21.15.104:6835/993026884] up:active 2024-04-12T21:27:40.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:40 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:27:40.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:40 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:42 smithi153 ceph-mon[36569]: pgmap v781: 129 pgs: 129 active+clean; 598 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 KiB/s wr, 36 op/s 2024-04-12T21:27:42.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:42 smithi104 ceph-mon[36624]: pgmap v781: 129 pgs: 129 active+clean; 598 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 KiB/s wr, 36 op/s 2024-04-12T21:27:42.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:42 smithi081 ceph-mon[31453]: pgmap v781: 129 pgs: 129 active+clean; 598 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 KiB/s wr, 36 op/s 2024-04-12T21:27:44.577 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:44 smithi153 ceph-mon[36569]: pgmap v782: 129 pgs: 129 active+clean; 598 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 93 KiB/s wr, 39 op/s 2024-04-12T21:27:44.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:44 smithi104 ceph-mon[36624]: pgmap v782: 129 pgs: 129 active+clean; 598 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 93 KiB/s wr, 39 op/s 2024-04-12T21:27:44.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:44 smithi081 ceph-mon[31453]: pgmap v782: 129 pgs: 129 active+clean; 598 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 93 KiB/s wr, 39 op/s 2024-04-12T21:27:46.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:46 smithi153 ceph-mon[36569]: pgmap v783: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 81 KiB/s wr, 37 op/s 2024-04-12T21:27:46.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:46 smithi104 ceph-mon[36624]: pgmap v783: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 81 KiB/s wr, 37 op/s 2024-04-12T21:27:46.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:46 smithi081 ceph-mon[31453]: pgmap v783: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 81 KiB/s wr, 37 op/s 2024-04-12T21:27:48.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:48 smithi153 ceph-mon[36569]: pgmap v784: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 78 KiB/s wr, 36 op/s 2024-04-12T21:27:48.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:48 smithi104 ceph-mon[36624]: pgmap v784: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 78 KiB/s wr, 36 op/s 2024-04-12T21:27:48.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:48 smithi081 ceph-mon[31453]: pgmap v784: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 78 KiB/s wr, 36 op/s 2024-04-12T21:27:50.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:50 smithi153 ceph-mon[36569]: pgmap v785: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 65 KiB/s wr, 31 op/s 2024-04-12T21:27:50.806 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:50 smithi104 ceph-mon[36624]: pgmap v785: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 65 KiB/s wr, 31 op/s 2024-04-12T21:27:50.807 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:50 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:27:50.807 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:50 smithi104 systemd[1]: Stopped Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:27:50.807 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:50 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 22.140s CPU time. 2024-04-12T21:27:50.807 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:50 smithi104 systemd[1]: Starting Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:27:50.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:50 smithi081 ceph-mon[31453]: pgmap v785: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 65 KiB/s wr, 31 op/s 2024-04-12T21:27:51.079 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:50 smithi104 podman[72094]: 2024-04-12 21:27:50.713570946 +0000 UTC m=+0.018599745 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:27:51.079 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:50 smithi104 podman[72094]: 2024-04-12 21:27:50.842917731 +0000 UTC m=+0.147946538 container create 20551596724de679730cd4e0355e862d2608566b76d3cc422f7ed731888ecd39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-12T21:27:51.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:27:51.128+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:51.490 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:51 smithi104 podman[72094]: 2024-04-12 21:27:51.087422187 +0000 UTC m=+0.392450988 container init 20551596724de679730cd4e0355e862d2608566b76d3cc422f7ed731888ecd39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T21:27:51.490 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:51 smithi104 podman[72094]: 2024-04-12 21:27:51.089931561 +0000 UTC m=+0.394960370 container start 20551596724de679730cd4e0355e862d2608566b76d3cc422f7ed731888ecd39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:27:51.490 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mds[72114]: -- 172.21.15.104:0/2066523182 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561ab431ba40 con 0x561ab42eec00 2024-04-12T21:27:51.490 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: starting mds.e at 2024-04-12T21:27:51.490 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:51 smithi104 bash[72094]: 20551596724de679730cd4e0355e862d2608566b76d3cc422f7ed731888ecd39 2024-04-12T21:27:51.491 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:27:51 smithi104 systemd[1]: Started Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:27:51.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: daemon mds.e restarted 2024-04-12T21:27:51.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:51.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:51.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: osdmap e121: 12 total, 12 up, 12 in 2024-04-12T21:27:51.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:boot 2024-04-12T21:27:51.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:27:51.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:51.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:51.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:27:51.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:51 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:27:51.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: daemon mds.e restarted 2024-04-12T21:27:51.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:51.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:51.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: osdmap e121: 12 total, 12 up, 12 in 2024-04-12T21:27:51.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:boot 2024-04-12T21:27:51.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:27:51.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:51.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:51.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:27:51.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:51 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: daemon mds.e restarted 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: osdmap e121: 12 total, 12 up, 12 in 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:boot 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:27:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:51 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:27:52.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:52 smithi153 ceph-mon[36569]: pgmap v787: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 409 B/s wr, 13 op/s 2024-04-12T21:27:52.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:52 smithi104 ceph-mon[36624]: pgmap v787: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 409 B/s wr, 13 op/s 2024-04-12T21:27:52.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:52 smithi081 ceph-mon[31453]: pgmap v787: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 409 B/s wr, 13 op/s 2024-04-12T21:27:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:53.295+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:27:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:53.295+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:27:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:27:53.295+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:27:53.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:53 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:reconnect 2024-04-12T21:27:53.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:53 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:27:53.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:53 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:27:53.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:53 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:reconnect 2024-04-12T21:27:53.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:53 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:27:53.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:53 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:27:53.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:53 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:reconnect 2024-04-12T21:27:53.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:53 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:27:53.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:53 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:27:54.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:54 smithi153 ceph-mon[36569]: pgmap v788: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:27:54.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:54 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:rejoin 2024-04-12T21:27:54.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:54 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:27:54.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:54 smithi104 ceph-mon[36624]: pgmap v788: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:27:54.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:54 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:rejoin 2024-04-12T21:27:54.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:54 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:27:54.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:54 smithi081 ceph-mon[31453]: pgmap v788: 129 pgs: 129 active+clean; 597 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:27:54.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:54 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:rejoin 2024-04-12T21:27:54.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:54 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:27:55.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:55 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:clientreplay 2024-04-12T21:27:55.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:55 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:27:55.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:55 smithi153 ceph-mon[36569]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:27:55.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:55 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:clientreplay 2024-04-12T21:27:55.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:55 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:27:55.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:55 smithi104 ceph-mon[36624]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:27:55.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:55 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:clientreplay 2024-04-12T21:27:55.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:55 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:27:55.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:55 smithi081 ceph-mon[31453]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:27:56.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-mon[36569]: pgmap v789: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 11 op/s 2024-04-12T21:27:56.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:56.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:active 2024-04-12T21:27:56.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:27:56.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:56.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:27:56.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: *** Caught signal (Aborted) ** 2024-04-12T21:27:56.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: in thread 7f7d061f9700 thread_name:ms_dispatch 2024-04-12T21:27:56.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:56.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7d13db3d20] 2024-04-12T21:27:56.806 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2: gsignal() 2024-04-12T21:27:56.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 3: abort() 2024-04-12T21:27:56.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55c524939268] 2024-04-12T21:27:56.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55c5249d50fc] 2024-04-12T21:27:56.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55c524d09f0a] 2024-04-12T21:27:56.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 7: (MDLog::trim()+0xca9) [0x55c524d0b0c9] 2024-04-12T21:27:56.807 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55c524d0b5a8] 2024-04-12T21:27:56.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f7d12c9fb23] 2024-04-12T21:27:56.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f7d13da91ca] 2024-04-12T21:27:56.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 11: clone() 2024-04-12T21:27:56.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2024-04-12T21:27:56.467+0000 7f7d061f9700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:56.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: in thread 7f7d061f9700 thread_name:ms_dispatch 2024-04-12T21:27:56.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2024-04-12T21:27:56.808 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:56.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7d13db3d20] 2024-04-12T21:27:56.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2: gsignal() 2024-04-12T21:27:56.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 3: abort() 2024-04-12T21:27:56.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55c524939268] 2024-04-12T21:27:56.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55c5249d50fc] 2024-04-12T21:27:56.809 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55c524d09f0a] 2024-04-12T21:27:56.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 7: (MDLog::trim()+0xca9) [0x55c524d0b0c9] 2024-04-12T21:27:56.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55c524d0b5a8] 2024-04-12T21:27:56.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f7d12c9fb23] 2024-04-12T21:27:56.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f7d13da91ca] 2024-04-12T21:27:56.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 11: clone() 2024-04-12T21:27:56.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:56.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2024-04-12T21:27:56.810 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 0> 2024-04-12T21:27:56.467+0000 7f7d061f9700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:56.811 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: in thread 7f7d061f9700 thread_name:ms_dispatch 2024-04-12T21:27:56.811 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2024-04-12T21:27:56.811 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:56.811 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7d13db3d20] 2024-04-12T21:27:56.811 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2: gsignal() 2024-04-12T21:27:56.811 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 3: abort() 2024-04-12T21:27:56.811 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55c524939268] 2024-04-12T21:27:56.812 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55c5249d50fc] 2024-04-12T21:27:56.812 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55c524d09f0a] 2024-04-12T21:27:56.812 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 7: (MDLog::trim()+0xca9) [0x55c524d0b0c9] 2024-04-12T21:27:56.812 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55c524d0b5a8] 2024-04-12T21:27:56.812 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f7d12c9fb23] 2024-04-12T21:27:56.812 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f7d13da91ca] 2024-04-12T21:27:56.812 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 11: clone() 2024-04-12T21:27:56.813 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:56.813 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2024-04-12T21:27:56.813 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: -10> 2024-04-12T21:27:56.467+0000 7f7d061f9700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:27:56.813 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: in thread 7f7d061f9700 thread_name:ms_dispatch 2024-04-12T21:27:56.813 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2024-04-12T21:27:56.813 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:27:56.814 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7d13db3d20] 2024-04-12T21:27:56.814 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2: gsignal() 2024-04-12T21:27:56.814 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 3: abort() 2024-04-12T21:27:56.814 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55c524939268] 2024-04-12T21:27:56.814 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55c5249d50fc] 2024-04-12T21:27:56.814 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55c524d09f0a] 2024-04-12T21:27:56.815 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 7: (MDLog::trim()+0xca9) [0x55c524d0b0c9] 2024-04-12T21:27:56.815 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55c524d0b5a8] 2024-04-12T21:27:56.815 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f7d12c9fb23] 2024-04-12T21:27:56.815 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f7d13da91ca] 2024-04-12T21:27:56.815 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 11: clone() 2024-04-12T21:27:56.815 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:27:56.815 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[73764]: 2024-04-12T21:27:56.815 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:27:56 smithi153 podman[74031]: 2024-04-12 21:27:56.576951564 +0000 UTC m=+0.031507718 container died 65daf3132e6fc4b3635ce798cd8484744c5d5cd58d64045f57492467d0fbf068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, CEPH_POINT_RELEASE=) 2024-04-12T21:27:56.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:56 smithi104 ceph-mon[36624]: pgmap v789: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 11 op/s 2024-04-12T21:27:56.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:56 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:56.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:56 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:active 2024-04-12T21:27:56.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:56 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:27:56.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:56 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:56.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:56 smithi081 ceph-mon[31453]: pgmap v789: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 11 op/s 2024-04-12T21:27:56.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:56 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:27:56.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:56 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4250244890,v1:172.21.15.153:6835/4250244890] up:active 2024-04-12T21:27:56.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:56 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:27:56.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:56 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:27:58.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:27:58 smithi153 ceph-mon[36569]: pgmap v790: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 33 KiB/s wr, 30 op/s 2024-04-12T21:27:58.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:27:58 smithi081 ceph-mon[31453]: pgmap v790: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 33 KiB/s wr, 30 op/s 2024-04-12T21:27:58.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:27:58 smithi104 ceph-mon[36624]: pgmap v790: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 33 KiB/s wr, 30 op/s 2024-04-12T21:28:00.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:00 smithi104 ceph-mon[36624]: pgmap v791: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 34 KiB/s wr, 36 op/s 2024-04-12T21:28:00.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:00 smithi153 ceph-mon[36569]: pgmap v791: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 34 KiB/s wr, 36 op/s 2024-04-12T21:28:00.839 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:00 smithi081 ceph-mon[31453]: pgmap v791: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 34 KiB/s wr, 36 op/s 2024-04-12T21:28:02.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:02 smithi104 ceph-mon[36624]: pgmap v792: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 113 KiB/s wr, 37 op/s 2024-04-12T21:28:02.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:02 smithi081 ceph-mon[31453]: pgmap v792: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 113 KiB/s wr, 37 op/s 2024-04-12T21:28:02.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:02 smithi153 ceph-mon[36569]: pgmap v792: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 113 KiB/s wr, 37 op/s 2024-04-12T21:28:02.937 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:02 smithi153 podman[74031]: 2024-04-12 21:28:02.705300718 +0000 UTC m=+6.159856862 container remove 65daf3132e6fc4b3635ce798cd8484744c5d5cd58d64045f57492467d0fbf068 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-12T21:28:02.937 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:28:03.303 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-12T21:28:03.304 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 24.118s CPU time. 2024-04-12T21:28:04.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:04 smithi104 ceph-mon[36624]: pgmap v793: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 97 KiB/s wr, 32 op/s 2024-04-12T21:28:04.870 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:04 smithi081 ceph-mon[31453]: pgmap v793: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 97 KiB/s wr, 32 op/s 2024-04-12T21:28:05.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:04 smithi153 ceph-mon[36569]: pgmap v793: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 97 KiB/s wr, 32 op/s 2024-04-12T21:28:06.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:06 smithi104 ceph-mon[36624]: pgmap v794: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 97 KiB/s wr, 33 op/s 2024-04-12T21:28:06.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:06 smithi081 ceph-mon[31453]: pgmap v794: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 97 KiB/s wr, 33 op/s 2024-04-12T21:28:07.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:06 smithi153 ceph-mon[36569]: pgmap v794: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 97 KiB/s wr, 33 op/s 2024-04-12T21:28:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:08.311+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:28:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:08.311+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:28:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:08.311+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:28:08.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:08 smithi104 ceph-mon[36624]: pgmap v795: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 97 KiB/s wr, 28 op/s 2024-04-12T21:28:08.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:08 smithi081 ceph-mon[31453]: pgmap v795: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 97 KiB/s wr, 28 op/s 2024-04-12T21:28:09.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:08 smithi153 ceph-mon[36569]: pgmap v795: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 97 KiB/s wr, 28 op/s 2024-04-12T21:28:10.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:10 smithi104 ceph-mon[36624]: pgmap v796: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 69 KiB/s wr, 11 op/s 2024-04-12T21:28:10.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:10 smithi081 ceph-mon[31453]: pgmap v796: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 69 KiB/s wr, 11 op/s 2024-04-12T21:28:11.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:10 smithi153 ceph-mon[36569]: pgmap v796: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 69 KiB/s wr, 11 op/s 2024-04-12T21:28:12.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:12.819+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006140 con 0x7f1704076980 2024-04-12T21:28:12.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:12.820+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006140 con 0x7f1704076980 2024-04-12T21:28:12.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:12.852+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d0005e60 con 0x563c9e245d20 2024-04-12T21:28:12.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:12.852+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d0005e60 con 0x563c9e245d20 2024-04-12T21:28:12.856 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:12 smithi153 ceph-mon[36569]: pgmap v797: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 69 KiB/s wr, 7 op/s 2024-04-12T21:28:12.856 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:12 smithi153 ceph-mon[36569]: Replacing daemon mds.f as rank 0 with standby daemon mds.e 2024-04-12T21:28:12.856 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:12 smithi153 ceph-mon[36569]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-12T21:28:12.856 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:12 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:12.856 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:12 smithi153 ceph-mon[36569]: osdmap e122: 12 total, 12 up, 12 in 2024-04-12T21:28:12.856 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:12 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:replay} 6 up:standby 2024-04-12T21:28:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:12 smithi081 ceph-mon[31453]: pgmap v797: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 69 KiB/s wr, 7 op/s 2024-04-12T21:28:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:12 smithi081 ceph-mon[31453]: Replacing daemon mds.f as rank 0 with standby daemon mds.e 2024-04-12T21:28:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:12 smithi081 ceph-mon[31453]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-12T21:28:12.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:12 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:12.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:12 smithi081 ceph-mon[31453]: osdmap e122: 12 total, 12 up, 12 in 2024-04-12T21:28:12.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:12 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:replay} 6 up:standby 2024-04-12T21:28:13.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:12 smithi104 ceph-mon[36624]: pgmap v797: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 69 KiB/s wr, 7 op/s 2024-04-12T21:28:13.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:12 smithi104 ceph-mon[36624]: Replacing daemon mds.f as rank 0 with standby daemon mds.e 2024-04-12T21:28:13.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:12 smithi104 ceph-mon[36624]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-12T21:28:13.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:12 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:13.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:12 smithi104 ceph-mon[36624]: osdmap e122: 12 total, 12 up, 12 in 2024-04-12T21:28:13.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:12 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:replay} 6 up:standby 2024-04-12T21:28:13.292 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:28:13.292 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 systemd[1]: Stopped Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:28:13.292 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 24.118s CPU time. 2024-04-12T21:28:13.292 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 systemd[1]: Starting Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:28:13.554 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 podman[74130]: 2024-04-12 21:28:13.198357792 +0000 UTC m=+0.018130373 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:28:13.554 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 podman[74130]: 2024-04-12 21:28:13.297409434 +0000 UTC m=+0.117181990 container create a0af144a5f3147a9cd7d28fcce8b8caa81649a1a27fd11f5e500f1983423fb6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:28:13.554 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 podman[74130]: 2024-04-12 21:28:13.54257843 +0000 UTC m=+0.362350995 container init a0af144a5f3147a9cd7d28fcce8b8caa81649a1a27fd11f5e500f1983423fb6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux ) 2024-04-12T21:28:13.554 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 podman[74130]: 2024-04-12 21:28:13.545436264 +0000 UTC m=+0.365208837 container start a0af144a5f3147a9cd7d28fcce8b8caa81649a1a27fd11f5e500f1983423fb6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:28:14.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 ceph-mds[74149]: -- 172.21.15.153:0/1839211319 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560b4778fa40 con 0x560b47762c00 2024-04-12T21:28:14.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: starting mds.f at 2024-04-12T21:28:14.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 bash[74130]: a0af144a5f3147a9cd7d28fcce8b8caa81649a1a27fd11f5e500f1983423fb6a 2024-04-12T21:28:14.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:13 smithi153 systemd[1]: Started Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:28:14.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:14 smithi081 ceph-mon[31453]: pgmap v799: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:28:14.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:14 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:reconnect 2024-04-12T21:28:14.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:14 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:reconnect} 6 up:standby 2024-04-12T21:28:14.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:14 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209998 2024-04-12T21:28:15.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:14 smithi153 ceph-mon[36569]: pgmap v799: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:28:15.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:14 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:reconnect 2024-04-12T21:28:15.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:14 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:reconnect} 6 up:standby 2024-04-12T21:28:15.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:14 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209998 2024-04-12T21:28:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:14 smithi104 ceph-mon[36624]: pgmap v799: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:28:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:14 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:reconnect 2024-04-12T21:28:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:14 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:reconnect} 6 up:standby 2024-04-12T21:28:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:14 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209998 2024-04-12T21:28:15.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:boot 2024-04-12T21:28:15.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:rejoin 2024-04-12T21:28:15.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:15 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:28:15.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:28:16.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:boot 2024-04-12T21:28:16.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:rejoin 2024-04-12T21:28:16.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:15 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:28:16.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:28:16.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:boot 2024-04-12T21:28:16.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:rejoin 2024-04-12T21:28:16.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:15 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:28:16.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:28:16.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:16 smithi081 ceph-mon[31453]: pgmap v800: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2 op/s 2024-04-12T21:28:16.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:16 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:clientreplay 2024-04-12T21:28:16.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:16 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:28:16.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:16 smithi081 ceph-mon[31453]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:28:17.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:16 smithi153 ceph-mon[36569]: pgmap v800: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2 op/s 2024-04-12T21:28:17.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:16 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:clientreplay 2024-04-12T21:28:17.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:16 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:28:17.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:16 smithi153 ceph-mon[36569]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:28:17.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:16 smithi104 ceph-mon[36624]: pgmap v800: 129 pgs: 129 active+clean; 596 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2 op/s 2024-04-12T21:28:17.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:16 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:clientreplay 2024-04-12T21:28:17.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:16 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:28:17.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:16 smithi104 ceph-mon[36624]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:28:17.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:17.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:active 2024-04-12T21:28:17.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:28:17.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:17.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: *** Caught signal (Aborted) ** 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: in thread 7fd3da361700 thread_name:ms_dispatch 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd3e7f1bd20] 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2: gsignal() 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 3: abort() 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561ab2254268] 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561ab22f00fc] 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561ab2624f0a] 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 7: (MDLog::trim()+0xca9) [0x561ab26260c9] 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561ab26265a8] 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fd3e6e07b23] 2024-04-12T21:28:17.831 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fd3e7f111ca] 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 11: clone() 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2024-04-12T21:28:17.557+0000 7fd3da361700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: in thread 7fd3da361700 thread_name:ms_dispatch 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd3e7f1bd20] 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2: gsignal() 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 3: abort() 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561ab2254268] 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561ab22f00fc] 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561ab2624f0a] 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 7: (MDLog::trim()+0xca9) [0x561ab26260c9] 2024-04-12T21:28:17.832 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561ab26265a8] 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fd3e6e07b23] 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fd3e7f111ca] 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 11: clone() 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 0> 2024-04-12T21:28:17.557+0000 7fd3da361700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: in thread 7fd3da361700 thread_name:ms_dispatch 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd3e7f1bd20] 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2: gsignal() 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 3: abort() 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561ab2254268] 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561ab22f00fc] 2024-04-12T21:28:17.833 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561ab2624f0a] 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 7: (MDLog::trim()+0xca9) [0x561ab26260c9] 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561ab26265a8] 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fd3e6e07b23] 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fd3e7f111ca] 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 11: clone() 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: -9> 2024-04-12T21:28:17.557+0000 7fd3da361700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: in thread 7fd3da361700 thread_name:ms_dispatch 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd3e7f1bd20] 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2: gsignal() 2024-04-12T21:28:17.834 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 3: abort() 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561ab2254268] 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561ab22f00fc] 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561ab2624f0a] 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 7: (MDLog::trim()+0xca9) [0x561ab26260c9] 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561ab26265a8] 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fd3e6e07b23] 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fd3e7f111ca] 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 11: clone() 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72110]: 2024-04-12T21:28:17.835 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:17 smithi104 podman[72150]: 2024-04-12 21:28:17.61559536 +0000 UTC m=+0.018556477 container died 20551596724de679730cd4e0355e862d2608566b76d3cc422f7ed731888ecd39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-12T21:28:17.863 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:17 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:17.863 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:17 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:active 2024-04-12T21:28:17.863 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:17 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:28:17.863 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:17 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:18.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:17 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:18.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:17 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3174062203,v1:172.21.15.104:6835/3174062203] up:active 2024-04-12T21:28:18.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:17 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:28:18.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:17 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:18.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:18 smithi104 podman[72150]: 2024-04-12 21:28:18.323100103 +0000 UTC m=+0.726061213 container remove 20551596724de679730cd4e0355e862d2608566b76d3cc422f7ed731888ecd39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS) 2024-04-12T21:28:18.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:18 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:28:18.579 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:18 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-12T21:28:18.580 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:18 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 23.119s CPU time. 2024-04-12T21:28:18.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:18 smithi081 ceph-mon[31453]: pgmap v801: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-12T21:28:19.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:18 smithi153 ceph-mon[36569]: pgmap v801: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-12T21:28:19.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:18 smithi104 ceph-mon[36624]: pgmap v801: 129 pgs: 129 active+clean; 595 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-12T21:28:20.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:20 smithi081 ceph-mon[31453]: pgmap v802: 129 pgs: 129 active+clean; 594 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 KiB/s wr, 18 op/s 2024-04-12T21:28:21.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:20 smithi153 ceph-mon[36569]: pgmap v802: 129 pgs: 129 active+clean; 594 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 KiB/s wr, 18 op/s 2024-04-12T21:28:21.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:20 smithi104 ceph-mon[36624]: pgmap v802: 129 pgs: 129 active+clean; 594 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 KiB/s wr, 18 op/s 2024-04-12T21:28:22.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:22 smithi081 ceph-mon[31453]: pgmap v803: 129 pgs: 129 active+clean; 592 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 72 KiB/s wr, 42 op/s 2024-04-12T21:28:23.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:22 smithi153 ceph-mon[36569]: pgmap v803: 129 pgs: 129 active+clean; 592 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 72 KiB/s wr, 42 op/s 2024-04-12T21:28:23.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:22 smithi104 ceph-mon[36624]: pgmap v803: 129 pgs: 129 active+clean; 592 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 72 KiB/s wr, 42 op/s 2024-04-12T21:28:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:23.325+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:28:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:23.325+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:28:23.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:23.325+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:28:24.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:24 smithi081 ceph-mon[31453]: pgmap v804: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 KiB/s wr, 41 op/s 2024-04-12T21:28:25.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:24 smithi153 ceph-mon[36569]: pgmap v804: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 KiB/s wr, 41 op/s 2024-04-12T21:28:25.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:24 smithi104 ceph-mon[36624]: pgmap v804: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 66 KiB/s wr, 41 op/s 2024-04-12T21:28:26.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:26 smithi081 ceph-mon[31453]: pgmap v805: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 60 KiB/s wr, 40 op/s 2024-04-12T21:28:27.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:26 smithi153 ceph-mon[36569]: pgmap v805: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 60 KiB/s wr, 40 op/s 2024-04-12T21:28:27.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:26 smithi104 ceph-mon[36624]: pgmap v805: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 60 KiB/s wr, 40 op/s 2024-04-12T21:28:28.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:28 smithi081 ceph-mon[31453]: pgmap v806: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 60 KiB/s wr, 40 op/s 2024-04-12T21:28:28.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:28 smithi104 ceph-mon[36624]: pgmap v806: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 60 KiB/s wr, 40 op/s 2024-04-12T21:28:28.960 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:28 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:28:28.960 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:28 smithi104 systemd[1]: Stopped Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:28:28.960 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:28 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 23.119s CPU time. 2024-04-12T21:28:28.960 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:28 smithi104 systemd[1]: Starting Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:28:28.960 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:28 smithi104 podman[72244]: 2024-04-12 21:28:28.718464163 +0000 UTC m=+0.018392401 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:28:28.960 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:28 smithi104 podman[72244]: 2024-04-12 21:28:28.823720436 +0000 UTC m=+0.123648673 container create a868d2789511bbfb13222ea4a72133e406dea1aed1a6831cccfd18900595ad54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-04-12T21:28:29.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:28 smithi153 ceph-mon[36569]: pgmap v806: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 60 KiB/s wr, 40 op/s 2024-04-12T21:28:29.329 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:29 smithi104 podman[72244]: 2024-04-12 21:28:29.093482838 +0000 UTC m=+0.393411089 container init a868d2789511bbfb13222ea4a72133e406dea1aed1a6831cccfd18900595ad54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-12T21:28:29.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:29 smithi104 podman[72244]: 2024-04-12 21:28:29.096133154 +0000 UTC m=+0.396061412 container start a868d2789511bbfb13222ea4a72133e406dea1aed1a6831cccfd18900595ad54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-04-12T21:28:29.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mds[72264]: -- 172.21.15.104:0/2803405546 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e25b37ba40 con 0x55e25b34ec00 2024-04-12T21:28:29.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: starting mds.e at 2024-04-12T21:28:29.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:29 smithi104 bash[72244]: a868d2789511bbfb13222ea4a72133e406dea1aed1a6831cccfd18900595ad54 2024-04-12T21:28:29.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:29 smithi104 systemd[1]: Started Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:28:29.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:28:29.133+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:28:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: daemon mds.e restarted 2024-04-12T21:28:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:28:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: osdmap e123: 12 total, 12 up, 12 in 2024-04-12T21:28:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:boot 2024-04-12T21:28:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:28:30.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:28:30.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:28:30.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:28:30.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:29 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:28:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: daemon mds.e restarted 2024-04-12T21:28:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:28:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: osdmap e123: 12 total, 12 up, 12 in 2024-04-12T21:28:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:boot 2024-04-12T21:28:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:28:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:28:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:28:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:28:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:29 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:28:30.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: daemon mds.e restarted 2024-04-12T21:28:30.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:30.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:28:30.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: osdmap e123: 12 total, 12 up, 12 in 2024-04-12T21:28:30.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:boot 2024-04-12T21:28:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:28:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:28:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:28:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:28:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:29 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:28:31.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:30 smithi153 ceph-mon[36569]: pgmap v808: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 71 KiB/s wr, 32 op/s 2024-04-12T21:28:31.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:30 smithi104 ceph-mon[36624]: pgmap v808: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 71 KiB/s wr, 32 op/s 2024-04-12T21:28:31.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:30 smithi081 ceph-mon[31453]: pgmap v808: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 71 KiB/s wr, 32 op/s 2024-04-12T21:28:32.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:31 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:reconnect 2024-04-12T21:28:32.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:31 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:28:32.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:31 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189997 2024-04-12T21:28:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:31 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:reconnect 2024-04-12T21:28:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:31 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:28:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:31 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189997 2024-04-12T21:28:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:31 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:reconnect 2024-04-12T21:28:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:31 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:28:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:31 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189997 2024-04-12T21:28:33.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:32 smithi153 ceph-mon[36569]: pgmap v809: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 204 B/s wr, 9 op/s 2024-04-12T21:28:33.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:32 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:rejoin 2024-04-12T21:28:33.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:32 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:28:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:32 smithi104 ceph-mon[36624]: pgmap v809: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 204 B/s wr, 9 op/s 2024-04-12T21:28:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:32 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:rejoin 2024-04-12T21:28:33.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:32 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:28:33.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:32 smithi081 ceph-mon[31453]: pgmap v809: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 204 B/s wr, 9 op/s 2024-04-12T21:28:33.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:32 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:rejoin 2024-04-12T21:28:33.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:32 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:28:34.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:33 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:clientreplay 2024-04-12T21:28:34.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:33 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:28:34.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:33 smithi153 ceph-mon[36569]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:28:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:33 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:clientreplay 2024-04-12T21:28:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:33 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:28:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:33 smithi104 ceph-mon[36624]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:28:34.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:33 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:clientreplay 2024-04-12T21:28:34.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:33 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:28:34.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:33 smithi081 ceph-mon[31453]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:28:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-mon[36569]: pgmap v810: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:28:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:active 2024-04-12T21:28:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:28:35.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2024-04-12T21:28:34.620+0000 7fbda4c23700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: *** Caught signal (Aborted) ** 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: in thread 7fbda2c1f700 thread_name:ms_dispatch 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbdb07d9d20] 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2: gsignal() 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 3: abort() 2024-04-12T21:28:35.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560b46293268] 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x560b4632f0fc] 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560b46663f0a] 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 7: (MDLog::trim()+0xca9) [0x560b466650c9] 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560b466655a8] 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbdaf6c5b23] 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbdb07cf1ca] 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 11: clone() 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2024-04-12T21:28:34.656+0000 7fbda2c1f700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: in thread 7fbda2c1f700 thread_name:ms_dispatch 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2024-04-12T21:28:35.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbdb07d9d20] 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2: gsignal() 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 3: abort() 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560b46293268] 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x560b4632f0fc] 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560b46663f0a] 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 7: (MDLog::trim()+0xca9) [0x560b466650c9] 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560b466655a8] 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbdaf6c5b23] 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbdb07cf1ca] 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 11: clone() 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:35.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: -6997> 2024-04-12T21:28:34.620+0000 7fbda4c23700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 0> 2024-04-12T21:28:34.656+0000 7fbda2c1f700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: in thread 7fbda2c1f700 thread_name:ms_dispatch 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbdb07d9d20] 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2: gsignal() 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 3: abort() 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560b46293268] 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x560b4632f0fc] 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560b46663f0a] 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 7: (MDLog::trim()+0xca9) [0x560b466650c9] 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560b466655a8] 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbdaf6c5b23] 2024-04-12T21:28:35.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbdb07cf1ca] 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 11: clone() 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: -6997> 2024-04-12T21:28:34.620+0000 7fbda4c23700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 0> 2024-04-12T21:28:34.656+0000 7fbda2c1f700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: in thread 7fbda2c1f700 thread_name:ms_dispatch 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbdb07d9d20] 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2: gsignal() 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 3: abort() 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560b46293268] 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x560b4632f0fc] 2024-04-12T21:28:35.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560b46663f0a] 2024-04-12T21:28:35.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 7: (MDLog::trim()+0xca9) [0x560b466650c9] 2024-04-12T21:28:35.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560b466655a8] 2024-04-12T21:28:35.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbdaf6c5b23] 2024-04-12T21:28:35.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbdb07cf1ca] 2024-04-12T21:28:35.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 11: clone() 2024-04-12T21:28:35.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:35.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74145]: 2024-04-12T21:28:35.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:34 smithi153 podman[74186]: 2024-04-12 21:28:34.773044038 +0000 UTC m=+0.018182392 container died a0af144a5f3147a9cd7d28fcce8b8caa81649a1a27fd11f5e500f1983423fb6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-12T21:28:35.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:34 smithi104 ceph-mon[36624]: pgmap v810: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:28:35.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:34 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:35.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:34 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:active 2024-04-12T21:28:35.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:34 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:28:35.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:34 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:34 smithi081 ceph-mon[31453]: pgmap v810: 129 pgs: 129 active+clean; 591 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:28:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:34 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:34 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/4173993540,v1:172.21.15.153:6835/4173993540] up:active 2024-04-12T21:28:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:34 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:28:35.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:34 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:35.702 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:35 smithi153 podman[74186]: 2024-04-12 21:28:35.45516708 +0000 UTC m=+0.700305420 container remove a0af144a5f3147a9cd7d28fcce8b8caa81649a1a27fd11f5e500f1983423fb6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=) 2024-04-12T21:28:35.702 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:35 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:28:35.702 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:35 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-12T21:28:36.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:35 smithi153 ceph-mon[36569]: pgmap v811: 129 pgs: 129 active+clean; 592 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 155 KiB/s wr, 19 op/s 2024-04-12T21:28:36.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:35 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 23.749s CPU time. 2024-04-12T21:28:36.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:35 smithi104 ceph-mon[36624]: pgmap v811: 129 pgs: 129 active+clean; 592 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 155 KiB/s wr, 19 op/s 2024-04-12T21:28:36.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:35 smithi081 ceph-mon[31453]: pgmap v811: 129 pgs: 129 active+clean; 592 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 155 KiB/s wr, 19 op/s 2024-04-12T21:28:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:38.340+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:28:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:38.340+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:28:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:38.340+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:28:38.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:38 smithi153 ceph-mon[36569]: pgmap v812: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 157 KiB/s wr, 30 op/s 2024-04-12T21:28:38.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:28:38.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:38 smithi104 ceph-mon[36624]: pgmap v812: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 157 KiB/s wr, 30 op/s 2024-04-12T21:28:38.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:28:38.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:38 smithi081 ceph-mon[31453]: pgmap v812: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 157 KiB/s wr, 30 op/s 2024-04-12T21:28:38.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:28:39.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:28:39.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:28:39.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:28:39.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:28:39.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:28:39.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:28:39.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:28:39.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:28:39.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:28:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:40 smithi153 ceph-mon[36569]: pgmap v813: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 159 KiB/s wr, 31 op/s 2024-04-12T21:28:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:40 smithi153 ceph-mon[36569]: pgmap v814: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 165 KiB/s wr, 32 op/s 2024-04-12T21:28:40.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:40 smithi104 ceph-mon[36624]: pgmap v813: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 159 KiB/s wr, 31 op/s 2024-04-12T21:28:40.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:40 smithi104 ceph-mon[36624]: pgmap v814: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 165 KiB/s wr, 32 op/s 2024-04-12T21:28:40.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:40 smithi081 ceph-mon[31453]: pgmap v813: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 159 KiB/s wr, 31 op/s 2024-04-12T21:28:40.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:40 smithi081 ceph-mon[31453]: pgmap v814: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 165 KiB/s wr, 32 op/s 2024-04-12T21:28:42.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:42 smithi153 ceph-mon[36569]: pgmap v815: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 165 KiB/s wr, 32 op/s 2024-04-12T21:28:42.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:42 smithi104 ceph-mon[36624]: pgmap v815: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 165 KiB/s wr, 32 op/s 2024-04-12T21:28:42.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:42 smithi081 ceph-mon[31453]: pgmap v815: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 165 KiB/s wr, 32 op/s 2024-04-12T21:28:44.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:44 smithi153 ceph-mon[36569]: pgmap v816: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 165 KiB/s wr, 33 op/s 2024-04-12T21:28:44.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:44 smithi104 ceph-mon[36624]: pgmap v816: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 165 KiB/s wr, 33 op/s 2024-04-12T21:28:44.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:44 smithi081 ceph-mon[31453]: pgmap v816: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 165 KiB/s wr, 33 op/s 2024-04-12T21:28:46.255 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:45 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:28:46.255 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:45 smithi153 systemd[1]: Stopped Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:28:46.255 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:45 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 23.749s CPU time. 2024-04-12T21:28:46.255 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:45 smithi153 systemd[1]: Starting Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:28:46.531 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:46 smithi153 podman[74410]: 2024-04-12 21:28:46.160669619 +0000 UTC m=+0.018155574 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:28:46.531 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:46 smithi153 podman[74410]: 2024-04-12 21:28:46.28957558 +0000 UTC m=+0.147061573 container create 4ce54a5bb2547ec0adff6a656ca2d801efe3f58646d86550ef2550b74ebb760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-04-12T21:28:46.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:46 smithi153 ceph-mon[36569]: pgmap v817: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-12T21:28:46.804 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:46 smithi153 podman[74410]: 2024-04-12 21:28:46.534107672 +0000 UTC m=+0.391593612 container init 4ce54a5bb2547ec0adff6a656ca2d801efe3f58646d86550ef2550b74ebb760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, ceph=True) 2024-04-12T21:28:46.804 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:46 smithi153 podman[74410]: 2024-04-12 21:28:46.536784785 +0000 UTC m=+0.394270743 container start 4ce54a5bb2547ec0adff6a656ca2d801efe3f58646d86550ef2550b74ebb760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409) 2024-04-12T21:28:46.804 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:46 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: starting mds.f at 2024-04-12T21:28:46.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:46 smithi153 bash[74410]: 4ce54a5bb2547ec0adff6a656ca2d801efe3f58646d86550ef2550b74ebb760b 2024-04-12T21:28:46.805 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:28:46 smithi153 systemd[1]: Started Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:28:46.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:46 smithi104 ceph-mon[36624]: pgmap v817: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-12T21:28:46.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:46 smithi081 ceph-mon[31453]: pgmap v817: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-12T21:28:46.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:28:46.580+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: daemon mds.f restarted 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: osdmap e124: 12 total, 12 up, 12 in 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:boot 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:28:47.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:47 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: daemon mds.f restarted 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: osdmap e124: 12 total, 12 up, 12 in 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:boot 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:28:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:47 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:28:48.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: daemon mds.f restarted 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: osdmap e124: 12 total, 12 up, 12 in 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:boot 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: Standby daemon mds.e assigned to filesystem cephfs as rank 0 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-12T21:28:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:47 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:replay} 7 up:standby 2024-04-12T21:28:48.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:48 smithi104 ceph-mon[36624]: pgmap v819: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 6 op/s 2024-04-12T21:28:48.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:48 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:reconnect 2024-04-12T21:28:48.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:48 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:28:48.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:48 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999988 2024-04-12T21:28:48.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:48 smithi081 ceph-mon[31453]: pgmap v819: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 6 op/s 2024-04-12T21:28:48.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:48 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:reconnect 2024-04-12T21:28:48.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:48 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:28:48.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:48 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999988 2024-04-12T21:28:49.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:48 smithi153 ceph-mon[36569]: pgmap v819: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 6 op/s 2024-04-12T21:28:49.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:48 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:reconnect 2024-04-12T21:28:49.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:48 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:reconnect} 7 up:standby 2024-04-12T21:28:49.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:48 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999988 2024-04-12T21:28:49.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:49 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:rejoin 2024-04-12T21:28:49.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:49 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:28:50.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:49 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:rejoin 2024-04-12T21:28:50.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:49 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:28:50.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:49 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:rejoin 2024-04-12T21:28:50.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:49 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:rejoin} 7 up:standby 2024-04-12T21:28:50.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:50 smithi081 ceph-mon[31453]: pgmap v820: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:28:50.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:50 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:clientreplay 2024-04-12T21:28:50.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:50 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:28:50.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:50 smithi081 ceph-mon[31453]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:28:51.074 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:50 smithi153 ceph-mon[36569]: pgmap v820: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:28:51.075 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:50 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:clientreplay 2024-04-12T21:28:51.075 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:50 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:28:51.075 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:50 smithi153 ceph-mon[36569]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:28:51.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-mon[36624]: pgmap v820: 129 pgs: 129 active+clean; 590 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:28:51.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:clientreplay 2024-04-12T21:28:51.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:clientreplay} 7 up:standby 2024-04-12T21:28:51.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-mon[36624]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:28:51.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:28:51.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: *** Caught signal (Aborted) ** 2024-04-12T21:28:51.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: in thread 7ff63886a700 thread_name:ms_dispatch 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff646424d20] 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2: gsignal() 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 3: abort() 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55e25971d268] 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55e2597b90fc] 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55e259aedf0a] 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 7: (MDLog::trim()+0xca9) [0x55e259aef0c9] 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55e259aef5a8] 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff645310b23] 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff64641a1ca] 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 11: clone() 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2024-04-12T21:28:50.733+0000 7ff63886a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: in thread 7ff63886a700 thread_name:ms_dispatch 2024-04-12T21:28:51.081 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff646424d20] 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2: gsignal() 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 3: abort() 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55e25971d268] 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55e2597b90fc] 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55e259aedf0a] 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 7: (MDLog::trim()+0xca9) [0x55e259aef0c9] 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55e259aef5a8] 2024-04-12T21:28:51.082 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff645310b23] 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff64641a1ca] 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 11: clone() 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 0> 2024-04-12T21:28:50.733+0000 7ff63886a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: in thread 7ff63886a700 thread_name:ms_dispatch 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff646424d20] 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2: gsignal() 2024-04-12T21:28:51.083 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 3: abort() 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55e25971d268] 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55e2597b90fc] 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55e259aedf0a] 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 7: (MDLog::trim()+0xca9) [0x55e259aef0c9] 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55e259aef5a8] 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff645310b23] 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff64641a1ca] 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 11: clone() 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: -10> 2024-04-12T21:28:50.733+0000 7ff63886a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: in thread 7ff63886a700 thread_name:ms_dispatch 2024-04-12T21:28:51.084 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff646424d20] 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2: gsignal() 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 3: abort() 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55e25971d268] 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55e2597b90fc] 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55e259aedf0a] 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 7: (MDLog::trim()+0xca9) [0x55e259aef0c9] 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55e259aef5a8] 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff645310b23] 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff64641a1ca] 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 11: clone() 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72260]: 2024-04-12T21:28:51.085 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:50 smithi104 podman[72441]: 2024-04-12 21:28:50.794130834 +0000 UTC m=+0.019114490 container died a868d2789511bbfb13222ea4a72133e406dea1aed1a6831cccfd18900595ad54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2) 2024-04-12T21:28:51.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:51 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:51.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:51 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:active 2024-04-12T21:28:51.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:51 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:28:51.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:51 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:51.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:51 smithi104 podman[72441]: 2024-04-12 21:28:51.507144734 +0000 UTC m=+0.732128389 container remove a868d2789511bbfb13222ea4a72133e406dea1aed1a6831cccfd18900595ad54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:28:51.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:51 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:28:51.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:51 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-12T21:28:51.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:28:51 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 22.051s CPU time. 2024-04-12T21:28:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:51 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:51 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:active 2024-04-12T21:28:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:51 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:28:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:51 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:51 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:28:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:51 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3830746769,v1:172.21.15.104:6835/3830746769] up:active 2024-04-12T21:28:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:51 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 7 up:standby 2024-04-12T21:28:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:51 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:28:53.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:52 smithi153 ceph-mon[36569]: pgmap v821: 129 pgs: 129 active+clean; 587 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 409 B/s wr, 23 op/s 2024-04-12T21:28:53.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:52 smithi104 ceph-mon[36624]: pgmap v821: 129 pgs: 129 active+clean; 587 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 409 B/s wr, 23 op/s 2024-04-12T21:28:53.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:52 smithi081 ceph-mon[31453]: pgmap v821: 129 pgs: 129 active+clean; 587 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 409 B/s wr, 23 op/s 2024-04-12T21:28:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:53.348+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:28:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:53.348+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:28:53.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:28:53.348+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:28:54.922 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:54 smithi081 ceph-mon[31453]: pgmap v822: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 KiB/s wr, 44 op/s 2024-04-12T21:28:55.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:54 smithi153 ceph-mon[36569]: pgmap v822: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 KiB/s wr, 44 op/s 2024-04-12T21:28:55.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:54 smithi104 ceph-mon[36624]: pgmap v822: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 5.0 KiB/s wr, 44 op/s 2024-04-12T21:28:57.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:56 smithi153 ceph-mon[36569]: pgmap v823: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.2 KiB/s wr, 53 op/s 2024-04-12T21:28:57.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:56 smithi104 ceph-mon[36624]: pgmap v823: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.2 KiB/s wr, 53 op/s 2024-04-12T21:28:57.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:56 smithi081 ceph-mon[31453]: pgmap v823: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 5.2 KiB/s wr, 53 op/s 2024-04-12T21:28:58.956 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:28:58 smithi081 ceph-mon[31453]: pgmap v824: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 29 KiB/s wr, 53 op/s 2024-04-12T21:28:59.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:28:58 smithi153 ceph-mon[36569]: pgmap v824: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 29 KiB/s wr, 53 op/s 2024-04-12T21:28:59.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:28:58 smithi104 ceph-mon[36624]: pgmap v824: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 29 KiB/s wr, 53 op/s 2024-04-12T21:29:01.005 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:00 smithi081 ceph-mon[31453]: pgmap v825: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 25 KiB/s wr, 46 op/s 2024-04-12T21:29:01.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:00 smithi153 ceph-mon[36569]: pgmap v825: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 25 KiB/s wr, 46 op/s 2024-04-12T21:29:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:00 smithi104 ceph-mon[36624]: pgmap v825: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 25 KiB/s wr, 46 op/s 2024-04-12T21:29:02.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:01 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:29:02.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:01 smithi104 systemd[1]: Stopped Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:29:02.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:01 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 22.051s CPU time. 2024-04-12T21:29:02.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:01 smithi104 systemd[1]: Starting Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:29:02.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:02 smithi104 podman[72536]: 2024-04-12 21:29:01.969389186 +0000 UTC m=+0.018191750 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:29:02.971 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:02 smithi104 ceph-mon[36624]: pgmap v826: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 25 KiB/s wr, 48 op/s 2024-04-12T21:29:03.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:02 smithi153 ceph-mon[36569]: pgmap v826: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 25 KiB/s wr, 48 op/s 2024-04-12T21:29:03.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:02 smithi081 ceph-mon[31453]: pgmap v826: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 25 KiB/s wr, 48 op/s 2024-04-12T21:29:03.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:02 smithi104 podman[72536]: 2024-04-12 21:29:02.974629581 +0000 UTC m=+1.023432151 container create 88b759dfa429d97b8d4cdb8e389c75e62698d61ecaa0e9f499937498ebba86f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-12T21:29:04.950 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:04 smithi081 ceph-mon[31453]: pgmap v827: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 25 KiB/s wr, 31 op/s 2024-04-12T21:29:05.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:04 smithi153 ceph-mon[36569]: pgmap v827: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 25 KiB/s wr, 31 op/s 2024-04-12T21:29:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:04 smithi104 ceph-mon[36624]: pgmap v827: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 25 KiB/s wr, 31 op/s 2024-04-12T21:29:05.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:04 smithi104 podman[72536]: 2024-04-12 21:29:04.903232283 +0000 UTC m=+2.952034837 container init 88b759dfa429d97b8d4cdb8e389c75e62698d61ecaa0e9f499937498ebba86f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-12T21:29:05.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:04 smithi104 podman[72536]: 2024-04-12 21:29:04.906565164 +0000 UTC m=+2.955367716 container start 88b759dfa429d97b8d4cdb8e389c75e62698d61ecaa0e9f499937498ebba86f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:29:05.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:04 smithi104 ceph-mds[72555]: -- 172.21.15.104:0/805056593 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556e6a69da40 con 0x556e6a670800 2024-04-12T21:29:05.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:04 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: starting mds.e at 2024-04-12T21:29:05.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:04 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:29:04.951+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: daemon mds.e restarted 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: osdmap e125: 12 total, 12 up, 12 in 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:boot 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:29:05.959 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:05 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:29:06.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: daemon mds.e restarted 2024-04-12T21:29:06.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:06.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:29:06.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: osdmap e125: 12 total, 12 up, 12 in 2024-04-12T21:29:06.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:boot 2024-04-12T21:29:06.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:29:06.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:29:06.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:29:06.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:29:06.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:05 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:29:06.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: daemon mds.e restarted 2024-04-12T21:29:06.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:06.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:29:06.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: osdmap e125: 12 total, 12 up, 12 in 2024-04-12T21:29:06.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:boot 2024-04-12T21:29:06.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-12T21:29:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:29:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 7 up:standby, 1 failed 2024-04-12T21:29:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-12T21:29:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:05 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:replay} 7 up:standby 2024-04-12T21:29:06.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:05 smithi104 bash[72536]: 88b759dfa429d97b8d4cdb8e389c75e62698d61ecaa0e9f499937498ebba86f2 2024-04-12T21:29:06.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:05 smithi104 systemd[1]: Started Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:29:07.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:06 smithi153 ceph-mon[36569]: pgmap v829: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 25 KiB/s wr, 7 op/s 2024-04-12T21:29:07.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:06 smithi104 ceph-mon[36624]: pgmap v829: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 25 KiB/s wr, 7 op/s 2024-04-12T21:29:07.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:06 smithi081 ceph-mon[31453]: pgmap v829: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 25 KiB/s wr, 7 op/s 2024-04-12T21:29:08.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:reconnect 2024-04-12T21:29:08.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:07 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:29:08.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:07 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:29:08.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:reconnect 2024-04-12T21:29:08.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:07 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:29:08.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:07 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:29:08.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:reconnect 2024-04-12T21:29:08.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:07 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:reconnect} 7 up:standby 2024-04-12T21:29:08.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:07 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:29:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:08.364+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:29:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:08.364+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:29:08.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:08.364+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:29:09.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:08 smithi153 ceph-mon[36569]: pgmap v830: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:29:09.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:08 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:rejoin 2024-04-12T21:29:09.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:08 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:29:09.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:08 smithi104 ceph-mon[36624]: pgmap v830: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:29:09.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:08 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:rejoin 2024-04-12T21:29:09.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:08 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:29:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:08 smithi081 ceph-mon[31453]: pgmap v830: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:29:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:08 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:rejoin 2024-04-12T21:29:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:08 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:rejoin} 7 up:standby 2024-04-12T21:29:10.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:09 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:clientreplay 2024-04-12T21:29:10.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:09 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:29:10.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:09 smithi153 ceph-mon[36569]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:29:10.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:09 smithi153 ceph-mon[36569]: pgmap v831: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4 op/s 2024-04-12T21:29:10.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:09 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:clientreplay 2024-04-12T21:29:10.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:09 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:29:10.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:09 smithi104 ceph-mon[36624]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:29:10.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:09 smithi104 ceph-mon[36624]: pgmap v831: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4 op/s 2024-04-12T21:29:10.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:09 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:clientreplay 2024-04-12T21:29:10.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:09 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=f=up:clientreplay} 7 up:standby 2024-04-12T21:29:10.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:09 smithi081 ceph-mon[31453]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-12T21:29:10.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:09 smithi081 ceph-mon[31453]: pgmap v831: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 4 op/s 2024-04-12T21:29:11.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:11.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:active 2024-04-12T21:29:11.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:29:11.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:29:11.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:29:11.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: *** Caught signal (Aborted) ** 2024-04-12T21:29:11.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: in thread 7fbf3affe700 thread_name:ms_dispatch 2024-04-12T21:29:11.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:11.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf48bb8d20] 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2: gsignal() 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 3: abort() 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560263835268] 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5602638d10fc] 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560263c05f0a] 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 7: (MDLog::trim()+0xca9) [0x560263c070c9] 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560263c075a8] 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbf47aa4b23] 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbf48bae1ca] 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 11: clone() 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2024-04-12T21:29:10.662+0000 7fbf3affe700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: in thread 7fbf3affe700 thread_name:ms_dispatch 2024-04-12T21:29:11.056 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf48bb8d20] 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2: gsignal() 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 3: abort() 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560263835268] 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5602638d10fc] 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560263c05f0a] 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 7: (MDLog::trim()+0xca9) [0x560263c070c9] 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560263c075a8] 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbf47aa4b23] 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbf48bae1ca] 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 11: clone() 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:11.057 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 0> 2024-04-12T21:29:10.662+0000 7fbf3affe700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: in thread 7fbf3affe700 thread_name:ms_dispatch 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf48bb8d20] 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2: gsignal() 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 3: abort() 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560263835268] 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5602638d10fc] 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560263c05f0a] 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 7: (MDLog::trim()+0xca9) [0x560263c070c9] 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560263c075a8] 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbf47aa4b23] 2024-04-12T21:29:11.058 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbf48bae1ca] 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 11: clone() 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: -1> 2024-04-12T21:29:10.662+0000 7fbf3affe700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: in thread 7fbf3affe700 thread_name:ms_dispatch 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf48bb8d20] 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2: gsignal() 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 3: abort() 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560263835268] 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5602638d10fc] 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560263c05f0a] 2024-04-12T21:29:11.059 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 7: (MDLog::trim()+0xca9) [0x560263c070c9] 2024-04-12T21:29:11.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560263c075a8] 2024-04-12T21:29:11.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbf47aa4b23] 2024-04-12T21:29:11.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbf48bae1ca] 2024-04-12T21:29:11.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 11: clone() 2024-04-12T21:29:11.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:11.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f[74426]: 2024-04-12T21:29:11.060 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:10 smithi153 podman[74480]: 2024-04-12 21:29:10.85542026 +0000 UTC m=+0.030127311 container died 4ce54a5bb2547ec0adff6a656ca2d801efe3f58646d86550ef2550b74ebb760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-12T21:29:11.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:10 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:11.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:10 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:active 2024-04-12T21:29:11.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:10 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:29:11.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:10 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:29:11.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:10 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:11.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:10 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6834/2846296933,v1:172.21.15.153:6835/2846296933] up:active 2024-04-12T21:29:11.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:10 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 7 up:standby 2024-04-12T21:29:11.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:10 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=f=up:active} 1 up:standby-replay 6 up:standby 2024-04-12T21:29:12.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:11 smithi153 ceph-mon[36569]: pgmap v832: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 18 KiB/s wr, 9 op/s 2024-04-12T21:29:12.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:11 smithi153 podman[74480]: 2024-04-12 21:29:11.566325231 +0000 UTC m=+0.741032273 container remove 4ce54a5bb2547ec0adff6a656ca2d801efe3f58646d86550ef2550b74ebb760b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-f, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-12T21:29:12.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:11 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:29:12.054 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:11 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-12T21:29:12.055 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:11 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 24.696s CPU time. 2024-04-12T21:29:12.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:11 smithi104 ceph-mon[36624]: pgmap v832: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 18 KiB/s wr, 9 op/s 2024-04-12T21:29:12.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:11 smithi081 ceph-mon[31453]: pgmap v832: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 18 KiB/s wr, 9 op/s 2024-04-12T21:29:14.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:14 smithi104 ceph-mon[36624]: pgmap v833: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 18 KiB/s wr, 12 op/s 2024-04-12T21:29:14.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:14 smithi081 ceph-mon[31453]: pgmap v833: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 18 KiB/s wr, 12 op/s 2024-04-12T21:29:14.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:14 smithi153 ceph-mon[36569]: pgmap v833: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 18 KiB/s wr, 12 op/s 2024-04-12T21:29:16.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:16 smithi104 ceph-mon[36624]: pgmap v834: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-12T21:29:16.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:16 smithi081 ceph-mon[31453]: pgmap v834: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-12T21:29:16.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:16 smithi153 ceph-mon[36569]: pgmap v834: 129 pgs: 129 active+clean; 582 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 18 KiB/s wr, 14 op/s 2024-04-12T21:29:18.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:18 smithi081 ceph-mon[31453]: pgmap v835: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 74 KiB/s wr, 13 op/s 2024-04-12T21:29:18.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:18 smithi153 ceph-mon[36569]: pgmap v835: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 74 KiB/s wr, 13 op/s 2024-04-12T21:29:18.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:18 smithi104 ceph-mon[36624]: pgmap v835: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 74 KiB/s wr, 13 op/s 2024-04-12T21:29:20.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:20 smithi081 ceph-mon[31453]: pgmap v836: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 74 KiB/s wr, 12 op/s 2024-04-12T21:29:20.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:20 smithi153 ceph-mon[36569]: pgmap v836: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 74 KiB/s wr, 12 op/s 2024-04-12T21:29:20.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:20 smithi104 ceph-mon[36624]: pgmap v836: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 74 KiB/s wr, 12 op/s 2024-04-12T21:29:22.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:22 smithi081 ceph-mon[31453]: pgmap v837: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-12T21:29:22.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:22 smithi153 ceph-mon[36569]: pgmap v837: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-12T21:29:22.555 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:22 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:29:22.555 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:22 smithi153 systemd[1]: Stopped Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:29:22.555 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:22 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Consumed 24.696s CPU time. 2024-04-12T21:29:22.555 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:22 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Start request repeated too quickly. 2024-04-12T21:29:22.555 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:22 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-12T21:29:22.555 INFO:journalctl@ceph.mds.f.smithi153.stdout:Apr 12 21:29:22 smithi153 systemd[1]: Failed to start Ceph mds.f for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:29:22.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:22 smithi104 ceph-mon[36624]: pgmap v837: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 74 KiB/s wr, 14 op/s 2024-04-12T21:29:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:23.377+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:29:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:23.377+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:29:23.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:23.377+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:29:24.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:24 smithi081 ceph-mon[31453]: pgmap v838: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 59 KiB/s wr, 8 op/s 2024-04-12T21:29:24.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:24 smithi153 ceph-mon[36569]: pgmap v838: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 59 KiB/s wr, 8 op/s 2024-04-12T21:29:24.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:24 smithi104 ceph-mon[36624]: pgmap v838: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 59 KiB/s wr, 8 op/s 2024-04-12T21:29:26.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:26 smithi081 ceph-mon[31453]: pgmap v839: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 59 KiB/s wr, 5 op/s 2024-04-12T21:29:26.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:26 smithi153 ceph-mon[36569]: pgmap v839: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 59 KiB/s wr, 5 op/s 2024-04-12T21:29:26.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:26 smithi104 ceph-mon[36624]: pgmap v839: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 59 KiB/s wr, 5 op/s 2024-04-12T21:29:28.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:28 smithi081 ceph-mon[31453]: pgmap v840: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 59 KiB/s wr, 5 op/s 2024-04-12T21:29:28.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:28 smithi081 ceph-mon[31453]: Replacing daemon mds.f as rank 0 with standby daemon mds.e 2024-04-12T21:29:28.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:28 smithi081 ceph-mon[31453]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-12T21:29:28.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:28 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:28.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:28 smithi081 ceph-mon[31453]: osdmap e126: 12 total, 12 up, 12 in 2024-04-12T21:29:28.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:28 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:replay} 6 up:standby 2024-04-12T21:29:28.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:28 smithi153 ceph-mon[36569]: pgmap v840: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 59 KiB/s wr, 5 op/s 2024-04-12T21:29:28.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:28 smithi153 ceph-mon[36569]: Replacing daemon mds.f as rank 0 with standby daemon mds.e 2024-04-12T21:29:28.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:28 smithi153 ceph-mon[36569]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-12T21:29:28.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:28 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:28.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:28 smithi153 ceph-mon[36569]: osdmap e126: 12 total, 12 up, 12 in 2024-04-12T21:29:28.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:28 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:replay} 6 up:standby 2024-04-12T21:29:28.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:28 smithi104 ceph-mon[36624]: pgmap v840: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 59 KiB/s wr, 5 op/s 2024-04-12T21:29:28.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:28 smithi104 ceph-mon[36624]: Replacing daemon mds.f as rank 0 with standby daemon mds.e 2024-04-12T21:29:28.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:28 smithi104 ceph-mon[36624]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-04-12T21:29:28.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:28 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:28.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:28 smithi104 ceph-mon[36624]: osdmap e126: 12 total, 12 up, 12 in 2024-04-12T21:29:28.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:28 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:replay} 6 up:standby 2024-04-12T21:29:29.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:29 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:reconnect 2024-04-12T21:29:29.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:29 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:reconnect} 6 up:standby 2024-04-12T21:29:29.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:29 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:29:29.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:29 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:reconnect 2024-04-12T21:29:29.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:29 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:reconnect} 6 up:standby 2024-04-12T21:29:29.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:29 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:29:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:29 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:reconnect 2024-04-12T21:29:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:29 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:reconnect} 6 up:standby 2024-04-12T21:29:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:29 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:29:30.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:30 smithi153 ceph-mon[36569]: pgmap v842: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3 op/s 2024-04-12T21:29:30.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:30 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:rejoin 2024-04-12T21:29:30.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:30 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:rejoin} 6 up:standby 2024-04-12T21:29:30.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:30 smithi104 ceph-mon[36624]: pgmap v842: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3 op/s 2024-04-12T21:29:30.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:30 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:rejoin 2024-04-12T21:29:30.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:30 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:rejoin} 6 up:standby 2024-04-12T21:29:30.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:30 smithi081 ceph-mon[31453]: pgmap v842: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3 op/s 2024-04-12T21:29:30.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:30 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:rejoin 2024-04-12T21:29:30.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:30 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:rejoin} 6 up:standby 2024-04-12T21:29:31.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:31 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:clientreplay 2024-04-12T21:29:31.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:31 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=e=up:clientreplay} 6 up:standby 2024-04-12T21:29:31.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:31 smithi153 ceph-mon[36569]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:29:31.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:clientreplay 2024-04-12T21:29:31.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=e=up:clientreplay} 6 up:standby 2024-04-12T21:29:31.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-mon[36624]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:29:31.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:31 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:clientreplay 2024-04-12T21:29:31.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:31 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=e=up:clientreplay} 6 up:standby 2024-04-12T21:29:31.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:31 smithi081 ceph-mon[31453]: daemon mds.e is now active in filesystem cephfs as rank 0 2024-04-12T21:29:32.330 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: *** Caught signal (Aborted) ** 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: in thread 7f01a61bd700 thread_name:ms_dispatch 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01b3d77d20] 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2: gsignal() 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 3: abort() 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x556e688de268] 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x556e6897a0fc] 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x556e68caef0a] 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 7: (MDLog::trim()+0xca9) [0x556e68cb00c9] 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 8: (MDLog::log_trim_upkeep()+0x358) [0x556e68cb05a8] 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f01b2c63b23] 2024-04-12T21:29:32.331 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f01b3d6d1ca] 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 11: clone() 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2024-04-12T21:29:31.904+0000 7f01a61bd700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: in thread 7f01a61bd700 thread_name:ms_dispatch 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01b3d77d20] 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2: gsignal() 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 3: abort() 2024-04-12T21:29:32.332 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x556e688de268] 2024-04-12T21:29:32.333 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x556e6897a0fc] 2024-04-12T21:29:32.333 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x556e68caef0a] 2024-04-12T21:29:32.333 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 7: (MDLog::trim()+0xca9) [0x556e68cb00c9] 2024-04-12T21:29:32.333 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 8: (MDLog::log_trim_upkeep()+0x358) [0x556e68cb05a8] 2024-04-12T21:29:32.333 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f01b2c63b23] 2024-04-12T21:29:32.334 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f01b3d6d1ca] 2024-04-12T21:29:32.334 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 11: clone() 2024-04-12T21:29:32.334 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:32.334 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2024-04-12T21:29:32.334 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 0> 2024-04-12T21:29:31.904+0000 7f01a61bd700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:32.334 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: in thread 7f01a61bd700 thread_name:ms_dispatch 2024-04-12T21:29:32.334 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2024-04-12T21:29:32.334 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01b3d77d20] 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2: gsignal() 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 3: abort() 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x556e688de268] 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x556e6897a0fc] 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x556e68caef0a] 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 7: (MDLog::trim()+0xca9) [0x556e68cb00c9] 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 8: (MDLog::log_trim_upkeep()+0x358) [0x556e68cb05a8] 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f01b2c63b23] 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f01b3d6d1ca] 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 11: clone() 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:32.335 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: -10> 2024-04-12T21:29:31.904+0000 7f01a61bd700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: in thread 7f01a61bd700 thread_name:ms_dispatch 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f01b3d77d20] 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2: gsignal() 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 3: abort() 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x556e688de268] 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x556e6897a0fc] 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x556e68caef0a] 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 7: (MDLog::trim()+0xca9) [0x556e68cb00c9] 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 8: (MDLog::log_trim_upkeep()+0x358) [0x556e68cb05a8] 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f01b2c63b23] 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f01b3d6d1ca] 2024-04-12T21:29:32.336 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 11: clone() 2024-04-12T21:29:32.337 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:32.337 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e[72551]: 2024-04-12T21:29:32.337 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:31 smithi104 podman[72602]: 2024-04-12 21:29:31.968778045 +0000 UTC m=+0.019190213 container died 88b759dfa429d97b8d4cdb8e389c75e62698d61ecaa0e9f499937498ebba86f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-12T21:29:32.788 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:32 smithi104 ceph-mon[36624]: pgmap v843: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:29:32.788 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:32 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:32.789 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:32 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:active 2024-04-12T21:29:32.789 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:32 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 6 up:standby 2024-04-12T21:29:32.789 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:32 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 5 up:standby 2024-04-12T21:29:32.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:32 smithi153 ceph-mon[36569]: pgmap v843: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:29:32.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:32 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:32.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:32 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:active 2024-04-12T21:29:32.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:32 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 6 up:standby 2024-04-12T21:29:32.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:32 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 5 up:standby 2024-04-12T21:29:32.873 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:32 smithi081 ceph-mon[31453]: pgmap v843: 129 pgs: 129 active+clean; 583 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:29:32.873 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:32 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:32.873 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:32 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6834/3178911137,v1:172.21.15.104:6835/3178911137] up:active 2024-04-12T21:29:32.873 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:32 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 6 up:standby 2024-04-12T21:29:32.873 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:32 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=e=up:active} 1 up:standby-replay 5 up:standby 2024-04-12T21:29:33.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:32 smithi104 podman[72602]: 2024-04-12 21:29:32.791325028 +0000 UTC m=+0.841737193 container remove 88b759dfa429d97b8d4cdb8e389c75e62698d61ecaa0e9f499937498ebba86f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:29:33.080 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:32 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:29:33.580 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:33 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-12T21:29:33.580 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:33 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 22.868s CPU time. 2024-04-12T21:29:34.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:34 smithi153 ceph-mon[36569]: pgmap v844: 129 pgs: 129 active+clean; 581 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-04-12T21:29:34.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:34 smithi104 ceph-mon[36624]: pgmap v844: 129 pgs: 129 active+clean; 581 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-04-12T21:29:34.855 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:34 smithi081 ceph-mon[31453]: pgmap v844: 129 pgs: 129 active+clean; 581 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 1.8 KiB/s wr, 13 op/s 2024-04-12T21:29:36.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:36 smithi153 ceph-mon[36569]: pgmap v845: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.0 KiB/s wr, 21 op/s 2024-04-12T21:29:36.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:36 smithi104 ceph-mon[36624]: pgmap v845: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.0 KiB/s wr, 21 op/s 2024-04-12T21:29:36.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:36 smithi081 ceph-mon[31453]: pgmap v845: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.0 KiB/s wr, 21 op/s 2024-04-12T21:29:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:38.391+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:29:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:38.391+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:29:38.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:38.391+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:29:38.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:38 smithi153 ceph-mon[36569]: pgmap v846: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 42 KiB/s wr, 30 op/s 2024-04-12T21:29:38.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:38 smithi104 ceph-mon[36624]: pgmap v846: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 42 KiB/s wr, 30 op/s 2024-04-12T21:29:38.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:38 smithi081 ceph-mon[31453]: pgmap v846: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 42 KiB/s wr, 30 op/s 2024-04-12T21:29:39.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:29:39.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:29:39.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:29:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:40 smithi153 ceph-mon[36569]: pgmap v847: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 37 KiB/s wr, 26 op/s 2024-04-12T21:29:40.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:40 smithi104 ceph-mon[36624]: pgmap v847: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 37 KiB/s wr, 26 op/s 2024-04-12T21:29:40.872 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:40 smithi081 ceph-mon[31453]: pgmap v847: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 37 KiB/s wr, 26 op/s 2024-04-12T21:29:41.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:29:41.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:29:41.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:29:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:29:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:29:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:29:42.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:29:42.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:29:42.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:29:42.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:42 smithi104 ceph-mon[36624]: pgmap v848: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 36 KiB/s wr, 28 op/s 2024-04-12T21:29:42.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:42 smithi104 ceph-mon[36624]: pgmap v849: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 43 KiB/s wr, 30 op/s 2024-04-12T21:29:42.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:42 smithi081 ceph-mon[31453]: pgmap v848: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 36 KiB/s wr, 28 op/s 2024-04-12T21:29:42.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:42 smithi081 ceph-mon[31453]: pgmap v849: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 43 KiB/s wr, 30 op/s 2024-04-12T21:29:42.955 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:42 smithi153 ceph-mon[36569]: pgmap v848: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 36 KiB/s wr, 28 op/s 2024-04-12T21:29:42.956 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:42 smithi153 ceph-mon[36569]: pgmap v849: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 43 KiB/s wr, 30 op/s 2024-04-12T21:29:43.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:43 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:29:43.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:43 smithi104 systemd[1]: Stopped Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:29:43.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:43 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Consumed 22.868s CPU time. 2024-04-12T21:29:43.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:43 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Start request repeated too quickly. 2024-04-12T21:29:43.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:43 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-04-12T21:29:43.830 INFO:journalctl@ceph.mds.e.smithi104.stdout:Apr 12 21:29:43 smithi104 systemd[1]: Failed to start Ceph mds.e for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:29:44.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:44 smithi104 ceph-mon[36624]: pgmap v850: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 42 KiB/s wr, 21 op/s 2024-04-12T21:29:44.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:44 smithi081 ceph-mon[31453]: pgmap v850: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 42 KiB/s wr, 21 op/s 2024-04-12T21:29:45.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:44 smithi153 ceph-mon[36569]: pgmap v850: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 42 KiB/s wr, 21 op/s 2024-04-12T21:29:46.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:46 smithi104 ceph-mon[36624]: pgmap v851: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-12T21:29:46.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:46 smithi081 ceph-mon[31453]: pgmap v851: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-12T21:29:47.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:46 smithi153 ceph-mon[36569]: pgmap v851: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 42 KiB/s wr, 14 op/s 2024-04-12T21:29:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:47 smithi081 ceph-mon[31453]: Replacing daemon mds.e as rank 0 with standby daemon mds.k 2024-04-12T21:29:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:47 smithi081 ceph-mon[31453]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-12T21:29:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:47 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:47 smithi081 ceph-mon[31453]: osdmap e127: 12 total, 12 up, 12 in 2024-04-12T21:29:47.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:47 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:29:48.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:47 smithi153 ceph-mon[36569]: Replacing daemon mds.e as rank 0 with standby daemon mds.k 2024-04-12T21:29:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:47 smithi153 ceph-mon[36569]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-12T21:29:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:47 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:47 smithi153 ceph-mon[36569]: osdmap e127: 12 total, 12 up, 12 in 2024-04-12T21:29:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:47 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:29:48.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:47 smithi104 ceph-mon[36624]: Replacing daemon mds.e as rank 0 with standby daemon mds.k 2024-04-12T21:29:48.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:47 smithi104 ceph-mon[36624]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-04-12T21:29:48.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:47 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:29:48.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:47 smithi104 ceph-mon[36624]: osdmap e127: 12 total, 12 up, 12 in 2024-04-12T21:29:48.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:47 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:29:48.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:48 smithi081 ceph-mon[31453]: pgmap v852: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:29:48.981 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:48 smithi104 ceph-mon[36624]: pgmap v852: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:29:49.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:48 smithi153 ceph-mon[36569]: pgmap v852: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:29:49.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:49 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:reconnect 2024-04-12T21:29:49.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:49 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:29:49.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:49 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0149998 2024-04-12T21:29:50.001 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:49 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:reconnect 2024-04-12T21:29:50.002 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:49 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:29:50.002 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:49 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0149998 2024-04-12T21:29:50.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:49 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:reconnect 2024-04-12T21:29:50.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:49 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:29:50.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:49 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0149998 2024-04-12T21:29:50.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:50 smithi104 ceph-mon[36624]: pgmap v854: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 2 op/s 2024-04-12T21:29:50.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:50 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:rejoin 2024-04-12T21:29:50.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:50 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:29:50.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:50 smithi081 ceph-mon[31453]: pgmap v854: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 2 op/s 2024-04-12T21:29:50.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:50 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:rejoin 2024-04-12T21:29:50.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:50 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:29:51.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:50 smithi153 ceph-mon[36569]: pgmap v854: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 2 op/s 2024-04-12T21:29:51.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:50 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:rejoin 2024-04-12T21:29:51.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:50 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:29:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:51 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:clientreplay 2024-04-12T21:29:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:51 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:29:51.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:51 smithi081 ceph-mon[31453]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:29:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:51 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:clientreplay 2024-04-12T21:29:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:51 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:29:52.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:51 smithi153 ceph-mon[36569]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:29:52.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:51 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:clientreplay 2024-04-12T21:29:52.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:51 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:29:52.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:51 smithi104 ceph-mon[36624]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:29:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-mon[36624]: pgmap v855: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:29:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:active 2024-04-12T21:29:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:29:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:29:52.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:29:52.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: *** Caught signal (Aborted) ** 2024-04-12T21:29:52.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: in thread 7f379ac58700 thread_name:ms_dispatch 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f37a8812d20] 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2: gsignal() 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 3: abort() 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563396921268] 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5633969bd0fc] 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563396cf1f0a] 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 7: (MDLog::trim()+0xca9) [0x563396cf30c9] 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563396cf35a8] 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f37a76feb23] 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f37a88081ca] 2024-04-12T21:29:52.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 11: clone() 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2024-04-12T21:29:52.544+0000 7f379ac58700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: in thread 7f379ac58700 thread_name:ms_dispatch 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f37a8812d20] 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2: gsignal() 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 3: abort() 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563396921268] 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5633969bd0fc] 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563396cf1f0a] 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 7: (MDLog::trim()+0xca9) [0x563396cf30c9] 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563396cf35a8] 2024-04-12T21:29:52.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f37a76feb23] 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f37a88081ca] 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 11: clone() 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 0> 2024-04-12T21:29:52.544+0000 7f379ac58700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: in thread 7f379ac58700 thread_name:ms_dispatch 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f37a8812d20] 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2: gsignal() 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 3: abort() 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563396921268] 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5633969bd0fc] 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563396cf1f0a] 2024-04-12T21:29:52.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 7: (MDLog::trim()+0xca9) [0x563396cf30c9] 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563396cf35a8] 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f37a76feb23] 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f37a88081ca] 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 11: clone() 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: -4> 2024-04-12T21:29:52.544+0000 7f379ac58700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: in thread 7f379ac58700 thread_name:ms_dispatch 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f37a8812d20] 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2: gsignal() 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 3: abort() 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563396921268] 2024-04-12T21:29:52.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5633969bd0fc] 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563396cf1f0a] 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 7: (MDLog::trim()+0xca9) [0x563396cf30c9] 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563396cf35a8] 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f37a76feb23] 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f37a88081ca] 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 11: clone() 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[65291]: 2024-04-12T21:29:52.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:52 smithi104 podman[72792]: 2024-04-12 21:29:52.605799803 +0000 UTC m=+0.020001612 container died 2e08d3c1310f9c38e9f7c06ec4ed37b502a9cb6ed80255ed61c6312877f417e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-12T21:29:52.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:52 smithi081 ceph-mon[31453]: pgmap v855: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:29:52.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:52 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:52.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:52 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:active 2024-04-12T21:29:52.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:52 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:29:52.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:52 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:29:53.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:52 smithi153 ceph-mon[36569]: pgmap v855: 129 pgs: 129 active+clean; 580 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:29:53.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:52 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:29:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:52 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1562734606,v1:172.21.15.104:6833/1562734606] up:active 2024-04-12T21:29:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:52 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:29:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:52 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:29:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:53.399+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:29:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:53.399+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:29:53.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:29:53.400+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:29:53.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:53 smithi104 podman[72792]: 2024-04-12 21:29:53.295536999 +0000 UTC m=+0.709738803 container remove 2e08d3c1310f9c38e9f7c06ec4ed37b502a9cb6ed80255ed61c6312877f417e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-12T21:29:53.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:29:53.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-12T21:29:53.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:29:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.526s CPU time. 2024-04-12T21:29:54.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:54 smithi081 ceph-mon[31453]: pgmap v856: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-12T21:29:55.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:54 smithi153 ceph-mon[36569]: pgmap v856: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-12T21:29:55.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:54 smithi104 ceph-mon[36624]: pgmap v856: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-12T21:29:56.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:56 smithi081 ceph-mon[31453]: pgmap v857: 129 pgs: 129 active+clean; 578 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-12T21:29:57.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:56 smithi153 ceph-mon[36569]: pgmap v857: 129 pgs: 129 active+clean; 578 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-12T21:29:57.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:56 smithi104 ceph-mon[36624]: pgmap v857: 129 pgs: 129 active+clean; 578 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-12T21:29:58.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:29:58 smithi081 ceph-mon[31453]: pgmap v858: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 98 KiB/s wr, 28 op/s 2024-04-12T21:29:59.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:29:58 smithi153 ceph-mon[36569]: pgmap v858: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 98 KiB/s wr, 28 op/s 2024-04-12T21:29:59.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:29:58 smithi104 ceph-mon[36624]: pgmap v858: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 98 KiB/s wr, 28 op/s 2024-04-12T21:30:01.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:00 smithi153 ceph-mon[36569]: pgmap v859: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 87 KiB/s wr, 25 op/s 2024-04-12T21:30:01.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:00 smithi153 ceph-mon[36569]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s) 2024-04-12T21:30:01.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:00 smithi153 ceph-mon[36569]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-04-12T21:30:01.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:00 smithi153 ceph-mon[36569]: daemon mds.h on smithi104 is in error state 2024-04-12T21:30:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:00 smithi153 ceph-mon[36569]: daemon mds.b on smithi104 is in error state 2024-04-12T21:30:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:00 smithi153 ceph-mon[36569]: daemon mds.c on smithi153 is in error state 2024-04-12T21:30:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:00 smithi153 ceph-mon[36569]: daemon mds.i on smithi153 is in error state 2024-04-12T21:30:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:00 smithi104 ceph-mon[36624]: pgmap v859: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 87 KiB/s wr, 25 op/s 2024-04-12T21:30:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:00 smithi104 ceph-mon[36624]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s) 2024-04-12T21:30:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:00 smithi104 ceph-mon[36624]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-04-12T21:30:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:00 smithi104 ceph-mon[36624]: daemon mds.h on smithi104 is in error state 2024-04-12T21:30:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:00 smithi104 ceph-mon[36624]: daemon mds.b on smithi104 is in error state 2024-04-12T21:30:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:00 smithi104 ceph-mon[36624]: daemon mds.c on smithi153 is in error state 2024-04-12T21:30:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:00 smithi104 ceph-mon[36624]: daemon mds.i on smithi153 is in error state 2024-04-12T21:30:01.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:00 smithi081 ceph-mon[31453]: pgmap v859: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 87 KiB/s wr, 25 op/s 2024-04-12T21:30:01.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:00 smithi081 ceph-mon[31453]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s) 2024-04-12T21:30:01.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:00 smithi081 ceph-mon[31453]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-04-12T21:30:01.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:00 smithi081 ceph-mon[31453]: daemon mds.h on smithi104 is in error state 2024-04-12T21:30:01.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:00 smithi081 ceph-mon[31453]: daemon mds.b on smithi104 is in error state 2024-04-12T21:30:01.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:00 smithi081 ceph-mon[31453]: daemon mds.c on smithi153 is in error state 2024-04-12T21:30:01.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:00 smithi081 ceph-mon[31453]: daemon mds.i on smithi153 is in error state 2024-04-12T21:30:03.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:02 smithi153 ceph-mon[36569]: pgmap v860: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 81 KiB/s wr, 25 op/s 2024-04-12T21:30:03.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:02 smithi104 ceph-mon[36624]: pgmap v860: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 81 KiB/s wr, 25 op/s 2024-04-12T21:30:03.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:02 smithi081 ceph-mon[31453]: pgmap v860: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 81 KiB/s wr, 25 op/s 2024-04-12T21:30:03.899 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:03 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:30:03.900 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:03 smithi104 systemd[1]: Stopped Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:30:03.900 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:03 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.526s CPU time. 2024-04-12T21:30:03.900 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:03 smithi104 systemd[1]: Starting Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:30:03.900 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:03 smithi104 podman[72886]: 2024-04-12 21:30:03.719282972 +0000 UTC m=+0.017984427 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:30:04.221 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:03 smithi104 podman[72886]: 2024-04-12 21:30:03.902337234 +0000 UTC m=+0.201038680 container create 9a57e799fb191fc8f1658a4b46445c38f0fb2b5cf3f66e234066d5cedd73bc18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:30:04.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:04 smithi104 podman[72886]: 2024-04-12 21:30:04.223908843 +0000 UTC m=+0.522610292 container init 9a57e799fb191fc8f1658a4b46445c38f0fb2b5cf3f66e234066d5cedd73bc18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True) 2024-04-12T21:30:04.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:04 smithi104 podman[72886]: 2024-04-12 21:30:04.2269473 +0000 UTC m=+0.525648755 container start 9a57e799fb191fc8f1658a4b46445c38f0fb2b5cf3f66e234066d5cedd73bc18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2) 2024-04-12T21:30:04.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mds[72905]: -- 172.21.15.104:0/2397656750 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558235b2ba40 con 0x558235afe800 2024-04-12T21:30:04.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: starting mds.k at 2024-04-12T21:30:04.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:04 smithi104 bash[72886]: 9a57e799fb191fc8f1658a4b46445c38f0fb2b5cf3f66e234066d5cedd73bc18 2024-04-12T21:30:04.580 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:04 smithi104 systemd[1]: Started Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:30:04.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:30:04.266+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: pgmap v861: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 81 KiB/s wr, 25 op/s 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: daemon mds.k restarted 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: osdmap e128: 12 total, 12 up, 12 in 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:boot 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:30:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:04 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:30:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: pgmap v861: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 81 KiB/s wr, 25 op/s 2024-04-12T21:30:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: daemon mds.k restarted 2024-04-12T21:30:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: osdmap e128: 12 total, 12 up, 12 in 2024-04-12T21:30:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:boot 2024-04-12T21:30:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:30:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:30:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:04 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:30:05.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: pgmap v861: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 81 KiB/s wr, 25 op/s 2024-04-12T21:30:05.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: daemon mds.k restarted 2024-04-12T21:30:05.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: osdmap e128: 12 total, 12 up, 12 in 2024-04-12T21:30:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:boot 2024-04-12T21:30:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:30:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:30:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:04 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:30:07.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:06 smithi153 ceph-mon[36569]: pgmap v863: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 96 KiB/s wr, 16 op/s 2024-04-12T21:30:07.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:06 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:reconnect 2024-04-12T21:30:07.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:06 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:30:07.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:06 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139998 2024-04-12T21:30:07.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:06 smithi104 ceph-mon[36624]: pgmap v863: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 96 KiB/s wr, 16 op/s 2024-04-12T21:30:07.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:06 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:reconnect 2024-04-12T21:30:07.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:06 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:30:07.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:06 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139998 2024-04-12T21:30:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:06 smithi081 ceph-mon[31453]: pgmap v863: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 96 KiB/s wr, 16 op/s 2024-04-12T21:30:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:06 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:reconnect 2024-04-12T21:30:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:06 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:30:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:06 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139998 2024-04-12T21:30:08.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:rejoin 2024-04-12T21:30:08.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:07 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:30:08.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:rejoin 2024-04-12T21:30:08.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:07 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:30:08.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:rejoin 2024-04-12T21:30:08.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:07 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:30:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:08.411+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:30:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:08.411+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:30:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:08.411+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:30:09.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:08 smithi153 ceph-mon[36569]: pgmap v864: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 5 op/s 2024-04-12T21:30:09.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:08 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:clientreplay 2024-04-12T21:30:09.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:08 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:30:09.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:08 smithi153 ceph-mon[36569]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:30:09.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:08 smithi104 ceph-mon[36624]: pgmap v864: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 5 op/s 2024-04-12T21:30:09.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:08 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:clientreplay 2024-04-12T21:30:09.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:08 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:30:09.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:08 smithi104 ceph-mon[36624]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:30:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:08 smithi081 ceph-mon[31453]: pgmap v864: 129 pgs: 129 active+clean; 579 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 5 op/s 2024-04-12T21:30:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:08 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:clientreplay 2024-04-12T21:30:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:08 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:30:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:08 smithi081 ceph-mon[31453]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:30:10.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:10.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:active 2024-04-12T21:30:10.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:30:10.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:10.040 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:30:10.040 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: *** Caught signal (Aborted) ** 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: in thread 7f696c221700 thread_name:ms_dispatch 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6979ddbd20] 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2: gsignal() 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 3: abort() 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562d20206268] 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562d202a20fc] 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562d205d6f0a] 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 7: (MDLog::trim()+0xca9) [0x562d205d80c9] 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562d205d85a8] 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6978cc7b23] 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f6979dd11ca] 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 11: clone() 2024-04-12T21:30:10.041 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2024-04-12T21:30:09.651+0000 7f696c221700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: in thread 7f696c221700 thread_name:ms_dispatch 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6979ddbd20] 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2: gsignal() 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 3: abort() 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562d20206268] 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562d202a20fc] 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562d205d6f0a] 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 7: (MDLog::trim()+0xca9) [0x562d205d80c9] 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562d205d85a8] 2024-04-12T21:30:10.042 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6978cc7b23] 2024-04-12T21:30:10.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f6979dd11ca] 2024-04-12T21:30:10.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 11: clone() 2024-04-12T21:30:10.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:10.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2024-04-12T21:30:10.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 0> 2024-04-12T21:30:09.651+0000 7f696c221700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:10.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: in thread 7f696c221700 thread_name:ms_dispatch 2024-04-12T21:30:10.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2024-04-12T21:30:10.043 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6979ddbd20] 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2: gsignal() 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 3: abort() 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562d20206268] 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562d202a20fc] 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562d205d6f0a] 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 7: (MDLog::trim()+0xca9) [0x562d205d80c9] 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562d205d85a8] 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6978cc7b23] 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f6979dd11ca] 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 11: clone() 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: -11> 2024-04-12T21:30:09.651+0000 7f696c221700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:10.044 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: in thread 7f696c221700 thread_name:ms_dispatch 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6979ddbd20] 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2: gsignal() 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 3: abort() 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562d20206268] 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562d202a20fc] 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562d205d6f0a] 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 7: (MDLog::trim()+0xca9) [0x562d205d80c9] 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562d205d85a8] 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6978cc7b23] 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f6979dd11ca] 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 11: clone() 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:10.045 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[67298]: 2024-04-12T21:30:10.046 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:09 smithi153 podman[74673]: 2024-04-12 21:30:09.794730107 +0000 UTC m=+0.037485582 container died 2811ba92030226bf07edc1dc711f8ad196c0c5fe83846d7906618ec0439ac606 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-12T21:30:10.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:09 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:10.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:09 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:active 2024-04-12T21:30:10.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:09 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:30:10.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:09 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:10.127 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:09 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:10.127 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:09 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/2327114952,v1:172.21.15.153:6833/2327114952] up:active 2024-04-12T21:30:10.127 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:09 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:30:10.127 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:09 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:10.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:10 smithi081 ceph-mon[31453]: pgmap v865: 129 pgs: 129 active+clean; 578 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:30:11.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:10 smithi153 ceph-mon[36569]: pgmap v865: 129 pgs: 129 active+clean; 578 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:30:11.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:10 smithi153 podman[74673]: 2024-04-12 21:30:10.876711686 +0000 UTC m=+1.119467139 container remove 2811ba92030226bf07edc1dc711f8ad196c0c5fe83846d7906618ec0439ac606 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-12T21:30:11.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:10 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:30:11.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:10 smithi104 ceph-mon[36624]: pgmap v865: 129 pgs: 129 active+clean; 578 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:30:11.554 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:11 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-12T21:30:11.555 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:11 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.723s CPU time. 2024-04-12T21:30:13.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:12 smithi153 ceph-mon[36569]: pgmap v866: 129 pgs: 129 active+clean; 575 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 818 B/s wr, 29 op/s 2024-04-12T21:30:13.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:12 smithi104 ceph-mon[36624]: pgmap v866: 129 pgs: 129 active+clean; 575 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 818 B/s wr, 29 op/s 2024-04-12T21:30:13.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:12 smithi081 ceph-mon[31453]: pgmap v866: 129 pgs: 129 active+clean; 575 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 818 B/s wr, 29 op/s 2024-04-12T21:30:15.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:14 smithi153 ceph-mon[36569]: pgmap v867: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 113 KiB/s wr, 47 op/s 2024-04-12T21:30:15.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:14 smithi104 ceph-mon[36624]: pgmap v867: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 113 KiB/s wr, 47 op/s 2024-04-12T21:30:15.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:14 smithi081 ceph-mon[31453]: pgmap v867: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 113 KiB/s wr, 47 op/s 2024-04-12T21:30:17.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:16 smithi153 ceph-mon[36569]: pgmap v868: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 108 KiB/s wr, 46 op/s 2024-04-12T21:30:17.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:16 smithi104 ceph-mon[36624]: pgmap v868: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 108 KiB/s wr, 46 op/s 2024-04-12T21:30:17.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:16 smithi081 ceph-mon[31453]: pgmap v868: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 108 KiB/s wr, 46 op/s 2024-04-12T21:30:18.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:17 smithi153 ceph-mon[36569]: pgmap v869: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 94 KiB/s wr, 42 op/s 2024-04-12T21:30:18.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:17 smithi104 ceph-mon[36624]: pgmap v869: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 94 KiB/s wr, 42 op/s 2024-04-12T21:30:18.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:17 smithi081 ceph-mon[31453]: pgmap v869: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 94 KiB/s wr, 42 op/s 2024-04-12T21:30:20.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:19 smithi153 ceph-mon[36569]: pgmap v870: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 94 KiB/s wr, 40 op/s 2024-04-12T21:30:20.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:19 smithi104 ceph-mon[36624]: pgmap v870: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 94 KiB/s wr, 40 op/s 2024-04-12T21:30:20.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:19 smithi081 ceph-mon[31453]: pgmap v870: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 94 KiB/s wr, 40 op/s 2024-04-12T21:30:21.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:30:21.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 systemd[1]: Stopped Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:30:21.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.723s CPU time. 2024-04-12T21:30:21.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 systemd[1]: Starting Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:30:21.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 podman[74782]: 2024-04-12 21:30:21.453640377 +0000 UTC m=+0.018566948 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:30:21.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 podman[74782]: 2024-04-12 21:30:21.585521491 +0000 UTC m=+0.150448057 container create 6b8fa148a723023df75eca7fb9bdd875715b75f013fd49b8f0531f270c1d295d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=) 2024-04-12T21:30:21.844 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:21 smithi153 ceph-mon[36569]: pgmap v871: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 94 KiB/s wr, 37 op/s 2024-04-12T21:30:22.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:21 smithi104 ceph-mon[36624]: pgmap v871: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 94 KiB/s wr, 37 op/s 2024-04-12T21:30:22.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:21 smithi081 ceph-mon[31453]: pgmap v871: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 94 KiB/s wr, 37 op/s 2024-04-12T21:30:22.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:30:21.896+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:22.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 podman[74782]: 2024-04-12 21:30:21.84681133 +0000 UTC m=+0.411737897 container init 6b8fa148a723023df75eca7fb9bdd875715b75f013fd49b8f0531f270c1d295d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T21:30:22.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 podman[74782]: 2024-04-12 21:30:21.849655097 +0000 UTC m=+0.414581663 container start 6b8fa148a723023df75eca7fb9bdd875715b75f013fd49b8f0531f270c1d295d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-12T21:30:22.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 ceph-mds[74802]: -- 172.21.15.153:0/1390381453 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5605119bfa40 con 0x560511992c00 2024-04-12T21:30:22.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: starting mds.l at 2024-04-12T21:30:22.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 bash[74782]: 6b8fa148a723023df75eca7fb9bdd875715b75f013fd49b8f0531f270c1d295d 2024-04-12T21:30:22.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:21 smithi153 systemd[1]: Started Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: daemon mds.l restarted 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: osdmap e129: 12 total, 12 up, 12 in 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:boot 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:30:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:22 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:30:23.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: daemon mds.l restarted 2024-04-12T21:30:23.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:23.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:23.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: osdmap e129: 12 total, 12 up, 12 in 2024-04-12T21:30:23.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:boot 2024-04-12T21:30:23.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:30:23.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:23.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:30:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:22 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:30:23.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: daemon mds.l restarted 2024-04-12T21:30:23.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:23.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:23.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: osdmap e129: 12 total, 12 up, 12 in 2024-04-12T21:30:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:boot 2024-04-12T21:30:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:30:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:30:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:22 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:30:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:23.426+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:30:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:23.426+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:30:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:23.426+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:30:24.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:23 smithi153 ceph-mon[36569]: pgmap v873: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:30:24.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:23 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:reconnect 2024-04-12T21:30:24.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:23 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:30:24.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:23 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169998 2024-04-12T21:30:24.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:23 smithi104 ceph-mon[36624]: pgmap v873: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:30:24.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:23 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:reconnect 2024-04-12T21:30:24.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:23 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:30:24.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:23 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169998 2024-04-12T21:30:24.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:23 smithi081 ceph-mon[31453]: pgmap v873: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:30:24.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:23 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:reconnect 2024-04-12T21:30:24.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:23 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:30:24.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:23 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169998 2024-04-12T21:30:25.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:24 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:rejoin 2024-04-12T21:30:25.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:24 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:30:25.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:24 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:rejoin 2024-04-12T21:30:25.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:24 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:30:25.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:24 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:rejoin 2024-04-12T21:30:25.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:24 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:30:26.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:25 smithi153 ceph-mon[36569]: pgmap v874: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:30:26.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:clientreplay 2024-04-12T21:30:26.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:25 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:30:26.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:25 smithi153 ceph-mon[36569]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:30:26.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:25 smithi104 ceph-mon[36624]: pgmap v874: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:30:26.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:clientreplay 2024-04-12T21:30:26.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:25 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:30:26.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:25 smithi104 ceph-mon[36624]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:30:26.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:25 smithi081 ceph-mon[31453]: pgmap v874: 129 pgs: 129 active+clean; 574 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:30:26.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:clientreplay 2024-04-12T21:30:26.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:25 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:30:26.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:25 smithi081 ceph-mon[31453]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:30:27.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:27.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:active 2024-04-12T21:30:27.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:30:27.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: *** Caught signal (Aborted) ** 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: in thread 7fb20f975700 thread_name:ms_dispatch 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb21d52fd20] 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2: gsignal() 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 3: abort() 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558234ae8268] 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558234b840fc] 2024-04-12T21:30:27.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558234eb8f0a] 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 7: (MDLog::trim()+0xca9) [0x558234eba0c9] 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558234eba5a8] 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb21c41bb23] 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb21d5251ca] 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 11: clone() 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2024-04-12T21:30:26.730+0000 7fb20f975700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: in thread 7fb20f975700 thread_name:ms_dispatch 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb21d52fd20] 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2: gsignal() 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 3: abort() 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558234ae8268] 2024-04-12T21:30:27.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558234b840fc] 2024-04-12T21:30:27.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558234eb8f0a] 2024-04-12T21:30:27.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 7: (MDLog::trim()+0xca9) [0x558234eba0c9] 2024-04-12T21:30:27.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558234eba5a8] 2024-04-12T21:30:27.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb21c41bb23] 2024-04-12T21:30:27.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb21d5251ca] 2024-04-12T21:30:27.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 11: clone() 2024-04-12T21:30:27.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 0> 2024-04-12T21:30:26.730+0000 7fb20f975700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: in thread 7fb20f975700 thread_name:ms_dispatch 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb21d52fd20] 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2: gsignal() 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 3: abort() 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558234ae8268] 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558234b840fc] 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558234eb8f0a] 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 7: (MDLog::trim()+0xca9) [0x558234eba0c9] 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558234eba5a8] 2024-04-12T21:30:27.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb21c41bb23] 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb21d5251ca] 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 11: clone() 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: -10> 2024-04-12T21:30:26.730+0000 7fb20f975700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: in thread 7fb20f975700 thread_name:ms_dispatch 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb21d52fd20] 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2: gsignal() 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 3: abort() 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558234ae8268] 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558234b840fc] 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558234eb8f0a] 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 7: (MDLog::trim()+0xca9) [0x558234eba0c9] 2024-04-12T21:30:27.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558234eba5a8] 2024-04-12T21:30:27.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb21c41bb23] 2024-04-12T21:30:27.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb21d5251ca] 2024-04-12T21:30:27.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 11: clone() 2024-04-12T21:30:27.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:27.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[72901]: 2024-04-12T21:30:27.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:26 smithi104 podman[72941]: 2024-04-12 21:30:26.790382629 +0000 UTC m=+0.018888656 container died 9a57e799fb191fc8f1658a4b46445c38f0fb2b5cf3f66e234066d5cedd73bc18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-12T21:30:27.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:26 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:26 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:active 2024-04-12T21:30:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:26 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:30:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:26 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:27.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:26 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:27.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:26 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/3260678460,v1:172.21.15.104:6833/3260678460] up:active 2024-04-12T21:30:27.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:26 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:30:27.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:26 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:27.977 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:27 smithi104 podman[72941]: 2024-04-12 21:30:27.610580048 +0000 UTC m=+0.839086108 container remove 9a57e799fb191fc8f1658a4b46445c38f0fb2b5cf3f66e234066d5cedd73bc18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux ) 2024-04-12T21:30:27.977 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:27 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:30:27.977 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:27 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-12T21:30:27.977 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:27 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.279s CPU time. 2024-04-12T21:30:28.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:27 smithi153 ceph-mon[36569]: pgmap v875: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.0 KiB/s wr, 22 op/s 2024-04-12T21:30:28.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:27 smithi104 ceph-mon[36624]: pgmap v875: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.0 KiB/s wr, 22 op/s 2024-04-12T21:30:28.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:27 smithi081 ceph-mon[31453]: pgmap v875: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.0 KiB/s wr, 22 op/s 2024-04-12T21:30:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:29 smithi153 ceph-mon[36569]: pgmap v876: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.0 KiB/s wr, 23 op/s 2024-04-12T21:30:30.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:29 smithi104 ceph-mon[36624]: pgmap v876: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.0 KiB/s wr, 23 op/s 2024-04-12T21:30:30.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:29 smithi081 ceph-mon[31453]: pgmap v876: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 2.0 KiB/s wr, 23 op/s 2024-04-12T21:30:32.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:31 smithi153 ceph-mon[36569]: pgmap v877: 129 pgs: 129 active+clean; 570 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.8 KiB/s wr, 29 op/s 2024-04-12T21:30:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:31 smithi104 ceph-mon[36624]: pgmap v877: 129 pgs: 129 active+clean; 570 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.8 KiB/s wr, 29 op/s 2024-04-12T21:30:32.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:31 smithi081 ceph-mon[31453]: pgmap v877: 129 pgs: 129 active+clean; 570 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.8 KiB/s wr, 29 op/s 2024-04-12T21:30:34.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:33 smithi153 ceph-mon[36569]: pgmap v878: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 77 KiB/s wr, 28 op/s 2024-04-12T21:30:34.348 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:33 smithi104 ceph-mon[36624]: pgmap v878: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 77 KiB/s wr, 28 op/s 2024-04-12T21:30:34.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:33 smithi081 ceph-mon[31453]: pgmap v878: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 77 KiB/s wr, 28 op/s 2024-04-12T21:30:36.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:35 smithi153 ceph-mon[36569]: pgmap v879: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 70 KiB/s wr, 26 op/s 2024-04-12T21:30:36.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:36 smithi104 ceph-mon[36624]: pgmap v879: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 70 KiB/s wr, 26 op/s 2024-04-12T21:30:36.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:35 smithi081 ceph-mon[31453]: pgmap v879: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 70 KiB/s wr, 26 op/s 2024-04-12T21:30:38.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:38 smithi153 ceph-mon[36569]: pgmap v880: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 70 KiB/s wr, 28 op/s 2024-04-12T21:30:38.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:38 smithi104 ceph-mon[36624]: pgmap v880: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 70 KiB/s wr, 28 op/s 2024-04-12T21:30:38.312 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:30:38.312 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 systemd[1]: Stopped Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:30:38.313 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.279s CPU time. 2024-04-12T21:30:38.313 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 systemd[1]: Starting Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:30:38.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:38 smithi081 ceph-mon[31453]: pgmap v880: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 70 KiB/s wr, 28 op/s 2024-04-12T21:30:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:38.442+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:30:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:38.442+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:30:38.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:38.442+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:30:38.577 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 podman[73038]: 2024-04-12 21:30:38.217558395 +0000 UTC m=+0.017877583 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:30:38.577 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 podman[73038]: 2024-04-12 21:30:38.327154295 +0000 UTC m=+0.127473456 container create 99077df301dfadb08e7787371d241463e3ec2d7f9320ff53bdd3e00d70c8ed78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:30:38.577 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 podman[73038]: 2024-04-12 21:30:38.554479561 +0000 UTC m=+0.354798733 container init 99077df301dfadb08e7787371d241463e3ec2d7f9320ff53bdd3e00d70c8ed78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-12T21:30:38.578 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 podman[73038]: 2024-04-12 21:30:38.557794782 +0000 UTC m=+0.358113943 container start 99077df301dfadb08e7787371d241463e3ec2d7f9320ff53bdd3e00d70c8ed78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-12T21:30:38.830 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 ceph-mds[73057]: -- 172.21.15.104:0/2228501063 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5642f5151a40 con 0x5642f5124c00 2024-04-12T21:30:38.830 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: starting mds.k at 2024-04-12T21:30:38.830 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 bash[73038]: 99077df301dfadb08e7787371d241463e3ec2d7f9320ff53bdd3e00d70c8ed78 2024-04-12T21:30:38.830 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:30:38 smithi104 systemd[1]: Started Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:30:38.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:38 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:30:38.597+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: daemon mds.k restarted 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: osdmap e130: 12 total, 12 up, 12 in 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:boot 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:30:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:39 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:30:39.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: daemon mds.k restarted 2024-04-12T21:30:39.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:39.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:39.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: osdmap e130: 12 total, 12 up, 12 in 2024-04-12T21:30:39.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:boot 2024-04-12T21:30:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:30:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:30:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:39 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: daemon mds.k restarted 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: osdmap e130: 12 total, 12 up, 12 in 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:boot 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:30:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:39 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:30:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:40 smithi081 ceph-mon[31453]: pgmap v882: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 82 KiB/s wr, 12 op/s 2024-04-12T21:30:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:40 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:reconnect 2024-04-12T21:30:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:40 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:30:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:40 smithi153 ceph-mon[36569]: pgmap v882: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 82 KiB/s wr, 12 op/s 2024-04-12T21:30:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:40 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:reconnect 2024-04-12T21:30:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:40 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:30:41.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:40 smithi104 ceph-mon[36624]: pgmap v882: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 82 KiB/s wr, 12 op/s 2024-04-12T21:30:41.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:40 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:reconnect 2024-04-12T21:30:41.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:40 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:30:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:41 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0179997 2024-04-12T21:30:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:41 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:rejoin 2024-04-12T21:30:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:41 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:30:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:30:42.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:41 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0179997 2024-04-12T21:30:42.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:41 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:rejoin 2024-04-12T21:30:42.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:41 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:30:42.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:30:42.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:41 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0179997 2024-04-12T21:30:42.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:41 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:rejoin 2024-04-12T21:30:42.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:41 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:30:42.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:30:42.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:42 smithi081 ceph-mon[31453]: pgmap v883: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 81 KiB/s wr, 6 op/s 2024-04-12T21:30:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:42 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:clientreplay 2024-04-12T21:30:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:42 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:30:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:42 smithi081 ceph-mon[31453]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:30:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:42 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:42 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:active 2024-04-12T21:30:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:42 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:42.977 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-mon[36569]: pgmap v883: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 81 KiB/s wr, 6 op/s 2024-04-12T21:30:42.978 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:clientreplay 2024-04-12T21:30:42.978 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:30:42.978 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-mon[36569]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:30:42.978 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:42.978 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:active 2024-04-12T21:30:42.978 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:43.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:42 smithi104 ceph-mon[36624]: pgmap v883: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 81 KiB/s wr, 6 op/s 2024-04-12T21:30:43.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:42 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:clientreplay 2024-04-12T21:30:43.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:42 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:30:43.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:42 smithi104 ceph-mon[36624]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:30:43.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:42 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:30:43.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:42 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/551587205,v1:172.21.15.153:6833/551587205] up:active 2024-04-12T21:30:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:42 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:30:43.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:30:43.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: *** Caught signal (Aborted) ** 2024-04-12T21:30:43.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: in thread 7f636afc8700 thread_name:ms_dispatch 2024-04-12T21:30:43.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:43.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6378b82d20] 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2: gsignal() 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 3: abort() 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56051037c268] 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5605104180fc] 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56051074cf0a] 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 7: (MDLog::trim()+0xca9) [0x56051074e0c9] 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56051074e5a8] 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6377a6eb23] 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f6378b781ca] 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 11: clone() 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2024-04-12T21:30:42.980+0000 7f636afc8700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: in thread 7f636afc8700 thread_name:ms_dispatch 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:43.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6378b82d20] 2024-04-12T21:30:43.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2: gsignal() 2024-04-12T21:30:43.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 3: abort() 2024-04-12T21:30:43.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56051037c268] 2024-04-12T21:30:43.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5605104180fc] 2024-04-12T21:30:43.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56051074cf0a] 2024-04-12T21:30:43.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 7: (MDLog::trim()+0xca9) [0x56051074e0c9] 2024-04-12T21:30:43.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56051074e5a8] 2024-04-12T21:30:43.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6377a6eb23] 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f6378b781ca] 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 11: clone() 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 0> 2024-04-12T21:30:42.980+0000 7f636afc8700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: in thread 7f636afc8700 thread_name:ms_dispatch 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6378b82d20] 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2: gsignal() 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 3: abort() 2024-04-12T21:30:43.308 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56051037c268] 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5605104180fc] 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56051074cf0a] 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 7: (MDLog::trim()+0xca9) [0x56051074e0c9] 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56051074e5a8] 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6377a6eb23] 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f6378b781ca] 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 11: clone() 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:42 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: -10> 2024-04-12T21:30:42.980+0000 7f636afc8700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: in thread 7f636afc8700 thread_name:ms_dispatch 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f6378b82d20] 2024-04-12T21:30:43.309 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2: gsignal() 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 3: abort() 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56051037c268] 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5605104180fc] 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56051074cf0a] 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 7: (MDLog::trim()+0xca9) [0x56051074e0c9] 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56051074e5a8] 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f6377a6eb23] 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f6378b781ca] 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 11: clone() 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[74798]: 2024-04-12T21:30:43.310 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:43 smithi153 podman[74935]: 2024-04-12 21:30:43.042468624 +0000 UTC m=+0.020441002 container died 6b8fa148a723023df75eca7fb9bdd875715b75f013fd49b8f0531f270c1d295d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:30:43.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:30:43.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:30:43.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:30:44.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:30:44.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:30:44.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:30:44.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:30:44.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:30:44.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:30:44.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:44 smithi081 ceph-mon[31453]: pgmap v884: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:30:44.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:44 smithi081 ceph-mon[31453]: pgmap v885: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5 op/s 2024-04-12T21:30:44.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:44 smithi081 ceph-mon[31453]: pgmap v886: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2 op/s 2024-04-12T21:30:44.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:44 smithi153 ceph-mon[36569]: pgmap v884: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:30:44.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:44 smithi153 ceph-mon[36569]: pgmap v885: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5 op/s 2024-04-12T21:30:44.940 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:44 smithi153 ceph-mon[36569]: pgmap v886: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2 op/s 2024-04-12T21:30:45.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:44 smithi104 ceph-mon[36624]: pgmap v884: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:30:45.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:44 smithi104 ceph-mon[36624]: pgmap v885: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 5 op/s 2024-04-12T21:30:45.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:44 smithi104 ceph-mon[36624]: pgmap v886: 129 pgs: 129 active+clean; 571 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2 op/s 2024-04-12T21:30:45.636 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:45 smithi153 podman[74935]: 2024-04-12 21:30:45.386362723 +0000 UTC m=+2.364335092 container remove 6b8fa148a723023df75eca7fb9bdd875715b75f013fd49b8f0531f270c1d295d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS) 2024-04-12T21:30:45.637 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:45 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:30:46.054 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:45 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-12T21:30:46.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:45 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.165s CPU time. 2024-04-12T21:30:46.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:46 smithi081 ceph-mon[31453]: pgmap v887: 129 pgs: 129 active+clean; 570 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 316 B/s wr, 13 op/s 2024-04-12T21:30:47.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:46 smithi153 ceph-mon[36569]: pgmap v887: 129 pgs: 129 active+clean; 570 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 316 B/s wr, 13 op/s 2024-04-12T21:30:47.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:46 smithi104 ceph-mon[36624]: pgmap v887: 129 pgs: 129 active+clean; 570 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 316 B/s wr, 13 op/s 2024-04-12T21:30:49.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:48 smithi153 ceph-mon[36569]: pgmap v888: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 59 KiB/s wr, 48 op/s 2024-04-12T21:30:49.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:48 smithi104 ceph-mon[36624]: pgmap v888: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 59 KiB/s wr, 48 op/s 2024-04-12T21:30:49.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:48 smithi081 ceph-mon[31453]: pgmap v888: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 59 KiB/s wr, 48 op/s 2024-04-12T21:30:51.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:50 smithi153 ceph-mon[36569]: pgmap v889: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 59 KiB/s wr, 48 op/s 2024-04-12T21:30:51.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:50 smithi104 ceph-mon[36624]: pgmap v889: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 59 KiB/s wr, 48 op/s 2024-04-12T21:30:51.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:50 smithi081 ceph-mon[31453]: pgmap v889: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 59 KiB/s wr, 48 op/s 2024-04-12T21:30:53.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:52 smithi153 ceph-mon[36569]: pgmap v890: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 59 KiB/s wr, 49 op/s 2024-04-12T21:30:53.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:52 smithi104 ceph-mon[36624]: pgmap v890: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 59 KiB/s wr, 49 op/s 2024-04-12T21:30:53.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:52 smithi081 ceph-mon[31453]: pgmap v890: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 59 KiB/s wr, 49 op/s 2024-04-12T21:30:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:53.454+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:30:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:53.454+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:30:53.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:30:53.454+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:30:55.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:54 smithi153 ceph-mon[36569]: pgmap v891: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 49 KiB/s wr, 41 op/s 2024-04-12T21:30:55.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:54 smithi104 ceph-mon[36624]: pgmap v891: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 49 KiB/s wr, 41 op/s 2024-04-12T21:30:55.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:54 smithi081 ceph-mon[31453]: pgmap v891: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 49 KiB/s wr, 41 op/s 2024-04-12T21:30:56.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:55 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:30:56.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:55 smithi153 systemd[1]: Stopped Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:30:56.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:55 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.165s CPU time. 2024-04-12T21:30:56.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:55 smithi153 systemd[1]: Starting Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:30:56.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:56 smithi153 podman[75064]: 2024-04-12 21:30:55.940882505 +0000 UTC m=+0.018118042 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:30:56.370 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:56 smithi153 podman[75064]: 2024-04-12 21:30:56.06147084 +0000 UTC m=+0.138706398 container create 176be319d8c8efcc7e70c9ec0c7fb9a433ce9956c06f10f02dd54900804ec79d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T21:30:56.370 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:56 smithi153 podman[75064]: 2024-04-12 21:30:56.263832755 +0000 UTC m=+0.341068274 container init 176be319d8c8efcc7e70c9ec0c7fb9a433ce9956c06f10f02dd54900804ec79d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409) 2024-04-12T21:30:56.370 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:56 smithi153 podman[75064]: 2024-04-12 21:30:56.266618865 +0000 UTC m=+0.343854385 container start 176be319d8c8efcc7e70c9ec0c7fb9a433ce9956c06f10f02dd54900804ec79d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-12T21:30:56.370 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mds[75083]: -- 172.21.15.153:0/2525803069 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562d54d7ba40 con 0x562d54d4ec00 2024-04-12T21:30:56.370 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: starting mds.l at 2024-04-12T21:30:56.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:30:56.314+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:56.677 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:56 smithi153 bash[75064]: 176be319d8c8efcc7e70c9ec0c7fb9a433ce9956c06f10f02dd54900804ec79d 2024-04-12T21:30:56.678 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:30:56 smithi153 systemd[1]: Started Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:30:56.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: pgmap v892: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 41 KiB/s wr, 34 op/s 2024-04-12T21:30:56.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: daemon mds.l restarted 2024-04-12T21:30:56.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:56.938 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:56.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: osdmap e131: 12 total, 12 up, 12 in 2024-04-12T21:30:56.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:boot 2024-04-12T21:30:56.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:30:56.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:56.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:56.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:30:56.939 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:56 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:30:57.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: pgmap v892: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 41 KiB/s wr, 34 op/s 2024-04-12T21:30:57.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: daemon mds.l restarted 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: osdmap e131: 12 total, 12 up, 12 in 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:boot 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:30:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:56 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:30:57.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: pgmap v892: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 41 KiB/s wr, 34 op/s 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: daemon mds.l restarted 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: osdmap e131: 12 total, 12 up, 12 in 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:boot 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:30:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:56 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:30:59.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:58 smithi153 ceph-mon[36569]: pgmap v894: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 5 op/s 2024-04-12T21:30:59.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:58 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:reconnect 2024-04-12T21:30:59.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:58 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:30:59.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:58 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:30:59.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:58 smithi104 ceph-mon[36624]: pgmap v894: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 5 op/s 2024-04-12T21:30:59.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:58 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:reconnect 2024-04-12T21:30:59.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:58 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:30:59.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:58 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:30:59.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:58 smithi081 ceph-mon[31453]: pgmap v894: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 5 op/s 2024-04-12T21:30:59.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:58 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:reconnect 2024-04-12T21:30:59.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:58 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:30:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:58 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159998 2024-04-12T21:31:00.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:59 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:rejoin 2024-04-12T21:31:00.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:30:59 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:31:00.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:59 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:rejoin 2024-04-12T21:31:00.055 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:30:59 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:31:00.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:59 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:rejoin 2024-04-12T21:31:00.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:30:59 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:31:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:00 smithi153 ceph-mon[36569]: pgmap v895: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:31:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:00 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:clientreplay 2024-04-12T21:31:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:00 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:31:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:00 smithi153 ceph-mon[36569]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:31:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:00 smithi104 ceph-mon[36624]: pgmap v895: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:31:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:00 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:clientreplay 2024-04-12T21:31:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:00 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:31:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:00 smithi104 ceph-mon[36624]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:31:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:00 smithi081 ceph-mon[31453]: pgmap v895: 129 pgs: 129 active+clean; 567 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:31:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:00 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:clientreplay 2024-04-12T21:31:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:00 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:31:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:00 smithi081 ceph-mon[31453]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:31:01.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:01.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:active 2024-04-12T21:31:01.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:31:01.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:01.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-mon[36624]: pgmap v896: 129 pgs: 129 active+clean; 566 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 307 B/s wr, 13 op/s 2024-04-12T21:31:01.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:31:01.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: *** Caught signal (Aborted) ** 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: in thread 7f322fb36700 thread_name:ms_dispatch 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f323d6f0d20] 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2: gsignal() 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 3: abort() 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5642f2683268] 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5642f271f0fc] 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5642f2a53f0a] 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 7: (MDLog::trim()+0xca9) [0x5642f2a550c9] 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5642f2a555a8] 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f323c5dcb23] 2024-04-12T21:31:01.832 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f323d6e61ca] 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 11: clone() 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2024-04-12T21:31:01.524+0000 7f322fb36700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: in thread 7f322fb36700 thread_name:ms_dispatch 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f323d6f0d20] 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2: gsignal() 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 3: abort() 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5642f2683268] 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5642f271f0fc] 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5642f2a53f0a] 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 7: (MDLog::trim()+0xca9) [0x5642f2a550c9] 2024-04-12T21:31:01.833 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5642f2a555a8] 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f323c5dcb23] 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f323d6e61ca] 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 11: clone() 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 0> 2024-04-12T21:31:01.524+0000 7f322fb36700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: in thread 7f322fb36700 thread_name:ms_dispatch 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f323d6f0d20] 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2: gsignal() 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 3: abort() 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5642f2683268] 2024-04-12T21:31:01.834 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5642f271f0fc] 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5642f2a53f0a] 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 7: (MDLog::trim()+0xca9) [0x5642f2a550c9] 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5642f2a555a8] 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f323c5dcb23] 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f323d6e61ca] 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 11: clone() 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: -1> 2024-04-12T21:31:01.524+0000 7f322fb36700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: in thread 7f322fb36700 thread_name:ms_dispatch 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f323d6f0d20] 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2: gsignal() 2024-04-12T21:31:01.835 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 3: abort() 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5642f2683268] 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5642f271f0fc] 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5642f2a53f0a] 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 7: (MDLog::trim()+0xca9) [0x5642f2a550c9] 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5642f2a555a8] 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f323c5dcb23] 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f323d6e61ca] 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 11: clone() 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73053]: 2024-04-12T21:31:01.836 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:01 smithi104 podman[73224]: 2024-04-12 21:31:01.584724386 +0000 UTC m=+0.019303403 container died 99077df301dfadb08e7787371d241463e3ec2d7f9320ff53bdd3e00d70c8ed78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-12T21:31:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:01 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:01 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:active 2024-04-12T21:31:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:01 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:31:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:01 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:01 smithi153 ceph-mon[36569]: pgmap v896: 129 pgs: 129 active+clean; 566 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 307 B/s wr, 13 op/s 2024-04-12T21:31:02.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:01 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:02.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:01 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/973396515,v1:172.21.15.104:6833/973396515] up:active 2024-04-12T21:31:02.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:01 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:31:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:01 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:01 smithi081 ceph-mon[31453]: pgmap v896: 129 pgs: 129 active+clean; 566 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 307 B/s wr, 13 op/s 2024-04-12T21:31:02.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:02 smithi104 podman[73224]: 2024-04-12 21:31:02.291903343 +0000 UTC m=+0.726482405 container remove 99077df301dfadb08e7787371d241463e3ec2d7f9320ff53bdd3e00d70c8ed78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409) 2024-04-12T21:31:02.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:02 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:31:02.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:02 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-12T21:31:02.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:02 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.556s CPU time. 2024-04-12T21:31:04.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:04 smithi081 ceph-mon[31453]: pgmap v897: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.8 KiB/s wr, 27 op/s 2024-04-12T21:31:04.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:04 smithi153 ceph-mon[36569]: pgmap v897: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.8 KiB/s wr, 27 op/s 2024-04-12T21:31:04.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:04 smithi104 ceph-mon[36624]: pgmap v897: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.8 KiB/s wr, 27 op/s 2024-04-12T21:31:06.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:06 smithi081 ceph-mon[31453]: pgmap v898: 129 pgs: 129 active+clean; 563 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 2.8 KiB/s wr, 34 op/s 2024-04-12T21:31:06.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:06 smithi153 ceph-mon[36569]: pgmap v898: 129 pgs: 129 active+clean; 563 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 2.8 KiB/s wr, 34 op/s 2024-04-12T21:31:06.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:06 smithi104 ceph-mon[36624]: pgmap v898: 129 pgs: 129 active+clean; 563 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 2.8 KiB/s wr, 34 op/s 2024-04-12T21:31:08.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:08 smithi153 ceph-mon[36569]: pgmap v899: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 87 KiB/s wr, 33 op/s 2024-04-12T21:31:08.457 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:08.458+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:31:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:08.458+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:31:08.458 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:08.458+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:31:08.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:08 smithi104 ceph-mon[36624]: pgmap v899: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 87 KiB/s wr, 33 op/s 2024-04-12T21:31:08.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:08 smithi081 ceph-mon[31453]: pgmap v899: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 87 KiB/s wr, 33 op/s 2024-04-12T21:31:10.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:10 smithi153 ceph-mon[36569]: pgmap v900: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 78 KiB/s wr, 30 op/s 2024-04-12T21:31:10.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:10 smithi104 ceph-mon[36624]: pgmap v900: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 78 KiB/s wr, 30 op/s 2024-04-12T21:31:10.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:10 smithi081 ceph-mon[31453]: pgmap v900: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 78 KiB/s wr, 30 op/s 2024-04-12T21:31:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 msgr2=0x7f16e80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-12T21:31:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 msgr2=0x7f16e80a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T21:31:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 0x7f16e80a6190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ce5f0 tx=0x7f16f40d4610 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:31:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 0x7f16e80a6190 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ce5f0 tx=0x7f16f40d4610 comp rx=0 tx=0).stop 2024-04-12T21:31:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 msgr2=0x7f16e80a6190 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-12T21:31:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 msgr2=0x7f16e80a6190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T21:31:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a3cd0 0x7f16e80a6190 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:31:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:31:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.086+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:31:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.088+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 0x7f16e80b1520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f16e80a40b0 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:31:12.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:12 smithi153 ceph-mon[36569]: pgmap v901: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 78 KiB/s wr, 32 op/s 2024-04-12T21:31:12.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:12 smithi104 ceph-mon[36624]: pgmap v901: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 78 KiB/s wr, 32 op/s 2024-04-12T21:31:12.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:12 smithi081 ceph-mon[31453]: pgmap v901: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 78 KiB/s wr, 32 op/s 2024-04-12T21:31:12.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.848+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006360 con 0x7f1704076980 2024-04-12T21:31:12.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.848+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006360 con 0x7f1704076980 2024-04-12T21:31:12.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.878+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d0006120 con 0x563c9e245d20 2024-04-12T21:31:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:12.878+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d0006120 con 0x563c9e245d20 2024-04-12T21:31:12.959 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:12 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:31:12.960 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:12 smithi104 systemd[1]: Stopped Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:31:12.960 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:12 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.556s CPU time. 2024-04-12T21:31:12.960 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:12 smithi104 systemd[1]: Starting Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:31:12.960 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:12 smithi104 podman[73323]: 2024-04-12 21:31:12.724172171 +0000 UTC m=+0.018198228 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:31:12.960 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:12 smithi104 podman[73323]: 2024-04-12 21:31:12.833434347 +0000 UTC m=+0.127460376 container create 54feaea4f6e0c11c0af52a177f656dfc898e167a4bfa14704c9614e7fe32fcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-12T21:31:13.224 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: daemon mds.k restarted 2024-04-12T21:31:13.224 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:13.224 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: osdmap e132: 12 total, 12 up, 12 in 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:boot 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:13 smithi104 podman[73323]: 2024-04-12 21:31:13.094526703 +0000 UTC m=+0.388552757 container init 54feaea4f6e0c11c0af52a177f656dfc898e167a4bfa14704c9614e7fe32fcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:13 smithi104 podman[73323]: 2024-04-12 21:31:13.097216899 +0000 UTC m=+0.391242950 container start 54feaea4f6e0c11c0af52a177f656dfc898e167a4bfa14704c9614e7fe32fcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-mds[73343]: -- 172.21.15.104:0/1916667155 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558a1e0bba40 con 0x558a1e08e800 2024-04-12T21:31:13.225 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:13 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: starting mds.k at 2024-04-12T21:31:13.226 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:13 smithi104 bash[73323]: 54feaea4f6e0c11c0af52a177f656dfc898e167a4bfa14704c9614e7fe32fcd2 2024-04-12T21:31:13.226 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:13 smithi104 systemd[1]: Started Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:31:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:31:13.136+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: daemon mds.k restarted 2024-04-12T21:31:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: osdmap e132: 12 total, 12 up, 12 in 2024-04-12T21:31:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:boot 2024-04-12T21:31:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:31:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:31:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:13 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: daemon mds.k restarted 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: osdmap e132: 12 total, 12 up, 12 in 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:boot 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:31:13.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:13 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:31:14.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:14 smithi153 ceph-mon[36569]: pgmap v902: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 78 KiB/s wr, 23 op/s 2024-04-12T21:31:14.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:14 smithi104 ceph-mon[36624]: pgmap v902: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 78 KiB/s wr, 23 op/s 2024-04-12T21:31:14.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:14 smithi081 ceph-mon[31453]: pgmap v902: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 78 KiB/s wr, 23 op/s 2024-04-12T21:31:15.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:reconnect 2024-04-12T21:31:15.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:15 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:31:15.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:15 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0249996 2024-04-12T21:31:15.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:reconnect 2024-04-12T21:31:15.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:15 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:31:15.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:15 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0249996 2024-04-12T21:31:15.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:reconnect 2024-04-12T21:31:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:15 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:31:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:15 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0249996 2024-04-12T21:31:16.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:16 smithi153 ceph-mon[36569]: pgmap v904: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 91 KiB/s wr, 6 op/s 2024-04-12T21:31:16.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:16 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:rejoin 2024-04-12T21:31:16.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:16 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:31:16.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:16 smithi104 ceph-mon[36624]: pgmap v904: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 91 KiB/s wr, 6 op/s 2024-04-12T21:31:16.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:16 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:rejoin 2024-04-12T21:31:16.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:16 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:31:16.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:16 smithi081 ceph-mon[31453]: pgmap v904: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 91 KiB/s wr, 6 op/s 2024-04-12T21:31:16.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:16 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:rejoin 2024-04-12T21:31:16.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:16 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:31:17.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:clientreplay 2024-04-12T21:31:17.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:31:17.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-mon[36569]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:31:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:17 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:clientreplay 2024-04-12T21:31:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:17 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:31:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:17 smithi104 ceph-mon[36624]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:31:17.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:17 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:clientreplay 2024-04-12T21:31:17.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:17 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:31:17.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:17 smithi081 ceph-mon[31453]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:31:18.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:31:18.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: *** Caught signal (Aborted) ** 2024-04-12T21:31:18.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: in thread 7f27beb9a700 thread_name:ms_dispatch 2024-04-12T21:31:18.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f27cc754d20] 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2: gsignal() 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 3: abort() 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562d538ff268] 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562d5399b0fc] 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562d53ccff0a] 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 7: (MDLog::trim()+0xca9) [0x562d53cd10c9] 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562d53cd15a8] 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f27cb640b23] 2024-04-12T21:31:18.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f27cc74a1ca] 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 11: clone() 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2024-04-12T21:31:17.772+0000 7f27beb9a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: in thread 7f27beb9a700 thread_name:ms_dispatch 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f27cc754d20] 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2: gsignal() 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 3: abort() 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562d538ff268] 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562d5399b0fc] 2024-04-12T21:31:18.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562d53ccff0a] 2024-04-12T21:31:18.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 7: (MDLog::trim()+0xca9) [0x562d53cd10c9] 2024-04-12T21:31:18.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562d53cd15a8] 2024-04-12T21:31:18.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f27cb640b23] 2024-04-12T21:31:18.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f27cc74a1ca] 2024-04-12T21:31:18.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 11: clone() 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 0> 2024-04-12T21:31:17.772+0000 7f27beb9a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: in thread 7f27beb9a700 thread_name:ms_dispatch 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f27cc754d20] 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2: gsignal() 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 3: abort() 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562d538ff268] 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562d5399b0fc] 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562d53ccff0a] 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 7: (MDLog::trim()+0xca9) [0x562d53cd10c9] 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562d53cd15a8] 2024-04-12T21:31:18.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f27cb640b23] 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f27cc74a1ca] 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 11: clone() 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: -10> 2024-04-12T21:31:17.772+0000 7f27beb9a700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: in thread 7f27beb9a700 thread_name:ms_dispatch 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f27cc754d20] 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2: gsignal() 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 3: abort() 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562d538ff268] 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562d5399b0fc] 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562d53ccff0a] 2024-04-12T21:31:18.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 7: (MDLog::trim()+0xca9) [0x562d53cd10c9] 2024-04-12T21:31:18.062 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562d53cd15a8] 2024-04-12T21:31:18.062 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f27cb640b23] 2024-04-12T21:31:18.062 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f27cc74a1ca] 2024-04-12T21:31:18.062 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 11: clone() 2024-04-12T21:31:18.062 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:18.062 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75079]: 2024-04-12T21:31:18.062 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:17 smithi153 podman[75131]: 2024-04-12 21:31:17.833710031 +0000 UTC m=+0.019341611 container died 176be319d8c8efcc7e70c9ec0c7fb9a433ce9956c06f10f02dd54900804ec79d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:31:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:18 smithi153 ceph-mon[36569]: pgmap v905: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:31:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:18 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:18 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:active 2024-04-12T21:31:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:18 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:31:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:18 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:18.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:18 smithi104 ceph-mon[36624]: pgmap v905: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:31:18.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:18 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:18.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:18 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:active 2024-04-12T21:31:18.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:18 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:31:18.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:18 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:18.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:18 smithi081 ceph-mon[31453]: pgmap v905: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 204 B/s wr, 7 op/s 2024-04-12T21:31:18.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:18 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:18.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:18 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3095147146,v1:172.21.15.153:6833/3095147146] up:active 2024-04-12T21:31:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:18 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:31:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:18 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:19.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:18 smithi153 podman[75131]: 2024-04-12 21:31:18.760933221 +0000 UTC m=+0.946564798 container remove 176be319d8c8efcc7e70c9ec0c7fb9a433ce9956c06f10f02dd54900804ec79d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-12T21:31:19.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:18 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:31:19.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:19 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-12T21:31:19.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:19 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.610s CPU time. 2024-04-12T21:31:20.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:20 smithi153 ceph-mon[36569]: pgmap v906: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 409 B/s wr, 11 op/s 2024-04-12T21:31:20.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:20 smithi104 ceph-mon[36624]: pgmap v906: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 409 B/s wr, 11 op/s 2024-04-12T21:31:20.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:20 smithi081 ceph-mon[31453]: pgmap v906: 129 pgs: 129 active+clean; 564 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 409 B/s wr, 11 op/s 2024-04-12T21:31:22.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:22 smithi153 ceph-mon[36569]: pgmap v907: 129 pgs: 129 active+clean; 563 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 46 KiB/s wr, 24 op/s 2024-04-12T21:31:22.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:22 smithi104 ceph-mon[36624]: pgmap v907: 129 pgs: 129 active+clean; 563 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 46 KiB/s wr, 24 op/s 2024-04-12T21:31:22.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:22 smithi081 ceph-mon[31453]: pgmap v907: 129 pgs: 129 active+clean; 563 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 46 KiB/s wr, 24 op/s 2024-04-12T21:31:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:23.470+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:31:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:23.471+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:31:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:23.471+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:31:24.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:24 smithi153 ceph-mon[36569]: pgmap v908: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 85 KiB/s wr, 39 op/s 2024-04-12T21:31:24.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:24 smithi104 ceph-mon[36624]: pgmap v908: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 85 KiB/s wr, 39 op/s 2024-04-12T21:31:24.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:24 smithi081 ceph-mon[31453]: pgmap v908: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 85 KiB/s wr, 39 op/s 2024-04-12T21:31:26.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:26 smithi153 ceph-mon[36569]: pgmap v909: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 71 KiB/s wr, 33 op/s 2024-04-12T21:31:26.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:26 smithi104 ceph-mon[36624]: pgmap v909: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 71 KiB/s wr, 33 op/s 2024-04-12T21:31:26.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:26 smithi081 ceph-mon[31453]: pgmap v909: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 71 KiB/s wr, 33 op/s 2024-04-12T21:31:28.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:28 smithi153 ceph-mon[36569]: pgmap v910: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 71 KiB/s wr, 35 op/s 2024-04-12T21:31:28.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:28 smithi104 ceph-mon[36624]: pgmap v910: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 71 KiB/s wr, 35 op/s 2024-04-12T21:31:28.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:28 smithi081 ceph-mon[31453]: pgmap v910: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 71 KiB/s wr, 35 op/s 2024-04-12T21:31:29.317 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:31:29.317 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 systemd[1]: Stopped Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:31:29.317 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.610s CPU time. 2024-04-12T21:31:29.318 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 systemd[1]: Starting Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:31:29.318 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 podman[75227]: 2024-04-12 21:31:29.198297472 +0000 UTC m=+0.018307247 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:31:29.582 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 podman[75227]: 2024-04-12 21:31:29.319963481 +0000 UTC m=+0.139973245 container create 69fdaa73d8d09ac4b5c71b4ecf9068f3c27cda5dc0e1c19f90cfefda5c5a2f6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409) 2024-04-12T21:31:29.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 podman[75227]: 2024-04-12 21:31:29.555710252 +0000 UTC m=+0.375720025 container init 69fdaa73d8d09ac4b5c71b4ecf9068f3c27cda5dc0e1c19f90cfefda5c5a2f6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:31:29.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 podman[75227]: 2024-04-12 21:31:29.559168916 +0000 UTC m=+0.379178683 container start 69fdaa73d8d09ac4b5c71b4ecf9068f3c27cda5dc0e1c19f90cfefda5c5a2f6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:31:29.583 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 ceph-mds[75246]: -- 172.21.15.153:0/1872189669 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563668dc1a40 con 0x563668d94800 2024-04-12T21:31:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:31:29.602+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:30.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: starting mds.l at 2024-04-12T21:31:30.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 bash[75227]: 69fdaa73d8d09ac4b5c71b4ecf9068f3c27cda5dc0e1c19f90cfefda5c5a2f6b 2024-04-12T21:31:30.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:29 smithi153 systemd[1]: Started Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:31:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: pgmap v911: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 70 KiB/s wr, 32 op/s 2024-04-12T21:31:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: daemon mds.l restarted 2024-04-12T21:31:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: osdmap e133: 12 total, 12 up, 12 in 2024-04-12T21:31:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:boot 2024-04-12T21:31:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:31:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:31:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:30 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:31:30.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: pgmap v911: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 70 KiB/s wr, 32 op/s 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: daemon mds.l restarted 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: osdmap e133: 12 total, 12 up, 12 in 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:boot 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:31:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:30 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:31:30.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: pgmap v911: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 70 KiB/s wr, 32 op/s 2024-04-12T21:31:30.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: daemon mds.l restarted 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: osdmap e133: 12 total, 12 up, 12 in 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:boot 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:31:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:30 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:31:32.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:32 smithi153 ceph-mon[36569]: pgmap v913: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 38 KiB/s wr, 20 op/s 2024-04-12T21:31:32.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:32 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:reconnect 2024-04-12T21:31:32.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:32 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:31:32.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:32 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209998 2024-04-12T21:31:32.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:32 smithi104 ceph-mon[36624]: pgmap v913: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 38 KiB/s wr, 20 op/s 2024-04-12T21:31:32.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:32 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:reconnect 2024-04-12T21:31:32.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:32 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:31:32.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:32 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209998 2024-04-12T21:31:32.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:32 smithi081 ceph-mon[31453]: pgmap v913: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 38 KiB/s wr, 20 op/s 2024-04-12T21:31:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:32 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:reconnect 2024-04-12T21:31:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:32 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:31:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:32 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0209998 2024-04-12T21:31:33.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:33 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:rejoin 2024-04-12T21:31:33.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:33 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:31:33.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:33 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:rejoin 2024-04-12T21:31:33.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:33 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:31:33.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:33 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:rejoin 2024-04-12T21:31:33.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:33 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:31:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:34 smithi153 ceph-mon[36569]: pgmap v914: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:31:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:34 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:clientreplay 2024-04-12T21:31:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:34 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:31:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:34 smithi153 ceph-mon[36569]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:31:34.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:34 smithi104 ceph-mon[36624]: pgmap v914: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:31:34.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:34 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:clientreplay 2024-04-12T21:31:34.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:34 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:31:34.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:34 smithi104 ceph-mon[36624]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:31:34.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:34 smithi081 ceph-mon[31453]: pgmap v914: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:31:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:34 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:clientreplay 2024-04-12T21:31:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:34 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:31:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:34 smithi081 ceph-mon[31453]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:31:35.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:35 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:35.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:35 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:active 2024-04-12T21:31:35.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:35 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:31:35.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:35 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:35.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:35.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:active 2024-04-12T21:31:35.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:31:35.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: *** Caught signal (Aborted) ** 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: in thread 7f93b2efe700 thread_name:ms_dispatch 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93c0ab8d20] 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2: gsignal() 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 3: abort() 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558a1ccb5268] 2024-04-12T21:31:35.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558a1cd510fc] 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558a1d085f0a] 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 7: (MDLog::trim()+0xca9) [0x558a1d0870c9] 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558a1d0875a8] 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f93bf9a4b23] 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f93c0aae1ca] 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 11: clone() 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2024-04-12T21:31:35.242+0000 7f93b2efe700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: in thread 7f93b2efe700 thread_name:ms_dispatch 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93c0ab8d20] 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2: gsignal() 2024-04-12T21:31:35.582 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 3: abort() 2024-04-12T21:31:35.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558a1ccb5268] 2024-04-12T21:31:35.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558a1cd510fc] 2024-04-12T21:31:35.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558a1d085f0a] 2024-04-12T21:31:35.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 7: (MDLog::trim()+0xca9) [0x558a1d0870c9] 2024-04-12T21:31:35.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558a1d0875a8] 2024-04-12T21:31:35.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f93bf9a4b23] 2024-04-12T21:31:35.583 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f93c0aae1ca] 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 11: clone() 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 0> 2024-04-12T21:31:35.242+0000 7f93b2efe700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: in thread 7f93b2efe700 thread_name:ms_dispatch 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93c0ab8d20] 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2: gsignal() 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 3: abort() 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558a1ccb5268] 2024-04-12T21:31:35.584 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558a1cd510fc] 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558a1d085f0a] 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 7: (MDLog::trim()+0xca9) [0x558a1d0870c9] 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558a1d0875a8] 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f93bf9a4b23] 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f93c0aae1ca] 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 11: clone() 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: -10> 2024-04-12T21:31:35.242+0000 7f93b2efe700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: in thread 7f93b2efe700 thread_name:ms_dispatch 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f93c0ab8d20] 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2: gsignal() 2024-04-12T21:31:35.585 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 3: abort() 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x558a1ccb5268] 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x558a1cd510fc] 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x558a1d085f0a] 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 7: (MDLog::trim()+0xca9) [0x558a1d0870c9] 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 8: (MDLog::log_trim_upkeep()+0x358) [0x558a1d0875a8] 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f93bf9a4b23] 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f93c0aae1ca] 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 11: clone() 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:35.586 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73338]: 2024-04-12T21:31:35.587 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:35 smithi104 podman[73390]: 2024-04-12 21:31:35.302848736 +0000 UTC m=+0.018890260 container died 54feaea4f6e0c11c0af52a177f656dfc898e167a4bfa14704c9614e7fe32fcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:31:35.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:35 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:35.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:35 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/1991998802,v1:172.21.15.104:6833/1991998802] up:active 2024-04-12T21:31:35.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:35 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:31:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:35 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:36.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:36 smithi104 ceph-mon[36624]: pgmap v915: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T21:31:36.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:36 smithi104 podman[73390]: 2024-04-12 21:31:36.140037912 +0000 UTC m=+0.856079492 container remove 54feaea4f6e0c11c0af52a177f656dfc898e167a4bfa14704c9614e7fe32fcd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-12T21:31:36.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:36 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:31:36.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:36 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-12T21:31:36.581 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:36 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.730s CPU time. 2024-04-12T21:31:36.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:36 smithi081 ceph-mon[31453]: pgmap v915: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T21:31:36.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:36 smithi153 ceph-mon[36569]: pgmap v915: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T21:31:38.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:38.486+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:31:38.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:38.486+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:31:38.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:38.486+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:31:38.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:38 smithi104 ceph-mon[36624]: pgmap v916: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 96 KiB/s wr, 24 op/s 2024-04-12T21:31:38.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:38 smithi081 ceph-mon[31453]: pgmap v916: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 96 KiB/s wr, 24 op/s 2024-04-12T21:31:38.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:38 smithi153 ceph-mon[36569]: pgmap v916: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 96 KiB/s wr, 24 op/s 2024-04-12T21:31:40.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:40 smithi081 ceph-mon[31453]: pgmap v917: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 97 KiB/s wr, 25 op/s 2024-04-12T21:31:40.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:40 smithi153 ceph-mon[36569]: pgmap v917: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 97 KiB/s wr, 25 op/s 2024-04-12T21:31:40.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:40 smithi104 ceph-mon[36624]: pgmap v917: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 97 KiB/s wr, 25 op/s 2024-04-12T21:31:42.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:42 smithi081 ceph-mon[31453]: pgmap v918: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 84 KiB/s wr, 23 op/s 2024-04-12T21:31:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:42 smithi153 ceph-mon[36569]: pgmap v918: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 84 KiB/s wr, 23 op/s 2024-04-12T21:31:42.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:42 smithi104 ceph-mon[36624]: pgmap v918: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 84 KiB/s wr, 23 op/s 2024-04-12T21:31:44.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:44 smithi153 ceph-mon[36569]: pgmap v919: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 81 KiB/s wr, 21 op/s 2024-04-12T21:31:44.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:31:44.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:44 smithi081 ceph-mon[31453]: pgmap v919: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 81 KiB/s wr, 21 op/s 2024-04-12T21:31:44.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:31:44.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:44 smithi104 ceph-mon[36624]: pgmap v919: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 81 KiB/s wr, 21 op/s 2024-04-12T21:31:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:31:45.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:31:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:31:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:45 smithi081 ceph-mon[31453]: pgmap v920: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 84 KiB/s wr, 22 op/s 2024-04-12T21:31:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:45 smithi081 ceph-mon[31453]: pgmap v921: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 102 KiB/s wr, 22 op/s 2024-04-12T21:31:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:31:45.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:31:45.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:31:45.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:45 smithi153 ceph-mon[36569]: pgmap v920: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 84 KiB/s wr, 22 op/s 2024-04-12T21:31:45.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:45 smithi153 ceph-mon[36569]: pgmap v921: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 102 KiB/s wr, 22 op/s 2024-04-12T21:31:45.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:31:45.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:31:45.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:31:45.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:45 smithi104 ceph-mon[36624]: pgmap v920: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 84 KiB/s wr, 22 op/s 2024-04-12T21:31:45.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:45 smithi104 ceph-mon[36624]: pgmap v921: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 102 KiB/s wr, 22 op/s 2024-04-12T21:31:45.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:31:46.872 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:46 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:31:46.872 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:46 smithi104 systemd[1]: Stopped Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:31:46.872 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:46 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.730s CPU time. 2024-04-12T21:31:46.872 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:46 smithi104 systemd[1]: Starting Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:31:46.872 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:46 smithi104 podman[73615]: 2024-04-12 21:31:46.717244248 +0000 UTC m=+0.018212796 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:31:46.872 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:46 smithi104 podman[73615]: 2024-04-12 21:31:46.87449425 +0000 UTC m=+0.175462814 container create f1fba8a56aea91a7e56224341ceb9f38131e19f722815ba0d17dba95383a8d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.build-date=20240409, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:31:47.404 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:47 smithi104 podman[73615]: 2024-04-12 21:31:47.135010585 +0000 UTC m=+0.435979135 container init f1fba8a56aea91a7e56224341ceb9f38131e19f722815ba0d17dba95383a8d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-12T21:31:47.404 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:47 smithi104 podman[73615]: 2024-04-12 21:31:47.137747982 +0000 UTC m=+0.438716532 container start f1fba8a56aea91a7e56224341ceb9f38131e19f722815ba0d17dba95383a8d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD) 2024-04-12T21:31:47.405 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mds[73634]: -- 172.21.15.104:0/838759362 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645206cfa40 con 0x5645206a3000 2024-04-12T21:31:47.405 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: starting mds.k at 2024-04-12T21:31:47.405 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:47 smithi104 bash[73615]: f1fba8a56aea91a7e56224341ceb9f38131e19f722815ba0d17dba95383a8d3a 2024-04-12T21:31:47.405 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:31:47 smithi104 systemd[1]: Started Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:31:47.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:31:47.177+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: pgmap v922: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 538 B/s wr, 7 op/s 2024-04-12T21:31:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: daemon mds.k restarted 2024-04-12T21:31:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: osdmap e134: 12 total, 12 up, 12 in 2024-04-12T21:31:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:boot 2024-04-12T21:31:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:31:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:31:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:47 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:31:48.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: pgmap v922: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 538 B/s wr, 7 op/s 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: daemon mds.k restarted 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: osdmap e134: 12 total, 12 up, 12 in 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:boot 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:31:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:47 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:31:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: pgmap v922: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 538 B/s wr, 7 op/s 2024-04-12T21:31:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: daemon mds.k restarted 2024-04-12T21:31:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:31:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:31:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: osdmap e134: 12 total, 12 up, 12 in 2024-04-12T21:31:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:boot 2024-04-12T21:31:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:31:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:31:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:31:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:31:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:47 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:31:50.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:49 smithi153 ceph-mon[36569]: pgmap v924: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:31:50.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:49 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:reconnect 2024-04-12T21:31:50.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:49 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:31:50.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:49 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:31:50.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:49 smithi104 ceph-mon[36624]: pgmap v924: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:31:50.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:49 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:reconnect 2024-04-12T21:31:50.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:49 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:31:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:49 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:31:50.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:49 smithi081 ceph-mon[31453]: pgmap v924: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:31:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:49 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:reconnect 2024-04-12T21:31:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:49 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:31:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:49 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:31:50.982 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:50 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:rejoin 2024-04-12T21:31:50.982 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:50 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:31:51.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:50 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:rejoin 2024-04-12T21:31:51.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:50 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:31:51.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:50 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:rejoin 2024-04-12T21:31:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:50 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:31:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:51 smithi153 ceph-mon[36569]: pgmap v925: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:31:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:51 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:clientreplay 2024-04-12T21:31:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:51 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:31:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:51 smithi153 ceph-mon[36569]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:31:52.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:51 smithi104 ceph-mon[36624]: pgmap v925: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:31:52.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:51 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:clientreplay 2024-04-12T21:31:52.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:51 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:31:52.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:51 smithi104 ceph-mon[36624]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:31:52.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:51 smithi081 ceph-mon[31453]: pgmap v925: 129 pgs: 129 active+clean; 561 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:31:52.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:51 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:clientreplay 2024-04-12T21:31:52.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:51 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:clientreplay} 5 up:standby 2024-04-12T21:31:52.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:51 smithi081 ceph-mon[31453]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: *** Caught signal (Aborted) ** 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: in thread 7fe7ca252700 thread_name:ms_dispatch 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe7d7e0cd20] 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2: gsignal() 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 3: abort() 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563667d7b268] 2024-04-12T21:31:52.715 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563667e170fc] 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56366814bf0a] 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 7: (MDLog::trim()+0xca9) [0x56366814d0c9] 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56366814d5a8] 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe7d6cf8b23] 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe7d7e021ca] 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 11: clone() 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2024-04-12T21:31:52.327+0000 7fe7ca252700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: in thread 7fe7ca252700 thread_name:ms_dispatch 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe7d7e0cd20] 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2: gsignal() 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 3: abort() 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563667d7b268] 2024-04-12T21:31:52.716 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563667e170fc] 2024-04-12T21:31:52.717 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56366814bf0a] 2024-04-12T21:31:52.717 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 7: (MDLog::trim()+0xca9) [0x56366814d0c9] 2024-04-12T21:31:52.717 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56366814d5a8] 2024-04-12T21:31:52.717 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe7d6cf8b23] 2024-04-12T21:31:52.717 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe7d7e021ca] 2024-04-12T21:31:52.717 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 11: clone() 2024-04-12T21:31:52.717 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 0> 2024-04-12T21:31:52.327+0000 7fe7ca252700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: in thread 7fe7ca252700 thread_name:ms_dispatch 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe7d7e0cd20] 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2: gsignal() 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 3: abort() 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563667d7b268] 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563667e170fc] 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56366814bf0a] 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 7: (MDLog::trim()+0xca9) [0x56366814d0c9] 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56366814d5a8] 2024-04-12T21:31:52.718 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe7d6cf8b23] 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe7d7e021ca] 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 11: clone() 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: -11> 2024-04-12T21:31:52.327+0000 7fe7ca252700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: in thread 7fe7ca252700 thread_name:ms_dispatch 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe7d7e0cd20] 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2: gsignal() 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 3: abort() 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563667d7b268] 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563667e170fc] 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56366814bf0a] 2024-04-12T21:31:52.719 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 7: (MDLog::trim()+0xca9) [0x56366814d0c9] 2024-04-12T21:31:52.720 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56366814d5a8] 2024-04-12T21:31:52.720 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fe7d6cf8b23] 2024-04-12T21:31:52.720 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fe7d7e021ca] 2024-04-12T21:31:52.720 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 11: clone() 2024-04-12T21:31:52.720 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:31:52.720 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75242]: 2024-04-12T21:31:53.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:52 smithi153 podman[75411]: 2024-04-12 21:31:52.718289455 +0000 UTC m=+0.250698471 container died 69fdaa73d8d09ac4b5c71b4ecf9068f3c27cda5dc0e1c19f90cfefda5c5a2f6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-04-12T21:31:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:active 2024-04-12T21:31:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:31:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:52 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:53.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:52 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:53.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:52 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:active 2024-04-12T21:31:53.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:52 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:31:53.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:52 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:53.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:52 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:31:53.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:52 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/3315171380,v1:172.21.15.153:6833/3315171380] up:active 2024-04-12T21:31:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:52 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:31:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:52 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:31:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:53.488+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:31:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:53.488+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:31:53.487 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:31:53.488+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:31:54.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:53 smithi153 ceph-mon[36569]: pgmap v926: 129 pgs: 129 active+clean; 558 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 KiB/s wr, 27 op/s 2024-04-12T21:31:54.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:53 smithi104 ceph-mon[36624]: pgmap v926: 129 pgs: 129 active+clean; 558 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 KiB/s wr, 27 op/s 2024-04-12T21:31:54.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:53 smithi081 ceph-mon[31453]: pgmap v926: 129 pgs: 129 active+clean; 558 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 KiB/s wr, 27 op/s 2024-04-12T21:31:55.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:54 smithi153 ceph-mon[36569]: pgmap v927: 129 pgs: 129 active+clean; 557 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.5 KiB/s wr, 31 op/s 2024-04-12T21:31:55.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:54 smithi104 ceph-mon[36624]: pgmap v927: 129 pgs: 129 active+clean; 557 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.5 KiB/s wr, 31 op/s 2024-04-12T21:31:55.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:54 smithi081 ceph-mon[31453]: pgmap v927: 129 pgs: 129 active+clean; 557 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3.5 KiB/s wr, 31 op/s 2024-04-12T21:31:57.311 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:57 smithi153 podman[75411]: 2024-04-12 21:31:57.062455894 +0000 UTC m=+4.594864927 container remove 69fdaa73d8d09ac4b5c71b4ecf9068f3c27cda5dc0e1c19f90cfefda5c5a2f6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-12T21:31:57.311 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:57 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:31:57.677 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:57 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-12T21:31:57.677 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:31:57 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.959s CPU time. 2024-04-12T21:31:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:57 smithi153 ceph-mon[36569]: pgmap v928: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 80 KiB/s wr, 57 op/s 2024-04-12T21:31:58.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:57 smithi104 ceph-mon[36624]: pgmap v928: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 80 KiB/s wr, 57 op/s 2024-04-12T21:31:58.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:57 smithi081 ceph-mon[31453]: pgmap v928: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 80 KiB/s wr, 57 op/s 2024-04-12T21:32:00.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:31:59 smithi153 ceph-mon[36569]: pgmap v929: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 70 KiB/s wr, 50 op/s 2024-04-12T21:32:00.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:31:59 smithi104 ceph-mon[36624]: pgmap v929: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 70 KiB/s wr, 50 op/s 2024-04-12T21:32:00.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:31:59 smithi081 ceph-mon[31453]: pgmap v929: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 70 KiB/s wr, 50 op/s 2024-04-12T21:32:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:01 smithi153 ceph-mon[36569]: pgmap v930: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 67 KiB/s wr, 49 op/s 2024-04-12T21:32:02.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:01 smithi104 ceph-mon[36624]: pgmap v930: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 67 KiB/s wr, 49 op/s 2024-04-12T21:32:02.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:01 smithi081 ceph-mon[31453]: pgmap v930: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 67 KiB/s wr, 49 op/s 2024-04-12T21:32:04.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:03 smithi153 ceph-mon[36569]: pgmap v931: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 67 KiB/s wr, 50 op/s 2024-04-12T21:32:04.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:03 smithi104 ceph-mon[36624]: pgmap v931: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 67 KiB/s wr, 50 op/s 2024-04-12T21:32:04.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:03 smithi081 ceph-mon[31453]: pgmap v931: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 67 KiB/s wr, 50 op/s 2024-04-12T21:32:06.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:05 smithi153 ceph-mon[36569]: pgmap v932: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 64 KiB/s wr, 34 op/s 2024-04-12T21:32:06.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:05 smithi104 ceph-mon[36624]: pgmap v932: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 64 KiB/s wr, 34 op/s 2024-04-12T21:32:06.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:05 smithi081 ceph-mon[31453]: pgmap v932: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 64 KiB/s wr, 34 op/s 2024-04-12T21:32:07.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:07 smithi153 ceph-mon[36569]: pgmap v933: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 64 KiB/s wr, 26 op/s 2024-04-12T21:32:07.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:07 smithi153 ceph-mon[36569]: Replacing daemon mds.l as rank 0 with standby daemon mds.k 2024-04-12T21:32:07.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:07 smithi153 ceph-mon[36569]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-12T21:32:07.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:07 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:07.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:07 smithi153 ceph-mon[36569]: osdmap e135: 12 total, 12 up, 12 in 2024-04-12T21:32:07.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:07 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:replay} 4 up:standby 2024-04-12T21:32:07.811 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:07 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:32:07.811 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:07 smithi153 systemd[1]: Stopped Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:32:07.812 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:07 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.959s CPU time. 2024-04-12T21:32:07.812 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:07 smithi153 systemd[1]: Starting Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:32:07.812 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:07 smithi153 podman[75508]: 2024-04-12 21:32:07.689800152 +0000 UTC m=+0.018075574 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:32:07.812 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:07 smithi153 podman[75508]: 2024-04-12 21:32:07.813140619 +0000 UTC m=+0.141416071 container create 1a18389d09ff4caae9c047ae0947edd52cd9d291db6b93668749fd72073a9cb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-12T21:32:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:07 smithi104 ceph-mon[36624]: pgmap v933: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 64 KiB/s wr, 26 op/s 2024-04-12T21:32:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:07 smithi104 ceph-mon[36624]: Replacing daemon mds.l as rank 0 with standby daemon mds.k 2024-04-12T21:32:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:07 smithi104 ceph-mon[36624]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-12T21:32:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:07 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:07 smithi104 ceph-mon[36624]: osdmap e135: 12 total, 12 up, 12 in 2024-04-12T21:32:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:07 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:replay} 4 up:standby 2024-04-12T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:07 smithi081 ceph-mon[31453]: pgmap v933: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 64 KiB/s wr, 26 op/s 2024-04-12T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:07 smithi081 ceph-mon[31453]: Replacing daemon mds.l as rank 0 with standby daemon mds.k 2024-04-12T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:07 smithi081 ceph-mon[31453]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-12T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:07 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:07 smithi081 ceph-mon[31453]: osdmap e135: 12 total, 12 up, 12 in 2024-04-12T21:32:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:07 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:replay} 4 up:standby 2024-04-12T21:32:08.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:08.503+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:32:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:08.503+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:32:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:08.503+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:32:08.555 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:08 smithi153 podman[75508]: 2024-04-12 21:32:08.073798799 +0000 UTC m=+0.402074215 container init 1a18389d09ff4caae9c047ae0947edd52cd9d291db6b93668749fd72073a9cb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-12T21:32:08.555 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:08 smithi153 podman[75508]: 2024-04-12 21:32:08.076524525 +0000 UTC m=+0.404799938 container start 1a18389d09ff4caae9c047ae0947edd52cd9d291db6b93668749fd72073a9cb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:32:08.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:08 smithi153 ceph-mds[75528]: -- 172.21.15.153:0/3913977418 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dce3bd5a40 con 0x55dce3ba8800 2024-04-12T21:32:08.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:08 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: starting mds.l at 2024-04-12T21:32:08.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:08 smithi153 bash[75508]: 1a18389d09ff4caae9c047ae0947edd52cd9d291db6b93668749fd72073a9cb0 2024-04-12T21:32:08.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:08 smithi153 systemd[1]: Started Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:32:10.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:09 smithi153 ceph-mon[36569]: pgmap v935: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:32:10.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:09 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:boot 2024-04-12T21:32:10.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:09 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:reconnect 2024-04-12T21:32:10.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:09 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:32:10.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:10.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:09 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:32:10.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:09 smithi104 ceph-mon[36624]: pgmap v935: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:32:10.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:09 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:boot 2024-04-12T21:32:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:09 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:reconnect 2024-04-12T21:32:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:09 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:32:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:09 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:32:10.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:09 smithi081 ceph-mon[31453]: pgmap v935: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:32:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:09 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:boot 2024-04-12T21:32:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:09 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:reconnect 2024-04-12T21:32:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:09 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:32:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:09 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:32:11.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:10 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:rejoin 2024-04-12T21:32:11.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:10 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:32:11.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:10 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:rejoin 2024-04-12T21:32:11.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:10 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:32:11.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:10 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:rejoin 2024-04-12T21:32:11.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:10 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:32:12.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:11 smithi153 ceph-mon[36569]: pgmap v936: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3 op/s 2024-04-12T21:32:12.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:11 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:clientreplay 2024-04-12T21:32:12.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:11 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:32:12.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:11 smithi153 ceph-mon[36569]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:32:12.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-mon[36624]: pgmap v936: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3 op/s 2024-04-12T21:32:12.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:clientreplay 2024-04-12T21:32:12.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:32:12.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-mon[36624]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: *** Caught signal (Aborted) ** 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: in thread 7f4d8a409700 thread_name:ms_dispatch 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d97fc3d20] 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2: gsignal() 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 3: abort() 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56451dc46268] 2024-04-12T21:32:12.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56451dce20fc] 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56451e016f0a] 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 7: (MDLog::trim()+0xca9) [0x56451e0180c9] 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56451e0185a8] 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4d96eafb23] 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4d97fb91ca] 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 11: clone() 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2024-04-12T21:32:11.828+0000 7f4d8a409700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: in thread 7f4d8a409700 thread_name:ms_dispatch 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d97fc3d20] 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2: gsignal() 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 3: abort() 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56451dc46268] 2024-04-12T21:32:12.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56451dce20fc] 2024-04-12T21:32:12.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56451e016f0a] 2024-04-12T21:32:12.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 7: (MDLog::trim()+0xca9) [0x56451e0180c9] 2024-04-12T21:32:12.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56451e0185a8] 2024-04-12T21:32:12.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4d96eafb23] 2024-04-12T21:32:12.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4d97fb91ca] 2024-04-12T21:32:12.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 11: clone() 2024-04-12T21:32:12.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:12.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 0> 2024-04-12T21:32:11.828+0000 7f4d8a409700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: in thread 7f4d8a409700 thread_name:ms_dispatch 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d97fc3d20] 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2: gsignal() 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 3: abort() 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56451dc46268] 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56451dce20fc] 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56451e016f0a] 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 7: (MDLog::trim()+0xca9) [0x56451e0180c9] 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56451e0185a8] 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4d96eafb23] 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4d97fb91ca] 2024-04-12T21:32:12.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 11: clone() 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: -23> 2024-04-12T21:32:11.828+0000 7f4d8a409700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: in thread 7f4d8a409700 thread_name:ms_dispatch 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4d97fc3d20] 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2: gsignal() 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 3: abort() 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x56451dc46268] 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56451dce20fc] 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56451e016f0a] 2024-04-12T21:32:12.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 7: (MDLog::trim()+0xca9) [0x56451e0180c9] 2024-04-12T21:32:12.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 8: (MDLog::log_trim_upkeep()+0x358) [0x56451e0185a8] 2024-04-12T21:32:12.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4d96eafb23] 2024-04-12T21:32:12.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f4d97fb91ca] 2024-04-12T21:32:12.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 11: clone() 2024-04-12T21:32:12.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:12.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73630]: 2024-04-12T21:32:12.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:11 smithi104 podman[73682]: 2024-04-12 21:32:11.891434098 +0000 UTC m=+0.019145702 container died f1fba8a56aea91a7e56224341ceb9f38131e19f722815ba0d17dba95383a8d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:32:12.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:11 smithi081 ceph-mon[31453]: pgmap v936: 129 pgs: 129 active+clean; 554 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3 op/s 2024-04-12T21:32:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:11 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:clientreplay 2024-04-12T21:32:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:11 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:clientreplay} 5 up:standby 2024-04-12T21:32:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:11 smithi081 ceph-mon[31453]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:32:13.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:12 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:13.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:12 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:active 2024-04-12T21:32:13.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:12 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:32:13.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:12 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:13.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:12 smithi104 podman[73682]: 2024-04-12 21:32:12.638197042 +0000 UTC m=+0.765908636 container remove f1fba8a56aea91a7e56224341ceb9f38131e19f722815ba0d17dba95383a8d3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-12T21:32:13.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:12 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:32:13.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:12 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-12T21:32:13.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:12 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 22.951s CPU time. 2024-04-12T21:32:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:12 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:12 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:active 2024-04-12T21:32:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:12 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:32:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:12 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:13.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:12 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:13.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:12 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/981185002,v1:172.21.15.104:6833/981185002] up:active 2024-04-12T21:32:13.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:12 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:32:13.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:12 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:13.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:13 smithi081 ceph-mon[31453]: pgmap v937: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-12T21:32:14.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:13 smithi153 ceph-mon[36569]: pgmap v937: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-12T21:32:14.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:13 smithi104 ceph-mon[36624]: pgmap v937: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-12T21:32:16.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:15 smithi153 ceph-mon[36569]: pgmap v938: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-04-12T21:32:16.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:15 smithi104 ceph-mon[36624]: pgmap v938: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-04-12T21:32:16.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:15 smithi081 ceph-mon[31453]: pgmap v938: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-04-12T21:32:18.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:17 smithi153 ceph-mon[36569]: pgmap v939: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.0 KiB/s wr, 24 op/s 2024-04-12T21:32:18.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:17 smithi104 ceph-mon[36624]: pgmap v939: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.0 KiB/s wr, 24 op/s 2024-04-12T21:32:18.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:17 smithi081 ceph-mon[31453]: pgmap v939: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 8.0 KiB/s wr, 24 op/s 2024-04-12T21:32:19.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:18 smithi153 ceph-mon[36569]: pgmap v940: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.2 KiB/s wr, 22 op/s 2024-04-12T21:32:19.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:18 smithi104 ceph-mon[36624]: pgmap v940: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.2 KiB/s wr, 22 op/s 2024-04-12T21:32:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:18 smithi081 ceph-mon[31453]: pgmap v940: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 7.2 KiB/s wr, 22 op/s 2024-04-12T21:32:22.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:21 smithi153 ceph-mon[36569]: pgmap v941: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.7 KiB/s wr, 21 op/s 2024-04-12T21:32:22.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:21 smithi104 ceph-mon[36624]: pgmap v941: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.7 KiB/s wr, 21 op/s 2024-04-12T21:32:22.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:21 smithi081 ceph-mon[31453]: pgmap v941: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 6.7 KiB/s wr, 21 op/s 2024-04-12T21:32:23.441 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:32:23.441 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 systemd[1]: Stopped Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:32:23.441 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 22.951s CPU time. 2024-04-12T21:32:23.441 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 systemd[1]: Starting Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:32:23.442 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 podman[73779]: 2024-04-12 21:32:23.221523727 +0000 UTC m=+0.018220652 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:32:23.442 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 podman[73779]: 2024-04-12 21:32:23.33024345 +0000 UTC m=+0.126940381 container create b97e5252808023db0e7a9d702bc44409cbb8a0c97498e9d63f784e01e70dbaa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-12T21:32:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:23.519+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:32:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:23.519+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:32:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:23.519+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:32:23.694 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 podman[73779]: 2024-04-12 21:32:23.594497464 +0000 UTC m=+0.391194375 container init b97e5252808023db0e7a9d702bc44409cbb8a0c97498e9d63f784e01e70dbaa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409) 2024-04-12T21:32:23.694 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 podman[73779]: 2024-04-12 21:32:23.59712133 +0000 UTC m=+0.393818242 container start b97e5252808023db0e7a9d702bc44409cbb8a0c97498e9d63f784e01e70dbaa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, ceph=True) 2024-04-12T21:32:23.694 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mds[73798]: -- 172.21.15.104:0/3621938708 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56271a643a40 con 0x56271a616800 2024-04-12T21:32:23.694 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: starting mds.k at 2024-04-12T21:32:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:32:23.637+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:32:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: pgmap v942: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 6.7 KiB/s wr, 22 op/s 2024-04-12T21:32:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: daemon mds.k restarted 2024-04-12T21:32:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:32:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: osdmap e136: 12 total, 12 up, 12 in 2024-04-12T21:32:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:boot 2024-04-12T21:32:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:32:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:32:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:32:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:32:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:23 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:32:24.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: pgmap v942: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 6.7 KiB/s wr, 22 op/s 2024-04-12T21:32:24.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: daemon mds.k restarted 2024-04-12T21:32:24.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:24.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:32:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: osdmap e136: 12 total, 12 up, 12 in 2024-04-12T21:32:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:boot 2024-04-12T21:32:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:32:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:32:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:32:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:32:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:23 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 bash[73779]: b97e5252808023db0e7a9d702bc44409cbb8a0c97498e9d63f784e01e70dbaa5 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:23 smithi104 systemd[1]: Started Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: pgmap v942: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 6.7 KiB/s wr, 22 op/s 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: daemon mds.k restarted 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: osdmap e136: 12 total, 12 up, 12 in 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:boot 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: Standby daemon mds.l assigned to filesystem cephfs as rank 0 2024-04-12T21:32:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:32:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:32:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-12T21:32:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:23 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:replay} 5 up:standby 2024-04-12T21:32:26.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:25 smithi153 ceph-mon[36569]: pgmap v944: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-04-12T21:32:26.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:reconnect 2024-04-12T21:32:26.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:25 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:32:26.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:25 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999983 2024-04-12T21:32:26.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:25 smithi104 ceph-mon[36624]: pgmap v944: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-04-12T21:32:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:reconnect 2024-04-12T21:32:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:25 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:32:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:25 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999983 2024-04-12T21:32:26.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:25 smithi081 ceph-mon[31453]: pgmap v944: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 6.6 KiB/s wr, 15 op/s 2024-04-12T21:32:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:reconnect 2024-04-12T21:32:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:25 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:reconnect} 5 up:standby 2024-04-12T21:32:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:25 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999983 2024-04-12T21:32:27.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:rejoin 2024-04-12T21:32:27.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:32:27.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-mon[36569]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:32:27.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:32:27.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: *** Caught signal (Aborted) ** 2024-04-12T21:32:27.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: in thread 7f130c551700 thread_name:ms_dispatch 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f131a10bd20] 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2: gsignal() 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 3: abort() 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55dce0d66268] 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55dce0e020fc] 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55dce1136f0a] 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 7: (MDLog::trim()+0xca9) [0x55dce11380c9] 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55dce11385a8] 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f1318ff7b23] 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f131a1011ca] 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 11: clone() 2024-04-12T21:32:27.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2024-04-12T21:32:26.848+0000 7f130c551700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: in thread 7f130c551700 thread_name:ms_dispatch 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f131a10bd20] 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2: gsignal() 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 3: abort() 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55dce0d66268] 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55dce0e020fc] 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55dce1136f0a] 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 7: (MDLog::trim()+0xca9) [0x55dce11380c9] 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55dce11385a8] 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f1318ff7b23] 2024-04-12T21:32:27.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f131a1011ca] 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 11: clone() 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 0> 2024-04-12T21:32:26.848+0000 7f130c551700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: in thread 7f130c551700 thread_name:ms_dispatch 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f131a10bd20] 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2: gsignal() 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 3: abort() 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55dce0d66268] 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55dce0e020fc] 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55dce1136f0a] 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 7: (MDLog::trim()+0xca9) [0x55dce11380c9] 2024-04-12T21:32:27.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55dce11385a8] 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f1318ff7b23] 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f131a1011ca] 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 11: clone() 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: -10> 2024-04-12T21:32:26.848+0000 7f130c551700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: in thread 7f130c551700 thread_name:ms_dispatch 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f131a10bd20] 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2: gsignal() 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 3: abort() 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55dce0d66268] 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55dce0e020fc] 2024-04-12T21:32:27.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55dce1136f0a] 2024-04-12T21:32:27.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 7: (MDLog::trim()+0xca9) [0x55dce11380c9] 2024-04-12T21:32:27.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55dce11385a8] 2024-04-12T21:32:27.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f1318ff7b23] 2024-04-12T21:32:27.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f131a1011ca] 2024-04-12T21:32:27.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 11: clone() 2024-04-12T21:32:27.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:27.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75524]: 2024-04-12T21:32:27.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:26 smithi153 podman[75578]: 2024-04-12 21:32:26.911269194 +0000 UTC m=+0.019251064 container died 1a18389d09ff4caae9c047ae0947edd52cd9d291db6b93668749fd72073a9cb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:32:27.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:26 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:rejoin 2024-04-12T21:32:27.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:26 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:32:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:26 smithi104 ceph-mon[36624]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:32:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:26 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:rejoin 2024-04-12T21:32:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:26 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:rejoin} 5 up:standby 2024-04-12T21:32:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:26 smithi081 ceph-mon[31453]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:32:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:27 smithi153 ceph-mon[36569]: pgmap v945: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:32:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:27 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:27 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:active 2024-04-12T21:32:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:27 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:32:28.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:27 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:28.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:27 smithi153 podman[75578]: 2024-04-12 21:32:27.612337462 +0000 UTC m=+0.720319336 container remove 1a18389d09ff4caae9c047ae0947edd52cd9d291db6b93668749fd72073a9cb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:32:28.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:27 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:32:28.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:27 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-12T21:32:28.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:27 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 22.841s CPU time. 2024-04-12T21:32:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:27 smithi104 ceph-mon[36624]: pgmap v945: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:32:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:27 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:27 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:active 2024-04-12T21:32:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:27 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:32:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:27 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:27 smithi081 ceph-mon[31453]: pgmap v945: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:32:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:27 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:27 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/4085134719,v1:172.21.15.153:6833/4085134719] up:active 2024-04-12T21:32:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:27 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 5 up:standby 2024-04-12T21:32:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:27 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:30.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:29 smithi153 ceph-mon[36569]: pgmap v946: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 5 op/s 2024-04-12T21:32:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:29 smithi104 ceph-mon[36624]: pgmap v946: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 5 op/s 2024-04-12T21:32:30.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:29 smithi081 ceph-mon[31453]: pgmap v946: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 5 op/s 2024-04-12T21:32:32.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:31 smithi153 ceph-mon[36569]: pgmap v947: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 818 B/s wr, 6 op/s 2024-04-12T21:32:32.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:31 smithi104 ceph-mon[36624]: pgmap v947: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 818 B/s wr, 6 op/s 2024-04-12T21:32:32.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:31 smithi081 ceph-mon[31453]: pgmap v947: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 818 B/s wr, 6 op/s 2024-04-12T21:32:34.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:33 smithi153 ceph-mon[36569]: pgmap v948: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-12T21:32:34.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:33 smithi104 ceph-mon[36624]: pgmap v948: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-12T21:32:34.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:33 smithi081 ceph-mon[31453]: pgmap v948: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-04-12T21:32:36.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:35 smithi153 ceph-mon[36569]: pgmap v949: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-12T21:32:36.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:35 smithi104 ceph-mon[36624]: pgmap v949: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-12T21:32:36.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:35 smithi081 ceph-mon[31453]: pgmap v949: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-04-12T21:32:38.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:37 smithi153 ceph-mon[36569]: pgmap v950: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-12T21:32:38.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:37 smithi104 ceph-mon[36624]: pgmap v950: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-12T21:32:38.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:37 smithi081 ceph-mon[31453]: pgmap v950: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-04-12T21:32:38.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:32:38.306 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 systemd[1]: Stopped Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:32:38.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 22.841s CPU time. 2024-04-12T21:32:38.307 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 systemd[1]: Starting Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:32:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:38.533+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:32:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:38.533+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:32:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:38.533+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:32:38.557 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 podman[75673]: 2024-04-12 21:32:38.211276342 +0000 UTC m=+0.018299683 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:32:38.558 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 podman[75673]: 2024-04-12 21:32:38.371334282 +0000 UTC m=+0.178357610 container create a8b5bea4e3388518493d76d89fce82ee533398daad38b323784748db433fdcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-12T21:32:38.558 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 podman[75673]: 2024-04-12 21:32:38.556905499 +0000 UTC m=+0.363928830 container init a8b5bea4e3388518493d76d89fce82ee533398daad38b323784748db433fdcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-12T21:32:38.558 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 podman[75673]: 2024-04-12 21:32:38.559434111 +0000 UTC m=+0.366457440 container start a8b5bea4e3388518493d76d89fce82ee533398daad38b323784748db433fdcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409) 2024-04-12T21:32:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:32:38.606+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: daemon mds.l restarted 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: osdmap e137: 12 total, 12 up, 12 in 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:boot 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:32:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:38 smithi081 ceph-mon[31453]: pgmap v952: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-12T21:32:39.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mds[75692]: -- 172.21.15.153:0/2439389737 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2e713da40 con 0x55d2e7110c00 2024-04-12T21:32:39.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: starting mds.l at 2024-04-12T21:32:39.055 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 bash[75673]: a8b5bea4e3388518493d76d89fce82ee533398daad38b323784748db433fdcd5 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:32:38 smithi153 systemd[1]: Started Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: daemon mds.l restarted 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: osdmap e137: 12 total, 12 up, 12 in 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:boot 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:38 smithi153 ceph-mon[36569]: pgmap v952: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-12T21:32:39.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: daemon mds.l restarted 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: osdmap e137: 12 total, 12 up, 12 in 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:boot 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: Standby daemon mds.k assigned to filesystem cephfs as rank 0 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 5 up:standby, 1 failed 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:replay} 5 up:standby 2024-04-12T21:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:38 smithi104 ceph-mon[36624]: pgmap v952: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-04-12T21:32:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:40 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:reconnect 2024-04-12T21:32:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:40 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:32:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:40 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999988 2024-04-12T21:32:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:40 smithi153 ceph-mon[36569]: pgmap v953: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 716 B/s wr, 7 op/s 2024-04-12T21:32:41.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:40 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:reconnect 2024-04-12T21:32:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:40 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:32:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:40 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999988 2024-04-12T21:32:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:40 smithi104 ceph-mon[36624]: pgmap v953: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 716 B/s wr, 7 op/s 2024-04-12T21:32:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:40 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:reconnect 2024-04-12T21:32:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:40 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:reconnect} 5 up:standby 2024-04-12T21:32:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:40 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999988 2024-04-12T21:32:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:40 smithi081 ceph-mon[31453]: pgmap v953: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 716 B/s wr, 7 op/s 2024-04-12T21:32:42.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:41 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:rejoin 2024-04-12T21:32:42.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:41 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:32:42.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:41 smithi153 ceph-mon[36569]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:32:42.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:41 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:rejoin 2024-04-12T21:32:42.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:41 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:32:42.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:41 smithi104 ceph-mon[36624]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:32:42.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:41 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:rejoin 2024-04-12T21:32:42.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:41 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=k=up:rejoin} 5 up:standby 2024-04-12T21:32:42.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:41 smithi081 ceph-mon[31453]: daemon mds.k is now active in filesystem cephfs as rank 0 2024-04-12T21:32:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:42.863+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:32:42.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:42.864+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:32:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:42.895+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:32:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:42.896+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:32:43.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:42 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:43.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:42 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:active 2024-04-12T21:32:43.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:42 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:32:43.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:42 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:43.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:42 smithi153 ceph-mon[36569]: pgmap v954: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:32:43.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:active 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-mon[36624]: pgmap v954: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: *** Caught signal (Aborted) ** 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: in thread 7f2a018da700 thread_name:ms_dispatch 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:43.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2a0f494d20] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2: gsignal() 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 3: abort() 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5627193f5268] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5627194910fc] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5627197c5f0a] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 7: (MDLog::trim()+0xca9) [0x5627197c70c9] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5627197c75a8] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f2a0e380b23] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f2a0f48a1ca] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 11: clone() 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2024-04-12T21:32:42.745+0000 7f2a018da700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: in thread 7f2a018da700 thread_name:ms_dispatch 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2a0f494d20] 2024-04-12T21:32:43.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2: gsignal() 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 3: abort() 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5627193f5268] 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5627194910fc] 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5627197c5f0a] 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 7: (MDLog::trim()+0xca9) [0x5627197c70c9] 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5627197c75a8] 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f2a0e380b23] 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f2a0f48a1ca] 2024-04-12T21:32:43.083 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 11: clone() 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 0> 2024-04-12T21:32:42.745+0000 7f2a018da700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: in thread 7f2a018da700 thread_name:ms_dispatch 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2a0f494d20] 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2: gsignal() 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 3: abort() 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5627193f5268] 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5627194910fc] 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5627197c5f0a] 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 7: (MDLog::trim()+0xca9) [0x5627197c70c9] 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5627197c75a8] 2024-04-12T21:32:43.084 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f2a0e380b23] 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f2a0f48a1ca] 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 11: clone() 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: -10> 2024-04-12T21:32:42.745+0000 7f2a018da700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: in thread 7f2a018da700 thread_name:ms_dispatch 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2a0f494d20] 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2: gsignal() 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 3: abort() 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x5627193f5268] 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x5627194910fc] 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5627197c5f0a] 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 7: (MDLog::trim()+0xca9) [0x5627197c70c9] 2024-04-12T21:32:43.085 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5627197c75a8] 2024-04-12T21:32:43.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f2a0e380b23] 2024-04-12T21:32:43.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f2a0f48a1ca] 2024-04-12T21:32:43.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 11: clone() 2024-04-12T21:32:43.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:32:43.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k[73794]: 2024-04-12T21:32:43.086 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:42 smithi104 podman[73834]: 2024-04-12 21:32:42.802392381 +0000 UTC m=+0.018318840 container died b97e5252808023db0e7a9d702bc44409cbb8a0c97498e9d63f784e01e70dbaa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-12T21:32:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:42 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:32:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:42 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.104:6832/873126231,v1:172.21.15.104:6833/873126231] up:active 2024-04-12T21:32:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:42 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 5 up:standby 2024-04-12T21:32:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:42 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=k=up:active} 1 up:standby-replay 4 up:standby 2024-04-12T21:32:43.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:42 smithi081 ceph-mon[31453]: pgmap v954: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:32:43.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:43 smithi104 podman[73834]: 2024-04-12 21:32:43.485771829 +0000 UTC m=+0.701698270 container remove b97e5252808023db0e7a9d702bc44409cbb8a0c97498e9d63f784e01e70dbaa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-12T21:32:43.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:43 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:32:43.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:43 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-12T21:32:43.831 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:43 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.822s CPU time. 2024-04-12T21:32:46.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: pgmap v955: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 9 op/s 2024-04-12T21:32:46.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:32:46.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: pgmap v955: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 9 op/s 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: pgmap v955: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 204 B/s wr, 9 op/s 2024-04-12T21:32:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:32:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:32:47.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:32:47.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:32:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:32:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:32:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:32:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:32:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:32:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:32:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:32:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:32:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:32:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:47 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:32:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:47 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:32:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:47 smithi153 ceph-mon[36569]: pgmap v956: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 91 KiB/s wr, 16 op/s 2024-04-12T21:32:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:47 smithi153 ceph-mon[36569]: pgmap v957: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 115 KiB/s wr, 20 op/s 2024-04-12T21:32:48.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:47 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:32:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:47 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:32:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:47 smithi104 ceph-mon[36624]: pgmap v956: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 91 KiB/s wr, 16 op/s 2024-04-12T21:32:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:47 smithi104 ceph-mon[36624]: pgmap v957: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 115 KiB/s wr, 20 op/s 2024-04-12T21:32:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:47 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:32:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:47 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:32:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:47 smithi081 ceph-mon[31453]: pgmap v956: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 91 KiB/s wr, 16 op/s 2024-04-12T21:32:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:47 smithi081 ceph-mon[31453]: pgmap v957: 129 pgs: 129 active+clean; 553 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 115 KiB/s wr, 20 op/s 2024-04-12T21:32:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:49 smithi153 ceph-mon[36569]: pgmap v958: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 92 KiB/s wr, 23 op/s 2024-04-12T21:32:50.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:49 smithi104 ceph-mon[36624]: pgmap v958: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 92 KiB/s wr, 23 op/s 2024-04-12T21:32:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:49 smithi081 ceph-mon[31453]: pgmap v958: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 92 KiB/s wr, 23 op/s 2024-04-12T21:32:52.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:51 smithi153 ceph-mon[36569]: pgmap v959: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 92 KiB/s wr, 23 op/s 2024-04-12T21:32:52.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:51 smithi104 ceph-mon[36624]: pgmap v959: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 92 KiB/s wr, 23 op/s 2024-04-12T21:32:52.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:51 smithi081 ceph-mon[31453]: pgmap v959: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 92 KiB/s wr, 23 op/s 2024-04-12T21:32:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:53.549+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:32:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:53.549+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:32:53.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:32:53.549+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:32:54.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:53 smithi153 ceph-mon[36569]: pgmap v960: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 92 KiB/s wr, 24 op/s 2024-04-12T21:32:54.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:53 smithi104 ceph-mon[36624]: pgmap v960: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 92 KiB/s wr, 24 op/s 2024-04-12T21:32:54.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:32:54.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:53 smithi104 systemd[1]: Stopped Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:32:54.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Consumed 23.822s CPU time. 2024-04-12T21:32:54.081 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Start request repeated too quickly. 2024-04-12T21:32:54.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:53 smithi104 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-12T21:32:54.082 INFO:journalctl@ceph.mds.k.smithi104.stdout:Apr 12 21:32:53 smithi104 systemd[1]: Failed to start Ceph mds.k for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:32:54.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:53 smithi081 ceph-mon[31453]: pgmap v960: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 92 KiB/s wr, 24 op/s 2024-04-12T21:32:56.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:55 smithi153 ceph-mon[36569]: pgmap v961: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 92 KiB/s wr, 20 op/s 2024-04-12T21:32:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:55 smithi104 ceph-mon[36624]: pgmap v961: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 92 KiB/s wr, 20 op/s 2024-04-12T21:32:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:55 smithi081 ceph-mon[31453]: pgmap v961: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 92 KiB/s wr, 20 op/s 2024-04-12T21:32:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:57 smithi153 ceph-mon[36569]: pgmap v962: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 921 B/s wr, 11 op/s 2024-04-12T21:32:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:57 smithi153 ceph-mon[36569]: Replacing daemon mds.k as rank 0 with standby daemon mds.l 2024-04-12T21:32:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:57 smithi153 ceph-mon[36569]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-12T21:32:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:57 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:57 smithi153 ceph-mon[36569]: osdmap e138: 12 total, 12 up, 12 in 2024-04-12T21:32:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:57 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:replay} 4 up:standby 2024-04-12T21:32:58.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:57 smithi104 ceph-mon[36624]: pgmap v962: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 921 B/s wr, 11 op/s 2024-04-12T21:32:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:57 smithi104 ceph-mon[36624]: Replacing daemon mds.k as rank 0 with standby daemon mds.l 2024-04-12T21:32:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:57 smithi104 ceph-mon[36624]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-12T21:32:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:57 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:57 smithi104 ceph-mon[36624]: osdmap e138: 12 total, 12 up, 12 in 2024-04-12T21:32:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:57 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:replay} 4 up:standby 2024-04-12T21:32:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:57 smithi081 ceph-mon[31453]: pgmap v962: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 921 B/s wr, 11 op/s 2024-04-12T21:32:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:57 smithi081 ceph-mon[31453]: Replacing daemon mds.k as rank 0 with standby daemon mds.l 2024-04-12T21:32:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:57 smithi081 ceph-mon[31453]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-12T21:32:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:57 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:32:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:57 smithi081 ceph-mon[31453]: osdmap e138: 12 total, 12 up, 12 in 2024-04-12T21:32:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:57 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:replay} 4 up:standby 2024-04-12T21:33:00.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:59 smithi153 ceph-mon[36569]: pgmap v964: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:33:00.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:59 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:reconnect 2024-04-12T21:33:00.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:59 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:reconnect} 4 up:standby 2024-04-12T21:33:00.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:32:59 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:33:00.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:59 smithi104 ceph-mon[36624]: pgmap v964: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:33:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:59 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:reconnect 2024-04-12T21:33:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:59 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:reconnect} 4 up:standby 2024-04-12T21:33:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:32:59 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:33:00.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:59 smithi081 ceph-mon[31453]: pgmap v964: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:33:00.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:59 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:reconnect 2024-04-12T21:33:00.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:59 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:reconnect} 4 up:standby 2024-04-12T21:33:00.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:32:59 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:33:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:00 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:rejoin 2024-04-12T21:33:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:00 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:rejoin} 4 up:standby 2024-04-12T21:33:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:00 smithi153 ceph-mon[36569]: pgmap v965: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:33:01.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:00 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:rejoin 2024-04-12T21:33:01.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:00 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:rejoin} 4 up:standby 2024-04-12T21:33:01.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:00 smithi081 ceph-mon[31453]: pgmap v965: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:33:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:00 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:rejoin 2024-04-12T21:33:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:00 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:rejoin} 4 up:standby 2024-04-12T21:33:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:00 smithi104 ceph-mon[36624]: pgmap v965: 129 pgs: 129 active+clean; 552 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:33:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:clientreplay 2024-04-12T21:33:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=l=up:clientreplay} 4 up:standby 2024-04-12T21:33:02.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-mon[36569]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:33:02.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:33:02.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: *** Caught signal (Aborted) ** 2024-04-12T21:33:02.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: in thread 7fc111af2700 thread_name:ms_dispatch 2024-04-12T21:33:02.056 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc11f6acd20] 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2: gsignal() 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 3: abort() 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d2e4466268] 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d2e45020fc] 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d2e4836f0a] 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 7: (MDLog::trim()+0xca9) [0x55d2e48380c9] 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d2e48385a8] 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc11e598b23] 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc11f6a21ca] 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 11: clone() 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2024-04-12T21:33:01.877+0000 7fc111af2700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:02.057 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: in thread 7fc111af2700 thread_name:ms_dispatch 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc11f6acd20] 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2: gsignal() 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 3: abort() 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d2e4466268] 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d2e45020fc] 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d2e4836f0a] 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 7: (MDLog::trim()+0xca9) [0x55d2e48380c9] 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d2e48385a8] 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc11e598b23] 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc11f6a21ca] 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 11: clone() 2024-04-12T21:33:02.058 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 0> 2024-04-12T21:33:01.877+0000 7fc111af2700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: in thread 7fc111af2700 thread_name:ms_dispatch 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc11f6acd20] 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2: gsignal() 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 3: abort() 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d2e4466268] 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d2e45020fc] 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d2e4836f0a] 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 7: (MDLog::trim()+0xca9) [0x55d2e48380c9] 2024-04-12T21:33:02.059 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d2e48385a8] 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc11e598b23] 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc11f6a21ca] 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 11: clone() 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: -23> 2024-04-12T21:33:01.877+0000 7fc111af2700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: in thread 7fc111af2700 thread_name:ms_dispatch 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc11f6acd20] 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2: gsignal() 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 3: abort() 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d2e4466268] 2024-04-12T21:33:02.060 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d2e45020fc] 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d2e4836f0a] 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 7: (MDLog::trim()+0xca9) [0x55d2e48380c9] 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d2e48385a8] 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fc11e598b23] 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fc11f6a21ca] 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 11: clone() 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l[75688]: 2024-04-12T21:33:02.061 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:01 smithi153 podman[75858]: 2024-04-12 21:33:01.948017628 +0000 UTC m=+0.019051232 container died a8b5bea4e3388518493d76d89fce82ee533398daad38b323784748db433fdcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-12T21:33:02.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:01 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:clientreplay 2024-04-12T21:33:02.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:01 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=l=up:clientreplay} 4 up:standby 2024-04-12T21:33:02.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:01 smithi104 ceph-mon[36624]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:33:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:01 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:clientreplay 2024-04-12T21:33:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:01 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=l=up:clientreplay} 4 up:standby 2024-04-12T21:33:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:01 smithi081 ceph-mon[31453]: daemon mds.l is now active in filesystem cephfs as rank 0 2024-04-12T21:33:02.936 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:02 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:02.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:02 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:active 2024-04-12T21:33:02.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:02 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 4 up:standby 2024-04-12T21:33:02.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:02 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 3 up:standby 2024-04-12T21:33:02.937 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:02 smithi153 ceph-mon[36569]: pgmap v966: 129 pgs: 129 active+clean; 551 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-12T21:33:02.937 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:02 smithi153 podman[75858]: 2024-04-12 21:33:02.67877214 +0000 UTC m=+0.749805746 container remove a8b5bea4e3388518493d76d89fce82ee533398daad38b323784748db433fdcd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:33:02.937 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:33:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:02 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:02 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:active 2024-04-12T21:33:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:02 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 4 up:standby 2024-04-12T21:33:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:02 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 3 up:standby 2024-04-12T21:33:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:02 smithi104 ceph-mon[36624]: pgmap v966: 129 pgs: 129 active+clean; 551 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-12T21:33:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:02 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:02 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.153:6832/1923941583,v1:172.21.15.153:6833/1923941583] up:active 2024-04-12T21:33:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:02 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 4 up:standby 2024-04-12T21:33:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:02 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=l=up:active} 1 up:standby-replay 3 up:standby 2024-04-12T21:33:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:02 smithi081 ceph-mon[31453]: pgmap v966: 129 pgs: 129 active+clean; 551 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.8 KiB/s wr, 12 op/s 2024-04-12T21:33:03.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-12T21:33:03.305 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:02 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.378s CPU time. 2024-04-12T21:33:05.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:05 smithi153 ceph-mon[36569]: pgmap v967: 129 pgs: 129 active+clean; 551 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.0 KiB/s wr, 17 op/s 2024-04-12T21:33:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:05 smithi104 ceph-mon[36624]: pgmap v967: 129 pgs: 129 active+clean; 551 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.0 KiB/s wr, 17 op/s 2024-04-12T21:33:05.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:05 smithi081 ceph-mon[31453]: pgmap v967: 129 pgs: 129 active+clean; 551 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.0 KiB/s wr, 17 op/s 2024-04-12T21:33:07.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:07 smithi153 ceph-mon[36569]: pgmap v968: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 KiB/s wr, 28 op/s 2024-04-12T21:33:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:07 smithi104 ceph-mon[36624]: pgmap v968: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 KiB/s wr, 28 op/s 2024-04-12T21:33:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:07 smithi081 ceph-mon[31453]: pgmap v968: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 12 KiB/s wr, 28 op/s 2024-04-12T21:33:08.563 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:08.564+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:33:08.563 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:08.564+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:33:08.563 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:08.564+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:33:09.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:09 smithi153 ceph-mon[36569]: pgmap v969: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 KiB/s wr, 27 op/s 2024-04-12T21:33:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:09 smithi104 ceph-mon[36624]: pgmap v969: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 KiB/s wr, 27 op/s 2024-04-12T21:33:09.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:09 smithi081 ceph-mon[31453]: pgmap v969: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 KiB/s wr, 27 op/s 2024-04-12T21:33:11.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:11 smithi153 ceph-mon[36569]: pgmap v970: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 KiB/s wr, 25 op/s 2024-04-12T21:33:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:11 smithi104 ceph-mon[36624]: pgmap v970: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 KiB/s wr, 25 op/s 2024-04-12T21:33:11.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:11 smithi081 ceph-mon[31453]: pgmap v970: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 KiB/s wr, 25 op/s 2024-04-12T21:33:13.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:13 smithi153 ceph-mon[36569]: pgmap v971: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 11 KiB/s wr, 27 op/s 2024-04-12T21:33:13.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:13 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:33:13.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:13 smithi153 systemd[1]: Stopped Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:33:13.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:13 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Consumed 23.378s CPU time. 2024-04-12T21:33:13.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:13 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Start request repeated too quickly. 2024-04-12T21:33:13.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:13 smithi153 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-12T21:33:13.556 INFO:journalctl@ceph.mds.l.smithi153.stdout:Apr 12 21:33:13 smithi153 systemd[1]: Failed to start Ceph mds.l for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:33:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:13 smithi104 ceph-mon[36624]: pgmap v971: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 11 KiB/s wr, 27 op/s 2024-04-12T21:33:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:13 smithi081 ceph-mon[31453]: pgmap v971: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 11 KiB/s wr, 27 op/s 2024-04-12T21:33:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:15 smithi153 ceph-mon[36569]: pgmap v972: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 9.1 KiB/s wr, 19 op/s 2024-04-12T21:33:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:15 smithi104 ceph-mon[36624]: pgmap v972: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 9.1 KiB/s wr, 19 op/s 2024-04-12T21:33:15.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:15 smithi081 ceph-mon[31453]: pgmap v972: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 9.1 KiB/s wr, 19 op/s 2024-04-12T21:33:17.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:17 smithi153 ceph-mon[36569]: pgmap v973: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 8.9 KiB/s wr, 15 op/s 2024-04-12T21:33:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:17 smithi104 ceph-mon[36624]: pgmap v973: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 8.9 KiB/s wr, 15 op/s 2024-04-12T21:33:17.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:17 smithi081 ceph-mon[31453]: pgmap v973: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 8.9 KiB/s wr, 15 op/s 2024-04-12T21:33:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:18 smithi153 ceph-mon[36569]: Replacing daemon mds.l as rank 0 with standby daemon mds.a 2024-04-12T21:33:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:18 smithi153 ceph-mon[36569]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-12T21:33:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:18 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:18 smithi153 ceph-mon[36569]: osdmap e139: 12 total, 12 up, 12 in 2024-04-12T21:33:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:18 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:33:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:18 smithi104 ceph-mon[36624]: Replacing daemon mds.l as rank 0 with standby daemon mds.a 2024-04-12T21:33:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:18 smithi104 ceph-mon[36624]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-12T21:33:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:18 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:18 smithi104 ceph-mon[36624]: osdmap e139: 12 total, 12 up, 12 in 2024-04-12T21:33:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:18 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:33:18.603 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:18 smithi081 ceph-mon[31453]: Replacing daemon mds.l as rank 0 with standby daemon mds.a 2024-04-12T21:33:18.603 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:18 smithi081 ceph-mon[31453]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-04-12T21:33:18.603 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:18 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:18.603 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:18 smithi081 ceph-mon[31453]: osdmap e139: 12 total, 12 up, 12 in 2024-04-12T21:33:18.604 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:18 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:33:19.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:19 smithi081 ceph-mon[31453]: pgmap v975: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:33:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:19 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:reconnect 2024-04-12T21:33:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:19 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:33:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:19 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:33:20.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:19 smithi153 ceph-mon[36569]: pgmap v975: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:33:20.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:19 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:reconnect 2024-04-12T21:33:20.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:19 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:33:20.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:19 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:33:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:19 smithi104 ceph-mon[36624]: pgmap v975: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:33:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:19 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:reconnect 2024-04-12T21:33:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:19 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:33:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:19 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:33:20.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:20 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:rejoin 2024-04-12T21:33:20.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:20 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:33:21.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:20 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:rejoin 2024-04-12T21:33:21.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:20 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:33:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:20 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:rejoin 2024-04-12T21:33:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:20 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:33:21.886 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:33:21.886 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: *** Caught signal (Aborted) ** 2024-04-12T21:33:21.886 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: in thread 7f0b87c00700 thread_name:ms_dispatch 2024-04-12T21:33:21.886 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:21.886 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0b97bd2d20] 2024-04-12T21:33:21.886 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2: gsignal() 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 3: abort() 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564fdb086268] 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564fdb1220fc] 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564fdb456f0a] 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 7: (MDLog::trim()+0xca9) [0x564fdb4580c9] 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564fdb4585a8] 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0b96abeb23] 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0b97bc81ca] 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 11: clone() 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2024-04-12T21:33:21.823+0000 7f0b87c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: in thread 7f0b87c00700 thread_name:ms_dispatch 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2024-04-12T21:33:21.887 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0b97bd2d20] 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2: gsignal() 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 3: abort() 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564fdb086268] 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564fdb1220fc] 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564fdb456f0a] 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 7: (MDLog::trim()+0xca9) [0x564fdb4580c9] 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564fdb4585a8] 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0b96abeb23] 2024-04-12T21:33:21.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-mon[31453]: pgmap v976: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:33:21.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:clientreplay 2024-04-12T21:33:21.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:33:21.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-mon[31453]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:33:21.889 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0b97bc81ca] 2024-04-12T21:33:21.889 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 11: clone() 2024-04-12T21:33:21.889 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:21.889 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 0> 2024-04-12T21:33:21.823+0000 7f0b87c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: in thread 7f0b87c00700 thread_name:ms_dispatch 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0b97bd2d20] 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2: gsignal() 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 3: abort() 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564fdb086268] 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564fdb1220fc] 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564fdb456f0a] 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 7: (MDLog::trim()+0xca9) [0x564fdb4580c9] 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564fdb4585a8] 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0b96abeb23] 2024-04-12T21:33:21.890 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0b97bc81ca] 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 11: clone() 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: -18> 2024-04-12T21:33:21.823+0000 7f0b87c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: in thread 7f0b87c00700 thread_name:ms_dispatch 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0b97bd2d20] 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2: gsignal() 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 3: abort() 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564fdb086268] 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564fdb1220fc] 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564fdb456f0a] 2024-04-12T21:33:21.891 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 7: (MDLog::trim()+0xca9) [0x564fdb4580c9] 2024-04-12T21:33:21.892 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564fdb4585a8] 2024-04-12T21:33:21.892 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0b96abeb23] 2024-04-12T21:33:21.892 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0b97bc81ca] 2024-04-12T21:33:21.892 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 11: clone() 2024-04-12T21:33:21.892 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:21.892 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[73099]: 2024-04-12T21:33:21.982 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:21 smithi104 ceph-mon[36624]: pgmap v976: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:33:21.982 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:21 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:clientreplay 2024-04-12T21:33:21.983 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:21 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:33:21.983 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:21 smithi104 ceph-mon[36624]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:33:22.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:21 smithi153 ceph-mon[36569]: pgmap v976: 129 pgs: 129 active+clean; 550 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:33:22.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:21 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:clientreplay 2024-04-12T21:33:22.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:21 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:33:22.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:21 smithi153 ceph-mon[36569]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:33:22.155 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:21 smithi081 podman[90948]: 2024-04-12 21:33:21.973622543 +0000 UTC m=+0.036526974 container died dfcc5c53b4f2b296dc3e2295c408fc86f0ca0efbbf441d85b081929657b2c02a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409) 2024-04-12T21:33:22.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:22 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:22.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:22 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:active 2024-04-12T21:33:22.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:22 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:33:22.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:22 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:33:22.996 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:22 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:22.996 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:22 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:active 2024-04-12T21:33:22.996 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:22 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:33:22.996 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:22 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:33:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:22 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:22 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2944644175,v1:172.21.15.81:6841/2944644175] up:active 2024-04-12T21:33:23.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:22 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:33:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:22 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:33:23.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:22 smithi081 podman[90948]: 2024-04-12 21:33:22.901869121 +0000 UTC m=+0.964773589 container remove dfcc5c53b4f2b296dc3e2295c408fc86f0ca0efbbf441d85b081929657b2c02a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-12T21:33:23.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:22 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:33:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:23.579+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:33:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:23.579+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:33:23.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:23.579+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:33:23.643 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:23 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-12T21:33:23.643 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:23 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 27.270s CPU time. 2024-04-12T21:33:23.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:23 smithi081 ceph-mon[31453]: pgmap v977: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-12T21:33:24.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:23 smithi153 ceph-mon[36569]: pgmap v977: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-12T21:33:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:23 smithi104 ceph-mon[36624]: pgmap v977: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-12T21:33:25.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:25 smithi081 ceph-mon[31453]: pgmap v978: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-12T21:33:26.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:25 smithi153 ceph-mon[36569]: pgmap v978: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-12T21:33:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:25 smithi104 ceph-mon[36624]: pgmap v978: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 1.6 KiB/s wr, 16 op/s 2024-04-12T21:33:27.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:27 smithi081 ceph-mon[31453]: pgmap v979: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-12T21:33:28.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:27 smithi153 ceph-mon[36569]: pgmap v979: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-12T21:33:28.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:27 smithi104 ceph-mon[36624]: pgmap v979: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 KiB/s wr, 25 op/s 2024-04-12T21:33:30.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:29 smithi153 ceph-mon[36569]: pgmap v980: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 KiB/s wr, 25 op/s 2024-04-12T21:33:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:29 smithi104 ceph-mon[36624]: pgmap v980: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 KiB/s wr, 25 op/s 2024-04-12T21:33:30.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:29 smithi081 ceph-mon[31453]: pgmap v980: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 21 KiB/s wr, 25 op/s 2024-04-12T21:33:32.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:31 smithi153 ceph-mon[36569]: pgmap v981: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 19 KiB/s wr, 23 op/s 2024-04-12T21:33:32.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:31 smithi081 ceph-mon[31453]: pgmap v981: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 19 KiB/s wr, 23 op/s 2024-04-12T21:33:32.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:31 smithi104 ceph-mon[36624]: pgmap v981: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 19 KiB/s wr, 23 op/s 2024-04-12T21:33:33.663 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:33 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:33:33.663 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:33 smithi081 systemd[1]: Stopped Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:33:33.663 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:33 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 27.270s CPU time. 2024-04-12T21:33:33.663 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:33 smithi081 systemd[1]: Starting Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:33:34.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:33 smithi153 ceph-mon[36569]: pgmap v982: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 19 KiB/s wr, 24 op/s 2024-04-12T21:33:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:33 smithi104 ceph-mon[36624]: pgmap v982: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 19 KiB/s wr, 24 op/s 2024-04-12T21:33:34.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:33 smithi081 ceph-mon[31453]: pgmap v982: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 19 KiB/s wr, 24 op/s 2024-04-12T21:33:34.155 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:33 smithi081 podman[91037]: 2024-04-12 21:33:33.694088191 +0000 UTC m=+0.033873059 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:33:35.632 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:35 smithi081 podman[91037]: 2024-04-12 21:33:35.370672077 +0000 UTC m=+1.710456974 container create c1ec501da6425e0f2e9a0afa696c0b69aad9d3a9c4f46ec572ab3051462a324c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:33:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:35 smithi081 ceph-mon[31453]: pgmap v983: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 18 KiB/s wr, 17 op/s 2024-04-12T21:33:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:33:35.699+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:33:35.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:35 smithi081 podman[91037]: 2024-04-12 21:33:35.633719884 +0000 UTC m=+1.973504818 container init c1ec501da6425e0f2e9a0afa696c0b69aad9d3a9c4f46ec572ab3051462a324c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:33:35.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:35 smithi081 podman[91037]: 2024-04-12 21:33:35.643884604 +0000 UTC m=+1.983669508 container start c1ec501da6425e0f2e9a0afa696c0b69aad9d3a9c4f46ec572ab3051462a324c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-12T21:33:35.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:35 smithi081 ceph-mds[91056]: -- 172.21.15.81:0/4010612774 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557e05b824e0 con 0x557e05b59000 2024-04-12T21:33:35.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:35 smithi081 ceph-mds[91056]: -- 172.21.15.81:0/4010612774 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e05b85a40 con 0x557e05b59000 2024-04-12T21:33:35.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: starting mds.a at 2024-04-12T21:33:35.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:35 smithi081 bash[91037]: c1ec501da6425e0f2e9a0afa696c0b69aad9d3a9c4f46ec572ab3051462a324c 2024-04-12T21:33:35.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:35 smithi081 systemd[1]: Started Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:33:36.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:35 smithi153 ceph-mon[36569]: pgmap v983: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 18 KiB/s wr, 17 op/s 2024-04-12T21:33:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:35 smithi104 ceph-mon[36624]: pgmap v983: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 18 KiB/s wr, 17 op/s 2024-04-12T21:33:37.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: daemon mds.a restarted 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: osdmap e140: 12 total, 12 up, 12 in 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:boot 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:36 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:33:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: daemon mds.a restarted 2024-04-12T21:33:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:33:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: osdmap e140: 12 total, 12 up, 12 in 2024-04-12T21:33:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:boot 2024-04-12T21:33:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:33:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:33:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:33:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:33:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:36 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:33:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: daemon mds.a restarted 2024-04-12T21:33:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:33:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: osdmap e140: 12 total, 12 up, 12 in 2024-04-12T21:33:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:boot 2024-04-12T21:33:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:33:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:33:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:33:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:33:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:36 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:33:38.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:37 smithi153 ceph-mon[36569]: pgmap v985: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 307 B/s wr, 6 op/s 2024-04-12T21:33:38.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:37 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:reconnect 2024-04-12T21:33:38.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:37 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:33:38.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:37 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:33:38.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:37 smithi104 ceph-mon[36624]: pgmap v985: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 307 B/s wr, 6 op/s 2024-04-12T21:33:38.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:37 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:reconnect 2024-04-12T21:33:38.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:37 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:33:38.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:37 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:33:38.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:37 smithi081 ceph-mon[31453]: pgmap v985: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 307 B/s wr, 6 op/s 2024-04-12T21:33:38.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:37 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:reconnect 2024-04-12T21:33:38.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:37 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:33:38.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:37 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:33:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:38.595+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:33:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:38.595+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:33:38.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:38.595+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:33:39.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:38 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:rejoin 2024-04-12T21:33:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:38 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:33:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:38 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:rejoin 2024-04-12T21:33:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:38 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:33:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:38 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:rejoin 2024-04-12T21:33:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:38 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:33:40.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:39 smithi153 ceph-mon[36569]: pgmap v986: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:33:40.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:39 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:clientreplay 2024-04-12T21:33:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:39 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:33:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:39 smithi153 ceph-mon[36569]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:33:40.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:39 smithi104 ceph-mon[36624]: pgmap v986: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:33:40.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:39 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:clientreplay 2024-04-12T21:33:40.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:39 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:33:40.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:39 smithi104 ceph-mon[36624]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:33:40.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:39 smithi081 ceph-mon[31453]: pgmap v986: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:33:40.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:39 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:clientreplay 2024-04-12T21:33:40.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:39 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:33:40.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:39 smithi081 ceph-mon[31453]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:33:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:active 2024-04-12T21:33:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:33:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:33:40.907 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: *** Caught signal (Aborted) ** 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: in thread 7fb25d600700 thread_name:ms_dispatch 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb26d616d20] 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2: gsignal() 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 3: abort() 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x559b5c35d268] 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x559b5c3f90fc] 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x559b5c72df0a] 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 7: (MDLog::trim()+0xca9) [0x559b5c72f0c9] 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 8: (MDLog::log_trim_upkeep()+0x358) [0x559b5c72f5a8] 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb26c502b23] 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb26d60c1ca] 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 11: clone() 2024-04-12T21:33:40.908 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2024-04-12T21:33:40.649+0000 7fb25d600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: in thread 7fb25d600700 thread_name:ms_dispatch 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb26d616d20] 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2: gsignal() 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 3: abort() 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x559b5c35d268] 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x559b5c3f90fc] 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x559b5c72df0a] 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 7: (MDLog::trim()+0xca9) [0x559b5c72f0c9] 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 8: (MDLog::log_trim_upkeep()+0x358) [0x559b5c72f5a8] 2024-04-12T21:33:40.909 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb26c502b23] 2024-04-12T21:33:40.910 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb26d60c1ca] 2024-04-12T21:33:40.910 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 11: clone() 2024-04-12T21:33:40.910 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:40.910 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 0> 2024-04-12T21:33:40.649+0000 7fb25d600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: in thread 7fb25d600700 thread_name:ms_dispatch 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb26d616d20] 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2: gsignal() 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 3: abort() 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x559b5c35d268] 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x559b5c3f90fc] 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x559b5c72df0a] 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 7: (MDLog::trim()+0xca9) [0x559b5c72f0c9] 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 8: (MDLog::log_trim_upkeep()+0x358) [0x559b5c72f5a8] 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb26c502b23] 2024-04-12T21:33:40.911 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb26d60c1ca] 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 11: clone() 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 0> 2024-04-12T21:33:40.649+0000 7fb25d600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: in thread 7fb25d600700 thread_name:ms_dispatch 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fb26d616d20] 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2: gsignal() 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 3: abort() 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x559b5c35d268] 2024-04-12T21:33:40.912 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x559b5c3f90fc] 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x559b5c72df0a] 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 7: (MDLog::trim()+0xca9) [0x559b5c72f0c9] 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 8: (MDLog::log_trim_upkeep()+0x358) [0x559b5c72f5a8] 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fb26c502b23] 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fb26d60c1ca] 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 11: clone() 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[72754]: 2024-04-12T21:33:40.913 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:40 smithi081 podman[91091]: 2024-04-12 21:33:40.816805217 +0000 UTC m=+0.038822610 container died 2a629bf61cef78527afcdb7eaf83f62eec80c3d33cb94d66d00a01e37ad7bfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2) 2024-04-12T21:33:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:40 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:40 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:active 2024-04-12T21:33:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:40 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:33:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:40 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:33:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:40 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:33:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:40 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/1348294553,v1:172.21.15.81:6839/1348294553] up:active 2024-04-12T21:33:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:40 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:33:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:40 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:33:41.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:41 smithi081 ceph-mon[31453]: pgmap v987: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:33:41.905 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:41 smithi081 podman[91091]: 2024-04-12 21:33:41.571691324 +0000 UTC m=+0.793708725 container remove 2a629bf61cef78527afcdb7eaf83f62eec80c3d33cb94d66d00a01e37ad7bfc9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-12T21:33:41.905 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:41 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:33:42.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:41 smithi153 ceph-mon[36569]: pgmap v987: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:33:42.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:41 smithi104 ceph-mon[36624]: pgmap v987: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:33:42.405 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:42 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-12T21:33:42.405 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:42 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 28.084s CPU time. 2024-04-12T21:33:43.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:42 smithi153 ceph-mon[36569]: pgmap v988: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 73 KiB/s wr, 17 op/s 2024-04-12T21:33:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:42 smithi104 ceph-mon[36624]: pgmap v988: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 73 KiB/s wr, 17 op/s 2024-04-12T21:33:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:42 smithi081 ceph-mon[31453]: pgmap v988: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 73 KiB/s wr, 17 op/s 2024-04-12T21:33:45.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:45 smithi153 ceph-mon[36569]: pgmap v989: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 73 KiB/s wr, 18 op/s 2024-04-12T21:33:45.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:45 smithi104 ceph-mon[36624]: pgmap v989: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 73 KiB/s wr, 18 op/s 2024-04-12T21:33:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:45 smithi081 ceph-mon[31453]: pgmap v989: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 73 KiB/s wr, 18 op/s 2024-04-12T21:33:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:47 smithi153 ceph-mon[36569]: pgmap v990: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 70 KiB/s wr, 18 op/s 2024-04-12T21:33:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:33:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:47 smithi104 ceph-mon[36624]: pgmap v990: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 70 KiB/s wr, 18 op/s 2024-04-12T21:33:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:33:47.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:47 smithi081 ceph-mon[31453]: pgmap v990: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 70 KiB/s wr, 18 op/s 2024-04-12T21:33:47.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:33:48.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:33:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:33:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:48 smithi153 ceph-mon[36569]: pgmap v991: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 64 KiB/s wr, 16 op/s 2024-04-12T21:33:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:48 smithi153 ceph-mon[36569]: pgmap v992: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 78 KiB/s wr, 20 op/s 2024-04-12T21:33:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:33:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:33:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:33:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:48 smithi104 ceph-mon[36624]: pgmap v991: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 64 KiB/s wr, 16 op/s 2024-04-12T21:33:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:48 smithi104 ceph-mon[36624]: pgmap v992: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 78 KiB/s wr, 20 op/s 2024-04-12T21:33:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:33:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:33:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:33:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:48 smithi081 ceph-mon[31453]: pgmap v991: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 64 KiB/s wr, 16 op/s 2024-04-12T21:33:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:48 smithi081 ceph-mon[31453]: pgmap v992: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 78 KiB/s wr, 20 op/s 2024-04-12T21:33:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:33:51.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:50 smithi153 ceph-mon[36569]: pgmap v993: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:33:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:50 smithi104 ceph-mon[36624]: pgmap v993: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:33:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:50 smithi081 ceph-mon[31453]: pgmap v993: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:33:52.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:52 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:33:52.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:52 smithi081 systemd[1]: Stopped Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:33:52.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:52 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 28.084s CPU time. 2024-04-12T21:33:52.546 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:52 smithi081 systemd[1]: Starting Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:33:52.905 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:52 smithi081 podman[91310]: 2024-04-12 21:33:52.453103685 +0000 UTC m=+0.033222076 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:33:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:52 smithi081 ceph-mon[31453]: pgmap v994: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 7 op/s 2024-04-12T21:33:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:52 smithi153 ceph-mon[36569]: pgmap v994: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 7 op/s 2024-04-12T21:33:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:52 smithi104 ceph-mon[36624]: pgmap v994: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s rd, 7 op/s 2024-04-12T21:33:53.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:53.610+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:33:53.610 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:53.610+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:33:53.610 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:33:53.610+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:33:54.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:54 smithi081 ceph-mon[31453]: pgmap v995: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5 op/s 2024-04-12T21:33:54.693 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:54 smithi081 podman[91310]: 2024-04-12 21:33:54.423966912 +0000 UTC m=+2.004085350 container create f97415d0b679ba163f9e1f5f299ba7d67464fbeabba6d200bcab9d75e94c4733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-12T21:33:54.693 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:54 smithi081 podman[91310]: 2024-04-12 21:33:54.69426624 +0000 UTC m=+2.274384672 container init f97415d0b679ba163f9e1f5f299ba7d67464fbeabba6d200bcab9d75e94c4733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409) 2024-04-12T21:33:55.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:54 smithi153 ceph-mon[36569]: pgmap v995: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5 op/s 2024-04-12T21:33:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:54 smithi104 ceph-mon[36624]: pgmap v995: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5 op/s 2024-04-12T21:33:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:54 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:33:54.759+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:33:55.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:54 smithi081 podman[91310]: 2024-04-12 21:33:54.705798465 +0000 UTC m=+2.285916872 container start f97415d0b679ba163f9e1f5f299ba7d67464fbeabba6d200bcab9d75e94c4733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-12T21:33:55.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:54 smithi081 ceph-mds[91328]: -- 172.21.15.81:0/3437834366 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561c86251a40 con 0x561c86224800 2024-04-12T21:33:55.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:54 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: starting mds.g at 2024-04-12T21:33:55.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:54 smithi081 bash[91310]: f97415d0b679ba163f9e1f5f299ba7d67464fbeabba6d200bcab9d75e94c4733 2024-04-12T21:33:55.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:33:54 smithi081 systemd[1]: Started Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: daemon mds.g restarted 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: osdmap e141: 12 total, 12 up, 12 in 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:boot 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:33:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:55 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:33:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: daemon mds.g restarted 2024-04-12T21:33:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:33:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: osdmap e141: 12 total, 12 up, 12 in 2024-04-12T21:33:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:boot 2024-04-12T21:33:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:33:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:33:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:33:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:33:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:55 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:33:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: daemon mds.g restarted 2024-04-12T21:33:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:33:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:33:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: osdmap e141: 12 total, 12 up, 12 in 2024-04-12T21:33:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:boot 2024-04-12T21:33:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:33:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:33:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:33:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:33:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:55 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:33:57.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:56 smithi153 ceph-mon[36569]: pgmap v997: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:33:57.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:56 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:reconnect 2024-04-12T21:33:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:56 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:33:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:56 smithi104 ceph-mon[36624]: pgmap v997: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:33:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:56 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:reconnect 2024-04-12T21:33:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:56 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:33:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:56 smithi081 ceph-mon[31453]: pgmap v997: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:33:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:56 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:reconnect 2024-04-12T21:33:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:56 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:33:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:57 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999981 2024-04-12T21:33:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:57 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:rejoin 2024-04-12T21:33:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:57 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:33:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:57 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999981 2024-04-12T21:33:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:57 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:rejoin 2024-04-12T21:33:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:57 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:33:58.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:57 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.00999981 2024-04-12T21:33:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:57 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:rejoin 2024-04-12T21:33:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:57 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:33:59.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:58 smithi153 ceph-mon[36569]: pgmap v998: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4 op/s 2024-04-12T21:33:59.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:58 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:clientreplay 2024-04-12T21:33:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:58 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:33:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:58 smithi153 ceph-mon[36569]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:33:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:58 smithi104 ceph-mon[36624]: pgmap v998: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4 op/s 2024-04-12T21:33:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:58 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:clientreplay 2024-04-12T21:33:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:58 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:33:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:58 smithi104 ceph-mon[36624]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:33:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:58 smithi081 ceph-mon[31453]: pgmap v998: 129 pgs: 129 active+clean; 549 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4 op/s 2024-04-12T21:33:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:58 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:clientreplay 2024-04-12T21:33:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:58 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:33:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:58 smithi081 ceph-mon[31453]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:34:00.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:59 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:59 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:active 2024-04-12T21:34:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:59 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:34:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:33:59 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:59 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:59 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:active 2024-04-12T21:34:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:59 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:34:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:33:59 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:00.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:00.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3590160334,v1:172.21.15.81:6841/3590160334] up:active 2024-04-12T21:34:00.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:34:00.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:00.146 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:34:00.146 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: *** Caught signal (Aborted) ** 2024-04-12T21:34:00.146 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: in thread 7fcfd2c00700 thread_name:ms_dispatch 2024-04-12T21:34:00.146 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:00.146 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcfe2d0bd20] 2024-04-12T21:34:00.146 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2: gsignal() 2024-04-12T21:34:00.146 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 3: abort() 2024-04-12T21:34:00.146 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x557dd252b268] 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x557dd25c70fc] 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x557dd28fbf0a] 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 7: (MDLog::trim()+0xca9) [0x557dd28fd0c9] 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 8: (MDLog::log_trim_upkeep()+0x358) [0x557dd28fd5a8] 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fcfe1bf7b23] 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fcfe2d011ca] 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 11: clone() 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2024-04-12T21:33:59.773+0000 7fcfd2c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: in thread 7fcfd2c00700 thread_name:ms_dispatch 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcfe2d0bd20] 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2: gsignal() 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 3: abort() 2024-04-12T21:34:00.147 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x557dd252b268] 2024-04-12T21:34:00.148 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x557dd25c70fc] 2024-04-12T21:34:00.148 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x557dd28fbf0a] 2024-04-12T21:34:00.148 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 7: (MDLog::trim()+0xca9) [0x557dd28fd0c9] 2024-04-12T21:34:00.148 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 8: (MDLog::log_trim_upkeep()+0x358) [0x557dd28fd5a8] 2024-04-12T21:34:00.148 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fcfe1bf7b23] 2024-04-12T21:34:00.148 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fcfe2d011ca] 2024-04-12T21:34:00.148 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 11: clone() 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 0> 2024-04-12T21:33:59.773+0000 7fcfd2c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: in thread 7fcfd2c00700 thread_name:ms_dispatch 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcfe2d0bd20] 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2: gsignal() 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 3: abort() 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x557dd252b268] 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x557dd25c70fc] 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x557dd28fbf0a] 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 7: (MDLog::trim()+0xca9) [0x557dd28fd0c9] 2024-04-12T21:34:00.149 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 8: (MDLog::log_trim_upkeep()+0x358) [0x557dd28fd5a8] 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fcfe1bf7b23] 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fcfe2d011ca] 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 11: clone() 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: -9> 2024-04-12T21:33:59.773+0000 7fcfd2c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: in thread 7fcfd2c00700 thread_name:ms_dispatch 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcfe2d0bd20] 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2: gsignal() 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 3: abort() 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x557dd252b268] 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x557dd25c70fc] 2024-04-12T21:34:00.150 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x557dd28fbf0a] 2024-04-12T21:34:00.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 7: (MDLog::trim()+0xca9) [0x557dd28fd0c9] 2024-04-12T21:34:00.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 8: (MDLog::log_trim_upkeep()+0x358) [0x557dd28fd5a8] 2024-04-12T21:34:00.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fcfe1bf7b23] 2024-04-12T21:34:00.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fcfe2d011ca] 2024-04-12T21:34:00.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 11: clone() 2024-04-12T21:34:00.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:00.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91052]: 2024-04-12T21:34:00.151 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:33:59 smithi081 podman[91363]: 2024-04-12 21:33:59.904739974 +0000 UTC m=+0.034756470 container died c1ec501da6425e0f2e9a0afa696c0b69aad9d3a9c4f46ec572ab3051462a324c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-12T21:34:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:00 smithi081 ceph-mon[31453]: pgmap v999: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T21:34:00.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:00 smithi081 podman[91363]: 2024-04-12 21:34:00.549788296 +0000 UTC m=+0.679804835 container remove c1ec501da6425e0f2e9a0afa696c0b69aad9d3a9c4f46ec572ab3051462a324c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:34:00.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:00 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:34:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:00 smithi153 ceph-mon[36569]: pgmap v999: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T21:34:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:00 smithi104 ceph-mon[36624]: pgmap v999: 129 pgs: 129 active+clean; 548 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 0 B/s wr, 9 op/s 2024-04-12T21:34:01.405 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:00 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-12T21:34:01.405 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:00 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 28.064s CPU time. 2024-04-12T21:34:03.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:02 smithi153 ceph-mon[36569]: pgmap v1000: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 111 KiB/s wr, 37 op/s 2024-04-12T21:34:03.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:02 smithi081 ceph-mon[31453]: pgmap v1000: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 111 KiB/s wr, 37 op/s 2024-04-12T21:34:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:02 smithi104 ceph-mon[36624]: pgmap v1000: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 111 KiB/s wr, 37 op/s 2024-04-12T21:34:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:04 smithi153 ceph-mon[36569]: pgmap v1001: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 111 KiB/s wr, 37 op/s 2024-04-12T21:34:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:04 smithi104 ceph-mon[36624]: pgmap v1001: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 111 KiB/s wr, 37 op/s 2024-04-12T21:34:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:04 smithi081 ceph-mon[31453]: pgmap v1001: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 111 KiB/s wr, 37 op/s 2024-04-12T21:34:07.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:06 smithi153 ceph-mon[36569]: pgmap v1002: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 KiB/s wr, 36 op/s 2024-04-12T21:34:07.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:06 smithi104 ceph-mon[36624]: pgmap v1002: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 KiB/s wr, 36 op/s 2024-04-12T21:34:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:06 smithi081 ceph-mon[31453]: pgmap v1002: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 102 KiB/s wr, 36 op/s 2024-04-12T21:34:08.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:08.625+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:34:08.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:08.625+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:34:08.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:08.625+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:34:09.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:08 smithi153 ceph-mon[36569]: pgmap v1003: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 92 KiB/s wr, 33 op/s 2024-04-12T21:34:09.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:08 smithi104 ceph-mon[36624]: pgmap v1003: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 92 KiB/s wr, 33 op/s 2024-04-12T21:34:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:08 smithi081 ceph-mon[31453]: pgmap v1003: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 92 KiB/s wr, 33 op/s 2024-04-12T21:34:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:10 smithi153 ceph-mon[36569]: pgmap v1004: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 92 KiB/s wr, 33 op/s 2024-04-12T21:34:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:10 smithi104 ceph-mon[36624]: pgmap v1004: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 92 KiB/s wr, 33 op/s 2024-04-12T21:34:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:10 smithi081 ceph-mon[31453]: pgmap v1004: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 92 KiB/s wr, 33 op/s 2024-04-12T21:34:11.562 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:11 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:34:11.562 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:11 smithi081 systemd[1]: Stopped Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:34:11.562 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:11 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 28.064s CPU time. 2024-04-12T21:34:11.562 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:11 smithi081 systemd[1]: Starting Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:34:11.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:11 smithi081 podman[91460]: 2024-04-12 21:34:11.468271884 +0000 UTC m=+0.036116761 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:34:11.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:11 smithi081 podman[91460]: 2024-04-12 21:34:11.74354266 +0000 UTC m=+0.311387574 container create 8e668aa9f308cdf6b9b4d091bff0d1b55b7fdbdf199284799b593815795701a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:34:12.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:34:12.110+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:12.405 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:12 smithi081 podman[91460]: 2024-04-12 21:34:12.045521746 +0000 UTC m=+0.613366666 container init 8e668aa9f308cdf6b9b4d091bff0d1b55b7fdbdf199284799b593815795701a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-04-12T21:34:12.406 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:12 smithi081 podman[91460]: 2024-04-12 21:34:12.055955997 +0000 UTC m=+0.623800918 container start 8e668aa9f308cdf6b9b4d091bff0d1b55b7fdbdf199284799b593815795701a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-12T21:34:12.406 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mds[91478]: -- 172.21.15.81:0/3006708208 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55af10153a40 con 0x55af10126800 2024-04-12T21:34:12.406 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: starting mds.a at 2024-04-12T21:34:12.406 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:12 smithi081 bash[91460]: 8e668aa9f308cdf6b9b4d091bff0d1b55b7fdbdf199284799b593815795701a4 2024-04-12T21:34:12.406 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:12 smithi081 systemd[1]: Started Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:34:12.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:12.876+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:34:12.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:12.877+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:34:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:12.914+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:34:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:12.914+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:34:13.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: pgmap v1005: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 92 KiB/s wr, 29 op/s 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: daemon mds.a restarted 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: osdmap e142: 12 total, 12 up, 12 in 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:boot 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:34:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:12 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:34:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: pgmap v1005: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 92 KiB/s wr, 29 op/s 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: daemon mds.a restarted 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: osdmap e142: 12 total, 12 up, 12 in 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:boot 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:12 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:34:13.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: pgmap v1005: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 92 KiB/s wr, 29 op/s 2024-04-12T21:34:13.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: daemon mds.a restarted 2024-04-12T21:34:13.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:13.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:13.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: osdmap e142: 12 total, 12 up, 12 in 2024-04-12T21:34:13.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:boot 2024-04-12T21:34:13.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:34:13.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:13.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:13.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:34:13.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:12 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:34:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:14 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:reconnect 2024-04-12T21:34:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:14 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:34:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:14 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:34:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:14 smithi081 ceph-mon[31453]: pgmap v1007: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:34:14.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:14 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:reconnect 2024-04-12T21:34:14.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:14 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:34:14.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:14 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:34:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:14 smithi153 ceph-mon[36569]: pgmap v1007: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:34:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:14 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:reconnect 2024-04-12T21:34:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:14 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:34:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:14 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:34:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:14 smithi104 ceph-mon[36624]: pgmap v1007: 129 pgs: 129 active+clean; 546 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 204 B/s wr, 5 op/s 2024-04-12T21:34:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:rejoin 2024-04-12T21:34:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:15 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:34:15.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:rejoin 2024-04-12T21:34:15.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:15 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:34:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:rejoin 2024-04-12T21:34:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:15 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:34:16.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:clientreplay 2024-04-12T21:34:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:34:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-mon[31453]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:34:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-mon[31453]: pgmap v1008: 129 pgs: 129 active+clean; 545 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 102 B/s wr, 8 op/s 2024-04-12T21:34:16.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:16 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:clientreplay 2024-04-12T21:34:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:16 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:34:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:16 smithi153 ceph-mon[36569]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:34:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:16 smithi153 ceph-mon[36569]: pgmap v1008: 129 pgs: 129 active+clean; 545 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 102 B/s wr, 8 op/s 2024-04-12T21:34:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:16 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:clientreplay 2024-04-12T21:34:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:16 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:34:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:16 smithi104 ceph-mon[36624]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:34:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:16 smithi104 ceph-mon[36624]: pgmap v1008: 129 pgs: 129 active+clean; 545 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 102 B/s wr, 8 op/s 2024-04-12T21:34:17.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:34:17.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: *** Caught signal (Aborted) ** 2024-04-12T21:34:17.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: in thread 7f8be3600700 thread_name:ms_dispatch 2024-04-12T21:34:17.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8bf36d6d20] 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2: gsignal() 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 3: abort() 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561c4aef5268] 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561c4af910fc] 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561c4b2c5f0a] 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 7: (MDLog::trim()+0xca9) [0x561c4b2c70c9] 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561c4b2c75a8] 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f8bf25c2b23] 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8bf36cc1ca] 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 11: clone() 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2024-04-12T21:34:16.822+0000 7f8be3600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: in thread 7f8be3600700 thread_name:ms_dispatch 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2024-04-12T21:34:17.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8bf36d6d20] 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2: gsignal() 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 3: abort() 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561c4aef5268] 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561c4af910fc] 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561c4b2c5f0a] 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 7: (MDLog::trim()+0xca9) [0x561c4b2c70c9] 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561c4b2c75a8] 2024-04-12T21:34:17.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f8bf25c2b23] 2024-04-12T21:34:17.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8bf36cc1ca] 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 11: clone() 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 0> 2024-04-12T21:34:16.822+0000 7f8be3600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: in thread 7f8be3600700 thread_name:ms_dispatch 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8bf36d6d20] 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2: gsignal() 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 3: abort() 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561c4aef5268] 2024-04-12T21:34:17.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561c4af910fc] 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561c4b2c5f0a] 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 7: (MDLog::trim()+0xca9) [0x561c4b2c70c9] 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561c4b2c75a8] 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f8bf25c2b23] 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8bf36cc1ca] 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 11: clone() 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: -1> 2024-04-12T21:34:16.822+0000 7f8be3600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: in thread 7f8be3600700 thread_name:ms_dispatch 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8bf36d6d20] 2024-04-12T21:34:17.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2: gsignal() 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 3: abort() 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x561c4aef5268] 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x561c4af910fc] 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x561c4b2c5f0a] 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 7: (MDLog::trim()+0xca9) [0x561c4b2c70c9] 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 8: (MDLog::log_trim_upkeep()+0x358) [0x561c4b2c75a8] 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f8bf25c2b23] 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f8bf36cc1ca] 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 11: clone() 2024-04-12T21:34:17.164 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:17.165 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91324]: 2024-04-12T21:34:17.165 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:16 smithi081 podman[91514]: 2024-04-12 21:34:16.973418146 +0000 UTC m=+0.041831593 container died f97415d0b679ba163f9e1f5f299ba7d67464fbeabba6d200bcab9d75e94c4733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:34:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:17 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:17 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:active 2024-04-12T21:34:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:17 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:34:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:17 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:17 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:17 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:active 2024-04-12T21:34:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:17 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:34:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:17 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:17.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:17 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:17.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:17 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3762748908,v1:172.21.15.81:6839/3762748908] up:active 2024-04-12T21:34:17.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:17 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:34:17.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:17 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:17.905 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:17 smithi081 podman[91514]: 2024-04-12 21:34:17.642051942 +0000 UTC m=+0.710465435 container remove f97415d0b679ba163f9e1f5f299ba7d67464fbeabba6d200bcab9d75e94c4733 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-12T21:34:17.905 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:17 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:34:18.406 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:18 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-12T21:34:18.406 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:18 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.871s CPU time. 2024-04-12T21:34:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:18 smithi081 ceph-mon[31453]: pgmap v1009: 129 pgs: 129 active+clean; 543 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.8 KiB/s wr, 24 op/s 2024-04-12T21:34:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:18 smithi153 ceph-mon[36569]: pgmap v1009: 129 pgs: 129 active+clean; 543 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.8 KiB/s wr, 24 op/s 2024-04-12T21:34:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:18 smithi104 ceph-mon[36624]: pgmap v1009: 129 pgs: 129 active+clean; 543 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.8 KiB/s wr, 24 op/s 2024-04-12T21:34:21.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:20 smithi153 ceph-mon[36569]: pgmap v1010: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.8 KiB/s wr, 31 op/s 2024-04-12T21:34:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:20 smithi104 ceph-mon[36624]: pgmap v1010: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.8 KiB/s wr, 31 op/s 2024-04-12T21:34:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:20 smithi081 ceph-mon[31453]: pgmap v1010: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.8 KiB/s wr, 31 op/s 2024-04-12T21:34:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:22 smithi153 ceph-mon[36569]: pgmap v1011: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 65 KiB/s wr, 36 op/s 2024-04-12T21:34:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:22 smithi104 ceph-mon[36624]: pgmap v1011: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 65 KiB/s wr, 36 op/s 2024-04-12T21:34:23.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:22 smithi081 ceph-mon[31453]: pgmap v1011: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 65 KiB/s wr, 36 op/s 2024-04-12T21:34:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:23.631+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:34:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:23.631+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:34:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:23.631+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:34:25.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:24 smithi153 ceph-mon[36569]: pgmap v1012: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 57 KiB/s wr, 32 op/s 2024-04-12T21:34:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:24 smithi104 ceph-mon[36624]: pgmap v1012: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 57 KiB/s wr, 32 op/s 2024-04-12T21:34:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:24 smithi081 ceph-mon[31453]: pgmap v1012: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 57 KiB/s wr, 32 op/s 2024-04-12T21:34:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:26 smithi153 ceph-mon[36569]: pgmap v1013: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 54 KiB/s wr, 32 op/s 2024-04-12T21:34:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:26 smithi104 ceph-mon[36624]: pgmap v1013: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 54 KiB/s wr, 32 op/s 2024-04-12T21:34:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:26 smithi081 ceph-mon[31453]: pgmap v1013: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 54 KiB/s wr, 32 op/s 2024-04-12T21:34:28.527 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:28 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:34:28.527 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:28 smithi081 systemd[1]: Stopped Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:34:28.527 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:28 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.871s CPU time. 2024-04-12T21:34:28.527 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:28 smithi081 systemd[1]: Starting Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:34:28.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:28 smithi081 ceph-mon[31453]: pgmap v1014: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-12T21:34:28.905 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:28 smithi081 podman[91603]: 2024-04-12 21:34:28.431972357 +0000 UTC m=+0.032679815 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:34:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:28 smithi153 ceph-mon[36569]: pgmap v1014: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-12T21:34:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:28 smithi104 ceph-mon[36624]: pgmap v1014: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 54 KiB/s wr, 30 op/s 2024-04-12T21:34:30.655 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:30 smithi081 podman[91603]: 2024-04-12 21:34:30.401791441 +0000 UTC m=+2.002498936 container create e9c88c08f6f193faee5175d22ed99a1c4141bbe66de6c3608b0fd91eafd4040c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-12T21:34:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:30 smithi153 ceph-mon[36569]: pgmap v1015: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 52 KiB/s wr, 15 op/s 2024-04-12T21:34:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:30 smithi104 ceph-mon[36624]: pgmap v1015: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 52 KiB/s wr, 15 op/s 2024-04-12T21:34:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:30 smithi081 ceph-mon[31453]: pgmap v1015: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 52 KiB/s wr, 15 op/s 2024-04-12T21:34:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:30 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:34:30.731+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:31.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:30 smithi081 podman[91603]: 2024-04-12 21:34:30.663979817 +0000 UTC m=+2.264687356 container init e9c88c08f6f193faee5175d22ed99a1c4141bbe66de6c3608b0fd91eafd4040c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-12T21:34:31.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:30 smithi081 podman[91603]: 2024-04-12 21:34:30.673646704 +0000 UTC m=+2.274354171 container start e9c88c08f6f193faee5175d22ed99a1c4141bbe66de6c3608b0fd91eafd4040c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, ceph=True) 2024-04-12T21:34:31.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:30 smithi081 ceph-mds[91623]: -- 172.21.15.81:0/3064844474 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563c4b60ba40 con 0x563c4b5dec00 2024-04-12T21:34:31.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:30 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: starting mds.g at 2024-04-12T21:34:31.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:30 smithi081 bash[91603]: e9c88c08f6f193faee5175d22ed99a1c4141bbe66de6c3608b0fd91eafd4040c 2024-04-12T21:34:31.156 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:30 smithi081 systemd[1]: Started Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: daemon mds.g restarted 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: osdmap e143: 12 total, 12 up, 12 in 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:boot 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:34:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:31 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: daemon mds.g restarted 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: osdmap e143: 12 total, 12 up, 12 in 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:boot 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:34:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:31 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: daemon mds.g restarted 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: osdmap e143: 12 total, 12 up, 12 in 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:boot 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:34:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:31 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:34:33.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:32 smithi153 ceph-mon[36569]: pgmap v1017: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:34:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:32 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:reconnect 2024-04-12T21:34:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:32 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:34:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:32 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:34:33.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:32 smithi081 ceph-mon[31453]: pgmap v1017: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:34:33.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:32 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:reconnect 2024-04-12T21:34:33.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:32 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:34:33.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:32 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:34:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:32 smithi104 ceph-mon[36624]: pgmap v1017: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:34:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:32 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:reconnect 2024-04-12T21:34:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:32 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:34:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:32 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:34:34.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:33 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:rejoin 2024-04-12T21:34:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:33 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:34:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:33 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:rejoin 2024-04-12T21:34:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:33 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:34:34.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:33 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:rejoin 2024-04-12T21:34:34.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:33 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:34:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:34 smithi153 ceph-mon[36569]: pgmap v1018: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:34:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:34 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:clientreplay 2024-04-12T21:34:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:34 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:34:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:34 smithi153 ceph-mon[36569]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:34:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:34 smithi104 ceph-mon[36624]: pgmap v1018: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:34:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:34 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:clientreplay 2024-04-12T21:34:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:34 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:34:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:34 smithi104 ceph-mon[36624]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:34:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:34 smithi081 ceph-mon[31453]: pgmap v1018: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:34:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:34 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:clientreplay 2024-04-12T21:34:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:34 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:34:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:34 smithi081 ceph-mon[31453]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:34:35.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:34:35.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: *** Caught signal (Aborted) ** 2024-04-12T21:34:35.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: in thread 7f40da000700 thread_name:ms_dispatch 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40ea0fed20] 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2: gsignal() 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 3: abort() 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55af03f6c268] 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55af040080fc] 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55af0433cf0a] 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 7: (MDLog::trim()+0xca9) [0x55af0433e0c9] 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55af0433e5a8] 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f40e8feab23] 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f40ea0f41ca] 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 11: clone() 2024-04-12T21:34:35.657 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2024-04-12T21:34:35.183+0000 7f40da000700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: in thread 7f40da000700 thread_name:ms_dispatch 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40ea0fed20] 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2: gsignal() 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 3: abort() 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55af03f6c268] 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55af040080fc] 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55af0433cf0a] 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 7: (MDLog::trim()+0xca9) [0x55af0433e0c9] 2024-04-12T21:34:35.658 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55af0433e5a8] 2024-04-12T21:34:35.659 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f40e8feab23] 2024-04-12T21:34:35.659 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f40ea0f41ca] 2024-04-12T21:34:35.659 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 11: clone() 2024-04-12T21:34:35.659 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:35.659 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2024-04-12T21:34:35.659 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 0> 2024-04-12T21:34:35.183+0000 7f40da000700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: in thread 7f40da000700 thread_name:ms_dispatch 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40ea0fed20] 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2: gsignal() 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 3: abort() 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55af03f6c268] 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55af040080fc] 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55af0433cf0a] 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 7: (MDLog::trim()+0xca9) [0x55af0433e0c9] 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55af0433e5a8] 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f40e8feab23] 2024-04-12T21:34:35.660 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f40ea0f41ca] 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 11: clone() 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: -3> 2024-04-12T21:34:35.183+0000 7f40da000700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: in thread 7f40da000700 thread_name:ms_dispatch 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40ea0fed20] 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2: gsignal() 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 3: abort() 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55af03f6c268] 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55af040080fc] 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55af0433cf0a] 2024-04-12T21:34:35.661 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 7: (MDLog::trim()+0xca9) [0x55af0433e0c9] 2024-04-12T21:34:35.662 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55af0433e5a8] 2024-04-12T21:34:35.662 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f40e8feab23] 2024-04-12T21:34:35.662 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f40ea0f41ca] 2024-04-12T21:34:35.662 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 11: clone() 2024-04-12T21:34:35.662 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:35.662 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91474]: 2024-04-12T21:34:35.662 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:35 smithi081 podman[91658]: 2024-04-12 21:34:35.339000781 +0000 UTC m=+0.041602403 container died 8e668aa9f308cdf6b9b4d091bff0d1b55b7fdbdf199284799b593815795701a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-12T21:34:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:35 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:35 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:active 2024-04-12T21:34:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:35 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:34:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:35 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:35 smithi153 ceph-mon[36569]: pgmap v1019: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 818 B/s wr, 11 op/s 2024-04-12T21:34:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:35 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:35 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:active 2024-04-12T21:34:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:35 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:34:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:35 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:35 smithi104 ceph-mon[36624]: pgmap v1019: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 818 B/s wr, 11 op/s 2024-04-12T21:34:36.110 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:36.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/3690792211,v1:172.21.15.81:6841/3690792211] up:active 2024-04-12T21:34:36.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:34:36.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:36.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:35 smithi081 ceph-mon[31453]: pgmap v1019: 129 pgs: 129 active+clean; 542 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 818 B/s wr, 11 op/s 2024-04-12T21:34:36.405 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:36 smithi081 podman[91658]: 2024-04-12 21:34:36.135596003 +0000 UTC m=+0.838197647 container remove 8e668aa9f308cdf6b9b4d091bff0d1b55b7fdbdf199284799b593815795701a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409) 2024-04-12T21:34:36.405 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:36 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:34:36.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:36 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-12T21:34:36.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:36 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 27.089s CPU time. 2024-04-12T21:34:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:38.646+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:34:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:38.646+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:34:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:38.646+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:34:39.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:38 smithi153 ceph-mon[36569]: pgmap v1020: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 38 KiB/s wr, 22 op/s 2024-04-12T21:34:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:38 smithi104 ceph-mon[36624]: pgmap v1020: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 38 KiB/s wr, 22 op/s 2024-04-12T21:34:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:38 smithi081 ceph-mon[31453]: pgmap v1020: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 38 KiB/s wr, 22 op/s 2024-04-12T21:34:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:40 smithi153 ceph-mon[36569]: pgmap v1021: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 38 KiB/s wr, 26 op/s 2024-04-12T21:34:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:40 smithi104 ceph-mon[36624]: pgmap v1021: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 38 KiB/s wr, 26 op/s 2024-04-12T21:34:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:40 smithi081 ceph-mon[31453]: pgmap v1021: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 38 KiB/s wr, 26 op/s 2024-04-12T21:34:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:42 smithi153 ceph-mon[36569]: pgmap v1022: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 35 KiB/s wr, 26 op/s 2024-04-12T21:34:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:42 smithi104 ceph-mon[36624]: pgmap v1022: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 35 KiB/s wr, 26 op/s 2024-04-12T21:34:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:42 smithi081 ceph-mon[31453]: pgmap v1022: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 35 KiB/s wr, 26 op/s 2024-04-12T21:34:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:44 smithi153 ceph-mon[36569]: pgmap v1023: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 32 KiB/s wr, 23 op/s 2024-04-12T21:34:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:44 smithi104 ceph-mon[36624]: pgmap v1023: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 32 KiB/s wr, 23 op/s 2024-04-12T21:34:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:44 smithi081 ceph-mon[31453]: pgmap v1023: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 32 KiB/s wr, 23 op/s 2024-04-12T21:34:47.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:46 smithi081 ceph-mon[31453]: pgmap v1024: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-12T21:34:47.020 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:46 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:34:47.020 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:46 smithi081 systemd[1]: Stopped Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:34:47.021 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:46 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 27.089s CPU time. 2024-04-12T21:34:47.021 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:46 smithi081 systemd[1]: Starting Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:34:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:46 smithi153 ceph-mon[36569]: pgmap v1024: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-12T21:34:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:46 smithi104 ceph-mon[36624]: pgmap v1024: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 32 KiB/s wr, 24 op/s 2024-04-12T21:34:47.405 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:47 smithi081 podman[91748]: 2024-04-12 21:34:46.92694954 +0000 UTC m=+0.032965408 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:34:48.398 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:48 smithi081 podman[91748]: 2024-04-12 21:34:48.120402191 +0000 UTC m=+1.226418072 container create 54649b7d9e1f5b8458a35e43a28f66a74182c6b0fbc7a00393c94a4efd94914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-12T21:34:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:34:48.463+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:48.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:48 smithi081 podman[91748]: 2024-04-12 21:34:48.399578208 +0000 UTC m=+1.505594119 container init 54649b7d9e1f5b8458a35e43a28f66a74182c6b0fbc7a00393c94a4efd94914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-12T21:34:48.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:48 smithi081 podman[91748]: 2024-04-12 21:34:48.408911431 +0000 UTC m=+1.514927305 container start 54649b7d9e1f5b8458a35e43a28f66a74182c6b0fbc7a00393c94a4efd94914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, ceph=True) 2024-04-12T21:34:48.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mds[91834]: -- 172.21.15.81:0/4177804862 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563df7633a40 con 0x563df7607000 2024-04-12T21:34:48.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: starting mds.a at 2024-04-12T21:34:48.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:48 smithi081 bash[91748]: 54649b7d9e1f5b8458a35e43a28f66a74182c6b0fbc7a00393c94a4efd94914c 2024-04-12T21:34:48.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:34:48 smithi081 systemd[1]: Started Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: pgmap v1025: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 31 KiB/s wr, 18 op/s 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: daemon mds.a restarted 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: osdmap e144: 12 total, 12 up, 12 in 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:boot 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:34:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:34:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:48 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:34:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: pgmap v1025: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 31 KiB/s wr, 18 op/s 2024-04-12T21:34:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:34:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: daemon mds.a restarted 2024-04-12T21:34:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: osdmap e144: 12 total, 12 up, 12 in 2024-04-12T21:34:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:boot 2024-04-12T21:34:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:34:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:34:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:48 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:34:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: pgmap v1025: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 31 KiB/s wr, 18 op/s 2024-04-12T21:34:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:34:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: daemon mds.a restarted 2024-04-12T21:34:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:34:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:34:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: osdmap e144: 12 total, 12 up, 12 in 2024-04-12T21:34:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:boot 2024-04-12T21:34:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:34:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:34:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:34:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:34:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:48 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:34:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:34:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:34:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:34:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:34:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:34:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:34:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:34:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:34:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:34:51.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:50 smithi153 ceph-mon[36569]: pgmap v1027: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5 op/s 2024-04-12T21:34:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:50 smithi153 ceph-mon[36569]: pgmap v1028: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:34:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:50 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:reconnect 2024-04-12T21:34:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:50 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:34:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:50 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0149997 2024-04-12T21:34:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:50 smithi104 ceph-mon[36624]: pgmap v1027: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5 op/s 2024-04-12T21:34:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:50 smithi104 ceph-mon[36624]: pgmap v1028: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:34:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:50 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:reconnect 2024-04-12T21:34:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:50 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:34:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:50 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0149997 2024-04-12T21:34:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:50 smithi081 ceph-mon[31453]: pgmap v1027: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 5 op/s 2024-04-12T21:34:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:50 smithi081 ceph-mon[31453]: pgmap v1028: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:34:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:50 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:reconnect 2024-04-12T21:34:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:50 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:34:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:50 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0149997 2024-04-12T21:34:52.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:51 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:rejoin 2024-04-12T21:34:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:51 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:34:52.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:51 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:rejoin 2024-04-12T21:34:52.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:51 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:34:52.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:51 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:rejoin 2024-04-12T21:34:52.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:51 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:34:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-mon[31453]: pgmap v1029: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:34:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:clientreplay 2024-04-12T21:34:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:34:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-mon[31453]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:34:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:active 2024-04-12T21:34:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:52 smithi153 ceph-mon[36569]: pgmap v1029: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:34:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:52 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:clientreplay 2024-04-12T21:34:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:52 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:34:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:52 smithi153 ceph-mon[36569]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:34:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:52 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:52 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:active 2024-04-12T21:34:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:52 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:52 smithi104 ceph-mon[36624]: pgmap v1029: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:34:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:52 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:clientreplay 2024-04-12T21:34:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:52 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:34:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:52 smithi104 ceph-mon[36624]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:34:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:52 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:34:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:52 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3647251695,v1:172.21.15.81:6839/3647251695] up:active 2024-04-12T21:34:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:52 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:34:53.327 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: *** Caught signal (Aborted) ** 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: in thread 7fbf1f600700 thread_name:ms_dispatch 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf2f74dd20] 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2: gsignal() 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 3: abort() 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563c37cbd268] 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563c37d590fc] 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563c3808df0a] 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 7: (MDLog::trim()+0xca9) [0x563c3808f0c9] 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563c3808f5a8] 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbf2e639b23] 2024-04-12T21:34:53.328 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbf2f7431ca] 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 11: clone() 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2024-04-12T21:34:52.925+0000 7fbf1f600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: in thread 7fbf1f600700 thread_name:ms_dispatch 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf2f74dd20] 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2: gsignal() 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 3: abort() 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563c37cbd268] 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563c37d590fc] 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563c3808df0a] 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 7: (MDLog::trim()+0xca9) [0x563c3808f0c9] 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563c3808f5a8] 2024-04-12T21:34:53.329 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbf2e639b23] 2024-04-12T21:34:53.330 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbf2f7431ca] 2024-04-12T21:34:53.330 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 11: clone() 2024-04-12T21:34:53.330 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:53.330 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2024-04-12T21:34:53.330 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 0> 2024-04-12T21:34:52.925+0000 7fbf1f600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:53.330 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: in thread 7fbf1f600700 thread_name:ms_dispatch 2024-04-12T21:34:53.330 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf2f74dd20] 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2: gsignal() 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 3: abort() 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563c37cbd268] 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563c37d590fc] 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563c3808df0a] 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 7: (MDLog::trim()+0xca9) [0x563c3808f0c9] 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563c3808f5a8] 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbf2e639b23] 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbf2f7431ca] 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 11: clone() 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:53.331 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: -10> 2024-04-12T21:34:52.925+0000 7fbf1f600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: in thread 7fbf1f600700 thread_name:ms_dispatch 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbf2f74dd20] 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2: gsignal() 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 3: abort() 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563c37cbd268] 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563c37d590fc] 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563c3808df0a] 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 7: (MDLog::trim()+0xca9) [0x563c3808f0c9] 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563c3808f5a8] 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fbf2e639b23] 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fbf2f7431ca] 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 11: clone() 2024-04-12T21:34:53.332 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:34:53.333 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:52 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[91619]: 2024-04-12T21:34:53.333 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:53 smithi081 podman[91933]: 2024-04-12 21:34:53.064552855 +0000 UTC m=+0.037650115 container died e9c88c08f6f193faee5175d22ed99a1c4141bbe66de6c3608b0fd91eafd4040c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-12T21:34:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:53.661+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:34:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:53.661+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:34:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:34:53.661+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:34:54.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:53 smithi081 podman[91933]: 2024-04-12 21:34:53.788352066 +0000 UTC m=+0.761449370 container remove e9c88c08f6f193faee5175d22ed99a1c4141bbe66de6c3608b0fd91eafd4040c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-12T21:34:54.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:53 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:34:54.655 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:54 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-12T21:34:54.655 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:34:54 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.205s CPU time. 2024-04-12T21:34:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:54 smithi153 ceph-mon[36569]: pgmap v1030: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 136 B/s wr, 3 op/s 2024-04-12T21:34:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:54 smithi104 ceph-mon[36624]: pgmap v1030: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 136 B/s wr, 3 op/s 2024-04-12T21:34:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:54 smithi081 ceph-mon[31453]: pgmap v1030: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 136 B/s wr, 3 op/s 2024-04-12T21:34:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:56 smithi153 ceph-mon[36569]: pgmap v1031: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 410 B/s wr, 8 op/s 2024-04-12T21:34:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:56 smithi104 ceph-mon[36624]: pgmap v1031: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 410 B/s wr, 8 op/s 2024-04-12T21:34:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:56 smithi081 ceph-mon[31453]: pgmap v1031: 129 pgs: 129 active+clean; 541 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 410 B/s wr, 8 op/s 2024-04-12T21:34:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:34:58 smithi153 ceph-mon[36569]: pgmap v1032: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 34 KiB/s wr, 28 op/s 2024-04-12T21:34:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:34:58 smithi104 ceph-mon[36624]: pgmap v1032: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 34 KiB/s wr, 28 op/s 2024-04-12T21:34:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:34:58 smithi081 ceph-mon[31453]: pgmap v1032: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 34 KiB/s wr, 28 op/s 2024-04-12T21:35:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:00 smithi153 ceph-mon[36569]: pgmap v1033: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 29 KiB/s wr, 26 op/s 2024-04-12T21:35:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:00 smithi104 ceph-mon[36624]: pgmap v1033: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 29 KiB/s wr, 26 op/s 2024-04-12T21:35:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:00 smithi081 ceph-mon[31453]: pgmap v1033: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 29 KiB/s wr, 26 op/s 2024-04-12T21:35:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:02 smithi153 ceph-mon[36569]: pgmap v1034: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:02 smithi104 ceph-mon[36624]: pgmap v1034: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:02 smithi081 ceph-mon[31453]: pgmap v1034: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:04.773 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:04 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:35:04.773 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:04 smithi081 systemd[1]: Stopped Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:35:04.773 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:04 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.205s CPU time. 2024-04-12T21:35:04.773 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:04 smithi081 systemd[1]: Starting Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:35:05.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:04 smithi153 ceph-mon[36569]: pgmap v1035: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:04 smithi104 ceph-mon[36624]: pgmap v1035: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:05.130 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:04 smithi081 podman[92020]: 2024-04-12 21:35:04.711144164 +0000 UTC m=+0.035492589 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:35:05.130 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:04 smithi081 ceph-mon[31453]: pgmap v1035: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:06 smithi153 ceph-mon[36569]: pgmap v1036: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:07.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:06 smithi104 ceph-mon[36624]: pgmap v1036: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:06 smithi081 ceph-mon[31453]: pgmap v1036: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-12T21:35:07.434 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:07 smithi081 podman[92020]: 2024-04-12 21:35:07.164761895 +0000 UTC m=+2.489110348 container create 3d440d7d2505a6cb62a2cb9535bcfc6ad72b006ab1825c62615aa5e670891fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:35:07.781 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:35:07.499+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:07.781 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:07 smithi081 podman[92020]: 2024-04-12 21:35:07.43529512 +0000 UTC m=+2.759643597 container init 3d440d7d2505a6cb62a2cb9535bcfc6ad72b006ab1825c62615aa5e670891fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-12T21:35:07.781 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:07 smithi081 podman[92020]: 2024-04-12 21:35:07.444593026 +0000 UTC m=+2.768941466 container start 3d440d7d2505a6cb62a2cb9535bcfc6ad72b006ab1825c62615aa5e670891fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-12T21:35:07.781 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mds[92051]: -- 172.21.15.81:0/2151266367 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5603a529da40 con 0x5603a5271000 2024-04-12T21:35:07.781 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: starting mds.g at 2024-04-12T21:35:07.781 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:07 smithi081 bash[92020]: 3d440d7d2505a6cb62a2cb9535bcfc6ad72b006ab1825c62615aa5e670891fe7 2024-04-12T21:35:07.781 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:07 smithi081 systemd[1]: Started Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: daemon mds.g restarted 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: osdmap e145: 12 total, 12 up, 12 in 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:boot 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:35:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:07 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:35:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: daemon mds.g restarted 2024-04-12T21:35:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: osdmap e145: 12 total, 12 up, 12 in 2024-04-12T21:35:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:boot 2024-04-12T21:35:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:35:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:35:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:07 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:35:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: daemon mds.g restarted 2024-04-12T21:35:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: osdmap e145: 12 total, 12 up, 12 in 2024-04-12T21:35:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:boot 2024-04-12T21:35:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:35:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:35:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:07 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:35:08.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:08.672+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:35:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:08.672+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:35:08.672 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:08.672+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:35:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:08 smithi153 ceph-mon[36569]: pgmap v1037: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-12T21:35:09.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:08 smithi081 ceph-mon[31453]: pgmap v1037: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-12T21:35:09.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:08 smithi104 ceph-mon[36624]: pgmap v1037: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 24 KiB/s wr, 20 op/s 2024-04-12T21:35:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:09 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:reconnect 2024-04-12T21:35:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:09 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:35:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:09 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:35:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:09 smithi104 ceph-mon[36624]: pgmap v1039: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:35:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:09 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:reconnect 2024-04-12T21:35:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:09 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:35:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:09 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:35:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:09 smithi081 ceph-mon[31453]: pgmap v1039: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:35:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:09 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:reconnect 2024-04-12T21:35:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:09 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:35:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:09 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0129998 2024-04-12T21:35:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:09 smithi153 ceph-mon[36569]: pgmap v1039: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:35:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:10 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:rejoin 2024-04-12T21:35:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:10 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:35:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:10 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:rejoin 2024-04-12T21:35:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:10 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:35:11.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:10 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:rejoin 2024-04-12T21:35:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:10 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:35:12.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:11 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:clientreplay 2024-04-12T21:35:12.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:11 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:35:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:11 smithi104 ceph-mon[36624]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:35:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:11 smithi104 ceph-mon[36624]: pgmap v1040: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:35:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:11 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:clientreplay 2024-04-12T21:35:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:11 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:35:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:11 smithi081 ceph-mon[31453]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:35:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:11 smithi081 ceph-mon[31453]: pgmap v1040: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:35:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:11 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:clientreplay 2024-04-12T21:35:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:11 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:35:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:11 smithi153 ceph-mon[36569]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:35:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:11 smithi153 ceph-mon[36569]: pgmap v1040: 129 pgs: 129 active+clean; 540 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:35:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:active 2024-04-12T21:35:12.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:35:12.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:12.903 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:35:12.903 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: *** Caught signal (Aborted) ** 2024-04-12T21:35:12.903 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: in thread 7efc38400700 thread_name:ms_dispatch 2024-04-12T21:35:12.903 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:12.903 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc484fbd20] 2024-04-12T21:35:12.903 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2: gsignal() 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 3: abort() 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563db7f03268] 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563db7f9f0fc] 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563db82d3f0a] 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 7: (MDLog::trim()+0xca9) [0x563db82d50c9] 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563db82d55a8] 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7efc473e7b23] 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7efc484f11ca] 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 11: clone() 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2024-04-12T21:35:12.630+0000 7efc38400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: in thread 7efc38400700 thread_name:ms_dispatch 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2024-04-12T21:35:12.904 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc484fbd20] 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2: gsignal() 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 3: abort() 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563db7f03268] 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563db7f9f0fc] 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563db82d3f0a] 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 7: (MDLog::trim()+0xca9) [0x563db82d50c9] 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563db82d55a8] 2024-04-12T21:35:12.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7efc473e7b23] 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7efc484f11ca] 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 11: clone() 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 0> 2024-04-12T21:35:12.630+0000 7efc38400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: in thread 7efc38400700 thread_name:ms_dispatch 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc484fbd20] 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2: gsignal() 2024-04-12T21:35:12.906 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 3: abort() 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563db7f03268] 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563db7f9f0fc] 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563db82d3f0a] 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 7: (MDLog::trim()+0xca9) [0x563db82d50c9] 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563db82d55a8] 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7efc473e7b23] 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7efc484f11ca] 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 11: clone() 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: -3> 2024-04-12T21:35:12.630+0000 7efc38400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: in thread 7efc38400700 thread_name:ms_dispatch 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:12.907 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc484fbd20] 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2: gsignal() 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 3: abort() 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563db7f03268] 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563db7f9f0fc] 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x563db82d3f0a] 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 7: (MDLog::trim()+0xca9) [0x563db82d50c9] 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 8: (MDLog::log_trim_upkeep()+0x358) [0x563db82d55a8] 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7efc473e7b23] 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7efc484f11ca] 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 11: clone() 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[91830]: 2024-04-12T21:35:12.908 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:12 smithi081 podman[92086]: 2024-04-12 21:35:12.771784823 +0000 UTC m=+0.035599412 container died 54649b7d9e1f5b8458a35e43a28f66a74182c6b0fbc7a00393c94a4efd94914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T21:35:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:12 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:12 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:active 2024-04-12T21:35:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:12 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:35:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:12 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:12 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:12 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2713201939,v1:172.21.15.81:6841/2713201939] up:active 2024-04-12T21:35:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:12 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 3 up:standby 2024-04-12T21:35:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:12 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:13.838 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:13 smithi081 podman[92086]: 2024-04-12 21:35:13.466404408 +0000 UTC m=+0.730218996 container remove 54649b7d9e1f5b8458a35e43a28f66a74182c6b0fbc7a00393c94a4efd94914c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-12T21:35:13.839 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:13 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:35:14.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:13 smithi081 ceph-mon[31453]: pgmap v1041: 129 pgs: 129 active+clean; 539 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-12T21:35:14.155 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:13 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-12T21:35:14.155 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:13 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 27.742s CPU time. 2024-04-12T21:35:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:13 smithi153 ceph-mon[36569]: pgmap v1041: 129 pgs: 129 active+clean; 539 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-12T21:35:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:13 smithi104 ceph-mon[36624]: pgmap v1041: 129 pgs: 129 active+clean; 539 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-12T21:35:16.469 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:16 smithi081 ceph-mon[31453]: pgmap v1042: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.0 KiB/s wr, 18 op/s 2024-04-12T21:35:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:16 smithi153 ceph-mon[36569]: pgmap v1042: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.0 KiB/s wr, 18 op/s 2024-04-12T21:35:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:16 smithi104 ceph-mon[36624]: pgmap v1042: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 2.0 KiB/s wr, 18 op/s 2024-04-12T21:35:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:18 smithi153 ceph-mon[36569]: pgmap v1043: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 74 KiB/s wr, 29 op/s 2024-04-12T21:35:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:18 smithi104 ceph-mon[36624]: pgmap v1043: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 74 KiB/s wr, 29 op/s 2024-04-12T21:35:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:18 smithi081 ceph-mon[31453]: pgmap v1043: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 74 KiB/s wr, 29 op/s 2024-04-12T21:35:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:20 smithi153 ceph-mon[36569]: pgmap v1044: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 63 KiB/s wr, 26 op/s 2024-04-12T21:35:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:20 smithi104 ceph-mon[36624]: pgmap v1044: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 63 KiB/s wr, 26 op/s 2024-04-12T21:35:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:20 smithi081 ceph-mon[31453]: pgmap v1044: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 63 KiB/s wr, 26 op/s 2024-04-12T21:35:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:22 smithi153 ceph-mon[36569]: pgmap v1045: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-12T21:35:22.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:22 smithi104 ceph-mon[36624]: pgmap v1045: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-12T21:35:22.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:22 smithi081 ceph-mon[31453]: pgmap v1045: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 61 KiB/s wr, 27 op/s 2024-04-12T21:35:23.684 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:23.685+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:35:23.685 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:23.685+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:35:23.685 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:23.685+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:35:24.196 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:23 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:35:24.196 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:23 smithi081 systemd[1]: Stopped Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:35:24.197 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:23 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 27.742s CPU time. 2024-04-12T21:35:24.197 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:23 smithi081 systemd[1]: Starting Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:35:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:24 smithi153 ceph-mon[36569]: pgmap v1046: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 26 op/s 2024-04-12T21:35:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:24 smithi104 ceph-mon[36624]: pgmap v1046: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 26 op/s 2024-04-12T21:35:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:24 smithi081 ceph-mon[31453]: pgmap v1046: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 26 op/s 2024-04-12T21:35:24.656 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:24 smithi081 podman[92179]: 2024-04-12 21:35:24.203293259 +0000 UTC m=+0.034223141 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:35:24.938 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:24 smithi081 podman[92179]: 2024-04-12 21:35:24.676777026 +0000 UTC m=+0.507706943 container create 44c7ce40b7a3a5e2841c39924cfdc5bf7cbe96c0bf4bc87f3af63d6e934b2a44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-12T21:35:25.201 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:35:25.001+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:25.201 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:24 smithi081 podman[92179]: 2024-04-12 21:35:24.939615024 +0000 UTC m=+0.770544951 container init 44c7ce40b7a3a5e2841c39924cfdc5bf7cbe96c0bf4bc87f3af63d6e934b2a44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-12T21:35:25.202 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:24 smithi081 podman[92179]: 2024-04-12 21:35:24.948008333 +0000 UTC m=+0.778938273 container start 44c7ce40b7a3a5e2841c39924cfdc5bf7cbe96c0bf4bc87f3af63d6e934b2a44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:35:25.202 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:24 smithi081 ceph-mds[92198]: -- 172.21.15.81:0/3100589394 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562e0a4d3a40 con 0x562e0a4a6800 2024-04-12T21:35:25.202 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:24 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: starting mds.a at 2024-04-12T21:35:25.202 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:25 smithi081 bash[92179]: 44c7ce40b7a3a5e2841c39924cfdc5bf7cbe96c0bf4bc87f3af63d6e934b2a44 2024-04-12T21:35:25.202 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:25 smithi081 systemd[1]: Started Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: daemon mds.a restarted 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: osdmap e146: 12 total, 12 up, 12 in 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:boot 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:25 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:35:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: daemon mds.a restarted 2024-04-12T21:35:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: osdmap e146: 12 total, 12 up, 12 in 2024-04-12T21:35:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:boot 2024-04-12T21:35:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:35:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:35:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:25 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:35:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: daemon mds.a restarted 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: osdmap e146: 12 total, 12 up, 12 in 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:boot 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-12T21:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:25 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:replay} 3 up:standby 2024-04-12T21:35:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:26 smithi153 ceph-mon[36569]: pgmap v1048: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-12T21:35:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:26 smithi104 ceph-mon[36624]: pgmap v1048: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-12T21:35:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:26 smithi081 ceph-mon[31453]: pgmap v1048: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 72 KiB/s wr, 16 op/s 2024-04-12T21:35:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:27 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:reconnect 2024-04-12T21:35:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:27 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:35:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:27 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:35:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:27 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:reconnect 2024-04-12T21:35:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:27 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:35:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:27 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:35:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:27 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:reconnect 2024-04-12T21:35:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:27 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:reconnect} 3 up:standby 2024-04-12T21:35:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:27 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:35:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:28 smithi153 ceph-mon[36569]: pgmap v1049: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:35:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:28 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:rejoin 2024-04-12T21:35:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:28 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:35:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:28 smithi104 ceph-mon[36624]: pgmap v1049: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:35:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:28 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:rejoin 2024-04-12T21:35:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:28 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:35:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:28 smithi081 ceph-mon[31453]: pgmap v1049: 129 pgs: 129 active+clean; 538 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 5 op/s 2024-04-12T21:35:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:28 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:rejoin 2024-04-12T21:35:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:28 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:rejoin} 3 up:standby 2024-04-12T21:35:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:29 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:clientreplay 2024-04-12T21:35:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:29 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:35:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:29 smithi153 ceph-mon[36569]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:35:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:29 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:clientreplay 2024-04-12T21:35:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:29 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:35:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:29 smithi104 ceph-mon[36624]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:35:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:clientreplay 2024-04-12T21:35:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:clientreplay} 3 up:standby 2024-04-12T21:35:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-mon[31453]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:35:30.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:35:30.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: *** Caught signal (Aborted) ** 2024-04-12T21:35:30.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: in thread 7f0aae600700 thread_name:ms_dispatch 2024-04-12T21:35:30.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:30.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0abe646d20] 2024-04-12T21:35:30.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2: gsignal() 2024-04-12T21:35:30.158 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 3: abort() 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560371881268] 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56037191d0fc] 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560371c51f0a] 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 7: (MDLog::trim()+0xca9) [0x560371c530c9] 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560371c535a8] 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0abd532b23] 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0abe63c1ca] 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 11: clone() 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2024-04-12T21:35:29.892+0000 7f0aae600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: in thread 7f0aae600700 thread_name:ms_dispatch 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0abe646d20] 2024-04-12T21:35:30.159 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2: gsignal() 2024-04-12T21:35:30.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 3: abort() 2024-04-12T21:35:30.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560371881268] 2024-04-12T21:35:30.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56037191d0fc] 2024-04-12T21:35:30.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560371c51f0a] 2024-04-12T21:35:30.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 7: (MDLog::trim()+0xca9) [0x560371c530c9] 2024-04-12T21:35:30.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560371c535a8] 2024-04-12T21:35:30.160 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0abd532b23] 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0abe63c1ca] 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 11: clone() 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 0> 2024-04-12T21:35:29.892+0000 7f0aae600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: in thread 7f0aae600700 thread_name:ms_dispatch 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0abe646d20] 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2: gsignal() 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 3: abort() 2024-04-12T21:35:30.161 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560371881268] 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56037191d0fc] 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560371c51f0a] 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 7: (MDLog::trim()+0xca9) [0x560371c530c9] 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560371c535a8] 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0abd532b23] 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0abe63c1ca] 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 11: clone() 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: -1> 2024-04-12T21:35:29.892+0000 7f0aae600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: in thread 7f0aae600700 thread_name:ms_dispatch 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:30.162 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0abe646d20] 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2: gsignal() 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 3: abort() 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x560371881268] 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x56037191d0fc] 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x560371c51f0a] 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 7: (MDLog::trim()+0xca9) [0x560371c530c9] 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 8: (MDLog::log_trim_upkeep()+0x358) [0x560371c535a8] 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0abd532b23] 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0abe63c1ca] 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 11: clone() 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:29 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92047]: 2024-04-12T21:35:30.163 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:30 smithi081 podman[92233]: 2024-04-12 21:35:30.032425232 +0000 UTC m=+0.036654546 container died 3d440d7d2505a6cb62a2cb9535bcfc6ad72b006ab1825c62615aa5e670891fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-12T21:35:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:30 smithi153 ceph-mon[36569]: pgmap v1050: 129 pgs: 129 active+clean; 537 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:35:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:30 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:30 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:active 2024-04-12T21:35:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:30 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:35:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:30 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:30 smithi104 ceph-mon[36624]: pgmap v1050: 129 pgs: 129 active+clean; 537 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:35:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:30 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:30 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:active 2024-04-12T21:35:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:30 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:35:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:30 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:30 smithi081 ceph-mon[31453]: pgmap v1050: 129 pgs: 129 active+clean; 537 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2024-04-12T21:35:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:30 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:30 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3883338435,v1:172.21.15.81:6839/3883338435] up:active 2024-04-12T21:35:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:30 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 3 up:standby 2024-04-12T21:35:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:30 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:31.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:30 smithi081 podman[92233]: 2024-04-12 21:35:30.852778059 +0000 UTC m=+0.857007367 container remove 3d440d7d2505a6cb62a2cb9535bcfc6ad72b006ab1825c62615aa5e670891fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD) 2024-04-12T21:35:31.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:30 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:35:31.655 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:31 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-12T21:35:31.655 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:31 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.729s CPU time. 2024-04-12T21:35:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:32 smithi153 ceph-mon[36569]: pgmap v1051: 129 pgs: 129 active+clean; 536 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 307 B/s wr, 20 op/s 2024-04-12T21:35:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:32 smithi104 ceph-mon[36624]: pgmap v1051: 129 pgs: 129 active+clean; 536 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 307 B/s wr, 20 op/s 2024-04-12T21:35:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:32 smithi081 ceph-mon[31453]: pgmap v1051: 129 pgs: 129 active+clean; 536 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 307 B/s wr, 20 op/s 2024-04-12T21:35:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:34 smithi153 ceph-mon[36569]: pgmap v1052: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 74 KiB/s wr, 31 op/s 2024-04-12T21:35:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:34 smithi104 ceph-mon[36624]: pgmap v1052: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 74 KiB/s wr, 31 op/s 2024-04-12T21:35:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:34 smithi081 ceph-mon[31453]: pgmap v1052: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 74 KiB/s wr, 31 op/s 2024-04-12T21:35:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:36 smithi153 ceph-mon[36569]: pgmap v1053: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 73 KiB/s wr, 32 op/s 2024-04-12T21:35:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:36 smithi104 ceph-mon[36624]: pgmap v1053: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 73 KiB/s wr, 32 op/s 2024-04-12T21:35:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:36 smithi081 ceph-mon[31453]: pgmap v1053: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 73 KiB/s wr, 32 op/s 2024-04-12T21:35:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:38 smithi153 ceph-mon[36569]: pgmap v1054: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 KiB/s wr, 28 op/s 2024-04-12T21:35:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:38 smithi104 ceph-mon[36624]: pgmap v1054: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 KiB/s wr, 28 op/s 2024-04-12T21:35:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:38 smithi081 ceph-mon[31453]: pgmap v1054: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 KiB/s wr, 28 op/s 2024-04-12T21:35:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:38.699+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:35:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:38.699+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:35:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:38.699+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:35:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:40 smithi153 ceph-mon[36569]: pgmap v1055: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 KiB/s wr, 28 op/s 2024-04-12T21:35:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:40 smithi104 ceph-mon[36624]: pgmap v1055: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 KiB/s wr, 28 op/s 2024-04-12T21:35:40.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:40 smithi081 ceph-mon[31453]: pgmap v1055: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 61 KiB/s wr, 28 op/s 2024-04-12T21:35:41.763 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:41 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:35:41.763 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:41 smithi081 systemd[1]: Stopped Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:35:41.764 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:41 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.729s CPU time. 2024-04-12T21:35:41.764 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:41 smithi081 systemd[1]: Starting Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:35:42.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:41 smithi081 podman[92327]: 2024-04-12 21:35:41.673347489 +0000 UTC m=+0.038565101 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:35:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:42 smithi153 ceph-mon[36569]: pgmap v1056: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 61 KiB/s wr, 24 op/s 2024-04-12T21:35:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:42 smithi104 ceph-mon[36624]: pgmap v1056: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 61 KiB/s wr, 24 op/s 2024-04-12T21:35:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:42 smithi081 ceph-mon[31453]: pgmap v1056: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 61 KiB/s wr, 24 op/s 2024-04-12T21:35:43.905 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:43 smithi081 podman[92327]: 2024-04-12 21:35:43.629523074 +0000 UTC m=+1.994740714 container create d07e008c823ed243095f1aaddf6761fee6ab37f765d83132d5ee4bc4405ca12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-12T21:35:44.283 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:43 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:35:43.989+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:44.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:43 smithi081 podman[92327]: 2024-04-12 21:35:43.92519386 +0000 UTC m=+2.290411504 container init d07e008c823ed243095f1aaddf6761fee6ab37f765d83132d5ee4bc4405ca12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-12T21:35:44.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:43 smithi081 podman[92327]: 2024-04-12 21:35:43.934859091 +0000 UTC m=+2.300076704 container start d07e008c823ed243095f1aaddf6761fee6ab37f765d83132d5ee4bc4405ca12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-12T21:35:44.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:43 smithi081 ceph-mds[92347]: -- 172.21.15.81:0/437150910 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d610249a40 con 0x55d61021d000 2024-04-12T21:35:44.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:43 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: starting mds.g at 2024-04-12T21:35:44.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:44 smithi081 bash[92327]: d07e008c823ed243095f1aaddf6761fee6ab37f765d83132d5ee4bc4405ca12c 2024-04-12T21:35:44.284 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:35:44 smithi081 systemd[1]: Started Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: pgmap v1057: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 15 op/s 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: daemon mds.g restarted 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: osdmap e147: 12 total, 12 up, 12 in 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:boot 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:35:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:44 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:35:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: pgmap v1057: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 15 op/s 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: daemon mds.g restarted 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: osdmap e147: 12 total, 12 up, 12 in 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:boot 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:44 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: pgmap v1057: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 15 op/s 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: daemon mds.g restarted 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: osdmap e147: 12 total, 12 up, 12 in 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:boot 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: Standby daemon mds.a assigned to filesystem cephfs as rank 0 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 3 up:standby, 1 failed 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:44 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:replay} 3 up:standby 2024-04-12T21:35:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:46 smithi104 ceph-mon[36624]: pgmap v1059: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 5 op/s 2024-04-12T21:35:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:46 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:reconnect 2024-04-12T21:35:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:46 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:35:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:46 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:35:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:46 smithi081 ceph-mon[31453]: pgmap v1059: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 5 op/s 2024-04-12T21:35:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:46 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:reconnect 2024-04-12T21:35:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:46 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:35:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:46 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:35:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:46 smithi153 ceph-mon[36569]: pgmap v1059: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 5 op/s 2024-04-12T21:35:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:46 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:reconnect 2024-04-12T21:35:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:46 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:reconnect} 3 up:standby 2024-04-12T21:35:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:46 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:35:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:47 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:rejoin 2024-04-12T21:35:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:47 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:35:47.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:47 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:rejoin 2024-04-12T21:35:47.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:47 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:35:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:47 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:rejoin 2024-04-12T21:35:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:47 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:rejoin} 3 up:standby 2024-04-12T21:35:48.344 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-mon[31453]: pgmap v1060: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4 op/s 2024-04-12T21:35:48.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:clientreplay 2024-04-12T21:35:48.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:35:48.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-mon[31453]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:35:48.345 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: *** Caught signal (Aborted) ** 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: in thread 7efcd0400700 thread_name:ms_dispatch 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efce043ad20] 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2: gsignal() 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 3: abort() 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562e05792268] 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562e0582e0fc] 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562e05b62f0a] 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 7: (MDLog::trim()+0xca9) [0x562e05b640c9] 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562e05b645a8] 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7efcdf326b23] 2024-04-12T21:35:48.346 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7efce04301ca] 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 11: clone() 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2024-04-12T21:35:48.277+0000 7efcd0400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: in thread 7efcd0400700 thread_name:ms_dispatch 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efce043ad20] 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2: gsignal() 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 3: abort() 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562e05792268] 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562e0582e0fc] 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562e05b62f0a] 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 7: (MDLog::trim()+0xca9) [0x562e05b640c9] 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562e05b645a8] 2024-04-12T21:35:48.347 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7efcdf326b23] 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7efce04301ca] 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 11: clone() 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 0> 2024-04-12T21:35:48.277+0000 7efcd0400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: in thread 7efcd0400700 thread_name:ms_dispatch 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:48.348 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efce043ad20] 2024-04-12T21:35:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:48 smithi104 ceph-mon[36624]: pgmap v1060: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4 op/s 2024-04-12T21:35:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:48 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:clientreplay 2024-04-12T21:35:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:48 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:35:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:48 smithi104 ceph-mon[36624]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:35:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:48 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:48 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:active 2024-04-12T21:35:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:48 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:48.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:48.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:active 2024-04-12T21:35:48.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:48.639 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2: gsignal() 2024-04-12T21:35:48.639 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 3: abort() 2024-04-12T21:35:48.639 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562e05792268] 2024-04-12T21:35:48.639 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562e0582e0fc] 2024-04-12T21:35:48.639 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562e05b62f0a] 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 7: (MDLog::trim()+0xca9) [0x562e05b640c9] 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562e05b645a8] 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7efcdf326b23] 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7efce04301ca] 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 11: clone() 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: -1> 2024-04-12T21:35:48.277+0000 7efcd0400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: in thread 7efcd0400700 thread_name:ms_dispatch 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efce043ad20] 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2: gsignal() 2024-04-12T21:35:48.640 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 3: abort() 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x562e05792268] 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x562e0582e0fc] 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x562e05b62f0a] 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 7: (MDLog::trim()+0xca9) [0x562e05b640c9] 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 8: (MDLog::log_trim_upkeep()+0x358) [0x562e05b645a8] 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7efcdf326b23] 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7efce04301ca] 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 11: clone() 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a[92194]: 2024-04-12T21:35:48.641 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:48 smithi081 podman[92382]: 2024-04-12 21:35:48.446041352 +0000 UTC m=+0.045737819 container died 44c7ce40b7a3a5e2841c39924cfdc5bf7cbe96c0bf4bc87f3af63d6e934b2a44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-12T21:35:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:48 smithi153 ceph-mon[36569]: pgmap v1060: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4 op/s 2024-04-12T21:35:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:48 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:clientreplay 2024-04-12T21:35:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:48 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=a=up:clientreplay} 3 up:standby 2024-04-12T21:35:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:48 smithi153 ceph-mon[36569]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-04-12T21:35:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:48 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:35:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:48 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6840/2202140417,v1:172.21.15.81:6841/2202140417] up:active 2024-04-12T21:35:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:48 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=a=up:active} 1 up:standby-replay 2 up:standby 2024-04-12T21:35:49.405 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:49 smithi081 podman[92382]: 2024-04-12 21:35:49.104491751 +0000 UTC m=+0.704188262 container remove 44c7ce40b7a3a5e2841c39924cfdc5bf7cbe96c0bf4bc87f3af63d6e934b2a44 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-a, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-12T21:35:49.405 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:49 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:35:49.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:49 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-12T21:35:49.905 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:49 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 28.505s CPU time. 2024-04-12T21:35:50.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:50 smithi104 ceph-mon[36624]: pgmap v1061: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 0 B/s wr, 7 op/s 2024-04-12T21:35:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:35:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:50 smithi081 ceph-mon[31453]: pgmap v1061: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 0 B/s wr, 7 op/s 2024-04-12T21:35:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:35:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:50 smithi153 ceph-mon[36569]: pgmap v1061: 129 pgs: 129 active+clean; 535 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 0 B/s wr, 7 op/s 2024-04-12T21:35:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:35:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:52 smithi153 ceph-mon[36569]: pgmap v1062: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 307 B/s wr, 13 op/s 2024-04-12T21:35:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:35:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:35:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:52 smithi153 ceph-mon[36569]: pgmap v1063: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-12T21:35:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:52 smithi153 ceph-mon[36569]: pgmap v1064: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 26 KiB/s wr, 19 op/s 2024-04-12T21:35:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:52.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:52 smithi104 ceph-mon[36624]: pgmap v1062: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 307 B/s wr, 13 op/s 2024-04-12T21:35:52.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:35:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:35:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:52 smithi104 ceph-mon[36624]: pgmap v1063: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-12T21:35:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:52 smithi104 ceph-mon[36624]: pgmap v1064: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 26 KiB/s wr, 19 op/s 2024-04-12T21:35:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:52 smithi081 ceph-mon[31453]: pgmap v1062: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 307 B/s wr, 13 op/s 2024-04-12T21:35:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:35:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:35:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:52 smithi081 ceph-mon[31453]: pgmap v1063: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 23 KiB/s wr, 17 op/s 2024-04-12T21:35:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:52 smithi081 ceph-mon[31453]: pgmap v1064: 129 pgs: 129 active+clean; 534 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 26 KiB/s wr, 19 op/s 2024-04-12T21:35:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:35:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:53.713+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:35:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:53.713+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:35:53.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:35:53.713+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:35:54.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:54 smithi081 ceph-mon[31453]: pgmap v1065: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 76 KiB/s wr, 24 op/s 2024-04-12T21:35:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:54 smithi153 ceph-mon[36569]: pgmap v1065: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 76 KiB/s wr, 24 op/s 2024-04-12T21:35:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:54 smithi104 ceph-mon[36624]: pgmap v1065: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 76 KiB/s wr, 24 op/s 2024-04-12T21:35:56.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:56 smithi081 ceph-mon[31453]: pgmap v1066: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 76 KiB/s wr, 23 op/s 2024-04-12T21:35:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:56 smithi153 ceph-mon[36569]: pgmap v1066: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 76 KiB/s wr, 23 op/s 2024-04-12T21:35:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:56 smithi104 ceph-mon[36624]: pgmap v1066: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 76 KiB/s wr, 23 op/s 2024-04-12T21:35:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:35:58 smithi081 ceph-mon[31453]: pgmap v1067: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 76 KiB/s wr, 22 op/s 2024-04-12T21:35:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:35:58 smithi153 ceph-mon[36569]: pgmap v1067: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 76 KiB/s wr, 22 op/s 2024-04-12T21:35:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:35:58 smithi104 ceph-mon[36624]: pgmap v1067: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 MiB/s rd, 76 KiB/s wr, 22 op/s 2024-04-12T21:36:00.156 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:59 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:36:00.156 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:59 smithi081 systemd[1]: Stopped Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:36:00.156 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:59 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Consumed 28.505s CPU time. 2024-04-12T21:36:00.156 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:59 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-04-12T21:36:00.156 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:59 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-04-12T21:36:00.157 INFO:journalctl@ceph.mds.a.smithi081.stdout:Apr 12 21:35:59 smithi081 systemd[1]: Failed to start Ceph mds.a for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:36:00.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:00 smithi081 ceph-mon[31453]: pgmap v1068: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 75 KiB/s wr, 13 op/s 2024-04-12T21:36:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:00 smithi153 ceph-mon[36569]: pgmap v1068: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 75 KiB/s wr, 13 op/s 2024-04-12T21:36:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:00 smithi104 ceph-mon[36624]: pgmap v1068: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 75 KiB/s wr, 13 op/s 2024-04-12T21:36:02.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:02 smithi081 ceph-mon[31453]: pgmap v1069: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-12T21:36:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:02 smithi153 ceph-mon[36569]: pgmap v1069: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-12T21:36:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:02 smithi104 ceph-mon[36624]: pgmap v1069: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 43 KiB/s wr, 10 op/s 2024-04-12T21:36:03.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:03 smithi081 ceph-mon[31453]: Replacing daemon mds.a as rank 0 with standby daemon mds.g 2024-04-12T21:36:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:03 smithi081 ceph-mon[31453]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-12T21:36:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:03 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:03 smithi081 ceph-mon[31453]: osdmap e148: 12 total, 12 up, 12 in 2024-04-12T21:36:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:03 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:replay} 2 up:standby 2024-04-12T21:36:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:03 smithi153 ceph-mon[36569]: Replacing daemon mds.a as rank 0 with standby daemon mds.g 2024-04-12T21:36:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:03 smithi153 ceph-mon[36569]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-12T21:36:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:03 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:03 smithi153 ceph-mon[36569]: osdmap e148: 12 total, 12 up, 12 in 2024-04-12T21:36:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:03 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:replay} 2 up:standby 2024-04-12T21:36:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:03 smithi104 ceph-mon[36624]: Replacing daemon mds.a as rank 0 with standby daemon mds.g 2024-04-12T21:36:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:03 smithi104 ceph-mon[36624]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-04-12T21:36:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:03 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:03 smithi104 ceph-mon[36624]: osdmap e148: 12 total, 12 up, 12 in 2024-04-12T21:36:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:03 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:replay} 2 up:standby 2024-04-12T21:36:04.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:04 smithi081 ceph-mon[31453]: pgmap v1071: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:36:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:04 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:reconnect 2024-04-12T21:36:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:04 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:reconnect} 2 up:standby 2024-04-12T21:36:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:04 smithi153 ceph-mon[36569]: pgmap v1071: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:36:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:04 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:reconnect 2024-04-12T21:36:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:04 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:reconnect} 2 up:standby 2024-04-12T21:36:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:04 smithi104 ceph-mon[36624]: pgmap v1071: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:36:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:04 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:reconnect 2024-04-12T21:36:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:04 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:reconnect} 2 up:standby 2024-04-12T21:36:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:05 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:36:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:05 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:rejoin 2024-04-12T21:36:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:05 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:rejoin} 2 up:standby 2024-04-12T21:36:06.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:05 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:36:06.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:05 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:rejoin 2024-04-12T21:36:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:05 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:rejoin} 2 up:standby 2024-04-12T21:36:06.118 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:05 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:36:06.118 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:05 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:rejoin 2024-04-12T21:36:06.118 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:05 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:rejoin} 2 up:standby 2024-04-12T21:36:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:06 smithi153 ceph-mon[36569]: pgmap v1072: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:36:07.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:06 smithi104 ceph-mon[36624]: pgmap v1072: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:36:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:06 smithi081 ceph-mon[31453]: pgmap v1072: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:36:07.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:clientreplay 2024-04-12T21:36:07.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:clientreplay} 2 up:standby 2024-04-12T21:36:07.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-mon[31453]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:36:07.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:07.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:active 2024-04-12T21:36:07.725 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:clientreplay 2024-04-12T21:36:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:07 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:clientreplay} 2 up:standby 2024-04-12T21:36:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:07 smithi153 ceph-mon[36569]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:36:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:07 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:active 2024-04-12T21:36:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:07 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:clientreplay 2024-04-12T21:36:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:07 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:clientreplay} 2 up:standby 2024-04-12T21:36:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:07 smithi104 ceph-mon[36624]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:36:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:07 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/3478767746,v1:172.21.15.81:6839/3478767746] up:active 2024-04-12T21:36:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:07 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:08.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: *** Caught signal (Aborted) ** 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: in thread 7f4757a00700 thread_name:ms_dispatch 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f47679d7d20] 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2: gsignal() 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 3: abort() 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d5fd8d0268] 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d5fd96c0fc] 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d5fdca0f0a] 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 7: (MDLog::trim()+0xca9) [0x55d5fdca20c9] 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d5fdca25a8] 2024-04-12T21:36:08.146 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f47668c3b23] 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f47679cd1ca] 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 11: clone() 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2024-04-12T21:36:07.727+0000 7f4757a00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: in thread 7f4757a00700 thread_name:ms_dispatch 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f47679d7d20] 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2: gsignal() 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 3: abort() 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d5fd8d0268] 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d5fd96c0fc] 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d5fdca0f0a] 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 7: (MDLog::trim()+0xca9) [0x55d5fdca20c9] 2024-04-12T21:36:08.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d5fdca25a8] 2024-04-12T21:36:08.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f47668c3b23] 2024-04-12T21:36:08.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f47679cd1ca] 2024-04-12T21:36:08.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 11: clone() 2024-04-12T21:36:08.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:08.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2024-04-12T21:36:08.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 0> 2024-04-12T21:36:07.727+0000 7f4757a00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:08.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: in thread 7f4757a00700 thread_name:ms_dispatch 2024-04-12T21:36:08.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f47679d7d20] 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2: gsignal() 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 3: abort() 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d5fd8d0268] 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d5fd96c0fc] 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d5fdca0f0a] 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 7: (MDLog::trim()+0xca9) [0x55d5fdca20c9] 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d5fdca25a8] 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f47668c3b23] 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f47679cd1ca] 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 11: clone() 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:08.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: -1> 2024-04-12T21:36:07.727+0000 7f4757a00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: in thread 7f4757a00700 thread_name:ms_dispatch 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f47679d7d20] 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2: gsignal() 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 3: abort() 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d5fd8d0268] 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d5fd96c0fc] 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d5fdca0f0a] 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 7: (MDLog::trim()+0xca9) [0x55d5fdca20c9] 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d5fdca25a8] 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f47668c3b23] 2024-04-12T21:36:08.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f47679cd1ca] 2024-04-12T21:36:08.151 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 11: clone() 2024-04-12T21:36:08.151 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:08.151 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92343]: 2024-04-12T21:36:08.151 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:07 smithi081 podman[92665]: 2024-04-12 21:36:07.862411666 +0000 UTC m=+0.033035219 container died d07e008c823ed243095f1aaddf6761fee6ab37f765d83132d5ee4bc4405ca12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-12T21:36:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:08.728+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:36:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:08.728+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:36:08.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:08.728+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:36:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:08 smithi153 ceph-mon[36569]: pgmap v1073: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:36:09.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:08 smithi104 ceph-mon[36624]: pgmap v1073: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:36:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:08 smithi081 ceph-mon[31453]: pgmap v1073: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 102 B/s wr, 3 op/s 2024-04-12T21:36:09.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:08 smithi081 podman[92665]: 2024-04-12 21:36:08.761965643 +0000 UTC m=+0.932589237 container remove d07e008c823ed243095f1aaddf6761fee6ab37f765d83132d5ee4bc4405ca12c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-12T21:36:09.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:08 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:36:09.655 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:09 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-12T21:36:09.655 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:09 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.004s CPU time. 2024-04-12T21:36:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:10 smithi153 ceph-mon[36569]: pgmap v1074: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 307 B/s wr, 9 op/s 2024-04-12T21:36:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:10 smithi104 ceph-mon[36624]: pgmap v1074: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 307 B/s wr, 9 op/s 2024-04-12T21:36:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:10 smithi081 ceph-mon[31453]: pgmap v1074: 129 pgs: 129 active+clean; 534 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 307 B/s wr, 9 op/s 2024-04-12T21:36:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:12 smithi153 ceph-mon[36569]: pgmap v1075: 129 pgs: 129 active+clean; 532 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 16 KiB/s wr, 25 op/s 2024-04-12T21:36:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:12 smithi104 ceph-mon[36624]: pgmap v1075: 129 pgs: 129 active+clean; 532 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 16 KiB/s wr, 25 op/s 2024-04-12T21:36:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:12 smithi081 ceph-mon[31453]: pgmap v1075: 129 pgs: 129 active+clean; 532 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 16 KiB/s wr, 25 op/s 2024-04-12T21:36:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:14 smithi153 ceph-mon[36569]: pgmap v1076: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 KiB/s wr, 35 op/s 2024-04-12T21:36:15.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:14 smithi104 ceph-mon[36624]: pgmap v1076: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 KiB/s wr, 35 op/s 2024-04-12T21:36:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:14 smithi081 ceph-mon[31453]: pgmap v1076: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 15 KiB/s wr, 35 op/s 2024-04-12T21:36:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:16 smithi153 ceph-mon[36569]: pgmap v1077: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 KiB/s wr, 32 op/s 2024-04-12T21:36:17.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:16 smithi104 ceph-mon[36624]: pgmap v1077: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 KiB/s wr, 32 op/s 2024-04-12T21:36:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:16 smithi081 ceph-mon[31453]: pgmap v1077: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 KiB/s wr, 32 op/s 2024-04-12T21:36:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:18 smithi153 ceph-mon[36569]: pgmap v1078: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 14 KiB/s wr, 33 op/s 2024-04-12T21:36:19.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:18 smithi104 ceph-mon[36624]: pgmap v1078: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 14 KiB/s wr, 33 op/s 2024-04-12T21:36:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:18 smithi081 ceph-mon[31453]: pgmap v1078: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 14 KiB/s wr, 33 op/s 2024-04-12T21:36:19.785 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:19 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:36:19.786 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:19 smithi081 systemd[1]: Stopped Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:36:19.786 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:19 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.004s CPU time. 2024-04-12T21:36:19.786 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:19 smithi081 systemd[1]: Starting Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:36:20.155 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:19 smithi081 podman[92757]: 2024-04-12 21:36:19.69419017 +0000 UTC m=+0.036487334 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:36:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:20 smithi153 ceph-mon[36569]: pgmap v1079: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 14 KiB/s wr, 32 op/s 2024-04-12T21:36:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:20 smithi104 ceph-mon[36624]: pgmap v1079: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 14 KiB/s wr, 32 op/s 2024-04-12T21:36:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:20 smithi081 ceph-mon[31453]: pgmap v1079: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 14 KiB/s wr, 32 op/s 2024-04-12T21:36:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:22 smithi153 ceph-mon[36569]: pgmap v1080: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 13 KiB/s wr, 29 op/s 2024-04-12T21:36:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:22 smithi153 ceph-mon[36569]: Replacing daemon mds.g as rank 0 with standby daemon mds.d 2024-04-12T21:36:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:22 smithi153 ceph-mon[36569]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-12T21:36:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:22 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:22 smithi153 ceph-mon[36569]: osdmap e149: 12 total, 12 up, 12 in 2024-04-12T21:36:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:22 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:36:23.077 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:22 smithi081 podman[92757]: 2024-04-12 21:36:22.913573444 +0000 UTC m=+3.255870648 container create 26ebb918b0fee217e0a6759ce764e2ce9eb6db7253aa6d3f86a40beb919cd4c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-12T21:36:23.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:22 smithi081 ceph-mon[31453]: pgmap v1080: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 13 KiB/s wr, 29 op/s 2024-04-12T21:36:23.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:22 smithi081 ceph-mon[31453]: Replacing daemon mds.g as rank 0 with standby daemon mds.d 2024-04-12T21:36:23.078 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:22 smithi081 ceph-mon[31453]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-12T21:36:23.078 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:22 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:23.078 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:22 smithi081 ceph-mon[31453]: osdmap e149: 12 total, 12 up, 12 in 2024-04-12T21:36:23.078 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:22 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:36:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:22 smithi104 ceph-mon[36624]: pgmap v1080: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 13 KiB/s wr, 29 op/s 2024-04-12T21:36:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:22 smithi104 ceph-mon[36624]: Replacing daemon mds.g as rank 0 with standby daemon mds.d 2024-04-12T21:36:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:22 smithi104 ceph-mon[36624]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-12T21:36:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:22 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:22 smithi104 ceph-mon[36624]: osdmap e149: 12 total, 12 up, 12 in 2024-04-12T21:36:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:22 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:36:23.656 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:23 smithi081 podman[92757]: 2024-04-12 21:36:23.193079101 +0000 UTC m=+3.535376285 container init 26ebb918b0fee217e0a6759ce764e2ce9eb6db7253aa6d3f86a40beb919cd4c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:36:23.656 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:23 smithi081 podman[92757]: 2024-04-12 21:36:23.204051003 +0000 UTC m=+3.546348213 container start 26ebb918b0fee217e0a6759ce764e2ce9eb6db7253aa6d3f86a40beb919cd4c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:36:23.657 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:23 smithi081 ceph-mds[92777]: -- 172.21.15.81:0/3679856305 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55607e92da40 con 0x55607e900800 2024-04-12T21:36:23.657 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: starting mds.g at 2024-04-12T21:36:23.657 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:23 smithi081 bash[92757]: 26ebb918b0fee217e0a6759ce764e2ce9eb6db7253aa6d3f86a40beb919cd4c3 2024-04-12T21:36:23.657 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:23 smithi081 systemd[1]: Started Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:36:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:23.743+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:36:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:23.744+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:36:23.744 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:23.744+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:36:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:24 smithi153 ceph-mon[36569]: pgmap v1082: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:36:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:24 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:boot 2024-04-12T21:36:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:24 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:reconnect 2024-04-12T21:36:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:24 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:reconnect} 2 up:standby 2024-04-12T21:36:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:36:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:24 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:36:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:24 smithi104 ceph-mon[36624]: pgmap v1082: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:36:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:24 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:boot 2024-04-12T21:36:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:24 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:reconnect 2024-04-12T21:36:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:24 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:reconnect} 2 up:standby 2024-04-12T21:36:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:36:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:24 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:36:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:24 smithi081 ceph-mon[31453]: pgmap v1082: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:36:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:24 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:boot 2024-04-12T21:36:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:24 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:reconnect 2024-04-12T21:36:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:24 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:reconnect} 2 up:standby 2024-04-12T21:36:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-12T21:36:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:24 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:rejoin 2024-04-12T21:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:25 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:rejoin} 2 up:standby 2024-04-12T21:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:25 smithi153 ceph-mon[36569]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:36:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:rejoin 2024-04-12T21:36:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:25 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:rejoin} 2 up:standby 2024-04-12T21:36:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:25 smithi104 ceph-mon[36624]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:36:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:rejoin 2024-04-12T21:36:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:25 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:rejoin} 2 up:standby 2024-04-12T21:36:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:25 smithi081 ceph-mon[31453]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:36:26.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2024-04-12T21:36:26.410+0000 7f94c4400700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:36:26.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:36:26.697 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: *** Caught signal (Aborted) ** 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: in thread 7f94c1c00700 thread_name:ms_dispatch 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f94d1d4ad20] 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2: gsignal() 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 3: abort() 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a5fa84a268] 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a5fa8e60fc] 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a5fac1af0a] 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 7: (MDLog::trim()+0xca9) [0x55a5fac1c0c9] 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a5fac1c5a8] 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f94d0c36b23] 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f94d1d401ca] 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 11: clone() 2024-04-12T21:36:26.698 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2024-04-12T21:36:26.650+0000 7f94c1c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: in thread 7f94c1c00700 thread_name:ms_dispatch 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f94d1d4ad20] 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2: gsignal() 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 3: abort() 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a5fa84a268] 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a5fa8e60fc] 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a5fac1af0a] 2024-04-12T21:36:26.699 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 7: (MDLog::trim()+0xca9) [0x55a5fac1c0c9] 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a5fac1c5a8] 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f94d0c36b23] 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f94d1d401ca] 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 11: clone() 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 0> 2024-04-12T21:36:26.650+0000 7f94c1c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: in thread 7f94c1c00700 thread_name:ms_dispatch 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f94d1d4ad20] 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2: gsignal() 2024-04-12T21:36:26.700 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 3: abort() 2024-04-12T21:36:26.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a5fa84a268] 2024-04-12T21:36:26.701 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a5fa8e60fc] 2024-04-12T21:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:26 smithi153 ceph-mon[36569]: pgmap v1083: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:26 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:26 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:active 2024-04-12T21:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:26 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 2 up:standby 2024-04-12T21:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:26 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:26 smithi104 ceph-mon[36624]: pgmap v1083: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:26 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:26 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:active 2024-04-12T21:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:26 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 2 up:standby 2024-04-12T21:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:26 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:27.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-mon[31453]: pgmap v1083: 129 pgs: 129 active+clean; 531 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:36:27.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:27.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/809891880,v1:172.21.15.81:6837/809891880] up:active 2024-04-12T21:36:27.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 2 up:standby 2024-04-12T21:36:27.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a5fac1af0a] 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 7: (MDLog::trim()+0xca9) [0x55a5fac1c0c9] 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a5fac1c5a8] 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f94d0c36b23] 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f94d1d401ca] 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 11: clone() 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: -12> 2024-04-12T21:36:26.650+0000 7f94c1c00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: in thread 7f94c1c00700 thread_name:ms_dispatch 2024-04-12T21:36:27.112 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f94d1d4ad20] 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2: gsignal() 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 3: abort() 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55a5fa84a268] 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55a5fa8e60fc] 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55a5fac1af0a] 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 7: (MDLog::trim()+0xca9) [0x55a5fac1c0c9] 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55a5fac1c5a8] 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f94d0c36b23] 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f94d1d401ca] 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 11: clone() 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[72115]: 2024-04-12T21:36:27.113 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:26 smithi081 podman[92812]: 2024-04-12 21:36:26.81039538 +0000 UTC m=+0.030970888 container died fa4bc41faa469be6dae2c2f6df19c18f720b3dbe9e6f15e3a73b4f0bfb1e7b26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-12T21:36:27.905 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:27 smithi081 podman[92812]: 2024-04-12 21:36:27.489359071 +0000 UTC m=+0.709934583 container remove fa4bc41faa469be6dae2c2f6df19c18f720b3dbe9e6f15e3a73b4f0bfb1e7b26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2) 2024-04-12T21:36:27.905 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:27 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:36:28.405 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:27 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-12T21:36:28.405 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:27 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 28.371s CPU time. 2024-04-12T21:36:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:28 smithi153 ceph-mon[36569]: pgmap v1084: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-12T21:36:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:28 smithi104 ceph-mon[36624]: pgmap v1084: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-12T21:36:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:28 smithi081 ceph-mon[31453]: pgmap v1084: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-12T21:36:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:30 smithi153 ceph-mon[36569]: pgmap v1085: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-04-12T21:36:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:30 smithi104 ceph-mon[36624]: pgmap v1085: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-04-12T21:36:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:30 smithi081 ceph-mon[31453]: pgmap v1085: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-04-12T21:36:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:32 smithi153 ceph-mon[36569]: pgmap v1086: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 74 KiB/s wr, 25 op/s 2024-04-12T21:36:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:32 smithi104 ceph-mon[36624]: pgmap v1086: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 74 KiB/s wr, 25 op/s 2024-04-12T21:36:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:32 smithi081 ceph-mon[31453]: pgmap v1086: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 74 KiB/s wr, 25 op/s 2024-04-12T21:36:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:34 smithi153 ceph-mon[36569]: pgmap v1087: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 24 op/s 2024-04-12T21:36:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:34 smithi104 ceph-mon[36624]: pgmap v1087: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 24 op/s 2024-04-12T21:36:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:34 smithi081 ceph-mon[31453]: pgmap v1087: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 70 KiB/s wr, 24 op/s 2024-04-12T21:36:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:36 smithi153 ceph-mon[36569]: pgmap v1088: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 62 KiB/s wr, 22 op/s 2024-04-12T21:36:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:36 smithi104 ceph-mon[36624]: pgmap v1088: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 62 KiB/s wr, 22 op/s 2024-04-12T21:36:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:36 smithi081 ceph-mon[31453]: pgmap v1088: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 62 KiB/s wr, 22 op/s 2024-04-12T21:36:38.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:38 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:36:38.541 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:38 smithi081 systemd[1]: Stopped Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:36:38.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:38 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 28.371s CPU time. 2024-04-12T21:36:38.542 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:38 smithi081 systemd[1]: Starting Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:36:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:38.756+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:36:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:38.756+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:36:38.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:38.756+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:36:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:38 smithi081 ceph-mon[31453]: pgmap v1089: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 62 KiB/s wr, 24 op/s 2024-04-12T21:36:38.906 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:38 smithi081 podman[92902]: 2024-04-12 21:36:38.446600845 +0000 UTC m=+0.036273310 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:36:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:38 smithi153 ceph-mon[36569]: pgmap v1089: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 62 KiB/s wr, 24 op/s 2024-04-12T21:36:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:38 smithi104 ceph-mon[36624]: pgmap v1089: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 62 KiB/s wr, 24 op/s 2024-04-12T21:36:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:40 smithi153 ceph-mon[36569]: pgmap v1090: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 61 KiB/s wr, 17 op/s 2024-04-12T21:36:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:40 smithi104 ceph-mon[36624]: pgmap v1090: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 61 KiB/s wr, 17 op/s 2024-04-12T21:36:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:40 smithi081 ceph-mon[31453]: pgmap v1090: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 61 KiB/s wr, 17 op/s 2024-04-12T21:36:42.027 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:41 smithi081 podman[92902]: 2024-04-12 21:36:41.748925427 +0000 UTC m=+3.338597921 container create 5fa293c44008e97acbdbff09e4605dce4e9f7845f18d501e2a58567409ae7c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-12T21:36:42.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:36:42.092+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:36:42.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:42 smithi081 podman[92902]: 2024-04-12 21:36:42.027611009 +0000 UTC m=+3.617283510 container init 5fa293c44008e97acbdbff09e4605dce4e9f7845f18d501e2a58567409ae7c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409) 2024-04-12T21:36:42.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:42 smithi081 podman[92902]: 2024-04-12 21:36:42.03808912 +0000 UTC m=+3.627761594 container start 5fa293c44008e97acbdbff09e4605dce4e9f7845f18d501e2a58567409ae7c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-04-12T21:36:42.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mds[92920]: -- 172.21.15.81:0/3428604786 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8a15aba40 con 0x55f8a157ec00 2024-04-12T21:36:42.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: starting mds.d at 2024-04-12T21:36:42.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:42 smithi081 bash[92902]: 5fa293c44008e97acbdbff09e4605dce4e9f7845f18d501e2a58567409ae7c5c 2024-04-12T21:36:42.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:36:42 smithi081 systemd[1]: Started Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:36:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: pgmap v1091: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 13 op/s 2024-04-12T21:36:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: daemon mds.d restarted 2024-04-12T21:36:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:36:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: osdmap e150: 12 total, 12 up, 12 in 2024-04-12T21:36:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:boot 2024-04-12T21:36:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:36:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:36:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 2 up:standby, 1 failed 2024-04-12T21:36:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:36:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:42 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:replay} 2 up:standby 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: pgmap v1091: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 13 op/s 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: daemon mds.d restarted 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: osdmap e150: 12 total, 12 up, 12 in 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:boot 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 2 up:standby, 1 failed 2024-04-12T21:36:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:36:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:42 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:replay} 2 up:standby 2024-04-12T21:36:43.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: pgmap v1091: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 13 op/s 2024-04-12T21:36:43.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: daemon mds.d restarted 2024-04-12T21:36:43.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:36:43.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:36:43.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: osdmap e150: 12 total, 12 up, 12 in 2024-04-12T21:36:43.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:boot 2024-04-12T21:36:43.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: Standby daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-12T21:36:43.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:36:43.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 2 up:standby, 1 failed 2024-04-12T21:36:43.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:36:43.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:42 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:replay} 2 up:standby 2024-04-12T21:36:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:44 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:reconnect 2024-04-12T21:36:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:44 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:reconnect} 2 up:standby 2024-04-12T21:36:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:44 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:36:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:44 smithi081 ceph-mon[31453]: pgmap v1093: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:36:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:44 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:reconnect 2024-04-12T21:36:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:44 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:reconnect} 2 up:standby 2024-04-12T21:36:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:44 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:36:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:44 smithi153 ceph-mon[36569]: pgmap v1093: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:36:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:44 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:reconnect 2024-04-12T21:36:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:44 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:reconnect} 2 up:standby 2024-04-12T21:36:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:44 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:36:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:44 smithi104 ceph-mon[36624]: pgmap v1093: 129 pgs: 129 active+clean; 530 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:36:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:45 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:rejoin 2024-04-12T21:36:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:45 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:rejoin} 2 up:standby 2024-04-12T21:36:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:45 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:rejoin 2024-04-12T21:36:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:45 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:rejoin} 2 up:standby 2024-04-12T21:36:45.610 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:45 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:rejoin 2024-04-12T21:36:45.610 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:45 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:rejoin} 2 up:standby 2024-04-12T21:36:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:clientreplay 2024-04-12T21:36:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=g=up:clientreplay} 2 up:standby 2024-04-12T21:36:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-mon[31453]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:36:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-mon[31453]: pgmap v1094: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 5 op/s 2024-04-12T21:36:46.457 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:46 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:clientreplay 2024-04-12T21:36:46.457 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:46 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=g=up:clientreplay} 2 up:standby 2024-04-12T21:36:46.457 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:46 smithi153 ceph-mon[36569]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:36:46.458 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:46 smithi153 ceph-mon[36569]: pgmap v1094: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 5 op/s 2024-04-12T21:36:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:46 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:clientreplay 2024-04-12T21:36:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:46 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=g=up:clientreplay} 2 up:standby 2024-04-12T21:36:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:46 smithi104 ceph-mon[36624]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-12T21:36:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:46 smithi104 ceph-mon[36624]: pgmap v1094: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 0 B/s wr, 5 op/s 2024-04-12T21:36:47.143 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:36:47.143 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: *** Caught signal (Aborted) ** 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: in thread 7ff6a5200700 thread_name:ms_dispatch 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6b5298d20] 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2: gsignal() 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 3: abort() 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x556062e66268] 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x556062f020fc] 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x556063236f0a] 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 7: (MDLog::trim()+0xca9) [0x5560632380c9] 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5560632385a8] 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff6b4184b23] 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b528e1ca] 2024-04-12T21:36:47.144 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 11: clone() 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2024-04-12T21:36:46.820+0000 7ff6a5200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: in thread 7ff6a5200700 thread_name:ms_dispatch 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6b5298d20] 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2: gsignal() 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 3: abort() 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x556062e66268] 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x556062f020fc] 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x556063236f0a] 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 7: (MDLog::trim()+0xca9) [0x5560632380c9] 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5560632385a8] 2024-04-12T21:36:47.145 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff6b4184b23] 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b528e1ca] 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 11: clone() 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 0> 2024-04-12T21:36:46.820+0000 7ff6a5200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: in thread 7ff6a5200700 thread_name:ms_dispatch 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6b5298d20] 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2: gsignal() 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 3: abort() 2024-04-12T21:36:47.147 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x556062e66268] 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x556062f020fc] 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x556063236f0a] 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 7: (MDLog::trim()+0xca9) [0x5560632380c9] 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5560632385a8] 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff6b4184b23] 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b528e1ca] 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 11: clone() 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: -21> 2024-04-12T21:36:46.820+0000 7ff6a5200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: in thread 7ff6a5200700 thread_name:ms_dispatch 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:36:47.148 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff6b5298d20] 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2: gsignal() 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 3: abort() 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x556062e66268] 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x556062f020fc] 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x556063236f0a] 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 7: (MDLog::trim()+0xca9) [0x5560632380c9] 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5560632385a8] 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff6b4184b23] 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff6b528e1ca] 2024-04-12T21:36:47.149 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 11: clone() 2024-04-12T21:36:47.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:36:47.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g[92773]: 2024-04-12T21:36:47.150 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:46 smithi081 podman[92955]: 2024-04-12 21:36:46.963464655 +0000 UTC m=+0.035867397 container died 26ebb918b0fee217e0a6759ce764e2ce9eb6db7253aa6d3f86a40beb919cd4c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T21:36:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:47 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:47 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:active 2024-04-12T21:36:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:47 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 2 up:standby 2024-04-12T21:36:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:47 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:47 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:47 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:active 2024-04-12T21:36:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:47 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 2 up:standby 2024-04-12T21:36:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:47 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:47 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:36:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:47 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6838/842833546,v1:172.21.15.81:6839/842833546] up:active 2024-04-12T21:36:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:47 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 2 up:standby 2024-04-12T21:36:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:47 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=g=up:active} 1 up:standby-replay 1 up:standby 2024-04-12T21:36:48.142 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:47 smithi081 podman[92955]: 2024-04-12 21:36:47.849313308 +0000 UTC m=+0.921716040 container remove 26ebb918b0fee217e0a6759ce764e2ce9eb6db7253aa6d3f86a40beb919cd4c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-g, ceph=True, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-12T21:36:48.142 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:47 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:36:48.405 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:48 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-12T21:36:48.405 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:48 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.866s CPU time. 2024-04-12T21:36:48.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:48 smithi081 ceph-mon[31453]: pgmap v1095: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-04-12T21:36:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:48 smithi153 ceph-mon[36569]: pgmap v1095: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-04-12T21:36:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:48 smithi104 ceph-mon[36624]: pgmap v1095: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-04-12T21:36:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:50 smithi081 ceph-mon[31453]: pgmap v1096: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 KiB/s wr, 18 op/s 2024-04-12T21:36:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:50 smithi153 ceph-mon[36569]: pgmap v1096: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 KiB/s wr, 18 op/s 2024-04-12T21:36:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:50 smithi104 ceph-mon[36624]: pgmap v1096: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.6 KiB/s wr, 18 op/s 2024-04-12T21:36:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:52 smithi081 ceph-mon[31453]: pgmap v1097: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 KiB/s wr, 23 op/s 2024-04-12T21:36:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:36:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:52 smithi153 ceph-mon[36569]: pgmap v1097: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 KiB/s wr, 23 op/s 2024-04-12T21:36:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:36:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.864 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:52 smithi104 ceph-mon[36624]: pgmap v1097: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 60 KiB/s wr, 23 op/s 2024-04-12T21:36:52.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:36:52.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:52.865 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:53.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:53.771+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:36:53.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:53.771+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:36:53.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:36:53.771+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:36:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:54 smithi153 ceph-mon[36569]: pgmap v1098: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 53 KiB/s wr, 21 op/s 2024-04-12T21:36:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:54 smithi104 ceph-mon[36624]: pgmap v1098: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 53 KiB/s wr, 21 op/s 2024-04-12T21:36:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:54 smithi081 ceph-mon[31453]: pgmap v1098: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 53 KiB/s wr, 21 op/s 2024-04-12T21:36:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:56 smithi153 ceph-mon[36569]: pgmap v1099: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 50 KiB/s wr, 20 op/s 2024-04-12T21:36:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:56 smithi104 ceph-mon[36624]: pgmap v1099: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 50 KiB/s wr, 20 op/s 2024-04-12T21:36:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:56 smithi081 ceph-mon[31453]: pgmap v1099: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 50 KiB/s wr, 20 op/s 2024-04-12T21:36:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:36:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:36:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:36:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:36:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:36:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:36:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:36:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:58 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:36:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:58 smithi081 systemd[1]: Stopped Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:36:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:58 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Consumed 27.866s CPU time. 2024-04-12T21:36:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:58 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-04-12T21:36:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:58 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-12T21:36:58.804 INFO:journalctl@ceph.mds.g.smithi081.stdout:Apr 12 21:36:58 smithi081 systemd[1]: Failed to start Ceph mds.g for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:36:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:58 smithi153 ceph-mon[36569]: pgmap v1100: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-12T21:36:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:58 smithi153 ceph-mon[36569]: pgmap v1101: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-12T21:36:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:36:58 smithi153 ceph-mon[36569]: Health check update: 10 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:36:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:58 smithi104 ceph-mon[36624]: pgmap v1100: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-12T21:36:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:58 smithi104 ceph-mon[36624]: pgmap v1101: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-12T21:36:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:36:58 smithi104 ceph-mon[36624]: Health check update: 10 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:36:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:58 smithi081 ceph-mon[31453]: pgmap v1100: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 52 KiB/s wr, 21 op/s 2024-04-12T21:36:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:58 smithi081 ceph-mon[31453]: pgmap v1101: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 61 KiB/s wr, 14 op/s 2024-04-12T21:36:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:36:58 smithi081 ceph-mon[31453]: Health check update: 10 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:37:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:00 smithi104 ceph-mon[36624]: pgmap v1102: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 61 KiB/s wr, 10 op/s 2024-04-12T21:37:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:00 smithi081 ceph-mon[31453]: pgmap v1102: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 61 KiB/s wr, 10 op/s 2024-04-12T21:37:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:00 smithi153 ceph-mon[36569]: pgmap v1102: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 61 KiB/s wr, 10 op/s 2024-04-12T21:37:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:01 smithi081 ceph-mon[31453]: pgmap v1103: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 213 B/s wr, 4 op/s 2024-04-12T21:37:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:01 smithi153 ceph-mon[36569]: pgmap v1103: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 213 B/s wr, 4 op/s 2024-04-12T21:37:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:01 smithi104 ceph-mon[36624]: pgmap v1103: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 213 B/s wr, 4 op/s 2024-04-12T21:37:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:02 smithi081 ceph-mon[31453]: Replacing daemon mds.g as rank 0 with standby daemon mds.d 2024-04-12T21:37:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:02 smithi081 ceph-mon[31453]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:02 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:02 smithi081 ceph-mon[31453]: osdmap e151: 12 total, 12 up, 12 in 2024-04-12T21:37:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:02 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:37:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:02 smithi153 ceph-mon[36569]: Replacing daemon mds.g as rank 0 with standby daemon mds.d 2024-04-12T21:37:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:02 smithi153 ceph-mon[36569]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:02 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:02 smithi153 ceph-mon[36569]: osdmap e151: 12 total, 12 up, 12 in 2024-04-12T21:37:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:02 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:37:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:02 smithi104 ceph-mon[36624]: Replacing daemon mds.g as rank 0 with standby daemon mds.d 2024-04-12T21:37:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:02 smithi104 ceph-mon[36624]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:02 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:02 smithi104 ceph-mon[36624]: osdmap e151: 12 total, 12 up, 12 in 2024-04-12T21:37:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:02 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:37:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:03 smithi081 ceph-mon[31453]: pgmap v1105: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:37:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:03 smithi153 ceph-mon[36569]: pgmap v1105: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:37:04.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:03 smithi104 ceph-mon[36624]: pgmap v1105: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:37:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:04 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:reconnect 2024-04-12T21:37:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:04 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:37:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:04 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:37:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:04 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:reconnect 2024-04-12T21:37:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:04 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:37:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:04 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:37:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:04 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:reconnect 2024-04-12T21:37:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:04 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:37:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:04 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:37:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:05 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:rejoin 2024-04-12T21:37:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:05 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:37:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:05 smithi081 ceph-mon[31453]: pgmap v1106: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1 op/s 2024-04-12T21:37:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:05 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:rejoin 2024-04-12T21:37:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:05 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:37:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:05 smithi153 ceph-mon[36569]: pgmap v1106: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1 op/s 2024-04-12T21:37:06.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:05 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:rejoin 2024-04-12T21:37:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:05 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:37:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:05 smithi104 ceph-mon[36624]: pgmap v1106: 129 pgs: 129 active+clean; 529 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 1 op/s 2024-04-12T21:37:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:06 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:clientreplay 2024-04-12T21:37:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:06 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:37:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:06 smithi081 ceph-mon[31453]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:37:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:06 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:clientreplay 2024-04-12T21:37:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:06 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:37:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:06 smithi153 ceph-mon[36569]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:37:07.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:06 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:clientreplay 2024-04-12T21:37:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:06 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:37:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:06 smithi104 ceph-mon[36624]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2024-04-12T21:37:07.360+0000 7ff40bc00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: *** Caught signal (Aborted) ** 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: in thread 7ff409400700 thread_name:ms_dispatch 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4194bad20] 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2: gsignal() 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 3: abort() 2024-04-12T21:37:07.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f870f9b268] 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f8710370fc] 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f87136bf0a] 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 7: (MDLog::trim()+0xca9) [0x55f87136d0c9] 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f87136d5a8] 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff4183a6b23] 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff4194b01ca] 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 11: clone() 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2024-04-12T21:37:07.395+0000 7ff409400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: in thread 7ff409400700 thread_name:ms_dispatch 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4194bad20] 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2: gsignal() 2024-04-12T21:37:07.659 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 3: abort() 2024-04-12T21:37:07.660 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f870f9b268] 2024-04-12T21:37:07.660 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f8710370fc] 2024-04-12T21:37:07.660 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f87136bf0a] 2024-04-12T21:37:07.660 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 7: (MDLog::trim()+0xca9) [0x55f87136d0c9] 2024-04-12T21:37:07.660 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f87136d5a8] 2024-04-12T21:37:07.660 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff4183a6b23] 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff4194b01ca] 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 11: clone() 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: -2619> 2024-04-12T21:37:07.360+0000 7ff40bc00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 0> 2024-04-12T21:37:07.395+0000 7ff409400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: in thread 7ff409400700 thread_name:ms_dispatch 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2024-04-12T21:37:07.661 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4194bad20] 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2: gsignal() 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 3: abort() 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f870f9b268] 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f8710370fc] 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f87136bf0a] 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 7: (MDLog::trim()+0xca9) [0x55f87136d0c9] 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f87136d5a8] 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff4183a6b23] 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff4194b01ca] 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 11: clone() 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:07.662 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: -2628> 2024-04-12T21:37:07.360+0000 7ff40bc00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: -9> 2024-04-12T21:37:07.395+0000 7ff409400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: in thread 7ff409400700 thread_name:ms_dispatch 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff4194bad20] 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2: gsignal() 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 3: abort() 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f870f9b268] 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f8710370fc] 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f87136bf0a] 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 7: (MDLog::trim()+0xca9) [0x55f87136d0c9] 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f87136d5a8] 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff4183a6b23] 2024-04-12T21:37:07.663 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff4194b01ca] 2024-04-12T21:37:07.664 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 11: clone() 2024-04-12T21:37:07.664 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:07.664 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[92916]: 2024-04-12T21:37:07.664 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:07 smithi081 podman[93581]: 2024-04-12 21:37:07.522311286 +0000 UTC m=+0.033467956 container died 5fa293c44008e97acbdbff09e4605dce4e9f7845f18d501e2a58567409ae7c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-12T21:37:08.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:08.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:active 2024-04-12T21:37:08.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:37:08.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:37:08.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:07 smithi081 ceph-mon[31453]: pgmap v1107: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-12T21:37:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:07 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:07 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:active 2024-04-12T21:37:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:07 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:37:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:07 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:37:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:07 smithi153 ceph-mon[36569]: pgmap v1107: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-12T21:37:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:07 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:07 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2109992149,v1:172.21.15.81:6837/2109992149] up:active 2024-04-12T21:37:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:07 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:37:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:07 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:37:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:07 smithi104 ceph-mon[36624]: pgmap v1107: 129 pgs: 129 active+clean; 528 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-12T21:37:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:08.786+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:37:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:08.786+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:37:08.786 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:08.786+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:37:09.405 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:09 smithi081 podman[93581]: 2024-04-12 21:37:09.037024854 +0000 UTC m=+1.548181559 container remove 5fa293c44008e97acbdbff09e4605dce4e9f7845f18d501e2a58567409ae7c5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, ceph=True) 2024-04-12T21:37:09.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:09 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:37:09.896 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:09 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-12T21:37:09.896 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:09 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 27.875s CPU time. 2024-04-12T21:37:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:09 smithi153 ceph-mon[36569]: pgmap v1108: 129 pgs: 129 active+clean; 527 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.4 KiB/s wr, 16 op/s 2024-04-12T21:37:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:09 smithi104 ceph-mon[36624]: pgmap v1108: 129 pgs: 129 active+clean; 527 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.4 KiB/s wr, 16 op/s 2024-04-12T21:37:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:09 smithi081 ceph-mon[31453]: pgmap v1108: 129 pgs: 129 active+clean; 527 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.4 KiB/s wr, 16 op/s 2024-04-12T21:37:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:12 smithi081 ceph-mon[31453]: pgmap v1109: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 2.6 KiB/s wr, 32 op/s 2024-04-12T21:37:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:12 smithi153 ceph-mon[36569]: pgmap v1109: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 2.6 KiB/s wr, 32 op/s 2024-04-12T21:37:12.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:12 smithi104 ceph-mon[36624]: pgmap v1109: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 2.6 KiB/s wr, 32 op/s 2024-04-12T21:37:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:12.902+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:37:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:12.903+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:12.937+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:37:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:12.937+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:37:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:14 smithi153 ceph-mon[36569]: pgmap v1110: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 32 KiB/s wr, 34 op/s 2024-04-12T21:37:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:14 smithi104 ceph-mon[36624]: pgmap v1110: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 32 KiB/s wr, 34 op/s 2024-04-12T21:37:14.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:14 smithi081 ceph-mon[31453]: pgmap v1110: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 32 KiB/s wr, 34 op/s 2024-04-12T21:37:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:16 smithi153 ceph-mon[36569]: pgmap v1111: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 27 KiB/s wr, 30 op/s 2024-04-12T21:37:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:16 smithi104 ceph-mon[36624]: pgmap v1111: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 27 KiB/s wr, 30 op/s 2024-04-12T21:37:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:16 smithi081 ceph-mon[31453]: pgmap v1111: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 27 KiB/s wr, 30 op/s 2024-04-12T21:37:18.982 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:18 smithi081 ceph-mon[31453]: pgmap v1112: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 27 KiB/s wr, 32 op/s 2024-04-12T21:37:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:18 smithi153 ceph-mon[36569]: pgmap v1112: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 27 KiB/s wr, 32 op/s 2024-04-12T21:37:19.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:18 smithi104 ceph-mon[36624]: pgmap v1112: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 27 KiB/s wr, 32 op/s 2024-04-12T21:37:20.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:19 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:37:20.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:19 smithi081 systemd[1]: Stopped Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:37:20.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:19 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 27.875s CPU time. 2024-04-12T21:37:20.026 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:19 smithi081 systemd[1]: Starting Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:37:20.405 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:20 smithi081 podman[93681]: 2024-04-12 21:37:19.931886025 +0000 UTC m=+0.035526722 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:37:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:20 smithi153 ceph-mon[36569]: pgmap v1113: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-12T21:37:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:20 smithi104 ceph-mon[36624]: pgmap v1113: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-12T21:37:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:20 smithi081 ceph-mon[31453]: pgmap v1113: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 25 KiB/s wr, 22 op/s 2024-04-12T21:37:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:22 smithi153 ceph-mon[36569]: pgmap v1114: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 25 KiB/s wr, 20 op/s 2024-04-12T21:37:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:22 smithi153 ceph-mon[36569]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:37:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:22 smithi153 ceph-mon[36569]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:22 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:22 smithi153 ceph-mon[36569]: osdmap e152: 12 total, 12 up, 12 in 2024-04-12T21:37:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:22 smithi104 ceph-mon[36624]: pgmap v1114: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 25 KiB/s wr, 20 op/s 2024-04-12T21:37:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:22 smithi104 ceph-mon[36624]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:37:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:22 smithi104 ceph-mon[36624]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:22 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:22 smithi104 ceph-mon[36624]: osdmap e152: 12 total, 12 up, 12 in 2024-04-12T21:37:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:22 smithi081 ceph-mon[31453]: pgmap v1114: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 25 KiB/s wr, 20 op/s 2024-04-12T21:37:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:22 smithi081 ceph-mon[31453]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:37:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:22 smithi081 ceph-mon[31453]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:22 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:22 smithi081 ceph-mon[31453]: osdmap e152: 12 total, 12 up, 12 in 2024-04-12T21:37:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:23.798+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:37:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:23.798+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:37:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:23.798+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:37:23.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:23 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:37:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:23 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:37:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:23 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:37:24.226 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:23 smithi081 podman[93681]: 2024-04-12 21:37:23.963578531 +0000 UTC m=+4.067219305 container create e726c0a7a71d26d8c83aff3d96ac43f4901cc117ab4715e1cde9b9902a7d7124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409) 2024-04-12T21:37:24.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:24 smithi081 podman[93681]: 2024-04-12 21:37:24.226546143 +0000 UTC m=+4.330186874 container init e726c0a7a71d26d8c83aff3d96ac43f4901cc117ab4715e1cde9b9902a7d7124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-04-12T21:37:24.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:24 smithi081 podman[93681]: 2024-04-12 21:37:24.237322622 +0000 UTC m=+4.340963373 container start e726c0a7a71d26d8c83aff3d96ac43f4901cc117ab4715e1cde9b9902a7d7124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-12T21:37:24.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:24 smithi081 ceph-mds[93701]: -- 172.21.15.81:0/1557740607 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564dc97fda40 con 0x564dc97d1000 2024-04-12T21:37:24.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:24 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: starting mds.d at 2024-04-12T21:37:24.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:24 smithi081 bash[93681]: e726c0a7a71d26d8c83aff3d96ac43f4901cc117ab4715e1cde9b9902a7d7124 2024-04-12T21:37:24.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:24 smithi081 systemd[1]: Started Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:37:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:24 smithi153 ceph-mon[36569]: pgmap v1116: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:37:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:24 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:reconnect 2024-04-12T21:37:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:24 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:37:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:24 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:37:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:24 smithi104 ceph-mon[36624]: pgmap v1116: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:37:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:24 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:reconnect 2024-04-12T21:37:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:24 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:37:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:24 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:37:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:24 smithi081 ceph-mon[31453]: pgmap v1116: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:37:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:24 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:reconnect 2024-04-12T21:37:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:24 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:37:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:24 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0109998 2024-04-12T21:37:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:rejoin 2024-04-12T21:37:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:25 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:boot 2024-04-12T21:37:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:25 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:rejoin} 1 up:standby 2024-04-12T21:37:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:37:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:rejoin 2024-04-12T21:37:26.253 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:25 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:boot 2024-04-12T21:37:26.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:25 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:rejoin} 1 up:standby 2024-04-12T21:37:26.254 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:37:26.254 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:rejoin 2024-04-12T21:37:26.255 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:25 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:boot 2024-04-12T21:37:26.255 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:25 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:rejoin} 1 up:standby 2024-04-12T21:37:26.255 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:37:27.006 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-mon[31453]: pgmap v1117: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:37:27.006 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:clientreplay 2024-04-12T21:37:27.006 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:37:27.006 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-mon[31453]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:37:27.008 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:37:27.008 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: *** Caught signal (Aborted) ** 2024-04-12T21:37:27.008 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: in thread 7f9e99800700 thread_name:ms_dispatch 2024-04-12T21:37:27.008 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:27.008 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9ea9798d20] 2024-04-12T21:37:27.008 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2: gsignal() 2024-04-12T21:37:27.008 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 3: abort() 2024-04-12T21:37:27.008 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55904b84c268] 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55904b8e80fc] 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55904bc1cf0a] 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 7: (MDLog::trim()+0xca9) [0x55904bc1e0c9] 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55904bc1e5a8] 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f9ea8684b23] 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f9ea978e1ca] 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 11: clone() 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2024-04-12T21:37:26.928+0000 7f9e99800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: in thread 7f9e99800700 thread_name:ms_dispatch 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9ea9798d20] 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2: gsignal() 2024-04-12T21:37:27.009 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 3: abort() 2024-04-12T21:37:27.010 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55904b84c268] 2024-04-12T21:37:27.010 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55904b8e80fc] 2024-04-12T21:37:27.010 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55904bc1cf0a] 2024-04-12T21:37:27.010 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 7: (MDLog::trim()+0xca9) [0x55904bc1e0c9] 2024-04-12T21:37:27.010 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55904bc1e5a8] 2024-04-12T21:37:27.010 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f9ea8684b23] 2024-04-12T21:37:27.010 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f9ea978e1ca] 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 11: clone() 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 0> 2024-04-12T21:37:26.928+0000 7f9e99800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: in thread 7f9e99800700 thread_name:ms_dispatch 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9ea9798d20] 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2: gsignal() 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 3: abort() 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55904b84c268] 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55904b8e80fc] 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55904bc1cf0a] 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 7: (MDLog::trim()+0xca9) [0x55904bc1e0c9] 2024-04-12T21:37:27.011 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55904bc1e5a8] 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f9ea8684b23] 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f9ea978e1ca] 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 11: clone() 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:26 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: -9> 2024-04-12T21:37:26.928+0000 7f9e99800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: in thread 7f9e99800700 thread_name:ms_dispatch 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9ea9798d20] 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2: gsignal() 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 3: abort() 2024-04-12T21:37:27.012 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55904b84c268] 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55904b8e80fc] 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55904bc1cf0a] 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 7: (MDLog::trim()+0xca9) [0x55904bc1e0c9] 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55904bc1e5a8] 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f9ea8684b23] 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f9ea978e1ca] 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 11: clone() 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:27.013 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[71771]: 2024-04-12T21:37:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:26 smithi153 ceph-mon[36569]: pgmap v1117: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:37:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:26 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:clientreplay 2024-04-12T21:37:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:26 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:37:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:26 smithi153 ceph-mon[36569]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:37:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:26 smithi104 ceph-mon[36624]: pgmap v1117: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 4 op/s 2024-04-12T21:37:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:26 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:clientreplay 2024-04-12T21:37:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:26 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:37:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:26 smithi104 ceph-mon[36624]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:37:27.391 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 podman[93736]: 2024-04-12 21:37:27.094869402 +0000 UTC m=+0.033600915 container died f82f7deb6ea97f3aac70ee05122ff72964f8fd69dc4762d0bd33885d6ca57c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:37:28.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:27 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:28.025 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:27 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:active 2024-04-12T21:37:28.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:27 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:37:28.026 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:27 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:37:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:27 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:27 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:active 2024-04-12T21:37:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:27 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:37:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:27 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:37:28.155 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 podman[93736]: 2024-04-12 21:37:27.797101556 +0000 UTC m=+0.735833076 container remove f82f7deb6ea97f3aac70ee05122ff72964f8fd69dc4762d0bd33885d6ca57c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409) 2024-04-12T21:37:28.155 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:27 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:37:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/1703364372,v1:172.21.15.81:6835/1703364372] up:active 2024-04-12T21:37:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:37:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:27 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:37:28.655 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:28 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-12T21:37:28.655 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:28 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 29.085s CPU time. 2024-04-12T21:37:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:28 smithi153 ceph-mon[36569]: pgmap v1118: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-12T21:37:29.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:28 smithi081 ceph-mon[31453]: pgmap v1118: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-12T21:37:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:28 smithi104 ceph-mon[36624]: pgmap v1118: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-12T21:37:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:30 smithi153 ceph-mon[36569]: pgmap v1119: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-12T21:37:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:30 smithi104 ceph-mon[36624]: pgmap v1119: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-12T21:37:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:30 smithi081 ceph-mon[31453]: pgmap v1119: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-12T21:37:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:32 smithi153 ceph-mon[36569]: pgmap v1120: 129 pgs: 129 active+clean; 525 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.1 KiB/s wr, 23 op/s 2024-04-12T21:37:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:32 smithi104 ceph-mon[36624]: pgmap v1120: 129 pgs: 129 active+clean; 525 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.1 KiB/s wr, 23 op/s 2024-04-12T21:37:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:32 smithi081 ceph-mon[31453]: pgmap v1120: 129 pgs: 129 active+clean; 525 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 2.1 KiB/s wr, 23 op/s 2024-04-12T21:37:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:34 smithi153 ceph-mon[36569]: pgmap v1121: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 20 KiB/s wr, 28 op/s 2024-04-12T21:37:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:34 smithi104 ceph-mon[36624]: pgmap v1121: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 20 KiB/s wr, 28 op/s 2024-04-12T21:37:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:34 smithi081 ceph-mon[31453]: pgmap v1121: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 20 KiB/s wr, 28 op/s 2024-04-12T21:37:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:36 smithi153 ceph-mon[36569]: pgmap v1122: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 KiB/s wr, 24 op/s 2024-04-12T21:37:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:36 smithi104 ceph-mon[36624]: pgmap v1122: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 KiB/s wr, 24 op/s 2024-04-12T21:37:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:36 smithi081 ceph-mon[31453]: pgmap v1122: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 17 KiB/s wr, 24 op/s 2024-04-12T21:37:38.779 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:38 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-04-12T21:37:38.779 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:38 smithi081 systemd[1]: Stopped Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:37:38.779 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:38 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 29.085s CPU time. 2024-04-12T21:37:38.779 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:38 smithi081 systemd[1]: Starting Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:37:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:38.809+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:37:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:38.809+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:37:38.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:38.809+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:37:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:38 smithi153 ceph-mon[36569]: pgmap v1123: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 17 KiB/s wr, 26 op/s 2024-04-12T21:37:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:38 smithi104 ceph-mon[36624]: pgmap v1123: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 17 KiB/s wr, 26 op/s 2024-04-12T21:37:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:38 smithi081 ceph-mon[31453]: pgmap v1123: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 17 KiB/s wr, 26 op/s 2024-04-12T21:37:39.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:38 smithi081 podman[93834]: 2024-04-12 21:37:38.685762293 +0000 UTC m=+0.035615976 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:37:39.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:38 smithi081 podman[93834]: 2024-04-12 21:37:38.824156668 +0000 UTC m=+0.174010391 container create 8f332831727a7c5b285edf64a66326a4d79500a2c441ff14a7c6fd4ab0ab2d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:37:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:40 smithi153 ceph-mon[36569]: pgmap v1124: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 16 KiB/s wr, 20 op/s 2024-04-12T21:37:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:40 smithi104 ceph-mon[36624]: pgmap v1124: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 16 KiB/s wr, 20 op/s 2024-04-12T21:37:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:40 smithi081 ceph-mon[31453]: pgmap v1124: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 16 KiB/s wr, 20 op/s 2024-04-12T21:37:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:42 smithi153 ceph-mon[36569]: pgmap v1125: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 16 KiB/s wr, 20 op/s 2024-04-12T21:37:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:42 smithi153 ceph-mon[36569]: Replacing daemon mds.j as rank 0 with standby daemon mds.d 2024-04-12T21:37:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:42 smithi153 ceph-mon[36569]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:42 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:42 smithi153 ceph-mon[36569]: osdmap e153: 12 total, 12 up, 12 in 2024-04-12T21:37:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:42 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:replay} 2024-04-12T21:37:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:42 smithi104 ceph-mon[36624]: pgmap v1125: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 16 KiB/s wr, 20 op/s 2024-04-12T21:37:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:42 smithi104 ceph-mon[36624]: Replacing daemon mds.j as rank 0 with standby daemon mds.d 2024-04-12T21:37:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:42 smithi104 ceph-mon[36624]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:42 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:42 smithi104 ceph-mon[36624]: osdmap e153: 12 total, 12 up, 12 in 2024-04-12T21:37:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:42 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:replay} 2024-04-12T21:37:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:42 smithi081 ceph-mon[31453]: pgmap v1125: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 16 KiB/s wr, 20 op/s 2024-04-12T21:37:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:42 smithi081 ceph-mon[31453]: Replacing daemon mds.j as rank 0 with standby daemon mds.d 2024-04-12T21:37:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:42 smithi081 ceph-mon[31453]: MDS daemon mds.j is removed because it is dead or otherwise unavailable. 2024-04-12T21:37:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:42 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:37:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:42 smithi081 ceph-mon[31453]: osdmap e153: 12 total, 12 up, 12 in 2024-04-12T21:37:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:42 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:replay} 2024-04-12T21:37:44.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:43 smithi081 podman[93834]: 2024-04-12 21:37:43.979406748 +0000 UTC m=+5.329260484 container init 8f332831727a7c5b285edf64a66326a4d79500a2c441ff14a7c6fd4ab0ab2d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-12T21:37:44.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:43 smithi081 podman[93834]: 2024-04-12 21:37:43.991163732 +0000 UTC m=+5.341017421 container start 8f332831727a7c5b285edf64a66326a4d79500a2c441ff14a7c6fd4ab0ab2d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:37:44.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:44 smithi081 ceph-mds[93853]: -- 172.21.15.81:0/532624045 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5636a70cda40 con 0x5636a70a0800 2024-04-12T21:37:44.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:44 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: starting mds.j at 2024-04-12T21:37:44.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:44 smithi081 bash[93834]: 8f332831727a7c5b285edf64a66326a4d79500a2c441ff14a7c6fd4ab0ab2d17 2024-04-12T21:37:44.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:37:44 smithi081 systemd[1]: Started Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:37:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:44 smithi153 ceph-mon[36569]: pgmap v1127: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:37:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:44 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:reconnect 2024-04-12T21:37:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:44 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:reconnect} 2024-04-12T21:37:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:44 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:37:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:44 smithi104 ceph-mon[36624]: pgmap v1127: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:37:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:44 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:reconnect 2024-04-12T21:37:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:44 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:reconnect} 2024-04-12T21:37:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:44 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:37:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:44 smithi081 ceph-mon[31453]: pgmap v1127: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:37:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:44 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:reconnect 2024-04-12T21:37:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:44 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:reconnect} 2024-04-12T21:37:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:44 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:37:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:45 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:rejoin 2024-04-12T21:37:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:45 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:boot 2024-04-12T21:37:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:45 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:37:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:37:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:45 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:rejoin 2024-04-12T21:37:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:45 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:boot 2024-04-12T21:37:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:45 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:37:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:37:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:45 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:rejoin 2024-04-12T21:37:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:45 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:boot 2024-04-12T21:37:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:45 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:37:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:37:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:46 smithi081 ceph-mon[31453]: pgmap v1128: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:37:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:46 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:clientreplay 2024-04-12T21:37:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:46 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:37:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:46 smithi081 ceph-mon[31453]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:37:47.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:46 smithi153 ceph-mon[36569]: pgmap v1128: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:37:47.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:46 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:clientreplay 2024-04-12T21:37:47.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:46 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:37:47.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:46 smithi153 ceph-mon[36569]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:37:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:46 smithi104 ceph-mon[36624]: pgmap v1128: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:37:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:46 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:clientreplay 2024-04-12T21:37:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:46 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:37:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:46 smithi104 ceph-mon[36624]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:37:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:active 2024-04-12T21:37:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:37:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:37:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-mon[31453]: pgmap v1129: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 KiB/s wr, 6 op/s 2024-04-12T21:37:48.156 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:37:48.156 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: *** Caught signal (Aborted) ** 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: in thread 7f0378000700 thread_name:ms_dispatch 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0387fb8d20] 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2: gsignal() 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 3: abort() 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564dab844268] 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564dab8e00fc] 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564dabc14f0a] 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 7: (MDLog::trim()+0xca9) [0x564dabc160c9] 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564dabc165a8] 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0386ea4b23] 2024-04-12T21:37:48.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0387fae1ca] 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 11: clone() 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2024-04-12T21:37:47.757+0000 7f0378000700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: in thread 7f0378000700 thread_name:ms_dispatch 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0387fb8d20] 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2: gsignal() 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 3: abort() 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564dab844268] 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564dab8e00fc] 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564dabc14f0a] 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 7: (MDLog::trim()+0xca9) [0x564dabc160c9] 2024-04-12T21:37:48.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564dabc165a8] 2024-04-12T21:37:48.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0386ea4b23] 2024-04-12T21:37:48.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0387fae1ca] 2024-04-12T21:37:48.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 11: clone() 2024-04-12T21:37:48.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 0> 2024-04-12T21:37:47.757+0000 7f0378000700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: in thread 7f0378000700 thread_name:ms_dispatch 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0387fb8d20] 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2: gsignal() 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 3: abort() 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564dab844268] 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564dab8e00fc] 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564dabc14f0a] 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 7: (MDLog::trim()+0xca9) [0x564dabc160c9] 2024-04-12T21:37:48.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564dabc165a8] 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0386ea4b23] 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0387fae1ca] 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 11: clone() 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 0> 2024-04-12T21:37:47.757+0000 7f0378000700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: in thread 7f0378000700 thread_name:ms_dispatch 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0387fb8d20] 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2: gsignal() 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 3: abort() 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x564dab844268] 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x564dab8e00fc] 2024-04-12T21:37:48.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x564dabc14f0a] 2024-04-12T21:37:48.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 7: (MDLog::trim()+0xca9) [0x564dabc160c9] 2024-04-12T21:37:48.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 8: (MDLog::log_trim_upkeep()+0x358) [0x564dabc165a8] 2024-04-12T21:37:48.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f0386ea4b23] 2024-04-12T21:37:48.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f0387fae1ca] 2024-04-12T21:37:48.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 11: clone() 2024-04-12T21:37:48.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:37:48.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[93697]: 2024-04-12T21:37:48.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:47 smithi081 podman[93888]: 2024-04-12 21:37:47.941088701 +0000 UTC m=+0.039756629 container died e726c0a7a71d26d8c83aff3d96ac43f4901cc117ab4715e1cde9b9902a7d7124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409) 2024-04-12T21:37:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:47 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:47 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:active 2024-04-12T21:37:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:47 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:37:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:47 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:37:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:47 smithi153 ceph-mon[36569]: pgmap v1129: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 KiB/s wr, 6 op/s 2024-04-12T21:37:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:47 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:37:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:47 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1438915245,v1:172.21.15.81:6837/1438915245] up:active 2024-04-12T21:37:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:47 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:37:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:47 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:37:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:47 smithi104 ceph-mon[36624]: pgmap v1129: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 KiB/s wr, 6 op/s 2024-04-12T21:37:48.652 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:48 smithi081 podman[93888]: 2024-04-12 21:37:48.652815134 +0000 UTC m=+0.751483122 container remove e726c0a7a71d26d8c83aff3d96ac43f4901cc117ab4715e1cde9b9902a7d7124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409) 2024-04-12T21:37:48.905 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:48 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:37:49.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:49 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-12T21:37:49.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:49 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 29.103s CPU time. 2024-04-12T21:37:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:50 smithi153 ceph-mon[36569]: pgmap v1130: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-12T21:37:50.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:50 smithi104 ceph-mon[36624]: pgmap v1130: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-12T21:37:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:50 smithi081 ceph-mon[31453]: pgmap v1130: 129 pgs: 129 active+clean; 526 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 KiB/s wr, 7 op/s 2024-04-12T21:37:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:52 smithi153 ceph-mon[36569]: pgmap v1131: 129 pgs: 129 active+clean; 525 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 KiB/s wr, 16 op/s 2024-04-12T21:37:52.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:52 smithi104 ceph-mon[36624]: pgmap v1131: 129 pgs: 129 active+clean; 525 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 KiB/s wr, 16 op/s 2024-04-12T21:37:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:52 smithi081 ceph-mon[31453]: pgmap v1131: 129 pgs: 129 active+clean; 525 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 KiB/s wr, 16 op/s 2024-04-12T21:37:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:53.824+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:37:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:53.824+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:37:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:37:53.824+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:37:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:54 smithi153 ceph-mon[36569]: pgmap v1132: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 97 KiB/s wr, 17 op/s 2024-04-12T21:37:54.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:54 smithi104 ceph-mon[36624]: pgmap v1132: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 97 KiB/s wr, 17 op/s 2024-04-12T21:37:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:54 smithi081 ceph-mon[31453]: pgmap v1132: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 97 KiB/s wr, 17 op/s 2024-04-12T21:37:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:56 smithi153 ceph-mon[36569]: pgmap v1133: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 83 KiB/s wr, 15 op/s 2024-04-12T21:37:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:56 smithi104 ceph-mon[36624]: pgmap v1133: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 83 KiB/s wr, 15 op/s 2024-04-12T21:37:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:56 smithi081 ceph-mon[31453]: pgmap v1133: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 83 KiB/s wr, 15 op/s 2024-04-12T21:37:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:58 smithi153 ceph-mon[36569]: pgmap v1134: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 83 KiB/s wr, 17 op/s 2024-04-12T21:37:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:37:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:58.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:58 smithi104 ceph-mon[36624]: pgmap v1134: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 83 KiB/s wr, 17 op/s 2024-04-12T21:37:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:37:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:58 smithi081 ceph-mon[31453]: pgmap v1134: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 83 KiB/s wr, 17 op/s 2024-04-12T21:37:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:37:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:37:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:37:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:59 smithi153 ceph-mon[36569]: pgmap v1135: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 74 KiB/s wr, 13 op/s 2024-04-12T21:37:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:59 smithi153 ceph-mon[36569]: pgmap v1136: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 89 KiB/s wr, 15 op/s 2024-04-12T21:37:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:37:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:37:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:37:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:59 smithi104 ceph-mon[36624]: pgmap v1135: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 74 KiB/s wr, 13 op/s 2024-04-12T21:37:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:59 smithi104 ceph-mon[36624]: pgmap v1136: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 89 KiB/s wr, 15 op/s 2024-04-12T21:37:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:37:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:37:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:37:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:59 smithi081 ceph-mon[31453]: pgmap v1135: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 74 KiB/s wr, 13 op/s 2024-04-12T21:37:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:59 smithi081 ceph-mon[31453]: pgmap v1136: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 89 KiB/s wr, 15 op/s 2024-04-12T21:37:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:37:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:37:59.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:59 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:37:59.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:59 smithi081 systemd[1]: Stopped Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:37:59.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:59 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 29.103s CPU time. 2024-04-12T21:37:59.801 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:59 smithi081 systemd[1]: Starting Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:38:00.156 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:37:59 smithi081 podman[94110]: 2024-04-12 21:37:59.705508866 +0000 UTC m=+0.039105293 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:38:01.065 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:00 smithi081 podman[94110]: 2024-04-12 21:38:00.759223272 +0000 UTC m=+1.092819763 container create 616209ec2b07dd2dd554033b140d250120736da040f4a34d3eed5759a8581b70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-04-12T21:38:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:38:01.127+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:01.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:01 smithi081 podman[94110]: 2024-04-12 21:38:01.065214133 +0000 UTC m=+1.398810637 container init 616209ec2b07dd2dd554033b140d250120736da040f4a34d3eed5759a8581b70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-12T21:38:01.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:01 smithi081 podman[94110]: 2024-04-12 21:38:01.075357835 +0000 UTC m=+1.408954267 container start 616209ec2b07dd2dd554033b140d250120736da040f4a34d3eed5759a8581b70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-12T21:38:01.407 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mds[94128]: -- 172.21.15.81:0/2845918503 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f48a345a40 con 0x55f48a319000 2024-04-12T21:38:01.407 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: starting mds.d at 2024-04-12T21:38:01.407 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:01 smithi081 bash[94110]: 616209ec2b07dd2dd554033b140d250120736da040f4a34d3eed5759a8581b70 2024-04-12T21:38:01.407 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:01 smithi081 systemd[1]: Started Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:38:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: pgmap v1137: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 89 KiB/s wr, 5 op/s 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: daemon mds.d restarted 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: osdmap e154: 12 total, 12 up, 12 in 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:boot 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:01 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:replay} 1 up:standby 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: pgmap v1137: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 89 KiB/s wr, 5 op/s 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: daemon mds.d restarted 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: osdmap e154: 12 total, 12 up, 12 in 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:boot 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:01 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:replay} 1 up:standby 2024-04-12T21:38:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: pgmap v1137: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 89 KiB/s wr, 5 op/s 2024-04-12T21:38:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: daemon mds.d restarted 2024-04-12T21:38:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: osdmap e154: 12 total, 12 up, 12 in 2024-04-12T21:38:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:boot 2024-04-12T21:38:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-04-12T21:38:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:38:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:01 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:replay} 1 up:standby 2024-04-12T21:38:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:03 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:reconnect 2024-04-12T21:38:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:03 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:reconnect} 1 up:standby 2024-04-12T21:38:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:03 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139997 2024-04-12T21:38:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:03 smithi081 ceph-mon[31453]: pgmap v1139: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:38:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:03 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:reconnect 2024-04-12T21:38:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:03 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:reconnect} 1 up:standby 2024-04-12T21:38:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:03 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139997 2024-04-12T21:38:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:03 smithi153 ceph-mon[36569]: pgmap v1139: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:38:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:03 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:reconnect 2024-04-12T21:38:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:03 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:reconnect} 1 up:standby 2024-04-12T21:38:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:03 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0139997 2024-04-12T21:38:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:03 smithi104 ceph-mon[36624]: pgmap v1139: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:38:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:04 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:rejoin 2024-04-12T21:38:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:04 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:rejoin} 1 up:standby 2024-04-12T21:38:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:04 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:rejoin 2024-04-12T21:38:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:04 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:rejoin} 1 up:standby 2024-04-12T21:38:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:04 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:rejoin 2024-04-12T21:38:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:04 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:rejoin} 1 up:standby 2024-04-12T21:38:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:05 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:clientreplay 2024-04-12T21:38:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:05 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:38:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:05 smithi153 ceph-mon[36569]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:38:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:05 smithi153 ceph-mon[36569]: pgmap v1140: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2 op/s 2024-04-12T21:38:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:05 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:clientreplay 2024-04-12T21:38:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:05 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:38:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:05 smithi104 ceph-mon[36624]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:38:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:05 smithi104 ceph-mon[36624]: pgmap v1140: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2 op/s 2024-04-12T21:38:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:clientreplay 2024-04-12T21:38:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:38:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-mon[31453]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:38:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-mon[31453]: pgmap v1140: 129 pgs: 129 active+clean; 526 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 2 op/s 2024-04-12T21:38:06.025 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:38:06.025 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: *** Caught signal (Aborted) ** 2024-04-12T21:38:06.025 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: in thread 7ff98b600700 thread_name:ms_dispatch 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff99b776d20] 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2: gsignal() 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 3: abort() 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563685ddf268] 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563685e7b0fc] 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5636861aff0a] 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 7: (MDLog::trim()+0xca9) [0x5636861b10c9] 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5636861b15a8] 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff99a662b23] 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff99b76c1ca] 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 11: clone() 2024-04-12T21:38:06.026 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2024-04-12T21:38:05.864+0000 7ff98b600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: in thread 7ff98b600700 thread_name:ms_dispatch 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff99b776d20] 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2: gsignal() 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 3: abort() 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563685ddf268] 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563685e7b0fc] 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5636861aff0a] 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 7: (MDLog::trim()+0xca9) [0x5636861b10c9] 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5636861b15a8] 2024-04-12T21:38:06.027 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff99a662b23] 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff99b76c1ca] 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 11: clone() 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 0> 2024-04-12T21:38:05.864+0000 7ff98b600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: in thread 7ff98b600700 thread_name:ms_dispatch 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff99b776d20] 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2: gsignal() 2024-04-12T21:38:06.028 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 3: abort() 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563685ddf268] 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563685e7b0fc] 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5636861aff0a] 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 7: (MDLog::trim()+0xca9) [0x5636861b10c9] 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5636861b15a8] 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff99a662b23] 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff99b76c1ca] 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 11: clone() 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: -19> 2024-04-12T21:38:05.864+0000 7ff98b600700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: in thread 7ff98b600700 thread_name:ms_dispatch 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2024-04-12T21:38:06.029 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff99b776d20] 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2: gsignal() 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 3: abort() 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x563685ddf268] 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x563685e7b0fc] 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x5636861aff0a] 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 7: (MDLog::trim()+0xca9) [0x5636861b10c9] 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 8: (MDLog::log_trim_upkeep()+0x358) [0x5636861b15a8] 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7ff99a662b23] 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7ff99b76c1ca] 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 11: clone() 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:06.030 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:05 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[93849]: 2024-04-12T21:38:06.031 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:06 smithi081 podman[94167]: 2024-04-12 21:38:06.025537361 +0000 UTC m=+0.044038958 container died 8f332831727a7c5b285edf64a66326a4d79500a2c441ff14a7c6fd4ab0ab2d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-12T21:38:06.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:06 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:06.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:06 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:active 2024-04-12T21:38:06.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:06 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:38:06.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:06 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:38:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:06 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:06 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:active 2024-04-12T21:38:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:06 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:38:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:06 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:38:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:06 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:06 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2905588902,v1:172.21.15.81:6835/2905588902] up:active 2024-04-12T21:38:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:06 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:38:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:06 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:38:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:07 smithi081 ceph-mon[31453]: pgmap v1141: 129 pgs: 129 active+clean; 525 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 93 KiB/s wr, 24 op/s 2024-04-12T21:38:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:07 smithi153 ceph-mon[36569]: pgmap v1141: 129 pgs: 129 active+clean; 525 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 93 KiB/s wr, 24 op/s 2024-04-12T21:38:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:07 smithi104 ceph-mon[36624]: pgmap v1141: 129 pgs: 129 active+clean; 525 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 93 KiB/s wr, 24 op/s 2024-04-12T21:38:08.105 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:07 smithi081 podman[94167]: 2024-04-12 21:38:07.660973168 +0000 UTC m=+1.679474805 container remove 8f332831727a7c5b285edf64a66326a4d79500a2c441ff14a7c6fd4ab0ab2d17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-12T21:38:08.106 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:07 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:38:08.406 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:08 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-12T21:38:08.406 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:08 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 28.462s CPU time. 2024-04-12T21:38:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:08.840+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:38:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:08.840+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:38:08.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:08.840+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:38:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:09 smithi153 ceph-mon[36569]: pgmap v1142: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 74 KiB/s wr, 27 op/s 2024-04-12T21:38:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:09 smithi104 ceph-mon[36624]: pgmap v1142: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 74 KiB/s wr, 27 op/s 2024-04-12T21:38:09.866 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:09 smithi081 ceph-mon[31453]: pgmap v1142: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 74 KiB/s wr, 27 op/s 2024-04-12T21:38:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:11 smithi104 ceph-mon[36624]: pgmap v1143: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 75 KiB/s wr, 32 op/s 2024-04-12T21:38:11.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:11 smithi081 ceph-mon[31453]: pgmap v1143: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 75 KiB/s wr, 32 op/s 2024-04-12T21:38:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:11 smithi153 ceph-mon[36569]: pgmap v1143: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 75 KiB/s wr, 32 op/s 2024-04-12T21:38:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:13 smithi104 ceph-mon[36624]: pgmap v1144: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 66 KiB/s wr, 29 op/s 2024-04-12T21:38:13.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:13 smithi081 ceph-mon[31453]: pgmap v1144: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 66 KiB/s wr, 29 op/s 2024-04-12T21:38:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:13 smithi153 ceph-mon[36569]: pgmap v1144: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 66 KiB/s wr, 29 op/s 2024-04-12T21:38:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:15 smithi104 ceph-mon[36624]: pgmap v1145: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 63 KiB/s wr, 28 op/s 2024-04-12T21:38:15.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:15 smithi081 ceph-mon[31453]: pgmap v1145: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 63 KiB/s wr, 28 op/s 2024-04-12T21:38:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:15 smithi153 ceph-mon[36569]: pgmap v1145: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 63 KiB/s wr, 28 op/s 2024-04-12T21:38:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:17 smithi104 ceph-mon[36624]: pgmap v1146: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 63 KiB/s wr, 30 op/s 2024-04-12T21:38:17.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:17 smithi153 ceph-mon[36569]: pgmap v1146: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 63 KiB/s wr, 30 op/s 2024-04-12T21:38:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:17 smithi081 ceph-mon[31453]: pgmap v1146: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 63 KiB/s wr, 30 op/s 2024-04-12T21:38:18.538 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-04-12T21:38:18.538 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 systemd[1]: Stopped Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:38:18.538 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 28.462s CPU time. 2024-04-12T21:38:18.538 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 systemd[1]: Starting Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:38:18.866 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 podman[94266]: 2024-04-12 21:38:18.443525194 +0000 UTC m=+0.033019483 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:38:18.866 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 podman[94266]: 2024-04-12 21:38:18.562543095 +0000 UTC m=+0.152037445 container create 34624ad0e10fe6ca593cde2b2454b5efd3e316e2c127d82817cf9e6b1f8a785f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-12T21:38:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:18 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:38:18.931+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:19.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 podman[94266]: 2024-04-12 21:38:18.866654023 +0000 UTC m=+0.456148352 container init 34624ad0e10fe6ca593cde2b2454b5efd3e316e2c127d82817cf9e6b1f8a785f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-12T21:38:19.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 podman[94266]: 2024-04-12 21:38:18.877303498 +0000 UTC m=+0.466797808 container start 34624ad0e10fe6ca593cde2b2454b5efd3e316e2c127d82817cf9e6b1f8a785f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-12T21:38:19.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 ceph-mds[94285]: -- 172.21.15.81:0/1757672898 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e6f41efa40 con 0x55e6f41c3000 2024-04-12T21:38:19.157 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: starting mds.j at 2024-04-12T21:38:19.157 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 bash[94266]: 34624ad0e10fe6ca593cde2b2454b5efd3e316e2c127d82817cf9e6b1f8a785f 2024-04-12T21:38:19.157 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:18 smithi081 systemd[1]: Started Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:38:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: pgmap v1147: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 852 B/s wr, 15 op/s 2024-04-12T21:38:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: daemon mds.j restarted 2024-04-12T21:38:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: osdmap e155: 12 total, 12 up, 12 in 2024-04-12T21:38:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:boot 2024-04-12T21:38:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:38:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:38:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:19 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:38:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: pgmap v1147: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 852 B/s wr, 15 op/s 2024-04-12T21:38:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: daemon mds.j restarted 2024-04-12T21:38:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: osdmap e155: 12 total, 12 up, 12 in 2024-04-12T21:38:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:boot 2024-04-12T21:38:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:38:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:38:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:19 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:38:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: pgmap v1147: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 852 B/s wr, 15 op/s 2024-04-12T21:38:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: daemon mds.j restarted 2024-04-12T21:38:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: osdmap e155: 12 total, 12 up, 12 in 2024-04-12T21:38:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:boot 2024-04-12T21:38:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:38:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:38:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:19 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:38:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:20 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:reconnect 2024-04-12T21:38:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:20 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:38:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:20 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:38:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:20 smithi153 ceph-mon[36569]: pgmap v1149: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:38:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:20 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:reconnect 2024-04-12T21:38:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:20 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:38:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:20 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:38:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:20 smithi104 ceph-mon[36624]: pgmap v1149: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:38:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:20 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:reconnect 2024-04-12T21:38:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:20 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:38:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:20 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:38:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:20 smithi081 ceph-mon[31453]: pgmap v1149: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:38:22.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:21 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:rejoin 2024-04-12T21:38:22.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:21 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:38:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:21 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:rejoin 2024-04-12T21:38:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:21 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:38:22.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:21 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:rejoin 2024-04-12T21:38:22.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:21 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:38:23.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:38:23.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: *** Caught signal (Aborted) ** 2024-04-12T21:38:23.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: in thread 7fda2a800700 thread_name:ms_dispatch 2024-04-12T21:38:23.238 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fda3a901d20] 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2: gsignal() 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 3: abort() 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f474137268] 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f4741d30fc] 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f474507f0a] 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 7: (MDLog::trim()+0xca9) [0x55f4745090c9] 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f4745095a8] 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fda397edb23] 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fda3a8f71ca] 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 11: clone() 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2024-04-12T21:38:23.210+0000 7fda2a800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: in thread 7fda2a800700 thread_name:ms_dispatch 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2024-04-12T21:38:23.239 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fda3a901d20] 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2: gsignal() 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 3: abort() 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f474137268] 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f4741d30fc] 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f474507f0a] 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 7: (MDLog::trim()+0xca9) [0x55f4745090c9] 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f4745095a8] 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fda397edb23] 2024-04-12T21:38:23.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:22 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:clientreplay 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:22 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:22 smithi081 ceph-mon[31453]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:22 smithi081 ceph-mon[31453]: pgmap v1150: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:22 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:22 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:active 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:22 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fda3a8f71ca] 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 11: clone() 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2024-04-12T21:38:23.241 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 0> 2024-04-12T21:38:23.210+0000 7fda2a800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: in thread 7fda2a800700 thread_name:ms_dispatch 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fda3a901d20] 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2: gsignal() 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 3: abort() 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f474137268] 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f4741d30fc] 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f474507f0a] 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 7: (MDLog::trim()+0xca9) [0x55f4745090c9] 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f4745095a8] 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fda397edb23] 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fda3a8f71ca] 2024-04-12T21:38:23.242 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 11: clone() 2024-04-12T21:38:23.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:23.243 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2024-04-12T21:38:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:22 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:clientreplay 2024-04-12T21:38:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:22 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:38:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:22 smithi153 ceph-mon[36569]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:38:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:22 smithi153 ceph-mon[36569]: pgmap v1150: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:38:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:22 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:22 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:active 2024-04-12T21:38:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:22 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:38:23.364 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:22 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:clientreplay 2024-04-12T21:38:23.365 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:22 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:38:23.365 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:22 smithi104 ceph-mon[36624]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:38:23.365 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:22 smithi104 ceph-mon[36624]: pgmap v1150: 129 pgs: 129 active+clean; 524 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:38:23.365 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:22 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:23.365 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:22 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/2548643237,v1:172.21.15.81:6837/2548643237] up:active 2024-04-12T21:38:23.365 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:22 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: -10> 2024-04-12T21:38:23.210+0000 7fda2a800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: in thread 7fda2a800700 thread_name:ms_dispatch 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fda3a901d20] 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2: gsignal() 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 3: abort() 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f474137268] 2024-04-12T21:38:23.657 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f4741d30fc] 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f474507f0a] 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 7: (MDLog::trim()+0xca9) [0x55f4745090c9] 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f4745095a8] 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fda397edb23] 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fda3a8f71ca] 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 11: clone() 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94124]: 2024-04-12T21:38:23.658 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:23 smithi081 podman[94320]: 2024-04-12 21:38:23.358734257 +0000 UTC m=+0.035428160 container died 616209ec2b07dd2dd554033b140d250120736da040f4a34d3eed5759a8581b70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:38:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:23.856+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:38:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:23.856+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:38:23.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:23.856+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:38:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:25 smithi104 ceph-mon[36624]: pgmap v1151: 129 pgs: 129 active+clean; 523 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 204 B/s wr, 12 op/s 2024-04-12T21:38:25.864 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:25 smithi081 ceph-mon[31453]: pgmap v1151: 129 pgs: 129 active+clean; 523 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 204 B/s wr, 12 op/s 2024-04-12T21:38:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:25 smithi153 ceph-mon[36569]: pgmap v1151: 129 pgs: 129 active+clean; 523 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 204 B/s wr, 12 op/s 2024-04-12T21:38:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:27 smithi104 ceph-mon[36624]: pgmap v1152: 129 pgs: 129 active+clean; 521 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 39 KiB/s wr, 38 op/s 2024-04-12T21:38:27.846 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:27 smithi081 ceph-mon[31453]: pgmap v1152: 129 pgs: 129 active+clean; 521 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 39 KiB/s wr, 38 op/s 2024-04-12T21:38:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:27 smithi153 ceph-mon[36569]: pgmap v1152: 129 pgs: 129 active+clean; 521 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 39 KiB/s wr, 38 op/s 2024-04-12T21:38:29.831 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:29 smithi081 podman[94320]: 2024-04-12 21:38:29.823202407 +0000 UTC m=+6.499896331 container remove 616209ec2b07dd2dd554033b140d250120736da040f4a34d3eed5759a8581b70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240409) 2024-04-12T21:38:29.831 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:29 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:38:29.831 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:29 smithi081 ceph-mon[31453]: pgmap v1153: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 KiB/s wr, 46 op/s 2024-04-12T21:38:29.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:29 smithi104 ceph-mon[36624]: pgmap v1153: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 KiB/s wr, 46 op/s 2024-04-12T21:38:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:29 smithi153 ceph-mon[36569]: pgmap v1153: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 60 KiB/s wr, 46 op/s 2024-04-12T21:38:30.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:30 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-12T21:38:30.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:30 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 28.609s CPU time. 2024-04-12T21:38:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:31 smithi104 ceph-mon[36624]: pgmap v1154: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 52 KiB/s wr, 41 op/s 2024-04-12T21:38:31.879 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:31 smithi081 ceph-mon[31453]: pgmap v1154: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 52 KiB/s wr, 41 op/s 2024-04-12T21:38:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:31 smithi153 ceph-mon[36569]: pgmap v1154: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 52 KiB/s wr, 41 op/s 2024-04-12T21:38:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:33 smithi081 ceph-mon[31453]: pgmap v1155: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 50 KiB/s wr, 40 op/s 2024-04-12T21:38:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:33 smithi153 ceph-mon[36569]: pgmap v1155: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 50 KiB/s wr, 40 op/s 2024-04-12T21:38:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:33 smithi104 ceph-mon[36624]: pgmap v1155: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 50 KiB/s wr, 40 op/s 2024-04-12T21:38:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:35 smithi081 ceph-mon[31453]: pgmap v1156: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 50 KiB/s wr, 40 op/s 2024-04-12T21:38:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:35 smithi153 ceph-mon[36569]: pgmap v1156: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 50 KiB/s wr, 40 op/s 2024-04-12T21:38:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:35 smithi104 ceph-mon[36624]: pgmap v1156: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s rd, 50 KiB/s wr, 40 op/s 2024-04-12T21:38:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:37 smithi081 ceph-mon[31453]: pgmap v1157: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 50 KiB/s wr, 35 op/s 2024-04-12T21:38:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:37 smithi153 ceph-mon[36569]: pgmap v1157: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 50 KiB/s wr, 35 op/s 2024-04-12T21:38:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:37 smithi104 ceph-mon[36624]: pgmap v1157: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 25 MiB/s rd, 50 KiB/s wr, 35 op/s 2024-04-12T21:38:38.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:38.871+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:38:38.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:38.871+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:38:38.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:38.871+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:38:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:38 smithi081 ceph-mon[31453]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:38:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:38 smithi081 ceph-mon[31453]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:38:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:38 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:38 smithi081 ceph-mon[31453]: osdmap e156: 12 total, 12 up, 12 in 2024-04-12T21:38:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:38 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:38:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:38 smithi153 ceph-mon[36569]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:38:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:38 smithi153 ceph-mon[36569]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:38:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:38 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:38 smithi153 ceph-mon[36569]: osdmap e156: 12 total, 12 up, 12 in 2024-04-12T21:38:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:38 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:38:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:38 smithi104 ceph-mon[36624]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:38:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:38 smithi104 ceph-mon[36624]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:38:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:38 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:38 smithi104 ceph-mon[36624]: osdmap e156: 12 total, 12 up, 12 in 2024-04-12T21:38:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:38 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:38:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:39 smithi153 ceph-mon[36569]: pgmap v1159: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 5 op/s 2024-04-12T21:38:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:39 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:reconnect 2024-04-12T21:38:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:39 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:38:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:39 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:38:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:39 smithi104 ceph-mon[36624]: pgmap v1159: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 5 op/s 2024-04-12T21:38:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:39 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:reconnect 2024-04-12T21:38:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:39 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:38:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:39 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:38:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:39 smithi081 ceph-mon[31453]: pgmap v1159: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 5 op/s 2024-04-12T21:38:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:39 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:reconnect 2024-04-12T21:38:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:39 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:38:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:39 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:38:40.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:40 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:38:40.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:40 smithi081 systemd[1]: Stopped Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:38:40.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:40 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 28.609s CPU time. 2024-04-12T21:38:40.739 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:40 smithi081 systemd[1]: Starting Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:38:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:40 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:rejoin 2024-04-12T21:38:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:40 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:38:41.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:40 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:rejoin 2024-04-12T21:38:41.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:40 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:38:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:40 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:rejoin 2024-04-12T21:38:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:40 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:38:41.156 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:40 smithi081 podman[94403]: 2024-04-12 21:38:40.778792334 +0000 UTC m=+0.032485769 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:38:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:41 smithi153 ceph-mon[36569]: pgmap v1160: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:38:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:41 smithi104 ceph-mon[36624]: pgmap v1160: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:38:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:41 smithi081 ceph-mon[31453]: pgmap v1160: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 4 op/s 2024-04-12T21:38:43.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:clientreplay 2024-04-12T21:38:43.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:38:43.038 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-mon[31453]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:38:43.039 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:38:43.039 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: *** Caught signal (Aborted) ** 2024-04-12T21:38:43.039 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: in thread 7f4744200700 thread_name:ms_dispatch 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4754296d20] 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2: gsignal() 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 3: abort() 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55e6b6b83268] 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55e6b6c1f0fc] 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55e6b6f53f0a] 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 7: (MDLog::trim()+0xca9) [0x55e6b6f550c9] 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55e6b6f555a8] 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4753182b23] 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f475428c1ca] 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 11: clone() 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2024-04-12T21:38:42.907+0000 7f4744200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:43.040 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: in thread 7f4744200700 thread_name:ms_dispatch 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4754296d20] 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2: gsignal() 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 3: abort() 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55e6b6b83268] 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55e6b6c1f0fc] 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55e6b6f53f0a] 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 7: (MDLog::trim()+0xca9) [0x55e6b6f550c9] 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55e6b6f555a8] 2024-04-12T21:38:43.041 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4753182b23] 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f475428c1ca] 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 11: clone() 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 0> 2024-04-12T21:38:42.907+0000 7f4744200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: in thread 7f4744200700 thread_name:ms_dispatch 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4754296d20] 2024-04-12T21:38:43.042 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2: gsignal() 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 3: abort() 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55e6b6b83268] 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55e6b6c1f0fc] 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55e6b6f53f0a] 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 7: (MDLog::trim()+0xca9) [0x55e6b6f550c9] 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55e6b6f555a8] 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4753182b23] 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f475428c1ca] 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 11: clone() 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: -8> 2024-04-12T21:38:42.907+0000 7f4744200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: in thread 7f4744200700 thread_name:ms_dispatch 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:38:43.043 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4754296d20] 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2: gsignal() 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 3: abort() 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55e6b6b83268] 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55e6b6c1f0fc] 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55e6b6f53f0a] 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 7: (MDLog::trim()+0xca9) [0x55e6b6f550c9] 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55e6b6f555a8] 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f4753182b23] 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f475428c1ca] 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 11: clone() 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:38:43.044 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:42 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94281]: 2024-04-12T21:38:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:42 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:clientreplay 2024-04-12T21:38:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:42 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:38:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:42 smithi153 ceph-mon[36569]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:38:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:42 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:clientreplay 2024-04-12T21:38:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:42 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:38:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:42 smithi104 ceph-mon[36624]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:38:44.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:43 smithi081 ceph-mon[31453]: pgmap v1161: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:38:44.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:43 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:44.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:43 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:active 2024-04-12T21:38:44.035 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:43 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:38:44.035 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:43 smithi081 podman[94403]: 2024-04-12 21:38:43.850148853 +0000 UTC m=+3.103842325 container create a069477693595fa8fa14e97eead0dcac53cde775115d6916d9c046342e8a40c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-12T21:38:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:43 smithi153 ceph-mon[36569]: pgmap v1161: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:38:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:43 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:43 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:active 2024-04-12T21:38:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:43 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:38:44.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:43 smithi104 ceph-mon[36624]: pgmap v1161: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 204 B/s wr, 4 op/s 2024-04-12T21:38:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:43 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:38:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:43 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/4022528594,v1:172.21.15.81:6835/4022528594] up:active 2024-04-12T21:38:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:43 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:38:44.334 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:44 smithi081 podman[94420]: 2024-04-12 21:38:44.035008825 +0000 UTC m=+0.977494752 container died 34624ad0e10fe6ca593cde2b2454b5efd3e316e2c127d82817cf9e6b1f8a785f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:38:44.937 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:44 smithi081 podman[94420]: 2024-04-12 21:38:44.773870647 +0000 UTC m=+1.716356597 container remove 34624ad0e10fe6ca593cde2b2454b5efd3e316e2c127d82817cf9e6b1f8a785f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.build-date=20240409, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True) 2024-04-12T21:38:44.937 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:44 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:38:44.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:44 smithi081 podman[94403]: 2024-04-12 21:38:44.884375567 +0000 UTC m=+4.138069035 container init a069477693595fa8fa14e97eead0dcac53cde775115d6916d9c046342e8a40c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, io.buildah.version=1.33.5) 2024-04-12T21:38:44.937 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:44 smithi081 podman[94403]: 2024-04-12 21:38:44.893141099 +0000 UTC m=+4.146834549 container start a069477693595fa8fa14e97eead0dcac53cde775115d6916d9c046342e8a40c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-12T21:38:44.938 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:44 smithi081 ceph-mds[94452]: -- 172.21.15.81:0/1802348977 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55afb3ad1a40 con 0x55afb3aa4c00 2024-04-12T21:38:45.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:44 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: starting mds.d at 2024-04-12T21:38:45.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:44 smithi081 bash[94403]: a069477693595fa8fa14e97eead0dcac53cde775115d6916d9c046342e8a40c8 2024-04-12T21:38:45.263 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:38:44 smithi081 systemd[1]: Started Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:38:45.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:45 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-12T21:38:45.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:45 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 27.942s CPU time. 2024-04-12T21:38:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:45 smithi153 ceph-mon[36569]: pgmap v1162: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 409 B/s wr, 6 op/s 2024-04-12T21:38:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:45 smithi104 ceph-mon[36624]: pgmap v1162: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 409 B/s wr, 6 op/s 2024-04-12T21:38:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:45 smithi081 ceph-mon[31453]: pgmap v1162: 129 pgs: 129 active+clean; 520 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 409 B/s wr, 6 op/s 2024-04-12T21:38:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:46 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:boot 2024-04-12T21:38:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:46 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:38:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:38:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:46 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:38:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:46 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:boot 2024-04-12T21:38:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:46 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:38:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:38:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:46 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:38:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:46 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:boot 2024-04-12T21:38:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:46 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:38:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:38:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:46 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:38:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:47 smithi153 ceph-mon[36569]: pgmap v1163: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-04-12T21:38:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:47 smithi104 ceph-mon[36624]: pgmap v1163: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-04-12T21:38:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:47 smithi081 ceph-mon[31453]: pgmap v1163: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 KiB/s wr, 14 op/s 2024-04-12T21:38:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:49 smithi153 ceph-mon[36569]: pgmap v1164: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 12 KiB/s wr, 17 op/s 2024-04-12T21:38:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:49 smithi104 ceph-mon[36624]: pgmap v1164: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 12 KiB/s wr, 17 op/s 2024-04-12T21:38:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:49 smithi081 ceph-mon[31453]: pgmap v1164: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 12 KiB/s wr, 17 op/s 2024-04-12T21:38:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:51 smithi153 ceph-mon[36569]: pgmap v1165: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:51 smithi104 ceph-mon[36624]: pgmap v1165: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:52.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:51 smithi081 ceph-mon[31453]: pgmap v1165: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:53.887+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:38:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:53.887+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:38:53.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:38:53.887+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:53 smithi104 ceph-mon[36624]: pgmap v1166: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-12T21:38:54.181 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:53 smithi081 ceph-mon[31453]: pgmap v1166: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-12T21:38:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:53 smithi153 ceph-mon[36569]: pgmap v1166: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 11 KiB/s wr, 18 op/s 2024-04-12T21:38:55.810 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:55 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-04-12T21:38:55.811 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:55 smithi081 systemd[1]: Stopped Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:38:55.811 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:55 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 27.942s CPU time. 2024-04-12T21:38:55.811 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:55 smithi081 systemd[1]: Starting Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:38:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:55 smithi104 ceph-mon[36624]: pgmap v1167: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:56.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:55 smithi081 ceph-mon[31453]: pgmap v1167: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:56.142 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:55 smithi081 podman[94542]: 2024-04-12 21:38:55.905585755 +0000 UTC m=+0.061817766 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:38:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:55 smithi153 ceph-mon[36569]: pgmap v1167: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:56.406 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:56 smithi081 podman[94542]: 2024-04-12 21:38:56.142087227 +0000 UTC m=+0.298319212 container create b587fa0ce5520b267bef7cb83e6185e30c0083109824fbc640fa7da34618ce6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-04-12T21:38:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:56 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:38:56.856+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:57.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:56 smithi081 podman[94542]: 2024-04-12 21:38:56.795703108 +0000 UTC m=+0.951935128 container init b587fa0ce5520b267bef7cb83e6185e30c0083109824fbc640fa7da34618ce6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-04-12T21:38:57.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:56 smithi081 podman[94542]: 2024-04-12 21:38:56.80503424 +0000 UTC m=+0.961266225 container start b587fa0ce5520b267bef7cb83e6185e30c0083109824fbc640fa7da34618ce6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240409) 2024-04-12T21:38:57.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:56 smithi081 ceph-mds[94562]: -- 172.21.15.81:0/2434356259 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d739d4ba40 con 0x55d739d1f000 2024-04-12T21:38:57.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:56 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: starting mds.j at 2024-04-12T21:38:57.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:56 smithi081 bash[94542]: b587fa0ce5520b267bef7cb83e6185e30c0083109824fbc640fa7da34618ce6e 2024-04-12T21:38:57.157 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:38:56 smithi081 systemd[1]: Started Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:38:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: pgmap v1168: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: daemon mds.j restarted 2024-04-12T21:38:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: osdmap e157: 12 total, 12 up, 12 in 2024-04-12T21:38:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:boot 2024-04-12T21:38:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:38:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:38:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:57 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: pgmap v1168: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: daemon mds.j restarted 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: osdmap e157: 12 total, 12 up, 12 in 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:boot 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:38:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:57 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:38:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: pgmap v1168: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 11 KiB/s wr, 17 op/s 2024-04-12T21:38:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: daemon mds.j restarted 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: osdmap e157: 12 total, 12 up, 12 in 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:boot 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:38:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:57 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:38:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:58 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:reconnect 2024-04-12T21:38:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:58 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:38:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:58 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:38:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:58 smithi081 ceph-mon[31453]: pgmap v1170: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 7 op/s 2024-04-12T21:38:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:38:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:58 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:reconnect 2024-04-12T21:38:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:58 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:38:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:58 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:38:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:58 smithi153 ceph-mon[36569]: pgmap v1170: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 7 op/s 2024-04-12T21:38:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:38:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:58 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:reconnect 2024-04-12T21:38:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:58 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:38:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:58 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:38:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:58 smithi104 ceph-mon[36624]: pgmap v1170: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 7 op/s 2024-04-12T21:38:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:39:00.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:59 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:rejoin 2024-04-12T21:39:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:38:59 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:39:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:59 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:rejoin 2024-04-12T21:39:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:38:59 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:39:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:59 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:rejoin 2024-04-12T21:39:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:38:59 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:00 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:clientreplay 2024-04-12T21:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:00 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:00 smithi081 ceph-mon[31453]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:00 smithi081 ceph-mon[31453]: pgmap v1171: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:00 smithi081 ceph-mon[31453]: pgmap v1172: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:39:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:00 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:clientreplay 2024-04-12T21:39:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:00 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:39:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:00 smithi153 ceph-mon[36569]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:39:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:39:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:39:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:00 smithi153 ceph-mon[36569]: pgmap v1171: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:39:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:00 smithi153 ceph-mon[36569]: pgmap v1172: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:39:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:00 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:clientreplay 2024-04-12T21:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:00 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:00 smithi104 ceph-mon[36624]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:00 smithi104 ceph-mon[36624]: pgmap v1171: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:00 smithi104 ceph-mon[36624]: pgmap v1172: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 0 B/s wr, 8 op/s 2024-04-12T21:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:39:02.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:02.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:02 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:active 2024-04-12T21:39:02.063 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:02 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:39:02.064 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:02 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:39:02.066 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:39:02.066 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: *** Caught signal (Aborted) ** 2024-04-12T21:39:02.066 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: in thread 7fcf98e00700 thread_name:ms_dispatch 2024-04-12T21:39:02.066 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:02.066 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcfa8f7cd20] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2: gsignal() 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 3: abort() 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55afb00df268] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55afb017b0fc] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55afb04aff0a] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 7: (MDLog::trim()+0xca9) [0x55afb04b10c9] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55afb04b15a8] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fcfa7e68b23] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fcfa8f721ca] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 11: clone() 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2024-04-12T21:39:01.799+0000 7fcf98e00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: in thread 7fcf98e00700 thread_name:ms_dispatch 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcfa8f7cd20] 2024-04-12T21:39:02.067 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2: gsignal() 2024-04-12T21:39:02.068 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 3: abort() 2024-04-12T21:39:02.068 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55afb00df268] 2024-04-12T21:39:02.068 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55afb017b0fc] 2024-04-12T21:39:02.068 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55afb04aff0a] 2024-04-12T21:39:02.068 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 7: (MDLog::trim()+0xca9) [0x55afb04b10c9] 2024-04-12T21:39:02.068 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55afb04b15a8] 2024-04-12T21:39:02.068 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fcfa7e68b23] 2024-04-12T21:39:02.069 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fcfa8f721ca] 2024-04-12T21:39:02.069 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 11: clone() 2024-04-12T21:39:02.069 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 0> 2024-04-12T21:39:01.799+0000 7fcf98e00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: in thread 7fcf98e00700 thread_name:ms_dispatch 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcfa8f7cd20] 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2: gsignal() 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 3: abort() 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55afb00df268] 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55afb017b0fc] 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55afb04aff0a] 2024-04-12T21:39:02.070 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 7: (MDLog::trim()+0xca9) [0x55afb04b10c9] 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55afb04b15a8] 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fcfa7e68b23] 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fcfa8f721ca] 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 11: clone() 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: -2> 2024-04-12T21:39:01.799+0000 7fcf98e00700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: in thread 7fcf98e00700 thread_name:ms_dispatch 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcfa8f7cd20] 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2: gsignal() 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 3: abort() 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55afb00df268] 2024-04-12T21:39:02.071 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55afb017b0fc] 2024-04-12T21:39:02.072 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55afb04aff0a] 2024-04-12T21:39:02.072 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 7: (MDLog::trim()+0xca9) [0x55afb04b10c9] 2024-04-12T21:39:02.072 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55afb04b15a8] 2024-04-12T21:39:02.072 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7fcfa7e68b23] 2024-04-12T21:39:02.072 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7fcfa8f721ca] 2024-04-12T21:39:02.072 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 11: clone() 2024-04-12T21:39:02.072 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:02.072 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:01 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94434]: 2024-04-12T21:39:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:01 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:01 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:active 2024-04-12T21:39:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:01 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:39:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:01 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:39:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:01 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:01 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/2592351136,v1:172.21.15.81:6835/2592351136] up:active 2024-04-12T21:39:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:01 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:39:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:01 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:39:02.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:02 smithi081 podman[94725]: 2024-04-12 21:39:02.122443557 +0000 UTC m=+0.039512307 container died a069477693595fa8fa14e97eead0dcac53cde775115d6916d9c046342e8a40c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:39:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:02 smithi081 ceph-mon[31453]: pgmap v1173: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 52 KiB/s wr, 25 op/s 2024-04-12T21:39:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:02 smithi153 ceph-mon[36569]: pgmap v1173: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 52 KiB/s wr, 25 op/s 2024-04-12T21:39:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:02 smithi104 ceph-mon[36624]: pgmap v1173: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 52 KiB/s wr, 25 op/s 2024-04-12T21:39:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:05 smithi153 ceph-mon[36569]: pgmap v1174: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 53 KiB/s wr, 28 op/s 2024-04-12T21:39:05.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:05 smithi104 ceph-mon[36624]: pgmap v1174: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 53 KiB/s wr, 28 op/s 2024-04-12T21:39:05.863 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:05 smithi081 ceph-mon[31453]: pgmap v1174: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 53 KiB/s wr, 28 op/s 2024-04-12T21:39:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:07 smithi153 ceph-mon[36569]: pgmap v1175: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 44 KiB/s wr, 27 op/s 2024-04-12T21:39:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:07 smithi104 ceph-mon[36624]: pgmap v1175: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 44 KiB/s wr, 27 op/s 2024-04-12T21:39:07.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:07 smithi081 ceph-mon[31453]: pgmap v1175: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 44 KiB/s wr, 27 op/s 2024-04-12T21:39:08.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:08.901+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:39:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:08.920+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:39:08.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:08.920+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:39:09.359 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:08 smithi081 podman[94725]: 2024-04-12 21:39:08.938231462 +0000 UTC m=+6.855300222 container remove a069477693595fa8fa14e97eead0dcac53cde775115d6916d9c046342e8a40c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240409, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-04-12T21:39:09.359 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:08 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:39:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:09 smithi081 ceph-mon[31453]: pgmap v1176: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 75 KiB/s wr, 27 op/s 2024-04-12T21:39:09.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:09 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-12T21:39:09.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:09 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 28.750s CPU time. 2024-04-12T21:39:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:09 smithi153 ceph-mon[36569]: pgmap v1176: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 75 KiB/s wr, 27 op/s 2024-04-12T21:39:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:09 smithi104 ceph-mon[36624]: pgmap v1176: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 75 KiB/s wr, 27 op/s 2024-04-12T21:39:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:11 smithi153 ceph-mon[36569]: pgmap v1177: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 73 KiB/s wr, 25 op/s 2024-04-12T21:39:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:11 smithi104 ceph-mon[36624]: pgmap v1177: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 73 KiB/s wr, 25 op/s 2024-04-12T21:39:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:11 smithi081 ceph-mon[31453]: pgmap v1177: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 73 KiB/s wr, 25 op/s 2024-04-12T21:39:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:13 smithi153 ceph-mon[36569]: pgmap v1178: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 22 op/s 2024-04-12T21:39:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:13 smithi104 ceph-mon[36624]: pgmap v1178: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 22 op/s 2024-04-12T21:39:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:13 smithi081 ceph-mon[31453]: pgmap v1178: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 61 KiB/s wr, 22 op/s 2024-04-12T21:39:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:15 smithi153 ceph-mon[36569]: pgmap v1179: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-12T21:39:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:15 smithi104 ceph-mon[36624]: pgmap v1179: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-12T21:39:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:15 smithi081 ceph-mon[31453]: pgmap v1179: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 27 KiB/s wr, 11 op/s 2024-04-12T21:39:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:17 smithi153 ceph-mon[36569]: pgmap v1180: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 27 KiB/s wr, 7 op/s 2024-04-12T21:39:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:17 smithi153 ceph-mon[36569]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:39:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:17 smithi153 ceph-mon[36569]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:39:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:17 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:17 smithi104 ceph-mon[36624]: pgmap v1180: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 27 KiB/s wr, 7 op/s 2024-04-12T21:39:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:17 smithi104 ceph-mon[36624]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:39:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:17 smithi104 ceph-mon[36624]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:39:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:17 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:17 smithi081 ceph-mon[31453]: pgmap v1180: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 27 KiB/s wr, 7 op/s 2024-04-12T21:39:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:17 smithi081 ceph-mon[31453]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:39:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:17 smithi081 ceph-mon[31453]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:39:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:17 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:18 smithi153 ceph-mon[36569]: osdmap e158: 12 total, 12 up, 12 in 2024-04-12T21:39:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:18 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:39:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:18 smithi104 ceph-mon[36624]: osdmap e158: 12 total, 12 up, 12 in 2024-04-12T21:39:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:18 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:39:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:18 smithi081 ceph-mon[31453]: osdmap e158: 12 total, 12 up, 12 in 2024-04-12T21:39:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:18 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:39:19.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:19 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:39:19.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:19 smithi081 systemd[1]: Stopped Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:39:19.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:19 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 28.750s CPU time. 2024-04-12T21:39:19.755 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:19 smithi081 systemd[1]: Starting Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:39:20.007 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:19 smithi081 ceph-mon[31453]: pgmap v1182: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:39:20.007 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:19 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:reconnect 2024-04-12T21:39:20.007 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:19 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:39:20.007 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:19 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:39:20.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:19 smithi081 podman[94817]: 2024-04-12 21:39:19.709063651 +0000 UTC m=+0.038013487 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:39:20.008 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:19 smithi081 podman[94817]: 2024-04-12 21:39:19.857074091 +0000 UTC m=+0.186023975 container create a6614fe2b4985ce33c8d1b30ba033f167894c312aaf9f42addef0cd83e1daa20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-12T21:39:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:19 smithi153 ceph-mon[36569]: pgmap v1182: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:39:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:19 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:reconnect 2024-04-12T21:39:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:19 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:39:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:19 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:39:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:19 smithi104 ceph-mon[36624]: pgmap v1182: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 4 op/s 2024-04-12T21:39:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:19 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:reconnect 2024-04-12T21:39:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:19 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:39:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:19 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0169997 2024-04-12T21:39:20.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:20 smithi081 podman[94817]: 2024-04-12 21:39:20.143392773 +0000 UTC m=+0.472342619 container init a6614fe2b4985ce33c8d1b30ba033f167894c312aaf9f42addef0cd83e1daa20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:39:20.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:20 smithi081 podman[94817]: 2024-04-12 21:39:20.163728502 +0000 UTC m=+0.492678334 container start a6614fe2b4985ce33c8d1b30ba033f167894c312aaf9f42addef0cd83e1daa20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-12T21:39:20.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:20 smithi081 ceph-mds[94835]: -- 172.21.15.81:0/251670447 <== mon.0 v2:172.21.15.81:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d1b579e4e0 con 0x55d1b5775000 2024-04-12T21:39:20.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:20 smithi081 ceph-mds[94835]: -- 172.21.15.81:0/251670447 <== mon.0 v2:172.21.15.81:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d1b57a1a40 con 0x55d1b5775000 2024-04-12T21:39:20.260 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:20 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: starting mds.d at 2024-04-12T21:39:20.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:20 smithi081 bash[94817]: a6614fe2b4985ce33c8d1b30ba033f167894c312aaf9f42addef0cd83e1daa20 2024-04-12T21:39:20.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:20 smithi081 systemd[1]: Started Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:39:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:20 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:rejoin 2024-04-12T21:39:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:20 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:39:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:20 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:rejoin 2024-04-12T21:39:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:20 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:39:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:20 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:rejoin 2024-04-12T21:39:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:20 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:39:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:21 smithi153 ceph-mon[36569]: pgmap v1183: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:39:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:21 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:boot 2024-04-12T21:39:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:21 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:clientreplay 2024-04-12T21:39:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:21 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:39:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:39:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:21 smithi153 ceph-mon[36569]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:39:22.058 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:21 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2024-04-12T21:39:21.873+0000 7f61ec000700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-12T21:39:22.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:21 smithi081 ceph-mon[31453]: pgmap v1183: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:39:22.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:21 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:boot 2024-04-12T21:39:22.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:21 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:clientreplay 2024-04-12T21:39:22.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:21 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:39:22.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:39:22.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:21 smithi081 ceph-mon[31453]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:39:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:21 smithi104 ceph-mon[36624]: pgmap v1183: 129 pgs: 129 active+clean; 519 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 4 op/s 2024-04-12T21:39:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:21 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:boot 2024-04-12T21:39:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:21 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:clientreplay 2024-04-12T21:39:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:21 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:clientreplay} 1 up:standby 2024-04-12T21:39:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-12T21:39:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:21 smithi104 ceph-mon[36624]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: *** Caught signal (Aborted) ** 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: in thread 7f61e9800700 thread_name:ms_dispatch 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f61f9960d20] 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2: gsignal() 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 3: abort() 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d7284ee268] 2024-04-12T21:39:22.407 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d72858a0fc] 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d7288bef0a] 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 7: (MDLog::trim()+0xca9) [0x55d7288c00c9] 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d7288c05a8] 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f61f884cb23] 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f61f99561ca] 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 11: clone() 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2024-04-12T21:39:22.073+0000 7f61e9800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: in thread 7f61e9800700 thread_name:ms_dispatch 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f61f9960d20] 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2: gsignal() 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 3: abort() 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d7284ee268] 2024-04-12T21:39:22.408 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d72858a0fc] 2024-04-12T21:39:22.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d7288bef0a] 2024-04-12T21:39:22.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 7: (MDLog::trim()+0xca9) [0x55d7288c00c9] 2024-04-12T21:39:22.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d7288c05a8] 2024-04-12T21:39:22.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f61f884cb23] 2024-04-12T21:39:22.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f61f99561ca] 2024-04-12T21:39:22.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 11: clone() 2024-04-12T21:39:22.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:22.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 0> 2024-04-12T21:39:22.073+0000 7f61e9800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: in thread 7f61e9800700 thread_name:ms_dispatch 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f61f9960d20] 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2: gsignal() 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 3: abort() 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d7284ee268] 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d72858a0fc] 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d7288bef0a] 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 7: (MDLog::trim()+0xca9) [0x55d7288c00c9] 2024-04-12T21:39:22.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d7288c05a8] 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f61f884cb23] 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f61f99561ca] 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 11: clone() 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: -4> 2024-04-12T21:39:22.073+0000 7f61e9800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: in thread 7f61e9800700 thread_name:ms_dispatch 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f61f9960d20] 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2: gsignal() 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 3: abort() 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d7284ee268] 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d72858a0fc] 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d7288bef0a] 2024-04-12T21:39:22.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 7: (MDLog::trim()+0xca9) [0x55d7288c00c9] 2024-04-12T21:39:22.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d7288c05a8] 2024-04-12T21:39:22.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f61f884cb23] 2024-04-12T21:39:22.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f61f99561ca] 2024-04-12T21:39:22.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 11: clone() 2024-04-12T21:39:22.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:22.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94558]: 2024-04-12T21:39:22.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:22 smithi081 podman[94871]: 2024-04-12 21:39:22.206455076 +0000 UTC m=+0.033524539 container died b587fa0ce5520b267bef7cb83e6185e30c0083109824fbc640fa7da34618ce6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2) 2024-04-12T21:39:22.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:22.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:active 2024-04-12T21:39:22.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:39:22.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:22 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:39:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:22 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:22 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:active 2024-04-12T21:39:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:22 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:39:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:22 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:39:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:22 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:22 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/4049206744,v1:172.21.15.81:6837/4049206744] up:active 2024-04-12T21:39:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:22 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 1 up:standby 2024-04-12T21:39:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:22 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 1 up:standby-replay 2024-04-12T21:39:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:23.936+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:39:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:23.936+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:39:23.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:23.936+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:39:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:23 smithi153 ceph-mon[36569]: pgmap v1184: 129 pgs: 129 active+clean; 518 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 38 KiB/s wr, 13 op/s 2024-04-12T21:39:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:23 smithi104 ceph-mon[36624]: pgmap v1184: 129 pgs: 129 active+clean; 518 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 38 KiB/s wr, 13 op/s 2024-04-12T21:39:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:23 smithi081 ceph-mon[31453]: pgmap v1184: 129 pgs: 129 active+clean; 518 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 38 KiB/s wr, 13 op/s 2024-04-12T21:39:25.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:24 smithi081 podman[94871]: 2024-04-12 21:39:24.701647528 +0000 UTC m=+2.528717053 container remove b587fa0ce5520b267bef7cb83e6185e30c0083109824fbc640fa7da34618ce6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-12T21:39:25.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:24 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:39:25.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:25 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-12T21:39:25.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:25 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 29.133s CPU time. 2024-04-12T21:39:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:25 smithi153 ceph-mon[36569]: pgmap v1185: 129 pgs: 129 active+clean; 518 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 38 KiB/s wr, 19 op/s 2024-04-12T21:39:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:25 smithi104 ceph-mon[36624]: pgmap v1185: 129 pgs: 129 active+clean; 518 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 38 KiB/s wr, 19 op/s 2024-04-12T21:39:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:25 smithi081 ceph-mon[31453]: pgmap v1185: 129 pgs: 129 active+clean; 518 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 38 KiB/s wr, 19 op/s 2024-04-12T21:39:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:27 smithi104 ceph-mon[36624]: pgmap v1186: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 39 KiB/s wr, 34 op/s 2024-04-12T21:39:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:27 smithi081 ceph-mon[31453]: pgmap v1186: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 39 KiB/s wr, 34 op/s 2024-04-12T21:39:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:27 smithi153 ceph-mon[36569]: pgmap v1186: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 39 KiB/s wr, 34 op/s 2024-04-12T21:39:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:29 smithi104 ceph-mon[36624]: pgmap v1187: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 KiB/s wr, 37 op/s 2024-04-12T21:39:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:29 smithi081 ceph-mon[31453]: pgmap v1187: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 KiB/s wr, 37 op/s 2024-04-12T21:39:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:29 smithi153 ceph-mon[36569]: pgmap v1187: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 57 KiB/s wr, 37 op/s 2024-04-12T21:39:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:31 smithi104 ceph-mon[36624]: pgmap v1188: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 50 KiB/s wr, 34 op/s 2024-04-12T21:39:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:31 smithi081 ceph-mon[31453]: pgmap v1188: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 50 KiB/s wr, 34 op/s 2024-04-12T21:39:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:31 smithi153 ceph-mon[36569]: pgmap v1188: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 50 KiB/s wr, 34 op/s 2024-04-12T21:39:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:33 smithi104 ceph-mon[36624]: pgmap v1189: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 50 KiB/s wr, 35 op/s 2024-04-12T21:39:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:33 smithi081 ceph-mon[31453]: pgmap v1189: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 50 KiB/s wr, 35 op/s 2024-04-12T21:39:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:33 smithi153 ceph-mon[36569]: pgmap v1189: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 50 KiB/s wr, 35 op/s 2024-04-12T21:39:35.827 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:35 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-04-12T21:39:35.827 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:35 smithi081 systemd[1]: Stopped Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:39:35.827 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:35 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 29.133s CPU time. 2024-04-12T21:39:35.827 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:35 smithi081 systemd[1]: Starting Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:39:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:35 smithi104 ceph-mon[36624]: pgmap v1190: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 19 KiB/s wr, 26 op/s 2024-04-12T21:39:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:35 smithi081 ceph-mon[31453]: pgmap v1190: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 19 KiB/s wr, 26 op/s 2024-04-12T21:39:36.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:35 smithi081 podman[94979]: 2024-04-12 21:39:35.77818695 +0000 UTC m=+0.036889960 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:39:36.156 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:35 smithi081 podman[94979]: 2024-04-12 21:39:35.920038901 +0000 UTC m=+0.178741948 container create 6c53e4c6e316aa991f19a05c40095064280c86e816c678780da1037882595ac7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-12T21:39:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:35 smithi153 ceph-mon[36569]: pgmap v1190: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 19 KiB/s wr, 26 op/s 2024-04-12T21:39:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:39:36.392+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:39:36.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:36 smithi081 podman[94979]: 2024-04-12 21:39:36.315674136 +0000 UTC m=+0.574377190 container init 6c53e4c6e316aa991f19a05c40095064280c86e816c678780da1037882595ac7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240409, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-12T21:39:36.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:36 smithi081 podman[94979]: 2024-04-12 21:39:36.327190115 +0000 UTC m=+0.585893131 container start 6c53e4c6e316aa991f19a05c40095064280c86e816c678780da1037882595ac7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240409, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-12T21:39:36.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mds[94998]: -- 172.21.15.81:0/1481808287 <== mon.2 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0680cba40 con 0x55f06809f000 2024-04-12T21:39:36.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: starting mds.j at 2024-04-12T21:39:36.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:36 smithi081 bash[94979]: 6c53e4c6e316aa991f19a05c40095064280c86e816c678780da1037882595ac7 2024-04-12T21:39:36.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:39:36 smithi081 systemd[1]: Started Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:39:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: daemon mds.j restarted 2024-04-12T21:39:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:39:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: osdmap e159: 12 total, 12 up, 12 in 2024-04-12T21:39:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:boot 2024-04-12T21:39:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:39:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:39:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:39:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:39:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:36 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:39:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: daemon mds.j restarted 2024-04-12T21:39:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:39:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: osdmap e159: 12 total, 12 up, 12 in 2024-04-12T21:39:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:boot 2024-04-12T21:39:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:39:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:39:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:39:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:39:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:36 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:39:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: daemon mds.j restarted 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: osdmap e159: 12 total, 12 up, 12 in 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:boot 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: Standby daemon mds.d assigned to filesystem cephfs as rank 0 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby-replay 1 up:standby, 1 failed 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:36 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:replay} 1 up:standby 2024-04-12T21:39:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:37 smithi104 ceph-mon[36624]: pgmap v1191: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 18 KiB/s wr, 21 op/s 2024-04-12T21:39:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:37 smithi081 ceph-mon[31453]: pgmap v1191: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 18 KiB/s wr, 21 op/s 2024-04-12T21:39:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:37 smithi153 ceph-mon[36569]: pgmap v1191: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 18 KiB/s wr, 21 op/s 2024-04-12T21:39:38.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:38.952+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:39:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:38.952+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:39:38.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:38.952+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:39:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:38 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:reconnect 2024-04-12T21:39:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:38 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:39:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:38 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:39:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:38 smithi081 ceph-mon[31453]: pgmap v1193: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:39:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:38 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:reconnect 2024-04-12T21:39:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:38 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:39:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:38 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:39:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:38 smithi153 ceph-mon[36569]: pgmap v1193: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:39:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:38 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:reconnect 2024-04-12T21:39:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:38 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:reconnect} 1 up:standby 2024-04-12T21:39:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:38 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0119998 2024-04-12T21:39:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:38 smithi104 ceph-mon[36624]: pgmap v1193: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 5 op/s 2024-04-12T21:39:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:39 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:rejoin 2024-04-12T21:39:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:39 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:39:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:39 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:rejoin 2024-04-12T21:39:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:39 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:39:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:39 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:rejoin 2024-04-12T21:39:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:39 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:rejoin} 1 up:standby 2024-04-12T21:39:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:40 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:clientreplay 2024-04-12T21:39:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:40 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:39:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:40 smithi081 ceph-mon[31453]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:39:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:40 smithi081 ceph-mon[31453]: pgmap v1194: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 204 B/s wr, 8 op/s 2024-04-12T21:39:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:40 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:clientreplay 2024-04-12T21:39:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:40 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:39:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:40 smithi153 ceph-mon[36569]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:39:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:40 smithi153 ceph-mon[36569]: pgmap v1194: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 204 B/s wr, 8 op/s 2024-04-12T21:39:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:40 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:clientreplay 2024-04-12T21:39:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:40 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=d=up:clientreplay} 1 up:standby 2024-04-12T21:39:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:40 smithi104 ceph-mon[36624]: daemon mds.d is now active in filesystem cephfs as rank 0 2024-04-12T21:39:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:40 smithi104 ceph-mon[36624]: pgmap v1194: 129 pgs: 129 active+clean; 517 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 204 B/s wr, 8 op/s 2024-04-12T21:39:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:active 2024-04-12T21:39:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:39:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:39:42.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:39:42.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: *** Caught signal (Aborted) ** 2024-04-12T21:39:42.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: in thread 7f868b200700 thread_name:ms_dispatch 2024-04-12T21:39:42.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:42.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f869b2edd20] 2024-04-12T21:39:42.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2: gsignal() 2024-04-12T21:39:42.157 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 3: abort() 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d193526268] 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d1935c20fc] 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d1938f6f0a] 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 7: (MDLog::trim()+0xca9) [0x55d1938f80c9] 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d1938f85a8] 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f869a1d9b23] 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f869b2e31ca] 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 11: clone() 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2024-04-12T21:39:41.804+0000 7f868b200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: in thread 7f868b200700 thread_name:ms_dispatch 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f869b2edd20] 2024-04-12T21:39:42.158 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2: gsignal() 2024-04-12T21:39:42.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 3: abort() 2024-04-12T21:39:42.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d193526268] 2024-04-12T21:39:42.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d1935c20fc] 2024-04-12T21:39:42.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d1938f6f0a] 2024-04-12T21:39:42.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 7: (MDLog::trim()+0xca9) [0x55d1938f80c9] 2024-04-12T21:39:42.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d1938f85a8] 2024-04-12T21:39:42.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f869a1d9b23] 2024-04-12T21:39:42.159 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f869b2e31ca] 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 11: clone() 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 0> 2024-04-12T21:39:41.804+0000 7f868b200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: in thread 7f868b200700 thread_name:ms_dispatch 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f869b2edd20] 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2: gsignal() 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 3: abort() 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d193526268] 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d1935c20fc] 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d1938f6f0a] 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 7: (MDLog::trim()+0xca9) [0x55d1938f80c9] 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d1938f85a8] 2024-04-12T21:39:42.160 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f869a1d9b23] 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f869b2e31ca] 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 11: clone() 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 0> 2024-04-12T21:39:41.804+0000 7f868b200700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: in thread 7f868b200700 thread_name:ms_dispatch 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f869b2edd20] 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2: gsignal() 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 3: abort() 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55d193526268] 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55d1935c20fc] 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55d1938f6f0a] 2024-04-12T21:39:42.161 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 7: (MDLog::trim()+0xca9) [0x55d1938f80c9] 2024-04-12T21:39:42.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55d1938f85a8] 2024-04-12T21:39:42.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f869a1d9b23] 2024-04-12T21:39:42.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f869b2e31ca] 2024-04-12T21:39:42.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 11: clone() 2024-04-12T21:39:42.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:39:42.162 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:41 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d[94831]: 2024-04-12T21:39:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:41 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:41 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:active 2024-04-12T21:39:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:41 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:39:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:41 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:39:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:41 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:39:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:41 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/793567891,v1:172.21.15.81:6835/793567891] up:active 2024-04-12T21:39:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:41 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby 2024-04-12T21:39:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:41 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=d=up:active} 1 up:standby-replay 2024-04-12T21:39:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:42 smithi081 ceph-mon[31453]: pgmap v1195: 129 pgs: 129 active+clean; 516 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 48 KiB/s wr, 23 op/s 2024-04-12T21:39:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:42 smithi153 ceph-mon[36569]: pgmap v1195: 129 pgs: 129 active+clean; 516 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 48 KiB/s wr, 23 op/s 2024-04-12T21:39:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:42 smithi104 ceph-mon[36624]: pgmap v1195: 129 pgs: 129 active+clean; 516 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 48 KiB/s wr, 23 op/s 2024-04-12T21:39:44.656 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:44 smithi081 podman[95033]: 2024-04-12 21:39:44.352942055 +0000 UTC m=+2.329984387 container died a6614fe2b4985ce33c8d1b30ba033f167894c312aaf9f42addef0cd83e1daa20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240409, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:39:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:45 smithi153 ceph-mon[36569]: pgmap v1196: 129 pgs: 129 active+clean; 516 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 49 KiB/s wr, 28 op/s 2024-04-12T21:39:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:45 smithi104 ceph-mon[36624]: pgmap v1196: 129 pgs: 129 active+clean; 516 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 49 KiB/s wr, 28 op/s 2024-04-12T21:39:45.863 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:45 smithi081 ceph-mon[31453]: pgmap v1196: 129 pgs: 129 active+clean; 516 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 49 KiB/s wr, 28 op/s 2024-04-12T21:39:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:47 smithi153 ceph-mon[36569]: pgmap v1197: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 49 KiB/s wr, 32 op/s 2024-04-12T21:39:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:47 smithi104 ceph-mon[36624]: pgmap v1197: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 49 KiB/s wr, 32 op/s 2024-04-12T21:39:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:47 smithi081 ceph-mon[31453]: pgmap v1197: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 49 KiB/s wr, 32 op/s 2024-04-12T21:39:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:49 smithi153 ceph-mon[36569]: pgmap v1198: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 41 KiB/s wr, 27 op/s 2024-04-12T21:39:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:49 smithi104 ceph-mon[36624]: pgmap v1198: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 41 KiB/s wr, 27 op/s 2024-04-12T21:39:49.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:49 smithi081 ceph-mon[31453]: pgmap v1198: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 41 KiB/s wr, 27 op/s 2024-04-12T21:39:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:51 smithi153 ceph-mon[36569]: pgmap v1199: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 41 KiB/s wr, 28 op/s 2024-04-12T21:39:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:51 smithi104 ceph-mon[36624]: pgmap v1199: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 41 KiB/s wr, 28 op/s 2024-04-12T21:39:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:51 smithi081 ceph-mon[31453]: pgmap v1199: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 41 KiB/s wr, 28 op/s 2024-04-12T21:39:52.405 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:52 smithi081 podman[95033]: 2024-04-12 21:39:52.047074738 +0000 UTC m=+10.024117094 container remove a6614fe2b4985ce33c8d1b30ba033f167894c312aaf9f42addef0cd83e1daa20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240409, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-12T21:39:52.406 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:52 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:39:52.906 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:52 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-12T21:39:52.906 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:39:52 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 29.497s CPU time. 2024-04-12T21:39:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:53 smithi153 ceph-mon[36569]: pgmap v1200: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 41 KiB/s wr, 25 op/s 2024-04-12T21:39:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:53 smithi104 ceph-mon[36624]: pgmap v1200: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 41 KiB/s wr, 25 op/s 2024-04-12T21:39:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:53 smithi081 ceph-mon[31453]: pgmap v1200: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 41 KiB/s wr, 25 op/s 2024-04-12T21:39:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:53.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:39:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:53.961+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:39:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:39:53.961+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:39:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:55 smithi153 ceph-mon[36569]: pgmap v1201: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 938 B/s wr, 12 op/s 2024-04-12T21:39:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:55 smithi104 ceph-mon[36624]: pgmap v1201: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 938 B/s wr, 12 op/s 2024-04-12T21:39:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:55 smithi081 ceph-mon[31453]: pgmap v1201: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 MiB/s rd, 938 B/s wr, 12 op/s 2024-04-12T21:39:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:57 smithi153 ceph-mon[36569]: pgmap v1202: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 767 B/s wr, 8 op/s 2024-04-12T21:39:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:57 smithi104 ceph-mon[36624]: pgmap v1202: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 767 B/s wr, 8 op/s 2024-04-12T21:39:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:57 smithi081 ceph-mon[31453]: pgmap v1202: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 767 B/s wr, 8 op/s 2024-04-12T21:39:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:58 smithi153 ceph-mon[36569]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:39:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:58 smithi153 ceph-mon[36569]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:39:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:58 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:58 smithi153 ceph-mon[36569]: osdmap e160: 12 total, 12 up, 12 in 2024-04-12T21:39:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:58 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:39:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:58 smithi104 ceph-mon[36624]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:39:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:58 smithi104 ceph-mon[36624]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:39:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:58 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:58 smithi104 ceph-mon[36624]: osdmap e160: 12 total, 12 up, 12 in 2024-04-12T21:39:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:58 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:39:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:58 smithi081 ceph-mon[31453]: Replacing daemon mds.d as rank 0 with standby daemon mds.j 2024-04-12T21:39:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:58 smithi081 ceph-mon[31453]: MDS daemon mds.d is removed because it is dead or otherwise unavailable. 2024-04-12T21:39:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:58 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:39:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:58 smithi081 ceph-mon[31453]: osdmap e160: 12 total, 12 up, 12 in 2024-04-12T21:39:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:58 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:40:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:59 smithi153 ceph-mon[36569]: pgmap v1204: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:40:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:59 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:reconnect 2024-04-12T21:40:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:39:59 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:40:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:59 smithi104 ceph-mon[36624]: pgmap v1204: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:40:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:59 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:reconnect 2024-04-12T21:40:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:39:59 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:40:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:59 smithi081 ceph-mon[31453]: pgmap v1204: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 4 op/s 2024-04-12T21:40:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:59 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:reconnect 2024-04-12T21:40:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:39:59 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:40:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: Health detail: HEALTH_WARN 10 failed cephadm daemon(s); 1 filesystem is degraded 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: [WRN] CEPHADM_FAILED_DAEMON: 10 failed cephadm daemon(s) 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.g on smithi081 is in unknown state 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.a on smithi081 is in error state 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.k on smithi104 is in error state 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.e on smithi104 is in error state 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.h on smithi104 is in error state 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.b on smithi104 is in error state 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.l on smithi153 is in error state 2024-04-12T21:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.f on smithi153 is in error state 2024-04-12T21:40:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.c on smithi153 is in error state 2024-04-12T21:40:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: daemon mds.i on smithi153 is in error state 2024-04-12T21:40:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T21:40:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: fs cephfs is degraded 2024-04-12T21:40:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:rejoin 2024-04-12T21:40:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:40:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: Health detail: HEALTH_WARN 10 failed cephadm daemon(s); 1 filesystem is degraded 2024-04-12T21:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: [WRN] CEPHADM_FAILED_DAEMON: 10 failed cephadm daemon(s) 2024-04-12T21:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.g on smithi081 is in unknown state 2024-04-12T21:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.a on smithi081 is in error state 2024-04-12T21:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.k on smithi104 is in error state 2024-04-12T21:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.e on smithi104 is in error state 2024-04-12T21:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.h on smithi104 is in error state 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.b on smithi104 is in error state 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.l on smithi153 is in error state 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.f on smithi153 is in error state 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.c on smithi153 is in error state 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: daemon mds.i on smithi153 is in error state 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: fs cephfs is degraded 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:rejoin 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0189996 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: Health detail: HEALTH_WARN 10 failed cephadm daemon(s); 1 filesystem is degraded 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: [WRN] CEPHADM_FAILED_DAEMON: 10 failed cephadm daemon(s) 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.g on smithi081 is in unknown state 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.a on smithi081 is in error state 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.k on smithi104 is in error state 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.e on smithi104 is in error state 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.h on smithi104 is in error state 2024-04-12T21:40:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.b on smithi104 is in error state 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.l on smithi153 is in error state 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.f on smithi153 is in error state 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.c on smithi153 is in error state 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: daemon mds.i on smithi153 is in error state 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: [WRN] FS_DEGRADED: 1 filesystem is degraded 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: fs cephfs is degraded 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:rejoin 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:40:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:40:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:01 smithi153 ceph-mon[36569]: pgmap v1205: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3 op/s 2024-04-12T21:40:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:01 smithi104 ceph-mon[36624]: pgmap v1205: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3 op/s 2024-04-12T21:40:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:01 smithi081 ceph-mon[31453]: pgmap v1205: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 3 op/s 2024-04-12T21:40:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:02 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:clientreplay 2024-04-12T21:40:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:02 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:40:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:02 smithi153 ceph-mon[36569]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:40:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:40:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:40:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:40:03.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:40:02 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:40:03.057 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:40:02 smithi081 systemd[1]: Stopped Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:40:02 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Consumed 29.497s CPU time. 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:40:02 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:40:02 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mds.d.smithi081.stdout:Apr 12 21:40:02 smithi081 systemd[1]: Failed to start Ceph mds.d for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:02 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:clientreplay 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:02 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:02 smithi081 ceph-mon[31453]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:40:03.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:40:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:02 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:clientreplay 2024-04-12T21:40:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:02 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:40:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:02 smithi104 ceph-mon[36624]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:40:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:40:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:40:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:40:03.409 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: *** Caught signal (Aborted) ** 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: in thread 7f7f13800700 thread_name:ms_dispatch 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7f2391cd20] 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2: gsignal() 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 3: abort() 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f02dad4268] 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f02db700fc] 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f02dea4f0a] 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 7: (MDLog::trim()+0xca9) [0x55f02dea60c9] 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f02dea65a8] 2024-04-12T21:40:03.410 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f7f22808b23] 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f7f239121ca] 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 11: clone() 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2024-04-12T21:40:03.056+0000 7f7f13800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: in thread 7f7f13800700 thread_name:ms_dispatch 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7f2391cd20] 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2: gsignal() 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 3: abort() 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f02dad4268] 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f02db700fc] 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f02dea4f0a] 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 7: (MDLog::trim()+0xca9) [0x55f02dea60c9] 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f02dea65a8] 2024-04-12T21:40:03.411 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f7f22808b23] 2024-04-12T21:40:03.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f7f239121ca] 2024-04-12T21:40:03.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 11: clone() 2024-04-12T21:40:03.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:40:03.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2024-04-12T21:40:03.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 0> 2024-04-12T21:40:03.056+0000 7f7f13800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:40:03.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: in thread 7f7f13800700 thread_name:ms_dispatch 2024-04-12T21:40:03.412 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7f2391cd20] 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2: gsignal() 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 3: abort() 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f02dad4268] 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f02db700fc] 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f02dea4f0a] 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 7: (MDLog::trim()+0xca9) [0x55f02dea60c9] 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f02dea65a8] 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f7f22808b23] 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f7f239121ca] 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 11: clone() 2024-04-12T21:40:03.413 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: -6> 2024-04-12T21:40:03.056+0000 7f7f13800700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: in thread 7f7f13800700 thread_name:ms_dispatch 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7f2391cd20] 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2: gsignal() 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 3: abort() 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55f02dad4268] 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55f02db700fc] 2024-04-12T21:40:03.414 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55f02dea4f0a] 2024-04-12T21:40:03.415 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 7: (MDLog::trim()+0xca9) [0x55f02dea60c9] 2024-04-12T21:40:03.415 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55f02dea65a8] 2024-04-12T21:40:03.415 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f7f22808b23] 2024-04-12T21:40:03.415 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f7f239121ca] 2024-04-12T21:40:03.415 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 11: clone() 2024-04-12T21:40:03.415 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:40:03.415 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[94994]: 2024-04-12T21:40:03.415 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:03 smithi081 podman[95211]: 2024-04-12 21:40:03.201545757 +0000 UTC m=+0.037415938 container died 6c53e4c6e316aa991f19a05c40095064280c86e816c678780da1037882595ac7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-12T21:40:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:03 smithi153 ceph-mon[36569]: pgmap v1206: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 103 B/s wr, 3 op/s 2024-04-12T21:40:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:03 smithi153 ceph-mon[36569]: pgmap v1207: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 129 B/s wr, 3 op/s 2024-04-12T21:40:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:03 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:40:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:03 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:active 2024-04-12T21:40:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:03 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:03 smithi104 ceph-mon[36624]: pgmap v1206: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 103 B/s wr, 3 op/s 2024-04-12T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:03 smithi104 ceph-mon[36624]: pgmap v1207: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 129 B/s wr, 3 op/s 2024-04-12T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:03 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:03 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:active 2024-04-12T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:03 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:40:04.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-mon[31453]: pgmap v1206: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 103 B/s wr, 3 op/s 2024-04-12T21:40:04.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-mon[31453]: pgmap v1207: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 129 B/s wr, 3 op/s 2024-04-12T21:40:04.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:40:04.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6836/1757171466,v1:172.21.15.81:6837/1757171466] up:active 2024-04-12T21:40:04.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:03 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:40:04.111 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:04 smithi081 podman[95211]: 2024-04-12 21:40:04.012180184 +0000 UTC m=+0.848050408 container remove 6c53e4c6e316aa991f19a05c40095064280c86e816c678780da1037882595ac7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240409, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-12T21:40:04.111 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:04 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:40:04.906 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:04 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-12T21:40:04.906 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:04 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 29.151s CPU time. 2024-04-12T21:40:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:05 smithi153 ceph-mon[36569]: pgmap v1208: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 387 B/s wr, 6 op/s 2024-04-12T21:40:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:05 smithi104 ceph-mon[36624]: pgmap v1208: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 387 B/s wr, 6 op/s 2024-04-12T21:40:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:05 smithi081 ceph-mon[31453]: pgmap v1208: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 387 B/s wr, 6 op/s 2024-04-12T21:40:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:07 smithi153 ceph-mon[36569]: pgmap v1209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 2.1 KiB/s wr, 14 op/s 2024-04-12T21:40:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:07 smithi104 ceph-mon[36624]: pgmap v1209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 2.1 KiB/s wr, 14 op/s 2024-04-12T21:40:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:07 smithi081 ceph-mon[31453]: pgmap v1209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 2.1 KiB/s wr, 14 op/s 2024-04-12T21:40:08.968 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:08.968+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:40:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:08.968+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:40:08.969 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:08.968+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:40:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:09 smithi153 ceph-mon[36569]: pgmap v1210: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:40:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:09 smithi104 ceph-mon[36624]: pgmap v1210: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:40:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:09 smithi081 ceph-mon[31453]: pgmap v1210: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:40:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:11 smithi104 ceph-mon[36624]: pgmap v1211: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:40:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:11 smithi081 ceph-mon[31453]: pgmap v1211: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:40:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:11 smithi153 ceph-mon[36569]: pgmap v1211: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 78 KiB/s wr, 16 op/s 2024-04-12T21:40:12.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:12.928+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:40:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:12.929+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:40:12.969 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:12.969+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:40:12.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:12.970+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:40:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:13 smithi104 ceph-mon[36624]: pgmap v1212: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 77 KiB/s wr, 15 op/s 2024-04-12T21:40:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:13 smithi081 ceph-mon[31453]: pgmap v1212: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 77 KiB/s wr, 15 op/s 2024-04-12T21:40:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:13 smithi153 ceph-mon[36569]: pgmap v1212: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 77 KiB/s wr, 15 op/s 2024-04-12T21:40:15.048 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:14 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-04-12T21:40:15.048 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:14 smithi081 systemd[1]: Stopped Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:40:15.048 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:14 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 29.151s CPU time. 2024-04-12T21:40:15.048 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:14 smithi081 systemd[1]: Starting Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968... 2024-04-12T21:40:15.396 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:15 smithi081 podman[95315]: 2024-04-12 21:40:14.952271742 +0000 UTC m=+0.032580167 image pull 8717720dc5e5c56303818ea4d4f0a11f6e2c03dee521fb668f411f1304e0afca quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63 2024-04-12T21:40:15.396 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:15 smithi081 podman[95315]: 2024-04-12 21:40:15.104216481 +0000 UTC m=+0.184524942 container create 52bdc45756816e4b9d3025db4c562471982df0230b41e54c3e8305f52c0c74c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, org.label-schema.build-date=20240409, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-12T21:40:15.396 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:15 smithi081 podman[95315]: 2024-04-12 21:40:15.393455103 +0000 UTC m=+0.473763571 container init 52bdc45756816e4b9d3025db4c562471982df0230b41e54c3e8305f52c0c74c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-12T21:40:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mon-a[31449]: 2024-04-12T21:40:15.459+0000 7f5209600700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:40:15.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:15 smithi081 podman[95315]: 2024-04-12 21:40:15.404499397 +0000 UTC m=+0.484807830 container start 52bdc45756816e4b9d3025db4c562471982df0230b41e54c3e8305f52c0c74c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-12T21:40:15.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mds[95334]: -- 172.21.15.81:0/178656653 <== mon.1 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b09bd7fa40 con 0x55b09bd52800 2024-04-12T21:40:15.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: starting mds.j at 2024-04-12T21:40:15.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:15 smithi081 bash[95315]: 52bdc45756816e4b9d3025db4c562471982df0230b41e54c3e8305f52c0c74c6 2024-04-12T21:40:15.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:15 smithi081 systemd[1]: Started Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:40:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: pgmap v1213: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 64 KiB/s wr, 12 op/s 2024-04-12T21:40:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: daemon mds.j restarted 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: osdmap e161: 12 total, 12 up, 12 in 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:boot 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: fsmap cephfs:0/1 1 up:standby, 1 failed 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:40:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:15 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:40:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: pgmap v1213: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 64 KiB/s wr, 12 op/s 2024-04-12T21:40:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: daemon mds.j restarted 2024-04-12T21:40:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:40:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:40:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: osdmap e161: 12 total, 12 up, 12 in 2024-04-12T21:40:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:boot 2024-04-12T21:40:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-04-12T21:40:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:40:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: fsmap cephfs:0/1 1 up:standby, 1 failed 2024-04-12T21:40:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:40:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:15 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:40:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: pgmap v1213: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 64 KiB/s wr, 12 op/s 2024-04-12T21:40:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: daemon mds.j restarted 2024-04-12T21:40:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-12T21:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-12T21:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: osdmap e161: 12 total, 12 up, 12 in 2024-04-12T21:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:boot 2024-04-12T21:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: Standby daemon mds.j assigned to filesystem cephfs as rank 0 2024-04-12T21:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-12T21:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: fsmap cephfs:0/1 1 up:standby, 1 failed 2024-04-12T21:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-12T21:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:15 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:replay} 2024-04-12T21:40:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:17 smithi104 ceph-mon[36624]: pgmap v1215: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 75 KiB/s wr, 5 op/s 2024-04-12T21:40:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:17 smithi153 ceph-mon[36569]: pgmap v1215: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 75 KiB/s wr, 5 op/s 2024-04-12T21:40:18.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:17 smithi081 ceph-mon[31453]: pgmap v1215: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 75 KiB/s wr, 5 op/s 2024-04-12T21:40:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:19 smithi104 ceph-mon[36624]: pgmap v1216: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1 op/s 2024-04-12T21:40:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:19 smithi081 ceph-mon[31453]: pgmap v1216: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1 op/s 2024-04-12T21:40:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:19 smithi153 ceph-mon[36569]: pgmap v1216: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1 op/s 2024-04-12T21:40:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:21 smithi081 ceph-mon[31453]: pgmap v1217: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3 op/s 2024-04-12T21:40:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:21 smithi153 ceph-mon[36569]: pgmap v1217: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3 op/s 2024-04-12T21:40:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:21 smithi104 ceph-mon[36624]: pgmap v1217: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3 op/s 2024-04-12T21:40:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:23.983+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:40:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:23.983+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:40:23.984 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:23.983+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:40:24.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:23 smithi104 ceph-mon[36624]: pgmap v1218: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:40:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:23 smithi081 ceph-mon[31453]: pgmap v1218: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:40:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:23 smithi153 ceph-mon[36569]: pgmap v1218: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 5 op/s 2024-04-12T21:40:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:25 smithi081 ceph-mon[31453]: pgmap v1219: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 6 op/s 2024-04-12T21:40:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:25 smithi153 ceph-mon[36569]: pgmap v1219: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 6 op/s 2024-04-12T21:40:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:25 smithi104 ceph-mon[36624]: pgmap v1219: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 MiB/s rd, 6 op/s 2024-04-12T21:40:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:27 smithi081 ceph-mon[31453]: pgmap v1220: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 7 op/s 2024-04-12T21:40:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:27 smithi153 ceph-mon[36569]: pgmap v1220: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 7 op/s 2024-04-12T21:40:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:27 smithi104 ceph-mon[36624]: pgmap v1220: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 MiB/s rd, 7 op/s 2024-04-12T21:40:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:28 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:reconnect 2024-04-12T21:40:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:28 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:40:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:28 smithi081 ceph-mon[31453]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:40:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:28 smithi081 ceph-mon[31453]: pgmap v1221: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 6 op/s 2024-04-12T21:40:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:28 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:reconnect 2024-04-12T21:40:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:28 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:40:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:28 smithi153 ceph-mon[36569]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:40:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:28 smithi153 ceph-mon[36569]: pgmap v1221: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 6 op/s 2024-04-12T21:40:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:28 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:reconnect 2024-04-12T21:40:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:28 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:reconnect} 2024-04-12T21:40:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:28 smithi104 ceph-mon[36624]: reconnect by client.25316 v1:172.21.15.81:0/3123326716 after 0.0159997 2024-04-12T21:40:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:28 smithi104 ceph-mon[36624]: pgmap v1221: 129 pgs: 129 active+clean; 516 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 MiB/s rd, 6 op/s 2024-04-12T21:40:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:29 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:rejoin 2024-04-12T21:40:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:29 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:40:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:29 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:rejoin 2024-04-12T21:40:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:29 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:40:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:29 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:rejoin 2024-04-12T21:40:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:29 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:rejoin} 2024-04-12T21:40:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:30 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:clientreplay 2024-04-12T21:40:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:30 smithi081 ceph-mon[31453]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:40:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:30 smithi081 ceph-mon[31453]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:40:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:30 smithi081 ceph-mon[31453]: pgmap v1222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 170 B/s wr, 11 op/s 2024-04-12T21:40:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:30 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:clientreplay 2024-04-12T21:40:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:30 smithi153 ceph-mon[36569]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:40:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:30 smithi153 ceph-mon[36569]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:40:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:30 smithi153 ceph-mon[36569]: pgmap v1222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 170 B/s wr, 11 op/s 2024-04-12T21:40:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:30 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:clientreplay 2024-04-12T21:40:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:30 smithi104 ceph-mon[36624]: fsmap cephfs:1/1 {0=j=up:clientreplay} 2024-04-12T21:40:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:30 smithi104 ceph-mon[36624]: daemon mds.j is now active in filesystem cephfs as rank 0 2024-04-12T21:40:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:30 smithi104 ceph-mon[36624]: pgmap v1222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 24 MiB/s rd, 170 B/s wr, 11 op/s 2024-04-12T21:40:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-mon[31453]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:40:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-mon[31453]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:active 2024-04-12T21:40:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1145: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::back() [with _Tp = CInodeCommitOperations; _Alloc = std::allocator; std::vector<_Tp, _Alloc>::reference = CInodeCommitOperations&]: Assertion '!this->empty()' failed. 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: *** Caught signal (Aborted) ** 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: in thread 7f9aa0400700 thread_name:ms_dispatch 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9ab0574d20] 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2: gsignal() 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 3: abort() 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55b0865c0268] 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55b08665c0fc] 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55b086990f0a] 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 7: (MDLog::trim()+0xca9) [0x55b0869920c9] 2024-04-12T21:40:32.158 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55b0869925a8] 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f9aaf460b23] 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f9ab056a1ca] 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 11: clone() 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2024-04-12T21:40:31.809+0000 7f9aa0400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: in thread 7f9aa0400700 thread_name:ms_dispatch 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9ab0574d20] 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2: gsignal() 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 3: abort() 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55b0865c0268] 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55b08665c0fc] 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55b086990f0a] 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 7: (MDLog::trim()+0xca9) [0x55b0869920c9] 2024-04-12T21:40:32.159 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55b0869925a8] 2024-04-12T21:40:32.160 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f9aaf460b23] 2024-04-12T21:40:32.160 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f9ab056a1ca] 2024-04-12T21:40:32.160 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 11: clone() 2024-04-12T21:40:32.160 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:40:32.160 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2024-04-12T21:40:32.160 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 0> 2024-04-12T21:40:31.809+0000 7f9aa0400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: in thread 7f9aa0400700 thread_name:ms_dispatch 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9ab0574d20] 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2: gsignal() 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 3: abort() 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55b0865c0268] 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55b08665c0fc] 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55b086990f0a] 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 7: (MDLog::trim()+0xca9) [0x55b0869920c9] 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55b0869925a8] 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f9aaf460b23] 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f9ab056a1ca] 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 11: clone() 2024-04-12T21:40:32.161 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: -13> 2024-04-12T21:40:31.809+0000 7f9aa0400700 -1 *** Caught signal (Aborted) ** 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: in thread 7f9aa0400700 thread_name:ms_dispatch 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: ceph version 19.0.0-2879-g324baf8d (324baf8ddb91dea46cac50cb535c306115791355) squid (dev) 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9ab0574d20] 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2: gsignal() 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 3: abort() 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 4: /usr/bin/ceph-mds(+0x1d9268) [0x55b0865c0268] 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 5: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2bdc) [0x55b08665c0fc] 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 6: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x55b086990f0a] 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 7: (MDLog::trim()+0xca9) [0x55b0869920c9] 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 8: (MDLog::log_trim_upkeep()+0x358) [0x55b0869925a8] 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 9: /lib64/libstdc++.so.6(+0xc2b23) [0x7f9aaf460b23] 2024-04-12T21:40:32.162 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 10: /lib64/libpthread.so.0(+0x81ca) [0x7f9ab056a1ca] 2024-04-12T21:40:32.163 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 11: clone() 2024-04-12T21:40:32.163 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-12T21:40:32.163 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:31 smithi081 ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j[95330]: 2024-04-12T21:40:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:31 smithi153 ceph-mon[36569]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:40:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:31 smithi153 ceph-mon[36569]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:active 2024-04-12T21:40:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:31 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:40:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:31 smithi104 ceph-mon[36624]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-12T21:40:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:31 smithi104 ceph-mon[36624]: mds.? [v2:172.21.15.81:6834/11538883,v1:172.21.15.81:6835/11538883] up:active 2024-04-12T21:40:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:31 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active} 2024-04-12T21:40:32.897 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:32 smithi081 podman[95369]: 2024-04-12 21:40:32.636023897 +0000 UTC m=+0.663272376 container died 52bdc45756816e4b9d3025db4c562471982df0230b41e54c3e8305f52c0c74c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240409, ceph=True, RELEASE=wip-vshankar-testing-20240411.061452-324baf8) 2024-04-12T21:40:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:32 smithi081 ceph-mon[31453]: pgmap v1223: 129 pgs: 129 active+clean; 515 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 48 KiB/s wr, 21 op/s 2024-04-12T21:40:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:32 smithi153 ceph-mon[36569]: pgmap v1223: 129 pgs: 129 active+clean; 515 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 48 KiB/s wr, 21 op/s 2024-04-12T21:40:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:32 smithi104 ceph-mon[36624]: pgmap v1223: 129 pgs: 129 active+clean; 515 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 MiB/s rd, 48 KiB/s wr, 21 op/s 2024-04-12T21:40:33.906 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:33 smithi081 podman[95369]: 2024-04-12 21:40:33.473275584 +0000 UTC m=+1.500524098 container remove 52bdc45756816e4b9d3025db4c562471982df0230b41e54c3e8305f52c0c74c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f4452def3be4d350f4a8e5feaf7ebb1511be9298fe68478ba7a2d6fbd7d69a63, name=ceph-11b8cc88-f910-11ee-bc8d-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240411.061452-324baf8, org.label-schema.build-date=20240409, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-12T21:40:33.906 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:33 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-04-12T21:40:34.406 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:33 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-12T21:40:34.406 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:33 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 27.425s CPU time. 2024-04-12T21:40:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:35 smithi081 ceph-mon[31453]: pgmap v1224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 48 KiB/s wr, 24 op/s 2024-04-12T21:40:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:35 smithi153 ceph-mon[36569]: pgmap v1224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 48 KiB/s wr, 24 op/s 2024-04-12T21:40:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:35 smithi104 ceph-mon[36624]: pgmap v1224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 48 KiB/s wr, 24 op/s 2024-04-12T21:40:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:37 smithi081 ceph-mon[31453]: pgmap v1225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 50 KiB/s wr, 25 op/s 2024-04-12T21:40:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:37 smithi153 ceph-mon[36569]: pgmap v1225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 50 KiB/s wr, 25 op/s 2024-04-12T21:40:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:37 smithi104 ceph-mon[36624]: pgmap v1225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 50 KiB/s wr, 25 op/s 2024-04-12T21:40:38.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:38.996+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:40:38.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:38.996+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:40:38.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:38.996+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:40:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:39 smithi081 ceph-mon[31453]: pgmap v1226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 90 KiB/s wr, 23 op/s 2024-04-12T21:40:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:39 smithi153 ceph-mon[36569]: pgmap v1226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 90 KiB/s wr, 23 op/s 2024-04-12T21:40:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:39 smithi104 ceph-mon[36624]: pgmap v1226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 90 KiB/s wr, 23 op/s 2024-04-12T21:40:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:41 smithi081 ceph-mon[31453]: pgmap v1227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 90 KiB/s wr, 23 op/s 2024-04-12T21:40:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:41 smithi153 ceph-mon[36569]: pgmap v1227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 90 KiB/s wr, 23 op/s 2024-04-12T21:40:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:41 smithi104 ceph-mon[36624]: pgmap v1227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 90 KiB/s wr, 23 op/s 2024-04-12T21:40:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:43 smithi081 ceph-mon[31453]: pgmap v1228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 90 KiB/s wr, 18 op/s 2024-04-12T21:40:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:43 smithi153 ceph-mon[36569]: pgmap v1228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 90 KiB/s wr, 18 op/s 2024-04-12T21:40:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:43 smithi104 ceph-mon[36624]: pgmap v1228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 90 KiB/s wr, 18 op/s 2024-04-12T21:40:44.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:44 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-04-12T21:40:44.656 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:44 smithi081 systemd[1]: Stopped Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:40:44.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:44 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Consumed 27.425s CPU time. 2024-04-12T21:40:44.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:44 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Start request repeated too quickly. 2024-04-12T21:40:44.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:44 smithi081 systemd[1]: ceph-11b8cc88-f910-11ee-bc8d-c7b262605968@mds.j.service: Failed with result 'exit-code'. 2024-04-12T21:40:44.657 INFO:journalctl@ceph.mds.j.smithi081.stdout:Apr 12 21:40:44 smithi081 systemd[1]: Failed to start Ceph mds.j for 11b8cc88-f910-11ee-bc8d-c7b262605968. 2024-04-12T21:40:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:45 smithi153 ceph-mon[36569]: pgmap v1229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:40:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:45 smithi104 ceph-mon[36624]: pgmap v1229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:40:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:45 smithi081 ceph-mon[31453]: pgmap v1229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 41 KiB/s wr, 5 op/s 2024-04-12T21:40:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:47 smithi153 ceph-mon[36569]: pgmap v1230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 41 KiB/s wr, 1 op/s 2024-04-12T21:40:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:47 smithi104 ceph-mon[36624]: pgmap v1230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 41 KiB/s wr, 1 op/s 2024-04-12T21:40:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:47 smithi081 ceph-mon[31453]: pgmap v1230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 41 KiB/s wr, 1 op/s 2024-04-12T21:40:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:48 smithi153 ceph-mon[36569]: fsmap cephfs:1 {0=j=up:active(laggy or crashed)} 2024-04-12T21:40:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:48 smithi104 ceph-mon[36624]: fsmap cephfs:1 {0=j=up:active(laggy or crashed)} 2024-04-12T21:40:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:48 smithi081 ceph-mon[31453]: fsmap cephfs:1 {0=j=up:active(laggy or crashed)} 2024-04-12T21:40:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:49 smithi153 ceph-mon[36569]: pgmap v1231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s wr, 0 op/s 2024-04-12T21:40:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:49 smithi104 ceph-mon[36624]: pgmap v1231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s wr, 0 op/s 2024-04-12T21:40:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:49 smithi081 ceph-mon[31453]: pgmap v1231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s wr, 0 op/s 2024-04-12T21:40:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:51 smithi153 ceph-mon[36569]: pgmap v1232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:51 smithi104 ceph-mon[36624]: pgmap v1232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:51 smithi081 ceph-mon[31453]: pgmap v1232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:54.008+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:40:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:54.008+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:40:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:54.008+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:40:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:53 smithi153 ceph-mon[36569]: pgmap v1233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:53 smithi104 ceph-mon[36624]: pgmap v1233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:53 smithi081 ceph-mon[31453]: pgmap v1233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:55 smithi153 ceph-mon[36569]: pgmap v1234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:55 smithi104 ceph-mon[36624]: pgmap v1234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:55 smithi081 ceph-mon[31453]: pgmap v1234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:57.934+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:40:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:57.935+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:40:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:57.976+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:40:57.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:40:57.977+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:40:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:57 smithi153 ceph-mon[36569]: pgmap v1235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:57 smithi104 ceph-mon[36624]: pgmap v1235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:40:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:57 smithi081 ceph-mon[31453]: pgmap v1235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:40:59 smithi153 ceph-mon[36569]: pgmap v1236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:40:59 smithi104 ceph-mon[36624]: pgmap v1236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:40:59 smithi081 ceph-mon[31453]: pgmap v1236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:01 smithi104 ceph-mon[36624]: pgmap v1237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:01 smithi081 ceph-mon[31453]: pgmap v1237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:01 smithi153 ceph-mon[36569]: pgmap v1237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:41:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:41:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:41:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:03 smithi104 ceph-mon[36624]: pgmap v1238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:03 smithi081 ceph-mon[31453]: pgmap v1238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:03 smithi153 ceph-mon[36569]: pgmap v1238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:41:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:41:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:41:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:41:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:41:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:41:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:41:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:41:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:41:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:05 smithi081 ceph-mon[31453]: pgmap v1239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:05 smithi081 ceph-mon[31453]: pgmap v1240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:05 smithi153 ceph-mon[36569]: pgmap v1239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:05 smithi153 ceph-mon[36569]: pgmap v1240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:05 smithi104 ceph-mon[36624]: pgmap v1239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:05 smithi104 ceph-mon[36624]: pgmap v1240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:07 smithi081 ceph-mon[31453]: pgmap v1241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:07 smithi153 ceph-mon[36569]: pgmap v1241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:07 smithi104 ceph-mon[36624]: pgmap v1241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:09.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:09.015+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:41:09.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:09.016+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:41:09.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:09.016+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:41:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:09 smithi081 ceph-mon[31453]: pgmap v1242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:09 smithi153 ceph-mon[36569]: pgmap v1242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:09 smithi104 ceph-mon[36624]: pgmap v1242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:11 smithi081 ceph-mon[31453]: pgmap v1243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:11 smithi153 ceph-mon[36569]: pgmap v1243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:11 smithi104 ceph-mon[36624]: pgmap v1243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:13 smithi081 ceph-mon[31453]: pgmap v1244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:13 smithi153 ceph-mon[36569]: pgmap v1244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:13 smithi104 ceph-mon[36624]: pgmap v1244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:15 smithi081 ceph-mon[31453]: pgmap v1245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:15 smithi153 ceph-mon[36569]: pgmap v1245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:15 smithi104 ceph-mon[36624]: pgmap v1245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:16 smithi081 ceph-mon[31453]: pgmap v1246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:16 smithi153 ceph-mon[36569]: pgmap v1246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:16 smithi104 ceph-mon[36624]: pgmap v1246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:19 smithi153 ceph-mon[36569]: pgmap v1247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:19 smithi104 ceph-mon[36624]: pgmap v1247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:19 smithi081 ceph-mon[31453]: pgmap v1247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:21 smithi153 ceph-mon[36569]: pgmap v1248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:21 smithi104 ceph-mon[36624]: pgmap v1248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:21 smithi081 ceph-mon[31453]: pgmap v1248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:23 smithi153 ceph-mon[36569]: pgmap v1249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:23 smithi104 ceph-mon[36624]: pgmap v1249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:23 smithi081 ceph-mon[31453]: pgmap v1249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:24.031+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:41:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:24.031+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:41:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:24.031+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:41:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:25 smithi153 ceph-mon[36569]: pgmap v1250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:25 smithi104 ceph-mon[36624]: pgmap v1250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:25 smithi081 ceph-mon[31453]: pgmap v1250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:27 smithi153 ceph-mon[36569]: pgmap v1251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:27 smithi104 ceph-mon[36624]: pgmap v1251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:27 smithi081 ceph-mon[31453]: pgmap v1251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:29 smithi153 ceph-mon[36569]: pgmap v1252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:29 smithi104 ceph-mon[36624]: pgmap v1252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:29 smithi081 ceph-mon[31453]: pgmap v1252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:31 smithi153 ceph-mon[36569]: pgmap v1253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:31 smithi104 ceph-mon[36624]: pgmap v1253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:31 smithi081 ceph-mon[31453]: pgmap v1253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:33 smithi153 ceph-mon[36569]: pgmap v1254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:33 smithi104 ceph-mon[36624]: pgmap v1254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:33 smithi081 ceph-mon[31453]: pgmap v1254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:35 smithi153 ceph-mon[36569]: pgmap v1255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:35 smithi104 ceph-mon[36624]: pgmap v1255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:35 smithi081 ceph-mon[31453]: pgmap v1255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:37 smithi153 ceph-mon[36569]: pgmap v1256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:37 smithi104 ceph-mon[36624]: pgmap v1256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:37 smithi081 ceph-mon[31453]: pgmap v1256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:39.046 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:39.046+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:41:39.047 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:39.046+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:41:39.047 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:39.046+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:41:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:39 smithi104 ceph-mon[36624]: pgmap v1257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:39 smithi081 ceph-mon[31453]: pgmap v1257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:39 smithi153 ceph-mon[36569]: pgmap v1257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:41 smithi104 ceph-mon[36624]: pgmap v1258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:41 smithi081 ceph-mon[31453]: pgmap v1258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:41 smithi153 ceph-mon[36569]: pgmap v1258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:43 smithi104 ceph-mon[36624]: pgmap v1259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:43 smithi081 ceph-mon[31453]: pgmap v1259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:43 smithi153 ceph-mon[36569]: pgmap v1259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:45 smithi104 ceph-mon[36624]: pgmap v1260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:45 smithi081 ceph-mon[31453]: pgmap v1260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:45 smithi153 ceph-mon[36569]: pgmap v1260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:47 smithi104 ceph-mon[36624]: pgmap v1261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:47 smithi081 ceph-mon[31453]: pgmap v1261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:47 smithi153 ceph-mon[36569]: pgmap v1261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:49.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:49 smithi153 ceph-mon[36569]: pgmap v1262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:49 smithi081 ceph-mon[31453]: pgmap v1262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:49 smithi104 ceph-mon[36624]: pgmap v1262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:51 smithi081 ceph-mon[31453]: pgmap v1263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:51 smithi153 ceph-mon[36569]: pgmap v1263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:51 smithi104 ceph-mon[36624]: pgmap v1263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:53 smithi081 ceph-mon[31453]: pgmap v1264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:53 smithi153 ceph-mon[36569]: pgmap v1264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:53 smithi104 ceph-mon[36624]: pgmap v1264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:54.054+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:41:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:54.054+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:41:54.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:41:54.054+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:41:55.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:55 smithi104 ceph-mon[36624]: pgmap v1265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:55 smithi081 ceph-mon[31453]: pgmap v1265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:55 smithi153 ceph-mon[36569]: pgmap v1265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:57 smithi081 ceph-mon[31453]: pgmap v1266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:57 smithi153 ceph-mon[36569]: pgmap v1266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:57 smithi104 ceph-mon[36624]: pgmap v1266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:41:59 smithi081 ceph-mon[31453]: pgmap v1267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:41:59 smithi153 ceph-mon[36569]: pgmap v1267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:41:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:41:59 smithi104 ceph-mon[36624]: pgmap v1267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:01 smithi081 ceph-mon[31453]: pgmap v1268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:01 smithi153 ceph-mon[36569]: pgmap v1268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:01 smithi104 ceph-mon[36624]: pgmap v1268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:03 smithi081 ceph-mon[31453]: pgmap v1269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:03 smithi153 ceph-mon[36569]: pgmap v1269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:03 smithi104 ceph-mon[36624]: pgmap v1269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:05 smithi081 ceph-mon[31453]: pgmap v1270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:42:05.552 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:05 smithi104 ceph-mon[36624]: pgmap v1270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:05.553 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:42:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:05 smithi153 ceph-mon[36569]: pgmap v1270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:42:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:07 smithi081 ceph-mon[31453]: pgmap v1271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:07.496 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:07 smithi104 ceph-mon[36624]: pgmap v1271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:07 smithi153 ceph-mon[36569]: pgmap v1271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:09.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:09.070+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:42:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:09.070+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:42:09.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:09.070+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:42:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:09 smithi081 ceph-mon[31453]: pgmap v1272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:09 smithi153 ceph-mon[36569]: pgmap v1272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:09 smithi104 ceph-mon[36624]: pgmap v1272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:11 smithi081 ceph-mon[31453]: pgmap v1273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:42:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:42:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:11 smithi153 ceph-mon[36569]: pgmap v1273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:42:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:42:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:11 smithi104 ceph-mon[36624]: pgmap v1273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:42:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:42:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:42:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:12 smithi081 ceph-mon[31453]: pgmap v1274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:12 smithi081 ceph-mon[31453]: pgmap v1275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:12 smithi153 ceph-mon[36569]: pgmap v1274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:12 smithi153 ceph-mon[36569]: pgmap v1275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:12 smithi104 ceph-mon[36624]: pgmap v1274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:12 smithi104 ceph-mon[36624]: pgmap v1275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:14 smithi081 ceph-mon[31453]: pgmap v1276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:14 smithi153 ceph-mon[36569]: pgmap v1276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:14 smithi104 ceph-mon[36624]: pgmap v1276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:16 smithi081 ceph-mon[31453]: pgmap v1277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:16 smithi153 ceph-mon[36569]: pgmap v1277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:16 smithi104 ceph-mon[36624]: pgmap v1277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:18 smithi104 ceph-mon[36624]: pgmap v1278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:19.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:18 smithi153 ceph-mon[36569]: pgmap v1278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:18 smithi081 ceph-mon[31453]: pgmap v1278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:20 smithi081 ceph-mon[31453]: pgmap v1279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:20 smithi153 ceph-mon[36569]: pgmap v1279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:20 smithi104 ceph-mon[36624]: pgmap v1279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:22 smithi081 ceph-mon[31453]: pgmap v1280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:22 smithi153 ceph-mon[36569]: pgmap v1280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:22 smithi104 ceph-mon[36624]: pgmap v1280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:24.083+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:42:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:24.083+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:42:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:24.083+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:42:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:23 smithi153 ceph-mon[36569]: pgmap v1281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:23 smithi104 ceph-mon[36624]: pgmap v1281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:23 smithi081 ceph-mon[31453]: pgmap v1281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:26 smithi104 ceph-mon[36624]: pgmap v1282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:26 smithi081 ceph-mon[31453]: pgmap v1282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:26 smithi153 ceph-mon[36569]: pgmap v1282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:28 smithi081 ceph-mon[31453]: pgmap v1283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:28 smithi153 ceph-mon[36569]: pgmap v1283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:28 smithi104 ceph-mon[36624]: pgmap v1283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:30 smithi081 ceph-mon[31453]: pgmap v1284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:30 smithi153 ceph-mon[36569]: pgmap v1284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:30 smithi104 ceph-mon[36624]: pgmap v1284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:32 smithi081 ceph-mon[31453]: pgmap v1285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:32 smithi153 ceph-mon[36569]: pgmap v1285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:32 smithi104 ceph-mon[36624]: pgmap v1285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:34 smithi081 ceph-mon[31453]: pgmap v1286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:34 smithi153 ceph-mon[36569]: pgmap v1286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:34 smithi104 ceph-mon[36624]: pgmap v1286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:36 smithi081 ceph-mon[31453]: pgmap v1287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:36 smithi153 ceph-mon[36569]: pgmap v1287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:36 smithi104 ceph-mon[36624]: pgmap v1287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:38 smithi081 ceph-mon[31453]: pgmap v1288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:38 smithi153 ceph-mon[36569]: pgmap v1288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:38 smithi104 ceph-mon[36624]: pgmap v1288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:39.095+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:42:39.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:39.095+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:42:39.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:39.095+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:42:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:40 smithi081 ceph-mon[31453]: pgmap v1289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:40 smithi153 ceph-mon[36569]: pgmap v1289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:40 smithi104 ceph-mon[36624]: pgmap v1289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:42 smithi081 ceph-mon[31453]: pgmap v1290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:42 smithi153 ceph-mon[36569]: pgmap v1290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:42 smithi104 ceph-mon[36624]: pgmap v1290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:44 smithi081 ceph-mon[31453]: pgmap v1291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:44 smithi153 ceph-mon[36569]: pgmap v1291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:44 smithi104 ceph-mon[36624]: pgmap v1291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:46 smithi081 ceph-mon[31453]: pgmap v1292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:46 smithi153 ceph-mon[36569]: pgmap v1292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:46 smithi104 ceph-mon[36624]: pgmap v1292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:48 smithi081 ceph-mon[31453]: pgmap v1293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:48 smithi153 ceph-mon[36569]: pgmap v1293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:48 smithi104 ceph-mon[36624]: pgmap v1293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:50 smithi081 ceph-mon[31453]: pgmap v1294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:50 smithi153 ceph-mon[36569]: pgmap v1294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:50 smithi104 ceph-mon[36624]: pgmap v1294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:52 smithi081 ceph-mon[31453]: pgmap v1295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:52 smithi153 ceph-mon[36569]: pgmap v1295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:52 smithi104 ceph-mon[36624]: pgmap v1295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:54.103+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:42:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:54.103+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:42:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:42:54.103+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:42:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:54 smithi081 ceph-mon[31453]: pgmap v1296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:54 smithi153 ceph-mon[36569]: pgmap v1296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:54 smithi104 ceph-mon[36624]: pgmap v1296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:56 smithi081 ceph-mon[31453]: pgmap v1297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:56 smithi153 ceph-mon[36569]: pgmap v1297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:56 smithi104 ceph-mon[36624]: pgmap v1297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:42:58 smithi153 ceph-mon[36569]: pgmap v1298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:42:58 smithi104 ceph-mon[36624]: pgmap v1298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:42:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:42:58 smithi081 ceph-mon[31453]: pgmap v1298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:00 smithi153 ceph-mon[36569]: pgmap v1299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:00 smithi104 ceph-mon[36624]: pgmap v1299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:00 smithi081 ceph-mon[31453]: pgmap v1299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:02 smithi153 ceph-mon[36569]: pgmap v1300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:02 smithi104 ceph-mon[36624]: pgmap v1300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:02 smithi081 ceph-mon[31453]: pgmap v1300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:04 smithi153 ceph-mon[36569]: pgmap v1301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:04 smithi104 ceph-mon[36624]: pgmap v1301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:04 smithi081 ceph-mon[31453]: pgmap v1301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:06.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:06 smithi081 ceph-mon[31453]: pgmap v1302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:06 smithi153 ceph-mon[36569]: pgmap v1302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:06 smithi104 ceph-mon[36624]: pgmap v1302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:08 smithi153 ceph-mon[36569]: pgmap v1303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:08 smithi104 ceph-mon[36624]: pgmap v1303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:09.117+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:43:09.117 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:09.117+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:43:09.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:09.117+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:43:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:08 smithi081 ceph-mon[31453]: pgmap v1303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:10 smithi153 ceph-mon[36569]: pgmap v1304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:10 smithi104 ceph-mon[36624]: pgmap v1304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:10 smithi081 ceph-mon[31453]: pgmap v1304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:12.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:12.956+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:43:12.958 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:12.957+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:43:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:12.997+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:43:12.998 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:12.997+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:43:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:12 smithi153 ceph-mon[36569]: pgmap v1305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:43:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:12 smithi104 ceph-mon[36624]: pgmap v1305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:43:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:12 smithi081 ceph-mon[31453]: pgmap v1305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:43:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:13 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:43:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:13 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:43:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:13 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:43:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:13 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:43:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:13 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:43:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:13 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:43:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:14 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:43:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:14 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:43:14.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:14 smithi153 ceph-mon[36569]: pgmap v1306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:14 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:43:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:14 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:43:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:14 smithi104 ceph-mon[36624]: pgmap v1306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:14 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:43:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:14 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:43:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:14 smithi081 ceph-mon[31453]: pgmap v1306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:17.020 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:16 smithi153 ceph-mon[36569]: pgmap v1307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:16 smithi104 ceph-mon[36624]: pgmap v1307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:16 smithi081 ceph-mon[31453]: pgmap v1307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:43:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:43:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:43:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:43:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: pgmap v1308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: pgmap v1309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:43:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: pgmap v1308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: pgmap v1309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:43:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:43:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:43:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:43:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:43:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: pgmap v1308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: pgmap v1309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:43:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:19 smithi081 ceph-mon[31453]: pgmap v1310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:19 smithi153 ceph-mon[36569]: pgmap v1310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:19 smithi104 ceph-mon[36624]: pgmap v1310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:22.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:21 smithi153 ceph-mon[36569]: pgmap v1311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:21 smithi081 ceph-mon[31453]: pgmap v1311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:21 smithi104 ceph-mon[36624]: pgmap v1311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:24.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:24.121+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:43:24.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:24.121+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:43:24.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:24.121+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:43:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:23 smithi081 ceph-mon[31453]: pgmap v1312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:23 smithi153 ceph-mon[36569]: pgmap v1312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:23 smithi104 ceph-mon[36624]: pgmap v1312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:26.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:25 smithi104 ceph-mon[36624]: pgmap v1313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:25 smithi081 ceph-mon[31453]: pgmap v1313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:25 smithi153 ceph-mon[36569]: pgmap v1313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:27 smithi081 ceph-mon[31453]: pgmap v1314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:27 smithi153 ceph-mon[36569]: pgmap v1314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:27 smithi104 ceph-mon[36624]: pgmap v1314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:29 smithi081 ceph-mon[31453]: pgmap v1315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:29 smithi153 ceph-mon[36569]: pgmap v1315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:29 smithi104 ceph-mon[36624]: pgmap v1315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:31 smithi081 ceph-mon[31453]: pgmap v1316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:31 smithi153 ceph-mon[36569]: pgmap v1316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:31 smithi104 ceph-mon[36624]: pgmap v1316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:32 smithi153 ceph-mon[36569]: pgmap v1317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:32 smithi104 ceph-mon[36624]: pgmap v1317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:32 smithi081 ceph-mon[31453]: pgmap v1317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:36.113 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:35 smithi081 ceph-mon[31453]: pgmap v1318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:35 smithi153 ceph-mon[36569]: pgmap v1318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:35 smithi104 ceph-mon[36624]: pgmap v1318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:37 smithi081 ceph-mon[31453]: pgmap v1319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:37 smithi153 ceph-mon[36569]: pgmap v1319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:37 smithi104 ceph-mon[36624]: pgmap v1319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:39.127+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:43:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:39.127+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:43:39.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:39.127+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:43:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:39 smithi081 ceph-mon[31453]: pgmap v1320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:39 smithi153 ceph-mon[36569]: pgmap v1320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:39 smithi104 ceph-mon[36624]: pgmap v1320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:41 smithi081 ceph-mon[31453]: pgmap v1321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:41 smithi153 ceph-mon[36569]: pgmap v1321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:41 smithi104 ceph-mon[36624]: pgmap v1321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:43 smithi081 ceph-mon[31453]: pgmap v1322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:43 smithi153 ceph-mon[36569]: pgmap v1322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:43 smithi104 ceph-mon[36624]: pgmap v1322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:45 smithi081 ceph-mon[31453]: pgmap v1323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:45 smithi153 ceph-mon[36569]: pgmap v1323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:45 smithi104 ceph-mon[36624]: pgmap v1323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:46 smithi153 ceph-mon[36569]: pgmap v1324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:46 smithi104 ceph-mon[36624]: pgmap v1324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:46 smithi081 ceph-mon[31453]: pgmap v1324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:49 smithi081 ceph-mon[31453]: pgmap v1325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:49 smithi153 ceph-mon[36569]: pgmap v1325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:49 smithi104 ceph-mon[36624]: pgmap v1325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:51 smithi081 ceph-mon[31453]: pgmap v1326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:51 smithi153 ceph-mon[36569]: pgmap v1326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:51 smithi104 ceph-mon[36624]: pgmap v1326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:54.141+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:43:54.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:54.141+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:43:54.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:43:54.141+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:43:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:53 smithi081 ceph-mon[31453]: pgmap v1327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:53 smithi153 ceph-mon[36569]: pgmap v1327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:53 smithi104 ceph-mon[36624]: pgmap v1327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:54 smithi153 ceph-mon[36569]: pgmap v1328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:54 smithi104 ceph-mon[36624]: pgmap v1328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:54 smithi081 ceph-mon[31453]: pgmap v1328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:57 smithi081 ceph-mon[31453]: pgmap v1329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:57 smithi153 ceph-mon[36569]: pgmap v1329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:43:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:57 smithi104 ceph-mon[36624]: pgmap v1329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:43:59 smithi081 ceph-mon[31453]: pgmap v1330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:43:59 smithi153 ceph-mon[36569]: pgmap v1330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:43:59 smithi104 ceph-mon[36624]: pgmap v1330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:01 smithi081 ceph-mon[31453]: pgmap v1331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:01 smithi153 ceph-mon[36569]: pgmap v1331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:01 smithi104 ceph-mon[36624]: pgmap v1331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:03 smithi081 ceph-mon[31453]: pgmap v1332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:03 smithi153 ceph-mon[36569]: pgmap v1332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:03 smithi104 ceph-mon[36624]: pgmap v1332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:04 smithi153 ceph-mon[36569]: pgmap v1333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:04 smithi104 ceph-mon[36624]: pgmap v1333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:04 smithi081 ceph-mon[31453]: pgmap v1333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:07 smithi081 ceph-mon[31453]: pgmap v1334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:07 smithi153 ceph-mon[36569]: pgmap v1334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:07 smithi104 ceph-mon[36624]: pgmap v1334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:09.154+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:44:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:09.154+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:44:09.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:09.154+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:44:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:09 smithi081 ceph-mon[31453]: pgmap v1335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:09 smithi153 ceph-mon[36569]: pgmap v1335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:09 smithi104 ceph-mon[36624]: pgmap v1335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:11 smithi081 ceph-mon[31453]: pgmap v1336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:11 smithi153 ceph-mon[36569]: pgmap v1336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:11 smithi104 ceph-mon[36624]: pgmap v1336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:13 smithi081 ceph-mon[31453]: pgmap v1337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:13 smithi153 ceph-mon[36569]: pgmap v1337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:13 smithi104 ceph-mon[36624]: pgmap v1337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:14 smithi153 ceph-mon[36569]: pgmap v1338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:14 smithi104 ceph-mon[36624]: pgmap v1338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:14 smithi081 ceph-mon[31453]: pgmap v1338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:17 smithi081 ceph-mon[31453]: pgmap v1339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:44:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:17 smithi153 ceph-mon[36569]: pgmap v1339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:44:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:17 smithi104 ceph-mon[36624]: pgmap v1339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:44:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:44:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:44:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:44:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:44:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:44:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:44:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:44:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:44:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:44:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:19 smithi081 ceph-mon[31453]: pgmap v1340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:19 smithi081 ceph-mon[31453]: pgmap v1341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:19 smithi153 ceph-mon[36569]: pgmap v1340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:19 smithi153 ceph-mon[36569]: pgmap v1341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:19 smithi104 ceph-mon[36624]: pgmap v1340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:19 smithi104 ceph-mon[36624]: pgmap v1341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:20 smithi153 ceph-mon[36569]: pgmap v1342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:20 smithi104 ceph-mon[36624]: pgmap v1342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:20 smithi081 ceph-mon[31453]: pgmap v1342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:23 smithi153 ceph-mon[36569]: pgmap v1343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:23 smithi104 ceph-mon[36624]: pgmap v1343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:23 smithi081 ceph-mon[31453]: pgmap v1343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:24.168+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:44:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:24.168+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:44:24.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:24.168+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:44:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:25 smithi153 ceph-mon[36569]: pgmap v1344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:25 smithi104 ceph-mon[36624]: pgmap v1344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:25 smithi081 ceph-mon[31453]: pgmap v1344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:27 smithi153 ceph-mon[36569]: pgmap v1345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:27 smithi104 ceph-mon[36624]: pgmap v1345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:27 smithi081 ceph-mon[31453]: pgmap v1345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:29 smithi153 ceph-mon[36569]: pgmap v1346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:29 smithi104 ceph-mon[36624]: pgmap v1346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:29 smithi081 ceph-mon[31453]: pgmap v1346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:31 smithi153 ceph-mon[36569]: pgmap v1347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:31 smithi104 ceph-mon[36624]: pgmap v1347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:31 smithi081 ceph-mon[31453]: pgmap v1347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:33 smithi153 ceph-mon[36569]: pgmap v1348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:33 smithi104 ceph-mon[36624]: pgmap v1348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:33 smithi081 ceph-mon[31453]: pgmap v1348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:35 smithi153 ceph-mon[36569]: pgmap v1349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:35 smithi104 ceph-mon[36624]: pgmap v1349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:35 smithi081 ceph-mon[31453]: pgmap v1349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:37 smithi153 ceph-mon[36569]: pgmap v1350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:37 smithi104 ceph-mon[36624]: pgmap v1350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:37 smithi081 ceph-mon[31453]: pgmap v1350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:39.182+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:44:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:39.182+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:44:39.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:39.182+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:44:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:39 smithi153 ceph-mon[36569]: pgmap v1351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:39 smithi104 ceph-mon[36624]: pgmap v1351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:39 smithi081 ceph-mon[31453]: pgmap v1351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:41 smithi081 ceph-mon[31453]: pgmap v1352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:41 smithi153 ceph-mon[36569]: pgmap v1352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:41 smithi104 ceph-mon[36624]: pgmap v1352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:43 smithi153 ceph-mon[36569]: pgmap v1353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:43 smithi104 ceph-mon[36624]: pgmap v1353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:43 smithi081 ceph-mon[31453]: pgmap v1353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:45 smithi153 ceph-mon[36569]: pgmap v1354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:45 smithi104 ceph-mon[36624]: pgmap v1354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:45 smithi081 ceph-mon[31453]: pgmap v1354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:47 smithi153 ceph-mon[36569]: pgmap v1355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:47 smithi104 ceph-mon[36624]: pgmap v1355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:47 smithi081 ceph-mon[31453]: pgmap v1355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:49 smithi153 ceph-mon[36569]: pgmap v1356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:49 smithi104 ceph-mon[36624]: pgmap v1356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:49 smithi081 ceph-mon[31453]: pgmap v1356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:51 smithi153 ceph-mon[36569]: pgmap v1357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:51 smithi104 ceph-mon[36624]: pgmap v1357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:51 smithi081 ceph-mon[31453]: pgmap v1357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:53 smithi153 ceph-mon[36569]: pgmap v1358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:53 smithi104 ceph-mon[36624]: pgmap v1358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:53 smithi081 ceph-mon[31453]: pgmap v1358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:54.198+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:44:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:54.198+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:44:54.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:44:54.198+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:44:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:55 smithi153 ceph-mon[36569]: pgmap v1359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:55 smithi104 ceph-mon[36624]: pgmap v1359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:55 smithi081 ceph-mon[31453]: pgmap v1359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:57 smithi153 ceph-mon[36569]: pgmap v1360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:57 smithi104 ceph-mon[36624]: pgmap v1360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:57 smithi081 ceph-mon[31453]: pgmap v1360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:44:59 smithi153 ceph-mon[36569]: pgmap v1361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:44:59 smithi104 ceph-mon[36624]: pgmap v1361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:44:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:44:59 smithi081 ceph-mon[31453]: pgmap v1361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:01 smithi153 ceph-mon[36569]: pgmap v1362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:01 smithi104 ceph-mon[36624]: pgmap v1362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:01 smithi081 ceph-mon[31453]: pgmap v1362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:03 smithi153 ceph-mon[36569]: pgmap v1363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:03 smithi104 ceph-mon[36624]: pgmap v1363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:03 smithi081 ceph-mon[31453]: pgmap v1363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:05 smithi153 ceph-mon[36569]: pgmap v1364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:05 smithi104 ceph-mon[36624]: pgmap v1364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:05 smithi081 ceph-mon[31453]: pgmap v1364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:07 smithi153 ceph-mon[36569]: pgmap v1365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:07 smithi104 ceph-mon[36624]: pgmap v1365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:07 smithi081 ceph-mon[31453]: pgmap v1365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:09.210+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:45:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:09.210+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:45:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:09.210+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:45:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:09 smithi153 ceph-mon[36569]: pgmap v1366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:09 smithi104 ceph-mon[36624]: pgmap v1366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:09 smithi081 ceph-mon[31453]: pgmap v1366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:11 smithi153 ceph-mon[36569]: pgmap v1367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:11 smithi104 ceph-mon[36624]: pgmap v1367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:11 smithi081 ceph-mon[31453]: pgmap v1367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:13 smithi153 ceph-mon[36569]: pgmap v1368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:13 smithi104 ceph-mon[36624]: pgmap v1368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:13 smithi081 ceph-mon[31453]: pgmap v1368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:15 smithi104 ceph-mon[36624]: pgmap v1369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:15 smithi081 ceph-mon[31453]: pgmap v1369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:15 smithi153 ceph-mon[36569]: pgmap v1369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:17 smithi104 ceph-mon[36624]: pgmap v1370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:17 smithi081 ceph-mon[31453]: pgmap v1370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:17 smithi153 ceph-mon[36569]: pgmap v1370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:19 smithi104 ceph-mon[36624]: pgmap v1371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:45:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:19 smithi081 ceph-mon[31453]: pgmap v1371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:45:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:19 smithi153 ceph-mon[36569]: pgmap v1371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:45:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:45:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:45:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:45:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:45:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:45:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:45:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:45:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:45:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:45:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:21 smithi081 ceph-mon[31453]: pgmap v1372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:21 smithi081 ceph-mon[31453]: pgmap v1373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:21 smithi153 ceph-mon[36569]: pgmap v1372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:21 smithi153 ceph-mon[36569]: pgmap v1373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:21 smithi104 ceph-mon[36624]: pgmap v1372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:21 smithi104 ceph-mon[36624]: pgmap v1373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:23.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:23 smithi153 ceph-mon[36569]: pgmap v1374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:23 smithi081 ceph-mon[31453]: pgmap v1374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:23 smithi104 ceph-mon[36624]: pgmap v1374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:24.223+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:45:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:24.223+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:45:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:24.223+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:45:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:25 smithi081 ceph-mon[31453]: pgmap v1375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:25 smithi153 ceph-mon[36569]: pgmap v1375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:25 smithi104 ceph-mon[36624]: pgmap v1375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:27 smithi081 ceph-mon[31453]: pgmap v1376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:27 smithi153 ceph-mon[36569]: pgmap v1376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:27 smithi104 ceph-mon[36624]: pgmap v1376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:29.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:29 smithi104 ceph-mon[36624]: pgmap v1377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:29 smithi081 ceph-mon[31453]: pgmap v1377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:29 smithi153 ceph-mon[36569]: pgmap v1377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:31 smithi081 ceph-mon[31453]: pgmap v1378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:31 smithi153 ceph-mon[36569]: pgmap v1378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:31 smithi104 ceph-mon[36624]: pgmap v1378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:33 smithi081 ceph-mon[31453]: pgmap v1379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:33 smithi153 ceph-mon[36569]: pgmap v1379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:33 smithi104 ceph-mon[36624]: pgmap v1379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:35 smithi081 ceph-mon[31453]: pgmap v1380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:35 smithi153 ceph-mon[36569]: pgmap v1380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:35 smithi104 ceph-mon[36624]: pgmap v1380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:37 smithi081 ceph-mon[31453]: pgmap v1381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:37 smithi153 ceph-mon[36569]: pgmap v1381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:37 smithi104 ceph-mon[36624]: pgmap v1381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:39.237+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:39.237+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:45:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:39.237+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:45:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:39 smithi081 ceph-mon[31453]: pgmap v1382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:39 smithi153 ceph-mon[36569]: pgmap v1382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:39 smithi104 ceph-mon[36624]: pgmap v1382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:41 smithi081 ceph-mon[31453]: pgmap v1383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:41 smithi153 ceph-mon[36569]: pgmap v1383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:41 smithi104 ceph-mon[36624]: pgmap v1383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:43 smithi081 ceph-mon[31453]: pgmap v1384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:43 smithi153 ceph-mon[36569]: pgmap v1384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:43 smithi104 ceph-mon[36624]: pgmap v1384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:45 smithi081 ceph-mon[31453]: pgmap v1385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:45 smithi153 ceph-mon[36569]: pgmap v1385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:45 smithi104 ceph-mon[36624]: pgmap v1385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:47 smithi081 ceph-mon[31453]: pgmap v1386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:47 smithi153 ceph-mon[36569]: pgmap v1386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:47 smithi104 ceph-mon[36624]: pgmap v1386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:49 smithi081 ceph-mon[31453]: pgmap v1387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:49 smithi153 ceph-mon[36569]: pgmap v1387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:49 smithi104 ceph-mon[36624]: pgmap v1387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:51 smithi153 ceph-mon[36569]: pgmap v1388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:52.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:51 smithi104 ceph-mon[36624]: pgmap v1388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:51 smithi081 ceph-mon[31453]: pgmap v1388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:53 smithi153 ceph-mon[36569]: pgmap v1389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:53 smithi104 ceph-mon[36624]: pgmap v1389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:53 smithi081 ceph-mon[31453]: pgmap v1389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:54.248+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:45:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:54.248+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:45:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:45:54.248+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:45:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:55 smithi153 ceph-mon[36569]: pgmap v1390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:55 smithi104 ceph-mon[36624]: pgmap v1390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:55 smithi081 ceph-mon[31453]: pgmap v1390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:57 smithi153 ceph-mon[36569]: pgmap v1391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:58.103 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:57 smithi104 ceph-mon[36624]: pgmap v1391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:45:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:57 smithi081 ceph-mon[31453]: pgmap v1391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:45:59 smithi153 ceph-mon[36569]: pgmap v1392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:45:59 smithi104 ceph-mon[36624]: pgmap v1392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:45:59 smithi081 ceph-mon[31453]: pgmap v1392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:01 smithi153 ceph-mon[36569]: pgmap v1393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:01 smithi104 ceph-mon[36624]: pgmap v1393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:01 smithi081 ceph-mon[31453]: pgmap v1393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:03 smithi153 ceph-mon[36569]: pgmap v1394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:03 smithi104 ceph-mon[36624]: pgmap v1394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:03 smithi081 ceph-mon[31453]: pgmap v1394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:05 smithi153 ceph-mon[36569]: pgmap v1395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:05 smithi104 ceph-mon[36624]: pgmap v1395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:05 smithi081 ceph-mon[31453]: pgmap v1395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:07 smithi153 ceph-mon[36569]: pgmap v1396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:07 smithi104 ceph-mon[36624]: pgmap v1396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:07 smithi081 ceph-mon[31453]: pgmap v1396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:09.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:09.263+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:46:09.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:09.263+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:46:09.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:09.263+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:46:09.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:09 smithi081 ceph-mon[31453]: pgmap v1397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:09 smithi153 ceph-mon[36569]: pgmap v1397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:09 smithi104 ceph-mon[36624]: pgmap v1397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:11 smithi153 ceph-mon[36569]: pgmap v1398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:11 smithi104 ceph-mon[36624]: pgmap v1398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T21:46:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T21:46:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 0x7f16e80b1520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f16e80a40b0 tx=0x7f16f40bd040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T21:46:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 0x7f16e80b1520 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f16e80a40b0 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-12T21:46:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-12T21:46:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T21:46:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af130 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T21:46:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 0x7f16e80a4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T21:46:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.089+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 0x7f16e80a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T21:46:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.091+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 0x7f16e80a4850 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f16d40034a0 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T21:46:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:11 smithi081 ceph-mon[31453]: pgmap v1398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.986+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:46:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:12.986+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:46:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:13.024+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:46:13.026 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:13.024+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:46:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:13 smithi153 ceph-mon[36569]: pgmap v1399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:13 smithi104 ceph-mon[36624]: pgmap v1399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:13 smithi081 ceph-mon[31453]: pgmap v1399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:15 smithi153 ceph-mon[36569]: pgmap v1400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:15 smithi104 ceph-mon[36624]: pgmap v1400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:15 smithi081 ceph-mon[31453]: pgmap v1400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:17 smithi153 ceph-mon[36569]: pgmap v1401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:17 smithi104 ceph-mon[36624]: pgmap v1401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:17 smithi081 ceph-mon[31453]: pgmap v1401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:19 smithi153 ceph-mon[36569]: pgmap v1402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:19 smithi104 ceph-mon[36624]: pgmap v1402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:19 smithi081 ceph-mon[31453]: pgmap v1402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:46:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:46:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:46:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:21 smithi153 ceph-mon[36569]: pgmap v1403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:46:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:46:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:46:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:21 smithi104 ceph-mon[36624]: pgmap v1403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:46:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:46:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:46:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:21 smithi081 ceph-mon[31453]: pgmap v1403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:46:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:46:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:46:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:22 smithi153 ceph-mon[36569]: pgmap v1404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:22 smithi153 ceph-mon[36569]: pgmap v1405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:22 smithi104 ceph-mon[36624]: pgmap v1404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:22 smithi104 ceph-mon[36624]: pgmap v1405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:22 smithi081 ceph-mon[31453]: pgmap v1404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:22 smithi081 ceph-mon[31453]: pgmap v1405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:24.266+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:46:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:24.266+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:46:24.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:24.266+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:46:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:24 smithi153 ceph-mon[36569]: pgmap v1406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:24 smithi104 ceph-mon[36624]: pgmap v1406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:24 smithi081 ceph-mon[31453]: pgmap v1406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:26 smithi153 ceph-mon[36569]: pgmap v1407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:26 smithi104 ceph-mon[36624]: pgmap v1407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:26 smithi081 ceph-mon[31453]: pgmap v1407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:28 smithi153 ceph-mon[36569]: pgmap v1408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:28 smithi104 ceph-mon[36624]: pgmap v1408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:28 smithi081 ceph-mon[31453]: pgmap v1408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:30 smithi153 ceph-mon[36569]: pgmap v1409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:30 smithi104 ceph-mon[36624]: pgmap v1409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:30 smithi081 ceph-mon[31453]: pgmap v1409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:32 smithi153 ceph-mon[36569]: pgmap v1410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:32 smithi104 ceph-mon[36624]: pgmap v1410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:32 smithi081 ceph-mon[31453]: pgmap v1410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:34 smithi153 ceph-mon[36569]: pgmap v1411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:34 smithi104 ceph-mon[36624]: pgmap v1411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:34 smithi081 ceph-mon[31453]: pgmap v1411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:36 smithi153 ceph-mon[36569]: pgmap v1412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:36 smithi104 ceph-mon[36624]: pgmap v1412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:36 smithi081 ceph-mon[31453]: pgmap v1412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:38 smithi153 ceph-mon[36569]: pgmap v1413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:38 smithi104 ceph-mon[36624]: pgmap v1413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:38 smithi081 ceph-mon[31453]: pgmap v1413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:39.278+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:46:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:39.278+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:46:39.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:39.278+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:46:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:40 smithi153 ceph-mon[36569]: pgmap v1414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:40 smithi104 ceph-mon[36624]: pgmap v1414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:40 smithi081 ceph-mon[31453]: pgmap v1414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:42 smithi104 ceph-mon[36624]: pgmap v1415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:42 smithi081 ceph-mon[31453]: pgmap v1415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:42 smithi153 ceph-mon[36569]: pgmap v1415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:44 smithi104 ceph-mon[36624]: pgmap v1416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:44 smithi081 ceph-mon[31453]: pgmap v1416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:44 smithi153 ceph-mon[36569]: pgmap v1416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:46 smithi104 ceph-mon[36624]: pgmap v1417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:46 smithi081 ceph-mon[31453]: pgmap v1417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:46 smithi153 ceph-mon[36569]: pgmap v1417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:48 smithi104 ceph-mon[36624]: pgmap v1418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:48 smithi081 ceph-mon[31453]: pgmap v1418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:48 smithi153 ceph-mon[36569]: pgmap v1418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:50 smithi104 ceph-mon[36624]: pgmap v1419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:50 smithi081 ceph-mon[31453]: pgmap v1419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:50 smithi153 ceph-mon[36569]: pgmap v1419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:52 smithi081 ceph-mon[31453]: pgmap v1420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:52 smithi153 ceph-mon[36569]: pgmap v1420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:52 smithi104 ceph-mon[36624]: pgmap v1420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:54.280+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:46:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:54.280+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:46:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:46:54.280+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:46:55.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:54 smithi153 ceph-mon[36569]: pgmap v1421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:54 smithi081 ceph-mon[31453]: pgmap v1421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:54 smithi104 ceph-mon[36624]: pgmap v1421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:56 smithi081 ceph-mon[31453]: pgmap v1422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:56 smithi153 ceph-mon[36569]: pgmap v1422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:56 smithi104 ceph-mon[36624]: pgmap v1422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:46:58 smithi081 ceph-mon[31453]: pgmap v1423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:46:58 smithi153 ceph-mon[36569]: pgmap v1423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:46:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:46:58 smithi104 ceph-mon[36624]: pgmap v1423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:01.128 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:00 smithi081 ceph-mon[31453]: pgmap v1424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:01.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:00 smithi104 ceph-mon[36624]: pgmap v1424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:00 smithi153 ceph-mon[36569]: pgmap v1424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:02 smithi081 ceph-mon[31453]: pgmap v1425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:02 smithi153 ceph-mon[36569]: pgmap v1425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:02 smithi104 ceph-mon[36624]: pgmap v1425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:04 smithi081 ceph-mon[31453]: pgmap v1426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:04 smithi153 ceph-mon[36569]: pgmap v1426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:04 smithi104 ceph-mon[36624]: pgmap v1426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:06 smithi081 ceph-mon[31453]: pgmap v1427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:06 smithi153 ceph-mon[36569]: pgmap v1427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:06 smithi104 ceph-mon[36624]: pgmap v1427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:08 smithi081 ceph-mon[31453]: pgmap v1428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:09.296+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:47:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:09.296+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:47:09.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:09.296+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:47:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:08 smithi153 ceph-mon[36569]: pgmap v1428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:08 smithi104 ceph-mon[36624]: pgmap v1428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:10 smithi081 ceph-mon[31453]: pgmap v1429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:10 smithi153 ceph-mon[36569]: pgmap v1429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:10 smithi104 ceph-mon[36624]: pgmap v1429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:12 smithi081 ceph-mon[31453]: pgmap v1430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:12 smithi153 ceph-mon[36569]: pgmap v1430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:12 smithi104 ceph-mon[36624]: pgmap v1430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:14 smithi081 ceph-mon[31453]: pgmap v1431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:14 smithi153 ceph-mon[36569]: pgmap v1431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:14 smithi104 ceph-mon[36624]: pgmap v1431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:16 smithi081 ceph-mon[31453]: pgmap v1432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:16 smithi153 ceph-mon[36569]: pgmap v1432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:16 smithi104 ceph-mon[36624]: pgmap v1432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:18 smithi081 ceph-mon[31453]: pgmap v1433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:18 smithi153 ceph-mon[36569]: pgmap v1433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:18 smithi104 ceph-mon[36624]: pgmap v1433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:20 smithi153 ceph-mon[36569]: pgmap v1434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:20 smithi104 ceph-mon[36624]: pgmap v1434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:20 smithi081 ceph-mon[31453]: pgmap v1434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:47:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:47:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:47:22.981 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:22 smithi153 ceph-mon[36569]: pgmap v1435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:22.981 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:22.981 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:22.981 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:22.981 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:23.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:22 smithi104 ceph-mon[36624]: pgmap v1435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:23.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:23.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:23.182 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:23.182 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:22 smithi081 ceph-mon[31453]: pgmap v1435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:24.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:24.311+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:47:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:24.311+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:47:24.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:24.311+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:47:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:24 smithi153 ceph-mon[36569]: pgmap v1436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:24 smithi104 ceph-mon[36624]: pgmap v1436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:24 smithi081 ceph-mon[31453]: pgmap v1436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:26 smithi153 ceph-mon[36569]: pgmap v1437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:26 smithi104 ceph-mon[36624]: pgmap v1437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:26 smithi081 ceph-mon[31453]: pgmap v1437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:28 smithi081 ceph-mon[31453]: pgmap v1438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:28 smithi153 ceph-mon[36569]: pgmap v1438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:28 smithi104 ceph-mon[36624]: pgmap v1438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:47:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:47:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:29 smithi081 ceph-mon[31453]: pgmap v1439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:29 smithi081 ceph-mon[31453]: pgmap v1440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:47:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:47:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:29 smithi153 ceph-mon[36569]: pgmap v1439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:29 smithi153 ceph-mon[36569]: pgmap v1440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:47:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:47:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:29 smithi104 ceph-mon[36624]: pgmap v1439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:29 smithi104 ceph-mon[36624]: pgmap v1440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:47:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:30 smithi081 ceph-mon[31453]: Health check update: 12 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:47:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:30 smithi153 ceph-mon[36569]: Health check update: 12 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:47:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:30 smithi104 ceph-mon[36624]: Health check update: 12 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-12T21:47:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:31 smithi081 ceph-mon[31453]: pgmap v1441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:31.484 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:31 smithi104 ceph-mon[36624]: pgmap v1441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:31 smithi153 ceph-mon[36569]: pgmap v1441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:33 smithi153 ceph-mon[36569]: pgmap v1442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:33 smithi104 ceph-mon[36624]: pgmap v1442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:33 smithi081 ceph-mon[31453]: pgmap v1442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:35 smithi153 ceph-mon[36569]: pgmap v1443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:35 smithi104 ceph-mon[36624]: pgmap v1443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:35 smithi081 ceph-mon[31453]: pgmap v1443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:37 smithi153 ceph-mon[36569]: pgmap v1444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:37 smithi104 ceph-mon[36624]: pgmap v1444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:37 smithi081 ceph-mon[31453]: pgmap v1444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:39.327+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:47:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:39.327+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:47:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:39.327+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:47:40.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:39 smithi081 ceph-mon[31453]: pgmap v1445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:39 smithi153 ceph-mon[36569]: pgmap v1445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:39 smithi104 ceph-mon[36624]: pgmap v1445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:40 smithi153 ceph-mon[36569]: pgmap v1446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:40 smithi104 ceph-mon[36624]: pgmap v1446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:40 smithi081 ceph-mon[31453]: pgmap v1446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:43 smithi081 ceph-mon[31453]: pgmap v1447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:43 smithi153 ceph-mon[36569]: pgmap v1447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:43 smithi104 ceph-mon[36624]: pgmap v1447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:45 smithi153 ceph-mon[36569]: pgmap v1448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:45 smithi104 ceph-mon[36624]: pgmap v1448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:45 smithi081 ceph-mon[31453]: pgmap v1448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:47 smithi081 ceph-mon[31453]: pgmap v1449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:47 smithi153 ceph-mon[36569]: pgmap v1449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:47 smithi104 ceph-mon[36624]: pgmap v1449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:49 smithi153 ceph-mon[36569]: pgmap v1450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:49 smithi104 ceph-mon[36624]: pgmap v1450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:49 smithi081 ceph-mon[31453]: pgmap v1450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:51 smithi153 ceph-mon[36569]: pgmap v1451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:51 smithi104 ceph-mon[36624]: pgmap v1451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:51 smithi081 ceph-mon[31453]: pgmap v1451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:53 smithi153 ceph-mon[36569]: pgmap v1452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:53 smithi104 ceph-mon[36624]: pgmap v1452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:53 smithi081 ceph-mon[31453]: pgmap v1452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:54.334+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:47:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:54.334+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:47:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:47:54.334+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:47:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:55 smithi153 ceph-mon[36569]: pgmap v1453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:55 smithi104 ceph-mon[36624]: pgmap v1453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:55 smithi081 ceph-mon[31453]: pgmap v1453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:57 smithi153 ceph-mon[36569]: pgmap v1454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:57 smithi104 ceph-mon[36624]: pgmap v1454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:57 smithi081 ceph-mon[31453]: pgmap v1454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:47:59 smithi153 ceph-mon[36569]: pgmap v1455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:47:59 smithi104 ceph-mon[36624]: pgmap v1455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:47:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:47:59 smithi081 ceph-mon[31453]: pgmap v1455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:01 smithi153 ceph-mon[36569]: pgmap v1456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:01 smithi104 ceph-mon[36624]: pgmap v1456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:01 smithi081 ceph-mon[31453]: pgmap v1456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:03 smithi153 ceph-mon[36569]: pgmap v1457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:03 smithi104 ceph-mon[36624]: pgmap v1457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:03 smithi081 ceph-mon[31453]: pgmap v1457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:05 smithi153 ceph-mon[36569]: pgmap v1458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:05 smithi104 ceph-mon[36624]: pgmap v1458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:05 smithi081 ceph-mon[31453]: pgmap v1458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:07 smithi153 ceph-mon[36569]: pgmap v1459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:07 smithi104 ceph-mon[36624]: pgmap v1459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:07 smithi081 ceph-mon[31453]: pgmap v1459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:09.350+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:48:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:09.350+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:48:09.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:09.350+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:48:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:09 smithi153 ceph-mon[36569]: pgmap v1460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:09 smithi104 ceph-mon[36624]: pgmap v1460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:09 smithi081 ceph-mon[31453]: pgmap v1460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:11 smithi153 ceph-mon[36569]: pgmap v1461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:11 smithi104 ceph-mon[36624]: pgmap v1461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:11 smithi081 ceph-mon[31453]: pgmap v1461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:13 smithi153 ceph-mon[36569]: pgmap v1462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:13 smithi104 ceph-mon[36624]: pgmap v1462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:13 smithi081 ceph-mon[31453]: pgmap v1462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:15 smithi153 ceph-mon[36569]: pgmap v1463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:15 smithi104 ceph-mon[36624]: pgmap v1463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:15 smithi081 ceph-mon[31453]: pgmap v1463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:17 smithi153 ceph-mon[36569]: pgmap v1464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:17 smithi104 ceph-mon[36624]: pgmap v1464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:17 smithi081 ceph-mon[31453]: pgmap v1464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:19 smithi153 ceph-mon[36569]: pgmap v1465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:19 smithi104 ceph-mon[36624]: pgmap v1465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:19 smithi081 ceph-mon[31453]: pgmap v1465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:21 smithi153 ceph-mon[36569]: pgmap v1466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:21 smithi104 ceph-mon[36624]: pgmap v1466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:21 smithi081 ceph-mon[31453]: pgmap v1466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:23 smithi153 ceph-mon[36569]: pgmap v1467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:23 smithi104 ceph-mon[36624]: pgmap v1467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:24.364+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:48:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:24.365+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:48:24.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:24.365+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:48:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:23 smithi081 ceph-mon[31453]: pgmap v1467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:25 smithi153 ceph-mon[36569]: pgmap v1468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:25 smithi104 ceph-mon[36624]: pgmap v1468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:25 smithi081 ceph-mon[31453]: pgmap v1468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:26 smithi153 ceph-mon[36569]: pgmap v1469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:26 smithi104 ceph-mon[36624]: pgmap v1469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:26 smithi081 ceph-mon[31453]: pgmap v1469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:29 smithi153 ceph-mon[36569]: pgmap v1470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:48:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:29 smithi104 ceph-mon[36624]: pgmap v1470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:48:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:29 smithi081 ceph-mon[31453]: pgmap v1470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:48:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:30.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:48:30.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:48:30.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:48:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:48:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:48:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:48:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:48:31.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:31 smithi104 ceph-mon[36624]: pgmap v1471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:31.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:31 smithi104 ceph-mon[36624]: pgmap v1472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:31 smithi153 ceph-mon[36569]: pgmap v1471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:31 smithi153 ceph-mon[36569]: pgmap v1472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:31 smithi081 ceph-mon[31453]: pgmap v1471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:31 smithi081 ceph-mon[31453]: pgmap v1472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:33 smithi081 ceph-mon[31453]: pgmap v1473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:33 smithi153 ceph-mon[36569]: pgmap v1473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:33 smithi104 ceph-mon[36624]: pgmap v1473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:35 smithi153 ceph-mon[36569]: pgmap v1474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:35 smithi104 ceph-mon[36624]: pgmap v1474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:35 smithi081 ceph-mon[31453]: pgmap v1474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:37 smithi153 ceph-mon[36569]: pgmap v1475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:37 smithi104 ceph-mon[36624]: pgmap v1475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:37 smithi081 ceph-mon[31453]: pgmap v1475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:39.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:39.380+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:48:39.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:39.380+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:48:39.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:39.380+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:48:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:39 smithi153 ceph-mon[36569]: pgmap v1476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:39 smithi104 ceph-mon[36624]: pgmap v1476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:39 smithi081 ceph-mon[31453]: pgmap v1476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:41 smithi153 ceph-mon[36569]: pgmap v1477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:41 smithi104 ceph-mon[36624]: pgmap v1477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:41 smithi081 ceph-mon[31453]: pgmap v1477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:43 smithi153 ceph-mon[36569]: pgmap v1478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:43 smithi104 ceph-mon[36624]: pgmap v1478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:43 smithi081 ceph-mon[31453]: pgmap v1478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:45 smithi153 ceph-mon[36569]: pgmap v1479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:45 smithi104 ceph-mon[36624]: pgmap v1479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:45 smithi081 ceph-mon[31453]: pgmap v1479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:47 smithi153 ceph-mon[36569]: pgmap v1480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:47 smithi104 ceph-mon[36624]: pgmap v1480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:47 smithi081 ceph-mon[31453]: pgmap v1480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:49 smithi153 ceph-mon[36569]: pgmap v1481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:49 smithi104 ceph-mon[36624]: pgmap v1481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:49 smithi081 ceph-mon[31453]: pgmap v1481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:51 smithi153 ceph-mon[36569]: pgmap v1482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:51 smithi104 ceph-mon[36624]: pgmap v1482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:51 smithi081 ceph-mon[31453]: pgmap v1482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:53 smithi153 ceph-mon[36569]: pgmap v1483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:53 smithi104 ceph-mon[36624]: pgmap v1483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:53 smithi081 ceph-mon[31453]: pgmap v1483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:54.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:54.385+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:48:54.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:54.385+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:48:54.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:48:54.385+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:48:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:55 smithi153 ceph-mon[36569]: pgmap v1484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:55 smithi104 ceph-mon[36624]: pgmap v1484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:55 smithi081 ceph-mon[31453]: pgmap v1484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:57 smithi153 ceph-mon[36569]: pgmap v1485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:57 smithi104 ceph-mon[36624]: pgmap v1485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:57 smithi081 ceph-mon[31453]: pgmap v1485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:48:59 smithi153 ceph-mon[36569]: pgmap v1486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:48:59 smithi104 ceph-mon[36624]: pgmap v1486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:48:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:48:59 smithi081 ceph-mon[31453]: pgmap v1486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:01 smithi153 ceph-mon[36569]: pgmap v1487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:01 smithi104 ceph-mon[36624]: pgmap v1487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:01 smithi081 ceph-mon[31453]: pgmap v1487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:03 smithi153 ceph-mon[36569]: pgmap v1488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:03 smithi104 ceph-mon[36624]: pgmap v1488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:03 smithi081 ceph-mon[31453]: pgmap v1488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:05 smithi153 ceph-mon[36569]: pgmap v1489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:05 smithi104 ceph-mon[36624]: pgmap v1489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:05 smithi081 ceph-mon[31453]: pgmap v1489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:07 smithi153 ceph-mon[36569]: pgmap v1490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:07 smithi104 ceph-mon[36624]: pgmap v1490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:07 smithi081 ceph-mon[31453]: pgmap v1490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:09.401+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:49:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:09.401+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:49:09.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:09.401+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:49:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:09 smithi153 ceph-mon[36569]: pgmap v1491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:09 smithi104 ceph-mon[36624]: pgmap v1491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:09 smithi081 ceph-mon[31453]: pgmap v1491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:11 smithi153 ceph-mon[36569]: pgmap v1492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:11 smithi104 ceph-mon[36624]: pgmap v1492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:11 smithi081 ceph-mon[31453]: pgmap v1492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:13.017+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:49:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:13.017+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T21:49:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:13.017+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:49:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:13.018+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:49:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:13.052+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:49:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:13.053+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T21:49:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:13.053+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:49:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:13.054+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:49:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:13 smithi153 ceph-mon[36569]: pgmap v1493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:13 smithi104 ceph-mon[36624]: pgmap v1493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:13 smithi081 ceph-mon[31453]: pgmap v1493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:15 smithi104 ceph-mon[36624]: pgmap v1494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:15 smithi081 ceph-mon[31453]: pgmap v1494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:15 smithi153 ceph-mon[36569]: pgmap v1494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:17 smithi104 ceph-mon[36624]: pgmap v1495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:17 smithi081 ceph-mon[31453]: pgmap v1495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:17 smithi153 ceph-mon[36569]: pgmap v1495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:19 smithi104 ceph-mon[36624]: pgmap v1496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:19.880 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:19 smithi081 ceph-mon[31453]: pgmap v1496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:19 smithi153 ceph-mon[36569]: pgmap v1496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:21 smithi104 ceph-mon[36624]: pgmap v1497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:21 smithi081 ceph-mon[31453]: pgmap v1497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:21 smithi153 ceph-mon[36569]: pgmap v1497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:23 smithi104 ceph-mon[36624]: pgmap v1498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:23 smithi081 ceph-mon[31453]: pgmap v1498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:23 smithi153 ceph-mon[36569]: pgmap v1498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:24.408+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:49:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:24.408+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:49:24.409 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:24.408+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:49:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:25 smithi081 ceph-mon[31453]: pgmap v1499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:25 smithi153 ceph-mon[36569]: pgmap v1499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:25 smithi104 ceph-mon[36624]: pgmap v1499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:27.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:27 smithi153 ceph-mon[36569]: pgmap v1500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:27.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:27 smithi081 ceph-mon[31453]: pgmap v1500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:27 smithi104 ceph-mon[36624]: pgmap v1500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:29 smithi081 ceph-mon[31453]: pgmap v1501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:29 smithi153 ceph-mon[36569]: pgmap v1501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:29 smithi104 ceph-mon[36624]: pgmap v1501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:49:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:49:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:49:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:31 smithi081 ceph-mon[31453]: pgmap v1502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:49:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:49:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:49:31.985 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:31 smithi104 ceph-mon[36624]: pgmap v1502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:31.985 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:49:31.985 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:49:31.985 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:49:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:31 smithi153 ceph-mon[36569]: pgmap v1502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:49:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:49:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:49:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:32 smithi081 ceph-mon[31453]: pgmap v1503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:32 smithi081 ceph-mon[31453]: pgmap v1504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:32 smithi153 ceph-mon[36569]: pgmap v1503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:32 smithi153 ceph-mon[36569]: pgmap v1504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:32 smithi104 ceph-mon[36624]: pgmap v1503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:32 smithi104 ceph-mon[36624]: pgmap v1504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:34 smithi081 ceph-mon[31453]: pgmap v1505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:34 smithi153 ceph-mon[36569]: pgmap v1505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:34 smithi104 ceph-mon[36624]: pgmap v1505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:36 smithi081 ceph-mon[31453]: pgmap v1506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:36 smithi153 ceph-mon[36569]: pgmap v1506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:36 smithi104 ceph-mon[36624]: pgmap v1506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:38 smithi081 ceph-mon[31453]: pgmap v1507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:38 smithi153 ceph-mon[36569]: pgmap v1507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:38 smithi104 ceph-mon[36624]: pgmap v1507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:39.423+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:49:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:39.423+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:49:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:39.423+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:49:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:40 smithi081 ceph-mon[31453]: pgmap v1508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:40 smithi153 ceph-mon[36569]: pgmap v1508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:40 smithi104 ceph-mon[36624]: pgmap v1508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:42 smithi081 ceph-mon[31453]: pgmap v1509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:42 smithi153 ceph-mon[36569]: pgmap v1509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:42 smithi104 ceph-mon[36624]: pgmap v1509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:44 smithi081 ceph-mon[31453]: pgmap v1510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:44 smithi153 ceph-mon[36569]: pgmap v1510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:44 smithi104 ceph-mon[36624]: pgmap v1510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:46 smithi081 ceph-mon[31453]: pgmap v1511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:46 smithi153 ceph-mon[36569]: pgmap v1511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:46 smithi104 ceph-mon[36624]: pgmap v1511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:48 smithi081 ceph-mon[31453]: pgmap v1512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:48 smithi153 ceph-mon[36569]: pgmap v1512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:48 smithi104 ceph-mon[36624]: pgmap v1512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:50 smithi153 ceph-mon[36569]: pgmap v1513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:50 smithi104 ceph-mon[36624]: pgmap v1513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:51.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:50 smithi081 ceph-mon[31453]: pgmap v1513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:52 smithi153 ceph-mon[36569]: pgmap v1514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:52 smithi104 ceph-mon[36624]: pgmap v1514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:52 smithi081 ceph-mon[31453]: pgmap v1514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:54.439+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:49:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:54.439+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:49:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:49:54.439+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:49:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:54 smithi153 ceph-mon[36569]: pgmap v1515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:54 smithi104 ceph-mon[36624]: pgmap v1515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:54 smithi081 ceph-mon[31453]: pgmap v1515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:56 smithi153 ceph-mon[36569]: pgmap v1516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:56 smithi104 ceph-mon[36624]: pgmap v1516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:56 smithi081 ceph-mon[31453]: pgmap v1516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:49:58 smithi153 ceph-mon[36569]: pgmap v1517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:49:58 smithi104 ceph-mon[36624]: pgmap v1517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:49:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:49:58 smithi081 ceph-mon[31453]: pgmap v1517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: pgmap v1518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: Health detail: HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: [WRN] CEPHADM_FAILED_DAEMON: 12 failed cephadm daemon(s) 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.j on smithi081 is in error state 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.d on smithi081 is in error state 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.g on smithi081 is in error state 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.a on smithi081 is in error state 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.k on smithi104 is in error state 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.e on smithi104 is in error state 2024-04-12T21:50:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.h on smithi104 is in error state 2024-04-12T21:50:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.b on smithi104 is in error state 2024-04-12T21:50:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.l on smithi153 is in error state 2024-04-12T21:50:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.f on smithi153 is in error state 2024-04-12T21:50:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.c on smithi153 is in error state 2024-04-12T21:50:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:00 smithi153 ceph-mon[36569]: daemon mds.i on smithi153 is in error state 2024-04-12T21:50:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: pgmap v1518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: Health detail: HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: [WRN] CEPHADM_FAILED_DAEMON: 12 failed cephadm daemon(s) 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.j on smithi081 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.d on smithi081 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.g on smithi081 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.a on smithi081 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.k on smithi104 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.e on smithi104 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.h on smithi104 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.b on smithi104 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.l on smithi153 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.f on smithi153 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.c on smithi153 is in error state 2024-04-12T21:50:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:00 smithi104 ceph-mon[36624]: daemon mds.i on smithi153 is in error state 2024-04-12T21:50:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: pgmap v1518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: Health detail: HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: [WRN] CEPHADM_FAILED_DAEMON: 12 failed cephadm daemon(s) 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.j on smithi081 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.d on smithi081 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.g on smithi081 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.a on smithi081 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.k on smithi104 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.e on smithi104 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.h on smithi104 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.b on smithi104 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.l on smithi153 is in error state 2024-04-12T21:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.f on smithi153 is in error state 2024-04-12T21:50:01.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.c on smithi153 is in error state 2024-04-12T21:50:01.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:00 smithi081 ceph-mon[31453]: daemon mds.i on smithi153 is in error state 2024-04-12T21:50:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:02 smithi153 ceph-mon[36569]: pgmap v1519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:02 smithi104 ceph-mon[36624]: pgmap v1519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:02 smithi081 ceph-mon[31453]: pgmap v1519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:04 smithi153 ceph-mon[36569]: pgmap v1520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:04 smithi104 ceph-mon[36624]: pgmap v1520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:04 smithi081 ceph-mon[31453]: pgmap v1520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:06.984 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:06 smithi081 ceph-mon[31453]: pgmap v1521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:06 smithi153 ceph-mon[36569]: pgmap v1521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:06 smithi104 ceph-mon[36624]: pgmap v1521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:08 smithi153 ceph-mon[36569]: pgmap v1522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:08 smithi104 ceph-mon[36624]: pgmap v1522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:08 smithi081 ceph-mon[31453]: pgmap v1522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:09.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:09.440+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:50:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:09.440+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:50:09.442 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:09.440+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:50:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:10 smithi153 ceph-mon[36569]: pgmap v1523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:10 smithi104 ceph-mon[36624]: pgmap v1523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:10 smithi081 ceph-mon[31453]: pgmap v1523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:12 smithi153 ceph-mon[36569]: pgmap v1524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:12 smithi104 ceph-mon[36624]: pgmap v1524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:12 smithi081 ceph-mon[31453]: pgmap v1524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:14 smithi153 ceph-mon[36569]: pgmap v1525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:14 smithi104 ceph-mon[36624]: pgmap v1525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:14 smithi081 ceph-mon[31453]: pgmap v1525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:16 smithi153 ceph-mon[36569]: pgmap v1526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:16 smithi104 ceph-mon[36624]: pgmap v1526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:16 smithi081 ceph-mon[31453]: pgmap v1526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:18 smithi153 ceph-mon[36569]: pgmap v1527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:18 smithi104 ceph-mon[36624]: pgmap v1527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:18 smithi081 ceph-mon[31453]: pgmap v1527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:20 smithi153 ceph-mon[36569]: pgmap v1528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:20 smithi104 ceph-mon[36624]: pgmap v1528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:20 smithi081 ceph-mon[31453]: pgmap v1528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:22 smithi153 ceph-mon[36569]: pgmap v1529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:22 smithi104 ceph-mon[36624]: pgmap v1529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:22 smithi081 ceph-mon[31453]: pgmap v1529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:24.455+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:50:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:24.455+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:50:24.457 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:24.455+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:50:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:24 smithi153 ceph-mon[36569]: pgmap v1530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:24 smithi104 ceph-mon[36624]: pgmap v1530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:24 smithi081 ceph-mon[31453]: pgmap v1530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:26 smithi153 ceph-mon[36569]: pgmap v1531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:26 smithi104 ceph-mon[36624]: pgmap v1531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:26 smithi081 ceph-mon[31453]: pgmap v1531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:28 smithi153 ceph-mon[36569]: pgmap v1532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:28 smithi104 ceph-mon[36624]: pgmap v1532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:28 smithi081 ceph-mon[31453]: pgmap v1532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:30 smithi153 ceph-mon[36569]: pgmap v1533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:30 smithi104 ceph-mon[36624]: pgmap v1533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:30 smithi081 ceph-mon[31453]: pgmap v1533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:50:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:32 smithi153 ceph-mon[36569]: pgmap v1534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:32 smithi104 ceph-mon[36624]: pgmap v1534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:32 smithi081 ceph-mon[31453]: pgmap v1534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:50:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:50:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:33 smithi153 ceph-mon[36569]: pgmap v1535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:33 smithi153 ceph-mon[36569]: pgmap v1536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:50:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:50:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:50:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:33 smithi104 ceph-mon[36624]: pgmap v1535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:33 smithi104 ceph-mon[36624]: pgmap v1536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:50:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:50:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:50:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:33 smithi081 ceph-mon[31453]: pgmap v1535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:33 smithi081 ceph-mon[31453]: pgmap v1536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:50:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:35 smithi153 ceph-mon[36569]: pgmap v1537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:35 smithi104 ceph-mon[36624]: pgmap v1537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:35 smithi081 ceph-mon[31453]: pgmap v1537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:37 smithi153 ceph-mon[36569]: pgmap v1538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:37 smithi104 ceph-mon[36624]: pgmap v1538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:37 smithi081 ceph-mon[31453]: pgmap v1538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:39 smithi153 ceph-mon[36569]: pgmap v1539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:39 smithi104 ceph-mon[36624]: pgmap v1539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:39 smithi081 ceph-mon[31453]: pgmap v1539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:39.466+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:50:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:39.466+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:50:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:39.466+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:50:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:41 smithi153 ceph-mon[36569]: pgmap v1540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:41 smithi104 ceph-mon[36624]: pgmap v1540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:41 smithi081 ceph-mon[31453]: pgmap v1540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:43 smithi153 ceph-mon[36569]: pgmap v1541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:43 smithi104 ceph-mon[36624]: pgmap v1541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:43 smithi081 ceph-mon[31453]: pgmap v1541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:45 smithi153 ceph-mon[36569]: pgmap v1542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:45 smithi104 ceph-mon[36624]: pgmap v1542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:45 smithi081 ceph-mon[31453]: pgmap v1542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:47 smithi153 ceph-mon[36569]: pgmap v1543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:47 smithi104 ceph-mon[36624]: pgmap v1543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:47 smithi081 ceph-mon[31453]: pgmap v1543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:49 smithi153 ceph-mon[36569]: pgmap v1544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:49 smithi104 ceph-mon[36624]: pgmap v1544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:49 smithi081 ceph-mon[31453]: pgmap v1544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:51 smithi153 ceph-mon[36569]: pgmap v1545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:51 smithi104 ceph-mon[36624]: pgmap v1545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:51 smithi081 ceph-mon[31453]: pgmap v1545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:53 smithi153 ceph-mon[36569]: pgmap v1546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:53 smithi104 ceph-mon[36624]: pgmap v1546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:53 smithi081 ceph-mon[31453]: pgmap v1546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:54.482+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:50:54.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:54.482+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:50:54.484 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:50:54.482+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:50:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:55 smithi153 ceph-mon[36569]: pgmap v1547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:55 smithi104 ceph-mon[36624]: pgmap v1547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:55 smithi081 ceph-mon[31453]: pgmap v1547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:57 smithi153 ceph-mon[36569]: pgmap v1548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:57 smithi104 ceph-mon[36624]: pgmap v1548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:57 smithi081 ceph-mon[31453]: pgmap v1548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:50:59 smithi153 ceph-mon[36569]: pgmap v1549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:50:59 smithi104 ceph-mon[36624]: pgmap v1549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:50:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:50:59 smithi081 ceph-mon[31453]: pgmap v1549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:01 smithi153 ceph-mon[36569]: pgmap v1550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:01 smithi104 ceph-mon[36624]: pgmap v1550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:01 smithi081 ceph-mon[31453]: pgmap v1550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:03 smithi153 ceph-mon[36569]: pgmap v1551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:03 smithi104 ceph-mon[36624]: pgmap v1551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:03 smithi081 ceph-mon[31453]: pgmap v1551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:05 smithi153 ceph-mon[36569]: pgmap v1552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:05 smithi104 ceph-mon[36624]: pgmap v1552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:05 smithi081 ceph-mon[31453]: pgmap v1552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:07 smithi153 ceph-mon[36569]: pgmap v1553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:07 smithi104 ceph-mon[36624]: pgmap v1553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:07 smithi081 ceph-mon[31453]: pgmap v1553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:09.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:09 smithi153 ceph-mon[36569]: pgmap v1554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:09 smithi104 ceph-mon[36624]: pgmap v1554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:09 smithi081 ceph-mon[31453]: pgmap v1554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:09.498+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:51:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:09.498+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:51:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:09.498+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:51:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:11 smithi153 ceph-mon[36569]: pgmap v1555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:11 smithi104 ceph-mon[36624]: pgmap v1555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:11 smithi081 ceph-mon[31453]: pgmap v1555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:13 smithi153 ceph-mon[36569]: pgmap v1556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:13 smithi104 ceph-mon[36624]: pgmap v1556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:13 smithi081 ceph-mon[31453]: pgmap v1556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:15 smithi153 ceph-mon[36569]: pgmap v1557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:15 smithi104 ceph-mon[36624]: pgmap v1557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:15 smithi081 ceph-mon[31453]: pgmap v1557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:17 smithi153 ceph-mon[36569]: pgmap v1558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:17 smithi104 ceph-mon[36624]: pgmap v1558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:17 smithi081 ceph-mon[31453]: pgmap v1558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:19 smithi153 ceph-mon[36569]: pgmap v1559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:19 smithi104 ceph-mon[36624]: pgmap v1559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:19 smithi081 ceph-mon[31453]: pgmap v1559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:21 smithi153 ceph-mon[36569]: pgmap v1560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:21 smithi104 ceph-mon[36624]: pgmap v1560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:21 smithi081 ceph-mon[31453]: pgmap v1560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:23 smithi153 ceph-mon[36569]: pgmap v1561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:23 smithi104 ceph-mon[36624]: pgmap v1561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:23 smithi081 ceph-mon[31453]: pgmap v1561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:24.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:24.513+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:51:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:24.513+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:51:24.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:24.513+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:51:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:25 smithi153 ceph-mon[36569]: pgmap v1562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:25.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:25 smithi081 ceph-mon[31453]: pgmap v1562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:25 smithi104 ceph-mon[36624]: pgmap v1562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:27 smithi153 ceph-mon[36569]: pgmap v1563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:27 smithi104 ceph-mon[36624]: pgmap v1563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:27 smithi081 ceph-mon[31453]: pgmap v1563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:29 smithi153 ceph-mon[36569]: pgmap v1564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:29 smithi104 ceph-mon[36624]: pgmap v1564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:29 smithi081 ceph-mon[31453]: pgmap v1564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:31 smithi153 ceph-mon[36569]: pgmap v1565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:31 smithi104 ceph-mon[36624]: pgmap v1565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:31 smithi081 ceph-mon[31453]: pgmap v1565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:33 smithi153 ceph-mon[36569]: pgmap v1566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:33 smithi104 ceph-mon[36624]: pgmap v1566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:33 smithi081 ceph-mon[31453]: pgmap v1566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:51:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:35 smithi104 ceph-mon[36624]: pgmap v1567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:35 smithi104 ceph-mon[36624]: pgmap v1568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:51:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:35 smithi081 ceph-mon[31453]: pgmap v1567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:35 smithi081 ceph-mon[31453]: pgmap v1568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:51:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:51:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:51:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:35 smithi153 ceph-mon[36569]: pgmap v1567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:35 smithi153 ceph-mon[36569]: pgmap v1568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:51:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:37 smithi081 ceph-mon[31453]: pgmap v1569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:37 smithi153 ceph-mon[36569]: pgmap v1569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:37 smithi104 ceph-mon[36624]: pgmap v1569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:39.525+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:51:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:39.525+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:51:39.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:39.525+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:51:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:39 smithi081 ceph-mon[31453]: pgmap v1570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:39 smithi153 ceph-mon[36569]: pgmap v1570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:39 smithi104 ceph-mon[36624]: pgmap v1570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:41 smithi153 ceph-mon[36569]: pgmap v1571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:41 smithi104 ceph-mon[36624]: pgmap v1571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:41 smithi081 ceph-mon[31453]: pgmap v1571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:43.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:43 smithi081 ceph-mon[31453]: pgmap v1572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:44.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:43 smithi153 ceph-mon[36569]: pgmap v1572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:43 smithi104 ceph-mon[36624]: pgmap v1572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:45 smithi153 ceph-mon[36569]: pgmap v1573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:45 smithi104 ceph-mon[36624]: pgmap v1573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:46.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:45 smithi081 ceph-mon[31453]: pgmap v1573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:47 smithi153 ceph-mon[36569]: pgmap v1574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:47 smithi104 ceph-mon[36624]: pgmap v1574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:47 smithi081 ceph-mon[31453]: pgmap v1574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:49 smithi153 ceph-mon[36569]: pgmap v1575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:49 smithi104 ceph-mon[36624]: pgmap v1575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:49 smithi081 ceph-mon[31453]: pgmap v1575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:51 smithi153 ceph-mon[36569]: pgmap v1576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:51 smithi104 ceph-mon[36624]: pgmap v1576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:51 smithi081 ceph-mon[31453]: pgmap v1576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:53 smithi153 ceph-mon[36569]: pgmap v1577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:53 smithi104 ceph-mon[36624]: pgmap v1577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:53 smithi081 ceph-mon[31453]: pgmap v1577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:54.536+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:51:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:54.537+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:51:54.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:51:54.537+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:51:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:55 smithi153 ceph-mon[36569]: pgmap v1578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:55 smithi104 ceph-mon[36624]: pgmap v1578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:55 smithi081 ceph-mon[31453]: pgmap v1578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:57 smithi153 ceph-mon[36569]: pgmap v1579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:57 smithi104 ceph-mon[36624]: pgmap v1579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:51:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:57 smithi081 ceph-mon[31453]: pgmap v1579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:51:59 smithi153 ceph-mon[36569]: pgmap v1580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:51:59 smithi104 ceph-mon[36624]: pgmap v1580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:51:59 smithi081 ceph-mon[31453]: pgmap v1580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:01 smithi153 ceph-mon[36569]: pgmap v1581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:01 smithi104 ceph-mon[36624]: pgmap v1581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:01 smithi081 ceph-mon[31453]: pgmap v1581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:03 smithi153 ceph-mon[36569]: pgmap v1582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:03 smithi104 ceph-mon[36624]: pgmap v1582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:03 smithi081 ceph-mon[31453]: pgmap v1582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:05 smithi153 ceph-mon[36569]: pgmap v1583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:05 smithi104 ceph-mon[36624]: pgmap v1583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:05 smithi081 ceph-mon[31453]: pgmap v1583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:07 smithi153 ceph-mon[36569]: pgmap v1584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:07 smithi104 ceph-mon[36624]: pgmap v1584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:07 smithi081 ceph-mon[31453]: pgmap v1584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:09 smithi153 ceph-mon[36569]: pgmap v1585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:09 smithi104 ceph-mon[36624]: pgmap v1585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:09 smithi081 ceph-mon[31453]: pgmap v1585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:09.552+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:52:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:09.552+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:52:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:09.552+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:52:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:11 smithi153 ceph-mon[36569]: pgmap v1586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:11 smithi104 ceph-mon[36624]: pgmap v1586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:11 smithi081 ceph-mon[31453]: pgmap v1586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:13.047 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:13.045+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:52:13.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:13.045+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:52:13.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:13.078+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:52:13.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:13.078+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:52:13.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:13 smithi081 ceph-mon[31453]: pgmap v1587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:14.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:13 smithi153 ceph-mon[36569]: pgmap v1587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:13 smithi104 ceph-mon[36624]: pgmap v1587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:15 smithi153 ceph-mon[36569]: pgmap v1588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:15 smithi104 ceph-mon[36624]: pgmap v1588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:15 smithi081 ceph-mon[31453]: pgmap v1588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:17 smithi153 ceph-mon[36569]: pgmap v1589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:17 smithi104 ceph-mon[36624]: pgmap v1589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:17 smithi081 ceph-mon[31453]: pgmap v1589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:19 smithi153 ceph-mon[36569]: pgmap v1590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:19 smithi104 ceph-mon[36624]: pgmap v1590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:19 smithi081 ceph-mon[31453]: pgmap v1590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:21 smithi153 ceph-mon[36569]: pgmap v1591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:21 smithi104 ceph-mon[36624]: pgmap v1591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:21 smithi081 ceph-mon[31453]: pgmap v1591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:23 smithi153 ceph-mon[36569]: pgmap v1592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:23 smithi104 ceph-mon[36624]: pgmap v1592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:23 smithi081 ceph-mon[31453]: pgmap v1592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:24.557+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:52:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:24.557+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:52:24.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:24.557+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:52:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:25 smithi153 ceph-mon[36569]: pgmap v1593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:25 smithi104 ceph-mon[36624]: pgmap v1593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:25 smithi081 ceph-mon[31453]: pgmap v1593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:27 smithi153 ceph-mon[36569]: pgmap v1594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:27 smithi104 ceph-mon[36624]: pgmap v1594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:27 smithi081 ceph-mon[31453]: pgmap v1594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:29 smithi153 ceph-mon[36569]: pgmap v1595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:29 smithi104 ceph-mon[36624]: pgmap v1595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:29 smithi081 ceph-mon[31453]: pgmap v1595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:31 smithi153 ceph-mon[36569]: pgmap v1596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:31 smithi104 ceph-mon[36624]: pgmap v1596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:31 smithi081 ceph-mon[31453]: pgmap v1596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:33 smithi153 ceph-mon[36569]: pgmap v1597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:33 smithi104 ceph-mon[36624]: pgmap v1597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:33 smithi081 ceph-mon[31453]: pgmap v1597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:35 smithi153 ceph-mon[36569]: pgmap v1598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:35 smithi104 ceph-mon[36624]: pgmap v1598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:35 smithi081 ceph-mon[31453]: pgmap v1598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:52:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:52:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:52:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:52:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:52:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:52:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:37 smithi153 ceph-mon[36569]: pgmap v1599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:37 smithi153 ceph-mon[36569]: pgmap v1600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:37 smithi104 ceph-mon[36624]: pgmap v1599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:37 smithi104 ceph-mon[36624]: pgmap v1600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:37 smithi081 ceph-mon[31453]: pgmap v1599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:37 smithi081 ceph-mon[31453]: pgmap v1600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:39.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:39.573+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:52:39.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:39.573+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:52:39.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:39.573+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:52:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:39 smithi153 ceph-mon[36569]: pgmap v1601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:39 smithi104 ceph-mon[36624]: pgmap v1601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:39 smithi081 ceph-mon[31453]: pgmap v1601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:41 smithi153 ceph-mon[36569]: pgmap v1602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:41 smithi104 ceph-mon[36624]: pgmap v1602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:41 smithi081 ceph-mon[31453]: pgmap v1602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:44.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:43 smithi153 ceph-mon[36569]: pgmap v1603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:43 smithi104 ceph-mon[36624]: pgmap v1603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:43 smithi081 ceph-mon[31453]: pgmap v1603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:45 smithi153 ceph-mon[36569]: pgmap v1604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:45 smithi104 ceph-mon[36624]: pgmap v1604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:45 smithi081 ceph-mon[31453]: pgmap v1604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:47 smithi153 ceph-mon[36569]: pgmap v1605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:47 smithi104 ceph-mon[36624]: pgmap v1605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:47 smithi081 ceph-mon[31453]: pgmap v1605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:49 smithi153 ceph-mon[36569]: pgmap v1606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:49 smithi104 ceph-mon[36624]: pgmap v1606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:49 smithi081 ceph-mon[31453]: pgmap v1606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:51 smithi153 ceph-mon[36569]: pgmap v1607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:51 smithi104 ceph-mon[36624]: pgmap v1607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:51 smithi081 ceph-mon[31453]: pgmap v1607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:53 smithi153 ceph-mon[36569]: pgmap v1608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:53 smithi104 ceph-mon[36624]: pgmap v1608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:54.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:53 smithi081 ceph-mon[31453]: pgmap v1608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:54.588+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:52:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:54.588+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:52:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:52:54.588+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:52:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:55 smithi153 ceph-mon[36569]: pgmap v1609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:55 smithi104 ceph-mon[36624]: pgmap v1609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:55 smithi081 ceph-mon[31453]: pgmap v1609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:57 smithi153 ceph-mon[36569]: pgmap v1610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:57 smithi104 ceph-mon[36624]: pgmap v1610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:52:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:57 smithi081 ceph-mon[31453]: pgmap v1610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:52:59 smithi153 ceph-mon[36569]: pgmap v1611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:52:59 smithi104 ceph-mon[36624]: pgmap v1611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:52:59 smithi081 ceph-mon[31453]: pgmap v1611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:01 smithi153 ceph-mon[36569]: pgmap v1612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:01 smithi104 ceph-mon[36624]: pgmap v1612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:01 smithi081 ceph-mon[31453]: pgmap v1612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:03 smithi153 ceph-mon[36569]: pgmap v1613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:03 smithi104 ceph-mon[36624]: pgmap v1613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:03 smithi081 ceph-mon[31453]: pgmap v1613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:05 smithi153 ceph-mon[36569]: pgmap v1614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:05 smithi104 ceph-mon[36624]: pgmap v1614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:05 smithi081 ceph-mon[31453]: pgmap v1614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:07 smithi153 ceph-mon[36569]: pgmap v1615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:07 smithi104 ceph-mon[36624]: pgmap v1615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:07 smithi081 ceph-mon[31453]: pgmap v1615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:09.594+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:53:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:09.594+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:53:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:09.594+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:53:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:09 smithi153 ceph-mon[36569]: pgmap v1616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:09 smithi104 ceph-mon[36624]: pgmap v1616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:09 smithi081 ceph-mon[31453]: pgmap v1616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:11 smithi104 ceph-mon[36624]: pgmap v1617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:11 smithi081 ceph-mon[31453]: pgmap v1617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:11 smithi153 ceph-mon[36569]: pgmap v1617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:13 smithi104 ceph-mon[36624]: pgmap v1618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:13 smithi081 ceph-mon[31453]: pgmap v1618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:13 smithi153 ceph-mon[36569]: pgmap v1618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:15 smithi104 ceph-mon[36624]: pgmap v1619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:15 smithi081 ceph-mon[31453]: pgmap v1619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:15 smithi153 ceph-mon[36569]: pgmap v1619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:17 smithi104 ceph-mon[36624]: pgmap v1620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:17 smithi081 ceph-mon[31453]: pgmap v1620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:17 smithi153 ceph-mon[36569]: pgmap v1620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:19 smithi104 ceph-mon[36624]: pgmap v1621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:19 smithi081 ceph-mon[31453]: pgmap v1621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:19 smithi153 ceph-mon[36569]: pgmap v1621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:21 smithi081 ceph-mon[31453]: pgmap v1622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:21 smithi153 ceph-mon[36569]: pgmap v1622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:21 smithi104 ceph-mon[36624]: pgmap v1622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:23 smithi081 ceph-mon[31453]: pgmap v1623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:23 smithi153 ceph-mon[36569]: pgmap v1623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:23 smithi104 ceph-mon[36624]: pgmap v1623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:24.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:24.610+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:53:24.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:24.610+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:53:24.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:24.610+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:53:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:25 smithi081 ceph-mon[31453]: pgmap v1624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:25 smithi153 ceph-mon[36569]: pgmap v1624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:25 smithi104 ceph-mon[36624]: pgmap v1624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:27 smithi081 ceph-mon[31453]: pgmap v1625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:27 smithi153 ceph-mon[36569]: pgmap v1625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:27 smithi104 ceph-mon[36624]: pgmap v1625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:30.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:29 smithi153 ceph-mon[36569]: pgmap v1626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:29 smithi081 ceph-mon[31453]: pgmap v1626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:29 smithi104 ceph-mon[36624]: pgmap v1626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:31 smithi081 ceph-mon[31453]: pgmap v1627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:31 smithi153 ceph-mon[36569]: pgmap v1627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:31 smithi104 ceph-mon[36624]: pgmap v1627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:34.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:33 smithi104 ceph-mon[36624]: pgmap v1628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:33 smithi081 ceph-mon[31453]: pgmap v1628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:33 smithi153 ceph-mon[36569]: pgmap v1628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:35 smithi081 ceph-mon[31453]: pgmap v1629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:35 smithi153 ceph-mon[36569]: pgmap v1629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:35 smithi104 ceph-mon[36624]: pgmap v1629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:53:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: pgmap v1630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:53:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: pgmap v1630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:53:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: pgmap v1630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T21:53:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:53:38.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:53:38.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:53:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:38 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:53:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:38 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:53:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:38 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:53:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:38 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:53:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:38 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:53:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:38 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:53:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:38 smithi081 ceph-mon[31453]: pgmap v1631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:38 smithi081 ceph-mon[31453]: pgmap v1632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:38 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:53:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:38 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:53:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:38 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:53:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:38 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:53:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:38 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:53:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:38 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:53:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:38 smithi153 ceph-mon[36569]: pgmap v1631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:38 smithi153 ceph-mon[36569]: pgmap v1632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:38 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T21:53:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:38 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T21:53:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:38 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T21:53:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:38 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T21:53:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:38 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T21:53:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:38 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T21:53:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:38 smithi104 ceph-mon[36624]: pgmap v1631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:38 smithi104 ceph-mon[36624]: pgmap v1632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:39.622+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:53:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:39.622+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:53:39.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:39.622+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:53:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:40 smithi081 ceph-mon[31453]: pgmap v1633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:40 smithi153 ceph-mon[36569]: pgmap v1633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:40 smithi104 ceph-mon[36624]: pgmap v1633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:42 smithi153 ceph-mon[36569]: pgmap v1634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:42 smithi104 ceph-mon[36624]: pgmap v1634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:42 smithi081 ceph-mon[31453]: pgmap v1634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:45.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:44 smithi081 ceph-mon[31453]: pgmap v1635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:44 smithi153 ceph-mon[36569]: pgmap v1635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:44 smithi104 ceph-mon[36624]: pgmap v1635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:46 smithi153 ceph-mon[36569]: pgmap v1636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:46 smithi104 ceph-mon[36624]: pgmap v1636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:46 smithi081 ceph-mon[31453]: pgmap v1636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:48 smithi153 ceph-mon[36569]: pgmap v1637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:48 smithi104 ceph-mon[36624]: pgmap v1637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:48 smithi081 ceph-mon[31453]: pgmap v1637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:50 smithi153 ceph-mon[36569]: pgmap v1638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:51.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:50 smithi104 ceph-mon[36624]: pgmap v1638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:51.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:50 smithi081 ceph-mon[31453]: pgmap v1638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:52 smithi153 ceph-mon[36569]: pgmap v1639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:52 smithi104 ceph-mon[36624]: pgmap v1639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:52 smithi081 ceph-mon[31453]: pgmap v1639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:54.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:54.628+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:53:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:54.628+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:53:54.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:53:54.628+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:53:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:54 smithi153 ceph-mon[36569]: pgmap v1640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:54 smithi104 ceph-mon[36624]: pgmap v1640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:54 smithi081 ceph-mon[31453]: pgmap v1640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:56 smithi153 ceph-mon[36569]: pgmap v1641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:56 smithi104 ceph-mon[36624]: pgmap v1641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:56 smithi081 ceph-mon[31453]: pgmap v1641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:53:58 smithi153 ceph-mon[36569]: pgmap v1642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:53:58 smithi104 ceph-mon[36624]: pgmap v1642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:53:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:53:58 smithi081 ceph-mon[31453]: pgmap v1642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:00 smithi153 ceph-mon[36569]: pgmap v1643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:00 smithi104 ceph-mon[36624]: pgmap v1643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:00 smithi081 ceph-mon[31453]: pgmap v1643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:02 smithi153 ceph-mon[36569]: pgmap v1644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:02 smithi104 ceph-mon[36624]: pgmap v1644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:02 smithi081 ceph-mon[31453]: pgmap v1644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:04 smithi153 ceph-mon[36569]: pgmap v1645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:04 smithi104 ceph-mon[36624]: pgmap v1645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:04 smithi081 ceph-mon[31453]: pgmap v1645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:06 smithi153 ceph-mon[36569]: pgmap v1646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:06 smithi104 ceph-mon[36624]: pgmap v1646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:06 smithi081 ceph-mon[31453]: pgmap v1646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:08 smithi153 ceph-mon[36569]: pgmap v1647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:08 smithi104 ceph-mon[36624]: pgmap v1647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:08 smithi081 ceph-mon[31453]: pgmap v1647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:09.644+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:54:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:09.644+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:54:09.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:09.644+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:54:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:10 smithi153 ceph-mon[36569]: pgmap v1648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:10 smithi104 ceph-mon[36624]: pgmap v1648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:10 smithi081 ceph-mon[31453]: pgmap v1648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:12 smithi153 ceph-mon[36569]: pgmap v1649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:12 smithi104 ceph-mon[36624]: pgmap v1649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:12 smithi081 ceph-mon[31453]: pgmap v1649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:15.305 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:14 smithi081 ceph-mon[31453]: pgmap v1650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:15.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:14 smithi153 ceph-mon[36569]: pgmap v1650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:14 smithi104 ceph-mon[36624]: pgmap v1650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:17.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:16 smithi153 ceph-mon[36569]: pgmap v1651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:16 smithi104 ceph-mon[36624]: pgmap v1651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:16 smithi081 ceph-mon[31453]: pgmap v1651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:19 smithi153 ceph-mon[36569]: pgmap v1652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:19 smithi104 ceph-mon[36624]: pgmap v1652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:19 smithi081 ceph-mon[31453]: pgmap v1652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:21 smithi153 ceph-mon[36569]: pgmap v1653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:21 smithi104 ceph-mon[36624]: pgmap v1653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:21 smithi081 ceph-mon[31453]: pgmap v1653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:23 smithi153 ceph-mon[36569]: pgmap v1654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:23 smithi104 ceph-mon[36624]: pgmap v1654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:23 smithi081 ceph-mon[31453]: pgmap v1654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:24.645+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:54:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:24.645+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:54:24.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:24.645+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:54:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:25 smithi153 ceph-mon[36569]: pgmap v1655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:25 smithi104 ceph-mon[36624]: pgmap v1655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:25 smithi081 ceph-mon[31453]: pgmap v1655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:27 smithi153 ceph-mon[36569]: pgmap v1656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:27 smithi104 ceph-mon[36624]: pgmap v1656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:27 smithi081 ceph-mon[31453]: pgmap v1656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:29 smithi153 ceph-mon[36569]: pgmap v1657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:29 smithi104 ceph-mon[36624]: pgmap v1657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:29.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:29 smithi081 ceph-mon[31453]: pgmap v1657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:30 smithi153 ceph-mon[36569]: pgmap v1658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:30 smithi104 ceph-mon[36624]: pgmap v1658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:30 smithi081 ceph-mon[31453]: pgmap v1658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:33.011 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:32 smithi153 ceph-mon[36569]: pgmap v1659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:33.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:32 smithi104 ceph-mon[36624]: pgmap v1659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:32 smithi081 ceph-mon[31453]: pgmap v1659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:34 smithi153 ceph-mon[36569]: pgmap v1660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:34 smithi104 ceph-mon[36624]: pgmap v1660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:34 smithi081 ceph-mon[31453]: pgmap v1660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:36 smithi153 ceph-mon[36569]: pgmap v1661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:36 smithi104 ceph-mon[36624]: pgmap v1661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:36 smithi081 ceph-mon[31453]: pgmap v1661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:38 smithi153 ceph-mon[36569]: pgmap v1662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:38 smithi104 ceph-mon[36624]: pgmap v1662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:38 smithi081 ceph-mon[31453]: pgmap v1662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:54:39.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:39.648+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:54:39.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:39.648+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:54:39.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:39.648+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:54:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:54:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:54:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:54:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:54:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:54:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:40 smithi153 ceph-mon[36569]: pgmap v1663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:40 smithi153 ceph-mon[36569]: pgmap v1664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:40 smithi104 ceph-mon[36624]: pgmap v1663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:40 smithi104 ceph-mon[36624]: pgmap v1664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:40 smithi081 ceph-mon[31453]: pgmap v1663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:40 smithi081 ceph-mon[31453]: pgmap v1664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:42 smithi153 ceph-mon[36569]: pgmap v1665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:42 smithi104 ceph-mon[36624]: pgmap v1665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:42 smithi081 ceph-mon[31453]: pgmap v1665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:44 smithi153 ceph-mon[36569]: pgmap v1666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:44 smithi104 ceph-mon[36624]: pgmap v1666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:44 smithi081 ceph-mon[31453]: pgmap v1666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:46 smithi153 ceph-mon[36569]: pgmap v1667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:46 smithi104 ceph-mon[36624]: pgmap v1667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:46 smithi081 ceph-mon[31453]: pgmap v1667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:48 smithi104 ceph-mon[36624]: pgmap v1668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:48 smithi081 ceph-mon[31453]: pgmap v1668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:48 smithi153 ceph-mon[36569]: pgmap v1668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:50 smithi104 ceph-mon[36624]: pgmap v1669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:50 smithi081 ceph-mon[31453]: pgmap v1669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:50 smithi153 ceph-mon[36569]: pgmap v1669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:52 smithi104 ceph-mon[36624]: pgmap v1670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:52 smithi081 ceph-mon[31453]: pgmap v1670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:52 smithi153 ceph-mon[36569]: pgmap v1670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:54.663+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:54:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:54.663+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:54:54.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:54:54.663+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:54:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:54 smithi104 ceph-mon[36624]: pgmap v1671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:55.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:54 smithi081 ceph-mon[31453]: pgmap v1671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:54 smithi153 ceph-mon[36569]: pgmap v1671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:56 smithi104 ceph-mon[36624]: pgmap v1672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:56 smithi081 ceph-mon[31453]: pgmap v1672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:56 smithi153 ceph-mon[36569]: pgmap v1672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:54:58 smithi081 ceph-mon[31453]: pgmap v1673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:54:58 smithi153 ceph-mon[36569]: pgmap v1673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:54:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:54:58 smithi104 ceph-mon[36624]: pgmap v1673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:00 smithi081 ceph-mon[31453]: pgmap v1674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:00 smithi153 ceph-mon[36569]: pgmap v1674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:00 smithi104 ceph-mon[36624]: pgmap v1674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:03.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:02 smithi153 ceph-mon[36569]: pgmap v1675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:02 smithi081 ceph-mon[31453]: pgmap v1675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:02 smithi104 ceph-mon[36624]: pgmap v1675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:04 smithi081 ceph-mon[31453]: pgmap v1676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:04 smithi153 ceph-mon[36569]: pgmap v1676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:04 smithi104 ceph-mon[36624]: pgmap v1676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:07.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:06 smithi104 ceph-mon[36624]: pgmap v1677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:06 smithi081 ceph-mon[31453]: pgmap v1677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:06 smithi153 ceph-mon[36569]: pgmap v1677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:08 smithi081 ceph-mon[31453]: pgmap v1678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:08 smithi153 ceph-mon[36569]: pgmap v1678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:08 smithi104 ceph-mon[36624]: pgmap v1678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:09.676+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:55:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:09.676+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:55:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:09.676+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:55:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:10 smithi081 ceph-mon[31453]: pgmap v1679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:10 smithi153 ceph-mon[36569]: pgmap v1679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:10 smithi104 ceph-mon[36624]: pgmap v1679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:13.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:13.076+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:55:13.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:13.076+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:55:13.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:13.106+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:55:13.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:13.107+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:55:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:12 smithi081 ceph-mon[31453]: pgmap v1680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:12 smithi153 ceph-mon[36569]: pgmap v1680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:12 smithi104 ceph-mon[36624]: pgmap v1680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:14 smithi081 ceph-mon[31453]: pgmap v1681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:14 smithi153 ceph-mon[36569]: pgmap v1681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:14 smithi104 ceph-mon[36624]: pgmap v1681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:16 smithi081 ceph-mon[31453]: pgmap v1682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:16 smithi153 ceph-mon[36569]: pgmap v1682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:16 smithi104 ceph-mon[36624]: pgmap v1682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:18 smithi081 ceph-mon[31453]: pgmap v1683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:18 smithi153 ceph-mon[36569]: pgmap v1683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:18 smithi104 ceph-mon[36624]: pgmap v1683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:20 smithi081 ceph-mon[31453]: pgmap v1684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:20 smithi153 ceph-mon[36569]: pgmap v1684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:20 smithi104 ceph-mon[36624]: pgmap v1684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:22 smithi081 ceph-mon[31453]: pgmap v1685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:22 smithi153 ceph-mon[36569]: pgmap v1685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:22 smithi104 ceph-mon[36624]: pgmap v1685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:24.687+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:55:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:24.687+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:55:24.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:24.687+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:55:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:24 smithi081 ceph-mon[31453]: pgmap v1686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:24 smithi153 ceph-mon[36569]: pgmap v1686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:24 smithi104 ceph-mon[36624]: pgmap v1686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:26 smithi081 ceph-mon[31453]: pgmap v1687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:26 smithi153 ceph-mon[36569]: pgmap v1687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:26 smithi104 ceph-mon[36624]: pgmap v1687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:28 smithi153 ceph-mon[36569]: pgmap v1688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:29.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:28 smithi104 ceph-mon[36624]: pgmap v1688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:28 smithi081 ceph-mon[31453]: pgmap v1688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:30 smithi153 ceph-mon[36569]: pgmap v1689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:30 smithi104 ceph-mon[36624]: pgmap v1689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:30 smithi081 ceph-mon[31453]: pgmap v1689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:33.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:32 smithi081 ceph-mon[31453]: pgmap v1690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:32 smithi153 ceph-mon[36569]: pgmap v1690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:32 smithi104 ceph-mon[36624]: pgmap v1690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:34 smithi153 ceph-mon[36569]: pgmap v1691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:34 smithi104 ceph-mon[36624]: pgmap v1691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:34 smithi081 ceph-mon[31453]: pgmap v1691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:36 smithi153 ceph-mon[36569]: pgmap v1692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:36 smithi104 ceph-mon[36624]: pgmap v1692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:36 smithi081 ceph-mon[31453]: pgmap v1692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:38 smithi153 ceph-mon[36569]: pgmap v1693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:38 smithi104 ceph-mon[36624]: pgmap v1693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:38 smithi081 ceph-mon[31453]: pgmap v1693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:39.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:39.701+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:55:39.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:39.701+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:55:39.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:39.701+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:55:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:55:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:40 smithi153 ceph-mon[36569]: pgmap v1694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:55:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:40 smithi104 ceph-mon[36624]: pgmap v1694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:55:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:40 smithi081 ceph-mon[31453]: pgmap v1694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:55:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:55:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:55:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:41 smithi153 ceph-mon[36569]: pgmap v1695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:41 smithi153 ceph-mon[36569]: pgmap v1696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:41 smithi104 ceph-mon[36624]: pgmap v1695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:41 smithi104 ceph-mon[36624]: pgmap v1696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:41 smithi081 ceph-mon[31453]: pgmap v1695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:41 smithi081 ceph-mon[31453]: pgmap v1696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:43 smithi153 ceph-mon[36569]: pgmap v1697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:43 smithi104 ceph-mon[36624]: pgmap v1697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:43 smithi081 ceph-mon[31453]: pgmap v1697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:45 smithi153 ceph-mon[36569]: pgmap v1698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:45 smithi104 ceph-mon[36624]: pgmap v1698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:45 smithi081 ceph-mon[31453]: pgmap v1698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:47 smithi153 ceph-mon[36569]: pgmap v1699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:47 smithi104 ceph-mon[36624]: pgmap v1699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:47 smithi081 ceph-mon[31453]: pgmap v1699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:49 smithi153 ceph-mon[36569]: pgmap v1700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:49 smithi104 ceph-mon[36624]: pgmap v1700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:49 smithi081 ceph-mon[31453]: pgmap v1700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:51 smithi153 ceph-mon[36569]: pgmap v1701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:51 smithi104 ceph-mon[36624]: pgmap v1701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:51 smithi081 ceph-mon[31453]: pgmap v1701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:54 smithi153 ceph-mon[36569]: pgmap v1702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:54 smithi104 ceph-mon[36624]: pgmap v1702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:54 smithi081 ceph-mon[31453]: pgmap v1702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:54.717+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:55:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:54.717+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:55:54.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:55:54.717+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:55:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:56 smithi153 ceph-mon[36569]: pgmap v1703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:56 smithi104 ceph-mon[36624]: pgmap v1703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:56 smithi081 ceph-mon[31453]: pgmap v1703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:55:58 smithi153 ceph-mon[36569]: pgmap v1704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:55:58 smithi104 ceph-mon[36624]: pgmap v1704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:55:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:55:58 smithi081 ceph-mon[31453]: pgmap v1704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:00 smithi153 ceph-mon[36569]: pgmap v1705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:00 smithi104 ceph-mon[36624]: pgmap v1705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:00 smithi081 ceph-mon[31453]: pgmap v1705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:02 smithi153 ceph-mon[36569]: pgmap v1706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:02 smithi104 ceph-mon[36624]: pgmap v1706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:02 smithi081 ceph-mon[31453]: pgmap v1706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:04 smithi081 ceph-mon[31453]: pgmap v1707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:04 smithi153 ceph-mon[36569]: pgmap v1707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:04 smithi104 ceph-mon[36624]: pgmap v1707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:06 smithi153 ceph-mon[36569]: pgmap v1708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:06 smithi104 ceph-mon[36624]: pgmap v1708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:06 smithi081 ceph-mon[31453]: pgmap v1708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:08 smithi153 ceph-mon[36569]: pgmap v1709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:08 smithi104 ceph-mon[36624]: pgmap v1709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:08 smithi081 ceph-mon[31453]: pgmap v1709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:09.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:09.718+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:56:09.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:09.718+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:56:09.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:09.718+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:56:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:10 smithi153 ceph-mon[36569]: pgmap v1710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:10 smithi104 ceph-mon[36624]: pgmap v1710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:10 smithi081 ceph-mon[31453]: pgmap v1710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:11 smithi081 ceph-mon[31453]: pgmap v1711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:11.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:11 smithi153 ceph-mon[36569]: pgmap v1711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:11 smithi104 ceph-mon[36624]: pgmap v1711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:13 smithi104 ceph-mon[36624]: pgmap v1712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:13 smithi081 ceph-mon[31453]: pgmap v1712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:13 smithi153 ceph-mon[36569]: pgmap v1712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:15 smithi104 ceph-mon[36624]: pgmap v1713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:15 smithi081 ceph-mon[31453]: pgmap v1713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:15 smithi153 ceph-mon[36569]: pgmap v1713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:17 smithi081 ceph-mon[31453]: pgmap v1714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:17 smithi153 ceph-mon[36569]: pgmap v1714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:17 smithi104 ceph-mon[36624]: pgmap v1714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:19.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:19 smithi081 ceph-mon[31453]: pgmap v1715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:19 smithi153 ceph-mon[36569]: pgmap v1715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:19 smithi104 ceph-mon[36624]: pgmap v1715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:21 smithi104 ceph-mon[36624]: pgmap v1716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:21 smithi081 ceph-mon[31453]: pgmap v1716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:21 smithi153 ceph-mon[36569]: pgmap v1716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:23 smithi081 ceph-mon[31453]: pgmap v1717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:23 smithi153 ceph-mon[36569]: pgmap v1717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:23 smithi104 ceph-mon[36624]: pgmap v1717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:24.731+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:56:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:24.731+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:56:24.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:24.731+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:56:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:25 smithi081 ceph-mon[31453]: pgmap v1718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:25 smithi153 ceph-mon[36569]: pgmap v1718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:25 smithi104 ceph-mon[36624]: pgmap v1718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:27 smithi081 ceph-mon[31453]: pgmap v1719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:27 smithi153 ceph-mon[36569]: pgmap v1719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:27 smithi104 ceph-mon[36624]: pgmap v1719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:29 smithi081 ceph-mon[31453]: pgmap v1720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:29 smithi153 ceph-mon[36569]: pgmap v1720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:29 smithi104 ceph-mon[36624]: pgmap v1720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:31 smithi081 ceph-mon[31453]: pgmap v1721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:31 smithi153 ceph-mon[36569]: pgmap v1721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:31 smithi104 ceph-mon[36624]: pgmap v1721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:33 smithi081 ceph-mon[31453]: pgmap v1722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:33 smithi153 ceph-mon[36569]: pgmap v1722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:33 smithi104 ceph-mon[36624]: pgmap v1722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:35 smithi081 ceph-mon[31453]: pgmap v1723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:35 smithi153 ceph-mon[36569]: pgmap v1723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:35 smithi104 ceph-mon[36624]: pgmap v1723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:38.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:37 smithi104 ceph-mon[36624]: pgmap v1724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:37 smithi081 ceph-mon[31453]: pgmap v1724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:37 smithi153 ceph-mon[36569]: pgmap v1724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:39.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:39.748+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:56:39.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:39.748+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:56:39.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:39.748+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:56:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:39 smithi081 ceph-mon[31453]: pgmap v1725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:39 smithi153 ceph-mon[36569]: pgmap v1725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:39 smithi104 ceph-mon[36624]: pgmap v1725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:41 smithi081 ceph-mon[31453]: pgmap v1726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:56:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:41 smithi153 ceph-mon[36569]: pgmap v1726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:56:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:41 smithi104 ceph-mon[36624]: pgmap v1726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:56:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:56:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:56:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:56:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:56:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:56:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:43 smithi081 ceph-mon[31453]: pgmap v1727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:43 smithi081 ceph-mon[31453]: pgmap v1728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:43 smithi153 ceph-mon[36569]: pgmap v1727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:43 smithi153 ceph-mon[36569]: pgmap v1728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:43 smithi104 ceph-mon[36624]: pgmap v1727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:43 smithi104 ceph-mon[36624]: pgmap v1728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:46.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:45 smithi081 ceph-mon[31453]: pgmap v1729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:45 smithi153 ceph-mon[36569]: pgmap v1729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:45 smithi104 ceph-mon[36624]: pgmap v1729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:47 smithi081 ceph-mon[31453]: pgmap v1730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:47 smithi153 ceph-mon[36569]: pgmap v1730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:47 smithi104 ceph-mon[36624]: pgmap v1730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:50.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:49 smithi153 ceph-mon[36569]: pgmap v1731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:49 smithi104 ceph-mon[36624]: pgmap v1731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:49 smithi081 ceph-mon[31453]: pgmap v1731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:51 smithi153 ceph-mon[36569]: pgmap v1732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:51 smithi104 ceph-mon[36624]: pgmap v1732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:51 smithi081 ceph-mon[31453]: pgmap v1732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:53 smithi153 ceph-mon[36569]: pgmap v1733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:53 smithi104 ceph-mon[36624]: pgmap v1733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:53 smithi081 ceph-mon[31453]: pgmap v1733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:54.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:54.763+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:56:54.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:54.763+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:56:54.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:56:54.763+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:56:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:55 smithi153 ceph-mon[36569]: pgmap v1734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:55 smithi104 ceph-mon[36624]: pgmap v1734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:55 smithi081 ceph-mon[31453]: pgmap v1734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:57 smithi153 ceph-mon[36569]: pgmap v1735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:57 smithi104 ceph-mon[36624]: pgmap v1735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:56:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:57 smithi081 ceph-mon[31453]: pgmap v1735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:56:59 smithi153 ceph-mon[36569]: pgmap v1736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:56:59 smithi104 ceph-mon[36624]: pgmap v1736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:56:59 smithi081 ceph-mon[31453]: pgmap v1736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:01 smithi153 ceph-mon[36569]: pgmap v1737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:01 smithi104 ceph-mon[36624]: pgmap v1737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:01 smithi081 ceph-mon[31453]: pgmap v1737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:03 smithi153 ceph-mon[36569]: pgmap v1738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:03 smithi104 ceph-mon[36624]: pgmap v1738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:03 smithi081 ceph-mon[31453]: pgmap v1738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:05 smithi153 ceph-mon[36569]: pgmap v1739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:05 smithi104 ceph-mon[36624]: pgmap v1739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:05 smithi081 ceph-mon[31453]: pgmap v1739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:07 smithi153 ceph-mon[36569]: pgmap v1740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:07 smithi104 ceph-mon[36624]: pgmap v1740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:07 smithi081 ceph-mon[31453]: pgmap v1740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:09.773+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:57:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:09.773+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:57:09.775 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:09.773+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:57:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:09 smithi153 ceph-mon[36569]: pgmap v1741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:09 smithi104 ceph-mon[36624]: pgmap v1741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:09 smithi081 ceph-mon[31453]: pgmap v1741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:11 smithi153 ceph-mon[36569]: pgmap v1742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:11 smithi104 ceph-mon[36624]: pgmap v1742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:11 smithi081 ceph-mon[31453]: pgmap v1742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:14.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:13 smithi153 ceph-mon[36569]: pgmap v1743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:13 smithi104 ceph-mon[36624]: pgmap v1743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:13 smithi081 ceph-mon[31453]: pgmap v1743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:15 smithi153 ceph-mon[36569]: pgmap v1744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:15 smithi104 ceph-mon[36624]: pgmap v1744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:15 smithi081 ceph-mon[31453]: pgmap v1744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:17 smithi153 ceph-mon[36569]: pgmap v1745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:17 smithi104 ceph-mon[36624]: pgmap v1745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:18.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:17 smithi081 ceph-mon[31453]: pgmap v1745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:19 smithi153 ceph-mon[36569]: pgmap v1746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:19 smithi104 ceph-mon[36624]: pgmap v1746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:19 smithi081 ceph-mon[31453]: pgmap v1746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:21 smithi153 ceph-mon[36569]: pgmap v1747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:21 smithi104 ceph-mon[36624]: pgmap v1747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:21 smithi081 ceph-mon[31453]: pgmap v1747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:23 smithi153 ceph-mon[36569]: pgmap v1748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:23 smithi104 ceph-mon[36624]: pgmap v1748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:23 smithi081 ceph-mon[31453]: pgmap v1748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:24.788+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:57:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:24.788+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:57:24.791 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:24.788+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:57:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:26 smithi153 ceph-mon[36569]: pgmap v1749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:26 smithi104 ceph-mon[36624]: pgmap v1749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:26 smithi081 ceph-mon[31453]: pgmap v1749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:28.098+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:57:28.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:28.098+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:57:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:28.127+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:57:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:28.128+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:57:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:28 smithi153 ceph-mon[36569]: pgmap v1750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:28 smithi104 ceph-mon[36624]: pgmap v1750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:28 smithi081 ceph-mon[31453]: pgmap v1750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:30 smithi153 ceph-mon[36569]: pgmap v1751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:30 smithi104 ceph-mon[36624]: pgmap v1751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:30 smithi081 ceph-mon[31453]: pgmap v1751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:32 smithi153 ceph-mon[36569]: pgmap v1752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:32 smithi104 ceph-mon[36624]: pgmap v1752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:32 smithi081 ceph-mon[31453]: pgmap v1752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:34 smithi153 ceph-mon[36569]: pgmap v1753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:34 smithi104 ceph-mon[36624]: pgmap v1753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:34 smithi081 ceph-mon[31453]: pgmap v1753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:36 smithi153 ceph-mon[36569]: pgmap v1754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:36 smithi104 ceph-mon[36624]: pgmap v1754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:36 smithi081 ceph-mon[31453]: pgmap v1754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:38 smithi153 ceph-mon[36569]: pgmap v1755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:38 smithi104 ceph-mon[36624]: pgmap v1755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:38 smithi081 ceph-mon[31453]: pgmap v1755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:39.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:39.804+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:57:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:39.804+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:57:39.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:39.804+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:57:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:40 smithi153 ceph-mon[36569]: pgmap v1756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:40 smithi104 ceph-mon[36624]: pgmap v1756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:40 smithi081 ceph-mon[31453]: pgmap v1756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:42 smithi153 ceph-mon[36569]: pgmap v1757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:42 smithi104 ceph-mon[36624]: pgmap v1757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:42 smithi081 ceph-mon[31453]: pgmap v1757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:57:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:57:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:57:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:44 smithi153 ceph-mon[36569]: pgmap v1758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:44 smithi104 ceph-mon[36624]: pgmap v1758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:44 smithi081 ceph-mon[31453]: pgmap v1758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:46 smithi104 ceph-mon[36624]: pgmap v1759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:46 smithi081 ceph-mon[31453]: pgmap v1759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:46 smithi153 ceph-mon[36569]: pgmap v1759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:48 smithi104 ceph-mon[36624]: pgmap v1760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:48 smithi081 ceph-mon[31453]: pgmap v1760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:48 smithi153 ceph-mon[36569]: pgmap v1760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:49 smithi153 ceph-mon[36569]: pgmap v1761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:49 smithi104 ceph-mon[36624]: pgmap v1761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:49 smithi081 ceph-mon[31453]: pgmap v1761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:50 smithi153 ceph-mon[36569]: pgmap v1762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:50 smithi153 ceph-mon[36569]: pgmap v1763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:50 smithi104 ceph-mon[36624]: pgmap v1762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:50 smithi104 ceph-mon[36624]: pgmap v1763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:57:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:57:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:50 smithi081 ceph-mon[31453]: pgmap v1762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:50 smithi081 ceph-mon[31453]: pgmap v1763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:57:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:52 smithi153 ceph-mon[36569]: pgmap v1764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:52.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:52 smithi104 ceph-mon[36624]: pgmap v1764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:52 smithi081 ceph-mon[31453]: pgmap v1764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:54 smithi153 ceph-mon[36569]: pgmap v1765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:54.820+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:57:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:54.820+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:57:54.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:57:54.820+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:54 smithi104 ceph-mon[36624]: pgmap v1765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:54 smithi081 ceph-mon[31453]: pgmap v1765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:56 smithi104 ceph-mon[36624]: pgmap v1766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:56.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:56 smithi081 ceph-mon[31453]: pgmap v1766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:56 smithi153 ceph-mon[36569]: pgmap v1766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:57:58 smithi104 ceph-mon[36624]: pgmap v1767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:57:58 smithi081 ceph-mon[31453]: pgmap v1767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:57:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:57:58 smithi153 ceph-mon[36569]: pgmap v1767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:00.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:00 smithi104 ceph-mon[36624]: pgmap v1768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:00 smithi081 ceph-mon[31453]: pgmap v1768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:00 smithi153 ceph-mon[36569]: pgmap v1768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:02 smithi104 ceph-mon[36624]: pgmap v1769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:02 smithi081 ceph-mon[31453]: pgmap v1769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:02 smithi153 ceph-mon[36569]: pgmap v1769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:04.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:04 smithi104 ceph-mon[36624]: pgmap v1770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:04.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:04 smithi153 ceph-mon[36569]: pgmap v1770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:04.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:04 smithi081 ceph-mon[31453]: pgmap v1770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:06 smithi081 ceph-mon[31453]: pgmap v1771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:06 smithi153 ceph-mon[36569]: pgmap v1771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:06 smithi104 ceph-mon[36624]: pgmap v1771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:08.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:08 smithi104 ceph-mon[36624]: pgmap v1772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:08 smithi081 ceph-mon[31453]: pgmap v1772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:08 smithi153 ceph-mon[36569]: pgmap v1772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:09.824+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:58:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:09.824+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:58:09.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:09.824+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:58:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:10 smithi081 ceph-mon[31453]: pgmap v1773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:10 smithi153 ceph-mon[36569]: pgmap v1773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:10 smithi104 ceph-mon[36624]: pgmap v1773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:12 smithi081 ceph-mon[31453]: pgmap v1774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:12 smithi153 ceph-mon[36569]: pgmap v1774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:12 smithi104 ceph-mon[36624]: pgmap v1774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:13.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:13.105+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:58:13.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:13.105+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T21:58:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:13.133+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:58:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:13.133+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T21:58:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:14 smithi081 ceph-mon[31453]: pgmap v1775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:14 smithi153 ceph-mon[36569]: pgmap v1775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:14 smithi104 ceph-mon[36624]: pgmap v1775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:16 smithi081 ceph-mon[31453]: pgmap v1776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:16 smithi153 ceph-mon[36569]: pgmap v1776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:16 smithi104 ceph-mon[36624]: pgmap v1776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:18 smithi081 ceph-mon[31453]: pgmap v1777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:18 smithi153 ceph-mon[36569]: pgmap v1777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:18 smithi104 ceph-mon[36624]: pgmap v1777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:20 smithi081 ceph-mon[31453]: pgmap v1778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:20 smithi153 ceph-mon[36569]: pgmap v1778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:20 smithi104 ceph-mon[36624]: pgmap v1778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:22 smithi081 ceph-mon[31453]: pgmap v1779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:22 smithi153 ceph-mon[36569]: pgmap v1779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:22 smithi104 ceph-mon[36624]: pgmap v1779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:24.826+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:58:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:24.826+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:58:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:24.826+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:58:24.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:24 smithi081 ceph-mon[31453]: pgmap v1780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:24 smithi153 ceph-mon[36569]: pgmap v1780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:24 smithi104 ceph-mon[36624]: pgmap v1780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:26 smithi081 ceph-mon[31453]: pgmap v1781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:26 smithi153 ceph-mon[36569]: pgmap v1781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:26 smithi104 ceph-mon[36624]: pgmap v1781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:28 smithi081 ceph-mon[31453]: pgmap v1782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:28 smithi153 ceph-mon[36569]: pgmap v1782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:28 smithi104 ceph-mon[36624]: pgmap v1782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:30 smithi081 ceph-mon[31453]: pgmap v1783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:30 smithi153 ceph-mon[36569]: pgmap v1783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:30 smithi104 ceph-mon[36624]: pgmap v1783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:32 smithi081 ceph-mon[31453]: pgmap v1784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:32 smithi153 ceph-mon[36569]: pgmap v1784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:32 smithi104 ceph-mon[36624]: pgmap v1784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:34 smithi081 ceph-mon[31453]: pgmap v1785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:34 smithi153 ceph-mon[36569]: pgmap v1785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:34 smithi104 ceph-mon[36624]: pgmap v1785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:36 smithi153 ceph-mon[36569]: pgmap v1786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:36 smithi104 ceph-mon[36624]: pgmap v1786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:36 smithi081 ceph-mon[31453]: pgmap v1786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:38.985 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:38 smithi081 ceph-mon[31453]: pgmap v1787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:38 smithi153 ceph-mon[36569]: pgmap v1787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:38 smithi104 ceph-mon[36624]: pgmap v1787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:39.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:39.835+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:58:39.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:39.835+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:58:39.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:39.835+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:58:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:40 smithi153 ceph-mon[36569]: pgmap v1788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:40 smithi104 ceph-mon[36624]: pgmap v1788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:40 smithi081 ceph-mon[31453]: pgmap v1788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:42 smithi153 ceph-mon[36569]: pgmap v1789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:42 smithi104 ceph-mon[36624]: pgmap v1789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:42 smithi081 ceph-mon[31453]: pgmap v1789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:44 smithi153 ceph-mon[36569]: pgmap v1790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:44 smithi104 ceph-mon[36624]: pgmap v1790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:44 smithi081 ceph-mon[31453]: pgmap v1790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:46 smithi153 ceph-mon[36569]: pgmap v1791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:46 smithi104 ceph-mon[36624]: pgmap v1791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:46 smithi081 ceph-mon[31453]: pgmap v1791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:48.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:48 smithi081 ceph-mon[31453]: pgmap v1792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:48 smithi153 ceph-mon[36569]: pgmap v1792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:48 smithi104 ceph-mon[36624]: pgmap v1792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:50 smithi153 ceph-mon[36569]: pgmap v1793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:58:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:50 smithi104 ceph-mon[36624]: pgmap v1793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:58:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:50 smithi081 ceph-mon[31453]: pgmap v1793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:58:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:58:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:58:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:58:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:58:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:58:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:58:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:58:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:52 smithi153 ceph-mon[36569]: pgmap v1794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:52 smithi153 ceph-mon[36569]: pgmap v1795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:52 smithi104 ceph-mon[36624]: pgmap v1794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:52 smithi104 ceph-mon[36624]: pgmap v1795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:52 smithi081 ceph-mon[31453]: pgmap v1794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:52 smithi081 ceph-mon[31453]: pgmap v1795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:54.850+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:58:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:54.850+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:58:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:58:54.850+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:58:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:54 smithi153 ceph-mon[36569]: pgmap v1796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:54 smithi104 ceph-mon[36624]: pgmap v1796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:54 smithi081 ceph-mon[31453]: pgmap v1796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:56 smithi153 ceph-mon[36569]: pgmap v1797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:56 smithi104 ceph-mon[36624]: pgmap v1797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:56 smithi081 ceph-mon[31453]: pgmap v1797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:58:58 smithi153 ceph-mon[36569]: pgmap v1798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:58:58 smithi104 ceph-mon[36624]: pgmap v1798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:58:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:58:58 smithi081 ceph-mon[31453]: pgmap v1798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:00 smithi153 ceph-mon[36569]: pgmap v1799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:00 smithi104 ceph-mon[36624]: pgmap v1799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:00 smithi081 ceph-mon[31453]: pgmap v1799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:02 smithi153 ceph-mon[36569]: pgmap v1800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:02 smithi104 ceph-mon[36624]: pgmap v1800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:02 smithi081 ceph-mon[31453]: pgmap v1800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:04 smithi153 ceph-mon[36569]: pgmap v1801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:04 smithi104 ceph-mon[36624]: pgmap v1801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:04 smithi081 ceph-mon[31453]: pgmap v1801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:06 smithi153 ceph-mon[36569]: pgmap v1802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:06 smithi104 ceph-mon[36624]: pgmap v1802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:06 smithi081 ceph-mon[31453]: pgmap v1802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:08 smithi153 ceph-mon[36569]: pgmap v1803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:08 smithi104 ceph-mon[36624]: pgmap v1803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:08 smithi081 ceph-mon[31453]: pgmap v1803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:09.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:09.866+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:59:09.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:09.866+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:59:09.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:09.866+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:59:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:10 smithi153 ceph-mon[36569]: pgmap v1804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:10 smithi104 ceph-mon[36624]: pgmap v1804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:10 smithi081 ceph-mon[31453]: pgmap v1804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:12 smithi153 ceph-mon[36569]: pgmap v1805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:12 smithi104 ceph-mon[36624]: pgmap v1805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:12 smithi081 ceph-mon[31453]: pgmap v1805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:14 smithi153 ceph-mon[36569]: pgmap v1806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:14 smithi104 ceph-mon[36624]: pgmap v1806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:14 smithi081 ceph-mon[31453]: pgmap v1806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:16 smithi153 ceph-mon[36569]: pgmap v1807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:16 smithi104 ceph-mon[36624]: pgmap v1807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:16 smithi081 ceph-mon[31453]: pgmap v1807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:18 smithi153 ceph-mon[36569]: pgmap v1808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:18 smithi104 ceph-mon[36624]: pgmap v1808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:18 smithi081 ceph-mon[31453]: pgmap v1808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:20 smithi153 ceph-mon[36569]: pgmap v1809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:20 smithi104 ceph-mon[36624]: pgmap v1809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:20 smithi081 ceph-mon[31453]: pgmap v1809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:22 smithi153 ceph-mon[36569]: pgmap v1810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:22 smithi104 ceph-mon[36624]: pgmap v1810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:22 smithi081 ceph-mon[31453]: pgmap v1810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:24.867+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:59:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:24.867+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:59:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:24.867+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:59:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:24 smithi153 ceph-mon[36569]: pgmap v1811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:24 smithi104 ceph-mon[36624]: pgmap v1811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:24 smithi081 ceph-mon[31453]: pgmap v1811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:26 smithi104 ceph-mon[36624]: pgmap v1812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:26 smithi081 ceph-mon[31453]: pgmap v1812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:26 smithi153 ceph-mon[36569]: pgmap v1812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:28 smithi104 ceph-mon[36624]: pgmap v1813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:28 smithi081 ceph-mon[31453]: pgmap v1813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:28 smithi153 ceph-mon[36569]: pgmap v1813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:30 smithi104 ceph-mon[36624]: pgmap v1814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:30 smithi081 ceph-mon[31453]: pgmap v1814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:30 smithi153 ceph-mon[36569]: pgmap v1814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:32 smithi104 ceph-mon[36624]: pgmap v1815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:32 smithi081 ceph-mon[31453]: pgmap v1815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:32 smithi153 ceph-mon[36569]: pgmap v1815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:34 smithi104 ceph-mon[36624]: pgmap v1816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:35.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:34 smithi153 ceph-mon[36569]: pgmap v1816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:34 smithi081 ceph-mon[31453]: pgmap v1816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:36 smithi081 ceph-mon[31453]: pgmap v1817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:36 smithi153 ceph-mon[36569]: pgmap v1817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:36 smithi104 ceph-mon[36624]: pgmap v1817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:39.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:38 smithi104 ceph-mon[36624]: pgmap v1818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:38 smithi081 ceph-mon[31453]: pgmap v1818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:38 smithi153 ceph-mon[36569]: pgmap v1818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:39.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:39.880+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:59:39.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:39.880+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:59:39.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:39.880+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:59:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:40 smithi104 ceph-mon[36624]: pgmap v1819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:40 smithi081 ceph-mon[31453]: pgmap v1819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:40 smithi153 ceph-mon[36569]: pgmap v1819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:42 smithi081 ceph-mon[31453]: pgmap v1820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:42 smithi153 ceph-mon[36569]: pgmap v1820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:42 smithi104 ceph-mon[36624]: pgmap v1820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:44 smithi081 ceph-mon[31453]: pgmap v1821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:44 smithi153 ceph-mon[36569]: pgmap v1821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:44 smithi104 ceph-mon[36624]: pgmap v1821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:46 smithi081 ceph-mon[31453]: pgmap v1822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:46 smithi153 ceph-mon[36569]: pgmap v1822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:46 smithi104 ceph-mon[36624]: pgmap v1822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:48 smithi081 ceph-mon[31453]: pgmap v1823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:48 smithi153 ceph-mon[36569]: pgmap v1823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:48 smithi104 ceph-mon[36624]: pgmap v1823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:50 smithi104 ceph-mon[36624]: pgmap v1824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:50 smithi081 ceph-mon[31453]: pgmap v1824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:50 smithi153 ceph-mon[36569]: pgmap v1824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:59:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:59:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T21:59:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:52 smithi081 ceph-mon[31453]: pgmap v1825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:59:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:59:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:59:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:52 smithi153 ceph-mon[36569]: pgmap v1825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:59:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:59:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:59:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:52 smithi104 ceph-mon[36624]: pgmap v1825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T21:59:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T21:59:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T21:59:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:53 smithi081 ceph-mon[31453]: pgmap v1826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:53 smithi081 ceph-mon[31453]: pgmap v1827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:53 smithi153 ceph-mon[36569]: pgmap v1826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:53 smithi153 ceph-mon[36569]: pgmap v1827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:53 smithi104 ceph-mon[36624]: pgmap v1826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:53 smithi104 ceph-mon[36624]: pgmap v1827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:54.894+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T21:59:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:54.894+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T21:59:54.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T21:59:54.894+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T21:59:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:55 smithi081 ceph-mon[31453]: pgmap v1828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:55 smithi153 ceph-mon[36569]: pgmap v1828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:55 smithi104 ceph-mon[36624]: pgmap v1828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:57 smithi081 ceph-mon[31453]: pgmap v1829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:57 smithi153 ceph-mon[36569]: pgmap v1829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T21:59:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:57 smithi104 ceph-mon[36624]: pgmap v1829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 21:59:59 smithi081 ceph-mon[31453]: pgmap v1830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 21:59:59 smithi153 ceph-mon[36569]: pgmap v1830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 21:59:59 smithi104 ceph-mon[36624]: pgmap v1830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:00:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:00:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:00:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:01 smithi153 ceph-mon[36569]: pgmap v1831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:01 smithi104 ceph-mon[36624]: pgmap v1831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:01 smithi081 ceph-mon[31453]: pgmap v1831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:03 smithi153 ceph-mon[36569]: pgmap v1832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:03 smithi104 ceph-mon[36624]: pgmap v1832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:03 smithi081 ceph-mon[31453]: pgmap v1832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:05 smithi153 ceph-mon[36569]: pgmap v1833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:05 smithi104 ceph-mon[36624]: pgmap v1833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:05 smithi081 ceph-mon[31453]: pgmap v1833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:07 smithi153 ceph-mon[36569]: pgmap v1834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:07 smithi104 ceph-mon[36624]: pgmap v1834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:07 smithi081 ceph-mon[31453]: pgmap v1834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:09 smithi153 ceph-mon[36569]: pgmap v1835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:09 smithi104 ceph-mon[36624]: pgmap v1835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:09 smithi081 ceph-mon[31453]: pgmap v1835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:09.901+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:00:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:09.901+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:00:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:09.901+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:00:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:11 smithi104 ceph-mon[36624]: pgmap v1836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:11 smithi081 ceph-mon[31453]: pgmap v1836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:11 smithi153 ceph-mon[36569]: pgmap v1836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:13 smithi104 ceph-mon[36624]: pgmap v1837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:13 smithi081 ceph-mon[31453]: pgmap v1837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:13 smithi153 ceph-mon[36569]: pgmap v1837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:15 smithi081 ceph-mon[31453]: pgmap v1838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:15 smithi153 ceph-mon[36569]: pgmap v1838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:15 smithi104 ceph-mon[36624]: pgmap v1838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:17 smithi081 ceph-mon[31453]: pgmap v1839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:17 smithi153 ceph-mon[36569]: pgmap v1839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:17 smithi104 ceph-mon[36624]: pgmap v1839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:19 smithi153 ceph-mon[36569]: pgmap v1840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:19 smithi104 ceph-mon[36624]: pgmap v1840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:19 smithi081 ceph-mon[31453]: pgmap v1840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:21 smithi104 ceph-mon[36624]: pgmap v1841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:21 smithi081 ceph-mon[31453]: pgmap v1841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:21 smithi153 ceph-mon[36569]: pgmap v1841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:23 smithi081 ceph-mon[31453]: pgmap v1842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:23 smithi153 ceph-mon[36569]: pgmap v1842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:23 smithi104 ceph-mon[36624]: pgmap v1842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:24.913+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:00:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:24.913+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:00:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:24.913+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:00:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:25 smithi081 ceph-mon[31453]: pgmap v1843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:25 smithi153 ceph-mon[36569]: pgmap v1843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:25 smithi104 ceph-mon[36624]: pgmap v1843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:27 smithi081 ceph-mon[31453]: pgmap v1844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:27 smithi153 ceph-mon[36569]: pgmap v1844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:27 smithi104 ceph-mon[36624]: pgmap v1844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:29 smithi081 ceph-mon[31453]: pgmap v1845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:29 smithi153 ceph-mon[36569]: pgmap v1845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:29 smithi104 ceph-mon[36624]: pgmap v1845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:31 smithi081 ceph-mon[31453]: pgmap v1846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:31 smithi153 ceph-mon[36569]: pgmap v1846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:31 smithi104 ceph-mon[36624]: pgmap v1846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:33 smithi081 ceph-mon[31453]: pgmap v1847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:33 smithi153 ceph-mon[36569]: pgmap v1847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:33 smithi104 ceph-mon[36624]: pgmap v1847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:35 smithi081 ceph-mon[31453]: pgmap v1848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:35 smithi153 ceph-mon[36569]: pgmap v1848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:35 smithi104 ceph-mon[36624]: pgmap v1848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:37 smithi081 ceph-mon[31453]: pgmap v1849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:37 smithi153 ceph-mon[36569]: pgmap v1849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:37 smithi104 ceph-mon[36624]: pgmap v1849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:39.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:39.928+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:00:39.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:39.928+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:00:39.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:39.928+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:00:40.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:39 smithi104 ceph-mon[36624]: pgmap v1850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:39 smithi081 ceph-mon[31453]: pgmap v1850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:39 smithi153 ceph-mon[36569]: pgmap v1850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:41 smithi081 ceph-mon[31453]: pgmap v1851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:41 smithi153 ceph-mon[36569]: pgmap v1851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:41 smithi104 ceph-mon[36624]: pgmap v1851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:43 smithi081 ceph-mon[31453]: pgmap v1852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:43 smithi153 ceph-mon[36569]: pgmap v1852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:43 smithi104 ceph-mon[36624]: pgmap v1852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:45 smithi081 ceph-mon[31453]: pgmap v1853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:45 smithi153 ceph-mon[36569]: pgmap v1853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:45 smithi104 ceph-mon[36624]: pgmap v1853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:47 smithi081 ceph-mon[31453]: pgmap v1854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:47 smithi153 ceph-mon[36569]: pgmap v1854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:47 smithi104 ceph-mon[36624]: pgmap v1854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:49 smithi081 ceph-mon[31453]: pgmap v1855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:49 smithi153 ceph-mon[36569]: pgmap v1855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:49 smithi104 ceph-mon[36624]: pgmap v1855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:51 smithi081 ceph-mon[31453]: pgmap v1856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:51 smithi153 ceph-mon[36569]: pgmap v1856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:51 smithi104 ceph-mon[36624]: pgmap v1856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:53 smithi153 ceph-mon[36569]: pgmap v1857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:00:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:53 smithi104 ceph-mon[36624]: pgmap v1857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:00:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:53 smithi081 ceph-mon[31453]: pgmap v1857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:00:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:54.943+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:00:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:54.943+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:00:54.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:00:54.943+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:00:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:00:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:00:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:00:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:00:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:00:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:55 smithi153 ceph-mon[36569]: pgmap v1858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:55 smithi153 ceph-mon[36569]: pgmap v1859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:55 smithi104 ceph-mon[36624]: pgmap v1858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:55 smithi104 ceph-mon[36624]: pgmap v1859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:55 smithi081 ceph-mon[31453]: pgmap v1858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:55 smithi081 ceph-mon[31453]: pgmap v1859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:58.248 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:57 smithi081 ceph-mon[31453]: pgmap v1860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:57 smithi153 ceph-mon[36569]: pgmap v1860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:00:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:57 smithi104 ceph-mon[36624]: pgmap v1860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:00:59 smithi153 ceph-mon[36569]: pgmap v1861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:00:59 smithi104 ceph-mon[36624]: pgmap v1861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:00:59 smithi081 ceph-mon[31453]: pgmap v1861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:01 smithi153 ceph-mon[36569]: pgmap v1862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:01 smithi104 ceph-mon[36624]: pgmap v1862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:01 smithi081 ceph-mon[31453]: pgmap v1862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:03 smithi153 ceph-mon[36569]: pgmap v1863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:03 smithi104 ceph-mon[36624]: pgmap v1863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:03 smithi081 ceph-mon[31453]: pgmap v1863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:05 smithi153 ceph-mon[36569]: pgmap v1864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:05 smithi104 ceph-mon[36624]: pgmap v1864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:05 smithi081 ceph-mon[31453]: pgmap v1864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:07 smithi153 ceph-mon[36569]: pgmap v1865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:07 smithi104 ceph-mon[36624]: pgmap v1865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:07 smithi081 ceph-mon[31453]: pgmap v1865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:09.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:09.954+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:01:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:09.954+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:01:09.956 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:09.954+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:01:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:09 smithi153 ceph-mon[36569]: pgmap v1866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:09 smithi104 ceph-mon[36624]: pgmap v1866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:09 smithi081 ceph-mon[31453]: pgmap v1866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.092+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 msgr2=0x7f16e80a4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T22:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.092+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 msgr2=0x7f16e80a4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T22:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.092+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 0x7f16e80a4850 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f16d40034a0 tx=0x7f16f40bd040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T22:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.092+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 0x7f16e80a4850 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f16d40034a0 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-12T22:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.092+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 msgr2=0x7f16e80a4850 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-12T22:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.092+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 msgr2=0x7f16e80a4850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T22:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.092+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4470 0x7f16e80a4850 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T22:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.092+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T22:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.093+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T22:01:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:12.094+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 0x7f16e80b1520 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f40d1000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T22:01:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:11 smithi153 ceph-mon[36569]: pgmap v1867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:11 smithi104 ceph-mon[36624]: pgmap v1867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:11 smithi081 ceph-mon[31453]: pgmap v1867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:13.139+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:01:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:13.139+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:01:13.162 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:13.161+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:01:13.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:13.162+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:01:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:13 smithi153 ceph-mon[36569]: pgmap v1868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:13 smithi104 ceph-mon[36624]: pgmap v1868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:13 smithi081 ceph-mon[31453]: pgmap v1868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:15 smithi153 ceph-mon[36569]: pgmap v1869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:15 smithi104 ceph-mon[36624]: pgmap v1869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:15 smithi081 ceph-mon[31453]: pgmap v1869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:17 smithi153 ceph-mon[36569]: pgmap v1870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:17 smithi104 ceph-mon[36624]: pgmap v1870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:17 smithi081 ceph-mon[31453]: pgmap v1870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:19 smithi153 ceph-mon[36569]: pgmap v1871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:19 smithi104 ceph-mon[36624]: pgmap v1871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:19 smithi081 ceph-mon[31453]: pgmap v1871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:21 smithi153 ceph-mon[36569]: pgmap v1872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:21 smithi104 ceph-mon[36624]: pgmap v1872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:21 smithi081 ceph-mon[31453]: pgmap v1872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:23 smithi153 ceph-mon[36569]: pgmap v1873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:23 smithi104 ceph-mon[36624]: pgmap v1873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:23 smithi081 ceph-mon[31453]: pgmap v1873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:24.969 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:24.968+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:01:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:24.968+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:01:24.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:24.968+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:01:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:26 smithi153 ceph-mon[36569]: pgmap v1874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:26 smithi104 ceph-mon[36624]: pgmap v1874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:25 smithi081 ceph-mon[31453]: pgmap v1874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:28 smithi153 ceph-mon[36569]: pgmap v1875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:28 smithi104 ceph-mon[36624]: pgmap v1875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:28 smithi081 ceph-mon[31453]: pgmap v1875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:30 smithi153 ceph-mon[36569]: pgmap v1876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:30 smithi104 ceph-mon[36624]: pgmap v1876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:30 smithi081 ceph-mon[31453]: pgmap v1876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:32 smithi153 ceph-mon[36569]: pgmap v1877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:32 smithi104 ceph-mon[36624]: pgmap v1877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:32 smithi081 ceph-mon[31453]: pgmap v1877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:34 smithi153 ceph-mon[36569]: pgmap v1878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:34 smithi104 ceph-mon[36624]: pgmap v1878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:34 smithi081 ceph-mon[31453]: pgmap v1878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:36 smithi153 ceph-mon[36569]: pgmap v1879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:36 smithi104 ceph-mon[36624]: pgmap v1879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:36 smithi081 ceph-mon[31453]: pgmap v1879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:38 smithi153 ceph-mon[36569]: pgmap v1880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:38 smithi104 ceph-mon[36624]: pgmap v1880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:38 smithi081 ceph-mon[31453]: pgmap v1880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:39.985 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:39.983+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:01:39.985 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:39.983+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:01:39.985 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:39.983+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:01:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:40 smithi153 ceph-mon[36569]: pgmap v1881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:40 smithi104 ceph-mon[36624]: pgmap v1881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:40 smithi081 ceph-mon[31453]: pgmap v1881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:42 smithi153 ceph-mon[36569]: pgmap v1882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:42 smithi104 ceph-mon[36624]: pgmap v1882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:42 smithi081 ceph-mon[31453]: pgmap v1882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:44 smithi153 ceph-mon[36569]: pgmap v1883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:44 smithi104 ceph-mon[36624]: pgmap v1883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:44 smithi081 ceph-mon[31453]: pgmap v1883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:46 smithi153 ceph-mon[36569]: pgmap v1884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:46 smithi104 ceph-mon[36624]: pgmap v1884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:46 smithi081 ceph-mon[31453]: pgmap v1884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:48 smithi153 ceph-mon[36569]: pgmap v1885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:48 smithi104 ceph-mon[36624]: pgmap v1885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:48 smithi081 ceph-mon[31453]: pgmap v1885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:50 smithi104 ceph-mon[36624]: pgmap v1886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:50 smithi081 ceph-mon[31453]: pgmap v1886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:50 smithi153 ceph-mon[36569]: pgmap v1886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:52 smithi104 ceph-mon[36624]: pgmap v1887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:52 smithi081 ceph-mon[31453]: pgmap v1887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:52 smithi153 ceph-mon[36569]: pgmap v1887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:54 smithi104 ceph-mon[36624]: pgmap v1888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:54 smithi081 ceph-mon[31453]: pgmap v1888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:54 smithi153 ceph-mon[36569]: pgmap v1888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:54.998+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:01:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:54.998+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:01:54.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:01:54.998+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:01:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:01:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:56 smithi081 ceph-mon[31453]: pgmap v1889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:01:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:01:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:56 smithi153 ceph-mon[36569]: pgmap v1889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:01:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:01:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:56 smithi104 ceph-mon[36624]: pgmap v1889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:01:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:01:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:01:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:57 smithi081 ceph-mon[31453]: pgmap v1890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:57 smithi081 ceph-mon[31453]: pgmap v1891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:57 smithi153 ceph-mon[36569]: pgmap v1890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:57 smithi153 ceph-mon[36569]: pgmap v1891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:57 smithi104 ceph-mon[36624]: pgmap v1890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:57 smithi104 ceph-mon[36624]: pgmap v1891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:01:59 smithi081 ceph-mon[31453]: pgmap v1892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:01:59 smithi153 ceph-mon[36569]: pgmap v1892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:01:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:01:59 smithi104 ceph-mon[36624]: pgmap v1892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:01 smithi081 ceph-mon[31453]: pgmap v1893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:01 smithi153 ceph-mon[36569]: pgmap v1893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:01 smithi104 ceph-mon[36624]: pgmap v1893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:03 smithi081 ceph-mon[31453]: pgmap v1894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:03 smithi153 ceph-mon[36569]: pgmap v1894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:03 smithi104 ceph-mon[36624]: pgmap v1894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:05 smithi081 ceph-mon[31453]: pgmap v1895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:05 smithi153 ceph-mon[36569]: pgmap v1895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:05 smithi104 ceph-mon[36624]: pgmap v1895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:07 smithi081 ceph-mon[31453]: pgmap v1896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:07 smithi153 ceph-mon[36569]: pgmap v1896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:07 smithi104 ceph-mon[36624]: pgmap v1896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:09 smithi081 ceph-mon[31453]: pgmap v1897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:09 smithi153 ceph-mon[36569]: pgmap v1897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:09 smithi104 ceph-mon[36624]: pgmap v1897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:10.008+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:02:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:10.008+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:02:10.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:10.008+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:02:11.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:11 smithi104 ceph-mon[36624]: pgmap v1898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:11 smithi081 ceph-mon[31453]: pgmap v1898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:11 smithi153 ceph-mon[36569]: pgmap v1898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:12 smithi153 ceph-mon[36569]: pgmap v1899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:12 smithi104 ceph-mon[36624]: pgmap v1899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:12 smithi081 ceph-mon[31453]: pgmap v1899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:14 smithi081 ceph-mon[31453]: pgmap v1900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:14 smithi153 ceph-mon[36569]: pgmap v1900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:14 smithi104 ceph-mon[36624]: pgmap v1900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:16 smithi081 ceph-mon[31453]: pgmap v1901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:16 smithi153 ceph-mon[36569]: pgmap v1901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:16 smithi104 ceph-mon[36624]: pgmap v1901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:18 smithi081 ceph-mon[31453]: pgmap v1902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:18 smithi153 ceph-mon[36569]: pgmap v1902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:18 smithi104 ceph-mon[36624]: pgmap v1902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:20 smithi153 ceph-mon[36569]: pgmap v1903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:20 smithi104 ceph-mon[36624]: pgmap v1903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:20 smithi081 ceph-mon[31453]: pgmap v1903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:22 smithi153 ceph-mon[36569]: pgmap v1904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:22 smithi104 ceph-mon[36624]: pgmap v1904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:22 smithi081 ceph-mon[31453]: pgmap v1904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:25.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:25.018+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:02:25.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:25.018+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:02:25.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:25.018+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:02:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:24 smithi153 ceph-mon[36569]: pgmap v1905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:24 smithi104 ceph-mon[36624]: pgmap v1905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:24 smithi081 ceph-mon[31453]: pgmap v1905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:26 smithi153 ceph-mon[36569]: pgmap v1906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:26 smithi104 ceph-mon[36624]: pgmap v1906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:26 smithi081 ceph-mon[31453]: pgmap v1906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:28 smithi153 ceph-mon[36569]: pgmap v1907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:28 smithi104 ceph-mon[36624]: pgmap v1907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:28 smithi081 ceph-mon[31453]: pgmap v1907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:30 smithi153 ceph-mon[36569]: pgmap v1908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:30 smithi104 ceph-mon[36624]: pgmap v1908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:30 smithi081 ceph-mon[31453]: pgmap v1908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:32 smithi153 ceph-mon[36569]: pgmap v1909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:32 smithi104 ceph-mon[36624]: pgmap v1909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:32 smithi081 ceph-mon[31453]: pgmap v1909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:34 smithi153 ceph-mon[36569]: pgmap v1910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:34 smithi104 ceph-mon[36624]: pgmap v1910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:34 smithi081 ceph-mon[31453]: pgmap v1910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:37.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:36 smithi153 ceph-mon[36569]: pgmap v1911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:36 smithi104 ceph-mon[36624]: pgmap v1911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:36 smithi081 ceph-mon[31453]: pgmap v1911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:38 smithi153 ceph-mon[36569]: pgmap v1912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:38 smithi104 ceph-mon[36624]: pgmap v1912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:38 smithi081 ceph-mon[31453]: pgmap v1912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:40.026+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:02:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:40.026+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:02:40.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:40.026+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:02:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:40 smithi153 ceph-mon[36569]: pgmap v1913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:40 smithi104 ceph-mon[36624]: pgmap v1913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:40 smithi081 ceph-mon[31453]: pgmap v1913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:42 smithi153 ceph-mon[36569]: pgmap v1914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:42 smithi104 ceph-mon[36624]: pgmap v1914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:42 smithi081 ceph-mon[31453]: pgmap v1914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:44 smithi153 ceph-mon[36569]: pgmap v1915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:44 smithi104 ceph-mon[36624]: pgmap v1915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:44 smithi081 ceph-mon[31453]: pgmap v1915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:46 smithi153 ceph-mon[36569]: pgmap v1916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:46 smithi104 ceph-mon[36624]: pgmap v1916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:46 smithi081 ceph-mon[31453]: pgmap v1916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:48 smithi153 ceph-mon[36569]: pgmap v1917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:48 smithi104 ceph-mon[36624]: pgmap v1917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:48 smithi081 ceph-mon[31453]: pgmap v1917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:50 smithi153 ceph-mon[36569]: pgmap v1918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:50 smithi104 ceph-mon[36624]: pgmap v1918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:50 smithi081 ceph-mon[31453]: pgmap v1918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:52 smithi153 ceph-mon[36569]: pgmap v1919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:52 smithi104 ceph-mon[36624]: pgmap v1919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:52 smithi081 ceph-mon[31453]: pgmap v1919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:55.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:55.041+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:02:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:55.041+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:02:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:02:55.041+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:02:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:54 smithi153 ceph-mon[36569]: pgmap v1920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:54 smithi104 ceph-mon[36624]: pgmap v1920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:54 smithi081 ceph-mon[31453]: pgmap v1920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:57 smithi153 ceph-mon[36569]: pgmap v1921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:57 smithi104 ceph-mon[36624]: pgmap v1921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:57 smithi081 ceph-mon[31453]: pgmap v1921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:02:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:02:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:02:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:02:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:02:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:02:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:59 smithi153 ceph-mon[36569]: pgmap v1922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:02:59 smithi153 ceph-mon[36569]: pgmap v1923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:59 smithi104 ceph-mon[36624]: pgmap v1922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:02:59 smithi104 ceph-mon[36624]: pgmap v1923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:59 smithi081 ceph-mon[31453]: pgmap v1922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:02:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:02:59 smithi081 ceph-mon[31453]: pgmap v1923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:01 smithi153 ceph-mon[36569]: pgmap v1924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:01 smithi104 ceph-mon[36624]: pgmap v1924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:01 smithi081 ceph-mon[31453]: pgmap v1924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:03 smithi153 ceph-mon[36569]: pgmap v1925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:03 smithi104 ceph-mon[36624]: pgmap v1925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:03 smithi081 ceph-mon[31453]: pgmap v1925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:05 smithi153 ceph-mon[36569]: pgmap v1926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:05 smithi104 ceph-mon[36624]: pgmap v1926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:05 smithi081 ceph-mon[31453]: pgmap v1926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:07 smithi153 ceph-mon[36569]: pgmap v1927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:07 smithi104 ceph-mon[36624]: pgmap v1927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:07 smithi081 ceph-mon[31453]: pgmap v1927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:09 smithi153 ceph-mon[36569]: pgmap v1928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:09 smithi104 ceph-mon[36624]: pgmap v1928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:09 smithi081 ceph-mon[31453]: pgmap v1928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:10.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:10.050+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:03:10.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:10.050+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:03:10.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:10.050+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:03:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:11 smithi153 ceph-mon[36569]: pgmap v1929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:11 smithi104 ceph-mon[36624]: pgmap v1929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:11 smithi081 ceph-mon[31453]: pgmap v1929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:13 smithi153 ceph-mon[36569]: pgmap v1930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:13 smithi104 ceph-mon[36624]: pgmap v1930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:13 smithi081 ceph-mon[31453]: pgmap v1930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:15 smithi104 ceph-mon[36624]: pgmap v1931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:15 smithi081 ceph-mon[31453]: pgmap v1931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:15 smithi153 ceph-mon[36569]: pgmap v1931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:17 smithi104 ceph-mon[36624]: pgmap v1932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:17 smithi081 ceph-mon[31453]: pgmap v1932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:17 smithi153 ceph-mon[36569]: pgmap v1932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:19 smithi104 ceph-mon[36624]: pgmap v1933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:19 smithi081 ceph-mon[31453]: pgmap v1933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:19 smithi153 ceph-mon[36569]: pgmap v1933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:21.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:21 smithi081 ceph-mon[31453]: pgmap v1934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:21 smithi153 ceph-mon[36569]: pgmap v1934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:21 smithi104 ceph-mon[36624]: pgmap v1934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:23 smithi081 ceph-mon[31453]: pgmap v1935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:23 smithi153 ceph-mon[36569]: pgmap v1935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:23 smithi104 ceph-mon[36624]: pgmap v1935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:25.066 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:25.065+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:03:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:25.065+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:03:25.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:25.065+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:03:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:25 smithi081 ceph-mon[31453]: pgmap v1936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:25 smithi153 ceph-mon[36569]: pgmap v1936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:25 smithi104 ceph-mon[36624]: pgmap v1936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:27 smithi081 ceph-mon[31453]: pgmap v1937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:27 smithi153 ceph-mon[36569]: pgmap v1937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:27 smithi104 ceph-mon[36624]: pgmap v1937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:29 smithi081 ceph-mon[31453]: pgmap v1938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:29 smithi153 ceph-mon[36569]: pgmap v1938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:29 smithi104 ceph-mon[36624]: pgmap v1938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:31 smithi081 ceph-mon[31453]: pgmap v1939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:31 smithi153 ceph-mon[36569]: pgmap v1939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:31 smithi104 ceph-mon[36624]: pgmap v1939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:33 smithi081 ceph-mon[31453]: pgmap v1940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:33 smithi153 ceph-mon[36569]: pgmap v1940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:33 smithi104 ceph-mon[36624]: pgmap v1940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:35 smithi081 ceph-mon[31453]: pgmap v1941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:35 smithi153 ceph-mon[36569]: pgmap v1941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:35 smithi104 ceph-mon[36624]: pgmap v1941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:37 smithi081 ceph-mon[31453]: pgmap v1942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:37 smithi153 ceph-mon[36569]: pgmap v1942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:37 smithi104 ceph-mon[36624]: pgmap v1942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:39 smithi081 ceph-mon[31453]: pgmap v1943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:39 smithi153 ceph-mon[36569]: pgmap v1943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:39 smithi104 ceph-mon[36624]: pgmap v1943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:40.071+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:03:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:40.071+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:03:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:40.071+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:03:41.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:41 smithi104 ceph-mon[36624]: pgmap v1944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:41 smithi081 ceph-mon[31453]: pgmap v1944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:41 smithi153 ceph-mon[36569]: pgmap v1944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:43 smithi081 ceph-mon[31453]: pgmap v1945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:43.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:43 smithi153 ceph-mon[36569]: pgmap v1945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:43 smithi104 ceph-mon[36624]: pgmap v1945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:44 smithi153 ceph-mon[36569]: pgmap v1946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:44 smithi104 ceph-mon[36624]: pgmap v1946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:44 smithi081 ceph-mon[31453]: pgmap v1946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:46 smithi153 ceph-mon[36569]: pgmap v1947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:46 smithi104 ceph-mon[36624]: pgmap v1947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:46 smithi081 ceph-mon[31453]: pgmap v1947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:48 smithi153 ceph-mon[36569]: pgmap v1948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:48 smithi104 ceph-mon[36624]: pgmap v1948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:48 smithi081 ceph-mon[31453]: pgmap v1948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:50 smithi153 ceph-mon[36569]: pgmap v1949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:50 smithi104 ceph-mon[36624]: pgmap v1949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:50 smithi081 ceph-mon[31453]: pgmap v1949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:52 smithi153 ceph-mon[36569]: pgmap v1950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:52 smithi104 ceph-mon[36624]: pgmap v1950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:52 smithi081 ceph-mon[31453]: pgmap v1950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:54 smithi104 ceph-mon[36624]: pgmap v1951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:54 smithi081 ceph-mon[31453]: pgmap v1951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:54 smithi153 ceph-mon[36569]: pgmap v1951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:55.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:55.085+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:03:55.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:55.085+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:03:55.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:03:55.085+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:03:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:56 smithi104 ceph-mon[36624]: pgmap v1952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:56 smithi081 ceph-mon[31453]: pgmap v1952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:56 smithi153 ceph-mon[36569]: pgmap v1952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: pgmap v1953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: pgmap v1953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: pgmap v1953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:03:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:03:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:03:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:03:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: pgmap v1954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: pgmap v1955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:03:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:03:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: pgmap v1954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: pgmap v1955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:03:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:04:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:04:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:04:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: pgmap v1954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: pgmap v1955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:03:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:04:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:02 smithi081 ceph-mon[31453]: pgmap v1956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:02 smithi153 ceph-mon[36569]: pgmap v1956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:02 smithi104 ceph-mon[36624]: pgmap v1956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:04 smithi081 ceph-mon[31453]: pgmap v1957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:04 smithi153 ceph-mon[36569]: pgmap v1957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:04 smithi104 ceph-mon[36624]: pgmap v1957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:06 smithi081 ceph-mon[31453]: pgmap v1958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:06 smithi153 ceph-mon[36569]: pgmap v1958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:06 smithi104 ceph-mon[36624]: pgmap v1958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:08.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:08 smithi153 ceph-mon[36569]: pgmap v1959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:08 smithi081 ceph-mon[31453]: pgmap v1959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:08 smithi104 ceph-mon[36624]: pgmap v1959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:10.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:10.101+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:04:10.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:10.101+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:04:10.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:10.101+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:04:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:10 smithi081 ceph-mon[31453]: pgmap v1960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:10 smithi153 ceph-mon[36569]: pgmap v1960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:10 smithi104 ceph-mon[36624]: pgmap v1960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:12.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:12 smithi104 ceph-mon[36624]: pgmap v1961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:12 smithi081 ceph-mon[31453]: pgmap v1961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:12 smithi153 ceph-mon[36569]: pgmap v1961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:13.168+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:04:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:13.168+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:04:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:13.187+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:04:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:13.188+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:04:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:14 smithi081 ceph-mon[31453]: pgmap v1962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:14 smithi153 ceph-mon[36569]: pgmap v1962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:14 smithi104 ceph-mon[36624]: pgmap v1962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:16 smithi081 ceph-mon[31453]: pgmap v1963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:16 smithi153 ceph-mon[36569]: pgmap v1963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:16 smithi104 ceph-mon[36624]: pgmap v1963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:18 smithi081 ceph-mon[31453]: pgmap v1964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:18 smithi153 ceph-mon[36569]: pgmap v1964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:18 smithi104 ceph-mon[36624]: pgmap v1964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:19 smithi153 ceph-mon[36569]: pgmap v1965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:19 smithi104 ceph-mon[36624]: pgmap v1965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:19 smithi081 ceph-mon[31453]: pgmap v1965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:22.360 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:22 smithi081 ceph-mon[31453]: pgmap v1966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:22 smithi153 ceph-mon[36569]: pgmap v1966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:22 smithi104 ceph-mon[36624]: pgmap v1966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:24 smithi081 ceph-mon[31453]: pgmap v1967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:24 smithi153 ceph-mon[36569]: pgmap v1967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:24 smithi104 ceph-mon[36624]: pgmap v1967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:25.104+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:04:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:25.104+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:04:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:25.104+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:04:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:26 smithi081 ceph-mon[31453]: pgmap v1968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:26 smithi153 ceph-mon[36569]: pgmap v1968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:26 smithi104 ceph-mon[36624]: pgmap v1968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:28 smithi081 ceph-mon[31453]: pgmap v1969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:28 smithi153 ceph-mon[36569]: pgmap v1969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:28 smithi104 ceph-mon[36624]: pgmap v1969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:30 smithi081 ceph-mon[31453]: pgmap v1970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:30 smithi153 ceph-mon[36569]: pgmap v1970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:30 smithi104 ceph-mon[36624]: pgmap v1970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:32 smithi081 ceph-mon[31453]: pgmap v1971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:32 smithi153 ceph-mon[36569]: pgmap v1971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:32 smithi104 ceph-mon[36624]: pgmap v1971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:34 smithi081 ceph-mon[31453]: pgmap v1972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:34 smithi153 ceph-mon[36569]: pgmap v1972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:34 smithi104 ceph-mon[36624]: pgmap v1972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:36 smithi081 ceph-mon[31453]: pgmap v1973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:36 smithi153 ceph-mon[36569]: pgmap v1973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:36 smithi104 ceph-mon[36624]: pgmap v1973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:38 smithi081 ceph-mon[31453]: pgmap v1974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:38 smithi153 ceph-mon[36569]: pgmap v1974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:38 smithi104 ceph-mon[36624]: pgmap v1974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:40.105+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:04:40.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:40.105+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:04:40.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:40.105+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:04:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:40 smithi081 ceph-mon[31453]: pgmap v1975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:40 smithi153 ceph-mon[36569]: pgmap v1975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:40 smithi104 ceph-mon[36624]: pgmap v1975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:41 smithi153 ceph-mon[36569]: pgmap v1976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:41 smithi104 ceph-mon[36624]: pgmap v1976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:41 smithi081 ceph-mon[31453]: pgmap v1976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:44 smithi081 ceph-mon[31453]: pgmap v1977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:44 smithi153 ceph-mon[36569]: pgmap v1977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:44 smithi104 ceph-mon[36624]: pgmap v1977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:46 smithi081 ceph-mon[31453]: pgmap v1978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:46 smithi153 ceph-mon[36569]: pgmap v1978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:46 smithi104 ceph-mon[36624]: pgmap v1978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:48 smithi081 ceph-mon[31453]: pgmap v1979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:48 smithi153 ceph-mon[36569]: pgmap v1979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:48 smithi104 ceph-mon[36624]: pgmap v1979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:50 smithi081 ceph-mon[31453]: pgmap v1980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:50 smithi153 ceph-mon[36569]: pgmap v1980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:50 smithi104 ceph-mon[36624]: pgmap v1980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:52 smithi081 ceph-mon[31453]: pgmap v1981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:52 smithi153 ceph-mon[36569]: pgmap v1981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:52 smithi104 ceph-mon[36624]: pgmap v1981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:54 smithi081 ceph-mon[31453]: pgmap v1982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:54 smithi153 ceph-mon[36569]: pgmap v1982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:54 smithi104 ceph-mon[36624]: pgmap v1982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:55.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:55.121+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:04:55.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:55.121+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:04:55.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:04:55.121+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:04:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:56 smithi081 ceph-mon[31453]: pgmap v1983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:56 smithi153 ceph-mon[36569]: pgmap v1983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:56 smithi104 ceph-mon[36624]: pgmap v1983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:58 smithi081 ceph-mon[31453]: pgmap v1984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:58 smithi153 ceph-mon[36569]: pgmap v1984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:58 smithi104 ceph-mon[36624]: pgmap v1984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:04:59 smithi153 ceph-mon[36569]: pgmap v1985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:04:59 smithi104 ceph-mon[36624]: pgmap v1985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:04:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:04:59 smithi081 ceph-mon[31453]: pgmap v1985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:00.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:05:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:01 smithi153 ceph-mon[36569]: pgmap v1986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:01 smithi153 ceph-mon[36569]: pgmap v1987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:05:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:01 smithi104 ceph-mon[36624]: pgmap v1986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:01 smithi104 ceph-mon[36624]: pgmap v1987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:05:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:05:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:05:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:01 smithi081 ceph-mon[31453]: pgmap v1986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:01 smithi081 ceph-mon[31453]: pgmap v1987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:05:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:03 smithi153 ceph-mon[36569]: pgmap v1988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:03 smithi104 ceph-mon[36624]: pgmap v1988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:03 smithi081 ceph-mon[31453]: pgmap v1988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:05 smithi153 ceph-mon[36569]: pgmap v1989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:05 smithi104 ceph-mon[36624]: pgmap v1989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:05 smithi081 ceph-mon[31453]: pgmap v1989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:07 smithi153 ceph-mon[36569]: pgmap v1990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:07 smithi104 ceph-mon[36624]: pgmap v1990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:08.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:07 smithi081 ceph-mon[31453]: pgmap v1990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:09 smithi153 ceph-mon[36569]: pgmap v1991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:09 smithi104 ceph-mon[36624]: pgmap v1991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:10.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:10.124+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:05:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:10.124+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:05:10.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:10.124+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:05:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:09 smithi081 ceph-mon[31453]: pgmap v1991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:11 smithi153 ceph-mon[36569]: pgmap v1992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:11 smithi104 ceph-mon[36624]: pgmap v1992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:11 smithi081 ceph-mon[31453]: pgmap v1992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:13 smithi153 ceph-mon[36569]: pgmap v1993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:13 smithi104 ceph-mon[36624]: pgmap v1993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:13 smithi081 ceph-mon[31453]: pgmap v1993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:15 smithi153 ceph-mon[36569]: pgmap v1994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:15 smithi104 ceph-mon[36624]: pgmap v1994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:15 smithi081 ceph-mon[31453]: pgmap v1994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:17 smithi153 ceph-mon[36569]: pgmap v1995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:17 smithi104 ceph-mon[36624]: pgmap v1995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:17 smithi081 ceph-mon[31453]: pgmap v1995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:19 smithi153 ceph-mon[36569]: pgmap v1996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:19 smithi104 ceph-mon[36624]: pgmap v1996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:19 smithi081 ceph-mon[31453]: pgmap v1996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:21 smithi153 ceph-mon[36569]: pgmap v1997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:21 smithi104 ceph-mon[36624]: pgmap v1997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:21 smithi081 ceph-mon[31453]: pgmap v1997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:23.976 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:23 smithi081 ceph-mon[31453]: pgmap v1998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:23 smithi153 ceph-mon[36569]: pgmap v1998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:23 smithi104 ceph-mon[36624]: pgmap v1998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:25.133+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:05:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:25.133+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:05:25.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:25.133+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:05:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:25 smithi153 ceph-mon[36569]: pgmap v1999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:25 smithi104 ceph-mon[36624]: pgmap v1999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:25 smithi081 ceph-mon[31453]: pgmap v1999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:27 smithi153 ceph-mon[36569]: pgmap v2000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:27 smithi104 ceph-mon[36624]: pgmap v2000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:27 smithi081 ceph-mon[31453]: pgmap v2000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:29 smithi153 ceph-mon[36569]: pgmap v2001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:29 smithi104 ceph-mon[36624]: pgmap v2001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:29 smithi081 ceph-mon[31453]: pgmap v2001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:31 smithi153 ceph-mon[36569]: pgmap v2002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:31 smithi104 ceph-mon[36624]: pgmap v2002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:31 smithi081 ceph-mon[31453]: pgmap v2002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:33 smithi153 ceph-mon[36569]: pgmap v2003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:33 smithi104 ceph-mon[36624]: pgmap v2003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:33 smithi081 ceph-mon[31453]: pgmap v2003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:35 smithi153 ceph-mon[36569]: pgmap v2004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:35 smithi104 ceph-mon[36624]: pgmap v2004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:35 smithi081 ceph-mon[31453]: pgmap v2004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:37 smithi153 ceph-mon[36569]: pgmap v2005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:37 smithi104 ceph-mon[36624]: pgmap v2005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:37 smithi081 ceph-mon[31453]: pgmap v2005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:39 smithi153 ceph-mon[36569]: pgmap v2006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:39 smithi104 ceph-mon[36624]: pgmap v2006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:40.149+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:05:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:40.149+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:05:40.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:40.149+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:05:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:39 smithi081 ceph-mon[31453]: pgmap v2006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:41 smithi153 ceph-mon[36569]: pgmap v2007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:41 smithi104 ceph-mon[36624]: pgmap v2007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:41 smithi081 ceph-mon[31453]: pgmap v2007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:43.184+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:05:43.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:43.185+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:05:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:43.201+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:05:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:43.202+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:05:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:43 smithi153 ceph-mon[36569]: pgmap v2008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:43 smithi104 ceph-mon[36624]: pgmap v2008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:43 smithi081 ceph-mon[31453]: pgmap v2008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:45 smithi153 ceph-mon[36569]: pgmap v2009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:45 smithi104 ceph-mon[36624]: pgmap v2009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:45 smithi081 ceph-mon[31453]: pgmap v2009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:47 smithi153 ceph-mon[36569]: pgmap v2010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:47 smithi104 ceph-mon[36624]: pgmap v2010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:47 smithi081 ceph-mon[31453]: pgmap v2010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:49 smithi153 ceph-mon[36569]: pgmap v2011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:49 smithi104 ceph-mon[36624]: pgmap v2011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:49 smithi081 ceph-mon[31453]: pgmap v2011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:51 smithi153 ceph-mon[36569]: pgmap v2012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:51 smithi104 ceph-mon[36624]: pgmap v2012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:51 smithi081 ceph-mon[31453]: pgmap v2012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:53 smithi153 ceph-mon[36569]: pgmap v2013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:53 smithi104 ceph-mon[36624]: pgmap v2013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:53 smithi081 ceph-mon[31453]: pgmap v2013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:55.164+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:05:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:55.164+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:05:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:05:55.164+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:05:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:55 smithi153 ceph-mon[36569]: pgmap v2014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:55 smithi104 ceph-mon[36624]: pgmap v2014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:55 smithi081 ceph-mon[31453]: pgmap v2014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:57 smithi153 ceph-mon[36569]: pgmap v2015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:57 smithi104 ceph-mon[36624]: pgmap v2015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:05:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:57 smithi081 ceph-mon[31453]: pgmap v2015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:05:59 smithi104 ceph-mon[36624]: pgmap v2016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:05:59 smithi081 ceph-mon[31453]: pgmap v2016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:05:59 smithi153 ceph-mon[36569]: pgmap v2016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:01 smithi104 ceph-mon[36624]: pgmap v2017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:06:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:01 smithi081 ceph-mon[31453]: pgmap v2017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:06:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:01 smithi153 ceph-mon[36569]: pgmap v2017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:06:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:04 smithi153 ceph-mon[36569]: pgmap v2018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:06:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:04 smithi153 ceph-mon[36569]: pgmap v2019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:04 smithi153 ceph-mon[36569]: pgmap v2020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:04 smithi104 ceph-mon[36624]: pgmap v2018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:06:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:04 smithi104 ceph-mon[36624]: pgmap v2019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:04 smithi104 ceph-mon[36624]: pgmap v2020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:04 smithi081 ceph-mon[31453]: pgmap v2018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:06:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:06:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:04 smithi081 ceph-mon[31453]: pgmap v2019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:04 smithi081 ceph-mon[31453]: pgmap v2020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:06:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:06 smithi081 ceph-mon[31453]: pgmap v2021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:06 smithi153 ceph-mon[36569]: pgmap v2021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:06 smithi104 ceph-mon[36624]: pgmap v2021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:08 smithi081 ceph-mon[31453]: pgmap v2022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:08 smithi153 ceph-mon[36569]: pgmap v2022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:08 smithi104 ceph-mon[36624]: pgmap v2022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:10.180+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:06:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:10.180+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:06:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:10.180+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:06:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:10 smithi081 ceph-mon[31453]: pgmap v2023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:10 smithi153 ceph-mon[36569]: pgmap v2023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:10 smithi104 ceph-mon[36624]: pgmap v2023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:12 smithi081 ceph-mon[31453]: pgmap v2024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:12 smithi153 ceph-mon[36569]: pgmap v2024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:12 smithi104 ceph-mon[36624]: pgmap v2024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:14 smithi081 ceph-mon[31453]: pgmap v2025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:14 smithi153 ceph-mon[36569]: pgmap v2025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:14 smithi104 ceph-mon[36624]: pgmap v2025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:16 smithi081 ceph-mon[31453]: pgmap v2026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:16 smithi153 ceph-mon[36569]: pgmap v2026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:16 smithi104 ceph-mon[36624]: pgmap v2026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:18 smithi081 ceph-mon[31453]: pgmap v2027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:18 smithi153 ceph-mon[36569]: pgmap v2027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:18 smithi104 ceph-mon[36624]: pgmap v2027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:20 smithi153 ceph-mon[36569]: pgmap v2028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:20 smithi104 ceph-mon[36624]: pgmap v2028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:20 smithi081 ceph-mon[31453]: pgmap v2028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:22 smithi153 ceph-mon[36569]: pgmap v2029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:22 smithi104 ceph-mon[36624]: pgmap v2029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:22 smithi081 ceph-mon[31453]: pgmap v2029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:24.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:24 smithi081 ceph-mon[31453]: pgmap v2030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:24 smithi153 ceph-mon[36569]: pgmap v2030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:24 smithi104 ceph-mon[36624]: pgmap v2030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:25.190+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:06:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:25.190+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:06:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:25.190+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:06:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:26 smithi081 ceph-mon[31453]: pgmap v2031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:26 smithi153 ceph-mon[36569]: pgmap v2031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:26 smithi104 ceph-mon[36624]: pgmap v2031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:27 smithi153 ceph-mon[36569]: pgmap v2032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:27 smithi104 ceph-mon[36624]: pgmap v2032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:27 smithi081 ceph-mon[31453]: pgmap v2032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:30 smithi081 ceph-mon[31453]: pgmap v2033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:30 smithi153 ceph-mon[36569]: pgmap v2033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:30 smithi104 ceph-mon[36624]: pgmap v2033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:32 smithi081 ceph-mon[31453]: pgmap v2034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:32 smithi153 ceph-mon[36569]: pgmap v2034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:32 smithi104 ceph-mon[36624]: pgmap v2034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:34 smithi081 ceph-mon[31453]: pgmap v2035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:34 smithi153 ceph-mon[36569]: pgmap v2035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:34 smithi104 ceph-mon[36624]: pgmap v2035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:36 smithi081 ceph-mon[31453]: pgmap v2036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:36 smithi153 ceph-mon[36569]: pgmap v2036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:36 smithi104 ceph-mon[36624]: pgmap v2036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:38 smithi081 ceph-mon[31453]: pgmap v2037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:38 smithi153 ceph-mon[36569]: pgmap v2037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:38 smithi104 ceph-mon[36624]: pgmap v2037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:40.203+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:06:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:40.203+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:06:40.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:40.203+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:06:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:40 smithi153 ceph-mon[36569]: pgmap v2038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:40 smithi104 ceph-mon[36624]: pgmap v2038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:40 smithi081 ceph-mon[31453]: pgmap v2038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:42 smithi153 ceph-mon[36569]: pgmap v2039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:42 smithi104 ceph-mon[36624]: pgmap v2039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:42 smithi081 ceph-mon[31453]: pgmap v2039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:44 smithi153 ceph-mon[36569]: pgmap v2040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:44 smithi104 ceph-mon[36624]: pgmap v2040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:44 smithi081 ceph-mon[31453]: pgmap v2040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:46 smithi153 ceph-mon[36569]: pgmap v2041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:46 smithi104 ceph-mon[36624]: pgmap v2041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:46 smithi081 ceph-mon[31453]: pgmap v2041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:47 smithi153 ceph-mon[36569]: pgmap v2042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:47 smithi104 ceph-mon[36624]: pgmap v2042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:47 smithi081 ceph-mon[31453]: pgmap v2042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:50 smithi081 ceph-mon[31453]: pgmap v2043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:50 smithi153 ceph-mon[36569]: pgmap v2043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:50 smithi104 ceph-mon[36624]: pgmap v2043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:52 smithi081 ceph-mon[31453]: pgmap v2044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:52 smithi153 ceph-mon[36569]: pgmap v2044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:52 smithi104 ceph-mon[36624]: pgmap v2044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:54 smithi153 ceph-mon[36569]: pgmap v2045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:54 smithi104 ceph-mon[36624]: pgmap v2045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:54 smithi081 ceph-mon[31453]: pgmap v2045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:55.218+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:06:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:55.218+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:06:55.219 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:06:55.218+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:06:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:56 smithi153 ceph-mon[36569]: pgmap v2046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:56 smithi104 ceph-mon[36624]: pgmap v2046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:56 smithi081 ceph-mon[31453]: pgmap v2046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:06:58 smithi153 ceph-mon[36569]: pgmap v2047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:06:58 smithi104 ceph-mon[36624]: pgmap v2047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:06:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:06:58 smithi081 ceph-mon[31453]: pgmap v2047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:00 smithi153 ceph-mon[36569]: pgmap v2048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:00 smithi104 ceph-mon[36624]: pgmap v2048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:00 smithi081 ceph-mon[31453]: pgmap v2048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:02 smithi153 ceph-mon[36569]: pgmap v2049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:02 smithi104 ceph-mon[36624]: pgmap v2049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:02 smithi081 ceph-mon[31453]: pgmap v2049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:04 smithi153 ceph-mon[36569]: pgmap v2050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:04 smithi104 ceph-mon[36624]: pgmap v2050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:04 smithi081 ceph-mon[31453]: pgmap v2050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:07:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: pgmap v2051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: pgmap v2052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:05.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: pgmap v2051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: pgmap v2052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:07:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:07:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: pgmap v2051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: pgmap v2052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:07:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:07 smithi153 ceph-mon[36569]: pgmap v2053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:07 smithi104 ceph-mon[36624]: pgmap v2053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:07 smithi081 ceph-mon[31453]: pgmap v2053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:09 smithi153 ceph-mon[36569]: pgmap v2054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:09 smithi104 ceph-mon[36624]: pgmap v2054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:09 smithi081 ceph-mon[31453]: pgmap v2054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:10.219+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:07:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:10.219+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:07:10.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:10.219+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:07:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:11 smithi153 ceph-mon[36569]: pgmap v2055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:11 smithi104 ceph-mon[36624]: pgmap v2055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:11 smithi081 ceph-mon[31453]: pgmap v2055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:13.199+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:07:13.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:13.200+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:07:13.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:13.215+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:07:13.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:13.216+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:07:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:13 smithi153 ceph-mon[36569]: pgmap v2056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:13 smithi104 ceph-mon[36624]: pgmap v2056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:13 smithi081 ceph-mon[31453]: pgmap v2056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:15 smithi153 ceph-mon[36569]: pgmap v2057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:15 smithi104 ceph-mon[36624]: pgmap v2057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:15 smithi081 ceph-mon[31453]: pgmap v2057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:17 smithi153 ceph-mon[36569]: pgmap v2058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:17 smithi104 ceph-mon[36624]: pgmap v2058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:17 smithi081 ceph-mon[31453]: pgmap v2058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:19 smithi153 ceph-mon[36569]: pgmap v2059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:19 smithi104 ceph-mon[36624]: pgmap v2059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:19 smithi081 ceph-mon[31453]: pgmap v2059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:21 smithi153 ceph-mon[36569]: pgmap v2060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:21 smithi104 ceph-mon[36624]: pgmap v2060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:21 smithi081 ceph-mon[31453]: pgmap v2060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:23 smithi153 ceph-mon[36569]: pgmap v2061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:23 smithi104 ceph-mon[36624]: pgmap v2061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:23 smithi081 ceph-mon[31453]: pgmap v2061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:25.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:25.223+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:07:25.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:25.223+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:07:25.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:25.223+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:07:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:25 smithi153 ceph-mon[36569]: pgmap v2062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:25 smithi104 ceph-mon[36624]: pgmap v2062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:25 smithi081 ceph-mon[31453]: pgmap v2062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:27 smithi153 ceph-mon[36569]: pgmap v2063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:27 smithi104 ceph-mon[36624]: pgmap v2063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:27 smithi081 ceph-mon[31453]: pgmap v2063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:29 smithi153 ceph-mon[36569]: pgmap v2064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:29 smithi104 ceph-mon[36624]: pgmap v2064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:29 smithi081 ceph-mon[31453]: pgmap v2064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:31 smithi153 ceph-mon[36569]: pgmap v2065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:31 smithi104 ceph-mon[36624]: pgmap v2065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:31 smithi081 ceph-mon[31453]: pgmap v2065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:33 smithi153 ceph-mon[36569]: pgmap v2066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:33 smithi104 ceph-mon[36624]: pgmap v2066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:33 smithi081 ceph-mon[31453]: pgmap v2066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:35 smithi153 ceph-mon[36569]: pgmap v2067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:35 smithi104 ceph-mon[36624]: pgmap v2067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:35 smithi081 ceph-mon[31453]: pgmap v2067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:37 smithi153 ceph-mon[36569]: pgmap v2068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:37 smithi104 ceph-mon[36624]: pgmap v2068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:37 smithi081 ceph-mon[31453]: pgmap v2068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:39 smithi104 ceph-mon[36624]: pgmap v2069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:39 smithi081 ceph-mon[31453]: pgmap v2069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:40.225+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:07:40.227 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:40.225+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:07:40.227 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:40.225+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:07:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:39 smithi153 ceph-mon[36569]: pgmap v2069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:41 smithi104 ceph-mon[36624]: pgmap v2070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:42.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:41 smithi153 ceph-mon[36569]: pgmap v2070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:41 smithi081 ceph-mon[31453]: pgmap v2070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:43 smithi104 ceph-mon[36624]: pgmap v2071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:43 smithi081 ceph-mon[31453]: pgmap v2071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:43 smithi153 ceph-mon[36569]: pgmap v2071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:45 smithi104 ceph-mon[36624]: pgmap v2072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:45 smithi081 ceph-mon[31453]: pgmap v2072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:45 smithi153 ceph-mon[36569]: pgmap v2072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:47 smithi104 ceph-mon[36624]: pgmap v2073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:47 smithi081 ceph-mon[31453]: pgmap v2073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:47 smithi153 ceph-mon[36569]: pgmap v2073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:49 smithi081 ceph-mon[31453]: pgmap v2074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:49 smithi153 ceph-mon[36569]: pgmap v2074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:49 smithi104 ceph-mon[36624]: pgmap v2074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:51 smithi081 ceph-mon[31453]: pgmap v2075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:51 smithi153 ceph-mon[36569]: pgmap v2075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:51 smithi104 ceph-mon[36624]: pgmap v2075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:53 smithi081 ceph-mon[31453]: pgmap v2076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:53 smithi153 ceph-mon[36569]: pgmap v2076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:53 smithi104 ceph-mon[36624]: pgmap v2076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:55.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:55.236+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:07:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:55.236+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:07:55.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:07:55.236+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:07:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:55 smithi081 ceph-mon[31453]: pgmap v2077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:55 smithi153 ceph-mon[36569]: pgmap v2077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:55 smithi104 ceph-mon[36624]: pgmap v2077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:57 smithi081 ceph-mon[31453]: pgmap v2078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:57 smithi153 ceph-mon[36569]: pgmap v2078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:07:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:57 smithi104 ceph-mon[36624]: pgmap v2078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:07:59 smithi081 ceph-mon[31453]: pgmap v2079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:07:59 smithi153 ceph-mon[36569]: pgmap v2079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:07:59 smithi104 ceph-mon[36624]: pgmap v2079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:01 smithi081 ceph-mon[31453]: pgmap v2080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:01 smithi153 ceph-mon[36569]: pgmap v2080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:01 smithi104 ceph-mon[36624]: pgmap v2080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:03 smithi081 ceph-mon[31453]: pgmap v2081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:03 smithi153 ceph-mon[36569]: pgmap v2081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:03 smithi104 ceph-mon[36624]: pgmap v2081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:05 smithi153 ceph-mon[36569]: pgmap v2082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:08:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:05 smithi081 ceph-mon[31453]: pgmap v2082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:08:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.290 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:05 smithi104 ceph-mon[36624]: pgmap v2082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:06.290 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:08:06.290 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.290 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.290 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:06.291 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:07 smithi081 ceph-mon[31453]: pgmap v2083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:07 smithi153 ceph-mon[36569]: pgmap v2083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:07 smithi104 ceph-mon[36624]: pgmap v2083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:09 smithi081 ceph-mon[31453]: pgmap v2084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:10.251+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:08:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:10.251+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:08:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:10.251+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:08:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:09 smithi153 ceph-mon[36569]: pgmap v2084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:09 smithi104 ceph-mon[36624]: pgmap v2084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:11 smithi153 ceph-mon[36569]: pgmap v2085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:11 smithi104 ceph-mon[36624]: pgmap v2085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:11 smithi081 ceph-mon[31453]: pgmap v2085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:12 smithi104 ceph-mon[36624]: pgmap v2086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:12 smithi104 ceph-mon[36624]: pgmap v2087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:12 smithi081 ceph-mon[31453]: pgmap v2086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:12 smithi081 ceph-mon[31453]: pgmap v2087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:08:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:08:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:12 smithi153 ceph-mon[36569]: pgmap v2086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:12 smithi153 ceph-mon[36569]: pgmap v2087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:08:14.984 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:14 smithi081 ceph-mon[31453]: pgmap v2088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:14 smithi153 ceph-mon[36569]: pgmap v2088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:14 smithi104 ceph-mon[36624]: pgmap v2088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:16 smithi153 ceph-mon[36569]: pgmap v2089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:16 smithi104 ceph-mon[36624]: pgmap v2089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:16 smithi081 ceph-mon[31453]: pgmap v2089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:18 smithi153 ceph-mon[36569]: pgmap v2090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:18 smithi104 ceph-mon[36624]: pgmap v2090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:18 smithi081 ceph-mon[31453]: pgmap v2090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:20 smithi153 ceph-mon[36569]: pgmap v2091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:20 smithi104 ceph-mon[36624]: pgmap v2091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:20 smithi081 ceph-mon[31453]: pgmap v2091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:22 smithi153 ceph-mon[36569]: pgmap v2092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:22 smithi104 ceph-mon[36624]: pgmap v2092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:22 smithi081 ceph-mon[31453]: pgmap v2092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:24 smithi153 ceph-mon[36569]: pgmap v2093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:24 smithi104 ceph-mon[36624]: pgmap v2093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:24 smithi081 ceph-mon[31453]: pgmap v2093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:25.267+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:25.267+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:25.267+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:08:26.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:26 smithi081 ceph-mon[31453]: pgmap v2094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:26 smithi153 ceph-mon[36569]: pgmap v2094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:26 smithi104 ceph-mon[36624]: pgmap v2094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:28 smithi153 ceph-mon[36569]: pgmap v2095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:28 smithi104 ceph-mon[36624]: pgmap v2095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:28 smithi081 ceph-mon[31453]: pgmap v2095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:30 smithi153 ceph-mon[36569]: pgmap v2096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:30 smithi104 ceph-mon[36624]: pgmap v2096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:30 smithi081 ceph-mon[31453]: pgmap v2096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:32 smithi153 ceph-mon[36569]: pgmap v2097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:32 smithi104 ceph-mon[36624]: pgmap v2097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:32 smithi081 ceph-mon[31453]: pgmap v2097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:34 smithi153 ceph-mon[36569]: pgmap v2098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:34 smithi104 ceph-mon[36624]: pgmap v2098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:34 smithi081 ceph-mon[31453]: pgmap v2098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:36 smithi153 ceph-mon[36569]: pgmap v2099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:36 smithi104 ceph-mon[36624]: pgmap v2099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:36 smithi081 ceph-mon[31453]: pgmap v2099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:38 smithi153 ceph-mon[36569]: pgmap v2100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:38 smithi104 ceph-mon[36624]: pgmap v2100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:38 smithi081 ceph-mon[31453]: pgmap v2100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:40.270+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:08:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:40.271+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:08:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:40.271+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:08:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:40 smithi153 ceph-mon[36569]: pgmap v2101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:40 smithi104 ceph-mon[36624]: pgmap v2101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:40 smithi081 ceph-mon[31453]: pgmap v2101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:42 smithi153 ceph-mon[36569]: pgmap v2102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:42 smithi104 ceph-mon[36624]: pgmap v2102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:42 smithi081 ceph-mon[31453]: pgmap v2102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:44 smithi153 ceph-mon[36569]: pgmap v2103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:44 smithi104 ceph-mon[36624]: pgmap v2103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:44 smithi081 ceph-mon[31453]: pgmap v2103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:46 smithi153 ceph-mon[36569]: pgmap v2104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:46 smithi104 ceph-mon[36624]: pgmap v2104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:46 smithi081 ceph-mon[31453]: pgmap v2104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:48 smithi153 ceph-mon[36569]: pgmap v2105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:48 smithi104 ceph-mon[36624]: pgmap v2105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:48 smithi081 ceph-mon[31453]: pgmap v2105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:50 smithi153 ceph-mon[36569]: pgmap v2106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:50 smithi104 ceph-mon[36624]: pgmap v2106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:50 smithi081 ceph-mon[31453]: pgmap v2106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:52 smithi153 ceph-mon[36569]: pgmap v2107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:52 smithi104 ceph-mon[36624]: pgmap v2107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:52 smithi081 ceph-mon[31453]: pgmap v2107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:54 smithi153 ceph-mon[36569]: pgmap v2108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:54 smithi104 ceph-mon[36624]: pgmap v2108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:54 smithi081 ceph-mon[31453]: pgmap v2108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:55.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:55.273+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:08:55.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:55.273+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:08:55.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:08:55.273+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:08:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:56 smithi153 ceph-mon[36569]: pgmap v2109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:56 smithi104 ceph-mon[36624]: pgmap v2109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:56 smithi081 ceph-mon[31453]: pgmap v2109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:08:58 smithi153 ceph-mon[36569]: pgmap v2110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:08:58 smithi104 ceph-mon[36624]: pgmap v2110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:08:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:08:58 smithi081 ceph-mon[31453]: pgmap v2110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:00 smithi153 ceph-mon[36569]: pgmap v2111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:00 smithi104 ceph-mon[36624]: pgmap v2111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:00 smithi081 ceph-mon[31453]: pgmap v2111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:02 smithi153 ceph-mon[36569]: pgmap v2112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:02 smithi104 ceph-mon[36624]: pgmap v2112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:02 smithi081 ceph-mon[31453]: pgmap v2112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:04 smithi153 ceph-mon[36569]: pgmap v2113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:04 smithi104 ceph-mon[36624]: pgmap v2113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:04 smithi081 ceph-mon[31453]: pgmap v2113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:06 smithi104 ceph-mon[36624]: pgmap v2114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:06 smithi081 ceph-mon[31453]: pgmap v2114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:06 smithi153 ceph-mon[36569]: pgmap v2114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:08 smithi081 ceph-mon[31453]: pgmap v2115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:08 smithi104 ceph-mon[36624]: pgmap v2115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:08 smithi153 ceph-mon[36569]: pgmap v2115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:10.285+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:09:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:10.285+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:09:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:10.285+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:09:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:10 smithi104 ceph-mon[36624]: pgmap v2116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:10 smithi081 ceph-mon[31453]: pgmap v2116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:10 smithi153 ceph-mon[36569]: pgmap v2116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:12 smithi104 ceph-mon[36624]: pgmap v2117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:09:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:13.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:12 smithi153 ceph-mon[36569]: pgmap v2117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:13.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:09:13.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:13.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:12 smithi081 ceph-mon[31453]: pgmap v2117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:09:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:09:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:09:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:09:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:09:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:09:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:09:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:09:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:14 smithi081 ceph-mon[31453]: pgmap v2118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:14 smithi081 ceph-mon[31453]: pgmap v2119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:14 smithi153 ceph-mon[36569]: pgmap v2118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:14 smithi153 ceph-mon[36569]: pgmap v2119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:14 smithi104 ceph-mon[36624]: pgmap v2118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:14 smithi104 ceph-mon[36624]: pgmap v2119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:16 smithi081 ceph-mon[31453]: pgmap v2120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:16 smithi153 ceph-mon[36569]: pgmap v2120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:16 smithi104 ceph-mon[36624]: pgmap v2120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:19.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:18 smithi104 ceph-mon[36624]: pgmap v2121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:18 smithi081 ceph-mon[31453]: pgmap v2121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:18 smithi153 ceph-mon[36569]: pgmap v2121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:20 smithi081 ceph-mon[31453]: pgmap v2122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:20 smithi153 ceph-mon[36569]: pgmap v2122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:20 smithi104 ceph-mon[36624]: pgmap v2122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:22 smithi081 ceph-mon[31453]: pgmap v2123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:22 smithi153 ceph-mon[36569]: pgmap v2123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:22 smithi104 ceph-mon[36624]: pgmap v2123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:24 smithi081 ceph-mon[31453]: pgmap v2124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:25.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:25.300+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:09:25.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:25.300+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:09:25.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:25.300+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:09:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:24 smithi153 ceph-mon[36569]: pgmap v2124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:24 smithi104 ceph-mon[36624]: pgmap v2124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:26 smithi081 ceph-mon[31453]: pgmap v2125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:26 smithi153 ceph-mon[36569]: pgmap v2125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:26 smithi104 ceph-mon[36624]: pgmap v2125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:28 smithi081 ceph-mon[31453]: pgmap v2126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:28 smithi153 ceph-mon[36569]: pgmap v2126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:28 smithi104 ceph-mon[36624]: pgmap v2126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:30 smithi081 ceph-mon[31453]: pgmap v2127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:30 smithi153 ceph-mon[36569]: pgmap v2127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:30 smithi104 ceph-mon[36624]: pgmap v2127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:32 smithi081 ceph-mon[31453]: pgmap v2128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:32 smithi153 ceph-mon[36569]: pgmap v2128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:32 smithi104 ceph-mon[36624]: pgmap v2128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:34 smithi081 ceph-mon[31453]: pgmap v2129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:34 smithi153 ceph-mon[36569]: pgmap v2129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:34 smithi104 ceph-mon[36624]: pgmap v2129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:36 smithi081 ceph-mon[31453]: pgmap v2130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:36 smithi153 ceph-mon[36569]: pgmap v2130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:36 smithi104 ceph-mon[36624]: pgmap v2130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:38 smithi081 ceph-mon[31453]: pgmap v2131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:38 smithi153 ceph-mon[36569]: pgmap v2131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:38 smithi104 ceph-mon[36624]: pgmap v2131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:40.315+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:09:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:40.315+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:09:40.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:40.315+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:09:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:40 smithi153 ceph-mon[36569]: pgmap v2132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:40 smithi104 ceph-mon[36624]: pgmap v2132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:40 smithi081 ceph-mon[31453]: pgmap v2132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:42 smithi153 ceph-mon[36569]: pgmap v2133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:42 smithi104 ceph-mon[36624]: pgmap v2133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:42 smithi081 ceph-mon[31453]: pgmap v2133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:44 smithi153 ceph-mon[36569]: pgmap v2134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:44 smithi104 ceph-mon[36624]: pgmap v2134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:44 smithi081 ceph-mon[31453]: pgmap v2134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:46 smithi153 ceph-mon[36569]: pgmap v2135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:46 smithi104 ceph-mon[36624]: pgmap v2135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:46 smithi081 ceph-mon[31453]: pgmap v2135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:49.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:48 smithi081 ceph-mon[31453]: pgmap v2136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:48 smithi153 ceph-mon[36569]: pgmap v2136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:48 smithi104 ceph-mon[36624]: pgmap v2136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:50 smithi153 ceph-mon[36569]: pgmap v2137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:50 smithi104 ceph-mon[36624]: pgmap v2137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:50 smithi081 ceph-mon[31453]: pgmap v2137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:52 smithi153 ceph-mon[36569]: pgmap v2138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:52 smithi104 ceph-mon[36624]: pgmap v2138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:52 smithi081 ceph-mon[31453]: pgmap v2138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:54 smithi153 ceph-mon[36569]: pgmap v2139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:55.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:55.320+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:09:55.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:55.320+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:09:55.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:09:55.320+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:09:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:54 smithi104 ceph-mon[36624]: pgmap v2139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:54 smithi081 ceph-mon[31453]: pgmap v2139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:57.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:56 smithi081 ceph-mon[31453]: pgmap v2140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:56 smithi153 ceph-mon[36569]: pgmap v2140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:56 smithi104 ceph-mon[36624]: pgmap v2140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:09:58 smithi153 ceph-mon[36569]: pgmap v2141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:09:58 smithi104 ceph-mon[36624]: pgmap v2141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:09:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:09:58 smithi081 ceph-mon[31453]: pgmap v2141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:00 smithi153 ceph-mon[36569]: pgmap v2142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:10:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:00 smithi104 ceph-mon[36624]: pgmap v2142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:10:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:00 smithi081 ceph-mon[31453]: pgmap v2142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:10:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:02 smithi153 ceph-mon[36569]: pgmap v2143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:02 smithi104 ceph-mon[36624]: pgmap v2143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:02 smithi081 ceph-mon[31453]: pgmap v2143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:04 smithi153 ceph-mon[36569]: pgmap v2144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:04 smithi104 ceph-mon[36624]: pgmap v2144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:04 smithi081 ceph-mon[31453]: pgmap v2144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:06 smithi153 ceph-mon[36569]: pgmap v2145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:06 smithi104 ceph-mon[36624]: pgmap v2145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:06 smithi081 ceph-mon[31453]: pgmap v2145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:08 smithi153 ceph-mon[36569]: pgmap v2146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:08 smithi104 ceph-mon[36624]: pgmap v2146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:08 smithi081 ceph-mon[31453]: pgmap v2146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:10.334+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:10:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:10.334+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:10:10.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:10.334+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:10:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:10 smithi153 ceph-mon[36569]: pgmap v2147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:10 smithi104 ceph-mon[36624]: pgmap v2147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:10 smithi081 ceph-mon[31453]: pgmap v2147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:13.229+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:10:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:13.229+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:10:13.242 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:13.242+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:10:13.243 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:13.243+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:10:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:12 smithi153 ceph-mon[36569]: pgmap v2148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:12 smithi104 ceph-mon[36624]: pgmap v2148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:12 smithi081 ceph-mon[31453]: pgmap v2148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:10:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:10:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:10:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:15 smithi153 ceph-mon[36569]: pgmap v2149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:10:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:10:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:10:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:15 smithi104 ceph-mon[36624]: pgmap v2149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:10:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:10:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:10:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:15 smithi081 ceph-mon[31453]: pgmap v2149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:10:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:10:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:10:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:16 smithi153 ceph-mon[36569]: pgmap v2150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:16 smithi153 ceph-mon[36569]: pgmap v2151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:16 smithi104 ceph-mon[36624]: pgmap v2150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:16 smithi104 ceph-mon[36624]: pgmap v2151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:16 smithi081 ceph-mon[31453]: pgmap v2150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:16 smithi081 ceph-mon[31453]: pgmap v2151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:18 smithi153 ceph-mon[36569]: pgmap v2152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:18 smithi104 ceph-mon[36624]: pgmap v2152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:18 smithi081 ceph-mon[31453]: pgmap v2152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:19 smithi153 ceph-mon[36569]: pgmap v2153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:19 smithi104 ceph-mon[36624]: pgmap v2153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:19 smithi081 ceph-mon[31453]: pgmap v2153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:21 smithi153 ceph-mon[36569]: pgmap v2154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:21 smithi104 ceph-mon[36624]: pgmap v2154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:21 smithi081 ceph-mon[31453]: pgmap v2154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:23 smithi153 ceph-mon[36569]: pgmap v2155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:23 smithi104 ceph-mon[36624]: pgmap v2155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:23 smithi081 ceph-mon[31453]: pgmap v2155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:25.349+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:10:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:25.349+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:10:25.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:25.349+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:10:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:25 smithi153 ceph-mon[36569]: pgmap v2156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:25 smithi104 ceph-mon[36624]: pgmap v2156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:25 smithi081 ceph-mon[31453]: pgmap v2156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:28.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:27 smithi081 ceph-mon[31453]: pgmap v2157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:27 smithi153 ceph-mon[36569]: pgmap v2157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:28.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:27 smithi104 ceph-mon[36624]: pgmap v2157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:29 smithi153 ceph-mon[36569]: pgmap v2158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:29 smithi104 ceph-mon[36624]: pgmap v2158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:29 smithi081 ceph-mon[31453]: pgmap v2158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:31 smithi153 ceph-mon[36569]: pgmap v2159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:31 smithi104 ceph-mon[36624]: pgmap v2159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:31 smithi081 ceph-mon[31453]: pgmap v2159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:33 smithi153 ceph-mon[36569]: pgmap v2160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:33 smithi104 ceph-mon[36624]: pgmap v2160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:33 smithi081 ceph-mon[31453]: pgmap v2160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:35 smithi153 ceph-mon[36569]: pgmap v2161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:35 smithi104 ceph-mon[36624]: pgmap v2161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:35 smithi081 ceph-mon[31453]: pgmap v2161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:37 smithi153 ceph-mon[36569]: pgmap v2162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:37 smithi104 ceph-mon[36624]: pgmap v2162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:37 smithi081 ceph-mon[31453]: pgmap v2162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:39 smithi153 ceph-mon[36569]: pgmap v2163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:39 smithi104 ceph-mon[36624]: pgmap v2163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:40.359+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:10:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:40.359+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:10:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:40.359+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:10:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:39 smithi081 ceph-mon[31453]: pgmap v2163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:41 smithi153 ceph-mon[36569]: pgmap v2164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:41 smithi104 ceph-mon[36624]: pgmap v2164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:41 smithi081 ceph-mon[31453]: pgmap v2164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:43 smithi153 ceph-mon[36569]: pgmap v2165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:43 smithi104 ceph-mon[36624]: pgmap v2165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:43 smithi081 ceph-mon[31453]: pgmap v2165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:45 smithi153 ceph-mon[36569]: pgmap v2166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:45 smithi104 ceph-mon[36624]: pgmap v2166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:45 smithi081 ceph-mon[31453]: pgmap v2166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:47 smithi153 ceph-mon[36569]: pgmap v2167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:47 smithi104 ceph-mon[36624]: pgmap v2167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:47 smithi081 ceph-mon[31453]: pgmap v2167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:49 smithi153 ceph-mon[36569]: pgmap v2168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:49 smithi104 ceph-mon[36624]: pgmap v2168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:49 smithi081 ceph-mon[31453]: pgmap v2168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:51 smithi153 ceph-mon[36569]: pgmap v2169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:51 smithi104 ceph-mon[36624]: pgmap v2169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:51 smithi081 ceph-mon[31453]: pgmap v2169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:53 smithi153 ceph-mon[36569]: pgmap v2170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:53 smithi104 ceph-mon[36624]: pgmap v2170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:53 smithi081 ceph-mon[31453]: pgmap v2170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:55.366+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:10:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:55.366+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:10:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:10:55.366+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:10:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:56 smithi153 ceph-mon[36569]: pgmap v2171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:56 smithi104 ceph-mon[36624]: pgmap v2171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:56 smithi081 ceph-mon[31453]: pgmap v2171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:10:58 smithi153 ceph-mon[36569]: pgmap v2172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:10:58 smithi104 ceph-mon[36624]: pgmap v2172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:10:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:10:58 smithi081 ceph-mon[31453]: pgmap v2172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:00 smithi153 ceph-mon[36569]: pgmap v2173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:00 smithi104 ceph-mon[36624]: pgmap v2173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:00 smithi081 ceph-mon[31453]: pgmap v2173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:02 smithi153 ceph-mon[36569]: pgmap v2174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:02 smithi104 ceph-mon[36624]: pgmap v2174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:02 smithi081 ceph-mon[31453]: pgmap v2174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:04 smithi153 ceph-mon[36569]: pgmap v2175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:04 smithi104 ceph-mon[36624]: pgmap v2175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:04 smithi081 ceph-mon[31453]: pgmap v2175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:06 smithi153 ceph-mon[36569]: pgmap v2176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:06 smithi104 ceph-mon[36624]: pgmap v2176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:06 smithi081 ceph-mon[31453]: pgmap v2176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:08 smithi153 ceph-mon[36569]: pgmap v2177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:08 smithi104 ceph-mon[36624]: pgmap v2177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:08 smithi081 ceph-mon[31453]: pgmap v2177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:10 smithi153 ceph-mon[36569]: pgmap v2178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:10 smithi104 ceph-mon[36624]: pgmap v2178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:10.378+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:11:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:10.378+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:11:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:10.378+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:11:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:10 smithi081 ceph-mon[31453]: pgmap v2178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:12 smithi153 ceph-mon[36569]: pgmap v2179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:12 smithi104 ceph-mon[36624]: pgmap v2179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:12 smithi081 ceph-mon[31453]: pgmap v2179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:14 smithi153 ceph-mon[36569]: pgmap v2180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:14 smithi104 ceph-mon[36624]: pgmap v2180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:14 smithi081 ceph-mon[31453]: pgmap v2180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:16 smithi104 ceph-mon[36624]: pgmap v2181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:16 smithi081 ceph-mon[31453]: pgmap v2181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:11:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:16 smithi153 ceph-mon[36569]: pgmap v2181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:11:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:11:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:11:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:11:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:11:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:11:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:11:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:11:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:11:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:18 smithi104 ceph-mon[36624]: pgmap v2182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:18 smithi104 ceph-mon[36624]: pgmap v2183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:18 smithi081 ceph-mon[31453]: pgmap v2182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:18 smithi081 ceph-mon[31453]: pgmap v2183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:18 smithi153 ceph-mon[36569]: pgmap v2182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:18 smithi153 ceph-mon[36569]: pgmap v2183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:20 smithi104 ceph-mon[36624]: pgmap v2184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:20 smithi081 ceph-mon[31453]: pgmap v2184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:20 smithi153 ceph-mon[36569]: pgmap v2184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:22 smithi081 ceph-mon[31453]: pgmap v2185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:22 smithi153 ceph-mon[36569]: pgmap v2185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:22 smithi104 ceph-mon[36624]: pgmap v2185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:24 smithi081 ceph-mon[31453]: pgmap v2186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:24 smithi153 ceph-mon[36569]: pgmap v2186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:24 smithi104 ceph-mon[36624]: pgmap v2186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:25.384+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:11:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:25.384+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:11:25.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:25.384+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:11:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:26 smithi081 ceph-mon[31453]: pgmap v2187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:26 smithi153 ceph-mon[36569]: pgmap v2187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:26 smithi104 ceph-mon[36624]: pgmap v2187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:28 smithi081 ceph-mon[31453]: pgmap v2188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:28 smithi153 ceph-mon[36569]: pgmap v2188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:28 smithi104 ceph-mon[36624]: pgmap v2188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:30 smithi081 ceph-mon[31453]: pgmap v2189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:30 smithi153 ceph-mon[36569]: pgmap v2189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:30 smithi104 ceph-mon[36624]: pgmap v2189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:32 smithi081 ceph-mon[31453]: pgmap v2190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:32 smithi153 ceph-mon[36569]: pgmap v2190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:32 smithi104 ceph-mon[36624]: pgmap v2190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:34 smithi081 ceph-mon[31453]: pgmap v2191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:34 smithi153 ceph-mon[36569]: pgmap v2191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:34 smithi104 ceph-mon[36624]: pgmap v2191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:36 smithi081 ceph-mon[31453]: pgmap v2192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:36 smithi153 ceph-mon[36569]: pgmap v2192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:36 smithi104 ceph-mon[36624]: pgmap v2192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:38 smithi081 ceph-mon[31453]: pgmap v2193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:38 smithi153 ceph-mon[36569]: pgmap v2193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:38 smithi104 ceph-mon[36624]: pgmap v2193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:40.400 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:40.400+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:11:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:40.400+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:11:40.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:40.400+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:11:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:40 smithi081 ceph-mon[31453]: pgmap v2194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:40 smithi153 ceph-mon[36569]: pgmap v2194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:40 smithi104 ceph-mon[36624]: pgmap v2194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:42 smithi081 ceph-mon[31453]: pgmap v2195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:42 smithi153 ceph-mon[36569]: pgmap v2195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:42 smithi104 ceph-mon[36624]: pgmap v2195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:44 smithi081 ceph-mon[31453]: pgmap v2196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:44 smithi153 ceph-mon[36569]: pgmap v2196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:44 smithi104 ceph-mon[36624]: pgmap v2196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:46 smithi081 ceph-mon[31453]: pgmap v2197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:46 smithi153 ceph-mon[36569]: pgmap v2197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:46 smithi104 ceph-mon[36624]: pgmap v2197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:48 smithi081 ceph-mon[31453]: pgmap v2198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:48 smithi153 ceph-mon[36569]: pgmap v2198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:48 smithi104 ceph-mon[36624]: pgmap v2198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:50 smithi081 ceph-mon[31453]: pgmap v2199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:50 smithi153 ceph-mon[36569]: pgmap v2199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:50 smithi104 ceph-mon[36624]: pgmap v2199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:52 smithi153 ceph-mon[36569]: pgmap v2200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:52 smithi104 ceph-mon[36624]: pgmap v2200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:52 smithi081 ceph-mon[31453]: pgmap v2200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:54 smithi153 ceph-mon[36569]: pgmap v2201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:54 smithi104 ceph-mon[36624]: pgmap v2201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:54 smithi081 ceph-mon[31453]: pgmap v2201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:55.410 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:55.410+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:11:55.410 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:55.410+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:11:55.410 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:11:55.410+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:11:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:56 smithi153 ceph-mon[36569]: pgmap v2202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:56 smithi104 ceph-mon[36624]: pgmap v2202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:56 smithi081 ceph-mon[31453]: pgmap v2202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:58.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:11:58 smithi081 ceph-mon[31453]: pgmap v2203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:11:58 smithi153 ceph-mon[36569]: pgmap v2203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:11:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:11:58 smithi104 ceph-mon[36624]: pgmap v2203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:00 smithi153 ceph-mon[36569]: pgmap v2204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:00 smithi104 ceph-mon[36624]: pgmap v2204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:00 smithi081 ceph-mon[31453]: pgmap v2204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:02 smithi153 ceph-mon[36569]: pgmap v2205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:02 smithi104 ceph-mon[36624]: pgmap v2205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:02 smithi081 ceph-mon[31453]: pgmap v2205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:04 smithi153 ceph-mon[36569]: pgmap v2206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:04 smithi104 ceph-mon[36624]: pgmap v2206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:04 smithi081 ceph-mon[31453]: pgmap v2206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:06 smithi153 ceph-mon[36569]: pgmap v2207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:06 smithi104 ceph-mon[36624]: pgmap v2207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:06 smithi081 ceph-mon[31453]: pgmap v2207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:08 smithi153 ceph-mon[36569]: pgmap v2208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:08 smithi104 ceph-mon[36624]: pgmap v2208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:08 smithi081 ceph-mon[31453]: pgmap v2208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:10.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:10.425+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:12:10.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:10.425+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:12:10.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:10.425+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:12:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:10 smithi153 ceph-mon[36569]: pgmap v2209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:10 smithi104 ceph-mon[36624]: pgmap v2209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:10 smithi081 ceph-mon[31453]: pgmap v2209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:12 smithi153 ceph-mon[36569]: pgmap v2210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:12 smithi104 ceph-mon[36624]: pgmap v2210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:12 smithi081 ceph-mon[31453]: pgmap v2210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:14 smithi153 ceph-mon[36569]: pgmap v2211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:14 smithi104 ceph-mon[36624]: pgmap v2211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:14 smithi081 ceph-mon[31453]: pgmap v2211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:16 smithi153 ceph-mon[36569]: pgmap v2212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:16 smithi104 ceph-mon[36624]: pgmap v2212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:16 smithi081 ceph-mon[31453]: pgmap v2212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:12:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:12:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:12:18.299 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:18 smithi104 ceph-mon[36624]: pgmap v2213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:18 smithi153 ceph-mon[36569]: pgmap v2213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:18 smithi081 ceph-mon[31453]: pgmap v2213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:20.489 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:20 smithi081 ceph-mon[31453]: pgmap v2214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:20 smithi153 ceph-mon[36569]: pgmap v2214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:20 smithi104 ceph-mon[36624]: pgmap v2214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:22 smithi081 ceph-mon[31453]: pgmap v2215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:22 smithi153 ceph-mon[36569]: pgmap v2215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:22 smithi104 ceph-mon[36624]: pgmap v2215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:24 smithi081 ceph-mon[31453]: pgmap v2216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:12:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:12:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:24 smithi153 ceph-mon[36569]: pgmap v2216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:12:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:12:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:24 smithi104 ceph-mon[36624]: pgmap v2216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:12:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:12:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:12:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:25 smithi081 ceph-mon[31453]: pgmap v2217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:25 smithi081 ceph-mon[31453]: pgmap v2218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:25.430+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:12:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:25.431+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:12:25.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:25.431+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:12:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:25 smithi153 ceph-mon[36569]: pgmap v2217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:25 smithi153 ceph-mon[36569]: pgmap v2218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:25 smithi104 ceph-mon[36624]: pgmap v2217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:25 smithi104 ceph-mon[36624]: pgmap v2218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:27 smithi081 ceph-mon[31453]: pgmap v2219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:27 smithi153 ceph-mon[36569]: pgmap v2219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:27 smithi104 ceph-mon[36624]: pgmap v2219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:29 smithi081 ceph-mon[31453]: pgmap v2220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:29 smithi153 ceph-mon[36569]: pgmap v2220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:29 smithi104 ceph-mon[36624]: pgmap v2220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:31 smithi081 ceph-mon[31453]: pgmap v2221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:31 smithi153 ceph-mon[36569]: pgmap v2221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:31 smithi104 ceph-mon[36624]: pgmap v2221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:33 smithi081 ceph-mon[31453]: pgmap v2222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:33 smithi153 ceph-mon[36569]: pgmap v2222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:33 smithi104 ceph-mon[36624]: pgmap v2222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:35 smithi081 ceph-mon[31453]: pgmap v2223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:35 smithi153 ceph-mon[36569]: pgmap v2223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:35 smithi104 ceph-mon[36624]: pgmap v2223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:37.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:37 smithi081 ceph-mon[31453]: pgmap v2224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:37 smithi153 ceph-mon[36569]: pgmap v2224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:37.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:37 smithi104 ceph-mon[36624]: pgmap v2224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:39 smithi081 ceph-mon[31453]: pgmap v2225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:39 smithi153 ceph-mon[36569]: pgmap v2225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:39 smithi104 ceph-mon[36624]: pgmap v2225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:40.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:40.441+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:12:40.442 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:40.441+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:12:40.442 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:40.441+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:12:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:41 smithi081 ceph-mon[31453]: pgmap v2226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:41 smithi153 ceph-mon[36569]: pgmap v2226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:41 smithi104 ceph-mon[36624]: pgmap v2226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:43 smithi081 ceph-mon[31453]: pgmap v2227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:43 smithi153 ceph-mon[36569]: pgmap v2227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:43 smithi104 ceph-mon[36624]: pgmap v2227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:45 smithi153 ceph-mon[36569]: pgmap v2228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:45 smithi104 ceph-mon[36624]: pgmap v2228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:45 smithi081 ceph-mon[31453]: pgmap v2228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:47 smithi153 ceph-mon[36569]: pgmap v2229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:47 smithi104 ceph-mon[36624]: pgmap v2229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:47 smithi081 ceph-mon[31453]: pgmap v2229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:49 smithi153 ceph-mon[36569]: pgmap v2230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:49 smithi104 ceph-mon[36624]: pgmap v2230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:49 smithi081 ceph-mon[31453]: pgmap v2230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:51 smithi153 ceph-mon[36569]: pgmap v2231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:51 smithi104 ceph-mon[36624]: pgmap v2231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:51 smithi081 ceph-mon[31453]: pgmap v2231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:53 smithi153 ceph-mon[36569]: pgmap v2232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:53 smithi104 ceph-mon[36624]: pgmap v2232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:53 smithi081 ceph-mon[31453]: pgmap v2232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:55.456+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:12:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:55.456+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:12:55.457 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:12:55.456+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:12:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:55 smithi153 ceph-mon[36569]: pgmap v2233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:55 smithi104 ceph-mon[36624]: pgmap v2233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:55.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:55 smithi081 ceph-mon[31453]: pgmap v2233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:57 smithi153 ceph-mon[36569]: pgmap v2234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:57 smithi104 ceph-mon[36624]: pgmap v2234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:57 smithi081 ceph-mon[31453]: pgmap v2234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:59.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:12:59 smithi081 ceph-mon[31453]: pgmap v2235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:59.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:12:59 smithi153 ceph-mon[36569]: pgmap v2235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:12:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:12:59 smithi104 ceph-mon[36624]: pgmap v2235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:01 smithi153 ceph-mon[36569]: pgmap v2236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:01 smithi104 ceph-mon[36624]: pgmap v2236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:01 smithi081 ceph-mon[31453]: pgmap v2236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:03 smithi153 ceph-mon[36569]: pgmap v2237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:03 smithi104 ceph-mon[36624]: pgmap v2237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:03 smithi081 ceph-mon[31453]: pgmap v2237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:05 smithi153 ceph-mon[36569]: pgmap v2238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:05 smithi104 ceph-mon[36624]: pgmap v2238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:05 smithi081 ceph-mon[31453]: pgmap v2238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:07 smithi153 ceph-mon[36569]: pgmap v2239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:07 smithi104 ceph-mon[36624]: pgmap v2239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:07 smithi081 ceph-mon[31453]: pgmap v2239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:09 smithi153 ceph-mon[36569]: pgmap v2240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:09 smithi104 ceph-mon[36624]: pgmap v2240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:09 smithi081 ceph-mon[31453]: pgmap v2240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:10.471+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:10.471+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:13:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:10.471+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:13:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:11 smithi153 ceph-mon[36569]: pgmap v2241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:11 smithi104 ceph-mon[36624]: pgmap v2241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:11 smithi081 ceph-mon[31453]: pgmap v2241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:13.258+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:13:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:13.259+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:13:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:13.269+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:13:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:13.270+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:13:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:13 smithi153 ceph-mon[36569]: pgmap v2242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:13 smithi104 ceph-mon[36624]: pgmap v2242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:13 smithi081 ceph-mon[31453]: pgmap v2242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:15 smithi153 ceph-mon[36569]: pgmap v2243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:15 smithi104 ceph-mon[36624]: pgmap v2243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:15 smithi081 ceph-mon[31453]: pgmap v2243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:17 smithi153 ceph-mon[36569]: pgmap v2244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:17 smithi104 ceph-mon[36624]: pgmap v2244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:17 smithi081 ceph-mon[31453]: pgmap v2244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:19 smithi153 ceph-mon[36569]: pgmap v2245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:19 smithi104 ceph-mon[36624]: pgmap v2245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:19 smithi081 ceph-mon[31453]: pgmap v2245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:21 smithi153 ceph-mon[36569]: pgmap v2246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:21 smithi104 ceph-mon[36624]: pgmap v2246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:21 smithi081 ceph-mon[31453]: pgmap v2246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:23 smithi153 ceph-mon[36569]: pgmap v2247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:23 smithi104 ceph-mon[36624]: pgmap v2247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:23 smithi081 ceph-mon[31453]: pgmap v2247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:25.486+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:13:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:25.486+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:13:25.487 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:25.486+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:13:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:25 smithi153 ceph-mon[36569]: pgmap v2248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:13:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:25 smithi104 ceph-mon[36624]: pgmap v2248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:13:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:25 smithi081 ceph-mon[31453]: pgmap v2248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:13:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:27 smithi153 ceph-mon[36569]: pgmap v2249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:27 smithi104 ceph-mon[36624]: pgmap v2249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:27 smithi081 ceph-mon[31453]: pgmap v2249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:28 smithi104 ceph-mon[36624]: pgmap v2250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:28 smithi081 ceph-mon[31453]: pgmap v2250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:28 smithi153 ceph-mon[36569]: pgmap v2250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:13:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:13:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:29 smithi153 ceph-mon[36569]: pgmap v2251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:29 smithi153 ceph-mon[36569]: pgmap v2252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:13:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:13:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:29 smithi104 ceph-mon[36624]: pgmap v2251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:29 smithi104 ceph-mon[36624]: pgmap v2252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:13:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:13:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:29 smithi081 ceph-mon[31453]: pgmap v2251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:29 smithi081 ceph-mon[31453]: pgmap v2252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:13:32.098 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:31 smithi081 ceph-mon[31453]: pgmap v2253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:31 smithi153 ceph-mon[36569]: pgmap v2253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:31 smithi104 ceph-mon[36624]: pgmap v2253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:33 smithi153 ceph-mon[36569]: pgmap v2254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:33 smithi104 ceph-mon[36624]: pgmap v2254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:33 smithi081 ceph-mon[31453]: pgmap v2254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:36 smithi153 ceph-mon[36569]: pgmap v2255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:36 smithi104 ceph-mon[36624]: pgmap v2255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:36 smithi081 ceph-mon[31453]: pgmap v2255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:38 smithi153 ceph-mon[36569]: pgmap v2256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:38 smithi104 ceph-mon[36624]: pgmap v2256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:38 smithi081 ceph-mon[31453]: pgmap v2256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:40 smithi153 ceph-mon[36569]: pgmap v2257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:40 smithi104 ceph-mon[36624]: pgmap v2257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:40 smithi081 ceph-mon[31453]: pgmap v2257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:40.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:40.494+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:13:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:40.494+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:13:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:40.494+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:13:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:42 smithi153 ceph-mon[36569]: pgmap v2258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:42 smithi104 ceph-mon[36624]: pgmap v2258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:42 smithi081 ceph-mon[31453]: pgmap v2258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:44 smithi153 ceph-mon[36569]: pgmap v2259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:44 smithi104 ceph-mon[36624]: pgmap v2259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:44 smithi081 ceph-mon[31453]: pgmap v2259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:46 smithi153 ceph-mon[36569]: pgmap v2260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:46 smithi104 ceph-mon[36624]: pgmap v2260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:46 smithi081 ceph-mon[31453]: pgmap v2260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:48 smithi153 ceph-mon[36569]: pgmap v2261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:48 smithi104 ceph-mon[36624]: pgmap v2261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:48 smithi081 ceph-mon[31453]: pgmap v2261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:50 smithi153 ceph-mon[36569]: pgmap v2262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:50 smithi104 ceph-mon[36624]: pgmap v2262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:50 smithi081 ceph-mon[31453]: pgmap v2262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:52 smithi153 ceph-mon[36569]: pgmap v2263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:52 smithi104 ceph-mon[36624]: pgmap v2263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:52 smithi081 ceph-mon[31453]: pgmap v2263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:54 smithi153 ceph-mon[36569]: pgmap v2264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:54 smithi104 ceph-mon[36624]: pgmap v2264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:54 smithi081 ceph-mon[31453]: pgmap v2264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:55.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:55.507+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:13:55.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:55.507+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:13:55.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:55.507+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:13:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:56 smithi153 ceph-mon[36569]: pgmap v2265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:56 smithi104 ceph-mon[36624]: pgmap v2265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:56 smithi081 ceph-mon[31453]: pgmap v2265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:58.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:58.266+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:13:58.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:58.267+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:13:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:58.276+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:13:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:13:58.277+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:13:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:13:58 smithi153 ceph-mon[36569]: pgmap v2266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:13:58 smithi104 ceph-mon[36624]: pgmap v2266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:13:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:13:58 smithi081 ceph-mon[31453]: pgmap v2266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:00 smithi104 ceph-mon[36624]: pgmap v2267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:00 smithi081 ceph-mon[31453]: pgmap v2267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:00 smithi153 ceph-mon[36569]: pgmap v2267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:02 smithi104 ceph-mon[36624]: pgmap v2268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:02 smithi081 ceph-mon[31453]: pgmap v2268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:02 smithi153 ceph-mon[36569]: pgmap v2268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:04 smithi104 ceph-mon[36624]: pgmap v2269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:04 smithi081 ceph-mon[31453]: pgmap v2269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:04 smithi153 ceph-mon[36569]: pgmap v2269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:06 smithi104 ceph-mon[36624]: pgmap v2270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:06 smithi081 ceph-mon[31453]: pgmap v2270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:06 smithi153 ceph-mon[36569]: pgmap v2270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:08 smithi081 ceph-mon[31453]: pgmap v2271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:08 smithi153 ceph-mon[36569]: pgmap v2271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:08 smithi104 ceph-mon[36624]: pgmap v2271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:10 smithi081 ceph-mon[31453]: pgmap v2272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:10.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:10.522+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:14:10.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:10.523+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:14:10.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:10.523+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:14:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:10 smithi153 ceph-mon[36569]: pgmap v2272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:10 smithi104 ceph-mon[36624]: pgmap v2272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:12 smithi081 ceph-mon[31453]: pgmap v2273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:12 smithi153 ceph-mon[36569]: pgmap v2273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:12 smithi104 ceph-mon[36624]: pgmap v2273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:14 smithi081 ceph-mon[31453]: pgmap v2274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:14 smithi153 ceph-mon[36569]: pgmap v2274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:14 smithi104 ceph-mon[36624]: pgmap v2274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:16.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:16 smithi153 ceph-mon[36569]: pgmap v2275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:16 smithi081 ceph-mon[31453]: pgmap v2275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:16 smithi104 ceph-mon[36624]: pgmap v2275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:18 smithi081 ceph-mon[31453]: pgmap v2276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:18 smithi153 ceph-mon[36569]: pgmap v2276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:18 smithi104 ceph-mon[36624]: pgmap v2276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:20 smithi081 ceph-mon[31453]: pgmap v2277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:20 smithi153 ceph-mon[36569]: pgmap v2277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:20 smithi104 ceph-mon[36624]: pgmap v2277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:22.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:22 smithi104 ceph-mon[36624]: pgmap v2278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:22 smithi081 ceph-mon[31453]: pgmap v2278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:22 smithi153 ceph-mon[36569]: pgmap v2278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:24 smithi081 ceph-mon[31453]: pgmap v2279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:24 smithi153 ceph-mon[36569]: pgmap v2279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:24 smithi104 ceph-mon[36624]: pgmap v2279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:25.532+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:14:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:25.532+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:14:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:25.532+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:14:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:26 smithi081 ceph-mon[31453]: pgmap v2280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:26 smithi153 ceph-mon[36569]: pgmap v2280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:26 smithi104 ceph-mon[36624]: pgmap v2280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:28 smithi081 ceph-mon[31453]: pgmap v2281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:28 smithi153 ceph-mon[36569]: pgmap v2281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:28 smithi104 ceph-mon[36624]: pgmap v2281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: pgmap v2282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: pgmap v2282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:14:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: pgmap v2282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:14:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:14:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:14:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:14:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:14:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:14:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:14:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:14:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:14:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:14:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:14:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:14:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:14:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:32 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:14:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:32 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:14:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:32 smithi081 ceph-mon[31453]: pgmap v2283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:32 smithi081 ceph-mon[31453]: pgmap v2284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:32 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:14:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:32 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:14:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:32 smithi153 ceph-mon[36569]: pgmap v2283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:32 smithi153 ceph-mon[36569]: pgmap v2284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:32 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:14:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:32 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:14:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:32 smithi104 ceph-mon[36624]: pgmap v2283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:32 smithi104 ceph-mon[36624]: pgmap v2284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:34 smithi153 ceph-mon[36569]: pgmap v2285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:34 smithi104 ceph-mon[36624]: pgmap v2285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:34 smithi081 ceph-mon[31453]: pgmap v2285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:36 smithi153 ceph-mon[36569]: pgmap v2286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:36 smithi104 ceph-mon[36624]: pgmap v2286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:36 smithi081 ceph-mon[31453]: pgmap v2286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:38 smithi153 ceph-mon[36569]: pgmap v2287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:38 smithi104 ceph-mon[36624]: pgmap v2287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:38 smithi081 ceph-mon[31453]: pgmap v2287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:40.546+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:14:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:40.546+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:14:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:40.546+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:14:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:40 smithi153 ceph-mon[36569]: pgmap v2288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:40 smithi104 ceph-mon[36624]: pgmap v2288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:40 smithi081 ceph-mon[31453]: pgmap v2288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:42 smithi153 ceph-mon[36569]: pgmap v2289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:42 smithi104 ceph-mon[36624]: pgmap v2289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:42 smithi081 ceph-mon[31453]: pgmap v2289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:44 smithi081 ceph-mon[31453]: pgmap v2290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:44 smithi153 ceph-mon[36569]: pgmap v2290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:44 smithi104 ceph-mon[36624]: pgmap v2290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:46 smithi153 ceph-mon[36569]: pgmap v2291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:46 smithi104 ceph-mon[36624]: pgmap v2291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:46 smithi081 ceph-mon[31453]: pgmap v2291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:48 smithi153 ceph-mon[36569]: pgmap v2292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:48 smithi104 ceph-mon[36624]: pgmap v2292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:48 smithi081 ceph-mon[31453]: pgmap v2292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:50 smithi153 ceph-mon[36569]: pgmap v2293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:50 smithi104 ceph-mon[36624]: pgmap v2293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:50 smithi081 ceph-mon[31453]: pgmap v2293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:52 smithi153 ceph-mon[36569]: pgmap v2294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:52 smithi104 ceph-mon[36624]: pgmap v2294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:52 smithi081 ceph-mon[31453]: pgmap v2294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:54 smithi153 ceph-mon[36569]: pgmap v2295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:54 smithi104 ceph-mon[36624]: pgmap v2295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:54 smithi081 ceph-mon[31453]: pgmap v2295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:55.550+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:14:55.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:55.550+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:14:55.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:14:55.550+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:14:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:56 smithi153 ceph-mon[36569]: pgmap v2296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:56 smithi104 ceph-mon[36624]: pgmap v2296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:56 smithi081 ceph-mon[31453]: pgmap v2296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:14:58 smithi153 ceph-mon[36569]: pgmap v2297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:14:58 smithi104 ceph-mon[36624]: pgmap v2297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:14:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:14:58 smithi081 ceph-mon[31453]: pgmap v2297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:00 smithi153 ceph-mon[36569]: pgmap v2298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:00 smithi104 ceph-mon[36624]: pgmap v2298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:00 smithi081 ceph-mon[31453]: pgmap v2298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:02 smithi153 ceph-mon[36569]: pgmap v2299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:02 smithi104 ceph-mon[36624]: pgmap v2299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:02 smithi081 ceph-mon[31453]: pgmap v2299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:04 smithi153 ceph-mon[36569]: pgmap v2300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:04 smithi104 ceph-mon[36624]: pgmap v2300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:04 smithi081 ceph-mon[31453]: pgmap v2300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:06 smithi153 ceph-mon[36569]: pgmap v2301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:06 smithi104 ceph-mon[36624]: pgmap v2301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:06 smithi081 ceph-mon[31453]: pgmap v2301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:08 smithi153 ceph-mon[36569]: pgmap v2302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:08 smithi104 ceph-mon[36624]: pgmap v2302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:08 smithi081 ceph-mon[31453]: pgmap v2302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:10 smithi153 ceph-mon[36569]: pgmap v2303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:10.565+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:15:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:10.565+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:15:10.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:10.565+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:15:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:10 smithi104 ceph-mon[36624]: pgmap v2303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:10 smithi081 ceph-mon[31453]: pgmap v2303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:12 smithi153 ceph-mon[36569]: pgmap v2304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:12 smithi104 ceph-mon[36624]: pgmap v2304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:12 smithi081 ceph-mon[31453]: pgmap v2304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:14 smithi153 ceph-mon[36569]: pgmap v2305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:14 smithi104 ceph-mon[36624]: pgmap v2305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:14 smithi081 ceph-mon[31453]: pgmap v2305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:16 smithi153 ceph-mon[36569]: pgmap v2306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:16 smithi104 ceph-mon[36624]: pgmap v2306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:16 smithi081 ceph-mon[31453]: pgmap v2306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:18 smithi153 ceph-mon[36569]: pgmap v2307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:18 smithi104 ceph-mon[36624]: pgmap v2307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:18 smithi081 ceph-mon[31453]: pgmap v2307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:20 smithi153 ceph-mon[36569]: pgmap v2308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:20 smithi104 ceph-mon[36624]: pgmap v2308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:20 smithi081 ceph-mon[31453]: pgmap v2308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:21 smithi081 ceph-mon[31453]: pgmap v2309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:21 smithi153 ceph-mon[36569]: pgmap v2309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:21 smithi104 ceph-mon[36624]: pgmap v2309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:23 smithi081 ceph-mon[31453]: pgmap v2310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:23 smithi153 ceph-mon[36569]: pgmap v2310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:23 smithi104 ceph-mon[36624]: pgmap v2310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:25.577+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:15:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:25.577+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:15:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:25.577+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:15:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:25 smithi081 ceph-mon[31453]: pgmap v2311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:25 smithi153 ceph-mon[36569]: pgmap v2311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:25 smithi104 ceph-mon[36624]: pgmap v2311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:27 smithi081 ceph-mon[31453]: pgmap v2312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:27 smithi153 ceph-mon[36569]: pgmap v2312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:27 smithi104 ceph-mon[36624]: pgmap v2312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:29 smithi081 ceph-mon[31453]: pgmap v2313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:29 smithi153 ceph-mon[36569]: pgmap v2313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:29 smithi104 ceph-mon[36624]: pgmap v2313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:31 smithi081 ceph-mon[31453]: pgmap v2314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:15:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:31 smithi153 ceph-mon[36569]: pgmap v2314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:15:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:31 smithi104 ceph-mon[36624]: pgmap v2314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:15:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:15:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:15:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:32 smithi081 ceph-mon[31453]: pgmap v2315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:32 smithi081 ceph-mon[31453]: pgmap v2316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:15:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:15:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:15:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:32 smithi153 ceph-mon[36569]: pgmap v2315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:32 smithi153 ceph-mon[36569]: pgmap v2316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:15:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:15:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:15:33.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:32 smithi104 ceph-mon[36624]: pgmap v2315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:33.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:32 smithi104 ceph-mon[36624]: pgmap v2316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:33.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:15:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:35 smithi081 ceph-mon[31453]: pgmap v2317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:35 smithi153 ceph-mon[36569]: pgmap v2317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:35 smithi104 ceph-mon[36624]: pgmap v2317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:37 smithi081 ceph-mon[31453]: pgmap v2318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:37 smithi153 ceph-mon[36569]: pgmap v2318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:37 smithi104 ceph-mon[36624]: pgmap v2318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:39 smithi081 ceph-mon[31453]: pgmap v2319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:39 smithi153 ceph-mon[36569]: pgmap v2319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:39 smithi104 ceph-mon[36624]: pgmap v2319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:40.593+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:15:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:40.593+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:15:40.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:40.593+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:15:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:41 smithi153 ceph-mon[36569]: pgmap v2320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:41 smithi104 ceph-mon[36624]: pgmap v2320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:41 smithi081 ceph-mon[31453]: pgmap v2320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:43 smithi153 ceph-mon[36569]: pgmap v2321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:43 smithi104 ceph-mon[36624]: pgmap v2321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:43 smithi081 ceph-mon[31453]: pgmap v2321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:45 smithi153 ceph-mon[36569]: pgmap v2322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:45 smithi104 ceph-mon[36624]: pgmap v2322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:45 smithi081 ceph-mon[31453]: pgmap v2322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:47 smithi081 ceph-mon[31453]: pgmap v2323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:47 smithi153 ceph-mon[36569]: pgmap v2323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:47 smithi104 ceph-mon[36624]: pgmap v2323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:49 smithi153 ceph-mon[36569]: pgmap v2324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:49 smithi104 ceph-mon[36624]: pgmap v2324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:49 smithi081 ceph-mon[31453]: pgmap v2324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:51 smithi153 ceph-mon[36569]: pgmap v2325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:51 smithi104 ceph-mon[36624]: pgmap v2325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:51 smithi081 ceph-mon[31453]: pgmap v2325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:53 smithi153 ceph-mon[36569]: pgmap v2326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:53 smithi104 ceph-mon[36624]: pgmap v2326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:53 smithi081 ceph-mon[31453]: pgmap v2326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:55 smithi153 ceph-mon[36569]: pgmap v2327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:55 smithi104 ceph-mon[36624]: pgmap v2327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:55.600 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:55.600+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:15:55.600 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:55.600+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:15:55.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:15:55.600+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:15:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:55 smithi081 ceph-mon[31453]: pgmap v2327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:57 smithi153 ceph-mon[36569]: pgmap v2328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:57 smithi104 ceph-mon[36624]: pgmap v2328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:57 smithi081 ceph-mon[31453]: pgmap v2328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:15:59 smithi153 ceph-mon[36569]: pgmap v2329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:15:59 smithi104 ceph-mon[36624]: pgmap v2329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:15:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:15:59 smithi081 ceph-mon[31453]: pgmap v2329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:01.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:01 smithi081 ceph-mon[31453]: pgmap v2330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:01 smithi153 ceph-mon[36569]: pgmap v2330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:01 smithi104 ceph-mon[36624]: pgmap v2330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:03 smithi153 ceph-mon[36569]: pgmap v2331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:03 smithi104 ceph-mon[36624]: pgmap v2331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:03 smithi081 ceph-mon[31453]: pgmap v2331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:05 smithi153 ceph-mon[36569]: pgmap v2332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:05 smithi104 ceph-mon[36624]: pgmap v2332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:05 smithi081 ceph-mon[31453]: pgmap v2332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:07 smithi153 ceph-mon[36569]: pgmap v2333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:07 smithi104 ceph-mon[36624]: pgmap v2333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:07 smithi081 ceph-mon[31453]: pgmap v2333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:09 smithi153 ceph-mon[36569]: pgmap v2334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:09 smithi104 ceph-mon[36624]: pgmap v2334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:09 smithi081 ceph-mon[31453]: pgmap v2334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:10.616+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:16:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:10.616+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:16:10.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:10.616+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:16:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:11 smithi153 ceph-mon[36569]: pgmap v2335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:11 smithi104 ceph-mon[36624]: pgmap v2335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:11 smithi081 ceph-mon[31453]: pgmap v2335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.095+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T22:16:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.095+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T22:16:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.095+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 0x7f16e80b1520 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f40d1000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T22:16:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.095+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 0x7f16e80b1520 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f40d1000 comp rx=0 tx=0).stop 2024-04-12T22:16:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.095+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-12T22:16:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.095+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T22:16:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.095+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a60f0 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T22:16:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.095+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 0x7f16e80a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T22:16:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.096+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 0x7f16e80a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T22:16:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:12.097+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 0x7f16e80a5040 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006620 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T22:16:13.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:13.289+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:16:13.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:13.289+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:16:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:13.296+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:16:13.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:13.296+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:16:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:13 smithi153 ceph-mon[36569]: pgmap v2336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:13 smithi104 ceph-mon[36624]: pgmap v2336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:13 smithi081 ceph-mon[31453]: pgmap v2336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:15 smithi153 ceph-mon[36569]: pgmap v2337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:15 smithi104 ceph-mon[36624]: pgmap v2337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:15 smithi081 ceph-mon[31453]: pgmap v2337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:17 smithi153 ceph-mon[36569]: pgmap v2338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:17 smithi104 ceph-mon[36624]: pgmap v2338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:17 smithi081 ceph-mon[31453]: pgmap v2338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:19 smithi153 ceph-mon[36569]: pgmap v2339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:19 smithi104 ceph-mon[36624]: pgmap v2339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:19 smithi081 ceph-mon[31453]: pgmap v2339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:21 smithi153 ceph-mon[36569]: pgmap v2340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:21 smithi104 ceph-mon[36624]: pgmap v2340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:21 smithi081 ceph-mon[31453]: pgmap v2340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:23 smithi153 ceph-mon[36569]: pgmap v2341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:23 smithi104 ceph-mon[36624]: pgmap v2341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:23 smithi081 ceph-mon[31453]: pgmap v2341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:24 smithi104 ceph-mon[36624]: pgmap v2342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:24 smithi081 ceph-mon[31453]: pgmap v2342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:24 smithi153 ceph-mon[36569]: pgmap v2342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:25.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:25.627+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:16:25.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:25.627+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:16:25.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:25.627+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:16:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:27 smithi153 ceph-mon[36569]: pgmap v2343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:27 smithi104 ceph-mon[36624]: pgmap v2343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:27 smithi081 ceph-mon[31453]: pgmap v2343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:29 smithi153 ceph-mon[36569]: pgmap v2344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:29 smithi104 ceph-mon[36624]: pgmap v2344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:29 smithi081 ceph-mon[31453]: pgmap v2344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:31.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:31 smithi081 ceph-mon[31453]: pgmap v2345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:31 smithi153 ceph-mon[36569]: pgmap v2345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:31 smithi104 ceph-mon[36624]: pgmap v2345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:33 smithi153 ceph-mon[36569]: pgmap v2346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:16:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:33 smithi104 ceph-mon[36624]: pgmap v2346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:16:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:33 smithi081 ceph-mon[31453]: pgmap v2346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:16:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:16:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:16:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:16:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:16:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:16:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:16:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:16:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:16:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:16:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:35 smithi153 ceph-mon[36569]: pgmap v2347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:35 smithi153 ceph-mon[36569]: pgmap v2348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:35 smithi104 ceph-mon[36624]: pgmap v2347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:35 smithi104 ceph-mon[36624]: pgmap v2348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:35 smithi081 ceph-mon[31453]: pgmap v2347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:35 smithi081 ceph-mon[31453]: pgmap v2348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:37 smithi153 ceph-mon[36569]: pgmap v2349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:37 smithi104 ceph-mon[36624]: pgmap v2349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:37 smithi081 ceph-mon[31453]: pgmap v2349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:39 smithi153 ceph-mon[36569]: pgmap v2350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:39 smithi104 ceph-mon[36624]: pgmap v2350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:39 smithi081 ceph-mon[31453]: pgmap v2350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:40.643+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:16:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:40.643+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:16:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:40.643+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:16:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:41 smithi153 ceph-mon[36569]: pgmap v2351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:41 smithi104 ceph-mon[36624]: pgmap v2351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:41 smithi081 ceph-mon[31453]: pgmap v2351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:43 smithi153 ceph-mon[36569]: pgmap v2352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:43 smithi104 ceph-mon[36624]: pgmap v2352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:43 smithi081 ceph-mon[31453]: pgmap v2352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:45 smithi153 ceph-mon[36569]: pgmap v2353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:45 smithi104 ceph-mon[36624]: pgmap v2353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:45 smithi081 ceph-mon[31453]: pgmap v2353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:47 smithi153 ceph-mon[36569]: pgmap v2354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:47 smithi104 ceph-mon[36624]: pgmap v2354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:47 smithi081 ceph-mon[31453]: pgmap v2354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:49 smithi081 ceph-mon[31453]: pgmap v2355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:49 smithi153 ceph-mon[36569]: pgmap v2355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:49 smithi104 ceph-mon[36624]: pgmap v2355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:51 smithi153 ceph-mon[36569]: pgmap v2356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:51 smithi104 ceph-mon[36624]: pgmap v2356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:51 smithi081 ceph-mon[31453]: pgmap v2356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:53 smithi153 ceph-mon[36569]: pgmap v2357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:53 smithi104 ceph-mon[36624]: pgmap v2357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:53 smithi081 ceph-mon[31453]: pgmap v2357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:55 smithi153 ceph-mon[36569]: pgmap v2358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:55 smithi104 ceph-mon[36624]: pgmap v2358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:55 smithi081 ceph-mon[31453]: pgmap v2358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:55.659+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:16:55.659 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:55.659+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:16:55.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:16:55.659+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:16:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:57 smithi153 ceph-mon[36569]: pgmap v2359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:57 smithi104 ceph-mon[36624]: pgmap v2359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:57 smithi081 ceph-mon[31453]: pgmap v2359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:16:59 smithi153 ceph-mon[36569]: pgmap v2360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:16:59 smithi104 ceph-mon[36624]: pgmap v2360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:16:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:16:59 smithi081 ceph-mon[31453]: pgmap v2360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:01 smithi153 ceph-mon[36569]: pgmap v2361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:01 smithi104 ceph-mon[36624]: pgmap v2361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:01 smithi081 ceph-mon[31453]: pgmap v2361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:03 smithi153 ceph-mon[36569]: pgmap v2362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:03 smithi104 ceph-mon[36624]: pgmap v2362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:03 smithi081 ceph-mon[31453]: pgmap v2362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:05 smithi153 ceph-mon[36569]: pgmap v2363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:05 smithi104 ceph-mon[36624]: pgmap v2363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:05 smithi081 ceph-mon[31453]: pgmap v2363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:07 smithi153 ceph-mon[36569]: pgmap v2364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:07 smithi104 ceph-mon[36624]: pgmap v2364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:07 smithi081 ceph-mon[31453]: pgmap v2364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:09 smithi153 ceph-mon[36569]: pgmap v2365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:09 smithi104 ceph-mon[36624]: pgmap v2365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:09 smithi081 ceph-mon[31453]: pgmap v2365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:10.675+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:17:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:10.675+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:17:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:10.675+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:17:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:11 smithi153 ceph-mon[36569]: pgmap v2366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:11 smithi104 ceph-mon[36624]: pgmap v2366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:11 smithi081 ceph-mon[31453]: pgmap v2366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:13 smithi153 ceph-mon[36569]: pgmap v2367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:13 smithi104 ceph-mon[36624]: pgmap v2367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:13 smithi081 ceph-mon[31453]: pgmap v2367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:15 smithi153 ceph-mon[36569]: pgmap v2368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:15 smithi104 ceph-mon[36624]: pgmap v2368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:15 smithi081 ceph-mon[31453]: pgmap v2368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:17 smithi104 ceph-mon[36624]: pgmap v2369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:17 smithi081 ceph-mon[31453]: pgmap v2369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:17 smithi153 ceph-mon[36569]: pgmap v2369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:19 smithi104 ceph-mon[36624]: pgmap v2370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:19 smithi081 ceph-mon[31453]: pgmap v2370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:19 smithi153 ceph-mon[36569]: pgmap v2370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:20 smithi104 ceph-mon[36624]: pgmap v2371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:20 smithi081 ceph-mon[31453]: pgmap v2371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:20 smithi153 ceph-mon[36569]: pgmap v2371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:22 smithi153 ceph-mon[36569]: pgmap v2372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:22 smithi104 ceph-mon[36624]: pgmap v2372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:22 smithi081 ceph-mon[31453]: pgmap v2372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:24 smithi153 ceph-mon[36569]: pgmap v2373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:24 smithi104 ceph-mon[36624]: pgmap v2373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:24 smithi081 ceph-mon[31453]: pgmap v2373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:25.690+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:17:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:25.690+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:17:25.691 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:25.690+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:17:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:26 smithi153 ceph-mon[36569]: pgmap v2374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:26 smithi104 ceph-mon[36624]: pgmap v2374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:26 smithi081 ceph-mon[31453]: pgmap v2374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:28 smithi153 ceph-mon[36569]: pgmap v2375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:28 smithi104 ceph-mon[36624]: pgmap v2375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:28 smithi081 ceph-mon[31453]: pgmap v2375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:30 smithi153 ceph-mon[36569]: pgmap v2376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:30 smithi104 ceph-mon[36624]: pgmap v2376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:30 smithi081 ceph-mon[31453]: pgmap v2376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:32 smithi153 ceph-mon[36569]: pgmap v2377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:32 smithi104 ceph-mon[36624]: pgmap v2377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:32 smithi081 ceph-mon[31453]: pgmap v2377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:34 smithi153 ceph-mon[36569]: pgmap v2378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:17:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:34 smithi104 ceph-mon[36624]: pgmap v2378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:17:35.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:34 smithi081 ceph-mon[31453]: pgmap v2378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:35.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:17:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:17:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:17:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:17:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:17:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:17:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:17:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:17:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:17:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:17:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:36 smithi153 ceph-mon[36569]: pgmap v2379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:36 smithi153 ceph-mon[36569]: pgmap v2380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:36 smithi104 ceph-mon[36624]: pgmap v2379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:36 smithi104 ceph-mon[36624]: pgmap v2380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:36 smithi081 ceph-mon[31453]: pgmap v2379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:36 smithi081 ceph-mon[31453]: pgmap v2380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:38 smithi153 ceph-mon[36569]: pgmap v2381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:38 smithi104 ceph-mon[36624]: pgmap v2381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:38 smithi081 ceph-mon[31453]: pgmap v2381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:40.705+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:17:40.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:40.705+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:17:40.706 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:40.705+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:17:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:40 smithi153 ceph-mon[36569]: pgmap v2382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:40 smithi104 ceph-mon[36624]: pgmap v2382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:40 smithi081 ceph-mon[31453]: pgmap v2382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:42 smithi153 ceph-mon[36569]: pgmap v2383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:42 smithi104 ceph-mon[36624]: pgmap v2383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:42 smithi081 ceph-mon[31453]: pgmap v2383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:44 smithi104 ceph-mon[36624]: pgmap v2384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:44 smithi081 ceph-mon[31453]: pgmap v2384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:44 smithi153 ceph-mon[36569]: pgmap v2384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:46 smithi104 ceph-mon[36624]: pgmap v2385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:47.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:46 smithi153 ceph-mon[36569]: pgmap v2385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:46 smithi081 ceph-mon[31453]: pgmap v2385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:48 smithi104 ceph-mon[36624]: pgmap v2386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:48 smithi081 ceph-mon[31453]: pgmap v2386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:48 smithi153 ceph-mon[36569]: pgmap v2386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:50 smithi104 ceph-mon[36624]: pgmap v2387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:50 smithi081 ceph-mon[31453]: pgmap v2387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:50 smithi153 ceph-mon[36569]: pgmap v2387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:52 smithi104 ceph-mon[36624]: pgmap v2388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:52 smithi081 ceph-mon[31453]: pgmap v2388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:52 smithi153 ceph-mon[36569]: pgmap v2388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:54 smithi081 ceph-mon[31453]: pgmap v2389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:54 smithi153 ceph-mon[36569]: pgmap v2389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:54 smithi104 ceph-mon[36624]: pgmap v2389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:55.714+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:17:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:55.714+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:17:55.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:17:55.714+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:17:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:56 smithi081 ceph-mon[31453]: pgmap v2390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:56 smithi153 ceph-mon[36569]: pgmap v2390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:56 smithi104 ceph-mon[36624]: pgmap v2390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:17:58 smithi081 ceph-mon[31453]: pgmap v2391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:17:58 smithi153 ceph-mon[36569]: pgmap v2391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:17:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:17:58 smithi104 ceph-mon[36624]: pgmap v2391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:00 smithi081 ceph-mon[31453]: pgmap v2392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:00 smithi153 ceph-mon[36569]: pgmap v2392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:00 smithi104 ceph-mon[36624]: pgmap v2392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:02 smithi081 ceph-mon[31453]: pgmap v2393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:02 smithi153 ceph-mon[36569]: pgmap v2393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:02 smithi104 ceph-mon[36624]: pgmap v2393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:04 smithi081 ceph-mon[31453]: pgmap v2394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:04 smithi153 ceph-mon[36569]: pgmap v2394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:04 smithi104 ceph-mon[36624]: pgmap v2394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:06 smithi081 ceph-mon[31453]: pgmap v2395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:06 smithi153 ceph-mon[36569]: pgmap v2395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:06 smithi104 ceph-mon[36624]: pgmap v2395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:08 smithi081 ceph-mon[31453]: pgmap v2396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:08 smithi153 ceph-mon[36569]: pgmap v2396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:08 smithi104 ceph-mon[36624]: pgmap v2396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:10.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:10.725+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:18:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:10.725+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:18:10.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:10.725+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:18:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:10 smithi081 ceph-mon[31453]: pgmap v2397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:10 smithi153 ceph-mon[36569]: pgmap v2397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:10 smithi104 ceph-mon[36624]: pgmap v2397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:12 smithi081 ceph-mon[31453]: pgmap v2398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:12 smithi153 ceph-mon[36569]: pgmap v2398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:12 smithi104 ceph-mon[36624]: pgmap v2398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:14 smithi081 ceph-mon[31453]: pgmap v2399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:14 smithi153 ceph-mon[36569]: pgmap v2399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:14 smithi104 ceph-mon[36624]: pgmap v2399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:16 smithi081 ceph-mon[31453]: pgmap v2400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:16 smithi153 ceph-mon[36569]: pgmap v2400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:16 smithi104 ceph-mon[36624]: pgmap v2400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:18 smithi081 ceph-mon[31453]: pgmap v2401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:18 smithi153 ceph-mon[36569]: pgmap v2401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:18 smithi104 ceph-mon[36624]: pgmap v2401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:20 smithi153 ceph-mon[36569]: pgmap v2402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:20 smithi104 ceph-mon[36624]: pgmap v2402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:20 smithi081 ceph-mon[31453]: pgmap v2402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:22 smithi153 ceph-mon[36569]: pgmap v2403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:22 smithi104 ceph-mon[36624]: pgmap v2403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:22 smithi081 ceph-mon[31453]: pgmap v2403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:24 smithi153 ceph-mon[36569]: pgmap v2404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:24 smithi104 ceph-mon[36624]: pgmap v2404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:24 smithi081 ceph-mon[31453]: pgmap v2404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:25.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:25.727+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:18:25.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:25.727+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:18:25.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:25.727+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:18:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:26 smithi153 ceph-mon[36569]: pgmap v2405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:26 smithi104 ceph-mon[36624]: pgmap v2405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:26 smithi081 ceph-mon[31453]: pgmap v2405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:28 smithi153 ceph-mon[36569]: pgmap v2406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:28 smithi104 ceph-mon[36624]: pgmap v2406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:28 smithi081 ceph-mon[31453]: pgmap v2406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:30 smithi153 ceph-mon[36569]: pgmap v2407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:30 smithi104 ceph-mon[36624]: pgmap v2407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:30 smithi081 ceph-mon[31453]: pgmap v2407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:33.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:32 smithi081 ceph-mon[31453]: pgmap v2408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:32 smithi153 ceph-mon[36569]: pgmap v2408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:32 smithi104 ceph-mon[36624]: pgmap v2408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:34 smithi153 ceph-mon[36569]: pgmap v2409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:34 smithi104 ceph-mon[36624]: pgmap v2409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:34 smithi081 ceph-mon[31453]: pgmap v2409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:18:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:18:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:18:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:37 smithi153 ceph-mon[36569]: pgmap v2410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:37 smithi104 ceph-mon[36624]: pgmap v2410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:37 smithi081 ceph-mon[31453]: pgmap v2410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:39 smithi081 ceph-mon[31453]: pgmap v2411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:39 smithi153 ceph-mon[36569]: pgmap v2411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:39 smithi104 ceph-mon[36624]: pgmap v2411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:40.742+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:18:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:40.742+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:18:40.743 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:40.742+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:18:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:41 smithi081 ceph-mon[31453]: pgmap v2412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:41 smithi153 ceph-mon[36569]: pgmap v2412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:41 smithi104 ceph-mon[36624]: pgmap v2412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:43 smithi081 ceph-mon[31453]: pgmap v2413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:18:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:18:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:43 smithi081 ceph-mon[31453]: pgmap v2414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:43 smithi081 ceph-mon[31453]: pgmap v2415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:43 smithi153 ceph-mon[36569]: pgmap v2413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:18:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:18:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:43 smithi153 ceph-mon[36569]: pgmap v2414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:43 smithi153 ceph-mon[36569]: pgmap v2415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:43 smithi104 ceph-mon[36624]: pgmap v2413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:18:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:18:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:43 smithi104 ceph-mon[36624]: pgmap v2414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:43 smithi104 ceph-mon[36624]: pgmap v2415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:18:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:45 smithi153 ceph-mon[36569]: pgmap v2416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:45 smithi104 ceph-mon[36624]: pgmap v2416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:45 smithi081 ceph-mon[31453]: pgmap v2416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:47 smithi153 ceph-mon[36569]: pgmap v2417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:47 smithi104 ceph-mon[36624]: pgmap v2417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:47 smithi081 ceph-mon[31453]: pgmap v2417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:49 smithi153 ceph-mon[36569]: pgmap v2418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:49 smithi104 ceph-mon[36624]: pgmap v2418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:49 smithi081 ceph-mon[31453]: pgmap v2418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:51 smithi104 ceph-mon[36624]: pgmap v2419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:51 smithi081 ceph-mon[31453]: pgmap v2419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:51 smithi153 ceph-mon[36569]: pgmap v2419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:53 smithi104 ceph-mon[36624]: pgmap v2420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:53 smithi081 ceph-mon[31453]: pgmap v2420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:53 smithi153 ceph-mon[36569]: pgmap v2420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:55 smithi104 ceph-mon[36624]: pgmap v2421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:55 smithi081 ceph-mon[31453]: pgmap v2421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:55.755+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:18:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:55.755+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:18:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:18:55.755+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:18:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:55 smithi153 ceph-mon[36569]: pgmap v2421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:57 smithi104 ceph-mon[36624]: pgmap v2422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:57 smithi081 ceph-mon[31453]: pgmap v2422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:57 smithi153 ceph-mon[36569]: pgmap v2422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:18:59 smithi104 ceph-mon[36624]: pgmap v2423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:18:59 smithi081 ceph-mon[31453]: pgmap v2423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:18:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:18:59 smithi153 ceph-mon[36569]: pgmap v2423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:00 smithi081 ceph-mon[31453]: pgmap v2424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:00 smithi153 ceph-mon[36569]: pgmap v2424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:00 smithi104 ceph-mon[36624]: pgmap v2424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:03 smithi153 ceph-mon[36569]: pgmap v2425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:03 smithi104 ceph-mon[36624]: pgmap v2425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:03 smithi081 ceph-mon[31453]: pgmap v2425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:05 smithi104 ceph-mon[36624]: pgmap v2426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:05 smithi081 ceph-mon[31453]: pgmap v2426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:05 smithi153 ceph-mon[36569]: pgmap v2426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:07 smithi104 ceph-mon[36624]: pgmap v2427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:07 smithi081 ceph-mon[31453]: pgmap v2427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:07 smithi153 ceph-mon[36569]: pgmap v2427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:09 smithi104 ceph-mon[36624]: pgmap v2428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:09 smithi081 ceph-mon[31453]: pgmap v2428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:09 smithi153 ceph-mon[36569]: pgmap v2428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:10.771+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:19:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:10.771+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:19:10.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:10.771+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:19:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:11 smithi104 ceph-mon[36624]: pgmap v2429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:11 smithi081 ceph-mon[31453]: pgmap v2429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:11 smithi153 ceph-mon[36569]: pgmap v2429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:13.317+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:19:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:13.318+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:19:13.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:13.321+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:19:13.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:13.322+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:19:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:13 smithi104 ceph-mon[36624]: pgmap v2430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:13 smithi081 ceph-mon[31453]: pgmap v2430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:13 smithi153 ceph-mon[36569]: pgmap v2430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:15 smithi104 ceph-mon[36624]: pgmap v2431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:15 smithi081 ceph-mon[31453]: pgmap v2431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:15 smithi153 ceph-mon[36569]: pgmap v2431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:16 smithi081 ceph-mon[31453]: pgmap v2432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:16 smithi153 ceph-mon[36569]: pgmap v2432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:16 smithi104 ceph-mon[36624]: pgmap v2432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:19 smithi104 ceph-mon[36624]: pgmap v2433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:19 smithi081 ceph-mon[31453]: pgmap v2433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:19 smithi153 ceph-mon[36569]: pgmap v2433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:21 smithi104 ceph-mon[36624]: pgmap v2434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:21 smithi081 ceph-mon[31453]: pgmap v2434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:21 smithi153 ceph-mon[36569]: pgmap v2434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:23 smithi104 ceph-mon[36624]: pgmap v2435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:23 smithi081 ceph-mon[31453]: pgmap v2435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:23 smithi153 ceph-mon[36569]: pgmap v2435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:25.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:25 smithi081 ceph-mon[31453]: pgmap v2436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:25 smithi104 ceph-mon[36624]: pgmap v2436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:25.787+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:19:25.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:25.787+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:19:25.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:25.787+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:19:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:25 smithi153 ceph-mon[36569]: pgmap v2436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:27 smithi081 ceph-mon[31453]: pgmap v2437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:27 smithi153 ceph-mon[36569]: pgmap v2437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:27 smithi104 ceph-mon[36624]: pgmap v2437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:29 smithi081 ceph-mon[31453]: pgmap v2438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:29 smithi153 ceph-mon[36569]: pgmap v2438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:29 smithi104 ceph-mon[36624]: pgmap v2438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:30 smithi081 ceph-mon[31453]: pgmap v2439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:30 smithi153 ceph-mon[36569]: pgmap v2439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:30 smithi104 ceph-mon[36624]: pgmap v2439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:33 smithi104 ceph-mon[36624]: pgmap v2440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:33 smithi081 ceph-mon[31453]: pgmap v2440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:33 smithi153 ceph-mon[36569]: pgmap v2440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:35 smithi104 ceph-mon[36624]: pgmap v2441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:35 smithi081 ceph-mon[31453]: pgmap v2441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:35 smithi153 ceph-mon[36569]: pgmap v2441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:37 smithi104 ceph-mon[36624]: pgmap v2442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:37 smithi081 ceph-mon[31453]: pgmap v2442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:37 smithi153 ceph-mon[36569]: pgmap v2442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:39 smithi081 ceph-mon[31453]: pgmap v2443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:39 smithi153 ceph-mon[36569]: pgmap v2443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:39 smithi104 ceph-mon[36624]: pgmap v2443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:40 smithi081 ceph-mon[31453]: pgmap v2444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:40.789+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:19:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:40.789+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:19:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:40.789+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:19:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:40 smithi153 ceph-mon[36569]: pgmap v2444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:40 smithi104 ceph-mon[36624]: pgmap v2444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:43 smithi104 ceph-mon[36624]: pgmap v2445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:19:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:43 smithi081 ceph-mon[31453]: pgmap v2445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:19:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:43 smithi153 ceph-mon[36569]: pgmap v2445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:19:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:19:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:19:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:19:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:19:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:19:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:19:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:19:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:45 smithi081 ceph-mon[31453]: pgmap v2446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:45 smithi081 ceph-mon[31453]: pgmap v2447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:45 smithi153 ceph-mon[36569]: pgmap v2446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:45 smithi153 ceph-mon[36569]: pgmap v2447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:45 smithi104 ceph-mon[36624]: pgmap v2446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:45 smithi104 ceph-mon[36624]: pgmap v2447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:46 smithi081 ceph-mon[31453]: pgmap v2448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:46 smithi153 ceph-mon[36569]: pgmap v2448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:46 smithi104 ceph-mon[36624]: pgmap v2448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:48 smithi081 ceph-mon[31453]: pgmap v2449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:48 smithi153 ceph-mon[36569]: pgmap v2449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:48 smithi104 ceph-mon[36624]: pgmap v2449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:50 smithi081 ceph-mon[31453]: pgmap v2450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:50 smithi153 ceph-mon[36569]: pgmap v2450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:50 smithi104 ceph-mon[36624]: pgmap v2450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:52 smithi081 ceph-mon[31453]: pgmap v2451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:52 smithi153 ceph-mon[36569]: pgmap v2451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:52 smithi104 ceph-mon[36624]: pgmap v2451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:55.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:54 smithi104 ceph-mon[36624]: pgmap v2452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:54 smithi081 ceph-mon[31453]: pgmap v2452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:54 smithi153 ceph-mon[36569]: pgmap v2452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:55.790+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:19:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:55.790+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:19:55.791 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:19:55.790+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:19:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:56 smithi081 ceph-mon[31453]: pgmap v2453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:56 smithi153 ceph-mon[36569]: pgmap v2453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:56 smithi104 ceph-mon[36624]: pgmap v2453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:19:58 smithi081 ceph-mon[31453]: pgmap v2454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:19:58 smithi153 ceph-mon[36569]: pgmap v2454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:19:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:19:58 smithi104 ceph-mon[36624]: pgmap v2454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:00 smithi081 ceph-mon[31453]: pgmap v2455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:20:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:00 smithi153 ceph-mon[36569]: pgmap v2455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:20:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:00 smithi104 ceph-mon[36624]: pgmap v2455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:20:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:02 smithi081 ceph-mon[31453]: pgmap v2456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:02 smithi153 ceph-mon[36569]: pgmap v2456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:02 smithi104 ceph-mon[36624]: pgmap v2456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:04 smithi153 ceph-mon[36569]: pgmap v2457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:04 smithi104 ceph-mon[36624]: pgmap v2457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:04 smithi081 ceph-mon[31453]: pgmap v2457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:06 smithi153 ceph-mon[36569]: pgmap v2458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:06 smithi104 ceph-mon[36624]: pgmap v2458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:06 smithi081 ceph-mon[31453]: pgmap v2458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:08 smithi153 ceph-mon[36569]: pgmap v2459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:08 smithi104 ceph-mon[36624]: pgmap v2459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:08 smithi081 ceph-mon[31453]: pgmap v2459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:10.801+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:20:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:10.801+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:20:10.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:10.801+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:20:11.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:10 smithi104 ceph-mon[36624]: pgmap v2460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:11.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:10 smithi081 ceph-mon[31453]: pgmap v2460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:10 smithi153 ceph-mon[36569]: pgmap v2460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:12 smithi153 ceph-mon[36569]: pgmap v2461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:12 smithi104 ceph-mon[36624]: pgmap v2461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:12 smithi081 ceph-mon[31453]: pgmap v2461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:14 smithi153 ceph-mon[36569]: pgmap v2462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:14 smithi104 ceph-mon[36624]: pgmap v2462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:14 smithi081 ceph-mon[31453]: pgmap v2462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:16 smithi153 ceph-mon[36569]: pgmap v2463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:16 smithi104 ceph-mon[36624]: pgmap v2463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:16 smithi081 ceph-mon[31453]: pgmap v2463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:19.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:18 smithi153 ceph-mon[36569]: pgmap v2464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:18 smithi104 ceph-mon[36624]: pgmap v2464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:18 smithi081 ceph-mon[31453]: pgmap v2464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:20 smithi153 ceph-mon[36569]: pgmap v2465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:20 smithi104 ceph-mon[36624]: pgmap v2465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:20 smithi081 ceph-mon[31453]: pgmap v2465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:22 smithi153 ceph-mon[36569]: pgmap v2466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:22 smithi104 ceph-mon[36624]: pgmap v2466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:22 smithi081 ceph-mon[31453]: pgmap v2466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:24 smithi153 ceph-mon[36569]: pgmap v2467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:24 smithi104 ceph-mon[36624]: pgmap v2467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:24 smithi081 ceph-mon[31453]: pgmap v2467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:25.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:25.814+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:20:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:25.814+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:20:25.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:25.814+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:20:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:26 smithi153 ceph-mon[36569]: pgmap v2468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:26 smithi104 ceph-mon[36624]: pgmap v2468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:26 smithi081 ceph-mon[31453]: pgmap v2468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:28 smithi153 ceph-mon[36569]: pgmap v2469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:28 smithi104 ceph-mon[36624]: pgmap v2469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:28 smithi081 ceph-mon[31453]: pgmap v2469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:30 smithi153 ceph-mon[36569]: pgmap v2470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:30 smithi104 ceph-mon[36624]: pgmap v2470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:30 smithi081 ceph-mon[31453]: pgmap v2470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:32 smithi153 ceph-mon[36569]: pgmap v2471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:32 smithi104 ceph-mon[36624]: pgmap v2471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:32 smithi081 ceph-mon[31453]: pgmap v2471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:34 smithi153 ceph-mon[36569]: pgmap v2472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:34 smithi104 ceph-mon[36624]: pgmap v2472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:34 smithi081 ceph-mon[31453]: pgmap v2472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:36 smithi153 ceph-mon[36569]: pgmap v2473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:36 smithi104 ceph-mon[36624]: pgmap v2473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:36 smithi081 ceph-mon[31453]: pgmap v2473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:38 smithi153 ceph-mon[36569]: pgmap v2474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:38 smithi104 ceph-mon[36624]: pgmap v2474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:38 smithi081 ceph-mon[31453]: pgmap v2474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:40.829+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:20:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:40.829+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:20:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:40.829+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:20:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:41 smithi153 ceph-mon[36569]: pgmap v2475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:41 smithi104 ceph-mon[36624]: pgmap v2475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:41 smithi081 ceph-mon[31453]: pgmap v2475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:43 smithi153 ceph-mon[36569]: pgmap v2476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:43 smithi104 ceph-mon[36624]: pgmap v2476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:43 smithi081 ceph-mon[31453]: pgmap v2476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:45 smithi153 ceph-mon[36569]: pgmap v2477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:20:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:45 smithi104 ceph-mon[36624]: pgmap v2477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:20:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:45 smithi081 ceph-mon[31453]: pgmap v2477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:20:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:20:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:20:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:46 smithi153 ceph-mon[36569]: pgmap v2478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:46 smithi153 ceph-mon[36569]: pgmap v2479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:20:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:20:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:20:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:46 smithi104 ceph-mon[36624]: pgmap v2478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:46 smithi104 ceph-mon[36624]: pgmap v2479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:20:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:20:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:20:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:46 smithi081 ceph-mon[31453]: pgmap v2478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:46 smithi081 ceph-mon[31453]: pgmap v2479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:20:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:47 smithi081 ceph-mon[31453]: pgmap v2480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:47 smithi153 ceph-mon[36569]: pgmap v2480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:47 smithi104 ceph-mon[36624]: pgmap v2480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:49 smithi081 ceph-mon[31453]: pgmap v2481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:49 smithi153 ceph-mon[36569]: pgmap v2481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:49 smithi104 ceph-mon[36624]: pgmap v2481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:51 smithi081 ceph-mon[31453]: pgmap v2482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:51 smithi153 ceph-mon[36569]: pgmap v2482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:51 smithi104 ceph-mon[36624]: pgmap v2482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:53 smithi081 ceph-mon[31453]: pgmap v2483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:53 smithi153 ceph-mon[36569]: pgmap v2483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:53 smithi104 ceph-mon[36624]: pgmap v2483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:55 smithi081 ceph-mon[31453]: pgmap v2484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:55 smithi153 ceph-mon[36569]: pgmap v2484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:55 smithi104 ceph-mon[36624]: pgmap v2484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:55.844+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:20:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:55.844+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:20:55.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:20:55.844+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:20:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:57 smithi081 ceph-mon[31453]: pgmap v2485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:57 smithi153 ceph-mon[36569]: pgmap v2485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:57 smithi104 ceph-mon[36624]: pgmap v2485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:20:59 smithi081 ceph-mon[31453]: pgmap v2486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:59.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:20:59 smithi153 ceph-mon[36569]: pgmap v2486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:20:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:20:59 smithi104 ceph-mon[36624]: pgmap v2486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:01 smithi081 ceph-mon[31453]: pgmap v2487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:01 smithi153 ceph-mon[36569]: pgmap v2487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:01 smithi104 ceph-mon[36624]: pgmap v2487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:03 smithi153 ceph-mon[36569]: pgmap v2488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:03 smithi104 ceph-mon[36624]: pgmap v2488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:03 smithi081 ceph-mon[31453]: pgmap v2488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:05.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:05 smithi081 ceph-mon[31453]: pgmap v2489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:05 smithi153 ceph-mon[36569]: pgmap v2489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:05 smithi104 ceph-mon[36624]: pgmap v2489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:07 smithi153 ceph-mon[36569]: pgmap v2490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:07 smithi104 ceph-mon[36624]: pgmap v2490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:07 smithi081 ceph-mon[31453]: pgmap v2490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:09 smithi153 ceph-mon[36569]: pgmap v2491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:09 smithi104 ceph-mon[36624]: pgmap v2491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:09 smithi081 ceph-mon[31453]: pgmap v2491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:10.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:10.853+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:21:10.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:10.853+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:21:10.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:10.853+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:21:11.744 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:11 smithi081 ceph-mon[31453]: pgmap v2492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:11 smithi153 ceph-mon[36569]: pgmap v2492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:11 smithi104 ceph-mon[36624]: pgmap v2492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:13 smithi081 ceph-mon[31453]: pgmap v2493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:13 smithi153 ceph-mon[36569]: pgmap v2493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:13 smithi104 ceph-mon[36624]: pgmap v2493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:15 smithi153 ceph-mon[36569]: pgmap v2494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:15 smithi104 ceph-mon[36624]: pgmap v2494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:15 smithi081 ceph-mon[31453]: pgmap v2494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:17 smithi153 ceph-mon[36569]: pgmap v2495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:17 smithi104 ceph-mon[36624]: pgmap v2495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:17 smithi081 ceph-mon[31453]: pgmap v2495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:19 smithi153 ceph-mon[36569]: pgmap v2496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:19 smithi104 ceph-mon[36624]: pgmap v2496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:19 smithi081 ceph-mon[31453]: pgmap v2496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:21.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:21 smithi153 ceph-mon[36569]: pgmap v2497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:21 smithi104 ceph-mon[36624]: pgmap v2497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:21 smithi081 ceph-mon[31453]: pgmap v2497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:23 smithi153 ceph-mon[36569]: pgmap v2498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:23 smithi104 ceph-mon[36624]: pgmap v2498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:23 smithi081 ceph-mon[31453]: pgmap v2498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:25 smithi153 ceph-mon[36569]: pgmap v2499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:25 smithi104 ceph-mon[36624]: pgmap v2499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:25.869+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:21:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:25.869+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:21:25.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:25.869+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:21:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:25 smithi081 ceph-mon[31453]: pgmap v2499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:27 smithi153 ceph-mon[36569]: pgmap v2500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:27 smithi104 ceph-mon[36624]: pgmap v2500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:27 smithi081 ceph-mon[31453]: pgmap v2500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:29 smithi153 ceph-mon[36569]: pgmap v2501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:29 smithi104 ceph-mon[36624]: pgmap v2501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:29 smithi081 ceph-mon[31453]: pgmap v2501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:31 smithi153 ceph-mon[36569]: pgmap v2502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:31 smithi104 ceph-mon[36624]: pgmap v2502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:31 smithi081 ceph-mon[31453]: pgmap v2502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:33 smithi153 ceph-mon[36569]: pgmap v2503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:33 smithi104 ceph-mon[36624]: pgmap v2503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:33 smithi081 ceph-mon[31453]: pgmap v2503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:35.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:35 smithi081 ceph-mon[31453]: pgmap v2504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:35 smithi153 ceph-mon[36569]: pgmap v2504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:35 smithi104 ceph-mon[36624]: pgmap v2504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:38 smithi153 ceph-mon[36569]: pgmap v2505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:38 smithi104 ceph-mon[36624]: pgmap v2505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:38 smithi081 ceph-mon[31453]: pgmap v2505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:40 smithi153 ceph-mon[36569]: pgmap v2506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:40 smithi104 ceph-mon[36624]: pgmap v2506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:40.875+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:21:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:40.876+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:21:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:40.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:21:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:40 smithi081 ceph-mon[31453]: pgmap v2506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:42 smithi153 ceph-mon[36569]: pgmap v2507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:42 smithi104 ceph-mon[36624]: pgmap v2507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:42 smithi081 ceph-mon[31453]: pgmap v2507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:44 smithi153 ceph-mon[36569]: pgmap v2508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:44 smithi104 ceph-mon[36624]: pgmap v2508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:44 smithi081 ceph-mon[31453]: pgmap v2508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:46 smithi153 ceph-mon[36569]: pgmap v2509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:21:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:46 smithi104 ceph-mon[36624]: pgmap v2509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:46 smithi081 ceph-mon[31453]: pgmap v2509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:21:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:21:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:21:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:47 smithi153 ceph-mon[36569]: pgmap v2510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:47 smithi153 ceph-mon[36569]: pgmap v2511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:21:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:21:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:21:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:47 smithi104 ceph-mon[36624]: pgmap v2510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:47 smithi104 ceph-mon[36624]: pgmap v2511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:21:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:21:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:47 smithi081 ceph-mon[31453]: pgmap v2510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:47 smithi081 ceph-mon[31453]: pgmap v2511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:21:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:49 smithi153 ceph-mon[36569]: pgmap v2512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:49 smithi104 ceph-mon[36624]: pgmap v2512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:49 smithi081 ceph-mon[31453]: pgmap v2512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:51 smithi153 ceph-mon[36569]: pgmap v2513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:51 smithi104 ceph-mon[36624]: pgmap v2513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:51 smithi081 ceph-mon[31453]: pgmap v2513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:53 smithi153 ceph-mon[36569]: pgmap v2514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:53 smithi104 ceph-mon[36624]: pgmap v2514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:53 smithi081 ceph-mon[31453]: pgmap v2514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:55.880+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:21:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:55.880+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:21:55.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:21:55.880+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:21:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:55 smithi153 ceph-mon[36569]: pgmap v2515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:55 smithi104 ceph-mon[36624]: pgmap v2515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:55 smithi081 ceph-mon[31453]: pgmap v2515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:57 smithi153 ceph-mon[36569]: pgmap v2516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:57 smithi104 ceph-mon[36624]: pgmap v2516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:21:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:57 smithi081 ceph-mon[31453]: pgmap v2516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:21:59 smithi153 ceph-mon[36569]: pgmap v2517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:00.328 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:21:59 smithi081 ceph-mon[31453]: pgmap v2517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:21:59 smithi104 ceph-mon[36624]: pgmap v2517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:01 smithi153 ceph-mon[36569]: pgmap v2518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:01 smithi104 ceph-mon[36624]: pgmap v2518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:01 smithi081 ceph-mon[31453]: pgmap v2518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:03 smithi153 ceph-mon[36569]: pgmap v2519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:03 smithi104 ceph-mon[36624]: pgmap v2519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:03 smithi081 ceph-mon[31453]: pgmap v2519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:05 smithi153 ceph-mon[36569]: pgmap v2520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:05 smithi104 ceph-mon[36624]: pgmap v2520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:05 smithi081 ceph-mon[31453]: pgmap v2520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:07 smithi153 ceph-mon[36569]: pgmap v2521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:07 smithi104 ceph-mon[36624]: pgmap v2521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:07 smithi081 ceph-mon[31453]: pgmap v2521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:10 smithi153 ceph-mon[36569]: pgmap v2522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:10 smithi104 ceph-mon[36624]: pgmap v2522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:10 smithi081 ceph-mon[31453]: pgmap v2522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:10.886+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:22:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:10.886+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:22:10.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:10.886+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:22:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:12 smithi153 ceph-mon[36569]: pgmap v2523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:12 smithi104 ceph-mon[36624]: pgmap v2523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:12 smithi081 ceph-mon[31453]: pgmap v2523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:13.347+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:22:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:13.348+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:22:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:13.348+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:22:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:13.348+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:22:13.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:13.348+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:22:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:13.348+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:22:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:13.349+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:22:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:13.349+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:22:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:14 smithi153 ceph-mon[36569]: pgmap v2524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:14 smithi104 ceph-mon[36624]: pgmap v2524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:14 smithi081 ceph-mon[31453]: pgmap v2524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:16 smithi153 ceph-mon[36569]: pgmap v2525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:16 smithi104 ceph-mon[36624]: pgmap v2525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:16 smithi081 ceph-mon[31453]: pgmap v2525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:18 smithi153 ceph-mon[36569]: pgmap v2526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:18 smithi104 ceph-mon[36624]: pgmap v2526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:18 smithi081 ceph-mon[31453]: pgmap v2526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:20 smithi153 ceph-mon[36569]: pgmap v2527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:20 smithi104 ceph-mon[36624]: pgmap v2527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:20 smithi081 ceph-mon[31453]: pgmap v2527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:22 smithi153 ceph-mon[36569]: pgmap v2528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:22 smithi104 ceph-mon[36624]: pgmap v2528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:22 smithi081 ceph-mon[31453]: pgmap v2528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:24 smithi153 ceph-mon[36569]: pgmap v2529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:24 smithi104 ceph-mon[36624]: pgmap v2529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:24 smithi081 ceph-mon[31453]: pgmap v2529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:25.887+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:22:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:25.887+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:22:25.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:25.887+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:22:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:26 smithi153 ceph-mon[36569]: pgmap v2530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:26 smithi104 ceph-mon[36624]: pgmap v2530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:26 smithi081 ceph-mon[31453]: pgmap v2530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:28 smithi153 ceph-mon[36569]: pgmap v2531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:28 smithi104 ceph-mon[36624]: pgmap v2531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:28 smithi081 ceph-mon[31453]: pgmap v2531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:30 smithi153 ceph-mon[36569]: pgmap v2532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:30 smithi104 ceph-mon[36624]: pgmap v2532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:30 smithi081 ceph-mon[31453]: pgmap v2532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:32 smithi104 ceph-mon[36624]: pgmap v2533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:32 smithi081 ceph-mon[31453]: pgmap v2533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:32 smithi153 ceph-mon[36569]: pgmap v2533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:34 smithi104 ceph-mon[36624]: pgmap v2534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:34 smithi081 ceph-mon[31453]: pgmap v2534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:34 smithi153 ceph-mon[36569]: pgmap v2534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:36 smithi104 ceph-mon[36624]: pgmap v2535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:36 smithi081 ceph-mon[31453]: pgmap v2535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:36 smithi153 ceph-mon[36569]: pgmap v2535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:38 smithi104 ceph-mon[36624]: pgmap v2536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:38 smithi081 ceph-mon[31453]: pgmap v2536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:38 smithi153 ceph-mon[36569]: pgmap v2536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:40 smithi104 ceph-mon[36624]: pgmap v2537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:40 smithi081 ceph-mon[31453]: pgmap v2537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:40 smithi153 ceph-mon[36569]: pgmap v2537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:40.897+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:22:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:40.897+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:22:40.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:40.897+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:22:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:42 smithi081 ceph-mon[31453]: pgmap v2538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:42 smithi153 ceph-mon[36569]: pgmap v2538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:42 smithi104 ceph-mon[36624]: pgmap v2538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:44 smithi081 ceph-mon[31453]: pgmap v2539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:44 smithi153 ceph-mon[36569]: pgmap v2539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:44 smithi104 ceph-mon[36624]: pgmap v2539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:46 smithi081 ceph-mon[31453]: pgmap v2540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:46 smithi153 ceph-mon[36569]: pgmap v2540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:46 smithi104 ceph-mon[36624]: pgmap v2540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:48 smithi081 ceph-mon[31453]: pgmap v2541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:22:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:48 smithi153 ceph-mon[36569]: pgmap v2541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:22:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:48 smithi104 ceph-mon[36624]: pgmap v2541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:22:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:22:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:22:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:22:49.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:22:49.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:22:49.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:22:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:22:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:22:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:22:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:50 smithi081 ceph-mon[31453]: pgmap v2542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:50 smithi081 ceph-mon[31453]: pgmap v2543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:50.457 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:50 smithi153 ceph-mon[36569]: pgmap v2542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:50.457 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:50 smithi153 ceph-mon[36569]: pgmap v2543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:50 smithi104 ceph-mon[36624]: pgmap v2542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:50 smithi104 ceph-mon[36624]: pgmap v2543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:52 smithi081 ceph-mon[31453]: pgmap v2544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:52 smithi153 ceph-mon[36569]: pgmap v2544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:52 smithi104 ceph-mon[36624]: pgmap v2544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:54 smithi081 ceph-mon[31453]: pgmap v2545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:54 smithi153 ceph-mon[36569]: pgmap v2545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:54 smithi104 ceph-mon[36624]: pgmap v2545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:55.904+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:55.904+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:22:55.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:22:55.904+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:22:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:56 smithi081 ceph-mon[31453]: pgmap v2546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:56 smithi153 ceph-mon[36569]: pgmap v2546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:56 smithi104 ceph-mon[36624]: pgmap v2546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:22:58 smithi081 ceph-mon[31453]: pgmap v2547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:22:58 smithi153 ceph-mon[36569]: pgmap v2547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:22:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:22:58 smithi104 ceph-mon[36624]: pgmap v2547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:00 smithi081 ceph-mon[31453]: pgmap v2548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:00 smithi153 ceph-mon[36569]: pgmap v2548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:00 smithi104 ceph-mon[36624]: pgmap v2548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:02 smithi153 ceph-mon[36569]: pgmap v2549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:02 smithi104 ceph-mon[36624]: pgmap v2549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:02 smithi081 ceph-mon[31453]: pgmap v2549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:04 smithi153 ceph-mon[36569]: pgmap v2550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:04 smithi104 ceph-mon[36624]: pgmap v2550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:04 smithi081 ceph-mon[31453]: pgmap v2550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:06.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:06 smithi081 ceph-mon[31453]: pgmap v2551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:06 smithi153 ceph-mon[36569]: pgmap v2551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:06 smithi104 ceph-mon[36624]: pgmap v2551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:08 smithi153 ceph-mon[36569]: pgmap v2552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:08 smithi104 ceph-mon[36624]: pgmap v2552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:08 smithi081 ceph-mon[31453]: pgmap v2552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:10 smithi153 ceph-mon[36569]: pgmap v2553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:10 smithi104 ceph-mon[36624]: pgmap v2553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:10 smithi081 ceph-mon[31453]: pgmap v2553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:10.920 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:10.920+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:23:10.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:10.920+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:23:10.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:10.920+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:23:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:12 smithi153 ceph-mon[36569]: pgmap v2554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:12 smithi104 ceph-mon[36624]: pgmap v2554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:12 smithi081 ceph-mon[31453]: pgmap v2554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:14 smithi153 ceph-mon[36569]: pgmap v2555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:14 smithi104 ceph-mon[36624]: pgmap v2555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:14 smithi081 ceph-mon[31453]: pgmap v2555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:16 smithi153 ceph-mon[36569]: pgmap v2556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:16 smithi104 ceph-mon[36624]: pgmap v2556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:16 smithi081 ceph-mon[31453]: pgmap v2556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:18 smithi081 ceph-mon[31453]: pgmap v2557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:18 smithi153 ceph-mon[36569]: pgmap v2557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:18 smithi104 ceph-mon[36624]: pgmap v2557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:20 smithi153 ceph-mon[36569]: pgmap v2558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:20 smithi104 ceph-mon[36624]: pgmap v2558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:20 smithi081 ceph-mon[31453]: pgmap v2558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:22 smithi153 ceph-mon[36569]: pgmap v2559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:22 smithi104 ceph-mon[36624]: pgmap v2559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:22 smithi081 ceph-mon[31453]: pgmap v2559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:24 smithi153 ceph-mon[36569]: pgmap v2560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:24 smithi104 ceph-mon[36624]: pgmap v2560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:24 smithi081 ceph-mon[31453]: pgmap v2560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:25.935+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:23:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:25.935+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:23:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:25.935+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:23:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:26 smithi153 ceph-mon[36569]: pgmap v2561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:26 smithi104 ceph-mon[36624]: pgmap v2561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:26 smithi081 ceph-mon[31453]: pgmap v2561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:28 smithi153 ceph-mon[36569]: pgmap v2562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:28 smithi104 ceph-mon[36624]: pgmap v2562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:28 smithi081 ceph-mon[31453]: pgmap v2562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:30 smithi153 ceph-mon[36569]: pgmap v2563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:30 smithi104 ceph-mon[36624]: pgmap v2563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:30 smithi081 ceph-mon[31453]: pgmap v2563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:32 smithi153 ceph-mon[36569]: pgmap v2564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:32 smithi104 ceph-mon[36624]: pgmap v2564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:32 smithi081 ceph-mon[31453]: pgmap v2564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:34 smithi081 ceph-mon[31453]: pgmap v2565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:34 smithi153 ceph-mon[36569]: pgmap v2565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:34 smithi104 ceph-mon[36624]: pgmap v2565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:36 smithi153 ceph-mon[36569]: pgmap v2566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:36 smithi104 ceph-mon[36624]: pgmap v2566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:36 smithi081 ceph-mon[31453]: pgmap v2566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:38 smithi153 ceph-mon[36569]: pgmap v2567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:38 smithi104 ceph-mon[36624]: pgmap v2567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:38 smithi081 ceph-mon[31453]: pgmap v2567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:40 smithi153 ceph-mon[36569]: pgmap v2568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:40 smithi104 ceph-mon[36624]: pgmap v2568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:40 smithi081 ceph-mon[31453]: pgmap v2568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:40.940+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:23:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:40.941+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:23:40.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:40.941+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:23:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:42 smithi153 ceph-mon[36569]: pgmap v2569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:42 smithi104 ceph-mon[36624]: pgmap v2569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:42 smithi081 ceph-mon[31453]: pgmap v2569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:44 smithi153 ceph-mon[36569]: pgmap v2570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:44 smithi104 ceph-mon[36624]: pgmap v2570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:44 smithi081 ceph-mon[31453]: pgmap v2570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:46 smithi153 ceph-mon[36569]: pgmap v2571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:46 smithi104 ceph-mon[36624]: pgmap v2571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:46 smithi081 ceph-mon[31453]: pgmap v2571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:48 smithi153 ceph-mon[36569]: pgmap v2572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:48 smithi104 ceph-mon[36624]: pgmap v2572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:48 smithi081 ceph-mon[31453]: pgmap v2572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:23:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:23:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:23:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:50 smithi153 ceph-mon[36569]: pgmap v2573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:23:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:23:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:23:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:50 smithi104 ceph-mon[36624]: pgmap v2573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:23:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:23:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:23:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:50 smithi081 ceph-mon[31453]: pgmap v2573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:23:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:23:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:23:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:51 smithi153 ceph-mon[36569]: pgmap v2574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:51 smithi153 ceph-mon[36569]: pgmap v2575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:51 smithi104 ceph-mon[36624]: pgmap v2574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:51 smithi104 ceph-mon[36624]: pgmap v2575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:51 smithi081 ceph-mon[31453]: pgmap v2574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:51 smithi081 ceph-mon[31453]: pgmap v2575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:53 smithi153 ceph-mon[36569]: pgmap v2576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:53 smithi104 ceph-mon[36624]: pgmap v2576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:53 smithi081 ceph-mon[31453]: pgmap v2576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:55 smithi104 ceph-mon[36624]: pgmap v2577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:55 smithi081 ceph-mon[31453]: pgmap v2577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:55 smithi153 ceph-mon[36569]: pgmap v2577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:55.953+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:23:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:55.953+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:23:55.954 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:23:55.953+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:23:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:57 smithi104 ceph-mon[36624]: pgmap v2578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:57 smithi081 ceph-mon[31453]: pgmap v2578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:57 smithi153 ceph-mon[36569]: pgmap v2578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:23:59 smithi104 ceph-mon[36624]: pgmap v2579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:23:59 smithi081 ceph-mon[31453]: pgmap v2579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:23:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:23:59 smithi153 ceph-mon[36569]: pgmap v2579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:01 smithi104 ceph-mon[36624]: pgmap v2580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:01 smithi081 ceph-mon[31453]: pgmap v2580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:01 smithi153 ceph-mon[36569]: pgmap v2580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:03 smithi104 ceph-mon[36624]: pgmap v2581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:03 smithi081 ceph-mon[31453]: pgmap v2581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:03 smithi153 ceph-mon[36569]: pgmap v2581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:05 smithi081 ceph-mon[31453]: pgmap v2582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:05.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:05 smithi153 ceph-mon[36569]: pgmap v2582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:05 smithi104 ceph-mon[36624]: pgmap v2582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:07 smithi081 ceph-mon[31453]: pgmap v2583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:07 smithi153 ceph-mon[36569]: pgmap v2583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:07 smithi104 ceph-mon[36624]: pgmap v2583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:09 smithi081 ceph-mon[31453]: pgmap v2584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:09 smithi153 ceph-mon[36569]: pgmap v2584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:09 smithi104 ceph-mon[36624]: pgmap v2584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:10.959+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:24:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:10.959+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:24:10.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:10.959+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:24:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:11 smithi081 ceph-mon[31453]: pgmap v2585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:11 smithi153 ceph-mon[36569]: pgmap v2585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:11 smithi104 ceph-mon[36624]: pgmap v2585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:13 smithi081 ceph-mon[31453]: pgmap v2586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:13 smithi153 ceph-mon[36569]: pgmap v2586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:13 smithi104 ceph-mon[36624]: pgmap v2586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:15 smithi081 ceph-mon[31453]: pgmap v2587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:15 smithi153 ceph-mon[36569]: pgmap v2587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:15 smithi104 ceph-mon[36624]: pgmap v2587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:17 smithi081 ceph-mon[31453]: pgmap v2588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:17 smithi153 ceph-mon[36569]: pgmap v2588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:17 smithi104 ceph-mon[36624]: pgmap v2588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:19 smithi081 ceph-mon[31453]: pgmap v2589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:19 smithi153 ceph-mon[36569]: pgmap v2589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:19 smithi104 ceph-mon[36624]: pgmap v2589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:21 smithi081 ceph-mon[31453]: pgmap v2590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:21 smithi153 ceph-mon[36569]: pgmap v2590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:21 smithi104 ceph-mon[36624]: pgmap v2590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:23 smithi081 ceph-mon[31453]: pgmap v2591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:23 smithi153 ceph-mon[36569]: pgmap v2591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:23 smithi104 ceph-mon[36624]: pgmap v2591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:25 smithi081 ceph-mon[31453]: pgmap v2592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:25 smithi153 ceph-mon[36569]: pgmap v2592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:25 smithi104 ceph-mon[36624]: pgmap v2592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:25.968 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:25.968+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:24:25.968 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:25.968+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:24:25.968 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:25.968+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:24:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:27 smithi081 ceph-mon[31453]: pgmap v2593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:27 smithi153 ceph-mon[36569]: pgmap v2593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:27 smithi104 ceph-mon[36624]: pgmap v2593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:29 smithi081 ceph-mon[31453]: pgmap v2594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:29 smithi153 ceph-mon[36569]: pgmap v2594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:29 smithi104 ceph-mon[36624]: pgmap v2594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:30 smithi153 ceph-mon[36569]: pgmap v2595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:30 smithi104 ceph-mon[36624]: pgmap v2595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:30 smithi081 ceph-mon[31453]: pgmap v2595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:33 smithi153 ceph-mon[36569]: pgmap v2596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:33 smithi104 ceph-mon[36624]: pgmap v2596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:33 smithi081 ceph-mon[31453]: pgmap v2596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:34 smithi153 ceph-mon[36569]: pgmap v2597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:34 smithi104 ceph-mon[36624]: pgmap v2597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:34 smithi081 ceph-mon[31453]: pgmap v2597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:37 smithi153 ceph-mon[36569]: pgmap v2598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:37.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:37 smithi081 ceph-mon[31453]: pgmap v2598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:37 smithi104 ceph-mon[36624]: pgmap v2598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:39 smithi153 ceph-mon[36569]: pgmap v2599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:39 smithi104 ceph-mon[36624]: pgmap v2599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:39 smithi081 ceph-mon[31453]: pgmap v2599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:40.980+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:24:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:40.980+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:24:40.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:40.980+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:24:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:41 smithi153 ceph-mon[36569]: pgmap v2600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:41 smithi104 ceph-mon[36624]: pgmap v2600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:41 smithi081 ceph-mon[31453]: pgmap v2600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:41.994 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-04-12T22:24:41.995 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-04-12T22:24:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:43 smithi153 ceph-mon[36569]: pgmap v2601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:43 smithi104 ceph-mon[36624]: pgmap v2601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:43 smithi081 ceph-mon[31453]: pgmap v2601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:45 smithi153 ceph-mon[36569]: pgmap v2602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:45 smithi104 ceph-mon[36624]: pgmap v2602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:45 smithi081 ceph-mon[31453]: pgmap v2602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:47 smithi153 ceph-mon[36569]: pgmap v2603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:47 smithi104 ceph-mon[36624]: pgmap v2603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:47 smithi081 ceph-mon[31453]: pgmap v2603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:49 smithi153 ceph-mon[36569]: pgmap v2604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:49 smithi104 ceph-mon[36624]: pgmap v2604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:49 smithi081 ceph-mon[31453]: pgmap v2604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: pgmap v2605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.336 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: pgmap v2605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: pgmap v2605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:24:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:24:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:24:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:24:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:53 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:24:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:53 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:24:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:53 smithi104 ceph-mon[36624]: pgmap v2606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:53 smithi104 ceph-mon[36624]: pgmap v2607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:53 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:24:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:53 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:24:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:53 smithi081 ceph-mon[31453]: pgmap v2606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:53 smithi081 ceph-mon[31453]: pgmap v2607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:53 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:24:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:53 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:24:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:53 smithi153 ceph-mon[36569]: pgmap v2606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:53 smithi153 ceph-mon[36569]: pgmap v2607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:55 smithi104 ceph-mon[36624]: pgmap v2608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:55 smithi081 ceph-mon[31453]: pgmap v2608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:55 smithi153 ceph-mon[36569]: pgmap v2608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:55.981+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:24:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:55.981+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:24:55.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:24:55.981+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:24:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:57 smithi081 ceph-mon[31453]: pgmap v2609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:57 smithi153 ceph-mon[36569]: pgmap v2609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:57 smithi104 ceph-mon[36624]: pgmap v2609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:24:59 smithi081 ceph-mon[31453]: pgmap v2610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:24:59 smithi153 ceph-mon[36569]: pgmap v2610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:24:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:24:59 smithi104 ceph-mon[36624]: pgmap v2610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:01 smithi081 ceph-mon[31453]: pgmap v2611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:01 smithi153 ceph-mon[36569]: pgmap v2611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:01 smithi104 ceph-mon[36624]: pgmap v2611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:03 smithi081 ceph-mon[31453]: pgmap v2612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:03 smithi153 ceph-mon[36569]: pgmap v2612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:03 smithi104 ceph-mon[36624]: pgmap v2612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:05 smithi081 ceph-mon[31453]: pgmap v2613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:05 smithi153 ceph-mon[36569]: pgmap v2613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:05 smithi104 ceph-mon[36624]: pgmap v2613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:07 smithi081 ceph-mon[31453]: pgmap v2614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:07 smithi153 ceph-mon[36569]: pgmap v2614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:07 smithi104 ceph-mon[36624]: pgmap v2614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:09 smithi081 ceph-mon[31453]: pgmap v2615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:09 smithi153 ceph-mon[36569]: pgmap v2615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:09 smithi104 ceph-mon[36624]: pgmap v2615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:10.985 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:10.985+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:25:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:10.985+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:25:10.986 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:10.985+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:25:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:11 smithi081 ceph-mon[31453]: pgmap v2616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:11 smithi153 ceph-mon[36569]: pgmap v2616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:11 smithi104 ceph-mon[36624]: pgmap v2616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:13.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:13.377+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:25:13.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:13.377+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:25:13.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:13.378+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:25:13.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:13.378+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:25:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:13 smithi081 ceph-mon[31453]: pgmap v2617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:13 smithi153 ceph-mon[36569]: pgmap v2617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:13 smithi104 ceph-mon[36624]: pgmap v2617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:15 smithi081 ceph-mon[31453]: pgmap v2618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:15 smithi153 ceph-mon[36569]: pgmap v2618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:15 smithi104 ceph-mon[36624]: pgmap v2618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:17 smithi081 ceph-mon[31453]: pgmap v2619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:17 smithi153 ceph-mon[36569]: pgmap v2619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:17 smithi104 ceph-mon[36624]: pgmap v2619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:19 smithi081 ceph-mon[31453]: pgmap v2620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:19 smithi153 ceph-mon[36569]: pgmap v2620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:19 smithi104 ceph-mon[36624]: pgmap v2620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:21 smithi081 ceph-mon[31453]: pgmap v2621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:21 smithi153 ceph-mon[36569]: pgmap v2621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:21 smithi104 ceph-mon[36624]: pgmap v2621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:23 smithi081 ceph-mon[31453]: pgmap v2622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:23 smithi153 ceph-mon[36569]: pgmap v2622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:23 smithi104 ceph-mon[36624]: pgmap v2622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:25 smithi153 ceph-mon[36569]: pgmap v2623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:25 smithi104 ceph-mon[36624]: pgmap v2623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:25 smithi081 ceph-mon[31453]: pgmap v2623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:25.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:25.988+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:25:25.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:25.988+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:25:25.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:25.988+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:25:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:27 smithi153 ceph-mon[36569]: pgmap v2624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:27 smithi104 ceph-mon[36624]: pgmap v2624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:27 smithi081 ceph-mon[31453]: pgmap v2624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:29 smithi153 ceph-mon[36569]: pgmap v2625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:29 smithi104 ceph-mon[36624]: pgmap v2625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:29 smithi081 ceph-mon[31453]: pgmap v2625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:31 smithi153 ceph-mon[36569]: pgmap v2626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:31 smithi104 ceph-mon[36624]: pgmap v2626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:31 smithi081 ceph-mon[31453]: pgmap v2626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:33 smithi153 ceph-mon[36569]: pgmap v2627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:33 smithi104 ceph-mon[36624]: pgmap v2627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:33 smithi081 ceph-mon[31453]: pgmap v2627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:35 smithi153 ceph-mon[36569]: pgmap v2628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:35 smithi104 ceph-mon[36624]: pgmap v2628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:35 smithi081 ceph-mon[31453]: pgmap v2628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:37.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:37 smithi081 ceph-mon[31453]: pgmap v2629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:37 smithi153 ceph-mon[36569]: pgmap v2629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:37 smithi104 ceph-mon[36624]: pgmap v2629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:39 smithi153 ceph-mon[36569]: pgmap v2630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:39 smithi104 ceph-mon[36624]: pgmap v2630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:39 smithi081 ceph-mon[31453]: pgmap v2630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:40.997+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:25:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:40.997+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:25:40.997 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:40.997+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:25:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:41 smithi153 ceph-mon[36569]: pgmap v2631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:41 smithi104 ceph-mon[36624]: pgmap v2631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:41 smithi081 ceph-mon[31453]: pgmap v2631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:43 smithi153 ceph-mon[36569]: pgmap v2632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:43 smithi104 ceph-mon[36624]: pgmap v2632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:43 smithi081 ceph-mon[31453]: pgmap v2632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:45 smithi153 ceph-mon[36569]: pgmap v2633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:45 smithi104 ceph-mon[36624]: pgmap v2633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:45 smithi081 ceph-mon[31453]: pgmap v2633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:47 smithi153 ceph-mon[36569]: pgmap v2634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:47 smithi104 ceph-mon[36624]: pgmap v2634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:47 smithi081 ceph-mon[31453]: pgmap v2634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:49 smithi153 ceph-mon[36569]: pgmap v2635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:49 smithi104 ceph-mon[36624]: pgmap v2635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:49 smithi081 ceph-mon[31453]: pgmap v2635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:51 smithi153 ceph-mon[36569]: pgmap v2636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:51 smithi104 ceph-mon[36624]: pgmap v2636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:51 smithi081 ceph-mon[31453]: pgmap v2636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:25:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:25:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:25:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:53 smithi153 ceph-mon[36569]: pgmap v2637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:25:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:25:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:25:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:53 smithi104 ceph-mon[36624]: pgmap v2637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:25:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:25:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:25:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:53 smithi081 ceph-mon[31453]: pgmap v2637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:25:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:25:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:25:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:54 smithi153 ceph-mon[36569]: pgmap v2638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:54 smithi153 ceph-mon[36569]: pgmap v2639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:54 smithi104 ceph-mon[36624]: pgmap v2638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:54 smithi104 ceph-mon[36624]: pgmap v2639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:54 smithi081 ceph-mon[31453]: pgmap v2638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:54 smithi081 ceph-mon[31453]: pgmap v2639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:55.998+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:25:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:55.998+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:25:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:25:55.998+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:25:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:56 smithi153 ceph-mon[36569]: pgmap v2640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:56 smithi104 ceph-mon[36624]: pgmap v2640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:56 smithi081 ceph-mon[31453]: pgmap v2640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:25:58 smithi153 ceph-mon[36569]: pgmap v2641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:25:58 smithi104 ceph-mon[36624]: pgmap v2641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:25:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:25:58 smithi081 ceph-mon[31453]: pgmap v2641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:00 smithi153 ceph-mon[36569]: pgmap v2642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:00 smithi104 ceph-mon[36624]: pgmap v2642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:00 smithi081 ceph-mon[31453]: pgmap v2642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:02 smithi153 ceph-mon[36569]: pgmap v2643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:02 smithi104 ceph-mon[36624]: pgmap v2643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:02 smithi081 ceph-mon[31453]: pgmap v2643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:04 smithi153 ceph-mon[36569]: pgmap v2644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:04 smithi104 ceph-mon[36624]: pgmap v2644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:04 smithi081 ceph-mon[31453]: pgmap v2644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:06 smithi153 ceph-mon[36569]: pgmap v2645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:06 smithi104 ceph-mon[36624]: pgmap v2645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:06 smithi081 ceph-mon[31453]: pgmap v2645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:08 smithi153 ceph-mon[36569]: pgmap v2646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:08 smithi104 ceph-mon[36624]: pgmap v2646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:08 smithi081 ceph-mon[31453]: pgmap v2646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:10 smithi153 ceph-mon[36569]: pgmap v2647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:10 smithi104 ceph-mon[36624]: pgmap v2647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:10 smithi081 ceph-mon[31453]: pgmap v2647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:11.005+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:26:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:11.005+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:26:11.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:11.005+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:26:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:12 smithi153 ceph-mon[36569]: pgmap v2648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:12 smithi104 ceph-mon[36624]: pgmap v2648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:12 smithi081 ceph-mon[31453]: pgmap v2648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:14 smithi153 ceph-mon[36569]: pgmap v2649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:14 smithi104 ceph-mon[36624]: pgmap v2649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:14 smithi081 ceph-mon[31453]: pgmap v2649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:16 smithi104 ceph-mon[36624]: pgmap v2650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:16 smithi081 ceph-mon[31453]: pgmap v2650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:16 smithi153 ceph-mon[36569]: pgmap v2650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:18 smithi104 ceph-mon[36624]: pgmap v2651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:18 smithi081 ceph-mon[31453]: pgmap v2651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:18 smithi153 ceph-mon[36569]: pgmap v2651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:20 smithi104 ceph-mon[36624]: pgmap v2652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:20 smithi081 ceph-mon[31453]: pgmap v2652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:20 smithi153 ceph-mon[36569]: pgmap v2652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:22 smithi104 ceph-mon[36624]: pgmap v2653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:22.608 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:22 smithi153 ceph-mon[36569]: pgmap v2653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:22 smithi081 ceph-mon[31453]: pgmap v2653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:24 smithi104 ceph-mon[36624]: pgmap v2654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:24 smithi081 ceph-mon[31453]: pgmap v2654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:24 smithi153 ceph-mon[36569]: pgmap v2654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:26.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:26.011+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:26:26.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:26.011+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:26:26.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:26.011+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:26:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:26 smithi081 ceph-mon[31453]: pgmap v2655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:26 smithi153 ceph-mon[36569]: pgmap v2655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:26 smithi104 ceph-mon[36624]: pgmap v2655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:28.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:28 smithi104 ceph-mon[36624]: pgmap v2656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:28 smithi081 ceph-mon[31453]: pgmap v2656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:28 smithi153 ceph-mon[36569]: pgmap v2656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:30 smithi081 ceph-mon[31453]: pgmap v2657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:30 smithi153 ceph-mon[36569]: pgmap v2657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:30 smithi104 ceph-mon[36624]: pgmap v2657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:32 smithi081 ceph-mon[31453]: pgmap v2658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:32 smithi153 ceph-mon[36569]: pgmap v2658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:32 smithi104 ceph-mon[36624]: pgmap v2658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:34 smithi081 ceph-mon[31453]: pgmap v2659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:34 smithi153 ceph-mon[36569]: pgmap v2659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:34 smithi104 ceph-mon[36624]: pgmap v2659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:36 smithi081 ceph-mon[31453]: pgmap v2660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:36 smithi153 ceph-mon[36569]: pgmap v2660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:36 smithi104 ceph-mon[36624]: pgmap v2660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:38 smithi081 ceph-mon[31453]: pgmap v2661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:38 smithi153 ceph-mon[36569]: pgmap v2661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:38 smithi104 ceph-mon[36624]: pgmap v2661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:40 smithi081 ceph-mon[31453]: pgmap v2662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:40 smithi153 ceph-mon[36569]: pgmap v2662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:40 smithi104 ceph-mon[36624]: pgmap v2662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:41.014+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:26:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:41.014+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:26:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:41.014+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:26:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:42 smithi081 ceph-mon[31453]: pgmap v2663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:42 smithi153 ceph-mon[36569]: pgmap v2663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:42 smithi104 ceph-mon[36624]: pgmap v2663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:44 smithi081 ceph-mon[31453]: pgmap v2664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:44 smithi153 ceph-mon[36569]: pgmap v2664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:44 smithi104 ceph-mon[36624]: pgmap v2664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:46 smithi081 ceph-mon[31453]: pgmap v2665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:46 smithi153 ceph-mon[36569]: pgmap v2665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:46 smithi104 ceph-mon[36624]: pgmap v2665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:48 smithi081 ceph-mon[31453]: pgmap v2666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:48 smithi153 ceph-mon[36569]: pgmap v2666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:48 smithi104 ceph-mon[36624]: pgmap v2666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:50 smithi081 ceph-mon[31453]: pgmap v2667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:50 smithi153 ceph-mon[36569]: pgmap v2667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:50 smithi104 ceph-mon[36624]: pgmap v2667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:52 smithi153 ceph-mon[36569]: pgmap v2668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:52 smithi104 ceph-mon[36624]: pgmap v2668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:52 smithi081 ceph-mon[31453]: pgmap v2668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:26:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:26:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:26:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:54 smithi153 ceph-mon[36569]: pgmap v2669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:54 smithi104 ceph-mon[36624]: pgmap v2669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:54 smithi081 ceph-mon[31453]: pgmap v2669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:26:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:26:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:26:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:26:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:26:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:26:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:26:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:26:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:26:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:56.016+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:26:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:56.016+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:26:56.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:26:56.016+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:26:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:56 smithi153 ceph-mon[36569]: pgmap v2670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:56 smithi153 ceph-mon[36569]: pgmap v2671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:56 smithi104 ceph-mon[36624]: pgmap v2670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:56 smithi104 ceph-mon[36624]: pgmap v2671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:56 smithi081 ceph-mon[31453]: pgmap v2670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:56 smithi081 ceph-mon[31453]: pgmap v2671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:57 smithi153 ceph-mon[36569]: pgmap v2672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:57 smithi104 ceph-mon[36624]: pgmap v2672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:26:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:57 smithi081 ceph-mon[31453]: pgmap v2672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:26:59 smithi153 ceph-mon[36569]: pgmap v2673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:26:59 smithi104 ceph-mon[36624]: pgmap v2673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:26:59 smithi081 ceph-mon[31453]: pgmap v2673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:01 smithi153 ceph-mon[36569]: pgmap v2674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:01 smithi104 ceph-mon[36624]: pgmap v2674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:01 smithi081 ceph-mon[31453]: pgmap v2674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:03 smithi153 ceph-mon[36569]: pgmap v2675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:03 smithi104 ceph-mon[36624]: pgmap v2675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:03 smithi081 ceph-mon[31453]: pgmap v2675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:05 smithi153 ceph-mon[36569]: pgmap v2676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:05 smithi104 ceph-mon[36624]: pgmap v2676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:05 smithi081 ceph-mon[31453]: pgmap v2676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:07 smithi153 ceph-mon[36569]: pgmap v2677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:07 smithi104 ceph-mon[36624]: pgmap v2677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:07 smithi081 ceph-mon[31453]: pgmap v2677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:09 smithi153 ceph-mon[36569]: pgmap v2678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:09 smithi104 ceph-mon[36624]: pgmap v2678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:09 smithi081 ceph-mon[31453]: pgmap v2678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:11.017+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:27:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:11.018+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:27:11.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:11.018+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:27:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:11 smithi153 ceph-mon[36569]: pgmap v2679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:11 smithi104 ceph-mon[36624]: pgmap v2679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:11 smithi081 ceph-mon[31453]: pgmap v2679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:13 smithi153 ceph-mon[36569]: pgmap v2680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:13 smithi104 ceph-mon[36624]: pgmap v2680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:13 smithi081 ceph-mon[31453]: pgmap v2680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:15 smithi153 ceph-mon[36569]: pgmap v2681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:15 smithi104 ceph-mon[36624]: pgmap v2681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:15 smithi081 ceph-mon[31453]: pgmap v2681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:17 smithi153 ceph-mon[36569]: pgmap v2682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:17 smithi104 ceph-mon[36624]: pgmap v2682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:17 smithi081 ceph-mon[31453]: pgmap v2682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:19 smithi153 ceph-mon[36569]: pgmap v2683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:19 smithi104 ceph-mon[36624]: pgmap v2683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:19 smithi081 ceph-mon[31453]: pgmap v2683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:21 smithi153 ceph-mon[36569]: pgmap v2684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:21 smithi104 ceph-mon[36624]: pgmap v2684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:21 smithi081 ceph-mon[31453]: pgmap v2684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:23 smithi153 ceph-mon[36569]: pgmap v2685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:23 smithi104 ceph-mon[36624]: pgmap v2685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:23 smithi081 ceph-mon[31453]: pgmap v2685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:26.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:26.023+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:27:26.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:26.023+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:27:26.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:26.023+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:27:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:25 smithi153 ceph-mon[36569]: pgmap v2686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:25 smithi104 ceph-mon[36624]: pgmap v2686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:25 smithi081 ceph-mon[31453]: pgmap v2686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:27 smithi153 ceph-mon[36569]: pgmap v2687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:27 smithi104 ceph-mon[36624]: pgmap v2687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:27 smithi081 ceph-mon[31453]: pgmap v2687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:29 smithi153 ceph-mon[36569]: pgmap v2688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:29 smithi104 ceph-mon[36624]: pgmap v2688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:29 smithi081 ceph-mon[31453]: pgmap v2688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:31 smithi153 ceph-mon[36569]: pgmap v2689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:31 smithi104 ceph-mon[36624]: pgmap v2689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:31 smithi081 ceph-mon[31453]: pgmap v2689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:33 smithi153 ceph-mon[36569]: pgmap v2690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:33 smithi104 ceph-mon[36624]: pgmap v2690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:33 smithi081 ceph-mon[31453]: pgmap v2690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:35 smithi153 ceph-mon[36569]: pgmap v2691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:35 smithi104 ceph-mon[36624]: pgmap v2691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:35 smithi081 ceph-mon[31453]: pgmap v2691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:37 smithi153 ceph-mon[36569]: pgmap v2692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:37 smithi104 ceph-mon[36624]: pgmap v2692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:37 smithi081 ceph-mon[31453]: pgmap v2692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:39 smithi153 ceph-mon[36569]: pgmap v2693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:39 smithi104 ceph-mon[36624]: pgmap v2693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:40.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:39 smithi081 ceph-mon[31453]: pgmap v2693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:41.038+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:27:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:41.038+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:27:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:41.038+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:27:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:41 smithi104 ceph-mon[36624]: pgmap v2694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:41 smithi081 ceph-mon[31453]: pgmap v2694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:41 smithi153 ceph-mon[36569]: pgmap v2694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:43 smithi104 ceph-mon[36624]: pgmap v2695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:43 smithi081 ceph-mon[31453]: pgmap v2695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:43 smithi153 ceph-mon[36569]: pgmap v2695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:45 smithi104 ceph-mon[36624]: pgmap v2696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:45 smithi081 ceph-mon[31453]: pgmap v2696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:45 smithi153 ceph-mon[36569]: pgmap v2696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:47 smithi104 ceph-mon[36624]: pgmap v2697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:47 smithi081 ceph-mon[31453]: pgmap v2697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:47 smithi153 ceph-mon[36569]: pgmap v2697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:49 smithi104 ceph-mon[36624]: pgmap v2698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:49 smithi081 ceph-mon[31453]: pgmap v2698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:49 smithi153 ceph-mon[36569]: pgmap v2698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:51 smithi081 ceph-mon[31453]: pgmap v2699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:51 smithi153 ceph-mon[36569]: pgmap v2699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:51 smithi104 ceph-mon[36624]: pgmap v2699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:54.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:53 smithi153 ceph-mon[36569]: pgmap v2700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:53 smithi081 ceph-mon[31453]: pgmap v2700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:53 smithi104 ceph-mon[36624]: pgmap v2700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:56.052+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:27:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:56.052+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:27:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:27:56.052+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:27:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:55 smithi081 ceph-mon[31453]: pgmap v2701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:27:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:55 smithi153 ceph-mon[36569]: pgmap v2701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:27:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:55 smithi104 ceph-mon[36624]: pgmap v2701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:27:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:27:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:27:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:56 smithi081 ceph-mon[31453]: pgmap v2702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:56 smithi081 ceph-mon[31453]: pgmap v2703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:27:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:27:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:27:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:56 smithi153 ceph-mon[36569]: pgmap v2702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:56 smithi153 ceph-mon[36569]: pgmap v2703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:27:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:27:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:27:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:56 smithi104 ceph-mon[36624]: pgmap v2702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:56 smithi104 ceph-mon[36624]: pgmap v2703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:27:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:27:59 smithi153 ceph-mon[36569]: pgmap v2704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:27:59 smithi104 ceph-mon[36624]: pgmap v2704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:27:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:27:59 smithi081 ceph-mon[31453]: pgmap v2704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:01 smithi153 ceph-mon[36569]: pgmap v2705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:01 smithi104 ceph-mon[36624]: pgmap v2705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:01 smithi081 ceph-mon[31453]: pgmap v2705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:03 smithi153 ceph-mon[36569]: pgmap v2706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:03 smithi104 ceph-mon[36624]: pgmap v2706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:03 smithi081 ceph-mon[31453]: pgmap v2706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:05 smithi153 ceph-mon[36569]: pgmap v2707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:05 smithi104 ceph-mon[36624]: pgmap v2707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:05 smithi081 ceph-mon[31453]: pgmap v2707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:07 smithi153 ceph-mon[36569]: pgmap v2708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:07 smithi104 ceph-mon[36624]: pgmap v2708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:07 smithi081 ceph-mon[31453]: pgmap v2708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:09 smithi153 ceph-mon[36569]: pgmap v2709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:09 smithi104 ceph-mon[36624]: pgmap v2709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:09 smithi081 ceph-mon[31453]: pgmap v2709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:11.061+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:28:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:11.061+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:28:11.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:11.061+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:28:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:11 smithi104 ceph-mon[36624]: pgmap v2710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:11 smithi081 ceph-mon[31453]: pgmap v2710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:11 smithi153 ceph-mon[36569]: pgmap v2710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:13.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:13.406+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:28:13.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:13.406+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:28:13.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:13.407+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:28:13.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:13.407+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:28:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:13 smithi104 ceph-mon[36624]: pgmap v2711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:13 smithi081 ceph-mon[31453]: pgmap v2711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:13 smithi153 ceph-mon[36569]: pgmap v2711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:15 smithi104 ceph-mon[36624]: pgmap v2712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:15 smithi081 ceph-mon[31453]: pgmap v2712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:15 smithi153 ceph-mon[36569]: pgmap v2712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:17 smithi104 ceph-mon[36624]: pgmap v2713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:17 smithi081 ceph-mon[31453]: pgmap v2713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:17 smithi153 ceph-mon[36569]: pgmap v2713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:19 smithi081 ceph-mon[31453]: pgmap v2714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:19 smithi153 ceph-mon[36569]: pgmap v2714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:19 smithi104 ceph-mon[36624]: pgmap v2714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:21 smithi081 ceph-mon[31453]: pgmap v2715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:21 smithi153 ceph-mon[36569]: pgmap v2715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:21 smithi104 ceph-mon[36624]: pgmap v2715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:23 smithi081 ceph-mon[31453]: pgmap v2716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:23 smithi153 ceph-mon[36569]: pgmap v2716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:23 smithi104 ceph-mon[36624]: pgmap v2716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:25.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:25 smithi153 ceph-mon[36569]: pgmap v2717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:25 smithi081 ceph-mon[31453]: pgmap v2717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:25 smithi104 ceph-mon[36624]: pgmap v2717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:26.077+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:28:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:26.077+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:28:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:26.077+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:28:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:27 smithi081 ceph-mon[31453]: pgmap v2718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:27 smithi153 ceph-mon[36569]: pgmap v2718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:27 smithi104 ceph-mon[36624]: pgmap v2718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:29 smithi081 ceph-mon[31453]: pgmap v2719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:29 smithi153 ceph-mon[36569]: pgmap v2719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:29 smithi104 ceph-mon[36624]: pgmap v2719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:31.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:31 smithi104 ceph-mon[36624]: pgmap v2720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:31 smithi081 ceph-mon[31453]: pgmap v2720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:31 smithi153 ceph-mon[36569]: pgmap v2720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:33 smithi081 ceph-mon[31453]: pgmap v2721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:33 smithi153 ceph-mon[36569]: pgmap v2721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:33 smithi104 ceph-mon[36624]: pgmap v2721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:35 smithi081 ceph-mon[31453]: pgmap v2722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:35 smithi153 ceph-mon[36569]: pgmap v2722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:35 smithi104 ceph-mon[36624]: pgmap v2722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:37 smithi081 ceph-mon[31453]: pgmap v2723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:37 smithi153 ceph-mon[36569]: pgmap v2723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:37 smithi104 ceph-mon[36624]: pgmap v2723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:39 smithi081 ceph-mon[31453]: pgmap v2724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:39 smithi153 ceph-mon[36569]: pgmap v2724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:39 smithi104 ceph-mon[36624]: pgmap v2724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:41.081+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:41.081+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:28:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:41.081+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:28:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:41 smithi081 ceph-mon[31453]: pgmap v2725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:41 smithi153 ceph-mon[36569]: pgmap v2725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:41 smithi104 ceph-mon[36624]: pgmap v2725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:43 smithi081 ceph-mon[31453]: pgmap v2726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:43 smithi153 ceph-mon[36569]: pgmap v2726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:43 smithi104 ceph-mon[36624]: pgmap v2726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:45 smithi081 ceph-mon[31453]: pgmap v2727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:45 smithi153 ceph-mon[36569]: pgmap v2727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:45 smithi104 ceph-mon[36624]: pgmap v2727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:47 smithi081 ceph-mon[31453]: pgmap v2728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:47 smithi153 ceph-mon[36569]: pgmap v2728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:47 smithi104 ceph-mon[36624]: pgmap v2728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:49 smithi153 ceph-mon[36569]: pgmap v2729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:49 smithi104 ceph-mon[36624]: pgmap v2729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:49 smithi081 ceph-mon[31453]: pgmap v2729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:51 smithi153 ceph-mon[36569]: pgmap v2730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:51 smithi104 ceph-mon[36624]: pgmap v2730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:51 smithi081 ceph-mon[31453]: pgmap v2730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:53 smithi153 ceph-mon[36569]: pgmap v2731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:53 smithi104 ceph-mon[36624]: pgmap v2731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:53 smithi081 ceph-mon[31453]: pgmap v2731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:55 smithi153 ceph-mon[36569]: pgmap v2732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:55 smithi104 ceph-mon[36624]: pgmap v2732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:55 smithi081 ceph-mon[31453]: pgmap v2732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:56.085+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:28:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:56.085+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:28:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:28:56.085+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:28:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:57 smithi153 ceph-mon[36569]: pgmap v2733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:28:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:57 smithi104 ceph-mon[36624]: pgmap v2733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:28:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:57 smithi081 ceph-mon[31453]: pgmap v2733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:28:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:59 smithi153 ceph-mon[36569]: pgmap v2734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:28:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:59 smithi104 ceph-mon[36624]: pgmap v2734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:28:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:59 smithi081 ceph-mon[31453]: pgmap v2734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:28:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:28:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:28:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:00 smithi104 ceph-mon[36624]: pgmap v2735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:00 smithi081 ceph-mon[31453]: pgmap v2735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:00 smithi153 ceph-mon[36569]: pgmap v2735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:03 smithi153 ceph-mon[36569]: pgmap v2736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:29:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:29:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:03 smithi104 ceph-mon[36624]: pgmap v2736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:29:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:29:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:03 smithi081 ceph-mon[31453]: pgmap v2736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:29:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:29:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:29:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:04 smithi153 ceph-mon[36569]: pgmap v2737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:04 smithi153 ceph-mon[36569]: pgmap v2738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:04 smithi104 ceph-mon[36624]: pgmap v2737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:04 smithi104 ceph-mon[36624]: pgmap v2738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:04 smithi081 ceph-mon[31453]: pgmap v2737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:04.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:04 smithi081 ceph-mon[31453]: pgmap v2738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:06 smithi153 ceph-mon[36569]: pgmap v2739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:06 smithi104 ceph-mon[36624]: pgmap v2739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:06 smithi081 ceph-mon[31453]: pgmap v2739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:07 smithi153 ceph-mon[36569]: pgmap v2740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:07 smithi104 ceph-mon[36624]: pgmap v2740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:07 smithi081 ceph-mon[31453]: pgmap v2740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:09 smithi153 ceph-mon[36569]: pgmap v2741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:09 smithi104 ceph-mon[36624]: pgmap v2741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:09 smithi081 ceph-mon[31453]: pgmap v2741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:11.089+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:29:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:11.089+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:29:11.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:11.089+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:29:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:11 smithi153 ceph-mon[36569]: pgmap v2742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:11 smithi104 ceph-mon[36624]: pgmap v2742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:11 smithi081 ceph-mon[31453]: pgmap v2742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:13 smithi153 ceph-mon[36569]: pgmap v2743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:13 smithi104 ceph-mon[36624]: pgmap v2743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:13 smithi081 ceph-mon[31453]: pgmap v2743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:15 smithi104 ceph-mon[36624]: pgmap v2744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:15 smithi081 ceph-mon[31453]: pgmap v2744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:15 smithi153 ceph-mon[36569]: pgmap v2744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:17 smithi104 ceph-mon[36624]: pgmap v2745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:17 smithi081 ceph-mon[31453]: pgmap v2745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:17 smithi153 ceph-mon[36569]: pgmap v2745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:19 smithi104 ceph-mon[36624]: pgmap v2746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:19 smithi081 ceph-mon[31453]: pgmap v2746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:19 smithi153 ceph-mon[36569]: pgmap v2746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:21 smithi104 ceph-mon[36624]: pgmap v2747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:21 smithi081 ceph-mon[31453]: pgmap v2747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:21 smithi153 ceph-mon[36569]: pgmap v2747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:23 smithi081 ceph-mon[31453]: pgmap v2748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:23 smithi153 ceph-mon[36569]: pgmap v2748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:23 smithi104 ceph-mon[36624]: pgmap v2748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:26.104+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:29:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:26.104+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:29:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:26.104+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:29:26.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:25 smithi153 ceph-mon[36569]: pgmap v2749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:25 smithi081 ceph-mon[31453]: pgmap v2749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:25 smithi104 ceph-mon[36624]: pgmap v2749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:27 smithi081 ceph-mon[31453]: pgmap v2750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:27 smithi153 ceph-mon[36569]: pgmap v2750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:27 smithi104 ceph-mon[36624]: pgmap v2750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:29 smithi081 ceph-mon[31453]: pgmap v2751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:29 smithi153 ceph-mon[36569]: pgmap v2751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:29 smithi104 ceph-mon[36624]: pgmap v2751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:32.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:31 smithi104 ceph-mon[36624]: pgmap v2752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:31 smithi081 ceph-mon[31453]: pgmap v2752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:31 smithi153 ceph-mon[36569]: pgmap v2752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:33 smithi081 ceph-mon[31453]: pgmap v2753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:33 smithi153 ceph-mon[36569]: pgmap v2753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:33 smithi104 ceph-mon[36624]: pgmap v2753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:35 smithi081 ceph-mon[31453]: pgmap v2754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:35 smithi153 ceph-mon[36569]: pgmap v2754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:35 smithi104 ceph-mon[36624]: pgmap v2754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:37 smithi081 ceph-mon[31453]: pgmap v2755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:37 smithi153 ceph-mon[36569]: pgmap v2755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:37 smithi104 ceph-mon[36624]: pgmap v2755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:39 smithi081 ceph-mon[31453]: pgmap v2756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:39 smithi153 ceph-mon[36569]: pgmap v2756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:39 smithi104 ceph-mon[36624]: pgmap v2756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:41.112+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:29:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:41.112+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:29:41.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:41.112+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:29:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:41 smithi081 ceph-mon[31453]: pgmap v2757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:41 smithi153 ceph-mon[36569]: pgmap v2757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:41 smithi104 ceph-mon[36624]: pgmap v2757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:43 smithi081 ceph-mon[31453]: pgmap v2758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:43 smithi153 ceph-mon[36569]: pgmap v2758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:43 smithi104 ceph-mon[36624]: pgmap v2758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:45 smithi081 ceph-mon[31453]: pgmap v2759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:45 smithi153 ceph-mon[36569]: pgmap v2759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:45 smithi104 ceph-mon[36624]: pgmap v2759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:47 smithi081 ceph-mon[31453]: pgmap v2760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:47 smithi153 ceph-mon[36569]: pgmap v2760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:47 smithi104 ceph-mon[36624]: pgmap v2760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:49 smithi081 ceph-mon[31453]: pgmap v2761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:49 smithi153 ceph-mon[36569]: pgmap v2761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:49 smithi104 ceph-mon[36624]: pgmap v2761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:51 smithi081 ceph-mon[31453]: pgmap v2762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:51 smithi153 ceph-mon[36569]: pgmap v2762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:51 smithi104 ceph-mon[36624]: pgmap v2762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:53 smithi153 ceph-mon[36569]: pgmap v2763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:53 smithi104 ceph-mon[36624]: pgmap v2763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:53 smithi081 ceph-mon[31453]: pgmap v2763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:56.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:56.121+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:29:56.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:56.121+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:29:56.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:29:56.121+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:29:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:55 smithi153 ceph-mon[36569]: pgmap v2764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:55 smithi104 ceph-mon[36624]: pgmap v2764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:55 smithi081 ceph-mon[31453]: pgmap v2764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:57 smithi153 ceph-mon[36569]: pgmap v2765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:57 smithi104 ceph-mon[36624]: pgmap v2765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:29:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:57 smithi081 ceph-mon[31453]: pgmap v2765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:29:59 smithi153 ceph-mon[36569]: pgmap v2766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:29:59 smithi104 ceph-mon[36624]: pgmap v2766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:29:59 smithi081 ceph-mon[31453]: pgmap v2766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:30:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:30:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:30:02.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:01 smithi104 ceph-mon[36624]: pgmap v2767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:01 smithi153 ceph-mon[36569]: pgmap v2767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:01 smithi081 ceph-mon[31453]: pgmap v2767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:03 smithi153 ceph-mon[36569]: pgmap v2768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:30:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:03 smithi104 ceph-mon[36624]: pgmap v2768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:30:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:03 smithi081 ceph-mon[31453]: pgmap v2768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:30:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:30:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:30:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:04 smithi153 ceph-mon[36569]: pgmap v2769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:04 smithi153 ceph-mon[36569]: pgmap v2770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:30:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:30:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:04 smithi104 ceph-mon[36624]: pgmap v2769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:04 smithi104 ceph-mon[36624]: pgmap v2770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:30:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:30:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:04 smithi081 ceph-mon[31453]: pgmap v2769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:04 smithi081 ceph-mon[31453]: pgmap v2770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:30:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:07 smithi153 ceph-mon[36569]: pgmap v2771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:07 smithi104 ceph-mon[36624]: pgmap v2771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:07 smithi081 ceph-mon[31453]: pgmap v2771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:09 smithi153 ceph-mon[36569]: pgmap v2772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:09 smithi104 ceph-mon[36624]: pgmap v2772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:09 smithi081 ceph-mon[31453]: pgmap v2772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:11.135+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:30:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:11.135+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:30:11.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:11.135+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:30:11.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:11 smithi081 ceph-mon[31453]: pgmap v2773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:11 smithi153 ceph-mon[36569]: pgmap v2773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:11 smithi104 ceph-mon[36624]: pgmap v2773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:13 smithi153 ceph-mon[36569]: pgmap v2774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:13 smithi104 ceph-mon[36624]: pgmap v2774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:13 smithi081 ceph-mon[31453]: pgmap v2774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:15 smithi153 ceph-mon[36569]: pgmap v2775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:15 smithi104 ceph-mon[36624]: pgmap v2775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:15 smithi081 ceph-mon[31453]: pgmap v2775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:17 smithi153 ceph-mon[36569]: pgmap v2776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:17 smithi104 ceph-mon[36624]: pgmap v2776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:17 smithi081 ceph-mon[31453]: pgmap v2776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:19 smithi153 ceph-mon[36569]: pgmap v2777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:19 smithi104 ceph-mon[36624]: pgmap v2777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:19 smithi081 ceph-mon[31453]: pgmap v2777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:21 smithi153 ceph-mon[36569]: pgmap v2778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:21 smithi104 ceph-mon[36624]: pgmap v2778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:21 smithi081 ceph-mon[31453]: pgmap v2778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:23 smithi153 ceph-mon[36569]: pgmap v2779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:23 smithi104 ceph-mon[36624]: pgmap v2779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:23 smithi081 ceph-mon[31453]: pgmap v2779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:25 smithi153 ceph-mon[36569]: pgmap v2780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:25 smithi104 ceph-mon[36624]: pgmap v2780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:25 smithi081 ceph-mon[31453]: pgmap v2780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:26.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:26.137+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:30:26.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:26.137+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:30:26.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:26.137+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:30:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:26 smithi153 ceph-mon[36569]: pgmap v2781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:26 smithi104 ceph-mon[36624]: pgmap v2781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:26 smithi081 ceph-mon[31453]: pgmap v2781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:28.428+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:30:28.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:28.428+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:30:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:28.436+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:30:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:28.436+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:30:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:29 smithi153 ceph-mon[36569]: pgmap v2782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:29 smithi104 ceph-mon[36624]: pgmap v2782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:29 smithi081 ceph-mon[31453]: pgmap v2782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:31 smithi153 ceph-mon[36569]: pgmap v2783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:31 smithi104 ceph-mon[36624]: pgmap v2783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:31 smithi081 ceph-mon[31453]: pgmap v2783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:33 smithi153 ceph-mon[36569]: pgmap v2784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:33 smithi104 ceph-mon[36624]: pgmap v2784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:33.864 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:33 smithi081 ceph-mon[31453]: pgmap v2784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:35 smithi153 ceph-mon[36569]: pgmap v2785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:35 smithi104 ceph-mon[36624]: pgmap v2785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:35 smithi081 ceph-mon[31453]: pgmap v2785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:37.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:37 smithi153 ceph-mon[36569]: pgmap v2786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:37 smithi104 ceph-mon[36624]: pgmap v2786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:37 smithi081 ceph-mon[31453]: pgmap v2786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:39 smithi153 ceph-mon[36569]: pgmap v2787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:39 smithi104 ceph-mon[36624]: pgmap v2787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:39 smithi081 ceph-mon[31453]: pgmap v2787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:41.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:41.152+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:30:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:41.153+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:30:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:41.153+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:30:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:41 smithi153 ceph-mon[36569]: pgmap v2788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:41 smithi104 ceph-mon[36624]: pgmap v2788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:41 smithi081 ceph-mon[31453]: pgmap v2788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:43 smithi153 ceph-mon[36569]: pgmap v2789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:43 smithi104 ceph-mon[36624]: pgmap v2789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:43 smithi081 ceph-mon[31453]: pgmap v2789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:45 smithi153 ceph-mon[36569]: pgmap v2790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:45 smithi104 ceph-mon[36624]: pgmap v2790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:45 smithi081 ceph-mon[31453]: pgmap v2790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:46 smithi153 ceph-mon[36569]: pgmap v2791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:46 smithi104 ceph-mon[36624]: pgmap v2791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:46 smithi081 ceph-mon[31453]: pgmap v2791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:49.610 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:49 smithi081 ceph-mon[31453]: pgmap v2792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:49 smithi153 ceph-mon[36569]: pgmap v2792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:49 smithi104 ceph-mon[36624]: pgmap v2792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:51 smithi153 ceph-mon[36569]: pgmap v2793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:51 smithi104 ceph-mon[36624]: pgmap v2793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:51 smithi081 ceph-mon[31453]: pgmap v2793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:53 smithi153 ceph-mon[36569]: pgmap v2794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:53 smithi104 ceph-mon[36624]: pgmap v2794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:53 smithi081 ceph-mon[31453]: pgmap v2794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:55 smithi153 ceph-mon[36569]: pgmap v2795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:55 smithi104 ceph-mon[36624]: pgmap v2795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:55 smithi081 ceph-mon[31453]: pgmap v2795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:56.168+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:30:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:56.168+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:30:56.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:30:56.168+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:30:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:57 smithi153 ceph-mon[36569]: pgmap v2796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:57 smithi104 ceph-mon[36624]: pgmap v2796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:57 smithi081 ceph-mon[31453]: pgmap v2796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:30:59 smithi153 ceph-mon[36569]: pgmap v2797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:30:59 smithi104 ceph-mon[36624]: pgmap v2797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:30:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:30:59 smithi081 ceph-mon[31453]: pgmap v2797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:01 smithi153 ceph-mon[36569]: pgmap v2798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:01 smithi104 ceph-mon[36624]: pgmap v2798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:01 smithi081 ceph-mon[31453]: pgmap v2798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:02 smithi153 ceph-mon[36569]: pgmap v2799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:02 smithi104 ceph-mon[36624]: pgmap v2799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:02 smithi081 ceph-mon[31453]: pgmap v2799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:05 smithi153 ceph-mon[36569]: pgmap v2800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:31:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:05 smithi104 ceph-mon[36624]: pgmap v2800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:31:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:05 smithi081 ceph-mon[31453]: pgmap v2800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:31:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:31:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:31:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:31:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:31:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:31:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:31:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:31:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:31:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:31:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:07 smithi153 ceph-mon[36569]: pgmap v2801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:07 smithi153 ceph-mon[36569]: pgmap v2802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:07 smithi104 ceph-mon[36624]: pgmap v2801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:07 smithi104 ceph-mon[36624]: pgmap v2802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:07 smithi081 ceph-mon[31453]: pgmap v2801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:07 smithi081 ceph-mon[31453]: pgmap v2802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:09 smithi153 ceph-mon[36569]: pgmap v2803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:09 smithi104 ceph-mon[36624]: pgmap v2803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:09 smithi081 ceph-mon[31453]: pgmap v2803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:11.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:11.177+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:31:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:11.178+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:31:11.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:11.178+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:31:11.738 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:11 smithi081 ceph-mon[31453]: pgmap v2804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:11 smithi153 ceph-mon[36569]: pgmap v2804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:11 smithi104 ceph-mon[36624]: pgmap v2804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.098+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 msgr2=0x7f16e80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T22:31:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.098+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 msgr2=0x7f16e80a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T22:31:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.098+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 0x7f16e80a5040 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006620 tx=0x7f16f40bd040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T22:31:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.098+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 0x7f16e80a5040 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006620 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-12T22:31:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.098+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 msgr2=0x7f16e80a5040 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-12T22:31:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.098+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 msgr2=0x7f16e80a5040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T22:31:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.098+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4850 0x7f16e80a5040 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T22:31:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.098+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T22:31:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.099+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T22:31:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:12.100+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T22:31:13.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:13.436+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:31:13.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:13.436+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:31:13.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:13.437+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:31:13.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:13.437+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:31:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:13 smithi153 ceph-mon[36569]: pgmap v2805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:13 smithi104 ceph-mon[36624]: pgmap v2805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:13 smithi081 ceph-mon[31453]: pgmap v2805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:14 smithi153 ceph-mon[36569]: pgmap v2806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:14 smithi104 ceph-mon[36624]: pgmap v2806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:14 smithi081 ceph-mon[31453]: pgmap v2806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:17 smithi153 ceph-mon[36569]: pgmap v2807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:17 smithi104 ceph-mon[36624]: pgmap v2807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:17 smithi081 ceph-mon[31453]: pgmap v2807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:19 smithi153 ceph-mon[36569]: pgmap v2808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:19 smithi104 ceph-mon[36624]: pgmap v2808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:19 smithi081 ceph-mon[31453]: pgmap v2808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:21 smithi153 ceph-mon[36569]: pgmap v2809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:21 smithi104 ceph-mon[36624]: pgmap v2809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:21 smithi081 ceph-mon[31453]: pgmap v2809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:23 smithi153 ceph-mon[36569]: pgmap v2810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:23 smithi104 ceph-mon[36624]: pgmap v2810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:23 smithi081 ceph-mon[31453]: pgmap v2810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:25 smithi104 ceph-mon[36624]: pgmap v2811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:25 smithi081 ceph-mon[31453]: pgmap v2811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:25 smithi153 ceph-mon[36569]: pgmap v2811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:26.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:26.186+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:31:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:26.186+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:31:26.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:26.186+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:31:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:27 smithi104 ceph-mon[36624]: pgmap v2812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:27.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:27 smithi153 ceph-mon[36569]: pgmap v2812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:27 smithi081 ceph-mon[31453]: pgmap v2812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:29 smithi104 ceph-mon[36624]: pgmap v2813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:29 smithi081 ceph-mon[31453]: pgmap v2813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:29 smithi153 ceph-mon[36569]: pgmap v2813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:31 smithi104 ceph-mon[36624]: pgmap v2814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:31 smithi081 ceph-mon[31453]: pgmap v2814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:31 smithi153 ceph-mon[36569]: pgmap v2814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:33 smithi104 ceph-mon[36624]: pgmap v2815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:33 smithi081 ceph-mon[31453]: pgmap v2815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:33 smithi153 ceph-mon[36569]: pgmap v2815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:35 smithi081 ceph-mon[31453]: pgmap v2816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:35 smithi153 ceph-mon[36569]: pgmap v2816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:35 smithi104 ceph-mon[36624]: pgmap v2816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:37 smithi081 ceph-mon[31453]: pgmap v2817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:37 smithi153 ceph-mon[36569]: pgmap v2817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:37 smithi104 ceph-mon[36624]: pgmap v2817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:39 smithi081 ceph-mon[31453]: pgmap v2818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:39 smithi153 ceph-mon[36569]: pgmap v2818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:39 smithi104 ceph-mon[36624]: pgmap v2818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:41.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:41.201+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:31:41.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:41.201+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:31:41.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:41.201+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:31:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:41 smithi081 ceph-mon[31453]: pgmap v2819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:41 smithi153 ceph-mon[36569]: pgmap v2819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:41 smithi104 ceph-mon[36624]: pgmap v2819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:43 smithi081 ceph-mon[31453]: pgmap v2820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:43 smithi153 ceph-mon[36569]: pgmap v2820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:43 smithi104 ceph-mon[36624]: pgmap v2820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:45 smithi081 ceph-mon[31453]: pgmap v2821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:45 smithi153 ceph-mon[36569]: pgmap v2821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:45 smithi104 ceph-mon[36624]: pgmap v2821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:47 smithi081 ceph-mon[31453]: pgmap v2822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:47 smithi153 ceph-mon[36569]: pgmap v2822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:47 smithi104 ceph-mon[36624]: pgmap v2822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:49 smithi081 ceph-mon[31453]: pgmap v2823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:49 smithi153 ceph-mon[36569]: pgmap v2823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:49 smithi104 ceph-mon[36624]: pgmap v2823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:51 smithi081 ceph-mon[31453]: pgmap v2824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:51 smithi153 ceph-mon[36569]: pgmap v2824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:51 smithi104 ceph-mon[36624]: pgmap v2824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:53 smithi081 ceph-mon[31453]: pgmap v2825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:53 smithi153 ceph-mon[36569]: pgmap v2825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:53 smithi104 ceph-mon[36624]: pgmap v2825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:55 smithi081 ceph-mon[31453]: pgmap v2826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:55 smithi153 ceph-mon[36569]: pgmap v2826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:55 smithi104 ceph-mon[36624]: pgmap v2826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:56.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:56.215+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:31:56.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:56.215+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:31:56.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:31:56.215+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:31:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:57 smithi081 ceph-mon[31453]: pgmap v2827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:57 smithi153 ceph-mon[36569]: pgmap v2827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:57 smithi104 ceph-mon[36624]: pgmap v2827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:31:59 smithi081 ceph-mon[31453]: pgmap v2828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:31:59 smithi153 ceph-mon[36569]: pgmap v2828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:31:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:31:59 smithi104 ceph-mon[36624]: pgmap v2828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:01 smithi081 ceph-mon[31453]: pgmap v2829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:01 smithi153 ceph-mon[36569]: pgmap v2829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:01 smithi104 ceph-mon[36624]: pgmap v2829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:03 smithi153 ceph-mon[36569]: pgmap v2830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:03 smithi104 ceph-mon[36624]: pgmap v2830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:03 smithi081 ceph-mon[31453]: pgmap v2830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:05 smithi153 ceph-mon[36569]: pgmap v2831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:05 smithi104 ceph-mon[36624]: pgmap v2831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:05 smithi081 ceph-mon[31453]: pgmap v2831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:07 smithi153 ceph-mon[36569]: pgmap v2832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:32:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:07 smithi104 ceph-mon[36624]: pgmap v2832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:32:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:07 smithi081 ceph-mon[31453]: pgmap v2832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:32:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:32:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:32:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:32:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:32:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:32:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:32:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:32:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:32:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:32:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:09 smithi153 ceph-mon[36569]: pgmap v2833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:09 smithi153 ceph-mon[36569]: pgmap v2834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:09 smithi104 ceph-mon[36624]: pgmap v2833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:09 smithi104 ceph-mon[36624]: pgmap v2834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:09 smithi081 ceph-mon[31453]: pgmap v2833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:09 smithi081 ceph-mon[31453]: pgmap v2834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:11.226+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:32:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:11.226+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:32:11.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:11.226+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:32:11.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:11 smithi081 ceph-mon[31453]: pgmap v2835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:11 smithi153 ceph-mon[36569]: pgmap v2835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:11 smithi104 ceph-mon[36624]: pgmap v2835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:13 smithi153 ceph-mon[36569]: pgmap v2836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:13 smithi104 ceph-mon[36624]: pgmap v2836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:13 smithi081 ceph-mon[31453]: pgmap v2836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:15 smithi153 ceph-mon[36569]: pgmap v2837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:15 smithi104 ceph-mon[36624]: pgmap v2837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:15 smithi081 ceph-mon[31453]: pgmap v2837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:17 smithi153 ceph-mon[36569]: pgmap v2838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:17 smithi104 ceph-mon[36624]: pgmap v2838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:17 smithi081 ceph-mon[31453]: pgmap v2838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:19 smithi153 ceph-mon[36569]: pgmap v2839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:19 smithi104 ceph-mon[36624]: pgmap v2839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:19 smithi081 ceph-mon[31453]: pgmap v2839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:21 smithi153 ceph-mon[36569]: pgmap v2840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:21 smithi104 ceph-mon[36624]: pgmap v2840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:21 smithi081 ceph-mon[31453]: pgmap v2840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:23 smithi153 ceph-mon[36569]: pgmap v2841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:23 smithi104 ceph-mon[36624]: pgmap v2841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:23 smithi081 ceph-mon[31453]: pgmap v2841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:25 smithi153 ceph-mon[36569]: pgmap v2842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:25 smithi104 ceph-mon[36624]: pgmap v2842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:25 smithi081 ceph-mon[31453]: pgmap v2842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:26.238+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:32:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:26.238+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:32:26.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:26.238+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:32:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:27 smithi153 ceph-mon[36569]: pgmap v2843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:27 smithi104 ceph-mon[36624]: pgmap v2843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:27 smithi081 ceph-mon[31453]: pgmap v2843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:29 smithi153 ceph-mon[36569]: pgmap v2844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:29 smithi104 ceph-mon[36624]: pgmap v2844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:29 smithi081 ceph-mon[31453]: pgmap v2844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:31 smithi153 ceph-mon[36569]: pgmap v2845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:31 smithi104 ceph-mon[36624]: pgmap v2845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:31 smithi081 ceph-mon[31453]: pgmap v2845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:33 smithi153 ceph-mon[36569]: pgmap v2846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:33 smithi104 ceph-mon[36624]: pgmap v2846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:33 smithi081 ceph-mon[31453]: pgmap v2846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:35 smithi153 ceph-mon[36569]: pgmap v2847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:35 smithi104 ceph-mon[36624]: pgmap v2847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:35 smithi081 ceph-mon[31453]: pgmap v2847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:37 smithi153 ceph-mon[36569]: pgmap v2848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:37.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:37 smithi104 ceph-mon[36624]: pgmap v2848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:37.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:37 smithi081 ceph-mon[31453]: pgmap v2848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:39 smithi153 ceph-mon[36569]: pgmap v2849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:39.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:39 smithi104 ceph-mon[36624]: pgmap v2849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:39.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:39 smithi081 ceph-mon[31453]: pgmap v2849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:41.246+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:32:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:41.246+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:32:41.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:41.246+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:32:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:41 smithi153 ceph-mon[36569]: pgmap v2850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:41 smithi104 ceph-mon[36624]: pgmap v2850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:41 smithi081 ceph-mon[31453]: pgmap v2850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:43 smithi153 ceph-mon[36569]: pgmap v2851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:43.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:43 smithi104 ceph-mon[36624]: pgmap v2851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:43 smithi081 ceph-mon[31453]: pgmap v2851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:45 smithi153 ceph-mon[36569]: pgmap v2852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:45.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:45 smithi104 ceph-mon[36624]: pgmap v2852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:45 smithi081 ceph-mon[31453]: pgmap v2852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:47 smithi153 ceph-mon[36569]: pgmap v2853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:47 smithi104 ceph-mon[36624]: pgmap v2853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:47.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:47 smithi081 ceph-mon[31453]: pgmap v2853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:49 smithi153 ceph-mon[36569]: pgmap v2854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:49 smithi104 ceph-mon[36624]: pgmap v2854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:49.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:49 smithi081 ceph-mon[31453]: pgmap v2854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:51 smithi153 ceph-mon[36569]: pgmap v2855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:51.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:51 smithi104 ceph-mon[36624]: pgmap v2855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:51.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:51 smithi081 ceph-mon[31453]: pgmap v2855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:53 smithi153 ceph-mon[36569]: pgmap v2856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:53.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:53 smithi104 ceph-mon[36624]: pgmap v2856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:53.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:53 smithi081 ceph-mon[31453]: pgmap v2856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:55 smithi153 ceph-mon[36569]: pgmap v2857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:55.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:55 smithi104 ceph-mon[36624]: pgmap v2857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:55.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:55 smithi081 ceph-mon[31453]: pgmap v2857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:56.260+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:32:56.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:56.260+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:32:56.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:32:56.260+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:32:57.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:57 smithi104 ceph-mon[36624]: pgmap v2858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:57 smithi081 ceph-mon[31453]: pgmap v2858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:57.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:57 smithi153 ceph-mon[36569]: pgmap v2858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:59.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:32:59 smithi104 ceph-mon[36624]: pgmap v2859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:59.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:32:59 smithi081 ceph-mon[31453]: pgmap v2859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:32:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:32:59 smithi153 ceph-mon[36569]: pgmap v2859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:01.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:01 smithi104 ceph-mon[36624]: pgmap v2860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:01.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:01 smithi081 ceph-mon[31453]: pgmap v2860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:01.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:01 smithi153 ceph-mon[36569]: pgmap v2860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:03 smithi104 ceph-mon[36624]: pgmap v2861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:03.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:03 smithi081 ceph-mon[31453]: pgmap v2861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:03.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:03 smithi153 ceph-mon[36569]: pgmap v2861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:05 smithi104 ceph-mon[36624]: pgmap v2862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:05 smithi081 ceph-mon[31453]: pgmap v2862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:05.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:05 smithi153 ceph-mon[36569]: pgmap v2862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:07 smithi081 ceph-mon[31453]: pgmap v2863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:07.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:07 smithi153 ceph-mon[36569]: pgmap v2863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:07.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:07 smithi104 ceph-mon[36624]: pgmap v2863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:08.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:33:08.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:33:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:33:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:09 smithi081 ceph-mon[31453]: pgmap v2864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:33:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:33:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:33:09.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:09 smithi153 ceph-mon[36569]: pgmap v2864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:09.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:33:09.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:33:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:33:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:09 smithi104 ceph-mon[36624]: pgmap v2864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:33:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:33:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:33:10.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:10 smithi081 ceph-mon[31453]: pgmap v2865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:10.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:10 smithi081 ceph-mon[31453]: pgmap v2866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:10.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:10 smithi153 ceph-mon[36569]: pgmap v2865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:10.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:10 smithi153 ceph-mon[36569]: pgmap v2866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:10 smithi104 ceph-mon[36624]: pgmap v2865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:10 smithi104 ceph-mon[36624]: pgmap v2866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:11.272 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:11.273+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:33:11.272 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:11.273+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:33:11.272 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:11.273+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:33:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:12 smithi081 ceph-mon[31453]: pgmap v2867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:12.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:12 smithi153 ceph-mon[36569]: pgmap v2867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:12 smithi104 ceph-mon[36624]: pgmap v2867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:14.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:14 smithi081 ceph-mon[31453]: pgmap v2868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:14.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:14 smithi153 ceph-mon[36569]: pgmap v2868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:14 smithi104 ceph-mon[36624]: pgmap v2868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:16.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:16 smithi081 ceph-mon[31453]: pgmap v2869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:16.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:16 smithi153 ceph-mon[36569]: pgmap v2869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:16 smithi104 ceph-mon[36624]: pgmap v2869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:18 smithi081 ceph-mon[31453]: pgmap v2870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:18.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:18 smithi153 ceph-mon[36569]: pgmap v2870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:18 smithi104 ceph-mon[36624]: pgmap v2870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:20 smithi081 ceph-mon[31453]: pgmap v2871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:20.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:20 smithi153 ceph-mon[36569]: pgmap v2871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:20 smithi104 ceph-mon[36624]: pgmap v2871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:22.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:22 smithi081 ceph-mon[31453]: pgmap v2872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:22 smithi153 ceph-mon[36569]: pgmap v2872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:22.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:22 smithi104 ceph-mon[36624]: pgmap v2872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:24 smithi081 ceph-mon[31453]: pgmap v2873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:24.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:24 smithi153 ceph-mon[36569]: pgmap v2873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:24.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:24 smithi104 ceph-mon[36624]: pgmap v2873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:26.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:26.288+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:33:26.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:26.288+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:33:26.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:26.288+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:33:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:26 smithi081 ceph-mon[31453]: pgmap v2874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:26.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:26 smithi153 ceph-mon[36569]: pgmap v2874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:26.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:26 smithi104 ceph-mon[36624]: pgmap v2874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:28.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:28 smithi081 ceph-mon[31453]: pgmap v2875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:28.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:28 smithi153 ceph-mon[36569]: pgmap v2875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:28 smithi104 ceph-mon[36624]: pgmap v2875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:30.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:30 smithi153 ceph-mon[36569]: pgmap v2876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:30.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:30 smithi104 ceph-mon[36624]: pgmap v2876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:30.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:30 smithi081 ceph-mon[31453]: pgmap v2876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:32.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:32 smithi153 ceph-mon[36569]: pgmap v2877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:32 smithi104 ceph-mon[36624]: pgmap v2877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:32.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:32 smithi081 ceph-mon[31453]: pgmap v2877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:34.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:34 smithi153 ceph-mon[36569]: pgmap v2878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:34.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:34 smithi104 ceph-mon[36624]: pgmap v2878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:34.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:34 smithi081 ceph-mon[31453]: pgmap v2878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:36.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:36 smithi153 ceph-mon[36569]: pgmap v2879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:36.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:36 smithi104 ceph-mon[36624]: pgmap v2879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:36.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:36 smithi081 ceph-mon[31453]: pgmap v2879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:38.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:38 smithi153 ceph-mon[36569]: pgmap v2880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:38.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:38 smithi104 ceph-mon[36624]: pgmap v2880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:38.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:38 smithi081 ceph-mon[31453]: pgmap v2880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:40.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:40 smithi081 ceph-mon[31453]: pgmap v2881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:40.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:40 smithi153 ceph-mon[36569]: pgmap v2881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:40.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:40 smithi104 ceph-mon[36624]: pgmap v2881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:41.289+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:33:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:41.289+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:33:41.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:41.289+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:33:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:42 smithi153 ceph-mon[36569]: pgmap v2882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:42.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:42 smithi104 ceph-mon[36624]: pgmap v2882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:42.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:42 smithi081 ceph-mon[31453]: pgmap v2882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:44.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:44 smithi153 ceph-mon[36569]: pgmap v2883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:44.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:44 smithi104 ceph-mon[36624]: pgmap v2883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:44.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:44 smithi081 ceph-mon[31453]: pgmap v2883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:46.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:46 smithi153 ceph-mon[36569]: pgmap v2884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:46.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:46 smithi104 ceph-mon[36624]: pgmap v2884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:46.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:46 smithi081 ceph-mon[31453]: pgmap v2884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:48.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:48 smithi153 ceph-mon[36569]: pgmap v2885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:48.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:48 smithi104 ceph-mon[36624]: pgmap v2885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:48.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:48 smithi081 ceph-mon[31453]: pgmap v2885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:50.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:50 smithi153 ceph-mon[36569]: pgmap v2886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:50.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:50 smithi104 ceph-mon[36624]: pgmap v2886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:50.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:50 smithi081 ceph-mon[31453]: pgmap v2886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:52.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:52 smithi153 ceph-mon[36569]: pgmap v2887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:52 smithi104 ceph-mon[36624]: pgmap v2887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:52.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:52 smithi081 ceph-mon[31453]: pgmap v2887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:54.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:54 smithi153 ceph-mon[36569]: pgmap v2888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:54.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:54 smithi104 ceph-mon[36624]: pgmap v2888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:54.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:54 smithi081 ceph-mon[31453]: pgmap v2888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:56.303+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:33:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:56.303+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:33:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:33:56.303+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:33:56.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:56 smithi153 ceph-mon[36569]: pgmap v2889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:56.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:56 smithi104 ceph-mon[36624]: pgmap v2889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:56.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:56 smithi081 ceph-mon[31453]: pgmap v2889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:58.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:33:58 smithi153 ceph-mon[36569]: pgmap v2890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:58.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:33:58 smithi104 ceph-mon[36624]: pgmap v2890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:33:58.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:33:58 smithi081 ceph-mon[31453]: pgmap v2890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:00 smithi153 ceph-mon[36569]: pgmap v2891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:00.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:00 smithi104 ceph-mon[36624]: pgmap v2891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:00.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:00 smithi081 ceph-mon[31453]: pgmap v2891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:02.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:02 smithi153 ceph-mon[36569]: pgmap v2892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:02.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:02 smithi104 ceph-mon[36624]: pgmap v2892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:02.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:02 smithi081 ceph-mon[31453]: pgmap v2892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:04.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:04 smithi153 ceph-mon[36569]: pgmap v2893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:04.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:04 smithi104 ceph-mon[36624]: pgmap v2893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:04.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:04 smithi081 ceph-mon[31453]: pgmap v2893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:06.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:06 smithi153 ceph-mon[36569]: pgmap v2894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:06.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:06 smithi104 ceph-mon[36624]: pgmap v2894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:06.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:06 smithi081 ceph-mon[31453]: pgmap v2894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:08.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:08 smithi153 ceph-mon[36569]: pgmap v2895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:08.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:08 smithi104 ceph-mon[36624]: pgmap v2895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:08.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:08 smithi081 ceph-mon[31453]: pgmap v2895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:09.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:34:09.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:34:09.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:34:10.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:10 smithi153 ceph-mon[36569]: pgmap v2896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:10.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:10 smithi104 ceph-mon[36624]: pgmap v2896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:10.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:10 smithi081 ceph-mon[31453]: pgmap v2896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:11.306+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:34:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:11.306+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:34:11.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:11.306+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:34:11.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:34:11.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:34:11.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:34:11.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:34:11.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:34:11.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:34:11.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:34:11.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:34:11.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:34:12.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:12 smithi153 ceph-mon[36569]: pgmap v2897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:12 smithi153 ceph-mon[36569]: pgmap v2898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:12.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:12 smithi104 ceph-mon[36624]: pgmap v2897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:12.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:12 smithi104 ceph-mon[36624]: pgmap v2898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:12.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:12 smithi081 ceph-mon[31453]: pgmap v2897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:12.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:12 smithi081 ceph-mon[31453]: pgmap v2898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:13.465+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:34:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:13.465+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:34:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:13.466+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:34:13.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:13.466+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:34:13.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:13 smithi153 ceph-mon[36569]: pgmap v2899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:13.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:13 smithi104 ceph-mon[36624]: pgmap v2899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:13.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:13 smithi081 ceph-mon[31453]: pgmap v2899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:16.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:15 smithi153 ceph-mon[36569]: pgmap v2900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:15 smithi104 ceph-mon[36624]: pgmap v2900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:16.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:15 smithi081 ceph-mon[31453]: pgmap v2900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:18.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:17 smithi153 ceph-mon[36569]: pgmap v2901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:18.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:17 smithi104 ceph-mon[36624]: pgmap v2901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:18.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:17 smithi081 ceph-mon[31453]: pgmap v2901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:20.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:19 smithi153 ceph-mon[36569]: pgmap v2902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:20.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:19 smithi104 ceph-mon[36624]: pgmap v2902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:20.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:19 smithi081 ceph-mon[31453]: pgmap v2902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:22.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:21 smithi153 ceph-mon[36569]: pgmap v2903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:22.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:21 smithi104 ceph-mon[36624]: pgmap v2903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:22.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:21 smithi081 ceph-mon[31453]: pgmap v2903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:24.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:23 smithi104 ceph-mon[36624]: pgmap v2904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:24.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:23 smithi081 ceph-mon[31453]: pgmap v2904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:24.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:23 smithi153 ceph-mon[36569]: pgmap v2904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:26.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:25 smithi104 ceph-mon[36624]: pgmap v2905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:26.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:25 smithi081 ceph-mon[31453]: pgmap v2905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:26.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:25 smithi153 ceph-mon[36569]: pgmap v2905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:26.313+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:34:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:26.313+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:34:26.311 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:26.313+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:34:28.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:27 smithi104 ceph-mon[36624]: pgmap v2906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:28.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:27 smithi153 ceph-mon[36569]: pgmap v2906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:28.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:27 smithi081 ceph-mon[31453]: pgmap v2906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:30.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:29 smithi104 ceph-mon[36624]: pgmap v2907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:30.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:29 smithi081 ceph-mon[31453]: pgmap v2907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:29 smithi153 ceph-mon[36569]: pgmap v2907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:31 smithi081 ceph-mon[31453]: pgmap v2908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:32.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:31 smithi153 ceph-mon[36569]: pgmap v2908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:32.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:31 smithi104 ceph-mon[36624]: pgmap v2908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:34.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:33 smithi081 ceph-mon[31453]: pgmap v2909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:34.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:33 smithi153 ceph-mon[36569]: pgmap v2909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:34.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:33 smithi104 ceph-mon[36624]: pgmap v2909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:36.132 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:35 smithi104 ceph-mon[36624]: pgmap v2910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:36.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:35 smithi081 ceph-mon[31453]: pgmap v2910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:36.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:35 smithi153 ceph-mon[36569]: pgmap v2910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:38.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:37 smithi081 ceph-mon[31453]: pgmap v2911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:38.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:37 smithi153 ceph-mon[36569]: pgmap v2911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:38.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:37 smithi104 ceph-mon[36624]: pgmap v2911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:39 smithi081 ceph-mon[31453]: pgmap v2912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:40.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:39 smithi153 ceph-mon[36569]: pgmap v2912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:40.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:39 smithi104 ceph-mon[36624]: pgmap v2912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:41.316+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:34:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:41.316+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:34:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:41.316+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:34:42.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:41 smithi081 ceph-mon[31453]: pgmap v2913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:42.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:41 smithi153 ceph-mon[36569]: pgmap v2913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:42.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:41 smithi104 ceph-mon[36624]: pgmap v2913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:44.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:43 smithi081 ceph-mon[31453]: pgmap v2914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:44.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:43 smithi153 ceph-mon[36569]: pgmap v2914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:44.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:43 smithi104 ceph-mon[36624]: pgmap v2914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:46.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:45 smithi081 ceph-mon[31453]: pgmap v2915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:46.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:45 smithi153 ceph-mon[36569]: pgmap v2915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:46.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:45 smithi104 ceph-mon[36624]: pgmap v2915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:48.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:47 smithi081 ceph-mon[31453]: pgmap v2916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:48.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:47 smithi153 ceph-mon[36569]: pgmap v2916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:47 smithi104 ceph-mon[36624]: pgmap v2916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:50.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:49 smithi081 ceph-mon[31453]: pgmap v2917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:50.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:49 smithi153 ceph-mon[36569]: pgmap v2917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:50.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:49 smithi104 ceph-mon[36624]: pgmap v2917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:51 smithi081 ceph-mon[31453]: pgmap v2918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:52.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:51 smithi153 ceph-mon[36569]: pgmap v2918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:52.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:51 smithi104 ceph-mon[36624]: pgmap v2918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:54.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:53 smithi081 ceph-mon[31453]: pgmap v2919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:54.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:53 smithi153 ceph-mon[36569]: pgmap v2919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:54.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:53 smithi104 ceph-mon[36624]: pgmap v2919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:56.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:55 smithi081 ceph-mon[31453]: pgmap v2920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:56.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:55 smithi153 ceph-mon[36569]: pgmap v2920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:56.329+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:34:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:56.329+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:34:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:34:56.329+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:34:56.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:55 smithi104 ceph-mon[36624]: pgmap v2920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:58.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:57 smithi081 ceph-mon[31453]: pgmap v2921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:58.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:57 smithi153 ceph-mon[36569]: pgmap v2921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:34:58.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:57 smithi104 ceph-mon[36624]: pgmap v2921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:00.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:34:59 smithi153 ceph-mon[36569]: pgmap v2922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:00.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:34:59 smithi104 ceph-mon[36624]: pgmap v2922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:00.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:34:59 smithi081 ceph-mon[31453]: pgmap v2922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:02.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:01 smithi153 ceph-mon[36569]: pgmap v2923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:02.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:01 smithi104 ceph-mon[36624]: pgmap v2923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:02.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:01 smithi081 ceph-mon[31453]: pgmap v2923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:04.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:03 smithi153 ceph-mon[36569]: pgmap v2924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:04.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:03 smithi104 ceph-mon[36624]: pgmap v2924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:04.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:03 smithi081 ceph-mon[31453]: pgmap v2924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:06.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:05 smithi153 ceph-mon[36569]: pgmap v2925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:06.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:05 smithi104 ceph-mon[36624]: pgmap v2925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:06.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:05 smithi081 ceph-mon[31453]: pgmap v2925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:08.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:07 smithi153 ceph-mon[36569]: pgmap v2926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:08.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:07 smithi104 ceph-mon[36624]: pgmap v2926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:08.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:07 smithi081 ceph-mon[31453]: pgmap v2926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:10.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:09 smithi153 ceph-mon[36569]: pgmap v2927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:10.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:09 smithi104 ceph-mon[36624]: pgmap v2927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:10.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:09 smithi081 ceph-mon[31453]: pgmap v2927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:11.335+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:35:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:11.335+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:35:11.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:11.335+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: pgmap v2928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: pgmap v2928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: pgmap v2928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:12.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:35:13.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:35:13.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:35:13.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: pgmap v2929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:13.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: pgmap v2930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:13.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:35:13.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:35:13.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:35:13.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: pgmap v2929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: pgmap v2930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:13.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:35:13.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: pgmap v2929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: pgmap v2930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:35:15.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:15 smithi153 ceph-mon[36569]: pgmap v2931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:15.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:15 smithi104 ceph-mon[36624]: pgmap v2931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:15.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:15 smithi081 ceph-mon[31453]: pgmap v2931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:17.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:17 smithi104 ceph-mon[36624]: pgmap v2932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:17 smithi081 ceph-mon[31453]: pgmap v2932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:17 smithi153 ceph-mon[36569]: pgmap v2932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:19.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:19 smithi104 ceph-mon[36624]: pgmap v2933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:19.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:19 smithi081 ceph-mon[31453]: pgmap v2933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:19.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:19 smithi153 ceph-mon[36569]: pgmap v2933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:21.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:21 smithi104 ceph-mon[36624]: pgmap v2934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:21.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:21 smithi081 ceph-mon[31453]: pgmap v2934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:21.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:21 smithi153 ceph-mon[36569]: pgmap v2934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:23.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:23 smithi104 ceph-mon[36624]: pgmap v2935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:23.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:23 smithi081 ceph-mon[31453]: pgmap v2935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:23.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:23 smithi153 ceph-mon[36569]: pgmap v2935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:25.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:25 smithi081 ceph-mon[31453]: pgmap v2936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:25.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:25 smithi153 ceph-mon[36569]: pgmap v2936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:25.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:25 smithi104 ceph-mon[36624]: pgmap v2936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:26.350+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:35:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:26.350+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:35:26.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:26.350+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:35:27.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:27 smithi081 ceph-mon[31453]: pgmap v2937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:27.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:27 smithi153 ceph-mon[36569]: pgmap v2937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:27.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:27 smithi104 ceph-mon[36624]: pgmap v2937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:29.606 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:29 smithi153 ceph-mon[36569]: pgmap v2938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:29.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:29 smithi081 ceph-mon[31453]: pgmap v2938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:29.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:29 smithi104 ceph-mon[36624]: pgmap v2938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:31.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:31 smithi081 ceph-mon[31453]: pgmap v2939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:31.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:31 smithi153 ceph-mon[36569]: pgmap v2939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:31.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:31 smithi104 ceph-mon[36624]: pgmap v2939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:33.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:33 smithi081 ceph-mon[31453]: pgmap v2940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:33.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:33 smithi153 ceph-mon[36569]: pgmap v2940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:33.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:33 smithi104 ceph-mon[36624]: pgmap v2940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:35.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:35 smithi081 ceph-mon[31453]: pgmap v2941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:35.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:35 smithi153 ceph-mon[36569]: pgmap v2941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:35.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:35 smithi104 ceph-mon[36624]: pgmap v2941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:37.631 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:37 smithi104 ceph-mon[36624]: pgmap v2942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:37.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:37 smithi081 ceph-mon[31453]: pgmap v2942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:37.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:37 smithi153 ceph-mon[36569]: pgmap v2942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:39.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:39 smithi081 ceph-mon[31453]: pgmap v2943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:39.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:39 smithi153 ceph-mon[36569]: pgmap v2943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:39.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:39 smithi104 ceph-mon[36624]: pgmap v2943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:41.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:41.367+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:35:41.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:41.367+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:35:41.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:41.367+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:35:41.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:41 smithi081 ceph-mon[31453]: pgmap v2944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:41.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:41 smithi153 ceph-mon[36569]: pgmap v2944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:41.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:41 smithi104 ceph-mon[36624]: pgmap v2944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:43.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:43 smithi081 ceph-mon[31453]: pgmap v2945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:43.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:43 smithi153 ceph-mon[36569]: pgmap v2945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:43.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:43 smithi104 ceph-mon[36624]: pgmap v2945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:45.455 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:45 smithi081 ceph-mon[31453]: pgmap v2946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:45.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:45 smithi153 ceph-mon[36569]: pgmap v2946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:45.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:45 smithi104 ceph-mon[36624]: pgmap v2946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:47.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:47 smithi081 ceph-mon[31453]: pgmap v2947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:47 smithi153 ceph-mon[36569]: pgmap v2947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:47.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:47 smithi104 ceph-mon[36624]: pgmap v2947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:49.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:49 smithi081 ceph-mon[31453]: pgmap v2948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:49 smithi153 ceph-mon[36569]: pgmap v2948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:49.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:49 smithi104 ceph-mon[36624]: pgmap v2948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:51 smithi081 ceph-mon[31453]: pgmap v2949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:51.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:51 smithi153 ceph-mon[36569]: pgmap v2949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:51.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:51 smithi104 ceph-mon[36624]: pgmap v2949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:53.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:53 smithi153 ceph-mon[36569]: pgmap v2950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:53.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:53 smithi104 ceph-mon[36624]: pgmap v2950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:53 smithi081 ceph-mon[31453]: pgmap v2950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:55.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:55 smithi153 ceph-mon[36569]: pgmap v2951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:55.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:55 smithi104 ceph-mon[36624]: pgmap v2951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:55.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:55 smithi081 ceph-mon[31453]: pgmap v2951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:56.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:56.377+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:35:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:56.377+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:35:56.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:35:56.377+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:35:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:57 smithi153 ceph-mon[36569]: pgmap v2952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:57.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:57 smithi104 ceph-mon[36624]: pgmap v2952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:57.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:57 smithi081 ceph-mon[31453]: pgmap v2952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:59.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:35:59 smithi153 ceph-mon[36569]: pgmap v2953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:59.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:35:59 smithi104 ceph-mon[36624]: pgmap v2953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:35:59.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:35:59 smithi081 ceph-mon[31453]: pgmap v2953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:01.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:01 smithi153 ceph-mon[36569]: pgmap v2954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:01 smithi104 ceph-mon[36624]: pgmap v2954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:01.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:01 smithi081 ceph-mon[31453]: pgmap v2954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:03 smithi153 ceph-mon[36569]: pgmap v2955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:03 smithi104 ceph-mon[36624]: pgmap v2955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:03.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:03 smithi081 ceph-mon[31453]: pgmap v2955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:05 smithi153 ceph-mon[36569]: pgmap v2956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:05 smithi104 ceph-mon[36624]: pgmap v2956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:05 smithi081 ceph-mon[31453]: pgmap v2956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:07 smithi153 ceph-mon[36569]: pgmap v2957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:07 smithi104 ceph-mon[36624]: pgmap v2957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:07 smithi081 ceph-mon[31453]: pgmap v2957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:09.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:09 smithi153 ceph-mon[36569]: pgmap v2958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:09 smithi104 ceph-mon[36624]: pgmap v2958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:09.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:09 smithi081 ceph-mon[31453]: pgmap v2958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:11.390+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:36:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:11.390+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:36:11.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:11.390+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:36:11.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:11 smithi153 ceph-mon[36569]: pgmap v2959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:11 smithi104 ceph-mon[36624]: pgmap v2959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:11.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:11 smithi081 ceph-mon[31453]: pgmap v2959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:13 smithi153 ceph-mon[36569]: pgmap v2960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:36:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:13 smithi104 ceph-mon[36624]: pgmap v2960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:36:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:13 smithi081 ceph-mon[31453]: pgmap v2960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:36:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:15 smithi153 ceph-mon[36569]: pgmap v2961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:36:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:36:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:15 smithi104 ceph-mon[36624]: pgmap v2961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:15.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:15.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:15.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:36:15.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:36:15.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:15.876 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:15 smithi081 ceph-mon[31453]: pgmap v2961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:15.876 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:15.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:15.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:36:15.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:36:15.877 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:36:16.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:16 smithi153 ceph-mon[36569]: pgmap v2962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:16.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:16 smithi153 ceph-mon[36569]: pgmap v2963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:16.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:16 smithi104 ceph-mon[36624]: pgmap v2962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:16.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:16 smithi104 ceph-mon[36624]: pgmap v2963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:16.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:16 smithi081 ceph-mon[31453]: pgmap v2962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:16.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:16 smithi081 ceph-mon[31453]: pgmap v2963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:18.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:18 smithi153 ceph-mon[36569]: pgmap v2964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:18.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:18 smithi104 ceph-mon[36624]: pgmap v2964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:18.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:18 smithi081 ceph-mon[31453]: pgmap v2964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:20.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:20 smithi153 ceph-mon[36569]: pgmap v2965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:20.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:20 smithi104 ceph-mon[36624]: pgmap v2965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:20.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:20 smithi081 ceph-mon[31453]: pgmap v2965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:22.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:22 smithi153 ceph-mon[36569]: pgmap v2966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:22.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:22 smithi104 ceph-mon[36624]: pgmap v2966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:22.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:22 smithi081 ceph-mon[31453]: pgmap v2966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:24.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:24 smithi153 ceph-mon[36569]: pgmap v2967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:24.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:24 smithi104 ceph-mon[36624]: pgmap v2967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:24.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:24 smithi081 ceph-mon[31453]: pgmap v2967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:26.394+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:36:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:26.394+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:36:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:26.394+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:36:26.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:26 smithi153 ceph-mon[36569]: pgmap v2968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:26.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:26 smithi104 ceph-mon[36624]: pgmap v2968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:26.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:26 smithi081 ceph-mon[31453]: pgmap v2968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:28.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:28 smithi153 ceph-mon[36569]: pgmap v2969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:28.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:28 smithi104 ceph-mon[36624]: pgmap v2969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:28.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:28 smithi081 ceph-mon[31453]: pgmap v2969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:30.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:30 smithi153 ceph-mon[36569]: pgmap v2970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:30.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:30 smithi104 ceph-mon[36624]: pgmap v2970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:30.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:30 smithi081 ceph-mon[31453]: pgmap v2970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:32.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:32 smithi153 ceph-mon[36569]: pgmap v2971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:32.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:32 smithi104 ceph-mon[36624]: pgmap v2971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:32.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:32 smithi081 ceph-mon[31453]: pgmap v2971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:34.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:34 smithi153 ceph-mon[36569]: pgmap v2972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:34.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:34 smithi104 ceph-mon[36624]: pgmap v2972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:34.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:34 smithi081 ceph-mon[31453]: pgmap v2972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:36.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:36 smithi153 ceph-mon[36569]: pgmap v2973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:36.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:36 smithi104 ceph-mon[36624]: pgmap v2973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:36 smithi081 ceph-mon[31453]: pgmap v2973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:38.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:38 smithi153 ceph-mon[36569]: pgmap v2974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:38.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:38 smithi104 ceph-mon[36624]: pgmap v2974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:38.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:38 smithi081 ceph-mon[31453]: pgmap v2974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:40.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:40 smithi153 ceph-mon[36569]: pgmap v2975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:40.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:40 smithi104 ceph-mon[36624]: pgmap v2975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:40.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:40 smithi081 ceph-mon[31453]: pgmap v2975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:41.409+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:36:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:41.410+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:36:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:41.410+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:36:42.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:42 smithi153 ceph-mon[36569]: pgmap v2976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:42.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:42 smithi104 ceph-mon[36624]: pgmap v2976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:42.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:42 smithi081 ceph-mon[31453]: pgmap v2976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:44.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:44 smithi153 ceph-mon[36569]: pgmap v2977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:44.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:44 smithi104 ceph-mon[36624]: pgmap v2977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:44.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:44 smithi081 ceph-mon[31453]: pgmap v2977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:46.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:46 smithi153 ceph-mon[36569]: pgmap v2978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:46.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:46 smithi104 ceph-mon[36624]: pgmap v2978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:46.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:46 smithi081 ceph-mon[31453]: pgmap v2978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:48.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:48 smithi153 ceph-mon[36569]: pgmap v2979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:48.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:48 smithi104 ceph-mon[36624]: pgmap v2979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:48.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:48 smithi081 ceph-mon[31453]: pgmap v2979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:50.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:50 smithi153 ceph-mon[36569]: pgmap v2980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:50.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:50 smithi104 ceph-mon[36624]: pgmap v2980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:50.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:50 smithi081 ceph-mon[31453]: pgmap v2980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:51.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:51 smithi081 ceph-mon[31453]: pgmap v2981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:52.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:51 smithi153 ceph-mon[36569]: pgmap v2981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:52.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:51 smithi104 ceph-mon[36624]: pgmap v2981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:54.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:53 smithi153 ceph-mon[36569]: pgmap v2982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:54.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:53 smithi104 ceph-mon[36624]: pgmap v2982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:54.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:53 smithi081 ceph-mon[31453]: pgmap v2982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:56.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:55 smithi153 ceph-mon[36569]: pgmap v2983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:56.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:55 smithi104 ceph-mon[36624]: pgmap v2983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:56.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:55 smithi081 ceph-mon[31453]: pgmap v2983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:56.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:56.422+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:36:56.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:56.422+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:36:56.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:36:56.422+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:36:58.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:57 smithi153 ceph-mon[36569]: pgmap v2984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:58.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:57 smithi104 ceph-mon[36624]: pgmap v2984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:36:58.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:57 smithi081 ceph-mon[31453]: pgmap v2984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:00.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:36:59 smithi153 ceph-mon[36569]: pgmap v2985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:00.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:36:59 smithi104 ceph-mon[36624]: pgmap v2985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:00.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:36:59 smithi081 ceph-mon[31453]: pgmap v2985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:02.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:01 smithi153 ceph-mon[36569]: pgmap v2986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:02.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:01 smithi104 ceph-mon[36624]: pgmap v2986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:02.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:01 smithi081 ceph-mon[31453]: pgmap v2986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:04.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:03 smithi153 ceph-mon[36569]: pgmap v2987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:04.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:03 smithi104 ceph-mon[36624]: pgmap v2987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:04.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:03 smithi081 ceph-mon[31453]: pgmap v2987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:06.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:05 smithi153 ceph-mon[36569]: pgmap v2988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:06.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:05 smithi104 ceph-mon[36624]: pgmap v2988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:06.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:05 smithi081 ceph-mon[31453]: pgmap v2988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:08.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:07 smithi153 ceph-mon[36569]: pgmap v2989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:08.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:07 smithi104 ceph-mon[36624]: pgmap v2989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:08.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:07 smithi081 ceph-mon[31453]: pgmap v2989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:10.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:09 smithi153 ceph-mon[36569]: pgmap v2990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:10.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:09 smithi104 ceph-mon[36624]: pgmap v2990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:10.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:09 smithi081 ceph-mon[31453]: pgmap v2990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:11.437+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:37:11.433 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:11.437+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:37:11.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:11.437+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:37:12.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:11 smithi153 ceph-mon[36569]: pgmap v2991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:12.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:11 smithi104 ceph-mon[36624]: pgmap v2991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:12.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:11 smithi081 ceph-mon[31453]: pgmap v2991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:13.495+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:37:13.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:13.495+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:37:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:13.495+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:37:13.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:13.495+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:37:14.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:13 smithi153 ceph-mon[36569]: pgmap v2992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:14.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:13 smithi104 ceph-mon[36624]: pgmap v2992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:14.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:13 smithi081 ceph-mon[31453]: pgmap v2992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:16.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:15 smithi153 ceph-mon[36569]: pgmap v2993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:16.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:37:16.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:15 smithi104 ceph-mon[36624]: pgmap v2993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:16.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:37:16.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:15 smithi081 ceph-mon[31453]: pgmap v2993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:16.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:37:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:37:17.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:37:17.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: pgmap v2994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:17.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: pgmap v2995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:17.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: pgmap v2994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: pgmap v2995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:17.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:37:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:37:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: pgmap v2994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:17.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: pgmap v2995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:37:19.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:19 smithi104 ceph-mon[36624]: pgmap v2996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:19.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:19 smithi081 ceph-mon[31453]: pgmap v2996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:19 smithi153 ceph-mon[36569]: pgmap v2996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:21.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:21 smithi104 ceph-mon[36624]: pgmap v2997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:21.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:21 smithi081 ceph-mon[31453]: pgmap v2997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:21 smithi153 ceph-mon[36569]: pgmap v2997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:23.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:23 smithi104 ceph-mon[36624]: pgmap v2998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:23.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:23 smithi081 ceph-mon[31453]: pgmap v2998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:23 smithi153 ceph-mon[36569]: pgmap v2998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:25 smithi104 ceph-mon[36624]: pgmap v2999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:25 smithi081 ceph-mon[31453]: pgmap v2999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:25 smithi153 ceph-mon[36569]: pgmap v2999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:26.439 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:26.443+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:37:26.439 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:26.443+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:37:26.439 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:26.443+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:37:27.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:27 smithi104 ceph-mon[36624]: pgmap v3000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:27.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:27 smithi081 ceph-mon[31453]: pgmap v3000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:27 smithi153 ceph-mon[36569]: pgmap v3000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:29.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:29 smithi081 ceph-mon[31453]: pgmap v3001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:29.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:29 smithi153 ceph-mon[36569]: pgmap v3001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:29.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:29 smithi104 ceph-mon[36624]: pgmap v3001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:31.604 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:31 smithi153 ceph-mon[36569]: pgmap v3002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:31.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:31 smithi081 ceph-mon[31453]: pgmap v3002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:31 smithi104 ceph-mon[36624]: pgmap v3002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:33.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:33 smithi081 ceph-mon[31453]: pgmap v3003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:33.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:33 smithi153 ceph-mon[36569]: pgmap v3003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:33 smithi104 ceph-mon[36624]: pgmap v3003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:35.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:35 smithi081 ceph-mon[31453]: pgmap v3004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:35.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:35 smithi153 ceph-mon[36569]: pgmap v3004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:35 smithi104 ceph-mon[36624]: pgmap v3004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:37.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:37 smithi081 ceph-mon[31453]: pgmap v3005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:37.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:37 smithi153 ceph-mon[36569]: pgmap v3005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:37 smithi104 ceph-mon[36624]: pgmap v3005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:39.629 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:39 smithi104 ceph-mon[36624]: pgmap v3006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:39.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:39 smithi081 ceph-mon[31453]: pgmap v3006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:39.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:39 smithi153 ceph-mon[36569]: pgmap v3006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:41.458+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:37:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:41.458+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:37:41.454 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:41.458+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:37:41.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:41 smithi081 ceph-mon[31453]: pgmap v3007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:41 smithi153 ceph-mon[36569]: pgmap v3007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:41 smithi104 ceph-mon[36624]: pgmap v3007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:43.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:43 smithi081 ceph-mon[31453]: pgmap v3008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:43.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:43 smithi153 ceph-mon[36569]: pgmap v3008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:43 smithi104 ceph-mon[36624]: pgmap v3008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:45.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:45 smithi081 ceph-mon[31453]: pgmap v3009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:45.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:45 smithi153 ceph-mon[36569]: pgmap v3009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:45.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:45 smithi104 ceph-mon[36624]: pgmap v3009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:47.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:47 smithi081 ceph-mon[31453]: pgmap v3010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:47.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:47 smithi153 ceph-mon[36569]: pgmap v3010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:47 smithi104 ceph-mon[36624]: pgmap v3010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:49.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:49 smithi081 ceph-mon[31453]: pgmap v3011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:49.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:49 smithi153 ceph-mon[36569]: pgmap v3011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:49 smithi104 ceph-mon[36624]: pgmap v3011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:51.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:51 smithi081 ceph-mon[31453]: pgmap v3012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:51.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:51 smithi153 ceph-mon[36569]: pgmap v3012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:51 smithi104 ceph-mon[36624]: pgmap v3012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:53.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:53 smithi081 ceph-mon[31453]: pgmap v3013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:53.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:53 smithi153 ceph-mon[36569]: pgmap v3013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:53.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:53 smithi104 ceph-mon[36624]: pgmap v3013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:55.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:55 smithi081 ceph-mon[31453]: pgmap v3014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:55.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:55 smithi153 ceph-mon[36569]: pgmap v3014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:55 smithi104 ceph-mon[36624]: pgmap v3014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:56.471+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:37:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:56.471+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:37:56.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:37:56.471+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:37:57.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:57 smithi081 ceph-mon[31453]: pgmap v3015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:57.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:57 smithi153 ceph-mon[36569]: pgmap v3015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:57.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:57 smithi104 ceph-mon[36624]: pgmap v3015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:59.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:37:59 smithi153 ceph-mon[36569]: pgmap v3016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:37:59 smithi104 ceph-mon[36624]: pgmap v3016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:37:59.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:37:59 smithi081 ceph-mon[31453]: pgmap v3016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:01.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:01 smithi153 ceph-mon[36569]: pgmap v3017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:01.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:01 smithi104 ceph-mon[36624]: pgmap v3017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:01.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:01 smithi081 ceph-mon[31453]: pgmap v3017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:03.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:03 smithi153 ceph-mon[36569]: pgmap v3018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:03.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:03 smithi104 ceph-mon[36624]: pgmap v3018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:03.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:03 smithi081 ceph-mon[31453]: pgmap v3018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:05.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:05 smithi153 ceph-mon[36569]: pgmap v3019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:05.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:05 smithi104 ceph-mon[36624]: pgmap v3019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:05.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:05 smithi081 ceph-mon[31453]: pgmap v3019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:07.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:07 smithi153 ceph-mon[36569]: pgmap v3020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:07.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:07 smithi104 ceph-mon[36624]: pgmap v3020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:07.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:07 smithi081 ceph-mon[31453]: pgmap v3020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:09.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:09 smithi153 ceph-mon[36569]: pgmap v3021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:09.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:09 smithi104 ceph-mon[36624]: pgmap v3021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:09.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:09 smithi081 ceph-mon[31453]: pgmap v3021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:11.484+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:38:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:11.484+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:38:11.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:11.484+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:38:11.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:11 smithi153 ceph-mon[36569]: pgmap v3022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:11.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:11 smithi104 ceph-mon[36624]: pgmap v3022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:11.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:11 smithi081 ceph-mon[31453]: pgmap v3022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:13.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:13 smithi153 ceph-mon[36569]: pgmap v3023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:13.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:13 smithi104 ceph-mon[36624]: pgmap v3023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:13.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:13 smithi081 ceph-mon[31453]: pgmap v3023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:15.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:15 smithi153 ceph-mon[36569]: pgmap v3024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:15.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:15 smithi104 ceph-mon[36624]: pgmap v3024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:15.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:15 smithi081 ceph-mon[31453]: pgmap v3024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:17.705 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:17 smithi081 ceph-mon[31453]: pgmap v3025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:17.705 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:38:17.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:17 smithi153 ceph-mon[36569]: pgmap v3025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:17.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:38:17.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:17 smithi104 ceph-mon[36624]: pgmap v3025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:17.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:38:18.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:38:18.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:38:18.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:38:18.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:38:18.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:38:18.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:38:18.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:38:18.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:38:18.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:38:19.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:19 smithi153 ceph-mon[36569]: pgmap v3026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:19 smithi153 ceph-mon[36569]: pgmap v3027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:19 smithi104 ceph-mon[36624]: pgmap v3026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:19.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:19 smithi104 ceph-mon[36624]: pgmap v3027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:19 smithi081 ceph-mon[31453]: pgmap v3026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:19.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:19 smithi081 ceph-mon[31453]: pgmap v3027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:21.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:21 smithi153 ceph-mon[36569]: pgmap v3028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:21 smithi104 ceph-mon[36624]: pgmap v3028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:21 smithi081 ceph-mon[31453]: pgmap v3028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:23.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:23 smithi153 ceph-mon[36569]: pgmap v3029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:23 smithi104 ceph-mon[36624]: pgmap v3029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:23 smithi081 ceph-mon[31453]: pgmap v3029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:25.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:25 smithi153 ceph-mon[36569]: pgmap v3030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:25 smithi104 ceph-mon[36624]: pgmap v3030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:25 smithi081 ceph-mon[31453]: pgmap v3030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:26.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:26.499+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:38:26.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:26.499+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:38:26.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:26.499+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:38:27.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:27 smithi153 ceph-mon[36569]: pgmap v3031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:27 smithi104 ceph-mon[36624]: pgmap v3031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:27.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:27 smithi081 ceph-mon[31453]: pgmap v3031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:29.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:29 smithi153 ceph-mon[36569]: pgmap v3032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:29.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:29 smithi104 ceph-mon[36624]: pgmap v3032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:29.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:29 smithi081 ceph-mon[31453]: pgmap v3032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:31.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:31 smithi153 ceph-mon[36569]: pgmap v3033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:31 smithi104 ceph-mon[36624]: pgmap v3033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:31.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:31 smithi081 ceph-mon[31453]: pgmap v3033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:33.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:33 smithi153 ceph-mon[36569]: pgmap v3034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:33.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:33 smithi104 ceph-mon[36624]: pgmap v3034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:33.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:33 smithi081 ceph-mon[31453]: pgmap v3034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:35.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:35 smithi153 ceph-mon[36569]: pgmap v3035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:35 smithi104 ceph-mon[36624]: pgmap v3035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:35.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:35 smithi081 ceph-mon[31453]: pgmap v3035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:37.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:37 smithi153 ceph-mon[36569]: pgmap v3036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:37 smithi104 ceph-mon[36624]: pgmap v3036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:37.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:37 smithi081 ceph-mon[31453]: pgmap v3036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:39.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:39 smithi153 ceph-mon[36569]: pgmap v3037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:39 smithi104 ceph-mon[36624]: pgmap v3037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:39.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:39 smithi081 ceph-mon[31453]: pgmap v3037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:41.515+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:38:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:41.515+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:38:41.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:41.515+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:38:41.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:41 smithi153 ceph-mon[36569]: pgmap v3038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:41 smithi104 ceph-mon[36624]: pgmap v3038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:41.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:41 smithi081 ceph-mon[31453]: pgmap v3038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:43.510+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:38:43.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:43.510+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:38:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:43.510+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:38:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:43.510+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:38:43.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:43 smithi153 ceph-mon[36569]: pgmap v3039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:43.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:43 smithi104 ceph-mon[36624]: pgmap v3039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:43.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:43 smithi081 ceph-mon[31453]: pgmap v3039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:45.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:45 smithi153 ceph-mon[36569]: pgmap v3040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:45.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:45 smithi104 ceph-mon[36624]: pgmap v3040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:45.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:45 smithi081 ceph-mon[31453]: pgmap v3040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:47.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:47 smithi153 ceph-mon[36569]: pgmap v3041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:47.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:47 smithi104 ceph-mon[36624]: pgmap v3041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:47.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:47 smithi081 ceph-mon[31453]: pgmap v3041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:49.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:49 smithi153 ceph-mon[36569]: pgmap v3042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:49 smithi104 ceph-mon[36624]: pgmap v3042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:49.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:49 smithi081 ceph-mon[31453]: pgmap v3042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:51.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:51 smithi153 ceph-mon[36569]: pgmap v3043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:51.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:51 smithi104 ceph-mon[36624]: pgmap v3043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:51.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:51 smithi081 ceph-mon[31453]: pgmap v3043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:53.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:53 smithi104 ceph-mon[36624]: pgmap v3044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:53.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:53 smithi081 ceph-mon[31453]: pgmap v3044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:54.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:53 smithi153 ceph-mon[36569]: pgmap v3044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:55.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:55 smithi104 ceph-mon[36624]: pgmap v3045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:55.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:55 smithi081 ceph-mon[31453]: pgmap v3045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:56.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:55 smithi153 ceph-mon[36569]: pgmap v3045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:56.530+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:38:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:56.530+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:38:56.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:38:56.530+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:38:57.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:57 smithi104 ceph-mon[36624]: pgmap v3046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:57.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:57 smithi081 ceph-mon[31453]: pgmap v3046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:58.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:57 smithi153 ceph-mon[36569]: pgmap v3046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:59.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:38:59 smithi104 ceph-mon[36624]: pgmap v3047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:38:59.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:38:59 smithi081 ceph-mon[31453]: pgmap v3047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:00.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:38:59 smithi153 ceph-mon[36569]: pgmap v3047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:01.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:01 smithi104 ceph-mon[36624]: pgmap v3048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:01.853 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:01 smithi153 ceph-mon[36569]: pgmap v3048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:01.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:01 smithi081 ceph-mon[31453]: pgmap v3048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:02.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:02 smithi081 ceph-mon[31453]: pgmap v3049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:03.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:02 smithi153 ceph-mon[36569]: pgmap v3049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:03.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:02 smithi104 ceph-mon[36624]: pgmap v3049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:05.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:04 smithi081 ceph-mon[31453]: pgmap v3050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:05.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:04 smithi153 ceph-mon[36569]: pgmap v3050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:05.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:04 smithi104 ceph-mon[36624]: pgmap v3050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:07.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:06 smithi081 ceph-mon[31453]: pgmap v3051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:07.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:06 smithi153 ceph-mon[36569]: pgmap v3051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:07.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:06 smithi104 ceph-mon[36624]: pgmap v3051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:09.129 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:08 smithi104 ceph-mon[36624]: pgmap v3052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:09.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:08 smithi081 ceph-mon[31453]: pgmap v3052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:09.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:08 smithi153 ceph-mon[36569]: pgmap v3052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:11.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:10 smithi081 ceph-mon[31453]: pgmap v3053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:11.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:10 smithi153 ceph-mon[36569]: pgmap v3053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:11.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:10 smithi104 ceph-mon[36624]: pgmap v3053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:11.542+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:39:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:11.542+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:39:11.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:11.542+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:39:13.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:12 smithi081 ceph-mon[31453]: pgmap v3054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:13.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:12 smithi153 ceph-mon[36569]: pgmap v3054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:13.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:12 smithi104 ceph-mon[36624]: pgmap v3054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:15.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:14 smithi081 ceph-mon[31453]: pgmap v3055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:15.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:14 smithi153 ceph-mon[36569]: pgmap v3055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:15.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:14 smithi104 ceph-mon[36624]: pgmap v3055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:17.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:16 smithi081 ceph-mon[31453]: pgmap v3056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:17.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:16 smithi153 ceph-mon[36569]: pgmap v3056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:17.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:16 smithi104 ceph-mon[36624]: pgmap v3056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:19.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:18 smithi081 ceph-mon[31453]: pgmap v3057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:19.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:39:19.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:19.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:19.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:18 smithi153 ceph-mon[36569]: pgmap v3057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:19.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:39:19.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:19.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:19.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:18 smithi104 ceph-mon[36624]: pgmap v3057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:19.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:39:19.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:19.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:20.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:20.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:20.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:20.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:20.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:20.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:21 smithi153 ceph-mon[36569]: pgmap v3058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:21.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:21 smithi104 ceph-mon[36624]: pgmap v3058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:21 smithi081 ceph-mon[31453]: pgmap v3058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:21.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:23.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:23 smithi153 ceph-mon[36569]: pgmap v3059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:23 smithi104 ceph-mon[36624]: pgmap v3059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:23.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:23.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:23 smithi081 ceph-mon[31453]: pgmap v3059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:24.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:39:24.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:39:24.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:39:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:39:24.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:24.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:39:24.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:39:24.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:39:25.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:25 smithi153 ceph-mon[36569]: pgmap v3060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:25.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:25 smithi153 ceph-mon[36569]: pgmap v3061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:25.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:25 smithi153 ceph-mon[36569]: pgmap v3062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:25.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:25 smithi104 ceph-mon[36624]: pgmap v3060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:25 smithi104 ceph-mon[36624]: pgmap v3061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:25.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:25 smithi104 ceph-mon[36624]: pgmap v3062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:25.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:25 smithi081 ceph-mon[31453]: pgmap v3060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:25 smithi081 ceph-mon[31453]: pgmap v3061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:25 smithi081 ceph-mon[31453]: pgmap v3062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:26.557+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:39:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:26.557+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:39:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:26.557+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:39:27.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:27 smithi153 ceph-mon[36569]: pgmap v3063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:27.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:27 smithi104 ceph-mon[36624]: pgmap v3063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:27.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:27 smithi081 ceph-mon[31453]: pgmap v3063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:29.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:29 smithi153 ceph-mon[36569]: pgmap v3064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:29.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:29 smithi104 ceph-mon[36624]: pgmap v3064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:29.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:29 smithi081 ceph-mon[31453]: pgmap v3064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:31.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:31 smithi153 ceph-mon[36569]: pgmap v3065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:31 smithi104 ceph-mon[36624]: pgmap v3065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:31.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:31 smithi081 ceph-mon[31453]: pgmap v3065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:33.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:33 smithi153 ceph-mon[36569]: pgmap v3066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:33.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:33 smithi104 ceph-mon[36624]: pgmap v3066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:33.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:33 smithi081 ceph-mon[31453]: pgmap v3066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:35.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:35 smithi153 ceph-mon[36569]: pgmap v3067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:35.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:35 smithi104 ceph-mon[36624]: pgmap v3067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:35.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:35 smithi081 ceph-mon[31453]: pgmap v3067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:37.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:37 smithi153 ceph-mon[36569]: pgmap v3068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:37.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:37 smithi104 ceph-mon[36624]: pgmap v3068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:37.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:37 smithi081 ceph-mon[31453]: pgmap v3068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:39.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:39 smithi153 ceph-mon[36569]: pgmap v3069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:39.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:39 smithi104 ceph-mon[36624]: pgmap v3069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:39.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:39 smithi081 ceph-mon[31453]: pgmap v3069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:41.570+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:39:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:41.570+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:39:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:41.570+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:39:41.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:41 smithi153 ceph-mon[36569]: pgmap v3070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:41.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:41 smithi104 ceph-mon[36624]: pgmap v3070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:41.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:41 smithi081 ceph-mon[31453]: pgmap v3070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:43.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:43 smithi153 ceph-mon[36569]: pgmap v3071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:43.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:43 smithi104 ceph-mon[36624]: pgmap v3071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:43.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:43 smithi081 ceph-mon[31453]: pgmap v3071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:45.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:45 smithi153 ceph-mon[36569]: pgmap v3072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:45.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:45 smithi104 ceph-mon[36624]: pgmap v3072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:45.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:45 smithi081 ceph-mon[31453]: pgmap v3072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:47.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:47 smithi153 ceph-mon[36569]: pgmap v3073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:47.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:47 smithi104 ceph-mon[36624]: pgmap v3073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:47.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:47 smithi081 ceph-mon[31453]: pgmap v3073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:49.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:49 smithi153 ceph-mon[36569]: pgmap v3074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:49.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:49 smithi104 ceph-mon[36624]: pgmap v3074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:49.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:49 smithi081 ceph-mon[31453]: pgmap v3074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:51.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:51 smithi153 ceph-mon[36569]: pgmap v3075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:51.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:51 smithi104 ceph-mon[36624]: pgmap v3075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:51.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:51 smithi081 ceph-mon[31453]: pgmap v3075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:53.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:53 smithi153 ceph-mon[36569]: pgmap v3076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:53.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:53 smithi104 ceph-mon[36624]: pgmap v3076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:53.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:53 smithi081 ceph-mon[31453]: pgmap v3076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:55.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:55 smithi104 ceph-mon[36624]: pgmap v3077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:55.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:55 smithi081 ceph-mon[31453]: pgmap v3077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:56.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:55 smithi153 ceph-mon[36569]: pgmap v3077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:56.586+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:39:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:56.586+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:39:56.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:39:56.586+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:39:57.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:57 smithi104 ceph-mon[36624]: pgmap v3078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:57.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:57 smithi081 ceph-mon[31453]: pgmap v3078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:58.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:57 smithi153 ceph-mon[36569]: pgmap v3078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:59.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:39:59 smithi104 ceph-mon[36624]: pgmap v3079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:39:59.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:39:59 smithi081 ceph-mon[31453]: pgmap v3079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:00.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:39:59 smithi153 ceph-mon[36569]: pgmap v3079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:00.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:40:00.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:40:01.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:40:01.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:01 smithi081 ceph-mon[31453]: pgmap v3080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:01.952 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:01 smithi153 ceph-mon[36569]: pgmap v3080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:02.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:01 smithi104 ceph-mon[36624]: pgmap v3080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:03.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:03 smithi081 ceph-mon[31453]: pgmap v3081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:04.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:03 smithi153 ceph-mon[36569]: pgmap v3081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:04.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:03 smithi104 ceph-mon[36624]: pgmap v3081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:05.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:05 smithi081 ceph-mon[31453]: pgmap v3082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:06.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:05 smithi153 ceph-mon[36569]: pgmap v3082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:06.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:05 smithi104 ceph-mon[36624]: pgmap v3082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:06.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:06 smithi081 ceph-mon[31453]: pgmap v3083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:07.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:06 smithi153 ceph-mon[36569]: pgmap v3083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:07.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:06 smithi104 ceph-mon[36624]: pgmap v3083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:09.604 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:09 smithi081 ceph-mon[31453]: pgmap v3084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:09.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:09 smithi153 ceph-mon[36569]: pgmap v3084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:09.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:09 smithi104 ceph-mon[36624]: pgmap v3084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:11.590+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:40:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:11.590+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:40:11.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:11.590+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:40:11.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:11 smithi153 ceph-mon[36569]: pgmap v3085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:11.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:11 smithi104 ceph-mon[36624]: pgmap v3085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:11.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:11 smithi081 ceph-mon[31453]: pgmap v3085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:13.525+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:40:13.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:13.525+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:40:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:13.526+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:40:13.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:13.526+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:40:13.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:13 smithi153 ceph-mon[36569]: pgmap v3086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:13.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:13 smithi104 ceph-mon[36624]: pgmap v3086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:13.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:13 smithi081 ceph-mon[31453]: pgmap v3086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:15.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:15 smithi104 ceph-mon[36624]: pgmap v3087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:15.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:15 smithi081 ceph-mon[31453]: pgmap v3087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:16.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:15 smithi153 ceph-mon[36569]: pgmap v3087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:17.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:17 smithi104 ceph-mon[36624]: pgmap v3088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:17.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:17 smithi081 ceph-mon[31453]: pgmap v3088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:18.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:17 smithi153 ceph-mon[36569]: pgmap v3088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:19.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:19 smithi104 ceph-mon[36624]: pgmap v3089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:19.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:19 smithi081 ceph-mon[31453]: pgmap v3089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:20.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:19 smithi153 ceph-mon[36569]: pgmap v3089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:21.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:21 smithi104 ceph-mon[36624]: pgmap v3090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:21.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:21 smithi081 ceph-mon[31453]: pgmap v3090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:22.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:21 smithi153 ceph-mon[36569]: pgmap v3090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:23.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:23 smithi104 ceph-mon[36624]: pgmap v3091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:23.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:23 smithi081 ceph-mon[31453]: pgmap v3091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:24.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:23 smithi153 ceph-mon[36569]: pgmap v3091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:25.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:25 smithi081 ceph-mon[31453]: pgmap v3092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:40:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:25.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:25 smithi153 ceph-mon[36569]: pgmap v3092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:40:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:26.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:25 smithi104 ceph-mon[36624]: pgmap v3092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:26.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:40:26.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:26.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:26.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:26.604+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:40:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:26.604+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:40:26.600 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:26.604+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:40:26.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:40:26.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:40:26.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:27.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:40:27.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:40:27.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:27.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:40:27.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:40:27.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:40:27.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:27 smithi081 ceph-mon[31453]: pgmap v3093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:27.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:27 smithi081 ceph-mon[31453]: pgmap v3094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:28.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:27 smithi153 ceph-mon[36569]: pgmap v3093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:28.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:27 smithi153 ceph-mon[36569]: pgmap v3094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:28.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:27 smithi104 ceph-mon[36624]: pgmap v3093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:28.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:27 smithi104 ceph-mon[36624]: pgmap v3094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:28.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:28 smithi081 ceph-mon[31453]: pgmap v3095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:29.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:28 smithi153 ceph-mon[36569]: pgmap v3095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:29.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:28 smithi104 ceph-mon[36624]: pgmap v3095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:31.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:31 smithi081 ceph-mon[31453]: pgmap v3096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:31.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:31 smithi153 ceph-mon[36569]: pgmap v3096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:31.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:31 smithi104 ceph-mon[36624]: pgmap v3096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:33.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:33 smithi153 ceph-mon[36569]: pgmap v3097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:33.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:33 smithi081 ceph-mon[31453]: pgmap v3097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:33.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:33 smithi104 ceph-mon[36624]: pgmap v3097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:35.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:35 smithi081 ceph-mon[31453]: pgmap v3098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:35.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:35 smithi153 ceph-mon[36569]: pgmap v3098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:35.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:35 smithi104 ceph-mon[36624]: pgmap v3098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:37.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:37 smithi081 ceph-mon[31453]: pgmap v3099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:37.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:37 smithi153 ceph-mon[36569]: pgmap v3099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:37.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:37 smithi104 ceph-mon[36624]: pgmap v3099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:39.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:39 smithi081 ceph-mon[31453]: pgmap v3100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:39.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:39 smithi153 ceph-mon[36569]: pgmap v3100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:39.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:39 smithi104 ceph-mon[36624]: pgmap v3100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:41.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:41 smithi081 ceph-mon[31453]: pgmap v3101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:41.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:41 smithi153 ceph-mon[36569]: pgmap v3101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:41.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:41 smithi104 ceph-mon[36624]: pgmap v3101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:41.610+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:40:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:41.610+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:40:41.604 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:41.610+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:40:43.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:43 smithi153 ceph-mon[36569]: pgmap v3102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:43.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:43 smithi081 ceph-mon[31453]: pgmap v3102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:43.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:43 smithi104 ceph-mon[36624]: pgmap v3102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:45.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:45 smithi153 ceph-mon[36569]: pgmap v3103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:45.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:45 smithi104 ceph-mon[36624]: pgmap v3103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:45.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:45 smithi081 ceph-mon[31453]: pgmap v3103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:47.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:47 smithi153 ceph-mon[36569]: pgmap v3104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:47.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:47 smithi104 ceph-mon[36624]: pgmap v3104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:47.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:47 smithi081 ceph-mon[31453]: pgmap v3104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:49.453 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:49 smithi081 ceph-mon[31453]: pgmap v3105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:49.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:49 smithi153 ceph-mon[36569]: pgmap v3105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:49.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:49 smithi104 ceph-mon[36624]: pgmap v3105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:51.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:51 smithi153 ceph-mon[36569]: pgmap v3106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:51.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:51 smithi104 ceph-mon[36624]: pgmap v3106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:51.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:51 smithi081 ceph-mon[31453]: pgmap v3106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:53.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:53 smithi153 ceph-mon[36569]: pgmap v3107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:53.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:53 smithi104 ceph-mon[36624]: pgmap v3107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:53.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:53 smithi081 ceph-mon[31453]: pgmap v3107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:55.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:55 smithi153 ceph-mon[36569]: pgmap v3108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:55.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:55 smithi104 ceph-mon[36624]: pgmap v3108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:55.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:55 smithi081 ceph-mon[31453]: pgmap v3108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:56.623+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:40:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:56.623+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:40:56.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:40:56.623+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:40:57.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:57 smithi081 ceph-mon[31453]: pgmap v3109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:57.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:57 smithi153 ceph-mon[36569]: pgmap v3109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:57.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:57 smithi104 ceph-mon[36624]: pgmap v3109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:59.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:40:59 smithi153 ceph-mon[36569]: pgmap v3110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:59.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:40:59 smithi104 ceph-mon[36624]: pgmap v3110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:40:59.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:40:59 smithi081 ceph-mon[31453]: pgmap v3110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:01.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:01 smithi153 ceph-mon[36569]: pgmap v3111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:01.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:01 smithi104 ceph-mon[36624]: pgmap v3111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:01.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:01 smithi081 ceph-mon[31453]: pgmap v3111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:03.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:03 smithi153 ceph-mon[36569]: pgmap v3112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:03.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:03 smithi104 ceph-mon[36624]: pgmap v3112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:03.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:03 smithi081 ceph-mon[31453]: pgmap v3112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:05.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:05 smithi153 ceph-mon[36569]: pgmap v3113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:05.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:05 smithi104 ceph-mon[36624]: pgmap v3113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:05.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:05 smithi081 ceph-mon[31453]: pgmap v3113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:07.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:07 smithi153 ceph-mon[36569]: pgmap v3114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:07.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:07 smithi104 ceph-mon[36624]: pgmap v3114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:07.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:07 smithi081 ceph-mon[31453]: pgmap v3114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:09.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:09 smithi153 ceph-mon[36569]: pgmap v3115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:09.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:09 smithi104 ceph-mon[36624]: pgmap v3115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:09.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:09 smithi081 ceph-mon[31453]: pgmap v3115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:11.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:11 smithi153 ceph-mon[36569]: pgmap v3116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:11.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:11 smithi104 ceph-mon[36624]: pgmap v3116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:11.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:11.633+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:41:11.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:11.633+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:41:11.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:11.633+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:41:11.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:11 smithi081 ceph-mon[31453]: pgmap v3116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:13.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:13 smithi153 ceph-mon[36569]: pgmap v3117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:13.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:13 smithi104 ceph-mon[36624]: pgmap v3117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:13.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:13 smithi081 ceph-mon[31453]: pgmap v3117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:15.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:15 smithi153 ceph-mon[36569]: pgmap v3118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:15.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:15 smithi104 ceph-mon[36624]: pgmap v3118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:15.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:15 smithi081 ceph-mon[31453]: pgmap v3118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:17.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:17 smithi153 ceph-mon[36569]: pgmap v3119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:17.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:17 smithi104 ceph-mon[36624]: pgmap v3119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:17.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:17 smithi081 ceph-mon[31453]: pgmap v3119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:19.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:19 smithi153 ceph-mon[36569]: pgmap v3120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:19.767 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:19 smithi081 ceph-mon[31453]: pgmap v3120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:19.767 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:19 smithi104 ceph-mon[36624]: pgmap v3120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:21.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:21 smithi153 ceph-mon[36569]: pgmap v3121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:21.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:21 smithi104 ceph-mon[36624]: pgmap v3121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:21.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:21 smithi081 ceph-mon[31453]: pgmap v3121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:23.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:23 smithi153 ceph-mon[36569]: pgmap v3122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:23.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:23 smithi104 ceph-mon[36624]: pgmap v3122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:23.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:23 smithi081 ceph-mon[31453]: pgmap v3122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:25.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:25 smithi153 ceph-mon[36569]: pgmap v3123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:25.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:25 smithi104 ceph-mon[36624]: pgmap v3123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:25.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:25 smithi081 ceph-mon[31453]: pgmap v3123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:26.648+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:41:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:26.648+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:41:26.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:26.648+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:41:27.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:27 smithi153 ceph-mon[36569]: pgmap v3124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:27.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:41:27.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:27 smithi104 ceph-mon[36624]: pgmap v3124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:27.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:41:27.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:27 smithi081 ceph-mon[31453]: pgmap v3124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:27.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:41:28.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:41:28.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:41:28.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:41:28.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:41:28.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:41:28.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:41:28.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:41:28.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:41:28.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:41:29.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:29 smithi153 ceph-mon[36569]: pgmap v3125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:29.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:29 smithi153 ceph-mon[36569]: pgmap v3126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:29.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:29 smithi104 ceph-mon[36624]: pgmap v3125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:29.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:29 smithi104 ceph-mon[36624]: pgmap v3126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:29.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:29 smithi081 ceph-mon[31453]: pgmap v3125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:29.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:29 smithi081 ceph-mon[31453]: pgmap v3126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:31.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:31 smithi153 ceph-mon[36569]: pgmap v3127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:31.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:31 smithi104 ceph-mon[36624]: pgmap v3127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:31.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:31 smithi081 ceph-mon[31453]: pgmap v3127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:33.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:33 smithi153 ceph-mon[36569]: pgmap v3128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:33.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:33 smithi104 ceph-mon[36624]: pgmap v3128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:33.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:33 smithi081 ceph-mon[31453]: pgmap v3128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:35.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:35 smithi153 ceph-mon[36569]: pgmap v3129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:35.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:35 smithi104 ceph-mon[36624]: pgmap v3129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:35.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:35 smithi081 ceph-mon[31453]: pgmap v3129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:37.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:37 smithi153 ceph-mon[36569]: pgmap v3130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:37.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:37 smithi104 ceph-mon[36624]: pgmap v3130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:37.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:37 smithi081 ceph-mon[31453]: pgmap v3130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:39.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:39 smithi153 ceph-mon[36569]: pgmap v3131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:39.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:39 smithi104 ceph-mon[36624]: pgmap v3131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:39.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:39 smithi081 ceph-mon[31453]: pgmap v3131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:41.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:41 smithi153 ceph-mon[36569]: pgmap v3132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:41.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:41 smithi104 ceph-mon[36624]: pgmap v3132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:41.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:41.651+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:41:41.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:41.651+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:41:41.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:41.651+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:41:41.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:41 smithi081 ceph-mon[31453]: pgmap v3132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:43.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:43 smithi104 ceph-mon[36624]: pgmap v3133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:43.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:43 smithi081 ceph-mon[31453]: pgmap v3133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:43.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:43 smithi153 ceph-mon[36569]: pgmap v3133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:45.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:45 smithi104 ceph-mon[36624]: pgmap v3134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:45.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:45 smithi081 ceph-mon[31453]: pgmap v3134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:45.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:45 smithi153 ceph-mon[36569]: pgmap v3134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:47.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:47 smithi104 ceph-mon[36624]: pgmap v3135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:47.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:47 smithi081 ceph-mon[31453]: pgmap v3135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:47.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:47 smithi153 ceph-mon[36569]: pgmap v3135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:49.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:49 smithi104 ceph-mon[36624]: pgmap v3136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:49.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:49 smithi081 ceph-mon[31453]: pgmap v3136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:49.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:49 smithi153 ceph-mon[36569]: pgmap v3136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:51.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:51 smithi104 ceph-mon[36624]: pgmap v3137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:51.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:51 smithi081 ceph-mon[31453]: pgmap v3137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:51.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:51 smithi153 ceph-mon[36569]: pgmap v3137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:53.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:53 smithi081 ceph-mon[31453]: pgmap v3138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:53.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:53 smithi153 ceph-mon[36569]: pgmap v3138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:53.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:53 smithi104 ceph-mon[36624]: pgmap v3138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:55.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:55 smithi081 ceph-mon[31453]: pgmap v3139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:55.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:55 smithi153 ceph-mon[36569]: pgmap v3139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:55.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:55 smithi104 ceph-mon[36624]: pgmap v3139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:56.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:56.652+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:41:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:56.652+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:41:56.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:41:56.652+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:41:57.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:57 smithi081 ceph-mon[31453]: pgmap v3140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:57.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:57 smithi153 ceph-mon[36569]: pgmap v3140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:57.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:57 smithi104 ceph-mon[36624]: pgmap v3140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:59.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:41:59 smithi081 ceph-mon[31453]: pgmap v3141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:59.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:41:59 smithi153 ceph-mon[36569]: pgmap v3141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:41:59.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:41:59 smithi104 ceph-mon[36624]: pgmap v3141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:01.602 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:01 smithi153 ceph-mon[36569]: pgmap v3142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:01.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:01 smithi081 ceph-mon[31453]: pgmap v3142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:01.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:01 smithi104 ceph-mon[36624]: pgmap v3142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:03.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:03 smithi081 ceph-mon[31453]: pgmap v3143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:03.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:03 smithi153 ceph-mon[36569]: pgmap v3143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:03.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:03 smithi104 ceph-mon[36624]: pgmap v3143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:05.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:05 smithi081 ceph-mon[31453]: pgmap v3144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:05.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:05 smithi153 ceph-mon[36569]: pgmap v3144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:05.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:05 smithi104 ceph-mon[36624]: pgmap v3144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:07.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:07 smithi081 ceph-mon[31453]: pgmap v3145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:07.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:07 smithi153 ceph-mon[36569]: pgmap v3145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:07.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:07 smithi104 ceph-mon[36624]: pgmap v3145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:09.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:09 smithi081 ceph-mon[31453]: pgmap v3146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:09.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:09 smithi153 ceph-mon[36569]: pgmap v3146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:09.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:09 smithi104 ceph-mon[36624]: pgmap v3146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:11.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:11 smithi081 ceph-mon[31453]: pgmap v3147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:11.664+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:42:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:11.664+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:42:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:11.664+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:42:11.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:11 smithi153 ceph-mon[36569]: pgmap v3147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:11.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:11 smithi104 ceph-mon[36624]: pgmap v3147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:13.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:13 smithi081 ceph-mon[31453]: pgmap v3148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:13.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:13 smithi153 ceph-mon[36569]: pgmap v3148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:13.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:13 smithi104 ceph-mon[36624]: pgmap v3148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:15.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:15 smithi081 ceph-mon[31453]: pgmap v3149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:15.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:15 smithi153 ceph-mon[36569]: pgmap v3149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:15.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:15 smithi104 ceph-mon[36624]: pgmap v3149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:17.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:17 smithi153 ceph-mon[36569]: pgmap v3150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:17.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:17 smithi104 ceph-mon[36624]: pgmap v3150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:17.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:17 smithi081 ceph-mon[31453]: pgmap v3150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:19.702 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:19 smithi081 ceph-mon[31453]: pgmap v3151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:19.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:19 smithi153 ceph-mon[36569]: pgmap v3151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:19.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:19 smithi104 ceph-mon[36624]: pgmap v3151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:21.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:21 smithi153 ceph-mon[36569]: pgmap v3152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:21.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:21 smithi104 ceph-mon[36624]: pgmap v3152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:21.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:21 smithi081 ceph-mon[31453]: pgmap v3152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:23.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:23 smithi153 ceph-mon[36569]: pgmap v3153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:23.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:23 smithi104 ceph-mon[36624]: pgmap v3153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:23.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:23 smithi081 ceph-mon[31453]: pgmap v3153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:25.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:25 smithi153 ceph-mon[36569]: pgmap v3154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:25.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:25 smithi104 ceph-mon[36624]: pgmap v3154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:25.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:25 smithi081 ceph-mon[31453]: pgmap v3154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:26.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:26.678+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:42:26.672 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:26.678+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:42:26.672 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:26.678+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:42:27.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:27 smithi153 ceph-mon[36569]: pgmap v3155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:27.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:27 smithi104 ceph-mon[36624]: pgmap v3155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:27.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:27 smithi081 ceph-mon[31453]: pgmap v3155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:28.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:42:28.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:42:28.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:42:29.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:29 smithi104 ceph-mon[36624]: pgmap v3156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:29.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:29 smithi153 ceph-mon[36569]: pgmap v3156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:29.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:29 smithi081 ceph-mon[31453]: pgmap v3156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:31.783 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:31 smithi104 ceph-mon[36624]: pgmap v3157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:31.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:31 smithi153 ceph-mon[36569]: pgmap v3157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:31.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:31 smithi081 ceph-mon[31453]: pgmap v3157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:33.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:33 smithi153 ceph-mon[36569]: pgmap v3158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:33.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:33.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:33.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:33 smithi104 ceph-mon[36624]: pgmap v3158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:33.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:33.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:33.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:33 smithi081 ceph-mon[31453]: pgmap v3158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:33.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:33.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:35.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:35 smithi153 ceph-mon[36569]: pgmap v3159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:35.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:35 smithi104 ceph-mon[36624]: pgmap v3159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:35.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:35 smithi081 ceph-mon[31453]: pgmap v3159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:36.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:36.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:36.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:42:36.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:42:36.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:36 smithi081 ceph-mon[31453]: pgmap v3160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:36.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:36 smithi081 ceph-mon[31453]: pgmap v3161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:36.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:37.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:37.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:37.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:42:37.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:42:37.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:36 smithi153 ceph-mon[36569]: pgmap v3160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:37.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:36 smithi153 ceph-mon[36569]: pgmap v3161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:37.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:37.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:37.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:37.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:42:37.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:42:37.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:36 smithi104 ceph-mon[36624]: pgmap v3160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:37.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:36 smithi104 ceph-mon[36624]: pgmap v3161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:37.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:42:39.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:38 smithi153 ceph-mon[36569]: pgmap v3162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:39.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:38 smithi104 ceph-mon[36624]: pgmap v3162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:39.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:38 smithi081 ceph-mon[31453]: pgmap v3162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:41.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:40 smithi153 ceph-mon[36569]: pgmap v3163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:41.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:40 smithi104 ceph-mon[36624]: pgmap v3163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:41.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:40 smithi081 ceph-mon[31453]: pgmap v3163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:41.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:41.694+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:42:41.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:41.694+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:42:41.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:41.694+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:42:43.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:42 smithi153 ceph-mon[36569]: pgmap v3164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:43.052 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:42 smithi104 ceph-mon[36624]: pgmap v3164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:43.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:42 smithi081 ceph-mon[31453]: pgmap v3164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:45.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:44 smithi153 ceph-mon[36569]: pgmap v3165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:45.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:44 smithi104 ceph-mon[36624]: pgmap v3165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:45.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:44 smithi081 ceph-mon[31453]: pgmap v3165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:47.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:46 smithi153 ceph-mon[36569]: pgmap v3166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:47.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:46 smithi104 ceph-mon[36624]: pgmap v3166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:47.076 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:46 smithi081 ceph-mon[31453]: pgmap v3166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:49.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:48 smithi153 ceph-mon[36569]: pgmap v3167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:49.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:48 smithi104 ceph-mon[36624]: pgmap v3167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:49.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:48 smithi081 ceph-mon[31453]: pgmap v3167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:51.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:50 smithi153 ceph-mon[36569]: pgmap v3168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:51.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:50 smithi104 ceph-mon[36624]: pgmap v3168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:51.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:50 smithi081 ceph-mon[31453]: pgmap v3168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:53.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:52 smithi153 ceph-mon[36569]: pgmap v3169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:53.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:52 smithi104 ceph-mon[36624]: pgmap v3169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:53.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:52 smithi081 ceph-mon[31453]: pgmap v3169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:55.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:54 smithi153 ceph-mon[36569]: pgmap v3170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:55.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:54 smithi104 ceph-mon[36624]: pgmap v3170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:55.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:54 smithi081 ceph-mon[31453]: pgmap v3170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:56.709+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:42:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:56.709+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:42:56.703 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:42:56.709+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:42:57.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:56 smithi153 ceph-mon[36569]: pgmap v3171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:57.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:56 smithi104 ceph-mon[36624]: pgmap v3171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:57.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:56 smithi081 ceph-mon[31453]: pgmap v3171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:59.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:42:58 smithi081 ceph-mon[31453]: pgmap v3172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:59.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:42:58 smithi153 ceph-mon[36569]: pgmap v3172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:42:59.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:42:58 smithi104 ceph-mon[36624]: pgmap v3172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:01.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:00 smithi153 ceph-mon[36569]: pgmap v3173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:01.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:00 smithi104 ceph-mon[36624]: pgmap v3173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:01.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:00 smithi081 ceph-mon[31453]: pgmap v3173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:03.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:02 smithi153 ceph-mon[36569]: pgmap v3174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:03.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:02 smithi104 ceph-mon[36624]: pgmap v3174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:03.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:02 smithi081 ceph-mon[31453]: pgmap v3174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:05.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:04 smithi153 ceph-mon[36569]: pgmap v3175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:05.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:04 smithi104 ceph-mon[36624]: pgmap v3175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:05.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:04 smithi081 ceph-mon[31453]: pgmap v3175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:07.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:06 smithi153 ceph-mon[36569]: pgmap v3176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:07.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:06 smithi104 ceph-mon[36624]: pgmap v3176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:07.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:06 smithi081 ceph-mon[31453]: pgmap v3176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:09.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:08 smithi153 ceph-mon[36569]: pgmap v3177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:09.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:08 smithi104 ceph-mon[36624]: pgmap v3177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:09.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:08 smithi081 ceph-mon[31453]: pgmap v3177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:11.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:10 smithi153 ceph-mon[36569]: pgmap v3178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:11.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:10 smithi104 ceph-mon[36624]: pgmap v3178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:11.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:10 smithi081 ceph-mon[31453]: pgmap v3178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:11.712+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:43:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:11.712+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:43:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:11.712+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:43:13.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:12 smithi153 ceph-mon[36569]: pgmap v3179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:13.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:12 smithi104 ceph-mon[36624]: pgmap v3179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:13.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:12 smithi081 ceph-mon[31453]: pgmap v3179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:13.554+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:43:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:13.554+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:43:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:13.554+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:43:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:13.555+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:43:15.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:14 smithi104 ceph-mon[36624]: pgmap v3180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:15.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:14 smithi081 ceph-mon[31453]: pgmap v3180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:15.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:14 smithi153 ceph-mon[36569]: pgmap v3180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:17.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:16 smithi104 ceph-mon[36624]: pgmap v3181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:17.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:16 smithi081 ceph-mon[31453]: pgmap v3181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:17.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:16 smithi153 ceph-mon[36569]: pgmap v3181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:19.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:18 smithi104 ceph-mon[36624]: pgmap v3182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:19.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:18 smithi081 ceph-mon[31453]: pgmap v3182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:19.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:18 smithi153 ceph-mon[36569]: pgmap v3182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:21.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:21 smithi153 ceph-mon[36569]: pgmap v3183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:21.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:21 smithi104 ceph-mon[36624]: pgmap v3183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:21.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:21 smithi081 ceph-mon[31453]: pgmap v3183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:23.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:23 smithi153 ceph-mon[36569]: pgmap v3184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:23.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:23 smithi104 ceph-mon[36624]: pgmap v3184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:23.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:23 smithi081 ceph-mon[31453]: pgmap v3184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:25.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:25 smithi153 ceph-mon[36569]: pgmap v3185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:25.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:25 smithi104 ceph-mon[36624]: pgmap v3185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:25.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:25 smithi081 ceph-mon[31453]: pgmap v3185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:26.728+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:43:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:26.728+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:43:26.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:26.728+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:43:27.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:27 smithi153 ceph-mon[36569]: pgmap v3186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:27.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:27 smithi104 ceph-mon[36624]: pgmap v3186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:27.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:27 smithi081 ceph-mon[31453]: pgmap v3186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:29.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:29 smithi153 ceph-mon[36569]: pgmap v3187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:29.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:29 smithi104 ceph-mon[36624]: pgmap v3187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:29.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:29 smithi081 ceph-mon[31453]: pgmap v3187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:31.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:31 smithi153 ceph-mon[36569]: pgmap v3188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:31.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:31 smithi104 ceph-mon[36624]: pgmap v3188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:31.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:31 smithi081 ceph-mon[31453]: pgmap v3188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:33.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:33 smithi153 ceph-mon[36569]: pgmap v3189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:33.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:33 smithi104 ceph-mon[36624]: pgmap v3189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:33.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:33 smithi081 ceph-mon[31453]: pgmap v3189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:35.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:35 smithi153 ceph-mon[36569]: pgmap v3190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:35.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:35 smithi104 ceph-mon[36624]: pgmap v3190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:35.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:35 smithi081 ceph-mon[31453]: pgmap v3190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:36.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:43:36.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:43:36.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:43:37.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:37 smithi153 ceph-mon[36569]: pgmap v3191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:37.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:37 smithi104 ceph-mon[36624]: pgmap v3191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:37.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:37 smithi081 ceph-mon[31453]: pgmap v3191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:39.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:39 smithi153 ceph-mon[36569]: pgmap v3192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:39.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:39 smithi104 ceph-mon[36624]: pgmap v3192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:39.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:39 smithi081 ceph-mon[31453]: pgmap v3192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:41.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:41 smithi153 ceph-mon[36569]: pgmap v3193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:41.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:43:41.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:43:41.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:41 smithi104 ceph-mon[36624]: pgmap v3193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:41.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:43:41.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:43:41.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:41 smithi081 ceph-mon[31453]: pgmap v3193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:41.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:43:41.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:43:41.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:43:41.733 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:41.740+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:43:41.733 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:41.740+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:43:41.733 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:41.740+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:43:42.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:42 smithi153 ceph-mon[36569]: pgmap v3194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:42.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:42 smithi153 ceph-mon[36569]: pgmap v3195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:42.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:42 smithi104 ceph-mon[36624]: pgmap v3194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:42.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:42 smithi104 ceph-mon[36624]: pgmap v3195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:42.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:42 smithi081 ceph-mon[31453]: pgmap v3194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:42.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:42 smithi081 ceph-mon[31453]: pgmap v3195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:44.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:44 smithi153 ceph-mon[36569]: pgmap v3196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:44.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:44 smithi104 ceph-mon[36624]: pgmap v3196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:44.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:44 smithi081 ceph-mon[31453]: pgmap v3196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:46.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:46 smithi153 ceph-mon[36569]: pgmap v3197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:46.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:46 smithi104 ceph-mon[36624]: pgmap v3197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:46.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:46 smithi081 ceph-mon[31453]: pgmap v3197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:48.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:48 smithi153 ceph-mon[36569]: pgmap v3198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:48.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:48 smithi104 ceph-mon[36624]: pgmap v3198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:48.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:48 smithi081 ceph-mon[31453]: pgmap v3198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:50.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:50 smithi153 ceph-mon[36569]: pgmap v3199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:50.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:50 smithi104 ceph-mon[36624]: pgmap v3199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:50.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:50 smithi081 ceph-mon[31453]: pgmap v3199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:52.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:52 smithi153 ceph-mon[36569]: pgmap v3200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:52.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:52 smithi104 ceph-mon[36624]: pgmap v3200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:52.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:52 smithi081 ceph-mon[31453]: pgmap v3200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:54.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:54 smithi153 ceph-mon[36569]: pgmap v3201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:54.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:54 smithi104 ceph-mon[36624]: pgmap v3201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:54.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:54 smithi081 ceph-mon[31453]: pgmap v3201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:56.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:56 smithi153 ceph-mon[36569]: pgmap v3202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:56.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:56 smithi104 ceph-mon[36624]: pgmap v3202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:56.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:56 smithi081 ceph-mon[31453]: pgmap v3202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:56.748+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:43:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:56.748+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:43:56.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:43:56.748+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:43:58.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:43:58 smithi153 ceph-mon[36569]: pgmap v3203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:58.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:43:58 smithi104 ceph-mon[36624]: pgmap v3203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:43:58.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:43:58 smithi081 ceph-mon[31453]: pgmap v3203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:00.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:00 smithi153 ceph-mon[36569]: pgmap v3204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:00.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:00 smithi104 ceph-mon[36624]: pgmap v3204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:00.591 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:00 smithi081 ceph-mon[31453]: pgmap v3204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:02.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:02 smithi153 ceph-mon[36569]: pgmap v3205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:02.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:02 smithi104 ceph-mon[36624]: pgmap v3205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:02.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:02 smithi081 ceph-mon[31453]: pgmap v3205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:04.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:04 smithi153 ceph-mon[36569]: pgmap v3206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:04.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:04 smithi104 ceph-mon[36624]: pgmap v3206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:04.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:04 smithi081 ceph-mon[31453]: pgmap v3206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:06.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:06 smithi153 ceph-mon[36569]: pgmap v3207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:06.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:06 smithi104 ceph-mon[36624]: pgmap v3207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:06.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:06 smithi081 ceph-mon[31453]: pgmap v3207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:08.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:08 smithi153 ceph-mon[36569]: pgmap v3208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:08.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:08 smithi104 ceph-mon[36624]: pgmap v3208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:08.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:08 smithi081 ceph-mon[31453]: pgmap v3208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:10.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:10 smithi153 ceph-mon[36569]: pgmap v3209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:10.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:10 smithi104 ceph-mon[36624]: pgmap v3209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:10.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:10 smithi081 ceph-mon[31453]: pgmap v3209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:11.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:11.760+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:44:11.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:11.760+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:44:11.754 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:11.761+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:44:12.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:12 smithi153 ceph-mon[36569]: pgmap v3210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:12.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:12 smithi104 ceph-mon[36624]: pgmap v3210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:12.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:12 smithi081 ceph-mon[31453]: pgmap v3210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:14.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:14 smithi153 ceph-mon[36569]: pgmap v3211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:14.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:14 smithi104 ceph-mon[36624]: pgmap v3211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:14.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:14 smithi081 ceph-mon[31453]: pgmap v3211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:16.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:16 smithi153 ceph-mon[36569]: pgmap v3212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:16.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:16 smithi104 ceph-mon[36624]: pgmap v3212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:16.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:16 smithi081 ceph-mon[31453]: pgmap v3212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:18.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:18 smithi104 ceph-mon[36624]: pgmap v3213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:18.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:18 smithi081 ceph-mon[31453]: pgmap v3213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:18.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:18 smithi153 ceph-mon[36569]: pgmap v3213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:20.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:20 smithi104 ceph-mon[36624]: pgmap v3214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:20.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:20 smithi081 ceph-mon[31453]: pgmap v3214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:20.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:20 smithi153 ceph-mon[36569]: pgmap v3214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:22.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:22 smithi104 ceph-mon[36624]: pgmap v3215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:22.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:22 smithi081 ceph-mon[31453]: pgmap v3215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:22 smithi153 ceph-mon[36569]: pgmap v3215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:24.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:24 smithi104 ceph-mon[36624]: pgmap v3216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:24.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:24 smithi081 ceph-mon[31453]: pgmap v3216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:24.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:24 smithi153 ceph-mon[36569]: pgmap v3216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:26.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:26 smithi104 ceph-mon[36624]: pgmap v3217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:26.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:26 smithi081 ceph-mon[31453]: pgmap v3217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:26.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:26.775+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:44:26.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:26.775+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:44:26.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:26.775+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:44:26.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:26 smithi153 ceph-mon[36569]: pgmap v3217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:28.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:28 smithi081 ceph-mon[31453]: pgmap v3218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:28.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:28 smithi153 ceph-mon[36569]: pgmap v3218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:28.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:28 smithi104 ceph-mon[36624]: pgmap v3218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:30.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:30 smithi081 ceph-mon[31453]: pgmap v3219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:30.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:30 smithi153 ceph-mon[36569]: pgmap v3219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:30.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:30 smithi104 ceph-mon[36624]: pgmap v3219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:32.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:32 smithi081 ceph-mon[31453]: pgmap v3220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:32.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:32 smithi153 ceph-mon[36569]: pgmap v3220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:32.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:32 smithi104 ceph-mon[36624]: pgmap v3220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:34.601 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:34 smithi153 ceph-mon[36569]: pgmap v3221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:34.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:34 smithi081 ceph-mon[31453]: pgmap v3221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:34.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:34 smithi104 ceph-mon[36624]: pgmap v3221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:36.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:36 smithi081 ceph-mon[31453]: pgmap v3222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:36.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:36 smithi153 ceph-mon[36569]: pgmap v3222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:36.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:36 smithi104 ceph-mon[36624]: pgmap v3222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:38.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:38 smithi081 ceph-mon[31453]: pgmap v3223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:38.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:38 smithi153 ceph-mon[36569]: pgmap v3223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:38.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:38 smithi104 ceph-mon[36624]: pgmap v3223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:40.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:40 smithi081 ceph-mon[31453]: pgmap v3224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:40.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:40 smithi153 ceph-mon[36569]: pgmap v3224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:40.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:40 smithi104 ceph-mon[36624]: pgmap v3224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:41.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:44:41.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:41.789+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:44:41.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:41.789+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:44:41.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:41.789+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:44:41.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:44:41.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:44:42.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:42 smithi081 ceph-mon[31453]: pgmap v3225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:42.725 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:42 smithi104 ceph-mon[36624]: pgmap v3225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:42.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:42 smithi153 ceph-mon[36569]: pgmap v3225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:43.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:44:43.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:44:43.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:43 smithi081 ceph-mon[31453]: pgmap v3226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:43.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:43 smithi081 ceph-mon[31453]: pgmap v3227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:43.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:44:43.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:44:43.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:44:43.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:43 smithi153 ceph-mon[36569]: pgmap v3226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:43.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:43 smithi153 ceph-mon[36569]: pgmap v3227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:43.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:44:43.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:44:43.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:44:43.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:43 smithi104 ceph-mon[36624]: pgmap v3226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:43.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:43 smithi104 ceph-mon[36624]: pgmap v3227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:43.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:44:45.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:45 smithi104 ceph-mon[36624]: pgmap v3228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:45.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:45 smithi081 ceph-mon[31453]: pgmap v3228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:46.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:45 smithi153 ceph-mon[36569]: pgmap v3228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:47.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:47 smithi104 ceph-mon[36624]: pgmap v3229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:47.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:47 smithi081 ceph-mon[31453]: pgmap v3229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:48.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:47 smithi153 ceph-mon[36569]: pgmap v3229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:49.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:49 smithi104 ceph-mon[36624]: pgmap v3230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:49.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:49 smithi081 ceph-mon[31453]: pgmap v3230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:50.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:49 smithi153 ceph-mon[36569]: pgmap v3230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:51.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:51 smithi104 ceph-mon[36624]: pgmap v3231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:51.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:51 smithi081 ceph-mon[31453]: pgmap v3231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:52.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:51 smithi153 ceph-mon[36569]: pgmap v3231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:53.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:53 smithi081 ceph-mon[31453]: pgmap v3232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:54.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:53 smithi153 ceph-mon[36569]: pgmap v3232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:54.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:53 smithi104 ceph-mon[36624]: pgmap v3232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:55.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:55 smithi081 ceph-mon[31453]: pgmap v3233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:56.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:55 smithi153 ceph-mon[36569]: pgmap v3233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:56.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:55 smithi104 ceph-mon[36624]: pgmap v3233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:56.802+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:44:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:56.803+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:44:56.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:44:56.803+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:44:57.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:57 smithi081 ceph-mon[31453]: pgmap v3234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:58.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:57 smithi153 ceph-mon[36569]: pgmap v3234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:58.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:57 smithi104 ceph-mon[36624]: pgmap v3234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:44:59.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:44:59 smithi081 ceph-mon[31453]: pgmap v3235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:00.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:44:59 smithi153 ceph-mon[36569]: pgmap v3235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:00.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:44:59 smithi104 ceph-mon[36624]: pgmap v3235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:01.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:01 smithi081 ceph-mon[31453]: pgmap v3236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:02.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:01 smithi153 ceph-mon[36569]: pgmap v3236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:02.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:01 smithi104 ceph-mon[36624]: pgmap v3236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:03.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:03 smithi081 ceph-mon[31453]: pgmap v3237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:04.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:03 smithi153 ceph-mon[36569]: pgmap v3237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:04.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:03 smithi104 ceph-mon[36624]: pgmap v3237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:05.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:05 smithi081 ceph-mon[31453]: pgmap v3238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:06.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:05 smithi153 ceph-mon[36569]: pgmap v3238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:06.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:05 smithi104 ceph-mon[36624]: pgmap v3238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:07.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:07 smithi081 ceph-mon[31453]: pgmap v3239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:08.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:07 smithi153 ceph-mon[36569]: pgmap v3239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:08.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:07 smithi104 ceph-mon[36624]: pgmap v3239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:09.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:09 smithi081 ceph-mon[31453]: pgmap v3240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:10.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:09 smithi153 ceph-mon[36569]: pgmap v3240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:10.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:09 smithi104 ceph-mon[36624]: pgmap v3240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:11.818+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:45:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:11.818+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:45:11.811 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:11.818+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:45:11.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:11 smithi081 ceph-mon[31453]: pgmap v3241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:12.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:11 smithi153 ceph-mon[36569]: pgmap v3241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:12.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:11 smithi104 ceph-mon[36624]: pgmap v3241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:13.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:13 smithi081 ceph-mon[31453]: pgmap v3242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:14.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:13 smithi153 ceph-mon[36569]: pgmap v3242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:14.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:13 smithi104 ceph-mon[36624]: pgmap v3242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:15.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:15 smithi081 ceph-mon[31453]: pgmap v3243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:16.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:15 smithi153 ceph-mon[36569]: pgmap v3243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:16.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:15 smithi104 ceph-mon[36624]: pgmap v3243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:17.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:17 smithi081 ceph-mon[31453]: pgmap v3244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:18.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:17 smithi153 ceph-mon[36569]: pgmap v3244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:18.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:17 smithi104 ceph-mon[36624]: pgmap v3244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:20.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:19 smithi153 ceph-mon[36569]: pgmap v3245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:20.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:19 smithi104 ceph-mon[36624]: pgmap v3245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:20.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:19 smithi081 ceph-mon[31453]: pgmap v3245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:21.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:20 smithi153 ceph-mon[36569]: pgmap v3246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:21.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:20 smithi104 ceph-mon[36624]: pgmap v3246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:21.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:20 smithi081 ceph-mon[31453]: pgmap v3246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:23.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:23 smithi081 ceph-mon[31453]: pgmap v3247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:24.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:23 smithi153 ceph-mon[36569]: pgmap v3247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:24.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:23 smithi104 ceph-mon[36624]: pgmap v3247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:25.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:25 smithi081 ceph-mon[31453]: pgmap v3248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:26.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:25 smithi153 ceph-mon[36569]: pgmap v3248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:26.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:25 smithi104 ceph-mon[36624]: pgmap v3248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:26.827+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:45:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:26.827+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:45:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:26.827+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:45:27.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:27 smithi081 ceph-mon[31453]: pgmap v3249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:28.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:27 smithi153 ceph-mon[36569]: pgmap v3249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:28.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:27 smithi104 ceph-mon[36624]: pgmap v3249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:29.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:29 smithi081 ceph-mon[31453]: pgmap v3250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:30.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:29 smithi153 ceph-mon[36569]: pgmap v3250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:30.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:29 smithi104 ceph-mon[36624]: pgmap v3250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:31.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:31 smithi081 ceph-mon[31453]: pgmap v3251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:32.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:31 smithi153 ceph-mon[36569]: pgmap v3251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:32.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:31 smithi104 ceph-mon[36624]: pgmap v3251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:33.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:33 smithi081 ceph-mon[31453]: pgmap v3252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:34.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:33 smithi153 ceph-mon[36569]: pgmap v3252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:34.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:33 smithi104 ceph-mon[36624]: pgmap v3252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:35.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:35 smithi081 ceph-mon[31453]: pgmap v3253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:36.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:35 smithi153 ceph-mon[36569]: pgmap v3253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:36.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:35 smithi104 ceph-mon[36624]: pgmap v3253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:37.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:37 smithi081 ceph-mon[31453]: pgmap v3254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:38.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:37 smithi153 ceph-mon[36569]: pgmap v3254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:38.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:37 smithi104 ceph-mon[36624]: pgmap v3254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:39.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:39 smithi081 ceph-mon[31453]: pgmap v3255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:40.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:39 smithi153 ceph-mon[36569]: pgmap v3255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:40.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:39 smithi104 ceph-mon[36624]: pgmap v3255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:41.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:41.834+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:45:41.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:41.834+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:45:41.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:41.834+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:45:41.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:41 smithi081 ceph-mon[31453]: pgmap v3256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:42.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:41 smithi153 ceph-mon[36569]: pgmap v3256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:42.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:41 smithi104 ceph-mon[36624]: pgmap v3256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:43.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: pgmap v3257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: pgmap v3257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:44.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:45:44.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: pgmap v3257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:44.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:45:44.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:45:44.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:45:45.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:45:45.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:45:45.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:45:45.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:45:45.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:45:45.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:45:45.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:45.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:45.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:45.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:45:45.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:45:45.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:45:45.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:45:46.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:45 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:45:46.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:45 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:45:46.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:45 smithi153 ceph-mon[36569]: pgmap v3258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:46.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:45 smithi153 ceph-mon[36569]: pgmap v3259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:46.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:45 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:45:46.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:45 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:45:46.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:45 smithi104 ceph-mon[36624]: pgmap v3258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:46.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:45 smithi104 ceph-mon[36624]: pgmap v3259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:46.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:45 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:45:46.197 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:45 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:45:46.197 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:45 smithi081 ceph-mon[31453]: pgmap v3258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:46.197 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:45 smithi081 ceph-mon[31453]: pgmap v3259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:48.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:47 smithi153 ceph-mon[36569]: pgmap v3260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:48.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:47 smithi104 ceph-mon[36624]: pgmap v3260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:48.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:47 smithi081 ceph-mon[31453]: pgmap v3260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:49.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:48 smithi153 ceph-mon[36569]: pgmap v3261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:49.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:48 smithi104 ceph-mon[36624]: pgmap v3261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:49.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:48 smithi081 ceph-mon[31453]: pgmap v3261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:51.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:51 smithi081 ceph-mon[31453]: pgmap v3262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:51.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:51 smithi153 ceph-mon[36569]: pgmap v3262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:51.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:51 smithi104 ceph-mon[36624]: pgmap v3262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:53.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:53 smithi081 ceph-mon[31453]: pgmap v3263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:53.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:53 smithi153 ceph-mon[36569]: pgmap v3263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:53.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:53 smithi104 ceph-mon[36624]: pgmap v3263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:55.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:55 smithi081 ceph-mon[31453]: pgmap v3264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:55.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:55 smithi153 ceph-mon[36569]: pgmap v3264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:55.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:55 smithi104 ceph-mon[36624]: pgmap v3264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:56.845+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:45:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:56.845+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:45:56.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:45:56.845+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:45:57.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:57 smithi081 ceph-mon[31453]: pgmap v3265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:57.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:57 smithi153 ceph-mon[36569]: pgmap v3265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:57.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:57 smithi104 ceph-mon[36624]: pgmap v3265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:59.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:45:59 smithi081 ceph-mon[31453]: pgmap v3266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:59.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:45:59 smithi153 ceph-mon[36569]: pgmap v3266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:45:59.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:45:59 smithi104 ceph-mon[36624]: pgmap v3266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:01.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:01 smithi081 ceph-mon[31453]: pgmap v3267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:01.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:01 smithi153 ceph-mon[36569]: pgmap v3267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:01.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:01 smithi104 ceph-mon[36624]: pgmap v3267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:03.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:03 smithi081 ceph-mon[31453]: pgmap v3268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:03.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:03 smithi153 ceph-mon[36569]: pgmap v3268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:03.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:03 smithi104 ceph-mon[36624]: pgmap v3268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:05.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:05 smithi081 ceph-mon[31453]: pgmap v3269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:05.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:05 smithi153 ceph-mon[36569]: pgmap v3269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:05.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:05 smithi104 ceph-mon[36624]: pgmap v3269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:07.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:07 smithi081 ceph-mon[31453]: pgmap v3270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:07.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:07 smithi153 ceph-mon[36569]: pgmap v3270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:07.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:07 smithi104 ceph-mon[36624]: pgmap v3270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:09.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:09 smithi081 ceph-mon[31453]: pgmap v3271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:09.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:09 smithi153 ceph-mon[36569]: pgmap v3271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:09.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:09 smithi104 ceph-mon[36624]: pgmap v3271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:11.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:11 smithi081 ceph-mon[31453]: pgmap v3272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:11.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:11 smithi153 ceph-mon[36569]: pgmap v3272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:11.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:11 smithi104 ceph-mon[36624]: pgmap v3272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:11.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:11.860+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:46:11.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:11.860+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:46:11.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:11.860+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:46:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.101+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T22:46:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.101+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T22:46:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.101+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f40bd040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T22:46:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.101+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-12T22:46:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.101+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-12T22:46:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.101+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T22:46:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.101+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T22:46:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.101+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T22:46:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.102+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T22:46:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:12.104+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f16d40065f0 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T22:46:13.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:13 smithi153 ceph-mon[36569]: pgmap v3273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:13.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:13 smithi104 ceph-mon[36624]: pgmap v3273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:13.584+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:46:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:13.584+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:46:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:13.584+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:46:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:13.584+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:46:13.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:13 smithi081 ceph-mon[31453]: pgmap v3273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:15.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:15 smithi153 ceph-mon[36569]: pgmap v3274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:15.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:15 smithi104 ceph-mon[36624]: pgmap v3274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:15.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:15 smithi081 ceph-mon[31453]: pgmap v3274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:17.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:17 smithi153 ceph-mon[36569]: pgmap v3275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:17.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:17 smithi104 ceph-mon[36624]: pgmap v3275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:17.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:17 smithi081 ceph-mon[31453]: pgmap v3275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:19.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:19 smithi153 ceph-mon[36569]: pgmap v3276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:19.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:19 smithi104 ceph-mon[36624]: pgmap v3276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:19.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:19 smithi081 ceph-mon[31453]: pgmap v3276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:21.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:21 smithi153 ceph-mon[36569]: pgmap v3277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:21.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:21 smithi104 ceph-mon[36624]: pgmap v3277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:21.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:21 smithi081 ceph-mon[31453]: pgmap v3277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:23.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:23 smithi081 ceph-mon[31453]: pgmap v3278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:23.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:23 smithi153 ceph-mon[36569]: pgmap v3278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:23.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:23 smithi104 ceph-mon[36624]: pgmap v3278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:25.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:25 smithi153 ceph-mon[36569]: pgmap v3279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:25.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:25 smithi104 ceph-mon[36624]: pgmap v3279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:25.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:25 smithi081 ceph-mon[31453]: pgmap v3279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:26.862+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:46:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:26.862+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:46:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:26.862+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:46:27.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:27 smithi153 ceph-mon[36569]: pgmap v3280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:27.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:27 smithi104 ceph-mon[36624]: pgmap v3280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:27.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:27 smithi081 ceph-mon[31453]: pgmap v3280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:29.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:29 smithi153 ceph-mon[36569]: pgmap v3281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:29.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:29 smithi104 ceph-mon[36624]: pgmap v3281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:29.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:29 smithi081 ceph-mon[31453]: pgmap v3281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:31.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:31 smithi153 ceph-mon[36569]: pgmap v3282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:31.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:31 smithi104 ceph-mon[36624]: pgmap v3282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:31.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:31 smithi081 ceph-mon[31453]: pgmap v3282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:33.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:33 smithi153 ceph-mon[36569]: pgmap v3283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:33.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:33 smithi104 ceph-mon[36624]: pgmap v3283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:33.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:33 smithi081 ceph-mon[31453]: pgmap v3283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:35.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:35 smithi153 ceph-mon[36569]: pgmap v3284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:35.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:35 smithi104 ceph-mon[36624]: pgmap v3284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:35.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:35 smithi081 ceph-mon[31453]: pgmap v3284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:37.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:37 smithi153 ceph-mon[36569]: pgmap v3285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:37.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:37 smithi104 ceph-mon[36624]: pgmap v3285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:37.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:37 smithi081 ceph-mon[31453]: pgmap v3285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:39.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:39 smithi081 ceph-mon[31453]: pgmap v3286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:39.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:39 smithi153 ceph-mon[36569]: pgmap v3286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:39.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:39 smithi104 ceph-mon[36624]: pgmap v3286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:41.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:41 smithi153 ceph-mon[36569]: pgmap v3287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:41.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:41 smithi104 ceph-mon[36624]: pgmap v3287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:41.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:41 smithi081 ceph-mon[31453]: pgmap v3287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:41.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:41.876+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:46:41.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:41.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:46:41.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:41.876+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:46:43.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:43 smithi153 ceph-mon[36569]: pgmap v3288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:43.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:43 smithi104 ceph-mon[36624]: pgmap v3288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:43.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:43 smithi081 ceph-mon[31453]: pgmap v3288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:45.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:45 smithi153 ceph-mon[36569]: pgmap v3289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:45.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:46:45.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:45 smithi104 ceph-mon[36624]: pgmap v3289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:45.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:46:45.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:45 smithi081 ceph-mon[31453]: pgmap v3289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:45.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:46:46.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:46:46.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:46:46.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:46 smithi153 ceph-mon[36569]: pgmap v3290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:46.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:46 smithi153 ceph-mon[36569]: pgmap v3291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:46.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:46:46.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:46:46.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:46:46.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:46 smithi104 ceph-mon[36624]: pgmap v3290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:46.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:46 smithi104 ceph-mon[36624]: pgmap v3291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:46.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:46:46.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:46:46.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:46:46.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:46 smithi081 ceph-mon[31453]: pgmap v3290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:46.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:46 smithi081 ceph-mon[31453]: pgmap v3291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:46.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:46:48.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:48 smithi081 ceph-mon[31453]: pgmap v3292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:49.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:48 smithi153 ceph-mon[36569]: pgmap v3292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:49.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:48 smithi104 ceph-mon[36624]: pgmap v3292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:50.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:50 smithi081 ceph-mon[31453]: pgmap v3293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:51.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:50 smithi153 ceph-mon[36569]: pgmap v3293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:51.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:50 smithi104 ceph-mon[36624]: pgmap v3293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:52.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:52 smithi081 ceph-mon[31453]: pgmap v3294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:53.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:52 smithi153 ceph-mon[36569]: pgmap v3294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:53.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:52 smithi104 ceph-mon[36624]: pgmap v3294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:54.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:54 smithi081 ceph-mon[31453]: pgmap v3295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:55.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:54 smithi153 ceph-mon[36569]: pgmap v3295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:55.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:54 smithi104 ceph-mon[36624]: pgmap v3295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:56.882+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:46:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:56.882+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:46:56.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:56.882+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:46:56.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:56 smithi081 ceph-mon[31453]: pgmap v3296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:57.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:56 smithi153 ceph-mon[36569]: pgmap v3296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:57.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:56 smithi104 ceph-mon[36624]: pgmap v3296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:58.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:58.592+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:46:58.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:58.592+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:46:58.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:58.592+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:46:58.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:46:58.592+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:46:58.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:46:58 smithi081 ceph-mon[31453]: pgmap v3297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:59.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:46:58 smithi153 ceph-mon[36569]: pgmap v3297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:46:59.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:46:58 smithi104 ceph-mon[36624]: pgmap v3297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:00.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:00 smithi081 ceph-mon[31453]: pgmap v3298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:01.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:00 smithi153 ceph-mon[36569]: pgmap v3298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:01.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:00 smithi104 ceph-mon[36624]: pgmap v3298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:02.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:02 smithi081 ceph-mon[31453]: pgmap v3299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:03.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:02 smithi153 ceph-mon[36569]: pgmap v3299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:03.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:02 smithi104 ceph-mon[36624]: pgmap v3299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:04.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:04 smithi081 ceph-mon[31453]: pgmap v3300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:05.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:04 smithi153 ceph-mon[36569]: pgmap v3300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:05.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:04 smithi104 ceph-mon[36624]: pgmap v3300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:06.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:06 smithi081 ceph-mon[31453]: pgmap v3301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:07.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:06 smithi153 ceph-mon[36569]: pgmap v3301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:07.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:06 smithi104 ceph-mon[36624]: pgmap v3301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:08.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:08 smithi081 ceph-mon[31453]: pgmap v3302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:09.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:08 smithi153 ceph-mon[36569]: pgmap v3302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:09.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:08 smithi104 ceph-mon[36624]: pgmap v3302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:11.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:10 smithi153 ceph-mon[36569]: pgmap v3303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:11.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:10 smithi104 ceph-mon[36624]: pgmap v3303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:11.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:10 smithi081 ceph-mon[31453]: pgmap v3303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:11.897+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:47:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:11.897+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:47:11.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:11.897+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:47:13.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:12 smithi153 ceph-mon[36569]: pgmap v3304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:13.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:12 smithi104 ceph-mon[36624]: pgmap v3304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:13.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:12 smithi081 ceph-mon[31453]: pgmap v3304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:15.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:14 smithi153 ceph-mon[36569]: pgmap v3305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:15.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:14 smithi104 ceph-mon[36624]: pgmap v3305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:15.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:14 smithi081 ceph-mon[31453]: pgmap v3305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:17.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:16 smithi153 ceph-mon[36569]: pgmap v3306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:17.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:16 smithi104 ceph-mon[36624]: pgmap v3306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:17.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:16 smithi081 ceph-mon[31453]: pgmap v3306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:19.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:18 smithi153 ceph-mon[36569]: pgmap v3307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:19.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:18 smithi104 ceph-mon[36624]: pgmap v3307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:19.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:18 smithi081 ceph-mon[31453]: pgmap v3307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:20.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:19 smithi153 ceph-mon[36569]: pgmap v3308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:20.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:19 smithi104 ceph-mon[36624]: pgmap v3308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:20.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:19 smithi081 ceph-mon[31453]: pgmap v3308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:22.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:22 smithi081 ceph-mon[31453]: pgmap v3309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:23.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:22 smithi153 ceph-mon[36569]: pgmap v3309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:23.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:22 smithi104 ceph-mon[36624]: pgmap v3309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:24.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:24 smithi081 ceph-mon[31453]: pgmap v3310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:25.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:24 smithi153 ceph-mon[36569]: pgmap v3310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:25.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:24 smithi104 ceph-mon[36624]: pgmap v3310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:26.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:26 smithi081 ceph-mon[31453]: pgmap v3311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:26.913+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:47:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:26.913+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:47:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:26.913+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:47:27.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:26 smithi153 ceph-mon[36569]: pgmap v3311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:27.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:26 smithi104 ceph-mon[36624]: pgmap v3311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:28.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:28 smithi081 ceph-mon[31453]: pgmap v3312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:29.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:28 smithi153 ceph-mon[36569]: pgmap v3312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:29.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:28 smithi104 ceph-mon[36624]: pgmap v3312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:30.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:30 smithi081 ceph-mon[31453]: pgmap v3313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:31.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:30 smithi153 ceph-mon[36569]: pgmap v3313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:31.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:30 smithi104 ceph-mon[36624]: pgmap v3313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:32.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:32 smithi081 ceph-mon[31453]: pgmap v3314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:33.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:32 smithi153 ceph-mon[36569]: pgmap v3314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:33.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:32 smithi104 ceph-mon[36624]: pgmap v3314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:34.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:34 smithi081 ceph-mon[31453]: pgmap v3315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:35.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:34 smithi153 ceph-mon[36569]: pgmap v3315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:35.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:34 smithi104 ceph-mon[36624]: pgmap v3315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:37.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:36 smithi153 ceph-mon[36569]: pgmap v3316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:37.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:36 smithi104 ceph-mon[36624]: pgmap v3316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:37.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:36 smithi081 ceph-mon[31453]: pgmap v3316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:39.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:38 smithi153 ceph-mon[36569]: pgmap v3317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:39.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:38 smithi104 ceph-mon[36624]: pgmap v3317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:39.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:38 smithi081 ceph-mon[31453]: pgmap v3317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:41.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:40 smithi153 ceph-mon[36569]: pgmap v3318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:41.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:40 smithi104 ceph-mon[36624]: pgmap v3318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:41.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:40 smithi081 ceph-mon[31453]: pgmap v3318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:41.920 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:41.929+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:47:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:41.929+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:47:41.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:41.929+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:47:43.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:42 smithi153 ceph-mon[36569]: pgmap v3319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:43.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:42 smithi104 ceph-mon[36624]: pgmap v3319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:43.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:42 smithi081 ceph-mon[31453]: pgmap v3319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:45.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:44 smithi153 ceph-mon[36569]: pgmap v3320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:45.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:44 smithi104 ceph-mon[36624]: pgmap v3320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:45.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:44 smithi081 ceph-mon[31453]: pgmap v3320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:47.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:46 smithi153 ceph-mon[36569]: pgmap v3321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:47.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:47:47.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:46 smithi104 ceph-mon[36624]: pgmap v3321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:47.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:47:47.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:46 smithi081 ceph-mon[31453]: pgmap v3321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:47.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:47:48.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:47:48.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:47:48.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:47:48.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:47:48.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:47:48.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:47:48.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:47:48.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:47:48.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:47:49.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:48 smithi153 ceph-mon[36569]: pgmap v3322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:49.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:48 smithi153 ceph-mon[36569]: pgmap v3323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:49.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:48 smithi104 ceph-mon[36624]: pgmap v3322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:49.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:48 smithi104 ceph-mon[36624]: pgmap v3323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:49.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:48 smithi081 ceph-mon[31453]: pgmap v3322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:49.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:48 smithi081 ceph-mon[31453]: pgmap v3323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:50.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:49 smithi153 ceph-mon[36569]: pgmap v3324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:50.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:49 smithi104 ceph-mon[36624]: pgmap v3324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:50.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:49 smithi081 ceph-mon[31453]: pgmap v3324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:52.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:52 smithi153 ceph-mon[36569]: pgmap v3325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:52.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:52 smithi104 ceph-mon[36624]: pgmap v3325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:52.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:52 smithi081 ceph-mon[31453]: pgmap v3325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:54.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:54 smithi153 ceph-mon[36569]: pgmap v3326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:54.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:54 smithi104 ceph-mon[36624]: pgmap v3326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:54.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:54 smithi081 ceph-mon[31453]: pgmap v3326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:56.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:56 smithi153 ceph-mon[36569]: pgmap v3327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:56.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:56 smithi104 ceph-mon[36624]: pgmap v3327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:56.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:56 smithi081 ceph-mon[31453]: pgmap v3327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:56.937+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:47:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:56.937+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:47:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:47:56.937+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:47:58.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:47:58 smithi153 ceph-mon[36569]: pgmap v3328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:58.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:47:58 smithi104 ceph-mon[36624]: pgmap v3328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:47:58.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:47:58 smithi081 ceph-mon[31453]: pgmap v3328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:00.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:00 smithi153 ceph-mon[36569]: pgmap v3329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:00.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:00 smithi104 ceph-mon[36624]: pgmap v3329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:00.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:00 smithi081 ceph-mon[31453]: pgmap v3329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:02.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:02 smithi153 ceph-mon[36569]: pgmap v3330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:02.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:02 smithi104 ceph-mon[36624]: pgmap v3330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:02.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:02 smithi081 ceph-mon[31453]: pgmap v3330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:04.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:04 smithi153 ceph-mon[36569]: pgmap v3331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:04.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:04 smithi104 ceph-mon[36624]: pgmap v3331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:04.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:04 smithi081 ceph-mon[31453]: pgmap v3331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:06.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:06 smithi153 ceph-mon[36569]: pgmap v3332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:06.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:06 smithi104 ceph-mon[36624]: pgmap v3332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:06.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:06 smithi081 ceph-mon[31453]: pgmap v3332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:08.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:08 smithi153 ceph-mon[36569]: pgmap v3333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:08.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:08 smithi104 ceph-mon[36624]: pgmap v3333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:08.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:08 smithi081 ceph-mon[31453]: pgmap v3333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:10.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:10 smithi153 ceph-mon[36569]: pgmap v3334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:10.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:10 smithi104 ceph-mon[36624]: pgmap v3334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:10.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:10 smithi081 ceph-mon[31453]: pgmap v3334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:11.943+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:11.943+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:48:11.936 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:11.943+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:48:12.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:12 smithi153 ceph-mon[36569]: pgmap v3335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:12.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:12 smithi104 ceph-mon[36624]: pgmap v3335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:12.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:12 smithi081 ceph-mon[31453]: pgmap v3335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:14.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:14 smithi153 ceph-mon[36569]: pgmap v3336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:14.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:14 smithi104 ceph-mon[36624]: pgmap v3336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:14.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:14 smithi081 ceph-mon[31453]: pgmap v3336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:16.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:16 smithi153 ceph-mon[36569]: pgmap v3337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:16.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:16 smithi104 ceph-mon[36624]: pgmap v3337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:16.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:16 smithi081 ceph-mon[31453]: pgmap v3337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:18.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:18 smithi153 ceph-mon[36569]: pgmap v3338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:18.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:18 smithi104 ceph-mon[36624]: pgmap v3338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:18.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:18 smithi081 ceph-mon[31453]: pgmap v3338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:20.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:20 smithi153 ceph-mon[36569]: pgmap v3339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:20.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:20 smithi104 ceph-mon[36624]: pgmap v3339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:20.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:20 smithi081 ceph-mon[31453]: pgmap v3339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:22.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:22 smithi153 ceph-mon[36569]: pgmap v3340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:22.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:22 smithi104 ceph-mon[36624]: pgmap v3340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:22.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:22 smithi081 ceph-mon[31453]: pgmap v3340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:24.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:24 smithi104 ceph-mon[36624]: pgmap v3341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:24.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:24 smithi081 ceph-mon[31453]: pgmap v3341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:24.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:24 smithi153 ceph-mon[36569]: pgmap v3341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:26.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:26 smithi104 ceph-mon[36624]: pgmap v3342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:26.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:26 smithi081 ceph-mon[31453]: pgmap v3342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:26.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:26 smithi153 ceph-mon[36569]: pgmap v3342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:26.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:48:26.951 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:26.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:48:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:26.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:48:28.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:28 smithi104 ceph-mon[36624]: pgmap v3343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:28.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:28 smithi081 ceph-mon[31453]: pgmap v3343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:28.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:28 smithi153 ceph-mon[36569]: pgmap v3343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:30.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:30 smithi104 ceph-mon[36624]: pgmap v3344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:30.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:30 smithi081 ceph-mon[31453]: pgmap v3344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:30.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:30 smithi153 ceph-mon[36569]: pgmap v3344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:32.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:32 smithi104 ceph-mon[36624]: pgmap v3345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:32.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:32 smithi081 ceph-mon[31453]: pgmap v3345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:32.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:32 smithi153 ceph-mon[36569]: pgmap v3345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:34.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:34 smithi081 ceph-mon[31453]: pgmap v3346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:34.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:34 smithi153 ceph-mon[36569]: pgmap v3346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:34.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:34 smithi104 ceph-mon[36624]: pgmap v3346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:36.600 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:36 smithi153 ceph-mon[36569]: pgmap v3347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:36.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:36 smithi081 ceph-mon[31453]: pgmap v3347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:36.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:36 smithi104 ceph-mon[36624]: pgmap v3347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:38.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:38 smithi081 ceph-mon[31453]: pgmap v3348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:38.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:38 smithi153 ceph-mon[36569]: pgmap v3348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:38.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:38 smithi104 ceph-mon[36624]: pgmap v3348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:40.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:40 smithi081 ceph-mon[31453]: pgmap v3349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:40.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:40 smithi153 ceph-mon[36569]: pgmap v3349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:40.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:40 smithi104 ceph-mon[36624]: pgmap v3349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:41.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:41.974+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:48:41.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:41.974+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:48:41.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:41.974+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:48:42.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:42 smithi081 ceph-mon[31453]: pgmap v3350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:42.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:42 smithi153 ceph-mon[36569]: pgmap v3350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:42.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:42 smithi104 ceph-mon[36624]: pgmap v3350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:44.625 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:44 smithi104 ceph-mon[36624]: pgmap v3351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:44.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:44 smithi081 ceph-mon[31453]: pgmap v3351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:44.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:44 smithi153 ceph-mon[36569]: pgmap v3351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:46.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:46 smithi081 ceph-mon[31453]: pgmap v3352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:46.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:46 smithi153 ceph-mon[36569]: pgmap v3352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:46.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:46 smithi104 ceph-mon[36624]: pgmap v3352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:48.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:48 smithi081 ceph-mon[31453]: pgmap v3353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:48.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:48:48.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:48 smithi153 ceph-mon[36569]: pgmap v3353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:48.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:48:48.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:48 smithi104 ceph-mon[36624]: pgmap v3353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:48.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:48:49.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:48:49.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:48:49.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:49 smithi081 ceph-mon[31453]: pgmap v3354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:49.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:49 smithi081 ceph-mon[31453]: pgmap v3355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:49.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:48:49.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:48:49.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:48:49.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:49 smithi153 ceph-mon[36569]: pgmap v3354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:49.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:49 smithi153 ceph-mon[36569]: pgmap v3355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:49.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:48:49.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:48:49.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:48:49.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:49 smithi104 ceph-mon[36624]: pgmap v3354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:49.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:49 smithi104 ceph-mon[36624]: pgmap v3355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:49.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:48:51.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:50 smithi153 ceph-mon[36569]: pgmap v3356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:51.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:50 smithi104 ceph-mon[36624]: pgmap v3356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:51.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:50 smithi081 ceph-mon[31453]: pgmap v3356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:53.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:52 smithi153 ceph-mon[36569]: pgmap v3357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:53.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:52 smithi104 ceph-mon[36624]: pgmap v3357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:53.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:52 smithi081 ceph-mon[31453]: pgmap v3357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:55.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:54 smithi153 ceph-mon[36569]: pgmap v3358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:55.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:54 smithi104 ceph-mon[36624]: pgmap v3358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:55.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:54 smithi081 ceph-mon[31453]: pgmap v3358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:56.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:56.987+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:48:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:56.987+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:48:56.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:48:56.987+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:48:57.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:56 smithi153 ceph-mon[36569]: pgmap v3359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:57.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:56 smithi104 ceph-mon[36624]: pgmap v3359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:57.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:56 smithi081 ceph-mon[31453]: pgmap v3359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:59.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:48:58 smithi153 ceph-mon[36569]: pgmap v3360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:59.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:48:58 smithi104 ceph-mon[36624]: pgmap v3360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:48:59.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:48:58 smithi081 ceph-mon[31453]: pgmap v3360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:01.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:00 smithi153 ceph-mon[36569]: pgmap v3361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:01.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:00 smithi104 ceph-mon[36624]: pgmap v3361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:01.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:00 smithi081 ceph-mon[31453]: pgmap v3361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:03.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:02 smithi153 ceph-mon[36569]: pgmap v3362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:03.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:02 smithi104 ceph-mon[36624]: pgmap v3362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:03.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:02 smithi081 ceph-mon[31453]: pgmap v3362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:05.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:04 smithi153 ceph-mon[36569]: pgmap v3363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:05.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:04 smithi104 ceph-mon[36624]: pgmap v3363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:05.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:04 smithi081 ceph-mon[31453]: pgmap v3363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:07.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:06 smithi153 ceph-mon[36569]: pgmap v3364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:07.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:06 smithi104 ceph-mon[36624]: pgmap v3364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:07.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:06 smithi081 ceph-mon[31453]: pgmap v3364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:09.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:08 smithi153 ceph-mon[36569]: pgmap v3365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:09.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:08 smithi104 ceph-mon[36624]: pgmap v3365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:09.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:08 smithi081 ceph-mon[31453]: pgmap v3365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:11.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:10 smithi153 ceph-mon[36569]: pgmap v3366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:11.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:10 smithi104 ceph-mon[36624]: pgmap v3366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:11.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:10 smithi081 ceph-mon[31453]: pgmap v3366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:11.989+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:49:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:11.989+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:49:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:11.989+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:49:13.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:12 smithi153 ceph-mon[36569]: pgmap v3367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:13.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:12 smithi104 ceph-mon[36624]: pgmap v3367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:13.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:12 smithi081 ceph-mon[31453]: pgmap v3367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:13.614+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:49:13.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:13.614+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:49:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:13.614+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:49:13.606 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:13.614+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:49:15.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:14 smithi153 ceph-mon[36569]: pgmap v3368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:15.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:14 smithi104 ceph-mon[36624]: pgmap v3368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:15.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:14 smithi081 ceph-mon[31453]: pgmap v3368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:17.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:16 smithi153 ceph-mon[36569]: pgmap v3369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:17.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:16 smithi104 ceph-mon[36624]: pgmap v3369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:17.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:16 smithi081 ceph-mon[31453]: pgmap v3369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:19.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:18 smithi153 ceph-mon[36569]: pgmap v3370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:19.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:18 smithi104 ceph-mon[36624]: pgmap v3370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:19.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:18 smithi081 ceph-mon[31453]: pgmap v3370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:21.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:20 smithi153 ceph-mon[36569]: pgmap v3371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:21.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:20 smithi104 ceph-mon[36624]: pgmap v3371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:21.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:20 smithi081 ceph-mon[31453]: pgmap v3371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:23.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:22 smithi153 ceph-mon[36569]: pgmap v3372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:23.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:22 smithi104 ceph-mon[36624]: pgmap v3372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:23.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:22 smithi081 ceph-mon[31453]: pgmap v3372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:25.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:24 smithi153 ceph-mon[36569]: pgmap v3373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:25.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:24 smithi104 ceph-mon[36624]: pgmap v3373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:25.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:24 smithi081 ceph-mon[31453]: pgmap v3373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:27.000+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:49:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:27.000+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:49:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:27.000+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:49:27.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:26 smithi153 ceph-mon[36569]: pgmap v3374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:27.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:26 smithi104 ceph-mon[36624]: pgmap v3374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:27.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:26 smithi081 ceph-mon[31453]: pgmap v3374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:29.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:28 smithi153 ceph-mon[36569]: pgmap v3375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:29.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:28 smithi104 ceph-mon[36624]: pgmap v3375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:29.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:28 smithi081 ceph-mon[31453]: pgmap v3375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:31.006 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:30 smithi081 ceph-mon[31453]: pgmap v3376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:31.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:30 smithi153 ceph-mon[36569]: pgmap v3376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:31.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:30 smithi104 ceph-mon[36624]: pgmap v3376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:33.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:32 smithi153 ceph-mon[36569]: pgmap v3377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:33.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:32 smithi104 ceph-mon[36624]: pgmap v3377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:33.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:32 smithi081 ceph-mon[31453]: pgmap v3377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:35.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:34 smithi153 ceph-mon[36569]: pgmap v3378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:35.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:34 smithi104 ceph-mon[36624]: pgmap v3378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:35.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:34 smithi081 ceph-mon[31453]: pgmap v3378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:37.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:36 smithi153 ceph-mon[36569]: pgmap v3379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:37.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:36 smithi104 ceph-mon[36624]: pgmap v3379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:37.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:36 smithi081 ceph-mon[31453]: pgmap v3379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:39.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:38 smithi153 ceph-mon[36569]: pgmap v3380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:39.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:38 smithi104 ceph-mon[36624]: pgmap v3380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:39.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:38 smithi081 ceph-mon[31453]: pgmap v3380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:41.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:40 smithi153 ceph-mon[36569]: pgmap v3381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:41.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:40 smithi104 ceph-mon[36624]: pgmap v3381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:41.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:40 smithi081 ceph-mon[31453]: pgmap v3381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:42.002+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:49:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:42.002+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:49:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:42.002+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:49:43.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:42 smithi153 ceph-mon[36569]: pgmap v3382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:43.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:42 smithi104 ceph-mon[36624]: pgmap v3382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:43.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:42 smithi081 ceph-mon[31453]: pgmap v3382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:45.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:44 smithi153 ceph-mon[36569]: pgmap v3383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:45.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:44 smithi104 ceph-mon[36624]: pgmap v3383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:45.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:44 smithi081 ceph-mon[31453]: pgmap v3383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:47.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:46 smithi153 ceph-mon[36569]: pgmap v3384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:47.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:46 smithi104 ceph-mon[36624]: pgmap v3384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:47.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:46 smithi081 ceph-mon[31453]: pgmap v3384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:49.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:48 smithi153 ceph-mon[36569]: pgmap v3385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:49.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:48 smithi104 ceph-mon[36624]: pgmap v3385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:49.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:48 smithi081 ceph-mon[31453]: pgmap v3385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:50.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:49:50.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:50.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:50.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:50.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:49:50.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:50.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:50.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:50.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:49:50.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:50.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:50.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:51.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:51.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:50 smithi153 ceph-mon[36569]: pgmap v3386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:51.050 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:51.050 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:50 smithi081 ceph-mon[31453]: pgmap v3386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:51.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:51.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:50 smithi104 ceph-mon[36624]: pgmap v3386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:52.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:52.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:53.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:53 smithi153 ceph-mon[36569]: pgmap v3387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:53.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:53 smithi104 ceph-mon[36624]: pgmap v3387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:53.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:53 smithi081 ceph-mon[31453]: pgmap v3387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:55.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:55.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:55.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:55 smithi153 ceph-mon[36569]: pgmap v3388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:55.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:55.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:55.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:55 smithi104 ceph-mon[36624]: pgmap v3388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:55.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:55.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:55.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:55 smithi081 ceph-mon[31453]: pgmap v3388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:56.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:49:56.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:49:56.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:56 smithi104 ceph-mon[36624]: pgmap v3389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:56.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:56 smithi104 ceph-mon[36624]: pgmap v3390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:56.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:56.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:49:56.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:49:56.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:56 smithi081 ceph-mon[31453]: pgmap v3389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:56.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:56 smithi081 ceph-mon[31453]: pgmap v3390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:56.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:56.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:49:56.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:49:56.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:56 smithi153 ceph-mon[36569]: pgmap v3389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:56.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:56 smithi153 ceph-mon[36569]: pgmap v3390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:56.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:49:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:57.008+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:49:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:57.008+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:49:57.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:49:57.008+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:49:58.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:49:58 smithi153 ceph-mon[36569]: pgmap v3391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:58.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:49:58 smithi104 ceph-mon[36624]: pgmap v3391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:49:58.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:49:58 smithi081 ceph-mon[31453]: pgmap v3391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:00.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:00 smithi104 ceph-mon[36624]: pgmap v3392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:00.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:50:00.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:00 smithi081 ceph-mon[31453]: pgmap v3392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:00.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:50:01.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:00 smithi153 ceph-mon[36569]: pgmap v3392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:01.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T22:50:02.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:02 smithi104 ceph-mon[36624]: pgmap v3393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:02.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:02 smithi081 ceph-mon[31453]: pgmap v3393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:03.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:02 smithi153 ceph-mon[36569]: pgmap v3393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:04.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:04 smithi104 ceph-mon[36624]: pgmap v3394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:04.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:04 smithi081 ceph-mon[31453]: pgmap v3394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:05.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:04 smithi153 ceph-mon[36569]: pgmap v3394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:06.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:06 smithi104 ceph-mon[36624]: pgmap v3395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:06.849 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:06 smithi153 ceph-mon[36569]: pgmap v3395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:06.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:06 smithi081 ceph-mon[31453]: pgmap v3395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:08.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:08 smithi081 ceph-mon[31453]: pgmap v3396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:09.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:08 smithi153 ceph-mon[36569]: pgmap v3396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:09.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:08 smithi104 ceph-mon[36624]: pgmap v3396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:10.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:10 smithi081 ceph-mon[31453]: pgmap v3397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:11.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:10 smithi153 ceph-mon[36569]: pgmap v3397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:11.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:10 smithi104 ceph-mon[36624]: pgmap v3397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:12.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:12.015+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:50:12.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:12.015+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:50:12.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:12.015+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:50:12.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:12 smithi081 ceph-mon[31453]: pgmap v3398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:13.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:12 smithi153 ceph-mon[36569]: pgmap v3398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:13.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:12 smithi104 ceph-mon[36624]: pgmap v3398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:14.875 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:14 smithi104 ceph-mon[36624]: pgmap v3399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:14.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:14 smithi081 ceph-mon[31453]: pgmap v3399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:15.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:14 smithi153 ceph-mon[36569]: pgmap v3399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:16.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:16 smithi081 ceph-mon[31453]: pgmap v3400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:17.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:16 smithi153 ceph-mon[36569]: pgmap v3400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:17.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:16 smithi104 ceph-mon[36624]: pgmap v3400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:18.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:18 smithi081 ceph-mon[31453]: pgmap v3401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:19.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:18 smithi153 ceph-mon[36569]: pgmap v3401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:19.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:18 smithi104 ceph-mon[36624]: pgmap v3401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:20.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:20 smithi081 ceph-mon[31453]: pgmap v3402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:21.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:20 smithi153 ceph-mon[36569]: pgmap v3402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:21.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:20 smithi104 ceph-mon[36624]: pgmap v3402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:22.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:22 smithi081 ceph-mon[31453]: pgmap v3403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:23.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:22 smithi153 ceph-mon[36569]: pgmap v3403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:23.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:22 smithi104 ceph-mon[36624]: pgmap v3403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:24.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:24 smithi081 ceph-mon[31453]: pgmap v3404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:25.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:24 smithi153 ceph-mon[36569]: pgmap v3404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:25.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:24 smithi104 ceph-mon[36624]: pgmap v3404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:26.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:26 smithi081 ceph-mon[31453]: pgmap v3405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:27.023+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:50:27.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:27.023+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:50:27.015 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:27.023+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:50:27.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:26 smithi153 ceph-mon[36569]: pgmap v3405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:27.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:26 smithi104 ceph-mon[36624]: pgmap v3405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:28.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:28 smithi081 ceph-mon[31453]: pgmap v3406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:29.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:28 smithi153 ceph-mon[36569]: pgmap v3406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:29.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:28 smithi104 ceph-mon[36624]: pgmap v3406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:30.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:30 smithi081 ceph-mon[31453]: pgmap v3407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:31.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:30 smithi153 ceph-mon[36569]: pgmap v3407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:31.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:30 smithi104 ceph-mon[36624]: pgmap v3407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:32.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:32 smithi081 ceph-mon[31453]: pgmap v3408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:33.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:32 smithi153 ceph-mon[36569]: pgmap v3408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:33.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:32 smithi104 ceph-mon[36624]: pgmap v3408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:34.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:34 smithi081 ceph-mon[31453]: pgmap v3409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:35.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:34 smithi153 ceph-mon[36569]: pgmap v3409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:35.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:34 smithi104 ceph-mon[36624]: pgmap v3409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:37.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:36 smithi153 ceph-mon[36569]: pgmap v3410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:37.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:36 smithi104 ceph-mon[36624]: pgmap v3410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:37.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:36 smithi081 ceph-mon[31453]: pgmap v3410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:39.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:38 smithi153 ceph-mon[36569]: pgmap v3411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:39.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:38 smithi104 ceph-mon[36624]: pgmap v3411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:39.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:38 smithi081 ceph-mon[31453]: pgmap v3411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:41.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:40 smithi153 ceph-mon[36569]: pgmap v3412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:41.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:40 smithi104 ceph-mon[36624]: pgmap v3412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:41.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:40 smithi081 ceph-mon[31453]: pgmap v3412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:42.036+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:50:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:42.036+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:50:42.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:42.036+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:50:43.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:42 smithi153 ceph-mon[36569]: pgmap v3413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:43.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:42 smithi104 ceph-mon[36624]: pgmap v3413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:43.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:42 smithi081 ceph-mon[31453]: pgmap v3413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:45.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:44 smithi153 ceph-mon[36569]: pgmap v3414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:45.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:44 smithi104 ceph-mon[36624]: pgmap v3414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:45.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:44 smithi081 ceph-mon[31453]: pgmap v3414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:47.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:46 smithi153 ceph-mon[36569]: pgmap v3415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:47.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:46 smithi104 ceph-mon[36624]: pgmap v3415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:47.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:46 smithi081 ceph-mon[31453]: pgmap v3415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:49.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:48 smithi153 ceph-mon[36569]: pgmap v3416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:49.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:48 smithi104 ceph-mon[36624]: pgmap v3416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:49.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:48 smithi081 ceph-mon[31453]: pgmap v3416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:51.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:50 smithi153 ceph-mon[36569]: pgmap v3417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:51.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:50 smithi104 ceph-mon[36624]: pgmap v3417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:51.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:50 smithi081 ceph-mon[31453]: pgmap v3417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:53.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:52 smithi153 ceph-mon[36569]: pgmap v3418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:53.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:52 smithi104 ceph-mon[36624]: pgmap v3418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:53.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:52 smithi081 ceph-mon[31453]: pgmap v3418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:55.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:54 smithi153 ceph-mon[36569]: pgmap v3419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:55.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:54 smithi104 ceph-mon[36624]: pgmap v3419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:55.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:54 smithi081 ceph-mon[31453]: pgmap v3419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:56.951 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:56 smithi081 ceph-mon[31453]: pgmap v3420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:56.951 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:50:56.951 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:56.951 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:57.051+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:50:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:57.051+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:50:57.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:50:57.051+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:50:57.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:56 smithi153 ceph-mon[36569]: pgmap v3420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:57.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:50:57.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:57.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:57.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:56 smithi104 ceph-mon[36624]: pgmap v3420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:57.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:50:57.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:57.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:58.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:50:58.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:50:58.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:58.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:50:58.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:50:58.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:58.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:50:58.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:50:58.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:50:59.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:58 smithi153 ceph-mon[36569]: pgmap v3421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:59.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:50:58 smithi153 ceph-mon[36569]: pgmap v3422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:59.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:58 smithi104 ceph-mon[36624]: pgmap v3421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:59.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:50:58 smithi104 ceph-mon[36624]: pgmap v3422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:59.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:58 smithi081 ceph-mon[31453]: pgmap v3421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:50:59.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:50:58 smithi081 ceph-mon[31453]: pgmap v3422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:01.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:00 smithi153 ceph-mon[36569]: pgmap v3423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:01.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:00 smithi104 ceph-mon[36624]: pgmap v3423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:01.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:00 smithi081 ceph-mon[31453]: pgmap v3423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:02.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:01 smithi153 ceph-mon[36569]: pgmap v3424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:02.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:01 smithi104 ceph-mon[36624]: pgmap v3424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:02.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:01 smithi081 ceph-mon[31453]: pgmap v3424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:04.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:04 smithi153 ceph-mon[36569]: pgmap v3425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:04.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:04 smithi104 ceph-mon[36624]: pgmap v3425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:04.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:04 smithi081 ceph-mon[31453]: pgmap v3425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:06.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:06 smithi153 ceph-mon[36569]: pgmap v3426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:06.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:06 smithi104 ceph-mon[36624]: pgmap v3426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:06.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:06 smithi081 ceph-mon[31453]: pgmap v3426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:08.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:08 smithi153 ceph-mon[36569]: pgmap v3427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:08.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:08 smithi104 ceph-mon[36624]: pgmap v3427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:08.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:08 smithi081 ceph-mon[31453]: pgmap v3427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:10.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:10 smithi153 ceph-mon[36569]: pgmap v3428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:10.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:10 smithi104 ceph-mon[36624]: pgmap v3428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:10.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:10 smithi081 ceph-mon[31453]: pgmap v3428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:12.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:12.063+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:51:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:12.064+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:51:12.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:12.064+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:51:12.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:12 smithi153 ceph-mon[36569]: pgmap v3429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:12.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:12 smithi104 ceph-mon[36624]: pgmap v3429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:12.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:12 smithi081 ceph-mon[31453]: pgmap v3429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:14.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:14 smithi153 ceph-mon[36569]: pgmap v3430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:14.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:14 smithi104 ceph-mon[36624]: pgmap v3430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:14.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:14 smithi081 ceph-mon[31453]: pgmap v3430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:16.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:16 smithi153 ceph-mon[36569]: pgmap v3431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:16.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:16 smithi104 ceph-mon[36624]: pgmap v3431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:16.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:16 smithi081 ceph-mon[31453]: pgmap v3431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:18.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:18 smithi153 ceph-mon[36569]: pgmap v3432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:18.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:18 smithi104 ceph-mon[36624]: pgmap v3432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:18.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:18 smithi081 ceph-mon[31453]: pgmap v3432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:20.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:20 smithi153 ceph-mon[36569]: pgmap v3433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:20.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:20 smithi104 ceph-mon[36624]: pgmap v3433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:20.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:20 smithi081 ceph-mon[31453]: pgmap v3433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:22.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:22 smithi153 ceph-mon[36569]: pgmap v3434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:22.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:22 smithi104 ceph-mon[36624]: pgmap v3434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:22.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:22 smithi081 ceph-mon[31453]: pgmap v3434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:24.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:24 smithi153 ceph-mon[36569]: pgmap v3435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:24.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:24 smithi104 ceph-mon[36624]: pgmap v3435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:24.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:24 smithi081 ceph-mon[31453]: pgmap v3435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:26.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:26 smithi153 ceph-mon[36569]: pgmap v3436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:26.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:26 smithi104 ceph-mon[36624]: pgmap v3436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:26.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:26 smithi081 ceph-mon[31453]: pgmap v3436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:27.073+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:51:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:27.073+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:51:27.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:27.073+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:51:28.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:28 smithi153 ceph-mon[36569]: pgmap v3437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:28.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:28 smithi104 ceph-mon[36624]: pgmap v3437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:28.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:28 smithi081 ceph-mon[31453]: pgmap v3437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:30.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:30 smithi153 ceph-mon[36569]: pgmap v3438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:30.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:30 smithi104 ceph-mon[36624]: pgmap v3438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:30.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:30 smithi081 ceph-mon[31453]: pgmap v3438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:32.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:32 smithi153 ceph-mon[36569]: pgmap v3439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:32.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:32 smithi104 ceph-mon[36624]: pgmap v3439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:32.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:32 smithi081 ceph-mon[31453]: pgmap v3439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:34.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:34 smithi104 ceph-mon[36624]: pgmap v3440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:34.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:34 smithi081 ceph-mon[31453]: pgmap v3440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:34.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:34 smithi153 ceph-mon[36569]: pgmap v3440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:36.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:36 smithi104 ceph-mon[36624]: pgmap v3441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:36.599 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:36 smithi153 ceph-mon[36569]: pgmap v3441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:36.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:36 smithi081 ceph-mon[31453]: pgmap v3441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:38.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:38 smithi104 ceph-mon[36624]: pgmap v3442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:38.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:38 smithi081 ceph-mon[31453]: pgmap v3442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:38.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:38 smithi153 ceph-mon[36569]: pgmap v3442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:40.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:40 smithi104 ceph-mon[36624]: pgmap v3443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:40.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:40 smithi081 ceph-mon[31453]: pgmap v3443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:40.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:40 smithi153 ceph-mon[36569]: pgmap v3443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:42.079+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:51:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:42.079+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:51:42.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:42.079+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:51:42.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:42 smithi104 ceph-mon[36624]: pgmap v3444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:42.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:42 smithi081 ceph-mon[31453]: pgmap v3444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:42.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:42 smithi153 ceph-mon[36569]: pgmap v3444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:44.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:44 smithi081 ceph-mon[31453]: pgmap v3445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:44.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:44 smithi153 ceph-mon[36569]: pgmap v3445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:44.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:44 smithi104 ceph-mon[36624]: pgmap v3445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:46.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:46 smithi081 ceph-mon[31453]: pgmap v3446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:46.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:46 smithi153 ceph-mon[36569]: pgmap v3446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:46.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:46 smithi104 ceph-mon[36624]: pgmap v3446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:48.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:48 smithi081 ceph-mon[31453]: pgmap v3447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:48.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:48 smithi153 ceph-mon[36569]: pgmap v3447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:48.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:48 smithi104 ceph-mon[36624]: pgmap v3447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:50.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:50 smithi081 ceph-mon[31453]: pgmap v3448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:50.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:50 smithi153 ceph-mon[36569]: pgmap v3448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:50.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:50 smithi104 ceph-mon[36624]: pgmap v3448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:52.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:52 smithi081 ceph-mon[31453]: pgmap v3449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:52.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:52 smithi153 ceph-mon[36569]: pgmap v3449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:52.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:52 smithi104 ceph-mon[36624]: pgmap v3449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:54.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:54 smithi081 ceph-mon[31453]: pgmap v3450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:54.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:54 smithi153 ceph-mon[36569]: pgmap v3450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:54.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:54 smithi104 ceph-mon[36624]: pgmap v3450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:56.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:56 smithi081 ceph-mon[31453]: pgmap v3451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:56.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:56 smithi153 ceph-mon[36569]: pgmap v3451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:56.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:56 smithi104 ceph-mon[36624]: pgmap v3451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:57.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:57.094+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:51:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:57.094+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:51:57.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:51:57.094+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:51:58.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:58 smithi081 ceph-mon[31453]: pgmap v3452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:58.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:51:58.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:58 smithi153 ceph-mon[36569]: pgmap v3452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:58.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:51:58.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:58 smithi104 ceph-mon[36624]: pgmap v3452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:58.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:51:59.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:51:59.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:51:59.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:59 smithi081 ceph-mon[31453]: pgmap v3453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:59.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:59 smithi081 ceph-mon[31453]: pgmap v3454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:59.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:51:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:51:59.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:51:59.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:51:59.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:59 smithi153 ceph-mon[36569]: pgmap v3453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:59.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:59 smithi153 ceph-mon[36569]: pgmap v3454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:59.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:51:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:51:59.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:51:59.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:51:59.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:59 smithi104 ceph-mon[36624]: pgmap v3453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:59.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:59 smithi104 ceph-mon[36624]: pgmap v3454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:51:59.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:51:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:52:02.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:01 smithi153 ceph-mon[36569]: pgmap v3455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:02.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:01 smithi104 ceph-mon[36624]: pgmap v3455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:02.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:01 smithi081 ceph-mon[31453]: pgmap v3455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:04.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:03 smithi153 ceph-mon[36569]: pgmap v3456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:04.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:03 smithi104 ceph-mon[36624]: pgmap v3456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:04.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:03 smithi081 ceph-mon[31453]: pgmap v3456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:05.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:04 smithi153 ceph-mon[36569]: pgmap v3457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:05.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:04 smithi104 ceph-mon[36624]: pgmap v3457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:05.118 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:04 smithi081 ceph-mon[31453]: pgmap v3457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:08.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:07 smithi153 ceph-mon[36569]: pgmap v3458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:08.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:07 smithi104 ceph-mon[36624]: pgmap v3458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:08.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:07 smithi081 ceph-mon[31453]: pgmap v3458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:10.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:09 smithi153 ceph-mon[36569]: pgmap v3459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:10.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:09 smithi104 ceph-mon[36624]: pgmap v3459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:10.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:09 smithi081 ceph-mon[31453]: pgmap v3459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:11.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:10 smithi153 ceph-mon[36569]: pgmap v3460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:11.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:10 smithi104 ceph-mon[36624]: pgmap v3460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:11.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:10 smithi081 ceph-mon[31453]: pgmap v3460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:12.109+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:52:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:12.109+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:52:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:12.109+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:52:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:13.644+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:52:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:13.644+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:52:13.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:13.645+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:52:13.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:13.645+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:52:14.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:13 smithi153 ceph-mon[36569]: pgmap v3461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:14.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:13 smithi104 ceph-mon[36624]: pgmap v3461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:14.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:13 smithi081 ceph-mon[31453]: pgmap v3461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:15.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:14 smithi153 ceph-mon[36569]: pgmap v3462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:15.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:14 smithi104 ceph-mon[36624]: pgmap v3462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:15.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:14 smithi081 ceph-mon[31453]: pgmap v3462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:18.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:17 smithi153 ceph-mon[36569]: pgmap v3463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:18.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:17 smithi104 ceph-mon[36624]: pgmap v3463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:18.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:17 smithi081 ceph-mon[31453]: pgmap v3463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:19.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:18 smithi153 ceph-mon[36569]: pgmap v3464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:19.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:18 smithi104 ceph-mon[36624]: pgmap v3464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:19.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:18 smithi081 ceph-mon[31453]: pgmap v3464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:22.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:21 smithi153 ceph-mon[36569]: pgmap v3465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:22.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:21 smithi104 ceph-mon[36624]: pgmap v3465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:22.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:21 smithi081 ceph-mon[31453]: pgmap v3465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:23.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:22 smithi153 ceph-mon[36569]: pgmap v3466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:23.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:22 smithi104 ceph-mon[36624]: pgmap v3466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:23.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:22 smithi081 ceph-mon[31453]: pgmap v3466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:25.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:24 smithi153 ceph-mon[36569]: pgmap v3467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:25.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:24 smithi104 ceph-mon[36624]: pgmap v3467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:25.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:24 smithi081 ceph-mon[31453]: pgmap v3467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:27.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:26 smithi153 ceph-mon[36569]: pgmap v3468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:27.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:26 smithi104 ceph-mon[36624]: pgmap v3468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:27.119+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:52:27.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:27.119+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:52:27.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:27.119+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:52:27.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:26 smithi081 ceph-mon[31453]: pgmap v3468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:29.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:28 smithi153 ceph-mon[36569]: pgmap v3469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:29.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:28 smithi104 ceph-mon[36624]: pgmap v3469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:29.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:28 smithi081 ceph-mon[31453]: pgmap v3469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:31.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:30 smithi153 ceph-mon[36569]: pgmap v3470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:31.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:30 smithi104 ceph-mon[36624]: pgmap v3470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:31.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:30 smithi081 ceph-mon[31453]: pgmap v3470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:33.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:32 smithi153 ceph-mon[36569]: pgmap v3471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:33.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:32 smithi104 ceph-mon[36624]: pgmap v3471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:33.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:32 smithi081 ceph-mon[31453]: pgmap v3471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:35.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:34 smithi153 ceph-mon[36569]: pgmap v3472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:35.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:34 smithi104 ceph-mon[36624]: pgmap v3472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:35.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:34 smithi081 ceph-mon[31453]: pgmap v3472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:37.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:36 smithi153 ceph-mon[36569]: pgmap v3473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:37.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:36 smithi104 ceph-mon[36624]: pgmap v3473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:37.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:36 smithi081 ceph-mon[31453]: pgmap v3473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:39.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:38 smithi153 ceph-mon[36569]: pgmap v3474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:39.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:38 smithi104 ceph-mon[36624]: pgmap v3474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:39.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:38 smithi081 ceph-mon[31453]: pgmap v3474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:41.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:40 smithi153 ceph-mon[36569]: pgmap v3475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:41.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:40 smithi104 ceph-mon[36624]: pgmap v3475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:41.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:40 smithi081 ceph-mon[31453]: pgmap v3475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:42.128+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:52:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:42.128+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:52:42.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:42.128+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:52:43.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:42 smithi153 ceph-mon[36569]: pgmap v3476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:43.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:42 smithi104 ceph-mon[36624]: pgmap v3476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:43.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:42 smithi081 ceph-mon[31453]: pgmap v3476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:45.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:44 smithi153 ceph-mon[36569]: pgmap v3477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:45.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:44 smithi104 ceph-mon[36624]: pgmap v3477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:45.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:44 smithi081 ceph-mon[31453]: pgmap v3477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:47.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:46 smithi153 ceph-mon[36569]: pgmap v3478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:47.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:46 smithi104 ceph-mon[36624]: pgmap v3478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:47.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:46 smithi081 ceph-mon[31453]: pgmap v3478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:49.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:48 smithi153 ceph-mon[36569]: pgmap v3479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:49.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:48 smithi104 ceph-mon[36624]: pgmap v3479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:49.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:48 smithi081 ceph-mon[31453]: pgmap v3479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:51.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:50 smithi153 ceph-mon[36569]: pgmap v3480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:51.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:50 smithi104 ceph-mon[36624]: pgmap v3480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:51.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:50 smithi081 ceph-mon[31453]: pgmap v3480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:53.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:52 smithi153 ceph-mon[36569]: pgmap v3481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:53.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:52 smithi104 ceph-mon[36624]: pgmap v3481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:53.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:52 smithi081 ceph-mon[31453]: pgmap v3481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:55.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:54 smithi153 ceph-mon[36569]: pgmap v3482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:55.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:54 smithi104 ceph-mon[36624]: pgmap v3482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:55.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:54 smithi081 ceph-mon[31453]: pgmap v3482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:57.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:56 smithi153 ceph-mon[36569]: pgmap v3483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:57.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:56 smithi104 ceph-mon[36624]: pgmap v3483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:57.144+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:52:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:57.144+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:52:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:52:57.144+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:52:57.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:56 smithi081 ceph-mon[31453]: pgmap v3483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:59.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:58 smithi153 ceph-mon[36569]: pgmap v3484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:59.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:58 smithi104 ceph-mon[36624]: pgmap v3484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:52:59.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:58 smithi081 ceph-mon[31453]: pgmap v3484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:00.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:52:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:53:00.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:52:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:53:00.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:52:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:53:01.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:53:01.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:53:01.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:00 smithi104 ceph-mon[36624]: pgmap v3485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:01.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:00 smithi104 ceph-mon[36624]: pgmap v3486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:01.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:53:01.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:53:01.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:53:01.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:00 smithi081 ceph-mon[31453]: pgmap v3485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:01.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:00 smithi081 ceph-mon[31453]: pgmap v3486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:01.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:53:01.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:53:01.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:53:01.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:00 smithi153 ceph-mon[36569]: pgmap v3485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:01.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:00 smithi153 ceph-mon[36569]: pgmap v3486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:01.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:53:03.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:03 smithi153 ceph-mon[36569]: pgmap v3487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:03.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:03 smithi104 ceph-mon[36624]: pgmap v3487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:03.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:03 smithi081 ceph-mon[31453]: pgmap v3487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:05.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:05 smithi153 ceph-mon[36569]: pgmap v3488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:05.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:05 smithi104 ceph-mon[36624]: pgmap v3488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:05.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:05 smithi081 ceph-mon[31453]: pgmap v3488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:07.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:07 smithi081 ceph-mon[31453]: pgmap v3489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:07.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:07 smithi153 ceph-mon[36569]: pgmap v3489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:07.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:07 smithi104 ceph-mon[36624]: pgmap v3489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:09.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:09 smithi153 ceph-mon[36569]: pgmap v3490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:09.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:09 smithi104 ceph-mon[36624]: pgmap v3490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:09.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:09 smithi081 ceph-mon[31453]: pgmap v3490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:11.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:11 smithi153 ceph-mon[36569]: pgmap v3491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:11.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:11 smithi104 ceph-mon[36624]: pgmap v3491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:11.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:11 smithi081 ceph-mon[31453]: pgmap v3491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:12.153+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:53:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:12.153+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:53:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:12.153+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:53:13.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:13 smithi153 ceph-mon[36569]: pgmap v3492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:13.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:13 smithi104 ceph-mon[36624]: pgmap v3492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:13.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:13 smithi081 ceph-mon[31453]: pgmap v3492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:15.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:15 smithi153 ceph-mon[36569]: pgmap v3493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:15.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:15 smithi104 ceph-mon[36624]: pgmap v3493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:15.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:15 smithi081 ceph-mon[31453]: pgmap v3493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:17.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:17 smithi153 ceph-mon[36569]: pgmap v3494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:17.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:17 smithi104 ceph-mon[36624]: pgmap v3494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:17.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:17 smithi081 ceph-mon[31453]: pgmap v3494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:19.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:19 smithi153 ceph-mon[36569]: pgmap v3495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:19.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:19 smithi104 ceph-mon[36624]: pgmap v3495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:19.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:19 smithi081 ceph-mon[31453]: pgmap v3495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:21.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:21 smithi153 ceph-mon[36569]: pgmap v3496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:21.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:21 smithi104 ceph-mon[36624]: pgmap v3496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:21.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:21 smithi081 ceph-mon[31453]: pgmap v3496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:23.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:23 smithi153 ceph-mon[36569]: pgmap v3497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:23.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:23 smithi104 ceph-mon[36624]: pgmap v3497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:23.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:23 smithi081 ceph-mon[31453]: pgmap v3497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:25.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:25 smithi153 ceph-mon[36569]: pgmap v3498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:25.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:25 smithi104 ceph-mon[36624]: pgmap v3498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:25.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:25 smithi081 ceph-mon[31453]: pgmap v3498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:27.167+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:53:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:27.167+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:53:27.159 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:27.167+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:53:27.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:27 smithi153 ceph-mon[36569]: pgmap v3499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:27.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:27 smithi104 ceph-mon[36624]: pgmap v3499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:27.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:27 smithi081 ceph-mon[31453]: pgmap v3499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:29.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:29 smithi153 ceph-mon[36569]: pgmap v3500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:29.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:29 smithi104 ceph-mon[36624]: pgmap v3500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:29.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:29 smithi081 ceph-mon[31453]: pgmap v3500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:31.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:31 smithi153 ceph-mon[36569]: pgmap v3501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:31.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:31 smithi104 ceph-mon[36624]: pgmap v3501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:31.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:31 smithi081 ceph-mon[31453]: pgmap v3501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:33.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:33 smithi153 ceph-mon[36569]: pgmap v3502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:33.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:33 smithi104 ceph-mon[36624]: pgmap v3502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:33.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:33 smithi081 ceph-mon[31453]: pgmap v3502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:35.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:35 smithi153 ceph-mon[36569]: pgmap v3503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:35.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:35 smithi104 ceph-mon[36624]: pgmap v3503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:35.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:35 smithi081 ceph-mon[31453]: pgmap v3503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:37.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:37 smithi153 ceph-mon[36569]: pgmap v3504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:37.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:37 smithi104 ceph-mon[36624]: pgmap v3504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:37.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:37 smithi081 ceph-mon[31453]: pgmap v3504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:39.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:39 smithi153 ceph-mon[36569]: pgmap v3505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:39.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:39 smithi104 ceph-mon[36624]: pgmap v3505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:39.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:39 smithi081 ceph-mon[31453]: pgmap v3505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:41.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:41 smithi153 ceph-mon[36569]: pgmap v3506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:41.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:41 smithi104 ceph-mon[36624]: pgmap v3506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:41.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:41 smithi081 ceph-mon[31453]: pgmap v3506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:42.184+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:53:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:42.184+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:53:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:42.184+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:53:43.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:43 smithi153 ceph-mon[36569]: pgmap v3507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:43.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:43 smithi104 ceph-mon[36624]: pgmap v3507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:43.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:43 smithi081 ceph-mon[31453]: pgmap v3507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:45.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:45 smithi104 ceph-mon[36624]: pgmap v3508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:45.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:45 smithi081 ceph-mon[31453]: pgmap v3508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:45.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:45 smithi153 ceph-mon[36569]: pgmap v3508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:47.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:47 smithi104 ceph-mon[36624]: pgmap v3509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:47.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:47 smithi081 ceph-mon[31453]: pgmap v3509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:47.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:47 smithi153 ceph-mon[36569]: pgmap v3509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:49.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:49 smithi104 ceph-mon[36624]: pgmap v3510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:49.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:49 smithi081 ceph-mon[31453]: pgmap v3510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:49.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:49 smithi153 ceph-mon[36569]: pgmap v3510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:51.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:51 smithi081 ceph-mon[31453]: pgmap v3511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:51.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:51 smithi153 ceph-mon[36569]: pgmap v3511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:51.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:51 smithi104 ceph-mon[36624]: pgmap v3511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:53.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:53 smithi081 ceph-mon[31453]: pgmap v3512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:53.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:53 smithi153 ceph-mon[36569]: pgmap v3512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:53.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:53 smithi104 ceph-mon[36624]: pgmap v3512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:55.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:55 smithi081 ceph-mon[31453]: pgmap v3513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:55.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:55 smithi153 ceph-mon[36569]: pgmap v3513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:55.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:55 smithi104 ceph-mon[36624]: pgmap v3513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:57.199+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:53:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:57.200+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:53:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:53:57.200+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:53:57.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:57 smithi081 ceph-mon[31453]: pgmap v3514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:57.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:57 smithi153 ceph-mon[36569]: pgmap v3514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:57.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:57 smithi104 ceph-mon[36624]: pgmap v3514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:59.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:53:59 smithi153 ceph-mon[36569]: pgmap v3515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:59.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:53:59 smithi104 ceph-mon[36624]: pgmap v3515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:53:59.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:53:59 smithi081 ceph-mon[31453]: pgmap v3515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:01.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:01 smithi153 ceph-mon[36569]: pgmap v3516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:01.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:54:01.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:01 smithi104 ceph-mon[36624]: pgmap v3516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:01.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:54:01.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:01 smithi081 ceph-mon[31453]: pgmap v3516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:01.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:54:02.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:54:02.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:54:02.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:02 smithi153 ceph-mon[36569]: pgmap v3517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:02.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:02 smithi153 ceph-mon[36569]: pgmap v3518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:02.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:54:02.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:54:02.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:54:02.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:02 smithi104 ceph-mon[36624]: pgmap v3517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:02.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:02 smithi104 ceph-mon[36624]: pgmap v3518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:02.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:54:02.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:54:02.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:54:02.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:02 smithi081 ceph-mon[31453]: pgmap v3517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:02.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:02 smithi081 ceph-mon[31453]: pgmap v3518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:02.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:54:05.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:04 smithi153 ceph-mon[36569]: pgmap v3519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:05.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:04 smithi104 ceph-mon[36624]: pgmap v3519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:05.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:04 smithi081 ceph-mon[31453]: pgmap v3519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:06.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:05 smithi104 ceph-mon[36624]: pgmap v3520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:06.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:05 smithi081 ceph-mon[31453]: pgmap v3520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:06.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:05 smithi153 ceph-mon[36569]: pgmap v3520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:09.023 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:08 smithi081 ceph-mon[31453]: pgmap v3521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:09.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:08 smithi153 ceph-mon[36569]: pgmap v3521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:09.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:08 smithi104 ceph-mon[36624]: pgmap v3521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:10.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:09 smithi104 ceph-mon[36624]: pgmap v3522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:10.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:09 smithi081 ceph-mon[31453]: pgmap v3522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:10.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:09 smithi153 ceph-mon[36569]: pgmap v3522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:12.209+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:54:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:12.209+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:54:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:12.209+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:54:13.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:12 smithi153 ceph-mon[36569]: pgmap v3523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:13.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:12 smithi104 ceph-mon[36624]: pgmap v3523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:13.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:12 smithi081 ceph-mon[31453]: pgmap v3523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:14.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:13 smithi153 ceph-mon[36569]: pgmap v3524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:14.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:13 smithi104 ceph-mon[36624]: pgmap v3524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:14.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:13 smithi081 ceph-mon[31453]: pgmap v3524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:17.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:16 smithi153 ceph-mon[36569]: pgmap v3525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:17.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:16 smithi104 ceph-mon[36624]: pgmap v3525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:17.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:16 smithi081 ceph-mon[31453]: pgmap v3525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:18.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:17 smithi153 ceph-mon[36569]: pgmap v3526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:18.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:17 smithi104 ceph-mon[36624]: pgmap v3526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:18.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:17 smithi081 ceph-mon[31453]: pgmap v3526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:20.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:20 smithi153 ceph-mon[36569]: pgmap v3527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:20.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:20 smithi104 ceph-mon[36624]: pgmap v3527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:20.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:20 smithi081 ceph-mon[31453]: pgmap v3527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:22.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:21 smithi104 ceph-mon[36624]: pgmap v3528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:22.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:21 smithi081 ceph-mon[31453]: pgmap v3528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:22.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:21 smithi153 ceph-mon[36569]: pgmap v3528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:24.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:23 smithi104 ceph-mon[36624]: pgmap v3529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:24.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:23 smithi081 ceph-mon[31453]: pgmap v3529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:24.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:23 smithi153 ceph-mon[36569]: pgmap v3529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:26.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:25 smithi104 ceph-mon[36624]: pgmap v3530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:26.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:25 smithi081 ceph-mon[31453]: pgmap v3530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:26.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:25 smithi153 ceph-mon[36569]: pgmap v3530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:27.213 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:27.223+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:54:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:27.223+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:54:27.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:27.223+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:54:28.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:27 smithi104 ceph-mon[36624]: pgmap v3531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:28.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:27 smithi081 ceph-mon[31453]: pgmap v3531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:28.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:27 smithi153 ceph-mon[36569]: pgmap v3531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:30.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:29 smithi104 ceph-mon[36624]: pgmap v3532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:30.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:29 smithi081 ceph-mon[31453]: pgmap v3532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:30.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:29 smithi153 ceph-mon[36569]: pgmap v3532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:32.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:31 smithi104 ceph-mon[36624]: pgmap v3533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:32.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:31 smithi081 ceph-mon[31453]: pgmap v3533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:32.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:31 smithi153 ceph-mon[36569]: pgmap v3533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:34.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:33 smithi104 ceph-mon[36624]: pgmap v3534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:34.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:33 smithi081 ceph-mon[31453]: pgmap v3534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:34.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:33 smithi153 ceph-mon[36569]: pgmap v3534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:36.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:35 smithi104 ceph-mon[36624]: pgmap v3535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:36.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:35 smithi081 ceph-mon[31453]: pgmap v3535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:36.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:35 smithi153 ceph-mon[36569]: pgmap v3535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:38.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:37 smithi104 ceph-mon[36624]: pgmap v3536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:38.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:37 smithi081 ceph-mon[31453]: pgmap v3536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:38.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:37 smithi153 ceph-mon[36569]: pgmap v3536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:40.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:39 smithi104 ceph-mon[36624]: pgmap v3537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:40.099 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:39 smithi153 ceph-mon[36569]: pgmap v3537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:40.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:39 smithi081 ceph-mon[31453]: pgmap v3537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:42.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:41 smithi104 ceph-mon[36624]: pgmap v3538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:42.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:41 smithi081 ceph-mon[31453]: pgmap v3538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:42.233+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:54:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:42.233+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:54:42.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:42.233+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:54:42.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:41 smithi153 ceph-mon[36569]: pgmap v3538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:44.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:43 smithi104 ceph-mon[36624]: pgmap v3539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:44.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:43 smithi081 ceph-mon[31453]: pgmap v3539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:44.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:43 smithi153 ceph-mon[36569]: pgmap v3539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:46.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:45 smithi104 ceph-mon[36624]: pgmap v3540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:46.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:45 smithi081 ceph-mon[31453]: pgmap v3540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:46.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:45 smithi153 ceph-mon[36569]: pgmap v3540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:48.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:47 smithi104 ceph-mon[36624]: pgmap v3541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:48.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:47 smithi081 ceph-mon[31453]: pgmap v3541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:48.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:47 smithi153 ceph-mon[36569]: pgmap v3541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:50.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:49 smithi104 ceph-mon[36624]: pgmap v3542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:50.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:49 smithi081 ceph-mon[31453]: pgmap v3542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:50.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:49 smithi153 ceph-mon[36569]: pgmap v3542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:52.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:51 smithi104 ceph-mon[36624]: pgmap v3543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:52.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:51 smithi081 ceph-mon[31453]: pgmap v3543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:52.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:51 smithi153 ceph-mon[36569]: pgmap v3543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:54.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:53 smithi104 ceph-mon[36624]: pgmap v3544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:54.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:53 smithi081 ceph-mon[31453]: pgmap v3544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:54.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:53 smithi153 ceph-mon[36569]: pgmap v3544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:56.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:55 smithi104 ceph-mon[36624]: pgmap v3545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:56.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:55 smithi081 ceph-mon[31453]: pgmap v3545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:56.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:55 smithi153 ceph-mon[36569]: pgmap v3545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:57.237 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:57.247+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:54:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:57.247+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:54:57.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:54:57.247+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:54:58.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:57 smithi104 ceph-mon[36624]: pgmap v3546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:58.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:57 smithi081 ceph-mon[31453]: pgmap v3546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:54:58.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:57 smithi153 ceph-mon[36569]: pgmap v3546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:00.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:54:59 smithi104 ceph-mon[36624]: pgmap v3547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:00.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:54:59 smithi081 ceph-mon[31453]: pgmap v3547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:00.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:54:59 smithi153 ceph-mon[36569]: pgmap v3547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:02.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:01 smithi104 ceph-mon[36624]: pgmap v3548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:02.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:01 smithi081 ceph-mon[31453]: pgmap v3548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:02.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:01 smithi153 ceph-mon[36569]: pgmap v3548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:03.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:55:03.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:55:03.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:55:04.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:55:04.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:55:04.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:03 smithi081 ceph-mon[31453]: pgmap v3549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:04.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:03 smithi081 ceph-mon[31453]: pgmap v3550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:04.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:55:04.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:55:04.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:55:04.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:03 smithi153 ceph-mon[36569]: pgmap v3549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:04.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:03 smithi153 ceph-mon[36569]: pgmap v3550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:04.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:55:04.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:55:04.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:55:04.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:03 smithi104 ceph-mon[36624]: pgmap v3549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:04.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:03 smithi104 ceph-mon[36624]: pgmap v3550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:04.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:55:06.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:06 smithi081 ceph-mon[31453]: pgmap v3551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:06.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:06 smithi153 ceph-mon[36569]: pgmap v3551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:06.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:06 smithi104 ceph-mon[36624]: pgmap v3551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:08.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:08 smithi081 ceph-mon[31453]: pgmap v3552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:08.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:08 smithi153 ceph-mon[36569]: pgmap v3552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:08.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:08 smithi104 ceph-mon[36624]: pgmap v3552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:10.599 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:10 smithi153 ceph-mon[36569]: pgmap v3553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:10.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:10 smithi081 ceph-mon[31453]: pgmap v3553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:10.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:10 smithi104 ceph-mon[36624]: pgmap v3553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:12.258+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:55:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:12.258+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:55:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:12.258+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:55:12.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:12 smithi081 ceph-mon[31453]: pgmap v3554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:12.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:12 smithi153 ceph-mon[36569]: pgmap v3554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:12.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:12 smithi104 ceph-mon[36624]: pgmap v3554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:13.674+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:55:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:13.675+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:55:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:13.675+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T22:55:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:13.675+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T22:55:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:13.675+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:55:13.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:13.675+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:55:13.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:13.676+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:55:13.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:13.676+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:55:14.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:14 smithi081 ceph-mon[31453]: pgmap v3555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:14.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:14 smithi153 ceph-mon[36569]: pgmap v3555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:14.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:14 smithi104 ceph-mon[36624]: pgmap v3555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:16.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:16 smithi081 ceph-mon[31453]: pgmap v3556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:16.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:16 smithi153 ceph-mon[36569]: pgmap v3556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:16.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:16 smithi104 ceph-mon[36624]: pgmap v3556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:18.624 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:18 smithi104 ceph-mon[36624]: pgmap v3557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:18.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:18 smithi081 ceph-mon[31453]: pgmap v3557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:18.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:18 smithi153 ceph-mon[36569]: pgmap v3557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:20.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:20 smithi081 ceph-mon[31453]: pgmap v3558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:20.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:20 smithi153 ceph-mon[36569]: pgmap v3558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:20.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:20 smithi104 ceph-mon[36624]: pgmap v3558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:22.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:22 smithi081 ceph-mon[31453]: pgmap v3559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:22.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:22 smithi153 ceph-mon[36569]: pgmap v3559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:22.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:22 smithi104 ceph-mon[36624]: pgmap v3559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:24.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:24 smithi081 ceph-mon[31453]: pgmap v3560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:24.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:24 smithi153 ceph-mon[36569]: pgmap v3560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:24.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:24 smithi104 ceph-mon[36624]: pgmap v3560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:26.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:26 smithi081 ceph-mon[31453]: pgmap v3561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:26.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:26 smithi153 ceph-mon[36569]: pgmap v3561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:26.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:26 smithi104 ceph-mon[36624]: pgmap v3561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:27.273+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:55:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:27.274+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:55:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:27.274+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:55:28.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:28 smithi081 ceph-mon[31453]: pgmap v3562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:28.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:28 smithi153 ceph-mon[36569]: pgmap v3562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:28.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:28 smithi104 ceph-mon[36624]: pgmap v3562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:30.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:30 smithi153 ceph-mon[36569]: pgmap v3563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:30.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:30 smithi104 ceph-mon[36624]: pgmap v3563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:30.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:30 smithi081 ceph-mon[31453]: pgmap v3563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:32.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:32 smithi153 ceph-mon[36569]: pgmap v3564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:32.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:32 smithi104 ceph-mon[36624]: pgmap v3564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:32.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:32 smithi081 ceph-mon[31453]: pgmap v3564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:34.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:34 smithi153 ceph-mon[36569]: pgmap v3565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:34.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:34 smithi104 ceph-mon[36624]: pgmap v3565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:34.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:34 smithi081 ceph-mon[31453]: pgmap v3565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:36.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:36 smithi153 ceph-mon[36569]: pgmap v3566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:36.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:36 smithi104 ceph-mon[36624]: pgmap v3566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:36.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:36 smithi081 ceph-mon[31453]: pgmap v3566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:38.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:38 smithi153 ceph-mon[36569]: pgmap v3567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:38.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:38 smithi104 ceph-mon[36624]: pgmap v3567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:38.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:38 smithi081 ceph-mon[31453]: pgmap v3567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:40.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:40 smithi153 ceph-mon[36569]: pgmap v3568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:40.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:40 smithi104 ceph-mon[36624]: pgmap v3568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:40.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:40 smithi081 ceph-mon[31453]: pgmap v3568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:42.283+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:55:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:42.283+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:55:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:42.283+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:55:42.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:42 smithi081 ceph-mon[31453]: pgmap v3569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:42.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:42 smithi153 ceph-mon[36569]: pgmap v3569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:42.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:42 smithi104 ceph-mon[36624]: pgmap v3569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:44.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:44 smithi153 ceph-mon[36569]: pgmap v3570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:44.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:44 smithi104 ceph-mon[36624]: pgmap v3570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:44.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:44 smithi081 ceph-mon[31453]: pgmap v3570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:46.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:46 smithi153 ceph-mon[36569]: pgmap v3571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:46.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:46 smithi104 ceph-mon[36624]: pgmap v3571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:46.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:46 smithi081 ceph-mon[31453]: pgmap v3571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:48.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:48 smithi153 ceph-mon[36569]: pgmap v3572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:48.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:48 smithi104 ceph-mon[36624]: pgmap v3572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:48.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:48 smithi081 ceph-mon[31453]: pgmap v3572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:50.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:50 smithi153 ceph-mon[36569]: pgmap v3573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:50.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:50 smithi104 ceph-mon[36624]: pgmap v3573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:50.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:50 smithi081 ceph-mon[31453]: pgmap v3573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:52.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:52 smithi153 ceph-mon[36569]: pgmap v3574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:52.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:52 smithi104 ceph-mon[36624]: pgmap v3574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:52.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:52 smithi081 ceph-mon[31453]: pgmap v3574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:54.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:54 smithi153 ceph-mon[36569]: pgmap v3575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:54.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:54 smithi104 ceph-mon[36624]: pgmap v3575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:54.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:54 smithi081 ceph-mon[31453]: pgmap v3575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:56.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:56 smithi153 ceph-mon[36569]: pgmap v3576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:56.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:56 smithi104 ceph-mon[36624]: pgmap v3576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:56.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:56 smithi081 ceph-mon[31453]: pgmap v3576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:57.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:57.297+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:55:57.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:57.297+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:55:57.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:55:57.297+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:55:58.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:55:58 smithi153 ceph-mon[36569]: pgmap v3577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:58.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:55:58 smithi104 ceph-mon[36624]: pgmap v3577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:55:58.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:55:58 smithi081 ceph-mon[31453]: pgmap v3577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:00.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:00 smithi153 ceph-mon[36569]: pgmap v3578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:00.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:00 smithi104 ceph-mon[36624]: pgmap v3578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:00.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:00 smithi081 ceph-mon[31453]: pgmap v3578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:02.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:02 smithi153 ceph-mon[36569]: pgmap v3579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:02.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:02 smithi104 ceph-mon[36624]: pgmap v3579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:02.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:02 smithi081 ceph-mon[31453]: pgmap v3579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:04.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: pgmap v3580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:04.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:56:04.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: pgmap v3580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:04.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:56:04.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: pgmap v3580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:04.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:56:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:56:05.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:56:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:56:05.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T22:56:05.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T22:56:05.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T22:56:05.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T22:56:05.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T22:56:05.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:56:05.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:56:05.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:56:06.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:06 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:56:06.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:06 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:56:06.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:06 smithi153 ceph-mon[36569]: pgmap v3581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:06.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:06 smithi153 ceph-mon[36569]: pgmap v3582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:06.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:06 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:56:06.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:06 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:56:06.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:06 smithi104 ceph-mon[36624]: pgmap v3581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:06.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:06 smithi104 ceph-mon[36624]: pgmap v3582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:06.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:06 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T22:56:06.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:06 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T22:56:06.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:06 smithi081 ceph-mon[31453]: pgmap v3581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:06.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:06 smithi081 ceph-mon[31453]: pgmap v3582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:08.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:08 smithi153 ceph-mon[36569]: pgmap v3583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:08.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:08 smithi104 ceph-mon[36624]: pgmap v3583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:08.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:08 smithi081 ceph-mon[31453]: pgmap v3583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:10.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:10 smithi153 ceph-mon[36569]: pgmap v3584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:10.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:10 smithi104 ceph-mon[36624]: pgmap v3584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:10.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:10 smithi081 ceph-mon[31453]: pgmap v3584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:12.309+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:56:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:12.309+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:56:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:12.309+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:56:12.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:12 smithi153 ceph-mon[36569]: pgmap v3585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:12.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:12 smithi104 ceph-mon[36624]: pgmap v3585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:12.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:12 smithi081 ceph-mon[31453]: pgmap v3585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:14.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:14 smithi153 ceph-mon[36569]: pgmap v3586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:14.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:14 smithi104 ceph-mon[36624]: pgmap v3586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:14.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:14 smithi081 ceph-mon[31453]: pgmap v3586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:16.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:16 smithi153 ceph-mon[36569]: pgmap v3587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:16.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:16 smithi104 ceph-mon[36624]: pgmap v3587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:16.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:16 smithi081 ceph-mon[31453]: pgmap v3587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:18.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:18 smithi153 ceph-mon[36569]: pgmap v3588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:18.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:18 smithi104 ceph-mon[36624]: pgmap v3588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:18.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:18 smithi081 ceph-mon[31453]: pgmap v3588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:20.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:20 smithi153 ceph-mon[36569]: pgmap v3589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:20.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:20 smithi104 ceph-mon[36624]: pgmap v3589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:20.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:20 smithi081 ceph-mon[31453]: pgmap v3589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:22.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:22 smithi153 ceph-mon[36569]: pgmap v3590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:22.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:22 smithi104 ceph-mon[36624]: pgmap v3590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:22.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:22 smithi081 ceph-mon[31453]: pgmap v3590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:24.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:24 smithi104 ceph-mon[36624]: pgmap v3591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:24.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:24 smithi081 ceph-mon[31453]: pgmap v3591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:25.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:24 smithi153 ceph-mon[36569]: pgmap v3591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:26.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:26 smithi104 ceph-mon[36624]: pgmap v3592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:26.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:26 smithi081 ceph-mon[31453]: pgmap v3592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:27.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:26 smithi153 ceph-mon[36569]: pgmap v3592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:27.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:27.312+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:56:27.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:27.312+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:56:27.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:27.312+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:56:28.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:28 smithi104 ceph-mon[36624]: pgmap v3593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:28.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:28 smithi081 ceph-mon[31453]: pgmap v3593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:29.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:28 smithi153 ceph-mon[36569]: pgmap v3593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:30.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:30 smithi104 ceph-mon[36624]: pgmap v3594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:30.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:30 smithi081 ceph-mon[31453]: pgmap v3594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:31.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:30 smithi153 ceph-mon[36569]: pgmap v3594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:32.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:32 smithi104 ceph-mon[36624]: pgmap v3595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:32.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:32 smithi081 ceph-mon[31453]: pgmap v3595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:33.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:32 smithi153 ceph-mon[36569]: pgmap v3595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:34.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:34 smithi081 ceph-mon[31453]: pgmap v3596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:35.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:34 smithi153 ceph-mon[36569]: pgmap v3596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:35.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:34 smithi104 ceph-mon[36624]: pgmap v3596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:36.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:36 smithi081 ceph-mon[31453]: pgmap v3597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:37.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:36 smithi153 ceph-mon[36569]: pgmap v3597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:37.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:36 smithi104 ceph-mon[36624]: pgmap v3597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:38.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:38 smithi081 ceph-mon[31453]: pgmap v3598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:39.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:38 smithi153 ceph-mon[36569]: pgmap v3598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:39.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:38 smithi104 ceph-mon[36624]: pgmap v3598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:40.849 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:40 smithi153 ceph-mon[36569]: pgmap v3599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:40.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:40 smithi081 ceph-mon[31453]: pgmap v3599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:41.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:40 smithi104 ceph-mon[36624]: pgmap v3599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:42.316+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:56:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:42.316+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:56:42.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:42.316+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:56:42.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:42 smithi081 ceph-mon[31453]: pgmap v3600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:43.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:42 smithi153 ceph-mon[36569]: pgmap v3600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:43.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:42 smithi104 ceph-mon[36624]: pgmap v3600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:44.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:44 smithi081 ceph-mon[31453]: pgmap v3601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:45.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:44 smithi153 ceph-mon[36569]: pgmap v3601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:45.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:44 smithi104 ceph-mon[36624]: pgmap v3601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:46.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:46 smithi081 ceph-mon[31453]: pgmap v3602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:47.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:46 smithi153 ceph-mon[36569]: pgmap v3602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:47.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:46 smithi104 ceph-mon[36624]: pgmap v3602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:48.874 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:48 smithi104 ceph-mon[36624]: pgmap v3603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:48.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:48 smithi081 ceph-mon[31453]: pgmap v3603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:49.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:48 smithi153 ceph-mon[36569]: pgmap v3603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:50.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:50 smithi081 ceph-mon[31453]: pgmap v3604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:51.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:50 smithi153 ceph-mon[36569]: pgmap v3604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:51.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:50 smithi104 ceph-mon[36624]: pgmap v3604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:52.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:52 smithi081 ceph-mon[31453]: pgmap v3605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:53.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:52 smithi153 ceph-mon[36569]: pgmap v3605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:53.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:52 smithi104 ceph-mon[36624]: pgmap v3605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:54.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:54 smithi081 ceph-mon[31453]: pgmap v3606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:55.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:54 smithi153 ceph-mon[36569]: pgmap v3606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:55.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:54 smithi104 ceph-mon[36624]: pgmap v3606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:56.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:56 smithi081 ceph-mon[31453]: pgmap v3607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:57.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:56 smithi153 ceph-mon[36569]: pgmap v3607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:57.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:56 smithi104 ceph-mon[36624]: pgmap v3607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:57.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:57.328+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:56:57.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:57.328+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:56:57.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:56:57.328+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:56:58.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:56:58 smithi081 ceph-mon[31453]: pgmap v3608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:59.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:56:58 smithi153 ceph-mon[36569]: pgmap v3608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:56:59.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:56:58 smithi104 ceph-mon[36624]: pgmap v3608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:00.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:00 smithi081 ceph-mon[31453]: pgmap v3609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:01.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:00 smithi153 ceph-mon[36569]: pgmap v3609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:01.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:00 smithi104 ceph-mon[36624]: pgmap v3609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:03.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:02 smithi153 ceph-mon[36569]: pgmap v3610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:03.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:02 smithi104 ceph-mon[36624]: pgmap v3610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:03.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:02 smithi081 ceph-mon[31453]: pgmap v3610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:05.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:04 smithi153 ceph-mon[36569]: pgmap v3611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:05.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:04 smithi104 ceph-mon[36624]: pgmap v3611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:05.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:04 smithi081 ceph-mon[31453]: pgmap v3611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:06.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:57:06.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:57:06.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:57:07.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:06 smithi153 ceph-mon[36569]: pgmap v3612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:07.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:57:07.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:57:07.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:57:07.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:06 smithi104 ceph-mon[36624]: pgmap v3612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:07.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:57:07.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:57:07.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:57:07.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:06 smithi081 ceph-mon[31453]: pgmap v3612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:07.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:57:07.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:57:07.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:57:08.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:07 smithi153 ceph-mon[36569]: pgmap v3613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:08.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:07 smithi153 ceph-mon[36569]: pgmap v3614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:08.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:07 smithi104 ceph-mon[36624]: pgmap v3613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:08.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:07 smithi104 ceph-mon[36624]: pgmap v3614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:08.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:07 smithi081 ceph-mon[31453]: pgmap v3613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:08.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:07 smithi081 ceph-mon[31453]: pgmap v3614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:10.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:09 smithi153 ceph-mon[36569]: pgmap v3615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:10.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:09 smithi104 ceph-mon[36624]: pgmap v3615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:10.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:09 smithi081 ceph-mon[31453]: pgmap v3615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:12.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:11 smithi153 ceph-mon[36569]: pgmap v3616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:12.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:11 smithi104 ceph-mon[36624]: pgmap v3616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:12.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:11 smithi081 ceph-mon[31453]: pgmap v3616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:12.340+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:57:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:12.341+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:57:12.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:12.341+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:57:14.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:13 smithi153 ceph-mon[36569]: pgmap v3617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:14.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:13 smithi104 ceph-mon[36624]: pgmap v3617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:14.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:13 smithi081 ceph-mon[31453]: pgmap v3617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:16.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:15 smithi153 ceph-mon[36569]: pgmap v3618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:16.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:15 smithi104 ceph-mon[36624]: pgmap v3618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:16.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:15 smithi081 ceph-mon[31453]: pgmap v3618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:18.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:17 smithi153 ceph-mon[36569]: pgmap v3619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:18.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:17 smithi104 ceph-mon[36624]: pgmap v3619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:18.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:17 smithi081 ceph-mon[31453]: pgmap v3619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:20.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:19 smithi153 ceph-mon[36569]: pgmap v3620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:20.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:19 smithi104 ceph-mon[36624]: pgmap v3620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:20.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:19 smithi081 ceph-mon[31453]: pgmap v3620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:22.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:21 smithi153 ceph-mon[36569]: pgmap v3621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:22.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:21 smithi104 ceph-mon[36624]: pgmap v3621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:22.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:21 smithi081 ceph-mon[31453]: pgmap v3621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:24.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:23 smithi153 ceph-mon[36569]: pgmap v3622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:24.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:23 smithi104 ceph-mon[36624]: pgmap v3622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:24.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:23 smithi081 ceph-mon[31453]: pgmap v3622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:26.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:25 smithi153 ceph-mon[36569]: pgmap v3623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:26.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:25 smithi104 ceph-mon[36624]: pgmap v3623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:26.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:25 smithi081 ceph-mon[31453]: pgmap v3623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:27.351+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:57:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:27.351+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:57:27.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:27.351+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:57:28.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:27 smithi153 ceph-mon[36569]: pgmap v3624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:28.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:27 smithi104 ceph-mon[36624]: pgmap v3624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:28.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:27 smithi081 ceph-mon[31453]: pgmap v3624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:30.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:29 smithi153 ceph-mon[36569]: pgmap v3625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:30.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:29 smithi104 ceph-mon[36624]: pgmap v3625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:30.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:29 smithi081 ceph-mon[31453]: pgmap v3625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:32.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:31 smithi153 ceph-mon[36569]: pgmap v3626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:32.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:31 smithi104 ceph-mon[36624]: pgmap v3626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:32.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:31 smithi081 ceph-mon[31453]: pgmap v3626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:34.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:33 smithi153 ceph-mon[36569]: pgmap v3627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:34.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:33 smithi104 ceph-mon[36624]: pgmap v3627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:34.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:33 smithi081 ceph-mon[31453]: pgmap v3627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:36.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:35 smithi153 ceph-mon[36569]: pgmap v3628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:36.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:35 smithi104 ceph-mon[36624]: pgmap v3628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:36.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:35 smithi081 ceph-mon[31453]: pgmap v3628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:38.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:37 smithi153 ceph-mon[36569]: pgmap v3629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:38.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:37 smithi104 ceph-mon[36624]: pgmap v3629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:38.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:37 smithi081 ceph-mon[31453]: pgmap v3629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:40.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:39 smithi153 ceph-mon[36569]: pgmap v3630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:40.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:39 smithi104 ceph-mon[36624]: pgmap v3630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:40.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:39 smithi081 ceph-mon[31453]: pgmap v3630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:42.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:41 smithi153 ceph-mon[36569]: pgmap v3631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:42.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:41 smithi104 ceph-mon[36624]: pgmap v3631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:42.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:41 smithi081 ceph-mon[31453]: pgmap v3631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:42.351+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:57:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:42.351+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:57:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:42.351+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:57:44.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:43 smithi153 ceph-mon[36569]: pgmap v3632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:44.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:43 smithi104 ceph-mon[36624]: pgmap v3632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:44.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:43 smithi081 ceph-mon[31453]: pgmap v3632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:46.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:45 smithi153 ceph-mon[36569]: pgmap v3633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:46.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:45 smithi104 ceph-mon[36624]: pgmap v3633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:46.109 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:45 smithi081 ceph-mon[31453]: pgmap v3633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:48.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:47 smithi104 ceph-mon[36624]: pgmap v3634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:48.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:47 smithi081 ceph-mon[31453]: pgmap v3634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:48.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:47 smithi153 ceph-mon[36569]: pgmap v3634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:50.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:49 smithi104 ceph-mon[36624]: pgmap v3635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:50.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:49 smithi081 ceph-mon[31453]: pgmap v3635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:50.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:49 smithi153 ceph-mon[36569]: pgmap v3635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:51.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:50 smithi104 ceph-mon[36624]: pgmap v3636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:51.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:50 smithi081 ceph-mon[31453]: pgmap v3636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:51.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:50 smithi153 ceph-mon[36569]: pgmap v3636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:53.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:53 smithi153 ceph-mon[36569]: pgmap v3637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:53.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:53 smithi104 ceph-mon[36624]: pgmap v3637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:53.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:53 smithi081 ceph-mon[31453]: pgmap v3637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:55.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:55 smithi153 ceph-mon[36569]: pgmap v3638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:55.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:55 smithi104 ceph-mon[36624]: pgmap v3638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:55.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:55 smithi081 ceph-mon[31453]: pgmap v3638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:57.357+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:57:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:57.357+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:57:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:57:57.357+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:57:57.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:57 smithi153 ceph-mon[36569]: pgmap v3639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:57.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:57 smithi104 ceph-mon[36624]: pgmap v3639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:57.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:57 smithi081 ceph-mon[31453]: pgmap v3639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:59.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:57:59 smithi153 ceph-mon[36569]: pgmap v3640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:59.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:57:59 smithi104 ceph-mon[36624]: pgmap v3640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:57:59.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:57:59 smithi081 ceph-mon[31453]: pgmap v3640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:01.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:01 smithi153 ceph-mon[36569]: pgmap v3641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:01.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:01 smithi104 ceph-mon[36624]: pgmap v3641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:01.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:01 smithi081 ceph-mon[31453]: pgmap v3641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:03.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:03 smithi153 ceph-mon[36569]: pgmap v3642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:03.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:03 smithi104 ceph-mon[36624]: pgmap v3642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:03.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:03 smithi081 ceph-mon[31453]: pgmap v3642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:05.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:05 smithi153 ceph-mon[36569]: pgmap v3643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:05.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:05 smithi104 ceph-mon[36624]: pgmap v3643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:05.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:05 smithi081 ceph-mon[31453]: pgmap v3643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:07.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:07 smithi153 ceph-mon[36569]: pgmap v3644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:07.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:58:07.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:07 smithi104 ceph-mon[36624]: pgmap v3644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:07.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:58:07.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:07 smithi081 ceph-mon[31453]: pgmap v3644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:07.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:58:08.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:58:08.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:58:08.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:58:08.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:58:08.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:58:08.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:58:08.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:58:08.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:58:08.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:58:09.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:09 smithi153 ceph-mon[36569]: pgmap v3645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:09.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:09 smithi153 ceph-mon[36569]: pgmap v3646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:09.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:09 smithi104 ceph-mon[36624]: pgmap v3645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:09.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:09 smithi104 ceph-mon[36624]: pgmap v3646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:09.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:09 smithi081 ceph-mon[31453]: pgmap v3645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:09.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:09 smithi081 ceph-mon[31453]: pgmap v3646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:11.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:11 smithi153 ceph-mon[36569]: pgmap v3647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:11.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:11 smithi104 ceph-mon[36624]: pgmap v3647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:11.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:11 smithi081 ceph-mon[31453]: pgmap v3647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:12.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:12.363+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:58:12.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:12.363+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:58:12.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:12.363+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:58:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:13.704+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:58:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:13.704+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:58:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:13.705+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T22:58:13.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:13.705+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T22:58:13.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:13 smithi153 ceph-mon[36569]: pgmap v3648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:13.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:13 smithi104 ceph-mon[36624]: pgmap v3648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:13.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:13 smithi081 ceph-mon[31453]: pgmap v3648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:15.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:15 smithi153 ceph-mon[36569]: pgmap v3649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:15.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:15 smithi104 ceph-mon[36624]: pgmap v3649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:15.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:15 smithi081 ceph-mon[31453]: pgmap v3649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:17.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:17 smithi081 ceph-mon[31453]: pgmap v3650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:17.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:17 smithi153 ceph-mon[36569]: pgmap v3650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:17.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:17 smithi104 ceph-mon[36624]: pgmap v3650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:19.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:19 smithi153 ceph-mon[36569]: pgmap v3651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:19.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:19 smithi104 ceph-mon[36624]: pgmap v3651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:19.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:19 smithi081 ceph-mon[31453]: pgmap v3651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:21.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:21 smithi153 ceph-mon[36569]: pgmap v3652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:21.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:21 smithi104 ceph-mon[36624]: pgmap v3652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:21.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:21 smithi081 ceph-mon[31453]: pgmap v3652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:23 smithi104 ceph-mon[36624]: pgmap v3653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:23 smithi081 ceph-mon[31453]: pgmap v3653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:24.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:23 smithi153 ceph-mon[36569]: pgmap v3653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:25.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:25 smithi104 ceph-mon[36624]: pgmap v3654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:25.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:25 smithi081 ceph-mon[31453]: pgmap v3654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:26.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:25 smithi153 ceph-mon[36569]: pgmap v3654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:27.379+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:58:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:27.379+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:58:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:27.379+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:58:27.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:27 smithi104 ceph-mon[36624]: pgmap v3655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:27.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:27 smithi081 ceph-mon[31453]: pgmap v3655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:28.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:27 smithi153 ceph-mon[36569]: pgmap v3655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:29.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:29 smithi104 ceph-mon[36624]: pgmap v3656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:29.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:29 smithi081 ceph-mon[31453]: pgmap v3656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:30.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:29 smithi153 ceph-mon[36569]: pgmap v3656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:31.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:31 smithi104 ceph-mon[36624]: pgmap v3657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:31.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:31 smithi081 ceph-mon[31453]: pgmap v3657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:32.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:31 smithi153 ceph-mon[36569]: pgmap v3657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:33.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:33 smithi081 ceph-mon[31453]: pgmap v3658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:34.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:33 smithi153 ceph-mon[36569]: pgmap v3658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:34.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:33 smithi104 ceph-mon[36624]: pgmap v3658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:35.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:35 smithi081 ceph-mon[31453]: pgmap v3659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:36.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:35 smithi153 ceph-mon[36569]: pgmap v3659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:36.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:35 smithi104 ceph-mon[36624]: pgmap v3659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:37.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:37 smithi081 ceph-mon[31453]: pgmap v3660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:38.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:37 smithi153 ceph-mon[36569]: pgmap v3660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:38.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:37 smithi104 ceph-mon[36624]: pgmap v3660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:39.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:39 smithi081 ceph-mon[31453]: pgmap v3661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:40.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:39 smithi153 ceph-mon[36569]: pgmap v3661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:40.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:39 smithi104 ceph-mon[36624]: pgmap v3661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:41.848 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:41 smithi153 ceph-mon[36569]: pgmap v3662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:41.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:41 smithi081 ceph-mon[31453]: pgmap v3662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:42.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:41 smithi104 ceph-mon[36624]: pgmap v3662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:42.394+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:58:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:42.394+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:58:42.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:42.394+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:58:43.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:43 smithi081 ceph-mon[31453]: pgmap v3663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:44.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:43 smithi153 ceph-mon[36569]: pgmap v3663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:44.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:43 smithi104 ceph-mon[36624]: pgmap v3663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:45.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:45 smithi081 ceph-mon[31453]: pgmap v3664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:46.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:45 smithi153 ceph-mon[36569]: pgmap v3664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:46.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:45 smithi104 ceph-mon[36624]: pgmap v3664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:47.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:47 smithi081 ceph-mon[31453]: pgmap v3665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:48.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:47 smithi153 ceph-mon[36569]: pgmap v3665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:48.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:47 smithi104 ceph-mon[36624]: pgmap v3665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:49.874 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:49 smithi104 ceph-mon[36624]: pgmap v3666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:49.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:49 smithi081 ceph-mon[31453]: pgmap v3666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:50.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:49 smithi153 ceph-mon[36569]: pgmap v3666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:51.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:51 smithi081 ceph-mon[31453]: pgmap v3667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:52.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:51 smithi153 ceph-mon[36569]: pgmap v3667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:52.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:51 smithi104 ceph-mon[36624]: pgmap v3667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:53.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:53 smithi081 ceph-mon[31453]: pgmap v3668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:54.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:53 smithi153 ceph-mon[36569]: pgmap v3668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:54.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:53 smithi104 ceph-mon[36624]: pgmap v3668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:55.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:55 smithi081 ceph-mon[31453]: pgmap v3669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:56.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:55 smithi153 ceph-mon[36569]: pgmap v3669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:56.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:55 smithi104 ceph-mon[36624]: pgmap v3669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:57.409+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:58:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:57.409+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:58:57.399 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:58:57.409+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:58:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:57 smithi081 ceph-mon[31453]: pgmap v3670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:58.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:57 smithi153 ceph-mon[36569]: pgmap v3670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:58.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:57 smithi104 ceph-mon[36624]: pgmap v3670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:58:59.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:58:59 smithi081 ceph-mon[31453]: pgmap v3671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:00.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:58:59 smithi153 ceph-mon[36569]: pgmap v3671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:00.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:58:59 smithi104 ceph-mon[36624]: pgmap v3671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:01.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:01 smithi081 ceph-mon[31453]: pgmap v3672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:02.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:01 smithi153 ceph-mon[36569]: pgmap v3672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:02.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:01 smithi104 ceph-mon[36624]: pgmap v3672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:04.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:03 smithi153 ceph-mon[36569]: pgmap v3673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:04.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:03 smithi104 ceph-mon[36624]: pgmap v3673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:04.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:03 smithi081 ceph-mon[31453]: pgmap v3673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:06.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:05 smithi153 ceph-mon[36569]: pgmap v3674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:06.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:05 smithi104 ceph-mon[36624]: pgmap v3674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:06.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:05 smithi081 ceph-mon[31453]: pgmap v3674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:08.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:07 smithi153 ceph-mon[36569]: pgmap v3675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:08.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:07 smithi104 ceph-mon[36624]: pgmap v3675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:08.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:07 smithi081 ceph-mon[31453]: pgmap v3675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:09.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:59:09.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:59:09.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T22:59:10.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:09 smithi153 ceph-mon[36569]: pgmap v3676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:10.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:59:10.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:59:10.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:59:10.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:09 smithi104 ceph-mon[36624]: pgmap v3676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:10.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:59:10.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:59:10.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:59:10.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:09 smithi081 ceph-mon[31453]: pgmap v3676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:10.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T22:59:10.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T22:59:10.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T22:59:11.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:10 smithi153 ceph-mon[36569]: pgmap v3677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:11.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:10 smithi153 ceph-mon[36569]: pgmap v3678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:11.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:10 smithi104 ceph-mon[36624]: pgmap v3677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:11.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:10 smithi104 ceph-mon[36624]: pgmap v3678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:11.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:10 smithi081 ceph-mon[31453]: pgmap v3677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:11.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:10 smithi081 ceph-mon[31453]: pgmap v3678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:12.425+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:59:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:12.425+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:59:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:12.425+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:59:13.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:12 smithi153 ceph-mon[36569]: pgmap v3679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:13.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:12 smithi104 ceph-mon[36624]: pgmap v3679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:13.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:12 smithi081 ceph-mon[31453]: pgmap v3679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:15.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:14 smithi153 ceph-mon[36569]: pgmap v3680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:15.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:14 smithi104 ceph-mon[36624]: pgmap v3680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:15.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:14 smithi081 ceph-mon[31453]: pgmap v3680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:17.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:16 smithi153 ceph-mon[36569]: pgmap v3681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:17.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:16 smithi104 ceph-mon[36624]: pgmap v3681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:17.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:16 smithi081 ceph-mon[31453]: pgmap v3681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:19.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:18 smithi153 ceph-mon[36569]: pgmap v3682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:19.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:18 smithi104 ceph-mon[36624]: pgmap v3682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:19.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:18 smithi081 ceph-mon[31453]: pgmap v3682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:21.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:20 smithi153 ceph-mon[36569]: pgmap v3683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:21.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:20 smithi104 ceph-mon[36624]: pgmap v3683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:21.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:20 smithi081 ceph-mon[31453]: pgmap v3683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:23.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:22 smithi153 ceph-mon[36569]: pgmap v3684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:23.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:22 smithi104 ceph-mon[36624]: pgmap v3684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:23.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:22 smithi081 ceph-mon[31453]: pgmap v3684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:25.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:24 smithi153 ceph-mon[36569]: pgmap v3685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:25.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:24 smithi104 ceph-mon[36624]: pgmap v3685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:25.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:24 smithi081 ceph-mon[31453]: pgmap v3685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:27.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:26 smithi153 ceph-mon[36569]: pgmap v3686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:27.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:26 smithi104 ceph-mon[36624]: pgmap v3686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:26 smithi081 ceph-mon[31453]: pgmap v3686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:27.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:27.438+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:59:27.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:27.438+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:59:27.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:27.438+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:59:29.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:28 smithi153 ceph-mon[36569]: pgmap v3687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:29.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:28 smithi104 ceph-mon[36624]: pgmap v3687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:28 smithi081 ceph-mon[31453]: pgmap v3687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:31.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:30 smithi153 ceph-mon[36569]: pgmap v3688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:31.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:30 smithi104 ceph-mon[36624]: pgmap v3688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:31.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:30 smithi081 ceph-mon[31453]: pgmap v3688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:33.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:32 smithi153 ceph-mon[36569]: pgmap v3689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:33.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:32 smithi104 ceph-mon[36624]: pgmap v3689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:33.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:32 smithi081 ceph-mon[31453]: pgmap v3689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:35.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:34 smithi153 ceph-mon[36569]: pgmap v3690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:35.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:34 smithi104 ceph-mon[36624]: pgmap v3690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:35.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:34 smithi081 ceph-mon[31453]: pgmap v3690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:37.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:36 smithi153 ceph-mon[36569]: pgmap v3691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:37.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:36 smithi104 ceph-mon[36624]: pgmap v3691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:37.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:36 smithi081 ceph-mon[31453]: pgmap v3691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:39.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:38 smithi153 ceph-mon[36569]: pgmap v3692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:39.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:38 smithi104 ceph-mon[36624]: pgmap v3692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:39.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:38 smithi081 ceph-mon[31453]: pgmap v3692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:41.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:40 smithi153 ceph-mon[36569]: pgmap v3693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:41.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:40 smithi104 ceph-mon[36624]: pgmap v3693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:41.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:40 smithi081 ceph-mon[31453]: pgmap v3693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:42.439 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:42.449+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:59:42.439 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:42.449+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:59:42.439 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:42.449+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:59:43.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:42 smithi153 ceph-mon[36569]: pgmap v3694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:43.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:42 smithi104 ceph-mon[36624]: pgmap v3694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:43.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:42 smithi081 ceph-mon[31453]: pgmap v3694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:45.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:44 smithi153 ceph-mon[36569]: pgmap v3695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:45.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:44 smithi104 ceph-mon[36624]: pgmap v3695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:45.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:44 smithi081 ceph-mon[31453]: pgmap v3695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:47.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:46 smithi153 ceph-mon[36569]: pgmap v3696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:47.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:46 smithi104 ceph-mon[36624]: pgmap v3696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:47.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:46 smithi081 ceph-mon[31453]: pgmap v3696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:49.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:48 smithi153 ceph-mon[36569]: pgmap v3697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:49.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:48 smithi104 ceph-mon[36624]: pgmap v3697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:49.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:48 smithi081 ceph-mon[31453]: pgmap v3697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:51.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:50 smithi153 ceph-mon[36569]: pgmap v3698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:51.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:50 smithi104 ceph-mon[36624]: pgmap v3698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:51.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:50 smithi081 ceph-mon[31453]: pgmap v3698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:53.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:52 smithi153 ceph-mon[36569]: pgmap v3699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:53.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:52 smithi104 ceph-mon[36624]: pgmap v3699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:53.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:52 smithi081 ceph-mon[31453]: pgmap v3699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:55.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:54 smithi104 ceph-mon[36624]: pgmap v3700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:55.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:54 smithi081 ceph-mon[31453]: pgmap v3700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:55.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:54 smithi153 ceph-mon[36569]: pgmap v3700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:57.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:56 smithi104 ceph-mon[36624]: pgmap v3701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:57.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:56 smithi081 ceph-mon[31453]: pgmap v3701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:57.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:56 smithi153 ceph-mon[36569]: pgmap v3701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:57.462+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T22:59:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:57.462+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T22:59:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T22:59:57.462+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T22:59:59.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 22:59:58 smithi104 ceph-mon[36624]: pgmap v3702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:59.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 22:59:58 smithi081 ceph-mon[31453]: pgmap v3702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T22:59:59.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 22:59:58 smithi153 ceph-mon[36569]: pgmap v3702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:01.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:00 smithi104 ceph-mon[36624]: pgmap v3703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:01.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:00 smithi081 ceph-mon[31453]: pgmap v3703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:00:01.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:00 smithi153 ceph-mon[36569]: pgmap v3703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:01.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:00:03.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:02 smithi081 ceph-mon[31453]: pgmap v3704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:03.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:02 smithi153 ceph-mon[36569]: pgmap v3704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:03.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:02 smithi104 ceph-mon[36624]: pgmap v3704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:05.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:04 smithi081 ceph-mon[31453]: pgmap v3705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:04 smithi153 ceph-mon[36569]: pgmap v3705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:05.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:04 smithi104 ceph-mon[36624]: pgmap v3705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:07.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:06 smithi081 ceph-mon[31453]: pgmap v3706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:07.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:06 smithi153 ceph-mon[36569]: pgmap v3706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:07.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:06 smithi104 ceph-mon[36624]: pgmap v3706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:08.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:07 smithi081 ceph-mon[31453]: pgmap v3707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:08.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:07 smithi153 ceph-mon[36569]: pgmap v3707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:08.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:07 smithi104 ceph-mon[36624]: pgmap v3707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:10.482 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:10 smithi153 ceph-mon[36569]: pgmap v3708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:10.482 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:00:10.482 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:10.482 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:10.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:10 smithi081 ceph-mon[31453]: pgmap v3708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:10.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:00:10.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:10.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:10.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:10 smithi104 ceph-mon[36624]: pgmap v3708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:10.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:00:10.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:10.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:11.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:11.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:11.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:11.792 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:11.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:11.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:12.477+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:00:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:12.477+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:00:12.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:12.477+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:00:12.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:12 smithi153 ceph-mon[36569]: pgmap v3709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:12.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:12.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:12.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:12 smithi104 ceph-mon[36624]: pgmap v3709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:12.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:12.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:12.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:12 smithi081 ceph-mon[31453]: pgmap v3709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:12.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:12.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:14.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:14.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:14.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:14.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:14.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:14.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:15.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:14 smithi153 ceph-mon[36569]: pgmap v3710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:15.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:14 smithi104 ceph-mon[36624]: pgmap v3710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:15.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:14 smithi081 ceph-mon[31453]: pgmap v3710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:16.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:16.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:16.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:15 smithi081 ceph-mon[31453]: pgmap v3711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:16.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:16.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:16.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:15 smithi153 ceph-mon[36569]: pgmap v3711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:16.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:16.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:16.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:15 smithi104 ceph-mon[36624]: pgmap v3711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:17.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:00:17.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:00:17.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:16 smithi081 ceph-mon[31453]: pgmap v3712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:17.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:16 smithi081 ceph-mon[31453]: pgmap v3713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:17.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:17.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:00:17.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:00:17.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:16 smithi153 ceph-mon[36569]: pgmap v3712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:17.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:16 smithi153 ceph-mon[36569]: pgmap v3713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:17.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:17.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:00:17.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:00:17.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:16 smithi104 ceph-mon[36624]: pgmap v3712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:17.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:16 smithi104 ceph-mon[36624]: pgmap v3713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:17.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:00:19.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:19 smithi153 ceph-mon[36569]: pgmap v3714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:19.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:19 smithi104 ceph-mon[36624]: pgmap v3714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:19.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:19 smithi081 ceph-mon[31453]: pgmap v3714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:21.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:21 smithi153 ceph-mon[36569]: pgmap v3715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:21.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:21 smithi104 ceph-mon[36624]: pgmap v3715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:21.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:21 smithi081 ceph-mon[31453]: pgmap v3715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:23.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:23 smithi153 ceph-mon[36569]: pgmap v3716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:23 smithi104 ceph-mon[36624]: pgmap v3716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:23 smithi081 ceph-mon[31453]: pgmap v3716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:25.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:25 smithi153 ceph-mon[36569]: pgmap v3717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:25.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:25 smithi104 ceph-mon[36624]: pgmap v3717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:25.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:25 smithi081 ceph-mon[31453]: pgmap v3717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:27.492+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:00:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:27.492+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:00:27.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:27.493+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:00:27.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:27 smithi153 ceph-mon[36569]: pgmap v3718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:27.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:27 smithi104 ceph-mon[36624]: pgmap v3718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:27.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:27 smithi081 ceph-mon[31453]: pgmap v3718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:29.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:29 smithi104 ceph-mon[36624]: pgmap v3719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:29.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:29 smithi081 ceph-mon[31453]: pgmap v3719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:30.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:29 smithi153 ceph-mon[36569]: pgmap v3719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:31.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:31 smithi104 ceph-mon[36624]: pgmap v3720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:31.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:31 smithi081 ceph-mon[31453]: pgmap v3720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:32.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:31 smithi153 ceph-mon[36569]: pgmap v3720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:33.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:33 smithi104 ceph-mon[36624]: pgmap v3721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:33.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:33 smithi081 ceph-mon[31453]: pgmap v3721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:34.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:33 smithi153 ceph-mon[36569]: pgmap v3721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:35.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:35 smithi104 ceph-mon[36624]: pgmap v3722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:35.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:35 smithi081 ceph-mon[31453]: pgmap v3722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:36.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:35 smithi153 ceph-mon[36569]: pgmap v3722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:37.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:37 smithi104 ceph-mon[36624]: pgmap v3723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:37.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:37 smithi081 ceph-mon[31453]: pgmap v3723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:38.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:37 smithi153 ceph-mon[36569]: pgmap v3723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:39.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:39 smithi081 ceph-mon[31453]: pgmap v3724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:40.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:39 smithi153 ceph-mon[36569]: pgmap v3724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:40.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:39 smithi104 ceph-mon[36624]: pgmap v3724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:41.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:41 smithi081 ceph-mon[31453]: pgmap v3725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:42.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:41 smithi153 ceph-mon[36569]: pgmap v3725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:42.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:41 smithi104 ceph-mon[36624]: pgmap v3725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:42.508+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:00:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:42.508+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:00:42.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:42.508+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:00:43.848 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:43 smithi153 ceph-mon[36569]: pgmap v3726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:43.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:43 smithi081 ceph-mon[31453]: pgmap v3726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:44.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:43 smithi104 ceph-mon[36624]: pgmap v3726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:45.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:45 smithi081 ceph-mon[31453]: pgmap v3727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:46.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:45 smithi153 ceph-mon[36569]: pgmap v3727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:46.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:45 smithi104 ceph-mon[36624]: pgmap v3727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:47.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:47 smithi081 ceph-mon[31453]: pgmap v3728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:48.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:47 smithi153 ceph-mon[36569]: pgmap v3728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:48.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:47 smithi104 ceph-mon[36624]: pgmap v3728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:49.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:49 smithi081 ceph-mon[31453]: pgmap v3729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:50.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:49 smithi153 ceph-mon[36569]: pgmap v3729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:50.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:49 smithi104 ceph-mon[36624]: pgmap v3729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:51.873 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:51 smithi104 ceph-mon[36624]: pgmap v3730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:51.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:51 smithi081 ceph-mon[31453]: pgmap v3730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:52.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:51 smithi153 ceph-mon[36569]: pgmap v3730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:53.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:53 smithi081 ceph-mon[31453]: pgmap v3731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:54.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:53 smithi153 ceph-mon[36569]: pgmap v3731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:54.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:53 smithi104 ceph-mon[36624]: pgmap v3731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:55.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:55 smithi081 ceph-mon[31453]: pgmap v3732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:56.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:55 smithi153 ceph-mon[36569]: pgmap v3732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:56.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:55 smithi104 ceph-mon[36624]: pgmap v3732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:57.524+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:00:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:57.524+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:00:57.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:00:57.524+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:00:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:57 smithi081 ceph-mon[31453]: pgmap v3733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:58.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:57 smithi153 ceph-mon[36569]: pgmap v3733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:58.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:57 smithi104 ceph-mon[36624]: pgmap v3733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:00:59.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:00:59 smithi081 ceph-mon[31453]: pgmap v3734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:00.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:00:59 smithi153 ceph-mon[36569]: pgmap v3734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:00.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:00:59 smithi104 ceph-mon[36624]: pgmap v3734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:01.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:01 smithi153 ceph-mon[36569]: pgmap v3735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:01.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:01 smithi081 ceph-mon[31453]: pgmap v3735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:02.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:01 smithi104 ceph-mon[36624]: pgmap v3735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:03.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:03 smithi081 ceph-mon[31453]: pgmap v3736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:03 smithi153 ceph-mon[36569]: pgmap v3736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:04.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:03 smithi104 ceph-mon[36624]: pgmap v3736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:05.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:05 smithi081 ceph-mon[31453]: pgmap v3737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:06.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:05 smithi153 ceph-mon[36569]: pgmap v3737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:06.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:05 smithi104 ceph-mon[36624]: pgmap v3737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:08.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:07 smithi153 ceph-mon[36569]: pgmap v3738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:08.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:07 smithi104 ceph-mon[36624]: pgmap v3738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:08.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:07 smithi081 ceph-mon[31453]: pgmap v3738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:10.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:09 smithi153 ceph-mon[36569]: pgmap v3739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:10.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:09 smithi104 ceph-mon[36624]: pgmap v3739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:10.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:09 smithi081 ceph-mon[31453]: pgmap v3739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:12.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:11 smithi153 ceph-mon[36569]: pgmap v3740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:12.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:11 smithi104 ceph-mon[36624]: pgmap v3740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.103+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T23:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.103+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T23:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.103+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f16d40065f0 tx=0x7f16f40bd040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T23:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.103+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f16d40065f0 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-12T23:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.103+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T23:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.103+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T23:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.103+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T23:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.104+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T23:01:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.105+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006620 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T23:01:12.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:11 smithi081 ceph-mon[31453]: pgmap v3740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:12.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.533+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:01:12.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.533+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:01:12.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:12.533+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:01:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:13.733+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:01:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:13.733+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:01:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:13.734+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:01:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:13.734+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:01:14.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:13 smithi153 ceph-mon[36569]: pgmap v3741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:14.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:13 smithi104 ceph-mon[36624]: pgmap v3741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:14.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:13 smithi081 ceph-mon[31453]: pgmap v3741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:16.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:15 smithi153 ceph-mon[36569]: pgmap v3742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:16.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:15 smithi104 ceph-mon[36624]: pgmap v3742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:16.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:15 smithi081 ceph-mon[31453]: pgmap v3742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:17.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:01:17.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:01:17.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:01:18.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:17 smithi153 ceph-mon[36569]: pgmap v3743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:18.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:18.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:18.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:17 smithi104 ceph-mon[36624]: pgmap v3743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:18.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:18.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:18.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:17 smithi081 ceph-mon[31453]: pgmap v3743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:18.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:18.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:19.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:01:19.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:01:19.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:19.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:01:19.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:01:19.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:19.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:01:19.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:01:19.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:01:20.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:19 smithi153 ceph-mon[36569]: pgmap v3744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:20.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:19 smithi153 ceph-mon[36569]: pgmap v3745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:20.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:19 smithi104 ceph-mon[36624]: pgmap v3744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:20.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:19 smithi104 ceph-mon[36624]: pgmap v3745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:20.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:19 smithi081 ceph-mon[31453]: pgmap v3744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:20.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:19 smithi081 ceph-mon[31453]: pgmap v3745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:22.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:21 smithi153 ceph-mon[36569]: pgmap v3746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:22.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:21 smithi104 ceph-mon[36624]: pgmap v3746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:22.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:21 smithi081 ceph-mon[31453]: pgmap v3746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:24.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:23 smithi153 ceph-mon[36569]: pgmap v3747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:24.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:23 smithi104 ceph-mon[36624]: pgmap v3747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:24.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:23 smithi081 ceph-mon[31453]: pgmap v3747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:26.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:25 smithi153 ceph-mon[36569]: pgmap v3748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:26.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:25 smithi104 ceph-mon[36624]: pgmap v3748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:25 smithi081 ceph-mon[31453]: pgmap v3748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:27.549+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:01:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:27.549+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:01:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:27.549+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:01:28.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:27 smithi153 ceph-mon[36569]: pgmap v3749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:28.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:27 smithi104 ceph-mon[36624]: pgmap v3749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:28.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:27 smithi081 ceph-mon[31453]: pgmap v3749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:30.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:29 smithi153 ceph-mon[36569]: pgmap v3750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:30.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:29 smithi104 ceph-mon[36624]: pgmap v3750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:30.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:29 smithi081 ceph-mon[31453]: pgmap v3750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:32.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:31 smithi153 ceph-mon[36569]: pgmap v3751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:32.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:31 smithi104 ceph-mon[36624]: pgmap v3751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:32.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:31 smithi081 ceph-mon[31453]: pgmap v3751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:34.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:33 smithi153 ceph-mon[36569]: pgmap v3752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:34.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:33 smithi104 ceph-mon[36624]: pgmap v3752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:34.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:33 smithi081 ceph-mon[31453]: pgmap v3752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:36.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:35 smithi153 ceph-mon[36569]: pgmap v3753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:36.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:35 smithi104 ceph-mon[36624]: pgmap v3753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:36.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:35 smithi081 ceph-mon[31453]: pgmap v3753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:38.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:37 smithi153 ceph-mon[36569]: pgmap v3754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:38.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:37 smithi104 ceph-mon[36624]: pgmap v3754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:38.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:37 smithi081 ceph-mon[31453]: pgmap v3754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:40.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:39 smithi153 ceph-mon[36569]: pgmap v3755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:40.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:39 smithi104 ceph-mon[36624]: pgmap v3755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:40.092 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:39 smithi081 ceph-mon[31453]: pgmap v3755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:42.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:41 smithi153 ceph-mon[36569]: pgmap v3756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:42.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:41 smithi104 ceph-mon[36624]: pgmap v3756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:42.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:41 smithi081 ceph-mon[31453]: pgmap v3756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:42.562+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:01:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:42.562+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:01:42.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:42.562+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:01:44.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:43 smithi153 ceph-mon[36569]: pgmap v3757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:44.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:43 smithi104 ceph-mon[36624]: pgmap v3757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:44.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:43 smithi081 ceph-mon[31453]: pgmap v3757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:46.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:45 smithi153 ceph-mon[36569]: pgmap v3758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:46.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:45 smithi104 ceph-mon[36624]: pgmap v3758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:46.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:45 smithi081 ceph-mon[31453]: pgmap v3758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:48.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:47 smithi153 ceph-mon[36569]: pgmap v3759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:48.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:47 smithi104 ceph-mon[36624]: pgmap v3759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:48.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:47 smithi081 ceph-mon[31453]: pgmap v3759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:50.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:49 smithi153 ceph-mon[36569]: pgmap v3760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:50.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:49 smithi104 ceph-mon[36624]: pgmap v3760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:50.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:49 smithi081 ceph-mon[31453]: pgmap v3760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:52.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:51 smithi153 ceph-mon[36569]: pgmap v3761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:52.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:51 smithi104 ceph-mon[36624]: pgmap v3761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:52.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:51 smithi081 ceph-mon[31453]: pgmap v3761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:54.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:53 smithi153 ceph-mon[36569]: pgmap v3762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:54.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:53 smithi104 ceph-mon[36624]: pgmap v3762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:54.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:53 smithi081 ceph-mon[31453]: pgmap v3762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:56.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:55 smithi153 ceph-mon[36569]: pgmap v3763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:56.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:55 smithi104 ceph-mon[36624]: pgmap v3763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:56.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:55 smithi081 ceph-mon[31453]: pgmap v3763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:57.577+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:01:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:57.577+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:01:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:01:57.577+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:01:58.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:57 smithi104 ceph-mon[36624]: pgmap v3764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:58.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:57 smithi081 ceph-mon[31453]: pgmap v3764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:01:58.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:57 smithi153 ceph-mon[36569]: pgmap v3764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:00.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:01:59 smithi104 ceph-mon[36624]: pgmap v3765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:00.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:01:59 smithi081 ceph-mon[31453]: pgmap v3765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:00.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:01:59 smithi153 ceph-mon[36569]: pgmap v3765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:02.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:01 smithi104 ceph-mon[36624]: pgmap v3766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:02.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:01 smithi081 ceph-mon[31453]: pgmap v3766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:02.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:01 smithi153 ceph-mon[36569]: pgmap v3766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:04.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:03 smithi104 ceph-mon[36624]: pgmap v3767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:04.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:03 smithi081 ceph-mon[31453]: pgmap v3767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:04.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:03 smithi153 ceph-mon[36569]: pgmap v3767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:06.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:05 smithi104 ceph-mon[36624]: pgmap v3768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:06.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:05 smithi081 ceph-mon[31453]: pgmap v3768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:06.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:05 smithi153 ceph-mon[36569]: pgmap v3768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:08.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:07 smithi081 ceph-mon[31453]: pgmap v3769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:08.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:07 smithi153 ceph-mon[36569]: pgmap v3769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:08.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:07 smithi104 ceph-mon[36624]: pgmap v3769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:10.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:09 smithi081 ceph-mon[31453]: pgmap v3770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:10.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:09 smithi153 ceph-mon[36569]: pgmap v3770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:10.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:09 smithi104 ceph-mon[36624]: pgmap v3770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:12.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:11 smithi081 ceph-mon[31453]: pgmap v3771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:12.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:11 smithi153 ceph-mon[36569]: pgmap v3771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:12.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:11 smithi104 ceph-mon[36624]: pgmap v3771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:12.583+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:02:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:12.584+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:02:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:12.584+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:02:14.098 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:13 smithi153 ceph-mon[36569]: pgmap v3772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:14.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:13 smithi081 ceph-mon[31453]: pgmap v3772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:14.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:13 smithi104 ceph-mon[36624]: pgmap v3772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:16.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:15 smithi081 ceph-mon[31453]: pgmap v3773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:16.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:15 smithi153 ceph-mon[36569]: pgmap v3773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:16.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:15 smithi104 ceph-mon[36624]: pgmap v3773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:18.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:17 smithi081 ceph-mon[31453]: pgmap v3774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:18.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:17 smithi153 ceph-mon[36569]: pgmap v3774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:18.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:17 smithi104 ceph-mon[36624]: pgmap v3774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:19.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:02:19.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:02:19.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:02:20.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:19 smithi081 ceph-mon[31453]: pgmap v3775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:20.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:02:20.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:02:20.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:02:20.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:19 smithi153 ceph-mon[36569]: pgmap v3775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:20.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:02:20.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:02:20.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:02:20.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:19 smithi104 ceph-mon[36624]: pgmap v3775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:20.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:02:20.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:02:20.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:02:21.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:20 smithi081 ceph-mon[31453]: pgmap v3776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:21.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:20 smithi081 ceph-mon[31453]: pgmap v3777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:21.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:20 smithi104 ceph-mon[36624]: pgmap v3776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:21.222 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:20 smithi104 ceph-mon[36624]: pgmap v3777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:21.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:20 smithi153 ceph-mon[36569]: pgmap v3776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:21.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:20 smithi153 ceph-mon[36569]: pgmap v3777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:22.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:21 smithi081 ceph-mon[31453]: pgmap v3778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:22.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:21 smithi153 ceph-mon[36569]: pgmap v3778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:22.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:21 smithi104 ceph-mon[36624]: pgmap v3778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:24.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:24 smithi153 ceph-mon[36569]: pgmap v3779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:24 smithi104 ceph-mon[36624]: pgmap v3779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:24 smithi081 ceph-mon[31453]: pgmap v3779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:26.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:26 smithi153 ceph-mon[36569]: pgmap v3780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:26.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:26 smithi104 ceph-mon[36624]: pgmap v3780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:26.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:26 smithi081 ceph-mon[31453]: pgmap v3780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:27.595+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:02:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:27.595+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:02:27.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:27.595+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:02:28.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:28 smithi153 ceph-mon[36569]: pgmap v3781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:28.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:28 smithi104 ceph-mon[36624]: pgmap v3781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:28.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:28 smithi081 ceph-mon[31453]: pgmap v3781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:30.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:30 smithi153 ceph-mon[36569]: pgmap v3782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:30.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:30 smithi104 ceph-mon[36624]: pgmap v3782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:30.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:30 smithi081 ceph-mon[31453]: pgmap v3782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:32.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:32 smithi153 ceph-mon[36569]: pgmap v3783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:32.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:32 smithi104 ceph-mon[36624]: pgmap v3783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:32.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:32 smithi081 ceph-mon[31453]: pgmap v3783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:34.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:34 smithi153 ceph-mon[36569]: pgmap v3784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:34.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:34 smithi104 ceph-mon[36624]: pgmap v3784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:34.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:34 smithi081 ceph-mon[31453]: pgmap v3784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:36.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:36 smithi153 ceph-mon[36569]: pgmap v3785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:36.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:36 smithi104 ceph-mon[36624]: pgmap v3785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:36.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:36 smithi081 ceph-mon[31453]: pgmap v3785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:38.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:38 smithi153 ceph-mon[36569]: pgmap v3786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:38.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:38 smithi104 ceph-mon[36624]: pgmap v3786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:38.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:38 smithi081 ceph-mon[31453]: pgmap v3786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:40.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:40 smithi153 ceph-mon[36569]: pgmap v3787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:40.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:40 smithi104 ceph-mon[36624]: pgmap v3787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:40.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:40 smithi081 ceph-mon[31453]: pgmap v3787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:42.602+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:02:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:42.602+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:02:42.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:42.602+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:02:42.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:42 smithi153 ceph-mon[36569]: pgmap v3788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:42.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:42 smithi104 ceph-mon[36624]: pgmap v3788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:42.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:42 smithi081 ceph-mon[31453]: pgmap v3788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:44.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:44 smithi153 ceph-mon[36569]: pgmap v3789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:44.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:44 smithi104 ceph-mon[36624]: pgmap v3789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:44.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:44 smithi081 ceph-mon[31453]: pgmap v3789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:46.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:46 smithi153 ceph-mon[36569]: pgmap v3790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:46.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:46 smithi104 ceph-mon[36624]: pgmap v3790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:46.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:46 smithi081 ceph-mon[31453]: pgmap v3790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:48.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:48 smithi153 ceph-mon[36569]: pgmap v3791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:48.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:48 smithi104 ceph-mon[36624]: pgmap v3791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:48.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:48 smithi081 ceph-mon[31453]: pgmap v3791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:50.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:50 smithi153 ceph-mon[36569]: pgmap v3792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:50.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:50 smithi104 ceph-mon[36624]: pgmap v3792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:50.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:50 smithi081 ceph-mon[31453]: pgmap v3792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:52.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:52 smithi153 ceph-mon[36569]: pgmap v3793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:52.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:52 smithi104 ceph-mon[36624]: pgmap v3793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:52.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:52 smithi081 ceph-mon[31453]: pgmap v3793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:54.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:54 smithi153 ceph-mon[36569]: pgmap v3794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:54.879 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:54 smithi104 ceph-mon[36624]: pgmap v3794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:54.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:54 smithi081 ceph-mon[31453]: pgmap v3794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:56.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:56 smithi153 ceph-mon[36569]: pgmap v3795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:56.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:56 smithi104 ceph-mon[36624]: pgmap v3795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:56.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:56 smithi081 ceph-mon[31453]: pgmap v3795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:57.605+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:57.605+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:02:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:02:57.605+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:02:58.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:02:58 smithi153 ceph-mon[36569]: pgmap v3796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:58.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:02:58 smithi104 ceph-mon[36624]: pgmap v3796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:02:58.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:02:58 smithi081 ceph-mon[31453]: pgmap v3796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:00.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:00 smithi104 ceph-mon[36624]: pgmap v3797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:00.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:00 smithi081 ceph-mon[31453]: pgmap v3797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:01.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:00 smithi153 ceph-mon[36569]: pgmap v3797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:02.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:02 smithi104 ceph-mon[36624]: pgmap v3798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:02.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:02 smithi081 ceph-mon[31453]: pgmap v3798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:03.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:02 smithi153 ceph-mon[36569]: pgmap v3798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:04.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:04 smithi104 ceph-mon[36624]: pgmap v3799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:04.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:04 smithi081 ceph-mon[31453]: pgmap v3799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:05.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:04 smithi153 ceph-mon[36569]: pgmap v3799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:06.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:06 smithi104 ceph-mon[36624]: pgmap v3800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:06.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:06 smithi081 ceph-mon[31453]: pgmap v3800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:07.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:06 smithi153 ceph-mon[36569]: pgmap v3800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:08.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:08 smithi104 ceph-mon[36624]: pgmap v3801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:08.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:08 smithi081 ceph-mon[31453]: pgmap v3801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:09.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:08 smithi153 ceph-mon[36569]: pgmap v3801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:10.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:10 smithi081 ceph-mon[31453]: pgmap v3802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:11.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:10 smithi153 ceph-mon[36569]: pgmap v3802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:11.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:10 smithi104 ceph-mon[36624]: pgmap v3802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:12.609+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:03:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:12.609+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:03:12.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:12.609+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:03:12.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:12 smithi081 ceph-mon[31453]: pgmap v3803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:13.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:12 smithi153 ceph-mon[36569]: pgmap v3803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:13.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:12 smithi104 ceph-mon[36624]: pgmap v3803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:14.847 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:14 smithi153 ceph-mon[36569]: pgmap v3804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:14.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:14 smithi081 ceph-mon[31453]: pgmap v3804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:15.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:14 smithi104 ceph-mon[36624]: pgmap v3804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:16.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:16 smithi081 ceph-mon[31453]: pgmap v3805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:17.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:16 smithi153 ceph-mon[36569]: pgmap v3805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:17.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:16 smithi104 ceph-mon[36624]: pgmap v3805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:18.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:18 smithi081 ceph-mon[31453]: pgmap v3806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:19.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:18 smithi153 ceph-mon[36569]: pgmap v3806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:19.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:18 smithi104 ceph-mon[36624]: pgmap v3806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:20.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:20 smithi081 ceph-mon[31453]: pgmap v3807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:20.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:03:21.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:20 smithi153 ceph-mon[36569]: pgmap v3807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:21.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:03:21.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:20 smithi104 ceph-mon[36624]: pgmap v3807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:21.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:03:21.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:03:21.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:03:21.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:03:22.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:03:22.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:03:22.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:03:22.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:03:22.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:03:22.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:03:22.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:22 smithi081 ceph-mon[31453]: pgmap v3808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:22.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:22 smithi081 ceph-mon[31453]: pgmap v3809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:22.972 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:22 smithi104 ceph-mon[36624]: pgmap v3808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:22.972 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:22 smithi104 ceph-mon[36624]: pgmap v3809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:23.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:22 smithi153 ceph-mon[36569]: pgmap v3808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:23.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:22 smithi153 ceph-mon[36569]: pgmap v3809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:24 smithi081 ceph-mon[31453]: pgmap v3810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:25.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:24 smithi153 ceph-mon[36569]: pgmap v3810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:25.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:24 smithi104 ceph-mon[36624]: pgmap v3810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:26.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:26 smithi081 ceph-mon[31453]: pgmap v3811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:27.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:26 smithi153 ceph-mon[36569]: pgmap v3811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:27.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:26 smithi104 ceph-mon[36624]: pgmap v3811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:27.621+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:03:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:27.621+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:03:27.610 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:27.621+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:03:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:28.756+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:03:28.745 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:28.756+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:03:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:28.757+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:03:28.746 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:28.757+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:03:28.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:28 smithi081 ceph-mon[31453]: pgmap v3812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:29.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:28 smithi153 ceph-mon[36569]: pgmap v3812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:29.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:28 smithi104 ceph-mon[36624]: pgmap v3812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:30.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:30 smithi081 ceph-mon[31453]: pgmap v3813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:31.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:30 smithi153 ceph-mon[36569]: pgmap v3813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:31.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:30 smithi104 ceph-mon[36624]: pgmap v3813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:32.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:32 smithi081 ceph-mon[31453]: pgmap v3814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:33.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:32 smithi153 ceph-mon[36569]: pgmap v3814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:33.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:32 smithi104 ceph-mon[36624]: pgmap v3814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:34.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:34 smithi081 ceph-mon[31453]: pgmap v3815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:35.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:34 smithi153 ceph-mon[36569]: pgmap v3815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:35.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:34 smithi104 ceph-mon[36624]: pgmap v3815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:37.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:36 smithi153 ceph-mon[36569]: pgmap v3816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:37.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:36 smithi104 ceph-mon[36624]: pgmap v3816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:37.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:36 smithi081 ceph-mon[31453]: pgmap v3816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:39.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:38 smithi153 ceph-mon[36569]: pgmap v3817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:39.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:38 smithi104 ceph-mon[36624]: pgmap v3817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:39.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:38 smithi081 ceph-mon[31453]: pgmap v3817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:41.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:40 smithi153 ceph-mon[36569]: pgmap v3818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:41.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:40 smithi104 ceph-mon[36624]: pgmap v3818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:41.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:40 smithi081 ceph-mon[31453]: pgmap v3818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:42.637+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:03:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:42.637+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:03:42.626 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:42.637+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:03:43.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:42 smithi153 ceph-mon[36569]: pgmap v3819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:43.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:42 smithi104 ceph-mon[36624]: pgmap v3819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:43.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:42 smithi081 ceph-mon[31453]: pgmap v3819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:45.020 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:44 smithi081 ceph-mon[31453]: pgmap v3820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:45.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:44 smithi153 ceph-mon[36569]: pgmap v3820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:45.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:44 smithi104 ceph-mon[36624]: pgmap v3820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:47.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:46 smithi153 ceph-mon[36569]: pgmap v3821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:47.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:46 smithi104 ceph-mon[36624]: pgmap v3821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:47.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:46 smithi081 ceph-mon[31453]: pgmap v3821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:49.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:48 smithi153 ceph-mon[36569]: pgmap v3822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:49.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:48 smithi104 ceph-mon[36624]: pgmap v3822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:49.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:48 smithi081 ceph-mon[31453]: pgmap v3822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:51.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:50 smithi153 ceph-mon[36569]: pgmap v3823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:51.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:50 smithi104 ceph-mon[36624]: pgmap v3823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:51.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:50 smithi081 ceph-mon[31453]: pgmap v3823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:53.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:52 smithi153 ceph-mon[36569]: pgmap v3824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:53.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:52 smithi104 ceph-mon[36624]: pgmap v3824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:53.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:52 smithi081 ceph-mon[31453]: pgmap v3824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:55.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:54 smithi153 ceph-mon[36569]: pgmap v3825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:55.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:54 smithi104 ceph-mon[36624]: pgmap v3825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:55.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:54 smithi081 ceph-mon[31453]: pgmap v3825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:57.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:56 smithi153 ceph-mon[36569]: pgmap v3826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:57.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:56 smithi104 ceph-mon[36624]: pgmap v3826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:57.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:56 smithi081 ceph-mon[31453]: pgmap v3826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:57.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:57.652+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:03:57.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:57.652+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:03:57.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:03:57.652+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:03:59.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:03:58 smithi153 ceph-mon[36569]: pgmap v3827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:59.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:03:58 smithi104 ceph-mon[36624]: pgmap v3827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:03:59.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:03:58 smithi081 ceph-mon[31453]: pgmap v3827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:01.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:00 smithi153 ceph-mon[36569]: pgmap v3828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:01.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:00 smithi104 ceph-mon[36624]: pgmap v3828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:01.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:00 smithi081 ceph-mon[31453]: pgmap v3828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:03.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:02 smithi153 ceph-mon[36569]: pgmap v3829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:03.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:02 smithi104 ceph-mon[36624]: pgmap v3829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:03.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:02 smithi081 ceph-mon[31453]: pgmap v3829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:05.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:04 smithi153 ceph-mon[36569]: pgmap v3830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:05.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:04 smithi104 ceph-mon[36624]: pgmap v3830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:05.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:04 smithi081 ceph-mon[31453]: pgmap v3830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:07.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:06 smithi153 ceph-mon[36569]: pgmap v3831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:07.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:06 smithi104 ceph-mon[36624]: pgmap v3831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:07.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:06 smithi081 ceph-mon[31453]: pgmap v3831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:09.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:08 smithi153 ceph-mon[36569]: pgmap v3832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:09.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:08 smithi104 ceph-mon[36624]: pgmap v3832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:09.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:08 smithi081 ceph-mon[31453]: pgmap v3832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:11.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:10 smithi153 ceph-mon[36569]: pgmap v3833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:11.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:10 smithi104 ceph-mon[36624]: pgmap v3833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:11.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:10 smithi081 ceph-mon[31453]: pgmap v3833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:12.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:12.656+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:04:12.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:12.656+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:04:12.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:12.656+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:04:13.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:12 smithi153 ceph-mon[36569]: pgmap v3834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:13.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:12 smithi104 ceph-mon[36624]: pgmap v3834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:13.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:12 smithi081 ceph-mon[31453]: pgmap v3834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:13.763+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:04:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:13.763+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:04:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:13.764+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:04:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:13.764+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:04:15.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:14 smithi153 ceph-mon[36569]: pgmap v3835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:15.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:14 smithi104 ceph-mon[36624]: pgmap v3835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:15.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:14 smithi081 ceph-mon[31453]: pgmap v3835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:17.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:16 smithi153 ceph-mon[36569]: pgmap v3836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:17.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:16 smithi104 ceph-mon[36624]: pgmap v3836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:17.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:16 smithi081 ceph-mon[31453]: pgmap v3836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:19.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:18 smithi153 ceph-mon[36569]: pgmap v3837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:19.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:18 smithi104 ceph-mon[36624]: pgmap v3837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:19.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:18 smithi081 ceph-mon[31453]: pgmap v3837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:21.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:20 smithi153 ceph-mon[36569]: pgmap v3838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:21.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:20 smithi104 ceph-mon[36624]: pgmap v3838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:21.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:20 smithi081 ceph-mon[31453]: pgmap v3838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:22.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:04:22.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:04:22.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:04:23.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:22 smithi153 ceph-mon[36569]: pgmap v3839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:23.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:04:23.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:04:23.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:04:23.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:22 smithi104 ceph-mon[36624]: pgmap v3839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:23.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:04:23.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:04:23.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:04:23.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:22 smithi081 ceph-mon[31453]: pgmap v3839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:23.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:04:23.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:04:23.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:04:24.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:23 smithi153 ceph-mon[36569]: pgmap v3840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:24.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:23 smithi153 ceph-mon[36569]: pgmap v3841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:24.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:23 smithi104 ceph-mon[36624]: pgmap v3840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:24.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:23 smithi104 ceph-mon[36624]: pgmap v3841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:24.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:23 smithi081 ceph-mon[31453]: pgmap v3840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:24.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:23 smithi081 ceph-mon[31453]: pgmap v3841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:26.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:25 smithi153 ceph-mon[36569]: pgmap v3842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:26.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:25 smithi104 ceph-mon[36624]: pgmap v3842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:26.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:25 smithi081 ceph-mon[31453]: pgmap v3842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:27.671+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:04:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:27.671+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:04:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:27.671+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:04:28.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:27 smithi104 ceph-mon[36624]: pgmap v3843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:28.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:27 smithi081 ceph-mon[31453]: pgmap v3843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:28.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:27 smithi153 ceph-mon[36569]: pgmap v3843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:30.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:29 smithi104 ceph-mon[36624]: pgmap v3844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:30.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:29 smithi081 ceph-mon[31453]: pgmap v3844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:29 smithi153 ceph-mon[36569]: pgmap v3844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:32.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:31 smithi104 ceph-mon[36624]: pgmap v3845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:32.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:31 smithi081 ceph-mon[31453]: pgmap v3845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:32.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:31 smithi153 ceph-mon[36569]: pgmap v3845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:34.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:33 smithi104 ceph-mon[36624]: pgmap v3846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:34.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:33 smithi081 ceph-mon[31453]: pgmap v3846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:34.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:33 smithi153 ceph-mon[36569]: pgmap v3846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:36.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:35 smithi104 ceph-mon[36624]: pgmap v3847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:36.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:35 smithi081 ceph-mon[31453]: pgmap v3847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:36.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:35 smithi153 ceph-mon[36569]: pgmap v3847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:38.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:37 smithi081 ceph-mon[31453]: pgmap v3848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:38.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:37 smithi153 ceph-mon[36569]: pgmap v3848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:38.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:37 smithi104 ceph-mon[36624]: pgmap v3848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:40.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:39 smithi081 ceph-mon[31453]: pgmap v3849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:40.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:39 smithi153 ceph-mon[36569]: pgmap v3849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:40.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:39 smithi104 ceph-mon[36624]: pgmap v3849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:42.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:41 smithi081 ceph-mon[31453]: pgmap v3850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:42.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:41 smithi153 ceph-mon[36569]: pgmap v3850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:42.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:41 smithi104 ceph-mon[36624]: pgmap v3850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:42.673 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:42.685+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:04:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:42.685+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:04:42.674 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:42.685+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:04:44.097 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:43 smithi153 ceph-mon[36569]: pgmap v3851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:44.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:43 smithi081 ceph-mon[31453]: pgmap v3851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:44.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:43 smithi104 ceph-mon[36624]: pgmap v3851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:46.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:45 smithi081 ceph-mon[31453]: pgmap v3852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:46.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:45 smithi153 ceph-mon[36569]: pgmap v3852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:46.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:45 smithi104 ceph-mon[36624]: pgmap v3852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:48.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:47 smithi081 ceph-mon[31453]: pgmap v3853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:48.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:47 smithi153 ceph-mon[36569]: pgmap v3853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:48.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:47 smithi104 ceph-mon[36624]: pgmap v3853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:50.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:49 smithi081 ceph-mon[31453]: pgmap v3854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:50.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:49 smithi153 ceph-mon[36569]: pgmap v3854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:50.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:49 smithi104 ceph-mon[36624]: pgmap v3854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:52.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:51 smithi104 ceph-mon[36624]: pgmap v3855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:52.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:51 smithi081 ceph-mon[31453]: pgmap v3855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:52.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:51 smithi153 ceph-mon[36569]: pgmap v3855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:54.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:53 smithi081 ceph-mon[31453]: pgmap v3856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:53 smithi153 ceph-mon[36569]: pgmap v3856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:54.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:53 smithi104 ceph-mon[36624]: pgmap v3856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:56.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:55 smithi081 ceph-mon[31453]: pgmap v3857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:56.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:55 smithi153 ceph-mon[36569]: pgmap v3857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:56.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:55 smithi104 ceph-mon[36624]: pgmap v3857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:57.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:57.695+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:04:57.684 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:57.695+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:04:57.684 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:04:57.695+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:04:58.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:57 smithi081 ceph-mon[31453]: pgmap v3858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:58.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:57 smithi153 ceph-mon[36569]: pgmap v3858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:04:58.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:57 smithi104 ceph-mon[36624]: pgmap v3858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:00.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:04:59 smithi081 ceph-mon[31453]: pgmap v3859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:00.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:04:59 smithi153 ceph-mon[36569]: pgmap v3859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:00.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:04:59 smithi104 ceph-mon[36624]: pgmap v3859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:02.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:01 smithi081 ceph-mon[31453]: pgmap v3860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:02.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:01 smithi153 ceph-mon[36569]: pgmap v3860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:02.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:01 smithi104 ceph-mon[36624]: pgmap v3860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:04.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:03 smithi081 ceph-mon[31453]: pgmap v3861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:04.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:03 smithi153 ceph-mon[36569]: pgmap v3861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:04.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:03 smithi104 ceph-mon[36624]: pgmap v3861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:05.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:04 smithi153 ceph-mon[36569]: pgmap v3862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:05.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:04 smithi104 ceph-mon[36624]: pgmap v3862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:05.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:04 smithi081 ceph-mon[31453]: pgmap v3862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:07.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:07 smithi104 ceph-mon[36624]: pgmap v3863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:07.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:07 smithi081 ceph-mon[31453]: pgmap v3863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:08.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:07 smithi153 ceph-mon[36569]: pgmap v3863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:09.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:09 smithi104 ceph-mon[36624]: pgmap v3864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:09.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:09 smithi081 ceph-mon[31453]: pgmap v3864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:10.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:09 smithi153 ceph-mon[36569]: pgmap v3864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:11.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:11 smithi104 ceph-mon[36624]: pgmap v3865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:11.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:11 smithi081 ceph-mon[31453]: pgmap v3865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:12.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:11 smithi153 ceph-mon[36569]: pgmap v3865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:12.698 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:12.709+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:05:12.698 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:12.709+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:05:12.698 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:12.709+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:05:13.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:13 smithi081 ceph-mon[31453]: pgmap v3866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:14.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:13 smithi153 ceph-mon[36569]: pgmap v3866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:14.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:13 smithi104 ceph-mon[36624]: pgmap v3866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:15.847 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:15 smithi153 ceph-mon[36569]: pgmap v3867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:15.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:15 smithi081 ceph-mon[31453]: pgmap v3867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:16.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:15 smithi104 ceph-mon[36624]: pgmap v3867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:17.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:17 smithi081 ceph-mon[31453]: pgmap v3868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:18.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:17 smithi153 ceph-mon[36569]: pgmap v3868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:18.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:17 smithi104 ceph-mon[36624]: pgmap v3868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:19.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:19 smithi081 ceph-mon[31453]: pgmap v3869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:20.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:19 smithi153 ceph-mon[36569]: pgmap v3869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:20.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:19 smithi104 ceph-mon[36624]: pgmap v3869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:22.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:21 smithi153 ceph-mon[36569]: pgmap v3870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:22.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:21 smithi104 ceph-mon[36624]: pgmap v3870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:22.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:21 smithi081 ceph-mon[31453]: pgmap v3870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:24.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:23 smithi153 ceph-mon[36569]: pgmap v3871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:24.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:05:24.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:23 smithi104 ceph-mon[36624]: pgmap v3871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:24.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:05:24.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:23 smithi081 ceph-mon[31453]: pgmap v3871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:24.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:05:25.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:05:25.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:05:25.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:05:25.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:05:25.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:05:25.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:05:25.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:05:25.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:05:25.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:05:25.702 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:25 smithi081 ceph-mon[31453]: pgmap v3872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:26.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:25 smithi153 ceph-mon[36569]: pgmap v3872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:26.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:25 smithi153 ceph-mon[36569]: pgmap v3873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:26.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:25 smithi104 ceph-mon[36624]: pgmap v3872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:26.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:25 smithi104 ceph-mon[36624]: pgmap v3873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:26.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:25 smithi081 ceph-mon[31453]: pgmap v3873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:27.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:27.723+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:05:27.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:27.723+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:05:27.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:27.723+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:05:28.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:27 smithi153 ceph-mon[36569]: pgmap v3874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:28.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:27 smithi104 ceph-mon[36624]: pgmap v3874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:28.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:27 smithi081 ceph-mon[31453]: pgmap v3874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:30.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:29 smithi153 ceph-mon[36569]: pgmap v3875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:30.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:29 smithi104 ceph-mon[36624]: pgmap v3875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:30.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:29 smithi081 ceph-mon[31453]: pgmap v3875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:32.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:31 smithi153 ceph-mon[36569]: pgmap v3876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:32.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:31 smithi104 ceph-mon[36624]: pgmap v3876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:32.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:31 smithi081 ceph-mon[31453]: pgmap v3876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:34.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:33 smithi153 ceph-mon[36569]: pgmap v3877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:34.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:33 smithi104 ceph-mon[36624]: pgmap v3877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:34.077 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:33 smithi081 ceph-mon[31453]: pgmap v3877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:36.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:35 smithi153 ceph-mon[36569]: pgmap v3878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:36.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:35 smithi104 ceph-mon[36624]: pgmap v3878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:36.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:35 smithi081 ceph-mon[31453]: pgmap v3878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:38.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:37 smithi153 ceph-mon[36569]: pgmap v3879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:38.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:37 smithi104 ceph-mon[36624]: pgmap v3879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:38.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:37 smithi081 ceph-mon[31453]: pgmap v3879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:40.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:39 smithi153 ceph-mon[36569]: pgmap v3880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:40.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:39 smithi104 ceph-mon[36624]: pgmap v3880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:40.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:39 smithi081 ceph-mon[31453]: pgmap v3880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:42.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:41 smithi153 ceph-mon[36569]: pgmap v3881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:42.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:41 smithi104 ceph-mon[36624]: pgmap v3881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:42.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:41 smithi081 ceph-mon[31453]: pgmap v3881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:42.739+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:05:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:42.739+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:05:42.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:42.739+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:05:44.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:43 smithi153 ceph-mon[36569]: pgmap v3882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:44.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:43 smithi104 ceph-mon[36624]: pgmap v3882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:44.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:43 smithi081 ceph-mon[31453]: pgmap v3882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:46.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:45 smithi153 ceph-mon[36569]: pgmap v3883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:46.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:45 smithi104 ceph-mon[36624]: pgmap v3883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:46.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:45 smithi081 ceph-mon[31453]: pgmap v3883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:48.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:47 smithi153 ceph-mon[36569]: pgmap v3884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:48.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:47 smithi104 ceph-mon[36624]: pgmap v3884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:48.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:47 smithi081 ceph-mon[31453]: pgmap v3884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:50.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:49 smithi153 ceph-mon[36569]: pgmap v3885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:50.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:49 smithi104 ceph-mon[36624]: pgmap v3885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:50.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:49 smithi081 ceph-mon[31453]: pgmap v3885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:52.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:51 smithi153 ceph-mon[36569]: pgmap v3886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:52.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:51 smithi104 ceph-mon[36624]: pgmap v3886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:52.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:51 smithi081 ceph-mon[31453]: pgmap v3886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:54.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:53 smithi153 ceph-mon[36569]: pgmap v3887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:54.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:53 smithi104 ceph-mon[36624]: pgmap v3887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:54.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:53 smithi081 ceph-mon[31453]: pgmap v3887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:56.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:55 smithi153 ceph-mon[36569]: pgmap v3888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:56.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:55 smithi104 ceph-mon[36624]: pgmap v3888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:56.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:55 smithi081 ceph-mon[31453]: pgmap v3888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:57.750+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:05:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:57.750+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:05:57.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:05:57.750+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:05:58.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:57 smithi153 ceph-mon[36569]: pgmap v3889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:58.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:57 smithi104 ceph-mon[36624]: pgmap v3889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:05:58.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:57 smithi081 ceph-mon[31453]: pgmap v3889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:00.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:05:59 smithi153 ceph-mon[36569]: pgmap v3890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:00.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:05:59 smithi104 ceph-mon[36624]: pgmap v3890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:00.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:05:59 smithi081 ceph-mon[31453]: pgmap v3890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:02.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:01 smithi153 ceph-mon[36569]: pgmap v3891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:02.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:01 smithi104 ceph-mon[36624]: pgmap v3891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:02.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:01 smithi081 ceph-mon[31453]: pgmap v3891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:04.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:03 smithi153 ceph-mon[36569]: pgmap v3892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:04.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:03 smithi104 ceph-mon[36624]: pgmap v3892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:04.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:03 smithi081 ceph-mon[31453]: pgmap v3892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:06.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:05 smithi153 ceph-mon[36569]: pgmap v3893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:06.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:05 smithi104 ceph-mon[36624]: pgmap v3893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:06.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:05 smithi081 ceph-mon[31453]: pgmap v3893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:08.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:07 smithi153 ceph-mon[36569]: pgmap v3894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:08.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:07 smithi104 ceph-mon[36624]: pgmap v3894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:08.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:07 smithi081 ceph-mon[31453]: pgmap v3894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:10.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:09 smithi153 ceph-mon[36569]: pgmap v3895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:10.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:09 smithi104 ceph-mon[36624]: pgmap v3895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:10.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:09 smithi081 ceph-mon[31453]: pgmap v3895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:12.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:11 smithi104 ceph-mon[36624]: pgmap v3896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:12.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:11 smithi081 ceph-mon[31453]: pgmap v3896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:12.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:11 smithi153 ceph-mon[36569]: pgmap v3896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:12.740 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:12.752+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:06:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:12.752+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:06:12.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:12.752+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:06:14.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:13 smithi104 ceph-mon[36624]: pgmap v3897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:14.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:13 smithi081 ceph-mon[31453]: pgmap v3897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:14.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:13 smithi153 ceph-mon[36569]: pgmap v3897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:16.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:15 smithi104 ceph-mon[36624]: pgmap v3898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:16.097 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:15 smithi153 ceph-mon[36569]: pgmap v3898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:16.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:15 smithi081 ceph-mon[31453]: pgmap v3898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:18.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:17 smithi104 ceph-mon[36624]: pgmap v3899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:18.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:17 smithi081 ceph-mon[31453]: pgmap v3899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:18.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:17 smithi153 ceph-mon[36569]: pgmap v3899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:20.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:19 smithi104 ceph-mon[36624]: pgmap v3900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:20.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:19 smithi081 ceph-mon[31453]: pgmap v3900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:20.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:19 smithi153 ceph-mon[36569]: pgmap v3900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:22.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:21 smithi081 ceph-mon[31453]: pgmap v3901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:22.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:21 smithi153 ceph-mon[36569]: pgmap v3901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:22.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:21 smithi104 ceph-mon[36624]: pgmap v3901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:24.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:23 smithi104 ceph-mon[36624]: pgmap v3902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:24.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:23 smithi081 ceph-mon[31453]: pgmap v3902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:24.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:23 smithi153 ceph-mon[36569]: pgmap v3902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:25.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:06:25.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:06:25.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:06:26.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: pgmap v3903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: pgmap v3903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: pgmap v3903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:26.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:06:27.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:06:27.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:06:27.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:06:27.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:06:27.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:06:27.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:06:27.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:06:27.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:06:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:27.766+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:06:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:27.767+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:06:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:27.767+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:06:28.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:27 smithi081 ceph-mon[31453]: pgmap v3904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:28.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:27 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:06:28.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:27 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:06:28.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:27 smithi081 ceph-mon[31453]: pgmap v3905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:28.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:27 smithi081 ceph-mon[31453]: pgmap v3906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:28.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:27 smithi153 ceph-mon[36569]: pgmap v3904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:28.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:27 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:06:28.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:27 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:06:28.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:27 smithi153 ceph-mon[36569]: pgmap v3905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:28.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:27 smithi153 ceph-mon[36569]: pgmap v3906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:28.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:27 smithi104 ceph-mon[36624]: pgmap v3904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:28.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:27 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:06:28.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:27 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:06:28.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:27 smithi104 ceph-mon[36624]: pgmap v3905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:28.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:27 smithi104 ceph-mon[36624]: pgmap v3906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:30.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:29 smithi081 ceph-mon[31453]: pgmap v3907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:29 smithi153 ceph-mon[36569]: pgmap v3907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:30.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:29 smithi104 ceph-mon[36624]: pgmap v3907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:32.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:31 smithi081 ceph-mon[31453]: pgmap v3908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:32.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:31 smithi153 ceph-mon[36569]: pgmap v3908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:32.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:31 smithi104 ceph-mon[36624]: pgmap v3908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:34.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:33 smithi081 ceph-mon[31453]: pgmap v3909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:34.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:33 smithi153 ceph-mon[36569]: pgmap v3909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:34.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:33 smithi104 ceph-mon[36624]: pgmap v3909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:36.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:35 smithi081 ceph-mon[31453]: pgmap v3910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:36.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:35 smithi153 ceph-mon[36569]: pgmap v3910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:36.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:35 smithi104 ceph-mon[36624]: pgmap v3910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:38.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:37 smithi081 ceph-mon[31453]: pgmap v3911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:38.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:37 smithi153 ceph-mon[36569]: pgmap v3911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:38.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:37 smithi104 ceph-mon[36624]: pgmap v3911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:40.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:39 smithi081 ceph-mon[31453]: pgmap v3912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:40.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:39 smithi153 ceph-mon[36569]: pgmap v3912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:40.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:39 smithi104 ceph-mon[36624]: pgmap v3912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:42.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:41 smithi081 ceph-mon[31453]: pgmap v3913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:42.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:41 smithi153 ceph-mon[36569]: pgmap v3913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:42.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:41 smithi104 ceph-mon[36624]: pgmap v3913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:42.768+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:06:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:42.768+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:06:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:42.768+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:06:44.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:43 smithi153 ceph-mon[36569]: pgmap v3914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:44.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:43 smithi104 ceph-mon[36624]: pgmap v3914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:44.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:43 smithi081 ceph-mon[31453]: pgmap v3914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:46.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:45 smithi153 ceph-mon[36569]: pgmap v3915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:46.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:45 smithi104 ceph-mon[36624]: pgmap v3915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:46.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:45 smithi081 ceph-mon[31453]: pgmap v3915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:48.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:47 smithi153 ceph-mon[36569]: pgmap v3916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:47 smithi104 ceph-mon[36624]: pgmap v3916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:48.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:47 smithi081 ceph-mon[31453]: pgmap v3916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:50.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:49 smithi153 ceph-mon[36569]: pgmap v3917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:50.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:49 smithi104 ceph-mon[36624]: pgmap v3917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:50.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:49 smithi081 ceph-mon[31453]: pgmap v3917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:51.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:50 smithi153 ceph-mon[36569]: pgmap v3918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:51.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:50 smithi104 ceph-mon[36624]: pgmap v3918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:51.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:50 smithi081 ceph-mon[31453]: pgmap v3918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:53.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:53 smithi081 ceph-mon[31453]: pgmap v3919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:53.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:53 smithi153 ceph-mon[36569]: pgmap v3919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:53.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:53 smithi104 ceph-mon[36624]: pgmap v3919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:55.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:55 smithi081 ceph-mon[31453]: pgmap v3920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:55.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:55 smithi153 ceph-mon[36569]: pgmap v3920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:55.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:55 smithi104 ceph-mon[36624]: pgmap v3920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:57.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:57 smithi081 ceph-mon[31453]: pgmap v3921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:57.784+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:06:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:57.784+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:06:57.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:06:57.784+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:06:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:57 smithi153 ceph-mon[36569]: pgmap v3921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:57 smithi104 ceph-mon[36624]: pgmap v3921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:59.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:06:59 smithi081 ceph-mon[31453]: pgmap v3922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:59.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:06:59 smithi153 ceph-mon[36569]: pgmap v3922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:06:59.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:06:59 smithi104 ceph-mon[36624]: pgmap v3922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:01.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:01 smithi081 ceph-mon[31453]: pgmap v3923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:01.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:01 smithi153 ceph-mon[36569]: pgmap v3923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:01.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:01 smithi104 ceph-mon[36624]: pgmap v3923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:03.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:03 smithi153 ceph-mon[36569]: pgmap v3924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:03.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:03 smithi104 ceph-mon[36624]: pgmap v3924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:03.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:03 smithi081 ceph-mon[31453]: pgmap v3924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:05.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:05 smithi153 ceph-mon[36569]: pgmap v3925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:05.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:05 smithi104 ceph-mon[36624]: pgmap v3925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:05.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:05 smithi081 ceph-mon[31453]: pgmap v3925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:07.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:07 smithi081 ceph-mon[31453]: pgmap v3926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:07.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:07 smithi153 ceph-mon[36569]: pgmap v3926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:07.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:07 smithi104 ceph-mon[36624]: pgmap v3926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:09.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:09 smithi153 ceph-mon[36569]: pgmap v3927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:09.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:09 smithi104 ceph-mon[36624]: pgmap v3927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:09.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:09 smithi081 ceph-mon[31453]: pgmap v3927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:11.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:11 smithi153 ceph-mon[36569]: pgmap v3928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:11.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:11 smithi104 ceph-mon[36624]: pgmap v3928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:11.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:11 smithi081 ceph-mon[31453]: pgmap v3928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:12.785+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:07:12.774 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:12.785+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:07:12.775 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:12.785+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:07:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:13.792+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:07:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:13.792+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:07:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:13.792+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:07:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:13.792+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:07:13.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:13 smithi153 ceph-mon[36569]: pgmap v3929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:13.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:13 smithi104 ceph-mon[36624]: pgmap v3929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:13.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:13 smithi081 ceph-mon[31453]: pgmap v3929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:15.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:15 smithi153 ceph-mon[36569]: pgmap v3930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:15.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:15 smithi104 ceph-mon[36624]: pgmap v3930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:15.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:15 smithi081 ceph-mon[31453]: pgmap v3930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:17.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:17 smithi153 ceph-mon[36569]: pgmap v3931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:17.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:17 smithi104 ceph-mon[36624]: pgmap v3931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:17.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:17 smithi081 ceph-mon[31453]: pgmap v3931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:19.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:19 smithi153 ceph-mon[36569]: pgmap v3932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:19.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:19 smithi104 ceph-mon[36624]: pgmap v3932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:19.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:19 smithi081 ceph-mon[31453]: pgmap v3932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:21.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:21 smithi153 ceph-mon[36569]: pgmap v3933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:21.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:21 smithi104 ceph-mon[36624]: pgmap v3933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:21.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:21 smithi081 ceph-mon[31453]: pgmap v3933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:23.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:23 smithi153 ceph-mon[36569]: pgmap v3934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:23.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:23 smithi104 ceph-mon[36624]: pgmap v3934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:23.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:23 smithi081 ceph-mon[31453]: pgmap v3934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:25.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:25 smithi153 ceph-mon[36569]: pgmap v3935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:25.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:25 smithi104 ceph-mon[36624]: pgmap v3935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:25.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:25 smithi081 ceph-mon[31453]: pgmap v3935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:27.801+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:07:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:27.801+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:07:27.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:27.801+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:07:27.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:27 smithi153 ceph-mon[36569]: pgmap v3936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:27.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:07:27.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:27 smithi104 ceph-mon[36624]: pgmap v3936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:27.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:07:27.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:27 smithi081 ceph-mon[31453]: pgmap v3936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:27.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:07:29.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:07:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:07:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: pgmap v3937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: pgmap v3938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:29.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:07:29.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:07:29.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: pgmap v3937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:29.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: pgmap v3938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:29.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: pgmap v3937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: pgmap v3938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:29.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:07:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:29 smithi153 ceph-mon[36569]: pgmap v3939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:30.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:29 smithi104 ceph-mon[36624]: pgmap v3939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:30.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:29 smithi081 ceph-mon[31453]: pgmap v3939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:32.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:31 smithi153 ceph-mon[36569]: pgmap v3940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:32.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:31 smithi104 ceph-mon[36624]: pgmap v3940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:32.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:31 smithi081 ceph-mon[31453]: pgmap v3940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:34.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:33 smithi153 ceph-mon[36569]: pgmap v3941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:34.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:33 smithi104 ceph-mon[36624]: pgmap v3941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:34.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:33 smithi081 ceph-mon[31453]: pgmap v3941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:36.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:35 smithi153 ceph-mon[36569]: pgmap v3942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:36.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:35 smithi104 ceph-mon[36624]: pgmap v3942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:36.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:35 smithi081 ceph-mon[31453]: pgmap v3942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:38.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:37 smithi153 ceph-mon[36569]: pgmap v3943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:38.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:37 smithi104 ceph-mon[36624]: pgmap v3943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:38.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:37 smithi081 ceph-mon[31453]: pgmap v3943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:40.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:39 smithi153 ceph-mon[36569]: pgmap v3944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:40.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:39 smithi104 ceph-mon[36624]: pgmap v3944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:40.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:39 smithi081 ceph-mon[31453]: pgmap v3944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:42.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:41 smithi153 ceph-mon[36569]: pgmap v3945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:42.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:41 smithi104 ceph-mon[36624]: pgmap v3945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:42.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:41 smithi081 ceph-mon[31453]: pgmap v3945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:42.815+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:07:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:42.815+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:07:42.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:42.815+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:07:44.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:43 smithi153 ceph-mon[36569]: pgmap v3946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:44.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:43 smithi104 ceph-mon[36624]: pgmap v3946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:44.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:43 smithi081 ceph-mon[31453]: pgmap v3946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:46.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:45 smithi153 ceph-mon[36569]: pgmap v3947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:46.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:45 smithi104 ceph-mon[36624]: pgmap v3947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:46.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:45 smithi081 ceph-mon[31453]: pgmap v3947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:48.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:47 smithi153 ceph-mon[36569]: pgmap v3948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:47 smithi104 ceph-mon[36624]: pgmap v3948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:48.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:47 smithi081 ceph-mon[31453]: pgmap v3948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:50.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:49 smithi153 ceph-mon[36569]: pgmap v3949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:50.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:49 smithi104 ceph-mon[36624]: pgmap v3949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:50.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:49 smithi081 ceph-mon[31453]: pgmap v3949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:52.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:51 smithi153 ceph-mon[36569]: pgmap v3950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:52.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:51 smithi104 ceph-mon[36624]: pgmap v3950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:52.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:51 smithi081 ceph-mon[31453]: pgmap v3950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:54.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:53 smithi081 ceph-mon[31453]: pgmap v3951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:53 smithi153 ceph-mon[36569]: pgmap v3951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:54.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:53 smithi104 ceph-mon[36624]: pgmap v3951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:56.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:55 smithi153 ceph-mon[36569]: pgmap v3952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:56.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:55 smithi104 ceph-mon[36624]: pgmap v3952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:56.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:55 smithi081 ceph-mon[31453]: pgmap v3952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:57.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:57.831+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:07:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:57.831+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:07:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:07:57.831+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:07:58.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:57 smithi153 ceph-mon[36569]: pgmap v3953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:58.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:57 smithi104 ceph-mon[36624]: pgmap v3953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:07:58.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:57 smithi081 ceph-mon[31453]: pgmap v3953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:00.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:07:59 smithi153 ceph-mon[36569]: pgmap v3954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:00.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:07:59 smithi104 ceph-mon[36624]: pgmap v3954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:00.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:07:59 smithi081 ceph-mon[31453]: pgmap v3954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:02.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:01 smithi153 ceph-mon[36569]: pgmap v3955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:02.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:01 smithi104 ceph-mon[36624]: pgmap v3955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:02.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:01 smithi081 ceph-mon[31453]: pgmap v3955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:04.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:03 smithi153 ceph-mon[36569]: pgmap v3956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:04.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:03 smithi104 ceph-mon[36624]: pgmap v3956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:04.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:03 smithi081 ceph-mon[31453]: pgmap v3956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:06.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:05 smithi153 ceph-mon[36569]: pgmap v3957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:06.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:05 smithi104 ceph-mon[36624]: pgmap v3957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:06.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:05 smithi081 ceph-mon[31453]: pgmap v3957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:08.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:07 smithi153 ceph-mon[36569]: pgmap v3958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:08.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:07 smithi104 ceph-mon[36624]: pgmap v3958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:08.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:07 smithi081 ceph-mon[31453]: pgmap v3958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:10.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:09 smithi153 ceph-mon[36569]: pgmap v3959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:10.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:09 smithi104 ceph-mon[36624]: pgmap v3959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:10.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:09 smithi081 ceph-mon[31453]: pgmap v3959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:11 smithi153 ceph-mon[36569]: pgmap v3960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:12.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:11 smithi104 ceph-mon[36624]: pgmap v3960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:12.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:11 smithi081 ceph-mon[31453]: pgmap v3960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:12.832+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:08:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:12.832+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:08:12.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:12.832+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:08:14.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:13 smithi153 ceph-mon[36569]: pgmap v3961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:14.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:13 smithi104 ceph-mon[36624]: pgmap v3961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:14.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:13 smithi081 ceph-mon[31453]: pgmap v3961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:16.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:15 smithi153 ceph-mon[36569]: pgmap v3962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:16.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:15 smithi104 ceph-mon[36624]: pgmap v3962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:16.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:15 smithi081 ceph-mon[31453]: pgmap v3962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:18.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:17 smithi153 ceph-mon[36569]: pgmap v3963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:18.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:17 smithi104 ceph-mon[36624]: pgmap v3963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:18.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:17 smithi081 ceph-mon[31453]: pgmap v3963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:21.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:20 smithi153 ceph-mon[36569]: pgmap v3964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:21.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:20 smithi104 ceph-mon[36624]: pgmap v3964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:21.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:20 smithi081 ceph-mon[31453]: pgmap v3964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:22.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:21 smithi153 ceph-mon[36569]: pgmap v3965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:22.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:21 smithi104 ceph-mon[36624]: pgmap v3965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:22.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:21 smithi081 ceph-mon[31453]: pgmap v3965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:24.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:23 smithi153 ceph-mon[36569]: pgmap v3966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:24.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:23 smithi104 ceph-mon[36624]: pgmap v3966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:24.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:23 smithi081 ceph-mon[31453]: pgmap v3966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:25 smithi153 ceph-mon[36569]: pgmap v3967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:26.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:25 smithi104 ceph-mon[36624]: pgmap v3967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:26.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:25 smithi081 ceph-mon[31453]: pgmap v3967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:27.848+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:08:27.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:27.848+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:08:27.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:27.848+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:08:28.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:27 smithi153 ceph-mon[36569]: pgmap v3968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:28.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:27 smithi104 ceph-mon[36624]: pgmap v3968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:28.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:27 smithi081 ceph-mon[31453]: pgmap v3968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:29.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:08:29.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:08:29.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:08:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:08:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:08:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:30 smithi153 ceph-mon[36569]: pgmap v3969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:30 smithi153 ceph-mon[36569]: pgmap v3970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:08:30.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:08:30.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:08:30.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:30 smithi104 ceph-mon[36624]: pgmap v3969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:30.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:30 smithi104 ceph-mon[36624]: pgmap v3970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:30.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:08:30.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:08:30.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:08:30.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:30 smithi081 ceph-mon[31453]: pgmap v3969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:30.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:30 smithi081 ceph-mon[31453]: pgmap v3970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:30.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:08:32.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:32 smithi153 ceph-mon[36569]: pgmap v3971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:32.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:32 smithi104 ceph-mon[36624]: pgmap v3971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:32.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:32 smithi081 ceph-mon[31453]: pgmap v3971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:34.699 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:34 smithi081 ceph-mon[31453]: pgmap v3972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:34.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:34 smithi153 ceph-mon[36569]: pgmap v3972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:34.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:34 smithi104 ceph-mon[36624]: pgmap v3972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:36.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:36 smithi153 ceph-mon[36569]: pgmap v3973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:36.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:36 smithi104 ceph-mon[36624]: pgmap v3973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:36.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:36 smithi081 ceph-mon[31453]: pgmap v3973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:38.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:38 smithi153 ceph-mon[36569]: pgmap v3974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:38.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:38 smithi104 ceph-mon[36624]: pgmap v3974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:38.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:38 smithi081 ceph-mon[31453]: pgmap v3974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:40.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:40 smithi153 ceph-mon[36569]: pgmap v3975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:40.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:40 smithi104 ceph-mon[36624]: pgmap v3975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:40.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:40 smithi081 ceph-mon[31453]: pgmap v3975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:42.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:42 smithi153 ceph-mon[36569]: pgmap v3976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:42.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:42 smithi104 ceph-mon[36624]: pgmap v3976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:42.862+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:08:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:42.862+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:08:42.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:42.862+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:08:42.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:42 smithi081 ceph-mon[31453]: pgmap v3976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:44.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:44 smithi153 ceph-mon[36569]: pgmap v3977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:44.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:44 smithi104 ceph-mon[36624]: pgmap v3977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:44.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:44 smithi081 ceph-mon[31453]: pgmap v3977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:46.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:46 smithi153 ceph-mon[36569]: pgmap v3978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:46.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:46 smithi104 ceph-mon[36624]: pgmap v3978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:46.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:46 smithi081 ceph-mon[31453]: pgmap v3978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:48.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:48 smithi153 ceph-mon[36569]: pgmap v3979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:48.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:48 smithi104 ceph-mon[36624]: pgmap v3979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:48.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:48 smithi081 ceph-mon[31453]: pgmap v3979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:50.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:50 smithi153 ceph-mon[36569]: pgmap v3980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:50.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:50 smithi104 ceph-mon[36624]: pgmap v3980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:50.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:50 smithi081 ceph-mon[31453]: pgmap v3980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:52.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:52 smithi153 ceph-mon[36569]: pgmap v3981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:52.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:52 smithi104 ceph-mon[36624]: pgmap v3981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:52.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:52 smithi081 ceph-mon[31453]: pgmap v3981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:54.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:54 smithi153 ceph-mon[36569]: pgmap v3982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:54.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:54 smithi104 ceph-mon[36624]: pgmap v3982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:54 smithi081 ceph-mon[31453]: pgmap v3982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:56.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:56 smithi081 ceph-mon[31453]: pgmap v3983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:56 smithi153 ceph-mon[36569]: pgmap v3983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:56.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:56 smithi104 ceph-mon[36624]: pgmap v3983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:57.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:57.870+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:08:57.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:57.870+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:08:57.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:08:57.870+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:08:58.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:08:58 smithi153 ceph-mon[36569]: pgmap v3984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:58.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:08:58 smithi104 ceph-mon[36624]: pgmap v3984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:08:58.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:08:58 smithi081 ceph-mon[31453]: pgmap v3984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:00.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:00 smithi153 ceph-mon[36569]: pgmap v3985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:00.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:00 smithi104 ceph-mon[36624]: pgmap v3985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:00.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:00 smithi081 ceph-mon[31453]: pgmap v3985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:02.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:02 smithi153 ceph-mon[36569]: pgmap v3986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:02.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:02 smithi104 ceph-mon[36624]: pgmap v3986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:02.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:02 smithi081 ceph-mon[31453]: pgmap v3986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:04 smithi153 ceph-mon[36569]: pgmap v3987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:04.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:04 smithi104 ceph-mon[36624]: pgmap v3987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:04.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:04 smithi081 ceph-mon[31453]: pgmap v3987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:06.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:06 smithi153 ceph-mon[36569]: pgmap v3988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:06.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:06 smithi104 ceph-mon[36624]: pgmap v3988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:06.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:06 smithi081 ceph-mon[31453]: pgmap v3988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:08.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:08 smithi153 ceph-mon[36569]: pgmap v3989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:08.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:08 smithi104 ceph-mon[36624]: pgmap v3989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:08.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:08 smithi081 ceph-mon[31453]: pgmap v3989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:10.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:10 smithi153 ceph-mon[36569]: pgmap v3990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:10.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:10 smithi104 ceph-mon[36624]: pgmap v3990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:10.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:10 smithi081 ceph-mon[31453]: pgmap v3990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:12.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:12 smithi104 ceph-mon[36624]: pgmap v3991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:12.882+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:09:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:12.883+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:09:12.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:12.883+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:09:12.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:12 smithi081 ceph-mon[31453]: pgmap v3991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:13.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:12 smithi153 ceph-mon[36569]: pgmap v3991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:14.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:14 smithi104 ceph-mon[36624]: pgmap v3992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:14.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:14 smithi081 ceph-mon[31453]: pgmap v3992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:15.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:14 smithi153 ceph-mon[36569]: pgmap v3992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:16.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:16 smithi104 ceph-mon[36624]: pgmap v3993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:16.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:16 smithi081 ceph-mon[31453]: pgmap v3993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:17.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:16 smithi153 ceph-mon[36569]: pgmap v3993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:18.850 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:18 smithi153 ceph-mon[36569]: pgmap v3994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:18.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:18 smithi081 ceph-mon[31453]: pgmap v3994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:19.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:18 smithi104 ceph-mon[36624]: pgmap v3994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:20.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:20 smithi081 ceph-mon[31453]: pgmap v3995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:21.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:20 smithi153 ceph-mon[36569]: pgmap v3995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:21.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:20 smithi104 ceph-mon[36624]: pgmap v3995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:22.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:22 smithi081 ceph-mon[31453]: pgmap v3996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:23.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:22 smithi153 ceph-mon[36569]: pgmap v3996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:23.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:22 smithi104 ceph-mon[36624]: pgmap v3996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:24.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:24 smithi081 ceph-mon[31453]: pgmap v3997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:24 smithi153 ceph-mon[36569]: pgmap v3997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:25.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:24 smithi104 ceph-mon[36624]: pgmap v3997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:26.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:26 smithi104 ceph-mon[36624]: pgmap v3998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:26.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:26 smithi081 ceph-mon[31453]: pgmap v3998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:27.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:26 smithi153 ceph-mon[36569]: pgmap v3998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:27.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:27.893+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:09:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:27.893+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:09:27.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:27.893+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:09:28.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:28 smithi081 ceph-mon[31453]: pgmap v3999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:28 smithi153 ceph-mon[36569]: pgmap v3999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:29.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:28 smithi104 ceph-mon[36624]: pgmap v3999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:30.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:30 smithi081 ceph-mon[31453]: pgmap v4000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:30.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:09:31.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:30 smithi153 ceph-mon[36569]: pgmap v4000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:31.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:09:31.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:30 smithi104 ceph-mon[36624]: pgmap v4000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:31.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:09:31.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:09:31.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:09:31.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:09:32.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:09:32.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:09:32.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:09:32.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:09:32.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:09:32.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:09:32.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:32 smithi081 ceph-mon[31453]: pgmap v4001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:32.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:32 smithi081 ceph-mon[31453]: pgmap v4002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:33.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:32 smithi153 ceph-mon[36569]: pgmap v4001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:33.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:32 smithi153 ceph-mon[36569]: pgmap v4002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:33.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:32 smithi104 ceph-mon[36624]: pgmap v4001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:33.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:32 smithi104 ceph-mon[36624]: pgmap v4002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:34.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:34 smithi081 ceph-mon[31453]: pgmap v4003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:35.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:34 smithi153 ceph-mon[36569]: pgmap v4003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:35.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:34 smithi104 ceph-mon[36624]: pgmap v4003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:36.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:36 smithi081 ceph-mon[31453]: pgmap v4004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:37.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:36 smithi153 ceph-mon[36569]: pgmap v4004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:37.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:36 smithi104 ceph-mon[36624]: pgmap v4004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:38.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:38 smithi081 ceph-mon[31453]: pgmap v4005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:39.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:38 smithi153 ceph-mon[36569]: pgmap v4005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:39.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:38 smithi104 ceph-mon[36624]: pgmap v4005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:40.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:40 smithi081 ceph-mon[31453]: pgmap v4006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:41.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:40 smithi153 ceph-mon[36569]: pgmap v4006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:41.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:40 smithi104 ceph-mon[36624]: pgmap v4006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:42.909+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:09:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:42.909+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:09:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:42.909+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:09:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:42 smithi081 ceph-mon[31453]: pgmap v4007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:43.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:42 smithi153 ceph-mon[36569]: pgmap v4007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:43.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:42 smithi104 ceph-mon[36624]: pgmap v4007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:45.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:44 smithi153 ceph-mon[36569]: pgmap v4008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:45.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:44 smithi104 ceph-mon[36624]: pgmap v4008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:45.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:44 smithi081 ceph-mon[31453]: pgmap v4008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:47.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:46 smithi153 ceph-mon[36569]: pgmap v4009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:47.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:46 smithi104 ceph-mon[36624]: pgmap v4009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:47.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:46 smithi081 ceph-mon[31453]: pgmap v4009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:49.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:48 smithi153 ceph-mon[36569]: pgmap v4010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:49.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:48 smithi104 ceph-mon[36624]: pgmap v4010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:49.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:48 smithi081 ceph-mon[31453]: pgmap v4010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:51.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:50 smithi153 ceph-mon[36569]: pgmap v4011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:51.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:50 smithi104 ceph-mon[36624]: pgmap v4011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:51.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:50 smithi081 ceph-mon[31453]: pgmap v4011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:53.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:52 smithi153 ceph-mon[36569]: pgmap v4012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:53.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:52 smithi104 ceph-mon[36624]: pgmap v4012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:53.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:52 smithi081 ceph-mon[31453]: pgmap v4012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:55.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:54 smithi153 ceph-mon[36569]: pgmap v4013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:55.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:54 smithi104 ceph-mon[36624]: pgmap v4013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:55.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:54 smithi081 ceph-mon[31453]: pgmap v4013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:57.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:56 smithi153 ceph-mon[36569]: pgmap v4014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:57.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:56 smithi104 ceph-mon[36624]: pgmap v4014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:57.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:56 smithi081 ceph-mon[31453]: pgmap v4014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:57.924+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:09:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:57.924+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:09:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:09:57.924+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:09:59.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:09:58 smithi153 ceph-mon[36569]: pgmap v4015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:59.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:09:58 smithi104 ceph-mon[36624]: pgmap v4015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:09:59.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:09:58 smithi081 ceph-mon[31453]: pgmap v4015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:01.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:00 smithi153 ceph-mon[36569]: pgmap v4016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:01.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:10:01.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:00 smithi104 ceph-mon[36624]: pgmap v4016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:01.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:10:01.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:00 smithi081 ceph-mon[31453]: pgmap v4016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:01.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:10:03.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:02 smithi153 ceph-mon[36569]: pgmap v4017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:03.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:02 smithi104 ceph-mon[36624]: pgmap v4017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:03.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:02 smithi081 ceph-mon[31453]: pgmap v4017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:05.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:04 smithi153 ceph-mon[36569]: pgmap v4018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:05.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:04 smithi104 ceph-mon[36624]: pgmap v4018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:05.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:04 smithi081 ceph-mon[31453]: pgmap v4018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:07.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:06 smithi153 ceph-mon[36569]: pgmap v4019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:07.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:06 smithi104 ceph-mon[36624]: pgmap v4019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:07.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:06 smithi081 ceph-mon[31453]: pgmap v4019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:09.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:08 smithi153 ceph-mon[36569]: pgmap v4020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:09.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:08 smithi104 ceph-mon[36624]: pgmap v4020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:09.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:08 smithi081 ceph-mon[31453]: pgmap v4020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:11.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:10 smithi153 ceph-mon[36569]: pgmap v4021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:11.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:10 smithi104 ceph-mon[36624]: pgmap v4021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:11.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:10 smithi081 ceph-mon[31453]: pgmap v4021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:12.928 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:12.939+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:10:12.928 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:12.939+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:10:12.928 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:12.939+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:10:13.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:12 smithi153 ceph-mon[36569]: pgmap v4022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:13.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:12 smithi104 ceph-mon[36624]: pgmap v4022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:13.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:12 smithi081 ceph-mon[31453]: pgmap v4022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:13.823+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:10:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:13.823+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:10:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:13.824+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:10:13.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:13.824+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:10:15.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:14 smithi153 ceph-mon[36569]: pgmap v4023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:15.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:14 smithi104 ceph-mon[36624]: pgmap v4023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:15.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:14 smithi081 ceph-mon[31453]: pgmap v4023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:17.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:16 smithi153 ceph-mon[36569]: pgmap v4024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:17.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:16 smithi104 ceph-mon[36624]: pgmap v4024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:17.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:16 smithi081 ceph-mon[31453]: pgmap v4024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:19.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:18 smithi153 ceph-mon[36569]: pgmap v4025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:19.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:18 smithi104 ceph-mon[36624]: pgmap v4025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:19.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:18 smithi081 ceph-mon[31453]: pgmap v4025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:21.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:20 smithi153 ceph-mon[36569]: pgmap v4026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:21.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:20 smithi104 ceph-mon[36624]: pgmap v4026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:21.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:20 smithi081 ceph-mon[31453]: pgmap v4026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:23.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:22 smithi153 ceph-mon[36569]: pgmap v4027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:23.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:22 smithi104 ceph-mon[36624]: pgmap v4027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:23.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:22 smithi081 ceph-mon[31453]: pgmap v4027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:24 smithi153 ceph-mon[36569]: pgmap v4028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:25.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:24 smithi104 ceph-mon[36624]: pgmap v4028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:25.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:24 smithi081 ceph-mon[31453]: pgmap v4028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:27.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:26 smithi153 ceph-mon[36569]: pgmap v4029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:27.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:26 smithi104 ceph-mon[36624]: pgmap v4029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:27.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:26 smithi081 ceph-mon[31453]: pgmap v4029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:27.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:27.949+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:10:27.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:27.949+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:10:27.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:27.949+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:10:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:28 smithi153 ceph-mon[36569]: pgmap v4030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:29.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:28 smithi104 ceph-mon[36624]: pgmap v4030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:29.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:28 smithi081 ceph-mon[31453]: pgmap v4030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:31.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:30 smithi153 ceph-mon[36569]: pgmap v4031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:31.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:30 smithi104 ceph-mon[36624]: pgmap v4031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:31.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:30 smithi081 ceph-mon[31453]: pgmap v4031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:32.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:10:32.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:10:32.085 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:10:33.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:32 smithi104 ceph-mon[36624]: pgmap v4032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:33.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:32 smithi081 ceph-mon[31453]: pgmap v4032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:33.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:32 smithi153 ceph-mon[36569]: pgmap v4032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:33.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:33.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:34 smithi153 ceph-mon[36569]: pgmap v4033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:35.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:34 smithi104 ceph-mon[36624]: pgmap v4033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:35.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:34 smithi081 ceph-mon[31453]: pgmap v4033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:35.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:35.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:37.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:36 smithi153 ceph-mon[36569]: pgmap v4034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:37.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:37.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:37.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:36 smithi104 ceph-mon[36624]: pgmap v4034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:37.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:37.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:37.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:36 smithi081 ceph-mon[31453]: pgmap v4034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:37.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:37.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:38.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:10:38.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:10:38.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:10:38.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:10:38.073 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:10:38.073 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:10:39.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:38 smithi153 ceph-mon[36569]: pgmap v4035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:38 smithi153 ceph-mon[36569]: pgmap v4036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:38 smithi153 ceph-mon[36569]: pgmap v4037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:39.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:38 smithi104 ceph-mon[36624]: pgmap v4035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:38 smithi104 ceph-mon[36624]: pgmap v4036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:38 smithi104 ceph-mon[36624]: pgmap v4037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:39.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:38 smithi081 ceph-mon[31453]: pgmap v4035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:38 smithi081 ceph-mon[31453]: pgmap v4036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:38 smithi081 ceph-mon[31453]: pgmap v4037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:39.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:10:41.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:40 smithi153 ceph-mon[36569]: pgmap v4038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:41.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:40 smithi104 ceph-mon[36624]: pgmap v4038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:41.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:40 smithi081 ceph-mon[31453]: pgmap v4038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:42.964+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:10:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:42.964+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:10:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:42.964+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:10:43.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:42 smithi153 ceph-mon[36569]: pgmap v4039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:43.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:42 smithi104 ceph-mon[36624]: pgmap v4039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:43.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:42 smithi081 ceph-mon[31453]: pgmap v4039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:45.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:44 smithi104 ceph-mon[36624]: pgmap v4040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:45.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:44 smithi081 ceph-mon[31453]: pgmap v4040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:44 smithi153 ceph-mon[36569]: pgmap v4040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:47.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:46 smithi104 ceph-mon[36624]: pgmap v4041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:47.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:46 smithi081 ceph-mon[31453]: pgmap v4041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:47.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:46 smithi153 ceph-mon[36569]: pgmap v4041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:49.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:48 smithi104 ceph-mon[36624]: pgmap v4042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:49.097 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:48 smithi153 ceph-mon[36569]: pgmap v4042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:49.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:48 smithi081 ceph-mon[31453]: pgmap v4042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:50.955 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:50 smithi153 ceph-mon[36569]: pgmap v4043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:51.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:50 smithi104 ceph-mon[36624]: pgmap v4043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:51.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:50 smithi081 ceph-mon[31453]: pgmap v4043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:53.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:52 smithi104 ceph-mon[36624]: pgmap v4044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:53.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:52 smithi081 ceph-mon[31453]: pgmap v4044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:53.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:52 smithi153 ceph-mon[36569]: pgmap v4044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:55.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:54 smithi081 ceph-mon[31453]: pgmap v4045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:55.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:54 smithi153 ceph-mon[36569]: pgmap v4045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:55.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:54 smithi104 ceph-mon[36624]: pgmap v4045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:57.122 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:56 smithi104 ceph-mon[36624]: pgmap v4046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:57.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:56 smithi081 ceph-mon[31453]: pgmap v4046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:57.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:56 smithi153 ceph-mon[36569]: pgmap v4046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:57.974+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:10:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:57.974+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:10:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:10:57.974+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:10:59.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:10:58 smithi081 ceph-mon[31453]: pgmap v4047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:10:58 smithi153 ceph-mon[36569]: pgmap v4047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:10:59.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:10:58 smithi104 ceph-mon[36624]: pgmap v4047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:01.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:00 smithi081 ceph-mon[31453]: pgmap v4048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:01.258 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:00 smithi153 ceph-mon[36569]: pgmap v4048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:01.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:00 smithi104 ceph-mon[36624]: pgmap v4048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:03.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:02 smithi081 ceph-mon[31453]: pgmap v4049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:03.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:02 smithi153 ceph-mon[36569]: pgmap v4049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:03.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:02 smithi104 ceph-mon[36624]: pgmap v4049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:05.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:04 smithi081 ceph-mon[31453]: pgmap v4050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:04 smithi153 ceph-mon[36569]: pgmap v4050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:05.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:04 smithi104 ceph-mon[36624]: pgmap v4050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:07.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:06 smithi081 ceph-mon[31453]: pgmap v4051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:07.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:06 smithi153 ceph-mon[36569]: pgmap v4051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:07.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:06 smithi104 ceph-mon[36624]: pgmap v4051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:09.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:08 smithi081 ceph-mon[31453]: pgmap v4052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:09.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:08 smithi153 ceph-mon[36569]: pgmap v4052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:09.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:08 smithi104 ceph-mon[36624]: pgmap v4052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:11.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:10 smithi081 ceph-mon[31453]: pgmap v4053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:11.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:10 smithi153 ceph-mon[36569]: pgmap v4053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:11.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:10 smithi104 ceph-mon[36624]: pgmap v4053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:12.990+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:11:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:12.990+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:11:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:12.990+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:11:13.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:12 smithi081 ceph-mon[31453]: pgmap v4054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:12 smithi153 ceph-mon[36569]: pgmap v4054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:13.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:12 smithi104 ceph-mon[36624]: pgmap v4054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:15.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:14 smithi081 ceph-mon[31453]: pgmap v4055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:15.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:14 smithi153 ceph-mon[36569]: pgmap v4055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:15.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:14 smithi104 ceph-mon[36624]: pgmap v4055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:17.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:16 smithi081 ceph-mon[31453]: pgmap v4056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:17.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:16 smithi153 ceph-mon[36569]: pgmap v4056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:17.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:16 smithi104 ceph-mon[36624]: pgmap v4056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:19.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:18 smithi081 ceph-mon[31453]: pgmap v4057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:19.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:18 smithi153 ceph-mon[36569]: pgmap v4057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:19.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:18 smithi104 ceph-mon[36624]: pgmap v4057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:21.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:20 smithi081 ceph-mon[31453]: pgmap v4058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:21.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:20 smithi153 ceph-mon[36569]: pgmap v4058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:21.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:20 smithi104 ceph-mon[36624]: pgmap v4058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:23.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:22 smithi081 ceph-mon[31453]: pgmap v4059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:23.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:22 smithi153 ceph-mon[36569]: pgmap v4059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:23.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:22 smithi104 ceph-mon[36624]: pgmap v4059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:25.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:24 smithi153 ceph-mon[36569]: pgmap v4060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:25.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:24 smithi104 ceph-mon[36624]: pgmap v4060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:25.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:24 smithi081 ceph-mon[31453]: pgmap v4060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:27.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:26 smithi153 ceph-mon[36569]: pgmap v4061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:27.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:26 smithi104 ceph-mon[36624]: pgmap v4061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:27.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:26 smithi081 ceph-mon[31453]: pgmap v4061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:27.991+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:11:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:27.991+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:11:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:27.991+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:11:29.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:28 smithi153 ceph-mon[36569]: pgmap v4062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:29.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:28 smithi104 ceph-mon[36624]: pgmap v4062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:29.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:28 smithi081 ceph-mon[31453]: pgmap v4062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:30 smithi153 ceph-mon[36569]: pgmap v4063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:31.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:30 smithi104 ceph-mon[36624]: pgmap v4063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:31.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:30 smithi081 ceph-mon[31453]: pgmap v4063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:33.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:32 smithi153 ceph-mon[36569]: pgmap v4064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:33.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:32 smithi104 ceph-mon[36624]: pgmap v4064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:33.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:32 smithi081 ceph-mon[31453]: pgmap v4064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:35.197 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:34 smithi081 ceph-mon[31453]: pgmap v4065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:35.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:34 smithi153 ceph-mon[36569]: pgmap v4065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:35.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:34 smithi104 ceph-mon[36624]: pgmap v4065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:37.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:36 smithi153 ceph-mon[36569]: pgmap v4066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:37.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:36 smithi104 ceph-mon[36624]: pgmap v4066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:37.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:36 smithi081 ceph-mon[31453]: pgmap v4066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:39.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:38 smithi153 ceph-mon[36569]: pgmap v4067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:39.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:11:39.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:39.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:39.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:38 smithi104 ceph-mon[36624]: pgmap v4067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:39.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:11:39.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:39.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:39.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:38 smithi081 ceph-mon[31453]: pgmap v4067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:39.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:11:39.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:39.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:40.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:11:40.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:11:40.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:40.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:11:40.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:11:40.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:40.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:11:40.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:11:40.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:11:41.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:40 smithi153 ceph-mon[36569]: pgmap v4068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:41.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:40 smithi153 ceph-mon[36569]: pgmap v4069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:41.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:40 smithi104 ceph-mon[36624]: pgmap v4068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:41.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:40 smithi104 ceph-mon[36624]: pgmap v4069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:41.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:40 smithi081 ceph-mon[31453]: pgmap v4068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:41.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:40 smithi081 ceph-mon[31453]: pgmap v4069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:42.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:43.006+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:11:42.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:43.007+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:11:42.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:43.007+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:11:43.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:42 smithi153 ceph-mon[36569]: pgmap v4070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:43.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:42 smithi104 ceph-mon[36624]: pgmap v4070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:43.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:42 smithi081 ceph-mon[31453]: pgmap v4070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:43.838+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:11:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:43.838+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:11:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:43.838+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:11:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:43.838+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:11:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:44 smithi153 ceph-mon[36569]: pgmap v4071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:45.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:44 smithi104 ceph-mon[36624]: pgmap v4071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:45.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:44 smithi081 ceph-mon[31453]: pgmap v4071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:47.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:46 smithi153 ceph-mon[36569]: pgmap v4072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:47.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:46 smithi104 ceph-mon[36624]: pgmap v4072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:47.323 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:46 smithi081 ceph-mon[31453]: pgmap v4072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:49.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:48 smithi153 ceph-mon[36569]: pgmap v4073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:49.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:48 smithi104 ceph-mon[36624]: pgmap v4073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:49.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:48 smithi081 ceph-mon[31453]: pgmap v4073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:51.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:50 smithi153 ceph-mon[36569]: pgmap v4074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:51.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:50 smithi104 ceph-mon[36624]: pgmap v4074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:51.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:50 smithi081 ceph-mon[31453]: pgmap v4074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:52.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:52 smithi153 ceph-mon[36569]: pgmap v4075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:52.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:52 smithi104 ceph-mon[36624]: pgmap v4075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:52.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:52 smithi081 ceph-mon[31453]: pgmap v4075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:54.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:54 smithi081 ceph-mon[31453]: pgmap v4076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:54.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:54 smithi153 ceph-mon[36569]: pgmap v4076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:54.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:54 smithi104 ceph-mon[36624]: pgmap v4076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:56.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:56 smithi081 ceph-mon[31453]: pgmap v4077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:56 smithi153 ceph-mon[36569]: pgmap v4077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:56.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:56 smithi104 ceph-mon[36624]: pgmap v4077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:58.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:58.022+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:11:58.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:58.022+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:11:58.011 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:11:58.022+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:11:58.622 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:11:58 smithi104 ceph-mon[36624]: pgmap v4078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:58.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:11:58 smithi081 ceph-mon[31453]: pgmap v4078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:11:58.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:11:58 smithi153 ceph-mon[36569]: pgmap v4078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:00.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:00 smithi081 ceph-mon[31453]: pgmap v4079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:00.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:00 smithi153 ceph-mon[36569]: pgmap v4079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:00.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:00 smithi104 ceph-mon[36624]: pgmap v4079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:02.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:02 smithi081 ceph-mon[31453]: pgmap v4080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:02.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:02 smithi153 ceph-mon[36569]: pgmap v4080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:02.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:02 smithi104 ceph-mon[36624]: pgmap v4080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:04.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:04 smithi081 ceph-mon[31453]: pgmap v4081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:04.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:04 smithi153 ceph-mon[36569]: pgmap v4081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:04.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:04 smithi104 ceph-mon[36624]: pgmap v4081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:06.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:06 smithi081 ceph-mon[31453]: pgmap v4082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:06.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:06 smithi153 ceph-mon[36569]: pgmap v4082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:06.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:06 smithi104 ceph-mon[36624]: pgmap v4082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:08.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:08 smithi081 ceph-mon[31453]: pgmap v4083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:08.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:08 smithi153 ceph-mon[36569]: pgmap v4083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:08.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:08 smithi104 ceph-mon[36624]: pgmap v4083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:10.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:10 smithi081 ceph-mon[31453]: pgmap v4084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:10.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:10 smithi153 ceph-mon[36569]: pgmap v4084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:10.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:10 smithi104 ceph-mon[36624]: pgmap v4084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:12.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:12 smithi081 ceph-mon[31453]: pgmap v4085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:12.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:12 smithi153 ceph-mon[36569]: pgmap v4085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:12.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:12 smithi104 ceph-mon[36624]: pgmap v4085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:13.037+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:12:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:13.037+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:12:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:13.037+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:12:14.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:14 smithi081 ceph-mon[31453]: pgmap v4086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:14.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:14 smithi153 ceph-mon[36569]: pgmap v4086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:14.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:14 smithi104 ceph-mon[36624]: pgmap v4086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:16.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:16 smithi081 ceph-mon[31453]: pgmap v4087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:16.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:16 smithi153 ceph-mon[36569]: pgmap v4087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:16.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:16 smithi104 ceph-mon[36624]: pgmap v4087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:18.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:18 smithi081 ceph-mon[31453]: pgmap v4088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:18.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:18 smithi153 ceph-mon[36569]: pgmap v4088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:18.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:18 smithi104 ceph-mon[36624]: pgmap v4088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:21.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:20 smithi104 ceph-mon[36624]: pgmap v4089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:21.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:20 smithi081 ceph-mon[31453]: pgmap v4089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:21.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:20 smithi153 ceph-mon[36569]: pgmap v4089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:23.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:22 smithi104 ceph-mon[36624]: pgmap v4090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:23.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:22 smithi081 ceph-mon[31453]: pgmap v4090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:23.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:22 smithi153 ceph-mon[36569]: pgmap v4090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:25.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:24 smithi104 ceph-mon[36624]: pgmap v4091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:25.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:24 smithi081 ceph-mon[31453]: pgmap v4091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:25.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:24 smithi153 ceph-mon[36569]: pgmap v4091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:27.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:26 smithi081 ceph-mon[31453]: pgmap v4092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:27.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:26 smithi153 ceph-mon[36569]: pgmap v4092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:27.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:26 smithi104 ceph-mon[36624]: pgmap v4092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:28.038+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:12:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:28.038+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:12:28.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:28.038+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:12:29.122 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:28 smithi104 ceph-mon[36624]: pgmap v4093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:29.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:28 smithi081 ceph-mon[31453]: pgmap v4093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:29.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:28 smithi153 ceph-mon[36569]: pgmap v4093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:31.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:30 smithi081 ceph-mon[31453]: pgmap v4094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:30 smithi153 ceph-mon[36569]: pgmap v4094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:31.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:30 smithi104 ceph-mon[36624]: pgmap v4094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:33.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:32 smithi081 ceph-mon[31453]: pgmap v4095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:33.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:32 smithi153 ceph-mon[36569]: pgmap v4095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:33.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:32 smithi104 ceph-mon[36624]: pgmap v4095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:35.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:34 smithi081 ceph-mon[31453]: pgmap v4096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:35.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:34 smithi153 ceph-mon[36569]: pgmap v4096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:35.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:34 smithi104 ceph-mon[36624]: pgmap v4096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:37.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:36 smithi081 ceph-mon[31453]: pgmap v4097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:37.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:36 smithi153 ceph-mon[36569]: pgmap v4097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:37.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:36 smithi104 ceph-mon[36624]: pgmap v4097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:39.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:38 smithi081 ceph-mon[31453]: pgmap v4098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:39.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:38 smithi153 ceph-mon[36569]: pgmap v4098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:39.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:38 smithi104 ceph-mon[36624]: pgmap v4098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:40.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:12:40.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:12:40.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:12:41.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:40 smithi081 ceph-mon[31453]: pgmap v4099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:41.159 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:40 smithi104 ceph-mon[36624]: pgmap v4099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:41.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:40 smithi153 ceph-mon[36569]: pgmap v4099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:43.055+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:12:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:43.055+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:12:43.043 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:43.055+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:12:43.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:42 smithi081 ceph-mon[31453]: pgmap v4100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:43.194 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:42 smithi104 ceph-mon[36624]: pgmap v4100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:43.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:42 smithi153 ceph-mon[36569]: pgmap v4100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:45.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:44 smithi081 ceph-mon[31453]: pgmap v4101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:45.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:45.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:44 smithi153 ceph-mon[36569]: pgmap v4101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:45.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:44 smithi104 ceph-mon[36624]: pgmap v4101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:45.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:45.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:47.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:46 smithi081 ceph-mon[31453]: pgmap v4102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:47.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:46 smithi153 ceph-mon[36569]: pgmap v4102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:47.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:46 smithi104 ceph-mon[36624]: pgmap v4102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:48.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:48.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:48.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:12:48.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:12:48.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:47 smithi153 ceph-mon[36569]: pgmap v4103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:48.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:47 smithi153 ceph-mon[36569]: pgmap v4104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:48.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:12:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:12:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:47 smithi104 ceph-mon[36624]: pgmap v4103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:47 smithi104 ceph-mon[36624]: pgmap v4104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:48.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:48.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:48.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:48.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:12:48.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:12:48.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:47 smithi081 ceph-mon[31453]: pgmap v4103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:48.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:47 smithi081 ceph-mon[31453]: pgmap v4104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:48.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:12:49.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:49 smithi153 ceph-mon[36569]: pgmap v4105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:49.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:49 smithi104 ceph-mon[36624]: pgmap v4105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:49.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:49 smithi081 ceph-mon[31453]: pgmap v4105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:51.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:51 smithi153 ceph-mon[36569]: pgmap v4106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:51.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:51 smithi104 ceph-mon[36624]: pgmap v4106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:51.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:51 smithi081 ceph-mon[31453]: pgmap v4106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:53 smithi153 ceph-mon[36569]: pgmap v4107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:53.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:53 smithi104 ceph-mon[36624]: pgmap v4107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:53.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:53 smithi081 ceph-mon[31453]: pgmap v4107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:55.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:55 smithi153 ceph-mon[36569]: pgmap v4108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:55.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:55 smithi104 ceph-mon[36624]: pgmap v4108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:55.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:55 smithi081 ceph-mon[31453]: pgmap v4108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:57.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:57 smithi153 ceph-mon[36569]: pgmap v4109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:57.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:57 smithi104 ceph-mon[36624]: pgmap v4109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:57.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:57 smithi081 ceph-mon[31453]: pgmap v4109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:58.058 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:58.070+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:12:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:58.070+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:12:58.059 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:12:58.070+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:12:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:12:59 smithi153 ceph-mon[36569]: pgmap v4110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:59.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:12:59 smithi104 ceph-mon[36624]: pgmap v4110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:12:59.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:12:59 smithi081 ceph-mon[31453]: pgmap v4110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:01 smithi153 ceph-mon[36569]: pgmap v4111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:01.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:01 smithi104 ceph-mon[36624]: pgmap v4111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:01.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:01 smithi081 ceph-mon[31453]: pgmap v4111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:03.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:03 smithi153 ceph-mon[36569]: pgmap v4112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:03.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:03 smithi104 ceph-mon[36624]: pgmap v4112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:03.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:03 smithi081 ceph-mon[31453]: pgmap v4112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:05.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:05 smithi104 ceph-mon[36624]: pgmap v4113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:05.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:05 smithi081 ceph-mon[31453]: pgmap v4113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:05.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:05 smithi153 ceph-mon[36569]: pgmap v4113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:07.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:07 smithi153 ceph-mon[36569]: pgmap v4114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:07.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:07 smithi104 ceph-mon[36624]: pgmap v4114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:07.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:07 smithi081 ceph-mon[31453]: pgmap v4114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:09.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:09 smithi153 ceph-mon[36569]: pgmap v4115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:09.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:09 smithi104 ceph-mon[36624]: pgmap v4115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:09.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:09 smithi081 ceph-mon[31453]: pgmap v4115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:11.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:11 smithi104 ceph-mon[36624]: pgmap v4116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:11.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:11 smithi081 ceph-mon[31453]: pgmap v4116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:11.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:11 smithi153 ceph-mon[36569]: pgmap v4116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:13.081+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:13:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:13.081+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:13:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:13.081+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:13:13.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:13 smithi104 ceph-mon[36624]: pgmap v4117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:13.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:13 smithi081 ceph-mon[31453]: pgmap v4117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:13.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:13 smithi153 ceph-mon[36569]: pgmap v4117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:13.852+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:13:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:13.852+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:13:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:13.853+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:13:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:13.853+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:13:15.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:15 smithi104 ceph-mon[36624]: pgmap v4118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:15.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:15 smithi081 ceph-mon[31453]: pgmap v4118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:15.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:15 smithi153 ceph-mon[36569]: pgmap v4118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:17.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:17 smithi104 ceph-mon[36624]: pgmap v4119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:17.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:17 smithi081 ceph-mon[31453]: pgmap v4119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:17.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:17 smithi153 ceph-mon[36569]: pgmap v4119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:19.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:19 smithi104 ceph-mon[36624]: pgmap v4120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:19.339 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:19 smithi081 ceph-mon[31453]: pgmap v4120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:19.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:19 smithi153 ceph-mon[36569]: pgmap v4120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:21.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:21 smithi104 ceph-mon[36624]: pgmap v4121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:21.346 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:21 smithi153 ceph-mon[36569]: pgmap v4121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:21.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:21 smithi081 ceph-mon[31453]: pgmap v4121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:23.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:23 smithi104 ceph-mon[36624]: pgmap v4122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:23.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:23 smithi081 ceph-mon[31453]: pgmap v4122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:23.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:23 smithi153 ceph-mon[36569]: pgmap v4122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:25.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:25 smithi104 ceph-mon[36624]: pgmap v4123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:25.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:25 smithi081 ceph-mon[31453]: pgmap v4123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:25.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:25 smithi153 ceph-mon[36569]: pgmap v4123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:27.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:27 smithi104 ceph-mon[36624]: pgmap v4124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:27.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:27 smithi081 ceph-mon[31453]: pgmap v4124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:27.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:27 smithi153 ceph-mon[36569]: pgmap v4124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:28.096+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:13:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:28.096+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:13:28.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:28.096+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:13:29.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:29 smithi104 ceph-mon[36624]: pgmap v4125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:29.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:29 smithi081 ceph-mon[31453]: pgmap v4125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:29.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:29 smithi153 ceph-mon[36569]: pgmap v4125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:31.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:31 smithi104 ceph-mon[36624]: pgmap v4126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:31.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:31 smithi081 ceph-mon[31453]: pgmap v4126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:31.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:31 smithi153 ceph-mon[36569]: pgmap v4126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:33.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:33 smithi104 ceph-mon[36624]: pgmap v4127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:33.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:33 smithi081 ceph-mon[31453]: pgmap v4127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:33.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:33 smithi153 ceph-mon[36569]: pgmap v4127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:35.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:35 smithi104 ceph-mon[36624]: pgmap v4128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:35.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:35 smithi081 ceph-mon[31453]: pgmap v4128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:35.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:35 smithi153 ceph-mon[36569]: pgmap v4128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:37.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:37 smithi081 ceph-mon[31453]: pgmap v4129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:37.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:37 smithi104 ceph-mon[36624]: pgmap v4129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:37.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:37 smithi153 ceph-mon[36569]: pgmap v4129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:39.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:39 smithi104 ceph-mon[36624]: pgmap v4130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:39.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:39 smithi081 ceph-mon[31453]: pgmap v4130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:39.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:39 smithi153 ceph-mon[36569]: pgmap v4130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:41.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:41 smithi104 ceph-mon[36624]: pgmap v4131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:41.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:41 smithi081 ceph-mon[31453]: pgmap v4131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:41.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:41 smithi153 ceph-mon[36569]: pgmap v4131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:43.111+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:13:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:43.111+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:13:43.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:43.111+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:13:43.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:43 smithi104 ceph-mon[36624]: pgmap v4132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:43.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:43 smithi081 ceph-mon[31453]: pgmap v4132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:43.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:43 smithi153 ceph-mon[36569]: pgmap v4132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:45.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:45 smithi104 ceph-mon[36624]: pgmap v4133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:45.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:45 smithi081 ceph-mon[31453]: pgmap v4133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:45.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:45 smithi153 ceph-mon[36569]: pgmap v4133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:47.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:47 smithi104 ceph-mon[36624]: pgmap v4134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:47.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:47 smithi081 ceph-mon[31453]: pgmap v4134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:47.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:47 smithi153 ceph-mon[36569]: pgmap v4134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:48.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:13:48.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:13:48.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:13:49.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:49 smithi153 ceph-mon[36569]: pgmap v4135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:49.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:49 smithi104 ceph-mon[36624]: pgmap v4135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:49.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:49 smithi081 ceph-mon[31453]: pgmap v4135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:51.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:51 smithi153 ceph-mon[36569]: pgmap v4136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:51.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:51 smithi104 ceph-mon[36624]: pgmap v4136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:51.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:51 smithi081 ceph-mon[31453]: pgmap v4136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:13:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:13:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:53 smithi153 ceph-mon[36569]: pgmap v4137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:53 smithi153 ceph-mon[36569]: pgmap v4138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:53.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:53.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:53.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:53.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:13:53.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:13:53.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:53 smithi104 ceph-mon[36624]: pgmap v4137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:53.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:53 smithi104 ceph-mon[36624]: pgmap v4138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:53.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:53.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:53.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:53.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:13:53.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:13:53.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:53 smithi081 ceph-mon[31453]: pgmap v4137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:53.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:53 smithi081 ceph-mon[31453]: pgmap v4138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:53.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:13:55.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:55 smithi081 ceph-mon[31453]: pgmap v4139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:55.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:55 smithi153 ceph-mon[36569]: pgmap v4139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:55.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:55 smithi104 ceph-mon[36624]: pgmap v4139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:57.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:57 smithi081 ceph-mon[31453]: pgmap v4140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:57.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:57 smithi153 ceph-mon[36569]: pgmap v4140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:57.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:57 smithi104 ceph-mon[36624]: pgmap v4140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:58.126+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:13:58.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:58.127+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:13:58.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:13:58.127+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:13:59.622 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:13:59 smithi104 ceph-mon[36624]: pgmap v4141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:59.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:13:59 smithi081 ceph-mon[31453]: pgmap v4141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:13:59.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:13:59 smithi153 ceph-mon[36569]: pgmap v4141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:01.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:01 smithi081 ceph-mon[31453]: pgmap v4142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:01.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:01 smithi153 ceph-mon[36569]: pgmap v4142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:01.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:01 smithi104 ceph-mon[36624]: pgmap v4142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:03.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:03 smithi081 ceph-mon[31453]: pgmap v4143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:03.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:03 smithi153 ceph-mon[36569]: pgmap v4143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:03.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:03 smithi104 ceph-mon[36624]: pgmap v4143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:05.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:05 smithi081 ceph-mon[31453]: pgmap v4144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:05.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:05 smithi153 ceph-mon[36569]: pgmap v4144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:05.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:05 smithi104 ceph-mon[36624]: pgmap v4144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:07.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:07 smithi081 ceph-mon[31453]: pgmap v4145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:07.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:07 smithi153 ceph-mon[36569]: pgmap v4145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:07.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:07 smithi104 ceph-mon[36624]: pgmap v4145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:09.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:09 smithi081 ceph-mon[31453]: pgmap v4146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:09.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:09 smithi153 ceph-mon[36569]: pgmap v4146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:09.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:09 smithi104 ceph-mon[36624]: pgmap v4146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:11.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:11 smithi081 ceph-mon[31453]: pgmap v4147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:11.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:11 smithi153 ceph-mon[36569]: pgmap v4147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:11.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:11 smithi104 ceph-mon[36624]: pgmap v4147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:13.135+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:14:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:13.135+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:14:13.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:13.135+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:14:13.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:13 smithi081 ceph-mon[31453]: pgmap v4148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:13.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:13 smithi153 ceph-mon[36569]: pgmap v4148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:13.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:13 smithi104 ceph-mon[36624]: pgmap v4148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:15.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:15 smithi081 ceph-mon[31453]: pgmap v4149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:15.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:15 smithi153 ceph-mon[36569]: pgmap v4149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:15.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:15 smithi104 ceph-mon[36624]: pgmap v4149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:17.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:17 smithi081 ceph-mon[31453]: pgmap v4150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:17.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:17 smithi153 ceph-mon[36569]: pgmap v4150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:17.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:17 smithi104 ceph-mon[36624]: pgmap v4150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:19.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:19 smithi081 ceph-mon[31453]: pgmap v4151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:19.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:19 smithi153 ceph-mon[36569]: pgmap v4151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:19.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:19 smithi104 ceph-mon[36624]: pgmap v4151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:21.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:21 smithi153 ceph-mon[36569]: pgmap v4152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:21.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:21 smithi104 ceph-mon[36624]: pgmap v4152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:21.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:21 smithi081 ceph-mon[31453]: pgmap v4152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:23.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:23 smithi153 ceph-mon[36569]: pgmap v4153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:23.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:23 smithi104 ceph-mon[36624]: pgmap v4153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:23.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:23 smithi081 ceph-mon[31453]: pgmap v4153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:25.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:25 smithi153 ceph-mon[36569]: pgmap v4154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:25.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:25 smithi104 ceph-mon[36624]: pgmap v4154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:25.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:25 smithi081 ceph-mon[31453]: pgmap v4154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:27.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:27 smithi153 ceph-mon[36569]: pgmap v4155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:27.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:27 smithi104 ceph-mon[36624]: pgmap v4155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:27.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:27 smithi081 ceph-mon[31453]: pgmap v4155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:28.145+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:14:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:28.145+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:14:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:28.145+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:14:29.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:29 smithi153 ceph-mon[36569]: pgmap v4156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:29.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:29 smithi104 ceph-mon[36624]: pgmap v4156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:29.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:29 smithi081 ceph-mon[31453]: pgmap v4156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:31.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:31 smithi153 ceph-mon[36569]: pgmap v4157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:31.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:31 smithi104 ceph-mon[36624]: pgmap v4157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:31.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:31 smithi081 ceph-mon[31453]: pgmap v4157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:33.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:33 smithi153 ceph-mon[36569]: pgmap v4158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:33.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:33 smithi104 ceph-mon[36624]: pgmap v4158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:33.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:33 smithi081 ceph-mon[31453]: pgmap v4158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:35.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:35 smithi153 ceph-mon[36569]: pgmap v4159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:35.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:35 smithi104 ceph-mon[36624]: pgmap v4159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:35.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:35 smithi081 ceph-mon[31453]: pgmap v4159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:37.697 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:37 smithi081 ceph-mon[31453]: pgmap v4160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:37.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:37 smithi153 ceph-mon[36569]: pgmap v4160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:37.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:37 smithi104 ceph-mon[36624]: pgmap v4160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:39.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:39 smithi153 ceph-mon[36569]: pgmap v4161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:39.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:39 smithi104 ceph-mon[36624]: pgmap v4161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:39.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:39 smithi081 ceph-mon[31453]: pgmap v4161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:41.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:41 smithi153 ceph-mon[36569]: pgmap v4162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:41.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:41 smithi104 ceph-mon[36624]: pgmap v4162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:41.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:41 smithi081 ceph-mon[31453]: pgmap v4162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:43.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:43.154+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:14:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:43.154+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:14:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:43.154+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:14:43.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:43 smithi153 ceph-mon[36569]: pgmap v4163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:43.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:43 smithi104 ceph-mon[36624]: pgmap v4163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:43.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:43 smithi081 ceph-mon[31453]: pgmap v4163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:45.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:45 smithi153 ceph-mon[36569]: pgmap v4164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:45.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:45 smithi104 ceph-mon[36624]: pgmap v4164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:45.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:45 smithi081 ceph-mon[31453]: pgmap v4164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:47.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:47 smithi153 ceph-mon[36569]: pgmap v4165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:47.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:47 smithi104 ceph-mon[36624]: pgmap v4165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:47.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:47 smithi081 ceph-mon[31453]: pgmap v4165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:49.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:49 smithi153 ceph-mon[36569]: pgmap v4166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:49.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:49 smithi104 ceph-mon[36624]: pgmap v4166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:49.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:49 smithi081 ceph-mon[31453]: pgmap v4166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:51.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:51 smithi153 ceph-mon[36569]: pgmap v4167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:51.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:51 smithi104 ceph-mon[36624]: pgmap v4167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:51.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:51 smithi081 ceph-mon[31453]: pgmap v4167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:53.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:53 smithi153 ceph-mon[36569]: pgmap v4168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:53.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:14:53.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:53 smithi104 ceph-mon[36624]: pgmap v4168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:53.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:14:53.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:53 smithi081 ceph-mon[31453]: pgmap v4168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:53.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:14:54.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:14:54.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:14:54.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:54 smithi153 ceph-mon[36569]: pgmap v4169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:54.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:54 smithi153 ceph-mon[36569]: pgmap v4170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:54.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:14:54.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:14:54.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:14:54.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:54 smithi104 ceph-mon[36624]: pgmap v4169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:54.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:54 smithi104 ceph-mon[36624]: pgmap v4170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:54.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:14:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:14:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:14:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:54 smithi081 ceph-mon[31453]: pgmap v4169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:54 smithi081 ceph-mon[31453]: pgmap v4170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:54.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:14:57.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:56 smithi081 ceph-mon[31453]: pgmap v4171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:57.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:56 smithi153 ceph-mon[36569]: pgmap v4171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:57.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:56 smithi104 ceph-mon[36624]: pgmap v4171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:58.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:58.170+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:14:58.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:58.170+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:14:58.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:14:58.170+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:14:59.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:14:58 smithi081 ceph-mon[31453]: pgmap v4172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:14:58 smithi153 ceph-mon[36569]: pgmap v4172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:14:59.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:14:58 smithi104 ceph-mon[36624]: pgmap v4172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:00 smithi153 ceph-mon[36569]: pgmap v4173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:01.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:00 smithi104 ceph-mon[36624]: pgmap v4173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:01.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:00 smithi081 ceph-mon[31453]: pgmap v4173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:03.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:02 smithi153 ceph-mon[36569]: pgmap v4174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:03.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:02 smithi104 ceph-mon[36624]: pgmap v4174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:03.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:02 smithi081 ceph-mon[31453]: pgmap v4174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:04 smithi153 ceph-mon[36569]: pgmap v4175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:05.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:04 smithi104 ceph-mon[36624]: pgmap v4175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:05.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:04 smithi081 ceph-mon[31453]: pgmap v4175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:07.196 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:06 smithi081 ceph-mon[31453]: pgmap v4176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:07.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:06 smithi153 ceph-mon[36569]: pgmap v4176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:07.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:06 smithi104 ceph-mon[36624]: pgmap v4176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:09.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:08 smithi153 ceph-mon[36569]: pgmap v4177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:09.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:08 smithi104 ceph-mon[36624]: pgmap v4177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:09.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:08 smithi081 ceph-mon[31453]: pgmap v4177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:11.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:10 smithi153 ceph-mon[36569]: pgmap v4178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:11.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:10 smithi104 ceph-mon[36624]: pgmap v4178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:11.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:10 smithi081 ceph-mon[31453]: pgmap v4178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:13.185+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:15:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:13.185+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:15:13.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:13.185+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:15:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:12 smithi153 ceph-mon[36569]: pgmap v4179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:13.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:12 smithi104 ceph-mon[36624]: pgmap v4179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:13.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:12 smithi081 ceph-mon[31453]: pgmap v4179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:15.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:14 smithi153 ceph-mon[36569]: pgmap v4180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:15.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:14 smithi104 ceph-mon[36624]: pgmap v4180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:15.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:14 smithi081 ceph-mon[31453]: pgmap v4180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:17.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:16 smithi153 ceph-mon[36569]: pgmap v4181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:17.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:16 smithi104 ceph-mon[36624]: pgmap v4181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:17.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:16 smithi081 ceph-mon[31453]: pgmap v4181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:19.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:18 smithi153 ceph-mon[36569]: pgmap v4182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:19.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:18 smithi104 ceph-mon[36624]: pgmap v4182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:19.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:18 smithi081 ceph-mon[31453]: pgmap v4182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:21.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:20 smithi153 ceph-mon[36569]: pgmap v4183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:21.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:20 smithi104 ceph-mon[36624]: pgmap v4183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:21.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:20 smithi081 ceph-mon[31453]: pgmap v4183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:23.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:22 smithi153 ceph-mon[36569]: pgmap v4184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:23.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:22 smithi104 ceph-mon[36624]: pgmap v4184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:23.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:22 smithi081 ceph-mon[31453]: pgmap v4184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:25.196 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:24 smithi081 ceph-mon[31453]: pgmap v4185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:25.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:24 smithi153 ceph-mon[36569]: pgmap v4185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:25.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:24 smithi104 ceph-mon[36624]: pgmap v4185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:27.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:26 smithi153 ceph-mon[36569]: pgmap v4186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:27.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:26 smithi104 ceph-mon[36624]: pgmap v4186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:27.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:26 smithi081 ceph-mon[31453]: pgmap v4186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:28.194+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:15:28.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:28.194+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:15:28.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:28.194+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:15:29.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:28 smithi153 ceph-mon[36569]: pgmap v4187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:29.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:28 smithi104 ceph-mon[36624]: pgmap v4187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:29.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:28 smithi081 ceph-mon[31453]: pgmap v4187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:30 smithi153 ceph-mon[36569]: pgmap v4188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:31.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:30 smithi104 ceph-mon[36624]: pgmap v4188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:31.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:30 smithi081 ceph-mon[31453]: pgmap v4188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:33.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:32 smithi153 ceph-mon[36569]: pgmap v4189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:33.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:32 smithi104 ceph-mon[36624]: pgmap v4189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:33.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:32 smithi081 ceph-mon[31453]: pgmap v4189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:35.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:35 smithi153 ceph-mon[36569]: pgmap v4190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:35.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:35 smithi104 ceph-mon[36624]: pgmap v4190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:35.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:35 smithi081 ceph-mon[31453]: pgmap v4190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:37.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:37 smithi153 ceph-mon[36569]: pgmap v4191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:37.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:37 smithi104 ceph-mon[36624]: pgmap v4191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:37.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:37 smithi081 ceph-mon[31453]: pgmap v4191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:39.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:39 smithi153 ceph-mon[36569]: pgmap v4192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:39.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:39 smithi104 ceph-mon[36624]: pgmap v4192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:39.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:39 smithi081 ceph-mon[31453]: pgmap v4192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:41.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:41 smithi153 ceph-mon[36569]: pgmap v4193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:41.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:41 smithi104 ceph-mon[36624]: pgmap v4193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:41.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:41 smithi081 ceph-mon[31453]: pgmap v4193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:43.208+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:15:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:43.208+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:15:43.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:43.208+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:15:43.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:43 smithi153 ceph-mon[36569]: pgmap v4194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:43.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:43 smithi104 ceph-mon[36624]: pgmap v4194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:43.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:43 smithi081 ceph-mon[31453]: pgmap v4194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:45 smithi153 ceph-mon[36569]: pgmap v4195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:45.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:45 smithi104 ceph-mon[36624]: pgmap v4195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:45.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:45 smithi081 ceph-mon[31453]: pgmap v4195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:47.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:47 smithi153 ceph-mon[36569]: pgmap v4196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:47.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:47 smithi104 ceph-mon[36624]: pgmap v4196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:47.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:47 smithi081 ceph-mon[31453]: pgmap v4196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:49.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:49 smithi153 ceph-mon[36569]: pgmap v4197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:49.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:49 smithi104 ceph-mon[36624]: pgmap v4197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:49.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:49 smithi081 ceph-mon[31453]: pgmap v4197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:50.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:50 smithi081 ceph-mon[31453]: pgmap v4198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:50.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:50 smithi153 ceph-mon[36569]: pgmap v4198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:50.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:50 smithi104 ceph-mon[36624]: pgmap v4198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:53.247 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:52 smithi153 ceph-mon[36569]: pgmap v4199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:53.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:52 smithi104 ceph-mon[36624]: pgmap v4199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:53.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:52 smithi081 ceph-mon[31453]: pgmap v4199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:55.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:54 smithi153 ceph-mon[36569]: pgmap v4200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:55.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:15:55.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:54 smithi104 ceph-mon[36624]: pgmap v4200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:55.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:15:55.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:54 smithi081 ceph-mon[31453]: pgmap v4200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:55.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:15:56.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:15:56.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:15:56.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:15:56.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:15:56.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:15:56.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:15:56.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:15:56.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:15:56.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:15:57.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:56 smithi153 ceph-mon[36569]: pgmap v4201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:57.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:56 smithi153 ceph-mon[36569]: pgmap v4202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:57.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:56 smithi104 ceph-mon[36624]: pgmap v4201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:57.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:56 smithi104 ceph-mon[36624]: pgmap v4202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:57.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:56 smithi081 ceph-mon[31453]: pgmap v4201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:57.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:56 smithi081 ceph-mon[31453]: pgmap v4202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:58.220+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:15:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:58.220+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:15:58.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:15:58.220+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:15:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:15:58 smithi153 ceph-mon[36569]: pgmap v4203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:59.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:15:58 smithi104 ceph-mon[36624]: pgmap v4203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:15:59.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:15:58 smithi081 ceph-mon[31453]: pgmap v4203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:00 smithi153 ceph-mon[36569]: pgmap v4204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:01.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:00 smithi104 ceph-mon[36624]: pgmap v4204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:01.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:00 smithi081 ceph-mon[31453]: pgmap v4204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:03.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:02 smithi153 ceph-mon[36569]: pgmap v4205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:03.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:02 smithi104 ceph-mon[36624]: pgmap v4205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:03.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:02 smithi081 ceph-mon[31453]: pgmap v4205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:04 smithi153 ceph-mon[36569]: pgmap v4206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:05.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:04 smithi104 ceph-mon[36624]: pgmap v4206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:05.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:04 smithi081 ceph-mon[31453]: pgmap v4206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:07.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:06 smithi153 ceph-mon[36569]: pgmap v4207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:07.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:06 smithi104 ceph-mon[36624]: pgmap v4207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:07.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:06 smithi081 ceph-mon[31453]: pgmap v4207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:09.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:08 smithi153 ceph-mon[36569]: pgmap v4208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:09.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:08 smithi104 ceph-mon[36624]: pgmap v4208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:09.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:08 smithi081 ceph-mon[31453]: pgmap v4208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:11.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:10 smithi153 ceph-mon[36569]: pgmap v4209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:11.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:10 smithi104 ceph-mon[36624]: pgmap v4209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:11.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:10 smithi081 ceph-mon[31453]: pgmap v4209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.106+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T23:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.106+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T23:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.106+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006620 tx=0x7f16f40bd040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T23:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.106+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006620 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-12T23:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.106+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-12T23:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.106+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T23:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.106+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5790 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T23:16:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.106+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T23:16:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.107+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T23:16:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:12.108+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T23:16:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:13.234+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:16:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:13.234+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:16:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:13.234+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:16:13.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:12 smithi153 ceph-mon[36569]: pgmap v4210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:13.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:12 smithi104 ceph-mon[36624]: pgmap v4210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:13.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:12 smithi081 ceph-mon[31453]: pgmap v4210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:13.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:13.883+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:16:13.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:13.883+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:16:13.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:13.884+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:16:13.871 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:13.884+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:16:15.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:14 smithi153 ceph-mon[36569]: pgmap v4211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:15.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:14 smithi104 ceph-mon[36624]: pgmap v4211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:15.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:14 smithi081 ceph-mon[31453]: pgmap v4211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:17.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:17 smithi153 ceph-mon[36569]: pgmap v4212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:17.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:17 smithi104 ceph-mon[36624]: pgmap v4212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:17.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:17 smithi081 ceph-mon[31453]: pgmap v4212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:19.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:19 smithi153 ceph-mon[36569]: pgmap v4213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:19.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:19 smithi104 ceph-mon[36624]: pgmap v4213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:19.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:19 smithi081 ceph-mon[31453]: pgmap v4213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:21.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:21 smithi153 ceph-mon[36569]: pgmap v4214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:21.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:21 smithi104 ceph-mon[36624]: pgmap v4214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:21.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:21 smithi081 ceph-mon[31453]: pgmap v4214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:23.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:23 smithi153 ceph-mon[36569]: pgmap v4215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:23.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:23 smithi104 ceph-mon[36624]: pgmap v4215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:23.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:23 smithi081 ceph-mon[31453]: pgmap v4215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:25.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:25 smithi153 ceph-mon[36569]: pgmap v4216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:25.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:25 smithi104 ceph-mon[36624]: pgmap v4216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:25.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:25 smithi081 ceph-mon[31453]: pgmap v4216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:27.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:27 smithi081 ceph-mon[31453]: pgmap v4217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:27.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:27 smithi153 ceph-mon[36569]: pgmap v4217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:27.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:27 smithi104 ceph-mon[36624]: pgmap v4217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:28.241+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:16:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:28.241+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:16:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:28.241+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:16:29.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:29 smithi153 ceph-mon[36569]: pgmap v4218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:29.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:29 smithi104 ceph-mon[36624]: pgmap v4218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:29.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:29 smithi081 ceph-mon[31453]: pgmap v4218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:31.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:31 smithi153 ceph-mon[36569]: pgmap v4219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:31.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:31 smithi104 ceph-mon[36624]: pgmap v4219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:31.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:31 smithi081 ceph-mon[31453]: pgmap v4219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:33.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:33 smithi153 ceph-mon[36569]: pgmap v4220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:33.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:33 smithi104 ceph-mon[36624]: pgmap v4220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:33.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:33 smithi081 ceph-mon[31453]: pgmap v4220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:34.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:34 smithi081 ceph-mon[31453]: pgmap v4221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:34.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:34 smithi153 ceph-mon[36569]: pgmap v4221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:34.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:34 smithi104 ceph-mon[36624]: pgmap v4221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:36.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:36 smithi153 ceph-mon[36569]: pgmap v4222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:36.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:36 smithi104 ceph-mon[36624]: pgmap v4222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:36.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:36 smithi081 ceph-mon[31453]: pgmap v4222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:38.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:38 smithi153 ceph-mon[36569]: pgmap v4223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:38.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:38 smithi104 ceph-mon[36624]: pgmap v4223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:38.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:38 smithi081 ceph-mon[31453]: pgmap v4223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:40.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:40 smithi153 ceph-mon[36569]: pgmap v4224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:40.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:40 smithi104 ceph-mon[36624]: pgmap v4224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:40.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:40 smithi081 ceph-mon[31453]: pgmap v4224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:42.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:42 smithi153 ceph-mon[36569]: pgmap v4225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:42.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:42 smithi104 ceph-mon[36624]: pgmap v4225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:42.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:42 smithi081 ceph-mon[31453]: pgmap v4225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:43.254+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:16:43.242 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:43.254+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:16:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:43.254+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:16:44.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:44 smithi153 ceph-mon[36569]: pgmap v4226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:44.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:44 smithi104 ceph-mon[36624]: pgmap v4226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:44.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:44 smithi081 ceph-mon[31453]: pgmap v4226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:46.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:46 smithi153 ceph-mon[36569]: pgmap v4227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:46.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:46 smithi104 ceph-mon[36624]: pgmap v4227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:46.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:46 smithi081 ceph-mon[31453]: pgmap v4227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:48.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:48 smithi153 ceph-mon[36569]: pgmap v4228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:48.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:48 smithi104 ceph-mon[36624]: pgmap v4228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:48.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:48 smithi081 ceph-mon[31453]: pgmap v4228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:50.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:50 smithi153 ceph-mon[36569]: pgmap v4229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:50.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:50 smithi104 ceph-mon[36624]: pgmap v4229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:50.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:50 smithi081 ceph-mon[31453]: pgmap v4229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:52.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:52 smithi153 ceph-mon[36569]: pgmap v4230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:52.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:52 smithi104 ceph-mon[36624]: pgmap v4230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:52.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:52 smithi081 ceph-mon[31453]: pgmap v4230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:54.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:54 smithi153 ceph-mon[36569]: pgmap v4231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:54.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:54 smithi104 ceph-mon[36624]: pgmap v4231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:54.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:54 smithi081 ceph-mon[31453]: pgmap v4231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: pgmap v4232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:16:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: pgmap v4232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:56.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:16:56.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: pgmap v4232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:56.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:16:57.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:16:57.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:16:57.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:16:57.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:16:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:16:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:16:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:16:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: pgmap v4233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: pgmap v4234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:57.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:16:57.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:16:57.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: pgmap v4233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: pgmap v4234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:57.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:16:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:16:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:16:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:16:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:16:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:16:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:16:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:16:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:16:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:16:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: pgmap v4233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: pgmap v4234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:16:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:16:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:58.270+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:16:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:58.270+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:16:58.258 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:16:58.270+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:17:00.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:16:59 smithi153 ceph-mon[36569]: pgmap v4235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:00.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:16:59 smithi104 ceph-mon[36624]: pgmap v4235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:00.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:16:59 smithi081 ceph-mon[31453]: pgmap v4235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:02.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:02 smithi153 ceph-mon[36569]: pgmap v4236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:02.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:02 smithi104 ceph-mon[36624]: pgmap v4236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:02.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:02 smithi081 ceph-mon[31453]: pgmap v4236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:04.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:04 smithi153 ceph-mon[36569]: pgmap v4237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:04.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:04 smithi104 ceph-mon[36624]: pgmap v4237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:04.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:04 smithi081 ceph-mon[31453]: pgmap v4237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:06.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:06 smithi153 ceph-mon[36569]: pgmap v4238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:06.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:06 smithi104 ceph-mon[36624]: pgmap v4238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:06.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:06 smithi081 ceph-mon[31453]: pgmap v4238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:08.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:08 smithi153 ceph-mon[36569]: pgmap v4239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:08.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:08 smithi104 ceph-mon[36624]: pgmap v4239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:08.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:08 smithi081 ceph-mon[31453]: pgmap v4239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:10 smithi153 ceph-mon[36569]: pgmap v4240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:10.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:10 smithi104 ceph-mon[36624]: pgmap v4240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:10.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:10 smithi081 ceph-mon[31453]: pgmap v4240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:12 smithi153 ceph-mon[36569]: pgmap v4241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:12.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:12 smithi104 ceph-mon[36624]: pgmap v4241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:12.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:12 smithi081 ceph-mon[31453]: pgmap v4241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:13.286+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:17:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:13.286+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:17:13.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:13.286+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:17:14.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:14 smithi153 ceph-mon[36569]: pgmap v4242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:14.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:14 smithi104 ceph-mon[36624]: pgmap v4242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:14.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:14 smithi081 ceph-mon[31453]: pgmap v4242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:16.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:16 smithi153 ceph-mon[36569]: pgmap v4243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:16.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:16 smithi104 ceph-mon[36624]: pgmap v4243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:16.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:16 smithi081 ceph-mon[31453]: pgmap v4243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:18.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:18 smithi153 ceph-mon[36569]: pgmap v4244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:18.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:18 smithi104 ceph-mon[36624]: pgmap v4244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:18.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:18 smithi081 ceph-mon[31453]: pgmap v4244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:20.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:20 smithi153 ceph-mon[36569]: pgmap v4245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:20.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:20 smithi104 ceph-mon[36624]: pgmap v4245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:20.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:20 smithi081 ceph-mon[31453]: pgmap v4245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:22.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:22 smithi153 ceph-mon[36569]: pgmap v4246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:22.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:22 smithi104 ceph-mon[36624]: pgmap v4246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:22.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:22 smithi081 ceph-mon[31453]: pgmap v4246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:24.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:24 smithi104 ceph-mon[36624]: pgmap v4247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:24.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:24 smithi081 ceph-mon[31453]: pgmap v4247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:24.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:24 smithi153 ceph-mon[36569]: pgmap v4247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:25.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:25 smithi104 ceph-mon[36624]: pgmap v4248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:25.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:25 smithi081 ceph-mon[31453]: pgmap v4248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:25.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:25 smithi153 ceph-mon[36569]: pgmap v4248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:28.293+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:17:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:28.293+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:17:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:28.293+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:17:28.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:28 smithi153 ceph-mon[36569]: pgmap v4249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:28.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:28 smithi104 ceph-mon[36624]: pgmap v4249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:28.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:28 smithi081 ceph-mon[31453]: pgmap v4249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:30.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:30 smithi153 ceph-mon[36569]: pgmap v4250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:30.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:30 smithi104 ceph-mon[36624]: pgmap v4250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:30.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:30 smithi081 ceph-mon[31453]: pgmap v4250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:32.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:32 smithi153 ceph-mon[36569]: pgmap v4251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:32.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:32 smithi104 ceph-mon[36624]: pgmap v4251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:32.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:32 smithi081 ceph-mon[31453]: pgmap v4251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:34.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:34 smithi153 ceph-mon[36569]: pgmap v4252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:34.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:34 smithi104 ceph-mon[36624]: pgmap v4252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:34.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:34 smithi081 ceph-mon[31453]: pgmap v4252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:36.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:36 smithi153 ceph-mon[36569]: pgmap v4253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:36.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:36 smithi104 ceph-mon[36624]: pgmap v4253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:36.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:36 smithi081 ceph-mon[31453]: pgmap v4253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:38.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:38 smithi153 ceph-mon[36569]: pgmap v4254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:38.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:38 smithi104 ceph-mon[36624]: pgmap v4254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:38.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:38 smithi081 ceph-mon[31453]: pgmap v4254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:40.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:40 smithi153 ceph-mon[36569]: pgmap v4255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:40.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:40 smithi104 ceph-mon[36624]: pgmap v4255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:40.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:40 smithi081 ceph-mon[31453]: pgmap v4255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:42.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:42 smithi153 ceph-mon[36569]: pgmap v4256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:42.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:42 smithi104 ceph-mon[36624]: pgmap v4256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:42.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:42 smithi081 ceph-mon[31453]: pgmap v4256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:43.299+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:17:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:43.299+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:17:43.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:43.299+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:17:44.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:44 smithi153 ceph-mon[36569]: pgmap v4257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:44.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:44 smithi104 ceph-mon[36624]: pgmap v4257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:44.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:44 smithi081 ceph-mon[31453]: pgmap v4257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:46.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:46 smithi104 ceph-mon[36624]: pgmap v4258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:46.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:46 smithi081 ceph-mon[31453]: pgmap v4258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:46.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:46 smithi153 ceph-mon[36569]: pgmap v4258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:47.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:47 smithi081 ceph-mon[31453]: pgmap v4259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:47 smithi153 ceph-mon[36569]: pgmap v4259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:47.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:47 smithi104 ceph-mon[36624]: pgmap v4259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:50.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:50 smithi153 ceph-mon[36569]: pgmap v4260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:50.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:50 smithi104 ceph-mon[36624]: pgmap v4260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:50.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:50 smithi081 ceph-mon[31453]: pgmap v4260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:52.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:52 smithi153 ceph-mon[36569]: pgmap v4261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:52.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:52 smithi104 ceph-mon[36624]: pgmap v4261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:52.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:52 smithi081 ceph-mon[31453]: pgmap v4261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:54.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:54 smithi153 ceph-mon[36569]: pgmap v4262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:54.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:54 smithi104 ceph-mon[36624]: pgmap v4262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:54.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:54 smithi081 ceph-mon[31453]: pgmap v4262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:56.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:56 smithi153 ceph-mon[36569]: pgmap v4263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:56.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:56 smithi104 ceph-mon[36624]: pgmap v4263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:56.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:56 smithi081 ceph-mon[31453]: pgmap v4263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:58.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:58 smithi153 ceph-mon[36569]: pgmap v4264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:58.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:17:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:58.315+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:17:58.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:58.315+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:17:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:17:58.315+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:17:58.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:58 smithi104 ceph-mon[36624]: pgmap v4264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:58.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:17:58.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:58 smithi081 ceph-mon[31453]: pgmap v4264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:17:58.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:17:59.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:17:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:17:59.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:17:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:17:59.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:17:59.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:17:59.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:17:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:17:59.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:17:59.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:17:59.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:17:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:18:00.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:00 smithi104 ceph-mon[36624]: pgmap v4265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:00.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:00 smithi104 ceph-mon[36624]: pgmap v4266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:00.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:00 smithi081 ceph-mon[31453]: pgmap v4265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:00.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:00 smithi081 ceph-mon[31453]: pgmap v4266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:00.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:00 smithi153 ceph-mon[36569]: pgmap v4265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:00.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:00 smithi153 ceph-mon[36569]: pgmap v4266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:02.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:02 smithi104 ceph-mon[36624]: pgmap v4267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:02.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:02 smithi081 ceph-mon[31453]: pgmap v4267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:02 smithi153 ceph-mon[36569]: pgmap v4267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:03.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:03 smithi081 ceph-mon[31453]: pgmap v4268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:03.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:03 smithi153 ceph-mon[36569]: pgmap v4268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:03.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:03 smithi104 ceph-mon[36624]: pgmap v4268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:05.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:05 smithi104 ceph-mon[36624]: pgmap v4269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:05.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:05 smithi081 ceph-mon[31453]: pgmap v4269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:06.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:05 smithi153 ceph-mon[36569]: pgmap v4269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:07.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:07 smithi104 ceph-mon[36624]: pgmap v4270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:07.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:07 smithi081 ceph-mon[31453]: pgmap v4270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:08.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:07 smithi153 ceph-mon[36569]: pgmap v4270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:09.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:09 smithi104 ceph-mon[36624]: pgmap v4271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:09.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:09 smithi081 ceph-mon[31453]: pgmap v4271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:10.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:09 smithi153 ceph-mon[36569]: pgmap v4271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:11.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:11 smithi104 ceph-mon[36624]: pgmap v4272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:11.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:11 smithi081 ceph-mon[31453]: pgmap v4272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:12.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:11 smithi153 ceph-mon[36569]: pgmap v4272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:13.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:13.327+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:18:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:13.327+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:18:13.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:13.327+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:18:13.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:13 smithi104 ceph-mon[36624]: pgmap v4273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:13.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:13 smithi081 ceph-mon[31453]: pgmap v4273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:14.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:13 smithi153 ceph-mon[36569]: pgmap v4273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:15.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:15 smithi081 ceph-mon[31453]: pgmap v4274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:16.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:15 smithi153 ceph-mon[36569]: pgmap v4274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:16.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:15 smithi104 ceph-mon[36624]: pgmap v4274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:17.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:17 smithi081 ceph-mon[31453]: pgmap v4275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:18.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:17 smithi153 ceph-mon[36569]: pgmap v4275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:18.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:17 smithi104 ceph-mon[36624]: pgmap v4275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:19.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:19 smithi081 ceph-mon[31453]: pgmap v4276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:20.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:19 smithi153 ceph-mon[36569]: pgmap v4276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:20.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:19 smithi104 ceph-mon[36624]: pgmap v4276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:21.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:21 smithi081 ceph-mon[31453]: pgmap v4277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:22.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:21 smithi153 ceph-mon[36569]: pgmap v4277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:22.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:21 smithi104 ceph-mon[36624]: pgmap v4277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:23.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:23 smithi081 ceph-mon[31453]: pgmap v4278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:24.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:23 smithi153 ceph-mon[36569]: pgmap v4278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:24.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:23 smithi104 ceph-mon[36624]: pgmap v4278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:25.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:25 smithi081 ceph-mon[31453]: pgmap v4279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:26.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:25 smithi153 ceph-mon[36569]: pgmap v4279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:26.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:25 smithi104 ceph-mon[36624]: pgmap v4279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:27.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:27 smithi081 ceph-mon[31453]: pgmap v4280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:28.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:27 smithi153 ceph-mon[36569]: pgmap v4280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:28.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:27 smithi104 ceph-mon[36624]: pgmap v4280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:28.342+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:18:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:28.342+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:18:28.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:28.342+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:18:29.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:29 smithi081 ceph-mon[31453]: pgmap v4281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:30.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:29 smithi153 ceph-mon[36569]: pgmap v4281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:30.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:29 smithi104 ceph-mon[36624]: pgmap v4281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:31.872 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:31 smithi104 ceph-mon[36624]: pgmap v4282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:31.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:31 smithi081 ceph-mon[31453]: pgmap v4282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:32.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:31 smithi153 ceph-mon[36569]: pgmap v4282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:33.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:33 smithi081 ceph-mon[31453]: pgmap v4283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:34.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:33 smithi153 ceph-mon[36569]: pgmap v4283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:34.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:33 smithi104 ceph-mon[36624]: pgmap v4283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:35.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:35 smithi081 ceph-mon[31453]: pgmap v4284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:36.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:35 smithi153 ceph-mon[36569]: pgmap v4284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:36.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:35 smithi104 ceph-mon[36624]: pgmap v4284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:37.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:37 smithi081 ceph-mon[31453]: pgmap v4285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:38.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:37 smithi153 ceph-mon[36569]: pgmap v4285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:38.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:37 smithi104 ceph-mon[36624]: pgmap v4285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:39.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:39 smithi081 ceph-mon[31453]: pgmap v4286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:40.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:39 smithi153 ceph-mon[36569]: pgmap v4286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:40.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:39 smithi104 ceph-mon[36624]: pgmap v4286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:41.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:41 smithi081 ceph-mon[31453]: pgmap v4287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:42.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:41 smithi153 ceph-mon[36569]: pgmap v4287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:42.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:41 smithi104 ceph-mon[36624]: pgmap v4287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:43.357+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:18:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:43.357+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:18:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:43.357+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:18:44.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:43 smithi153 ceph-mon[36569]: pgmap v4288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:44.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:43 smithi104 ceph-mon[36624]: pgmap v4288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:44.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:43 smithi081 ceph-mon[31453]: pgmap v4288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:46.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:45 smithi153 ceph-mon[36569]: pgmap v4289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:46.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:45 smithi104 ceph-mon[36624]: pgmap v4289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:46.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:45 smithi081 ceph-mon[31453]: pgmap v4289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:48.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:47 smithi153 ceph-mon[36569]: pgmap v4290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:48.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:47 smithi104 ceph-mon[36624]: pgmap v4290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:48.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:47 smithi081 ceph-mon[31453]: pgmap v4290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:50.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:49 smithi153 ceph-mon[36569]: pgmap v4291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:50.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:49 smithi104 ceph-mon[36624]: pgmap v4291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:50.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:49 smithi081 ceph-mon[31453]: pgmap v4291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:52.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:51 smithi153 ceph-mon[36569]: pgmap v4292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:52.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:51 smithi104 ceph-mon[36624]: pgmap v4292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:52.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:51 smithi081 ceph-mon[31453]: pgmap v4292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:54.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:53 smithi153 ceph-mon[36569]: pgmap v4293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:54.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:53 smithi104 ceph-mon[36624]: pgmap v4293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:54.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:53 smithi081 ceph-mon[31453]: pgmap v4293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:56.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:55 smithi153 ceph-mon[36569]: pgmap v4294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:56.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:55 smithi104 ceph-mon[36624]: pgmap v4294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:56.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:55 smithi081 ceph-mon[31453]: pgmap v4294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:58.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:57 smithi153 ceph-mon[36569]: pgmap v4295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:58.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:57 smithi104 ceph-mon[36624]: pgmap v4295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:58.372+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:18:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:58.372+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:18:58.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:18:58.372+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:18:58.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:57 smithi081 ceph-mon[31453]: pgmap v4295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:18:59.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:18:59.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:18:59.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:19:00.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:18:59 smithi153 ceph-mon[36569]: pgmap v4296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:00.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:18:59 smithi104 ceph-mon[36624]: pgmap v4296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:00.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:18:59 smithi081 ceph-mon[31453]: pgmap v4296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:01.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:19:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:19:01.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:19:01.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:19:01.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:19:01.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:19:01.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:19:01.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:19:01.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:19:02.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:01 smithi153 ceph-mon[36569]: pgmap v4297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:02.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:01 smithi153 ceph-mon[36569]: pgmap v4298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:02.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:01 smithi104 ceph-mon[36624]: pgmap v4297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:02.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:01 smithi104 ceph-mon[36624]: pgmap v4298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:02.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:01 smithi081 ceph-mon[31453]: pgmap v4297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:02.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:01 smithi081 ceph-mon[31453]: pgmap v4298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:04.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:03 smithi153 ceph-mon[36569]: pgmap v4299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:04.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:03 smithi104 ceph-mon[36624]: pgmap v4299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:04.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:03 smithi081 ceph-mon[31453]: pgmap v4299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:06.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:05 smithi153 ceph-mon[36569]: pgmap v4300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:06.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:05 smithi104 ceph-mon[36624]: pgmap v4300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:06.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:05 smithi081 ceph-mon[31453]: pgmap v4300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:08.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:08 smithi153 ceph-mon[36569]: pgmap v4301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:08.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:08 smithi104 ceph-mon[36624]: pgmap v4301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:08.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:08 smithi081 ceph-mon[31453]: pgmap v4301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:10.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:10 smithi153 ceph-mon[36569]: pgmap v4302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:10.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:10 smithi104 ceph-mon[36624]: pgmap v4302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:10.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:10 smithi081 ceph-mon[31453]: pgmap v4302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:12.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:12 smithi153 ceph-mon[36569]: pgmap v4303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:12.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:12 smithi104 ceph-mon[36624]: pgmap v4303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:12.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:12 smithi081 ceph-mon[31453]: pgmap v4303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:13.380+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:19:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:13.380+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:19:13.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:13.380+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:19:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:13.913+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:19:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:13.913+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:19:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:13.913+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:19:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:13.913+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:19:14.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:14 smithi153 ceph-mon[36569]: pgmap v4304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:14.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:14 smithi104 ceph-mon[36624]: pgmap v4304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:14.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:14 smithi081 ceph-mon[31453]: pgmap v4304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:16.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:16 smithi153 ceph-mon[36569]: pgmap v4305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:16.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:16 smithi104 ceph-mon[36624]: pgmap v4305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:16.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:16 smithi081 ceph-mon[31453]: pgmap v4305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:18.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:18 smithi081 ceph-mon[31453]: pgmap v4306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:18.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:18 smithi153 ceph-mon[36569]: pgmap v4306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:18.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:18 smithi104 ceph-mon[36624]: pgmap v4306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:20.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:20 smithi153 ceph-mon[36569]: pgmap v4307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:20.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:20 smithi104 ceph-mon[36624]: pgmap v4307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:20.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:20 smithi081 ceph-mon[31453]: pgmap v4307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:22.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:22 smithi153 ceph-mon[36569]: pgmap v4308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:22.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:22 smithi104 ceph-mon[36624]: pgmap v4308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:22.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:22 smithi081 ceph-mon[31453]: pgmap v4308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:24.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:24 smithi153 ceph-mon[36569]: pgmap v4309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:24.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:24 smithi104 ceph-mon[36624]: pgmap v4309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:24.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:24 smithi081 ceph-mon[31453]: pgmap v4309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:26.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:26 smithi153 ceph-mon[36569]: pgmap v4310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:26.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:26 smithi104 ceph-mon[36624]: pgmap v4310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:26.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:26 smithi081 ceph-mon[31453]: pgmap v4310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:28.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:28 smithi153 ceph-mon[36569]: pgmap v4311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:28.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:28 smithi104 ceph-mon[36624]: pgmap v4311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:28.396+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:19:28.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:28.396+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:19:28.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:28.396+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:19:28.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:28 smithi081 ceph-mon[31453]: pgmap v4311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:30.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:30 smithi104 ceph-mon[36624]: pgmap v4312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:30.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:30 smithi081 ceph-mon[31453]: pgmap v4312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:30.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:30 smithi153 ceph-mon[36569]: pgmap v4312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:32.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:32 smithi104 ceph-mon[36624]: pgmap v4313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:32.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:32 smithi081 ceph-mon[31453]: pgmap v4313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:32.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:32 smithi153 ceph-mon[36569]: pgmap v4313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:34.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:34 smithi104 ceph-mon[36624]: pgmap v4314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:34.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:34 smithi081 ceph-mon[31453]: pgmap v4314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:34.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:34 smithi153 ceph-mon[36569]: pgmap v4314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:36.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:36 smithi104 ceph-mon[36624]: pgmap v4315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:36.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:36 smithi081 ceph-mon[31453]: pgmap v4315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:36.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:36 smithi153 ceph-mon[36569]: pgmap v4315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:38.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:38 smithi104 ceph-mon[36624]: pgmap v4316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:38.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:38 smithi081 ceph-mon[31453]: pgmap v4316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:38.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:38 smithi153 ceph-mon[36569]: pgmap v4316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:40.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:40 smithi081 ceph-mon[31453]: pgmap v4317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:40.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:40 smithi153 ceph-mon[36569]: pgmap v4317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:40.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:40 smithi104 ceph-mon[36624]: pgmap v4317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:41.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:41 smithi081 ceph-mon[31453]: pgmap v4318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:41 smithi153 ceph-mon[36569]: pgmap v4318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:41.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:41 smithi104 ceph-mon[36624]: pgmap v4318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:43.410+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:19:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:43.410+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:19:43.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:43.410+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:19:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:43 smithi153 ceph-mon[36569]: pgmap v4319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:43.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:43 smithi104 ceph-mon[36624]: pgmap v4319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:43.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:43 smithi081 ceph-mon[31453]: pgmap v4319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:45 smithi153 ceph-mon[36569]: pgmap v4320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:45.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:45 smithi104 ceph-mon[36624]: pgmap v4320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:45.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:45 smithi081 ceph-mon[31453]: pgmap v4320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:47 smithi153 ceph-mon[36569]: pgmap v4321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:47.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:47 smithi104 ceph-mon[36624]: pgmap v4321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:47.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:47 smithi081 ceph-mon[31453]: pgmap v4321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:49.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:49 smithi081 ceph-mon[31453]: pgmap v4322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:49 smithi153 ceph-mon[36569]: pgmap v4322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:49.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:49 smithi104 ceph-mon[36624]: pgmap v4322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:51 smithi153 ceph-mon[36569]: pgmap v4323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:51.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:51 smithi104 ceph-mon[36624]: pgmap v4323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:51.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:51 smithi081 ceph-mon[31453]: pgmap v4323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:53 smithi153 ceph-mon[36569]: pgmap v4324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:53.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:53 smithi104 ceph-mon[36624]: pgmap v4324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:53.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:53 smithi081 ceph-mon[31453]: pgmap v4324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:55 smithi153 ceph-mon[36569]: pgmap v4325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:55.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:55 smithi104 ceph-mon[36624]: pgmap v4325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:55.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:55 smithi081 ceph-mon[31453]: pgmap v4325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:57.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:57 smithi153 ceph-mon[36569]: pgmap v4326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:57.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:57 smithi104 ceph-mon[36624]: pgmap v4326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:57.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:57 smithi081 ceph-mon[31453]: pgmap v4326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:58.417+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:19:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:58.417+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:19:58.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:58.417+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:19:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:58.921+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:19:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:58.921+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:19:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:58.922+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:19:58.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:19:58.922+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:19:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:19:59 smithi153 ceph-mon[36569]: pgmap v4327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:59.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:19:59 smithi104 ceph-mon[36624]: pgmap v4327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:19:59.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:19:59 smithi081 ceph-mon[31453]: pgmap v4327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:00.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:20:00.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:20:00.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:20:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:01 smithi153 ceph-mon[36569]: pgmap v4328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:01.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:20:01.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:01 smithi104 ceph-mon[36624]: pgmap v4328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:01.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:20:01.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:01 smithi081 ceph-mon[31453]: pgmap v4328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:01.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:20:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:20:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:20:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:02 smithi153 ceph-mon[36569]: pgmap v4329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:02 smithi153 ceph-mon[36569]: pgmap v4330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:02.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:20:02.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:20:02.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:20:02.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:02 smithi104 ceph-mon[36624]: pgmap v4329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:02.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:02 smithi104 ceph-mon[36624]: pgmap v4330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:02.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:20:02.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:20:02.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:20:02.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:02 smithi081 ceph-mon[31453]: pgmap v4329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:02.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:02 smithi081 ceph-mon[31453]: pgmap v4330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:02.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:20:04.986 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:04 smithi081 ceph-mon[31453]: pgmap v4331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:05.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:04 smithi153 ceph-mon[36569]: pgmap v4331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:05.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:04 smithi104 ceph-mon[36624]: pgmap v4331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:07.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:06 smithi153 ceph-mon[36569]: pgmap v4332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:07.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:06 smithi104 ceph-mon[36624]: pgmap v4332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:07.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:06 smithi081 ceph-mon[31453]: pgmap v4332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:09.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:08 smithi153 ceph-mon[36569]: pgmap v4333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:09.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:08 smithi104 ceph-mon[36624]: pgmap v4333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:09.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:08 smithi081 ceph-mon[31453]: pgmap v4333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:10.946 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:10 smithi081 ceph-mon[31453]: pgmap v4334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:11.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:10 smithi153 ceph-mon[36569]: pgmap v4334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:11.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:10 smithi104 ceph-mon[36624]: pgmap v4334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:13.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:12 smithi153 ceph-mon[36569]: pgmap v4335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:13.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:12 smithi104 ceph-mon[36624]: pgmap v4335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:13.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:12 smithi081 ceph-mon[31453]: pgmap v4335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:13.416 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:13.428+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:20:13.416 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:13.428+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:20:13.416 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:13.428+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:20:15.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:14 smithi153 ceph-mon[36569]: pgmap v4336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:15.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:14 smithi104 ceph-mon[36624]: pgmap v4336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:15.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:14 smithi081 ceph-mon[31453]: pgmap v4336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:17.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:16 smithi153 ceph-mon[36569]: pgmap v4337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:17.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:16 smithi104 ceph-mon[36624]: pgmap v4337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:17.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:16 smithi081 ceph-mon[31453]: pgmap v4337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:19.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:18 smithi153 ceph-mon[36569]: pgmap v4338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:19.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:18 smithi104 ceph-mon[36624]: pgmap v4338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:19.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:18 smithi081 ceph-mon[31453]: pgmap v4338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:21.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:20 smithi153 ceph-mon[36569]: pgmap v4339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:21.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:20 smithi104 ceph-mon[36624]: pgmap v4339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:21.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:20 smithi081 ceph-mon[31453]: pgmap v4339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:23.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:22 smithi153 ceph-mon[36569]: pgmap v4340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:23.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:22 smithi104 ceph-mon[36624]: pgmap v4340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:23.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:22 smithi081 ceph-mon[31453]: pgmap v4340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:25.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:25 smithi153 ceph-mon[36569]: pgmap v4341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:25.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:25 smithi104 ceph-mon[36624]: pgmap v4341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:25.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:25 smithi081 ceph-mon[31453]: pgmap v4341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:27.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:27 smithi153 ceph-mon[36569]: pgmap v4342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:27.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:27 smithi104 ceph-mon[36624]: pgmap v4342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:27.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:27 smithi081 ceph-mon[31453]: pgmap v4342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:28.437+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:20:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:28.437+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:20:28.424 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:28.437+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:20:29.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:29 smithi153 ceph-mon[36569]: pgmap v4343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:29.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:29 smithi104 ceph-mon[36624]: pgmap v4343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:29.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:29 smithi081 ceph-mon[31453]: pgmap v4343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:31.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:31 smithi153 ceph-mon[36569]: pgmap v4344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:31.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:31 smithi104 ceph-mon[36624]: pgmap v4344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:31.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:31 smithi081 ceph-mon[31453]: pgmap v4344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:33.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:33 smithi153 ceph-mon[36569]: pgmap v4345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:33.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:33 smithi104 ceph-mon[36624]: pgmap v4345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:33.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:33 smithi081 ceph-mon[31453]: pgmap v4345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:35.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:35 smithi153 ceph-mon[36569]: pgmap v4346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:35.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:35 smithi104 ceph-mon[36624]: pgmap v4346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:35.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:35 smithi081 ceph-mon[31453]: pgmap v4346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:37.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:37 smithi153 ceph-mon[36569]: pgmap v4347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:37.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:37 smithi104 ceph-mon[36624]: pgmap v4347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:37.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:37 smithi081 ceph-mon[31453]: pgmap v4347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:39 smithi153 ceph-mon[36569]: pgmap v4348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:39.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:39 smithi104 ceph-mon[36624]: pgmap v4348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:39.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:39 smithi081 ceph-mon[31453]: pgmap v4348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:41.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:41 smithi153 ceph-mon[36569]: pgmap v4349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:41.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:41 smithi104 ceph-mon[36624]: pgmap v4349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:41.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:41 smithi081 ceph-mon[31453]: pgmap v4349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:43.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:43 smithi153 ceph-mon[36569]: pgmap v4350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:43.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:43 smithi104 ceph-mon[36624]: pgmap v4350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:43.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:43 smithi081 ceph-mon[31453]: pgmap v4350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:43.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:43.438+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:20:43.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:43.438+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:20:43.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:43.438+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:20:45.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:45 smithi153 ceph-mon[36569]: pgmap v4351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:45.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:45 smithi104 ceph-mon[36624]: pgmap v4351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:45.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:45 smithi081 ceph-mon[31453]: pgmap v4351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:47.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:47 smithi104 ceph-mon[36624]: pgmap v4352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:47.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:47 smithi081 ceph-mon[31453]: pgmap v4352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:47 smithi153 ceph-mon[36569]: pgmap v4352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:49.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:49 smithi104 ceph-mon[36624]: pgmap v4353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:49.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:49 smithi081 ceph-mon[31453]: pgmap v4353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:49 smithi153 ceph-mon[36569]: pgmap v4353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:51.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:51 smithi104 ceph-mon[36624]: pgmap v4354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:51.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:51 smithi081 ceph-mon[31453]: pgmap v4354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:51 smithi153 ceph-mon[36569]: pgmap v4354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:53.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:53 smithi104 ceph-mon[36624]: pgmap v4355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:53.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:53 smithi081 ceph-mon[31453]: pgmap v4355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:53 smithi153 ceph-mon[36569]: pgmap v4355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:55.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:55 smithi104 ceph-mon[36624]: pgmap v4356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:55.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:55 smithi081 ceph-mon[31453]: pgmap v4356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:55 smithi153 ceph-mon[36569]: pgmap v4356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:57.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:57 smithi104 ceph-mon[36624]: pgmap v4357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:57.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:57 smithi153 ceph-mon[36569]: pgmap v4357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:57.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:57 smithi081 ceph-mon[31453]: pgmap v4357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:58.454+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:20:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:58.454+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:20:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:20:58.454+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:20:59.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:20:59 smithi081 ceph-mon[31453]: pgmap v4358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:59.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:20:59 smithi153 ceph-mon[36569]: pgmap v4358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:20:59.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:20:59 smithi104 ceph-mon[36624]: pgmap v4358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:00.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:00 smithi081 ceph-mon[31453]: pgmap v4359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:00.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:00 smithi153 ceph-mon[36569]: pgmap v4359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:00.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:00 smithi104 ceph-mon[36624]: pgmap v4359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:02.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:02 smithi153 ceph-mon[36569]: pgmap v4360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:02.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:21:02.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:02.994 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:03.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:02 smithi104 ceph-mon[36624]: pgmap v4360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:03.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:21:03.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:03.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:03.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:02 smithi081 ceph-mon[31453]: pgmap v4360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:03.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:21:03.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:03.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:04.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:04.067 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:04.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:04.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:05.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:04 smithi153 ceph-mon[36569]: pgmap v4361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:05.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:05.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:05.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:04 smithi104 ceph-mon[36624]: pgmap v4361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:05.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:05.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:05.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:04 smithi081 ceph-mon[31453]: pgmap v4361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:05.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:05.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:06.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:06.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:06.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:06.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:06.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:06.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:07.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:06 smithi153 ceph-mon[36569]: pgmap v4362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:07.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:06 smithi104 ceph-mon[36624]: pgmap v4362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:07.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:06 smithi081 ceph-mon[31453]: pgmap v4362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:08.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:08.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:08.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:08 smithi153 ceph-mon[36569]: pgmap v4363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:08.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:08.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:08.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:08 smithi104 ceph-mon[36624]: pgmap v4363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:08.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:08.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:08.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:08 smithi081 ceph-mon[31453]: pgmap v4363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:21:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:21:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:09 smithi153 ceph-mon[36569]: pgmap v4364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:09.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:09 smithi153 ceph-mon[36569]: pgmap v4365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:09.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:09.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:21:09.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:21:09.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:09 smithi104 ceph-mon[36624]: pgmap v4364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:09.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:09 smithi104 ceph-mon[36624]: pgmap v4365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:09.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:09.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:21:09.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:21:09.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:09 smithi081 ceph-mon[31453]: pgmap v4364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:09.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:09 smithi081 ceph-mon[31453]: pgmap v4365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:09.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:21:11.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:11 smithi081 ceph-mon[31453]: pgmap v4366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:12.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:11 smithi153 ceph-mon[36569]: pgmap v4366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:12.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:11 smithi104 ceph-mon[36624]: pgmap v4366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:13.455 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:13.468+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:21:13.455 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:13.468+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:21:13.455 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:13.468+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:21:13.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:13 smithi081 ceph-mon[31453]: pgmap v4367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:14.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:13 smithi153 ceph-mon[36569]: pgmap v4367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:14.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:13 smithi104 ceph-mon[36624]: pgmap v4367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:15.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:15 smithi081 ceph-mon[31453]: pgmap v4368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:16.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:15 smithi153 ceph-mon[36569]: pgmap v4368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:16.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:15 smithi104 ceph-mon[36624]: pgmap v4368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:18.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:17 smithi153 ceph-mon[36569]: pgmap v4369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:18.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:17 smithi104 ceph-mon[36624]: pgmap v4369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:18.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:17 smithi081 ceph-mon[31453]: pgmap v4369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:20.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:19 smithi153 ceph-mon[36569]: pgmap v4370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:20.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:19 smithi104 ceph-mon[36624]: pgmap v4370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:20.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:19 smithi081 ceph-mon[31453]: pgmap v4370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:22.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:21 smithi153 ceph-mon[36569]: pgmap v4371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:22.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:21 smithi104 ceph-mon[36624]: pgmap v4371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:22.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:21 smithi081 ceph-mon[31453]: pgmap v4371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:24.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:23 smithi153 ceph-mon[36569]: pgmap v4372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:24.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:23 smithi104 ceph-mon[36624]: pgmap v4372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:24.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:23 smithi081 ceph-mon[31453]: pgmap v4372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:25.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:25 smithi081 ceph-mon[31453]: pgmap v4373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:26.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:25 smithi153 ceph-mon[36569]: pgmap v4373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:26.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:25 smithi104 ceph-mon[36624]: pgmap v4373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:28.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:27 smithi153 ceph-mon[36569]: pgmap v4374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:28.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:27 smithi104 ceph-mon[36624]: pgmap v4374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:28.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:27 smithi081 ceph-mon[31453]: pgmap v4374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:28.479+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:21:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:28.479+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:21:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:28.479+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:21:30.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:29 smithi153 ceph-mon[36569]: pgmap v4375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:30.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:29 smithi104 ceph-mon[36624]: pgmap v4375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:30.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:29 smithi081 ceph-mon[31453]: pgmap v4375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:32.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:31 smithi153 ceph-mon[36569]: pgmap v4376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:32.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:31 smithi104 ceph-mon[36624]: pgmap v4376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:32.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:31 smithi081 ceph-mon[31453]: pgmap v4376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:34.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:33 smithi153 ceph-mon[36569]: pgmap v4377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:34.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:33 smithi104 ceph-mon[36624]: pgmap v4377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:34.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:33 smithi081 ceph-mon[31453]: pgmap v4377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:36.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:35 smithi153 ceph-mon[36569]: pgmap v4378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:36.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:35 smithi104 ceph-mon[36624]: pgmap v4378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:36.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:35 smithi081 ceph-mon[31453]: pgmap v4378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:38.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:37 smithi153 ceph-mon[36569]: pgmap v4379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:38.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:37 smithi104 ceph-mon[36624]: pgmap v4379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:38.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:37 smithi081 ceph-mon[31453]: pgmap v4379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:40.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:39 smithi153 ceph-mon[36569]: pgmap v4380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:40.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:39 smithi104 ceph-mon[36624]: pgmap v4380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:40.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:39 smithi081 ceph-mon[31453]: pgmap v4380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:42.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:41 smithi153 ceph-mon[36569]: pgmap v4381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:42.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:41 smithi104 ceph-mon[36624]: pgmap v4381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:42.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:41 smithi081 ceph-mon[31453]: pgmap v4381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:43.491+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:21:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:43.491+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:21:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:43.491+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:21:44.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:43 smithi153 ceph-mon[36569]: pgmap v4382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:44.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:43 smithi104 ceph-mon[36624]: pgmap v4382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:44.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:43 smithi081 ceph-mon[31453]: pgmap v4382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:46.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:45 smithi153 ceph-mon[36569]: pgmap v4383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:46.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:45 smithi104 ceph-mon[36624]: pgmap v4383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:46.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:45 smithi081 ceph-mon[31453]: pgmap v4383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:48.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:47 smithi153 ceph-mon[36569]: pgmap v4384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:48.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:47 smithi104 ceph-mon[36624]: pgmap v4384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:48.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:47 smithi081 ceph-mon[31453]: pgmap v4384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:50.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:49 smithi153 ceph-mon[36569]: pgmap v4385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:50.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:49 smithi104 ceph-mon[36624]: pgmap v4385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:50.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:49 smithi081 ceph-mon[31453]: pgmap v4385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:52.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:51 smithi153 ceph-mon[36569]: pgmap v4386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:52.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:51 smithi104 ceph-mon[36624]: pgmap v4386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:52.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:51 smithi081 ceph-mon[31453]: pgmap v4386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:54.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:53 smithi153 ceph-mon[36569]: pgmap v4387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:54.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:53 smithi104 ceph-mon[36624]: pgmap v4387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:54.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:53 smithi081 ceph-mon[31453]: pgmap v4387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:56.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:55 smithi153 ceph-mon[36569]: pgmap v4388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:56.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:55 smithi104 ceph-mon[36624]: pgmap v4388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:56.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:55 smithi081 ceph-mon[31453]: pgmap v4388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:58.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:57 smithi153 ceph-mon[36569]: pgmap v4389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:58.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:57 smithi104 ceph-mon[36624]: pgmap v4389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:58.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:57 smithi081 ceph-mon[31453]: pgmap v4389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:21:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:58.503+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:21:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:58.503+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:21:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:21:58.503+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:22:00.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:21:59 smithi153 ceph-mon[36569]: pgmap v4390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:00.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:21:59 smithi104 ceph-mon[36624]: pgmap v4390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:00.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:21:59 smithi081 ceph-mon[31453]: pgmap v4390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:02.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:01 smithi153 ceph-mon[36569]: pgmap v4391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:02.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:01 smithi104 ceph-mon[36624]: pgmap v4391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:02.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:01 smithi081 ceph-mon[31453]: pgmap v4391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:03 smithi153 ceph-mon[36569]: pgmap v4392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:04.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:03 smithi104 ceph-mon[36624]: pgmap v4392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:04.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:03 smithi081 ceph-mon[31453]: pgmap v4392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:06.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:05 smithi153 ceph-mon[36569]: pgmap v4393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:06.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:05 smithi104 ceph-mon[36624]: pgmap v4393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:06.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:05 smithi081 ceph-mon[31453]: pgmap v4393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:08.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:07 smithi153 ceph-mon[36569]: pgmap v4394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:08.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:07 smithi104 ceph-mon[36624]: pgmap v4394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:08.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:07 smithi081 ceph-mon[31453]: pgmap v4394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:10.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:09 smithi153 ceph-mon[36569]: pgmap v4395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:10.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:22:10.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:10.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:10.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:09 smithi104 ceph-mon[36624]: pgmap v4395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:10.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:22:10.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:10.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:10.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:09 smithi081 ceph-mon[31453]: pgmap v4395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:10.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:22:10.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:10.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:11.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:22:11.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:22:11.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:11.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:22:11.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:22:11.045 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:11.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:22:11.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:22:11.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:22:12.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:11 smithi153 ceph-mon[36569]: pgmap v4396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:12.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:11 smithi153 ceph-mon[36569]: pgmap v4397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:12.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:11 smithi104 ceph-mon[36624]: pgmap v4396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:12.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:11 smithi104 ceph-mon[36624]: pgmap v4397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:12.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:11 smithi081 ceph-mon[31453]: pgmap v4396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:12.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:11 smithi081 ceph-mon[31453]: pgmap v4397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:13.519+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:22:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:13.519+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:22:13.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:13.519+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:22:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:13.944+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:22:13.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:13.944+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:22:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:13.945+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:22:13.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:13.945+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:22:14.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:13 smithi104 ceph-mon[36624]: pgmap v4398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:14.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:13 smithi081 ceph-mon[31453]: pgmap v4398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:14.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:13 smithi153 ceph-mon[36569]: pgmap v4398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:16.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:15 smithi104 ceph-mon[36624]: pgmap v4399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:16.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:15 smithi081 ceph-mon[31453]: pgmap v4399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:16.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:15 smithi153 ceph-mon[36569]: pgmap v4399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:18.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:17 smithi104 ceph-mon[36624]: pgmap v4400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:18.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:17 smithi081 ceph-mon[31453]: pgmap v4400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:18.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:17 smithi153 ceph-mon[36569]: pgmap v4400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:20.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:19 smithi104 ceph-mon[36624]: pgmap v4401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:20.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:19 smithi081 ceph-mon[31453]: pgmap v4401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:20.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:19 smithi153 ceph-mon[36569]: pgmap v4401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:22.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:21 smithi104 ceph-mon[36624]: pgmap v4402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:22.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:21 smithi081 ceph-mon[31453]: pgmap v4402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:22.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:21 smithi153 ceph-mon[36569]: pgmap v4402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:24.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:23 smithi081 ceph-mon[31453]: pgmap v4403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:24.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:23 smithi153 ceph-mon[36569]: pgmap v4403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:24.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:23 smithi104 ceph-mon[36624]: pgmap v4403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:26.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:25 smithi081 ceph-mon[31453]: pgmap v4404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:26.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:25 smithi153 ceph-mon[36569]: pgmap v4404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:26.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:25 smithi104 ceph-mon[36624]: pgmap v4404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:28.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:27 smithi153 ceph-mon[36569]: pgmap v4405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:28.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:27 smithi081 ceph-mon[31453]: pgmap v4405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:28.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:27 smithi104 ceph-mon[36624]: pgmap v4405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:28.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:28.532+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:22:28.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:28.532+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:22:28.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:28.532+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:22:30.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:29 smithi081 ceph-mon[31453]: pgmap v4406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:30.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:29 smithi153 ceph-mon[36569]: pgmap v4406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:30.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:29 smithi104 ceph-mon[36624]: pgmap v4406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:32.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:31 smithi081 ceph-mon[31453]: pgmap v4407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:31 smithi153 ceph-mon[36569]: pgmap v4407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:32.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:31 smithi104 ceph-mon[36624]: pgmap v4407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:34.121 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:33 smithi104 ceph-mon[36624]: pgmap v4408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:34.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:33 smithi081 ceph-mon[31453]: pgmap v4408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:34.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:33 smithi153 ceph-mon[36569]: pgmap v4408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:36.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:35 smithi081 ceph-mon[31453]: pgmap v4409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:35 smithi153 ceph-mon[36569]: pgmap v4409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:36.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:35 smithi104 ceph-mon[36624]: pgmap v4409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:38.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:37 smithi081 ceph-mon[31453]: pgmap v4410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:38.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:37 smithi153 ceph-mon[36569]: pgmap v4410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:38.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:37 smithi104 ceph-mon[36624]: pgmap v4410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:40.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:39 smithi081 ceph-mon[31453]: pgmap v4411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:40.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:39 smithi153 ceph-mon[36569]: pgmap v4411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:40.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:39 smithi104 ceph-mon[36624]: pgmap v4411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:42.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:41 smithi081 ceph-mon[31453]: pgmap v4412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:42.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:41 smithi153 ceph-mon[36569]: pgmap v4412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:42.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:41 smithi104 ceph-mon[36624]: pgmap v4412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:43.546+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:22:43.532 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:43.546+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:22:43.533 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:43.546+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:22:44.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:43 smithi081 ceph-mon[31453]: pgmap v4413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:44.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:43 smithi153 ceph-mon[36569]: pgmap v4413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:44.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:43 smithi104 ceph-mon[36624]: pgmap v4413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:46.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:45 smithi081 ceph-mon[31453]: pgmap v4414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:46.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:45 smithi153 ceph-mon[36569]: pgmap v4414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:46.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:45 smithi104 ceph-mon[36624]: pgmap v4414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:48.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:47 smithi081 ceph-mon[31453]: pgmap v4415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:48.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:47 smithi153 ceph-mon[36569]: pgmap v4415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:48.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:47 smithi104 ceph-mon[36624]: pgmap v4415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:50.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:49 smithi081 ceph-mon[31453]: pgmap v4416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:50.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:49 smithi153 ceph-mon[36569]: pgmap v4416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:50.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:49 smithi104 ceph-mon[36624]: pgmap v4416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:52.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:51 smithi153 ceph-mon[36569]: pgmap v4417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:52.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:51 smithi104 ceph-mon[36624]: pgmap v4417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:52.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:51 smithi081 ceph-mon[31453]: pgmap v4417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:53 smithi153 ceph-mon[36569]: pgmap v4418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:54.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:53 smithi104 ceph-mon[36624]: pgmap v4418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:54.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:53 smithi081 ceph-mon[31453]: pgmap v4418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:56.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:55 smithi153 ceph-mon[36569]: pgmap v4419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:56.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:55 smithi104 ceph-mon[36624]: pgmap v4419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:56.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:55 smithi081 ceph-mon[31453]: pgmap v4419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:58.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:57 smithi153 ceph-mon[36569]: pgmap v4420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:58.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:57 smithi104 ceph-mon[36624]: pgmap v4420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:58.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:57 smithi081 ceph-mon[31453]: pgmap v4420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:22:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:58.559+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:22:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:58.559+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:22:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:22:58.559+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:23:00.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:22:59 smithi153 ceph-mon[36569]: pgmap v4421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:00.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:22:59 smithi104 ceph-mon[36624]: pgmap v4421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:00.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:22:59 smithi081 ceph-mon[31453]: pgmap v4421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:02.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:01 smithi153 ceph-mon[36569]: pgmap v4422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:02.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:01 smithi104 ceph-mon[36624]: pgmap v4422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:02.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:01 smithi081 ceph-mon[31453]: pgmap v4422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:04.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:03 smithi153 ceph-mon[36569]: pgmap v4423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:04.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:03 smithi104 ceph-mon[36624]: pgmap v4423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:04.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:03 smithi081 ceph-mon[31453]: pgmap v4423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:06.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:05 smithi153 ceph-mon[36569]: pgmap v4424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:06.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:05 smithi104 ceph-mon[36624]: pgmap v4424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:06.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:05 smithi081 ceph-mon[31453]: pgmap v4424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:08.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:07 smithi153 ceph-mon[36569]: pgmap v4425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:08.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:07 smithi104 ceph-mon[36624]: pgmap v4425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:08.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:07 smithi081 ceph-mon[31453]: pgmap v4425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:10.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:09 smithi153 ceph-mon[36569]: pgmap v4426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:10.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:09 smithi104 ceph-mon[36624]: pgmap v4426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:10.330 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:09 smithi081 ceph-mon[31453]: pgmap v4426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:11.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:23:11.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:23:11.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:23:12.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:11 smithi153 ceph-mon[36569]: pgmap v4427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:12.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:23:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:23:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:23:12.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:11 smithi104 ceph-mon[36624]: pgmap v4427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:12.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:23:12.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:23:12.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:23:12.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:11 smithi081 ceph-mon[31453]: pgmap v4427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:12.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:23:12.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:23:12.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:23:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:12 smithi153 ceph-mon[36569]: pgmap v4428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:13.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:12 smithi153 ceph-mon[36569]: pgmap v4429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:13.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:12 smithi104 ceph-mon[36624]: pgmap v4428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:13.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:12 smithi104 ceph-mon[36624]: pgmap v4429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:13.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:12 smithi081 ceph-mon[31453]: pgmap v4428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:13.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:12 smithi081 ceph-mon[31453]: pgmap v4429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:13.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:23:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:13.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:23:13.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:13.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:23:15.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:14 smithi153 ceph-mon[36569]: pgmap v4430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:15.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:14 smithi104 ceph-mon[36624]: pgmap v4430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:15.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:14 smithi081 ceph-mon[31453]: pgmap v4430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:17.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:16 smithi153 ceph-mon[36569]: pgmap v4431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:17.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:16 smithi104 ceph-mon[36624]: pgmap v4431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:17.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:16 smithi081 ceph-mon[31453]: pgmap v4431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:19.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:18 smithi153 ceph-mon[36569]: pgmap v4432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:19.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:18 smithi104 ceph-mon[36624]: pgmap v4432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:19.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:18 smithi081 ceph-mon[31453]: pgmap v4432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:21.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:20 smithi153 ceph-mon[36569]: pgmap v4433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:21.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:20 smithi104 ceph-mon[36624]: pgmap v4433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:21.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:20 smithi081 ceph-mon[31453]: pgmap v4433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:23.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:23 smithi153 ceph-mon[36569]: pgmap v4434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:23.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:23 smithi104 ceph-mon[36624]: pgmap v4434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:23.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:23 smithi081 ceph-mon[31453]: pgmap v4434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:25.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:25 smithi153 ceph-mon[36569]: pgmap v4435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:25.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:25 smithi104 ceph-mon[36624]: pgmap v4435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:25.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:25 smithi081 ceph-mon[31453]: pgmap v4435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:27.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:27 smithi153 ceph-mon[36569]: pgmap v4436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:27.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:27 smithi104 ceph-mon[36624]: pgmap v4436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:27.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:27 smithi081 ceph-mon[31453]: pgmap v4436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:28.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:28.572+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:23:28.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:28.572+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:23:28.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:28.572+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:23:29.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:29 smithi153 ceph-mon[36569]: pgmap v4437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:29.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:29 smithi104 ceph-mon[36624]: pgmap v4437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:29.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:29 smithi081 ceph-mon[31453]: pgmap v4437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:31.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:31 smithi153 ceph-mon[36569]: pgmap v4438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:31.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:31 smithi104 ceph-mon[36624]: pgmap v4438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:31.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:31 smithi081 ceph-mon[31453]: pgmap v4438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:33.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:33 smithi153 ceph-mon[36569]: pgmap v4439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:33.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:33 smithi104 ceph-mon[36624]: pgmap v4439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:33.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:33 smithi081 ceph-mon[31453]: pgmap v4439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:35.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:35 smithi153 ceph-mon[36569]: pgmap v4440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:35.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:35 smithi104 ceph-mon[36624]: pgmap v4440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:35.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:35 smithi081 ceph-mon[31453]: pgmap v4440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:37.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:37 smithi153 ceph-mon[36569]: pgmap v4441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:37.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:37 smithi104 ceph-mon[36624]: pgmap v4441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:37.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:37 smithi081 ceph-mon[31453]: pgmap v4441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:39 smithi153 ceph-mon[36569]: pgmap v4442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:39.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:39 smithi104 ceph-mon[36624]: pgmap v4442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:39.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:39 smithi081 ceph-mon[31453]: pgmap v4442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:41.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:41 smithi153 ceph-mon[36569]: pgmap v4443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:41.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:41 smithi104 ceph-mon[36624]: pgmap v4443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:41.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:41 smithi081 ceph-mon[31453]: pgmap v4443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:43.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:43 smithi153 ceph-mon[36569]: pgmap v4444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:43.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:43 smithi104 ceph-mon[36624]: pgmap v4444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:43.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:43 smithi081 ceph-mon[31453]: pgmap v4444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:43.585+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:23:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:43.585+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:23:43.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:43.585+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:23:45.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:45 smithi104 ceph-mon[36624]: pgmap v4445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:45.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:45 smithi081 ceph-mon[31453]: pgmap v4445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:45 smithi153 ceph-mon[36569]: pgmap v4445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:47.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:47 smithi104 ceph-mon[36624]: pgmap v4446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:47.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:47 smithi081 ceph-mon[31453]: pgmap v4446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:47 smithi153 ceph-mon[36569]: pgmap v4446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:49.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:49 smithi104 ceph-mon[36624]: pgmap v4447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:49.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:49 smithi081 ceph-mon[31453]: pgmap v4447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:49.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:49 smithi153 ceph-mon[36569]: pgmap v4447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:51.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:51 smithi104 ceph-mon[36624]: pgmap v4448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:51.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:51 smithi081 ceph-mon[31453]: pgmap v4448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:51 smithi153 ceph-mon[36569]: pgmap v4448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:53.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:53 smithi081 ceph-mon[31453]: pgmap v4449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:53 smithi153 ceph-mon[36569]: pgmap v4449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:53.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:53 smithi104 ceph-mon[36624]: pgmap v4449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:55.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:55 smithi081 ceph-mon[31453]: pgmap v4450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:55 smithi153 ceph-mon[36569]: pgmap v4450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:55.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:55 smithi104 ceph-mon[36624]: pgmap v4450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:57.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:57 smithi081 ceph-mon[31453]: pgmap v4451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:57.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:57 smithi153 ceph-mon[36569]: pgmap v4451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:57.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:57 smithi104 ceph-mon[36624]: pgmap v4451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:58.589+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:23:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:58.589+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:23:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:23:58.589+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:23:59.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:23:59 smithi153 ceph-mon[36569]: pgmap v4452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:59.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:23:59 smithi081 ceph-mon[31453]: pgmap v4452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:23:59.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:23:59 smithi104 ceph-mon[36624]: pgmap v4452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:01.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:01 smithi081 ceph-mon[31453]: pgmap v4453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:01.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:01 smithi153 ceph-mon[36569]: pgmap v4453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:01.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:01 smithi104 ceph-mon[36624]: pgmap v4453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:03.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:03 smithi081 ceph-mon[31453]: pgmap v4454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:03 smithi153 ceph-mon[36569]: pgmap v4454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:03.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:03 smithi104 ceph-mon[36624]: pgmap v4454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:05.371 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:05 smithi104 ceph-mon[36624]: pgmap v4455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:05.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:05 smithi081 ceph-mon[31453]: pgmap v4455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:05.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:05 smithi153 ceph-mon[36569]: pgmap v4455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:07.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:07 smithi081 ceph-mon[31453]: pgmap v4456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:07.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:07 smithi153 ceph-mon[36569]: pgmap v4456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:07.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:07 smithi104 ceph-mon[36624]: pgmap v4456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:09.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:09 smithi081 ceph-mon[31453]: pgmap v4457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:09 smithi153 ceph-mon[36569]: pgmap v4457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:09.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:09 smithi104 ceph-mon[36624]: pgmap v4457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:11.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:11 smithi081 ceph-mon[31453]: pgmap v4458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:11 smithi153 ceph-mon[36569]: pgmap v4458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:11.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:11 smithi104 ceph-mon[36624]: pgmap v4458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:12.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:24:12.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:24:12.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:24:13.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:13 smithi081 ceph-mon[31453]: pgmap v4459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:13.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:13 smithi153 ceph-mon[36569]: pgmap v4459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:13.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:13 smithi104 ceph-mon[36624]: pgmap v4459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:13.605+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:24:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:13.605+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:24:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:13.605+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:24:14.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:24:14.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:24:14.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:24:14.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:24:14.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:24:14.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:24:14.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:24:14.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:24:14.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:24:15.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:15 smithi081 ceph-mon[31453]: pgmap v4460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:15.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:15 smithi081 ceph-mon[31453]: pgmap v4461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:15 smithi153 ceph-mon[36569]: pgmap v4460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:15.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:15 smithi153 ceph-mon[36569]: pgmap v4461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:15.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:15 smithi104 ceph-mon[36624]: pgmap v4460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:15.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:15 smithi104 ceph-mon[36624]: pgmap v4461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:16.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:16 smithi153 ceph-mon[36569]: pgmap v4462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:16.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:16 smithi104 ceph-mon[36624]: pgmap v4462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:16.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:16 smithi081 ceph-mon[31453]: pgmap v4462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:18.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:18 smithi153 ceph-mon[36569]: pgmap v4463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:18.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:18 smithi104 ceph-mon[36624]: pgmap v4463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:18.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:18 smithi081 ceph-mon[31453]: pgmap v4463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:21.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:21 smithi081 ceph-mon[31453]: pgmap v4464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:21 smithi153 ceph-mon[36569]: pgmap v4464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:21.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:21 smithi104 ceph-mon[36624]: pgmap v4464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:23.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:23 smithi081 ceph-mon[31453]: pgmap v4465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:23 smithi153 ceph-mon[36569]: pgmap v4465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:23.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:23 smithi104 ceph-mon[36624]: pgmap v4465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:25.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:25 smithi081 ceph-mon[31453]: pgmap v4466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:25 smithi153 ceph-mon[36569]: pgmap v4466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:25.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:25 smithi104 ceph-mon[36624]: pgmap v4466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:27.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:27 smithi081 ceph-mon[31453]: pgmap v4467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:27 smithi153 ceph-mon[36569]: pgmap v4467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:27.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:27 smithi104 ceph-mon[36624]: pgmap v4467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:28.620+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:24:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:28.620+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:24:28.607 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:28.620+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:24:29.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:29 smithi153 ceph-mon[36569]: pgmap v4468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:29.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:29 smithi081 ceph-mon[31453]: pgmap v4468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:29.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:29 smithi104 ceph-mon[36624]: pgmap v4468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:31.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:31 smithi081 ceph-mon[31453]: pgmap v4469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:31.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:31 smithi153 ceph-mon[36569]: pgmap v4469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:31.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:31 smithi104 ceph-mon[36624]: pgmap v4469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:33.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:33 smithi081 ceph-mon[31453]: pgmap v4470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:33.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:33 smithi153 ceph-mon[36569]: pgmap v4470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:33 smithi104 ceph-mon[36624]: pgmap v4470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:35.371 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:35 smithi104 ceph-mon[36624]: pgmap v4471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:35.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:35 smithi081 ceph-mon[31453]: pgmap v4471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:35 smithi153 ceph-mon[36569]: pgmap v4471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:37.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:37 smithi081 ceph-mon[31453]: pgmap v4472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:37 smithi153 ceph-mon[36569]: pgmap v4472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:37.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:37 smithi104 ceph-mon[36624]: pgmap v4472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:39 smithi081 ceph-mon[31453]: pgmap v4473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:39 smithi153 ceph-mon[36569]: pgmap v4473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:39.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:39 smithi104 ceph-mon[36624]: pgmap v4473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:41.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:41 smithi081 ceph-mon[31453]: pgmap v4474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:41 smithi153 ceph-mon[36569]: pgmap v4474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:41.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:41 smithi104 ceph-mon[36624]: pgmap v4474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:43.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:43 smithi081 ceph-mon[31453]: pgmap v4475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:43.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:43 smithi153 ceph-mon[36569]: pgmap v4475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:43.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:43 smithi104 ceph-mon[36624]: pgmap v4475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:43.628+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:24:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:43.628+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:24:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:43.628+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:24:44.295 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:44 smithi081 ceph-mon[31453]: pgmap v4476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:44.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:44 smithi153 ceph-mon[36569]: pgmap v4476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:44.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:44 smithi104 ceph-mon[36624]: pgmap v4476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:46.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:46 smithi153 ceph-mon[36569]: pgmap v4477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:46.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:46 smithi104 ceph-mon[36624]: pgmap v4477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:46.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:46 smithi081 ceph-mon[31453]: pgmap v4477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:48.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:48 smithi153 ceph-mon[36569]: pgmap v4478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:48.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:48 smithi104 ceph-mon[36624]: pgmap v4478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:48.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:48 smithi081 ceph-mon[31453]: pgmap v4478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:50.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:50 smithi153 ceph-mon[36569]: pgmap v4479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:50.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:50 smithi104 ceph-mon[36624]: pgmap v4479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:50.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:50 smithi081 ceph-mon[31453]: pgmap v4479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:52.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:52 smithi153 ceph-mon[36569]: pgmap v4480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:52.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:52 smithi104 ceph-mon[36624]: pgmap v4480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:52.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:52 smithi081 ceph-mon[31453]: pgmap v4480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:54.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:54 smithi153 ceph-mon[36569]: pgmap v4481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:54.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:54 smithi104 ceph-mon[36624]: pgmap v4481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:54.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:54 smithi081 ceph-mon[31453]: pgmap v4481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:56.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:56 smithi153 ceph-mon[36569]: pgmap v4482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:56.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:56 smithi104 ceph-mon[36624]: pgmap v4482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:56.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:56 smithi081 ceph-mon[31453]: pgmap v4482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:58.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:58.638+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:24:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:58.638+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:24:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:24:58.638+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:24:58.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:24:58 smithi153 ceph-mon[36569]: pgmap v4483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:58.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:24:58 smithi104 ceph-mon[36624]: pgmap v4483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:24:58.874 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:24:58 smithi081 ceph-mon[31453]: pgmap v4483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:00.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:00 smithi153 ceph-mon[36569]: pgmap v4484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:00.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:00 smithi104 ceph-mon[36624]: pgmap v4484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:00.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:00 smithi081 ceph-mon[31453]: pgmap v4484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:02.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:02 smithi153 ceph-mon[36569]: pgmap v4485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:02.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:02 smithi104 ceph-mon[36624]: pgmap v4485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:02.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:02 smithi081 ceph-mon[31453]: pgmap v4485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:04.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:04 smithi153 ceph-mon[36569]: pgmap v4486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:04.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:04 smithi104 ceph-mon[36624]: pgmap v4486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:04.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:04 smithi081 ceph-mon[31453]: pgmap v4486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:06.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:06 smithi153 ceph-mon[36569]: pgmap v4487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:06.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:06 smithi104 ceph-mon[36624]: pgmap v4487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:06.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:06 smithi081 ceph-mon[31453]: pgmap v4487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:08.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:08 smithi153 ceph-mon[36569]: pgmap v4488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:08.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:08 smithi104 ceph-mon[36624]: pgmap v4488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:08.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:08 smithi081 ceph-mon[31453]: pgmap v4488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:10.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:10 smithi153 ceph-mon[36569]: pgmap v4489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:10.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:10 smithi104 ceph-mon[36624]: pgmap v4489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:10.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:10 smithi081 ceph-mon[31453]: pgmap v4489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:12.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:12 smithi153 ceph-mon[36569]: pgmap v4490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:12.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:12 smithi104 ceph-mon[36624]: pgmap v4490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:12.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:12 smithi081 ceph-mon[31453]: pgmap v4490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:13.639 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:13.653+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:25:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:13.653+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:25:13.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:13.653+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:25:13.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:13.975+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:25:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:13.975+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:25:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:13.975+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:25:13.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:13.975+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:25:14.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:14 smithi153 ceph-mon[36569]: pgmap v4491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:14.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:25:14.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:14 smithi104 ceph-mon[36624]: pgmap v4491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:14.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:25:14.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:14 smithi081 ceph-mon[31453]: pgmap v4491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:14.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:25:15.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:25:15.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:25:15.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:15 smithi153 ceph-mon[36569]: pgmap v4492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:15.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:15 smithi153 ceph-mon[36569]: pgmap v4493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:15.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:25:15.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:25:15.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:25:15.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:15 smithi104 ceph-mon[36624]: pgmap v4492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:15.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:15 smithi104 ceph-mon[36624]: pgmap v4493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:15.819 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:25:15.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:25:15.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:25:15.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:15 smithi081 ceph-mon[31453]: pgmap v4492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:15.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:15 smithi081 ceph-mon[31453]: pgmap v4493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:15.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:25:18.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:17 smithi153 ceph-mon[36569]: pgmap v4494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:18.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:17 smithi104 ceph-mon[36624]: pgmap v4494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:18.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:17 smithi081 ceph-mon[31453]: pgmap v4494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:20.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:19 smithi153 ceph-mon[36569]: pgmap v4495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:20.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:19 smithi104 ceph-mon[36624]: pgmap v4495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:20.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:19 smithi081 ceph-mon[31453]: pgmap v4495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:22.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:21 smithi153 ceph-mon[36569]: pgmap v4496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:22.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:21 smithi104 ceph-mon[36624]: pgmap v4496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:22.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:21 smithi081 ceph-mon[31453]: pgmap v4496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:24.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:23 smithi153 ceph-mon[36569]: pgmap v4497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:24.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:23 smithi104 ceph-mon[36624]: pgmap v4497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:24.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:23 smithi081 ceph-mon[31453]: pgmap v4497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:26.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:25 smithi153 ceph-mon[36569]: pgmap v4498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:26.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:25 smithi104 ceph-mon[36624]: pgmap v4498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:26.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:25 smithi081 ceph-mon[31453]: pgmap v4498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:28.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:27 smithi153 ceph-mon[36569]: pgmap v4499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:28.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:27 smithi104 ceph-mon[36624]: pgmap v4499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:28.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:27 smithi081 ceph-mon[31453]: pgmap v4499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:28.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:28.653+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:25:28.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:28.653+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:25:28.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:28.653+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:25:30.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:29 smithi153 ceph-mon[36569]: pgmap v4500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:30.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:29 smithi104 ceph-mon[36624]: pgmap v4500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:30.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:29 smithi081 ceph-mon[31453]: pgmap v4500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:31 smithi153 ceph-mon[36569]: pgmap v4501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:32.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:31 smithi104 ceph-mon[36624]: pgmap v4501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:32.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:31 smithi081 ceph-mon[31453]: pgmap v4501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:34.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:33 smithi153 ceph-mon[36569]: pgmap v4502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:34.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:33 smithi104 ceph-mon[36624]: pgmap v4502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:34.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:33 smithi081 ceph-mon[31453]: pgmap v4502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:36.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:35 smithi153 ceph-mon[36569]: pgmap v4503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:36.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:35 smithi104 ceph-mon[36624]: pgmap v4503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:36.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:35 smithi081 ceph-mon[31453]: pgmap v4503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:38.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:38 smithi153 ceph-mon[36569]: pgmap v4504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:38.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:38 smithi104 ceph-mon[36624]: pgmap v4504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:38.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:38 smithi081 ceph-mon[31453]: pgmap v4504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:40.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:40 smithi153 ceph-mon[36569]: pgmap v4505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:40.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:40 smithi104 ceph-mon[36624]: pgmap v4505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:40.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:40 smithi081 ceph-mon[31453]: pgmap v4505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:42.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:42 smithi153 ceph-mon[36569]: pgmap v4506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:42.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:42 smithi104 ceph-mon[36624]: pgmap v4506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:42.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:42 smithi081 ceph-mon[31453]: pgmap v4506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:43.669+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:25:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:43.669+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:25:43.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:43.669+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:25:44.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:44 smithi153 ceph-mon[36569]: pgmap v4507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:44.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:44 smithi104 ceph-mon[36624]: pgmap v4507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:44.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:44 smithi081 ceph-mon[31453]: pgmap v4507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:46.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:46 smithi153 ceph-mon[36569]: pgmap v4508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:46.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:46 smithi104 ceph-mon[36624]: pgmap v4508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:46.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:46 smithi081 ceph-mon[31453]: pgmap v4508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:48.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:48 smithi153 ceph-mon[36569]: pgmap v4509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:48.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:48 smithi104 ceph-mon[36624]: pgmap v4509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:48.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:48 smithi081 ceph-mon[31453]: pgmap v4509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:50.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:50 smithi153 ceph-mon[36569]: pgmap v4510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:50.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:50 smithi104 ceph-mon[36624]: pgmap v4510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:50.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:50 smithi081 ceph-mon[31453]: pgmap v4510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:52.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:52 smithi153 ceph-mon[36569]: pgmap v4511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:52.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:52 smithi104 ceph-mon[36624]: pgmap v4511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:52.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:52 smithi081 ceph-mon[31453]: pgmap v4511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:54.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:54 smithi153 ceph-mon[36569]: pgmap v4512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:54.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:54 smithi104 ceph-mon[36624]: pgmap v4512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:54.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:54 smithi081 ceph-mon[31453]: pgmap v4512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:56.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:56 smithi153 ceph-mon[36569]: pgmap v4513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:56.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:56 smithi104 ceph-mon[36624]: pgmap v4513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:56.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:56 smithi081 ceph-mon[31453]: pgmap v4513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:58.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:25:58 smithi153 ceph-mon[36569]: pgmap v4514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:58.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:25:58 smithi104 ceph-mon[36624]: pgmap v4514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:58.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:25:58 smithi081 ceph-mon[31453]: pgmap v4514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:25:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:58.684+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:25:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:58.684+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:25:58.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:25:58.684+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:00.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:00 smithi104 ceph-mon[36624]: pgmap v4515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:00.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:00 smithi081 ceph-mon[31453]: pgmap v4515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:00.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:00 smithi153 ceph-mon[36569]: pgmap v4515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:02.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:02 smithi104 ceph-mon[36624]: pgmap v4516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:02.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:02 smithi081 ceph-mon[31453]: pgmap v4516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:02.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:02 smithi153 ceph-mon[36569]: pgmap v4516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:04.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:04 smithi104 ceph-mon[36624]: pgmap v4517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:04.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:04 smithi081 ceph-mon[31453]: pgmap v4517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:04.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:04 smithi153 ceph-mon[36569]: pgmap v4517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:06.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:06 smithi104 ceph-mon[36624]: pgmap v4518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:06.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:06 smithi081 ceph-mon[31453]: pgmap v4518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:06.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:06 smithi153 ceph-mon[36569]: pgmap v4518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:08.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:08 smithi104 ceph-mon[36624]: pgmap v4519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:08.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:08 smithi081 ceph-mon[31453]: pgmap v4519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:08.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:08 smithi153 ceph-mon[36569]: pgmap v4519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:10.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:10 smithi081 ceph-mon[31453]: pgmap v4520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:10.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:10 smithi153 ceph-mon[36569]: pgmap v4520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:10.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:10 smithi104 ceph-mon[36624]: pgmap v4520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:12.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:12 smithi081 ceph-mon[31453]: pgmap v4521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:12.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:12 smithi153 ceph-mon[36569]: pgmap v4521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:12.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:12 smithi104 ceph-mon[36624]: pgmap v4521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:13.696+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:13.696+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:13.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:13.696+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:26:14.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:14 smithi081 ceph-mon[31453]: pgmap v4522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:14.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:14 smithi153 ceph-mon[36569]: pgmap v4522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:14.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:14 smithi104 ceph-mon[36624]: pgmap v4522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:16.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:16 smithi081 ceph-mon[31453]: pgmap v4523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:16.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:26:16.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:16 smithi153 ceph-mon[36569]: pgmap v4523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:16.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:26:16.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:16 smithi104 ceph-mon[36624]: pgmap v4523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:16.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:26:17.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:26:17.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:26:17.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:26:17.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:26:17.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:26:17.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:26:17.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:26:17.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:26:17.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:26:18.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:18 smithi081 ceph-mon[31453]: pgmap v4524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:18.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:18 smithi081 ceph-mon[31453]: pgmap v4525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:18 smithi153 ceph-mon[36569]: pgmap v4524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:18 smithi153 ceph-mon[36569]: pgmap v4525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:18 smithi104 ceph-mon[36624]: pgmap v4524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:18 smithi104 ceph-mon[36624]: pgmap v4525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:20.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:20 smithi081 ceph-mon[31453]: pgmap v4526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:20 smithi153 ceph-mon[36569]: pgmap v4526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:20.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:20 smithi104 ceph-mon[36624]: pgmap v4526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:22.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:22 smithi081 ceph-mon[31453]: pgmap v4527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:22.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:22 smithi153 ceph-mon[36569]: pgmap v4527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:22.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:22 smithi104 ceph-mon[36624]: pgmap v4527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:24.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:24 smithi081 ceph-mon[31453]: pgmap v4528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:24.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:24 smithi153 ceph-mon[36569]: pgmap v4528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:24.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:24 smithi104 ceph-mon[36624]: pgmap v4528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:26.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:26 smithi081 ceph-mon[31453]: pgmap v4529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:26.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:26 smithi153 ceph-mon[36569]: pgmap v4529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:26.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:26 smithi104 ceph-mon[36624]: pgmap v4529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:28.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:28 smithi081 ceph-mon[31453]: pgmap v4530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:28.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:28 smithi153 ceph-mon[36569]: pgmap v4530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:28.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:28 smithi104 ceph-mon[36624]: pgmap v4530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:28.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:28.709+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:28.709+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:28.709+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:26:30.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:30 smithi081 ceph-mon[31453]: pgmap v4531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:30.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:30 smithi153 ceph-mon[36569]: pgmap v4531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:30.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:30 smithi104 ceph-mon[36624]: pgmap v4531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:32.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:32 smithi081 ceph-mon[31453]: pgmap v4532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:32.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:32 smithi153 ceph-mon[36569]: pgmap v4532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:32.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:32 smithi104 ceph-mon[36624]: pgmap v4532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:34.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:34 smithi081 ceph-mon[31453]: pgmap v4533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:34.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:34 smithi153 ceph-mon[36569]: pgmap v4533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:34.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:34 smithi104 ceph-mon[36624]: pgmap v4533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:36.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:36 smithi153 ceph-mon[36569]: pgmap v4534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:36.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:36 smithi104 ceph-mon[36624]: pgmap v4534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:36.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:36 smithi081 ceph-mon[31453]: pgmap v4534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:38.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:38 smithi153 ceph-mon[36569]: pgmap v4535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:38.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:38 smithi104 ceph-mon[36624]: pgmap v4535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:38.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:38 smithi081 ceph-mon[31453]: pgmap v4535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:40 smithi153 ceph-mon[36569]: pgmap v4536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:40 smithi104 ceph-mon[36624]: pgmap v4536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:40.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:40 smithi081 ceph-mon[31453]: pgmap v4536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:42.446 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:42 smithi081 ceph-mon[31453]: pgmap v4537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:42.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:42 smithi153 ceph-mon[36569]: pgmap v4537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:42.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:42 smithi104 ceph-mon[36624]: pgmap v4537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:43 smithi153 ceph-mon[36569]: pgmap v4538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:43.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:43 smithi104 ceph-mon[36624]: pgmap v4538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:43.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:43 smithi081 ceph-mon[31453]: pgmap v4538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:43.713+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:43.713+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:26:43.699 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:43.713+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:45.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:45 smithi153 ceph-mon[36569]: pgmap v4539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:45.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:45 smithi104 ceph-mon[36624]: pgmap v4539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:45.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:45 smithi081 ceph-mon[31453]: pgmap v4539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:47.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:47 smithi153 ceph-mon[36569]: pgmap v4540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:47.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:47 smithi104 ceph-mon[36624]: pgmap v4540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:47.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:47 smithi081 ceph-mon[31453]: pgmap v4540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:49.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:49 smithi153 ceph-mon[36569]: pgmap v4541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:49.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:49 smithi104 ceph-mon[36624]: pgmap v4541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:49.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:49 smithi081 ceph-mon[31453]: pgmap v4541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:51.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:51 smithi153 ceph-mon[36569]: pgmap v4542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:51.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:51 smithi104 ceph-mon[36624]: pgmap v4542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:51.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:51 smithi081 ceph-mon[31453]: pgmap v4542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:53.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:53 smithi153 ceph-mon[36569]: pgmap v4543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:53.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:53 smithi104 ceph-mon[36624]: pgmap v4543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:53.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:53 smithi081 ceph-mon[31453]: pgmap v4543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:55.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:55 smithi153 ceph-mon[36569]: pgmap v4544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:55.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:55 smithi104 ceph-mon[36624]: pgmap v4544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:55.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:55 smithi081 ceph-mon[31453]: pgmap v4544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:57.793 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:57 smithi153 ceph-mon[36569]: pgmap v4545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:57.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:57 smithi104 ceph-mon[36624]: pgmap v4545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:57.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:57 smithi081 ceph-mon[31453]: pgmap v4545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:58.727+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:58.727+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:26:58.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:26:58.727+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:26:59.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:26:59 smithi104 ceph-mon[36624]: pgmap v4546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:26:59.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:26:59 smithi081 ceph-mon[31453]: pgmap v4546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:00.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:26:59 smithi153 ceph-mon[36569]: pgmap v4546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:01.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:01 smithi104 ceph-mon[36624]: pgmap v4547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:01.844 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:01 smithi153 ceph-mon[36569]: pgmap v4547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:01.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:01 smithi081 ceph-mon[31453]: pgmap v4547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:03.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:03 smithi104 ceph-mon[36624]: pgmap v4548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:03.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:03 smithi081 ceph-mon[31453]: pgmap v4548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:04.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:03 smithi153 ceph-mon[36569]: pgmap v4548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:05.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:05 smithi104 ceph-mon[36624]: pgmap v4549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:05.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:05 smithi081 ceph-mon[31453]: pgmap v4549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:06.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:05 smithi153 ceph-mon[36569]: pgmap v4549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:07.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:07 smithi104 ceph-mon[36624]: pgmap v4550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:07.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:07 smithi081 ceph-mon[31453]: pgmap v4550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:08.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:07 smithi153 ceph-mon[36569]: pgmap v4550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:09.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:09 smithi081 ceph-mon[31453]: pgmap v4551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:10.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:09 smithi153 ceph-mon[36569]: pgmap v4551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:10.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:09 smithi104 ceph-mon[36624]: pgmap v4551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:11.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:11 smithi081 ceph-mon[31453]: pgmap v4552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:12.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:11 smithi153 ceph-mon[36569]: pgmap v4552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:12.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:11 smithi104 ceph-mon[36624]: pgmap v4552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:13.741+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:27:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:13.741+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:27:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:13.741+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:27:13.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:13 smithi081 ceph-mon[31453]: pgmap v4553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:14.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:13 smithi153 ceph-mon[36569]: pgmap v4553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:14.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:13 smithi104 ceph-mon[36624]: pgmap v4553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:16.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:16 smithi081 ceph-mon[31453]: pgmap v4554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:16.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:16 smithi153 ceph-mon[36569]: pgmap v4554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:16.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:16 smithi104 ceph-mon[36624]: pgmap v4554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:17.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:27:17.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:27:17.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: pgmap v4555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:27:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:27:18.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:27:18.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: pgmap v4555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:27:18.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: pgmap v4555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:27:18.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:27:18.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:27:18.644 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:27:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:19 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:27:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:19 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:27:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:19 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:27:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:19 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:27:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:19 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:27:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:19 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:27:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:19 smithi153 ceph-mon[36569]: pgmap v4556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:19 smithi153 ceph-mon[36569]: pgmap v4557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:19.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:19 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:27:19.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:19 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:27:19.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:19 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:27:19.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:19 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:27:19.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:19 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:27:19.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:19 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:27:19.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:19 smithi104 ceph-mon[36624]: pgmap v4556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:19.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:19 smithi104 ceph-mon[36624]: pgmap v4557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:19.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:19 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:27:19.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:19 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:27:19.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:19 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:27:19.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:19 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:27:19.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:19 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:27:19.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:19 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:27:19.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:19 smithi081 ceph-mon[31453]: pgmap v4556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:19.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:19 smithi081 ceph-mon[31453]: pgmap v4557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:21 smithi153 ceph-mon[36569]: pgmap v4558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:21.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:21 smithi104 ceph-mon[36624]: pgmap v4558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:21.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:21 smithi081 ceph-mon[31453]: pgmap v4558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:23 smithi153 ceph-mon[36569]: pgmap v4559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:23.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:23 smithi104 ceph-mon[36624]: pgmap v4559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:23.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:23 smithi081 ceph-mon[31453]: pgmap v4559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:24.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:24 smithi153 ceph-mon[36569]: pgmap v4560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:24.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:24 smithi104 ceph-mon[36624]: pgmap v4560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:24.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:24 smithi081 ceph-mon[31453]: pgmap v4560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:27.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:27 smithi081 ceph-mon[31453]: pgmap v4561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:27 smithi153 ceph-mon[36569]: pgmap v4561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:27.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:27 smithi104 ceph-mon[36624]: pgmap v4561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:28.752+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:27:28.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:28.753+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:27:28.740 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:28.753+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:27:29.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:29 smithi081 ceph-mon[31453]: pgmap v4562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:29 smithi153 ceph-mon[36569]: pgmap v4562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:29.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:29 smithi104 ceph-mon[36624]: pgmap v4562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:31.345 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:31 smithi153 ceph-mon[36569]: pgmap v4563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:31.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:31 smithi081 ceph-mon[31453]: pgmap v4563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:31.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:31 smithi104 ceph-mon[36624]: pgmap v4563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:33.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:33 smithi153 ceph-mon[36569]: pgmap v4564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:33.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:33 smithi081 ceph-mon[31453]: pgmap v4564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:33 smithi104 ceph-mon[36624]: pgmap v4564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:35.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:35 smithi081 ceph-mon[31453]: pgmap v4565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:35.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:35 smithi153 ceph-mon[36569]: pgmap v4565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:35.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:35 smithi104 ceph-mon[36624]: pgmap v4565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:37.371 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:37 smithi104 ceph-mon[36624]: pgmap v4566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:37.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:37 smithi081 ceph-mon[31453]: pgmap v4566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:37.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:37 smithi153 ceph-mon[36569]: pgmap v4566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:39 smithi081 ceph-mon[31453]: pgmap v4567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:39.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:39 smithi153 ceph-mon[36569]: pgmap v4567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:39.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:39 smithi104 ceph-mon[36624]: pgmap v4567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:41.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:41 smithi081 ceph-mon[31453]: pgmap v4568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:41 smithi153 ceph-mon[36569]: pgmap v4568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:41.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:41 smithi104 ceph-mon[36624]: pgmap v4568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:43.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:43 smithi153 ceph-mon[36569]: pgmap v4569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:43.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:43 smithi081 ceph-mon[31453]: pgmap v4569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:43.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:43 smithi104 ceph-mon[36624]: pgmap v4569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:43.763+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:27:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:43.763+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:27:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:43.763+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:27:45.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:45 smithi081 ceph-mon[31453]: pgmap v4570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:45 smithi153 ceph-mon[36569]: pgmap v4570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:45.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:45 smithi104 ceph-mon[36624]: pgmap v4570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:47.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:47 smithi081 ceph-mon[31453]: pgmap v4571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:47.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:47 smithi153 ceph-mon[36569]: pgmap v4571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:47.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:47 smithi104 ceph-mon[36624]: pgmap v4571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:49.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:49 smithi081 ceph-mon[31453]: pgmap v4572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:49.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:49 smithi153 ceph-mon[36569]: pgmap v4572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:49.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:49 smithi104 ceph-mon[36624]: pgmap v4572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:51.195 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:51 smithi081 ceph-mon[31453]: pgmap v4573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:51.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:51 smithi153 ceph-mon[36569]: pgmap v4573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:51.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:51 smithi104 ceph-mon[36624]: pgmap v4573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:53.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:53 smithi153 ceph-mon[36569]: pgmap v4574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:53.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:53 smithi104 ceph-mon[36624]: pgmap v4574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:53.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:53 smithi081 ceph-mon[31453]: pgmap v4574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:55 smithi153 ceph-mon[36569]: pgmap v4575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:55.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:55 smithi104 ceph-mon[36624]: pgmap v4575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:55.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:55 smithi081 ceph-mon[31453]: pgmap v4575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:57.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:57 smithi153 ceph-mon[36569]: pgmap v4576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:57.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:57 smithi104 ceph-mon[36624]: pgmap v4576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:57.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:57 smithi081 ceph-mon[31453]: pgmap v4576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:58.779+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:27:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:58.779+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:27:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:27:58.779+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:27:59.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:27:59 smithi153 ceph-mon[36569]: pgmap v4577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:59.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:27:59 smithi104 ceph-mon[36624]: pgmap v4577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:27:59.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:27:59 smithi081 ceph-mon[31453]: pgmap v4577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:01.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:01 smithi153 ceph-mon[36569]: pgmap v4578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:01.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:01 smithi104 ceph-mon[36624]: pgmap v4578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:01.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:01 smithi081 ceph-mon[31453]: pgmap v4578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:03.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:03 smithi153 ceph-mon[36569]: pgmap v4579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:03.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:03 smithi104 ceph-mon[36624]: pgmap v4579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:03.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:03 smithi081 ceph-mon[31453]: pgmap v4579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:05.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:05 smithi153 ceph-mon[36569]: pgmap v4580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:05.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:05 smithi104 ceph-mon[36624]: pgmap v4580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:05.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:05 smithi081 ceph-mon[31453]: pgmap v4580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:06.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:06 smithi153 ceph-mon[36569]: pgmap v4581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:06.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:06 smithi104 ceph-mon[36624]: pgmap v4581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:06.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:06 smithi081 ceph-mon[31453]: pgmap v4581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:09.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:09 smithi081 ceph-mon[31453]: pgmap v4582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:09.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:09 smithi153 ceph-mon[36569]: pgmap v4582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:09.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:09 smithi104 ceph-mon[36624]: pgmap v4582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:11.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:11 smithi081 ceph-mon[31453]: pgmap v4583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:11.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:11 smithi153 ceph-mon[36569]: pgmap v4583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:11.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:11 smithi104 ceph-mon[36624]: pgmap v4583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:13.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:13 smithi081 ceph-mon[31453]: pgmap v4584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:13.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:13 smithi153 ceph-mon[36569]: pgmap v4584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:13.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:13 smithi104 ceph-mon[36624]: pgmap v4584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:13.785+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:28:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:13.785+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:28:13.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:13.785+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:28:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:14.005+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:28:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:14.005+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:28:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:14.006+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-12T23:28:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:14.006+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:28:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:14.006+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-12T23:28:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:14.006+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:28:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:14.006+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:28:13.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:14.007+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:28:15.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:15 smithi081 ceph-mon[31453]: pgmap v4585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:15.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:15 smithi153 ceph-mon[36569]: pgmap v4585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:15.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:15 smithi104 ceph-mon[36624]: pgmap v4585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:17.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:17 smithi081 ceph-mon[31453]: pgmap v4586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:17.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:17 smithi153 ceph-mon[36569]: pgmap v4586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:17.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:17 smithi104 ceph-mon[36624]: pgmap v4586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:19.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:19 smithi081 ceph-mon[31453]: pgmap v4587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:19.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:28:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:19 smithi153 ceph-mon[36569]: pgmap v4587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:28:19.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:19 smithi104 ceph-mon[36624]: pgmap v4587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:19.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:28:20.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:28:20.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:28:20.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:28:20.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:28:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:28:20.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:28:20.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:28:20.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:28:20.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:28:21.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:21 smithi153 ceph-mon[36569]: pgmap v4588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:21.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:21 smithi153 ceph-mon[36569]: pgmap v4589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:21.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:21 smithi104 ceph-mon[36624]: pgmap v4588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:21.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:21 smithi104 ceph-mon[36624]: pgmap v4589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:21.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:21 smithi081 ceph-mon[31453]: pgmap v4588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:21.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:21 smithi081 ceph-mon[31453]: pgmap v4589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:23 smithi153 ceph-mon[36569]: pgmap v4590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:23.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:23 smithi104 ceph-mon[36624]: pgmap v4590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:23.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:23 smithi081 ceph-mon[31453]: pgmap v4590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:25 smithi153 ceph-mon[36569]: pgmap v4591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:25.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:25 smithi104 ceph-mon[36624]: pgmap v4591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:25.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:25 smithi081 ceph-mon[31453]: pgmap v4591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:26.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:26 smithi153 ceph-mon[36569]: pgmap v4592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:26.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:26 smithi104 ceph-mon[36624]: pgmap v4592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:26.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:26 smithi081 ceph-mon[31453]: pgmap v4592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:28.786 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:28.800+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:28:28.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:28.800+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:28:28.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:28.800+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:28:28.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:28 smithi081 ceph-mon[31453]: pgmap v4593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:29.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:28 smithi153 ceph-mon[36569]: pgmap v4593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:29.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:28 smithi104 ceph-mon[36624]: pgmap v4593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:30.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:30 smithi081 ceph-mon[31453]: pgmap v4594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:31.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:30 smithi153 ceph-mon[36569]: pgmap v4594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:31.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:30 smithi104 ceph-mon[36624]: pgmap v4594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:33.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:32 smithi153 ceph-mon[36569]: pgmap v4595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:33.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:32 smithi104 ceph-mon[36624]: pgmap v4595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:33.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:32 smithi081 ceph-mon[31453]: pgmap v4595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:35.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:34 smithi153 ceph-mon[36569]: pgmap v4596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:35.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:34 smithi104 ceph-mon[36624]: pgmap v4596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:35.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:34 smithi081 ceph-mon[31453]: pgmap v4596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:37.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:36 smithi153 ceph-mon[36569]: pgmap v4597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:37.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:36 smithi104 ceph-mon[36624]: pgmap v4597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:37.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:36 smithi081 ceph-mon[31453]: pgmap v4597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:39.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:38 smithi153 ceph-mon[36569]: pgmap v4598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:39.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:38 smithi104 ceph-mon[36624]: pgmap v4598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:39.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:38 smithi081 ceph-mon[31453]: pgmap v4598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:41.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:40 smithi153 ceph-mon[36569]: pgmap v4599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:41.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:40 smithi104 ceph-mon[36624]: pgmap v4599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:41.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:40 smithi081 ceph-mon[31453]: pgmap v4599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:43.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:42 smithi153 ceph-mon[36569]: pgmap v4600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:43.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:42 smithi104 ceph-mon[36624]: pgmap v4600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:43.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:42 smithi081 ceph-mon[31453]: pgmap v4600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:43.814+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:28:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:43.814+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:28:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:43.814+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:28:44.945 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:44 smithi081 ceph-mon[31453]: pgmap v4601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:45.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:44 smithi153 ceph-mon[36569]: pgmap v4601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:45.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:44 smithi104 ceph-mon[36624]: pgmap v4601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:47.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:46 smithi153 ceph-mon[36569]: pgmap v4602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:47.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:46 smithi104 ceph-mon[36624]: pgmap v4602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:47.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:46 smithi081 ceph-mon[31453]: pgmap v4602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:49.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:48 smithi153 ceph-mon[36569]: pgmap v4603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:49.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:48 smithi104 ceph-mon[36624]: pgmap v4603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:49.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:48 smithi081 ceph-mon[31453]: pgmap v4603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:51.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:50 smithi153 ceph-mon[36569]: pgmap v4604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:51.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:50 smithi104 ceph-mon[36624]: pgmap v4604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:51.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:50 smithi081 ceph-mon[31453]: pgmap v4604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:53.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:52 smithi153 ceph-mon[36569]: pgmap v4605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:53.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:52 smithi104 ceph-mon[36624]: pgmap v4605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:53.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:52 smithi081 ceph-mon[31453]: pgmap v4605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:55.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:54 smithi153 ceph-mon[36569]: pgmap v4606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:55.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:54 smithi104 ceph-mon[36624]: pgmap v4606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:55.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:54 smithi081 ceph-mon[31453]: pgmap v4606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:57.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:56 smithi153 ceph-mon[36569]: pgmap v4607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:57.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:56 smithi104 ceph-mon[36624]: pgmap v4607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:57.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:56 smithi081 ceph-mon[31453]: pgmap v4607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:58.829+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:28:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:58.829+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:28:58.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:28:58.829+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:28:59.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:28:58 smithi153 ceph-mon[36569]: pgmap v4608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:59.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:28:58 smithi104 ceph-mon[36624]: pgmap v4608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:28:59.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:28:58 smithi081 ceph-mon[31453]: pgmap v4608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:01.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:00 smithi153 ceph-mon[36569]: pgmap v4609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:01.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:00 smithi104 ceph-mon[36624]: pgmap v4609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:01.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:00 smithi081 ceph-mon[31453]: pgmap v4609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:03.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:02 smithi153 ceph-mon[36569]: pgmap v4610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:03.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:02 smithi104 ceph-mon[36624]: pgmap v4610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:03.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:02 smithi081 ceph-mon[31453]: pgmap v4610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:05.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:04 smithi153 ceph-mon[36569]: pgmap v4611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:05.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:04 smithi104 ceph-mon[36624]: pgmap v4611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:05.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:04 smithi081 ceph-mon[31453]: pgmap v4611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:07.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:06 smithi153 ceph-mon[36569]: pgmap v4612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:07.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:06 smithi104 ceph-mon[36624]: pgmap v4612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:07.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:06 smithi081 ceph-mon[31453]: pgmap v4612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:09.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:08 smithi153 ceph-mon[36569]: pgmap v4613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:09.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:08 smithi104 ceph-mon[36624]: pgmap v4613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:09.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:08 smithi081 ceph-mon[31453]: pgmap v4613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:11.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:10 smithi153 ceph-mon[36569]: pgmap v4614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:11.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:10 smithi104 ceph-mon[36624]: pgmap v4614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:11.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:10 smithi081 ceph-mon[31453]: pgmap v4614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:13.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:12 smithi153 ceph-mon[36569]: pgmap v4615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:13.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:12 smithi104 ceph-mon[36624]: pgmap v4615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:13.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:12 smithi081 ceph-mon[31453]: pgmap v4615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:13.834+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:29:13.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:13.834+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:29:13.821 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:13.834+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:29:15.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:14 smithi153 ceph-mon[36569]: pgmap v4616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:15.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:14 smithi104 ceph-mon[36624]: pgmap v4616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:15.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:14 smithi081 ceph-mon[31453]: pgmap v4616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:17.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:16 smithi153 ceph-mon[36569]: pgmap v4617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:17.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:16 smithi104 ceph-mon[36624]: pgmap v4617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:17.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:16 smithi081 ceph-mon[31453]: pgmap v4617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:19.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:18 smithi153 ceph-mon[36569]: pgmap v4618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:19.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:18 smithi104 ceph-mon[36624]: pgmap v4618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:19.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:18 smithi081 ceph-mon[31453]: pgmap v4618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:21.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:20 smithi153 ceph-mon[36569]: pgmap v4619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:21.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:29:21.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:20 smithi104 ceph-mon[36624]: pgmap v4619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:21.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:29:21.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:20 smithi081 ceph-mon[31453]: pgmap v4619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:21.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:29:22.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:29:22.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:29:22.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:21 smithi153 ceph-mon[36569]: pgmap v4620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:22.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:21 smithi153 ceph-mon[36569]: pgmap v4621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:22.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:29:22.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:29:22.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:29:22.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:21 smithi104 ceph-mon[36624]: pgmap v4620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:22.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:21 smithi104 ceph-mon[36624]: pgmap v4621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:22.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:29:22.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:29:22.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:29:22.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:21 smithi081 ceph-mon[31453]: pgmap v4620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:22.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:21 smithi081 ceph-mon[31453]: pgmap v4621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:22.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:29:23.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:23 smithi153 ceph-mon[36569]: pgmap v4622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:23.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:23 smithi104 ceph-mon[36624]: pgmap v4622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:23.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:23 smithi081 ceph-mon[31453]: pgmap v4622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:25.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:25 smithi153 ceph-mon[36569]: pgmap v4623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:25.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:25 smithi104 ceph-mon[36624]: pgmap v4623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:25.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:25 smithi081 ceph-mon[31453]: pgmap v4623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:27.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:27 smithi153 ceph-mon[36569]: pgmap v4624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:27.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:27 smithi104 ceph-mon[36624]: pgmap v4624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:27.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:27 smithi081 ceph-mon[31453]: pgmap v4624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:28.849+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:29:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:28.849+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:29:28.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:28.849+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:29:29.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:29 smithi153 ceph-mon[36569]: pgmap v4625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:29.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:29 smithi104 ceph-mon[36624]: pgmap v4625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:29.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:29 smithi081 ceph-mon[31453]: pgmap v4625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:31.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:31 smithi153 ceph-mon[36569]: pgmap v4626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:31.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:31 smithi104 ceph-mon[36624]: pgmap v4626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:31.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:31 smithi081 ceph-mon[31453]: pgmap v4626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:33.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:33 smithi153 ceph-mon[36569]: pgmap v4627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:33 smithi104 ceph-mon[36624]: pgmap v4627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:33.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:33 smithi081 ceph-mon[31453]: pgmap v4627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:35.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:35 smithi153 ceph-mon[36569]: pgmap v4628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:35.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:35 smithi104 ceph-mon[36624]: pgmap v4628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:35.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:35 smithi081 ceph-mon[31453]: pgmap v4628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:37.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:37 smithi153 ceph-mon[36569]: pgmap v4629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:37.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:37 smithi104 ceph-mon[36624]: pgmap v4629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:37.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:37 smithi081 ceph-mon[31453]: pgmap v4629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:39 smithi153 ceph-mon[36569]: pgmap v4630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:39.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:39 smithi104 ceph-mon[36624]: pgmap v4630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:39.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:39 smithi081 ceph-mon[31453]: pgmap v4630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:41 smithi153 ceph-mon[36569]: pgmap v4631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:41.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:41 smithi104 ceph-mon[36624]: pgmap v4631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:41 smithi081 ceph-mon[31453]: pgmap v4631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:43 smithi153 ceph-mon[36569]: pgmap v4632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:43.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:43 smithi104 ceph-mon[36624]: pgmap v4632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:43.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:43 smithi081 ceph-mon[31453]: pgmap v4632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:43.864+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:29:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:43.864+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:29:43.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:43.864+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:29:45.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:45 smithi153 ceph-mon[36569]: pgmap v4633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:45.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:45 smithi104 ceph-mon[36624]: pgmap v4633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:45.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:45 smithi081 ceph-mon[31453]: pgmap v4633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:47.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:47 smithi153 ceph-mon[36569]: pgmap v4634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:47.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:47 smithi104 ceph-mon[36624]: pgmap v4634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:47.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:47 smithi081 ceph-mon[31453]: pgmap v4634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:49.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:49 smithi153 ceph-mon[36569]: pgmap v4635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:49.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:49 smithi104 ceph-mon[36624]: pgmap v4635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:49.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:49 smithi081 ceph-mon[31453]: pgmap v4635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:51.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:51 smithi153 ceph-mon[36569]: pgmap v4636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:51.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:51 smithi104 ceph-mon[36624]: pgmap v4636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:51.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:51 smithi081 ceph-mon[31453]: pgmap v4636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:53.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:53 smithi153 ceph-mon[36569]: pgmap v4637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:53.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:53 smithi104 ceph-mon[36624]: pgmap v4637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:53.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:53 smithi081 ceph-mon[31453]: pgmap v4637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:55.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:55 smithi153 ceph-mon[36569]: pgmap v4638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:55.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:55 smithi104 ceph-mon[36624]: pgmap v4638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:55.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:55 smithi081 ceph-mon[31453]: pgmap v4638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:57.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:57 smithi153 ceph-mon[36569]: pgmap v4639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:57.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:57 smithi104 ceph-mon[36624]: pgmap v4639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:57.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:57 smithi081 ceph-mon[31453]: pgmap v4639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:58.868+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:29:58.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:58.868+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:29:58.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:29:58.868+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:29:59.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:29:59 smithi153 ceph-mon[36569]: pgmap v4640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:59.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:29:59 smithi104 ceph-mon[36624]: pgmap v4640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:29:59.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:29:59 smithi081 ceph-mon[31453]: pgmap v4640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:00.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:30:00.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:30:00.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:30:01.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:01 smithi153 ceph-mon[36569]: pgmap v4641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:01.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:01 smithi104 ceph-mon[36624]: pgmap v4641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:01.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:01 smithi081 ceph-mon[31453]: pgmap v4641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:03.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:03 smithi153 ceph-mon[36569]: pgmap v4642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:03.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:03 smithi104 ceph-mon[36624]: pgmap v4642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:03.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:03 smithi081 ceph-mon[31453]: pgmap v4642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:05.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:05 smithi153 ceph-mon[36569]: pgmap v4643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:05.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:05 smithi104 ceph-mon[36624]: pgmap v4643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:05.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:05 smithi081 ceph-mon[31453]: pgmap v4643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:07.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:07 smithi153 ceph-mon[36569]: pgmap v4644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:07.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:07 smithi104 ceph-mon[36624]: pgmap v4644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:07.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:07 smithi081 ceph-mon[31453]: pgmap v4644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:09.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:09 smithi153 ceph-mon[36569]: pgmap v4645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:09.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:09 smithi104 ceph-mon[36624]: pgmap v4645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:09.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:09 smithi081 ceph-mon[31453]: pgmap v4645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:11.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:11 smithi153 ceph-mon[36569]: pgmap v4646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:11.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:11 smithi104 ceph-mon[36624]: pgmap v4646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:11.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:11 smithi081 ceph-mon[31453]: pgmap v4646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:13.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:13 smithi153 ceph-mon[36569]: pgmap v4647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:13.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:13 smithi104 ceph-mon[36624]: pgmap v4647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:13.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:13 smithi081 ceph-mon[31453]: pgmap v4647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:13.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:13.875+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:30:13.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:13.875+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:30:13.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:13.875+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:30:15.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:15 smithi153 ceph-mon[36569]: pgmap v4648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:15.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:15 smithi104 ceph-mon[36624]: pgmap v4648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:15.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:15 smithi081 ceph-mon[31453]: pgmap v4648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:17.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:17 smithi153 ceph-mon[36569]: pgmap v4649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:17.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:17 smithi104 ceph-mon[36624]: pgmap v4649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:17.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:17 smithi081 ceph-mon[31453]: pgmap v4649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:19.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:19 smithi153 ceph-mon[36569]: pgmap v4650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:19.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:19 smithi104 ceph-mon[36624]: pgmap v4650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:19.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:19 smithi081 ceph-mon[31453]: pgmap v4650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:21.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:21 smithi153 ceph-mon[36569]: pgmap v4651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:21.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:21 smithi104 ceph-mon[36624]: pgmap v4651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:21.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:21 smithi081 ceph-mon[31453]: pgmap v4651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:22.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:30:22.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:30:22.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:30:23.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:30:23.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:30:23.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:23 smithi153 ceph-mon[36569]: pgmap v4652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:23.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:23 smithi153 ceph-mon[36569]: pgmap v4653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:23.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:30:23.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:30:23.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:30:23.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:23 smithi104 ceph-mon[36624]: pgmap v4652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:23.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:23 smithi104 ceph-mon[36624]: pgmap v4653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:23.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:30:23.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:30:23.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:30:23.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:23 smithi081 ceph-mon[31453]: pgmap v4652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:23.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:23 smithi081 ceph-mon[31453]: pgmap v4653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:23.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:30:26.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:25 smithi153 ceph-mon[36569]: pgmap v4654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:26.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:25 smithi104 ceph-mon[36624]: pgmap v4654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:26.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:25 smithi081 ceph-mon[31453]: pgmap v4654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:28.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:27 smithi153 ceph-mon[36569]: pgmap v4655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:28.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:27 smithi104 ceph-mon[36624]: pgmap v4655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:28.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:27 smithi081 ceph-mon[31453]: pgmap v4655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:28.887+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:30:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:28.887+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:30:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:28.887+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:30:30.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:29 smithi104 ceph-mon[36624]: pgmap v4656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:30.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:29 smithi081 ceph-mon[31453]: pgmap v4656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:30.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:29 smithi153 ceph-mon[36569]: pgmap v4656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:32.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:31 smithi104 ceph-mon[36624]: pgmap v4657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:32.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:31 smithi081 ceph-mon[31453]: pgmap v4657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:32.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:31 smithi153 ceph-mon[36569]: pgmap v4657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:34.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:33 smithi104 ceph-mon[36624]: pgmap v4658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:34.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:33 smithi153 ceph-mon[36569]: pgmap v4658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:34.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:33 smithi081 ceph-mon[31453]: pgmap v4658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:36.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:35 smithi104 ceph-mon[36624]: pgmap v4659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:36.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:35 smithi081 ceph-mon[31453]: pgmap v4659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:36.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:35 smithi153 ceph-mon[36569]: pgmap v4659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:38.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:37 smithi104 ceph-mon[36624]: pgmap v4660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:38.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:37 smithi081 ceph-mon[31453]: pgmap v4660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:38.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:37 smithi153 ceph-mon[36569]: pgmap v4660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:40.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:39 smithi081 ceph-mon[31453]: pgmap v4661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:40.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:39 smithi153 ceph-mon[36569]: pgmap v4661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:40.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:39 smithi104 ceph-mon[36624]: pgmap v4661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:42.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:41 smithi104 ceph-mon[36624]: pgmap v4662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:42.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:41 smithi081 ceph-mon[31453]: pgmap v4662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:42.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:41 smithi153 ceph-mon[36569]: pgmap v4662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:43.903+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:30:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:43.903+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:30:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:43.903+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:30:44.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:43 smithi081 ceph-mon[31453]: pgmap v4663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:44.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:43 smithi153 ceph-mon[36569]: pgmap v4663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:44.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:43 smithi104 ceph-mon[36624]: pgmap v4663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:46.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:45 smithi081 ceph-mon[31453]: pgmap v4664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:46.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:45 smithi153 ceph-mon[36569]: pgmap v4664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:46.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:45 smithi104 ceph-mon[36624]: pgmap v4664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:48.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:47 smithi081 ceph-mon[31453]: pgmap v4665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:48.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:47 smithi153 ceph-mon[36569]: pgmap v4665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:48.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:47 smithi104 ceph-mon[36624]: pgmap v4665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:50.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:49 smithi081 ceph-mon[31453]: pgmap v4666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:50.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:49 smithi153 ceph-mon[36569]: pgmap v4666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:50.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:49 smithi104 ceph-mon[36624]: pgmap v4666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:52.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:51 smithi081 ceph-mon[31453]: pgmap v4667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:52.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:51 smithi153 ceph-mon[36569]: pgmap v4667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:52.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:51 smithi104 ceph-mon[36624]: pgmap v4667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:54.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:53 smithi081 ceph-mon[31453]: pgmap v4668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:53 smithi153 ceph-mon[36569]: pgmap v4668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:54.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:53 smithi104 ceph-mon[36624]: pgmap v4668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:56.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:55 smithi081 ceph-mon[31453]: pgmap v4669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:56.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:55 smithi153 ceph-mon[36569]: pgmap v4669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:56.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:55 smithi104 ceph-mon[36624]: pgmap v4669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:58.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:57 smithi081 ceph-mon[31453]: pgmap v4670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:58.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:57 smithi153 ceph-mon[36569]: pgmap v4670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:58.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:57 smithi104 ceph-mon[36624]: pgmap v4670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:30:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:58.918+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:30:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:58.918+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:30:58.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:30:58.918+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:31:00.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:30:59 smithi081 ceph-mon[31453]: pgmap v4671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:00.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:30:59 smithi153 ceph-mon[36569]: pgmap v4671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:00.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:30:59 smithi104 ceph-mon[36624]: pgmap v4671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:02.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:01 smithi081 ceph-mon[31453]: pgmap v4672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:02.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:01 smithi153 ceph-mon[36569]: pgmap v4672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:02.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:01 smithi104 ceph-mon[36624]: pgmap v4672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:04.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:03 smithi081 ceph-mon[31453]: pgmap v4673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:04.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:03 smithi153 ceph-mon[36569]: pgmap v4673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:04.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:03 smithi104 ceph-mon[36624]: pgmap v4673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:06.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:05 smithi081 ceph-mon[31453]: pgmap v4674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:06.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:05 smithi153 ceph-mon[36569]: pgmap v4674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:06.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:05 smithi104 ceph-mon[36624]: pgmap v4674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:08.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:07 smithi153 ceph-mon[36569]: pgmap v4675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:08.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:07 smithi104 ceph-mon[36624]: pgmap v4675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:08.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:07 smithi081 ceph-mon[31453]: pgmap v4675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:10.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:09 smithi153 ceph-mon[36569]: pgmap v4676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:10.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:09 smithi104 ceph-mon[36624]: pgmap v4676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:10.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:09 smithi081 ceph-mon[31453]: pgmap v4676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.109+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-12T23:31:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.109+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-12T23:31:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.109+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f40bd040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-12T23:31:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.109+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-12T23:31:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.109+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-12T23:31:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.109+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T23:31:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.109+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T23:31:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.109+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T23:31:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.110+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T23:31:12.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:12.111+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T23:31:12.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:11 smithi153 ceph-mon[36569]: pgmap v4677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:12.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:11 smithi104 ceph-mon[36624]: pgmap v4677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:12.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:11 smithi081 ceph-mon[31453]: pgmap v4677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:13.927+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:31:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:13.927+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:31:13.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:13.927+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:14.035+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:31:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:14.035+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:14.036+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:31:14.022 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:14.036+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:31:14.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:13 smithi153 ceph-mon[36569]: pgmap v4678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:14.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:13 smithi104 ceph-mon[36624]: pgmap v4678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:14.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:13 smithi081 ceph-mon[31453]: pgmap v4678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:16.195 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:15 smithi081 ceph-mon[31453]: pgmap v4679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:16.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:15 smithi153 ceph-mon[36569]: pgmap v4679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:16.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:15 smithi104 ceph-mon[36624]: pgmap v4679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:18.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:17 smithi153 ceph-mon[36569]: pgmap v4680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:18.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:17 smithi104 ceph-mon[36624]: pgmap v4680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:18.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:17 smithi081 ceph-mon[31453]: pgmap v4680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:20.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:19 smithi153 ceph-mon[36569]: pgmap v4681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:20.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:19 smithi104 ceph-mon[36624]: pgmap v4681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:20.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:19 smithi081 ceph-mon[31453]: pgmap v4681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:22.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:21 smithi153 ceph-mon[36569]: pgmap v4682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:22.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:21 smithi104 ceph-mon[36624]: pgmap v4682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:22.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:21 smithi081 ceph-mon[31453]: pgmap v4682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:24.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:23 smithi153 ceph-mon[36569]: pgmap v4683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:31:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:23 smithi104 ceph-mon[36624]: pgmap v4683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:24.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:31:24.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:23 smithi081 ceph-mon[31453]: pgmap v4683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:24.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:31:24.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:24.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:25 smithi153 ceph-mon[36569]: pgmap v4684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:26.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:25 smithi104 ceph-mon[36624]: pgmap v4684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:26.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:25 smithi081 ceph-mon[31453]: pgmap v4684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:26.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:26.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:28.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:27 smithi153 ceph-mon[36569]: pgmap v4685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:28.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:27 smithi104 ceph-mon[36624]: pgmap v4685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:27 smithi081 ceph-mon[31453]: pgmap v4685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:28.936+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:31:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:28.936+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:31:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:28.936+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:31:29.294 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:29.294 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:29.294 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:29 smithi081 ceph-mon[31453]: pgmap v4686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:29.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:29.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:29 smithi153 ceph-mon[36569]: pgmap v4686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:29.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:29.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:29.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:29 smithi104 ceph-mon[36624]: pgmap v4686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:30.537 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:31:30.547 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:31:30.547 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:30 smithi081 ceph-mon[31453]: pgmap v4687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:30.547 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:30 smithi081 ceph-mon[31453]: pgmap v4688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:30.547 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:30.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:31:30.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:31:30.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:30 smithi153 ceph-mon[36569]: pgmap v4687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:30.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:30 smithi153 ceph-mon[36569]: pgmap v4688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:30.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:30.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:31:30.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:31:30.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:30 smithi104 ceph-mon[36624]: pgmap v4687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:30.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:30 smithi104 ceph-mon[36624]: pgmap v4688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:30.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:31:33.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:32 smithi153 ceph-mon[36569]: pgmap v4689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:33.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:32 smithi104 ceph-mon[36624]: pgmap v4689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:33.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:32 smithi081 ceph-mon[31453]: pgmap v4689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:35.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:34 smithi153 ceph-mon[36569]: pgmap v4690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:35.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:34 smithi104 ceph-mon[36624]: pgmap v4690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:35.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:34 smithi081 ceph-mon[31453]: pgmap v4690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:37.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:36 smithi153 ceph-mon[36569]: pgmap v4691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:37.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:36 smithi104 ceph-mon[36624]: pgmap v4691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:37.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:36 smithi081 ceph-mon[31453]: pgmap v4691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:39.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:38 smithi153 ceph-mon[36569]: pgmap v4692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:39.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:38 smithi104 ceph-mon[36624]: pgmap v4692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:39.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:38 smithi081 ceph-mon[31453]: pgmap v4692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:40.923 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:40 smithi081 ceph-mon[31453]: pgmap v4693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:41.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:40 smithi153 ceph-mon[36569]: pgmap v4693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:41.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:40 smithi104 ceph-mon[36624]: pgmap v4693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:43.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:42 smithi153 ceph-mon[36569]: pgmap v4694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:43.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:42 smithi104 ceph-mon[36624]: pgmap v4694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:43.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:42 smithi081 ceph-mon[31453]: pgmap v4694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:43.952+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:31:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:43.952+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:31:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:43.952+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:31:45.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:44 smithi153 ceph-mon[36569]: pgmap v4695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:45.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:44 smithi104 ceph-mon[36624]: pgmap v4695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:45.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:44 smithi081 ceph-mon[31453]: pgmap v4695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:46.945 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:46 smithi081 ceph-mon[31453]: pgmap v4696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:47.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:46 smithi153 ceph-mon[36569]: pgmap v4696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:47.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:46 smithi104 ceph-mon[36624]: pgmap v4696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:49.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:48 smithi153 ceph-mon[36569]: pgmap v4697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:49.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:48 smithi104 ceph-mon[36624]: pgmap v4697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:49.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:48 smithi081 ceph-mon[31453]: pgmap v4697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:51.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:50 smithi153 ceph-mon[36569]: pgmap v4698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:51.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:50 smithi104 ceph-mon[36624]: pgmap v4698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:51.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:50 smithi081 ceph-mon[31453]: pgmap v4698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:53.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:52 smithi153 ceph-mon[36569]: pgmap v4699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:53.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:52 smithi104 ceph-mon[36624]: pgmap v4699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:53.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:52 smithi081 ceph-mon[31453]: pgmap v4699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:55.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:54 smithi153 ceph-mon[36569]: pgmap v4700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:55.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:54 smithi104 ceph-mon[36624]: pgmap v4700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:55.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:54 smithi081 ceph-mon[31453]: pgmap v4700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:57.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:56 smithi153 ceph-mon[36569]: pgmap v4701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:57.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:56 smithi104 ceph-mon[36624]: pgmap v4701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:57.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:56 smithi081 ceph-mon[31453]: pgmap v4701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:58.967+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:31:58.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:58.967+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:31:58.954 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:31:58.967+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:31:59.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:31:58 smithi153 ceph-mon[36569]: pgmap v4702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:59.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:31:58 smithi104 ceph-mon[36624]: pgmap v4702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:31:59.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:31:58 smithi081 ceph-mon[31453]: pgmap v4702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:01.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:00 smithi153 ceph-mon[36569]: pgmap v4703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:01.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:00 smithi104 ceph-mon[36624]: pgmap v4703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:01.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:00 smithi081 ceph-mon[31453]: pgmap v4703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:03.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:02 smithi153 ceph-mon[36569]: pgmap v4704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:03.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:02 smithi104 ceph-mon[36624]: pgmap v4704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:03.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:02 smithi081 ceph-mon[31453]: pgmap v4704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:05.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:04 smithi153 ceph-mon[36569]: pgmap v4705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:05.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:04 smithi104 ceph-mon[36624]: pgmap v4705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:05.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:04 smithi081 ceph-mon[31453]: pgmap v4705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:07.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:06 smithi153 ceph-mon[36569]: pgmap v4706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:07.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:06 smithi104 ceph-mon[36624]: pgmap v4706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:07.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:06 smithi081 ceph-mon[31453]: pgmap v4706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:09.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:08 smithi153 ceph-mon[36569]: pgmap v4707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:09.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:08 smithi104 ceph-mon[36624]: pgmap v4707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:09.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:08 smithi081 ceph-mon[31453]: pgmap v4707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:11.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:10 smithi153 ceph-mon[36569]: pgmap v4708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:11.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:10 smithi104 ceph-mon[36624]: pgmap v4708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:11.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:10 smithi081 ceph-mon[31453]: pgmap v4708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:13.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:12 smithi153 ceph-mon[36569]: pgmap v4709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:13.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:12 smithi104 ceph-mon[36624]: pgmap v4709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:13.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:12 smithi081 ceph-mon[31453]: pgmap v4709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:13.983+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:32:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:13.983+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:32:13.969 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:13.983+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:32:15.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:14 smithi153 ceph-mon[36569]: pgmap v4710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:15.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:14 smithi104 ceph-mon[36624]: pgmap v4710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:15.098 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:14 smithi081 ceph-mon[31453]: pgmap v4710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:17.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:16 smithi153 ceph-mon[36569]: pgmap v4711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:17.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:16 smithi104 ceph-mon[36624]: pgmap v4711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:17.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:16 smithi081 ceph-mon[31453]: pgmap v4711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:19.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:18 smithi153 ceph-mon[36569]: pgmap v4712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:19.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:18 smithi104 ceph-mon[36624]: pgmap v4712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:19.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:18 smithi081 ceph-mon[31453]: pgmap v4712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:21.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:20 smithi153 ceph-mon[36569]: pgmap v4713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:21.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:20 smithi104 ceph-mon[36624]: pgmap v4713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:21.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:20 smithi081 ceph-mon[31453]: pgmap v4713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:23.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:22 smithi153 ceph-mon[36569]: pgmap v4714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:23.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:22 smithi104 ceph-mon[36624]: pgmap v4714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:23.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:22 smithi081 ceph-mon[31453]: pgmap v4714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:25.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:24 smithi153 ceph-mon[36569]: pgmap v4715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:25.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:24 smithi104 ceph-mon[36624]: pgmap v4715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:25.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:24 smithi081 ceph-mon[31453]: pgmap v4715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:27.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:26 smithi153 ceph-mon[36569]: pgmap v4716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:27.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:26 smithi104 ceph-mon[36624]: pgmap v4716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:27.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:26 smithi081 ceph-mon[31453]: pgmap v4716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:28.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:28.992+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:32:28.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:28.992+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:32:28.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:28.992+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:32:29.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:28 smithi104 ceph-mon[36624]: pgmap v4717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:29.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:28 smithi081 ceph-mon[31453]: pgmap v4717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:29.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:28 smithi153 ceph-mon[36569]: pgmap v4717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:31.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:30 smithi104 ceph-mon[36624]: pgmap v4718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:31.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:32:31.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:31.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:31.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:30 smithi081 ceph-mon[31453]: pgmap v4718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:31.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:32:31.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:31.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:31.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:30 smithi153 ceph-mon[36569]: pgmap v4718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:31.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:32:31.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:31.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:32.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:32:32.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:32:32.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:31 smithi104 ceph-mon[36624]: pgmap v4719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:32.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:31 smithi104 ceph-mon[36624]: pgmap v4720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:32.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:32.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:32:32.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:32:32.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:31 smithi081 ceph-mon[31453]: pgmap v4719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:32.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:31 smithi081 ceph-mon[31453]: pgmap v4720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:32.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:32.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:32:32.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:32:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:31 smithi153 ceph-mon[36569]: pgmap v4719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:31 smithi153 ceph-mon[36569]: pgmap v4720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:32.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:32:33.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:33 smithi153 ceph-mon[36569]: pgmap v4721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:33 smithi104 ceph-mon[36624]: pgmap v4721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:33.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:33 smithi081 ceph-mon[31453]: pgmap v4721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:35.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:35 smithi153 ceph-mon[36569]: pgmap v4722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:35.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:35 smithi104 ceph-mon[36624]: pgmap v4722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:35.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:35 smithi081 ceph-mon[31453]: pgmap v4722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:37.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:37 smithi153 ceph-mon[36569]: pgmap v4723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:37.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:37 smithi104 ceph-mon[36624]: pgmap v4723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:37.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:37 smithi081 ceph-mon[31453]: pgmap v4723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:39 smithi153 ceph-mon[36569]: pgmap v4724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:39.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:39 smithi104 ceph-mon[36624]: pgmap v4724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:39.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:39 smithi081 ceph-mon[31453]: pgmap v4724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:41 smithi153 ceph-mon[36569]: pgmap v4725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:41.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:41 smithi104 ceph-mon[36624]: pgmap v4725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:41 smithi081 ceph-mon[31453]: pgmap v4725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:43 smithi153 ceph-mon[36569]: pgmap v4726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:43.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:43 smithi104 ceph-mon[36624]: pgmap v4726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:43.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:43 smithi081 ceph-mon[31453]: pgmap v4726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:44.005+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:32:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:44.005+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:32:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:44.005+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:32:45.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:45 smithi153 ceph-mon[36569]: pgmap v4727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:45.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:45 smithi104 ceph-mon[36624]: pgmap v4727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:45.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:45 smithi081 ceph-mon[31453]: pgmap v4727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:47.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:47 smithi153 ceph-mon[36569]: pgmap v4728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:47.544 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:47 smithi081 ceph-mon[31453]: pgmap v4728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:47.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:47 smithi104 ceph-mon[36624]: pgmap v4728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:49.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:49 smithi153 ceph-mon[36569]: pgmap v4729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:49.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:49 smithi104 ceph-mon[36624]: pgmap v4729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:49.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:49 smithi081 ceph-mon[31453]: pgmap v4729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:51.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:51 smithi153 ceph-mon[36569]: pgmap v4730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:51.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:51 smithi104 ceph-mon[36624]: pgmap v4730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:51.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:51 smithi081 ceph-mon[31453]: pgmap v4730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:53.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:53 smithi153 ceph-mon[36569]: pgmap v4731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:53.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:53 smithi104 ceph-mon[36624]: pgmap v4731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:53.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:53 smithi081 ceph-mon[31453]: pgmap v4731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:55.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:55 smithi153 ceph-mon[36569]: pgmap v4732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:55.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:55 smithi104 ceph-mon[36624]: pgmap v4732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:55.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:55 smithi081 ceph-mon[31453]: pgmap v4732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:57.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:57 smithi153 ceph-mon[36569]: pgmap v4733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:57.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:57 smithi104 ceph-mon[36624]: pgmap v4733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:57.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:57 smithi081 ceph-mon[31453]: pgmap v4733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:59.020+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:32:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:59.020+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:32:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:32:59.020+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:32:59.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:32:59 smithi153 ceph-mon[36569]: pgmap v4734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:59.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:32:59 smithi104 ceph-mon[36624]: pgmap v4734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:32:59.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:32:59 smithi081 ceph-mon[31453]: pgmap v4734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:01.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:01 smithi153 ceph-mon[36569]: pgmap v4735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:01.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:01 smithi104 ceph-mon[36624]: pgmap v4735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:01.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:01 smithi081 ceph-mon[31453]: pgmap v4735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:03.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:03 smithi153 ceph-mon[36569]: pgmap v4736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:03.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:03 smithi104 ceph-mon[36624]: pgmap v4736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:03.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:03 smithi081 ceph-mon[31453]: pgmap v4736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:05.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:05 smithi153 ceph-mon[36569]: pgmap v4737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:05.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:05 smithi104 ceph-mon[36624]: pgmap v4737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:05.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:05 smithi081 ceph-mon[31453]: pgmap v4737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:07.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:07 smithi153 ceph-mon[36569]: pgmap v4738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:07.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:07 smithi104 ceph-mon[36624]: pgmap v4738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:07.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:07 smithi081 ceph-mon[31453]: pgmap v4738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:09.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:09 smithi153 ceph-mon[36569]: pgmap v4739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:09.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:09 smithi104 ceph-mon[36624]: pgmap v4739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:09.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:09 smithi081 ceph-mon[31453]: pgmap v4739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:11.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:11 smithi153 ceph-mon[36569]: pgmap v4740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:11.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:11 smithi104 ceph-mon[36624]: pgmap v4740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:11.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:11 smithi081 ceph-mon[31453]: pgmap v4740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:13.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:13 smithi153 ceph-mon[36569]: pgmap v4741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:13.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:13 smithi104 ceph-mon[36624]: pgmap v4741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:13.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:13 smithi081 ceph-mon[31453]: pgmap v4741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:14.020 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:14.033+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:33:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:14.034+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:33:14.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:14.034+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:33:15.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:15 smithi153 ceph-mon[36569]: pgmap v4742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:15.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:15 smithi104 ceph-mon[36624]: pgmap v4742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:15.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:15 smithi081 ceph-mon[31453]: pgmap v4742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:17.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:17 smithi153 ceph-mon[36569]: pgmap v4743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:17.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:17 smithi104 ceph-mon[36624]: pgmap v4743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:17.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:17 smithi081 ceph-mon[31453]: pgmap v4743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:19.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:19 smithi153 ceph-mon[36569]: pgmap v4744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:19.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:19 smithi104 ceph-mon[36624]: pgmap v4744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:19.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:19 smithi081 ceph-mon[31453]: pgmap v4744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:21.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:21 smithi153 ceph-mon[36569]: pgmap v4745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:21.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:21 smithi104 ceph-mon[36624]: pgmap v4745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:21.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:21 smithi081 ceph-mon[31453]: pgmap v4745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:23.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:23 smithi153 ceph-mon[36569]: pgmap v4746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:23.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:23 smithi104 ceph-mon[36624]: pgmap v4746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:23.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:23 smithi081 ceph-mon[31453]: pgmap v4746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:25.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:25 smithi104 ceph-mon[36624]: pgmap v4747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:25.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:25 smithi081 ceph-mon[31453]: pgmap v4747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:25.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:25 smithi153 ceph-mon[36569]: pgmap v4747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:27.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:27 smithi104 ceph-mon[36624]: pgmap v4748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:27.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:27 smithi081 ceph-mon[31453]: pgmap v4748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:27.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:27 smithi153 ceph-mon[36569]: pgmap v4748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:29.050+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:33:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:29.050+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:33:29.036 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:29.050+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:33:29.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:29 smithi104 ceph-mon[36624]: pgmap v4749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:29.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:29 smithi081 ceph-mon[31453]: pgmap v4749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:29.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:29 smithi153 ceph-mon[36569]: pgmap v4749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:31.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:31 smithi104 ceph-mon[36624]: pgmap v4750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:31.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:31 smithi081 ceph-mon[31453]: pgmap v4750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:31.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:31 smithi153 ceph-mon[36569]: pgmap v4750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:32.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:33:32.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:33:32.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:33:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:33:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:33:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:33 smithi104 ceph-mon[36624]: pgmap v4751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:33 smithi104 ceph-mon[36624]: pgmap v4752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:33.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:33:33.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:33:33.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:33:33.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:33 smithi081 ceph-mon[31453]: pgmap v4751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:33.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:33 smithi081 ceph-mon[31453]: pgmap v4752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:33.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:33:33.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:33:33.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:33:33.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:33 smithi153 ceph-mon[36569]: pgmap v4751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:33.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:33 smithi153 ceph-mon[36569]: pgmap v4752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:33.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:33:36.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:35 smithi153 ceph-mon[36569]: pgmap v4753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:36.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:35 smithi104 ceph-mon[36624]: pgmap v4753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:36.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:35 smithi081 ceph-mon[31453]: pgmap v4753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:38.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:37 smithi153 ceph-mon[36569]: pgmap v4754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:38.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:37 smithi104 ceph-mon[36624]: pgmap v4754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:38.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:37 smithi081 ceph-mon[31453]: pgmap v4754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:40.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:39 smithi153 ceph-mon[36569]: pgmap v4755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:40.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:39 smithi104 ceph-mon[36624]: pgmap v4755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:40.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:39 smithi081 ceph-mon[31453]: pgmap v4755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:42.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:41 smithi153 ceph-mon[36569]: pgmap v4756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:42.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:41 smithi104 ceph-mon[36624]: pgmap v4756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:42.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:41 smithi081 ceph-mon[31453]: pgmap v4756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:44.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:43 smithi153 ceph-mon[36569]: pgmap v4757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:44.059+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:44.059+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:33:44.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:44.059+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:33:44.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:43 smithi104 ceph-mon[36624]: pgmap v4757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:44.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:43 smithi081 ceph-mon[31453]: pgmap v4757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:46.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:45 smithi153 ceph-mon[36569]: pgmap v4758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:46.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:45 smithi104 ceph-mon[36624]: pgmap v4758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:46.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:45 smithi081 ceph-mon[31453]: pgmap v4758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:48.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:47 smithi153 ceph-mon[36569]: pgmap v4759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:48.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:47 smithi104 ceph-mon[36624]: pgmap v4759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:48.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:47 smithi081 ceph-mon[31453]: pgmap v4759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:50.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:49 smithi104 ceph-mon[36624]: pgmap v4760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:50.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:49 smithi081 ceph-mon[31453]: pgmap v4760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:50.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:49 smithi153 ceph-mon[36569]: pgmap v4760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:52.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:51 smithi104 ceph-mon[36624]: pgmap v4761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:51 smithi081 ceph-mon[31453]: pgmap v4761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:52.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:51 smithi153 ceph-mon[36569]: pgmap v4761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:54.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:53 smithi104 ceph-mon[36624]: pgmap v4762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:54.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:53 smithi081 ceph-mon[31453]: pgmap v4762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:53 smithi153 ceph-mon[36569]: pgmap v4762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:56.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:55 smithi104 ceph-mon[36624]: pgmap v4763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:56.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:55 smithi081 ceph-mon[31453]: pgmap v4763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:56.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:55 smithi153 ceph-mon[36569]: pgmap v4763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:58.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:57 smithi104 ceph-mon[36624]: pgmap v4764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:58.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:57 smithi081 ceph-mon[31453]: pgmap v4764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:58.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:57 smithi153 ceph-mon[36569]: pgmap v4764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:33:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:59.074+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:33:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:59.074+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:33:59.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:33:59.074+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:34:00.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:33:59 smithi081 ceph-mon[31453]: pgmap v4765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:00.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:33:59 smithi153 ceph-mon[36569]: pgmap v4765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:00.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:33:59 smithi104 ceph-mon[36624]: pgmap v4765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:02.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:01 smithi081 ceph-mon[31453]: pgmap v4766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:02.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:01 smithi153 ceph-mon[36569]: pgmap v4766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:02.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:01 smithi104 ceph-mon[36624]: pgmap v4766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:04.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:03 smithi081 ceph-mon[31453]: pgmap v4767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:04.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:03 smithi153 ceph-mon[36569]: pgmap v4767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:04.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:03 smithi104 ceph-mon[36624]: pgmap v4767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:06.095 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:05 smithi153 ceph-mon[36569]: pgmap v4768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:06.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:05 smithi081 ceph-mon[31453]: pgmap v4768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:06.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:05 smithi104 ceph-mon[36624]: pgmap v4768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:08.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:07 smithi081 ceph-mon[31453]: pgmap v4769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:08.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:07 smithi153 ceph-mon[36569]: pgmap v4769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:08.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:07 smithi104 ceph-mon[36624]: pgmap v4769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:10.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:09 smithi081 ceph-mon[31453]: pgmap v4770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:10.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:09 smithi153 ceph-mon[36569]: pgmap v4770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:10.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:09 smithi104 ceph-mon[36624]: pgmap v4770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:12.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:11 smithi081 ceph-mon[31453]: pgmap v4771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:12.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:11 smithi153 ceph-mon[36569]: pgmap v4771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:12.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:11 smithi104 ceph-mon[36624]: pgmap v4771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:14.065+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:34:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:14.065+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:34:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:14.065+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:34:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:14.065+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:34:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:14.089+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:34:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:14.089+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:34:14.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:14.089+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:34:14.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:13 smithi104 ceph-mon[36624]: pgmap v4772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:14.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:13 smithi081 ceph-mon[31453]: pgmap v4772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:14.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:13 smithi153 ceph-mon[36569]: pgmap v4772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:16.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:15 smithi081 ceph-mon[31453]: pgmap v4773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:16.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:15 smithi153 ceph-mon[36569]: pgmap v4773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:16.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:15 smithi104 ceph-mon[36624]: pgmap v4773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:18.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:17 smithi081 ceph-mon[31453]: pgmap v4774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:18.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:17 smithi153 ceph-mon[36569]: pgmap v4774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:18.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:17 smithi104 ceph-mon[36624]: pgmap v4774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:20.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:19 smithi081 ceph-mon[31453]: pgmap v4775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:20.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:19 smithi153 ceph-mon[36569]: pgmap v4775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:20.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:19 smithi104 ceph-mon[36624]: pgmap v4775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:22.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:21 smithi081 ceph-mon[31453]: pgmap v4776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:22.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:21 smithi153 ceph-mon[36569]: pgmap v4776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:22.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:21 smithi104 ceph-mon[36624]: pgmap v4776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:24.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:23 smithi081 ceph-mon[31453]: pgmap v4777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:24.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:23 smithi153 ceph-mon[36569]: pgmap v4777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:24.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:23 smithi104 ceph-mon[36624]: pgmap v4777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:26.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:25 smithi081 ceph-mon[31453]: pgmap v4778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:26.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:25 smithi153 ceph-mon[36569]: pgmap v4778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:26.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:25 smithi104 ceph-mon[36624]: pgmap v4778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:28.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:27 smithi153 ceph-mon[36569]: pgmap v4779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:28.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:27 smithi104 ceph-mon[36624]: pgmap v4779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:27 smithi081 ceph-mon[31453]: pgmap v4779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:29.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:29.105+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:34:29.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:29.105+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:34:29.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:29.105+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:34:30.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:29 smithi153 ceph-mon[36569]: pgmap v4780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:30.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:29 smithi104 ceph-mon[36624]: pgmap v4780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:30.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:29 smithi081 ceph-mon[31453]: pgmap v4780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:32.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:31 smithi153 ceph-mon[36569]: pgmap v4781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:32.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:31 smithi104 ceph-mon[36624]: pgmap v4781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:32.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:31 smithi081 ceph-mon[31453]: pgmap v4781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:34.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:33 smithi153 ceph-mon[36569]: pgmap v4782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:34.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:34:34.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:33 smithi104 ceph-mon[36624]: pgmap v4782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:34.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:34:34.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:33 smithi081 ceph-mon[31453]: pgmap v4782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:34.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:34:35.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:34:35.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:34:35.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:34:35.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:34:35.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:34:35.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:34:35.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:34:35.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:34:35.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:34:36.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:35 smithi153 ceph-mon[36569]: pgmap v4783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:36.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:35 smithi153 ceph-mon[36569]: pgmap v4784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:36.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:35 smithi104 ceph-mon[36624]: pgmap v4783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:36.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:35 smithi104 ceph-mon[36624]: pgmap v4784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:36.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:35 smithi081 ceph-mon[31453]: pgmap v4783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:36.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:35 smithi081 ceph-mon[31453]: pgmap v4784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:38.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:37 smithi153 ceph-mon[36569]: pgmap v4785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:38.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:37 smithi104 ceph-mon[36624]: pgmap v4785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:38.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:37 smithi081 ceph-mon[31453]: pgmap v4785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:40.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:39 smithi153 ceph-mon[36569]: pgmap v4786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:40.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:39 smithi104 ceph-mon[36624]: pgmap v4786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:40.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:39 smithi081 ceph-mon[31453]: pgmap v4786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:42.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:41 smithi153 ceph-mon[36569]: pgmap v4787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:42.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:41 smithi104 ceph-mon[36624]: pgmap v4787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:42.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:41 smithi081 ceph-mon[31453]: pgmap v4787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:44.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:44.117+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:34:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:44.117+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:34:44.103 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:44.117+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:34:44.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:43 smithi153 ceph-mon[36569]: pgmap v4788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:44.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:43 smithi104 ceph-mon[36624]: pgmap v4788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:44.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:43 smithi081 ceph-mon[31453]: pgmap v4788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:46.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:45 smithi153 ceph-mon[36569]: pgmap v4789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:46.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:45 smithi104 ceph-mon[36624]: pgmap v4789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:46.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:45 smithi081 ceph-mon[31453]: pgmap v4789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:48.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:47 smithi153 ceph-mon[36569]: pgmap v4790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:48.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:47 smithi104 ceph-mon[36624]: pgmap v4790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:48.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:47 smithi081 ceph-mon[31453]: pgmap v4790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:50.233 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:49 smithi081 ceph-mon[31453]: pgmap v4791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:50.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:49 smithi153 ceph-mon[36569]: pgmap v4791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:50.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:49 smithi104 ceph-mon[36624]: pgmap v4791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:52.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:51 smithi153 ceph-mon[36569]: pgmap v4792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:52.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:51 smithi104 ceph-mon[36624]: pgmap v4792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:52.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:51 smithi081 ceph-mon[31453]: pgmap v4792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:54.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:53 smithi153 ceph-mon[36569]: pgmap v4793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:54.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:53 smithi104 ceph-mon[36624]: pgmap v4793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:54.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:53 smithi081 ceph-mon[31453]: pgmap v4793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:56.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:55 smithi153 ceph-mon[36569]: pgmap v4794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:56.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:55 smithi104 ceph-mon[36624]: pgmap v4794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:56.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:55 smithi081 ceph-mon[31453]: pgmap v4794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:58.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:34:57 smithi153 ceph-mon[36569]: pgmap v4795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:58.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:34:57 smithi104 ceph-mon[36624]: pgmap v4795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:58.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:34:57 smithi081 ceph-mon[31453]: pgmap v4795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:34:59.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:59.132+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:34:59.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:59.132+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:34:59.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:34:59.132+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:00.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:00 smithi153 ceph-mon[36569]: pgmap v4796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:00.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:00 smithi104 ceph-mon[36624]: pgmap v4796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:00.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:00 smithi081 ceph-mon[31453]: pgmap v4796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:02.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:02 smithi153 ceph-mon[36569]: pgmap v4797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:02.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:02 smithi104 ceph-mon[36624]: pgmap v4797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:02.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:02 smithi081 ceph-mon[31453]: pgmap v4797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:04.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:04 smithi153 ceph-mon[36569]: pgmap v4798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:04.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:04 smithi104 ceph-mon[36624]: pgmap v4798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:04.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:04 smithi081 ceph-mon[31453]: pgmap v4798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:06.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:06 smithi153 ceph-mon[36569]: pgmap v4799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:06.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:06 smithi104 ceph-mon[36624]: pgmap v4799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:06.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:06 smithi081 ceph-mon[31453]: pgmap v4799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:08.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:08 smithi153 ceph-mon[36569]: pgmap v4800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:08.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:08 smithi104 ceph-mon[36624]: pgmap v4800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:08.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:08 smithi081 ceph-mon[31453]: pgmap v4800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:10.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:10 smithi153 ceph-mon[36569]: pgmap v4801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:10.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:10 smithi104 ceph-mon[36624]: pgmap v4801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:10.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:10 smithi081 ceph-mon[31453]: pgmap v4801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:12.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:12 smithi153 ceph-mon[36569]: pgmap v4802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:12.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:12 smithi104 ceph-mon[36624]: pgmap v4802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:12.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:12 smithi081 ceph-mon[31453]: pgmap v4802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:14.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:14.134+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:14.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:14.134+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:14.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:14.134+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:35:14.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:14 smithi153 ceph-mon[36569]: pgmap v4803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:14.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:14 smithi104 ceph-mon[36624]: pgmap v4803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:14.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:14 smithi081 ceph-mon[31453]: pgmap v4803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:16.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:16 smithi153 ceph-mon[36569]: pgmap v4804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:16.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:16 smithi104 ceph-mon[36624]: pgmap v4804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:16.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:16 smithi081 ceph-mon[31453]: pgmap v4804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:18.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:18 smithi153 ceph-mon[36569]: pgmap v4805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:18.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:18 smithi104 ceph-mon[36624]: pgmap v4805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:18.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:18 smithi081 ceph-mon[31453]: pgmap v4805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:20.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:20 smithi153 ceph-mon[36569]: pgmap v4806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:20.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:20 smithi104 ceph-mon[36624]: pgmap v4806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:20.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:20 smithi081 ceph-mon[31453]: pgmap v4806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:22.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:22 smithi104 ceph-mon[36624]: pgmap v4807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:22.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:22 smithi081 ceph-mon[31453]: pgmap v4807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:22.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:22 smithi153 ceph-mon[36569]: pgmap v4807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:24.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:24 smithi104 ceph-mon[36624]: pgmap v4808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:24.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:24 smithi081 ceph-mon[31453]: pgmap v4808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:24.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:24 smithi153 ceph-mon[36569]: pgmap v4808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:26.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:26 smithi104 ceph-mon[36624]: pgmap v4809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:26.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:26 smithi081 ceph-mon[31453]: pgmap v4809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:26.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:26 smithi153 ceph-mon[36569]: pgmap v4809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:28.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:28 smithi104 ceph-mon[36624]: pgmap v4810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:28 smithi081 ceph-mon[31453]: pgmap v4810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:28.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:28 smithi153 ceph-mon[36569]: pgmap v4810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:29.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:29.143+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:29.143+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:35:29.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:29.143+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:30.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:30 smithi104 ceph-mon[36624]: pgmap v4811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:30.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:30 smithi081 ceph-mon[31453]: pgmap v4811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:30.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:30 smithi153 ceph-mon[36569]: pgmap v4811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:32.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:32 smithi081 ceph-mon[31453]: pgmap v4812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:32.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:32 smithi153 ceph-mon[36569]: pgmap v4812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:32.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:32 smithi104 ceph-mon[36624]: pgmap v4812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:34.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:34 smithi081 ceph-mon[31453]: pgmap v4813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:34.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:34 smithi153 ceph-mon[36569]: pgmap v4813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:34.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:34 smithi104 ceph-mon[36624]: pgmap v4813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:35.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:35:35.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:35:35.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:35:36.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:36 smithi081 ceph-mon[31453]: pgmap v4814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:36.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:35:36.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:35:36.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:35:36.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:36 smithi153 ceph-mon[36569]: pgmap v4814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:36.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:35:36.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:35:36.443 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:35:36.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:36 smithi104 ceph-mon[36624]: pgmap v4814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:36.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:35:36.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:35:36.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:35:37.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:37 smithi081 ceph-mon[31453]: pgmap v4815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:37.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:37 smithi081 ceph-mon[31453]: pgmap v4816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:37.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:37 smithi153 ceph-mon[36569]: pgmap v4815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:37.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:37 smithi153 ceph-mon[36569]: pgmap v4816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:37.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:37 smithi104 ceph-mon[36624]: pgmap v4815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:37.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:37 smithi104 ceph-mon[36624]: pgmap v4816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:39.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:39 smithi081 ceph-mon[31453]: pgmap v4817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:39 smithi153 ceph-mon[36569]: pgmap v4817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:39.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:39 smithi104 ceph-mon[36624]: pgmap v4817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:41.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:41 smithi081 ceph-mon[31453]: pgmap v4818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:41 smithi153 ceph-mon[36569]: pgmap v4818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:41.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:41 smithi104 ceph-mon[36624]: pgmap v4818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:43.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:43 smithi081 ceph-mon[31453]: pgmap v4819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:43 smithi153 ceph-mon[36569]: pgmap v4819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:43.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:43 smithi104 ceph-mon[36624]: pgmap v4819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:44.159+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:44.159+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:44.159+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:35:45.370 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:45 smithi104 ceph-mon[36624]: pgmap v4820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:45.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:45 smithi081 ceph-mon[31453]: pgmap v4820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:45.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:45 smithi153 ceph-mon[36569]: pgmap v4820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:47.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:47 smithi081 ceph-mon[31453]: pgmap v4821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:47.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:47 smithi153 ceph-mon[36569]: pgmap v4821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:47.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:47 smithi104 ceph-mon[36624]: pgmap v4821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:49.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:49 smithi081 ceph-mon[31453]: pgmap v4822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:49.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:49 smithi153 ceph-mon[36569]: pgmap v4822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:49.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:49 smithi104 ceph-mon[36624]: pgmap v4822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:51.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:51 smithi081 ceph-mon[31453]: pgmap v4823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:51.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:51 smithi153 ceph-mon[36569]: pgmap v4823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:51.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:51 smithi104 ceph-mon[36624]: pgmap v4823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:53.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:53 smithi081 ceph-mon[31453]: pgmap v4824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:53.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:53 smithi153 ceph-mon[36569]: pgmap v4824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:53.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:53 smithi104 ceph-mon[36624]: pgmap v4824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:55.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:55 smithi081 ceph-mon[31453]: pgmap v4825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:55.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:55 smithi153 ceph-mon[36569]: pgmap v4825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:55.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:55 smithi104 ceph-mon[36624]: pgmap v4825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:57.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:57 smithi081 ceph-mon[31453]: pgmap v4826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:57.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:57 smithi153 ceph-mon[36569]: pgmap v4826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:57.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:57 smithi104 ceph-mon[36624]: pgmap v4826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:59.174+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:59.174+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:35:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:35:59.174+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:35:59.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:35:59 smithi153 ceph-mon[36569]: pgmap v4827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:59.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:35:59 smithi104 ceph-mon[36624]: pgmap v4827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:35:59.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:35:59 smithi081 ceph-mon[31453]: pgmap v4827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:01.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:01 smithi153 ceph-mon[36569]: pgmap v4828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:01.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:01 smithi104 ceph-mon[36624]: pgmap v4828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:01.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:01 smithi081 ceph-mon[31453]: pgmap v4828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:03.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:03 smithi153 ceph-mon[36569]: pgmap v4829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:03.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:03 smithi104 ceph-mon[36624]: pgmap v4829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:03.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:03 smithi081 ceph-mon[31453]: pgmap v4829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:05.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:05 smithi153 ceph-mon[36569]: pgmap v4830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:05.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:05 smithi104 ceph-mon[36624]: pgmap v4830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:05.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:05 smithi081 ceph-mon[31453]: pgmap v4830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:07.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:07 smithi153 ceph-mon[36569]: pgmap v4831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:07.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:07 smithi104 ceph-mon[36624]: pgmap v4831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:07.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:07 smithi081 ceph-mon[31453]: pgmap v4831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:09.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:09 smithi153 ceph-mon[36569]: pgmap v4832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:09.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:09 smithi104 ceph-mon[36624]: pgmap v4832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:09.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:09 smithi081 ceph-mon[31453]: pgmap v4832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:11.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:11 smithi153 ceph-mon[36569]: pgmap v4833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:11.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:11 smithi104 ceph-mon[36624]: pgmap v4833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:11.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:11 smithi081 ceph-mon[31453]: pgmap v4833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:13.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:13 smithi153 ceph-mon[36569]: pgmap v4834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:13.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:13 smithi104 ceph-mon[36624]: pgmap v4834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:13.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:13 smithi081 ceph-mon[31453]: pgmap v4834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:14.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:14.189+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:36:14.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:14.189+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:36:14.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:14.189+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:36:15.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:15 smithi153 ceph-mon[36569]: pgmap v4835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:15.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:15 smithi104 ceph-mon[36624]: pgmap v4835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:15.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:15 smithi081 ceph-mon[31453]: pgmap v4835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:17.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:17 smithi153 ceph-mon[36569]: pgmap v4836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:17.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:17 smithi104 ceph-mon[36624]: pgmap v4836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:17.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:17 smithi081 ceph-mon[31453]: pgmap v4836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:19.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:19 smithi153 ceph-mon[36569]: pgmap v4837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:19.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:19 smithi104 ceph-mon[36624]: pgmap v4837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:19.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:19 smithi081 ceph-mon[31453]: pgmap v4837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:21.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:21 smithi153 ceph-mon[36569]: pgmap v4838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:21.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:21 smithi104 ceph-mon[36624]: pgmap v4838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:21.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:21 smithi081 ceph-mon[31453]: pgmap v4838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:23.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:23 smithi153 ceph-mon[36569]: pgmap v4839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:23.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:23 smithi104 ceph-mon[36624]: pgmap v4839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:23.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:23 smithi081 ceph-mon[31453]: pgmap v4839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:25.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:25 smithi153 ceph-mon[36569]: pgmap v4840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:25.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:25 smithi104 ceph-mon[36624]: pgmap v4840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:25.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:25 smithi081 ceph-mon[31453]: pgmap v4840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:27.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:27 smithi153 ceph-mon[36569]: pgmap v4841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:27.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:27 smithi104 ceph-mon[36624]: pgmap v4841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:27.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:27 smithi081 ceph-mon[31453]: pgmap v4841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:29.088+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:36:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:29.088+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:36:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:29.088+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:36:29.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:29.088+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:36:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:29.196+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:36:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:29.196+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:36:29.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:29.196+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:36:29.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:29 smithi153 ceph-mon[36569]: pgmap v4842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:29.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:29 smithi104 ceph-mon[36624]: pgmap v4842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:29.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:29 smithi081 ceph-mon[31453]: pgmap v4842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:31.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:31 smithi153 ceph-mon[36569]: pgmap v4843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:31.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:31 smithi104 ceph-mon[36624]: pgmap v4843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:31.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:31 smithi081 ceph-mon[31453]: pgmap v4843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:33.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:33 smithi153 ceph-mon[36569]: pgmap v4844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:33.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:33 smithi104 ceph-mon[36624]: pgmap v4844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:33.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:33 smithi081 ceph-mon[31453]: pgmap v4844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:35.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:35 smithi153 ceph-mon[36569]: pgmap v4845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:35.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:35 smithi104 ceph-mon[36624]: pgmap v4845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:35.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:35 smithi081 ceph-mon[31453]: pgmap v4845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:36.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:36:36.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:36:36.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:36:37.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:37 smithi153 ceph-mon[36569]: pgmap v4846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:37.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:37 smithi104 ceph-mon[36624]: pgmap v4846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:37.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:37 smithi081 ceph-mon[31453]: pgmap v4846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:39 smithi081 ceph-mon[31453]: pgmap v4847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:36:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:36:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:39 smithi081 ceph-mon[31453]: pgmap v4848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:39 smithi081 ceph-mon[31453]: pgmap v4849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:39 smithi153 ceph-mon[36569]: pgmap v4847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:36:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:36:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:39 smithi153 ceph-mon[36569]: pgmap v4848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:39 smithi153 ceph-mon[36569]: pgmap v4849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:39.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:39 smithi104 ceph-mon[36624]: pgmap v4847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:39.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:39.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:36:39.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:36:39.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:39 smithi104 ceph-mon[36624]: pgmap v4848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:39 smithi104 ceph-mon[36624]: pgmap v4849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:39.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:36:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:41 smithi153 ceph-mon[36569]: pgmap v4850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:41.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:41 smithi104 ceph-mon[36624]: pgmap v4850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:41.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:41 smithi081 ceph-mon[31453]: pgmap v4850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:43 smithi153 ceph-mon[36569]: pgmap v4851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:43.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:43 smithi104 ceph-mon[36624]: pgmap v4851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:43.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:43 smithi081 ceph-mon[31453]: pgmap v4851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:44.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:44.199+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:36:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:44.199+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:36:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:44.199+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:36:45.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:45 smithi153 ceph-mon[36569]: pgmap v4852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:45.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:45 smithi104 ceph-mon[36624]: pgmap v4852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:45.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:45 smithi081 ceph-mon[31453]: pgmap v4852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:47.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:47 smithi153 ceph-mon[36569]: pgmap v4853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:47.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:47 smithi104 ceph-mon[36624]: pgmap v4853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:47.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:47 smithi081 ceph-mon[31453]: pgmap v4853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:49.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:49 smithi153 ceph-mon[36569]: pgmap v4854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:49.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:49 smithi104 ceph-mon[36624]: pgmap v4854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:49.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:49 smithi081 ceph-mon[31453]: pgmap v4854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:51.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:51 smithi153 ceph-mon[36569]: pgmap v4855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:51.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:51 smithi104 ceph-mon[36624]: pgmap v4855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:51.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:51 smithi081 ceph-mon[31453]: pgmap v4855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:53.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:53 smithi153 ceph-mon[36569]: pgmap v4856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:53.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:53 smithi104 ceph-mon[36624]: pgmap v4856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:53.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:53 smithi081 ceph-mon[31453]: pgmap v4856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:55.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:55 smithi153 ceph-mon[36569]: pgmap v4857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:55.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:55 smithi104 ceph-mon[36624]: pgmap v4857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:55.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:55 smithi081 ceph-mon[31453]: pgmap v4857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:56.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:56 smithi081 ceph-mon[31453]: pgmap v4858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:56.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:56 smithi153 ceph-mon[36569]: pgmap v4858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:56.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:56 smithi104 ceph-mon[36624]: pgmap v4858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:59.215+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:36:59.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:59.215+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:36:59.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:36:59.215+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:36:59.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:36:59 smithi153 ceph-mon[36569]: pgmap v4859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:59.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:36:59 smithi104 ceph-mon[36624]: pgmap v4859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:36:59.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:36:59 smithi081 ceph-mon[31453]: pgmap v4859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:01.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:01 smithi153 ceph-mon[36569]: pgmap v4860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:01.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:01 smithi104 ceph-mon[36624]: pgmap v4860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:01.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:01 smithi081 ceph-mon[31453]: pgmap v4860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:03.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:03 smithi153 ceph-mon[36569]: pgmap v4861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:03.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:03 smithi104 ceph-mon[36624]: pgmap v4861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:03.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:03 smithi081 ceph-mon[31453]: pgmap v4861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:05.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:05 smithi153 ceph-mon[36569]: pgmap v4862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:05.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:05 smithi104 ceph-mon[36624]: pgmap v4862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:05.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:05 smithi081 ceph-mon[31453]: pgmap v4862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:07.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:07 smithi153 ceph-mon[36569]: pgmap v4863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:07.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:07 smithi104 ceph-mon[36624]: pgmap v4863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:07.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:07 smithi081 ceph-mon[31453]: pgmap v4863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:09.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:09 smithi153 ceph-mon[36569]: pgmap v4864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:09.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:09 smithi104 ceph-mon[36624]: pgmap v4864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:09.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:09 smithi081 ceph-mon[31453]: pgmap v4864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:11.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:11 smithi153 ceph-mon[36569]: pgmap v4865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:11.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:11 smithi104 ceph-mon[36624]: pgmap v4865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:11.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:11 smithi081 ceph-mon[31453]: pgmap v4865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:13.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:13 smithi153 ceph-mon[36569]: pgmap v4866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:13.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:13 smithi104 ceph-mon[36624]: pgmap v4866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:13.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:13 smithi081 ceph-mon[31453]: pgmap v4866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:14.094+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:37:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:14.094+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:37:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:14.095+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:37:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:14.095+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:37:14.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:14.230+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:37:14.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:14.231+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:37:14.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:14.231+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:37:15.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:15 smithi153 ceph-mon[36569]: pgmap v4867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:15.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:15 smithi104 ceph-mon[36624]: pgmap v4867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:15.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:15 smithi081 ceph-mon[31453]: pgmap v4867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:17.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:17 smithi153 ceph-mon[36569]: pgmap v4868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:17.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:17 smithi104 ceph-mon[36624]: pgmap v4868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:17.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:17 smithi081 ceph-mon[31453]: pgmap v4868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:19.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:19 smithi153 ceph-mon[36569]: pgmap v4869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:19.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:19 smithi104 ceph-mon[36624]: pgmap v4869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:19.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:19 smithi081 ceph-mon[31453]: pgmap v4869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:21.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:21 smithi153 ceph-mon[36569]: pgmap v4870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:21.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:21 smithi104 ceph-mon[36624]: pgmap v4870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:21.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:21 smithi081 ceph-mon[31453]: pgmap v4870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:23.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:23 smithi153 ceph-mon[36569]: pgmap v4871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:23.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:23 smithi104 ceph-mon[36624]: pgmap v4871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:23.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:23 smithi081 ceph-mon[31453]: pgmap v4871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:25.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:25 smithi153 ceph-mon[36569]: pgmap v4872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:25.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:25 smithi104 ceph-mon[36624]: pgmap v4872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:25.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:25 smithi081 ceph-mon[31453]: pgmap v4872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:27.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:27 smithi153 ceph-mon[36569]: pgmap v4873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:27.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:27 smithi104 ceph-mon[36624]: pgmap v4873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:27.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:27 smithi081 ceph-mon[31453]: pgmap v4873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:29.243+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:37:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:29.243+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:37:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:29.243+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:37:29.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:29 smithi153 ceph-mon[36569]: pgmap v4874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:29.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:29 smithi104 ceph-mon[36624]: pgmap v4874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:29.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:29 smithi081 ceph-mon[31453]: pgmap v4874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:30.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:30 smithi081 ceph-mon[31453]: pgmap v4875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:30.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:30 smithi153 ceph-mon[36569]: pgmap v4875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:30.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:30 smithi104 ceph-mon[36624]: pgmap v4875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:33.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:33 smithi153 ceph-mon[36569]: pgmap v4876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:33.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:33 smithi104 ceph-mon[36624]: pgmap v4876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:33.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:33 smithi081 ceph-mon[31453]: pgmap v4876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:35.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:35 smithi153 ceph-mon[36569]: pgmap v4877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:35.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:35 smithi104 ceph-mon[36624]: pgmap v4877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:35.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:35 smithi081 ceph-mon[31453]: pgmap v4877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:37.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:37 smithi153 ceph-mon[36569]: pgmap v4878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:37.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:37 smithi104 ceph-mon[36624]: pgmap v4878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:37.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:37 smithi081 ceph-mon[31453]: pgmap v4878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:39.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:39 smithi153 ceph-mon[36569]: pgmap v4879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:39.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:37:39.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:39 smithi104 ceph-mon[36624]: pgmap v4879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:39.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:37:39.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:39 smithi081 ceph-mon[31453]: pgmap v4879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:39.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:37:40.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.544 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.544 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.544 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.546 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.546 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.546 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.546 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:37:40.546 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:37:40.546 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:37:40.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:37:40.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:37:40.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:37:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:41 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:37:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:41 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:37:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:41 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:37:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:41 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:37:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:41 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:37:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:41 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:37:41.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:41 smithi153 ceph-mon[36569]: pgmap v4880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:41.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:41 smithi153 ceph-mon[36569]: pgmap v4881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:41.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:41 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:37:41.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:41 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:37:41.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:41 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:37:41.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:41 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:37:41.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:41 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:37:41.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:41 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:37:41.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:41 smithi104 ceph-mon[36624]: pgmap v4880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:41.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:41 smithi104 ceph-mon[36624]: pgmap v4881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:41 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:37:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:41 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:37:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:41 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:37:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:41 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:37:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:41 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:37:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:41 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:37:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:41 smithi081 ceph-mon[31453]: pgmap v4880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:41.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:41 smithi081 ceph-mon[31453]: pgmap v4881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:43 smithi153 ceph-mon[36569]: pgmap v4882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:43.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:43 smithi104 ceph-mon[36624]: pgmap v4882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:43.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:43 smithi081 ceph-mon[31453]: pgmap v4882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:44.258+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:37:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:44.258+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:37:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:44.258+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:37:45.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:45 smithi153 ceph-mon[36569]: pgmap v4883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:45.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:45 smithi104 ceph-mon[36624]: pgmap v4883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:45.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:45 smithi081 ceph-mon[31453]: pgmap v4883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:47.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:47 smithi153 ceph-mon[36569]: pgmap v4884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:47.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:47 smithi104 ceph-mon[36624]: pgmap v4884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:47.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:47 smithi081 ceph-mon[31453]: pgmap v4884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:49.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:49 smithi153 ceph-mon[36569]: pgmap v4885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:49.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:49 smithi104 ceph-mon[36624]: pgmap v4885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:49.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:49 smithi081 ceph-mon[31453]: pgmap v4885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:51.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:51 smithi153 ceph-mon[36569]: pgmap v4886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:51.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:51 smithi104 ceph-mon[36624]: pgmap v4886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:51.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:51 smithi081 ceph-mon[31453]: pgmap v4886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:53.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:53 smithi153 ceph-mon[36569]: pgmap v4887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:53.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:53 smithi104 ceph-mon[36624]: pgmap v4887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:53.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:53 smithi081 ceph-mon[31453]: pgmap v4887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:55.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:55 smithi153 ceph-mon[36569]: pgmap v4888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:55.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:55 smithi104 ceph-mon[36624]: pgmap v4888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:55.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:55 smithi081 ceph-mon[31453]: pgmap v4888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:57.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:57 smithi153 ceph-mon[36569]: pgmap v4889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:57.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:57 smithi104 ceph-mon[36624]: pgmap v4889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:57.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:57 smithi081 ceph-mon[31453]: pgmap v4889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:59.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:59.262+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:37:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:59.262+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:37:59.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:37:59.262+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:37:59.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:37:59 smithi104 ceph-mon[36624]: pgmap v4890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:59.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:37:59 smithi081 ceph-mon[31453]: pgmap v4890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:37:59.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:37:59 smithi153 ceph-mon[36569]: pgmap v4890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:00.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:00 smithi104 ceph-mon[36624]: pgmap v4891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:00.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:00 smithi081 ceph-mon[31453]: pgmap v4891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:00.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:00 smithi153 ceph-mon[36569]: pgmap v4891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:03.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:02 smithi081 ceph-mon[31453]: pgmap v4892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:03.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:02 smithi153 ceph-mon[36569]: pgmap v4892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:03.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:02 smithi104 ceph-mon[36624]: pgmap v4892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:05.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:04 smithi081 ceph-mon[31453]: pgmap v4893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:05.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:04 smithi153 ceph-mon[36569]: pgmap v4893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:05.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:04 smithi104 ceph-mon[36624]: pgmap v4893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:07.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:06 smithi081 ceph-mon[31453]: pgmap v4894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:07.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:06 smithi153 ceph-mon[36569]: pgmap v4894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:07.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:06 smithi104 ceph-mon[36624]: pgmap v4894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:09.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:08 smithi081 ceph-mon[31453]: pgmap v4895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:09.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:08 smithi153 ceph-mon[36569]: pgmap v4895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:09.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:08 smithi104 ceph-mon[36624]: pgmap v4895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:11.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:10 smithi081 ceph-mon[31453]: pgmap v4896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:11.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:10 smithi153 ceph-mon[36569]: pgmap v4896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:11.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:10 smithi104 ceph-mon[36624]: pgmap v4896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:13.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:12 smithi081 ceph-mon[31453]: pgmap v4897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:13.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:12 smithi153 ceph-mon[36569]: pgmap v4897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:13.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:12 smithi104 ceph-mon[36624]: pgmap v4897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:14.273+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:38:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:14.273+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:38:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:14.273+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:38:15.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:14 smithi081 ceph-mon[31453]: pgmap v4898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:15.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:14 smithi153 ceph-mon[36569]: pgmap v4898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:15.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:14 smithi104 ceph-mon[36624]: pgmap v4898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:16 smithi153 ceph-mon[36569]: pgmap v4899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:17.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:16 smithi104 ceph-mon[36624]: pgmap v4899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:17.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:16 smithi081 ceph-mon[31453]: pgmap v4899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:19.194 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:18 smithi081 ceph-mon[31453]: pgmap v4900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:19.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:18 smithi153 ceph-mon[36569]: pgmap v4900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:19.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:18 smithi104 ceph-mon[36624]: pgmap v4900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:21.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:20 smithi153 ceph-mon[36569]: pgmap v4901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:21.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:20 smithi104 ceph-mon[36624]: pgmap v4901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:21.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:20 smithi081 ceph-mon[31453]: pgmap v4901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:23.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:22 smithi153 ceph-mon[36569]: pgmap v4902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:23.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:22 smithi104 ceph-mon[36624]: pgmap v4902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:23.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:22 smithi081 ceph-mon[31453]: pgmap v4902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:25.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:24 smithi153 ceph-mon[36569]: pgmap v4903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:25.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:24 smithi104 ceph-mon[36624]: pgmap v4903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:25.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:24 smithi081 ceph-mon[31453]: pgmap v4903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:27.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:26 smithi153 ceph-mon[36569]: pgmap v4904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:27.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:26 smithi104 ceph-mon[36624]: pgmap v4904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:27.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:26 smithi081 ceph-mon[31453]: pgmap v4904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:29.286+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:38:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:29.286+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:38:29.272 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:29.286+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:38:29.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:28 smithi153 ceph-mon[36569]: pgmap v4905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:29.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:28 smithi104 ceph-mon[36624]: pgmap v4905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:29.393 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:28 smithi081 ceph-mon[31453]: pgmap v4905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:31.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:30 smithi153 ceph-mon[36569]: pgmap v4906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:31.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:30 smithi104 ceph-mon[36624]: pgmap v4906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:31.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:30 smithi081 ceph-mon[31453]: pgmap v4906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:33.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:32 smithi153 ceph-mon[36569]: pgmap v4907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:33.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:32 smithi104 ceph-mon[36624]: pgmap v4907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:33.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:32 smithi081 ceph-mon[31453]: pgmap v4907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:35.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:34 smithi153 ceph-mon[36569]: pgmap v4908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:35.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:34 smithi104 ceph-mon[36624]: pgmap v4908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:35.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:34 smithi081 ceph-mon[31453]: pgmap v4908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:37.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:36 smithi153 ceph-mon[36569]: pgmap v4909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:37.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:36 smithi104 ceph-mon[36624]: pgmap v4909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:37.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:36 smithi081 ceph-mon[31453]: pgmap v4909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:39.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:38 smithi153 ceph-mon[36569]: pgmap v4910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:39.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:38 smithi104 ceph-mon[36624]: pgmap v4910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:39.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:38 smithi081 ceph-mon[31453]: pgmap v4910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:41.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:40 smithi153 ceph-mon[36569]: pgmap v4911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:41.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:38:41.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:40 smithi104 ceph-mon[36624]: pgmap v4911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:41.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:38:41.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:40 smithi081 ceph-mon[31453]: pgmap v4911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:41.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:38:42.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:38:42.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:38:42.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:38:42.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:38:42.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:38:42.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:38:42.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:38:42.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:38:42.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:38:43.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:42 smithi153 ceph-mon[36569]: pgmap v4912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:43.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:42 smithi153 ceph-mon[36569]: pgmap v4913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:43.293 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:42 smithi081 ceph-mon[31453]: pgmap v4912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:43.293 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:42 smithi081 ceph-mon[31453]: pgmap v4913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:43.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:42 smithi104 ceph-mon[36624]: pgmap v4912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:43.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:42 smithi104 ceph-mon[36624]: pgmap v4913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:44.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:44.292+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:38:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:44.292+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:38:44.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:44.292+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:38:45.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:44 smithi153 ceph-mon[36569]: pgmap v4914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:45.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:44 smithi104 ceph-mon[36624]: pgmap v4914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:45.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:44 smithi081 ceph-mon[31453]: pgmap v4914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:47.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:46 smithi153 ceph-mon[36569]: pgmap v4915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:47.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:46 smithi104 ceph-mon[36624]: pgmap v4915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:47.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:46 smithi081 ceph-mon[31453]: pgmap v4915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:49.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:48 smithi153 ceph-mon[36569]: pgmap v4916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:49.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:48 smithi104 ceph-mon[36624]: pgmap v4916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:49.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:48 smithi081 ceph-mon[31453]: pgmap v4916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:51.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:51 smithi153 ceph-mon[36569]: pgmap v4917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:51.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:51 smithi104 ceph-mon[36624]: pgmap v4917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:51.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:51 smithi081 ceph-mon[31453]: pgmap v4917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:53.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:53 smithi153 ceph-mon[36569]: pgmap v4918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:53.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:53 smithi104 ceph-mon[36624]: pgmap v4918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:53.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:53 smithi081 ceph-mon[31453]: pgmap v4918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:55.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:55 smithi153 ceph-mon[36569]: pgmap v4919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:55.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:55 smithi104 ceph-mon[36624]: pgmap v4919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:55.394 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:55 smithi081 ceph-mon[31453]: pgmap v4919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:57.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:57 smithi153 ceph-mon[36569]: pgmap v4920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:57.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:57 smithi104 ceph-mon[36624]: pgmap v4920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:57.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:57 smithi081 ceph-mon[31453]: pgmap v4920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:59.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:38:59 smithi153 ceph-mon[36569]: pgmap v4921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:59.308+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:38:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:59.308+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:38:59.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:38:59.308+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:38:59.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:38:59 smithi104 ceph-mon[36624]: pgmap v4921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:38:59.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:38:59 smithi081 ceph-mon[31453]: pgmap v4921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:01.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:01 smithi153 ceph-mon[36569]: pgmap v4922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:01.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:01 smithi104 ceph-mon[36624]: pgmap v4922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:01.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:01 smithi081 ceph-mon[31453]: pgmap v4922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:03.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:03 smithi153 ceph-mon[36569]: pgmap v4923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:03.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:03 smithi104 ceph-mon[36624]: pgmap v4923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:03.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:03 smithi081 ceph-mon[31453]: pgmap v4923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:05.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:05 smithi153 ceph-mon[36569]: pgmap v4924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:05.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:05 smithi104 ceph-mon[36624]: pgmap v4924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:05.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:05 smithi081 ceph-mon[31453]: pgmap v4924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:07.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:07 smithi153 ceph-mon[36569]: pgmap v4925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:07.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:07 smithi104 ceph-mon[36624]: pgmap v4925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:07.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:07 smithi081 ceph-mon[31453]: pgmap v4925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:09.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:09 smithi153 ceph-mon[36569]: pgmap v4926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:09.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:09 smithi104 ceph-mon[36624]: pgmap v4926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:09.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:09 smithi081 ceph-mon[31453]: pgmap v4926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:11.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:11 smithi153 ceph-mon[36569]: pgmap v4927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:11.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:11 smithi104 ceph-mon[36624]: pgmap v4927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:11.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:11 smithi081 ceph-mon[31453]: pgmap v4927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:13.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:13 smithi104 ceph-mon[36624]: pgmap v4928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:13.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:13 smithi081 ceph-mon[31453]: pgmap v4928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:13.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:13 smithi153 ceph-mon[36569]: pgmap v4928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:14.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:14.323+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:39:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:14.323+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:39:14.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:14.323+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:39:15.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:15 smithi104 ceph-mon[36624]: pgmap v4929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:15.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:15 smithi081 ceph-mon[31453]: pgmap v4929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:15.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:15 smithi153 ceph-mon[36569]: pgmap v4929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:17.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:17 smithi104 ceph-mon[36624]: pgmap v4930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:17.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:17 smithi081 ceph-mon[31453]: pgmap v4930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:17.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:17 smithi153 ceph-mon[36569]: pgmap v4930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:19.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:19 smithi104 ceph-mon[36624]: pgmap v4931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:19.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:19 smithi081 ceph-mon[31453]: pgmap v4931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:19.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:19 smithi153 ceph-mon[36569]: pgmap v4931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:21.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:21 smithi104 ceph-mon[36624]: pgmap v4932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:21.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:21 smithi081 ceph-mon[31453]: pgmap v4932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:21.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:21 smithi153 ceph-mon[36569]: pgmap v4932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:23.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:23 smithi081 ceph-mon[31453]: pgmap v4933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:23.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:23 smithi153 ceph-mon[36569]: pgmap v4933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:23.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:23 smithi104 ceph-mon[36624]: pgmap v4933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:25.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:25 smithi081 ceph-mon[31453]: pgmap v4934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:25.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:25 smithi153 ceph-mon[36569]: pgmap v4934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:25.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:25 smithi104 ceph-mon[36624]: pgmap v4934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:27.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:27 smithi081 ceph-mon[31453]: pgmap v4935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:27.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:27 smithi153 ceph-mon[36569]: pgmap v4935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:27.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:27 smithi104 ceph-mon[36624]: pgmap v4935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:29.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:29.327+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:39:29.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:29.327+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:39:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:29.327+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:39:29.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:29 smithi081 ceph-mon[31453]: pgmap v4936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:29.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:29 smithi153 ceph-mon[36569]: pgmap v4936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:29.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:29 smithi104 ceph-mon[36624]: pgmap v4936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:31.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:31 smithi081 ceph-mon[31453]: pgmap v4937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:31.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:31 smithi153 ceph-mon[36569]: pgmap v4937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:31.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:31 smithi104 ceph-mon[36624]: pgmap v4937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:33.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:33 smithi081 ceph-mon[31453]: pgmap v4938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:33.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:33 smithi153 ceph-mon[36569]: pgmap v4938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:33.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:33 smithi104 ceph-mon[36624]: pgmap v4938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:35.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:35 smithi081 ceph-mon[31453]: pgmap v4939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:35.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:35 smithi153 ceph-mon[36569]: pgmap v4939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:35.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:35 smithi104 ceph-mon[36624]: pgmap v4939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:37.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:37 smithi081 ceph-mon[31453]: pgmap v4940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:37.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:37 smithi153 ceph-mon[36569]: pgmap v4940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:37.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:37 smithi104 ceph-mon[36624]: pgmap v4940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:39.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:39 smithi081 ceph-mon[31453]: pgmap v4941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:39.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:39 smithi153 ceph-mon[36569]: pgmap v4941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:39.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:39 smithi104 ceph-mon[36624]: pgmap v4941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:41.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:41 smithi081 ceph-mon[31453]: pgmap v4942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:41.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:41 smithi153 ceph-mon[36569]: pgmap v4942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:41.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:41 smithi104 ceph-mon[36624]: pgmap v4942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:42.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:39:42.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:39:42.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:39:43.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:43 smithi081 ceph-mon[31453]: pgmap v4943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:43.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:39:43.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:39:43.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:39:43.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:43 smithi153 ceph-mon[36569]: pgmap v4943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:39:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:39:43.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:39:43.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:43 smithi104 ceph-mon[36624]: pgmap v4943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:43.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:39:43.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:39:43.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:44.343+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:44.343+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:39:44.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:44.343+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:39:44.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:44 smithi081 ceph-mon[31453]: pgmap v4944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:44.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:44 smithi081 ceph-mon[31453]: pgmap v4945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:44.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:44 smithi153 ceph-mon[36569]: pgmap v4944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:44.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:44 smithi153 ceph-mon[36569]: pgmap v4945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:44.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:44 smithi104 ceph-mon[36624]: pgmap v4944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:44.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:44 smithi104 ceph-mon[36624]: pgmap v4945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:46.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:46 smithi153 ceph-mon[36569]: pgmap v4946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:46.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:46 smithi104 ceph-mon[36624]: pgmap v4946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:46.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:46 smithi081 ceph-mon[31453]: pgmap v4946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:48.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:48 smithi153 ceph-mon[36569]: pgmap v4947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:48.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:48 smithi104 ceph-mon[36624]: pgmap v4947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:48.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:48 smithi081 ceph-mon[31453]: pgmap v4947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:50.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:50 smithi081 ceph-mon[31453]: pgmap v4948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:50.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:50 smithi153 ceph-mon[36569]: pgmap v4948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:50.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:50 smithi104 ceph-mon[36624]: pgmap v4948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:52.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:52 smithi153 ceph-mon[36569]: pgmap v4949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:52.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:52 smithi104 ceph-mon[36624]: pgmap v4949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:52.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:52 smithi081 ceph-mon[31453]: pgmap v4949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:54.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:54 smithi153 ceph-mon[36569]: pgmap v4950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:54.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:54 smithi104 ceph-mon[36624]: pgmap v4950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:54.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:54 smithi081 ceph-mon[31453]: pgmap v4950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:56.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:56 smithi153 ceph-mon[36569]: pgmap v4951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:56.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:56 smithi104 ceph-mon[36624]: pgmap v4951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:56.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:56 smithi081 ceph-mon[31453]: pgmap v4951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:58.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:39:58 smithi153 ceph-mon[36569]: pgmap v4952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:58.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:39:58 smithi104 ceph-mon[36624]: pgmap v4952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:58.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:39:58 smithi081 ceph-mon[31453]: pgmap v4952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:39:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:59.358+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:39:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:59.358+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:39:59.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:39:59.358+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:40:00.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:00 smithi153 ceph-mon[36569]: pgmap v4953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:00.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:40:00.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:00 smithi104 ceph-mon[36624]: pgmap v4953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:00.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:40:00.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:00 smithi081 ceph-mon[31453]: pgmap v4953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:00.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:40:02.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:02 smithi153 ceph-mon[36569]: pgmap v4954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:02.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:02 smithi104 ceph-mon[36624]: pgmap v4954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:02.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:02 smithi081 ceph-mon[31453]: pgmap v4954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:04.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:04 smithi153 ceph-mon[36569]: pgmap v4955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:04.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:04 smithi104 ceph-mon[36624]: pgmap v4955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:04.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:04 smithi081 ceph-mon[31453]: pgmap v4955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:06.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:06 smithi153 ceph-mon[36569]: pgmap v4956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:06.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:06 smithi104 ceph-mon[36624]: pgmap v4956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:06.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:06 smithi081 ceph-mon[31453]: pgmap v4956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:08.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:08 smithi153 ceph-mon[36569]: pgmap v4957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:08.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:08 smithi104 ceph-mon[36624]: pgmap v4957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:08.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:08 smithi081 ceph-mon[31453]: pgmap v4957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:10.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:10 smithi153 ceph-mon[36569]: pgmap v4958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:10.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:10 smithi104 ceph-mon[36624]: pgmap v4958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:10.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:10 smithi081 ceph-mon[31453]: pgmap v4958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:12.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:12 smithi153 ceph-mon[36569]: pgmap v4959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:12.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:12 smithi104 ceph-mon[36624]: pgmap v4959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:12.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:12 smithi081 ceph-mon[31453]: pgmap v4959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:14.123+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:40:14.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:14.123+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:40:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:14.123+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:40:14.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:14.123+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:40:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:14.372+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:40:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:14.372+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:40:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:14.372+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:40:14.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:14 smithi153 ceph-mon[36569]: pgmap v4960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:14.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:14 smithi104 ceph-mon[36624]: pgmap v4960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:14.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:14 smithi081 ceph-mon[31453]: pgmap v4960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:16.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:16 smithi153 ceph-mon[36569]: pgmap v4961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:16.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:16 smithi104 ceph-mon[36624]: pgmap v4961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:16.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:16 smithi081 ceph-mon[31453]: pgmap v4961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:18.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:18 smithi153 ceph-mon[36569]: pgmap v4962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:18.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:18 smithi104 ceph-mon[36624]: pgmap v4962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:18.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:18 smithi081 ceph-mon[31453]: pgmap v4962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:20.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:20 smithi153 ceph-mon[36569]: pgmap v4963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:20.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:20 smithi104 ceph-mon[36624]: pgmap v4963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:20.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:20 smithi081 ceph-mon[31453]: pgmap v4963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:21.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:21 smithi153 ceph-mon[36569]: pgmap v4964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:21.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:21 smithi104 ceph-mon[36624]: pgmap v4964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:21.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:21 smithi081 ceph-mon[31453]: pgmap v4964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:24.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:24 smithi104 ceph-mon[36624]: pgmap v4965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:24.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:24 smithi081 ceph-mon[31453]: pgmap v4965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:24.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:24 smithi153 ceph-mon[36569]: pgmap v4965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:26.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:26 smithi104 ceph-mon[36624]: pgmap v4966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:26.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:26 smithi081 ceph-mon[31453]: pgmap v4966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:26.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:26 smithi153 ceph-mon[36569]: pgmap v4966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:28.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:28 smithi104 ceph-mon[36624]: pgmap v4967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:28 smithi081 ceph-mon[31453]: pgmap v4967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:28.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:28 smithi153 ceph-mon[36569]: pgmap v4967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:29.379+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:40:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:29.379+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:40:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:29.379+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:40:30.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:30 smithi104 ceph-mon[36624]: pgmap v4968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:30.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:30 smithi081 ceph-mon[31453]: pgmap v4968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:30.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:30 smithi153 ceph-mon[36569]: pgmap v4968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:32.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:32 smithi104 ceph-mon[36624]: pgmap v4969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:32.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:32 smithi081 ceph-mon[31453]: pgmap v4969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:32.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:32 smithi153 ceph-mon[36569]: pgmap v4969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:34.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:34 smithi081 ceph-mon[31453]: pgmap v4970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:34.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:34 smithi153 ceph-mon[36569]: pgmap v4970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:34.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:34 smithi104 ceph-mon[36624]: pgmap v4970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:36.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:36 smithi081 ceph-mon[31453]: pgmap v4971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:36.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:36 smithi153 ceph-mon[36569]: pgmap v4971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:36.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:36 smithi104 ceph-mon[36624]: pgmap v4971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:38.343 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:38 smithi153 ceph-mon[36569]: pgmap v4972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:38.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:38 smithi081 ceph-mon[31453]: pgmap v4972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:38.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:38 smithi104 ceph-mon[36624]: pgmap v4972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:40.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:40 smithi081 ceph-mon[31453]: pgmap v4973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:40.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:40 smithi153 ceph-mon[36569]: pgmap v4973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:40.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:40 smithi104 ceph-mon[36624]: pgmap v4973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:42.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:42 smithi081 ceph-mon[31453]: pgmap v4974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:42.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:42 smithi153 ceph-mon[36569]: pgmap v4974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:42.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:42 smithi104 ceph-mon[36624]: pgmap v4974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:44.394+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:40:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:44.394+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:40:44.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:44.394+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:40:44.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:44 smithi081 ceph-mon[31453]: pgmap v4975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:44.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:40:44.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:44 smithi153 ceph-mon[36569]: pgmap v4975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:44.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:40:44.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:44 smithi104 ceph-mon[36624]: pgmap v4975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:44.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:40:45.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:40:45.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:40:45.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:40:45.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:40:45.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:40:45.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:40:45.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:40:45.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:40:45.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:40:46.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:46 smithi081 ceph-mon[31453]: pgmap v4976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:46.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:46 smithi081 ceph-mon[31453]: pgmap v4977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:46.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:46 smithi104 ceph-mon[36624]: pgmap v4976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:46.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:46 smithi104 ceph-mon[36624]: pgmap v4977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:46.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:46 smithi153 ceph-mon[36569]: pgmap v4976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:46.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:46 smithi153 ceph-mon[36569]: pgmap v4977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:48.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:48 smithi081 ceph-mon[31453]: pgmap v4978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:48.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:48 smithi153 ceph-mon[36569]: pgmap v4978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:48.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:48 smithi104 ceph-mon[36624]: pgmap v4978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:50.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:50 smithi081 ceph-mon[31453]: pgmap v4979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:50.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:50 smithi153 ceph-mon[36569]: pgmap v4979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:50.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:50 smithi104 ceph-mon[36624]: pgmap v4979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:52.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:52 smithi081 ceph-mon[31453]: pgmap v4980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:52.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:52 smithi153 ceph-mon[36569]: pgmap v4980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:52.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:52 smithi104 ceph-mon[36624]: pgmap v4980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:54.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:54 smithi081 ceph-mon[31453]: pgmap v4981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:54.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:54 smithi153 ceph-mon[36569]: pgmap v4981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:54.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:54 smithi104 ceph-mon[36624]: pgmap v4981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:56.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:56 smithi081 ceph-mon[31453]: pgmap v4982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:56.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:56 smithi153 ceph-mon[36569]: pgmap v4982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:56.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:56 smithi104 ceph-mon[36624]: pgmap v4982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:58.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:40:58 smithi081 ceph-mon[31453]: pgmap v4983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:58.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:40:58 smithi153 ceph-mon[36569]: pgmap v4983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:58.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:40:58 smithi104 ceph-mon[36624]: pgmap v4983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:40:59.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:59.397+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:40:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:59.398+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:40:59.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:40:59.398+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:41:00.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:00 smithi153 ceph-mon[36569]: pgmap v4984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:00.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:00 smithi104 ceph-mon[36624]: pgmap v4984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:00.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:00 smithi081 ceph-mon[31453]: pgmap v4984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:02.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:02 smithi153 ceph-mon[36569]: pgmap v4985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:02.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:02 smithi104 ceph-mon[36624]: pgmap v4985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:02.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:02 smithi081 ceph-mon[31453]: pgmap v4985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:04.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:04 smithi153 ceph-mon[36569]: pgmap v4986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:04.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:04 smithi104 ceph-mon[36624]: pgmap v4986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:04.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:04 smithi081 ceph-mon[31453]: pgmap v4986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:06.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:06 smithi153 ceph-mon[36569]: pgmap v4987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:06.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:06 smithi104 ceph-mon[36624]: pgmap v4987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:06.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:06 smithi081 ceph-mon[31453]: pgmap v4987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:08.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:08 smithi153 ceph-mon[36569]: pgmap v4988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:08.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:08 smithi104 ceph-mon[36624]: pgmap v4988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:08.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:08 smithi081 ceph-mon[31453]: pgmap v4988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:10.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:10 smithi153 ceph-mon[36569]: pgmap v4989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:10.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:10 smithi104 ceph-mon[36624]: pgmap v4989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:10.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:10 smithi081 ceph-mon[31453]: pgmap v4989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:12.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:12 smithi153 ceph-mon[36569]: pgmap v4990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:12.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:12 smithi104 ceph-mon[36624]: pgmap v4990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:12.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:12 smithi081 ceph-mon[31453]: pgmap v4990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:14.405+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:41:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:14.405+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:41:14.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:14.405+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:41:14.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:14 smithi153 ceph-mon[36569]: pgmap v4991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:14.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:14 smithi104 ceph-mon[36624]: pgmap v4991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:14.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:14 smithi081 ceph-mon[31453]: pgmap v4991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:16.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:16 smithi153 ceph-mon[36569]: pgmap v4992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:16.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:16 smithi104 ceph-mon[36624]: pgmap v4992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:16.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:16 smithi081 ceph-mon[31453]: pgmap v4992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:18.443 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:18 smithi081 ceph-mon[31453]: pgmap v4993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:18.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:18 smithi153 ceph-mon[36569]: pgmap v4993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:18.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:18 smithi104 ceph-mon[36624]: pgmap v4993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:20.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:20 smithi153 ceph-mon[36569]: pgmap v4994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:20.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:20 smithi104 ceph-mon[36624]: pgmap v4994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:20.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:20 smithi081 ceph-mon[31453]: pgmap v4994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:22.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:22 smithi153 ceph-mon[36569]: pgmap v4995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:22.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:22 smithi104 ceph-mon[36624]: pgmap v4995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:22.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:22 smithi081 ceph-mon[31453]: pgmap v4995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:24.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:24 smithi153 ceph-mon[36569]: pgmap v4996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:24.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:24 smithi104 ceph-mon[36624]: pgmap v4996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:24.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:24 smithi081 ceph-mon[31453]: pgmap v4996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:26.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:26 smithi153 ceph-mon[36569]: pgmap v4997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:26.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:26 smithi104 ceph-mon[36624]: pgmap v4997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:26.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:26 smithi081 ceph-mon[31453]: pgmap v4997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:28.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:28 smithi153 ceph-mon[36569]: pgmap v4998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:28.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:28 smithi104 ceph-mon[36624]: pgmap v4998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:28.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:28 smithi081 ceph-mon[31453]: pgmap v4998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:29.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:29.421+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:41:29.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:29.421+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:41:29.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:29.421+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:41:30.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:30 smithi153 ceph-mon[36569]: pgmap v4999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:30.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:30 smithi104 ceph-mon[36624]: pgmap v4999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:30.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:30 smithi081 ceph-mon[31453]: pgmap v4999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:32.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:32 smithi153 ceph-mon[36569]: pgmap v5000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:32.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:32 smithi104 ceph-mon[36624]: pgmap v5000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:32.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:32 smithi081 ceph-mon[31453]: pgmap v5000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:34.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:34 smithi153 ceph-mon[36569]: pgmap v5001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:34.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:34 smithi104 ceph-mon[36624]: pgmap v5001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:34.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:34 smithi081 ceph-mon[31453]: pgmap v5001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:36.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:36 smithi153 ceph-mon[36569]: pgmap v5002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:36.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:36 smithi104 ceph-mon[36624]: pgmap v5002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:36.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:36 smithi081 ceph-mon[31453]: pgmap v5002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:38.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:38 smithi153 ceph-mon[36569]: pgmap v5003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:38.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:38 smithi104 ceph-mon[36624]: pgmap v5003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:38.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:38 smithi081 ceph-mon[31453]: pgmap v5003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:40.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:40 smithi153 ceph-mon[36569]: pgmap v5004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:40.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:40 smithi104 ceph-mon[36624]: pgmap v5004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:40.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:40 smithi081 ceph-mon[31453]: pgmap v5004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:42.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:42 smithi153 ceph-mon[36569]: pgmap v5005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:42.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:42 smithi104 ceph-mon[36624]: pgmap v5005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:42.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:42 smithi081 ceph-mon[31453]: pgmap v5005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:44.436+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:41:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:44.436+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:41:44.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:44.436+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:41:44.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:44 smithi153 ceph-mon[36569]: pgmap v5006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:44.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:44 smithi104 ceph-mon[36624]: pgmap v5006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:44.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:44 smithi081 ceph-mon[31453]: pgmap v5006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:45.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:41:45.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:41:45.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:41:46.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:46 smithi104 ceph-mon[36624]: pgmap v5007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:46.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.468 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:46 smithi081 ceph-mon[31453]: pgmap v5007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:46.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:46 smithi153 ceph-mon[36569]: pgmap v5007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:46.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:46.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:48 smithi153 ceph-mon[36569]: pgmap v5008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:48.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:48 smithi104 ceph-mon[36624]: pgmap v5008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:48.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:48 smithi081 ceph-mon[31453]: pgmap v5008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:48.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:48.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:50.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:50 smithi081 ceph-mon[31453]: pgmap v5009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:50.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:50.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:50.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:50 smithi153 ceph-mon[36569]: pgmap v5009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:50.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:50.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:50.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:50 smithi104 ceph-mon[36624]: pgmap v5009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:50.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:50.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:52.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:52 smithi153 ceph-mon[36569]: pgmap v5010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:41:52.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:41:52.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:52 smithi153 ceph-mon[36569]: pgmap v5011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:52 smithi153 ceph-mon[36569]: pgmap v5012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:52.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:52 smithi104 ceph-mon[36624]: pgmap v5010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:41:52.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:41:52.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:52 smithi104 ceph-mon[36624]: pgmap v5011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:52 smithi104 ceph-mon[36624]: pgmap v5012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:52.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:52 smithi081 ceph-mon[31453]: pgmap v5010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:41:52.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:41:52.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:52 smithi081 ceph-mon[31453]: pgmap v5011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:52 smithi081 ceph-mon[31453]: pgmap v5012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:52.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:41:54.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:54 smithi081 ceph-mon[31453]: pgmap v5013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:54.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:54 smithi153 ceph-mon[36569]: pgmap v5013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:54.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:54 smithi104 ceph-mon[36624]: pgmap v5013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:56.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:56 smithi081 ceph-mon[31453]: pgmap v5014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:56.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:56 smithi153 ceph-mon[36569]: pgmap v5014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:56.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:56 smithi104 ceph-mon[36624]: pgmap v5014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:58.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:41:58 smithi081 ceph-mon[31453]: pgmap v5015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:58.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:41:58 smithi153 ceph-mon[36569]: pgmap v5015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:58.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:41:58 smithi104 ceph-mon[36624]: pgmap v5015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:41:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:59.449+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:41:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:59.449+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:41:59.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:41:59.449+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:42:00.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:00 smithi081 ceph-mon[31453]: pgmap v5016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:00.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:00 smithi153 ceph-mon[36569]: pgmap v5016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:00.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:00 smithi104 ceph-mon[36624]: pgmap v5016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:01.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:01 smithi153 ceph-mon[36569]: pgmap v5017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:01.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:01 smithi104 ceph-mon[36624]: pgmap v5017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:01.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:01 smithi081 ceph-mon[31453]: pgmap v5017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:04.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:04 smithi081 ceph-mon[31453]: pgmap v5018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:04.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:04 smithi153 ceph-mon[36569]: pgmap v5018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:04.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:04 smithi104 ceph-mon[36624]: pgmap v5018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:06.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:06 smithi081 ceph-mon[31453]: pgmap v5019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:06.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:06 smithi153 ceph-mon[36569]: pgmap v5019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:06.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:06 smithi104 ceph-mon[36624]: pgmap v5019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:08.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:08 smithi081 ceph-mon[31453]: pgmap v5020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:08.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:08 smithi153 ceph-mon[36569]: pgmap v5020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:08.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:08 smithi104 ceph-mon[36624]: pgmap v5020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:09.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:09 smithi153 ceph-mon[36569]: pgmap v5021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:09.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:09 smithi104 ceph-mon[36624]: pgmap v5021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:09.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:09 smithi081 ceph-mon[31453]: pgmap v5021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:12.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:12 smithi081 ceph-mon[31453]: pgmap v5022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:12.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:12 smithi153 ceph-mon[36569]: pgmap v5022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:12.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:12 smithi104 ceph-mon[36624]: pgmap v5022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:14.461+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:42:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:14.461+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:42:14.445 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:14.461+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:42:14.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:14 smithi081 ceph-mon[31453]: pgmap v5023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:14.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:14 smithi153 ceph-mon[36569]: pgmap v5023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:14.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:14 smithi104 ceph-mon[36624]: pgmap v5023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:16.618 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:16 smithi104 ceph-mon[36624]: pgmap v5024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:16.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:16 smithi081 ceph-mon[31453]: pgmap v5024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:16.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:16 smithi153 ceph-mon[36569]: pgmap v5024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:17.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:17 smithi081 ceph-mon[31453]: pgmap v5025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:17.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:17 smithi153 ceph-mon[36569]: pgmap v5025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:17.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:17 smithi104 ceph-mon[36624]: pgmap v5025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:20.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:20 smithi081 ceph-mon[31453]: pgmap v5026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:20.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:20 smithi153 ceph-mon[36569]: pgmap v5026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:20.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:20 smithi104 ceph-mon[36624]: pgmap v5026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:22.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:22 smithi081 ceph-mon[31453]: pgmap v5027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:22.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:22 smithi153 ceph-mon[36569]: pgmap v5027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:22.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:22 smithi104 ceph-mon[36624]: pgmap v5027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:23.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:23 smithi153 ceph-mon[36569]: pgmap v5028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:23.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:23 smithi104 ceph-mon[36624]: pgmap v5028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:23.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:23 smithi081 ceph-mon[31453]: pgmap v5028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:26.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:26 smithi081 ceph-mon[31453]: pgmap v5029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:26.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:26 smithi153 ceph-mon[36569]: pgmap v5029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:26.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:26 smithi104 ceph-mon[36624]: pgmap v5029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:28.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:28 smithi081 ceph-mon[31453]: pgmap v5030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:28.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:28 smithi153 ceph-mon[36569]: pgmap v5030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:28.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:28 smithi104 ceph-mon[36624]: pgmap v5030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:29.465+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:42:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:29.465+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:42:29.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:29.465+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:42:29.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:29 smithi081 ceph-mon[31453]: pgmap v5031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:29.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:29 smithi153 ceph-mon[36569]: pgmap v5031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:29.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:29 smithi104 ceph-mon[36624]: pgmap v5031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:32.643 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:32 smithi081 ceph-mon[31453]: pgmap v5032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:32.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:32 smithi153 ceph-mon[36569]: pgmap v5032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:32.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:32 smithi104 ceph-mon[36624]: pgmap v5032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:33.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:33 smithi153 ceph-mon[36569]: pgmap v5033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:33.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:33 smithi104 ceph-mon[36624]: pgmap v5033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:33.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:33 smithi081 ceph-mon[31453]: pgmap v5033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:36.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:36 smithi081 ceph-mon[31453]: pgmap v5034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:36.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:36 smithi153 ceph-mon[36569]: pgmap v5034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:36.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:36 smithi104 ceph-mon[36624]: pgmap v5034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:37.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:37 smithi081 ceph-mon[31453]: pgmap v5035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:37.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:37 smithi153 ceph-mon[36569]: pgmap v5035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:37.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:37 smithi104 ceph-mon[36624]: pgmap v5035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:40.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:40 smithi081 ceph-mon[31453]: pgmap v5036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:40.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:40 smithi153 ceph-mon[36569]: pgmap v5036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:40.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:40 smithi104 ceph-mon[36624]: pgmap v5036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:41.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:41 smithi081 ceph-mon[31453]: pgmap v5037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:41.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:41 smithi153 ceph-mon[36569]: pgmap v5037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:41.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:41 smithi104 ceph-mon[36624]: pgmap v5037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:43.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:43 smithi153 ceph-mon[36569]: pgmap v5038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:43.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:43 smithi104 ceph-mon[36624]: pgmap v5038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:43.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:43 smithi081 ceph-mon[31453]: pgmap v5038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:44.481+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:42:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:44.481+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:42:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:44.481+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:42:45.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:45 smithi153 ceph-mon[36569]: pgmap v5039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:45.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:45 smithi104 ceph-mon[36624]: pgmap v5039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:45.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:45 smithi081 ceph-mon[31453]: pgmap v5039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:47.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:47 smithi153 ceph-mon[36569]: pgmap v5040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:47.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:47 smithi104 ceph-mon[36624]: pgmap v5040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:47.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:47 smithi081 ceph-mon[31453]: pgmap v5040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:49.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:49 smithi153 ceph-mon[36569]: pgmap v5041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:49.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:49 smithi104 ceph-mon[36624]: pgmap v5041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:49.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:49 smithi081 ceph-mon[31453]: pgmap v5041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:51.692 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:51 smithi081 ceph-mon[31453]: pgmap v5042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:51.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:51 smithi153 ceph-mon[36569]: pgmap v5042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:51.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:51 smithi104 ceph-mon[36624]: pgmap v5042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:52.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:42:52.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:52.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:52.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:42:52.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:52.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:52.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:42:52.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:52.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:53.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:53 smithi153 ceph-mon[36569]: pgmap v5043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:53.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:53 smithi104 ceph-mon[36624]: pgmap v5043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:53.834 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:53 smithi081 ceph-mon[31453]: pgmap v5043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:55.492 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:55 smithi081 ceph-mon[31453]: pgmap v5044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:55.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:55 smithi153 ceph-mon[36569]: pgmap v5044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:55.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:55 smithi104 ceph-mon[36624]: pgmap v5044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:57.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:57.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:57.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:57.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:57.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:57.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:42:58.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:58 smithi153 ceph-mon[36569]: pgmap v5045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:58.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:58 smithi104 ceph-mon[36624]: pgmap v5045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:58.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:58 smithi081 ceph-mon[31453]: pgmap v5045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:42:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:59.496+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:42:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:59.496+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:42:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:42:59.496+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:00.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:00.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:00.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:43:00.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:43:00.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:59 smithi104 ceph-mon[36624]: pgmap v5046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:00.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:59 smithi104 ceph-mon[36624]: pgmap v5047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:00.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:00.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:00.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:00.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:43:00.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:43:00.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:59 smithi081 ceph-mon[31453]: pgmap v5046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:00.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:59 smithi081 ceph-mon[31453]: pgmap v5047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:00.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:00.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:00.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:00.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:43:00.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:43:00.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:59 smithi153 ceph-mon[36569]: pgmap v5046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:00.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:59 smithi153 ceph-mon[36569]: pgmap v5047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:00.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:43:02.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:01 smithi153 ceph-mon[36569]: pgmap v5048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:02.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:01 smithi104 ceph-mon[36624]: pgmap v5048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:02.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:01 smithi081 ceph-mon[31453]: pgmap v5048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:04.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:03 smithi153 ceph-mon[36569]: pgmap v5049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:04.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:03 smithi104 ceph-mon[36624]: pgmap v5049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:04.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:03 smithi081 ceph-mon[31453]: pgmap v5049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:06.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:05 smithi153 ceph-mon[36569]: pgmap v5050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:06.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:05 smithi104 ceph-mon[36624]: pgmap v5050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:06.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:05 smithi081 ceph-mon[31453]: pgmap v5050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:08.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:07 smithi153 ceph-mon[36569]: pgmap v5051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:08.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:07 smithi104 ceph-mon[36624]: pgmap v5051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:08.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:07 smithi081 ceph-mon[31453]: pgmap v5051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:10.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:09 smithi153 ceph-mon[36569]: pgmap v5052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:10.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:09 smithi104 ceph-mon[36624]: pgmap v5052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:10.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:09 smithi081 ceph-mon[31453]: pgmap v5052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:12.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:11 smithi153 ceph-mon[36569]: pgmap v5053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:12.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:11 smithi104 ceph-mon[36624]: pgmap v5053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:12.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:11 smithi081 ceph-mon[31453]: pgmap v5053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:14.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:14.154+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:43:14.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:14.154+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:43:14.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:14.155+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:43:14.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:14.155+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:43:14.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:13 smithi153 ceph-mon[36569]: pgmap v5054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:14.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:13 smithi104 ceph-mon[36624]: pgmap v5054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:14.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:13 smithi081 ceph-mon[31453]: pgmap v5054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:14.490 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:14.507+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:14.507+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:14.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:14.507+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:43:16.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:15 smithi153 ceph-mon[36569]: pgmap v5055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:16.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:15 smithi104 ceph-mon[36624]: pgmap v5055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:16.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:15 smithi081 ceph-mon[31453]: pgmap v5055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:18.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:17 smithi153 ceph-mon[36569]: pgmap v5056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:18.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:17 smithi104 ceph-mon[36624]: pgmap v5056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:18.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:17 smithi081 ceph-mon[31453]: pgmap v5056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:20.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:19 smithi104 ceph-mon[36624]: pgmap v5057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:20.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:19 smithi153 ceph-mon[36569]: pgmap v5057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:20.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:19 smithi081 ceph-mon[31453]: pgmap v5057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:22.222 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:21 smithi081 ceph-mon[31453]: pgmap v5058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:22.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:21 smithi153 ceph-mon[36569]: pgmap v5058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:22.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:21 smithi104 ceph-mon[36624]: pgmap v5058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:24.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:23 smithi153 ceph-mon[36569]: pgmap v5059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:24.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:23 smithi104 ceph-mon[36624]: pgmap v5059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:24.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:23 smithi081 ceph-mon[31453]: pgmap v5059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:26.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:25 smithi153 ceph-mon[36569]: pgmap v5060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:26.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:25 smithi104 ceph-mon[36624]: pgmap v5060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:26.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:25 smithi081 ceph-mon[31453]: pgmap v5060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:28.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:27 smithi153 ceph-mon[36569]: pgmap v5061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:28.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:27 smithi104 ceph-mon[36624]: pgmap v5061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:28.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:27 smithi081 ceph-mon[31453]: pgmap v5061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:29.522+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:29.522+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:43:29.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:29.522+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:30.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:29 smithi153 ceph-mon[36569]: pgmap v5062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:30.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:29 smithi104 ceph-mon[36624]: pgmap v5062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:30.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:29 smithi081 ceph-mon[31453]: pgmap v5062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:32.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:31 smithi153 ceph-mon[36569]: pgmap v5063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:32.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:31 smithi104 ceph-mon[36624]: pgmap v5063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:32.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:31 smithi081 ceph-mon[31453]: pgmap v5063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:34.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:33 smithi153 ceph-mon[36569]: pgmap v5064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:34.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:33 smithi104 ceph-mon[36624]: pgmap v5064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:34.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:33 smithi081 ceph-mon[31453]: pgmap v5064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:36.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:36 smithi153 ceph-mon[36569]: pgmap v5065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:36.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:36 smithi104 ceph-mon[36624]: pgmap v5065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:36.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:36 smithi081 ceph-mon[31453]: pgmap v5065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:38.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:38 smithi153 ceph-mon[36569]: pgmap v5066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:38.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:38 smithi104 ceph-mon[36624]: pgmap v5066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:38.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:38 smithi081 ceph-mon[31453]: pgmap v5066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:40.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:40 smithi153 ceph-mon[36569]: pgmap v5067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:40.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:40 smithi104 ceph-mon[36624]: pgmap v5067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:40.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:40 smithi081 ceph-mon[31453]: pgmap v5067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:42.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:42 smithi153 ceph-mon[36569]: pgmap v5068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:42.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:42 smithi104 ceph-mon[36624]: pgmap v5068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:42.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:42 smithi081 ceph-mon[31453]: pgmap v5068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:44.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:44 smithi153 ceph-mon[36569]: pgmap v5069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:44.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:44 smithi104 ceph-mon[36624]: pgmap v5069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:44.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:44 smithi081 ceph-mon[31453]: pgmap v5069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:44.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:44.524+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:44.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:44.524+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:44.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:44.524+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:43:46.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:46 smithi153 ceph-mon[36569]: pgmap v5070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:46.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:46 smithi104 ceph-mon[36624]: pgmap v5070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:46.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:46 smithi081 ceph-mon[31453]: pgmap v5070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:48.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:48 smithi153 ceph-mon[36569]: pgmap v5071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:48.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:48 smithi104 ceph-mon[36624]: pgmap v5071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:48.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:48 smithi081 ceph-mon[31453]: pgmap v5071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:50.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:50 smithi153 ceph-mon[36569]: pgmap v5072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:50.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:50 smithi104 ceph-mon[36624]: pgmap v5072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:50.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:50 smithi081 ceph-mon[31453]: pgmap v5072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:52.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:52 smithi153 ceph-mon[36569]: pgmap v5073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:52.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:52 smithi104 ceph-mon[36624]: pgmap v5073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:52.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:52 smithi081 ceph-mon[31453]: pgmap v5073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:54.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:54 smithi081 ceph-mon[31453]: pgmap v5074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:54.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:54 smithi153 ceph-mon[36569]: pgmap v5074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:54.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:54 smithi104 ceph-mon[36624]: pgmap v5074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:56.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:56 smithi153 ceph-mon[36569]: pgmap v5075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:56.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:56 smithi104 ceph-mon[36624]: pgmap v5075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:56.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:56 smithi081 ceph-mon[31453]: pgmap v5075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:58.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:43:58 smithi104 ceph-mon[36624]: pgmap v5076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:58.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:43:58 smithi081 ceph-mon[31453]: pgmap v5076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:58.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:43:58 smithi153 ceph-mon[36569]: pgmap v5076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:43:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:59.526+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:59.527+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:43:59.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:43:59.527+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:44:00.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:00 smithi104 ceph-mon[36624]: pgmap v5077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:00.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:44:00.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:00 smithi081 ceph-mon[31453]: pgmap v5077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:00.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:44:00.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:00 smithi153 ceph-mon[36569]: pgmap v5077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:00.456 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:44:02.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:02 smithi104 ceph-mon[36624]: pgmap v5078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:02.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:02 smithi081 ceph-mon[31453]: pgmap v5078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:02.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:02 smithi153 ceph-mon[36569]: pgmap v5078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:04.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:04 smithi104 ceph-mon[36624]: pgmap v5079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:04.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:04.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:04.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:44:04.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:44:04.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:04.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:04 smithi081 ceph-mon[31453]: pgmap v5079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:04.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:04.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:04.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:44:04.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:44:04.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:04.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:04 smithi153 ceph-mon[36569]: pgmap v5079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:04.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:04.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:04.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:44:04.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:44:04.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:44:05.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:05 smithi153 ceph-mon[36569]: pgmap v5080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:05.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:05 smithi153 ceph-mon[36569]: pgmap v5081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:05.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:05 smithi104 ceph-mon[36624]: pgmap v5080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:05.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:05 smithi104 ceph-mon[36624]: pgmap v5081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:05.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:05 smithi081 ceph-mon[31453]: pgmap v5080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:05.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:05 smithi081 ceph-mon[31453]: pgmap v5081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:07.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:07 smithi104 ceph-mon[36624]: pgmap v5082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:07.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:07 smithi081 ceph-mon[31453]: pgmap v5082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:07.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:07 smithi153 ceph-mon[36569]: pgmap v5082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:09.335 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:09 smithi081 ceph-mon[31453]: pgmap v5083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:09.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:09 smithi153 ceph-mon[36569]: pgmap v5083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:09.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:09 smithi104 ceph-mon[36624]: pgmap v5083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:11.342 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:11 smithi153 ceph-mon[36569]: pgmap v5084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:11.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:11 smithi081 ceph-mon[31453]: pgmap v5084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:11.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:11 smithi104 ceph-mon[36624]: pgmap v5084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:13.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:13 smithi081 ceph-mon[31453]: pgmap v5085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:13.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:13 smithi153 ceph-mon[36569]: pgmap v5085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:13.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:13 smithi104 ceph-mon[36624]: pgmap v5085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:14.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:14.528+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:44:14.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:14.528+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:44:14.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:14.528+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:44:15.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:15 smithi081 ceph-mon[31453]: pgmap v5086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:15.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:15 smithi153 ceph-mon[36569]: pgmap v5086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:15.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:15 smithi104 ceph-mon[36624]: pgmap v5086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:17.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:17 smithi081 ceph-mon[31453]: pgmap v5087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:17.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:17 smithi153 ceph-mon[36569]: pgmap v5087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:17.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:17 smithi104 ceph-mon[36624]: pgmap v5087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:19.367 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:19 smithi104 ceph-mon[36624]: pgmap v5088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:19.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:19 smithi081 ceph-mon[31453]: pgmap v5088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:19.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:19 smithi153 ceph-mon[36569]: pgmap v5088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:21.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:21 smithi081 ceph-mon[31453]: pgmap v5089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:21.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:21 smithi153 ceph-mon[36569]: pgmap v5089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:21.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:21 smithi104 ceph-mon[36624]: pgmap v5089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:23.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:23 smithi081 ceph-mon[31453]: pgmap v5090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:23.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:23 smithi153 ceph-mon[36569]: pgmap v5090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:23.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:23 smithi104 ceph-mon[36624]: pgmap v5090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:25.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:25 smithi081 ceph-mon[31453]: pgmap v5091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:25.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:25 smithi153 ceph-mon[36569]: pgmap v5091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:25.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:25 smithi104 ceph-mon[36624]: pgmap v5091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:27.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:27 smithi081 ceph-mon[31453]: pgmap v5092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:27.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:27 smithi153 ceph-mon[36569]: pgmap v5092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:27.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:27 smithi104 ceph-mon[36624]: pgmap v5092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:29.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:29 smithi081 ceph-mon[31453]: pgmap v5093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:29.537+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:44:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:29.537+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:44:29.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:29.537+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:44:29.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:29 smithi153 ceph-mon[36569]: pgmap v5093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:29.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:29 smithi104 ceph-mon[36624]: pgmap v5093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:31.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:31 smithi081 ceph-mon[31453]: pgmap v5094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:31.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:31 smithi153 ceph-mon[36569]: pgmap v5094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:31.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:31 smithi104 ceph-mon[36624]: pgmap v5094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:33.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:33 smithi081 ceph-mon[31453]: pgmap v5095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:33.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:33 smithi153 ceph-mon[36569]: pgmap v5095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:33.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:33 smithi104 ceph-mon[36624]: pgmap v5095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:35.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:35 smithi081 ceph-mon[31453]: pgmap v5096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:35.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:35 smithi153 ceph-mon[36569]: pgmap v5096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:35.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:35 smithi104 ceph-mon[36624]: pgmap v5096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:37.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:37 smithi153 ceph-mon[36569]: pgmap v5097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:37.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:37 smithi104 ceph-mon[36624]: pgmap v5097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:37.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:37 smithi081 ceph-mon[31453]: pgmap v5097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:39.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:39 smithi153 ceph-mon[36569]: pgmap v5098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:39.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:39 smithi104 ceph-mon[36624]: pgmap v5098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:39.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:39 smithi081 ceph-mon[31453]: pgmap v5098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:41.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:41 smithi081 ceph-mon[31453]: pgmap v5099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:41.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:41 smithi153 ceph-mon[36569]: pgmap v5099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:41.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:41 smithi104 ceph-mon[36624]: pgmap v5099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:43.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:43 smithi153 ceph-mon[36569]: pgmap v5100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:43.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:43 smithi104 ceph-mon[36624]: pgmap v5100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:43.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:43 smithi081 ceph-mon[31453]: pgmap v5100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:44.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:44.169+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:44:44.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:44.169+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:44:44.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:44.170+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:44:44.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:44.170+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:44:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:44.553+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:44:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:44.553+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:44:44.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:44.553+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:44:45.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:45 smithi153 ceph-mon[36569]: pgmap v5101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:45.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:45 smithi104 ceph-mon[36624]: pgmap v5101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:45.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:45 smithi081 ceph-mon[31453]: pgmap v5101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:47.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:47 smithi153 ceph-mon[36569]: pgmap v5102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:47.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:47 smithi104 ceph-mon[36624]: pgmap v5102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:47.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:47 smithi081 ceph-mon[31453]: pgmap v5102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:49.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:49 smithi153 ceph-mon[36569]: pgmap v5103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:49.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:49 smithi104 ceph-mon[36624]: pgmap v5103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:49.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:49 smithi081 ceph-mon[31453]: pgmap v5103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:51.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:51 smithi153 ceph-mon[36569]: pgmap v5104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:51.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:51 smithi104 ceph-mon[36624]: pgmap v5104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:51.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:51 smithi081 ceph-mon[31453]: pgmap v5104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:53.442 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:53 smithi081 ceph-mon[31453]: pgmap v5105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:53.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:53 smithi153 ceph-mon[36569]: pgmap v5105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:53.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:53 smithi104 ceph-mon[36624]: pgmap v5105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:55.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:55 smithi153 ceph-mon[36569]: pgmap v5106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:55.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:55 smithi104 ceph-mon[36624]: pgmap v5106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:55.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:55 smithi081 ceph-mon[31453]: pgmap v5106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:57.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:57 smithi081 ceph-mon[31453]: pgmap v5107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:57.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:57 smithi153 ceph-mon[36569]: pgmap v5107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:57.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:57 smithi104 ceph-mon[36624]: pgmap v5107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:44:59 smithi153 ceph-mon[36569]: pgmap v5108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:59.568+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:44:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:59.568+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:44:59.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:44:59.568+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:44:59.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:44:59 smithi104 ceph-mon[36624]: pgmap v5108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:44:59.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:44:59 smithi081 ceph-mon[31453]: pgmap v5108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:01.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:01 smithi153 ceph-mon[36569]: pgmap v5109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:01.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:01 smithi104 ceph-mon[36624]: pgmap v5109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:01.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:01 smithi081 ceph-mon[31453]: pgmap v5109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:03.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:03 smithi153 ceph-mon[36569]: pgmap v5110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:03.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:03 smithi104 ceph-mon[36624]: pgmap v5110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:03.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:03 smithi081 ceph-mon[31453]: pgmap v5110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:04.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:45:04.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:45:04.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:45:05.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:05 smithi153 ceph-mon[36569]: pgmap v5111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:05.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:05 smithi104 ceph-mon[36624]: pgmap v5111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:05.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:05 smithi081 ceph-mon[31453]: pgmap v5111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:06.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:45:06.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:45:06.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:45:06.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:45:06.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:45:06.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:45:06.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:45:06.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:45:06.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:45:07.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:07 smithi153 ceph-mon[36569]: pgmap v5112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:07.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:07 smithi153 ceph-mon[36569]: pgmap v5113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:07.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:07 smithi104 ceph-mon[36624]: pgmap v5112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:07.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:07 smithi104 ceph-mon[36624]: pgmap v5113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:07.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:07 smithi081 ceph-mon[31453]: pgmap v5112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:07.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:07 smithi081 ceph-mon[31453]: pgmap v5113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:09.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:09 smithi153 ceph-mon[36569]: pgmap v5114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:09.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:09 smithi104 ceph-mon[36624]: pgmap v5114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:09.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:09 smithi081 ceph-mon[31453]: pgmap v5114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:11.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:11 smithi153 ceph-mon[36569]: pgmap v5115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:11.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:11 smithi104 ceph-mon[36624]: pgmap v5115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:11.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:11 smithi081 ceph-mon[31453]: pgmap v5115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:13.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:13 smithi153 ceph-mon[36569]: pgmap v5116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:13.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:13 smithi104 ceph-mon[36624]: pgmap v5116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:13.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:13 smithi081 ceph-mon[31453]: pgmap v5116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:14.583+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:45:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:14.583+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:45:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:14.583+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:45:15.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:15 smithi153 ceph-mon[36569]: pgmap v5117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:15.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:15 smithi104 ceph-mon[36624]: pgmap v5117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:15.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:15 smithi081 ceph-mon[31453]: pgmap v5117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:17.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:17 smithi153 ceph-mon[36569]: pgmap v5118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:17.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:17 smithi104 ceph-mon[36624]: pgmap v5118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:17.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:17 smithi081 ceph-mon[31453]: pgmap v5118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:19.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:19 smithi153 ceph-mon[36569]: pgmap v5119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:19.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:19 smithi104 ceph-mon[36624]: pgmap v5119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:19.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:19 smithi081 ceph-mon[31453]: pgmap v5119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:21.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:21 smithi153 ceph-mon[36569]: pgmap v5120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:21.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:21 smithi104 ceph-mon[36624]: pgmap v5120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:21.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:21 smithi081 ceph-mon[31453]: pgmap v5120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:23.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:23 smithi153 ceph-mon[36569]: pgmap v5121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:23.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:23 smithi104 ceph-mon[36624]: pgmap v5121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:23.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:23 smithi081 ceph-mon[31453]: pgmap v5121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:25.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:25 smithi153 ceph-mon[36569]: pgmap v5122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:25.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:25 smithi104 ceph-mon[36624]: pgmap v5122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:25.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:25 smithi081 ceph-mon[31453]: pgmap v5122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:27.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:27 smithi153 ceph-mon[36569]: pgmap v5123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:27.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:27 smithi104 ceph-mon[36624]: pgmap v5123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:27.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:27 smithi081 ceph-mon[31453]: pgmap v5123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:29.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:29 smithi104 ceph-mon[36624]: pgmap v5124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:29.593+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:45:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:29.593+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:45:29.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:29.593+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:45:29.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:29 smithi081 ceph-mon[31453]: pgmap v5124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:29.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:29 smithi153 ceph-mon[36569]: pgmap v5124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:31.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:31 smithi104 ceph-mon[36624]: pgmap v5125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:31.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:31 smithi081 ceph-mon[31453]: pgmap v5125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:31.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:31 smithi153 ceph-mon[36569]: pgmap v5125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:33.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:33 smithi104 ceph-mon[36624]: pgmap v5126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:33.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:33 smithi081 ceph-mon[31453]: pgmap v5126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:33.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:33 smithi153 ceph-mon[36569]: pgmap v5126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:35.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:35 smithi104 ceph-mon[36624]: pgmap v5127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:35.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:35 smithi081 ceph-mon[31453]: pgmap v5127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:35.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:35 smithi153 ceph-mon[36569]: pgmap v5127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:37.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:37 smithi104 ceph-mon[36624]: pgmap v5128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:37.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:37 smithi081 ceph-mon[31453]: pgmap v5128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:37.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:37 smithi153 ceph-mon[36569]: pgmap v5128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:39.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:39 smithi081 ceph-mon[31453]: pgmap v5129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:39.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:39 smithi153 ceph-mon[36569]: pgmap v5129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:39.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:39 smithi104 ceph-mon[36624]: pgmap v5129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:41.591 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:41 smithi153 ceph-mon[36569]: pgmap v5130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:41.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:41 smithi081 ceph-mon[31453]: pgmap v5130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:41.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:41 smithi104 ceph-mon[36624]: pgmap v5130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:43.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:43 smithi081 ceph-mon[31453]: pgmap v5131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:43.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:43 smithi153 ceph-mon[36569]: pgmap v5131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:43.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:43 smithi104 ceph-mon[36624]: pgmap v5131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:44.599+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:45:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:44.599+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:45:44.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:44.599+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:45:45.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:45 smithi081 ceph-mon[31453]: pgmap v5132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:45.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:45 smithi153 ceph-mon[36569]: pgmap v5132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:45.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:45 smithi104 ceph-mon[36624]: pgmap v5132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:47.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:47 smithi081 ceph-mon[31453]: pgmap v5133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:47.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:47 smithi153 ceph-mon[36569]: pgmap v5133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:47.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:47 smithi104 ceph-mon[36624]: pgmap v5133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:49.617 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:49 smithi104 ceph-mon[36624]: pgmap v5134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:49.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:49 smithi081 ceph-mon[31453]: pgmap v5134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:49.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:49 smithi153 ceph-mon[36569]: pgmap v5134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:51.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:51 smithi081 ceph-mon[31453]: pgmap v5135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:51.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:51 smithi153 ceph-mon[36569]: pgmap v5135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:51.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:51 smithi104 ceph-mon[36624]: pgmap v5135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:53.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:53 smithi081 ceph-mon[31453]: pgmap v5136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:53.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:53 smithi153 ceph-mon[36569]: pgmap v5136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:53.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:53 smithi104 ceph-mon[36624]: pgmap v5136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:55.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:55 smithi081 ceph-mon[31453]: pgmap v5137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:55.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:55 smithi153 ceph-mon[36569]: pgmap v5137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:55.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:55 smithi104 ceph-mon[36624]: pgmap v5137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:57.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:57 smithi081 ceph-mon[31453]: pgmap v5138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:57.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:57 smithi153 ceph-mon[36569]: pgmap v5138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:57.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:57 smithi104 ceph-mon[36624]: pgmap v5138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:59.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:59.611+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:45:59.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:59.611+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:45:59.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:45:59.611+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:45:59.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:45:59 smithi081 ceph-mon[31453]: pgmap v5139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:59.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:45:59 smithi153 ceph-mon[36569]: pgmap v5139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:45:59.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:45:59 smithi104 ceph-mon[36624]: pgmap v5139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:01.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:01 smithi081 ceph-mon[31453]: pgmap v5140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:01.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:01 smithi153 ceph-mon[36569]: pgmap v5140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:01.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:01 smithi104 ceph-mon[36624]: pgmap v5140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:03.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:03 smithi081 ceph-mon[31453]: pgmap v5141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:03.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:03 smithi153 ceph-mon[36569]: pgmap v5141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:03.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:03 smithi104 ceph-mon[36624]: pgmap v5141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:05.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:05 smithi081 ceph-mon[31453]: pgmap v5142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:05.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:05 smithi153 ceph-mon[36569]: pgmap v5142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:05.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:05 smithi104 ceph-mon[36624]: pgmap v5142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:06.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:46:06.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:46:06.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:46:07.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:07 smithi153 ceph-mon[36569]: pgmap v5143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:46:07.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:46:07.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:07 smithi153 ceph-mon[36569]: pgmap v5144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:07 smithi153 ceph-mon[36569]: pgmap v5145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:46:07.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:07 smithi104 ceph-mon[36624]: pgmap v5143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:46:07.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:46:07.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:07 smithi104 ceph-mon[36624]: pgmap v5144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:07 smithi104 ceph-mon[36624]: pgmap v5145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:46:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:07 smithi081 ceph-mon[31453]: pgmap v5143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:46:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:46:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:07 smithi081 ceph-mon[31453]: pgmap v5144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:07 smithi081 ceph-mon[31453]: pgmap v5145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:46:10.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:10 smithi153 ceph-mon[36569]: pgmap v5146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:10.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:10 smithi104 ceph-mon[36624]: pgmap v5146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:10.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:10 smithi081 ceph-mon[31453]: pgmap v5146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:12.112+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000693) for more than 900000000 us, fault. 2024-04-12T23:46:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:12.112+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-12T23:46:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:12.112+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-12T23:46:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:12.112+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-12T23:46:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:12.112+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-12T23:46:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:12.113+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-12T23:46:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:12.114+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f16f40d42d0 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-12T23:46:12.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:12 smithi153 ceph-mon[36569]: pgmap v5147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:12.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:12 smithi104 ceph-mon[36624]: pgmap v5147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:12.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:12 smithi081 ceph-mon[31453]: pgmap v5147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:14.183+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:46:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:14.183+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:46:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:14.184+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:46:14.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:14.184+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:46:14.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:14 smithi153 ceph-mon[36569]: pgmap v5148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:14.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:14 smithi104 ceph-mon[36624]: pgmap v5148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:14.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:14 smithi081 ceph-mon[31453]: pgmap v5148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:14.626+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:46:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:14.626+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:46:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:14.626+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:46:16.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:16 smithi153 ceph-mon[36569]: pgmap v5149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:16.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:16 smithi104 ceph-mon[36624]: pgmap v5149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:16.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:16 smithi081 ceph-mon[31453]: pgmap v5149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:18.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:18 smithi153 ceph-mon[36569]: pgmap v5150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:18.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:18 smithi104 ceph-mon[36624]: pgmap v5150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:18.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:18 smithi081 ceph-mon[31453]: pgmap v5150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:20.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:20 smithi153 ceph-mon[36569]: pgmap v5151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:20.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:20 smithi104 ceph-mon[36624]: pgmap v5151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:20.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:20 smithi081 ceph-mon[31453]: pgmap v5151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:22.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:22 smithi153 ceph-mon[36569]: pgmap v5152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:22.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:22 smithi104 ceph-mon[36624]: pgmap v5152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:22.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:22 smithi081 ceph-mon[31453]: pgmap v5152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:24.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:24 smithi104 ceph-mon[36624]: pgmap v5153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:24.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:24 smithi081 ceph-mon[31453]: pgmap v5153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:24.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:24 smithi153 ceph-mon[36569]: pgmap v5153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:26.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:26 smithi104 ceph-mon[36624]: pgmap v5154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:26.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:26 smithi081 ceph-mon[31453]: pgmap v5154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:26.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:26 smithi153 ceph-mon[36569]: pgmap v5154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:28.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:28 smithi104 ceph-mon[36624]: pgmap v5155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:28.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:28 smithi081 ceph-mon[31453]: pgmap v5155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:28.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:28 smithi153 ceph-mon[36569]: pgmap v5155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:29.629+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:46:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:29.629+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:46:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:29.629+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:46:30.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:30 smithi104 ceph-mon[36624]: pgmap v5156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:30.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:30 smithi081 ceph-mon[31453]: pgmap v5156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:30.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:30 smithi153 ceph-mon[36569]: pgmap v5156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:32.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:32 smithi104 ceph-mon[36624]: pgmap v5157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:32.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:32 smithi081 ceph-mon[31453]: pgmap v5157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:32.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:32 smithi153 ceph-mon[36569]: pgmap v5157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:34.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:34 smithi081 ceph-mon[31453]: pgmap v5158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:34.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:34 smithi153 ceph-mon[36569]: pgmap v5158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:34.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:34 smithi104 ceph-mon[36624]: pgmap v5158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:36.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:36 smithi081 ceph-mon[31453]: pgmap v5159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:36.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:36 smithi153 ceph-mon[36569]: pgmap v5159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:36.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:36 smithi104 ceph-mon[36624]: pgmap v5159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:38.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:38 smithi081 ceph-mon[31453]: pgmap v5160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:38.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:38 smithi153 ceph-mon[36569]: pgmap v5160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:38.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:38 smithi104 ceph-mon[36624]: pgmap v5160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:40.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:40 smithi081 ceph-mon[31453]: pgmap v5161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:40.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:40 smithi153 ceph-mon[36569]: pgmap v5161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:40.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:40 smithi104 ceph-mon[36624]: pgmap v5161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:42.341 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:42 smithi153 ceph-mon[36569]: pgmap v5162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:42.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:42 smithi081 ceph-mon[31453]: pgmap v5162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:42.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:42 smithi104 ceph-mon[36624]: pgmap v5162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:44.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:44 smithi081 ceph-mon[31453]: pgmap v5163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:44.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:44 smithi153 ceph-mon[36569]: pgmap v5163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:44.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:44 smithi104 ceph-mon[36624]: pgmap v5163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:44.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:44.645+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:46:44.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:44.645+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:46:44.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:44.645+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:46:46.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:46 smithi081 ceph-mon[31453]: pgmap v5164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:46.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:46 smithi153 ceph-mon[36569]: pgmap v5164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:46.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:46 smithi104 ceph-mon[36624]: pgmap v5164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:48.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:48 smithi081 ceph-mon[31453]: pgmap v5165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:48.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:48 smithi153 ceph-mon[36569]: pgmap v5165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:48.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:48 smithi104 ceph-mon[36624]: pgmap v5165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:50.366 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:50 smithi104 ceph-mon[36624]: pgmap v5166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:50.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:50 smithi081 ceph-mon[31453]: pgmap v5166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:50.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:50 smithi153 ceph-mon[36569]: pgmap v5166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:52.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:52 smithi081 ceph-mon[31453]: pgmap v5167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:52.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:52 smithi153 ceph-mon[36569]: pgmap v5167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:52.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:52 smithi104 ceph-mon[36624]: pgmap v5167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:54.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:54 smithi081 ceph-mon[31453]: pgmap v5168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:54.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:54 smithi153 ceph-mon[36569]: pgmap v5168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:54.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:54 smithi104 ceph-mon[36624]: pgmap v5168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:56.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:56 smithi081 ceph-mon[31453]: pgmap v5169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:56.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:56 smithi153 ceph-mon[36569]: pgmap v5169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:56.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:56 smithi104 ceph-mon[36624]: pgmap v5169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:58.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:46:58 smithi081 ceph-mon[31453]: pgmap v5170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:58.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:46:58 smithi153 ceph-mon[36569]: pgmap v5170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:58.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:46:58 smithi104 ceph-mon[36624]: pgmap v5170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:46:59.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:59.659+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:46:59.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:59.659+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:46:59.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:46:59.659+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:47:00.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:00 smithi081 ceph-mon[31453]: pgmap v5171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:00.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:00 smithi153 ceph-mon[36569]: pgmap v5171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:00 smithi104 ceph-mon[36624]: pgmap v5171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:02.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:02 smithi153 ceph-mon[36569]: pgmap v5172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:02.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:02 smithi104 ceph-mon[36624]: pgmap v5172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:02.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:02 smithi081 ceph-mon[31453]: pgmap v5172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:04.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:04 smithi153 ceph-mon[36569]: pgmap v5173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:04.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:04 smithi104 ceph-mon[36624]: pgmap v5173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:04.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:04 smithi081 ceph-mon[31453]: pgmap v5173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:06.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:06 smithi153 ceph-mon[36569]: pgmap v5174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:06.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:06 smithi104 ceph-mon[36624]: pgmap v5174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:06.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:06 smithi081 ceph-mon[31453]: pgmap v5174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:08.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:08 smithi153 ceph-mon[36569]: pgmap v5175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:08.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:47:08.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:08 smithi104 ceph-mon[36624]: pgmap v5175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:08.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:47:08.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:08 smithi081 ceph-mon[31453]: pgmap v5175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:08.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:47:09.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:47:09.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:47:09.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:47:09.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:47:09.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:47:09.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:47:09.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:47:09.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:47:09.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:47:10.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:10 smithi153 ceph-mon[36569]: pgmap v5176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:10.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:10 smithi153 ceph-mon[36569]: pgmap v5177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:10.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:10 smithi104 ceph-mon[36624]: pgmap v5176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:10.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:10 smithi104 ceph-mon[36624]: pgmap v5177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:10.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:10 smithi081 ceph-mon[31453]: pgmap v5176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:10.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:10 smithi081 ceph-mon[31453]: pgmap v5177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:12.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:12 smithi153 ceph-mon[36569]: pgmap v5178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:12.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:12 smithi104 ceph-mon[36624]: pgmap v5178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:12.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:12 smithi081 ceph-mon[31453]: pgmap v5178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:14.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:14 smithi153 ceph-mon[36569]: pgmap v5179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:14.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:14 smithi104 ceph-mon[36624]: pgmap v5179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:14.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:14 smithi081 ceph-mon[31453]: pgmap v5179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:14.675+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:47:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:14.675+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:47:14.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:14.675+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:47:16.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:16 smithi153 ceph-mon[36569]: pgmap v5180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:16.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:16 smithi104 ceph-mon[36624]: pgmap v5180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:16.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:16 smithi081 ceph-mon[31453]: pgmap v5180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:18.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:18 smithi153 ceph-mon[36569]: pgmap v5181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:18.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:18 smithi104 ceph-mon[36624]: pgmap v5181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:18.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:18 smithi081 ceph-mon[31453]: pgmap v5181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:20.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:20 smithi153 ceph-mon[36569]: pgmap v5182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:20.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:20 smithi104 ceph-mon[36624]: pgmap v5182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:20.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:20 smithi081 ceph-mon[31453]: pgmap v5182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:22.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:22 smithi153 ceph-mon[36569]: pgmap v5183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:22.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:22 smithi104 ceph-mon[36624]: pgmap v5183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:22.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:22 smithi081 ceph-mon[31453]: pgmap v5183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:24.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:24 smithi153 ceph-mon[36569]: pgmap v5184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:24.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:24 smithi104 ceph-mon[36624]: pgmap v5184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:24.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:24 smithi081 ceph-mon[31453]: pgmap v5184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:26.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:26 smithi153 ceph-mon[36569]: pgmap v5185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:26.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:26 smithi104 ceph-mon[36624]: pgmap v5185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:26.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:26 smithi081 ceph-mon[31453]: pgmap v5185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:28.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:28 smithi153 ceph-mon[36569]: pgmap v5186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:28.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:28 smithi104 ceph-mon[36624]: pgmap v5186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:28.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:28 smithi081 ceph-mon[31453]: pgmap v5186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:29.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:29.688+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:47:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:29.688+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:47:29.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:29.688+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:47:30.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:30 smithi153 ceph-mon[36569]: pgmap v5187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:30.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:30 smithi104 ceph-mon[36624]: pgmap v5187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:30.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:30 smithi081 ceph-mon[31453]: pgmap v5187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:32.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:32 smithi081 ceph-mon[31453]: pgmap v5188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:32.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:32 smithi153 ceph-mon[36569]: pgmap v5188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:32.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:32 smithi104 ceph-mon[36624]: pgmap v5188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:34.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:34 smithi153 ceph-mon[36569]: pgmap v5189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:34.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:34 smithi104 ceph-mon[36624]: pgmap v5189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:34.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:34 smithi081 ceph-mon[31453]: pgmap v5189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:36.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:36 smithi153 ceph-mon[36569]: pgmap v5190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:36.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:36 smithi104 ceph-mon[36624]: pgmap v5190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:36.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:36 smithi081 ceph-mon[31453]: pgmap v5190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:38.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:38 smithi153 ceph-mon[36569]: pgmap v5191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:38.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:38 smithi104 ceph-mon[36624]: pgmap v5191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:38.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:38 smithi081 ceph-mon[31453]: pgmap v5191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:40.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:40 smithi153 ceph-mon[36569]: pgmap v5192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:40.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:40 smithi104 ceph-mon[36624]: pgmap v5192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:40.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:40 smithi081 ceph-mon[31453]: pgmap v5192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:42.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:42 smithi153 ceph-mon[36569]: pgmap v5193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:42.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:42 smithi104 ceph-mon[36624]: pgmap v5193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:42.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:42 smithi081 ceph-mon[31453]: pgmap v5193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:44.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:44 smithi153 ceph-mon[36569]: pgmap v5194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:44.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:44 smithi104 ceph-mon[36624]: pgmap v5194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:44.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:44 smithi081 ceph-mon[31453]: pgmap v5194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:44.704+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:47:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:44.704+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:47:44.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:44.704+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:47:46.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:46 smithi153 ceph-mon[36569]: pgmap v5195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:46.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:46 smithi104 ceph-mon[36624]: pgmap v5195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:46.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:46 smithi081 ceph-mon[31453]: pgmap v5195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:48.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:48 smithi153 ceph-mon[36569]: pgmap v5196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:48.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:48 smithi104 ceph-mon[36624]: pgmap v5196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:48.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:48 smithi081 ceph-mon[31453]: pgmap v5196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:50.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:50 smithi153 ceph-mon[36569]: pgmap v5197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:50.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:50 smithi104 ceph-mon[36624]: pgmap v5197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:50.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:50 smithi081 ceph-mon[31453]: pgmap v5197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:52.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:52 smithi153 ceph-mon[36569]: pgmap v5198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:52.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:52 smithi104 ceph-mon[36624]: pgmap v5198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:52.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:52 smithi081 ceph-mon[31453]: pgmap v5198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:54.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:54 smithi153 ceph-mon[36569]: pgmap v5199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:54.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:54 smithi104 ceph-mon[36624]: pgmap v5199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:54.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:54 smithi081 ceph-mon[31453]: pgmap v5199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:56.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:56 smithi104 ceph-mon[36624]: pgmap v5200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:56.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:56 smithi081 ceph-mon[31453]: pgmap v5200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:56.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:56 smithi153 ceph-mon[36569]: pgmap v5200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:58.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:47:58 smithi104 ceph-mon[36624]: pgmap v5201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:58.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:47:58 smithi081 ceph-mon[31453]: pgmap v5201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:58.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:47:58 smithi153 ceph-mon[36569]: pgmap v5201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:47:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:59.718+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:47:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:59.718+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:47:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:47:59.718+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:48:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:00 smithi104 ceph-mon[36624]: pgmap v5202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:00.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:00 smithi081 ceph-mon[31453]: pgmap v5202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:00.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:00 smithi153 ceph-mon[36569]: pgmap v5202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:02.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:02 smithi104 ceph-mon[36624]: pgmap v5203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:02.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:02 smithi081 ceph-mon[31453]: pgmap v5203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:02.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:02 smithi153 ceph-mon[36569]: pgmap v5203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:04.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:04 smithi081 ceph-mon[31453]: pgmap v5204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:04.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:04 smithi153 ceph-mon[36569]: pgmap v5204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:04.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:04 smithi104 ceph-mon[36624]: pgmap v5204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:06.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:06 smithi081 ceph-mon[31453]: pgmap v5205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:06.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:06 smithi153 ceph-mon[36569]: pgmap v5205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:06.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:06 smithi104 ceph-mon[36624]: pgmap v5205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:08.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:08 smithi081 ceph-mon[31453]: pgmap v5206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:08.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:08 smithi153 ceph-mon[36569]: pgmap v5206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:08.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:08 smithi104 ceph-mon[36624]: pgmap v5206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:09.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:48:09.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:48:09.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:48:10.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: pgmap v5207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:10.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:48:10.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:48:10.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:48:10.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: pgmap v5207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:10.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:48:10.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:48:10.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: pgmap v5207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:10.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:48:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:48:11.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:11 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:48:11.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:11 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:48:11.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:11 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:48:11.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:11 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:48:11.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:11 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:48:11.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:11 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:48:11.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:11 smithi081 ceph-mon[31453]: pgmap v5208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:11.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:11 smithi081 ceph-mon[31453]: pgmap v5209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:11.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:11 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:48:11.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:11 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:48:11.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:11 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:48:11.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:11 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:48:11.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:11 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:48:11.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:11 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:48:11.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:11 smithi153 ceph-mon[36569]: pgmap v5208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:11.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:11 smithi153 ceph-mon[36569]: pgmap v5209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:11.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:11 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:48:11.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:11 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:48:11.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:11 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:48:11.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:11 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:48:11.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:11 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:48:11.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:11 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:48:11.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:11 smithi104 ceph-mon[36624]: pgmap v5208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:11.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:11 smithi104 ceph-mon[36624]: pgmap v5209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:13.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:13 smithi081 ceph-mon[31453]: pgmap v5210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:13.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:13 smithi153 ceph-mon[36569]: pgmap v5210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:13.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:13 smithi104 ceph-mon[36624]: pgmap v5210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:14.721+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:48:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:14.722+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:48:14.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:14.722+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:48:15.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:15 smithi081 ceph-mon[31453]: pgmap v5211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:15.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:15 smithi153 ceph-mon[36569]: pgmap v5211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:15.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:15 smithi104 ceph-mon[36624]: pgmap v5211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:17.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:17 smithi081 ceph-mon[31453]: pgmap v5212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:17.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:17 smithi153 ceph-mon[36569]: pgmap v5212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:17.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:17 smithi104 ceph-mon[36624]: pgmap v5212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:19.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:19 smithi081 ceph-mon[31453]: pgmap v5213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:19.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:19 smithi153 ceph-mon[36569]: pgmap v5213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:19.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:19 smithi104 ceph-mon[36624]: pgmap v5213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:21.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:21 smithi081 ceph-mon[31453]: pgmap v5214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:21.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:21 smithi153 ceph-mon[36569]: pgmap v5214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:21.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:21 smithi104 ceph-mon[36624]: pgmap v5214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:23.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:23 smithi081 ceph-mon[31453]: pgmap v5215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:23.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:23 smithi153 ceph-mon[36569]: pgmap v5215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:23.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:23 smithi104 ceph-mon[36624]: pgmap v5215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:25.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:25 smithi153 ceph-mon[36569]: pgmap v5216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:25.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:25 smithi104 ceph-mon[36624]: pgmap v5216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:25.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:25 smithi081 ceph-mon[31453]: pgmap v5216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:27.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:27 smithi153 ceph-mon[36569]: pgmap v5217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:27.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:27 smithi104 ceph-mon[36624]: pgmap v5217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:27.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:27 smithi081 ceph-mon[31453]: pgmap v5217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:29.737+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:48:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:29.737+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:48:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:29.737+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:48:29.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:29 smithi153 ceph-mon[36569]: pgmap v5218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:29.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:29 smithi104 ceph-mon[36624]: pgmap v5218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:29.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:29 smithi081 ceph-mon[31453]: pgmap v5218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:31.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:31 smithi153 ceph-mon[36569]: pgmap v5219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:31.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:31 smithi104 ceph-mon[36624]: pgmap v5219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:31.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:31 smithi081 ceph-mon[31453]: pgmap v5219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:33.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:33 smithi153 ceph-mon[36569]: pgmap v5220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:33.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:33 smithi104 ceph-mon[36624]: pgmap v5220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:33.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:33 smithi081 ceph-mon[31453]: pgmap v5220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:35.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:35 smithi153 ceph-mon[36569]: pgmap v5221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:35.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:35 smithi104 ceph-mon[36624]: pgmap v5221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:35.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:35 smithi081 ceph-mon[31453]: pgmap v5221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:37.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:37 smithi153 ceph-mon[36569]: pgmap v5222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:37.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:37 smithi104 ceph-mon[36624]: pgmap v5222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:37.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:37 smithi081 ceph-mon[31453]: pgmap v5222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:38.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:38 smithi153 ceph-mon[36569]: pgmap v5223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:38.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:38 smithi104 ceph-mon[36624]: pgmap v5223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:38.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:38 smithi081 ceph-mon[31453]: pgmap v5223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:41.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:41 smithi153 ceph-mon[36569]: pgmap v5224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:41.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:41 smithi104 ceph-mon[36624]: pgmap v5224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:41.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:41 smithi081 ceph-mon[31453]: pgmap v5224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:43.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:43 smithi153 ceph-mon[36569]: pgmap v5225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:43.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:43 smithi104 ceph-mon[36624]: pgmap v5225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:43.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:43 smithi081 ceph-mon[31453]: pgmap v5225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:44.753+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:48:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:44.753+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:48:44.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:44.753+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:48:45.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:45 smithi153 ceph-mon[36569]: pgmap v5226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:45.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:45 smithi104 ceph-mon[36624]: pgmap v5226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:45.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:45 smithi081 ceph-mon[31453]: pgmap v5226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:47.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:47 smithi153 ceph-mon[36569]: pgmap v5227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:47.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:47 smithi104 ceph-mon[36624]: pgmap v5227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:47.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:47 smithi081 ceph-mon[31453]: pgmap v5227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:49.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:49 smithi153 ceph-mon[36569]: pgmap v5228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:49.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:49 smithi104 ceph-mon[36624]: pgmap v5228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:49.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:49 smithi081 ceph-mon[31453]: pgmap v5228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:51.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:51 smithi153 ceph-mon[36569]: pgmap v5229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:51.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:51 smithi104 ceph-mon[36624]: pgmap v5229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:51.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:51 smithi081 ceph-mon[31453]: pgmap v5229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:53.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:53 smithi153 ceph-mon[36569]: pgmap v5230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:53.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:53 smithi104 ceph-mon[36624]: pgmap v5230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:53.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:53 smithi081 ceph-mon[31453]: pgmap v5230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:55.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:55 smithi153 ceph-mon[36569]: pgmap v5231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:55.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:55 smithi104 ceph-mon[36624]: pgmap v5231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:55.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:55 smithi081 ceph-mon[31453]: pgmap v5231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:57.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:57 smithi153 ceph-mon[36569]: pgmap v5232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:57.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:57 smithi104 ceph-mon[36624]: pgmap v5232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:57.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:57 smithi081 ceph-mon[31453]: pgmap v5232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:59.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:48:59 smithi153 ceph-mon[36569]: pgmap v5233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:59.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:48:59 smithi104 ceph-mon[36624]: pgmap v5233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:59.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:48:59 smithi081 ceph-mon[31453]: pgmap v5233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:48:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:59.768+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:48:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:59.768+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:48:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:48:59.768+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:01.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:01 smithi153 ceph-mon[36569]: pgmap v5234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:01.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:01 smithi104 ceph-mon[36624]: pgmap v5234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:01.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:01 smithi081 ceph-mon[31453]: pgmap v5234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:03.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:03 smithi153 ceph-mon[36569]: pgmap v5235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:03.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:03 smithi104 ceph-mon[36624]: pgmap v5235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:03.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:03 smithi081 ceph-mon[31453]: pgmap v5235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:05.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:05 smithi153 ceph-mon[36569]: pgmap v5236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:05.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:05 smithi104 ceph-mon[36624]: pgmap v5236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:05.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:05 smithi081 ceph-mon[31453]: pgmap v5236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:07.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:07 smithi153 ceph-mon[36569]: pgmap v5237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:07.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:07 smithi104 ceph-mon[36624]: pgmap v5237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:07.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:07 smithi081 ceph-mon[31453]: pgmap v5237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:09.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:09 smithi153 ceph-mon[36569]: pgmap v5238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:09.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:09 smithi104 ceph-mon[36624]: pgmap v5238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:09.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:09 smithi081 ceph-mon[31453]: pgmap v5238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:11.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:11 smithi153 ceph-mon[36569]: pgmap v5239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:11.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:49:11.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:11 smithi104 ceph-mon[36624]: pgmap v5239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:11.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:49:11.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:11 smithi081 ceph-mon[31453]: pgmap v5239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:11.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:49:12.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:49:12.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:49:12.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:49:12.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:49:12.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:49:12.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:49:12.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:49:12.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:49:12.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:49:13.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:13 smithi153 ceph-mon[36569]: pgmap v5240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:13.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:13 smithi153 ceph-mon[36569]: pgmap v5241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:13.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:13 smithi104 ceph-mon[36624]: pgmap v5240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:13.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:13 smithi104 ceph-mon[36624]: pgmap v5241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:13.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:13 smithi081 ceph-mon[31453]: pgmap v5240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:13.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:13 smithi081 ceph-mon[31453]: pgmap v5241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:14.214+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:49:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:14.214+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:49:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:14.215+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:49:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:14.215+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:49:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:14.784+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:14.784+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:49:14.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:14.784+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:15.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:15 smithi153 ceph-mon[36569]: pgmap v5242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:15.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:15 smithi104 ceph-mon[36624]: pgmap v5242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:15.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:15 smithi081 ceph-mon[31453]: pgmap v5242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:17.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:17 smithi153 ceph-mon[36569]: pgmap v5243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:17.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:17 smithi104 ceph-mon[36624]: pgmap v5243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:17.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:17 smithi081 ceph-mon[31453]: pgmap v5243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:19.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:19 smithi153 ceph-mon[36569]: pgmap v5244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:19.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:19 smithi104 ceph-mon[36624]: pgmap v5244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:19.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:19 smithi081 ceph-mon[31453]: pgmap v5244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:21.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:21 smithi153 ceph-mon[36569]: pgmap v5245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:21.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:21 smithi104 ceph-mon[36624]: pgmap v5245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:21.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:21 smithi081 ceph-mon[31453]: pgmap v5245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:23.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:23 smithi153 ceph-mon[36569]: pgmap v5246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:23.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:23 smithi104 ceph-mon[36624]: pgmap v5246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:23.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:23 smithi081 ceph-mon[31453]: pgmap v5246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:25.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:25 smithi104 ceph-mon[36624]: pgmap v5247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:25.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:25 smithi081 ceph-mon[31453]: pgmap v5247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:25.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:25 smithi153 ceph-mon[36569]: pgmap v5247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:27.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:27 smithi104 ceph-mon[36624]: pgmap v5248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:27.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:27 smithi081 ceph-mon[31453]: pgmap v5248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:27.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:27 smithi153 ceph-mon[36569]: pgmap v5248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:29.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:29 smithi104 ceph-mon[36624]: pgmap v5249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:29.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:29 smithi081 ceph-mon[31453]: pgmap v5249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:29.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:29.800+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:29.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:29.800+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:29.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:29.800+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:49:29.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:29 smithi153 ceph-mon[36569]: pgmap v5249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:31.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:31 smithi104 ceph-mon[36624]: pgmap v5250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:31.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:31 smithi081 ceph-mon[31453]: pgmap v5250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:31.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:31 smithi153 ceph-mon[36569]: pgmap v5250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:33.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:33 smithi104 ceph-mon[36624]: pgmap v5251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:33.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:33 smithi081 ceph-mon[31453]: pgmap v5251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:33.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:33 smithi153 ceph-mon[36569]: pgmap v5251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:35.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:35 smithi081 ceph-mon[31453]: pgmap v5252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:35.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:35 smithi153 ceph-mon[36569]: pgmap v5252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:35.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:35 smithi104 ceph-mon[36624]: pgmap v5252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:37.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:37 smithi081 ceph-mon[31453]: pgmap v5253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:37.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:37 smithi153 ceph-mon[36569]: pgmap v5253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:37.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:37 smithi104 ceph-mon[36624]: pgmap v5253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:39.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:39 smithi081 ceph-mon[31453]: pgmap v5254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:39.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:39 smithi153 ceph-mon[36569]: pgmap v5254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:39.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:39 smithi104 ceph-mon[36624]: pgmap v5254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:41.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:41 smithi081 ceph-mon[31453]: pgmap v5255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:41.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:41 smithi153 ceph-mon[36569]: pgmap v5255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:41.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:41 smithi104 ceph-mon[36624]: pgmap v5255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:43.590 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:43 smithi153 ceph-mon[36569]: pgmap v5256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:43.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:43 smithi081 ceph-mon[31453]: pgmap v5256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:43.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:43 smithi104 ceph-mon[36624]: pgmap v5256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:44.806+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:44.806+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:49:44.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:44.806+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:45.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:45 smithi081 ceph-mon[31453]: pgmap v5257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:45.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:45 smithi153 ceph-mon[36569]: pgmap v5257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:45.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:45 smithi104 ceph-mon[36624]: pgmap v5257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:47.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:47 smithi081 ceph-mon[31453]: pgmap v5258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:47.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:47 smithi153 ceph-mon[36569]: pgmap v5258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:47.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:47 smithi104 ceph-mon[36624]: pgmap v5258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:49.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:49 smithi081 ceph-mon[31453]: pgmap v5259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:49.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:49 smithi153 ceph-mon[36569]: pgmap v5259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:49.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:49 smithi104 ceph-mon[36624]: pgmap v5259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:51.616 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:51 smithi104 ceph-mon[36624]: pgmap v5260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:51.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:51 smithi081 ceph-mon[31453]: pgmap v5260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:51.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:51 smithi153 ceph-mon[36569]: pgmap v5260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:53.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:53 smithi081 ceph-mon[31453]: pgmap v5261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:53.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:53 smithi153 ceph-mon[36569]: pgmap v5261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:53.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:53 smithi104 ceph-mon[36624]: pgmap v5261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:55.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:55 smithi081 ceph-mon[31453]: pgmap v5262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:55.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:55 smithi153 ceph-mon[36569]: pgmap v5262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:55.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:55 smithi104 ceph-mon[36624]: pgmap v5262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:57.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:57 smithi081 ceph-mon[31453]: pgmap v5263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:57.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:57 smithi153 ceph-mon[36569]: pgmap v5263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:57.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:57 smithi104 ceph-mon[36624]: pgmap v5263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:59.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:49:59 smithi081 ceph-mon[31453]: pgmap v5264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:59.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:49:59 smithi153 ceph-mon[36569]: pgmap v5264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:49:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:59.819+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:59.819+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:49:59.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:49:59.819+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:49:59.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:49:59 smithi104 ceph-mon[36624]: pgmap v5264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:00.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:50:00.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:50:00.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-12T23:50:01.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:01 smithi153 ceph-mon[36569]: pgmap v5265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:01.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:01 smithi104 ceph-mon[36624]: pgmap v5265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:01.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:01 smithi081 ceph-mon[31453]: pgmap v5265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:03.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:03 smithi153 ceph-mon[36569]: pgmap v5266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:03.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:03 smithi104 ceph-mon[36624]: pgmap v5266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:03.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:03 smithi081 ceph-mon[31453]: pgmap v5266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:05.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:05 smithi153 ceph-mon[36569]: pgmap v5267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:05.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:05 smithi104 ceph-mon[36624]: pgmap v5267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:05.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:05 smithi081 ceph-mon[31453]: pgmap v5267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:07.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:07 smithi081 ceph-mon[31453]: pgmap v5268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:07.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:07 smithi153 ceph-mon[36569]: pgmap v5268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:07.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:07 smithi104 ceph-mon[36624]: pgmap v5268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:09.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:09 smithi153 ceph-mon[36569]: pgmap v5269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:09.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:09 smithi104 ceph-mon[36624]: pgmap v5269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:09.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:09 smithi081 ceph-mon[31453]: pgmap v5269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:11.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:11 smithi153 ceph-mon[36569]: pgmap v5270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:11 smithi104 ceph-mon[36624]: pgmap v5270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:11.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:11 smithi081 ceph-mon[31453]: pgmap v5270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:12.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:50:12.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:50:12.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:50:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:13 smithi153 ceph-mon[36569]: pgmap v5271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:50:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:50:13.689 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:50:13.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:13 smithi104 ceph-mon[36624]: pgmap v5271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:13.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:50:13.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:50:13.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:50:13.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:13 smithi081 ceph-mon[31453]: pgmap v5271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:13.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:50:13.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:50:13.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:50:14.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:14 smithi153 ceph-mon[36569]: pgmap v5272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:14.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:14 smithi153 ceph-mon[36569]: pgmap v5273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:14.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:14 smithi104 ceph-mon[36624]: pgmap v5272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:14.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:14 smithi104 ceph-mon[36624]: pgmap v5273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:14.813 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:14.832+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:50:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:14.832+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:50:14.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:14.832+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:50:14.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:14 smithi081 ceph-mon[31453]: pgmap v5272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:14.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:14 smithi081 ceph-mon[31453]: pgmap v5273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:15.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:15 smithi153 ceph-mon[36569]: pgmap v5274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:15.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:15 smithi104 ceph-mon[36624]: pgmap v5274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:15.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:15 smithi081 ceph-mon[31453]: pgmap v5274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:18.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:18 smithi104 ceph-mon[36624]: pgmap v5275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:18.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:18 smithi081 ceph-mon[31453]: pgmap v5275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:18.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:18 smithi153 ceph-mon[36569]: pgmap v5275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:20.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:20 smithi104 ceph-mon[36624]: pgmap v5276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:20.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:20 smithi081 ceph-mon[31453]: pgmap v5276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:20.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:20 smithi153 ceph-mon[36569]: pgmap v5276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:22.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:22 smithi104 ceph-mon[36624]: pgmap v5277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:22.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:22 smithi081 ceph-mon[31453]: pgmap v5277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:22.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:22 smithi153 ceph-mon[36569]: pgmap v5277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:24.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:24 smithi104 ceph-mon[36624]: pgmap v5278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:24.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:24 smithi081 ceph-mon[31453]: pgmap v5278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:24.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:24 smithi153 ceph-mon[36569]: pgmap v5278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:26.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:26 smithi104 ceph-mon[36624]: pgmap v5279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:26.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:26 smithi081 ceph-mon[31453]: pgmap v5279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:26.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:26 smithi153 ceph-mon[36569]: pgmap v5279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:28.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:28 smithi081 ceph-mon[31453]: pgmap v5280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:28.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:28 smithi153 ceph-mon[36569]: pgmap v5280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:28.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:28 smithi104 ceph-mon[36624]: pgmap v5280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:29.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:29.847+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:50:29.829 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:29.847+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:50:29.829 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:29.847+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:50:30.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:30 smithi081 ceph-mon[31453]: pgmap v5281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:30.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:30 smithi153 ceph-mon[36569]: pgmap v5281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:30.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:30 smithi104 ceph-mon[36624]: pgmap v5281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:32.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:32 smithi081 ceph-mon[31453]: pgmap v5282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:32.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:32 smithi153 ceph-mon[36569]: pgmap v5282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:32.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:32 smithi104 ceph-mon[36624]: pgmap v5282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:34.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:34 smithi081 ceph-mon[31453]: pgmap v5283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:34.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:34 smithi153 ceph-mon[36569]: pgmap v5283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:34.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:34 smithi104 ceph-mon[36624]: pgmap v5283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:36.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:36 smithi081 ceph-mon[31453]: pgmap v5284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:36.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:36 smithi153 ceph-mon[36569]: pgmap v5284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:36.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:36 smithi104 ceph-mon[36624]: pgmap v5284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:38.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:38 smithi081 ceph-mon[31453]: pgmap v5285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:38.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:38 smithi153 ceph-mon[36569]: pgmap v5285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:38.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:38 smithi104 ceph-mon[36624]: pgmap v5285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:40.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:40 smithi081 ceph-mon[31453]: pgmap v5286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:40.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:40 smithi153 ceph-mon[36569]: pgmap v5286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:40.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:40 smithi104 ceph-mon[36624]: pgmap v5286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:42.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:42 smithi081 ceph-mon[31453]: pgmap v5287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:42.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:42 smithi153 ceph-mon[36569]: pgmap v5287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:42.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:42 smithi104 ceph-mon[36624]: pgmap v5287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:44.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:44 smithi081 ceph-mon[31453]: pgmap v5288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:44.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:44 smithi153 ceph-mon[36569]: pgmap v5288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:44.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:44 smithi104 ceph-mon[36624]: pgmap v5288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:44.863+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:50:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:44.863+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:50:44.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:44.863+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:50:46.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:46 smithi081 ceph-mon[31453]: pgmap v5289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:46.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:46 smithi153 ceph-mon[36569]: pgmap v5289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:46.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:46 smithi104 ceph-mon[36624]: pgmap v5289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:48.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:48 smithi081 ceph-mon[31453]: pgmap v5290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:48.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:48 smithi153 ceph-mon[36569]: pgmap v5290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:48.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:48 smithi104 ceph-mon[36624]: pgmap v5290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:50.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:50 smithi081 ceph-mon[31453]: pgmap v5291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:50.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:50 smithi153 ceph-mon[36569]: pgmap v5291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:50.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:50 smithi104 ceph-mon[36624]: pgmap v5291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:52.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:52 smithi081 ceph-mon[31453]: pgmap v5292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:52.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:52 smithi153 ceph-mon[36569]: pgmap v5292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:52.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:52 smithi104 ceph-mon[36624]: pgmap v5292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:54.637 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:54 smithi081 ceph-mon[31453]: pgmap v5293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:54.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:54 smithi153 ceph-mon[36569]: pgmap v5293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:54.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:54 smithi104 ceph-mon[36624]: pgmap v5293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:56.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:56 smithi153 ceph-mon[36569]: pgmap v5294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:56.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:56 smithi104 ceph-mon[36624]: pgmap v5294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:56.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:56 smithi081 ceph-mon[31453]: pgmap v5294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:58.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:50:58 smithi153 ceph-mon[36569]: pgmap v5295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:58.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:50:58 smithi104 ceph-mon[36624]: pgmap v5295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:58.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:50:58 smithi081 ceph-mon[31453]: pgmap v5295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:50:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:59.876+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:50:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:59.876+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:50:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:50:59.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:51:00.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:00 smithi153 ceph-mon[36569]: pgmap v5296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:00.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:00 smithi104 ceph-mon[36624]: pgmap v5296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:00.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:00 smithi081 ceph-mon[31453]: pgmap v5296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:02.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:02 smithi153 ceph-mon[36569]: pgmap v5297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:02.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:02 smithi104 ceph-mon[36624]: pgmap v5297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:02.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:02 smithi081 ceph-mon[31453]: pgmap v5297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:04.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:04 smithi153 ceph-mon[36569]: pgmap v5298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:04.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:04 smithi104 ceph-mon[36624]: pgmap v5298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:04.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:04 smithi081 ceph-mon[31453]: pgmap v5298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:06.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:06 smithi153 ceph-mon[36569]: pgmap v5299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:06.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:06 smithi104 ceph-mon[36624]: pgmap v5299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:06.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:06 smithi081 ceph-mon[31453]: pgmap v5299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:08.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:08 smithi153 ceph-mon[36569]: pgmap v5300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:08.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:08 smithi104 ceph-mon[36624]: pgmap v5300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:08.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:08 smithi081 ceph-mon[31453]: pgmap v5300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:10.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:10 smithi081 ceph-mon[31453]: pgmap v5301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:10.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:10 smithi153 ceph-mon[36569]: pgmap v5301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:10.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:10 smithi104 ceph-mon[36624]: pgmap v5301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:12.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:12 smithi153 ceph-mon[36569]: pgmap v5302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:12.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:12 smithi104 ceph-mon[36624]: pgmap v5302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:12.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:12 smithi081 ceph-mon[31453]: pgmap v5302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:14.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:14 smithi153 ceph-mon[36569]: pgmap v5303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:14.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:51:14.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:14 smithi104 ceph-mon[36624]: pgmap v5303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:14.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:51:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:14.891+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:51:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:14.891+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:51:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:14.891+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:51:14.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:14 smithi081 ceph-mon[31453]: pgmap v5303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:14.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:51:15.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:51:15.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:51:15.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:51:15.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:51:15.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:51:15.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:51:15.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:51:15.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:51:15.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:51:16.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:16 smithi153 ceph-mon[36569]: pgmap v5304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:16.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:16 smithi153 ceph-mon[36569]: pgmap v5305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:16.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:16 smithi104 ceph-mon[36624]: pgmap v5304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:16.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:16 smithi104 ceph-mon[36624]: pgmap v5305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:16.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:16 smithi081 ceph-mon[31453]: pgmap v5304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:16.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:16 smithi081 ceph-mon[31453]: pgmap v5305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:18.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:18 smithi153 ceph-mon[36569]: pgmap v5306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:18.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:18 smithi104 ceph-mon[36624]: pgmap v5306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:18.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:18 smithi081 ceph-mon[31453]: pgmap v5306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:19.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:19 smithi153 ceph-mon[36569]: pgmap v5307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:19.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:19 smithi104 ceph-mon[36624]: pgmap v5307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:19.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:19 smithi081 ceph-mon[31453]: pgmap v5307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:22.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:21 smithi081 ceph-mon[31453]: pgmap v5308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:22.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:21 smithi153 ceph-mon[36569]: pgmap v5308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:22.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:21 smithi104 ceph-mon[36624]: pgmap v5308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:24.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:23 smithi081 ceph-mon[31453]: pgmap v5309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:24.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:23 smithi153 ceph-mon[36569]: pgmap v5309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:24.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:23 smithi104 ceph-mon[36624]: pgmap v5309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:26.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:25 smithi081 ceph-mon[31453]: pgmap v5310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:26.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:25 smithi153 ceph-mon[36569]: pgmap v5310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:26.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:25 smithi104 ceph-mon[36624]: pgmap v5310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:28.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:27 smithi081 ceph-mon[31453]: pgmap v5311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:28.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:27 smithi153 ceph-mon[36569]: pgmap v5311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:28.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:27 smithi104 ceph-mon[36624]: pgmap v5311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:29.906+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:51:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:29.906+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:51:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:29.906+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:51:30.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:29 smithi081 ceph-mon[31453]: pgmap v5312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:30.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:29 smithi153 ceph-mon[36569]: pgmap v5312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:30.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:29 smithi104 ceph-mon[36624]: pgmap v5312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:32.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:31 smithi081 ceph-mon[31453]: pgmap v5313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:32.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:31 smithi153 ceph-mon[36569]: pgmap v5313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:32.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:31 smithi104 ceph-mon[36624]: pgmap v5313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:34.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:33 smithi081 ceph-mon[31453]: pgmap v5314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:34.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:33 smithi153 ceph-mon[36569]: pgmap v5314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:34.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:33 smithi104 ceph-mon[36624]: pgmap v5314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:36.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:35 smithi153 ceph-mon[36569]: pgmap v5315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:36.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:35 smithi104 ceph-mon[36624]: pgmap v5315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:36.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:35 smithi081 ceph-mon[31453]: pgmap v5315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:38.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:37 smithi153 ceph-mon[36569]: pgmap v5316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:38.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:37 smithi104 ceph-mon[36624]: pgmap v5316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:38.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:37 smithi081 ceph-mon[31453]: pgmap v5316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:40.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:39 smithi153 ceph-mon[36569]: pgmap v5317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:40.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:39 smithi104 ceph-mon[36624]: pgmap v5317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:40.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:39 smithi081 ceph-mon[31453]: pgmap v5317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:42.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:41 smithi081 ceph-mon[31453]: pgmap v5318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:42.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:41 smithi153 ceph-mon[36569]: pgmap v5318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:42.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:41 smithi104 ceph-mon[36624]: pgmap v5318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:44.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:43 smithi153 ceph-mon[36569]: pgmap v5319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:44.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:43 smithi104 ceph-mon[36624]: pgmap v5319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:44.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:43 smithi081 ceph-mon[31453]: pgmap v5319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:44.922+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:51:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:44.922+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:51:44.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:44.922+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:51:46.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:45 smithi153 ceph-mon[36569]: pgmap v5320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:46.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:45 smithi104 ceph-mon[36624]: pgmap v5320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:46.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:45 smithi081 ceph-mon[31453]: pgmap v5320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:48.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:47 smithi153 ceph-mon[36569]: pgmap v5321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:48.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:47 smithi104 ceph-mon[36624]: pgmap v5321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:48.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:47 smithi081 ceph-mon[31453]: pgmap v5321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:50.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:49 smithi153 ceph-mon[36569]: pgmap v5322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:50.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:49 smithi104 ceph-mon[36624]: pgmap v5322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:50.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:49 smithi081 ceph-mon[31453]: pgmap v5322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:52.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:51 smithi153 ceph-mon[36569]: pgmap v5323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:52.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:51 smithi104 ceph-mon[36624]: pgmap v5323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:52.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:51 smithi081 ceph-mon[31453]: pgmap v5323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:53 smithi153 ceph-mon[36569]: pgmap v5324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:54.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:53 smithi104 ceph-mon[36624]: pgmap v5324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:54.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:53 smithi081 ceph-mon[31453]: pgmap v5324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:56.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:55 smithi153 ceph-mon[36569]: pgmap v5325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:56.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:55 smithi104 ceph-mon[36624]: pgmap v5325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:56.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:55 smithi081 ceph-mon[31453]: pgmap v5325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:58.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:57 smithi153 ceph-mon[36569]: pgmap v5326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:58.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:57 smithi104 ceph-mon[36624]: pgmap v5326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:58.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:57 smithi081 ceph-mon[31453]: pgmap v5326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:51:59.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:59.938+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:51:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:59.938+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:51:59.919 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:51:59.938+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:52:00.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:51:59 smithi153 ceph-mon[36569]: pgmap v5327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:00.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:51:59 smithi104 ceph-mon[36624]: pgmap v5327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:00.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:51:59 smithi081 ceph-mon[31453]: pgmap v5327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:02.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:01 smithi153 ceph-mon[36569]: pgmap v5328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:02.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:01 smithi104 ceph-mon[36624]: pgmap v5328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:02.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:01 smithi081 ceph-mon[31453]: pgmap v5328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:04.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:03 smithi153 ceph-mon[36569]: pgmap v5329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:04.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:03 smithi104 ceph-mon[36624]: pgmap v5329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:04.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:03 smithi081 ceph-mon[31453]: pgmap v5329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:06.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:05 smithi153 ceph-mon[36569]: pgmap v5330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:06.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:05 smithi104 ceph-mon[36624]: pgmap v5330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:06.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:05 smithi081 ceph-mon[31453]: pgmap v5330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:08.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:08 smithi153 ceph-mon[36569]: pgmap v5331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:08.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:08 smithi104 ceph-mon[36624]: pgmap v5331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:08.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:08 smithi081 ceph-mon[31453]: pgmap v5331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:10.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:10 smithi153 ceph-mon[36569]: pgmap v5332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:10.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:10 smithi104 ceph-mon[36624]: pgmap v5332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:10.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:10 smithi081 ceph-mon[31453]: pgmap v5332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:12.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:12 smithi153 ceph-mon[36569]: pgmap v5333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:12.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:12 smithi104 ceph-mon[36624]: pgmap v5333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:12.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:12 smithi081 ceph-mon[31453]: pgmap v5333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:14.243+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:52:14.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:14.243+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:52:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:14.244+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:52:14.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:14.244+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:52:14.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:14 smithi153 ceph-mon[36569]: pgmap v5334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:14.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:14 smithi104 ceph-mon[36624]: pgmap v5334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:14.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:14 smithi081 ceph-mon[31453]: pgmap v5334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:14.952+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:52:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:14.952+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:52:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:14.952+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:52:16.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:16 smithi153 ceph-mon[36569]: pgmap v5335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:16.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:52:16.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:16 smithi104 ceph-mon[36624]: pgmap v5335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:16.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:52:16.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:16 smithi081 ceph-mon[31453]: pgmap v5335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:16.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:52:16.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:16.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:18 smithi153 ceph-mon[36569]: pgmap v5336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:18.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:18 smithi104 ceph-mon[36624]: pgmap v5336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:18.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:18 smithi081 ceph-mon[31453]: pgmap v5336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:18.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:18.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:20.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:20 smithi153 ceph-mon[36569]: pgmap v5337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:20.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:20 smithi104 ceph-mon[36624]: pgmap v5337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:20.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:20 smithi081 ceph-mon[31453]: pgmap v5337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:21.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:21.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:21.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:21 smithi153 ceph-mon[36569]: pgmap v5338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:21.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:21.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:21.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:21 smithi104 ceph-mon[36624]: pgmap v5338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:21.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:21.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:21.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:21 smithi081 ceph-mon[31453]: pgmap v5338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:23.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:52:23.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:52:23.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:22 smithi153 ceph-mon[36569]: pgmap v5339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:23.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:22 smithi153 ceph-mon[36569]: pgmap v5340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:23.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:23.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:52:23.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:52:23.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:22 smithi104 ceph-mon[36624]: pgmap v5339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:23.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:22 smithi104 ceph-mon[36624]: pgmap v5340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:23.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:23.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:52:23.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:52:23.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:22 smithi081 ceph-mon[31453]: pgmap v5339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:23.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:22 smithi081 ceph-mon[31453]: pgmap v5340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:23.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:52:25.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:24 smithi153 ceph-mon[36569]: pgmap v5341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:25.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:24 smithi104 ceph-mon[36624]: pgmap v5341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:25.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:24 smithi081 ceph-mon[31453]: pgmap v5341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:27.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:26 smithi153 ceph-mon[36569]: pgmap v5342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:27.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:26 smithi104 ceph-mon[36624]: pgmap v5342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:27.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:26 smithi081 ceph-mon[31453]: pgmap v5342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:29.190 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:28 smithi081 ceph-mon[31453]: pgmap v5343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:29.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:28 smithi153 ceph-mon[36569]: pgmap v5343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:29.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:28 smithi104 ceph-mon[36624]: pgmap v5343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:29.967+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:52:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:29.968+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:52:29.949 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:29.968+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:52:31.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:30 smithi153 ceph-mon[36569]: pgmap v5344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:31.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:30 smithi104 ceph-mon[36624]: pgmap v5344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:31.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:30 smithi081 ceph-mon[31453]: pgmap v5344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:33.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:32 smithi153 ceph-mon[36569]: pgmap v5345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:33.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:32 smithi104 ceph-mon[36624]: pgmap v5345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:33.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:32 smithi081 ceph-mon[31453]: pgmap v5345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:35.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:34 smithi153 ceph-mon[36569]: pgmap v5346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:35.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:34 smithi104 ceph-mon[36624]: pgmap v5346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:35.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:34 smithi081 ceph-mon[31453]: pgmap v5346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:37.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:36 smithi153 ceph-mon[36569]: pgmap v5347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:37.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:36 smithi104 ceph-mon[36624]: pgmap v5347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:37.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:36 smithi081 ceph-mon[31453]: pgmap v5347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:39.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:38 smithi153 ceph-mon[36569]: pgmap v5348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:39.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:38 smithi104 ceph-mon[36624]: pgmap v5348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:39.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:38 smithi081 ceph-mon[31453]: pgmap v5348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:41.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:40 smithi153 ceph-mon[36569]: pgmap v5349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:41.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:40 smithi104 ceph-mon[36624]: pgmap v5349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:41.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:40 smithi081 ceph-mon[31453]: pgmap v5349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:43.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:42 smithi153 ceph-mon[36569]: pgmap v5350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:43.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:42 smithi104 ceph-mon[36624]: pgmap v5350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:43.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:42 smithi081 ceph-mon[31453]: pgmap v5350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:44.976+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:52:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:44.976+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:52:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:44.976+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:52:45.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:44 smithi153 ceph-mon[36569]: pgmap v5351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:45.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:44 smithi104 ceph-mon[36624]: pgmap v5351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:45.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:44 smithi081 ceph-mon[31453]: pgmap v5351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:47.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:46 smithi153 ceph-mon[36569]: pgmap v5352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:47.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:46 smithi104 ceph-mon[36624]: pgmap v5352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:47.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:46 smithi081 ceph-mon[31453]: pgmap v5352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:49.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:48 smithi153 ceph-mon[36569]: pgmap v5353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:49.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:48 smithi104 ceph-mon[36624]: pgmap v5353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:49.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:48 smithi081 ceph-mon[31453]: pgmap v5353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:51.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:51 smithi153 ceph-mon[36569]: pgmap v5354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:51.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:51 smithi104 ceph-mon[36624]: pgmap v5354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:51.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:51 smithi081 ceph-mon[31453]: pgmap v5354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:53.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:53 smithi153 ceph-mon[36569]: pgmap v5355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:53.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:53 smithi104 ceph-mon[36624]: pgmap v5355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:53 smithi081 ceph-mon[31453]: pgmap v5355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:55.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:55 smithi153 ceph-mon[36569]: pgmap v5356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:55.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:55 smithi104 ceph-mon[36624]: pgmap v5356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:55.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:55 smithi081 ceph-mon[31453]: pgmap v5356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:57.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:57 smithi153 ceph-mon[36569]: pgmap v5357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:57.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:57 smithi104 ceph-mon[36624]: pgmap v5357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:57.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:57 smithi081 ceph-mon[31453]: pgmap v5357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:59.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:59.251+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:52:59.232 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:59.251+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:52:59.232 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:59.252+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:52:59.232 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:59.252+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:52:59.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:52:59 smithi153 ceph-mon[36569]: pgmap v5358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:59.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:52:59 smithi104 ceph-mon[36624]: pgmap v5358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:59.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:52:59 smithi081 ceph-mon[31453]: pgmap v5358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:52:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:59.981+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:52:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:59.981+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:52:59.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:52:59.981+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:01.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:01 smithi153 ceph-mon[36569]: pgmap v5359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:01.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:01 smithi104 ceph-mon[36624]: pgmap v5359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:01.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:01 smithi081 ceph-mon[31453]: pgmap v5359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:03.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:03 smithi153 ceph-mon[36569]: pgmap v5360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:03.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:03 smithi104 ceph-mon[36624]: pgmap v5360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:03.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:03 smithi081 ceph-mon[31453]: pgmap v5360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:05.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:05 smithi153 ceph-mon[36569]: pgmap v5361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:05.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:05 smithi104 ceph-mon[36624]: pgmap v5361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:05.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:05 smithi081 ceph-mon[31453]: pgmap v5361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:07.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:07 smithi153 ceph-mon[36569]: pgmap v5362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:07.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:07 smithi104 ceph-mon[36624]: pgmap v5362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:07.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:07 smithi081 ceph-mon[31453]: pgmap v5362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:09.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:09 smithi153 ceph-mon[36569]: pgmap v5363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:09.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:09 smithi104 ceph-mon[36624]: pgmap v5363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:09.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:09 smithi081 ceph-mon[31453]: pgmap v5363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:11.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:11 smithi104 ceph-mon[36624]: pgmap v5364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:11.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:11 smithi081 ceph-mon[31453]: pgmap v5364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:11.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:11 smithi153 ceph-mon[36569]: pgmap v5364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:13.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:13 smithi104 ceph-mon[36624]: pgmap v5365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:13.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:13 smithi081 ceph-mon[31453]: pgmap v5365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:13.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:13 smithi153 ceph-mon[36569]: pgmap v5365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:14.995+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:14.995+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:53:14.976 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:14.995+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:15.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:15 smithi104 ceph-mon[36624]: pgmap v5366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:15.339 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:15 smithi153 ceph-mon[36569]: pgmap v5366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:15.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:15 smithi081 ceph-mon[31453]: pgmap v5366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:17.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:17 smithi104 ceph-mon[36624]: pgmap v5367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:17.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:17 smithi081 ceph-mon[31453]: pgmap v5367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:17.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:17 smithi153 ceph-mon[36569]: pgmap v5367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:19.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:19 smithi104 ceph-mon[36624]: pgmap v5368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:19.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:19 smithi081 ceph-mon[31453]: pgmap v5368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:19.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:19 smithi153 ceph-mon[36569]: pgmap v5368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:21.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:21 smithi104 ceph-mon[36624]: pgmap v5369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:21.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:21 smithi081 ceph-mon[31453]: pgmap v5369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:21.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:21 smithi153 ceph-mon[36569]: pgmap v5369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:23.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:23 smithi081 ceph-mon[31453]: pgmap v5370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:23.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:53:23.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:23.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:23.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:23 smithi153 ceph-mon[36569]: pgmap v5370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:23.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:53:23.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:23.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:23.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:23 smithi104 ceph-mon[36624]: pgmap v5370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:23.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:53:23.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:23.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:24.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:53:24.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:53:24.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:24 smithi081 ceph-mon[31453]: pgmap v5371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:24.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:24 smithi081 ceph-mon[31453]: pgmap v5372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:24.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:24.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:53:24.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:53:24.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:24 smithi153 ceph-mon[36569]: pgmap v5371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:24.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:24 smithi153 ceph-mon[36569]: pgmap v5372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:24.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:24.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:53:24.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:53:24.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:24 smithi104 ceph-mon[36624]: pgmap v5371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:24.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:24 smithi104 ceph-mon[36624]: pgmap v5372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:24.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:53:26.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:26 smithi153 ceph-mon[36569]: pgmap v5373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:26.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:26 smithi104 ceph-mon[36624]: pgmap v5373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:26.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:26 smithi081 ceph-mon[31453]: pgmap v5373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:28.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:28 smithi153 ceph-mon[36569]: pgmap v5374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:28.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:28 smithi104 ceph-mon[36624]: pgmap v5374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:28.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:28 smithi081 ceph-mon[31453]: pgmap v5374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:29.986 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:30.006+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:30.006+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:53:29.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:30.006+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:30.689 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:30 smithi081 ceph-mon[31453]: pgmap v5375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:30.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:30 smithi153 ceph-mon[36569]: pgmap v5375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:30.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:30 smithi104 ceph-mon[36624]: pgmap v5375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:32 smithi153 ceph-mon[36569]: pgmap v5376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:32 smithi104 ceph-mon[36624]: pgmap v5376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:32 smithi081 ceph-mon[31453]: pgmap v5376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:34.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:34 smithi153 ceph-mon[36569]: pgmap v5377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:34.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:34 smithi104 ceph-mon[36624]: pgmap v5377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:34.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:34 smithi081 ceph-mon[31453]: pgmap v5377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:36.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:36 smithi153 ceph-mon[36569]: pgmap v5378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:36.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:36 smithi104 ceph-mon[36624]: pgmap v5378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:36.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:36 smithi081 ceph-mon[31453]: pgmap v5378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:38.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:38 smithi153 ceph-mon[36569]: pgmap v5379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:38.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:38 smithi104 ceph-mon[36624]: pgmap v5379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:38.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:38 smithi081 ceph-mon[31453]: pgmap v5379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:40.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:40 smithi153 ceph-mon[36569]: pgmap v5380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:40.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:40 smithi104 ceph-mon[36624]: pgmap v5380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:40.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:40 smithi081 ceph-mon[31453]: pgmap v5380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:42.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:42 smithi153 ceph-mon[36569]: pgmap v5381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:42.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:42 smithi104 ceph-mon[36624]: pgmap v5381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:42.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:42 smithi081 ceph-mon[31453]: pgmap v5381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:44.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:44 smithi153 ceph-mon[36569]: pgmap v5382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:44.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:44 smithi104 ceph-mon[36624]: pgmap v5382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:44.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:44 smithi081 ceph-mon[31453]: pgmap v5382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:45.008+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:45.008+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:53:45.008+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:53:46.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:46 smithi153 ceph-mon[36569]: pgmap v5383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:46.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:46 smithi104 ceph-mon[36624]: pgmap v5383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:46.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:46 smithi081 ceph-mon[31453]: pgmap v5383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:48.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:48 smithi153 ceph-mon[36569]: pgmap v5384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:48.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:48 smithi104 ceph-mon[36624]: pgmap v5384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:48.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:48 smithi081 ceph-mon[31453]: pgmap v5384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:50.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:50 smithi153 ceph-mon[36569]: pgmap v5385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:50.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:50 smithi104 ceph-mon[36624]: pgmap v5385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:50.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:50 smithi081 ceph-mon[31453]: pgmap v5385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:51.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:51 smithi153 ceph-mon[36569]: pgmap v5386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:51.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:51 smithi104 ceph-mon[36624]: pgmap v5386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:51.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:51 smithi081 ceph-mon[31453]: pgmap v5386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:54.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:54 smithi153 ceph-mon[36569]: pgmap v5387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:54.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:54 smithi104 ceph-mon[36624]: pgmap v5387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:54.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:54 smithi081 ceph-mon[31453]: pgmap v5387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:56.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:56 smithi153 ceph-mon[36569]: pgmap v5388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:56.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:56 smithi104 ceph-mon[36624]: pgmap v5388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:56.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:56 smithi081 ceph-mon[31453]: pgmap v5388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:58.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:53:58 smithi153 ceph-mon[36569]: pgmap v5389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:58.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:53:58 smithi104 ceph-mon[36624]: pgmap v5389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:58.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:53:58 smithi081 ceph-mon[31453]: pgmap v5389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:53:59.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:00.014+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:59.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:00.014+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:53:59.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:00.014+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:54:00.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:00 smithi153 ceph-mon[36569]: pgmap v5390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:00.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:00 smithi104 ceph-mon[36624]: pgmap v5390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:00.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:00 smithi081 ceph-mon[31453]: pgmap v5390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:02.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:02 smithi153 ceph-mon[36569]: pgmap v5391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:02.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:02 smithi104 ceph-mon[36624]: pgmap v5391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:02.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:02 smithi081 ceph-mon[31453]: pgmap v5391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:04.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:04 smithi153 ceph-mon[36569]: pgmap v5392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:04.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:04 smithi104 ceph-mon[36624]: pgmap v5392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:04.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:04 smithi081 ceph-mon[31453]: pgmap v5392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:06.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:06 smithi153 ceph-mon[36569]: pgmap v5393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:06.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:06 smithi104 ceph-mon[36624]: pgmap v5393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:06.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:06 smithi081 ceph-mon[31453]: pgmap v5393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:08.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:08 smithi153 ceph-mon[36569]: pgmap v5394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:08.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:08 smithi104 ceph-mon[36624]: pgmap v5394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:08.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:08 smithi081 ceph-mon[31453]: pgmap v5394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:10.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:10 smithi153 ceph-mon[36569]: pgmap v5395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:10.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:10 smithi104 ceph-mon[36624]: pgmap v5395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:10.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:10 smithi081 ceph-mon[31453]: pgmap v5395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:12.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:12 smithi153 ceph-mon[36569]: pgmap v5396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:12.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:12 smithi104 ceph-mon[36624]: pgmap v5396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:12.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:12 smithi081 ceph-mon[31453]: pgmap v5396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:13.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:13 smithi104 ceph-mon[36624]: pgmap v5397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:13.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:13 smithi081 ceph-mon[31453]: pgmap v5397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:14.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:13 smithi153 ceph-mon[36569]: pgmap v5397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:15.031+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:54:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:15.031+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:54:15.011 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:15.031+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:54:16.740 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:16 smithi153 ceph-mon[36569]: pgmap v5398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:16.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:16 smithi104 ceph-mon[36624]: pgmap v5398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:16.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:16 smithi081 ceph-mon[31453]: pgmap v5398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:18.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:18 smithi153 ceph-mon[36569]: pgmap v5399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:18 smithi104 ceph-mon[36624]: pgmap v5399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:18.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:18 smithi081 ceph-mon[31453]: pgmap v5399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:20.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:20 smithi153 ceph-mon[36569]: pgmap v5400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:20.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:20 smithi104 ceph-mon[36624]: pgmap v5400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:20.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:20 smithi081 ceph-mon[31453]: pgmap v5400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:22.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:22 smithi153 ceph-mon[36569]: pgmap v5401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:22.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:22 smithi104 ceph-mon[36624]: pgmap v5401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:22.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:22 smithi081 ceph-mon[31453]: pgmap v5401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:24.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:24 smithi153 ceph-mon[36569]: pgmap v5402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:24.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:54:24.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:24 smithi104 ceph-mon[36624]: pgmap v5402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:24.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:54:24.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:24 smithi081 ceph-mon[31453]: pgmap v5402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:24.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:54:25.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:54:25.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:54:25.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:54:25.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:54:25.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:54:25.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:54:25.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:54:25.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:54:25.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:54:26.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:26 smithi153 ceph-mon[36569]: pgmap v5403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:26.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:26 smithi153 ceph-mon[36569]: pgmap v5404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:26.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:26 smithi104 ceph-mon[36624]: pgmap v5403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:26.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:26 smithi104 ceph-mon[36624]: pgmap v5404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:26.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:26 smithi081 ceph-mon[31453]: pgmap v5403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:26.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:26 smithi081 ceph-mon[31453]: pgmap v5404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:27.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:27 smithi104 ceph-mon[36624]: pgmap v5405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:27.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:27 smithi081 ceph-mon[31453]: pgmap v5405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:28.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:27 smithi153 ceph-mon[36569]: pgmap v5405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:30.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:30.046+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:54:30.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:30.046+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:54:30.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:30.046+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:54:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:29 smithi153 ceph-mon[36569]: pgmap v5406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:29 smithi104 ceph-mon[36624]: pgmap v5406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:30.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:29 smithi081 ceph-mon[31453]: pgmap v5406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:32.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:31 smithi153 ceph-mon[36569]: pgmap v5407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:32.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:31 smithi104 ceph-mon[36624]: pgmap v5407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:32.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:31 smithi081 ceph-mon[31453]: pgmap v5407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:34.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:33 smithi153 ceph-mon[36569]: pgmap v5408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:34.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:33 smithi104 ceph-mon[36624]: pgmap v5408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:34.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:33 smithi081 ceph-mon[31453]: pgmap v5408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:36.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:35 smithi153 ceph-mon[36569]: pgmap v5409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:36.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:35 smithi104 ceph-mon[36624]: pgmap v5409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:36.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:35 smithi081 ceph-mon[31453]: pgmap v5409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:38.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:37 smithi153 ceph-mon[36569]: pgmap v5410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:38.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:37 smithi104 ceph-mon[36624]: pgmap v5410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:38.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:37 smithi081 ceph-mon[31453]: pgmap v5410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:40.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:39 smithi153 ceph-mon[36569]: pgmap v5411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:40.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:39 smithi104 ceph-mon[36624]: pgmap v5411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:40.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:39 smithi081 ceph-mon[31453]: pgmap v5411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:42.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:42 smithi153 ceph-mon[36569]: pgmap v5412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:42.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:42 smithi104 ceph-mon[36624]: pgmap v5412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:42.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:42 smithi081 ceph-mon[31453]: pgmap v5412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:44.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:44 smithi153 ceph-mon[36569]: pgmap v5413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:44.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:44 smithi104 ceph-mon[36624]: pgmap v5413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:44.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:44 smithi081 ceph-mon[31453]: pgmap v5413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:45.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:45.049+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:54:45.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:45.049+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:54:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:54:45.049+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:54:46.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:46 smithi153 ceph-mon[36569]: pgmap v5414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:46.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:46 smithi104 ceph-mon[36624]: pgmap v5414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:46.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:46 smithi081 ceph-mon[31453]: pgmap v5414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:48.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:48 smithi153 ceph-mon[36569]: pgmap v5415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:48.307 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:48 smithi081 ceph-mon[31453]: pgmap v5415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:48.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:48 smithi104 ceph-mon[36624]: pgmap v5415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:50.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:50 smithi153 ceph-mon[36569]: pgmap v5416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:50.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:50 smithi104 ceph-mon[36624]: pgmap v5416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:50.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:50 smithi081 ceph-mon[31453]: pgmap v5416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:52.006 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:52 smithi104 ceph-mon[36624]: pgmap v5417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:52.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:52 smithi153 ceph-mon[36569]: pgmap v5417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:52.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:52 smithi081 ceph-mon[31453]: pgmap v5417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:54.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:54 smithi153 ceph-mon[36569]: pgmap v5418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:54.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:54 smithi104 ceph-mon[36624]: pgmap v5418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:54.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:54 smithi081 ceph-mon[31453]: pgmap v5418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:56.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:56 smithi153 ceph-mon[36569]: pgmap v5419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:56.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:56 smithi104 ceph-mon[36624]: pgmap v5419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:56.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:56 smithi081 ceph-mon[31453]: pgmap v5419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:58.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:54:58 smithi153 ceph-mon[36569]: pgmap v5420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:54:58 smithi104 ceph-mon[36624]: pgmap v5420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:54:58.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:54:58 smithi081 ceph-mon[31453]: pgmap v5420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:00.064+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:55:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:00.064+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:55:00.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:00.064+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:55:00.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:00 smithi153 ceph-mon[36569]: pgmap v5421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:00.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:00 smithi104 ceph-mon[36624]: pgmap v5421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:00.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:00 smithi081 ceph-mon[31453]: pgmap v5421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:02.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:02 smithi153 ceph-mon[36569]: pgmap v5422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:02.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:02 smithi104 ceph-mon[36624]: pgmap v5422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:02.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:02 smithi081 ceph-mon[31453]: pgmap v5422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:04.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:04 smithi153 ceph-mon[36569]: pgmap v5423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:04.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:04 smithi104 ceph-mon[36624]: pgmap v5423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:04.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:04 smithi081 ceph-mon[31453]: pgmap v5423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:06.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:06 smithi104 ceph-mon[36624]: pgmap v5424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:06.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:06 smithi081 ceph-mon[31453]: pgmap v5424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:06.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:06 smithi153 ceph-mon[36569]: pgmap v5424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:08.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:08 smithi104 ceph-mon[36624]: pgmap v5425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:08.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:08 smithi081 ceph-mon[31453]: pgmap v5425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:08.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:08 smithi153 ceph-mon[36569]: pgmap v5425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:10.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:10 smithi104 ceph-mon[36624]: pgmap v5426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:10.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:10 smithi081 ceph-mon[31453]: pgmap v5426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:10.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:10 smithi153 ceph-mon[36569]: pgmap v5426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:12.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:12 smithi104 ceph-mon[36624]: pgmap v5427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:12.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:12 smithi081 ceph-mon[31453]: pgmap v5427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:12.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:12 smithi153 ceph-mon[36569]: pgmap v5427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:14.273+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:55:14.253 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:14.273+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:55:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:14.274+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:55:14.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:14.274+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:55:14.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:14 smithi081 ceph-mon[31453]: pgmap v5428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:14.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:14 smithi153 ceph-mon[36569]: pgmap v5428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:14.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:14 smithi104 ceph-mon[36624]: pgmap v5428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:15.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:15.080+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:55:15.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:15.080+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:55:15.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:15.080+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:55:16.339 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:16 smithi153 ceph-mon[36569]: pgmap v5429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:16.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:16 smithi081 ceph-mon[31453]: pgmap v5429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:16.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:16 smithi104 ceph-mon[36624]: pgmap v5429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:18.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:18 smithi081 ceph-mon[31453]: pgmap v5430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:18.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:18 smithi153 ceph-mon[36569]: pgmap v5430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:18.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:18 smithi104 ceph-mon[36624]: pgmap v5430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:20.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:20 smithi081 ceph-mon[31453]: pgmap v5431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:20.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:20 smithi153 ceph-mon[36569]: pgmap v5431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:20.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:20 smithi104 ceph-mon[36624]: pgmap v5431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:22.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:22 smithi081 ceph-mon[31453]: pgmap v5432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:22.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:22 smithi153 ceph-mon[36569]: pgmap v5432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:22.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:22 smithi104 ceph-mon[36624]: pgmap v5432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:24.364 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:24 smithi104 ceph-mon[36624]: pgmap v5433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:24.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:24 smithi081 ceph-mon[31453]: pgmap v5433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:24.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:24 smithi153 ceph-mon[36569]: pgmap v5433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:26.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:26 smithi081 ceph-mon[31453]: pgmap v5434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:26.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:55:26.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:26 smithi153 ceph-mon[36569]: pgmap v5434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:26.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:55:26.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:26 smithi104 ceph-mon[36624]: pgmap v5434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:26.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:55:27.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:55:27.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:55:27.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:27 smithi081 ceph-mon[31453]: pgmap v5435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:27.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:27 smithi081 ceph-mon[31453]: pgmap v5436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:27.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:55:27.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:55:27.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:55:27.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:27 smithi153 ceph-mon[36569]: pgmap v5435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:27.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:27 smithi153 ceph-mon[36569]: pgmap v5436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:27.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:55:27.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:55:27.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:55:27.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:27 smithi104 ceph-mon[36624]: pgmap v5435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:27.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:27 smithi104 ceph-mon[36624]: pgmap v5436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:27.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:55:28.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:28 smithi104 ceph-mon[36624]: pgmap v5437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:28.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:28 smithi081 ceph-mon[31453]: pgmap v5437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:29.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:28 smithi153 ceph-mon[36569]: pgmap v5437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:30.096+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:55:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:30.096+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:55:30.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:30.096+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:55:30.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:30 smithi104 ceph-mon[36624]: pgmap v5438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:30.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:30 smithi081 ceph-mon[31453]: pgmap v5438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:31.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:30 smithi153 ceph-mon[36569]: pgmap v5438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:32 smithi104 ceph-mon[36624]: pgmap v5439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:32 smithi081 ceph-mon[31453]: pgmap v5439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:33.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:32 smithi153 ceph-mon[36569]: pgmap v5439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:34.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:34 smithi104 ceph-mon[36624]: pgmap v5440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:34.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:34 smithi081 ceph-mon[31453]: pgmap v5440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:35.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:34 smithi153 ceph-mon[36569]: pgmap v5440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:36.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:36 smithi104 ceph-mon[36624]: pgmap v5441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:36.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:36 smithi081 ceph-mon[31453]: pgmap v5441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:37.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:36 smithi153 ceph-mon[36569]: pgmap v5441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:38.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:38 smithi104 ceph-mon[36624]: pgmap v5442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:38.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:38 smithi081 ceph-mon[31453]: pgmap v5442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:39.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:38 smithi153 ceph-mon[36569]: pgmap v5442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:40.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:40 smithi104 ceph-mon[36624]: pgmap v5443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:40.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:40 smithi081 ceph-mon[31453]: pgmap v5443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:41.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:40 smithi153 ceph-mon[36569]: pgmap v5443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:42.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:42 smithi104 ceph-mon[36624]: pgmap v5444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:42.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:42 smithi081 ceph-mon[31453]: pgmap v5444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:43.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:42 smithi153 ceph-mon[36569]: pgmap v5444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:44.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:44 smithi104 ceph-mon[36624]: pgmap v5445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:44.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:44 smithi081 ceph-mon[31453]: pgmap v5445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:45.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:44 smithi153 ceph-mon[36569]: pgmap v5445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:45.104+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:55:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:45.104+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:55:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:55:45.104+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:55:46.838 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:46 smithi153 ceph-mon[36569]: pgmap v5446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:46.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:46 smithi081 ceph-mon[31453]: pgmap v5446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:47.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:46 smithi104 ceph-mon[36624]: pgmap v5446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:48.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:48 smithi104 ceph-mon[36624]: pgmap v5447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:48.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:48 smithi081 ceph-mon[31453]: pgmap v5447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:49.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:48 smithi153 ceph-mon[36569]: pgmap v5447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:50.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:50 smithi104 ceph-mon[36624]: pgmap v5448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:50.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:50 smithi081 ceph-mon[31453]: pgmap v5448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:51.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:50 smithi153 ceph-mon[36569]: pgmap v5448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:52.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:52 smithi081 ceph-mon[31453]: pgmap v5449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:53.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:52 smithi153 ceph-mon[36569]: pgmap v5449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:53.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:52 smithi104 ceph-mon[36624]: pgmap v5449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:54.864 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:54 smithi104 ceph-mon[36624]: pgmap v5450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:54.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:54 smithi081 ceph-mon[31453]: pgmap v5450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:55.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:54 smithi153 ceph-mon[36569]: pgmap v5450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:56.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:56 smithi081 ceph-mon[31453]: pgmap v5451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:57.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:56 smithi153 ceph-mon[36569]: pgmap v5451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:57.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:56 smithi104 ceph-mon[36624]: pgmap v5451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:58.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:55:58 smithi081 ceph-mon[31453]: pgmap v5452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:59.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:55:58 smithi153 ceph-mon[36569]: pgmap v5452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:55:59.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:55:58 smithi104 ceph-mon[36624]: pgmap v5452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:00.119+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:56:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:00.119+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:56:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:00.120+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:56:00.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:00 smithi081 ceph-mon[31453]: pgmap v5453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:01.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:00 smithi153 ceph-mon[36569]: pgmap v5453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:01.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:00 smithi104 ceph-mon[36624]: pgmap v5453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:02.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:02 smithi081 ceph-mon[31453]: pgmap v5454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:03.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:02 smithi153 ceph-mon[36569]: pgmap v5454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:03.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:02 smithi104 ceph-mon[36624]: pgmap v5454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:04.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:04 smithi081 ceph-mon[31453]: pgmap v5455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:05.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:04 smithi153 ceph-mon[36569]: pgmap v5455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:05.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:04 smithi104 ceph-mon[36624]: pgmap v5455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:06.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:06 smithi081 ceph-mon[31453]: pgmap v5456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:07.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:06 smithi153 ceph-mon[36569]: pgmap v5456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:07.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:06 smithi104 ceph-mon[36624]: pgmap v5456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:08.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:08 smithi081 ceph-mon[31453]: pgmap v5457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:09.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:08 smithi153 ceph-mon[36569]: pgmap v5457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:09.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:08 smithi104 ceph-mon[36624]: pgmap v5457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:10.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:10 smithi081 ceph-mon[31453]: pgmap v5458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:11.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:10 smithi153 ceph-mon[36569]: pgmap v5458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:11.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:10 smithi104 ceph-mon[36624]: pgmap v5458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:12.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:12 smithi081 ceph-mon[31453]: pgmap v5459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:13.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:12 smithi153 ceph-mon[36569]: pgmap v5459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:13.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:12 smithi104 ceph-mon[36624]: pgmap v5459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:14.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:14 smithi081 ceph-mon[31453]: pgmap v5460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:15.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:14 smithi153 ceph-mon[36569]: pgmap v5460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:15.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:14 smithi104 ceph-mon[36624]: pgmap v5460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:15.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:15.132+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:56:15.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:15.132+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:56:15.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:15.132+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:56:16.838 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:16 smithi153 ceph-mon[36569]: pgmap v5461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:16.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:16 smithi081 ceph-mon[31453]: pgmap v5461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:17.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:16 smithi104 ceph-mon[36624]: pgmap v5461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:18.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:18 smithi081 ceph-mon[31453]: pgmap v5462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:19.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:18 smithi153 ceph-mon[36569]: pgmap v5462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:19.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:18 smithi104 ceph-mon[36624]: pgmap v5462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:20.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:20 smithi081 ceph-mon[31453]: pgmap v5463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:21.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:20 smithi153 ceph-mon[36569]: pgmap v5463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:21.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:20 smithi104 ceph-mon[36624]: pgmap v5463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:22.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:22 smithi081 ceph-mon[31453]: pgmap v5464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:23.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:22 smithi153 ceph-mon[36569]: pgmap v5464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:23.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:22 smithi104 ceph-mon[36624]: pgmap v5464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:24.864 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:24 smithi104 ceph-mon[36624]: pgmap v5465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:24.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:24 smithi081 ceph-mon[31453]: pgmap v5465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:25.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:24 smithi153 ceph-mon[36569]: pgmap v5465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:26.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:26 smithi081 ceph-mon[31453]: pgmap v5466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:27.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:26 smithi153 ceph-mon[36569]: pgmap v5466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:27.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:26 smithi104 ceph-mon[36624]: pgmap v5466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:27.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:56:28.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:56:28.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:56:28.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:56:28.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:56:28.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:28 smithi081 ceph-mon[31453]: pgmap v5467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:28.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:28 smithi081 ceph-mon[31453]: pgmap v5468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:28.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:56:29.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:56:29.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:56:29.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:28 smithi153 ceph-mon[36569]: pgmap v5467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:29.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:28 smithi153 ceph-mon[36569]: pgmap v5468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:29.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:56:29.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:56:29.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:56:29.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:28 smithi104 ceph-mon[36624]: pgmap v5467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:29.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:28 smithi104 ceph-mon[36624]: pgmap v5468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:29.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:56:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:30.141+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:56:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:30.141+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:56:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:30.141+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:56:31.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:31 smithi081 ceph-mon[31453]: pgmap v5469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:31.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:31 smithi153 ceph-mon[36569]: pgmap v5469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:31.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:31 smithi104 ceph-mon[36624]: pgmap v5469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:33.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:33 smithi081 ceph-mon[31453]: pgmap v5470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:33.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:33 smithi153 ceph-mon[36569]: pgmap v5470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:33.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:33 smithi104 ceph-mon[36624]: pgmap v5470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:35.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:35 smithi081 ceph-mon[31453]: pgmap v5471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:35.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:35 smithi153 ceph-mon[36569]: pgmap v5471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:35.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:35 smithi104 ceph-mon[36624]: pgmap v5471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:37.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:37 smithi081 ceph-mon[31453]: pgmap v5472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:37.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:37 smithi153 ceph-mon[36569]: pgmap v5472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:37.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:37 smithi104 ceph-mon[36624]: pgmap v5472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:39.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:39 smithi081 ceph-mon[31453]: pgmap v5473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:39.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:39 smithi153 ceph-mon[36569]: pgmap v5473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:39.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:39 smithi104 ceph-mon[36624]: pgmap v5473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:41.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:41 smithi081 ceph-mon[31453]: pgmap v5474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:41.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:41 smithi153 ceph-mon[36569]: pgmap v5474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:41.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:41 smithi104 ceph-mon[36624]: pgmap v5474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:43.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:43 smithi081 ceph-mon[31453]: pgmap v5475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:43.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:43 smithi153 ceph-mon[36569]: pgmap v5475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:43.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:43 smithi104 ceph-mon[36624]: pgmap v5475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:45.153+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:56:45.133 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:45.153+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:56:45.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:56:45.153+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:56:45.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:45 smithi081 ceph-mon[31453]: pgmap v5476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:45.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:45 smithi153 ceph-mon[36569]: pgmap v5476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:45.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:45 smithi104 ceph-mon[36624]: pgmap v5476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:47.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:47 smithi081 ceph-mon[31453]: pgmap v5477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:47.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:47 smithi153 ceph-mon[36569]: pgmap v5477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:47.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:47 smithi104 ceph-mon[36624]: pgmap v5477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:49.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:49 smithi081 ceph-mon[31453]: pgmap v5478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:49.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:49 smithi153 ceph-mon[36569]: pgmap v5478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:49.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:49 smithi104 ceph-mon[36624]: pgmap v5478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:51.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:51 smithi081 ceph-mon[31453]: pgmap v5479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:51.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:51 smithi153 ceph-mon[36569]: pgmap v5479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:51.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:51 smithi104 ceph-mon[36624]: pgmap v5479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:53 smithi081 ceph-mon[31453]: pgmap v5480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:53.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:53 smithi153 ceph-mon[36569]: pgmap v5480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:53 smithi104 ceph-mon[36624]: pgmap v5480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:55.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:55 smithi153 ceph-mon[36569]: pgmap v5481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:55.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:55 smithi104 ceph-mon[36624]: pgmap v5481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:55.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:55 smithi081 ceph-mon[31453]: pgmap v5481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:57.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:57 smithi153 ceph-mon[36569]: pgmap v5482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:57.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:57 smithi104 ceph-mon[36624]: pgmap v5482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:57.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:57 smithi081 ceph-mon[31453]: pgmap v5482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:59.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:56:59 smithi153 ceph-mon[36569]: pgmap v5483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:59.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:56:59 smithi104 ceph-mon[36624]: pgmap v5483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:56:59.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:56:59 smithi081 ceph-mon[31453]: pgmap v5483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:00.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:00.166+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:57:00.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:00.166+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:57:00.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:00.166+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:57:01.439 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:01 smithi081 ceph-mon[31453]: pgmap v5484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:01.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:01 smithi153 ceph-mon[36569]: pgmap v5484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:01.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:01 smithi104 ceph-mon[36624]: pgmap v5484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:03.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:03 smithi153 ceph-mon[36569]: pgmap v5485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:03.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:03 smithi104 ceph-mon[36624]: pgmap v5485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:03.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:03 smithi081 ceph-mon[31453]: pgmap v5485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:05.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:05 smithi153 ceph-mon[36569]: pgmap v5486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:05.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:05 smithi104 ceph-mon[36624]: pgmap v5486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:05.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:05 smithi081 ceph-mon[31453]: pgmap v5486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:07.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:07 smithi153 ceph-mon[36569]: pgmap v5487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:07.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:07 smithi104 ceph-mon[36624]: pgmap v5487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:07.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:07 smithi081 ceph-mon[31453]: pgmap v5487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:09.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:09 smithi153 ceph-mon[36569]: pgmap v5488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:09.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:09 smithi104 ceph-mon[36624]: pgmap v5488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:09.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:09 smithi081 ceph-mon[31453]: pgmap v5488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:11.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:11 smithi153 ceph-mon[36569]: pgmap v5489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:11.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:11 smithi104 ceph-mon[36624]: pgmap v5489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:11.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:11 smithi081 ceph-mon[31453]: pgmap v5489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:13.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:13 smithi153 ceph-mon[36569]: pgmap v5490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:13.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:13 smithi104 ceph-mon[36624]: pgmap v5490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:13.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:13 smithi081 ceph-mon[31453]: pgmap v5490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:15.176+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:57:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:15.176+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:57:15.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:15.176+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:57:15.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:15 smithi153 ceph-mon[36569]: pgmap v5491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:15.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:15 smithi104 ceph-mon[36624]: pgmap v5491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:15.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:15 smithi081 ceph-mon[31453]: pgmap v5491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:17.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:17 smithi153 ceph-mon[36569]: pgmap v5492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:17.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:17 smithi104 ceph-mon[36624]: pgmap v5492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:17.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:17 smithi081 ceph-mon[31453]: pgmap v5492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:19.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:19 smithi153 ceph-mon[36569]: pgmap v5493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:19.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:19 smithi104 ceph-mon[36624]: pgmap v5493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:19.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:19 smithi081 ceph-mon[31453]: pgmap v5493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:21.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:21 smithi153 ceph-mon[36569]: pgmap v5494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:21.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:21 smithi104 ceph-mon[36624]: pgmap v5494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:21.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:21 smithi081 ceph-mon[31453]: pgmap v5494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:23.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:23 smithi153 ceph-mon[36569]: pgmap v5495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:23.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:23 smithi104 ceph-mon[36624]: pgmap v5495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:23.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:23 smithi081 ceph-mon[31453]: pgmap v5495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:25.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:25 smithi153 ceph-mon[36569]: pgmap v5496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:25.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:25 smithi104 ceph-mon[36624]: pgmap v5496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:25.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:25 smithi081 ceph-mon[31453]: pgmap v5496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:27.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:27 smithi153 ceph-mon[36569]: pgmap v5497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:27.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:27 smithi104 ceph-mon[36624]: pgmap v5497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:27.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:27 smithi081 ceph-mon[31453]: pgmap v5497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:29.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:29 smithi153 ceph-mon[36569]: pgmap v5498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:29.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:57:29.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:29 smithi104 ceph-mon[36624]: pgmap v5498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:29.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:57:29.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:29 smithi081 ceph-mon[31453]: pgmap v5498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:29.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:57:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:30.183+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:57:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:30.183+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:57:30.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:30.183+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:57:30.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:57:30.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:57:30.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:57:30.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:57:30.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:57:30.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:57:30.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:57:30.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:57:30.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:57:31.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:31 smithi153 ceph-mon[36569]: pgmap v5499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:31.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:31 smithi153 ceph-mon[36569]: pgmap v5500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:31.537 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:31 smithi081 ceph-mon[31453]: pgmap v5499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:31.537 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:31 smithi081 ceph-mon[31453]: pgmap v5500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:31.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:31 smithi104 ceph-mon[36624]: pgmap v5499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:31.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:31 smithi104 ceph-mon[36624]: pgmap v5500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:33.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:33 smithi153 ceph-mon[36569]: pgmap v5501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:33.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:33 smithi104 ceph-mon[36624]: pgmap v5501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:33.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:33 smithi081 ceph-mon[31453]: pgmap v5501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:35.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:35 smithi153 ceph-mon[36569]: pgmap v5502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:35.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:35 smithi104 ceph-mon[36624]: pgmap v5502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:35.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:35 smithi081 ceph-mon[31453]: pgmap v5502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:37.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:37 smithi153 ceph-mon[36569]: pgmap v5503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:37.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:37 smithi104 ceph-mon[36624]: pgmap v5503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:37.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:37 smithi081 ceph-mon[31453]: pgmap v5503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:39.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:39 smithi153 ceph-mon[36569]: pgmap v5504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:39.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:39 smithi104 ceph-mon[36624]: pgmap v5504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:39.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:39 smithi081 ceph-mon[31453]: pgmap v5504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:41.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:41 smithi153 ceph-mon[36569]: pgmap v5505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:41.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:41 smithi104 ceph-mon[36624]: pgmap v5505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:41.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:41 smithi081 ceph-mon[31453]: pgmap v5505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:43.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:43 smithi153 ceph-mon[36569]: pgmap v5506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:43.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:43 smithi104 ceph-mon[36624]: pgmap v5506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:43.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:43 smithi081 ceph-mon[31453]: pgmap v5506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:45.196+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:57:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:45.196+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:57:45.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:57:45.196+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:57:45.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:45 smithi153 ceph-mon[36569]: pgmap v5507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:45.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:45 smithi104 ceph-mon[36624]: pgmap v5507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:45.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:45 smithi081 ceph-mon[31453]: pgmap v5507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:47.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:47 smithi153 ceph-mon[36569]: pgmap v5508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:47.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:47 smithi104 ceph-mon[36624]: pgmap v5508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:47.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:47 smithi081 ceph-mon[31453]: pgmap v5508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:49.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:49 smithi104 ceph-mon[36624]: pgmap v5509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:49.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:49 smithi081 ceph-mon[31453]: pgmap v5509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:49.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:49 smithi153 ceph-mon[36569]: pgmap v5509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:51.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:51 smithi104 ceph-mon[36624]: pgmap v5510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:51.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:51 smithi081 ceph-mon[31453]: pgmap v5510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:51.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:51 smithi153 ceph-mon[36569]: pgmap v5510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:53 smithi104 ceph-mon[36624]: pgmap v5511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:53.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:53 smithi081 ceph-mon[31453]: pgmap v5511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:53.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:53 smithi153 ceph-mon[36569]: pgmap v5511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:55.438 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:55 smithi081 ceph-mon[31453]: pgmap v5512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:55.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:55 smithi104 ceph-mon[36624]: pgmap v5512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:55.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:55 smithi153 ceph-mon[36569]: pgmap v5512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:57.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:57 smithi104 ceph-mon[36624]: pgmap v5513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:57.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:57 smithi081 ceph-mon[31453]: pgmap v5513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:57.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:57 smithi153 ceph-mon[36569]: pgmap v5513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:59.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:57:59 smithi081 ceph-mon[31453]: pgmap v5514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:59.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:57:59 smithi153 ceph-mon[36569]: pgmap v5514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:57:59.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:57:59 smithi104 ceph-mon[36624]: pgmap v5514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:00.206+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:58:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:00.206+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:58:00.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:00.206+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:58:01.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:01 smithi081 ceph-mon[31453]: pgmap v5515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:01.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:01 smithi153 ceph-mon[36569]: pgmap v5515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:01.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:01 smithi104 ceph-mon[36624]: pgmap v5515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:03.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:03 smithi081 ceph-mon[31453]: pgmap v5516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:03.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:03 smithi153 ceph-mon[36569]: pgmap v5516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:03.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:03 smithi104 ceph-mon[36624]: pgmap v5516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:05.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:05 smithi081 ceph-mon[31453]: pgmap v5517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:05.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:05 smithi153 ceph-mon[36569]: pgmap v5517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:05.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:05 smithi104 ceph-mon[36624]: pgmap v5517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:07.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:07 smithi081 ceph-mon[31453]: pgmap v5518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:07.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:07 smithi153 ceph-mon[36569]: pgmap v5518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:07.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:07 smithi104 ceph-mon[36624]: pgmap v5518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:09.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:09 smithi081 ceph-mon[31453]: pgmap v5519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:09.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:09 smithi153 ceph-mon[36569]: pgmap v5519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:09.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:09 smithi104 ceph-mon[36624]: pgmap v5519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:11.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:11 smithi081 ceph-mon[31453]: pgmap v5520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:11.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:11 smithi153 ceph-mon[36569]: pgmap v5520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:11.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:11 smithi104 ceph-mon[36624]: pgmap v5520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:13.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:13 smithi081 ceph-mon[31453]: pgmap v5521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:13.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:13 smithi153 ceph-mon[36569]: pgmap v5521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:13.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:13 smithi104 ceph-mon[36624]: pgmap v5521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:14.304+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:58:14.283 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:14.304+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:58:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:14.304+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-12T23:58:14.284 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:14.304+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-12T23:58:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:15.212+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:58:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:15.212+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:58:15.191 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:15.212+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:58:15.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:15 smithi081 ceph-mon[31453]: pgmap v5522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:15.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:15 smithi153 ceph-mon[36569]: pgmap v5522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:15.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:15 smithi104 ceph-mon[36624]: pgmap v5522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:17.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:17 smithi081 ceph-mon[31453]: pgmap v5523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:17.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:17 smithi153 ceph-mon[36569]: pgmap v5523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:17.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:17 smithi104 ceph-mon[36624]: pgmap v5523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:19.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:19 smithi081 ceph-mon[31453]: pgmap v5524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:19.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:19 smithi153 ceph-mon[36569]: pgmap v5524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:19.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:19 smithi104 ceph-mon[36624]: pgmap v5524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:21.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:21 smithi081 ceph-mon[31453]: pgmap v5525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:21.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:21 smithi153 ceph-mon[36569]: pgmap v5525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:21.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:21 smithi104 ceph-mon[36624]: pgmap v5525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:23.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:23 smithi081 ceph-mon[31453]: pgmap v5526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:23.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:23 smithi153 ceph-mon[36569]: pgmap v5526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:23.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:23 smithi104 ceph-mon[36624]: pgmap v5526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:25.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:25 smithi081 ceph-mon[31453]: pgmap v5527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:25.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:25 smithi153 ceph-mon[36569]: pgmap v5527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:25.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:25 smithi104 ceph-mon[36624]: pgmap v5527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:27.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:27 smithi153 ceph-mon[36569]: pgmap v5528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:27.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:27 smithi104 ceph-mon[36624]: pgmap v5528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:27.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:27 smithi081 ceph-mon[31453]: pgmap v5528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:29.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:29 smithi153 ceph-mon[36569]: pgmap v5529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:29.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:29 smithi104 ceph-mon[36624]: pgmap v5529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:29.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:29 smithi081 ceph-mon[31453]: pgmap v5529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:30.206 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:30.227+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:58:30.206 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:30.227+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:58:30.206 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:30.227+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:58:30.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:58:30.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:58:30.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: pgmap v5530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:58:31.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:58:31.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:58:31.787 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: pgmap v5530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:58:31.788 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:58:31.789 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:58:31.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: pgmap v5530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:31.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:58:31.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:58:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:32 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:58:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:32 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:58:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:32 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:58:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:32 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:58:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:32 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:58:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:32 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:58:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:32 smithi153 ceph-mon[36569]: pgmap v5531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:32 smithi153 ceph-mon[36569]: pgmap v5532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:32 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:58:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:32 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:58:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:32 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:58:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:32 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:58:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:32 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:58:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:32 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:58:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:32 smithi104 ceph-mon[36624]: pgmap v5531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:32.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:32 smithi104 ceph-mon[36624]: pgmap v5532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:32.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:32 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-12T23:58:32.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:32 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-12T23:58:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:32 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-12T23:58:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:32 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-12T23:58:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:32 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-12T23:58:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:32 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-12T23:58:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:32 smithi081 ceph-mon[31453]: pgmap v5531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:32 smithi081 ceph-mon[31453]: pgmap v5532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:34.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:34 smithi153 ceph-mon[36569]: pgmap v5533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:34.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:34 smithi104 ceph-mon[36624]: pgmap v5533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:34.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:34 smithi081 ceph-mon[31453]: pgmap v5533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:36.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:36 smithi153 ceph-mon[36569]: pgmap v5534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:36.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:36 smithi104 ceph-mon[36624]: pgmap v5534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:36.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:36 smithi081 ceph-mon[31453]: pgmap v5534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:38.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:38 smithi153 ceph-mon[36569]: pgmap v5535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:38.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:38 smithi104 ceph-mon[36624]: pgmap v5535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:38.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:38 smithi081 ceph-mon[31453]: pgmap v5535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:40.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:40 smithi153 ceph-mon[36569]: pgmap v5536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:40.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:40 smithi104 ceph-mon[36624]: pgmap v5536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:40.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:40 smithi081 ceph-mon[31453]: pgmap v5536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:42.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:42 smithi153 ceph-mon[36569]: pgmap v5537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:42.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:42 smithi104 ceph-mon[36624]: pgmap v5537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:42.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:42 smithi081 ceph-mon[31453]: pgmap v5537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:44.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:44 smithi153 ceph-mon[36569]: pgmap v5538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:44.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:44 smithi104 ceph-mon[36624]: pgmap v5538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:44.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:44 smithi081 ceph-mon[31453]: pgmap v5538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:45.242+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:58:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:45.242+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:58:45.222 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:58:45.242+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:58:46.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:46 smithi153 ceph-mon[36569]: pgmap v5539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:46.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:46 smithi104 ceph-mon[36624]: pgmap v5539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:46.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:46 smithi081 ceph-mon[31453]: pgmap v5539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:48.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:48 smithi153 ceph-mon[36569]: pgmap v5540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:48.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:48 smithi104 ceph-mon[36624]: pgmap v5540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:48.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:48 smithi081 ceph-mon[31453]: pgmap v5540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:50.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:50 smithi153 ceph-mon[36569]: pgmap v5541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:50.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:50 smithi104 ceph-mon[36624]: pgmap v5541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:50.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:50 smithi081 ceph-mon[31453]: pgmap v5541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:52.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:52 smithi153 ceph-mon[36569]: pgmap v5542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:52.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:52 smithi104 ceph-mon[36624]: pgmap v5542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:52 smithi081 ceph-mon[31453]: pgmap v5542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:54.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:54 smithi153 ceph-mon[36569]: pgmap v5543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:54.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:54 smithi104 ceph-mon[36624]: pgmap v5543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:54.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:54 smithi081 ceph-mon[31453]: pgmap v5543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:56.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:56 smithi153 ceph-mon[36569]: pgmap v5544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:56.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:56 smithi104 ceph-mon[36624]: pgmap v5544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:56.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:56 smithi081 ceph-mon[31453]: pgmap v5544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:58.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:58:58 smithi153 ceph-mon[36569]: pgmap v5545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:58.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:58:58 smithi104 ceph-mon[36624]: pgmap v5545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:58:58.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:58:58 smithi081 ceph-mon[31453]: pgmap v5545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:00.227 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:00.248+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:59:00.227 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:00.248+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:59:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:00.248+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:59:00.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:00 smithi153 ceph-mon[36569]: pgmap v5546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:00.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:00 smithi104 ceph-mon[36624]: pgmap v5546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:00.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:00 smithi081 ceph-mon[31453]: pgmap v5546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:02.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:02 smithi153 ceph-mon[36569]: pgmap v5547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:02.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:02 smithi104 ceph-mon[36624]: pgmap v5547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:02.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:02 smithi081 ceph-mon[31453]: pgmap v5547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:04.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:04 smithi153 ceph-mon[36569]: pgmap v5548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:04.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:04 smithi104 ceph-mon[36624]: pgmap v5548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:04.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:04 smithi081 ceph-mon[31453]: pgmap v5548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:06.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:06 smithi153 ceph-mon[36569]: pgmap v5549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:06.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:06 smithi104 ceph-mon[36624]: pgmap v5549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:06.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:06 smithi081 ceph-mon[31453]: pgmap v5549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:08.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:08 smithi153 ceph-mon[36569]: pgmap v5550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:08.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:08 smithi104 ceph-mon[36624]: pgmap v5550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:08.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:08 smithi081 ceph-mon[31453]: pgmap v5550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:10.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:10 smithi153 ceph-mon[36569]: pgmap v5551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:10.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:10 smithi104 ceph-mon[36624]: pgmap v5551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:10.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:10 smithi081 ceph-mon[31453]: pgmap v5551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:12.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:12 smithi153 ceph-mon[36569]: pgmap v5552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:12.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:12 smithi104 ceph-mon[36624]: pgmap v5552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:12.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:12 smithi081 ceph-mon[31453]: pgmap v5552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:14.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:14 smithi153 ceph-mon[36569]: pgmap v5553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:14.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:14 smithi104 ceph-mon[36624]: pgmap v5553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:14.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:14 smithi081 ceph-mon[31453]: pgmap v5553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:15.258+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:59:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:15.258+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:59:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:15.258+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:59:16.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:16 smithi153 ceph-mon[36569]: pgmap v5554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:16.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:16 smithi104 ceph-mon[36624]: pgmap v5554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:16.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:16 smithi081 ceph-mon[31453]: pgmap v5554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:18.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:18 smithi153 ceph-mon[36569]: pgmap v5555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:18.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:18 smithi104 ceph-mon[36624]: pgmap v5555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:18.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:18 smithi081 ceph-mon[31453]: pgmap v5555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:20.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:20 smithi153 ceph-mon[36569]: pgmap v5556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:20.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:20 smithi104 ceph-mon[36624]: pgmap v5556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:20.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:20 smithi081 ceph-mon[31453]: pgmap v5556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:22.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:22 smithi104 ceph-mon[36624]: pgmap v5557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:22.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:22 smithi081 ceph-mon[31453]: pgmap v5557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:23.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:22 smithi153 ceph-mon[36569]: pgmap v5557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:24.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:24 smithi104 ceph-mon[36624]: pgmap v5558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:24.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:24 smithi081 ceph-mon[31453]: pgmap v5558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:25.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:24 smithi153 ceph-mon[36569]: pgmap v5558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:26.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:26 smithi104 ceph-mon[36624]: pgmap v5559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:26.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:26 smithi081 ceph-mon[31453]: pgmap v5559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:27.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:26 smithi153 ceph-mon[36569]: pgmap v5559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:28.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:28 smithi104 ceph-mon[36624]: pgmap v5560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:28.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:28 smithi081 ceph-mon[31453]: pgmap v5560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:29.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:28 smithi153 ceph-mon[36569]: pgmap v5560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:29.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:29 smithi081 ceph-mon[31453]: pgmap v5561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:30.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:29 smithi153 ceph-mon[36569]: pgmap v5561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:30.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:29 smithi104 ceph-mon[36624]: pgmap v5561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:30.260+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:59:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:30.260+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:59:30.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:30.260+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:59:32.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:32 smithi153 ceph-mon[36569]: pgmap v5562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:32.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:59:32.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:32 smithi104 ceph-mon[36624]: pgmap v5562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:32.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:59:32.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:32 smithi081 ceph-mon[31453]: pgmap v5562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:32.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-12T23:59:33.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:59:33.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:59:33.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:59:33.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:59:33.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:59:33.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:59:33.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-12T23:59:33.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-12T23:59:33.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-12T23:59:34.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:34 smithi153 ceph-mon[36569]: pgmap v5563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:34.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:34 smithi153 ceph-mon[36569]: pgmap v5564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:34.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:34 smithi104 ceph-mon[36624]: pgmap v5563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:34.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:34 smithi104 ceph-mon[36624]: pgmap v5564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:34.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:34 smithi081 ceph-mon[31453]: pgmap v5563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:34.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:34 smithi081 ceph-mon[31453]: pgmap v5564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:36.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:36 smithi153 ceph-mon[36569]: pgmap v5565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:36.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:36 smithi104 ceph-mon[36624]: pgmap v5565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:36.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:36 smithi081 ceph-mon[31453]: pgmap v5565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:38.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:38 smithi153 ceph-mon[36569]: pgmap v5566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:38.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:38 smithi104 ceph-mon[36624]: pgmap v5566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:38.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:38 smithi081 ceph-mon[31453]: pgmap v5566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:40.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:40 smithi153 ceph-mon[36569]: pgmap v5567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:40.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:40 smithi104 ceph-mon[36624]: pgmap v5567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:40.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:40 smithi081 ceph-mon[31453]: pgmap v5567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:42.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:42 smithi153 ceph-mon[36569]: pgmap v5568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:42.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:42 smithi104 ceph-mon[36624]: pgmap v5568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:42.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:42 smithi081 ceph-mon[31453]: pgmap v5568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:44.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:44 smithi153 ceph-mon[36569]: pgmap v5569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:44.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:44 smithi104 ceph-mon[36624]: pgmap v5569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:44.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:44 smithi081 ceph-mon[31453]: pgmap v5569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:45.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:45.276+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-12T23:59:45.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:45.276+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-12T23:59:45.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-12T23:59:45.276+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-12T23:59:46.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:46 smithi153 ceph-mon[36569]: pgmap v5570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:46.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:46 smithi104 ceph-mon[36624]: pgmap v5570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:46.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:46 smithi081 ceph-mon[31453]: pgmap v5570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:48.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:48 smithi153 ceph-mon[36569]: pgmap v5571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:48.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:48 smithi104 ceph-mon[36624]: pgmap v5571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:48.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:48 smithi081 ceph-mon[31453]: pgmap v5571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:50.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:50 smithi153 ceph-mon[36569]: pgmap v5572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:50.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:50 smithi104 ceph-mon[36624]: pgmap v5572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:50.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:50 smithi081 ceph-mon[31453]: pgmap v5572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:52.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:52 smithi153 ceph-mon[36569]: pgmap v5573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:52.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:52 smithi104 ceph-mon[36624]: pgmap v5573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:52 smithi081 ceph-mon[31453]: pgmap v5573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:54.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:54 smithi153 ceph-mon[36569]: pgmap v5574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:54.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:54 smithi104 ceph-mon[36624]: pgmap v5574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:54.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:54 smithi081 ceph-mon[31453]: pgmap v5574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:56.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:56 smithi153 ceph-mon[36569]: pgmap v5575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:56.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:56 smithi104 ceph-mon[36624]: pgmap v5575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:56.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:56 smithi081 ceph-mon[31453]: pgmap v5575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:58.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 12 23:59:58 smithi153 ceph-mon[36569]: pgmap v5576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:58.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 12 23:59:58 smithi104 ceph-mon[36624]: pgmap v5576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-12T23:59:58.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 12 23:59:58 smithi081 ceph-mon[31453]: pgmap v5576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:00.281+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:00:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:00.281+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:00:00.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:00.281+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:00:00.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:00 smithi153 ceph-mon[36569]: pgmap v5577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:00.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:00:00.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:00 smithi104 ceph-mon[36624]: pgmap v5577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:00.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:00:00.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:00 smithi081 ceph-mon[31453]: pgmap v5577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:00.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:00:02.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:02 smithi153 ceph-mon[36569]: pgmap v5578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:02.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:02 smithi104 ceph-mon[36624]: pgmap v5578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:02.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:02 smithi081 ceph-mon[31453]: pgmap v5578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:04.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:04 smithi153 ceph-mon[36569]: pgmap v5579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:04.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:04 smithi104 ceph-mon[36624]: pgmap v5579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:04.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:04 smithi081 ceph-mon[31453]: pgmap v5579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:06.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:06 smithi153 ceph-mon[36569]: pgmap v5580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:06.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:06 smithi104 ceph-mon[36624]: pgmap v5580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:06.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:06 smithi081 ceph-mon[31453]: pgmap v5580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:08.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:08 smithi153 ceph-mon[36569]: pgmap v5581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:08.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:08 smithi104 ceph-mon[36624]: pgmap v5581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:08.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:08 smithi081 ceph-mon[31453]: pgmap v5581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:10.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:10 smithi153 ceph-mon[36569]: pgmap v5582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:10.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:10 smithi104 ceph-mon[36624]: pgmap v5582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:10.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:10 smithi081 ceph-mon[31453]: pgmap v5582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:12.781 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:12 smithi081 ceph-mon[31453]: pgmap v5583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:12.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:12 smithi153 ceph-mon[36569]: pgmap v5583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:12.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:12 smithi104 ceph-mon[36624]: pgmap v5583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:14.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:14 smithi153 ceph-mon[36569]: pgmap v5584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:14.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:14 smithi104 ceph-mon[36624]: pgmap v5584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:14.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:14 smithi081 ceph-mon[31453]: pgmap v5584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:15.288+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:00:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:15.288+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:00:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:15.288+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:00:16.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:16 smithi104 ceph-mon[36624]: pgmap v5585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:16.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:16 smithi081 ceph-mon[31453]: pgmap v5585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:17.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:16 smithi153 ceph-mon[36569]: pgmap v5585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:18.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:18 smithi104 ceph-mon[36624]: pgmap v5586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:18.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:18 smithi081 ceph-mon[31453]: pgmap v5586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:18.915 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:18 smithi153 ceph-mon[36569]: pgmap v5586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:20.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:20 smithi104 ceph-mon[36624]: pgmap v5587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:20.846 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:20 smithi153 ceph-mon[36569]: pgmap v5587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:20.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:20 smithi081 ceph-mon[31453]: pgmap v5587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:22.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:21 smithi153 ceph-mon[36569]: pgmap v5588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:22.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:21 smithi104 ceph-mon[36624]: pgmap v5588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:22.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:21 smithi081 ceph-mon[31453]: pgmap v5588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:24.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:23 smithi153 ceph-mon[36569]: pgmap v5589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:24.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:23 smithi104 ceph-mon[36624]: pgmap v5589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:24.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:23 smithi081 ceph-mon[31453]: pgmap v5589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:26.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:25 smithi153 ceph-mon[36569]: pgmap v5590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:26.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:25 smithi104 ceph-mon[36624]: pgmap v5590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:26.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:25 smithi081 ceph-mon[31453]: pgmap v5590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:28.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:28 smithi153 ceph-mon[36569]: pgmap v5591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:28.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:28 smithi104 ceph-mon[36624]: pgmap v5591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:28.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:28 smithi081 ceph-mon[31453]: pgmap v5591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:30.303+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:00:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:30.303+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:00:30.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:30.303+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:00:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:30 smithi153 ceph-mon[36569]: pgmap v5592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:30 smithi104 ceph-mon[36624]: pgmap v5592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:30.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:30 smithi081 ceph-mon[31453]: pgmap v5592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:32.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:32 smithi153 ceph-mon[36569]: pgmap v5593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:32.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:32 smithi104 ceph-mon[36624]: pgmap v5593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:32.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:32 smithi081 ceph-mon[31453]: pgmap v5593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:34.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:34 smithi153 ceph-mon[36569]: pgmap v5594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:34.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:00:34.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:34 smithi104 ceph-mon[36624]: pgmap v5594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:34.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:00:34.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:34 smithi081 ceph-mon[31453]: pgmap v5594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:34.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:00:35.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:00:35.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:00:35.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:35 smithi153 ceph-mon[36569]: pgmap v5595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:35.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:35 smithi153 ceph-mon[36569]: pgmap v5596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:35.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:00:35.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:00:35.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:00:35.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:35 smithi104 ceph-mon[36624]: pgmap v5595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:35.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:35 smithi104 ceph-mon[36624]: pgmap v5596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:35.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:00:35.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:00:35.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:00:35.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:35 smithi081 ceph-mon[31453]: pgmap v5595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:35.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:35 smithi081 ceph-mon[31453]: pgmap v5596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:35.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:00:37.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:36 smithi153 ceph-mon[36569]: pgmap v5597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:37.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:36 smithi104 ceph-mon[36624]: pgmap v5597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:37.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:36 smithi081 ceph-mon[31453]: pgmap v5597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:39.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:39 smithi104 ceph-mon[36624]: pgmap v5598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:39.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:39 smithi081 ceph-mon[31453]: pgmap v5598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:40.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:39 smithi153 ceph-mon[36569]: pgmap v5598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:41.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:41 smithi104 ceph-mon[36624]: pgmap v5599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:41.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:41 smithi081 ceph-mon[31453]: pgmap v5599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:42.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:41 smithi153 ceph-mon[36569]: pgmap v5599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:43.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:43 smithi104 ceph-mon[36624]: pgmap v5600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:43.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:43 smithi081 ceph-mon[31453]: pgmap v5600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:44.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:43 smithi153 ceph-mon[36569]: pgmap v5600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:45.311+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:00:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:45.311+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:00:45.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:00:45.311+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:00:45.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:45 smithi081 ceph-mon[31453]: pgmap v5601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:46.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:45 smithi153 ceph-mon[36569]: pgmap v5601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:46.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:45 smithi104 ceph-mon[36624]: pgmap v5601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:47.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:47 smithi081 ceph-mon[31453]: pgmap v5602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:48.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:47 smithi153 ceph-mon[36569]: pgmap v5602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:48.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:47 smithi104 ceph-mon[36624]: pgmap v5602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:49.837 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:49 smithi153 ceph-mon[36569]: pgmap v5603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:49.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:49 smithi081 ceph-mon[31453]: pgmap v5603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:50.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:49 smithi104 ceph-mon[36624]: pgmap v5603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:50.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:50 smithi081 ceph-mon[31453]: pgmap v5604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:51.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:50 smithi153 ceph-mon[36569]: pgmap v5604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:51.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:50 smithi104 ceph-mon[36624]: pgmap v5604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:53.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:53 smithi104 ceph-mon[36624]: pgmap v5605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:53.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:53 smithi081 ceph-mon[31453]: pgmap v5605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:54.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:53 smithi153 ceph-mon[36569]: pgmap v5605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:55.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:55 smithi104 ceph-mon[36624]: pgmap v5606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:55.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:55 smithi081 ceph-mon[31453]: pgmap v5606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:56.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:55 smithi153 ceph-mon[36569]: pgmap v5606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:57.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:57 smithi081 ceph-mon[31453]: pgmap v5607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:58.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:57 smithi153 ceph-mon[36569]: pgmap v5607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:58.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:57 smithi104 ceph-mon[36624]: pgmap v5607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:00:59.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:00:59 smithi081 ceph-mon[31453]: pgmap v5608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:00.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:00:59 smithi153 ceph-mon[36569]: pgmap v5608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:00.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:00:59 smithi104 ceph-mon[36624]: pgmap v5608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:00.316+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:01:00.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:00.316+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:01:00.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:00.316+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:01:01.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:01 smithi153 ceph-mon[36569]: pgmap v5609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:01.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:01 smithi081 ceph-mon[31453]: pgmap v5609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:02.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:01 smithi104 ceph-mon[36624]: pgmap v5609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:02.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:02 smithi081 ceph-mon[31453]: pgmap v5610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:03.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:02 smithi153 ceph-mon[36569]: pgmap v5610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:03.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:02 smithi104 ceph-mon[36624]: pgmap v5610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:05.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:05 smithi081 ceph-mon[31453]: pgmap v5611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:06.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:05 smithi153 ceph-mon[36569]: pgmap v5611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:06.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:05 smithi104 ceph-mon[36624]: pgmap v5611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:07.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:06 smithi153 ceph-mon[36569]: pgmap v5612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:07.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:06 smithi104 ceph-mon[36624]: pgmap v5612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:07.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:06 smithi081 ceph-mon[31453]: pgmap v5612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:09.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:09 smithi081 ceph-mon[31453]: pgmap v5613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:10.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:09 smithi153 ceph-mon[36569]: pgmap v5613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:10.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:09 smithi104 ceph-mon[36624]: pgmap v5613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:11.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:11 smithi081 ceph-mon[31453]: pgmap v5614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:12.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:11 smithi153 ceph-mon[36569]: pgmap v5614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:12.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:11 smithi104 ceph-mon[36624]: pgmap v5614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.114+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-13T00:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.114+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-13T00:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.114+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f16f40d42d0 tx=0x7f16f40bd040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-13T00:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.114+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f16f40d42d0 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-13T00:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.114+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-13T00:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.114+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T00:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.114+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5040 0x7f16e80b3a70 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T00:01:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.114+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T00:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.115+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T00:01:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:12.116+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T00:01:13.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:13 smithi081 ceph-mon[31453]: pgmap v5615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:14.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:13 smithi153 ceph-mon[36569]: pgmap v5615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:14.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:13 smithi104 ceph-mon[36624]: pgmap v5615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:14.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:14.334+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:01:14.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:14.334+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:01:14.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:14.335+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:01:14.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:14.335+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:01:14.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:14.335+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:01:14.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:14.335+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:01:14.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:14.336+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:01:14.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:14.336+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:01:14.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:14 smithi081 ceph-mon[31453]: pgmap v5616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:15.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:14 smithi153 ceph-mon[36569]: pgmap v5616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:15.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:14 smithi104 ceph-mon[36624]: pgmap v5616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:15.317+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:01:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:15.317+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:01:15.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:15.317+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:01:17.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:17 smithi081 ceph-mon[31453]: pgmap v5617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:18.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:17 smithi153 ceph-mon[36569]: pgmap v5617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:18.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:17 smithi104 ceph-mon[36624]: pgmap v5617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:19.837 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:19 smithi153 ceph-mon[36569]: pgmap v5618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:19.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:19 smithi081 ceph-mon[31453]: pgmap v5618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:20.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:19 smithi104 ceph-mon[36624]: pgmap v5618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:20.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:20 smithi081 ceph-mon[31453]: pgmap v5619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:21.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:20 smithi153 ceph-mon[36569]: pgmap v5619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:21.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:20 smithi104 ceph-mon[36624]: pgmap v5619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:23.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:23 smithi081 ceph-mon[31453]: pgmap v5620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:24.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:23 smithi153 ceph-mon[36569]: pgmap v5620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:24.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:23 smithi104 ceph-mon[36624]: pgmap v5620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:25.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:25 smithi081 ceph-mon[31453]: pgmap v5621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:26.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:25 smithi153 ceph-mon[36569]: pgmap v5621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:26.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:25 smithi104 ceph-mon[36624]: pgmap v5621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:26.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:26 smithi081 ceph-mon[31453]: pgmap v5622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:27.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:26 smithi153 ceph-mon[36569]: pgmap v5622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:27.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:26 smithi104 ceph-mon[36624]: pgmap v5622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:29.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:29 smithi081 ceph-mon[31453]: pgmap v5623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:30.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:29 smithi153 ceph-mon[36569]: pgmap v5623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:30.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:29 smithi104 ceph-mon[36624]: pgmap v5623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:30.327+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:01:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:30.327+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:01:30.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:30.327+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:01:31.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:31 smithi081 ceph-mon[31453]: pgmap v5624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:32.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:31 smithi153 ceph-mon[36569]: pgmap v5624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:32.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:31 smithi104 ceph-mon[36624]: pgmap v5624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:32.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:32 smithi081 ceph-mon[31453]: pgmap v5625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:33.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:32 smithi153 ceph-mon[36569]: pgmap v5625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:33.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:32 smithi104 ceph-mon[36624]: pgmap v5625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:35.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:35 smithi081 ceph-mon[31453]: pgmap v5626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:35.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:01:36.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:35 smithi153 ceph-mon[36569]: pgmap v5626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:36.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:01:36.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:35 smithi104 ceph-mon[36624]: pgmap v5626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:36.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:01:36.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:01:36.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:01:36.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:01:37.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:01:37.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:01:37.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:01:37.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:01:37.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:01:37.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:01:37.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:37 smithi081 ceph-mon[31453]: pgmap v5627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:37.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:37 smithi081 ceph-mon[31453]: pgmap v5628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:38.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:37 smithi153 ceph-mon[36569]: pgmap v5627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:38.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:37 smithi153 ceph-mon[36569]: pgmap v5628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:38.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:37 smithi104 ceph-mon[36624]: pgmap v5627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:38.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:37 smithi104 ceph-mon[36624]: pgmap v5628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:38.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:38 smithi081 ceph-mon[31453]: pgmap v5629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:39.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:38 smithi153 ceph-mon[36569]: pgmap v5629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:39.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:38 smithi104 ceph-mon[36624]: pgmap v5629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:41.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:41 smithi081 ceph-mon[31453]: pgmap v5630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:41.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:41 smithi153 ceph-mon[36569]: pgmap v5630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:41.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:41 smithi104 ceph-mon[36624]: pgmap v5630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:43.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:43 smithi081 ceph-mon[31453]: pgmap v5631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:43.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:43 smithi153 ceph-mon[36569]: pgmap v5631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:43.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:43 smithi104 ceph-mon[36624]: pgmap v5631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:45.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:45.339+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:01:45.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:45.339+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:01:45.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:01:45.339+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:01:45.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:45 smithi081 ceph-mon[31453]: pgmap v5632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:45.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:45 smithi153 ceph-mon[36569]: pgmap v5632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:45.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:45 smithi104 ceph-mon[36624]: pgmap v5632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:47.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:47 smithi081 ceph-mon[31453]: pgmap v5633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:47.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:47 smithi153 ceph-mon[36569]: pgmap v5633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:47.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:47 smithi104 ceph-mon[36624]: pgmap v5633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:49.337 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:49 smithi153 ceph-mon[36569]: pgmap v5634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:49.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:49 smithi081 ceph-mon[31453]: pgmap v5634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:49.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:49 smithi104 ceph-mon[36624]: pgmap v5634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:51.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:51 smithi081 ceph-mon[31453]: pgmap v5635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:51.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:51 smithi153 ceph-mon[36569]: pgmap v5635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:51.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:51 smithi104 ceph-mon[36624]: pgmap v5635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:53 smithi081 ceph-mon[31453]: pgmap v5636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:53 smithi153 ceph-mon[36569]: pgmap v5636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:53 smithi104 ceph-mon[36624]: pgmap v5636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:55.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:55 smithi081 ceph-mon[31453]: pgmap v5637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:55.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:55 smithi153 ceph-mon[36569]: pgmap v5637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:55.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:55 smithi104 ceph-mon[36624]: pgmap v5637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:57.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:57 smithi081 ceph-mon[31453]: pgmap v5638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:57.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:57 smithi153 ceph-mon[36569]: pgmap v5638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:57.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:57 smithi104 ceph-mon[36624]: pgmap v5638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:59.363 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:01:59 smithi104 ceph-mon[36624]: pgmap v5639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:59.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:01:59 smithi081 ceph-mon[31453]: pgmap v5639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:01:59.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:01:59 smithi153 ceph-mon[36569]: pgmap v5639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:00.354+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:02:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:00.354+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:02:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:00.354+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:02:01.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:01 smithi081 ceph-mon[31453]: pgmap v5640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:01.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:01 smithi153 ceph-mon[36569]: pgmap v5640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:01.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:01 smithi104 ceph-mon[36624]: pgmap v5640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:03.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:03 smithi081 ceph-mon[31453]: pgmap v5641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:03.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:03 smithi153 ceph-mon[36569]: pgmap v5641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:03.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:03 smithi104 ceph-mon[36624]: pgmap v5641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:05.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:05 smithi081 ceph-mon[31453]: pgmap v5642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:05.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:05 smithi153 ceph-mon[36569]: pgmap v5642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:05.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:05 smithi104 ceph-mon[36624]: pgmap v5642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:07.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:07 smithi081 ceph-mon[31453]: pgmap v5643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:07.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:07 smithi153 ceph-mon[36569]: pgmap v5643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:07.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:07 smithi104 ceph-mon[36624]: pgmap v5643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:09.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:09 smithi153 ceph-mon[36569]: pgmap v5644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:09.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:09 smithi104 ceph-mon[36624]: pgmap v5644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:09.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:09 smithi081 ceph-mon[31453]: pgmap v5644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:11.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:11 smithi153 ceph-mon[36569]: pgmap v5645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:11.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:11 smithi104 ceph-mon[36624]: pgmap v5645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:11.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:11 smithi081 ceph-mon[31453]: pgmap v5645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:13.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:13 smithi153 ceph-mon[36569]: pgmap v5646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:13.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:13 smithi104 ceph-mon[36624]: pgmap v5646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:13.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:13 smithi081 ceph-mon[31453]: pgmap v5646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:15.369+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:02:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:15.369+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:02:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:15.369+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:02:15.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:15 smithi153 ceph-mon[36569]: pgmap v5647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:15.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:15 smithi104 ceph-mon[36624]: pgmap v5647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:15.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:15 smithi081 ceph-mon[31453]: pgmap v5647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:17.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:17 smithi153 ceph-mon[36569]: pgmap v5648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:17.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:17 smithi104 ceph-mon[36624]: pgmap v5648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:17.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:17 smithi081 ceph-mon[31453]: pgmap v5648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:19.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:19 smithi153 ceph-mon[36569]: pgmap v5649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:19.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:19 smithi104 ceph-mon[36624]: pgmap v5649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:19.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:19 smithi081 ceph-mon[31453]: pgmap v5649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:21.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:21 smithi153 ceph-mon[36569]: pgmap v5650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:21.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:21 smithi104 ceph-mon[36624]: pgmap v5650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:21.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:21 smithi081 ceph-mon[31453]: pgmap v5650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:23.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:23 smithi153 ceph-mon[36569]: pgmap v5651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:23.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:23 smithi104 ceph-mon[36624]: pgmap v5651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:23.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:23 smithi081 ceph-mon[31453]: pgmap v5651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:25.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:25 smithi153 ceph-mon[36569]: pgmap v5652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:25.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:25 smithi104 ceph-mon[36624]: pgmap v5652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:25.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:25 smithi081 ceph-mon[31453]: pgmap v5652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:27.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:27 smithi153 ceph-mon[36569]: pgmap v5653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:27.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:27 smithi104 ceph-mon[36624]: pgmap v5653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:27.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:27 smithi081 ceph-mon[31453]: pgmap v5653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:29.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:29 smithi153 ceph-mon[36569]: pgmap v5654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:29.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:29 smithi104 ceph-mon[36624]: pgmap v5654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:29.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:29 smithi081 ceph-mon[31453]: pgmap v5654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:30.373+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:02:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:30.373+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:02:30.351 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:30.373+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:02:31.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:31 smithi153 ceph-mon[36569]: pgmap v5655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:31.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:31 smithi104 ceph-mon[36624]: pgmap v5655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:31.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:31 smithi081 ceph-mon[31453]: pgmap v5655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:33.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:33 smithi153 ceph-mon[36569]: pgmap v5656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:33.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:33 smithi104 ceph-mon[36624]: pgmap v5656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:33.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:33 smithi081 ceph-mon[31453]: pgmap v5656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:35.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:35 smithi153 ceph-mon[36569]: pgmap v5657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:35.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:35 smithi104 ceph-mon[36624]: pgmap v5657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:35.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:35 smithi081 ceph-mon[31453]: pgmap v5657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:37.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:37 smithi153 ceph-mon[36569]: pgmap v5658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:37.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:02:37.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:37 smithi104 ceph-mon[36624]: pgmap v5658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:37.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:02:37.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:37 smithi081 ceph-mon[31453]: pgmap v5658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:37.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:02:37.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:37.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:39.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:39 smithi153 ceph-mon[36569]: pgmap v5659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:39.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:39.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:39.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:39 smithi104 ceph-mon[36624]: pgmap v5659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:39.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:39.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:39.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:39 smithi081 ceph-mon[31453]: pgmap v5659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:39.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:39.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:40.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:40.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:40.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:40.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:40.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:40.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:41.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:41 smithi153 ceph-mon[36569]: pgmap v5660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:41.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:41 smithi104 ceph-mon[36624]: pgmap v5660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:41.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:41 smithi081 ceph-mon[31453]: pgmap v5660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:42.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:42.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:43.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:43.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:43.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:43.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:43.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:43 smithi081 ceph-mon[31453]: pgmap v5661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:43.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:02:43.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:02:43.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:44.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:43 smithi153 ceph-mon[36569]: pgmap v5661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:44.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:02:44.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:02:44.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:44.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:43 smithi104 ceph-mon[36624]: pgmap v5661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:44.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:02:44.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:02:44.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:02:44.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:44 smithi081 ceph-mon[31453]: pgmap v5662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:44.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:44 smithi081 ceph-mon[31453]: pgmap v5663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:45.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:44 smithi153 ceph-mon[36569]: pgmap v5662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:45.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:44 smithi153 ceph-mon[36569]: pgmap v5663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:45.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:44 smithi104 ceph-mon[36624]: pgmap v5662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:45.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:44 smithi104 ceph-mon[36624]: pgmap v5663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:45.388+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:02:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:45.388+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:02:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:02:45.388+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:02:46.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:45 smithi153 ceph-mon[36569]: pgmap v5664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:46.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:45 smithi104 ceph-mon[36624]: pgmap v5664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:46.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:45 smithi081 ceph-mon[31453]: pgmap v5664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:48.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:47 smithi153 ceph-mon[36569]: pgmap v5665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:48.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:47 smithi104 ceph-mon[36624]: pgmap v5665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:48.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:47 smithi081 ceph-mon[31453]: pgmap v5665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:50.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:49 smithi153 ceph-mon[36569]: pgmap v5666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:50.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:49 smithi104 ceph-mon[36624]: pgmap v5666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:50.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:49 smithi081 ceph-mon[31453]: pgmap v5666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:52.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:51 smithi153 ceph-mon[36569]: pgmap v5667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:52.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:51 smithi104 ceph-mon[36624]: pgmap v5667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:52.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:51 smithi081 ceph-mon[31453]: pgmap v5667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:54.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:53 smithi153 ceph-mon[36569]: pgmap v5668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:54.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:53 smithi104 ceph-mon[36624]: pgmap v5668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:54.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:53 smithi081 ceph-mon[31453]: pgmap v5668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:56.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:55 smithi153 ceph-mon[36569]: pgmap v5669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:56.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:55 smithi104 ceph-mon[36624]: pgmap v5669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:56.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:55 smithi081 ceph-mon[31453]: pgmap v5669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:58.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:57 smithi153 ceph-mon[36569]: pgmap v5670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:58.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:57 smithi104 ceph-mon[36624]: pgmap v5670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:02:58.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:57 smithi081 ceph-mon[31453]: pgmap v5670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:00.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:02:59 smithi153 ceph-mon[36569]: pgmap v5671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:00.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:02:59 smithi104 ceph-mon[36624]: pgmap v5671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:00.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:00.404+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:03:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:00.404+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:03:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:00.404+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:03:00.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:02:59 smithi081 ceph-mon[31453]: pgmap v5671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:02.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:01 smithi153 ceph-mon[36569]: pgmap v5672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:02.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:01 smithi104 ceph-mon[36624]: pgmap v5672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:02.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:01 smithi081 ceph-mon[31453]: pgmap v5672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:04.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:03 smithi153 ceph-mon[36569]: pgmap v5673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:04.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:03 smithi104 ceph-mon[36624]: pgmap v5673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:04.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:03 smithi081 ceph-mon[31453]: pgmap v5673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:06.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:05 smithi153 ceph-mon[36569]: pgmap v5674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:06.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:05 smithi104 ceph-mon[36624]: pgmap v5674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:06.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:05 smithi081 ceph-mon[31453]: pgmap v5674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:08.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:08 smithi153 ceph-mon[36569]: pgmap v5675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:08.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:08 smithi104 ceph-mon[36624]: pgmap v5675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:08 smithi081 ceph-mon[31453]: pgmap v5675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:10.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:10 smithi153 ceph-mon[36569]: pgmap v5676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:10.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:10 smithi104 ceph-mon[36624]: pgmap v5676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:10.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:10 smithi081 ceph-mon[31453]: pgmap v5676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:12.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:12 smithi153 ceph-mon[36569]: pgmap v5677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:12.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:12 smithi104 ceph-mon[36624]: pgmap v5677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:12.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:12 smithi081 ceph-mon[31453]: pgmap v5677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:14.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:14 smithi153 ceph-mon[36569]: pgmap v5678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:14.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:14 smithi104 ceph-mon[36624]: pgmap v5678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:14.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:14 smithi081 ceph-mon[31453]: pgmap v5678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:15.409+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:03:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:15.409+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:03:15.387 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:15.409+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:03:16.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:16 smithi153 ceph-mon[36569]: pgmap v5679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:16.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:16 smithi104 ceph-mon[36624]: pgmap v5679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:16.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:16 smithi081 ceph-mon[31453]: pgmap v5679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:18.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:18 smithi153 ceph-mon[36569]: pgmap v5680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:18.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:18 smithi104 ceph-mon[36624]: pgmap v5680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:18.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:18 smithi081 ceph-mon[31453]: pgmap v5680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:20.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:20 smithi153 ceph-mon[36569]: pgmap v5681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:20.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:20 smithi104 ceph-mon[36624]: pgmap v5681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:20.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:20 smithi081 ceph-mon[31453]: pgmap v5681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:22.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:22 smithi153 ceph-mon[36569]: pgmap v5682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:22.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:22 smithi104 ceph-mon[36624]: pgmap v5682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:22.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:22 smithi081 ceph-mon[31453]: pgmap v5682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:24.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:24 smithi153 ceph-mon[36569]: pgmap v5683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:24.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:24 smithi104 ceph-mon[36624]: pgmap v5683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:24.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:24 smithi081 ceph-mon[31453]: pgmap v5683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:26.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:26 smithi153 ceph-mon[36569]: pgmap v5684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:26.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:26 smithi104 ceph-mon[36624]: pgmap v5684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:26.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:26 smithi081 ceph-mon[31453]: pgmap v5684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:28.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:28 smithi153 ceph-mon[36569]: pgmap v5685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:28.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:28 smithi104 ceph-mon[36624]: pgmap v5685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:28.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:28 smithi081 ceph-mon[31453]: pgmap v5685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:30.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:30 smithi104 ceph-mon[36624]: pgmap v5686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:30.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:30 smithi081 ceph-mon[31453]: pgmap v5686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:30.423+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:03:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:30.423+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:03:30.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:30.423+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:03:30.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:30 smithi153 ceph-mon[36569]: pgmap v5686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:32.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:32 smithi104 ceph-mon[36624]: pgmap v5687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:32.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:32 smithi081 ceph-mon[31453]: pgmap v5687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:32.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:32 smithi153 ceph-mon[36569]: pgmap v5687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:34.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:34 smithi104 ceph-mon[36624]: pgmap v5688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:34.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:34 smithi081 ceph-mon[31453]: pgmap v5688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:34.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:34 smithi153 ceph-mon[36569]: pgmap v5688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:36.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:36 smithi081 ceph-mon[31453]: pgmap v5689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:36.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:36 smithi153 ceph-mon[36569]: pgmap v5689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:36.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:36 smithi104 ceph-mon[36624]: pgmap v5689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:38.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:38 smithi081 ceph-mon[31453]: pgmap v5690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:38.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:38 smithi153 ceph-mon[36569]: pgmap v5690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:38.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:38 smithi104 ceph-mon[36624]: pgmap v5690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:40.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:40 smithi081 ceph-mon[31453]: pgmap v5691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:40.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:40 smithi153 ceph-mon[36569]: pgmap v5691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:40.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:40 smithi104 ceph-mon[36624]: pgmap v5691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:42.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:42 smithi081 ceph-mon[31453]: pgmap v5692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:42.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:42 smithi153 ceph-mon[36569]: pgmap v5692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:42.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:42 smithi104 ceph-mon[36624]: pgmap v5692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:44.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:44 smithi081 ceph-mon[31453]: pgmap v5693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:44.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:03:44.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:44.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:44.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:44 smithi153 ceph-mon[36569]: pgmap v5693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:44.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:03:44.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:44.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:44.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:44 smithi104 ceph-mon[36624]: pgmap v5693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:44.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:03:44.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:44.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:45.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:03:45.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:03:45.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:45 smithi081 ceph-mon[31453]: pgmap v5694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:45.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:45 smithi081 ceph-mon[31453]: pgmap v5695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:45.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:45.438+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:03:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:45.438+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:03:45.417 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:03:45.438+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:03:45.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:03:45.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:03:45.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:45 smithi153 ceph-mon[36569]: pgmap v5694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:45.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:45 smithi153 ceph-mon[36569]: pgmap v5695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:45.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:45.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:03:45.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:03:45.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:45 smithi104 ceph-mon[36624]: pgmap v5694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:45.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:45 smithi104 ceph-mon[36624]: pgmap v5695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:45.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:03:47.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:47 smithi104 ceph-mon[36624]: pgmap v5696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:47.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:47 smithi081 ceph-mon[31453]: pgmap v5696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:48.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:47 smithi153 ceph-mon[36569]: pgmap v5696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:49.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:49 smithi104 ceph-mon[36624]: pgmap v5697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:49.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:49 smithi081 ceph-mon[31453]: pgmap v5697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:50.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:49 smithi153 ceph-mon[36569]: pgmap v5697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:51.837 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:51 smithi153 ceph-mon[36569]: pgmap v5698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:51.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:51 smithi081 ceph-mon[31453]: pgmap v5698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:52.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:51 smithi104 ceph-mon[36624]: pgmap v5698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:53.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:53 smithi081 ceph-mon[31453]: pgmap v5699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:54.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:53 smithi153 ceph-mon[36569]: pgmap v5699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:54.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:53 smithi104 ceph-mon[36624]: pgmap v5699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:55.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:55 smithi081 ceph-mon[31453]: pgmap v5700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:56.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:55 smithi153 ceph-mon[36569]: pgmap v5700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:56.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:55 smithi104 ceph-mon[36624]: pgmap v5700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:57.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:57 smithi081 ceph-mon[31453]: pgmap v5701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:58.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:57 smithi153 ceph-mon[36569]: pgmap v5701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:58.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:57 smithi104 ceph-mon[36624]: pgmap v5701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:03:59.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:03:59 smithi081 ceph-mon[31453]: pgmap v5702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:00.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:03:59 smithi153 ceph-mon[36569]: pgmap v5702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:00.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:03:59 smithi104 ceph-mon[36624]: pgmap v5702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:00.448+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:04:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:00.448+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:04:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:00.448+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:04:01.863 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:01 smithi104 ceph-mon[36624]: pgmap v5703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:01.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:01 smithi081 ceph-mon[31453]: pgmap v5703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:02.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:01 smithi153 ceph-mon[36569]: pgmap v5703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:03.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:03 smithi081 ceph-mon[31453]: pgmap v5704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:04.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:03 smithi153 ceph-mon[36569]: pgmap v5704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:04.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:03 smithi104 ceph-mon[36624]: pgmap v5704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:05.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:05 smithi081 ceph-mon[31453]: pgmap v5705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:06.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:05 smithi153 ceph-mon[36569]: pgmap v5705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:06.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:05 smithi104 ceph-mon[36624]: pgmap v5705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:07.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:07 smithi081 ceph-mon[31453]: pgmap v5706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:08.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:07 smithi153 ceph-mon[36569]: pgmap v5706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:08.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:07 smithi104 ceph-mon[36624]: pgmap v5706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:09.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:09 smithi081 ceph-mon[31453]: pgmap v5707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:10.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:09 smithi153 ceph-mon[36569]: pgmap v5707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:10.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:09 smithi104 ceph-mon[36624]: pgmap v5707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:11.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:11 smithi081 ceph-mon[31453]: pgmap v5708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:12.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:11 smithi153 ceph-mon[36569]: pgmap v5708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:12.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:11 smithi104 ceph-mon[36624]: pgmap v5708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:12.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:12 smithi081 ceph-mon[31453]: pgmap v5709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:13.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:12 smithi153 ceph-mon[36569]: pgmap v5709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:13.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:12 smithi104 ceph-mon[36624]: pgmap v5709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:14.364+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:04:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:14.364+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:04:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:14.364+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:04:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:14.364+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:04:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:15.449+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:04:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:15.449+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:04:15.427 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:15.449+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:04:15.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:15 smithi081 ceph-mon[31453]: pgmap v5710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:16.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:15 smithi153 ceph-mon[36569]: pgmap v5710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:16.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:15 smithi104 ceph-mon[36624]: pgmap v5710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:17.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:17 smithi081 ceph-mon[31453]: pgmap v5711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:18.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:17 smithi153 ceph-mon[36569]: pgmap v5711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:18.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:17 smithi104 ceph-mon[36624]: pgmap v5711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:19.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:19 smithi081 ceph-mon[31453]: pgmap v5712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:20.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:19 smithi153 ceph-mon[36569]: pgmap v5712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:20.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:19 smithi104 ceph-mon[36624]: pgmap v5712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:21.837 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:21 smithi153 ceph-mon[36569]: pgmap v5713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:21.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:21 smithi081 ceph-mon[31453]: pgmap v5713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:22.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:21 smithi104 ceph-mon[36624]: pgmap v5713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:23.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:23 smithi081 ceph-mon[31453]: pgmap v5714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:24.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:23 smithi153 ceph-mon[36569]: pgmap v5714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:24.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:23 smithi104 ceph-mon[36624]: pgmap v5714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:25.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:25 smithi081 ceph-mon[31453]: pgmap v5715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:26.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:25 smithi153 ceph-mon[36569]: pgmap v5715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:26.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:25 smithi104 ceph-mon[36624]: pgmap v5715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:27.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:27 smithi081 ceph-mon[31453]: pgmap v5716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:28.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:27 smithi153 ceph-mon[36569]: pgmap v5716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:28.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:27 smithi104 ceph-mon[36624]: pgmap v5716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:29.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:29 smithi081 ceph-mon[31453]: pgmap v5717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:30.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:29 smithi153 ceph-mon[36569]: pgmap v5717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:30.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:29 smithi104 ceph-mon[36624]: pgmap v5717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:30.464+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:04:30.442 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:30.464+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:04:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:30.464+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:04:31.863 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:31 smithi104 ceph-mon[36624]: pgmap v5718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:31.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:31 smithi081 ceph-mon[31453]: pgmap v5718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:32.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:31 smithi153 ceph-mon[36569]: pgmap v5718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:33.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:33 smithi081 ceph-mon[31453]: pgmap v5719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:34.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:33 smithi153 ceph-mon[36569]: pgmap v5719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:34.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:33 smithi104 ceph-mon[36624]: pgmap v5719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:35.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:35 smithi081 ceph-mon[31453]: pgmap v5720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:36.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:35 smithi153 ceph-mon[36569]: pgmap v5720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:36.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:35 smithi104 ceph-mon[36624]: pgmap v5720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:37.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:36 smithi153 ceph-mon[36569]: pgmap v5721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:37.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:36 smithi104 ceph-mon[36624]: pgmap v5721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:37.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:36 smithi081 ceph-mon[31453]: pgmap v5721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:39.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:39 smithi081 ceph-mon[31453]: pgmap v5722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:40.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:39 smithi153 ceph-mon[36569]: pgmap v5722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:40.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:39 smithi104 ceph-mon[36624]: pgmap v5722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:41.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:41 smithi081 ceph-mon[31453]: pgmap v5723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:42.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:41 smithi153 ceph-mon[36569]: pgmap v5723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:42.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:41 smithi104 ceph-mon[36624]: pgmap v5723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:43.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:43 smithi081 ceph-mon[31453]: pgmap v5724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:44.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:43 smithi153 ceph-mon[36569]: pgmap v5724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:44.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:43 smithi104 ceph-mon[36624]: pgmap v5724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:45.479+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:04:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:45.479+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:04:45.458 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:04:45.479+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:04:45.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:45 smithi081 ceph-mon[31453]: pgmap v5725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:45.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:04:46.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:45 smithi153 ceph-mon[36569]: pgmap v5725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:46.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:04:46.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:45 smithi104 ceph-mon[36624]: pgmap v5725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:46.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:04:46.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:04:46.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:04:46.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:04:47.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:04:47.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:04:47.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:04:47.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:04:47.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:04:47.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:04:47.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:47 smithi081 ceph-mon[31453]: pgmap v5726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:47.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:47 smithi081 ceph-mon[31453]: pgmap v5727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:48.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:47 smithi153 ceph-mon[36569]: pgmap v5726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:48.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:47 smithi153 ceph-mon[36569]: pgmap v5727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:48.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:47 smithi104 ceph-mon[36624]: pgmap v5726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:48.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:47 smithi104 ceph-mon[36624]: pgmap v5727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:49.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:49 smithi081 ceph-mon[31453]: pgmap v5728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:50.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:49 smithi153 ceph-mon[36569]: pgmap v5728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:50.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:49 smithi104 ceph-mon[36624]: pgmap v5728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:51.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:51 smithi081 ceph-mon[31453]: pgmap v5729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:52.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:51 smithi153 ceph-mon[36569]: pgmap v5729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:52.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:51 smithi104 ceph-mon[36624]: pgmap v5729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:52.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:52 smithi081 ceph-mon[31453]: pgmap v5730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:53.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:52 smithi153 ceph-mon[36569]: pgmap v5730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:53.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:52 smithi104 ceph-mon[36624]: pgmap v5730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:55.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:55 smithi081 ceph-mon[31453]: pgmap v5731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:55.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:55 smithi153 ceph-mon[36569]: pgmap v5731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:55.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:55 smithi104 ceph-mon[36624]: pgmap v5731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:57.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:57 smithi081 ceph-mon[31453]: pgmap v5732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:57.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:57 smithi153 ceph-mon[36569]: pgmap v5732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:57.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:57 smithi104 ceph-mon[36624]: pgmap v5732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:59.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:04:59 smithi081 ceph-mon[31453]: pgmap v5733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:59.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:04:59 smithi153 ceph-mon[36569]: pgmap v5733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:04:59.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:04:59 smithi104 ceph-mon[36624]: pgmap v5733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:00.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:00.488+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:05:00.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:00.488+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:05:00.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:00.488+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:05:01.362 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:01 smithi104 ceph-mon[36624]: pgmap v5734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:01.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:01 smithi081 ceph-mon[31453]: pgmap v5734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:01.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:01 smithi153 ceph-mon[36569]: pgmap v5734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:03.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:03 smithi081 ceph-mon[31453]: pgmap v5735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:03.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:03 smithi153 ceph-mon[36569]: pgmap v5735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:03.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:03 smithi104 ceph-mon[36624]: pgmap v5735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:05.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:05 smithi081 ceph-mon[31453]: pgmap v5736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:05.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:05 smithi153 ceph-mon[36569]: pgmap v5736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:05.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:05 smithi104 ceph-mon[36624]: pgmap v5736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:07.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:07 smithi081 ceph-mon[31453]: pgmap v5737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:07.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:07 smithi153 ceph-mon[36569]: pgmap v5737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:07.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:07 smithi104 ceph-mon[36624]: pgmap v5737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:09 smithi081 ceph-mon[31453]: pgmap v5738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:09.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:09 smithi153 ceph-mon[36569]: pgmap v5738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:09.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:09 smithi104 ceph-mon[36624]: pgmap v5738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:11.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:11 smithi081 ceph-mon[31453]: pgmap v5739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:11.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:11 smithi153 ceph-mon[36569]: pgmap v5739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:11.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:11 smithi104 ceph-mon[36624]: pgmap v5739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:13.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:13 smithi081 ceph-mon[31453]: pgmap v5740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:13.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:13 smithi153 ceph-mon[36569]: pgmap v5740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:13.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:13 smithi104 ceph-mon[36624]: pgmap v5740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:15.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:15 smithi081 ceph-mon[31453]: pgmap v5741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:15.490+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:05:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:15.490+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:05:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:15.490+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:05:15.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:15 smithi153 ceph-mon[36569]: pgmap v5741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:15.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:15 smithi104 ceph-mon[36624]: pgmap v5741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:17.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:17 smithi081 ceph-mon[31453]: pgmap v5742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:17.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:17 smithi153 ceph-mon[36569]: pgmap v5742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:17.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:17 smithi104 ceph-mon[36624]: pgmap v5742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:19.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:19 smithi081 ceph-mon[31453]: pgmap v5743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:19.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:19 smithi153 ceph-mon[36569]: pgmap v5743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:19.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:19 smithi104 ceph-mon[36624]: pgmap v5743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:21.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:21 smithi081 ceph-mon[31453]: pgmap v5744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:21.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:21 smithi153 ceph-mon[36569]: pgmap v5744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:21.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:21 smithi104 ceph-mon[36624]: pgmap v5744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:23.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:23 smithi153 ceph-mon[36569]: pgmap v5745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:23.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:23 smithi104 ceph-mon[36624]: pgmap v5745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:23.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:23 smithi081 ceph-mon[31453]: pgmap v5745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:25.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:25 smithi153 ceph-mon[36569]: pgmap v5746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:25.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:25 smithi104 ceph-mon[36624]: pgmap v5746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:25.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:25 smithi081 ceph-mon[31453]: pgmap v5746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:27.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:27 smithi153 ceph-mon[36569]: pgmap v5747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:27.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:27 smithi104 ceph-mon[36624]: pgmap v5747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:27.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:27 smithi081 ceph-mon[31453]: pgmap v5747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:29.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:29 smithi153 ceph-mon[36569]: pgmap v5748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:29.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:29 smithi104 ceph-mon[36624]: pgmap v5748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:29.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:29 smithi081 ceph-mon[31453]: pgmap v5748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:30.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:30.498+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:05:30.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:30.498+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:05:30.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:30.498+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:05:31.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:31 smithi153 ceph-mon[36569]: pgmap v5749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:31.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:31 smithi104 ceph-mon[36624]: pgmap v5749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:31.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:31 smithi081 ceph-mon[31453]: pgmap v5749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:33.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:33 smithi153 ceph-mon[36569]: pgmap v5750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:33.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:33 smithi104 ceph-mon[36624]: pgmap v5750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:33.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:33 smithi081 ceph-mon[31453]: pgmap v5750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:35.437 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:35 smithi081 ceph-mon[31453]: pgmap v5751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:35.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:35 smithi153 ceph-mon[36569]: pgmap v5751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:35.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:35 smithi104 ceph-mon[36624]: pgmap v5751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:37.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:37 smithi153 ceph-mon[36569]: pgmap v5752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:37.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:37 smithi104 ceph-mon[36624]: pgmap v5752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:37.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:37 smithi081 ceph-mon[31453]: pgmap v5752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:39.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:39 smithi153 ceph-mon[36569]: pgmap v5753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:39.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:39 smithi104 ceph-mon[36624]: pgmap v5753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:39.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:39 smithi081 ceph-mon[31453]: pgmap v5753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:41.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:41 smithi153 ceph-mon[36569]: pgmap v5754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:41.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:41 smithi104 ceph-mon[36624]: pgmap v5754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:41.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:41 smithi081 ceph-mon[31453]: pgmap v5754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:43.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:43 smithi153 ceph-mon[36569]: pgmap v5755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:43.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:43 smithi104 ceph-mon[36624]: pgmap v5755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:43.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:43 smithi081 ceph-mon[31453]: pgmap v5755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:45.514+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:05:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:45.514+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:05:45.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:05:45.514+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:05:45.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:45 smithi153 ceph-mon[36569]: pgmap v5756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:45.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:45 smithi104 ceph-mon[36624]: pgmap v5756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:45.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:45 smithi081 ceph-mon[31453]: pgmap v5756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:47.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:47 smithi153 ceph-mon[36569]: pgmap v5757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:47.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:05:47.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:47 smithi104 ceph-mon[36624]: pgmap v5757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:47.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:05:47.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:47 smithi081 ceph-mon[31453]: pgmap v5757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:47.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:05:48.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:05:48.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:05:48.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:48 smithi153 ceph-mon[36569]: pgmap v5758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:48.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:48 smithi153 ceph-mon[36569]: pgmap v5759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:48.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:05:48.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:05:48.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:05:48.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:48 smithi104 ceph-mon[36624]: pgmap v5758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:48.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:48 smithi104 ceph-mon[36624]: pgmap v5759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:48.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:05:48.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:05:48.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:05:48.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:48 smithi081 ceph-mon[31453]: pgmap v5758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:48.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:48 smithi081 ceph-mon[31453]: pgmap v5759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:48.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:05:50.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:50 smithi081 ceph-mon[31453]: pgmap v5760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:51.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:50 smithi153 ceph-mon[36569]: pgmap v5760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:51.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:50 smithi104 ceph-mon[36624]: pgmap v5760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:52.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:52 smithi081 ceph-mon[31453]: pgmap v5761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:53.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:52 smithi153 ceph-mon[36569]: pgmap v5761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:53.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:52 smithi104 ceph-mon[36624]: pgmap v5761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:54.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:54 smithi081 ceph-mon[31453]: pgmap v5762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:55.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:54 smithi153 ceph-mon[36569]: pgmap v5762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:55.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:54 smithi104 ceph-mon[36624]: pgmap v5762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:56.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:56 smithi081 ceph-mon[31453]: pgmap v5763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:57.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:56 smithi153 ceph-mon[36569]: pgmap v5763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:57.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:56 smithi104 ceph-mon[36624]: pgmap v5763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:58.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:05:58 smithi081 ceph-mon[31453]: pgmap v5764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:59.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:05:58 smithi153 ceph-mon[36569]: pgmap v5764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:05:59.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:05:58 smithi104 ceph-mon[36624]: pgmap v5764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:00.525+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:06:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:00.525+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:06:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:00.525+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:06:00.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:00 smithi081 ceph-mon[31453]: pgmap v5765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:01.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:00 smithi153 ceph-mon[36569]: pgmap v5765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:01.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:00 smithi104 ceph-mon[36624]: pgmap v5765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:02.862 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:02 smithi104 ceph-mon[36624]: pgmap v5766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:02.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:02 smithi081 ceph-mon[31453]: pgmap v5766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:03.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:02 smithi153 ceph-mon[36569]: pgmap v5766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:04.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:04 smithi081 ceph-mon[31453]: pgmap v5767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:05.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:04 smithi153 ceph-mon[36569]: pgmap v5767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:05.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:04 smithi104 ceph-mon[36624]: pgmap v5767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:06.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:06 smithi081 ceph-mon[31453]: pgmap v5768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:07.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:06 smithi153 ceph-mon[36569]: pgmap v5768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:07.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:06 smithi104 ceph-mon[36624]: pgmap v5768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:08.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:08 smithi081 ceph-mon[31453]: pgmap v5769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:09.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:08 smithi153 ceph-mon[36569]: pgmap v5769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:09.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:08 smithi104 ceph-mon[36624]: pgmap v5769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:10.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:10 smithi081 ceph-mon[31453]: pgmap v5770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:11.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:10 smithi153 ceph-mon[36569]: pgmap v5770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:11.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:10 smithi104 ceph-mon[36624]: pgmap v5770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:13.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:12 smithi153 ceph-mon[36569]: pgmap v5771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:13.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:12 smithi104 ceph-mon[36624]: pgmap v5771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:13.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:12 smithi081 ceph-mon[31453]: pgmap v5771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:14.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:13 smithi153 ceph-mon[36569]: pgmap v5772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:14.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:13 smithi104 ceph-mon[36624]: pgmap v5772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:14.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:13 smithi081 ceph-mon[31453]: pgmap v5772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:15.529+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:06:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:15.529+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:06:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:15.529+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:06:16.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:16 smithi081 ceph-mon[31453]: pgmap v5773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:17.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:16 smithi153 ceph-mon[36569]: pgmap v5773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:17.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:16 smithi104 ceph-mon[36624]: pgmap v5773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:18.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:18 smithi081 ceph-mon[31453]: pgmap v5774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:19.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:18 smithi153 ceph-mon[36569]: pgmap v5774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:19.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:18 smithi104 ceph-mon[36624]: pgmap v5774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:20.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:20 smithi081 ceph-mon[31453]: pgmap v5775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:21.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:20 smithi153 ceph-mon[36569]: pgmap v5775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:21.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:20 smithi104 ceph-mon[36624]: pgmap v5775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:22.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:22 smithi081 ceph-mon[31453]: pgmap v5776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:23.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:22 smithi153 ceph-mon[36569]: pgmap v5776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:23.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:22 smithi104 ceph-mon[36624]: pgmap v5776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:24.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:24 smithi081 ceph-mon[31453]: pgmap v5777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:25.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:24 smithi153 ceph-mon[36569]: pgmap v5777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:25.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:24 smithi104 ceph-mon[36624]: pgmap v5777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:26.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:26 smithi081 ceph-mon[31453]: pgmap v5778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:27.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:26 smithi153 ceph-mon[36569]: pgmap v5778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:27.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:26 smithi104 ceph-mon[36624]: pgmap v5778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:28.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:28 smithi081 ceph-mon[31453]: pgmap v5779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:29.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:28 smithi153 ceph-mon[36569]: pgmap v5779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:29.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:28 smithi104 ceph-mon[36624]: pgmap v5779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:30.544+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:06:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:30.544+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:06:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:30.544+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:06:30.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:30 smithi081 ceph-mon[31453]: pgmap v5780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:31.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:30 smithi153 ceph-mon[36569]: pgmap v5780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:31.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:30 smithi104 ceph-mon[36624]: pgmap v5780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:33.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:32 smithi153 ceph-mon[36569]: pgmap v5781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:33.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:32 smithi104 ceph-mon[36624]: pgmap v5781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:33.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:32 smithi081 ceph-mon[31453]: pgmap v5781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:35.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:34 smithi153 ceph-mon[36569]: pgmap v5782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:35.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:34 smithi104 ceph-mon[36624]: pgmap v5782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:35.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:34 smithi081 ceph-mon[31453]: pgmap v5782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:36.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:35 smithi153 ceph-mon[36569]: pgmap v5783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:36.036 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:35 smithi081 ceph-mon[31453]: pgmap v5783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:36.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:35 smithi104 ceph-mon[36624]: pgmap v5783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:38.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:38 smithi081 ceph-mon[31453]: pgmap v5784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:39.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:38 smithi153 ceph-mon[36569]: pgmap v5784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:39.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:38 smithi104 ceph-mon[36624]: pgmap v5784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:40.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:40 smithi081 ceph-mon[31453]: pgmap v5785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:41.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:40 smithi153 ceph-mon[36569]: pgmap v5785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:41.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:40 smithi104 ceph-mon[36624]: pgmap v5785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:42.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:42 smithi081 ceph-mon[31453]: pgmap v5786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:43.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:42 smithi153 ceph-mon[36569]: pgmap v5786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:43.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:42 smithi104 ceph-mon[36624]: pgmap v5786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:44.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:44 smithi081 ceph-mon[31453]: pgmap v5787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:45.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:44 smithi153 ceph-mon[36569]: pgmap v5787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:45.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:44 smithi104 ceph-mon[36624]: pgmap v5787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:45.547+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:06:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:45.547+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:06:45.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:06:45.547+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:06:46.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:46 smithi081 ceph-mon[31453]: pgmap v5788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:47.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:46 smithi153 ceph-mon[36569]: pgmap v5788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:47.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:46 smithi104 ceph-mon[36624]: pgmap v5788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:49.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:48 smithi153 ceph-mon[36569]: pgmap v5789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:49.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:06:49.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:48 smithi104 ceph-mon[36624]: pgmap v5789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:49.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:06:49.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:48 smithi081 ceph-mon[31453]: pgmap v5789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:49.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:06:51.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:50 smithi153 ceph-mon[36569]: pgmap v5790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:51.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:51.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:51.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:06:51.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:06:51.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:51.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:50 smithi104 ceph-mon[36624]: pgmap v5790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:51.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:51.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:51.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:06:51.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:06:51.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:51.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:50 smithi081 ceph-mon[31453]: pgmap v5790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:51.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:51.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:51.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:06:51.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:06:51.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:06:52.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:51 smithi153 ceph-mon[36569]: pgmap v5791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:52.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:51 smithi153 ceph-mon[36569]: pgmap v5792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:52.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:51 smithi104 ceph-mon[36624]: pgmap v5791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:52.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:51 smithi104 ceph-mon[36624]: pgmap v5792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:52.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:51 smithi081 ceph-mon[31453]: pgmap v5791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:52.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:51 smithi081 ceph-mon[31453]: pgmap v5792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:53.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:52 smithi153 ceph-mon[36569]: pgmap v5793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:53.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:52 smithi104 ceph-mon[36624]: pgmap v5793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:53.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:52 smithi081 ceph-mon[31453]: pgmap v5793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:55.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:54 smithi153 ceph-mon[36569]: pgmap v5794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:55.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:54 smithi081 ceph-mon[31453]: pgmap v5794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:55.406 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:54 smithi104 ceph-mon[36624]: pgmap v5794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:57.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:56 smithi153 ceph-mon[36569]: pgmap v5795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:57.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:56 smithi104 ceph-mon[36624]: pgmap v5795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:57.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:56 smithi081 ceph-mon[31453]: pgmap v5795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:59.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:06:58 smithi153 ceph-mon[36569]: pgmap v5796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:59.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:06:58 smithi104 ceph-mon[36624]: pgmap v5796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:06:59.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:06:58 smithi081 ceph-mon[31453]: pgmap v5796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:00.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:00.555+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:07:00.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:00.555+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:07:00.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:00.555+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:07:01.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:00 smithi153 ceph-mon[36569]: pgmap v5797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:01.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:00 smithi104 ceph-mon[36624]: pgmap v5797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:01.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:00 smithi081 ceph-mon[31453]: pgmap v5797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:03.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:02 smithi153 ceph-mon[36569]: pgmap v5798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:03.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:02 smithi104 ceph-mon[36624]: pgmap v5798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:03.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:02 smithi081 ceph-mon[31453]: pgmap v5798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:05.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:04 smithi153 ceph-mon[36569]: pgmap v5799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:05.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:04 smithi104 ceph-mon[36624]: pgmap v5799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:05.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:04 smithi081 ceph-mon[31453]: pgmap v5799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:07.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:06 smithi153 ceph-mon[36569]: pgmap v5800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:07.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:06 smithi104 ceph-mon[36624]: pgmap v5800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:07.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:06 smithi081 ceph-mon[31453]: pgmap v5800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:09.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:08 smithi153 ceph-mon[36569]: pgmap v5801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:08 smithi104 ceph-mon[36624]: pgmap v5801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:09 smithi081 ceph-mon[31453]: pgmap v5801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:11.260 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:11 smithi104 ceph-mon[36624]: pgmap v5802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:11.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:11 smithi153 ceph-mon[36569]: pgmap v5802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:11.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:11 smithi081 ceph-mon[31453]: pgmap v5802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:13.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:13 smithi153 ceph-mon[36569]: pgmap v5803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:13.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:13 smithi104 ceph-mon[36624]: pgmap v5803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:13.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:13 smithi081 ceph-mon[31453]: pgmap v5803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:14.394+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:07:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:14.394+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:07:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:14.395+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:07:14.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:14.395+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:07:15.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:15 smithi153 ceph-mon[36569]: pgmap v5804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:15.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:15 smithi104 ceph-mon[36624]: pgmap v5804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:15.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:15 smithi081 ceph-mon[31453]: pgmap v5804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:15.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:15.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:07:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:15.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:07:17.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:17 smithi153 ceph-mon[36569]: pgmap v5805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:17.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:17 smithi104 ceph-mon[36624]: pgmap v5805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:17.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:17 smithi081 ceph-mon[31453]: pgmap v5805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:19.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:19 smithi153 ceph-mon[36569]: pgmap v5806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:19.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:19 smithi104 ceph-mon[36624]: pgmap v5806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:19.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:19 smithi081 ceph-mon[31453]: pgmap v5806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:21.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:21 smithi153 ceph-mon[36569]: pgmap v5807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:21.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:21 smithi104 ceph-mon[36624]: pgmap v5807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:21.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:21 smithi081 ceph-mon[31453]: pgmap v5807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:23.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:23 smithi153 ceph-mon[36569]: pgmap v5808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:23.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:23 smithi104 ceph-mon[36624]: pgmap v5808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:23.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:23 smithi081 ceph-mon[31453]: pgmap v5808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:25.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:25 smithi153 ceph-mon[36569]: pgmap v5809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:25.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:25 smithi104 ceph-mon[36624]: pgmap v5809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:25.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:25 smithi081 ceph-mon[31453]: pgmap v5809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:27.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:27 smithi153 ceph-mon[36569]: pgmap v5810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:27.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:27 smithi104 ceph-mon[36624]: pgmap v5810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:27.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:27 smithi081 ceph-mon[31453]: pgmap v5810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:29.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:29 smithi153 ceph-mon[36569]: pgmap v5811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:29.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:29 smithi104 ceph-mon[36624]: pgmap v5811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:29.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:29 smithi081 ceph-mon[31453]: pgmap v5811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:30.562+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:07:30.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:30.562+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:07:30.541 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:30.562+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:07:31.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:31 smithi104 ceph-mon[36624]: pgmap v5812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:31.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:31 smithi081 ceph-mon[31453]: pgmap v5812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:31.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:31 smithi153 ceph-mon[36569]: pgmap v5812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:33.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:33 smithi104 ceph-mon[36624]: pgmap v5813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:33.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:33 smithi081 ceph-mon[31453]: pgmap v5813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:33.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:33 smithi153 ceph-mon[36569]: pgmap v5813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:35.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:35 smithi104 ceph-mon[36624]: pgmap v5814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:35.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:35 smithi081 ceph-mon[31453]: pgmap v5814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:35.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:35 smithi153 ceph-mon[36569]: pgmap v5814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:37.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:37 smithi104 ceph-mon[36624]: pgmap v5815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:37.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:37 smithi081 ceph-mon[31453]: pgmap v5815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:37.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:37 smithi153 ceph-mon[36569]: pgmap v5815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:39.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:39 smithi081 ceph-mon[31453]: pgmap v5816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:39.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:39 smithi153 ceph-mon[36569]: pgmap v5816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:39.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:39 smithi104 ceph-mon[36624]: pgmap v5816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:41.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:41 smithi081 ceph-mon[31453]: pgmap v5817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:41.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:41 smithi153 ceph-mon[36569]: pgmap v5817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:41.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:41 smithi104 ceph-mon[36624]: pgmap v5817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:43.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:43 smithi081 ceph-mon[31453]: pgmap v5818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:43.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:43 smithi153 ceph-mon[36569]: pgmap v5818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:43.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:43 smithi104 ceph-mon[36624]: pgmap v5818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:45.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:45 smithi081 ceph-mon[31453]: pgmap v5819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:45.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:45 smithi153 ceph-mon[36569]: pgmap v5819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:45.575+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:07:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:45.575+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:07:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:07:45.575+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:07:45.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:45 smithi104 ceph-mon[36624]: pgmap v5819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:47.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:47 smithi081 ceph-mon[31453]: pgmap v5820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:47.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:47 smithi153 ceph-mon[36569]: pgmap v5820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:47.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:47 smithi104 ceph-mon[36624]: pgmap v5820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:49.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:49 smithi081 ceph-mon[31453]: pgmap v5821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:49.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:49 smithi153 ceph-mon[36569]: pgmap v5821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:49.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:49 smithi104 ceph-mon[36624]: pgmap v5821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:51.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:51 smithi081 ceph-mon[31453]: pgmap v5822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:51.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:07:51.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:51 smithi153 ceph-mon[36569]: pgmap v5822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:51.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:07:51.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:51 smithi104 ceph-mon[36624]: pgmap v5822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:51.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:07:52.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:07:52.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:07:52.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: pgmap v5823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:52.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: pgmap v5824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:52.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:07:52.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:07:52.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: pgmap v5823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:52.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: pgmap v5824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:52.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:07:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:07:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: pgmap v5823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: pgmap v5824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:07:54.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:54 smithi081 ceph-mon[31453]: pgmap v5825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:55.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:54 smithi153 ceph-mon[36569]: pgmap v5825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:55.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:54 smithi104 ceph-mon[36624]: pgmap v5825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:56.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:56 smithi081 ceph-mon[31453]: pgmap v5826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:57.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:56 smithi153 ceph-mon[36569]: pgmap v5826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:57.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:56 smithi104 ceph-mon[36624]: pgmap v5826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:58.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:07:58 smithi081 ceph-mon[31453]: pgmap v5827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:59.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:07:58 smithi153 ceph-mon[36569]: pgmap v5827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:07:59.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:07:58 smithi104 ceph-mon[36624]: pgmap v5827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:00.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:00.586+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:08:00.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:00.586+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:08:00.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:00.586+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:08:00.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:00 smithi081 ceph-mon[31453]: pgmap v5828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:01.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:00 smithi153 ceph-mon[36569]: pgmap v5828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:01.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:00 smithi104 ceph-mon[36624]: pgmap v5828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:02.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:02 smithi081 ceph-mon[31453]: pgmap v5829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:03.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:02 smithi153 ceph-mon[36569]: pgmap v5829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:03.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:02 smithi104 ceph-mon[36624]: pgmap v5829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:04.862 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:04 smithi104 ceph-mon[36624]: pgmap v5830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:04.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:04 smithi081 ceph-mon[31453]: pgmap v5830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:05.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:04 smithi153 ceph-mon[36569]: pgmap v5830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:06.866 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:06 smithi081 ceph-mon[31453]: pgmap v5831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:06.866 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 13 00:08:06 smithi081 sudo[134720]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:06.866 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 13 00:08:06 smithi081 sudo[134720]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:06.866 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 13 00:08:06 smithi081 sudo[134720]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:06.866 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 13 00:08:06 smithi081 sudo[134720]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:06.938 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:06 smithi104 ceph-mon[36624]: pgmap v5831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:07.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:06 smithi153 ceph-mon[36569]: pgmap v5831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:07.133 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 13 00:08:06 smithi081 sudo[134723]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.134 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 13 00:08:06 smithi081 sudo[134723]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.134 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 13 00:08:06 smithi081 sudo[134723]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.134 INFO:journalctl@ceph.osd.0.smithi081.stdout:Apr 13 00:08:06 smithi081 sudo[134723]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.284 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116717]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.284 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116717]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.284 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116717]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.284 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116717]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.284 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116720]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.284 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116720]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.285 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116720]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.285 INFO:journalctl@ceph.osd.2.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116720]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.294 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 13 00:08:06 smithi104 sudo[114126]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.294 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 13 00:08:06 smithi104 sudo[114126]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.294 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 13 00:08:06 smithi104 sudo[114126]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.295 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 13 00:08:06 smithi104 sudo[114126]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.295 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 13 00:08:06 smithi104 sudo[114129]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.295 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 13 00:08:06 smithi104 sudo[114129]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.295 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 13 00:08:06 smithi104 sudo[114129]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.295 INFO:journalctl@ceph.osd.1.smithi104.stdout:Apr 13 00:08:06 smithi104 sudo[114129]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.459 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134726]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.459 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134726]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.459 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134726]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.459 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134726]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.459 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134729]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.460 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134729]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.460 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134729]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.460 INFO:journalctl@ceph.osd.3.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134729]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.559 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114132]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.560 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114132]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.560 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114132]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.560 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114132]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.560 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114135]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.560 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114135]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.560 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114135]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.560 INFO:journalctl@ceph.osd.4.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114135]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.634 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116723]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.634 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116723]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.634 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116723]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.634 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116723]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.634 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116726]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.635 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116726]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.635 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116726]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.635 INFO:journalctl@ceph.osd.5.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116726]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.730 INFO:journalctl@ceph.osd.6.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134732]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.731 INFO:journalctl@ceph.osd.6.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.731 INFO:journalctl@ceph.osd.6.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134732]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.731 INFO:journalctl@ceph.osd.6.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134732]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.731 INFO:journalctl@ceph.osd.6.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134735]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.731 INFO:journalctl@ceph.osd.6.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134735]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.731 INFO:journalctl@ceph.osd.6.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134735]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.731 INFO:journalctl@ceph.osd.6.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134735]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.849 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114138]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.849 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114138]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.849 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114138]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.849 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114138]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.849 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114141]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.849 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114141]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.849 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114141]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.849 INFO:journalctl@ceph.osd.7.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114141]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.901 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116729]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.901 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116729]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.901 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116729]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.901 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116729]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.901 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116732]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.901 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.901 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116732]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.902 INFO:journalctl@ceph.osd.8.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116732]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:07.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:07.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:07.988 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:07 smithi081 ceph-mon[31453]: pgmap v5832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:07.988 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134738]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:07.988 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134738]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.988 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134738]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.988 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134738]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:07.989 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134741]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:07.989 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:07.989 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134741]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:07.989 INFO:journalctl@ceph.osd.9.smithi081.stdout:Apr 13 00:08:07 smithi081 sudo[134741]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.138 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:08.138 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:08.139 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:08.139 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:07 smithi104 ceph-mon[36624]: pgmap v5832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:08.139 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114144]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:08.139 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114144]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.139 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114144]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.139 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114144]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.139 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114147]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:08.139 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114147]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.139 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114147]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.139 INFO:journalctl@ceph.osd.10.smithi104.stdout:Apr 13 00:08:07 smithi104 sudo[114147]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.284 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116735]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:08.284 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116735]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.284 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116735]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.284 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116735]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.285 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116738]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:08.285 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116738]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.285 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116738]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.285 INFO:journalctl@ceph.osd.11.smithi153.stdout:Apr 13 00:08:07 smithi153 sudo[116738]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:08.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:08.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:08.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:07 smithi153 ceph-mon[36569]: pgmap v5832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:08.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 sudo[116741]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:08.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 sudo[116741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 sudo[116741]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 sudo[116741]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 sudo[116744]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:08.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 sudo[116744]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 sudo[116744]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 sudo[116744]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 sudo[134744]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 sudo[134744]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 sudo[134744]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 sudo[134744]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 sudo[134747]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 sudo[134747]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 sudo[134747]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 sudo[134747]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 sudo[114150]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-13T00:08:08.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 sudo[114150]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 sudo[114150]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 sudo[114150]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:08.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 sudo[114153]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-13T00:08:08.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 sudo[114153]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-13T00:08:08.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 sudo[114153]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-13T00:08:08.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 sudo[114153]: pam_unix(sudo:session): session closed for user root 2024-04-13T00:08:09.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:09.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:08 smithi153 ceph-mon[36569]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:09.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:08 smithi104 ceph-mon[36624]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:09.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-13T00:08:09.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-13T00:08:09.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-13T00:08:09.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-13T00:08:09.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:08 smithi081 ceph-mon[31453]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-13T00:08:10.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:09 smithi153 ceph-mon[36569]: pgmap v5833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:10.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:09 smithi104 ceph-mon[36624]: pgmap v5833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:10.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:09 smithi081 ceph-mon[31453]: pgmap v5833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:12.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:12 smithi081 ceph-mon[31453]: pgmap v5834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:13.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:12 smithi153 ceph-mon[36569]: pgmap v5834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:13.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:12 smithi104 ceph-mon[36624]: pgmap v5834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:14.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:14 smithi081 ceph-mon[31453]: pgmap v5835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:15.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:14 smithi153 ceph-mon[36569]: pgmap v5835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:15.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:14 smithi104 ceph-mon[36624]: pgmap v5835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:15.591+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:08:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:15.591+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:08:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:15.591+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:08:16.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:16 smithi081 ceph-mon[31453]: pgmap v5836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:17.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:16 smithi153 ceph-mon[36569]: pgmap v5836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:17.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:16 smithi104 ceph-mon[36624]: pgmap v5836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:18.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:18 smithi081 ceph-mon[31453]: pgmap v5837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:19.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:18 smithi153 ceph-mon[36569]: pgmap v5837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:19.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:18 smithi104 ceph-mon[36624]: pgmap v5837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:20.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:20 smithi081 ceph-mon[31453]: pgmap v5838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:21.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:20 smithi153 ceph-mon[36569]: pgmap v5838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:21.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:20 smithi104 ceph-mon[36624]: pgmap v5838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:22.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:22 smithi081 ceph-mon[31453]: pgmap v5839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:22.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:22 smithi153 ceph-mon[36569]: pgmap v5839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:22.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:22 smithi104 ceph-mon[36624]: pgmap v5839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 120 KiB/s wr, 8 op/s 2024-04-13T00:08:24.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:24 smithi081 ceph-mon[31453]: pgmap v5840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:25.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:24 smithi153 ceph-mon[36569]: pgmap v5840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:25.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:24 smithi104 ceph-mon[36624]: pgmap v5840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:26.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:26 smithi081 ceph-mon[31453]: pgmap v5841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:27.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:26 smithi153 ceph-mon[36569]: pgmap v5841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:27.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:26 smithi104 ceph-mon[36624]: pgmap v5841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:28.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:28 smithi081 ceph-mon[31453]: pgmap v5842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:29.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:28 smithi153 ceph-mon[36569]: pgmap v5842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:29.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:28 smithi104 ceph-mon[36624]: pgmap v5842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:30.605+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:08:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:30.605+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:08:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:30.605+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:08:30.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:30 smithi081 ceph-mon[31453]: pgmap v5843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:31.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:30 smithi153 ceph-mon[36569]: pgmap v5843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:31.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:30 smithi104 ceph-mon[36624]: pgmap v5843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:32.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:32 smithi081 ceph-mon[31453]: pgmap v5844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:33.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:32 smithi153 ceph-mon[36569]: pgmap v5844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:33.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:32 smithi104 ceph-mon[36624]: pgmap v5844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:34.862 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:34 smithi104 ceph-mon[36624]: pgmap v5845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:34.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:34 smithi081 ceph-mon[31453]: pgmap v5845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:35.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:34 smithi153 ceph-mon[36569]: pgmap v5845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:36.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:36 smithi081 ceph-mon[31453]: pgmap v5846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:37.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:36 smithi153 ceph-mon[36569]: pgmap v5846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:37.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:36 smithi104 ceph-mon[36624]: pgmap v5846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:38.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:38 smithi081 ceph-mon[31453]: pgmap v5847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:39.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:38 smithi153 ceph-mon[36569]: pgmap v5847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:39.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:38 smithi104 ceph-mon[36624]: pgmap v5847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:40.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:40 smithi081 ceph-mon[31453]: pgmap v5848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:41.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:40 smithi153 ceph-mon[36569]: pgmap v5848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:41.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:40 smithi104 ceph-mon[36624]: pgmap v5848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:42.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:42 smithi081 ceph-mon[31453]: pgmap v5849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:43.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:42 smithi153 ceph-mon[36569]: pgmap v5849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:43.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:42 smithi104 ceph-mon[36624]: pgmap v5849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:44.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:44 smithi081 ceph-mon[31453]: pgmap v5850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:45.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:44 smithi153 ceph-mon[36569]: pgmap v5850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:45.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:44 smithi104 ceph-mon[36624]: pgmap v5850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:45.618+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:08:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:45.618+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:08:45.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:08:45.618+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:08:47.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:46 smithi153 ceph-mon[36569]: pgmap v5851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:47.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:46 smithi104 ceph-mon[36624]: pgmap v5851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:47.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:46 smithi081 ceph-mon[31453]: pgmap v5851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:49.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:48 smithi153 ceph-mon[36569]: pgmap v5852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:49.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:48 smithi104 ceph-mon[36624]: pgmap v5852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:49.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:48 smithi081 ceph-mon[31453]: pgmap v5852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:51.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:50 smithi153 ceph-mon[36569]: pgmap v5853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:51.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:50 smithi104 ceph-mon[36624]: pgmap v5853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:51.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:50 smithi081 ceph-mon[31453]: pgmap v5853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:52.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:52 smithi081 ceph-mon[31453]: pgmap v5854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:52.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:08:52.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:52 smithi153 ceph-mon[36569]: pgmap v5854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:52.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:08:52.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:52 smithi104 ceph-mon[36624]: pgmap v5854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:52.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:08:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:08:53.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:08:53.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:53.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:08:53.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:08:53.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:08:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:08:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:08:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:08:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:08:54.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:54 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:08:54.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:54 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:08:54.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:54 smithi081 ceph-mon[31453]: pgmap v5855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:54.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:54 smithi081 ceph-mon[31453]: pgmap v5856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:54.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:54 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:08:54.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:54 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:08:54.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:54 smithi153 ceph-mon[36569]: pgmap v5855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:54.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:54 smithi153 ceph-mon[36569]: pgmap v5856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:54.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:54 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:08:54.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:54 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:08:54.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:54 smithi104 ceph-mon[36624]: pgmap v5855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:54.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:54 smithi104 ceph-mon[36624]: pgmap v5856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:56.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:56 smithi081 ceph-mon[31453]: pgmap v5857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:56.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:56 smithi153 ceph-mon[36569]: pgmap v5857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:56.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:56 smithi104 ceph-mon[36624]: pgmap v5857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:58.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:08:58 smithi081 ceph-mon[31453]: pgmap v5858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:58.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:08:58 smithi153 ceph-mon[36569]: pgmap v5858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:08:58.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:08:58 smithi104 ceph-mon[36624]: pgmap v5858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:00.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:00 smithi081 ceph-mon[31453]: pgmap v5859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:00.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:00 smithi153 ceph-mon[36569]: pgmap v5859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:00.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:00 smithi104 ceph-mon[36624]: pgmap v5859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:00.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:00.634+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:09:00.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:00.634+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:09:00.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:00.634+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:09:02.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:02 smithi081 ceph-mon[31453]: pgmap v5860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:02.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:02 smithi153 ceph-mon[36569]: pgmap v5860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:02.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:02 smithi104 ceph-mon[36624]: pgmap v5860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:04.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:04 smithi081 ceph-mon[31453]: pgmap v5861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:04.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:04 smithi153 ceph-mon[36569]: pgmap v5861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:04.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:04 smithi104 ceph-mon[36624]: pgmap v5861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:06.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:06 smithi081 ceph-mon[31453]: pgmap v5862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:06.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:06 smithi153 ceph-mon[36569]: pgmap v5862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:06.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:06 smithi104 ceph-mon[36624]: pgmap v5862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:08.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:08 smithi153 ceph-mon[36569]: pgmap v5863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:08.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:08 smithi104 ceph-mon[36624]: pgmap v5863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:08.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:08 smithi081 ceph-mon[31453]: pgmap v5863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:10.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:10 smithi153 ceph-mon[36569]: pgmap v5864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:10.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:10 smithi104 ceph-mon[36624]: pgmap v5864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:10.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:10 smithi081 ceph-mon[31453]: pgmap v5864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:12.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:12 smithi153 ceph-mon[36569]: pgmap v5865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:12.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:12 smithi104 ceph-mon[36624]: pgmap v5865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:12.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:12 smithi081 ceph-mon[31453]: pgmap v5865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:14.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:14 smithi153 ceph-mon[36569]: pgmap v5866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:14.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:14 smithi104 ceph-mon[36624]: pgmap v5866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:14.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:14 smithi081 ceph-mon[31453]: pgmap v5866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:15.646+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:09:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:15.647+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:09:15.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:15.647+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:09:16.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:16 smithi153 ceph-mon[36569]: pgmap v5867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:16.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:16 smithi104 ceph-mon[36624]: pgmap v5867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:16.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:16 smithi081 ceph-mon[31453]: pgmap v5867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:18.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:18 smithi081 ceph-mon[31453]: pgmap v5868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:18.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:18 smithi153 ceph-mon[36569]: pgmap v5868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:18.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:18 smithi104 ceph-mon[36624]: pgmap v5868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:20.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:20 smithi153 ceph-mon[36569]: pgmap v5869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:20.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:20 smithi104 ceph-mon[36624]: pgmap v5869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:20.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:20 smithi081 ceph-mon[31453]: pgmap v5869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:22.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:22 smithi153 ceph-mon[36569]: pgmap v5870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:22.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:22 smithi104 ceph-mon[36624]: pgmap v5870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:22.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:22 smithi081 ceph-mon[31453]: pgmap v5870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:24.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:24 smithi153 ceph-mon[36569]: pgmap v5871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:24.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:24 smithi104 ceph-mon[36624]: pgmap v5871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:24.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:24 smithi081 ceph-mon[31453]: pgmap v5871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:26.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:26 smithi153 ceph-mon[36569]: pgmap v5872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:26.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:26 smithi104 ceph-mon[36624]: pgmap v5872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:26.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:26 smithi081 ceph-mon[31453]: pgmap v5872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:28.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:28 smithi153 ceph-mon[36569]: pgmap v5873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:28.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:28 smithi104 ceph-mon[36624]: pgmap v5873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:28.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:28 smithi081 ceph-mon[31453]: pgmap v5873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:29.417+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:09:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:29.417+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:09:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:29.417+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:09:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:29.418+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:09:30.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:30 smithi153 ceph-mon[36569]: pgmap v5874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:30.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:30 smithi104 ceph-mon[36624]: pgmap v5874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:30.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:30 smithi081 ceph-mon[31453]: pgmap v5874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:30.661+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:09:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:30.661+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:09:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:30.661+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:09:32.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:32 smithi153 ceph-mon[36569]: pgmap v5875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:32.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:32 smithi104 ceph-mon[36624]: pgmap v5875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:32.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:32 smithi081 ceph-mon[31453]: pgmap v5875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:34.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:34 smithi081 ceph-mon[31453]: pgmap v5876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:34.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:34 smithi153 ceph-mon[36569]: pgmap v5876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:34.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:34 smithi104 ceph-mon[36624]: pgmap v5876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:36.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:36 smithi153 ceph-mon[36569]: pgmap v5877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:36.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:36 smithi104 ceph-mon[36624]: pgmap v5877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:36.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:36 smithi081 ceph-mon[31453]: pgmap v5877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:38.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:38 smithi153 ceph-mon[36569]: pgmap v5878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:38.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:38 smithi104 ceph-mon[36624]: pgmap v5878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:38.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:38 smithi081 ceph-mon[31453]: pgmap v5878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:40.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:40 smithi153 ceph-mon[36569]: pgmap v5879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:40.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:40 smithi104 ceph-mon[36624]: pgmap v5879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:40.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:40 smithi081 ceph-mon[31453]: pgmap v5879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:42.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:42 smithi153 ceph-mon[36569]: pgmap v5880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:42.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:42 smithi104 ceph-mon[36624]: pgmap v5880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:42.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:42 smithi081 ceph-mon[31453]: pgmap v5880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:44.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:44 smithi153 ceph-mon[36569]: pgmap v5881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:44.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:44 smithi104 ceph-mon[36624]: pgmap v5881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:44.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:44 smithi081 ceph-mon[31453]: pgmap v5881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:45.643 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:45.666+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:09:45.643 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:45.666+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:09:45.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:09:45.666+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:09:46.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:46 smithi153 ceph-mon[36569]: pgmap v5882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:46.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:46 smithi104 ceph-mon[36624]: pgmap v5882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:46.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:46 smithi081 ceph-mon[31453]: pgmap v5882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:48.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:48 smithi153 ceph-mon[36569]: pgmap v5883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:48.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:48 smithi104 ceph-mon[36624]: pgmap v5883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:48.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:48 smithi081 ceph-mon[31453]: pgmap v5883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:50.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:50 smithi153 ceph-mon[36569]: pgmap v5884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:50.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:50 smithi104 ceph-mon[36624]: pgmap v5884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:50.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:50 smithi081 ceph-mon[31453]: pgmap v5884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:52.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:52 smithi153 ceph-mon[36569]: pgmap v5885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:52.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:52 smithi104 ceph-mon[36624]: pgmap v5885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:52.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:52 smithi081 ceph-mon[31453]: pgmap v5885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:54.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:54 smithi153 ceph-mon[36569]: pgmap v5886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:54.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:09:54.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:54 smithi104 ceph-mon[36624]: pgmap v5886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:54.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:09:54.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:54 smithi081 ceph-mon[31453]: pgmap v5886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:54.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:09:55.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:09:55.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:09:55.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:55 smithi153 ceph-mon[36569]: pgmap v5887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:55.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:55 smithi153 ceph-mon[36569]: pgmap v5888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:55.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:09:55.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:09:55.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:09:55.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:55 smithi104 ceph-mon[36624]: pgmap v5887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:55.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:55 smithi104 ceph-mon[36624]: pgmap v5888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:55.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:09:55.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:09:55.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:09:55.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:55 smithi081 ceph-mon[31453]: pgmap v5887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:55.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:55 smithi081 ceph-mon[31453]: pgmap v5888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:55.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:09:58.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:57 smithi153 ceph-mon[36569]: pgmap v5889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:58.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:57 smithi104 ceph-mon[36624]: pgmap v5889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:09:58.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:57 smithi081 ceph-mon[31453]: pgmap v5889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:00.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:09:59 smithi153 ceph-mon[36569]: pgmap v5890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:00.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:09:59 smithi104 ceph-mon[36624]: pgmap v5890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:00.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:09:59 smithi081 ceph-mon[31453]: pgmap v5890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:00.681+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:10:00.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:00.681+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:10:00.659 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:00.681+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:10:01.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:10:01.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:10:01.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:10:02.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:01 smithi153 ceph-mon[36569]: pgmap v5891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:02.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:01 smithi104 ceph-mon[36624]: pgmap v5891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:02.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:01 smithi081 ceph-mon[31453]: pgmap v5891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:04.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:03 smithi153 ceph-mon[36569]: pgmap v5892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:04.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:03 smithi104 ceph-mon[36624]: pgmap v5892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:04.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:03 smithi081 ceph-mon[31453]: pgmap v5892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:06.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:05 smithi153 ceph-mon[36569]: pgmap v5893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:06.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:05 smithi104 ceph-mon[36624]: pgmap v5893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:06.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:05 smithi081 ceph-mon[31453]: pgmap v5893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:08.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:07 smithi153 ceph-mon[36569]: pgmap v5894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:08.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:07 smithi104 ceph-mon[36624]: pgmap v5894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:08.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:07 smithi081 ceph-mon[31453]: pgmap v5894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:09.936 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:09 smithi081 ceph-mon[31453]: pgmap v5895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:10.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:09 smithi153 ceph-mon[36569]: pgmap v5895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:10.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:09 smithi104 ceph-mon[36624]: pgmap v5895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:11.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:10 smithi153 ceph-mon[36569]: pgmap v5896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:11.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:10 smithi104 ceph-mon[36624]: pgmap v5896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:11.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:10 smithi081 ceph-mon[31453]: pgmap v5896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:14.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:13 smithi153 ceph-mon[36569]: pgmap v5897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:14.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:13 smithi104 ceph-mon[36624]: pgmap v5897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:14.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:13 smithi081 ceph-mon[31453]: pgmap v5897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:14.424+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:10:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:14.424+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:10:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:14.425+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:10:14.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:14.425+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:10:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:15.696+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:10:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:15.696+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:10:15.673 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:15.696+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:10:16.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:15 smithi153 ceph-mon[36569]: pgmap v5898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:16.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:15 smithi104 ceph-mon[36624]: pgmap v5898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:16.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:15 smithi081 ceph-mon[31453]: pgmap v5898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:18.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:17 smithi153 ceph-mon[36569]: pgmap v5899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:18.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:17 smithi104 ceph-mon[36624]: pgmap v5899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:18.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:17 smithi081 ceph-mon[31453]: pgmap v5899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:20.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:19 smithi153 ceph-mon[36569]: pgmap v5900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:20.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:19 smithi104 ceph-mon[36624]: pgmap v5900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:20.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:19 smithi081 ceph-mon[31453]: pgmap v5900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:22.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:21 smithi153 ceph-mon[36569]: pgmap v5901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:22.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:21 smithi104 ceph-mon[36624]: pgmap v5901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:22.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:21 smithi081 ceph-mon[31453]: pgmap v5901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:24.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:23 smithi153 ceph-mon[36569]: pgmap v5902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:24.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:23 smithi104 ceph-mon[36624]: pgmap v5902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:24.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:23 smithi081 ceph-mon[31453]: pgmap v5902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:26.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:25 smithi153 ceph-mon[36569]: pgmap v5903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:26.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:25 smithi104 ceph-mon[36624]: pgmap v5903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:26.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:25 smithi081 ceph-mon[31453]: pgmap v5903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:28.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:27 smithi153 ceph-mon[36569]: pgmap v5904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:28.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:27 smithi104 ceph-mon[36624]: pgmap v5904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:28.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:27 smithi081 ceph-mon[31453]: pgmap v5904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:29.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:28 smithi153 ceph-mon[36569]: pgmap v5905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:29.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:28 smithi104 ceph-mon[36624]: pgmap v5905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:29.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:28 smithi081 ceph-mon[31453]: pgmap v5905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:30.710+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:10:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:30.710+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:10:30.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:30.710+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:10:32.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:31 smithi153 ceph-mon[36569]: pgmap v5906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:32.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:31 smithi104 ceph-mon[36624]: pgmap v5906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:32.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:31 smithi081 ceph-mon[31453]: pgmap v5906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:34.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:33 smithi153 ceph-mon[36569]: pgmap v5907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:34.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:33 smithi104 ceph-mon[36624]: pgmap v5907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:34.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:33 smithi081 ceph-mon[31453]: pgmap v5907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:36.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:35 smithi153 ceph-mon[36569]: pgmap v5908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:36.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:35 smithi104 ceph-mon[36624]: pgmap v5908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:36.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:35 smithi081 ceph-mon[31453]: pgmap v5908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:38.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:37 smithi153 ceph-mon[36569]: pgmap v5909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:38.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:37 smithi104 ceph-mon[36624]: pgmap v5909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:38.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:37 smithi081 ceph-mon[31453]: pgmap v5909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:39.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:39 smithi081 ceph-mon[31453]: pgmap v5910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:40.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:39 smithi153 ceph-mon[36569]: pgmap v5910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:40.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:39 smithi104 ceph-mon[36624]: pgmap v5910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:42.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:41 smithi153 ceph-mon[36569]: pgmap v5911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:42.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:41 smithi104 ceph-mon[36624]: pgmap v5911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:42.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:41 smithi081 ceph-mon[31453]: pgmap v5911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:44.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:43 smithi153 ceph-mon[36569]: pgmap v5912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:44.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:43 smithi104 ceph-mon[36624]: pgmap v5912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:44.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:43 smithi081 ceph-mon[31453]: pgmap v5912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:45.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:44 smithi153 ceph-mon[36569]: pgmap v5913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:45.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:44 smithi104 ceph-mon[36624]: pgmap v5913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:45.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:44 smithi081 ceph-mon[31453]: pgmap v5913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:45.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:45.712+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:10:45.690 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:45.712+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:10:45.690 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:10:45.712+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:10:48.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:47 smithi153 ceph-mon[36569]: pgmap v5914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:48.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:47 smithi104 ceph-mon[36624]: pgmap v5914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:48.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:47 smithi081 ceph-mon[31453]: pgmap v5914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:50.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:49 smithi153 ceph-mon[36569]: pgmap v5915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:50.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:49 smithi104 ceph-mon[36624]: pgmap v5915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:50.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:49 smithi081 ceph-mon[31453]: pgmap v5915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:51.936 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:51 smithi081 ceph-mon[31453]: pgmap v5916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:52.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:51 smithi153 ceph-mon[36569]: pgmap v5916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:52.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:51 smithi104 ceph-mon[36624]: pgmap v5916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:54.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:53 smithi153 ceph-mon[36569]: pgmap v5917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:54.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:53 smithi104 ceph-mon[36624]: pgmap v5917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:54.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:53 smithi081 ceph-mon[31453]: pgmap v5917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:56.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:55 smithi153 ceph-mon[36569]: pgmap v5918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:56.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:10:56.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:55 smithi104 ceph-mon[36624]: pgmap v5918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:56.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:10:56.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:55 smithi081 ceph-mon[31453]: pgmap v5918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:56.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:10:57.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:10:57.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:10:57.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:10:57.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:10:57.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:10:57.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:10:57.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:10:57.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:10:57.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:10:58.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:57 smithi153 ceph-mon[36569]: pgmap v5919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:58.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:57 smithi153 ceph-mon[36569]: pgmap v5920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:58.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:57 smithi104 ceph-mon[36624]: pgmap v5919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:58.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:57 smithi104 ceph-mon[36624]: pgmap v5920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:58.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:57 smithi081 ceph-mon[31453]: pgmap v5919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:58.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:57 smithi081 ceph-mon[31453]: pgmap v5920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:59.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:10:58 smithi153 ceph-mon[36569]: pgmap v5921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:59.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:10:58 smithi104 ceph-mon[36624]: pgmap v5921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:10:59.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:10:58 smithi081 ceph-mon[31453]: pgmap v5921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:00.715+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:11:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:00.715+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:11:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:00.715+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:11:01.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:01 smithi153 ceph-mon[36569]: pgmap v5922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:01.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:01 smithi104 ceph-mon[36624]: pgmap v5922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:01.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:01 smithi081 ceph-mon[31453]: pgmap v5922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:03.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:03 smithi153 ceph-mon[36569]: pgmap v5923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:03.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:03 smithi104 ceph-mon[36624]: pgmap v5923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:03.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:03 smithi081 ceph-mon[31453]: pgmap v5923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:05.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:05 smithi153 ceph-mon[36569]: pgmap v5924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:05.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:05 smithi104 ceph-mon[36624]: pgmap v5924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:05.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:05 smithi081 ceph-mon[31453]: pgmap v5924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:07.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:07 smithi153 ceph-mon[36569]: pgmap v5925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:07.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:07 smithi104 ceph-mon[36624]: pgmap v5925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:07.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:07 smithi081 ceph-mon[31453]: pgmap v5925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:09.436 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:09 smithi081 ceph-mon[31453]: pgmap v5926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:09.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:09 smithi153 ceph-mon[36569]: pgmap v5926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:09.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:09 smithi104 ceph-mon[36624]: pgmap v5926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:11.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:11 smithi153 ceph-mon[36569]: pgmap v5927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:11.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:11 smithi104 ceph-mon[36624]: pgmap v5927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:11.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:11 smithi081 ceph-mon[31453]: pgmap v5927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:13.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:13 smithi153 ceph-mon[36569]: pgmap v5928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:13.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:13 smithi104 ceph-mon[36624]: pgmap v5928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:13.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:13 smithi081 ceph-mon[31453]: pgmap v5928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:15.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:15 smithi153 ceph-mon[36569]: pgmap v5929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:15.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:15 smithi104 ceph-mon[36624]: pgmap v5929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:15.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:15 smithi081 ceph-mon[31453]: pgmap v5929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:15.731+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:11:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:15.731+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:11:15.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:15.731+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:11:17.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:17 smithi153 ceph-mon[36569]: pgmap v5930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:17.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:17 smithi104 ceph-mon[36624]: pgmap v5930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:17.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:17 smithi081 ceph-mon[31453]: pgmap v5930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:19.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:19 smithi153 ceph-mon[36569]: pgmap v5931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:19.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:19 smithi104 ceph-mon[36624]: pgmap v5931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:19.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:19 smithi081 ceph-mon[31453]: pgmap v5931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:21.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:21 smithi153 ceph-mon[36569]: pgmap v5932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:21.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:21 smithi104 ceph-mon[36624]: pgmap v5932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:21.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:21 smithi081 ceph-mon[31453]: pgmap v5932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:23.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:23 smithi081 ceph-mon[31453]: pgmap v5933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:23.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:23 smithi153 ceph-mon[36569]: pgmap v5933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:23.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:23 smithi104 ceph-mon[36624]: pgmap v5933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:25.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:25 smithi153 ceph-mon[36569]: pgmap v5934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:25.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:25 smithi104 ceph-mon[36624]: pgmap v5934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:25.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:25 smithi081 ceph-mon[31453]: pgmap v5934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:27.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:27 smithi153 ceph-mon[36569]: pgmap v5935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:27.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:27 smithi104 ceph-mon[36624]: pgmap v5935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:27.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:27 smithi081 ceph-mon[31453]: pgmap v5935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:29.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:29 smithi153 ceph-mon[36569]: pgmap v5936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:29.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:29 smithi104 ceph-mon[36624]: pgmap v5936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:29.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:29 smithi081 ceph-mon[31453]: pgmap v5936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:30.735+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:11:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:30.735+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:11:30.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:30.735+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:11:31.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:31 smithi153 ceph-mon[36569]: pgmap v5937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:31.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:31 smithi104 ceph-mon[36624]: pgmap v5937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:31.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:31 smithi081 ceph-mon[31453]: pgmap v5937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:33.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:33 smithi153 ceph-mon[36569]: pgmap v5938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:33.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:33 smithi104 ceph-mon[36624]: pgmap v5938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:33.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:33 smithi081 ceph-mon[31453]: pgmap v5938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:35.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:35 smithi153 ceph-mon[36569]: pgmap v5939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:35.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:35 smithi104 ceph-mon[36624]: pgmap v5939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:35.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:35 smithi081 ceph-mon[31453]: pgmap v5939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:37.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:37 smithi153 ceph-mon[36569]: pgmap v5940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:37.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:37 smithi104 ceph-mon[36624]: pgmap v5940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:37.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:37 smithi081 ceph-mon[31453]: pgmap v5940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:39.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:39 smithi153 ceph-mon[36569]: pgmap v5941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:39.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:39 smithi104 ceph-mon[36624]: pgmap v5941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:39.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:39 smithi081 ceph-mon[31453]: pgmap v5941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:41.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:41 smithi153 ceph-mon[36569]: pgmap v5942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:41.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:41 smithi104 ceph-mon[36624]: pgmap v5942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:41.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:41 smithi081 ceph-mon[31453]: pgmap v5942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:43.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:43 smithi153 ceph-mon[36569]: pgmap v5943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:43.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:43 smithi104 ceph-mon[36624]: pgmap v5943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:43.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:43 smithi081 ceph-mon[31453]: pgmap v5943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:45.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:45 smithi153 ceph-mon[36569]: pgmap v5944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:45.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:45 smithi104 ceph-mon[36624]: pgmap v5944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:45.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:45 smithi081 ceph-mon[31453]: pgmap v5944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:45.751+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:11:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:45.751+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:11:45.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:11:45.751+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:11:47.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:47 smithi104 ceph-mon[36624]: pgmap v5945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:47.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:47 smithi081 ceph-mon[31453]: pgmap v5945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:47.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:47 smithi153 ceph-mon[36569]: pgmap v5945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:49.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:49 smithi104 ceph-mon[36624]: pgmap v5946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:49.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:49 smithi081 ceph-mon[31453]: pgmap v5946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:49.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:49 smithi153 ceph-mon[36569]: pgmap v5946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:51.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:51 smithi104 ceph-mon[36624]: pgmap v5947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:51.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:51 smithi081 ceph-mon[31453]: pgmap v5947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:51.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:51 smithi153 ceph-mon[36569]: pgmap v5947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:53.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:53 smithi104 ceph-mon[36624]: pgmap v5948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:53.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:53 smithi081 ceph-mon[31453]: pgmap v5948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:53.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:53 smithi153 ceph-mon[36569]: pgmap v5948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:55.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:55 smithi104 ceph-mon[36624]: pgmap v5949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:55.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:55 smithi081 ceph-mon[31453]: pgmap v5949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:55.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:55 smithi153 ceph-mon[36569]: pgmap v5949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:57.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:57 smithi081 ceph-mon[31453]: pgmap v5950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:57.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:11:57.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:57 smithi153 ceph-mon[36569]: pgmap v5950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:57.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:11:57.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:57 smithi104 ceph-mon[36624]: pgmap v5950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:57.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:11:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:11:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:11:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:58 smithi081 ceph-mon[31453]: pgmap v5951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:58 smithi081 ceph-mon[31453]: pgmap v5952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:58.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:11:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:11:58.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:11:58.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:11:58.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:58 smithi153 ceph-mon[36569]: pgmap v5951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:58.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:58 smithi153 ceph-mon[36569]: pgmap v5952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:58.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:11:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:11:58.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:11:58.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:11:58.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:58 smithi104 ceph-mon[36624]: pgmap v5951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:58.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:58 smithi104 ceph-mon[36624]: pgmap v5952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:11:58.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:11:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:00.763+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:12:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:00.763+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:12:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:00.763+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:12:01.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:00 smithi153 ceph-mon[36569]: pgmap v5953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:01.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:00 smithi104 ceph-mon[36624]: pgmap v5953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:01.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:00 smithi081 ceph-mon[31453]: pgmap v5953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:03.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:02 smithi153 ceph-mon[36569]: pgmap v5954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:03.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:02 smithi104 ceph-mon[36624]: pgmap v5954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:03.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:02 smithi081 ceph-mon[31453]: pgmap v5954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:05.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:04 smithi153 ceph-mon[36569]: pgmap v5955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:05.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:04 smithi104 ceph-mon[36624]: pgmap v5955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:05.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:04 smithi081 ceph-mon[31453]: pgmap v5955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:07.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:06 smithi153 ceph-mon[36569]: pgmap v5956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:07.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:06 smithi104 ceph-mon[36624]: pgmap v5956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:07.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:06 smithi081 ceph-mon[31453]: pgmap v5956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:09.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:08 smithi153 ceph-mon[36569]: pgmap v5957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:09.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:08 smithi104 ceph-mon[36624]: pgmap v5957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:09.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:08 smithi081 ceph-mon[31453]: pgmap v5957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:11.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:10 smithi153 ceph-mon[36569]: pgmap v5958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:11.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:10 smithi104 ceph-mon[36624]: pgmap v5958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:11.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:10 smithi081 ceph-mon[31453]: pgmap v5958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:13.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:12 smithi153 ceph-mon[36569]: pgmap v5959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:13.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:12 smithi104 ceph-mon[36624]: pgmap v5959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:13.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:12 smithi081 ceph-mon[31453]: pgmap v5959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:15.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:14 smithi153 ceph-mon[36569]: pgmap v5960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:15.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:14 smithi104 ceph-mon[36624]: pgmap v5960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:15.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:14 smithi081 ceph-mon[31453]: pgmap v5960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:15.778+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:12:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:15.778+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:12:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:15.778+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:12:17.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:16 smithi153 ceph-mon[36569]: pgmap v5961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:17.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:16 smithi104 ceph-mon[36624]: pgmap v5961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:17.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:16 smithi081 ceph-mon[31453]: pgmap v5961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:19.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:18 smithi153 ceph-mon[36569]: pgmap v5962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:19.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:18 smithi104 ceph-mon[36624]: pgmap v5962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:19.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:18 smithi081 ceph-mon[31453]: pgmap v5962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:20.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:19 smithi153 ceph-mon[36569]: pgmap v5963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:20.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:19 smithi104 ceph-mon[36624]: pgmap v5963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:20.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:19 smithi081 ceph-mon[31453]: pgmap v5963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:22.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:22 smithi153 ceph-mon[36569]: pgmap v5964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:22.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:22 smithi104 ceph-mon[36624]: pgmap v5964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:22.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:22 smithi081 ceph-mon[31453]: pgmap v5964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:25.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:24 smithi153 ceph-mon[36569]: pgmap v5965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:25.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:24 smithi104 ceph-mon[36624]: pgmap v5965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:25.072 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:24 smithi081 ceph-mon[31453]: pgmap v5965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:27.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:26 smithi153 ceph-mon[36569]: pgmap v5966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:27.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:26 smithi104 ceph-mon[36624]: pgmap v5966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:27.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:26 smithi081 ceph-mon[31453]: pgmap v5966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:29.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:28 smithi153 ceph-mon[36569]: pgmap v5967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:29.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:28 smithi104 ceph-mon[36624]: pgmap v5967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:29.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:28 smithi081 ceph-mon[31453]: pgmap v5967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:30.794+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:12:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:30.794+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:12:30.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:30.794+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:12:31.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:30 smithi153 ceph-mon[36569]: pgmap v5968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:31.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:30 smithi104 ceph-mon[36624]: pgmap v5968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:31.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:30 smithi081 ceph-mon[31453]: pgmap v5968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:33.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:32 smithi153 ceph-mon[36569]: pgmap v5969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:33.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:32 smithi104 ceph-mon[36624]: pgmap v5969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:32 smithi081 ceph-mon[31453]: pgmap v5969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:35.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:34 smithi153 ceph-mon[36569]: pgmap v5970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:35.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:34 smithi104 ceph-mon[36624]: pgmap v5970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:35.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:34 smithi081 ceph-mon[31453]: pgmap v5970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:37.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:36 smithi153 ceph-mon[36569]: pgmap v5971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:37.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:36 smithi104 ceph-mon[36624]: pgmap v5971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:37.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:36 smithi081 ceph-mon[31453]: pgmap v5971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:39.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:38 smithi153 ceph-mon[36569]: pgmap v5972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:39.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:38 smithi104 ceph-mon[36624]: pgmap v5972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:39.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:38 smithi081 ceph-mon[31453]: pgmap v5972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:40.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:39 smithi153 ceph-mon[36569]: pgmap v5973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:40.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:39 smithi104 ceph-mon[36624]: pgmap v5973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:40.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:39 smithi081 ceph-mon[31453]: pgmap v5973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:43.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:42 smithi153 ceph-mon[36569]: pgmap v5974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:43.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:42 smithi104 ceph-mon[36624]: pgmap v5974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:43.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:42 smithi081 ceph-mon[31453]: pgmap v5974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:45.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:44 smithi153 ceph-mon[36569]: pgmap v5975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:45.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:44 smithi104 ceph-mon[36624]: pgmap v5975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:45.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:44 smithi081 ceph-mon[31453]: pgmap v5975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:45.807+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:12:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:45.807+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:12:45.784 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:12:45.807+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:12:47.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:46 smithi153 ceph-mon[36569]: pgmap v5976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:47.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:46 smithi104 ceph-mon[36624]: pgmap v5976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:47.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:46 smithi081 ceph-mon[31453]: pgmap v5976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:49.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:48 smithi153 ceph-mon[36569]: pgmap v5977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:49.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:48 smithi104 ceph-mon[36624]: pgmap v5977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:49.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:48 smithi081 ceph-mon[31453]: pgmap v5977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:51.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:50 smithi153 ceph-mon[36569]: pgmap v5978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:51.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:50 smithi104 ceph-mon[36624]: pgmap v5978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:51.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:50 smithi081 ceph-mon[31453]: pgmap v5978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:53.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:52 smithi153 ceph-mon[36569]: pgmap v5979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:53.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:52 smithi104 ceph-mon[36624]: pgmap v5979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:53.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:52 smithi081 ceph-mon[31453]: pgmap v5979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:55.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:54 smithi153 ceph-mon[36569]: pgmap v5980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:55.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:54 smithi104 ceph-mon[36624]: pgmap v5980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:55.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:54 smithi081 ceph-mon[31453]: pgmap v5980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:56.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:55 smithi153 ceph-mon[36569]: pgmap v5981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:56.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:55 smithi104 ceph-mon[36624]: pgmap v5981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:56.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:55 smithi081 ceph-mon[31453]: pgmap v5981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:59.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:58 smithi153 ceph-mon[36569]: pgmap v5982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:59.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:12:59.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:59.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:59.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:58 smithi104 ceph-mon[36624]: pgmap v5982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:59.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:12:59.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:59.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:59.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:58 smithi081 ceph-mon[31453]: pgmap v5982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:12:59.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:12:59.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:59.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:59.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:59.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:12:59.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:12:59 smithi104 ceph-mon[36624]: pgmap v5983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:00.027 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:00.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:00.028 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:12:59 smithi081 ceph-mon[31453]: pgmap v5983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:00.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:00.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:00.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:12:59 smithi153 ceph-mon[36569]: pgmap v5983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:00.822+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:13:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:00.822+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:13:00.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:00.822+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:13:01.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:01.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:02.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:02 smithi104 ceph-mon[36624]: pgmap v5984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:02.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:02 smithi153 ceph-mon[36569]: pgmap v5984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:02.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:02 smithi081 ceph-mon[31453]: pgmap v5984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:04.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:04.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:04.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:04 smithi104 ceph-mon[36624]: pgmap v5985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:04.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:04.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:04.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:04 smithi153 ceph-mon[36569]: pgmap v5985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:04.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:04.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:04.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:04 smithi081 ceph-mon[31453]: pgmap v5985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:06.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:06.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:06.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:06 smithi153 ceph-mon[36569]: pgmap v5986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:06.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:06.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:06.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:06 smithi104 ceph-mon[36624]: pgmap v5986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:06.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:06.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:06.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:06 smithi081 ceph-mon[31453]: pgmap v5986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:09.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:08 smithi153 ceph-mon[36569]: pgmap v5987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:09.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:08 smithi104 ceph-mon[36624]: pgmap v5987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:09.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:08 smithi081 ceph-mon[31453]: pgmap v5987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:11.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:10 smithi153 ceph-mon[36569]: pgmap v5988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:11.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:11.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:11.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:13:11.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:13:11.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:11.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:10 smithi104 ceph-mon[36624]: pgmap v5988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:11.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:11.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:11.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:13:11.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:13:11.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:11.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:10 smithi081 ceph-mon[31453]: pgmap v5988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:11.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:11.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:11.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:13:11.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:13:11.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:13:12.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:11 smithi153 ceph-mon[36569]: pgmap v5989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:12.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:11 smithi153 ceph-mon[36569]: pgmap v5990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:12.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:11 smithi104 ceph-mon[36624]: pgmap v5989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:12.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:11 smithi104 ceph-mon[36624]: pgmap v5990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:12.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:11 smithi081 ceph-mon[31453]: pgmap v5989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:12.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:11 smithi081 ceph-mon[31453]: pgmap v5990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:14.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:13 smithi153 ceph-mon[36569]: pgmap v5991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:14.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:13 smithi104 ceph-mon[36624]: pgmap v5991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:14.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:13 smithi081 ceph-mon[31453]: pgmap v5991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:14.430 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:14.453+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:13:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:14.453+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:13:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:14.454+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:13:14.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:14.454+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:13:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:15.837+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:13:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:15.838+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:13:15.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:15.838+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:13:16.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:15 smithi153 ceph-mon[36569]: pgmap v5992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:16.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:15 smithi104 ceph-mon[36624]: pgmap v5992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:16.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:15 smithi081 ceph-mon[31453]: pgmap v5992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:17.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:16 smithi153 ceph-mon[36569]: pgmap v5993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:17.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:16 smithi104 ceph-mon[36624]: pgmap v5993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:17.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:16 smithi081 ceph-mon[31453]: pgmap v5993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:19.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:19 smithi081 ceph-mon[31453]: pgmap v5994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:19.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:19 smithi153 ceph-mon[36569]: pgmap v5994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:19.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:19 smithi104 ceph-mon[36624]: pgmap v5994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:21.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:21 smithi081 ceph-mon[31453]: pgmap v5995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:21.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:21 smithi153 ceph-mon[36569]: pgmap v5995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:21.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:21 smithi104 ceph-mon[36624]: pgmap v5995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:23.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:23 smithi081 ceph-mon[31453]: pgmap v5996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:23.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:23 smithi153 ceph-mon[36569]: pgmap v5996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:23.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:23 smithi104 ceph-mon[36624]: pgmap v5996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:25.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:25 smithi081 ceph-mon[31453]: pgmap v5997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:25.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:25 smithi153 ceph-mon[36569]: pgmap v5997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:25 smithi104 ceph-mon[36624]: pgmap v5997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:27.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:27 smithi081 ceph-mon[31453]: pgmap v5998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:27.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:27 smithi153 ceph-mon[36569]: pgmap v5998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:27.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:27 smithi104 ceph-mon[36624]: pgmap v5998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:29.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:29 smithi081 ceph-mon[31453]: pgmap v5999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:29.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:29 smithi153 ceph-mon[36569]: pgmap v5999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:29.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:29 smithi104 ceph-mon[36624]: pgmap v5999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:30.854+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:13:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:30.854+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:13:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:30.854+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:13:31.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:31 smithi081 ceph-mon[31453]: pgmap v6000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:31.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:31 smithi153 ceph-mon[36569]: pgmap v6000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:31.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:31 smithi104 ceph-mon[36624]: pgmap v6000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:33.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:33 smithi081 ceph-mon[31453]: pgmap v6001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:33.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:33 smithi153 ceph-mon[36569]: pgmap v6001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:33.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:33 smithi104 ceph-mon[36624]: pgmap v6001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:35.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:35 smithi081 ceph-mon[31453]: pgmap v6002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:35.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:35 smithi153 ceph-mon[36569]: pgmap v6002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:35.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:35 smithi104 ceph-mon[36624]: pgmap v6002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:37.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:37 smithi081 ceph-mon[31453]: pgmap v6003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:37.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:37 smithi153 ceph-mon[36569]: pgmap v6003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:37.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:37 smithi104 ceph-mon[36624]: pgmap v6003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:39.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:39 smithi081 ceph-mon[31453]: pgmap v6004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:39.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:39 smithi153 ceph-mon[36569]: pgmap v6004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:39.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:39 smithi104 ceph-mon[36624]: pgmap v6004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:41.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:41 smithi081 ceph-mon[31453]: pgmap v6005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:41.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:41 smithi153 ceph-mon[36569]: pgmap v6005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:41.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:41 smithi104 ceph-mon[36624]: pgmap v6005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:43.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:43 smithi153 ceph-mon[36569]: pgmap v6006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:43.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:43 smithi104 ceph-mon[36624]: pgmap v6006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:43.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:43 smithi081 ceph-mon[31453]: pgmap v6006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:45.685 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:45 smithi081 ceph-mon[31453]: pgmap v6007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:45.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:45 smithi153 ceph-mon[36569]: pgmap v6007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:45.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:45 smithi104 ceph-mon[36624]: pgmap v6007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:45.858+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:13:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:45.858+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:13:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:13:45.858+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:13:47.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:47 smithi153 ceph-mon[36569]: pgmap v6008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:47.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:47 smithi104 ceph-mon[36624]: pgmap v6008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:47.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:47 smithi081 ceph-mon[31453]: pgmap v6008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:49.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:49 smithi153 ceph-mon[36569]: pgmap v6009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:49.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:49 smithi104 ceph-mon[36624]: pgmap v6009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:49.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:49 smithi081 ceph-mon[31453]: pgmap v6009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:51.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:51 smithi153 ceph-mon[36569]: pgmap v6010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:51.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:51 smithi104 ceph-mon[36624]: pgmap v6010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:51.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:51 smithi081 ceph-mon[31453]: pgmap v6010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:53.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:53 smithi153 ceph-mon[36569]: pgmap v6011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:53.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:53 smithi104 ceph-mon[36624]: pgmap v6011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:53.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:53 smithi081 ceph-mon[31453]: pgmap v6011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:55.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:55 smithi153 ceph-mon[36569]: pgmap v6012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:55.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:55 smithi104 ceph-mon[36624]: pgmap v6012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:55.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:55 smithi081 ceph-mon[31453]: pgmap v6012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:57.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:57 smithi153 ceph-mon[36569]: pgmap v6013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:57.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:57 smithi104 ceph-mon[36624]: pgmap v6013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:57.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:57 smithi081 ceph-mon[31453]: pgmap v6013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:59.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:13:59 smithi153 ceph-mon[36569]: pgmap v6014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:59.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:13:59 smithi104 ceph-mon[36624]: pgmap v6014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:13:59.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:13:59 smithi081 ceph-mon[31453]: pgmap v6014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:00.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:00.865+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:14:00.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:00.865+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:14:00.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:00.865+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:14:01.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:01 smithi153 ceph-mon[36569]: pgmap v6015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:01.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:01 smithi104 ceph-mon[36624]: pgmap v6015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:01.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:01 smithi081 ceph-mon[31453]: pgmap v6015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:03.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:03 smithi153 ceph-mon[36569]: pgmap v6016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:03.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:03 smithi104 ceph-mon[36624]: pgmap v6016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:03.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:03 smithi081 ceph-mon[31453]: pgmap v6016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:05.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:05 smithi153 ceph-mon[36569]: pgmap v6017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:05.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:05 smithi104 ceph-mon[36624]: pgmap v6017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:05.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:05 smithi081 ceph-mon[31453]: pgmap v6017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:07.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:07 smithi153 ceph-mon[36569]: pgmap v6018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:07.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:07 smithi104 ceph-mon[36624]: pgmap v6018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:07.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:07 smithi081 ceph-mon[31453]: pgmap v6018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:09.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:09 smithi153 ceph-mon[36569]: pgmap v6019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:09.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:09 smithi104 ceph-mon[36624]: pgmap v6019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:09.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:09 smithi081 ceph-mon[31453]: pgmap v6019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:11.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:11 smithi153 ceph-mon[36569]: pgmap v6020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:11.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:14:11.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:11.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:11.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:11 smithi104 ceph-mon[36624]: pgmap v6020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:11.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:14:11.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:11.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:11.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:11 smithi081 ceph-mon[31453]: pgmap v6020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:11.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:14:11.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:11.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:13.657 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:13 smithi153 ceph-mon[36569]: pgmap v6021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:13.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:13 smithi104 ceph-mon[36624]: pgmap v6021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:13.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:13 smithi081 ceph-mon[31453]: pgmap v6021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:15.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:15 smithi153 ceph-mon[36569]: pgmap v6022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:15.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:15 smithi104 ceph-mon[36624]: pgmap v6022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:15.881+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:14:15.858 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:15.881+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:14:15.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:15.881+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:14:15.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:15 smithi081 ceph-mon[31453]: pgmap v6022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:17.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:17.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:17.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:14:17.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:14:17.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:17 smithi081 ceph-mon[31453]: pgmap v6023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:17.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:17 smithi081 ceph-mon[31453]: pgmap v6024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:17.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:17.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:17.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:17.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:14:17.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:14:17.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:17 smithi153 ceph-mon[36569]: pgmap v6023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:17.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:17 smithi153 ceph-mon[36569]: pgmap v6024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:17.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:17.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:17.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:17.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:14:17.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:14:17.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:17 smithi104 ceph-mon[36624]: pgmap v6023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:17.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:17 smithi104 ceph-mon[36624]: pgmap v6024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:17.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:14:19.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:19 smithi153 ceph-mon[36569]: pgmap v6025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:19.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:19 smithi104 ceph-mon[36624]: pgmap v6025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:19.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:19 smithi081 ceph-mon[31453]: pgmap v6025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:21.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:21 smithi153 ceph-mon[36569]: pgmap v6026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:21.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:21 smithi104 ceph-mon[36624]: pgmap v6026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:21.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:21 smithi081 ceph-mon[31453]: pgmap v6026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:23.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:23 smithi153 ceph-mon[36569]: pgmap v6027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:23.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:23 smithi104 ceph-mon[36624]: pgmap v6027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:23.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:23 smithi081 ceph-mon[31453]: pgmap v6027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:25.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:25 smithi153 ceph-mon[36569]: pgmap v6028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:25.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:25 smithi104 ceph-mon[36624]: pgmap v6028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:25.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:25 smithi081 ceph-mon[31453]: pgmap v6028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:27.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:27 smithi153 ceph-mon[36569]: pgmap v6029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:27.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:27 smithi104 ceph-mon[36624]: pgmap v6029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:27.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:27 smithi081 ceph-mon[31453]: pgmap v6029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:29.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:29 smithi153 ceph-mon[36569]: pgmap v6030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:29.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:29 smithi104 ceph-mon[36624]: pgmap v6030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:29.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:29 smithi081 ceph-mon[31453]: pgmap v6030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:30.892+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:14:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:30.892+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:14:30.870 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:30.892+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:14:31.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:31 smithi153 ceph-mon[36569]: pgmap v6031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:31.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:31 smithi104 ceph-mon[36624]: pgmap v6031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:31.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:31 smithi081 ceph-mon[31453]: pgmap v6031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:33.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:33 smithi153 ceph-mon[36569]: pgmap v6032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:33.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:33 smithi104 ceph-mon[36624]: pgmap v6032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:33.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:33 smithi081 ceph-mon[31453]: pgmap v6032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:35.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:35 smithi153 ceph-mon[36569]: pgmap v6033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:35.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:35 smithi104 ceph-mon[36624]: pgmap v6033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:35.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:35 smithi081 ceph-mon[31453]: pgmap v6033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:37.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:37 smithi153 ceph-mon[36569]: pgmap v6034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:37.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:37 smithi104 ceph-mon[36624]: pgmap v6034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:37.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:37 smithi081 ceph-mon[31453]: pgmap v6034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:39.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:39 smithi153 ceph-mon[36569]: pgmap v6035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:39.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:39 smithi104 ceph-mon[36624]: pgmap v6035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:39.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:39 smithi081 ceph-mon[31453]: pgmap v6035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:41.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:41 smithi153 ceph-mon[36569]: pgmap v6036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:41.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:41 smithi104 ceph-mon[36624]: pgmap v6036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:41.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:41 smithi081 ceph-mon[31453]: pgmap v6036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:43.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:43 smithi153 ceph-mon[36569]: pgmap v6037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:43.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:43 smithi104 ceph-mon[36624]: pgmap v6037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:43.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:43 smithi081 ceph-mon[31453]: pgmap v6037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:45.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:45 smithi153 ceph-mon[36569]: pgmap v6038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:45.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:45 smithi104 ceph-mon[36624]: pgmap v6038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:45.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:45 smithi081 ceph-mon[31453]: pgmap v6038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:45.904+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:14:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:45.904+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:14:45.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:14:45.904+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:14:47.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:47 smithi153 ceph-mon[36569]: pgmap v6039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:47.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:47 smithi104 ceph-mon[36624]: pgmap v6039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:47.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:47 smithi081 ceph-mon[31453]: pgmap v6039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:49.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:49 smithi153 ceph-mon[36569]: pgmap v6040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:49.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:49 smithi104 ceph-mon[36624]: pgmap v6040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:49.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:49 smithi081 ceph-mon[31453]: pgmap v6040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:51.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:51 smithi153 ceph-mon[36569]: pgmap v6041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:51.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:51 smithi104 ceph-mon[36624]: pgmap v6041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:51.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:51 smithi081 ceph-mon[31453]: pgmap v6041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:53.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:53 smithi104 ceph-mon[36624]: pgmap v6042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:53.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:53 smithi081 ceph-mon[31453]: pgmap v6042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:53.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:53 smithi153 ceph-mon[36569]: pgmap v6042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:55.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:55 smithi104 ceph-mon[36624]: pgmap v6043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:55.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:55 smithi081 ceph-mon[31453]: pgmap v6043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:55.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:55 smithi153 ceph-mon[36569]: pgmap v6043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:57.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:57 smithi104 ceph-mon[36624]: pgmap v6044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:57.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:57 smithi081 ceph-mon[31453]: pgmap v6044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:57.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:57 smithi153 ceph-mon[36569]: pgmap v6044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:59.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:14:59 smithi104 ceph-mon[36624]: pgmap v6045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:59.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:14:59 smithi081 ceph-mon[31453]: pgmap v6045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:14:59.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:14:59 smithi153 ceph-mon[36569]: pgmap v6045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:00.921+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:15:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:00.921+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:15:00.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:00.921+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:15:01.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:01 smithi104 ceph-mon[36624]: pgmap v6046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:01.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:01 smithi081 ceph-mon[31453]: pgmap v6046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:01.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:01 smithi153 ceph-mon[36569]: pgmap v6046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:03.588 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:03 smithi153 ceph-mon[36569]: pgmap v6047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:03.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:03 smithi081 ceph-mon[31453]: pgmap v6047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:03.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:03 smithi104 ceph-mon[36624]: pgmap v6047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:05.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:05 smithi081 ceph-mon[31453]: pgmap v6048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:05.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:05 smithi153 ceph-mon[36569]: pgmap v6048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:05.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:05 smithi104 ceph-mon[36624]: pgmap v6048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:07.614 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:07 smithi104 ceph-mon[36624]: pgmap v6049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:07.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:07 smithi081 ceph-mon[31453]: pgmap v6049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:07.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:07 smithi153 ceph-mon[36569]: pgmap v6049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:09.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:09 smithi081 ceph-mon[31453]: pgmap v6050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:09.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:09 smithi153 ceph-mon[36569]: pgmap v6050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:09.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:09 smithi104 ceph-mon[36624]: pgmap v6050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:11.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:11 smithi081 ceph-mon[31453]: pgmap v6051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:11.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:11 smithi153 ceph-mon[36569]: pgmap v6051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:11.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:11 smithi104 ceph-mon[36624]: pgmap v6051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:13.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:13 smithi081 ceph-mon[31453]: pgmap v6052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:13.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:13 smithi153 ceph-mon[36569]: pgmap v6052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:13.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:13 smithi104 ceph-mon[36624]: pgmap v6052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:15.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:15 smithi081 ceph-mon[31453]: pgmap v6053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:15.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:15 smithi153 ceph-mon[36569]: pgmap v6053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:15.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:15 smithi104 ceph-mon[36624]: pgmap v6053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:15.929+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:15:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:15.929+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:15:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:15.929+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:15:17.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:17 smithi081 ceph-mon[31453]: pgmap v6054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:17.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:15:17.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:17 smithi153 ceph-mon[36569]: pgmap v6054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:17.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:15:17.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:17 smithi104 ceph-mon[36624]: pgmap v6054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:17.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:15:18.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:15:18.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:15:18.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:18 smithi081 ceph-mon[31453]: pgmap v6055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:18.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:18 smithi081 ceph-mon[31453]: pgmap v6056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:18.636 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:15:18.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:15:18.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:15:18.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:18 smithi153 ceph-mon[36569]: pgmap v6055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:18.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:18 smithi153 ceph-mon[36569]: pgmap v6056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:18.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:15:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:15:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:15:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:18 smithi104 ceph-mon[36624]: pgmap v6055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:18 smithi104 ceph-mon[36624]: pgmap v6056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:15:20.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:19 smithi104 ceph-mon[36624]: pgmap v6057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:20.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:19 smithi081 ceph-mon[31453]: pgmap v6057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:20.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:19 smithi153 ceph-mon[36569]: pgmap v6057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:22.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:21 smithi104 ceph-mon[36624]: pgmap v6058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:22.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:21 smithi081 ceph-mon[31453]: pgmap v6058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:22.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:21 smithi153 ceph-mon[36569]: pgmap v6058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:24.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:23 smithi104 ceph-mon[36624]: pgmap v6059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:24.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:23 smithi081 ceph-mon[31453]: pgmap v6059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:24.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:23 smithi153 ceph-mon[36569]: pgmap v6059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:26.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:25 smithi104 ceph-mon[36624]: pgmap v6060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:26.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:25 smithi081 ceph-mon[31453]: pgmap v6060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:26.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:25 smithi153 ceph-mon[36569]: pgmap v6060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:28.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:27 smithi104 ceph-mon[36624]: pgmap v6061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:28.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:27 smithi081 ceph-mon[31453]: pgmap v6061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:28.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:27 smithi153 ceph-mon[36569]: pgmap v6061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:30.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:29 smithi104 ceph-mon[36624]: pgmap v6062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:29 smithi081 ceph-mon[31453]: pgmap v6062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:30.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:29 smithi153 ceph-mon[36569]: pgmap v6062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:30.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:30.943+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:15:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:30.943+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:15:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:30.943+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:15:32.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:31 smithi104 ceph-mon[36624]: pgmap v6063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:32.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:31 smithi081 ceph-mon[31453]: pgmap v6063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:32.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:31 smithi153 ceph-mon[36569]: pgmap v6063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:34.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:33 smithi104 ceph-mon[36624]: pgmap v6064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:34.088 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:33 smithi153 ceph-mon[36569]: pgmap v6064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:34.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:33 smithi081 ceph-mon[31453]: pgmap v6064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:36.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:35 smithi081 ceph-mon[31453]: pgmap v6065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:36.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:35 smithi153 ceph-mon[36569]: pgmap v6065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:36.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:35 smithi104 ceph-mon[36624]: pgmap v6065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:38.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:37 smithi104 ceph-mon[36624]: pgmap v6066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:38.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:37 smithi081 ceph-mon[31453]: pgmap v6066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:38.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:37 smithi153 ceph-mon[36569]: pgmap v6066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:40.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:39 smithi104 ceph-mon[36624]: pgmap v6067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:40.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:39 smithi081 ceph-mon[31453]: pgmap v6067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:40.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:39 smithi153 ceph-mon[36569]: pgmap v6067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:42.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:41 smithi104 ceph-mon[36624]: pgmap v6068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:42.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:41 smithi081 ceph-mon[31453]: pgmap v6068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:42.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:41 smithi153 ceph-mon[36569]: pgmap v6068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:44.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:43 smithi104 ceph-mon[36624]: pgmap v6069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:44.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:43 smithi081 ceph-mon[31453]: pgmap v6069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:44.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:43 smithi153 ceph-mon[36569]: pgmap v6069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:45.953+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:15:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:45.953+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:15:45.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:15:45.953+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:15:46.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:45 smithi104 ceph-mon[36624]: pgmap v6070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:46.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:45 smithi081 ceph-mon[31453]: pgmap v6070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:46.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:45 smithi153 ceph-mon[36569]: pgmap v6070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:48.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:47 smithi104 ceph-mon[36624]: pgmap v6071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:48.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:47 smithi081 ceph-mon[31453]: pgmap v6071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:48.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:47 smithi153 ceph-mon[36569]: pgmap v6071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:50.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:49 smithi104 ceph-mon[36624]: pgmap v6072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:50.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:49 smithi081 ceph-mon[31453]: pgmap v6072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:50.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:49 smithi153 ceph-mon[36569]: pgmap v6072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:52.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:51 smithi104 ceph-mon[36624]: pgmap v6073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:52.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:51 smithi081 ceph-mon[31453]: pgmap v6073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:52.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:51 smithi153 ceph-mon[36569]: pgmap v6073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:54.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:53 smithi104 ceph-mon[36624]: pgmap v6074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:54.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:53 smithi081 ceph-mon[31453]: pgmap v6074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:53 smithi153 ceph-mon[36569]: pgmap v6074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:56.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:55 smithi081 ceph-mon[31453]: pgmap v6075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:56.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:55 smithi153 ceph-mon[36569]: pgmap v6075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:56.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:55 smithi104 ceph-mon[36624]: pgmap v6075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:58.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:57 smithi081 ceph-mon[31453]: pgmap v6076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:58.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:57 smithi153 ceph-mon[36569]: pgmap v6076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:15:58.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:57 smithi104 ceph-mon[36624]: pgmap v6076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:00.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:15:59 smithi081 ceph-mon[31453]: pgmap v6077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:00.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:15:59 smithi153 ceph-mon[36569]: pgmap v6077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:00.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:15:59 smithi104 ceph-mon[36624]: pgmap v6077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:00.968+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:16:00.950 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:00.969+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:16:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:00.969+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:16:02.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:01 smithi081 ceph-mon[31453]: pgmap v6078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:02.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:01 smithi153 ceph-mon[36569]: pgmap v6078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:02.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:01 smithi104 ceph-mon[36624]: pgmap v6078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:04.090 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:03 smithi153 ceph-mon[36569]: pgmap v6079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:04.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:03 smithi081 ceph-mon[31453]: pgmap v6079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:04.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:03 smithi104 ceph-mon[36624]: pgmap v6079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:06.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:05 smithi081 ceph-mon[31453]: pgmap v6080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:06.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:05 smithi153 ceph-mon[36569]: pgmap v6080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:06.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:05 smithi104 ceph-mon[36624]: pgmap v6080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:08.116 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:07 smithi104 ceph-mon[36624]: pgmap v6081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:08.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:07 smithi081 ceph-mon[31453]: pgmap v6081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:08.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:07 smithi153 ceph-mon[36569]: pgmap v6081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:10.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:09 smithi081 ceph-mon[31453]: pgmap v6082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:10.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:09 smithi153 ceph-mon[36569]: pgmap v6082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:10.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:09 smithi104 ceph-mon[36624]: pgmap v6082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:12.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:12.117+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000658) for more than 900000000 us, fault. 2024-04-13T00:16:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:12.117+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-13T00:16:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:12.117+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T00:16:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:12.117+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T00:16:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:12.117+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T00:16:12.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:12.117+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T00:16:12.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:12.119+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e9c0 tx=0x7f16f40d6000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T00:16:12.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:11 smithi081 ceph-mon[31453]: pgmap v6083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:12.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:11 smithi153 ceph-mon[36569]: pgmap v6083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:12.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:11 smithi104 ceph-mon[36624]: pgmap v6083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:14.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:13 smithi081 ceph-mon[31453]: pgmap v6084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:14.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:13 smithi153 ceph-mon[36569]: pgmap v6084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:14.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:13 smithi104 ceph-mon[36624]: pgmap v6084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:14.483+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:14.483+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:14.483+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:14.484+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:16:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:15.983+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:16:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:15.983+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:16:15.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:15.983+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:16:16.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:15 smithi081 ceph-mon[31453]: pgmap v6085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:16.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:15 smithi153 ceph-mon[36569]: pgmap v6085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:16.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:15 smithi104 ceph-mon[36624]: pgmap v6085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:18.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:17 smithi081 ceph-mon[31453]: pgmap v6086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:18.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:17 smithi153 ceph-mon[36569]: pgmap v6086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:18.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:17 smithi104 ceph-mon[36624]: pgmap v6086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:19.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:16:19.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:16:19.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:16:20.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:16:20.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:16:20.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:19 smithi081 ceph-mon[31453]: pgmap v6087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:20.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:19 smithi081 ceph-mon[31453]: pgmap v6088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:20.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:16:20.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:16:20.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:16:20.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:19 smithi153 ceph-mon[36569]: pgmap v6087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:20.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:19 smithi153 ceph-mon[36569]: pgmap v6088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:20.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:16:20.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:16:20.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:16:20.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:19 smithi104 ceph-mon[36624]: pgmap v6087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:20.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:19 smithi104 ceph-mon[36624]: pgmap v6088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:20.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:16:22.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:22 smithi153 ceph-mon[36569]: pgmap v6089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:22.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:22 smithi104 ceph-mon[36624]: pgmap v6089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:22.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:22 smithi081 ceph-mon[31453]: pgmap v6089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:24.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:24 smithi081 ceph-mon[31453]: pgmap v6090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:24.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:24 smithi153 ceph-mon[36569]: pgmap v6090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:24.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:24 smithi104 ceph-mon[36624]: pgmap v6090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:26.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:26 smithi081 ceph-mon[31453]: pgmap v6091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:26.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:26 smithi153 ceph-mon[36569]: pgmap v6091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:26.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:26 smithi104 ceph-mon[36624]: pgmap v6091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:28.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:28 smithi081 ceph-mon[31453]: pgmap v6092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:28.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:28 smithi153 ceph-mon[36569]: pgmap v6092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:28.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:28 smithi104 ceph-mon[36624]: pgmap v6092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:30.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:30 smithi081 ceph-mon[31453]: pgmap v6093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:30.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:30 smithi153 ceph-mon[36569]: pgmap v6093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:30.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:30 smithi104 ceph-mon[36624]: pgmap v6093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:30.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:30.998+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:16:30.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:30.998+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:16:30.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:30.998+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:16:32.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:32 smithi081 ceph-mon[31453]: pgmap v6094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:32.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:32 smithi153 ceph-mon[36569]: pgmap v6094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:32.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:32 smithi104 ceph-mon[36624]: pgmap v6094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:34.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:34 smithi081 ceph-mon[31453]: pgmap v6095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:34.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:34 smithi153 ceph-mon[36569]: pgmap v6095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:34.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:34 smithi104 ceph-mon[36624]: pgmap v6095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:36.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:36 smithi081 ceph-mon[31453]: pgmap v6096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:36.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:36 smithi153 ceph-mon[36569]: pgmap v6096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:36.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:36 smithi104 ceph-mon[36624]: pgmap v6096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:38.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:38 smithi081 ceph-mon[31453]: pgmap v6097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:38.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:38 smithi153 ceph-mon[36569]: pgmap v6097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:38.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:38 smithi104 ceph-mon[36624]: pgmap v6097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:40.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:40 smithi081 ceph-mon[31453]: pgmap v6098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:40.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:40 smithi153 ceph-mon[36569]: pgmap v6098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:40.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:40 smithi104 ceph-mon[36624]: pgmap v6098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:42.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:42 smithi081 ceph-mon[31453]: pgmap v6099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:42.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:42 smithi153 ceph-mon[36569]: pgmap v6099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:42.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:42 smithi104 ceph-mon[36624]: pgmap v6099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:44.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:44 smithi081 ceph-mon[31453]: pgmap v6100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:44.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:44 smithi153 ceph-mon[36569]: pgmap v6100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:44.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:44 smithi104 ceph-mon[36624]: pgmap v6100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:46.014+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:16:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:46.014+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:16:45.998 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:16:46.014+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:16:46.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:46 smithi153 ceph-mon[36569]: pgmap v6101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:46.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:46 smithi104 ceph-mon[36624]: pgmap v6101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:46.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:46 smithi081 ceph-mon[31453]: pgmap v6101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:48.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:48 smithi153 ceph-mon[36569]: pgmap v6102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:48.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:48 smithi104 ceph-mon[36624]: pgmap v6102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:48.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:48 smithi081 ceph-mon[31453]: pgmap v6102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:50.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:50 smithi153 ceph-mon[36569]: pgmap v6103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:50.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:50 smithi104 ceph-mon[36624]: pgmap v6103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:50.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:50 smithi081 ceph-mon[31453]: pgmap v6103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:52.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:52 smithi153 ceph-mon[36569]: pgmap v6104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:52.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:52 smithi104 ceph-mon[36624]: pgmap v6104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:52.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:52 smithi081 ceph-mon[31453]: pgmap v6104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:54.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:54 smithi153 ceph-mon[36569]: pgmap v6105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:54.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:54 smithi104 ceph-mon[36624]: pgmap v6105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:54.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:54 smithi081 ceph-mon[31453]: pgmap v6105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:56.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:56 smithi153 ceph-mon[36569]: pgmap v6106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:56.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:56 smithi104 ceph-mon[36624]: pgmap v6106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:56.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:56 smithi081 ceph-mon[31453]: pgmap v6106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:58.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:16:58 smithi153 ceph-mon[36569]: pgmap v6107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:58.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:16:58 smithi104 ceph-mon[36624]: pgmap v6107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:16:58.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:16:58 smithi081 ceph-mon[31453]: pgmap v6107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:00.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:00 smithi153 ceph-mon[36569]: pgmap v6108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:00.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:00 smithi104 ceph-mon[36624]: pgmap v6108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:00.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:00 smithi081 ceph-mon[31453]: pgmap v6108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:01.024+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:17:01.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:01.024+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:17:01.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:01.024+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:17:02.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:02 smithi153 ceph-mon[36569]: pgmap v6109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:02.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:02 smithi104 ceph-mon[36624]: pgmap v6109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:02.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:02 smithi081 ceph-mon[31453]: pgmap v6109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:04.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:04 smithi153 ceph-mon[36569]: pgmap v6110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:04.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:04 smithi104 ceph-mon[36624]: pgmap v6110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:04.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:04 smithi081 ceph-mon[31453]: pgmap v6110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:06.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:06 smithi153 ceph-mon[36569]: pgmap v6111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:06.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:06 smithi104 ceph-mon[36624]: pgmap v6111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:06.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:06 smithi081 ceph-mon[31453]: pgmap v6111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:08.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:08 smithi153 ceph-mon[36569]: pgmap v6112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:08.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:08 smithi104 ceph-mon[36624]: pgmap v6112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:08.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:08 smithi081 ceph-mon[31453]: pgmap v6112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:10.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:10 smithi153 ceph-mon[36569]: pgmap v6113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:10.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:10 smithi104 ceph-mon[36624]: pgmap v6113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:10.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:10 smithi081 ceph-mon[31453]: pgmap v6113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:12.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:12 smithi153 ceph-mon[36569]: pgmap v6114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:12.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:12 smithi104 ceph-mon[36624]: pgmap v6114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:12.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:12 smithi081 ceph-mon[31453]: pgmap v6114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:14.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:14 smithi153 ceph-mon[36569]: pgmap v6115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:14.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:14 smithi104 ceph-mon[36624]: pgmap v6115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:14.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:14 smithi081 ceph-mon[31453]: pgmap v6115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:16.037+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:17:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:16.037+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:17:16.022 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:16.037+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:17:16.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:16 smithi153 ceph-mon[36569]: pgmap v6116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:16.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:16 smithi104 ceph-mon[36624]: pgmap v6116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:16.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:16 smithi081 ceph-mon[31453]: pgmap v6116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:18.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:18 smithi153 ceph-mon[36569]: pgmap v6117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:18.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:18 smithi104 ceph-mon[36624]: pgmap v6117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:18.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:18 smithi081 ceph-mon[31453]: pgmap v6117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:20.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:20 smithi153 ceph-mon[36569]: pgmap v6118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:20.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:17:20.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:20 smithi104 ceph-mon[36624]: pgmap v6118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:20.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:17:20.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:20 smithi081 ceph-mon[31453]: pgmap v6118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:20.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:17:21.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:17:21.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:17:21.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:17:21.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:17:21.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:17:21.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:17:21.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:17:21.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:17:21.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:17:22.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:22 smithi153 ceph-mon[36569]: pgmap v6119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:22.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:22 smithi153 ceph-mon[36569]: pgmap v6120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:22.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:22 smithi104 ceph-mon[36624]: pgmap v6119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:22.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:22 smithi104 ceph-mon[36624]: pgmap v6120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:22.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:22 smithi081 ceph-mon[31453]: pgmap v6119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:22.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:22 smithi081 ceph-mon[31453]: pgmap v6120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:24.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:24 smithi153 ceph-mon[36569]: pgmap v6121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:24.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:24 smithi104 ceph-mon[36624]: pgmap v6121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:24.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:24 smithi081 ceph-mon[31453]: pgmap v6121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:26.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:26 smithi153 ceph-mon[36569]: pgmap v6122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:26.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:26 smithi104 ceph-mon[36624]: pgmap v6122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:26.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:26 smithi081 ceph-mon[31453]: pgmap v6122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:28.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:28 smithi153 ceph-mon[36569]: pgmap v6123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:28.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:28 smithi104 ceph-mon[36624]: pgmap v6123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:28.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:28 smithi081 ceph-mon[31453]: pgmap v6123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:30.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:30 smithi153 ceph-mon[36569]: pgmap v6124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:30.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:30 smithi104 ceph-mon[36624]: pgmap v6124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:30.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:30 smithi081 ceph-mon[31453]: pgmap v6124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:31.053+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:17:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:31.053+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:17:31.038 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:31.053+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:17:32.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:32 smithi153 ceph-mon[36569]: pgmap v6125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:32.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:32 smithi104 ceph-mon[36624]: pgmap v6125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:32.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:32 smithi081 ceph-mon[31453]: pgmap v6125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:34.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:34 smithi153 ceph-mon[36569]: pgmap v6126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:34.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:34 smithi104 ceph-mon[36624]: pgmap v6126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:34.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:34 smithi081 ceph-mon[31453]: pgmap v6126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:36.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:36 smithi153 ceph-mon[36569]: pgmap v6127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:36.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:36 smithi104 ceph-mon[36624]: pgmap v6127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:36.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:36 smithi081 ceph-mon[31453]: pgmap v6127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:38.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:38 smithi153 ceph-mon[36569]: pgmap v6128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:38.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:38 smithi104 ceph-mon[36624]: pgmap v6128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:38.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:38 smithi081 ceph-mon[31453]: pgmap v6128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:40.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:40 smithi153 ceph-mon[36569]: pgmap v6129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:40.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:40 smithi104 ceph-mon[36624]: pgmap v6129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:40.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:40 smithi081 ceph-mon[31453]: pgmap v6129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:42.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:42 smithi104 ceph-mon[36624]: pgmap v6130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:42.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:42 smithi081 ceph-mon[31453]: pgmap v6130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:43.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:42 smithi153 ceph-mon[36569]: pgmap v6130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:44.497+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:17:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:44.497+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:17:44.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:44.498+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:17:44.484 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:44.498+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:17:44.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:44 smithi104 ceph-mon[36624]: pgmap v6131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:44.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:44 smithi081 ceph-mon[31453]: pgmap v6131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:45.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:44 smithi153 ceph-mon[36569]: pgmap v6131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:46.056+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:17:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:46.056+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:17:46.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:17:46.056+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:17:46.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:46 smithi104 ceph-mon[36624]: pgmap v6132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:46.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:46 smithi081 ceph-mon[31453]: pgmap v6132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:47.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:46 smithi153 ceph-mon[36569]: pgmap v6132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:48.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:48 smithi104 ceph-mon[36624]: pgmap v6133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:48.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:48 smithi081 ceph-mon[31453]: pgmap v6133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:49.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:48 smithi153 ceph-mon[36569]: pgmap v6133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:50.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:50 smithi104 ceph-mon[36624]: pgmap v6134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:50.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:50 smithi081 ceph-mon[31453]: pgmap v6134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:51.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:50 smithi153 ceph-mon[36569]: pgmap v6134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:52.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:52 smithi081 ceph-mon[31453]: pgmap v6135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:53.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:52 smithi153 ceph-mon[36569]: pgmap v6135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:53.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:52 smithi104 ceph-mon[36624]: pgmap v6135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:54.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:54 smithi081 ceph-mon[31453]: pgmap v6136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:55.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:54 smithi153 ceph-mon[36569]: pgmap v6136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:55.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:54 smithi104 ceph-mon[36624]: pgmap v6136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:56.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:56 smithi081 ceph-mon[31453]: pgmap v6137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:57.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:56 smithi153 ceph-mon[36569]: pgmap v6137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:57.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:56 smithi104 ceph-mon[36624]: pgmap v6137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:58.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:17:58 smithi081 ceph-mon[31453]: pgmap v6138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:59.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:17:58 smithi153 ceph-mon[36569]: pgmap v6138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:17:59.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:17:58 smithi104 ceph-mon[36624]: pgmap v6138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:00.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:00 smithi081 ceph-mon[31453]: pgmap v6139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:01.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:00 smithi153 ceph-mon[36569]: pgmap v6139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:01.069+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:18:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:01.069+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:18:01.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:01.069+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:18:01.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:00 smithi104 ceph-mon[36624]: pgmap v6139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:02.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:02 smithi081 ceph-mon[31453]: pgmap v6140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:03.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:02 smithi153 ceph-mon[36569]: pgmap v6140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:03.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:02 smithi104 ceph-mon[36624]: pgmap v6140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:04.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:04 smithi081 ceph-mon[31453]: pgmap v6141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:05.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:04 smithi153 ceph-mon[36569]: pgmap v6141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:05.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:04 smithi104 ceph-mon[36624]: pgmap v6141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:06.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:06 smithi081 ceph-mon[31453]: pgmap v6142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:07.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:06 smithi153 ceph-mon[36569]: pgmap v6142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:07.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:06 smithi104 ceph-mon[36624]: pgmap v6142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:08.870 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:08 smithi104 ceph-mon[36624]: pgmap v6143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:08.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:08 smithi081 ceph-mon[31453]: pgmap v6143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:09.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:08 smithi153 ceph-mon[36569]: pgmap v6143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:10.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:10 smithi081 ceph-mon[31453]: pgmap v6144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:11.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:10 smithi153 ceph-mon[36569]: pgmap v6144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:11.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:10 smithi104 ceph-mon[36624]: pgmap v6144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:12.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:12 smithi081 ceph-mon[31453]: pgmap v6145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:13.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:12 smithi153 ceph-mon[36569]: pgmap v6145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:13.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:12 smithi104 ceph-mon[36624]: pgmap v6145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:14.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:14 smithi081 ceph-mon[31453]: pgmap v6146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:15.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:14 smithi153 ceph-mon[36569]: pgmap v6146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:15.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:14 smithi104 ceph-mon[36624]: pgmap v6146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:16.082+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:18:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:16.082+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:18:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:16.082+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:18:16.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:16 smithi081 ceph-mon[31453]: pgmap v6147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:17.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:16 smithi153 ceph-mon[36569]: pgmap v6147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:17.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:16 smithi104 ceph-mon[36624]: pgmap v6147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:18.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:18 smithi081 ceph-mon[31453]: pgmap v6148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:19.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:18 smithi153 ceph-mon[36569]: pgmap v6148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:19.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:18 smithi104 ceph-mon[36624]: pgmap v6148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:21.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:20 smithi153 ceph-mon[36569]: pgmap v6149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:21.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:20 smithi104 ceph-mon[36624]: pgmap v6149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:21.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:20 smithi081 ceph-mon[31453]: pgmap v6149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:22.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:18:22.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:18:22.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:18:22.970 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:22 smithi081 ceph-mon[31453]: pgmap v6150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:22.970 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:18:22.970 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:18:22.970 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:18:23.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:22 smithi153 ceph-mon[36569]: pgmap v6150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:23.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:18:23.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:18:23.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:18:23.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:22 smithi104 ceph-mon[36624]: pgmap v6150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:23.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:18:23.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:18:23.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:18:24.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:23 smithi153 ceph-mon[36569]: pgmap v6151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:24.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:23 smithi153 ceph-mon[36569]: pgmap v6152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:24.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:23 smithi104 ceph-mon[36624]: pgmap v6151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:24.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:23 smithi104 ceph-mon[36624]: pgmap v6152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:24.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:23 smithi081 ceph-mon[31453]: pgmap v6151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:24.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:23 smithi081 ceph-mon[31453]: pgmap v6152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:26.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:25 smithi153 ceph-mon[36569]: pgmap v6153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:26.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:25 smithi104 ceph-mon[36624]: pgmap v6153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:26.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:25 smithi081 ceph-mon[31453]: pgmap v6153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:28.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:27 smithi153 ceph-mon[36569]: pgmap v6154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:28.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:27 smithi104 ceph-mon[36624]: pgmap v6154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:28.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:27 smithi081 ceph-mon[31453]: pgmap v6154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:30.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:29 smithi153 ceph-mon[36569]: pgmap v6155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:30.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:29 smithi104 ceph-mon[36624]: pgmap v6155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:30.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:29 smithi081 ceph-mon[31453]: pgmap v6155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:31.098+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:18:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:31.098+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:18:31.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:31.098+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:18:32.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:31 smithi153 ceph-mon[36569]: pgmap v6156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:32.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:31 smithi104 ceph-mon[36624]: pgmap v6156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:32.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:31 smithi081 ceph-mon[31453]: pgmap v6156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:34.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:33 smithi153 ceph-mon[36569]: pgmap v6157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:34.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:33 smithi104 ceph-mon[36624]: pgmap v6157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:34.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:33 smithi081 ceph-mon[31453]: pgmap v6157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:36.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:35 smithi153 ceph-mon[36569]: pgmap v6158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:36.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:35 smithi104 ceph-mon[36624]: pgmap v6158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:36.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:35 smithi081 ceph-mon[31453]: pgmap v6158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:38.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:37 smithi153 ceph-mon[36569]: pgmap v6159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:38.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:37 smithi104 ceph-mon[36624]: pgmap v6159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:38.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:37 smithi081 ceph-mon[31453]: pgmap v6159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:40.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:39 smithi153 ceph-mon[36569]: pgmap v6160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:40.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:39 smithi104 ceph-mon[36624]: pgmap v6160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:40.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:39 smithi081 ceph-mon[31453]: pgmap v6160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:42.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:41 smithi153 ceph-mon[36569]: pgmap v6161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:42.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:41 smithi104 ceph-mon[36624]: pgmap v6161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:42.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:41 smithi081 ceph-mon[31453]: pgmap v6161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:44.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:43 smithi153 ceph-mon[36569]: pgmap v6162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:44.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:43 smithi104 ceph-mon[36624]: pgmap v6162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:44.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:43 smithi081 ceph-mon[31453]: pgmap v6162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:46.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:45 smithi153 ceph-mon[36569]: pgmap v6163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:46.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:45 smithi104 ceph-mon[36624]: pgmap v6163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:46.114+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:18:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:46.114+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:18:46.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:18:46.114+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:18:46.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:45 smithi081 ceph-mon[31453]: pgmap v6163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:48.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:47 smithi153 ceph-mon[36569]: pgmap v6164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:48.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:47 smithi104 ceph-mon[36624]: pgmap v6164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:48.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:47 smithi081 ceph-mon[31453]: pgmap v6164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:50.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:49 smithi153 ceph-mon[36569]: pgmap v6165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:50.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:49 smithi104 ceph-mon[36624]: pgmap v6165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:50.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:49 smithi081 ceph-mon[31453]: pgmap v6165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:52.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:51 smithi153 ceph-mon[36569]: pgmap v6166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:52.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:51 smithi104 ceph-mon[36624]: pgmap v6166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:52.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:51 smithi081 ceph-mon[31453]: pgmap v6166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:54.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:53 smithi153 ceph-mon[36569]: pgmap v6167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:54.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:53 smithi104 ceph-mon[36624]: pgmap v6167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:54.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:53 smithi081 ceph-mon[31453]: pgmap v6167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:56.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:55 smithi153 ceph-mon[36569]: pgmap v6168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:56.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:55 smithi104 ceph-mon[36624]: pgmap v6168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:56.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:55 smithi081 ceph-mon[31453]: pgmap v6168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:58.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:57 smithi153 ceph-mon[36569]: pgmap v6169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:58.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:57 smithi104 ceph-mon[36624]: pgmap v6169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:18:58.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:57 smithi081 ceph-mon[31453]: pgmap v6169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:00.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:18:59 smithi153 ceph-mon[36569]: pgmap v6170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:00.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:18:59 smithi104 ceph-mon[36624]: pgmap v6170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:00.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:18:59 smithi081 ceph-mon[31453]: pgmap v6170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:01.120+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:19:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:01.120+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:19:01.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:01.120+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:19:02.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:01 smithi153 ceph-mon[36569]: pgmap v6171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:02.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:01 smithi104 ceph-mon[36624]: pgmap v6171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:02.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:01 smithi081 ceph-mon[31453]: pgmap v6171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:03 smithi153 ceph-mon[36569]: pgmap v6172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:04.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:03 smithi104 ceph-mon[36624]: pgmap v6172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:04.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:03 smithi081 ceph-mon[31453]: pgmap v6172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:06.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:05 smithi153 ceph-mon[36569]: pgmap v6173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:06.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:05 smithi104 ceph-mon[36624]: pgmap v6173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:06.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:05 smithi081 ceph-mon[31453]: pgmap v6173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:08.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:07 smithi153 ceph-mon[36569]: pgmap v6174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:08.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:07 smithi104 ceph-mon[36624]: pgmap v6174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:08.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:07 smithi081 ceph-mon[31453]: pgmap v6174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:10.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:09 smithi104 ceph-mon[36624]: pgmap v6175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:10.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:09 smithi081 ceph-mon[31453]: pgmap v6175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:09 smithi153 ceph-mon[36569]: pgmap v6175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:12.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:11 smithi104 ceph-mon[36624]: pgmap v6176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:12.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:11 smithi081 ceph-mon[31453]: pgmap v6176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:11 smithi153 ceph-mon[36569]: pgmap v6176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:14.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:13 smithi104 ceph-mon[36624]: pgmap v6177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:14.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:13 smithi081 ceph-mon[31453]: pgmap v6177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:14.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:13 smithi153 ceph-mon[36569]: pgmap v6177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:14.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:14.512+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:19:14.501 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:14.512+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:19:14.501 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:14.513+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:19:14.501 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:14.513+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:19:15.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:14 smithi081 ceph-mon[31453]: pgmap v6178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:15.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:14 smithi153 ceph-mon[36569]: pgmap v6178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:15.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:14 smithi104 ceph-mon[36624]: pgmap v6178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:16.136+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:19:16.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:16.136+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:19:16.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:16.136+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:19:17.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:17 smithi153 ceph-mon[36569]: pgmap v6179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:17.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:17 smithi104 ceph-mon[36624]: pgmap v6179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:17.848 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:17 smithi081 ceph-mon[31453]: pgmap v6179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:19.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:19 smithi153 ceph-mon[36569]: pgmap v6180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:19.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:19 smithi104 ceph-mon[36624]: pgmap v6180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:19.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:19 smithi081 ceph-mon[31453]: pgmap v6180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:21.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:21 smithi153 ceph-mon[36569]: pgmap v6181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:21.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:21 smithi104 ceph-mon[36624]: pgmap v6181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:21.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:21 smithi081 ceph-mon[31453]: pgmap v6181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:23.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: pgmap v6182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:23.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:19:23.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: pgmap v6182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: pgmap v6182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:23.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:19:24.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:19:24.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:19:24.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:19:24.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:19:24.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:19:24.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:19:24.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:19:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:19:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:19:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:19:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:19:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:19:24.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:19:24.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:19:24.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:19:25.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:25 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:19:25.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:25 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:19:25.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:25 smithi153 ceph-mon[36569]: pgmap v6183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:25.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:25 smithi153 ceph-mon[36569]: pgmap v6184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:25.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:25 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:19:25.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:25 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:19:25.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:25 smithi104 ceph-mon[36624]: pgmap v6183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:25.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:25 smithi104 ceph-mon[36624]: pgmap v6184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:25.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:25 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:19:25.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:25 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:19:25.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:25 smithi081 ceph-mon[31453]: pgmap v6183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:25.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:25 smithi081 ceph-mon[31453]: pgmap v6184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:27.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:27 smithi153 ceph-mon[36569]: pgmap v6185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:27.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:27 smithi104 ceph-mon[36624]: pgmap v6185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:27.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:27 smithi081 ceph-mon[31453]: pgmap v6185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:29.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:29 smithi153 ceph-mon[36569]: pgmap v6186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:29.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:29 smithi104 ceph-mon[36624]: pgmap v6186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:29.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:29 smithi081 ceph-mon[31453]: pgmap v6186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:31.138+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:19:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:31.138+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:19:31.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:31.138+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:19:31.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:31 smithi153 ceph-mon[36569]: pgmap v6187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:31.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:31 smithi104 ceph-mon[36624]: pgmap v6187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:31.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:31 smithi081 ceph-mon[31453]: pgmap v6187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:33.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:33 smithi153 ceph-mon[36569]: pgmap v6188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:33.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:33 smithi104 ceph-mon[36624]: pgmap v6188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:33.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:33 smithi081 ceph-mon[31453]: pgmap v6188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:35.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:35 smithi153 ceph-mon[36569]: pgmap v6189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:35.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:35 smithi104 ceph-mon[36624]: pgmap v6189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:35.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:35 smithi081 ceph-mon[31453]: pgmap v6189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:37.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:37 smithi153 ceph-mon[36569]: pgmap v6190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:37.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:37 smithi104 ceph-mon[36624]: pgmap v6190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:37.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:37 smithi081 ceph-mon[31453]: pgmap v6190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:39.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:39 smithi153 ceph-mon[36569]: pgmap v6191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:39.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:39 smithi104 ceph-mon[36624]: pgmap v6191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:39.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:39 smithi081 ceph-mon[31453]: pgmap v6191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:41.645 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:41 smithi081 ceph-mon[31453]: pgmap v6192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:41.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:41 smithi153 ceph-mon[36569]: pgmap v6192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:41.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:41 smithi104 ceph-mon[36624]: pgmap v6192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:43.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:43 smithi153 ceph-mon[36569]: pgmap v6193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:43.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:43 smithi104 ceph-mon[36624]: pgmap v6193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:43.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:43 smithi081 ceph-mon[31453]: pgmap v6193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:45.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:45 smithi153 ceph-mon[36569]: pgmap v6194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:45.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:45 smithi104 ceph-mon[36624]: pgmap v6194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:45.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:45 smithi081 ceph-mon[31453]: pgmap v6194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:46.154+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:19:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:46.154+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:19:46.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:19:46.154+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:19:47.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:47 smithi153 ceph-mon[36569]: pgmap v6195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:47.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:47 smithi104 ceph-mon[36624]: pgmap v6195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:47.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:47 smithi081 ceph-mon[31453]: pgmap v6195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:49.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:49 smithi153 ceph-mon[36569]: pgmap v6196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:49.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:49 smithi104 ceph-mon[36624]: pgmap v6196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:49.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:49 smithi081 ceph-mon[31453]: pgmap v6196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:51.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:51 smithi153 ceph-mon[36569]: pgmap v6197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:51.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:51 smithi104 ceph-mon[36624]: pgmap v6197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:51.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:51 smithi081 ceph-mon[31453]: pgmap v6197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:53.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:53 smithi153 ceph-mon[36569]: pgmap v6198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:53.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:53 smithi104 ceph-mon[36624]: pgmap v6198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:53.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:53 smithi081 ceph-mon[31453]: pgmap v6198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:55.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:55 smithi153 ceph-mon[36569]: pgmap v6199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:55.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:55 smithi104 ceph-mon[36624]: pgmap v6199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:55.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:55 smithi081 ceph-mon[31453]: pgmap v6199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:57.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:57 smithi153 ceph-mon[36569]: pgmap v6200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:57.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:57 smithi104 ceph-mon[36624]: pgmap v6200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:57.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:57 smithi081 ceph-mon[31453]: pgmap v6200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:59.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:19:59 smithi153 ceph-mon[36569]: pgmap v6201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:59.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:19:59 smithi104 ceph-mon[36624]: pgmap v6201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:19:59.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:19:59 smithi081 ceph-mon[31453]: pgmap v6201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:00.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:20:00.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:20:00.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:20:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:01.167+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:20:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:01.167+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:20:01.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:01.167+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:20:01.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:01 smithi153 ceph-mon[36569]: pgmap v6202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:01.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:01 smithi104 ceph-mon[36624]: pgmap v6202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:01.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:01 smithi081 ceph-mon[31453]: pgmap v6202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:03.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:03 smithi153 ceph-mon[36569]: pgmap v6203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:03.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:03 smithi104 ceph-mon[36624]: pgmap v6203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:03.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:03 smithi081 ceph-mon[31453]: pgmap v6203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:05.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:05 smithi153 ceph-mon[36569]: pgmap v6204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:05.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:05 smithi104 ceph-mon[36624]: pgmap v6204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:05.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:05 smithi081 ceph-mon[31453]: pgmap v6204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:07.796 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:07 smithi153 ceph-mon[36569]: pgmap v6205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:07.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:07 smithi104 ceph-mon[36624]: pgmap v6205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:07.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:07 smithi081 ceph-mon[31453]: pgmap v6205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:09.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:09 smithi104 ceph-mon[36624]: pgmap v6206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:09.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:09 smithi081 ceph-mon[31453]: pgmap v6206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:10.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:09 smithi153 ceph-mon[36569]: pgmap v6206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:11.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:11 smithi104 ceph-mon[36624]: pgmap v6207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:11.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:11 smithi081 ceph-mon[31453]: pgmap v6207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:12.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:11 smithi153 ceph-mon[36569]: pgmap v6207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:13.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:13 smithi104 ceph-mon[36624]: pgmap v6208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:13.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:13 smithi081 ceph-mon[31453]: pgmap v6208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:14.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:13 smithi153 ceph-mon[36569]: pgmap v6208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:15.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:15 smithi104 ceph-mon[36624]: pgmap v6209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:15.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:15 smithi081 ceph-mon[31453]: pgmap v6209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:16.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:15 smithi153 ceph-mon[36569]: pgmap v6209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:16.177+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:20:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:16.177+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:20:16.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:16.177+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:20:17.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:17 smithi081 ceph-mon[31453]: pgmap v6210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:17.930 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:17 smithi153 ceph-mon[36569]: pgmap v6210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:18.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:17 smithi104 ceph-mon[36624]: pgmap v6210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:19.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:19 smithi081 ceph-mon[31453]: pgmap v6211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:20.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:19 smithi153 ceph-mon[36569]: pgmap v6211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:20.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:19 smithi104 ceph-mon[36624]: pgmap v6211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:21.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:21 smithi081 ceph-mon[31453]: pgmap v6212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:22.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:21 smithi153 ceph-mon[36569]: pgmap v6212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:22.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:21 smithi104 ceph-mon[36624]: pgmap v6212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:23.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:23 smithi081 ceph-mon[31453]: pgmap v6213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:24.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:23 smithi153 ceph-mon[36569]: pgmap v6213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:24.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:23 smithi104 ceph-mon[36624]: pgmap v6213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:24.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:20:25.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:20:25.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:20:25.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:25 smithi081 ceph-mon[31453]: pgmap v6214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:25.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:20:25.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:20:25.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:20:26.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:25 smithi153 ceph-mon[36569]: pgmap v6214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:26.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:20:26.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:20:26.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:20:26.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:25 smithi104 ceph-mon[36624]: pgmap v6214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:26.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:20:26.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:20:26.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:20:26.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:26 smithi081 ceph-mon[31453]: pgmap v6215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:26.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:26 smithi081 ceph-mon[31453]: pgmap v6216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:27.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:26 smithi153 ceph-mon[36569]: pgmap v6215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:27.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:26 smithi153 ceph-mon[36569]: pgmap v6216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:27.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:26 smithi104 ceph-mon[36624]: pgmap v6215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:27.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:26 smithi104 ceph-mon[36624]: pgmap v6216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:28.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:28 smithi081 ceph-mon[31453]: pgmap v6217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:29.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:28 smithi153 ceph-mon[36569]: pgmap v6217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:29.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:28 smithi104 ceph-mon[36624]: pgmap v6217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:30.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:30 smithi081 ceph-mon[31453]: pgmap v6218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:31.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:30 smithi153 ceph-mon[36569]: pgmap v6218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:31.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:30 smithi104 ceph-mon[36624]: pgmap v6218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:31.192+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:20:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:31.192+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:20:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:31.192+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:20:32.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:32 smithi081 ceph-mon[31453]: pgmap v6219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:33.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:32 smithi153 ceph-mon[36569]: pgmap v6219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:33.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:32 smithi104 ceph-mon[36624]: pgmap v6219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:34.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:34 smithi081 ceph-mon[31453]: pgmap v6220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:35.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:34 smithi153 ceph-mon[36569]: pgmap v6220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:35.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:34 smithi104 ceph-mon[36624]: pgmap v6220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:36.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:36 smithi081 ceph-mon[31453]: pgmap v6221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:37.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:36 smithi153 ceph-mon[36569]: pgmap v6221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:37.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:36 smithi104 ceph-mon[36624]: pgmap v6221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:38.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:38 smithi081 ceph-mon[31453]: pgmap v6222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:39.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:38 smithi153 ceph-mon[36569]: pgmap v6222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:39.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:38 smithi104 ceph-mon[36624]: pgmap v6222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:41.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:40 smithi153 ceph-mon[36569]: pgmap v6223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:41.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:40 smithi104 ceph-mon[36624]: pgmap v6223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:41.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:40 smithi081 ceph-mon[31453]: pgmap v6223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:43.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:42 smithi153 ceph-mon[36569]: pgmap v6224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:43.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:42 smithi104 ceph-mon[36624]: pgmap v6224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:43.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:42 smithi081 ceph-mon[31453]: pgmap v6224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:45.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:44 smithi153 ceph-mon[36569]: pgmap v6225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:45.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:44 smithi104 ceph-mon[36624]: pgmap v6225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:45.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:44 smithi081 ceph-mon[31453]: pgmap v6225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:46.208+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:20:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:46.208+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:20:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:20:46.208+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:20:47.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:46 smithi153 ceph-mon[36569]: pgmap v6226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:47.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:46 smithi104 ceph-mon[36624]: pgmap v6226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:47.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:46 smithi081 ceph-mon[31453]: pgmap v6226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:48.950 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:48 smithi081 ceph-mon[31453]: pgmap v6227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:49.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:48 smithi153 ceph-mon[36569]: pgmap v6227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:49.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:48 smithi104 ceph-mon[36624]: pgmap v6227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:51.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:50 smithi153 ceph-mon[36569]: pgmap v6228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:51.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:50 smithi104 ceph-mon[36624]: pgmap v6228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:51.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:50 smithi081 ceph-mon[31453]: pgmap v6228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:53.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:52 smithi153 ceph-mon[36569]: pgmap v6229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:53.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:52 smithi104 ceph-mon[36624]: pgmap v6229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:53.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:52 smithi081 ceph-mon[31453]: pgmap v6229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:55.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:54 smithi153 ceph-mon[36569]: pgmap v6230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:55.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:54 smithi104 ceph-mon[36624]: pgmap v6230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:55.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:54 smithi081 ceph-mon[31453]: pgmap v6230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:57.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:56 smithi153 ceph-mon[36569]: pgmap v6231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:57.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:56 smithi104 ceph-mon[36624]: pgmap v6231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:57.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:56 smithi081 ceph-mon[31453]: pgmap v6231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:59.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:20:58 smithi153 ceph-mon[36569]: pgmap v6232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:59.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:20:58 smithi104 ceph-mon[36624]: pgmap v6232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:20:59.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:20:58 smithi081 ceph-mon[31453]: pgmap v6232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:01.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:00 smithi153 ceph-mon[36569]: pgmap v6233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:01.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:00 smithi104 ceph-mon[36624]: pgmap v6233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:01.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:00 smithi081 ceph-mon[31453]: pgmap v6233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:01.223+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:21:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:01.224+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:21:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:01.224+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:21:03.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:02 smithi153 ceph-mon[36569]: pgmap v6234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:03.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:02 smithi104 ceph-mon[36624]: pgmap v6234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:03.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:02 smithi081 ceph-mon[31453]: pgmap v6234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:05.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:04 smithi153 ceph-mon[36569]: pgmap v6235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:05.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:04 smithi104 ceph-mon[36624]: pgmap v6235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:05.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:04 smithi081 ceph-mon[31453]: pgmap v6235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:07.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:06 smithi153 ceph-mon[36569]: pgmap v6236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:07.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:06 smithi104 ceph-mon[36624]: pgmap v6236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:07.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:06 smithi081 ceph-mon[31453]: pgmap v6236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:09.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:08 smithi153 ceph-mon[36569]: pgmap v6237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:09.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:08 smithi104 ceph-mon[36624]: pgmap v6237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:09.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:08 smithi081 ceph-mon[31453]: pgmap v6237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:11.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:10 smithi153 ceph-mon[36569]: pgmap v6238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:11.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:10 smithi104 ceph-mon[36624]: pgmap v6238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:11.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:10 smithi081 ceph-mon[31453]: pgmap v6238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:13.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:12 smithi153 ceph-mon[36569]: pgmap v6239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:13.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:12 smithi104 ceph-mon[36624]: pgmap v6239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:13.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:12 smithi081 ceph-mon[31453]: pgmap v6239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:15.008 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:14 smithi081 ceph-mon[31453]: pgmap v6240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:15.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:14 smithi153 ceph-mon[36569]: pgmap v6240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:15.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:14 smithi104 ceph-mon[36624]: pgmap v6240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:16.239+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:21:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:16.239+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:21:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:16.239+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:21:17.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:16 smithi153 ceph-mon[36569]: pgmap v6241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:17.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:16 smithi104 ceph-mon[36624]: pgmap v6241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:17.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:16 smithi081 ceph-mon[31453]: pgmap v6241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:19.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:18 smithi153 ceph-mon[36569]: pgmap v6242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:19.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:18 smithi104 ceph-mon[36624]: pgmap v6242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:19.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:18 smithi081 ceph-mon[31453]: pgmap v6242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:21.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:20 smithi153 ceph-mon[36569]: pgmap v6243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:21.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:20 smithi104 ceph-mon[36624]: pgmap v6243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:21.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:20 smithi081 ceph-mon[31453]: pgmap v6243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:23.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:22 smithi153 ceph-mon[36569]: pgmap v6244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:23.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:22 smithi104 ceph-mon[36624]: pgmap v6244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:23.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:22 smithi081 ceph-mon[31453]: pgmap v6244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:25.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:24 smithi153 ceph-mon[36569]: pgmap v6245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:25.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:24 smithi104 ceph-mon[36624]: pgmap v6245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:25.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:24 smithi081 ceph-mon[31453]: pgmap v6245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:26.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:21:26.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:21:26.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:21:27.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:26 smithi153 ceph-mon[36569]: pgmap v6246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:27.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:26 smithi104 ceph-mon[36624]: pgmap v6246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:27.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:26 smithi081 ceph-mon[31453]: pgmap v6246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:28.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:21:28.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:21:28.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:21:28.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:21:28.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:21:28.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:21:28.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:21:28.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:21:28.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:21:29.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:28 smithi153 ceph-mon[36569]: pgmap v6247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:29.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:28 smithi153 ceph-mon[36569]: pgmap v6248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:29.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:28 smithi104 ceph-mon[36624]: pgmap v6247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:29.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:28 smithi104 ceph-mon[36624]: pgmap v6248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:29.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:28 smithi081 ceph-mon[31453]: pgmap v6247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:29.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:28 smithi081 ceph-mon[31453]: pgmap v6248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:31.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:30 smithi153 ceph-mon[36569]: pgmap v6249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:31.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:30 smithi104 ceph-mon[36624]: pgmap v6249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:31.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:30 smithi081 ceph-mon[31453]: pgmap v6249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:31.255+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:21:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:31.255+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:21:31.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:31.255+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:21:33.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:32 smithi104 ceph-mon[36624]: pgmap v6250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:33.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:32 smithi081 ceph-mon[31453]: pgmap v6250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:33.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:32 smithi153 ceph-mon[36569]: pgmap v6250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:35.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:34 smithi104 ceph-mon[36624]: pgmap v6251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:35.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:34 smithi081 ceph-mon[31453]: pgmap v6251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:35.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:34 smithi153 ceph-mon[36569]: pgmap v6251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:37.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:36 smithi104 ceph-mon[36624]: pgmap v6252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:37.100 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:36 smithi153 ceph-mon[36569]: pgmap v6252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:37.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:36 smithi081 ceph-mon[31453]: pgmap v6252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:39.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:38 smithi104 ceph-mon[36624]: pgmap v6253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:39.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:38 smithi081 ceph-mon[31453]: pgmap v6253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:39.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:38 smithi153 ceph-mon[36569]: pgmap v6253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:41.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:40 smithi104 ceph-mon[36624]: pgmap v6254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:41.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:40 smithi081 ceph-mon[31453]: pgmap v6254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:41.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:40 smithi153 ceph-mon[36569]: pgmap v6254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:43.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:42 smithi081 ceph-mon[31453]: pgmap v6255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:43.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:42 smithi153 ceph-mon[36569]: pgmap v6255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:43.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:42 smithi104 ceph-mon[36624]: pgmap v6255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:45.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:44 smithi081 ceph-mon[31453]: pgmap v6256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:45.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:44 smithi153 ceph-mon[36569]: pgmap v6256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:45.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:44 smithi104 ceph-mon[36624]: pgmap v6256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:46.270+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:21:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:46.270+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:21:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:21:46.270+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:21:47.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:46 smithi081 ceph-mon[31453]: pgmap v6257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:46 smithi153 ceph-mon[36569]: pgmap v6257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:46 smithi104 ceph-mon[36624]: pgmap v6257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:49.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:48 smithi081 ceph-mon[31453]: pgmap v6258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:49.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:48 smithi153 ceph-mon[36569]: pgmap v6258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:49.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:48 smithi104 ceph-mon[36624]: pgmap v6258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:50.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:49 smithi081 ceph-mon[31453]: pgmap v6259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:50.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:49 smithi153 ceph-mon[36569]: pgmap v6259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:50.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:49 smithi104 ceph-mon[36624]: pgmap v6259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:52.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:52 smithi081 ceph-mon[31453]: pgmap v6260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:52.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:52 smithi153 ceph-mon[36569]: pgmap v6260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:52.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:52 smithi104 ceph-mon[36624]: pgmap v6260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:54.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:54 smithi081 ceph-mon[31453]: pgmap v6261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:54.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:54 smithi153 ceph-mon[36569]: pgmap v6261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:54.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:54 smithi104 ceph-mon[36624]: pgmap v6261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:56.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:56 smithi081 ceph-mon[31453]: pgmap v6262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:56.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:56 smithi153 ceph-mon[36569]: pgmap v6262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:56.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:56 smithi104 ceph-mon[36624]: pgmap v6262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:58.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:21:58 smithi081 ceph-mon[31453]: pgmap v6263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:58.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:21:58 smithi153 ceph-mon[36569]: pgmap v6263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:21:58.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:21:58 smithi104 ceph-mon[36624]: pgmap v6263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:00.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:00 smithi081 ceph-mon[31453]: pgmap v6264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:00.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:00 smithi153 ceph-mon[36569]: pgmap v6264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:00.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:00 smithi104 ceph-mon[36624]: pgmap v6264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:01.285+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:22:01.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:01.285+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:22:01.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:01.285+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:22:02.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:02 smithi081 ceph-mon[31453]: pgmap v6265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:02.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:02 smithi153 ceph-mon[36569]: pgmap v6265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:02.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:02 smithi104 ceph-mon[36624]: pgmap v6265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:04.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:04 smithi081 ceph-mon[31453]: pgmap v6266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:04.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:04 smithi153 ceph-mon[36569]: pgmap v6266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:04.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:04 smithi104 ceph-mon[36624]: pgmap v6266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:06.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:06 smithi081 ceph-mon[31453]: pgmap v6267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:06.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:06 smithi153 ceph-mon[36569]: pgmap v6267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:06.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:06 smithi104 ceph-mon[36624]: pgmap v6267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:08.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:08 smithi081 ceph-mon[31453]: pgmap v6268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:08.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:08 smithi153 ceph-mon[36569]: pgmap v6268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:08.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:08 smithi104 ceph-mon[36624]: pgmap v6268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:10.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:10 smithi153 ceph-mon[36569]: pgmap v6269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:10.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:10 smithi104 ceph-mon[36624]: pgmap v6269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:10.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:10 smithi081 ceph-mon[31453]: pgmap v6269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:12.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:12 smithi153 ceph-mon[36569]: pgmap v6270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:12.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:12 smithi104 ceph-mon[36624]: pgmap v6270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:12.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:12 smithi081 ceph-mon[31453]: pgmap v6270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:14.543+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:22:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:14.543+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:22:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:14.543+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:22:14.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:14.543+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:22:14.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:14 smithi153 ceph-mon[36569]: pgmap v6271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:14.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:14 smithi104 ceph-mon[36624]: pgmap v6271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:14.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:14 smithi081 ceph-mon[31453]: pgmap v6271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:16.300+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:22:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:16.301+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:22:16.294 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:16.301+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:22:16.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:16 smithi081 ceph-mon[31453]: pgmap v6272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:16.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:16 smithi153 ceph-mon[36569]: pgmap v6272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:16.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:16 smithi104 ceph-mon[36624]: pgmap v6272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:18.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:18 smithi153 ceph-mon[36569]: pgmap v6273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:18.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:18 smithi104 ceph-mon[36624]: pgmap v6273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:18.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:18 smithi081 ceph-mon[31453]: pgmap v6273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:20.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:20 smithi153 ceph-mon[36569]: pgmap v6274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:20.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:20 smithi104 ceph-mon[36624]: pgmap v6274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:20.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:20 smithi081 ceph-mon[31453]: pgmap v6274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:22.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:22 smithi081 ceph-mon[31453]: pgmap v6275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:22 smithi153 ceph-mon[36569]: pgmap v6275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:22.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:22 smithi104 ceph-mon[36624]: pgmap v6275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:24.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:24 smithi153 ceph-mon[36569]: pgmap v6276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:24.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:24 smithi104 ceph-mon[36624]: pgmap v6276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:24.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:24 smithi081 ceph-mon[31453]: pgmap v6276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:26.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:26 smithi153 ceph-mon[36569]: pgmap v6277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:26.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:26 smithi104 ceph-mon[36624]: pgmap v6277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:26.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:26 smithi081 ceph-mon[31453]: pgmap v6277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:27.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:22:27.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:22:27.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:22:28.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:28 smithi153 ceph-mon[36569]: pgmap v6278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:28.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:28 smithi104 ceph-mon[36624]: pgmap v6278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:28.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:28 smithi081 ceph-mon[31453]: pgmap v6278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:29.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:22:29.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:22:29.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:29 smithi153 ceph-mon[36569]: pgmap v6279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:29.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:29 smithi153 ceph-mon[36569]: pgmap v6280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:29.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:22:29.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:22:29.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:22:29.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:29 smithi104 ceph-mon[36624]: pgmap v6279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:29.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:29 smithi104 ceph-mon[36624]: pgmap v6280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:29.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:22:29.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:22:29.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:22:29.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:29 smithi081 ceph-mon[31453]: pgmap v6279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:29.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:29 smithi081 ceph-mon[31453]: pgmap v6280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:29.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:22:31.303 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:31.310+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:22:31.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:31.310+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:22:31.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:31.310+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:22:31.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:31 smithi081 ceph-mon[31453]: pgmap v6281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:32.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:31 smithi153 ceph-mon[36569]: pgmap v6281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:32.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:31 smithi104 ceph-mon[36624]: pgmap v6281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:33.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:33 smithi081 ceph-mon[31453]: pgmap v6282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:34.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:33 smithi153 ceph-mon[36569]: pgmap v6282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:34.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:33 smithi104 ceph-mon[36624]: pgmap v6282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:35.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:35 smithi081 ceph-mon[31453]: pgmap v6283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:36.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:35 smithi153 ceph-mon[36569]: pgmap v6283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:36.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:35 smithi104 ceph-mon[36624]: pgmap v6283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:37.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:37 smithi081 ceph-mon[31453]: pgmap v6284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:38.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:37 smithi153 ceph-mon[36569]: pgmap v6284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:38.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:37 smithi104 ceph-mon[36624]: pgmap v6284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:39.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:39 smithi081 ceph-mon[31453]: pgmap v6285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:40.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:39 smithi153 ceph-mon[36569]: pgmap v6285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:40.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:39 smithi104 ceph-mon[36624]: pgmap v6285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:41.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:41 smithi081 ceph-mon[31453]: pgmap v6286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:42.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:41 smithi153 ceph-mon[36569]: pgmap v6286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:42.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:41 smithi104 ceph-mon[36624]: pgmap v6286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:43.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:43 smithi081 ceph-mon[31453]: pgmap v6287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:44.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:43 smithi153 ceph-mon[36569]: pgmap v6287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:44.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:43 smithi104 ceph-mon[36624]: pgmap v6287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:46.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:45 smithi153 ceph-mon[36569]: pgmap v6288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:46.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:45 smithi104 ceph-mon[36624]: pgmap v6288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:46.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:45 smithi081 ceph-mon[31453]: pgmap v6288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:46.324+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:22:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:46.325+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:22:46.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:22:46.325+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:22:48.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:47 smithi153 ceph-mon[36569]: pgmap v6289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:48.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:47 smithi104 ceph-mon[36624]: pgmap v6289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:48.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:47 smithi081 ceph-mon[31453]: pgmap v6289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:50.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:49 smithi153 ceph-mon[36569]: pgmap v6290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:50.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:49 smithi104 ceph-mon[36624]: pgmap v6290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:50.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:49 smithi081 ceph-mon[31453]: pgmap v6290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:52.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:51 smithi153 ceph-mon[36569]: pgmap v6291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:52.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:51 smithi104 ceph-mon[36624]: pgmap v6291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:52.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:51 smithi081 ceph-mon[31453]: pgmap v6291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:54.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:53 smithi153 ceph-mon[36569]: pgmap v6292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:54.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:53 smithi104 ceph-mon[36624]: pgmap v6292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:54.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:53 smithi081 ceph-mon[31453]: pgmap v6292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:56.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:55 smithi153 ceph-mon[36569]: pgmap v6293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:56.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:55 smithi104 ceph-mon[36624]: pgmap v6293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:56.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:55 smithi081 ceph-mon[31453]: pgmap v6293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:58.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:57 smithi153 ceph-mon[36569]: pgmap v6294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:58.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:57 smithi104 ceph-mon[36624]: pgmap v6294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:22:58.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:57 smithi081 ceph-mon[31453]: pgmap v6294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:00.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:22:59 smithi153 ceph-mon[36569]: pgmap v6295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:00.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:22:59 smithi104 ceph-mon[36624]: pgmap v6295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:00.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:22:59 smithi081 ceph-mon[31453]: pgmap v6295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:01.336+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:23:01.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:01.336+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:23:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:01.336+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:23:02.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:01 smithi153 ceph-mon[36569]: pgmap v6296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:02.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:01 smithi104 ceph-mon[36624]: pgmap v6296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:02.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:01 smithi081 ceph-mon[31453]: pgmap v6296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:04.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:03 smithi153 ceph-mon[36569]: pgmap v6297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:04.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:03 smithi104 ceph-mon[36624]: pgmap v6297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:04.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:03 smithi081 ceph-mon[31453]: pgmap v6297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:06.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:05 smithi153 ceph-mon[36569]: pgmap v6298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:06.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:05 smithi104 ceph-mon[36624]: pgmap v6298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:06.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:05 smithi081 ceph-mon[31453]: pgmap v6298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:08.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:07 smithi153 ceph-mon[36569]: pgmap v6299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:08.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:07 smithi104 ceph-mon[36624]: pgmap v6299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:08.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:07 smithi081 ceph-mon[31453]: pgmap v6299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:10.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:09 smithi153 ceph-mon[36569]: pgmap v6300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:10.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:09 smithi104 ceph-mon[36624]: pgmap v6300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:10.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:09 smithi081 ceph-mon[31453]: pgmap v6300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:12.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:11 smithi153 ceph-mon[36569]: pgmap v6301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:12.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:11 smithi104 ceph-mon[36624]: pgmap v6301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:12.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:11 smithi081 ceph-mon[31453]: pgmap v6301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:14.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:13 smithi153 ceph-mon[36569]: pgmap v6302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:14.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:13 smithi104 ceph-mon[36624]: pgmap v6302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:14.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:13 smithi081 ceph-mon[31453]: pgmap v6302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:16.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:15 smithi153 ceph-mon[36569]: pgmap v6303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:16.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:15 smithi104 ceph-mon[36624]: pgmap v6303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:16.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:15 smithi081 ceph-mon[31453]: pgmap v6303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:16.340+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:23:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:16.340+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:23:16.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:16.340+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:23:18.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:17 smithi153 ceph-mon[36569]: pgmap v6304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:18.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:17 smithi104 ceph-mon[36624]: pgmap v6304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:18.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:17 smithi081 ceph-mon[31453]: pgmap v6304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:20.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:19 smithi153 ceph-mon[36569]: pgmap v6305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:20.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:19 smithi104 ceph-mon[36624]: pgmap v6305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:20.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:19 smithi081 ceph-mon[31453]: pgmap v6305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:22.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:21 smithi153 ceph-mon[36569]: pgmap v6306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:22.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:21 smithi104 ceph-mon[36624]: pgmap v6306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:22.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:21 smithi081 ceph-mon[31453]: pgmap v6306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:24.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:23 smithi153 ceph-mon[36569]: pgmap v6307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:24.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:23 smithi104 ceph-mon[36624]: pgmap v6307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:24.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:23 smithi081 ceph-mon[31453]: pgmap v6307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:25 smithi153 ceph-mon[36569]: pgmap v6308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:26.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:25 smithi104 ceph-mon[36624]: pgmap v6308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:26.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:25 smithi081 ceph-mon[31453]: pgmap v6308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:28.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:27 smithi153 ceph-mon[36569]: pgmap v6309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:28.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:27 smithi104 ceph-mon[36624]: pgmap v6309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:28.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:27 smithi081 ceph-mon[31453]: pgmap v6309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:29.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:29 smithi153 ceph-mon[36569]: pgmap v6310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:29.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:23:29.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:29.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:29.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:29.945 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:30.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:29 smithi104 ceph-mon[36624]: pgmap v6310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:30.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:23:30.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:30.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:30.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:30.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:30.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:29 smithi081 ceph-mon[31453]: pgmap v6310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:30.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:23:30.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:30.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:30.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:30.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:31.349+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:23:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:31.349+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:23:31.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:31.349+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:23:32.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:31 smithi153 ceph-mon[36569]: pgmap v6311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:32.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:31 smithi104 ceph-mon[36624]: pgmap v6311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:32.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:31 smithi081 ceph-mon[31453]: pgmap v6311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:32.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:34.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:33 smithi153 ceph-mon[36569]: pgmap v6312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:34.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:33 smithi104 ceph-mon[36624]: pgmap v6312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:34.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:33 smithi081 ceph-mon[31453]: pgmap v6312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:35.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:35.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:35.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:35 smithi153 ceph-mon[36569]: pgmap v6313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:35.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:35.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:35.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:35 smithi104 ceph-mon[36624]: pgmap v6313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:35.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:35.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:35.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:35 smithi081 ceph-mon[31453]: pgmap v6313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:36.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:23:36.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:23:36.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:36 smithi104 ceph-mon[36624]: pgmap v6314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:36.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:36 smithi104 ceph-mon[36624]: pgmap v6315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:36.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:36.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:23:36.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:23:36.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:36 smithi081 ceph-mon[31453]: pgmap v6314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:36.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:36 smithi081 ceph-mon[31453]: pgmap v6315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:36.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:36.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:23:36.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:23:36.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:36 smithi153 ceph-mon[36569]: pgmap v6314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:36.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:36 smithi153 ceph-mon[36569]: pgmap v6315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:36.801 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:23:38.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:38 smithi081 ceph-mon[31453]: pgmap v6316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:39.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:38 smithi153 ceph-mon[36569]: pgmap v6316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:39.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:38 smithi104 ceph-mon[36624]: pgmap v6316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:40.853 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:40 smithi153 ceph-mon[36569]: pgmap v6317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:40.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:40 smithi081 ceph-mon[31453]: pgmap v6317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:41.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:40 smithi104 ceph-mon[36624]: pgmap v6317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:42.879 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:42 smithi104 ceph-mon[36624]: pgmap v6318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:42.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:42 smithi081 ceph-mon[31453]: pgmap v6318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:43.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:42 smithi153 ceph-mon[36569]: pgmap v6318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:44.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:44 smithi081 ceph-mon[31453]: pgmap v6319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:45.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:44 smithi153 ceph-mon[36569]: pgmap v6319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:45.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:44 smithi104 ceph-mon[36624]: pgmap v6319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:46.366+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:23:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:46.366+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:23:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:23:46.366+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:23:46.900 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:46 smithi081 ceph-mon[31453]: pgmap v6320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:47.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:46 smithi153 ceph-mon[36569]: pgmap v6320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:47.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:46 smithi104 ceph-mon[36624]: pgmap v6320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:48.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:48 smithi081 ceph-mon[31453]: pgmap v6321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:49.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:48 smithi153 ceph-mon[36569]: pgmap v6321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:49.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:48 smithi104 ceph-mon[36624]: pgmap v6321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:50.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:50 smithi081 ceph-mon[31453]: pgmap v6322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:51.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:50 smithi153 ceph-mon[36569]: pgmap v6322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:51.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:50 smithi104 ceph-mon[36624]: pgmap v6322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:52.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:52 smithi081 ceph-mon[31453]: pgmap v6323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:53.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:52 smithi153 ceph-mon[36569]: pgmap v6323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:53.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:52 smithi104 ceph-mon[36624]: pgmap v6323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:54.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:54 smithi081 ceph-mon[31453]: pgmap v6324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:55.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:54 smithi153 ceph-mon[36569]: pgmap v6324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:55.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:54 smithi104 ceph-mon[36624]: pgmap v6324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:56.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:56 smithi081 ceph-mon[31453]: pgmap v6325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:57.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:56 smithi153 ceph-mon[36569]: pgmap v6325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:57.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:56 smithi104 ceph-mon[36624]: pgmap v6325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:58.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:23:58 smithi081 ceph-mon[31453]: pgmap v6326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:59.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:23:58 smithi153 ceph-mon[36569]: pgmap v6326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:23:59.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:23:58 smithi104 ceph-mon[36624]: pgmap v6326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:00.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:00 smithi081 ceph-mon[31453]: pgmap v6327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:01.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:00 smithi153 ceph-mon[36569]: pgmap v6327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:01.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:00 smithi104 ceph-mon[36624]: pgmap v6327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:01.371+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:24:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:01.371+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:24:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:01.371+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:24:03.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:02 smithi153 ceph-mon[36569]: pgmap v6328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:03.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:02 smithi104 ceph-mon[36624]: pgmap v6328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:03.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:02 smithi081 ceph-mon[31453]: pgmap v6328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:04.994 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:04 smithi081 ceph-mon[31453]: pgmap v6329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:05.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:04 smithi153 ceph-mon[36569]: pgmap v6329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:05.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:04 smithi104 ceph-mon[36624]: pgmap v6329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:07.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:06 smithi153 ceph-mon[36569]: pgmap v6330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:07.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:06 smithi104 ceph-mon[36624]: pgmap v6330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:06 smithi081 ceph-mon[31453]: pgmap v6330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:09.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:08 smithi153 ceph-mon[36569]: pgmap v6331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:09.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:08 smithi104 ceph-mon[36624]: pgmap v6331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:09.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:08 smithi081 ceph-mon[31453]: pgmap v6331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:11.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:10 smithi153 ceph-mon[36569]: pgmap v6332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:11.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:10 smithi104 ceph-mon[36624]: pgmap v6332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:11.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:10 smithi081 ceph-mon[31453]: pgmap v6332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:13.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:12 smithi153 ceph-mon[36569]: pgmap v6333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:12 smithi104 ceph-mon[36624]: pgmap v6333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:13.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:12 smithi081 ceph-mon[31453]: pgmap v6333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:14 smithi153 ceph-mon[36569]: pgmap v6334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:15.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:14 smithi104 ceph-mon[36624]: pgmap v6334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:15.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:14 smithi081 ceph-mon[31453]: pgmap v6334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:16.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:16.381+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:24:16.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:16.381+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:24:16.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:16.381+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:24:17.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:16 smithi153 ceph-mon[36569]: pgmap v6335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:17.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:16 smithi104 ceph-mon[36624]: pgmap v6335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:17.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:16 smithi081 ceph-mon[31453]: pgmap v6335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:18.954 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:18 smithi081 ceph-mon[31453]: pgmap v6336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:19.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:18 smithi153 ceph-mon[36569]: pgmap v6336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:19.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:18 smithi104 ceph-mon[36624]: pgmap v6336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:21.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:20 smithi153 ceph-mon[36569]: pgmap v6337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:21.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:20 smithi104 ceph-mon[36624]: pgmap v6337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:21.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:20 smithi081 ceph-mon[31453]: pgmap v6337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:22.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:22 smithi153 ceph-mon[36569]: pgmap v6338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:22.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:22 smithi104 ceph-mon[36624]: pgmap v6338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:22.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:22 smithi081 ceph-mon[31453]: pgmap v6338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:24.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:24 smithi081 ceph-mon[31453]: pgmap v6339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:25.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:24 smithi153 ceph-mon[36569]: pgmap v6339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:25.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:24 smithi104 ceph-mon[36624]: pgmap v6339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:26.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:26 smithi081 ceph-mon[31453]: pgmap v6340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:27.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:26 smithi153 ceph-mon[36569]: pgmap v6340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:27.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:26 smithi104 ceph-mon[36624]: pgmap v6340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:28.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:28 smithi081 ceph-mon[31453]: pgmap v6341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:29.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:28 smithi153 ceph-mon[36569]: pgmap v6341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:29.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:28 smithi104 ceph-mon[36624]: pgmap v6341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:30.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:30 smithi081 ceph-mon[31453]: pgmap v6342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:31.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:30 smithi153 ceph-mon[36569]: pgmap v6342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:31.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:30 smithi104 ceph-mon[36624]: pgmap v6342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:31.390+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:24:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:31.390+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:24:31.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:31.390+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:24:32.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:32 smithi081 ceph-mon[31453]: pgmap v6343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:33.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:32 smithi153 ceph-mon[36569]: pgmap v6343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:33.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:32 smithi104 ceph-mon[36624]: pgmap v6343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:34.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:34 smithi081 ceph-mon[31453]: pgmap v6344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:35.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:34 smithi153 ceph-mon[36569]: pgmap v6344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:35.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:34 smithi104 ceph-mon[36624]: pgmap v6344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:36 smithi081 ceph-mon[31453]: pgmap v6345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:24:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:36.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:37.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:36 smithi153 ceph-mon[36569]: pgmap v6345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:37.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:24:37.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:37.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:37.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:36 smithi104 ceph-mon[36624]: pgmap v6345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:37.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:24:37.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:37.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:38.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:24:38.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:24:38.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:24:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:24:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:24:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:24:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:24:39.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:38 smithi153 ceph-mon[36569]: pgmap v6346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:39.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:38 smithi153 ceph-mon[36569]: pgmap v6347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:39.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:38 smithi104 ceph-mon[36624]: pgmap v6346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:39.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:38 smithi104 ceph-mon[36624]: pgmap v6347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:39.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:38 smithi081 ceph-mon[31453]: pgmap v6346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:39.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:38 smithi081 ceph-mon[31453]: pgmap v6347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:41.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:40 smithi153 ceph-mon[36569]: pgmap v6348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:41.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:40 smithi104 ceph-mon[36624]: pgmap v6348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:41.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:40 smithi081 ceph-mon[31453]: pgmap v6348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:42 smithi153 ceph-mon[36569]: pgmap v6349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:43.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:42 smithi104 ceph-mon[36624]: pgmap v6349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:42 smithi081 ceph-mon[31453]: pgmap v6349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:45.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:44 smithi153 ceph-mon[36569]: pgmap v6350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:45.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:44 smithi104 ceph-mon[36624]: pgmap v6350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:45.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:44 smithi081 ceph-mon[31453]: pgmap v6350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:46.402+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:24:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:46.402+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:24:46.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:24:46.402+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:24:47.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:46 smithi153 ceph-mon[36569]: pgmap v6351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:47.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:46 smithi104 ceph-mon[36624]: pgmap v6351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:47.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:46 smithi081 ceph-mon[31453]: pgmap v6351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:48.955 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:48 smithi081 ceph-mon[31453]: pgmap v6352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:49.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:48 smithi153 ceph-mon[36569]: pgmap v6352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:49.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:48 smithi104 ceph-mon[36624]: pgmap v6352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:51.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:50 smithi153 ceph-mon[36569]: pgmap v6353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:51.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:50 smithi081 ceph-mon[31453]: pgmap v6353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:51.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:50 smithi104 ceph-mon[36624]: pgmap v6353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:53.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:52 smithi153 ceph-mon[36569]: pgmap v6354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:53.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:52 smithi104 ceph-mon[36624]: pgmap v6354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:53.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:52 smithi081 ceph-mon[31453]: pgmap v6354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:55.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:54 smithi153 ceph-mon[36569]: pgmap v6355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:55.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:54 smithi104 ceph-mon[36624]: pgmap v6355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:55.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:54 smithi081 ceph-mon[31453]: pgmap v6355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:57.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:56 smithi153 ceph-mon[36569]: pgmap v6356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:57.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:56 smithi104 ceph-mon[36624]: pgmap v6356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:57.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:56 smithi081 ceph-mon[31453]: pgmap v6356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:59.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:24:58 smithi153 ceph-mon[36569]: pgmap v6357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:59.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:24:58 smithi104 ceph-mon[36624]: pgmap v6357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:24:59.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:24:58 smithi081 ceph-mon[31453]: pgmap v6357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:01.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:00 smithi153 ceph-mon[36569]: pgmap v6358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:01.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:00 smithi104 ceph-mon[36624]: pgmap v6358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:01.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:00 smithi081 ceph-mon[31453]: pgmap v6358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:01.411+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:25:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:01.411+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:25:01.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:01.411+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:25:03.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:02 smithi153 ceph-mon[36569]: pgmap v6359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:03.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:02 smithi104 ceph-mon[36624]: pgmap v6359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:03.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:02 smithi081 ceph-mon[31453]: pgmap v6359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:05.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:04 smithi153 ceph-mon[36569]: pgmap v6360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:05.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:04 smithi104 ceph-mon[36624]: pgmap v6360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:05.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:04 smithi081 ceph-mon[31453]: pgmap v6360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:07.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:06 smithi153 ceph-mon[36569]: pgmap v6361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:07.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:06 smithi104 ceph-mon[36624]: pgmap v6361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:07.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:06 smithi081 ceph-mon[31453]: pgmap v6361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:09.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:08 smithi153 ceph-mon[36569]: pgmap v6362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:09.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:08 smithi104 ceph-mon[36624]: pgmap v6362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:09.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:08 smithi081 ceph-mon[31453]: pgmap v6362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:11.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:10 smithi153 ceph-mon[36569]: pgmap v6363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:11.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:10 smithi104 ceph-mon[36624]: pgmap v6363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:11.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:10 smithi081 ceph-mon[31453]: pgmap v6363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:13.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:12 smithi153 ceph-mon[36569]: pgmap v6364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:13.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:12 smithi104 ceph-mon[36624]: pgmap v6364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:13.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:12 smithi081 ceph-mon[31453]: pgmap v6364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:14.572+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:25:14.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:14.572+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:25:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:14.572+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:25:14.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:14.572+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:25:15.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:14 smithi153 ceph-mon[36569]: pgmap v6365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:15.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:14 smithi104 ceph-mon[36624]: pgmap v6365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:15.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:14 smithi081 ceph-mon[31453]: pgmap v6365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:16.424+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:25:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:16.424+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:25:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:16.424+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:25:17.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:16 smithi153 ceph-mon[36569]: pgmap v6366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:17.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:16 smithi104 ceph-mon[36624]: pgmap v6366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:17.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:16 smithi081 ceph-mon[31453]: pgmap v6366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:19.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:18 smithi153 ceph-mon[36569]: pgmap v6367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:18 smithi104 ceph-mon[36624]: pgmap v6367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:19.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:18 smithi081 ceph-mon[31453]: pgmap v6367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:21.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:20 smithi153 ceph-mon[36569]: pgmap v6368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:21.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:20 smithi104 ceph-mon[36624]: pgmap v6368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:21.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:20 smithi081 ceph-mon[31453]: pgmap v6368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:23.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:22 smithi153 ceph-mon[36569]: pgmap v6369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:23.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:22 smithi104 ceph-mon[36624]: pgmap v6369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:23.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:22 smithi081 ceph-mon[31453]: pgmap v6369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:25.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:24 smithi153 ceph-mon[36569]: pgmap v6370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:25.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:24 smithi104 ceph-mon[36624]: pgmap v6370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:25.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:24 smithi081 ceph-mon[31453]: pgmap v6370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:27.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:26 smithi153 ceph-mon[36569]: pgmap v6371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:27.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:26 smithi104 ceph-mon[36624]: pgmap v6371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:27.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:26 smithi081 ceph-mon[31453]: pgmap v6371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:29.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:28 smithi153 ceph-mon[36569]: pgmap v6372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:29.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:28 smithi104 ceph-mon[36624]: pgmap v6372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:29.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:28 smithi081 ceph-mon[31453]: pgmap v6372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:31.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:30 smithi153 ceph-mon[36569]: pgmap v6373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:31.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:30 smithi104 ceph-mon[36624]: pgmap v6373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:31.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:30 smithi081 ceph-mon[31453]: pgmap v6373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:31.424 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:31.427+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:25:31.424 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:31.427+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:25:31.424 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:31.427+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:25:33.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:32 smithi153 ceph-mon[36569]: pgmap v6374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:33.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:32 smithi104 ceph-mon[36624]: pgmap v6374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:33.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:32 smithi081 ceph-mon[31453]: pgmap v6374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:35.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:34 smithi153 ceph-mon[36569]: pgmap v6375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:35.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:34 smithi104 ceph-mon[36624]: pgmap v6375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:35.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:34 smithi081 ceph-mon[31453]: pgmap v6375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:37.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:36 smithi104 ceph-mon[36624]: pgmap v6376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:37.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:36 smithi081 ceph-mon[31453]: pgmap v6376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:37.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:36 smithi153 ceph-mon[36569]: pgmap v6376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:38.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:25:38.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:25:38.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:25:39.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:38 smithi104 ceph-mon[36624]: pgmap v6377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:39.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:25:39.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:25:39.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:25:39.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:38 smithi081 ceph-mon[31453]: pgmap v6377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:39.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:25:39.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:25:39.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:25:39.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:38 smithi153 ceph-mon[36569]: pgmap v6377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:39.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:25:39.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:25:39.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:25:40.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:39 smithi104 ceph-mon[36624]: pgmap v6378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:40.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:39 smithi104 ceph-mon[36624]: pgmap v6379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:39 smithi081 ceph-mon[31453]: pgmap v6378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:40.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:39 smithi081 ceph-mon[31453]: pgmap v6379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:40.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:39 smithi153 ceph-mon[36569]: pgmap v6378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:40.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:39 smithi153 ceph-mon[36569]: pgmap v6379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:42.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:41 smithi081 ceph-mon[31453]: pgmap v6380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:42.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:41 smithi153 ceph-mon[36569]: pgmap v6380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:42.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:41 smithi104 ceph-mon[36624]: pgmap v6380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:44.132 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:43 smithi104 ceph-mon[36624]: pgmap v6381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:44.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:43 smithi081 ceph-mon[31453]: pgmap v6381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:44.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:43 smithi153 ceph-mon[36569]: pgmap v6381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:46.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:45 smithi081 ceph-mon[31453]: pgmap v6382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:46.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:45 smithi153 ceph-mon[36569]: pgmap v6382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:46.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:45 smithi104 ceph-mon[36624]: pgmap v6382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:46.443+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:25:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:46.443+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:25:46.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:46.443+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:25:48.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:47 smithi081 ceph-mon[31453]: pgmap v6383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:48.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:47 smithi153 ceph-mon[36569]: pgmap v6383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:47 smithi104 ceph-mon[36624]: pgmap v6383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:50.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:49 smithi081 ceph-mon[31453]: pgmap v6384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:50.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:49 smithi153 ceph-mon[36569]: pgmap v6384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:50.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:49 smithi104 ceph-mon[36624]: pgmap v6384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:51 smithi081 ceph-mon[31453]: pgmap v6385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:52.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:51 smithi153 ceph-mon[36569]: pgmap v6385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:52.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:51 smithi104 ceph-mon[36624]: pgmap v6385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:54.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:53 smithi081 ceph-mon[31453]: pgmap v6386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:54.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:53 smithi153 ceph-mon[36569]: pgmap v6386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:54.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:53 smithi104 ceph-mon[36624]: pgmap v6386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:56.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:55 smithi081 ceph-mon[31453]: pgmap v6387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:56.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:55 smithi153 ceph-mon[36569]: pgmap v6387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:56.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:55 smithi104 ceph-mon[36624]: pgmap v6387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:58.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:57 smithi081 ceph-mon[31453]: pgmap v6388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:58.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:57 smithi153 ceph-mon[36569]: pgmap v6388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:58.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:57 smithi104 ceph-mon[36624]: pgmap v6388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:25:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:59.580+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:25:59.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:59.580+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:25:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:59.580+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:25:59.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:25:59.580+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:26:00.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:25:59 smithi081 ceph-mon[31453]: pgmap v6389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:00.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:25:59 smithi153 ceph-mon[36569]: pgmap v6389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:00.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:25:59 smithi104 ceph-mon[36624]: pgmap v6389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:01.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:01.454+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:26:01.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:01.454+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:26:01.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:01.454+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:26:02.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:01 smithi081 ceph-mon[31453]: pgmap v6390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:02.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:01 smithi153 ceph-mon[36569]: pgmap v6390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:02.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:01 smithi104 ceph-mon[36624]: pgmap v6390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:04.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:03 smithi081 ceph-mon[31453]: pgmap v6391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:04.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:03 smithi153 ceph-mon[36569]: pgmap v6391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:04.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:03 smithi104 ceph-mon[36624]: pgmap v6391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:06.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:05 smithi081 ceph-mon[31453]: pgmap v6392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:06.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:05 smithi153 ceph-mon[36569]: pgmap v6392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:06.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:05 smithi104 ceph-mon[36624]: pgmap v6392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:08.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:07 smithi081 ceph-mon[31453]: pgmap v6393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:08.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:07 smithi153 ceph-mon[36569]: pgmap v6393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:08.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:07 smithi104 ceph-mon[36624]: pgmap v6393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:09.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:08 smithi081 ceph-mon[31453]: pgmap v6394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:09.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:08 smithi153 ceph-mon[36569]: pgmap v6394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:09.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:08 smithi104 ceph-mon[36624]: pgmap v6394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:12.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:11 smithi153 ceph-mon[36569]: pgmap v6395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:12.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:11 smithi104 ceph-mon[36624]: pgmap v6395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:12.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:11 smithi081 ceph-mon[31453]: pgmap v6395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:14.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:13 smithi153 ceph-mon[36569]: pgmap v6396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:14.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:13 smithi104 ceph-mon[36624]: pgmap v6396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:14.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:13 smithi081 ceph-mon[31453]: pgmap v6396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:16.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:15 smithi153 ceph-mon[36569]: pgmap v6397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:16.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:15 smithi104 ceph-mon[36624]: pgmap v6397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:16.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:15 smithi081 ceph-mon[31453]: pgmap v6397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:16.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:16.470+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:16.470+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:26:16.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:16.470+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:26:18.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:17 smithi153 ceph-mon[36569]: pgmap v6398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:18.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:17 smithi104 ceph-mon[36624]: pgmap v6398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:18.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:17 smithi081 ceph-mon[31453]: pgmap v6398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:20.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:19 smithi153 ceph-mon[36569]: pgmap v6399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:20.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:19 smithi104 ceph-mon[36624]: pgmap v6399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:20.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:19 smithi081 ceph-mon[31453]: pgmap v6399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:22.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:21 smithi153 ceph-mon[36569]: pgmap v6400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:22.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:21 smithi104 ceph-mon[36624]: pgmap v6400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:22.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:21 smithi081 ceph-mon[31453]: pgmap v6400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:24.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:23 smithi153 ceph-mon[36569]: pgmap v6401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:24.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:23 smithi104 ceph-mon[36624]: pgmap v6401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:24.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:23 smithi081 ceph-mon[31453]: pgmap v6401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:26.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:25 smithi153 ceph-mon[36569]: pgmap v6402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:26.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:25 smithi104 ceph-mon[36624]: pgmap v6402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:26.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:25 smithi081 ceph-mon[31453]: pgmap v6402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:28.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:27 smithi153 ceph-mon[36569]: pgmap v6403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:28.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:27 smithi104 ceph-mon[36624]: pgmap v6403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:28.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:27 smithi081 ceph-mon[31453]: pgmap v6403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:30.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:29 smithi104 ceph-mon[36624]: pgmap v6404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:30.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:29 smithi081 ceph-mon[31453]: pgmap v6404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:29 smithi153 ceph-mon[36569]: pgmap v6404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:31.481+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:26:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:31.481+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:26:31.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:31.481+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:26:32.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:31 smithi104 ceph-mon[36624]: pgmap v6405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:32.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:31 smithi081 ceph-mon[31453]: pgmap v6405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:32.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:31 smithi153 ceph-mon[36569]: pgmap v6405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:34.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:33 smithi104 ceph-mon[36624]: pgmap v6406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:34.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:33 smithi081 ceph-mon[31453]: pgmap v6406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:34.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:33 smithi153 ceph-mon[36569]: pgmap v6406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:36.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:35 smithi104 ceph-mon[36624]: pgmap v6407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:36.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:35 smithi081 ceph-mon[31453]: pgmap v6407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:36.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:35 smithi153 ceph-mon[36569]: pgmap v6407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:38.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:37 smithi104 ceph-mon[36624]: pgmap v6408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:38.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:37 smithi081 ceph-mon[31453]: pgmap v6408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:38.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:37 smithi153 ceph-mon[36569]: pgmap v6408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:40.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:39 smithi104 ceph-mon[36624]: pgmap v6409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:40.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:26:40.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:39 smithi081 ceph-mon[31453]: pgmap v6409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:40.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:26:40.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:39 smithi153 ceph-mon[36569]: pgmap v6409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:40.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:26:41.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:26:41.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:26:41.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:26:41.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:26:41.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:26:41.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:26:41.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:26:41.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:26:41.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:26:42.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:41 smithi081 ceph-mon[31453]: pgmap v6410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:42.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:41 smithi081 ceph-mon[31453]: pgmap v6411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:42.205 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:41 smithi153 ceph-mon[36569]: pgmap v6410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:42.205 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:41 smithi153 ceph-mon[36569]: pgmap v6411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:42.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:41 smithi104 ceph-mon[36624]: pgmap v6410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:42.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:41 smithi104 ceph-mon[36624]: pgmap v6411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:44.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:43 smithi081 ceph-mon[31453]: pgmap v6412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:44.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:43 smithi153 ceph-mon[36569]: pgmap v6412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:44.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:43 smithi104 ceph-mon[36624]: pgmap v6412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:46.132 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:45 smithi104 ceph-mon[36624]: pgmap v6413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:46.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:45 smithi081 ceph-mon[31453]: pgmap v6413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:46.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:45 smithi153 ceph-mon[36569]: pgmap v6413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:46.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:46.497+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:26:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:46.497+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:26:46.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:26:46.497+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:26:48.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:47 smithi081 ceph-mon[31453]: pgmap v6414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:48.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:47 smithi153 ceph-mon[36569]: pgmap v6414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:48.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:47 smithi104 ceph-mon[36624]: pgmap v6414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:50.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:49 smithi081 ceph-mon[31453]: pgmap v6415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:50.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:49 smithi153 ceph-mon[36569]: pgmap v6415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:50.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:49 smithi104 ceph-mon[36624]: pgmap v6415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:52.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:51 smithi081 ceph-mon[31453]: pgmap v6416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:52.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:51 smithi153 ceph-mon[36569]: pgmap v6416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:52.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:51 smithi104 ceph-mon[36624]: pgmap v6416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:54.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:53 smithi081 ceph-mon[31453]: pgmap v6417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:54.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:53 smithi153 ceph-mon[36569]: pgmap v6417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:54.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:53 smithi104 ceph-mon[36624]: pgmap v6417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:56.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:55 smithi081 ceph-mon[31453]: pgmap v6418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:56.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:55 smithi153 ceph-mon[36569]: pgmap v6418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:56.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:55 smithi104 ceph-mon[36624]: pgmap v6418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:58.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:57 smithi081 ceph-mon[31453]: pgmap v6419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:58.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:57 smithi153 ceph-mon[36569]: pgmap v6419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:26:58.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:57 smithi104 ceph-mon[36624]: pgmap v6419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:00.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:26:59 smithi081 ceph-mon[31453]: pgmap v6420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:00.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:26:59 smithi153 ceph-mon[36569]: pgmap v6420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:00.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:26:59 smithi104 ceph-mon[36624]: pgmap v6420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:01.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:01.504+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:27:01.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:01.504+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:27:01.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:01.504+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:27:02.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:01 smithi153 ceph-mon[36569]: pgmap v6421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:02.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:01 smithi104 ceph-mon[36624]: pgmap v6421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:02.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:01 smithi081 ceph-mon[31453]: pgmap v6421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:04.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:03 smithi153 ceph-mon[36569]: pgmap v6422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:04.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:03 smithi104 ceph-mon[36624]: pgmap v6422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:04.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:03 smithi081 ceph-mon[31453]: pgmap v6422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:06.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:05 smithi153 ceph-mon[36569]: pgmap v6423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:06.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:05 smithi104 ceph-mon[36624]: pgmap v6423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:06.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:05 smithi081 ceph-mon[31453]: pgmap v6423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:07.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:06 smithi153 ceph-mon[36569]: pgmap v6424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:07.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:06 smithi104 ceph-mon[36624]: pgmap v6424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:07.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:06 smithi081 ceph-mon[31453]: pgmap v6424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:09.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:09 smithi081 ceph-mon[31453]: pgmap v6425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:09.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:09 smithi153 ceph-mon[36569]: pgmap v6425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:09.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:09 smithi104 ceph-mon[36624]: pgmap v6425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:11.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:11 smithi081 ceph-mon[31453]: pgmap v6426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:11.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:11 smithi153 ceph-mon[36569]: pgmap v6426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:11.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:11 smithi104 ceph-mon[36624]: pgmap v6426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:13.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:13 smithi081 ceph-mon[31453]: pgmap v6427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:13.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:13 smithi153 ceph-mon[36569]: pgmap v6427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:13.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:13 smithi104 ceph-mon[36624]: pgmap v6427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:15.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:15 smithi081 ceph-mon[31453]: pgmap v6428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:15.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:15 smithi153 ceph-mon[36569]: pgmap v6428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:15.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:15 smithi104 ceph-mon[36624]: pgmap v6428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:16.509+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:27:16.508 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:16.509+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:27:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:16.509+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:27:17.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:17 smithi081 ceph-mon[31453]: pgmap v6429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:17.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:17 smithi153 ceph-mon[36569]: pgmap v6429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:17.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:17 smithi104 ceph-mon[36624]: pgmap v6429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:19.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:19 smithi081 ceph-mon[31453]: pgmap v6430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:19 smithi153 ceph-mon[36569]: pgmap v6430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:19.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:19 smithi104 ceph-mon[36624]: pgmap v6430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:21.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:21 smithi153 ceph-mon[36569]: pgmap v6431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:21.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:21 smithi104 ceph-mon[36624]: pgmap v6431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:21.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:21 smithi081 ceph-mon[31453]: pgmap v6431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:23.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:23 smithi153 ceph-mon[36569]: pgmap v6432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:23.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:23 smithi104 ceph-mon[36624]: pgmap v6432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:23.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:23 smithi081 ceph-mon[31453]: pgmap v6432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:25.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:25 smithi153 ceph-mon[36569]: pgmap v6433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:25.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:25 smithi104 ceph-mon[36624]: pgmap v6433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:25.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:25 smithi081 ceph-mon[31453]: pgmap v6433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:27.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:27 smithi153 ceph-mon[36569]: pgmap v6434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:27.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:27 smithi104 ceph-mon[36624]: pgmap v6434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:27.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:27 smithi081 ceph-mon[31453]: pgmap v6434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:29.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:29 smithi153 ceph-mon[36569]: pgmap v6435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:29.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:29 smithi104 ceph-mon[36624]: pgmap v6435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:29.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:29 smithi081 ceph-mon[31453]: pgmap v6435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:31.510+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:27:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:31.510+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:27:31.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:31.510+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:27:31.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:31 smithi153 ceph-mon[36569]: pgmap v6436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:31.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:31 smithi104 ceph-mon[36624]: pgmap v6436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:31.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:31 smithi081 ceph-mon[31453]: pgmap v6436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:33.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:33 smithi153 ceph-mon[36569]: pgmap v6437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:33.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:33 smithi104 ceph-mon[36624]: pgmap v6437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:33.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:33 smithi081 ceph-mon[31453]: pgmap v6437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:35.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:35 smithi153 ceph-mon[36569]: pgmap v6438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:35.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:35 smithi104 ceph-mon[36624]: pgmap v6438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:35.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:35 smithi081 ceph-mon[31453]: pgmap v6438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:37.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:37 smithi153 ceph-mon[36569]: pgmap v6439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:37.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:37 smithi104 ceph-mon[36624]: pgmap v6439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:37.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:37 smithi081 ceph-mon[31453]: pgmap v6439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:39.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:39 smithi153 ceph-mon[36569]: pgmap v6440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:39.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:39 smithi104 ceph-mon[36624]: pgmap v6440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:39.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:39 smithi081 ceph-mon[31453]: pgmap v6440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:41.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:41 smithi153 ceph-mon[36569]: pgmap v6441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:41.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:27:41.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:41 smithi104 ceph-mon[36624]: pgmap v6441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:41.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:27:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:41 smithi081 ceph-mon[31453]: pgmap v6441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:41.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:27:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:27:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:27:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:42 smithi153 ceph-mon[36569]: pgmap v6442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:42 smithi153 ceph-mon[36569]: pgmap v6443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:27:42.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:27:42.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:27:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:42 smithi104 ceph-mon[36624]: pgmap v6442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:42 smithi104 ceph-mon[36624]: pgmap v6443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:27:42.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:27:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:27:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:42 smithi081 ceph-mon[31453]: pgmap v6442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:42 smithi081 ceph-mon[31453]: pgmap v6443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:27:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:44 smithi081 ceph-mon[31453]: pgmap v6444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:45.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:44 smithi153 ceph-mon[36569]: pgmap v6444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:45.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:44 smithi104 ceph-mon[36624]: pgmap v6444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:46.524 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:46.525+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:27:46.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:46.525+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:27:46.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:27:46.525+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:27:47.133 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:46 smithi104 ceph-mon[36624]: pgmap v6445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:47.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:46 smithi081 ceph-mon[31453]: pgmap v6445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:47.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:46 smithi153 ceph-mon[36569]: pgmap v6445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:49.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:48 smithi081 ceph-mon[31453]: pgmap v6446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:49.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:48 smithi153 ceph-mon[36569]: pgmap v6446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:49.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:48 smithi104 ceph-mon[36624]: pgmap v6446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:51.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:50 smithi081 ceph-mon[31453]: pgmap v6447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:51.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:50 smithi153 ceph-mon[36569]: pgmap v6447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:51.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:50 smithi104 ceph-mon[36624]: pgmap v6447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:53.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:52 smithi081 ceph-mon[31453]: pgmap v6448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:53.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:52 smithi153 ceph-mon[36569]: pgmap v6448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:53.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:52 smithi104 ceph-mon[36624]: pgmap v6448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:55.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:54 smithi081 ceph-mon[31453]: pgmap v6449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:55.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:54 smithi153 ceph-mon[36569]: pgmap v6449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:55.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:54 smithi104 ceph-mon[36624]: pgmap v6449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:57.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:56 smithi153 ceph-mon[36569]: pgmap v6450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:57.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:56 smithi104 ceph-mon[36624]: pgmap v6450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:57.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:56 smithi081 ceph-mon[31453]: pgmap v6450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:59.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:27:58 smithi153 ceph-mon[36569]: pgmap v6451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:59.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:27:58 smithi104 ceph-mon[36624]: pgmap v6451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:27:59.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:27:58 smithi081 ceph-mon[31453]: pgmap v6451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:01.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:00 smithi153 ceph-mon[36569]: pgmap v6452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:01.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:00 smithi104 ceph-mon[36624]: pgmap v6452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:01.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:00 smithi081 ceph-mon[31453]: pgmap v6452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:01.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:01.538+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:01.538+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:28:01.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:01.538+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:28:03.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:02 smithi153 ceph-mon[36569]: pgmap v6453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:03.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:02 smithi104 ceph-mon[36624]: pgmap v6453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:03.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:02 smithi081 ceph-mon[31453]: pgmap v6453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:05.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:04 smithi153 ceph-mon[36569]: pgmap v6454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:04 smithi104 ceph-mon[36624]: pgmap v6454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:05.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:04 smithi081 ceph-mon[31453]: pgmap v6454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:06.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:05 smithi153 ceph-mon[36569]: pgmap v6455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:06.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:05 smithi104 ceph-mon[36624]: pgmap v6455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:06.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:05 smithi081 ceph-mon[31453]: pgmap v6455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:08 smithi081 ceph-mon[31453]: pgmap v6456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:09.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:08 smithi153 ceph-mon[36569]: pgmap v6456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:09.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:08 smithi104 ceph-mon[36624]: pgmap v6456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:10 smithi081 ceph-mon[31453]: pgmap v6457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:11.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:10 smithi153 ceph-mon[36569]: pgmap v6457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:11.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:10 smithi104 ceph-mon[36624]: pgmap v6457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:12 smithi081 ceph-mon[31453]: pgmap v6458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:13.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:12 smithi153 ceph-mon[36569]: pgmap v6458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:12 smithi104 ceph-mon[36624]: pgmap v6458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:14.602+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:28:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:14.602+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:28:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:14.603+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:28:14.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:14.603+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:28:15.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:14 smithi153 ceph-mon[36569]: pgmap v6459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:15.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:14 smithi104 ceph-mon[36624]: pgmap v6459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:14 smithi081 ceph-mon[31453]: pgmap v6459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:16.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:16.551+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:28:16.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:16.551+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:28:16.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:16.551+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:28:17.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:16 smithi153 ceph-mon[36569]: pgmap v6460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:17.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:16 smithi104 ceph-mon[36624]: pgmap v6460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:16 smithi081 ceph-mon[31453]: pgmap v6460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:19.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:18 smithi153 ceph-mon[36569]: pgmap v6461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:19.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:18 smithi104 ceph-mon[36624]: pgmap v6461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:18 smithi081 ceph-mon[31453]: pgmap v6461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:21.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:20 smithi081 ceph-mon[31453]: pgmap v6462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:21.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:20 smithi153 ceph-mon[36569]: pgmap v6462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:21.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:20 smithi104 ceph-mon[36624]: pgmap v6462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:22 smithi153 ceph-mon[36569]: pgmap v6463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:22 smithi104 ceph-mon[36624]: pgmap v6463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:22 smithi081 ceph-mon[31453]: pgmap v6463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:24 smithi081 ceph-mon[31453]: pgmap v6464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:25.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:24 smithi153 ceph-mon[36569]: pgmap v6464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:25.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:24 smithi104 ceph-mon[36624]: pgmap v6464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:27.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:26 smithi153 ceph-mon[36569]: pgmap v6465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:26 smithi104 ceph-mon[36624]: pgmap v6465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:26 smithi081 ceph-mon[31453]: pgmap v6465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:29.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:28 smithi153 ceph-mon[36569]: pgmap v6466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:28 smithi104 ceph-mon[36624]: pgmap v6466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:28 smithi081 ceph-mon[31453]: pgmap v6466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:30 smithi153 ceph-mon[36569]: pgmap v6467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:30 smithi104 ceph-mon[36624]: pgmap v6467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:30 smithi081 ceph-mon[31453]: pgmap v6467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:31.557+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:28:31.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:31.557+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:28:31.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:31.557+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:28:33.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:32 smithi153 ceph-mon[36569]: pgmap v6468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:33.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:32 smithi104 ceph-mon[36624]: pgmap v6468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:32 smithi081 ceph-mon[31453]: pgmap v6468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:35.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:34 smithi153 ceph-mon[36569]: pgmap v6469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:35.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:34 smithi104 ceph-mon[36624]: pgmap v6469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:34 smithi081 ceph-mon[31453]: pgmap v6469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:37.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:36 smithi153 ceph-mon[36569]: pgmap v6470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:36 smithi104 ceph-mon[36624]: pgmap v6470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:36 smithi081 ceph-mon[31453]: pgmap v6470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:38 smithi153 ceph-mon[36569]: pgmap v6471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:38 smithi104 ceph-mon[36624]: pgmap v6471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:38 smithi081 ceph-mon[31453]: pgmap v6471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:40.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:39 smithi153 ceph-mon[36569]: pgmap v6472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:40.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:39 smithi104 ceph-mon[36624]: pgmap v6472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:40.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:39 smithi081 ceph-mon[31453]: pgmap v6472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:43.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:42 smithi153 ceph-mon[36569]: pgmap v6473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:28:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:42 smithi104 ceph-mon[36624]: pgmap v6473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:28:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:42 smithi081 ceph-mon[31453]: pgmap v6473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:28:44.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:28:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:28:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:28:44.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:28:44.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:28:44.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:28:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:28:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:28:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:28:45.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:44 smithi153 ceph-mon[36569]: pgmap v6474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:45.207 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:44 smithi153 ceph-mon[36569]: pgmap v6475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:44 smithi104 ceph-mon[36624]: pgmap v6474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:44 smithi104 ceph-mon[36624]: pgmap v6475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:44 smithi081 ceph-mon[31453]: pgmap v6474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:44 smithi081 ceph-mon[31453]: pgmap v6475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:46.573+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:28:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:46.573+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:28:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:28:46.573+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:28:47.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:46 smithi153 ceph-mon[36569]: pgmap v6476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:46 smithi104 ceph-mon[36624]: pgmap v6476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:46 smithi081 ceph-mon[31453]: pgmap v6476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:49.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:48 smithi153 ceph-mon[36569]: pgmap v6477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:48 smithi104 ceph-mon[36624]: pgmap v6477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:49.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:48 smithi081 ceph-mon[31453]: pgmap v6477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:50.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:49 smithi153 ceph-mon[36569]: pgmap v6478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:50.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:49 smithi104 ceph-mon[36624]: pgmap v6478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:49 smithi081 ceph-mon[31453]: pgmap v6478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:52 smithi153 ceph-mon[36569]: pgmap v6479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:52 smithi104 ceph-mon[36624]: pgmap v6479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:52 smithi081 ceph-mon[31453]: pgmap v6479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:54.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:54 smithi153 ceph-mon[36569]: pgmap v6480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:54 smithi104 ceph-mon[36624]: pgmap v6480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:54 smithi081 ceph-mon[31453]: pgmap v6480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:56 smithi153 ceph-mon[36569]: pgmap v6481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:56 smithi104 ceph-mon[36624]: pgmap v6481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:56.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:56 smithi081 ceph-mon[31453]: pgmap v6481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:28:58 smithi153 ceph-mon[36569]: pgmap v6482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:28:58 smithi104 ceph-mon[36624]: pgmap v6482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:28:58.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:28:58 smithi081 ceph-mon[31453]: pgmap v6482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:00 smithi153 ceph-mon[36569]: pgmap v6483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:00 smithi104 ceph-mon[36624]: pgmap v6483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:00 smithi081 ceph-mon[31453]: pgmap v6483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:01.586+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:29:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:01.586+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:29:01.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:01.586+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:29:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:02 smithi153 ceph-mon[36569]: pgmap v6484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:02 smithi104 ceph-mon[36624]: pgmap v6484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:02 smithi081 ceph-mon[31453]: pgmap v6484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:04 smithi153 ceph-mon[36569]: pgmap v6485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:04 smithi104 ceph-mon[36624]: pgmap v6485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:04 smithi081 ceph-mon[31453]: pgmap v6485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:06 smithi153 ceph-mon[36569]: pgmap v6486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:06 smithi104 ceph-mon[36624]: pgmap v6486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:06 smithi081 ceph-mon[31453]: pgmap v6486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:08 smithi153 ceph-mon[36569]: pgmap v6487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:08 smithi104 ceph-mon[36624]: pgmap v6487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:08 smithi081 ceph-mon[31453]: pgmap v6487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:10 smithi153 ceph-mon[36569]: pgmap v6488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:10 smithi104 ceph-mon[36624]: pgmap v6488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:10 smithi081 ceph-mon[31453]: pgmap v6488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:12 smithi153 ceph-mon[36569]: pgmap v6489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:12 smithi104 ceph-mon[36624]: pgmap v6489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:12 smithi081 ceph-mon[31453]: pgmap v6489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:14.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:14 smithi081 ceph-mon[31453]: pgmap v6490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:14 smithi153 ceph-mon[36569]: pgmap v6490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:14 smithi104 ceph-mon[36624]: pgmap v6490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:16.592+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:29:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:16.592+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:29:16.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:16.592+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:29:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:16 smithi153 ceph-mon[36569]: pgmap v6491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:16 smithi104 ceph-mon[36624]: pgmap v6491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:16 smithi081 ceph-mon[31453]: pgmap v6491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:18 smithi153 ceph-mon[36569]: pgmap v6492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:18 smithi104 ceph-mon[36624]: pgmap v6492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:18 smithi081 ceph-mon[31453]: pgmap v6492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:20 smithi153 ceph-mon[36569]: pgmap v6493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:20 smithi104 ceph-mon[36624]: pgmap v6493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:20 smithi081 ceph-mon[31453]: pgmap v6493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:22 smithi153 ceph-mon[36569]: pgmap v6494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:22 smithi104 ceph-mon[36624]: pgmap v6494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:22 smithi081 ceph-mon[31453]: pgmap v6494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:24 smithi153 ceph-mon[36569]: pgmap v6495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:24 smithi104 ceph-mon[36624]: pgmap v6495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:24 smithi081 ceph-mon[31453]: pgmap v6495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:26 smithi153 ceph-mon[36569]: pgmap v6496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:26 smithi104 ceph-mon[36624]: pgmap v6496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:26 smithi081 ceph-mon[31453]: pgmap v6496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:28 smithi153 ceph-mon[36569]: pgmap v6497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:28 smithi104 ceph-mon[36624]: pgmap v6497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:28.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:28 smithi081 ceph-mon[31453]: pgmap v6497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:30 smithi153 ceph-mon[36569]: pgmap v6498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:30 smithi104 ceph-mon[36624]: pgmap v6498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:30 smithi081 ceph-mon[31453]: pgmap v6498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:31.597 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:31.596+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:29:31.597 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:31.596+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:29:31.597 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:31.596+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:29:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:32 smithi153 ceph-mon[36569]: pgmap v6499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:32 smithi104 ceph-mon[36624]: pgmap v6499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:32 smithi081 ceph-mon[31453]: pgmap v6499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:34 smithi153 ceph-mon[36569]: pgmap v6500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:34 smithi104 ceph-mon[36624]: pgmap v6500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:34 smithi081 ceph-mon[31453]: pgmap v6500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:36 smithi153 ceph-mon[36569]: pgmap v6501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:36.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:36 smithi104 ceph-mon[36624]: pgmap v6501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:36 smithi081 ceph-mon[31453]: pgmap v6501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:38 smithi153 ceph-mon[36569]: pgmap v6502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:38 smithi104 ceph-mon[36624]: pgmap v6502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:38 smithi081 ceph-mon[31453]: pgmap v6502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:40 smithi104 ceph-mon[36624]: pgmap v6503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:40 smithi081 ceph-mon[31453]: pgmap v6503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:40 smithi153 ceph-mon[36569]: pgmap v6503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:42 smithi104 ceph-mon[36624]: pgmap v6504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:42 smithi081 ceph-mon[31453]: pgmap v6504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:42 smithi153 ceph-mon[36569]: pgmap v6504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: pgmap v6505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:29:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: pgmap v6505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:29:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: pgmap v6505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:29:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:29:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:29:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:29:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:29:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:29:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:29:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:29:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:29:45.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: pgmap v6506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:45.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: pgmap v6507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:45.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:29:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:29:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:29:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:29:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:29:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: pgmap v6506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: pgmap v6507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:29:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:29:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:29:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: pgmap v6506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: pgmap v6507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:29:46.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:46.611+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:29:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:46.612+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:29:46.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:29:46.612+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:29:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:47 smithi153 ceph-mon[36569]: pgmap v6508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:47 smithi104 ceph-mon[36624]: pgmap v6508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:47 smithi081 ceph-mon[31453]: pgmap v6508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:49 smithi153 ceph-mon[36569]: pgmap v6509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:49 smithi104 ceph-mon[36624]: pgmap v6509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:49 smithi081 ceph-mon[31453]: pgmap v6509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:51 smithi153 ceph-mon[36569]: pgmap v6510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:51 smithi104 ceph-mon[36624]: pgmap v6510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:51 smithi081 ceph-mon[31453]: pgmap v6510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:53 smithi153 ceph-mon[36569]: pgmap v6511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:53 smithi104 ceph-mon[36624]: pgmap v6511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:53 smithi081 ceph-mon[31453]: pgmap v6511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:55 smithi153 ceph-mon[36569]: pgmap v6512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:55 smithi104 ceph-mon[36624]: pgmap v6512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:55 smithi081 ceph-mon[31453]: pgmap v6512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:57 smithi153 ceph-mon[36569]: pgmap v6513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:57 smithi104 ceph-mon[36624]: pgmap v6513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:57 smithi081 ceph-mon[31453]: pgmap v6513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:29:59 smithi153 ceph-mon[36569]: pgmap v6514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:29:59 smithi104 ceph-mon[36624]: pgmap v6514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:29:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:29:59 smithi081 ceph-mon[31453]: pgmap v6514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:30:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:30:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:30:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:01 smithi153 ceph-mon[36569]: pgmap v6515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:01 smithi104 ceph-mon[36624]: pgmap v6515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:01.351 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:01 smithi081 ceph-mon[31453]: pgmap v6515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:01.620 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:01.620+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:30:01.620 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:01.620+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:30:01.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:01.620+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:30:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:03 smithi153 ceph-mon[36569]: pgmap v6516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:03 smithi104 ceph-mon[36624]: pgmap v6516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:03 smithi081 ceph-mon[31453]: pgmap v6516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:05 smithi153 ceph-mon[36569]: pgmap v6517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:05 smithi104 ceph-mon[36624]: pgmap v6517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:05 smithi081 ceph-mon[31453]: pgmap v6517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:07 smithi153 ceph-mon[36569]: pgmap v6518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:07 smithi104 ceph-mon[36624]: pgmap v6518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:07 smithi081 ceph-mon[31453]: pgmap v6518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:09 smithi153 ceph-mon[36569]: pgmap v6519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:09 smithi104 ceph-mon[36624]: pgmap v6519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:09 smithi081 ceph-mon[31453]: pgmap v6519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:11 smithi153 ceph-mon[36569]: pgmap v6520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:11 smithi104 ceph-mon[36624]: pgmap v6520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:11 smithi081 ceph-mon[31453]: pgmap v6520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:13 smithi153 ceph-mon[36569]: pgmap v6521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:13 smithi104 ceph-mon[36624]: pgmap v6521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:13 smithi081 ceph-mon[31453]: pgmap v6521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:15 smithi153 ceph-mon[36569]: pgmap v6522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:15 smithi104 ceph-mon[36624]: pgmap v6522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:15 smithi081 ceph-mon[31453]: pgmap v6522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:16.632+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:30:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:16.632+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:30:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:16.632+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:30:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:17 smithi153 ceph-mon[36569]: pgmap v6523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:17 smithi104 ceph-mon[36624]: pgmap v6523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:17 smithi081 ceph-mon[31453]: pgmap v6523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:19 smithi153 ceph-mon[36569]: pgmap v6524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:19 smithi104 ceph-mon[36624]: pgmap v6524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:19 smithi081 ceph-mon[31453]: pgmap v6524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:21 smithi153 ceph-mon[36569]: pgmap v6525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:21 smithi104 ceph-mon[36624]: pgmap v6525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:21 smithi081 ceph-mon[31453]: pgmap v6525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:23 smithi153 ceph-mon[36569]: pgmap v6526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:23 smithi104 ceph-mon[36624]: pgmap v6526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:23 smithi081 ceph-mon[31453]: pgmap v6526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:25 smithi153 ceph-mon[36569]: pgmap v6527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:25 smithi104 ceph-mon[36624]: pgmap v6527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:25 smithi081 ceph-mon[31453]: pgmap v6527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:27 smithi153 ceph-mon[36569]: pgmap v6528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:27 smithi104 ceph-mon[36624]: pgmap v6528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:27 smithi081 ceph-mon[31453]: pgmap v6528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:29 smithi153 ceph-mon[36569]: pgmap v6529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:29 smithi104 ceph-mon[36624]: pgmap v6529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:29 smithi081 ceph-mon[31453]: pgmap v6529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:31 smithi153 ceph-mon[36569]: pgmap v6530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:31 smithi104 ceph-mon[36624]: pgmap v6530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:31 smithi081 ceph-mon[31453]: pgmap v6530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:31.641+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:30:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:31.641+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:30:31.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:31.641+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:30:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:33 smithi153 ceph-mon[36569]: pgmap v6531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:33 smithi104 ceph-mon[36624]: pgmap v6531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:33 smithi081 ceph-mon[31453]: pgmap v6531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:35 smithi153 ceph-mon[36569]: pgmap v6532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:35 smithi104 ceph-mon[36624]: pgmap v6532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:35 smithi081 ceph-mon[31453]: pgmap v6532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:37 smithi153 ceph-mon[36569]: pgmap v6533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:37 smithi104 ceph-mon[36624]: pgmap v6533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:37 smithi081 ceph-mon[31453]: pgmap v6533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:39 smithi153 ceph-mon[36569]: pgmap v6534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:39 smithi104 ceph-mon[36624]: pgmap v6534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:39 smithi081 ceph-mon[31453]: pgmap v6534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:41 smithi153 ceph-mon[36569]: pgmap v6535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:41 smithi104 ceph-mon[36624]: pgmap v6535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:41 smithi081 ceph-mon[31453]: pgmap v6535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:43 smithi153 ceph-mon[36569]: pgmap v6536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:43 smithi104 ceph-mon[36624]: pgmap v6536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:43 smithi081 ceph-mon[31453]: pgmap v6536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:45 smithi153 ceph-mon[36569]: pgmap v6537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:45 smithi104 ceph-mon[36624]: pgmap v6537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:45 smithi081 ceph-mon[31453]: pgmap v6537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:30:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:30:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:30:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:46.655+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:30:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:46.655+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:30:46.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:30:46.655+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:30:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:30:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:30:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:47 smithi153 ceph-mon[36569]: pgmap v6538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:47 smithi153 ceph-mon[36569]: pgmap v6539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:30:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:30:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:30:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:47 smithi104 ceph-mon[36624]: pgmap v6538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:47 smithi104 ceph-mon[36624]: pgmap v6539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:30:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:30:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:30:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:47 smithi081 ceph-mon[31453]: pgmap v6538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:47 smithi081 ceph-mon[31453]: pgmap v6539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:30:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:49 smithi153 ceph-mon[36569]: pgmap v6540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:49 smithi104 ceph-mon[36624]: pgmap v6540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:49 smithi081 ceph-mon[31453]: pgmap v6540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:51 smithi153 ceph-mon[36569]: pgmap v6541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:51 smithi104 ceph-mon[36624]: pgmap v6541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:51 smithi081 ceph-mon[31453]: pgmap v6541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:53 smithi153 ceph-mon[36569]: pgmap v6542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:53 smithi104 ceph-mon[36624]: pgmap v6542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:53 smithi081 ceph-mon[31453]: pgmap v6542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:55.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:55 smithi153 ceph-mon[36569]: pgmap v6543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:55 smithi104 ceph-mon[36624]: pgmap v6543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:55 smithi081 ceph-mon[31453]: pgmap v6543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:57 smithi153 ceph-mon[36569]: pgmap v6544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:57 smithi104 ceph-mon[36624]: pgmap v6544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:57 smithi081 ceph-mon[31453]: pgmap v6544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:30:59 smithi153 ceph-mon[36569]: pgmap v6545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:30:59 smithi104 ceph-mon[36624]: pgmap v6545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:30:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:30:59 smithi081 ceph-mon[31453]: pgmap v6545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:01.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:01.669+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:31:01.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:01.669+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:31:01.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:01.669+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:31:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:01 smithi153 ceph-mon[36569]: pgmap v6546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:01 smithi104 ceph-mon[36624]: pgmap v6546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:01 smithi081 ceph-mon[31453]: pgmap v6546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:03 smithi153 ceph-mon[36569]: pgmap v6547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:03 smithi104 ceph-mon[36624]: pgmap v6547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:03 smithi081 ceph-mon[31453]: pgmap v6547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:05 smithi153 ceph-mon[36569]: pgmap v6548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:05 smithi104 ceph-mon[36624]: pgmap v6548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:05 smithi081 ceph-mon[31453]: pgmap v6548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:07 smithi153 ceph-mon[36569]: pgmap v6549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:07 smithi104 ceph-mon[36624]: pgmap v6549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:07 smithi081 ceph-mon[31453]: pgmap v6549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:09 smithi153 ceph-mon[36569]: pgmap v6550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:09 smithi104 ceph-mon[36624]: pgmap v6550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:09 smithi081 ceph-mon[31453]: pgmap v6550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:11 smithi153 ceph-mon[36569]: pgmap v6551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:11 smithi104 ceph-mon[36624]: pgmap v6551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:11 smithi081 ceph-mon[31453]: pgmap v6551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:12.119+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000665) for more than 900000000 us, fault. 2024-04-13T00:31:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:12.119+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f16f400e9c0 tx=0x7f16f40d6000 comp rx=0 tx=0).stop 2024-04-13T00:31:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:12.119+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T00:31:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:12.119+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T00:31:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:12.119+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T00:31:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:12.120+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T00:31:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:12.121+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006550 tx=0x7f16f40ba000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T00:31:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:13 smithi153 ceph-mon[36569]: pgmap v6552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:13 smithi104 ceph-mon[36624]: pgmap v6552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:13 smithi081 ceph-mon[31453]: pgmap v6552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:14.631+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:31:14.633 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:14.631+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:31:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:14.632+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:31:14.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:14.632+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:31:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:15 smithi153 ceph-mon[36569]: pgmap v6553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:15 smithi104 ceph-mon[36624]: pgmap v6553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:15 smithi081 ceph-mon[31453]: pgmap v6553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:16.673+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:31:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:16.673+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:31:16.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:16.673+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:31:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:17 smithi104 ceph-mon[36624]: pgmap v6554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:17 smithi081 ceph-mon[31453]: pgmap v6554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:17 smithi153 ceph-mon[36569]: pgmap v6554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:19 smithi104 ceph-mon[36624]: pgmap v6555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:19 smithi081 ceph-mon[31453]: pgmap v6555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:19 smithi153 ceph-mon[36569]: pgmap v6555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:21 smithi104 ceph-mon[36624]: pgmap v6556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:21 smithi081 ceph-mon[31453]: pgmap v6556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:21 smithi153 ceph-mon[36569]: pgmap v6556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:23 smithi104 ceph-mon[36624]: pgmap v6557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:23 smithi081 ceph-mon[31453]: pgmap v6557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:23 smithi153 ceph-mon[36569]: pgmap v6557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:25 smithi104 ceph-mon[36624]: pgmap v6558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:25 smithi081 ceph-mon[31453]: pgmap v6558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:25 smithi153 ceph-mon[36569]: pgmap v6558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:27 smithi081 ceph-mon[31453]: pgmap v6559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:27 smithi153 ceph-mon[36569]: pgmap v6559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:27 smithi104 ceph-mon[36624]: pgmap v6559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:29 smithi081 ceph-mon[31453]: pgmap v6560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:29 smithi153 ceph-mon[36569]: pgmap v6560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:29 smithi104 ceph-mon[36624]: pgmap v6560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:31.681+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:31:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:31.681+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:31:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:31.681+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:31:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:31 smithi081 ceph-mon[31453]: pgmap v6561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:31 smithi153 ceph-mon[36569]: pgmap v6561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:31 smithi104 ceph-mon[36624]: pgmap v6561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:33 smithi081 ceph-mon[31453]: pgmap v6562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:33 smithi153 ceph-mon[36569]: pgmap v6562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:33 smithi104 ceph-mon[36624]: pgmap v6562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:35 smithi081 ceph-mon[31453]: pgmap v6563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:35 smithi153 ceph-mon[36569]: pgmap v6563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:35 smithi104 ceph-mon[36624]: pgmap v6563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:37 smithi081 ceph-mon[31453]: pgmap v6564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:37 smithi153 ceph-mon[36569]: pgmap v6564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:37 smithi104 ceph-mon[36624]: pgmap v6564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:39 smithi081 ceph-mon[31453]: pgmap v6565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:39 smithi153 ceph-mon[36569]: pgmap v6565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:39 smithi104 ceph-mon[36624]: pgmap v6565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:41 smithi081 ceph-mon[31453]: pgmap v6566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:41 smithi153 ceph-mon[36569]: pgmap v6566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:41 smithi104 ceph-mon[36624]: pgmap v6566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:43 smithi081 ceph-mon[31453]: pgmap v6567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:43 smithi153 ceph-mon[36569]: pgmap v6567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:43 smithi104 ceph-mon[36624]: pgmap v6567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:45 smithi081 ceph-mon[31453]: pgmap v6568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:45 smithi153 ceph-mon[36569]: pgmap v6568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:45 smithi104 ceph-mon[36624]: pgmap v6568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:46.697+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:31:46.698 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:46.697+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:31:46.699 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:31:46.697+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:31:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:47 smithi081 ceph-mon[31453]: pgmap v6569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:31:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:47 smithi153 ceph-mon[36569]: pgmap v6569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:31:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:47 smithi104 ceph-mon[36624]: pgmap v6569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:31:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:31:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:31:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:31:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:31:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:31:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:31:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:31:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:31:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:31:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:49 smithi081 ceph-mon[31453]: pgmap v6570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:49 smithi081 ceph-mon[31453]: pgmap v6571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:49.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:49 smithi104 ceph-mon[36624]: pgmap v6570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:49.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:49 smithi104 ceph-mon[36624]: pgmap v6571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:49 smithi153 ceph-mon[36569]: pgmap v6570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:49 smithi153 ceph-mon[36569]: pgmap v6571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:51 smithi081 ceph-mon[31453]: pgmap v6572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:51 smithi153 ceph-mon[36569]: pgmap v6572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:51 smithi104 ceph-mon[36624]: pgmap v6572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:53 smithi153 ceph-mon[36569]: pgmap v6573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:53 smithi104 ceph-mon[36624]: pgmap v6573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:53 smithi081 ceph-mon[31453]: pgmap v6573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:55 smithi153 ceph-mon[36569]: pgmap v6574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:55 smithi104 ceph-mon[36624]: pgmap v6574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:55 smithi081 ceph-mon[31453]: pgmap v6574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:57 smithi153 ceph-mon[36569]: pgmap v6575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:57 smithi104 ceph-mon[36624]: pgmap v6575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:31:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:57 smithi081 ceph-mon[31453]: pgmap v6575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:31:59 smithi153 ceph-mon[36569]: pgmap v6576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:31:59 smithi104 ceph-mon[36624]: pgmap v6576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:31:59 smithi081 ceph-mon[31453]: pgmap v6576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:01.706+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:32:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:01.706+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:32:01.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:01.706+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:32:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:01 smithi153 ceph-mon[36569]: pgmap v6577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:01 smithi104 ceph-mon[36624]: pgmap v6577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:01 smithi081 ceph-mon[31453]: pgmap v6577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:03 smithi153 ceph-mon[36569]: pgmap v6578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:03 smithi104 ceph-mon[36624]: pgmap v6578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:03 smithi081 ceph-mon[31453]: pgmap v6578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:05 smithi153 ceph-mon[36569]: pgmap v6579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:05 smithi104 ceph-mon[36624]: pgmap v6579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:05 smithi081 ceph-mon[31453]: pgmap v6579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:07 smithi153 ceph-mon[36569]: pgmap v6580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:07 smithi104 ceph-mon[36624]: pgmap v6580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:07 smithi081 ceph-mon[31453]: pgmap v6580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:09 smithi153 ceph-mon[36569]: pgmap v6581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:09 smithi104 ceph-mon[36624]: pgmap v6581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:09 smithi081 ceph-mon[31453]: pgmap v6581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:11 smithi153 ceph-mon[36569]: pgmap v6582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:11 smithi104 ceph-mon[36624]: pgmap v6582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:11 smithi081 ceph-mon[31453]: pgmap v6582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:13 smithi153 ceph-mon[36569]: pgmap v6583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:13 smithi104 ceph-mon[36624]: pgmap v6583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:13 smithi081 ceph-mon[31453]: pgmap v6583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:15 smithi153 ceph-mon[36569]: pgmap v6584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:15 smithi104 ceph-mon[36624]: pgmap v6584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:15 smithi081 ceph-mon[31453]: pgmap v6584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:16.718+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:32:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:16.718+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:32:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:16.718+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:32:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:17 smithi153 ceph-mon[36569]: pgmap v6585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:17 smithi104 ceph-mon[36624]: pgmap v6585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:17 smithi081 ceph-mon[31453]: pgmap v6585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:19 smithi153 ceph-mon[36569]: pgmap v6586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:19 smithi104 ceph-mon[36624]: pgmap v6586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:19 smithi081 ceph-mon[31453]: pgmap v6586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:21 smithi153 ceph-mon[36569]: pgmap v6587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:21 smithi104 ceph-mon[36624]: pgmap v6587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:22.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:21 smithi081 ceph-mon[31453]: pgmap v6587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:23 smithi153 ceph-mon[36569]: pgmap v6588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:23 smithi104 ceph-mon[36624]: pgmap v6588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:23 smithi081 ceph-mon[31453]: pgmap v6588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:25 smithi153 ceph-mon[36569]: pgmap v6589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:25 smithi104 ceph-mon[36624]: pgmap v6589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:25 smithi081 ceph-mon[31453]: pgmap v6589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:27 smithi153 ceph-mon[36569]: pgmap v6590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:27 smithi104 ceph-mon[36624]: pgmap v6590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:27 smithi081 ceph-mon[31453]: pgmap v6590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:29 smithi153 ceph-mon[36569]: pgmap v6591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:29 smithi104 ceph-mon[36624]: pgmap v6591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:29 smithi081 ceph-mon[31453]: pgmap v6591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:31.734+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:32:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:31.734+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:32:31.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:31.734+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:32:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:31 smithi153 ceph-mon[36569]: pgmap v6592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:31 smithi104 ceph-mon[36624]: pgmap v6592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:31 smithi081 ceph-mon[31453]: pgmap v6592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:33 smithi153 ceph-mon[36569]: pgmap v6593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:33 smithi104 ceph-mon[36624]: pgmap v6593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:33 smithi081 ceph-mon[31453]: pgmap v6593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:35 smithi153 ceph-mon[36569]: pgmap v6594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:35 smithi104 ceph-mon[36624]: pgmap v6594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:35 smithi081 ceph-mon[31453]: pgmap v6594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:37 smithi153 ceph-mon[36569]: pgmap v6595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:37 smithi104 ceph-mon[36624]: pgmap v6595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:37 smithi081 ceph-mon[31453]: pgmap v6595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:39 smithi153 ceph-mon[36569]: pgmap v6596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:39 smithi104 ceph-mon[36624]: pgmap v6596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:39 smithi081 ceph-mon[31453]: pgmap v6596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:41 smithi153 ceph-mon[36569]: pgmap v6597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:41 smithi104 ceph-mon[36624]: pgmap v6597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:41 smithi081 ceph-mon[31453]: pgmap v6597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:44.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:43 smithi153 ceph-mon[36569]: pgmap v6598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:43 smithi104 ceph-mon[36624]: pgmap v6598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:43 smithi081 ceph-mon[31453]: pgmap v6598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:45 smithi153 ceph-mon[36569]: pgmap v6599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:45 smithi104 ceph-mon[36624]: pgmap v6599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:45 smithi081 ceph-mon[31453]: pgmap v6599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:46.748+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:32:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:46.748+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:32:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:32:46.748+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:32:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:47 smithi153 ceph-mon[36569]: pgmap v6600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:47 smithi104 ceph-mon[36624]: pgmap v6600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:47 smithi081 ceph-mon[31453]: pgmap v6600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:49.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:32:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:32:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:32:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:49 smithi104 ceph-mon[36624]: pgmap v6601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:50.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:32:50.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:32:50.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:32:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:49 smithi081 ceph-mon[31453]: pgmap v6601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:32:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:32:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:32:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:49 smithi153 ceph-mon[36569]: pgmap v6601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:32:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:32:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:32:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:50 smithi104 ceph-mon[36624]: pgmap v6602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:50 smithi104 ceph-mon[36624]: pgmap v6603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:50 smithi081 ceph-mon[31453]: pgmap v6602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:50 smithi081 ceph-mon[31453]: pgmap v6603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:50 smithi153 ceph-mon[36569]: pgmap v6602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:50 smithi153 ceph-mon[36569]: pgmap v6603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:52 smithi104 ceph-mon[36624]: pgmap v6604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:52 smithi081 ceph-mon[31453]: pgmap v6604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:52 smithi153 ceph-mon[36569]: pgmap v6604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:54 smithi081 ceph-mon[31453]: pgmap v6605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:54 smithi153 ceph-mon[36569]: pgmap v6605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:54 smithi104 ceph-mon[36624]: pgmap v6605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:56 smithi081 ceph-mon[31453]: pgmap v6606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:56 smithi153 ceph-mon[36569]: pgmap v6606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:56 smithi104 ceph-mon[36624]: pgmap v6606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:32:58 smithi081 ceph-mon[31453]: pgmap v6607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:32:58 smithi153 ceph-mon[36569]: pgmap v6607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:32:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:32:58 smithi104 ceph-mon[36624]: pgmap v6607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:00 smithi081 ceph-mon[31453]: pgmap v6608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:00 smithi153 ceph-mon[36569]: pgmap v6608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:00 smithi104 ceph-mon[36624]: pgmap v6608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:01.752+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:33:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:01.752+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:33:01.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:01.752+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:33:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:02 smithi081 ceph-mon[31453]: pgmap v6609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:02 smithi153 ceph-mon[36569]: pgmap v6609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:02 smithi104 ceph-mon[36624]: pgmap v6609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:04 smithi081 ceph-mon[31453]: pgmap v6610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:04 smithi153 ceph-mon[36569]: pgmap v6610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:04 smithi104 ceph-mon[36624]: pgmap v6610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:06 smithi081 ceph-mon[31453]: pgmap v6611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:06 smithi153 ceph-mon[36569]: pgmap v6611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:06 smithi104 ceph-mon[36624]: pgmap v6611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:08 smithi081 ceph-mon[31453]: pgmap v6612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:08 smithi153 ceph-mon[36569]: pgmap v6612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:08 smithi104 ceph-mon[36624]: pgmap v6612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:10 smithi081 ceph-mon[31453]: pgmap v6613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:10 smithi153 ceph-mon[36569]: pgmap v6613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:10 smithi104 ceph-mon[36624]: pgmap v6613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:12 smithi081 ceph-mon[31453]: pgmap v6614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:12 smithi153 ceph-mon[36569]: pgmap v6614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:12 smithi104 ceph-mon[36624]: pgmap v6614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:14 smithi081 ceph-mon[31453]: pgmap v6615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:15.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:14 smithi153 ceph-mon[36569]: pgmap v6615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:14 smithi104 ceph-mon[36624]: pgmap v6615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:16.764+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:33:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:16.764+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:33:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:16.764+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:33:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:16 smithi081 ceph-mon[31453]: pgmap v6616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:16 smithi153 ceph-mon[36569]: pgmap v6616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:16 smithi104 ceph-mon[36624]: pgmap v6616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:18 smithi081 ceph-mon[31453]: pgmap v6617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:18 smithi153 ceph-mon[36569]: pgmap v6617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:18 smithi104 ceph-mon[36624]: pgmap v6617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:20 smithi081 ceph-mon[31453]: pgmap v6618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:20 smithi153 ceph-mon[36569]: pgmap v6618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:20 smithi104 ceph-mon[36624]: pgmap v6618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:22 smithi153 ceph-mon[36569]: pgmap v6619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:22 smithi104 ceph-mon[36624]: pgmap v6619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:22 smithi081 ceph-mon[31453]: pgmap v6619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:24 smithi153 ceph-mon[36569]: pgmap v6620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:25.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:24 smithi081 ceph-mon[31453]: pgmap v6620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:24 smithi104 ceph-mon[36624]: pgmap v6620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:26 smithi153 ceph-mon[36569]: pgmap v6621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:26 smithi104 ceph-mon[36624]: pgmap v6621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:26 smithi081 ceph-mon[31453]: pgmap v6621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:28 smithi153 ceph-mon[36569]: pgmap v6622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:28 smithi104 ceph-mon[36624]: pgmap v6622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:28 smithi081 ceph-mon[31453]: pgmap v6622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:30 smithi153 ceph-mon[36569]: pgmap v6623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:30 smithi104 ceph-mon[36624]: pgmap v6623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:30 smithi081 ceph-mon[31453]: pgmap v6623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:31.776+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:33:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:31.776+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:33:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:31.777+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:33:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:32 smithi153 ceph-mon[36569]: pgmap v6624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:33.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:32 smithi104 ceph-mon[36624]: pgmap v6624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:32 smithi081 ceph-mon[31453]: pgmap v6624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:34 smithi153 ceph-mon[36569]: pgmap v6625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:34 smithi104 ceph-mon[36624]: pgmap v6625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:34 smithi081 ceph-mon[31453]: pgmap v6625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:36 smithi153 ceph-mon[36569]: pgmap v6626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:36 smithi104 ceph-mon[36624]: pgmap v6626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:37.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:36 smithi081 ceph-mon[31453]: pgmap v6626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:38 smithi153 ceph-mon[36569]: pgmap v6627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:38 smithi104 ceph-mon[36624]: pgmap v6627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:39.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:38 smithi081 ceph-mon[31453]: pgmap v6627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:40 smithi153 ceph-mon[36569]: pgmap v6628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:40 smithi104 ceph-mon[36624]: pgmap v6628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:40 smithi081 ceph-mon[31453]: pgmap v6628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:42 smithi153 ceph-mon[36569]: pgmap v6629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:42 smithi104 ceph-mon[36624]: pgmap v6629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:43.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:42 smithi081 ceph-mon[31453]: pgmap v6629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:44 smithi153 ceph-mon[36569]: pgmap v6630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:44 smithi104 ceph-mon[36624]: pgmap v6630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:45.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:44 smithi081 ceph-mon[31453]: pgmap v6630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:46.790+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:33:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:46.790+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:33:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:33:46.790+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:33:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:46 smithi153 ceph-mon[36569]: pgmap v6631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:46 smithi104 ceph-mon[36624]: pgmap v6631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:46 smithi081 ceph-mon[31453]: pgmap v6631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:48 smithi153 ceph-mon[36569]: pgmap v6632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:48 smithi104 ceph-mon[36624]: pgmap v6632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:48 smithi081 ceph-mon[31453]: pgmap v6632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:33:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:33:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:33:51.292 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:50 smithi104 ceph-mon[36624]: pgmap v6633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:51.292 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.292 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.292 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.292 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:50 smithi153 ceph-mon[36569]: pgmap v6633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:50 smithi081 ceph-mon[31453]: pgmap v6633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:52 smithi153 ceph-mon[36569]: pgmap v6634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:52 smithi104 ceph-mon[36624]: pgmap v6634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:52 smithi081 ceph-mon[31453]: pgmap v6634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:53.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:55 smithi153 ceph-mon[36569]: pgmap v6635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:55 smithi104 ceph-mon[36624]: pgmap v6635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:55.339 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:55 smithi081 ceph-mon[31453]: pgmap v6635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:55.339 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:57 smithi153 ceph-mon[36569]: pgmap v6636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:33:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:33:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:57 smithi153 ceph-mon[36569]: pgmap v6637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:57 smithi153 ceph-mon[36569]: pgmap v6638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:57 smithi104 ceph-mon[36624]: pgmap v6636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:33:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:33:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:57 smithi104 ceph-mon[36624]: pgmap v6637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:57 smithi104 ceph-mon[36624]: pgmap v6638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:57.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:57 smithi081 ceph-mon[31453]: pgmap v6636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:33:57.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:33:57.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:57 smithi081 ceph-mon[31453]: pgmap v6637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:57 smithi081 ceph-mon[31453]: pgmap v6638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:57.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:33:59.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:33:59 smithi081 ceph-mon[31453]: pgmap v6639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:33:59 smithi153 ceph-mon[36569]: pgmap v6639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:33:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:33:59 smithi104 ceph-mon[36624]: pgmap v6639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:01.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:01 smithi081 ceph-mon[31453]: pgmap v6640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:01.808 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:01.805+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:34:01.808 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:01.805+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:34:01.808 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:01.805+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:34:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:01 smithi153 ceph-mon[36569]: pgmap v6640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:01.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:01 smithi104 ceph-mon[36624]: pgmap v6640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:03.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:03 smithi081 ceph-mon[31453]: pgmap v6641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:03 smithi153 ceph-mon[36569]: pgmap v6641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:03.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:03 smithi104 ceph-mon[36624]: pgmap v6641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:05 smithi081 ceph-mon[31453]: pgmap v6642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:05 smithi153 ceph-mon[36569]: pgmap v6642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:05 smithi104 ceph-mon[36624]: pgmap v6642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:07 smithi081 ceph-mon[31453]: pgmap v6643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:07.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:07 smithi153 ceph-mon[36569]: pgmap v6643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:07.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:07 smithi104 ceph-mon[36624]: pgmap v6643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:09.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:09 smithi081 ceph-mon[31453]: pgmap v6644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:09 smithi153 ceph-mon[36569]: pgmap v6644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:09.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:09 smithi104 ceph-mon[36624]: pgmap v6644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:11.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:11 smithi081 ceph-mon[31453]: pgmap v6645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:11.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:11 smithi153 ceph-mon[36569]: pgmap v6645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:11 smithi104 ceph-mon[36624]: pgmap v6645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:13.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:13 smithi081 ceph-mon[31453]: pgmap v6646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:13 smithi153 ceph-mon[36569]: pgmap v6646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:13 smithi104 ceph-mon[36624]: pgmap v6646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:14.663 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:14.661+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:34:14.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:14.661+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:34:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:14.662+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T00:34:14.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:14.662+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:34:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:14.662+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T00:34:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:14.662+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:34:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:14.662+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:34:14.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:14.662+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:34:15.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:15 smithi081 ceph-mon[31453]: pgmap v6647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:15 smithi153 ceph-mon[36569]: pgmap v6647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:15 smithi104 ceph-mon[36624]: pgmap v6647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:16.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:16.813+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:34:16.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:16.813+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:34:16.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:16.813+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:34:17.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:17 smithi081 ceph-mon[31453]: pgmap v6648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:17 smithi153 ceph-mon[36569]: pgmap v6648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:17.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:17 smithi104 ceph-mon[36624]: pgmap v6648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:19.653 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:19 smithi104 ceph-mon[36624]: pgmap v6649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:19 smithi081 ceph-mon[31453]: pgmap v6649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:19.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:19 smithi153 ceph-mon[36569]: pgmap v6649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:21.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:21 smithi081 ceph-mon[31453]: pgmap v6650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:21.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:21 smithi153 ceph-mon[36569]: pgmap v6650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:21.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:21 smithi104 ceph-mon[36624]: pgmap v6650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:23.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:23 smithi081 ceph-mon[31453]: pgmap v6651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:23.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:23 smithi153 ceph-mon[36569]: pgmap v6651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:23.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:23 smithi104 ceph-mon[36624]: pgmap v6651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:25.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:25 smithi153 ceph-mon[36569]: pgmap v6652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:25 smithi104 ceph-mon[36624]: pgmap v6652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:25 smithi081 ceph-mon[31453]: pgmap v6652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:27 smithi153 ceph-mon[36569]: pgmap v6653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:27 smithi104 ceph-mon[36624]: pgmap v6653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:27 smithi081 ceph-mon[31453]: pgmap v6653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:29.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:29 smithi153 ceph-mon[36569]: pgmap v6654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:29 smithi104 ceph-mon[36624]: pgmap v6654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:29.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:29 smithi081 ceph-mon[31453]: pgmap v6654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:31.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:31 smithi153 ceph-mon[36569]: pgmap v6655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:31.828+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:34:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:31.828+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:34:31.831 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:31.828+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:34:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:31 smithi104 ceph-mon[36624]: pgmap v6655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:31 smithi081 ceph-mon[31453]: pgmap v6655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:33.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:33 smithi153 ceph-mon[36569]: pgmap v6656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:33 smithi104 ceph-mon[36624]: pgmap v6656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:33 smithi081 ceph-mon[31453]: pgmap v6656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:35.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:35 smithi153 ceph-mon[36569]: pgmap v6657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:35.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:35 smithi104 ceph-mon[36624]: pgmap v6657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:35.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:35 smithi081 ceph-mon[31453]: pgmap v6657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:37.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:37 smithi153 ceph-mon[36569]: pgmap v6658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:37 smithi104 ceph-mon[36624]: pgmap v6658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:37 smithi081 ceph-mon[31453]: pgmap v6658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:39 smithi153 ceph-mon[36569]: pgmap v6659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:39.859 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:39 smithi104 ceph-mon[36624]: pgmap v6659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:39.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:39 smithi081 ceph-mon[31453]: pgmap v6659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:41.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:41 smithi153 ceph-mon[36569]: pgmap v6660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:41.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:41 smithi104 ceph-mon[36624]: pgmap v6660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:41 smithi081 ceph-mon[31453]: pgmap v6660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:43.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:43 smithi153 ceph-mon[36569]: pgmap v6661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:43 smithi104 ceph-mon[36624]: pgmap v6661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:43.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:43 smithi081 ceph-mon[31453]: pgmap v6661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:45.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:45 smithi153 ceph-mon[36569]: pgmap v6662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:45.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:45 smithi104 ceph-mon[36624]: pgmap v6662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:45 smithi081 ceph-mon[31453]: pgmap v6662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:46.840+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:34:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:46.840+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:34:46.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:34:46.840+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:34:47.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:47 smithi153 ceph-mon[36569]: pgmap v6663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:47 smithi104 ceph-mon[36624]: pgmap v6663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:47 smithi081 ceph-mon[31453]: pgmap v6663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:49.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:49 smithi153 ceph-mon[36569]: pgmap v6664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:49 smithi104 ceph-mon[36624]: pgmap v6664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:49 smithi081 ceph-mon[31453]: pgmap v6664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:51.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:51 smithi153 ceph-mon[36569]: pgmap v6665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:51 smithi104 ceph-mon[36624]: pgmap v6665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:51.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:51 smithi081 ceph-mon[31453]: pgmap v6665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:53.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:53 smithi153 ceph-mon[36569]: pgmap v6666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:53.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:53 smithi104 ceph-mon[36624]: pgmap v6666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:53 smithi081 ceph-mon[31453]: pgmap v6666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:55.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:55 smithi153 ceph-mon[36569]: pgmap v6667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:55 smithi104 ceph-mon[36624]: pgmap v6667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:55 smithi081 ceph-mon[31453]: pgmap v6667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:57 smithi081 ceph-mon[31453]: pgmap v6668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:57.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:34:57.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:34:57.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:34:57.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:57 smithi153 ceph-mon[36569]: pgmap v6668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:57.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:34:57.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:34:57.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:34:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:57 smithi104 ceph-mon[36624]: pgmap v6668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:34:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:34:57.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:34:58.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:34:58.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:34:58.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:58 smithi153 ceph-mon[36569]: pgmap v6669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:58.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:58 smithi153 ceph-mon[36569]: pgmap v6670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:58.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:34:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:34:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:34:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:34:58.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:58 smithi104 ceph-mon[36624]: pgmap v6669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:58.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:58 smithi104 ceph-mon[36624]: pgmap v6670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:58.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:34:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:34:58.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:34:58.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:34:58.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:58 smithi081 ceph-mon[31453]: pgmap v6669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:58.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:58 smithi081 ceph-mon[31453]: pgmap v6670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:34:58.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:34:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:35:01.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:00 smithi153 ceph-mon[36569]: pgmap v6671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:00 smithi104 ceph-mon[36624]: pgmap v6671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:00 smithi081 ceph-mon[31453]: pgmap v6671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:01.855+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:35:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:01.855+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:35:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:01.855+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:35:03.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:02 smithi153 ceph-mon[36569]: pgmap v6672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:02 smithi104 ceph-mon[36624]: pgmap v6672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:02 smithi081 ceph-mon[31453]: pgmap v6672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:05.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:04 smithi153 ceph-mon[36569]: pgmap v6673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:04 smithi104 ceph-mon[36624]: pgmap v6673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:05.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:04 smithi081 ceph-mon[31453]: pgmap v6673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:07.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:06 smithi153 ceph-mon[36569]: pgmap v6674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:06 smithi104 ceph-mon[36624]: pgmap v6674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:06 smithi081 ceph-mon[31453]: pgmap v6674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:08 smithi153 ceph-mon[36569]: pgmap v6675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:09.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:08 smithi104 ceph-mon[36624]: pgmap v6675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:08 smithi081 ceph-mon[31453]: pgmap v6675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:11.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:10 smithi153 ceph-mon[36569]: pgmap v6676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:10 smithi104 ceph-mon[36624]: pgmap v6676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:10 smithi081 ceph-mon[31453]: pgmap v6676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:12 smithi153 ceph-mon[36569]: pgmap v6677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:12 smithi104 ceph-mon[36624]: pgmap v6677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:12 smithi081 ceph-mon[31453]: pgmap v6677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:15.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:15 smithi153 ceph-mon[36569]: pgmap v6678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:15 smithi104 ceph-mon[36624]: pgmap v6678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:15.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:15 smithi081 ceph-mon[31453]: pgmap v6678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:16.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:16.871+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:35:16.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:16.871+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:35:16.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:16.871+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:35:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:17 smithi153 ceph-mon[36569]: pgmap v6679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:17 smithi104 ceph-mon[36624]: pgmap v6679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:17.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:17 smithi081 ceph-mon[31453]: pgmap v6679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:19 smithi153 ceph-mon[36569]: pgmap v6680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:19.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:19 smithi104 ceph-mon[36624]: pgmap v6680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:19.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:19 smithi081 ceph-mon[31453]: pgmap v6680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:21 smithi153 ceph-mon[36569]: pgmap v6681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:21.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:21 smithi104 ceph-mon[36624]: pgmap v6681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:21.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:21 smithi081 ceph-mon[31453]: pgmap v6681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:23 smithi153 ceph-mon[36569]: pgmap v6682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:23 smithi104 ceph-mon[36624]: pgmap v6682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:23.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:23 smithi081 ceph-mon[31453]: pgmap v6682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:24.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:24 smithi153 ceph-mon[36569]: pgmap v6683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:24.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:24 smithi104 ceph-mon[36624]: pgmap v6683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:24.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:24 smithi081 ceph-mon[31453]: pgmap v6683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:26 smithi153 ceph-mon[36569]: pgmap v6684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:27.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:26 smithi104 ceph-mon[36624]: pgmap v6684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:27.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:26 smithi081 ceph-mon[31453]: pgmap v6684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:28 smithi153 ceph-mon[36569]: pgmap v6685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:28 smithi104 ceph-mon[36624]: pgmap v6685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:29.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:28 smithi081 ceph-mon[31453]: pgmap v6685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:31.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:30 smithi153 ceph-mon[36569]: pgmap v6686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:30 smithi104 ceph-mon[36624]: pgmap v6686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:31.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:30 smithi081 ceph-mon[31453]: pgmap v6686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:31.871+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:35:31.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:31.871+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:35:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:31.871+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:35:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:32 smithi153 ceph-mon[36569]: pgmap v6687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:33.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:32 smithi104 ceph-mon[36624]: pgmap v6687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:33.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:32 smithi081 ceph-mon[31453]: pgmap v6687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:35.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:35 smithi153 ceph-mon[36569]: pgmap v6688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:35.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:35 smithi104 ceph-mon[36624]: pgmap v6688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:35 smithi081 ceph-mon[31453]: pgmap v6688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:37.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:37 smithi153 ceph-mon[36569]: pgmap v6689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:37 smithi104 ceph-mon[36624]: pgmap v6689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:37.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:37 smithi081 ceph-mon[31453]: pgmap v6689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:39.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:39 smithi153 ceph-mon[36569]: pgmap v6690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:39.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:39 smithi104 ceph-mon[36624]: pgmap v6690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:39 smithi081 ceph-mon[31453]: pgmap v6690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:41 smithi153 ceph-mon[36569]: pgmap v6691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:41 smithi104 ceph-mon[36624]: pgmap v6691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:41.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:41 smithi081 ceph-mon[31453]: pgmap v6691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:43.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:43 smithi153 ceph-mon[36569]: pgmap v6692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:43.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:43 smithi104 ceph-mon[36624]: pgmap v6692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:43 smithi081 ceph-mon[31453]: pgmap v6692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:44.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:44 smithi081 ceph-mon[31453]: pgmap v6693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:44.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:44 smithi104 ceph-mon[36624]: pgmap v6693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:44.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:44 smithi153 ceph-mon[36569]: pgmap v6693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:46.887+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:35:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:46.887+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:35:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:35:46.887+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:35:47.263 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:46 smithi153 ceph-mon[36569]: pgmap v6694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:47.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:46 smithi104 ceph-mon[36624]: pgmap v6694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:47.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:46 smithi081 ceph-mon[31453]: pgmap v6694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:49.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:48 smithi153 ceph-mon[36569]: pgmap v6695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:49.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:48 smithi104 ceph-mon[36624]: pgmap v6695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:49.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:48 smithi081 ceph-mon[31453]: pgmap v6695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:51.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:51 smithi153 ceph-mon[36569]: pgmap v6696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:51.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:51 smithi104 ceph-mon[36624]: pgmap v6696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:51.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:51 smithi081 ceph-mon[31453]: pgmap v6696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:53 smithi153 ceph-mon[36569]: pgmap v6697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:53.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:53 smithi104 ceph-mon[36624]: pgmap v6697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:53.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:53 smithi081 ceph-mon[31453]: pgmap v6697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:55.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:55 smithi153 ceph-mon[36569]: pgmap v6698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:55.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:55 smithi104 ceph-mon[36624]: pgmap v6698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:55.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:55 smithi081 ceph-mon[31453]: pgmap v6698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:57 smithi153 ceph-mon[36569]: pgmap v6699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:57.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:57 smithi104 ceph-mon[36624]: pgmap v6699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:57.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:57 smithi081 ceph-mon[31453]: pgmap v6699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:59.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:59 smithi153 ceph-mon[36569]: pgmap v6700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:59.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:35:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:35:59.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:59 smithi104 ceph-mon[36624]: pgmap v6700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:59.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:35:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:35:59.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:59 smithi081 ceph-mon[31453]: pgmap v6700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:35:59.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:35:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:36:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:36:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:36:00.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:36:00.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:36:00.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:36:00.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:36:00.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:36:00.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:36:00.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:36:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:01 smithi153 ceph-mon[36569]: pgmap v6701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:01.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:01 smithi153 ceph-mon[36569]: pgmap v6702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:01 smithi104 ceph-mon[36624]: pgmap v6701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:01 smithi104 ceph-mon[36624]: pgmap v6702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:01.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:01 smithi081 ceph-mon[31453]: pgmap v6701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:01.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:01 smithi081 ceph-mon[31453]: pgmap v6702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:01.902+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:36:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:01.902+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:36:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:01.902+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:36:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:02 smithi153 ceph-mon[36569]: pgmap v6703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:02.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:02 smithi104 ceph-mon[36624]: pgmap v6703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:02.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:02 smithi081 ceph-mon[31453]: pgmap v6703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:04.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:04 smithi153 ceph-mon[36569]: pgmap v6704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:04.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:04 smithi104 ceph-mon[36624]: pgmap v6704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:04.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:04 smithi081 ceph-mon[31453]: pgmap v6704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:06.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:06 smithi153 ceph-mon[36569]: pgmap v6705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:06.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:06 smithi104 ceph-mon[36624]: pgmap v6705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:06.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:06 smithi081 ceph-mon[31453]: pgmap v6705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:08.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:08 smithi153 ceph-mon[36569]: pgmap v6706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:08.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:08 smithi104 ceph-mon[36624]: pgmap v6706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:08.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:08 smithi081 ceph-mon[31453]: pgmap v6706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:10.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:10 smithi153 ceph-mon[36569]: pgmap v6707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:10.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:10 smithi104 ceph-mon[36624]: pgmap v6707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:10.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:10 smithi081 ceph-mon[31453]: pgmap v6707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:12.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:12 smithi153 ceph-mon[36569]: pgmap v6708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:12.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:12 smithi104 ceph-mon[36624]: pgmap v6708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:12.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:12 smithi081 ceph-mon[31453]: pgmap v6708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:14.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:14 smithi153 ceph-mon[36569]: pgmap v6709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:14.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:14 smithi104 ceph-mon[36624]: pgmap v6709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:14.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:14 smithi081 ceph-mon[31453]: pgmap v6709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:16.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:16 smithi153 ceph-mon[36569]: pgmap v6710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:16.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:16 smithi104 ceph-mon[36624]: pgmap v6710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:16.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:16 smithi081 ceph-mon[31453]: pgmap v6710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:16.918+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:36:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:16.918+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:36:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:16.918+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:36:18.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:18 smithi153 ceph-mon[36569]: pgmap v6711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:18.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:18 smithi104 ceph-mon[36624]: pgmap v6711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:18.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:18 smithi081 ceph-mon[31453]: pgmap v6711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:20.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:20 smithi153 ceph-mon[36569]: pgmap v6712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:20.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:20 smithi104 ceph-mon[36624]: pgmap v6712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:20.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:20 smithi081 ceph-mon[31453]: pgmap v6712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:23.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:22 smithi153 ceph-mon[36569]: pgmap v6713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:23.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:22 smithi104 ceph-mon[36624]: pgmap v6713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:23.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:22 smithi081 ceph-mon[31453]: pgmap v6713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:25.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:24 smithi153 ceph-mon[36569]: pgmap v6714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:24 smithi104 ceph-mon[36624]: pgmap v6714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:24 smithi081 ceph-mon[31453]: pgmap v6714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:27.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:26 smithi153 ceph-mon[36569]: pgmap v6715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:27.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:26 smithi104 ceph-mon[36624]: pgmap v6715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:27.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:26 smithi081 ceph-mon[31453]: pgmap v6715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:29.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:28 smithi153 ceph-mon[36569]: pgmap v6716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:29.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:28 smithi104 ceph-mon[36624]: pgmap v6716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:29.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:28 smithi081 ceph-mon[31453]: pgmap v6716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:31.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:30 smithi153 ceph-mon[36569]: pgmap v6717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:31.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:30 smithi104 ceph-mon[36624]: pgmap v6717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:31.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:30 smithi081 ceph-mon[31453]: pgmap v6717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:31.927+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:36:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:31.927+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:36:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:31.927+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:36:33.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:32 smithi153 ceph-mon[36569]: pgmap v6718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:33.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:32 smithi104 ceph-mon[36624]: pgmap v6718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:33.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:32 smithi081 ceph-mon[31453]: pgmap v6718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:34 smithi153 ceph-mon[36569]: pgmap v6719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:35.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:34 smithi104 ceph-mon[36624]: pgmap v6719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:34 smithi081 ceph-mon[31453]: pgmap v6719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:37.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:36 smithi153 ceph-mon[36569]: pgmap v6720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:37.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:36 smithi104 ceph-mon[36624]: pgmap v6720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:37.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:36 smithi081 ceph-mon[31453]: pgmap v6720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:39.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:38 smithi153 ceph-mon[36569]: pgmap v6721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:39.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:38 smithi104 ceph-mon[36624]: pgmap v6721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:39.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:38 smithi081 ceph-mon[31453]: pgmap v6721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:41.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:40 smithi153 ceph-mon[36569]: pgmap v6722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:41.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:40 smithi104 ceph-mon[36624]: pgmap v6722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:41.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:40 smithi081 ceph-mon[31453]: pgmap v6722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:43.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:42 smithi153 ceph-mon[36569]: pgmap v6723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:43.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:42 smithi104 ceph-mon[36624]: pgmap v6723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:43.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:42 smithi081 ceph-mon[31453]: pgmap v6723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:45.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:44 smithi153 ceph-mon[36569]: pgmap v6724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:45.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:44 smithi104 ceph-mon[36624]: pgmap v6724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:45.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:44 smithi081 ceph-mon[31453]: pgmap v6724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:46.935+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:36:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:46.935+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:36:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:36:46.935+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:36:47.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:46 smithi153 ceph-mon[36569]: pgmap v6725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:47.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:46 smithi104 ceph-mon[36624]: pgmap v6725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:47.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:46 smithi081 ceph-mon[31453]: pgmap v6725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:49.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:48 smithi153 ceph-mon[36569]: pgmap v6726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:49.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:48 smithi104 ceph-mon[36624]: pgmap v6726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:49.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:48 smithi081 ceph-mon[31453]: pgmap v6726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:51.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:50 smithi153 ceph-mon[36569]: pgmap v6727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:51.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:50 smithi104 ceph-mon[36624]: pgmap v6727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:51.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:50 smithi081 ceph-mon[31453]: pgmap v6727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:53.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:52 smithi153 ceph-mon[36569]: pgmap v6728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:53.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:52 smithi104 ceph-mon[36624]: pgmap v6728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:53.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:52 smithi081 ceph-mon[31453]: pgmap v6728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:55.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:54 smithi153 ceph-mon[36569]: pgmap v6729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:55.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:54 smithi104 ceph-mon[36624]: pgmap v6729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:55.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:54 smithi081 ceph-mon[31453]: pgmap v6729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:56 smithi104 ceph-mon[36624]: pgmap v6730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:56 smithi081 ceph-mon[31453]: pgmap v6730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:57.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:56 smithi153 ceph-mon[36569]: pgmap v6730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:59.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:58 smithi104 ceph-mon[36624]: pgmap v6731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:59.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:58 smithi081 ceph-mon[31453]: pgmap v6731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:36:59.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:58 smithi153 ceph-mon[36569]: pgmap v6731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:00.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:36:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:37:00.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:36:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:37:00.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:36:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:37:01.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:00 smithi104 ceph-mon[36624]: pgmap v6732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:01.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:00 smithi081 ceph-mon[31453]: pgmap v6732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:01.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:00 smithi153 ceph-mon[36569]: pgmap v6732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:01.941+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:37:01.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:01.941+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:37:01.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:01.941+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:37:03.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:02 smithi153 ceph-mon[36569]: pgmap v6733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:03.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:03.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:37:03.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:37:03.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:02 smithi153 ceph-mon[36569]: pgmap v6734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:02 smithi153 ceph-mon[36569]: pgmap v6735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:03.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:02 smithi104 ceph-mon[36624]: pgmap v6733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:03.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:03.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:37:03.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:37:03.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:02 smithi104 ceph-mon[36624]: pgmap v6734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:02 smithi104 ceph-mon[36624]: pgmap v6735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:02 smithi081 ceph-mon[31453]: pgmap v6733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:37:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:37:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:02 smithi081 ceph-mon[31453]: pgmap v6734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:02 smithi081 ceph-mon[31453]: pgmap v6735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:37:05.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:04 smithi081 ceph-mon[31453]: pgmap v6736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:05.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:04 smithi153 ceph-mon[36569]: pgmap v6736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:05.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:04 smithi104 ceph-mon[36624]: pgmap v6736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:07.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:06 smithi081 ceph-mon[31453]: pgmap v6737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:07.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:06 smithi153 ceph-mon[36569]: pgmap v6737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:07.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:06 smithi104 ceph-mon[36624]: pgmap v6737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:09.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:08 smithi081 ceph-mon[31453]: pgmap v6738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:09.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:08 smithi153 ceph-mon[36569]: pgmap v6738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:09.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:08 smithi104 ceph-mon[36624]: pgmap v6738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:11.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:10 smithi081 ceph-mon[31453]: pgmap v6739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:11.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:10 smithi153 ceph-mon[36569]: pgmap v6739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:11.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:10 smithi104 ceph-mon[36624]: pgmap v6739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:13.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:12 smithi081 ceph-mon[31453]: pgmap v6740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:13.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:12 smithi153 ceph-mon[36569]: pgmap v6740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:13.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:12 smithi104 ceph-mon[36624]: pgmap v6740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:14.691+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:37:14.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:14.691+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:37:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:14.691+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:37:14.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:14.692+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:37:15.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:14 smithi081 ceph-mon[31453]: pgmap v6741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:15.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:14 smithi153 ceph-mon[36569]: pgmap v6741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:15.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:14 smithi104 ceph-mon[36624]: pgmap v6741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:16.956+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:37:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:16.956+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:37:16.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:16.956+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:37:17.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:16 smithi081 ceph-mon[31453]: pgmap v6742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:17.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:16 smithi153 ceph-mon[36569]: pgmap v6742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:17.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:16 smithi104 ceph-mon[36624]: pgmap v6742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:19.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:18 smithi081 ceph-mon[31453]: pgmap v6743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:19.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:18 smithi153 ceph-mon[36569]: pgmap v6743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:19.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:18 smithi104 ceph-mon[36624]: pgmap v6743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:21.138 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:20 smithi104 ceph-mon[36624]: pgmap v6744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:21.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:20 smithi081 ceph-mon[31453]: pgmap v6744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:21.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:20 smithi153 ceph-mon[36569]: pgmap v6744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:23.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:22 smithi081 ceph-mon[31453]: pgmap v6745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:23.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:22 smithi153 ceph-mon[36569]: pgmap v6745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:22 smithi104 ceph-mon[36624]: pgmap v6745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:25.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:24 smithi081 ceph-mon[31453]: pgmap v6746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:25.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:24 smithi153 ceph-mon[36569]: pgmap v6746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:25.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:24 smithi104 ceph-mon[36624]: pgmap v6746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:27.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:26 smithi081 ceph-mon[31453]: pgmap v6747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:27.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:26 smithi153 ceph-mon[36569]: pgmap v6747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:27.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:26 smithi104 ceph-mon[36624]: pgmap v6747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:29.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:28 smithi081 ceph-mon[31453]: pgmap v6748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:29.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:28 smithi153 ceph-mon[36569]: pgmap v6748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:28 smithi104 ceph-mon[36624]: pgmap v6748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:31.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:30 smithi081 ceph-mon[31453]: pgmap v6749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:31.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:30 smithi153 ceph-mon[36569]: pgmap v6749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:31.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:30 smithi104 ceph-mon[36624]: pgmap v6749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:31.964+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:37:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:31.964+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:37:31.968 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:31.964+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:37:33.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:32 smithi153 ceph-mon[36569]: pgmap v6750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:33.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:32 smithi104 ceph-mon[36624]: pgmap v6750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:33.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:32 smithi081 ceph-mon[31453]: pgmap v6750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:35.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:34 smithi153 ceph-mon[36569]: pgmap v6751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:35.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:34 smithi104 ceph-mon[36624]: pgmap v6751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:34 smithi081 ceph-mon[31453]: pgmap v6751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:37.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:36 smithi153 ceph-mon[36569]: pgmap v6752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:37.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:36 smithi104 ceph-mon[36624]: pgmap v6752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:37.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:36 smithi081 ceph-mon[31453]: pgmap v6752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:39.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:38 smithi153 ceph-mon[36569]: pgmap v6753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:39.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:38 smithi104 ceph-mon[36624]: pgmap v6753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:38 smithi081 ceph-mon[31453]: pgmap v6753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:41.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:40 smithi153 ceph-mon[36569]: pgmap v6754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:41.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:40 smithi104 ceph-mon[36624]: pgmap v6754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:41.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:40 smithi081 ceph-mon[31453]: pgmap v6754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:43.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:42 smithi153 ceph-mon[36569]: pgmap v6755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:43.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:42 smithi104 ceph-mon[36624]: pgmap v6755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:43.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:42 smithi081 ceph-mon[31453]: pgmap v6755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:44 smithi153 ceph-mon[36569]: pgmap v6756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:45.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:44 smithi104 ceph-mon[36624]: pgmap v6756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:45.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:44 smithi081 ceph-mon[31453]: pgmap v6756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:46.983 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:46.979+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:37:46.983 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:46.979+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:37:46.983 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:37:46.979+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:37:47.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:46 smithi153 ceph-mon[36569]: pgmap v6757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:47.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:46 smithi104 ceph-mon[36624]: pgmap v6757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:47.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:46 smithi081 ceph-mon[31453]: pgmap v6757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:49.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:48 smithi153 ceph-mon[36569]: pgmap v6758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:49.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:48 smithi104 ceph-mon[36624]: pgmap v6758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:49.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:48 smithi081 ceph-mon[31453]: pgmap v6758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:51.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:50 smithi153 ceph-mon[36569]: pgmap v6759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:51.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:50 smithi104 ceph-mon[36624]: pgmap v6759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:51.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:50 smithi081 ceph-mon[31453]: pgmap v6759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:53.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:52 smithi153 ceph-mon[36569]: pgmap v6760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:53.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:52 smithi104 ceph-mon[36624]: pgmap v6760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:53.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:52 smithi081 ceph-mon[31453]: pgmap v6760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:55.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:54 smithi153 ceph-mon[36569]: pgmap v6761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:55.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:54 smithi104 ceph-mon[36624]: pgmap v6761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:55.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:54 smithi081 ceph-mon[31453]: pgmap v6761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:57.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:56 smithi153 ceph-mon[36569]: pgmap v6762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:57.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:56 smithi104 ceph-mon[36624]: pgmap v6762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:57.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:56 smithi081 ceph-mon[31453]: pgmap v6762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:59.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:37:58 smithi153 ceph-mon[36569]: pgmap v6763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:59.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:37:58 smithi104 ceph-mon[36624]: pgmap v6763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:37:59.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:37:58 smithi081 ceph-mon[31453]: pgmap v6763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:01.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:00 smithi153 ceph-mon[36569]: pgmap v6764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:01.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:00 smithi104 ceph-mon[36624]: pgmap v6764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:01.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:00 smithi081 ceph-mon[31453]: pgmap v6764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:01.995+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:38:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:01.995+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:38:01.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:01.995+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:38:03.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:02 smithi153 ceph-mon[36569]: pgmap v6765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:03.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:38:03.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:02 smithi104 ceph-mon[36624]: pgmap v6765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:03.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:38:03.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:02 smithi081 ceph-mon[31453]: pgmap v6765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:03.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:38:04.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:38:04.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:38:04.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: pgmap v6766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:04.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: pgmap v6767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:04.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:38:04.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:38:04.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: pgmap v6766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:04.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: pgmap v6767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:04.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: pgmap v6766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: pgmap v6767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:04.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:38:06.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:06 smithi153 ceph-mon[36569]: pgmap v6768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:06.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:06 smithi104 ceph-mon[36624]: pgmap v6768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:06.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:06 smithi081 ceph-mon[31453]: pgmap v6768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:08.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:08 smithi153 ceph-mon[36569]: pgmap v6769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:08.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:08 smithi104 ceph-mon[36624]: pgmap v6769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:08.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:08 smithi081 ceph-mon[31453]: pgmap v6769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:10.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:10 smithi153 ceph-mon[36569]: pgmap v6770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:10.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:10 smithi104 ceph-mon[36624]: pgmap v6770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:10.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:10 smithi081 ceph-mon[31453]: pgmap v6770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:12.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:12 smithi153 ceph-mon[36569]: pgmap v6771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:12.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:12 smithi104 ceph-mon[36624]: pgmap v6771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:12.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:12 smithi081 ceph-mon[31453]: pgmap v6771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:14.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:14 smithi153 ceph-mon[36569]: pgmap v6772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:14.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:14 smithi104 ceph-mon[36624]: pgmap v6772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:14.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:14 smithi081 ceph-mon[31453]: pgmap v6772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:16.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:16 smithi153 ceph-mon[36569]: pgmap v6773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:16.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:16 smithi104 ceph-mon[36624]: pgmap v6773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:16.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:16 smithi081 ceph-mon[31453]: pgmap v6773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:17.003+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:38:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:17.003+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:38:17.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:17.003+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:38:18.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:18 smithi153 ceph-mon[36569]: pgmap v6774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:18.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:18 smithi104 ceph-mon[36624]: pgmap v6774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:18.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:18 smithi081 ceph-mon[31453]: pgmap v6774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:20.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:20 smithi153 ceph-mon[36569]: pgmap v6775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:20.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:20 smithi104 ceph-mon[36624]: pgmap v6775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:20.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:20 smithi081 ceph-mon[31453]: pgmap v6775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:22.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:22 smithi153 ceph-mon[36569]: pgmap v6776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:22.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:22 smithi104 ceph-mon[36624]: pgmap v6776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:22.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:22 smithi081 ceph-mon[31453]: pgmap v6776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:24.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:24 smithi153 ceph-mon[36569]: pgmap v6777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:24.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:24 smithi104 ceph-mon[36624]: pgmap v6777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:24.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:24 smithi081 ceph-mon[31453]: pgmap v6777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:26.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:26 smithi153 ceph-mon[36569]: pgmap v6778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:26.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:26 smithi104 ceph-mon[36624]: pgmap v6778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:26.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:26 smithi081 ceph-mon[31453]: pgmap v6778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:28.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:28 smithi153 ceph-mon[36569]: pgmap v6779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:28.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:28 smithi104 ceph-mon[36624]: pgmap v6779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:28.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:28 smithi081 ceph-mon[31453]: pgmap v6779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:30.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:30 smithi153 ceph-mon[36569]: pgmap v6780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:30.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:30 smithi104 ceph-mon[36624]: pgmap v6780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:30.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:30 smithi081 ceph-mon[31453]: pgmap v6780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:32.018+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:38:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:32.018+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:38:32.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:32.018+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:38:32.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:32 smithi153 ceph-mon[36569]: pgmap v6781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:32.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:32 smithi104 ceph-mon[36624]: pgmap v6781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:32.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:32 smithi081 ceph-mon[31453]: pgmap v6781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:34.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:34 smithi153 ceph-mon[36569]: pgmap v6782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:34.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:34 smithi104 ceph-mon[36624]: pgmap v6782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:34.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:34 smithi081 ceph-mon[31453]: pgmap v6782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:36.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:36 smithi153 ceph-mon[36569]: pgmap v6783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:36.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:36 smithi104 ceph-mon[36624]: pgmap v6783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:36.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:36 smithi081 ceph-mon[31453]: pgmap v6783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:38.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:38 smithi153 ceph-mon[36569]: pgmap v6784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:38.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:38 smithi104 ceph-mon[36624]: pgmap v6784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:38.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:38 smithi081 ceph-mon[31453]: pgmap v6784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:40.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:40 smithi153 ceph-mon[36569]: pgmap v6785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:40.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:40 smithi104 ceph-mon[36624]: pgmap v6785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:40.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:40 smithi081 ceph-mon[31453]: pgmap v6785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:42.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:42 smithi104 ceph-mon[36624]: pgmap v6786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:42.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:42 smithi081 ceph-mon[31453]: pgmap v6786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:43.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:42 smithi153 ceph-mon[36569]: pgmap v6786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:44.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:44 smithi104 ceph-mon[36624]: pgmap v6787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:44.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:44 smithi081 ceph-mon[31453]: pgmap v6787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:45.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:44 smithi153 ceph-mon[36569]: pgmap v6787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:46.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:46 smithi104 ceph-mon[36624]: pgmap v6788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:46.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:46 smithi081 ceph-mon[31453]: pgmap v6788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:47.021+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:38:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:47.021+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:38:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:38:47.021+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:38:47.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:46 smithi153 ceph-mon[36569]: pgmap v6788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:48.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:48 smithi104 ceph-mon[36624]: pgmap v6789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:48.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:48 smithi081 ceph-mon[31453]: pgmap v6789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:49.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:48 smithi153 ceph-mon[36569]: pgmap v6789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:50.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:50 smithi104 ceph-mon[36624]: pgmap v6790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:50.862 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:50 smithi153 ceph-mon[36569]: pgmap v6790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:50.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:50 smithi081 ceph-mon[31453]: pgmap v6790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:52.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:52 smithi104 ceph-mon[36624]: pgmap v6791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:52.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:52 smithi081 ceph-mon[31453]: pgmap v6791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:53.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:52 smithi153 ceph-mon[36569]: pgmap v6791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:54.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:54 smithi081 ceph-mon[31453]: pgmap v6792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:55.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:54 smithi153 ceph-mon[36569]: pgmap v6792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:55.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:54 smithi104 ceph-mon[36624]: pgmap v6792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:56.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:56 smithi081 ceph-mon[31453]: pgmap v6793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:57.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:56 smithi153 ceph-mon[36569]: pgmap v6793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:57.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:56 smithi104 ceph-mon[36624]: pgmap v6793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:58.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:38:58 smithi081 ceph-mon[31453]: pgmap v6794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:59.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:38:58 smithi153 ceph-mon[36569]: pgmap v6794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:38:59.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:38:58 smithi104 ceph-mon[36624]: pgmap v6794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:00.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:00 smithi081 ceph-mon[31453]: pgmap v6795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:01.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:00 smithi153 ceph-mon[36569]: pgmap v6795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:01.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:00 smithi104 ceph-mon[36624]: pgmap v6795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:02.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:02.024+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:39:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:02.024+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:39:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:02.024+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:39:02.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:02 smithi081 ceph-mon[31453]: pgmap v6796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:03.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:02 smithi153 ceph-mon[36569]: pgmap v6796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:03.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:02 smithi104 ceph-mon[36624]: pgmap v6796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:04.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:04 smithi081 ceph-mon[31453]: pgmap v6797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:04.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:39:05.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:04 smithi153 ceph-mon[36569]: pgmap v6797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:05.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:39:05.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:04 smithi104 ceph-mon[36624]: pgmap v6797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:05.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:39:05.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:39:05.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:39:05.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:39:06.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:39:06.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:39:06.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:39:06.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:39:06.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:39:06.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:39:06.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:06 smithi081 ceph-mon[31453]: pgmap v6798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:06.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:06 smithi081 ceph-mon[31453]: pgmap v6799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:07.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:06 smithi153 ceph-mon[36569]: pgmap v6798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:07.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:06 smithi153 ceph-mon[36569]: pgmap v6799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:07.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:06 smithi104 ceph-mon[36624]: pgmap v6798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:07.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:06 smithi104 ceph-mon[36624]: pgmap v6799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:08.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:08 smithi081 ceph-mon[31453]: pgmap v6800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:09.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:08 smithi153 ceph-mon[36569]: pgmap v6800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:09.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:08 smithi104 ceph-mon[36624]: pgmap v6800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:10.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:10 smithi081 ceph-mon[31453]: pgmap v6801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:11.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:10 smithi153 ceph-mon[36569]: pgmap v6801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:11.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:10 smithi104 ceph-mon[36624]: pgmap v6801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:12.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:12 smithi081 ceph-mon[31453]: pgmap v6802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:13.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:12 smithi153 ceph-mon[36569]: pgmap v6802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:13.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:12 smithi104 ceph-mon[36624]: pgmap v6802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:14.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:14 smithi081 ceph-mon[31453]: pgmap v6803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:15.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:14 smithi153 ceph-mon[36569]: pgmap v6803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:15.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:14 smithi104 ceph-mon[36624]: pgmap v6803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:16.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:16 smithi081 ceph-mon[31453]: pgmap v6804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:17.035+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:39:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:17.035+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:39:17.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:17.035+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:39:17.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:16 smithi153 ceph-mon[36569]: pgmap v6804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:17.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:16 smithi104 ceph-mon[36624]: pgmap v6804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:19.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:18 smithi153 ceph-mon[36569]: pgmap v6805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:19.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:18 smithi104 ceph-mon[36624]: pgmap v6805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:19.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:18 smithi081 ceph-mon[31453]: pgmap v6805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:21.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:20 smithi153 ceph-mon[36569]: pgmap v6806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:21.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:20 smithi104 ceph-mon[36624]: pgmap v6806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:21.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:20 smithi081 ceph-mon[31453]: pgmap v6806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:23.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:22 smithi153 ceph-mon[36569]: pgmap v6807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:23.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:22 smithi104 ceph-mon[36624]: pgmap v6807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:23.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:22 smithi081 ceph-mon[31453]: pgmap v6807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:25.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:24 smithi153 ceph-mon[36569]: pgmap v6808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:25.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:24 smithi104 ceph-mon[36624]: pgmap v6808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:25.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:24 smithi081 ceph-mon[31453]: pgmap v6808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:27.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:26 smithi153 ceph-mon[36569]: pgmap v6809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:27.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:26 smithi104 ceph-mon[36624]: pgmap v6809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:27.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:26 smithi081 ceph-mon[31453]: pgmap v6809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:29.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:28 smithi153 ceph-mon[36569]: pgmap v6810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:29.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:28 smithi104 ceph-mon[36624]: pgmap v6810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:29.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:28 smithi081 ceph-mon[31453]: pgmap v6810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:30.963 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:30 smithi081 ceph-mon[31453]: pgmap v6811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:31.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:30 smithi153 ceph-mon[36569]: pgmap v6811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:31.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:30 smithi104 ceph-mon[36624]: pgmap v6811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:32.043+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:39:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:32.043+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:39:32.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:32.043+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:39:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:32 smithi153 ceph-mon[36569]: pgmap v6812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:33.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:32 smithi104 ceph-mon[36624]: pgmap v6812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:33.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:32 smithi081 ceph-mon[31453]: pgmap v6812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:35.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:34 smithi153 ceph-mon[36569]: pgmap v6813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:35.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:34 smithi104 ceph-mon[36624]: pgmap v6813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:35.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:34 smithi081 ceph-mon[31453]: pgmap v6813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:37.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:36 smithi153 ceph-mon[36569]: pgmap v6814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:37.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:36 smithi104 ceph-mon[36624]: pgmap v6814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:37.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:36 smithi081 ceph-mon[31453]: pgmap v6814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:39.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:38 smithi153 ceph-mon[36569]: pgmap v6815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:39.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:38 smithi104 ceph-mon[36624]: pgmap v6815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:39.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:38 smithi081 ceph-mon[31453]: pgmap v6815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:41.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:40 smithi153 ceph-mon[36569]: pgmap v6816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:41.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:40 smithi104 ceph-mon[36624]: pgmap v6816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:41.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:40 smithi081 ceph-mon[31453]: pgmap v6816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:43.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:42 smithi153 ceph-mon[36569]: pgmap v6817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:43.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:42 smithi104 ceph-mon[36624]: pgmap v6817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:43.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:42 smithi081 ceph-mon[31453]: pgmap v6817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:45.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:44 smithi153 ceph-mon[36569]: pgmap v6818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:45.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:44 smithi104 ceph-mon[36624]: pgmap v6818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:45.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:44 smithi081 ceph-mon[31453]: pgmap v6818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:47.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:46 smithi153 ceph-mon[36569]: pgmap v6819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:47.059+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:39:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:47.059+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:39:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:39:47.059+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:39:47.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:46 smithi104 ceph-mon[36624]: pgmap v6819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:47.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:46 smithi081 ceph-mon[31453]: pgmap v6819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:49.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:48 smithi153 ceph-mon[36569]: pgmap v6820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:49.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:48 smithi104 ceph-mon[36624]: pgmap v6820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:49.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:48 smithi081 ceph-mon[31453]: pgmap v6820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:51.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:50 smithi153 ceph-mon[36569]: pgmap v6821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:51.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:50 smithi104 ceph-mon[36624]: pgmap v6821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:51.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:50 smithi081 ceph-mon[31453]: pgmap v6821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:53.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:52 smithi153 ceph-mon[36569]: pgmap v6822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:53.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:52 smithi104 ceph-mon[36624]: pgmap v6822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:53.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:52 smithi081 ceph-mon[31453]: pgmap v6822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:55.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:54 smithi153 ceph-mon[36569]: pgmap v6823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:55.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:54 smithi104 ceph-mon[36624]: pgmap v6823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:55.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:54 smithi081 ceph-mon[31453]: pgmap v6823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:57.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:56 smithi153 ceph-mon[36569]: pgmap v6824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:57.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:56 smithi104 ceph-mon[36624]: pgmap v6824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:57.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:56 smithi081 ceph-mon[31453]: pgmap v6824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:59.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:39:58 smithi153 ceph-mon[36569]: pgmap v6825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:59.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:39:58 smithi104 ceph-mon[36624]: pgmap v6825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:39:59.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:39:58 smithi081 ceph-mon[31453]: pgmap v6825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:00 smithi153 ceph-mon[36569]: pgmap v6826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:40:01.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:00 smithi104 ceph-mon[36624]: pgmap v6826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:01.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:40:01.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:00 smithi081 ceph-mon[31453]: pgmap v6826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:01.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:40:02.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:02.074+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:40:02.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:02.074+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:40:02.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:02.074+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:40:03.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:02 smithi153 ceph-mon[36569]: pgmap v6827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:03.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:02 smithi104 ceph-mon[36624]: pgmap v6827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:02 smithi081 ceph-mon[31453]: pgmap v6827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:05.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:04 smithi153 ceph-mon[36569]: pgmap v6828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:05.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:04 smithi104 ceph-mon[36624]: pgmap v6828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:05.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:04 smithi081 ceph-mon[31453]: pgmap v6828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:06.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:40:06.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:40:06.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:40:07.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: pgmap v6829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:07.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:40:07.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:40:07.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:40:07.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: pgmap v6829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:07.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:40:07.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:40:07.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:40:07.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: pgmap v6829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:07.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:40:07.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:40:07.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:40:07.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:40:08.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:07 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:40:08.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:07 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:40:08.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:07 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:40:08.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:07 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:40:08.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:07 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:40:08.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:07 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:40:08.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:07 smithi153 ceph-mon[36569]: pgmap v6830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:08.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:07 smithi153 ceph-mon[36569]: pgmap v6831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:08.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:07 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:40:08.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:07 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:40:08.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:07 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:40:08.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:07 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:40:08.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:07 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:40:08.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:07 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:40:08.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:07 smithi104 ceph-mon[36624]: pgmap v6830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:08.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:07 smithi104 ceph-mon[36624]: pgmap v6831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:07 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:40:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:07 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:40:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:07 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:40:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:07 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:40:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:07 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:40:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:07 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:40:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:07 smithi081 ceph-mon[31453]: pgmap v6830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:07 smithi081 ceph-mon[31453]: pgmap v6831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:10.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:09 smithi153 ceph-mon[36569]: pgmap v6832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:10.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:09 smithi104 ceph-mon[36624]: pgmap v6832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:10.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:09 smithi081 ceph-mon[31453]: pgmap v6832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:12.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:11 smithi104 ceph-mon[36624]: pgmap v6833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:12.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:11 smithi081 ceph-mon[31453]: pgmap v6833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:12.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:11 smithi153 ceph-mon[36569]: pgmap v6833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:14.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:13 smithi104 ceph-mon[36624]: pgmap v6834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:14.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:13 smithi081 ceph-mon[31453]: pgmap v6834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:14.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:13 smithi153 ceph-mon[36569]: pgmap v6834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:14.720+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:40:14.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:14.720+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:40:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:14.721+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:40:14.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:14.721+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:40:16.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:15 smithi104 ceph-mon[36624]: pgmap v6835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:16.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:15 smithi081 ceph-mon[31453]: pgmap v6835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:16.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:15 smithi153 ceph-mon[36569]: pgmap v6835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:17.090+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:40:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:17.090+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:40:17.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:17.090+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:40:18.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:17 smithi104 ceph-mon[36624]: pgmap v6836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:18.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:17 smithi081 ceph-mon[31453]: pgmap v6836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:18.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:17 smithi153 ceph-mon[36569]: pgmap v6836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:20.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:19 smithi104 ceph-mon[36624]: pgmap v6837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:20.115 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:19 smithi153 ceph-mon[36569]: pgmap v6837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:20.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:19 smithi081 ceph-mon[31453]: pgmap v6837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:22.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:21 smithi081 ceph-mon[31453]: pgmap v6838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:22.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:21 smithi153 ceph-mon[36569]: pgmap v6838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:22.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:21 smithi104 ceph-mon[36624]: pgmap v6838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:24.139 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:23 smithi104 ceph-mon[36624]: pgmap v6839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:24.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:23 smithi081 ceph-mon[31453]: pgmap v6839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:24.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:23 smithi153 ceph-mon[36569]: pgmap v6839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:26.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:25 smithi081 ceph-mon[31453]: pgmap v6840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:26.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:25 smithi153 ceph-mon[36569]: pgmap v6840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:26.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:25 smithi104 ceph-mon[36624]: pgmap v6840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:28.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:27 smithi081 ceph-mon[31453]: pgmap v6841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:28.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:27 smithi153 ceph-mon[36569]: pgmap v6841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:28.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:27 smithi104 ceph-mon[36624]: pgmap v6841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:30.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:29 smithi081 ceph-mon[31453]: pgmap v6842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:30.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:29 smithi153 ceph-mon[36569]: pgmap v6842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:30.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:29 smithi104 ceph-mon[36624]: pgmap v6842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:32.102+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:40:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:32.102+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:40:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:32.102+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:40:32.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:31 smithi081 ceph-mon[31453]: pgmap v6843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:32.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:31 smithi153 ceph-mon[36569]: pgmap v6843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:32.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:31 smithi104 ceph-mon[36624]: pgmap v6843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:34.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:33 smithi081 ceph-mon[31453]: pgmap v6844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:33 smithi153 ceph-mon[36569]: pgmap v6844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:34.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:33 smithi104 ceph-mon[36624]: pgmap v6844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:36.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:35 smithi081 ceph-mon[31453]: pgmap v6845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:36.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:35 smithi153 ceph-mon[36569]: pgmap v6845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:36.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:35 smithi104 ceph-mon[36624]: pgmap v6845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:38.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:37 smithi081 ceph-mon[31453]: pgmap v6846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:38.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:37 smithi153 ceph-mon[36569]: pgmap v6846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:38.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:37 smithi104 ceph-mon[36624]: pgmap v6846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:40.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:39 smithi081 ceph-mon[31453]: pgmap v6847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:40.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:39 smithi153 ceph-mon[36569]: pgmap v6847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:40.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:39 smithi104 ceph-mon[36624]: pgmap v6847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:42.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:41 smithi081 ceph-mon[31453]: pgmap v6848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:42.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:41 smithi153 ceph-mon[36569]: pgmap v6848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:42.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:41 smithi104 ceph-mon[36624]: pgmap v6848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:44.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:43 smithi081 ceph-mon[31453]: pgmap v6849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:44.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:43 smithi153 ceph-mon[36569]: pgmap v6849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:44.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:43 smithi104 ceph-mon[36624]: pgmap v6849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:46.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:45 smithi081 ceph-mon[31453]: pgmap v6850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:46.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:45 smithi153 ceph-mon[36569]: pgmap v6850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:46.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:45 smithi104 ceph-mon[36624]: pgmap v6850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:47.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:47.117+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:40:47.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:47.117+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:40:47.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:40:47.117+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:40:48.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:47 smithi081 ceph-mon[31453]: pgmap v6851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:48.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:47 smithi153 ceph-mon[36569]: pgmap v6851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:48.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:47 smithi104 ceph-mon[36624]: pgmap v6851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:50.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:49 smithi081 ceph-mon[31453]: pgmap v6852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:50.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:49 smithi153 ceph-mon[36569]: pgmap v6852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:50.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:49 smithi104 ceph-mon[36624]: pgmap v6852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:52.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:51 smithi153 ceph-mon[36569]: pgmap v6853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:52.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:51 smithi104 ceph-mon[36624]: pgmap v6853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:52.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:51 smithi081 ceph-mon[31453]: pgmap v6853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:54.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:53 smithi153 ceph-mon[36569]: pgmap v6854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:54.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:53 smithi104 ceph-mon[36624]: pgmap v6854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:54.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:53 smithi081 ceph-mon[31453]: pgmap v6854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:56.204 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:55 smithi081 ceph-mon[31453]: pgmap v6855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:56.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:55 smithi153 ceph-mon[36569]: pgmap v6855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:56.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:55 smithi104 ceph-mon[36624]: pgmap v6855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:58.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:57 smithi153 ceph-mon[36569]: pgmap v6856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:58.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:57 smithi104 ceph-mon[36624]: pgmap v6856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:40:58.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:57 smithi081 ceph-mon[31453]: pgmap v6856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:00.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:40:59 smithi153 ceph-mon[36569]: pgmap v6857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:00.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:40:59 smithi104 ceph-mon[36624]: pgmap v6857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:00.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:40:59 smithi081 ceph-mon[31453]: pgmap v6857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:02.130+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:41:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:02.130+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:41:02.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:02.130+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:41:02.215 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:01 smithi081 ceph-mon[31453]: pgmap v6858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:02.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:01 smithi153 ceph-mon[36569]: pgmap v6858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:02.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:01 smithi104 ceph-mon[36624]: pgmap v6858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:04.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:03 smithi153 ceph-mon[36569]: pgmap v6859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:04.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:03 smithi104 ceph-mon[36624]: pgmap v6859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:04.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:03 smithi081 ceph-mon[31453]: pgmap v6859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:06.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:05 smithi153 ceph-mon[36569]: pgmap v6860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:06.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:05 smithi104 ceph-mon[36624]: pgmap v6860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:06.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:05 smithi081 ceph-mon[31453]: pgmap v6860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:08.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:07 smithi153 ceph-mon[36569]: pgmap v6861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:08.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:41:08.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:07 smithi104 ceph-mon[36624]: pgmap v6861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:08.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:41:08.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:07 smithi081 ceph-mon[31453]: pgmap v6861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:08.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:41:09.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:41:09.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:41:09.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:41:09.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:41:09.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:41:09.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:41:09.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:41:09.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:41:09.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:41:10.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:09 smithi153 ceph-mon[36569]: pgmap v6862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:10.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:09 smithi153 ceph-mon[36569]: pgmap v6863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:10.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:09 smithi104 ceph-mon[36624]: pgmap v6862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:10.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:09 smithi104 ceph-mon[36624]: pgmap v6863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:10.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:09 smithi081 ceph-mon[31453]: pgmap v6862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:10.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:09 smithi081 ceph-mon[31453]: pgmap v6863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:12.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:11 smithi153 ceph-mon[36569]: pgmap v6864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:12.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:11 smithi104 ceph-mon[36624]: pgmap v6864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:12.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:11 smithi081 ceph-mon[31453]: pgmap v6864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:14.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:13 smithi153 ceph-mon[36569]: pgmap v6865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:14.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:13 smithi104 ceph-mon[36624]: pgmap v6865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:14.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:13 smithi081 ceph-mon[31453]: pgmap v6865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:16.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:15 smithi153 ceph-mon[36569]: pgmap v6866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:16.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:15 smithi104 ceph-mon[36624]: pgmap v6866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:16.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:15 smithi081 ceph-mon[31453]: pgmap v6866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:17.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:17.138+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:41:17.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:17.138+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:41:17.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:17.138+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:41:18.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:17 smithi153 ceph-mon[36569]: pgmap v6867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:18.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:17 smithi104 ceph-mon[36624]: pgmap v6867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:18.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:17 smithi081 ceph-mon[31453]: pgmap v6867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:20.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:19 smithi153 ceph-mon[36569]: pgmap v6868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:20.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:19 smithi104 ceph-mon[36624]: pgmap v6868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:20.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:19 smithi081 ceph-mon[31453]: pgmap v6868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:22.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:21 smithi153 ceph-mon[36569]: pgmap v6869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:22.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:21 smithi104 ceph-mon[36624]: pgmap v6869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:22.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:21 smithi081 ceph-mon[31453]: pgmap v6869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:24.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:24 smithi153 ceph-mon[36569]: pgmap v6870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:24.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:24 smithi104 ceph-mon[36624]: pgmap v6870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:24.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:24 smithi081 ceph-mon[31453]: pgmap v6870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:26.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:26 smithi153 ceph-mon[36569]: pgmap v6871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:26.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:26 smithi104 ceph-mon[36624]: pgmap v6871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:26.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:26 smithi081 ceph-mon[31453]: pgmap v6871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:28.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:28 smithi153 ceph-mon[36569]: pgmap v6872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:28.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:28 smithi104 ceph-mon[36624]: pgmap v6872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:28.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:28 smithi081 ceph-mon[31453]: pgmap v6872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:30.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:30 smithi153 ceph-mon[36569]: pgmap v6873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:30.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:30 smithi104 ceph-mon[36624]: pgmap v6873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:30.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:30 smithi081 ceph-mon[31453]: pgmap v6873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:32.149+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:41:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:32.149+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:41:32.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:32.149+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:41:32.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:32 smithi153 ceph-mon[36569]: pgmap v6874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:32.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:32 smithi104 ceph-mon[36624]: pgmap v6874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:32.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:32 smithi081 ceph-mon[31453]: pgmap v6874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:34 smithi153 ceph-mon[36569]: pgmap v6875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:34 smithi104 ceph-mon[36624]: pgmap v6875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:34.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:34 smithi081 ceph-mon[31453]: pgmap v6875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:36.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:36 smithi153 ceph-mon[36569]: pgmap v6876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:36.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:36 smithi104 ceph-mon[36624]: pgmap v6876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:36.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:36 smithi081 ceph-mon[31453]: pgmap v6876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:38.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:38 smithi153 ceph-mon[36569]: pgmap v6877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:38.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:38 smithi104 ceph-mon[36624]: pgmap v6877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:38.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:38 smithi081 ceph-mon[31453]: pgmap v6877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:40.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:40 smithi153 ceph-mon[36569]: pgmap v6878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:40.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:40 smithi104 ceph-mon[36624]: pgmap v6878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:40.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:40 smithi081 ceph-mon[31453]: pgmap v6878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:42.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:42 smithi153 ceph-mon[36569]: pgmap v6879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:42.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:42 smithi104 ceph-mon[36624]: pgmap v6879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:42.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:42 smithi081 ceph-mon[31453]: pgmap v6879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:44.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:44 smithi153 ceph-mon[36569]: pgmap v6880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:44.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:44 smithi104 ceph-mon[36624]: pgmap v6880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:44.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:44 smithi081 ceph-mon[31453]: pgmap v6880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:46.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:46 smithi104 ceph-mon[36624]: pgmap v6881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:46.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:46 smithi081 ceph-mon[31453]: pgmap v6881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:46.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:46 smithi153 ceph-mon[36569]: pgmap v6881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:47.164+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:41:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:47.164+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:41:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:41:47.164+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:41:48.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:48 smithi104 ceph-mon[36624]: pgmap v6882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:48.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:48 smithi081 ceph-mon[31453]: pgmap v6882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:48.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:48 smithi153 ceph-mon[36569]: pgmap v6882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:50.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:50 smithi104 ceph-mon[36624]: pgmap v6883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:50.364 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:50 smithi153 ceph-mon[36569]: pgmap v6883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:50.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:50 smithi081 ceph-mon[31453]: pgmap v6883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:52.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:52 smithi104 ceph-mon[36624]: pgmap v6884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:52.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:52 smithi081 ceph-mon[31453]: pgmap v6884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:52.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:52 smithi153 ceph-mon[36569]: pgmap v6884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:54.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:54 smithi104 ceph-mon[36624]: pgmap v6885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:54.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:54 smithi081 ceph-mon[31453]: pgmap v6885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:54.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:54 smithi153 ceph-mon[36569]: pgmap v6885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:56.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:56 smithi081 ceph-mon[31453]: pgmap v6886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:56.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:56 smithi153 ceph-mon[36569]: pgmap v6886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:56.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:56 smithi104 ceph-mon[36624]: pgmap v6886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:58.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:41:58 smithi081 ceph-mon[31453]: pgmap v6887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:58.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:41:58 smithi153 ceph-mon[36569]: pgmap v6887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:41:58.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:41:58 smithi104 ceph-mon[36624]: pgmap v6887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:00.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:00 smithi081 ceph-mon[31453]: pgmap v6888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:00.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:00 smithi153 ceph-mon[36569]: pgmap v6888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:00.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:00 smithi104 ceph-mon[36624]: pgmap v6888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:02.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:02.175+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:42:02.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:02.175+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:42:02.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:02.175+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:42:02.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:02 smithi081 ceph-mon[31453]: pgmap v6889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:02.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:02 smithi153 ceph-mon[36569]: pgmap v6889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:02.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:02 smithi104 ceph-mon[36624]: pgmap v6889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:04.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:04 smithi081 ceph-mon[31453]: pgmap v6890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:04.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:04 smithi153 ceph-mon[36569]: pgmap v6890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:04.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:04 smithi104 ceph-mon[36624]: pgmap v6890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:05.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:05 smithi153 ceph-mon[36569]: pgmap v6891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:05.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:05 smithi104 ceph-mon[36624]: pgmap v6891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:05.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:05 smithi081 ceph-mon[31453]: pgmap v6891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:07 smithi081 ceph-mon[31453]: pgmap v6892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:08.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:07 smithi153 ceph-mon[36569]: pgmap v6892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:08.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:07 smithi104 ceph-mon[36624]: pgmap v6892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:09.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:42:09.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:42:09.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:42:10.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:09 smithi081 ceph-mon[31453]: pgmap v6893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:10.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:42:10.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:42:10.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:42:10.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:09 smithi153 ceph-mon[36569]: pgmap v6893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:10.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:42:10.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:42:10.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:42:10.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:09 smithi104 ceph-mon[36624]: pgmap v6893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:10.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:42:10.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:42:10.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:42:11.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:10 smithi081 ceph-mon[31453]: pgmap v6894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:11.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:10 smithi081 ceph-mon[31453]: pgmap v6895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:11.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:10 smithi153 ceph-mon[36569]: pgmap v6894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:11.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:10 smithi153 ceph-mon[36569]: pgmap v6895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:11.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:10 smithi104 ceph-mon[36624]: pgmap v6894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:11.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:10 smithi104 ceph-mon[36624]: pgmap v6895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:13.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:12 smithi081 ceph-mon[31453]: pgmap v6896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:13.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:12 smithi153 ceph-mon[36569]: pgmap v6896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:13.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:12 smithi104 ceph-mon[36624]: pgmap v6896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:15.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:14 smithi081 ceph-mon[31453]: pgmap v6897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:15.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:14 smithi153 ceph-mon[36569]: pgmap v6897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:15.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:14 smithi104 ceph-mon[36624]: pgmap v6897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:17.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:16 smithi081 ceph-mon[31453]: pgmap v6898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:17.180+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:42:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:17.180+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:42:17.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:17.180+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:42:17.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:16 smithi153 ceph-mon[36569]: pgmap v6898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:17.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:16 smithi104 ceph-mon[36624]: pgmap v6898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:19.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:18 smithi081 ceph-mon[31453]: pgmap v6899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:19.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:18 smithi153 ceph-mon[36569]: pgmap v6899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:19.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:18 smithi104 ceph-mon[36624]: pgmap v6899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:21.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:20 smithi081 ceph-mon[31453]: pgmap v6900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:21.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:20 smithi153 ceph-mon[36569]: pgmap v6900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:21.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:20 smithi104 ceph-mon[36624]: pgmap v6900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:23.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:22 smithi081 ceph-mon[31453]: pgmap v6901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:23.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:22 smithi153 ceph-mon[36569]: pgmap v6901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:23.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:22 smithi104 ceph-mon[36624]: pgmap v6901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:25.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:24 smithi153 ceph-mon[36569]: pgmap v6902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:25.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:24 smithi104 ceph-mon[36624]: pgmap v6902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:25.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:24 smithi081 ceph-mon[31453]: pgmap v6902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:27.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:26 smithi153 ceph-mon[36569]: pgmap v6903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:27.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:26 smithi104 ceph-mon[36624]: pgmap v6903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:27.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:26 smithi081 ceph-mon[31453]: pgmap v6903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:29.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:28 smithi081 ceph-mon[31453]: pgmap v6904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:29.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:28 smithi153 ceph-mon[36569]: pgmap v6904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:29.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:28 smithi104 ceph-mon[36624]: pgmap v6904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:29.742+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:42:29.748 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:29.742+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:42:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:29.743+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:42:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:29.743+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:42:31.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:30 smithi153 ceph-mon[36569]: pgmap v6905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:31.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:30 smithi104 ceph-mon[36624]: pgmap v6905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:31.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:30 smithi081 ceph-mon[31453]: pgmap v6905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:32.195+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:42:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:32.195+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:42:32.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:32.195+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:42:33.215 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:32 smithi081 ceph-mon[31453]: pgmap v6906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:33.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:32 smithi153 ceph-mon[36569]: pgmap v6906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:33.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:32 smithi104 ceph-mon[36624]: pgmap v6906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:35.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:34 smithi153 ceph-mon[36569]: pgmap v6907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:35.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:34 smithi104 ceph-mon[36624]: pgmap v6907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:35.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:34 smithi081 ceph-mon[31453]: pgmap v6907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:37.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:36 smithi153 ceph-mon[36569]: pgmap v6908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:37.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:36 smithi104 ceph-mon[36624]: pgmap v6908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:37.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:36 smithi081 ceph-mon[31453]: pgmap v6908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:39.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:38 smithi153 ceph-mon[36569]: pgmap v6909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:39.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:38 smithi104 ceph-mon[36624]: pgmap v6909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:39.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:38 smithi081 ceph-mon[31453]: pgmap v6909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:41.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:40 smithi153 ceph-mon[36569]: pgmap v6910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:41.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:40 smithi104 ceph-mon[36624]: pgmap v6910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:41.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:40 smithi081 ceph-mon[31453]: pgmap v6910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:43.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:42 smithi153 ceph-mon[36569]: pgmap v6911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:43.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:42 smithi104 ceph-mon[36624]: pgmap v6911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:43.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:42 smithi081 ceph-mon[31453]: pgmap v6911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:45.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:44 smithi153 ceph-mon[36569]: pgmap v6912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:45.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:44 smithi104 ceph-mon[36624]: pgmap v6912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:45.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:44 smithi081 ceph-mon[31453]: pgmap v6912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:47.211+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:42:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:47.211+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:42:47.218 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:42:47.211+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:42:47.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:46 smithi153 ceph-mon[36569]: pgmap v6913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:47.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:46 smithi104 ceph-mon[36624]: pgmap v6913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:47.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:46 smithi081 ceph-mon[31453]: pgmap v6913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:49.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:48 smithi153 ceph-mon[36569]: pgmap v6914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:49.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:48 smithi104 ceph-mon[36624]: pgmap v6914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:49.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:48 smithi081 ceph-mon[31453]: pgmap v6914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:51.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:50 smithi153 ceph-mon[36569]: pgmap v6915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:51.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:50 smithi104 ceph-mon[36624]: pgmap v6915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:51.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:50 smithi081 ceph-mon[31453]: pgmap v6915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:53.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:52 smithi153 ceph-mon[36569]: pgmap v6916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:53.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:52 smithi104 ceph-mon[36624]: pgmap v6916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:53.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:52 smithi081 ceph-mon[31453]: pgmap v6916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:55.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:54 smithi153 ceph-mon[36569]: pgmap v6917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:55.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:54 smithi104 ceph-mon[36624]: pgmap v6917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:55.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:54 smithi081 ceph-mon[31453]: pgmap v6917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:57.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:56 smithi153 ceph-mon[36569]: pgmap v6918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:57.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:56 smithi104 ceph-mon[36624]: pgmap v6918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:57.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:56 smithi081 ceph-mon[31453]: pgmap v6918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:59.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:42:59 smithi153 ceph-mon[36569]: pgmap v6919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:59.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:42:59 smithi104 ceph-mon[36624]: pgmap v6919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:42:59.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:42:59 smithi081 ceph-mon[31453]: pgmap v6919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:01.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:01 smithi153 ceph-mon[36569]: pgmap v6920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:01.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:01 smithi104 ceph-mon[36624]: pgmap v6920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:01.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:01 smithi081 ceph-mon[31453]: pgmap v6920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:02.220+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:43:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:02.220+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:43:02.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:02.220+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:43:03.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:03 smithi153 ceph-mon[36569]: pgmap v6921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:03.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:03 smithi104 ceph-mon[36624]: pgmap v6921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:03.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:03 smithi081 ceph-mon[31453]: pgmap v6921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:05.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:05 smithi153 ceph-mon[36569]: pgmap v6922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:05.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:05 smithi104 ceph-mon[36624]: pgmap v6922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:05.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:05 smithi081 ceph-mon[31453]: pgmap v6922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:07.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:07 smithi153 ceph-mon[36569]: pgmap v6923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:07.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:07 smithi104 ceph-mon[36624]: pgmap v6923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:07.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:07 smithi081 ceph-mon[31453]: pgmap v6923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:09.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:09 smithi153 ceph-mon[36569]: pgmap v6924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:09.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:09 smithi104 ceph-mon[36624]: pgmap v6924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:09.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:09 smithi081 ceph-mon[31453]: pgmap v6924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:11.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:11 smithi153 ceph-mon[36569]: pgmap v6925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:11.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:43:11.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:11 smithi104 ceph-mon[36624]: pgmap v6925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:11.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:43:11.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:11 smithi081 ceph-mon[31453]: pgmap v6925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:11.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:43:13.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:13 smithi153 ceph-mon[36569]: pgmap v6926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:13.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:13 smithi104 ceph-mon[36624]: pgmap v6926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:13.380 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:13 smithi081 ceph-mon[31453]: pgmap v6926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:14.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:14.750+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:43:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:14.750+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:43:14.757 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:14.752+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:43:14.758 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:14.752+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:43:15.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:15 smithi153 ceph-mon[36569]: pgmap v6927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:15.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:15.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:15.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:15 smithi104 ceph-mon[36624]: pgmap v6927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:15.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:15.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:15.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:15 smithi081 ceph-mon[31453]: pgmap v6927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:15.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:15.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:17.232+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:17.232+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:43:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:17.232+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:43:17.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:17 smithi153 ceph-mon[36569]: pgmap v6928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:17.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:17 smithi104 ceph-mon[36624]: pgmap v6928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:17.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:17 smithi081 ceph-mon[31453]: pgmap v6928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:18.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:18.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:18.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:43:18.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:43:18.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:18 smithi153 ceph-mon[36569]: pgmap v6929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:18.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:18 smithi153 ceph-mon[36569]: pgmap v6930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:18.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:18.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:18.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:18.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:43:18.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:43:18.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:18 smithi104 ceph-mon[36624]: pgmap v6929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:18.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:18 smithi104 ceph-mon[36624]: pgmap v6930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:18.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:18.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:18.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:18.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:43:18.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:43:18.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:18 smithi081 ceph-mon[31453]: pgmap v6929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:18.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:18 smithi081 ceph-mon[31453]: pgmap v6930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:18.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:43:20.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:20 smithi081 ceph-mon[31453]: pgmap v6931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:21.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:20 smithi153 ceph-mon[36569]: pgmap v6931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:21.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:20 smithi104 ceph-mon[36624]: pgmap v6931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:22.864 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:22 smithi153 ceph-mon[36569]: pgmap v6932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:22.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:22 smithi081 ceph-mon[31453]: pgmap v6932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:23.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:22 smithi104 ceph-mon[36624]: pgmap v6932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:24.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:24 smithi081 ceph-mon[31453]: pgmap v6933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:25.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:24 smithi153 ceph-mon[36569]: pgmap v6933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:25.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:24 smithi104 ceph-mon[36624]: pgmap v6933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:26.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:26 smithi081 ceph-mon[31453]: pgmap v6934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:27.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:26 smithi153 ceph-mon[36569]: pgmap v6934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:27.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:26 smithi104 ceph-mon[36624]: pgmap v6934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:28.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:28 smithi081 ceph-mon[31453]: pgmap v6935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:29.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:28 smithi153 ceph-mon[36569]: pgmap v6935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:29.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:28 smithi104 ceph-mon[36624]: pgmap v6935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:30.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:30 smithi081 ceph-mon[31453]: pgmap v6936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:31.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:30 smithi153 ceph-mon[36569]: pgmap v6936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:31.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:30 smithi104 ceph-mon[36624]: pgmap v6936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:32.248+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:43:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:32.248+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:43:32.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:32.248+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:43:32.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:32 smithi081 ceph-mon[31453]: pgmap v6937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:33.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:32 smithi153 ceph-mon[36569]: pgmap v6937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:33.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:32 smithi104 ceph-mon[36624]: pgmap v6937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:34.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:34 smithi081 ceph-mon[31453]: pgmap v6938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:35.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:34 smithi153 ceph-mon[36569]: pgmap v6938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:35.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:34 smithi104 ceph-mon[36624]: pgmap v6938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:36.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:36 smithi081 ceph-mon[31453]: pgmap v6939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:37.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:36 smithi153 ceph-mon[36569]: pgmap v6939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:37.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:36 smithi104 ceph-mon[36624]: pgmap v6939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:38.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:38 smithi081 ceph-mon[31453]: pgmap v6940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:39.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:38 smithi153 ceph-mon[36569]: pgmap v6940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:39.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:38 smithi104 ceph-mon[36624]: pgmap v6940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:40.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:40 smithi081 ceph-mon[31453]: pgmap v6941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:41.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:40 smithi153 ceph-mon[36569]: pgmap v6941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:41.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:40 smithi104 ceph-mon[36624]: pgmap v6941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:42.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:42 smithi081 ceph-mon[31453]: pgmap v6942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:43.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:42 smithi153 ceph-mon[36569]: pgmap v6942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:43.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:42 smithi104 ceph-mon[36624]: pgmap v6942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:44.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:44 smithi081 ceph-mon[31453]: pgmap v6943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:45.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:44 smithi153 ceph-mon[36569]: pgmap v6943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:45.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:44 smithi104 ceph-mon[36624]: pgmap v6943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:47.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:46 smithi153 ceph-mon[36569]: pgmap v6944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:47.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:46 smithi104 ceph-mon[36624]: pgmap v6944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:47.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:46 smithi081 ceph-mon[31453]: pgmap v6944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:47.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:47.264+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:43:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:47.264+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:43:47.270 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:43:47.264+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:43:49.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:48 smithi153 ceph-mon[36569]: pgmap v6945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:49.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:48 smithi104 ceph-mon[36624]: pgmap v6945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:49.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:48 smithi081 ceph-mon[31453]: pgmap v6945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:51.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:50 smithi153 ceph-mon[36569]: pgmap v6946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:51.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:50 smithi104 ceph-mon[36624]: pgmap v6946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:51.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:50 smithi081 ceph-mon[31453]: pgmap v6946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:53.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:52 smithi153 ceph-mon[36569]: pgmap v6947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:53.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:52 smithi104 ceph-mon[36624]: pgmap v6947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:53.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:52 smithi081 ceph-mon[31453]: pgmap v6947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:55.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:54 smithi153 ceph-mon[36569]: pgmap v6948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:55.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:54 smithi104 ceph-mon[36624]: pgmap v6948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:55.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:54 smithi081 ceph-mon[31453]: pgmap v6948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:57.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:56 smithi153 ceph-mon[36569]: pgmap v6949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:57.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:56 smithi104 ceph-mon[36624]: pgmap v6949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:57.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:56 smithi081 ceph-mon[31453]: pgmap v6949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:59.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:43:58 smithi153 ceph-mon[36569]: pgmap v6950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:59.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:43:58 smithi104 ceph-mon[36624]: pgmap v6950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:43:59.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:43:58 smithi081 ceph-mon[31453]: pgmap v6950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:01.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:00 smithi153 ceph-mon[36569]: pgmap v6951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:01.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:00 smithi104 ceph-mon[36624]: pgmap v6951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:01.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:00 smithi081 ceph-mon[31453]: pgmap v6951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:02.280+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:44:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:02.280+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:44:02.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:02.280+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:44:03.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:02 smithi153 ceph-mon[36569]: pgmap v6952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:03.085 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:02 smithi081 ceph-mon[31453]: pgmap v6952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:03.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:02 smithi104 ceph-mon[36624]: pgmap v6952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:05.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:04 smithi153 ceph-mon[36569]: pgmap v6953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:05.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:04 smithi104 ceph-mon[36624]: pgmap v6953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:05.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:04 smithi081 ceph-mon[31453]: pgmap v6953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:07.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:06 smithi153 ceph-mon[36569]: pgmap v6954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:07.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:06 smithi104 ceph-mon[36624]: pgmap v6954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:07.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:06 smithi081 ceph-mon[31453]: pgmap v6954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:09.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:08 smithi153 ceph-mon[36569]: pgmap v6955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:09.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:08 smithi104 ceph-mon[36624]: pgmap v6955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:09.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:08 smithi081 ceph-mon[31453]: pgmap v6955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:11.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:10 smithi153 ceph-mon[36569]: pgmap v6956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:11.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:10 smithi104 ceph-mon[36624]: pgmap v6956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:11.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:10 smithi081 ceph-mon[31453]: pgmap v6956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:13.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:12 smithi153 ceph-mon[36569]: pgmap v6957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:13.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:12 smithi104 ceph-mon[36624]: pgmap v6957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:13.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:12 smithi081 ceph-mon[31453]: pgmap v6957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:15.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:14 smithi153 ceph-mon[36569]: pgmap v6958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:15.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:14 smithi104 ceph-mon[36624]: pgmap v6958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:15.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:14 smithi081 ceph-mon[31453]: pgmap v6958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:17.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:16 smithi153 ceph-mon[36569]: pgmap v6959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:17.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:16 smithi104 ceph-mon[36624]: pgmap v6959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:17.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:16 smithi081 ceph-mon[31453]: pgmap v6959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:17.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:17.289+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:44:17.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:17.289+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:44:17.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:17.289+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:44:19.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:18 smithi153 ceph-mon[36569]: pgmap v6960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:19.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:44:19.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:19.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:19.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:18 smithi104 ceph-mon[36624]: pgmap v6960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:19.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:44:19.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:19.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:19.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:18 smithi081 ceph-mon[31453]: pgmap v6960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:19.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:44:19.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:19.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:20.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:20.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:20.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:20.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:20.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:20.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:20 smithi081 ceph-mon[31453]: pgmap v6961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:21.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:20 smithi153 ceph-mon[36569]: pgmap v6961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:21.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:20 smithi104 ceph-mon[36624]: pgmap v6961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:21.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:21.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:22.965 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:22 smithi153 ceph-mon[36569]: pgmap v6962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:23.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:22 smithi081 ceph-mon[31453]: pgmap v6962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:23.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:22 smithi104 ceph-mon[36624]: pgmap v6962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:24.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:24.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:24.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:24 smithi081 ceph-mon[31453]: pgmap v6963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:24.486 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:24.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:24.487 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:24 smithi153 ceph-mon[36569]: pgmap v6963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:24.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:24.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:24.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:24 smithi104 ceph-mon[36624]: pgmap v6963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:25.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:25.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:25.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:44:25.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:44:25.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:25 smithi104 ceph-mon[36624]: pgmap v6964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:25.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:25 smithi104 ceph-mon[36624]: pgmap v6965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:25.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:25.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:25.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:25.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:44:25.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:44:25.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:25 smithi081 ceph-mon[31453]: pgmap v6964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:25.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:25 smithi081 ceph-mon[31453]: pgmap v6965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:25.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:26.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:26.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:26.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:44:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:44:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:25 smithi153 ceph-mon[36569]: pgmap v6964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:25 smithi153 ceph-mon[36569]: pgmap v6965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:44:27.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:27 smithi081 ceph-mon[31453]: pgmap v6966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:28.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:27 smithi153 ceph-mon[36569]: pgmap v6966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:28.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:27 smithi104 ceph-mon[36624]: pgmap v6966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:29.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:29 smithi081 ceph-mon[31453]: pgmap v6967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:30.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:29 smithi153 ceph-mon[36569]: pgmap v6967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:30.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:29 smithi104 ceph-mon[36624]: pgmap v6967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:32.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:31 smithi153 ceph-mon[36569]: pgmap v6968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:32.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:31 smithi104 ceph-mon[36624]: pgmap v6968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:32.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:31 smithi081 ceph-mon[31453]: pgmap v6968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:32.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:32.306+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:44:32.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:32.306+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:44:32.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:32.306+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:44:34.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:33 smithi153 ceph-mon[36569]: pgmap v6969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:34.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:33 smithi104 ceph-mon[36624]: pgmap v6969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:34.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:33 smithi081 ceph-mon[31453]: pgmap v6969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:36.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:35 smithi153 ceph-mon[36569]: pgmap v6970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:36.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:35 smithi104 ceph-mon[36624]: pgmap v6970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:36.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:35 smithi081 ceph-mon[31453]: pgmap v6970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:38.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:37 smithi153 ceph-mon[36569]: pgmap v6971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:38.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:37 smithi104 ceph-mon[36624]: pgmap v6971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:38.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:37 smithi081 ceph-mon[31453]: pgmap v6971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:40.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:39 smithi153 ceph-mon[36569]: pgmap v6972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:40.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:39 smithi104 ceph-mon[36624]: pgmap v6972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:40.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:39 smithi081 ceph-mon[31453]: pgmap v6972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:42.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:41 smithi153 ceph-mon[36569]: pgmap v6973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:42.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:41 smithi104 ceph-mon[36624]: pgmap v6973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:42.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:41 smithi081 ceph-mon[31453]: pgmap v6973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:44.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:43 smithi153 ceph-mon[36569]: pgmap v6974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:44.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:43 smithi104 ceph-mon[36624]: pgmap v6974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:44.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:43 smithi081 ceph-mon[31453]: pgmap v6974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:46.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:45 smithi153 ceph-mon[36569]: pgmap v6975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:46.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:45 smithi104 ceph-mon[36624]: pgmap v6975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:46.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:45 smithi081 ceph-mon[31453]: pgmap v6975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:47.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:47.321+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:44:47.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:47.322+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:44:47.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:44:47.322+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:44:48.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:47 smithi153 ceph-mon[36569]: pgmap v6976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:48.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:47 smithi104 ceph-mon[36624]: pgmap v6976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:48.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:47 smithi081 ceph-mon[31453]: pgmap v6976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:50.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:49 smithi153 ceph-mon[36569]: pgmap v6977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:50.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:49 smithi104 ceph-mon[36624]: pgmap v6977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:50.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:49 smithi081 ceph-mon[31453]: pgmap v6977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:52.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:51 smithi153 ceph-mon[36569]: pgmap v6978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:52.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:51 smithi104 ceph-mon[36624]: pgmap v6978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:52.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:51 smithi081 ceph-mon[31453]: pgmap v6978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:54.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:53 smithi153 ceph-mon[36569]: pgmap v6979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:54.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:53 smithi104 ceph-mon[36624]: pgmap v6979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:53 smithi081 ceph-mon[31453]: pgmap v6979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:56.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:55 smithi153 ceph-mon[36569]: pgmap v6980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:56.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:55 smithi104 ceph-mon[36624]: pgmap v6980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:56.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:55 smithi081 ceph-mon[31453]: pgmap v6980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:58.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:57 smithi153 ceph-mon[36569]: pgmap v6981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:58.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:57 smithi104 ceph-mon[36624]: pgmap v6981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:44:58.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:57 smithi081 ceph-mon[31453]: pgmap v6981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:00.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:44:59 smithi153 ceph-mon[36569]: pgmap v6982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:00.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:44:59 smithi104 ceph-mon[36624]: pgmap v6982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:00.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:44:59 smithi081 ceph-mon[31453]: pgmap v6982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:02.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:01 smithi153 ceph-mon[36569]: pgmap v6983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:02.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:01 smithi104 ceph-mon[36624]: pgmap v6983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:02.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:01 smithi081 ceph-mon[31453]: pgmap v6983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:02.337+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:45:02.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:02.337+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:45:02.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:02.337+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:45:04.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:03 smithi153 ceph-mon[36569]: pgmap v6984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:04.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:03 smithi104 ceph-mon[36624]: pgmap v6984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:04.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:03 smithi081 ceph-mon[31453]: pgmap v6984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:06.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:05 smithi153 ceph-mon[36569]: pgmap v6985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:06.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:05 smithi104 ceph-mon[36624]: pgmap v6985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:06.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:05 smithi081 ceph-mon[31453]: pgmap v6985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:08.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:07 smithi153 ceph-mon[36569]: pgmap v6986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:08.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:07 smithi104 ceph-mon[36624]: pgmap v6986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:08.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:07 smithi081 ceph-mon[31453]: pgmap v6986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:10.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:09 smithi153 ceph-mon[36569]: pgmap v6987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:10.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:09 smithi104 ceph-mon[36624]: pgmap v6987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:10.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:09 smithi081 ceph-mon[31453]: pgmap v6987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:12.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:11 smithi153 ceph-mon[36569]: pgmap v6988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:12.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:11 smithi104 ceph-mon[36624]: pgmap v6988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:12.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:11 smithi081 ceph-mon[31453]: pgmap v6988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:14.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:13 smithi153 ceph-mon[36569]: pgmap v6989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:14.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:13 smithi104 ceph-mon[36624]: pgmap v6989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:14.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:13 smithi081 ceph-mon[31453]: pgmap v6989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:16.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:15 smithi153 ceph-mon[36569]: pgmap v6990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:16.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:15 smithi104 ceph-mon[36624]: pgmap v6990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:16.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:15 smithi081 ceph-mon[31453]: pgmap v6990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:17.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:17.339+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:45:17.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:17.339+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:45:17.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:17.339+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:45:18.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:17 smithi153 ceph-mon[36569]: pgmap v6991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:18.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:17 smithi104 ceph-mon[36624]: pgmap v6991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:18.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:17 smithi081 ceph-mon[31453]: pgmap v6991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:20.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:19 smithi153 ceph-mon[36569]: pgmap v6992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:20.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:19 smithi104 ceph-mon[36624]: pgmap v6992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:20.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:19 smithi081 ceph-mon[31453]: pgmap v6992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:22.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:21 smithi153 ceph-mon[36569]: pgmap v6993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:22.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:21 smithi104 ceph-mon[36624]: pgmap v6993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:22.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:21 smithi081 ceph-mon[31453]: pgmap v6993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:24.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:23 smithi153 ceph-mon[36569]: pgmap v6994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:24.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:23 smithi104 ceph-mon[36624]: pgmap v6994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:24.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:23 smithi081 ceph-mon[31453]: pgmap v6994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:26.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:25 smithi153 ceph-mon[36569]: pgmap v6995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:45:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:26.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:25 smithi104 ceph-mon[36624]: pgmap v6995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:26.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:45:26.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:26.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:26.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:25 smithi081 ceph-mon[31453]: pgmap v6995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:26.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:45:26.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:26.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:27.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:45:27.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:45:27.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:27.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:45:27.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:45:27.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:27.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:45:27.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:45:27.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:45:28.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:27 smithi104 ceph-mon[36624]: pgmap v6996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:28.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:27 smithi104 ceph-mon[36624]: pgmap v6997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:28.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:27 smithi081 ceph-mon[31453]: pgmap v6996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:28.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:27 smithi081 ceph-mon[31453]: pgmap v6997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:28.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:27 smithi153 ceph-mon[36569]: pgmap v6996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:28.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:27 smithi153 ceph-mon[36569]: pgmap v6997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:30.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:29 smithi104 ceph-mon[36624]: pgmap v6998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:30.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:29 smithi081 ceph-mon[31453]: pgmap v6998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:30.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:29 smithi153 ceph-mon[36569]: pgmap v6998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:32.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:31 smithi104 ceph-mon[36624]: pgmap v6999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:32.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:31 smithi081 ceph-mon[31453]: pgmap v6999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:32.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:31 smithi153 ceph-mon[36569]: pgmap v6999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:32.347+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:45:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:32.347+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:45:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:32.347+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:45:34.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:33 smithi104 ceph-mon[36624]: pgmap v7000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:34.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:33 smithi081 ceph-mon[31453]: pgmap v7000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:34.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:33 smithi153 ceph-mon[36569]: pgmap v7000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:36.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:35 smithi104 ceph-mon[36624]: pgmap v7001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:36.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:35 smithi081 ceph-mon[31453]: pgmap v7001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:36.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:35 smithi153 ceph-mon[36569]: pgmap v7001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:38.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:37 smithi081 ceph-mon[31453]: pgmap v7002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:38.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:37 smithi153 ceph-mon[36569]: pgmap v7002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:38.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:37 smithi104 ceph-mon[36624]: pgmap v7002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:40.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:39 smithi081 ceph-mon[31453]: pgmap v7003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:40.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:39 smithi153 ceph-mon[36569]: pgmap v7003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:40.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:39 smithi104 ceph-mon[36624]: pgmap v7003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:42.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:41 smithi081 ceph-mon[31453]: pgmap v7004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:42.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:41 smithi153 ceph-mon[36569]: pgmap v7004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:42.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:41 smithi104 ceph-mon[36624]: pgmap v7004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:44.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:43 smithi081 ceph-mon[31453]: pgmap v7005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:44.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:43 smithi153 ceph-mon[36569]: pgmap v7005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:44.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:43 smithi104 ceph-mon[36624]: pgmap v7005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:46.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:45 smithi081 ceph-mon[31453]: pgmap v7006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:46.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:45 smithi153 ceph-mon[36569]: pgmap v7006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:46.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:45 smithi104 ceph-mon[36624]: pgmap v7006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:47.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:47.352+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:45:47.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:47.352+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:45:47.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:45:47.352+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:45:48.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:47 smithi081 ceph-mon[31453]: pgmap v7007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:48.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:47 smithi153 ceph-mon[36569]: pgmap v7007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:48.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:47 smithi104 ceph-mon[36624]: pgmap v7007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:50.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:49 smithi081 ceph-mon[31453]: pgmap v7008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:50.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:49 smithi153 ceph-mon[36569]: pgmap v7008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:50.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:49 smithi104 ceph-mon[36624]: pgmap v7008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:52.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:51 smithi081 ceph-mon[31453]: pgmap v7009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:52.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:51 smithi153 ceph-mon[36569]: pgmap v7009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:52.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:51 smithi104 ceph-mon[36624]: pgmap v7009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:53 smithi081 ceph-mon[31453]: pgmap v7010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:54.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:53 smithi153 ceph-mon[36569]: pgmap v7010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:54.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:53 smithi104 ceph-mon[36624]: pgmap v7010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:56.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:55 smithi081 ceph-mon[31453]: pgmap v7011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:56.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:55 smithi153 ceph-mon[36569]: pgmap v7011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:56.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:55 smithi104 ceph-mon[36624]: pgmap v7011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:58.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:57 smithi081 ceph-mon[31453]: pgmap v7012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:58.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:57 smithi153 ceph-mon[36569]: pgmap v7012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:45:58.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:57 smithi104 ceph-mon[36624]: pgmap v7012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:00.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:45:59 smithi081 ceph-mon[31453]: pgmap v7013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:00.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:45:59 smithi153 ceph-mon[36569]: pgmap v7013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:00.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:45:59 smithi104 ceph-mon[36624]: pgmap v7013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:02.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:01 smithi081 ceph-mon[31453]: pgmap v7014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:02.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:01 smithi153 ceph-mon[36569]: pgmap v7014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:02.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:01 smithi104 ceph-mon[36624]: pgmap v7014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:02.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:02.358+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:46:02.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:02.358+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:46:02.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:02.358+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:46:04.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:03 smithi081 ceph-mon[31453]: pgmap v7015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:04.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:03 smithi153 ceph-mon[36569]: pgmap v7015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:04.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:03 smithi104 ceph-mon[36624]: pgmap v7015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:06.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:05 smithi153 ceph-mon[36569]: pgmap v7016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:06.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:05 smithi104 ceph-mon[36624]: pgmap v7016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:06.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:05 smithi081 ceph-mon[31453]: pgmap v7016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:08.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:07 smithi153 ceph-mon[36569]: pgmap v7017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:08.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:07 smithi104 ceph-mon[36624]: pgmap v7017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:08.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:07 smithi081 ceph-mon[31453]: pgmap v7017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:10.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:09 smithi153 ceph-mon[36569]: pgmap v7018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:10.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:09 smithi104 ceph-mon[36624]: pgmap v7018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:10.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:09 smithi081 ceph-mon[31453]: pgmap v7018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:12.122+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000687) for more than 900000000 us, fault. 2024-04-13T00:46:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:12.122+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006550 tx=0x7f16f40ba000 comp rx=0 tx=0).stop 2024-04-13T00:46:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:12.122+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T00:46:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:12.122+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T00:46:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:12.122+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T00:46:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:12.122+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T00:46:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:12.124+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c9f0 tx=0x7f16f4039000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T00:46:12.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:11 smithi153 ceph-mon[36569]: pgmap v7019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:12.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:11 smithi104 ceph-mon[36624]: pgmap v7019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:12.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:11 smithi081 ceph-mon[31453]: pgmap v7019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:14.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:13 smithi153 ceph-mon[36569]: pgmap v7020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:14.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:13 smithi104 ceph-mon[36624]: pgmap v7020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:14.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:13 smithi081 ceph-mon[31453]: pgmap v7020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:14.781+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:46:14.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:14.781+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:46:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:14.782+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:46:14.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:14.782+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:46:16.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:15 smithi153 ceph-mon[36569]: pgmap v7021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:16.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:15 smithi104 ceph-mon[36624]: pgmap v7021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:16.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:15 smithi081 ceph-mon[31453]: pgmap v7021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:17.368+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:46:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:17.368+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:46:17.375 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:17.368+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:46:18.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:17 smithi153 ceph-mon[36569]: pgmap v7022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:18.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:17 smithi104 ceph-mon[36624]: pgmap v7022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:18.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:17 smithi081 ceph-mon[31453]: pgmap v7022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:20.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:19 smithi153 ceph-mon[36569]: pgmap v7023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:20.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:19 smithi104 ceph-mon[36624]: pgmap v7023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:20.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:19 smithi081 ceph-mon[31453]: pgmap v7023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:22.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:21 smithi153 ceph-mon[36569]: pgmap v7024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:22.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:21 smithi104 ceph-mon[36624]: pgmap v7024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:22.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:21 smithi081 ceph-mon[31453]: pgmap v7024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:24.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:23 smithi153 ceph-mon[36569]: pgmap v7025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:24.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:23 smithi104 ceph-mon[36624]: pgmap v7025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:24.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:23 smithi081 ceph-mon[31453]: pgmap v7025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:26.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:25 smithi153 ceph-mon[36569]: pgmap v7026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:26.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:25 smithi104 ceph-mon[36624]: pgmap v7026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:26.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:25 smithi081 ceph-mon[31453]: pgmap v7026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:27.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:46:27.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:46:27.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:46:28.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:27 smithi153 ceph-mon[36569]: pgmap v7027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:28.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:46:28.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:46:28.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:46:28.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:27 smithi104 ceph-mon[36624]: pgmap v7027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:28.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:46:28.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:46:28.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:46:28.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:27 smithi081 ceph-mon[31453]: pgmap v7027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:28.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:46:28.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:46:28.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:46:29.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:28 smithi153 ceph-mon[36569]: pgmap v7028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:29.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:28 smithi153 ceph-mon[36569]: pgmap v7029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:29.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:28 smithi104 ceph-mon[36624]: pgmap v7028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:29.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:28 smithi104 ceph-mon[36624]: pgmap v7029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:29.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:28 smithi081 ceph-mon[31453]: pgmap v7028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:29.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:28 smithi081 ceph-mon[31453]: pgmap v7029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:31.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:30 smithi153 ceph-mon[36569]: pgmap v7030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:31.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:30 smithi104 ceph-mon[36624]: pgmap v7030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:31.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:30 smithi081 ceph-mon[31453]: pgmap v7030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:32.376+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:46:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:32.376+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:46:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:32.376+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:46:33.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:33 smithi153 ceph-mon[36569]: pgmap v7031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:33.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:33 smithi104 ceph-mon[36624]: pgmap v7031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:33.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:33 smithi081 ceph-mon[31453]: pgmap v7031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:35.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:35 smithi153 ceph-mon[36569]: pgmap v7032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:35 smithi104 ceph-mon[36624]: pgmap v7032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:35 smithi081 ceph-mon[31453]: pgmap v7032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:37.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:37 smithi153 ceph-mon[36569]: pgmap v7033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:37.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:37 smithi104 ceph-mon[36624]: pgmap v7033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:37.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:37 smithi081 ceph-mon[31453]: pgmap v7033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:39.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:39 smithi153 ceph-mon[36569]: pgmap v7034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:39.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:39 smithi104 ceph-mon[36624]: pgmap v7034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:39.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:39 smithi081 ceph-mon[31453]: pgmap v7034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:41.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:41 smithi153 ceph-mon[36569]: pgmap v7035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:41.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:41 smithi104 ceph-mon[36624]: pgmap v7035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:41.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:41 smithi081 ceph-mon[31453]: pgmap v7035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:43.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:43 smithi153 ceph-mon[36569]: pgmap v7036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:43.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:43 smithi104 ceph-mon[36624]: pgmap v7036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:43.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:43 smithi081 ceph-mon[31453]: pgmap v7036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:45.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:45 smithi153 ceph-mon[36569]: pgmap v7037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:45.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:45 smithi104 ceph-mon[36624]: pgmap v7037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:45.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:45 smithi081 ceph-mon[31453]: pgmap v7037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:47.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:47 smithi153 ceph-mon[36569]: pgmap v7038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:47.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:47 smithi104 ceph-mon[36624]: pgmap v7038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:47.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:47.392+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:46:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:47.392+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:46:47.399 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:46:47.392+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:46:47.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:47 smithi081 ceph-mon[31453]: pgmap v7038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:49.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:49 smithi153 ceph-mon[36569]: pgmap v7039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:49.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:49 smithi104 ceph-mon[36624]: pgmap v7039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:49.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:49 smithi081 ceph-mon[31453]: pgmap v7039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:51.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:51 smithi153 ceph-mon[36569]: pgmap v7040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:51.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:51 smithi104 ceph-mon[36624]: pgmap v7040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:51.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:51 smithi081 ceph-mon[31453]: pgmap v7040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:53.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:53 smithi153 ceph-mon[36569]: pgmap v7041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:53.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:53 smithi104 ceph-mon[36624]: pgmap v7041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:53.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:53 smithi081 ceph-mon[31453]: pgmap v7041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:55.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:55 smithi104 ceph-mon[36624]: pgmap v7042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:55.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:55 smithi081 ceph-mon[31453]: pgmap v7042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:55.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:55 smithi153 ceph-mon[36569]: pgmap v7042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:57.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:57 smithi104 ceph-mon[36624]: pgmap v7043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:57.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:57 smithi081 ceph-mon[31453]: pgmap v7043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:57.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:57 smithi153 ceph-mon[36569]: pgmap v7043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:59.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:46:59 smithi104 ceph-mon[36624]: pgmap v7044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:59.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:46:59 smithi081 ceph-mon[31453]: pgmap v7044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:46:59.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:46:59 smithi153 ceph-mon[36569]: pgmap v7044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:01.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:01 smithi104 ceph-mon[36624]: pgmap v7045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:01.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:01 smithi081 ceph-mon[31453]: pgmap v7045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:01.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:01 smithi153 ceph-mon[36569]: pgmap v7045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:02.400+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:47:02.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:02.400+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:47:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:02.400+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:47:03.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:03 smithi104 ceph-mon[36624]: pgmap v7046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:03.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:03 smithi081 ceph-mon[31453]: pgmap v7046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:03.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:03 smithi153 ceph-mon[36569]: pgmap v7046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:05.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:05 smithi081 ceph-mon[31453]: pgmap v7047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:05.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:05 smithi153 ceph-mon[36569]: pgmap v7047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:05.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:05 smithi104 ceph-mon[36624]: pgmap v7047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:07.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:07 smithi081 ceph-mon[31453]: pgmap v7048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:07.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:07 smithi153 ceph-mon[36569]: pgmap v7048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:07.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:07 smithi104 ceph-mon[36624]: pgmap v7048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:09.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:09 smithi081 ceph-mon[31453]: pgmap v7049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:09.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:09 smithi153 ceph-mon[36569]: pgmap v7049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:09.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:09 smithi104 ceph-mon[36624]: pgmap v7049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:11.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:11 smithi081 ceph-mon[31453]: pgmap v7050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:11.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:11 smithi153 ceph-mon[36569]: pgmap v7050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:11.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:11 smithi104 ceph-mon[36624]: pgmap v7050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:13.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:13 smithi081 ceph-mon[31453]: pgmap v7051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:13.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:13 smithi153 ceph-mon[36569]: pgmap v7051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:13.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:13 smithi104 ceph-mon[36624]: pgmap v7051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:15.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:15 smithi081 ceph-mon[31453]: pgmap v7052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:15.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:15 smithi153 ceph-mon[36569]: pgmap v7052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:15.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:15 smithi104 ceph-mon[36624]: pgmap v7052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:17.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:17 smithi081 ceph-mon[31453]: pgmap v7053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:17.415+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:47:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:17.415+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:47:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:17.415+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:47:17.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:17 smithi153 ceph-mon[36569]: pgmap v7053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:17.615 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:17 smithi104 ceph-mon[36624]: pgmap v7053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:19.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:19 smithi081 ceph-mon[31453]: pgmap v7054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:19.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:19 smithi153 ceph-mon[36569]: pgmap v7054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:19.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:19 smithi104 ceph-mon[36624]: pgmap v7054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:21.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:21 smithi081 ceph-mon[31453]: pgmap v7055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:21.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:21 smithi153 ceph-mon[36569]: pgmap v7055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:21.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:21 smithi104 ceph-mon[36624]: pgmap v7055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:23.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:23 smithi081 ceph-mon[31453]: pgmap v7056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:23.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:23 smithi153 ceph-mon[36569]: pgmap v7056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:23.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:23 smithi104 ceph-mon[36624]: pgmap v7056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:25.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:25 smithi081 ceph-mon[31453]: pgmap v7057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:25.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:25 smithi153 ceph-mon[36569]: pgmap v7057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:25.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:25 smithi104 ceph-mon[36624]: pgmap v7057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:27.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:27 smithi081 ceph-mon[31453]: pgmap v7058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:27.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:27 smithi153 ceph-mon[36569]: pgmap v7058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:27.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:27 smithi104 ceph-mon[36624]: pgmap v7058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:28.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:47:28.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:47:28.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:47:29.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:29 smithi081 ceph-mon[31453]: pgmap v7059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:29.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:29 smithi153 ceph-mon[36569]: pgmap v7059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:29.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:29 smithi104 ceph-mon[36624]: pgmap v7059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:30.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:47:30.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:47:30.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:47:30.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:47:30.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:47:30.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:47:30.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:47:30.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:47:30.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:47:31.490 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:31 smithi104 ceph-mon[36624]: pgmap v7060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:31.490 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:31 smithi104 ceph-mon[36624]: pgmap v7061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:31.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:31 smithi153 ceph-mon[36569]: pgmap v7060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:31.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:31 smithi153 ceph-mon[36569]: pgmap v7061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:31.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:31 smithi081 ceph-mon[31453]: pgmap v7060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:31.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:31 smithi081 ceph-mon[31453]: pgmap v7061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:32.428+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:47:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:32.428+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:47:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:32.428+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:47:33.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:33 smithi153 ceph-mon[36569]: pgmap v7062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:33.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:33 smithi104 ceph-mon[36624]: pgmap v7062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:33.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:33 smithi081 ceph-mon[31453]: pgmap v7062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:34.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:34 smithi153 ceph-mon[36569]: pgmap v7063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:34.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:34 smithi104 ceph-mon[36624]: pgmap v7063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:34.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:34 smithi081 ceph-mon[31453]: pgmap v7063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:36.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:36 smithi153 ceph-mon[36569]: pgmap v7064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:36.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:36 smithi104 ceph-mon[36624]: pgmap v7064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:36.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:36 smithi081 ceph-mon[31453]: pgmap v7064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:38.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:38 smithi153 ceph-mon[36569]: pgmap v7065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:38.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:38 smithi104 ceph-mon[36624]: pgmap v7065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:38.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:38 smithi081 ceph-mon[31453]: pgmap v7065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:40.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:40 smithi153 ceph-mon[36569]: pgmap v7066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:40.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:40 smithi104 ceph-mon[36624]: pgmap v7066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:40.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:40 smithi081 ceph-mon[31453]: pgmap v7066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:42.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:42 smithi153 ceph-mon[36569]: pgmap v7067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:42.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:42 smithi104 ceph-mon[36624]: pgmap v7067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:42.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:42 smithi081 ceph-mon[31453]: pgmap v7067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:44.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:44 smithi153 ceph-mon[36569]: pgmap v7068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:44.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:44 smithi104 ceph-mon[36624]: pgmap v7068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:44.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:44 smithi081 ceph-mon[31453]: pgmap v7068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:46.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:46 smithi153 ceph-mon[36569]: pgmap v7069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:46.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:46 smithi104 ceph-mon[36624]: pgmap v7069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:46.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:46 smithi081 ceph-mon[31453]: pgmap v7069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:47.430+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:47:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:47.430+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:47:47.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:47:47.430+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:47:48.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:48 smithi153 ceph-mon[36569]: pgmap v7070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:48.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:48 smithi104 ceph-mon[36624]: pgmap v7070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:48.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:48 smithi081 ceph-mon[31453]: pgmap v7070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:50.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:50 smithi153 ceph-mon[36569]: pgmap v7071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:50.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:50 smithi104 ceph-mon[36624]: pgmap v7071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:50.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:50 smithi081 ceph-mon[31453]: pgmap v7071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:52.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:52 smithi153 ceph-mon[36569]: pgmap v7072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:52.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:52 smithi104 ceph-mon[36624]: pgmap v7072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:52.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:52 smithi081 ceph-mon[31453]: pgmap v7072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:54.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:54 smithi153 ceph-mon[36569]: pgmap v7073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:54.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:54 smithi104 ceph-mon[36624]: pgmap v7073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:54.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:54 smithi081 ceph-mon[31453]: pgmap v7073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:56.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:56 smithi153 ceph-mon[36569]: pgmap v7074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:56.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:56 smithi104 ceph-mon[36624]: pgmap v7074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:56.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:56 smithi081 ceph-mon[31453]: pgmap v7074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:47:58 smithi153 ceph-mon[36569]: pgmap v7075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:47:58 smithi104 ceph-mon[36624]: pgmap v7075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:47:58.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:47:58 smithi081 ceph-mon[31453]: pgmap v7075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:00.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:00 smithi153 ceph-mon[36569]: pgmap v7076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:00.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:00 smithi104 ceph-mon[36624]: pgmap v7076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:00.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:00 smithi081 ceph-mon[31453]: pgmap v7076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:02.447+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:48:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:02.447+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:48:02.454 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:02.447+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:48:02.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:02 smithi153 ceph-mon[36569]: pgmap v7077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:02.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:02 smithi104 ceph-mon[36624]: pgmap v7077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:02.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:02 smithi081 ceph-mon[31453]: pgmap v7077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:04.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:04 smithi153 ceph-mon[36569]: pgmap v7078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:04.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:04 smithi104 ceph-mon[36624]: pgmap v7078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:04.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:04 smithi081 ceph-mon[31453]: pgmap v7078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:06.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:06 smithi153 ceph-mon[36569]: pgmap v7079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:06.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:06 smithi104 ceph-mon[36624]: pgmap v7079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:06.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:06 smithi081 ceph-mon[31453]: pgmap v7079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:08.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:08 smithi104 ceph-mon[36624]: pgmap v7080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:08.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:08 smithi081 ceph-mon[31453]: pgmap v7080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:09.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:08 smithi153 ceph-mon[36569]: pgmap v7080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:10.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:10 smithi104 ceph-mon[36624]: pgmap v7081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:10.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:10 smithi081 ceph-mon[31453]: pgmap v7081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:11.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:10 smithi153 ceph-mon[36569]: pgmap v7081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:12.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:12 smithi104 ceph-mon[36624]: pgmap v7082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:12.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:12 smithi081 ceph-mon[31453]: pgmap v7082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:13.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:12 smithi153 ceph-mon[36569]: pgmap v7082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:14.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:14 smithi104 ceph-mon[36624]: pgmap v7083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:14.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:14 smithi081 ceph-mon[31453]: pgmap v7083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:15.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:14 smithi153 ceph-mon[36569]: pgmap v7083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:16.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:16 smithi104 ceph-mon[36624]: pgmap v7084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:16.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:16 smithi081 ceph-mon[31453]: pgmap v7084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:17.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:16 smithi153 ceph-mon[36569]: pgmap v7084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:17.461+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:48:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:17.462+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:48:17.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:17.462+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:48:18.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:18 smithi081 ceph-mon[31453]: pgmap v7085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:19.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:18 smithi153 ceph-mon[36569]: pgmap v7085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:19.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:18 smithi104 ceph-mon[36624]: pgmap v7085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:20.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:20 smithi081 ceph-mon[31453]: pgmap v7086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:21.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:20 smithi153 ceph-mon[36569]: pgmap v7086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:21.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:20 smithi104 ceph-mon[36624]: pgmap v7086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:23.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:22 smithi153 ceph-mon[36569]: pgmap v7087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:23.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:22 smithi104 ceph-mon[36624]: pgmap v7087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:23.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:22 smithi081 ceph-mon[31453]: pgmap v7087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:25.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:24 smithi153 ceph-mon[36569]: pgmap v7088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:25.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:24 smithi104 ceph-mon[36624]: pgmap v7088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:25.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:24 smithi081 ceph-mon[31453]: pgmap v7088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:27.279 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:26 smithi081 ceph-mon[31453]: pgmap v7089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:27.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:26 smithi153 ceph-mon[36569]: pgmap v7089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:27.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:26 smithi104 ceph-mon[36624]: pgmap v7089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:29.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:28 smithi153 ceph-mon[36569]: pgmap v7090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:29.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:28 smithi104 ceph-mon[36624]: pgmap v7090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:29.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:28 smithi081 ceph-mon[31453]: pgmap v7090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:30.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:48:30.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:48:30.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:48:31.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:31 smithi153 ceph-mon[36569]: pgmap v7091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:31.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:31 smithi104 ceph-mon[36624]: pgmap v7091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:31.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:31 smithi081 ceph-mon[31453]: pgmap v7091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:32.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:48:32.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:48:32.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:32 smithi153 ceph-mon[36569]: pgmap v7092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:32.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:32 smithi153 ceph-mon[36569]: pgmap v7093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:32.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:48:32.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:48:32.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:48:32.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:32 smithi104 ceph-mon[36624]: pgmap v7092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:32.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:32 smithi104 ceph-mon[36624]: pgmap v7093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:32.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:48:32.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:48:32.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:48:32.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:32 smithi081 ceph-mon[31453]: pgmap v7092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:32.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:32 smithi081 ceph-mon[31453]: pgmap v7093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:32.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:48:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:32.475+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:48:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:32.475+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:48:32.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:32.475+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:48:34.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:34 smithi081 ceph-mon[31453]: pgmap v7094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:34.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:34 smithi153 ceph-mon[36569]: pgmap v7094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:34.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:34 smithi104 ceph-mon[36624]: pgmap v7094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:36.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:36 smithi081 ceph-mon[31453]: pgmap v7095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:36.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:36 smithi153 ceph-mon[36569]: pgmap v7095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:36.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:36 smithi104 ceph-mon[36624]: pgmap v7095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:38.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:38 smithi081 ceph-mon[31453]: pgmap v7096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:38.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:38 smithi153 ceph-mon[36569]: pgmap v7096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:38.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:38 smithi104 ceph-mon[36624]: pgmap v7096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:40.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:40 smithi081 ceph-mon[31453]: pgmap v7097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:40.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:40 smithi153 ceph-mon[36569]: pgmap v7097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:40.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:40 smithi104 ceph-mon[36624]: pgmap v7097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:42.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:42 smithi081 ceph-mon[31453]: pgmap v7098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:42.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:42 smithi153 ceph-mon[36569]: pgmap v7098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:42.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:42 smithi104 ceph-mon[36624]: pgmap v7098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:44.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:44 smithi081 ceph-mon[31453]: pgmap v7099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:44.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:44 smithi153 ceph-mon[36569]: pgmap v7099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:44.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:44 smithi104 ceph-mon[36624]: pgmap v7099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:46.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:46 smithi081 ceph-mon[31453]: pgmap v7100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:46.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:46 smithi153 ceph-mon[36569]: pgmap v7100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:46.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:46 smithi104 ceph-mon[36624]: pgmap v7100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:47.490 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:47.483+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:48:47.490 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:47.483+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:48:47.490 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:48:47.483+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:48:48.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:48 smithi081 ceph-mon[31453]: pgmap v7101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:48.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:48 smithi153 ceph-mon[36569]: pgmap v7101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:48.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:48 smithi104 ceph-mon[36624]: pgmap v7101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:50.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:50 smithi081 ceph-mon[31453]: pgmap v7102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:50.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:50 smithi153 ceph-mon[36569]: pgmap v7102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:50.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:50 smithi104 ceph-mon[36624]: pgmap v7102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:52.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:52 smithi081 ceph-mon[31453]: pgmap v7103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:52.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:52 smithi153 ceph-mon[36569]: pgmap v7103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:52.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:52 smithi104 ceph-mon[36624]: pgmap v7103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:54.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:54 smithi081 ceph-mon[31453]: pgmap v7104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:54.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:54 smithi153 ceph-mon[36569]: pgmap v7104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:54.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:54 smithi104 ceph-mon[36624]: pgmap v7104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:56.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:56 smithi081 ceph-mon[31453]: pgmap v7105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:56.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:56 smithi153 ceph-mon[36569]: pgmap v7105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:56.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:56 smithi104 ceph-mon[36624]: pgmap v7105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:58.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:48:58 smithi081 ceph-mon[31453]: pgmap v7106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:58.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:48:58 smithi153 ceph-mon[36569]: pgmap v7106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:48:58.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:48:58 smithi104 ceph-mon[36624]: pgmap v7106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:00.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:00 smithi081 ceph-mon[31453]: pgmap v7107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:00.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:00 smithi153 ceph-mon[36569]: pgmap v7107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:00.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:00 smithi104 ceph-mon[36624]: pgmap v7107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:02.497+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:49:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:02.497+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:49:02.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:02.497+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:49:02.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:02 smithi153 ceph-mon[36569]: pgmap v7108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:02.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:02 smithi104 ceph-mon[36624]: pgmap v7108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:02.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:02 smithi081 ceph-mon[31453]: pgmap v7108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:04.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:04 smithi153 ceph-mon[36569]: pgmap v7109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:04.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:04 smithi104 ceph-mon[36624]: pgmap v7109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:04.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:04 smithi081 ceph-mon[31453]: pgmap v7109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:06.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:06 smithi153 ceph-mon[36569]: pgmap v7110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:06.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:06 smithi104 ceph-mon[36624]: pgmap v7110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:06.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:06 smithi081 ceph-mon[31453]: pgmap v7110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:08.465 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:08 smithi081 ceph-mon[31453]: pgmap v7111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:08.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:08 smithi153 ceph-mon[36569]: pgmap v7111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:08.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:08 smithi104 ceph-mon[36624]: pgmap v7111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:10.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:10 smithi153 ceph-mon[36569]: pgmap v7112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:10.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:10 smithi104 ceph-mon[36624]: pgmap v7112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:10.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:10 smithi081 ceph-mon[31453]: pgmap v7112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:12.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:12 smithi153 ceph-mon[36569]: pgmap v7113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:12.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:12 smithi104 ceph-mon[36624]: pgmap v7113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:12.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:12 smithi081 ceph-mon[31453]: pgmap v7113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:14.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:14 smithi153 ceph-mon[36569]: pgmap v7114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:14.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:14 smithi104 ceph-mon[36624]: pgmap v7114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:14.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:14 smithi081 ceph-mon[31453]: pgmap v7114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:14.812+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:49:14.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:14.812+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:49:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:14.812+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:49:14.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:14.812+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:49:16.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:16 smithi153 ceph-mon[36569]: pgmap v7115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:16.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:16 smithi104 ceph-mon[36624]: pgmap v7115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:16.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:16 smithi081 ceph-mon[31453]: pgmap v7115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:17.500+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:49:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:17.500+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:49:17.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:17.500+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:49:17.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:17 smithi153 ceph-mon[36569]: pgmap v7116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:17.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:17 smithi104 ceph-mon[36624]: pgmap v7116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:17.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:17 smithi081 ceph-mon[31453]: pgmap v7116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:20.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:20 smithi081 ceph-mon[31453]: pgmap v7117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:20.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:20 smithi153 ceph-mon[36569]: pgmap v7117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:20.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:20 smithi104 ceph-mon[36624]: pgmap v7117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:22.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:22 smithi081 ceph-mon[31453]: pgmap v7118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:22.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:22 smithi153 ceph-mon[36569]: pgmap v7118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:22.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:22 smithi104 ceph-mon[36624]: pgmap v7118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:24.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:24 smithi081 ceph-mon[31453]: pgmap v7119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:24.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:24 smithi153 ceph-mon[36569]: pgmap v7119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:24.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:24 smithi104 ceph-mon[36624]: pgmap v7119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:26.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:26 smithi081 ceph-mon[31453]: pgmap v7120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:26.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:26 smithi153 ceph-mon[36569]: pgmap v7120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:26.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:26 smithi104 ceph-mon[36624]: pgmap v7120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:28.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:28 smithi081 ceph-mon[31453]: pgmap v7121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:28.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:28 smithi153 ceph-mon[36569]: pgmap v7121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:28.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:28 smithi104 ceph-mon[36624]: pgmap v7121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:30.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:30 smithi081 ceph-mon[31453]: pgmap v7122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:30.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:30 smithi153 ceph-mon[36569]: pgmap v7122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:30.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:30 smithi104 ceph-mon[36624]: pgmap v7122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:32.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:32 smithi081 ceph-mon[31453]: pgmap v7123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:32.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:49:32.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:32.513+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:49:32.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:32.513+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:49:32.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:32.513+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:49:32.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:32 smithi153 ceph-mon[36569]: pgmap v7123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:32.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:49:32.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:32 smithi104 ceph-mon[36624]: pgmap v7123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:32.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:49:33.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:49:33.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:49:33.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:49:33.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:49:33.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:49:33.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:49:33.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:49:33.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:49:33.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:49:34.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:34 smithi081 ceph-mon[31453]: pgmap v7124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:34.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:34 smithi081 ceph-mon[31453]: pgmap v7125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:34.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:34 smithi153 ceph-mon[36569]: pgmap v7124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:34.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:34 smithi153 ceph-mon[36569]: pgmap v7125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:34.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:34 smithi104 ceph-mon[36624]: pgmap v7124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:34.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:34 smithi104 ceph-mon[36624]: pgmap v7125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:36.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:36 smithi153 ceph-mon[36569]: pgmap v7126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:36.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:36 smithi104 ceph-mon[36624]: pgmap v7126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:36.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:36 smithi081 ceph-mon[31453]: pgmap v7126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:38.464 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:38 smithi081 ceph-mon[31453]: pgmap v7127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:38.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:38 smithi153 ceph-mon[36569]: pgmap v7127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:38.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:38 smithi104 ceph-mon[36624]: pgmap v7127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:40.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:40 smithi153 ceph-mon[36569]: pgmap v7128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:40.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:40 smithi104 ceph-mon[36624]: pgmap v7128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:40.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:40 smithi081 ceph-mon[31453]: pgmap v7128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:42.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:42 smithi153 ceph-mon[36569]: pgmap v7129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:42.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:42 smithi104 ceph-mon[36624]: pgmap v7129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:42.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:42 smithi081 ceph-mon[31453]: pgmap v7129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:44.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:44 smithi153 ceph-mon[36569]: pgmap v7130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:44.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:44 smithi104 ceph-mon[36624]: pgmap v7130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:44.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:44 smithi081 ceph-mon[31453]: pgmap v7130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:46.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:46 smithi153 ceph-mon[36569]: pgmap v7131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:46.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:46 smithi104 ceph-mon[36624]: pgmap v7131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:46.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:46 smithi081 ceph-mon[31453]: pgmap v7131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:47.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:47.529+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:49:47.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:47.529+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:49:47.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:49:47.529+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:49:48.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:48 smithi153 ceph-mon[36569]: pgmap v7132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:48.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:48 smithi104 ceph-mon[36624]: pgmap v7132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:48.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:48 smithi081 ceph-mon[31453]: pgmap v7132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:50.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:50 smithi153 ceph-mon[36569]: pgmap v7133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:50.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:50 smithi104 ceph-mon[36624]: pgmap v7133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:50.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:50 smithi081 ceph-mon[31453]: pgmap v7133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:52.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:52 smithi153 ceph-mon[36569]: pgmap v7134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:52.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:52 smithi104 ceph-mon[36624]: pgmap v7134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:52.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:52 smithi081 ceph-mon[31453]: pgmap v7134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:53.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:53 smithi153 ceph-mon[36569]: pgmap v7135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:53.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:53 smithi104 ceph-mon[36624]: pgmap v7135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:53.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:53 smithi081 ceph-mon[31453]: pgmap v7135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:56.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:55 smithi153 ceph-mon[36569]: pgmap v7136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:56.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:55 smithi104 ceph-mon[36624]: pgmap v7136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:56.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:55 smithi081 ceph-mon[31453]: pgmap v7136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:58.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:57 smithi153 ceph-mon[36569]: pgmap v7137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:58.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:57 smithi104 ceph-mon[36624]: pgmap v7137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:49:58.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:57 smithi081 ceph-mon[31453]: pgmap v7137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:00.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:49:59 smithi153 ceph-mon[36569]: pgmap v7138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:00.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:49:59 smithi104 ceph-mon[36624]: pgmap v7138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:00.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:49:59 smithi081 ceph-mon[31453]: pgmap v7138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:00.972 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:50:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:50:01.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T00:50:02.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:01 smithi153 ceph-mon[36569]: pgmap v7139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:02.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:01 smithi104 ceph-mon[36624]: pgmap v7139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:02.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:01 smithi081 ceph-mon[31453]: pgmap v7139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:02.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:02.545+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:50:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:02.545+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:50:02.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:02.545+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:50:04.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:03 smithi153 ceph-mon[36569]: pgmap v7140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:04.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:03 smithi104 ceph-mon[36624]: pgmap v7140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:04.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:03 smithi081 ceph-mon[31453]: pgmap v7140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:06.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:05 smithi153 ceph-mon[36569]: pgmap v7141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:06.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:05 smithi104 ceph-mon[36624]: pgmap v7141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:06.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:05 smithi081 ceph-mon[31453]: pgmap v7141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:08.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:07 smithi153 ceph-mon[36569]: pgmap v7142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:08.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:07 smithi104 ceph-mon[36624]: pgmap v7142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:08.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:07 smithi081 ceph-mon[31453]: pgmap v7142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:10.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:09 smithi153 ceph-mon[36569]: pgmap v7143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:10.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:09 smithi104 ceph-mon[36624]: pgmap v7143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:10.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:09 smithi081 ceph-mon[31453]: pgmap v7143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:12.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:11 smithi153 ceph-mon[36569]: pgmap v7144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:12.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:11 smithi104 ceph-mon[36624]: pgmap v7144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:12.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:11 smithi081 ceph-mon[31453]: pgmap v7144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:14.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:13 smithi153 ceph-mon[36569]: pgmap v7145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:14.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:13 smithi104 ceph-mon[36624]: pgmap v7145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:14.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:13 smithi081 ceph-mon[31453]: pgmap v7145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:16.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:15 smithi153 ceph-mon[36569]: pgmap v7146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:16.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:15 smithi104 ceph-mon[36624]: pgmap v7146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:16.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:15 smithi081 ceph-mon[31453]: pgmap v7146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:17.548+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:50:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:17.548+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:50:17.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:17.548+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:50:18.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:17 smithi153 ceph-mon[36569]: pgmap v7147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:18.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:17 smithi104 ceph-mon[36624]: pgmap v7147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:18.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:17 smithi081 ceph-mon[31453]: pgmap v7147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:20.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:19 smithi153 ceph-mon[36569]: pgmap v7148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:20.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:19 smithi104 ceph-mon[36624]: pgmap v7148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:20.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:19 smithi081 ceph-mon[31453]: pgmap v7148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:22.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:21 smithi153 ceph-mon[36569]: pgmap v7149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:22.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:21 smithi104 ceph-mon[36624]: pgmap v7149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:22.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:21 smithi081 ceph-mon[31453]: pgmap v7149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:24.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:23 smithi153 ceph-mon[36569]: pgmap v7150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:23 smithi104 ceph-mon[36624]: pgmap v7150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:24.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:23 smithi081 ceph-mon[31453]: pgmap v7150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:26.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:25 smithi153 ceph-mon[36569]: pgmap v7151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:26.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:25 smithi104 ceph-mon[36624]: pgmap v7151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:26.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:25 smithi081 ceph-mon[31453]: pgmap v7151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:28.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:28 smithi153 ceph-mon[36569]: pgmap v7152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:28.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:28 smithi104 ceph-mon[36624]: pgmap v7152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:28.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:28 smithi081 ceph-mon[31453]: pgmap v7152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:30.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:30 smithi153 ceph-mon[36569]: pgmap v7153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:30.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:30 smithi104 ceph-mon[36624]: pgmap v7153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:30.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:30 smithi081 ceph-mon[31453]: pgmap v7153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:32.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:32 smithi153 ceph-mon[36569]: pgmap v7154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:32.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:32 smithi104 ceph-mon[36624]: pgmap v7154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:32.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:32 smithi081 ceph-mon[31453]: pgmap v7154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:32.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:50:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:32.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:50:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:32.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:50:33.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:50:33.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:50:33.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:50:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: pgmap v7155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: pgmap v7155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: pgmap v7155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:34.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:34.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:50:35.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:50:35.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:50:35.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:50:35.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: pgmap v7156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: pgmap v7157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:35.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:50:35.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:50:35.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:50:35.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: pgmap v7156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:35.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: pgmap v7157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:35.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:50:35.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: pgmap v7156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: pgmap v7157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:35.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:50:36.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:36 smithi153 ceph-mon[36569]: pgmap v7158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:36.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:36 smithi104 ceph-mon[36624]: pgmap v7158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:36.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:36 smithi081 ceph-mon[31453]: pgmap v7158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:38.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:38 smithi153 ceph-mon[36569]: pgmap v7159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:38.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:38 smithi104 ceph-mon[36624]: pgmap v7159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:38.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:38 smithi081 ceph-mon[31453]: pgmap v7159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:40.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:40 smithi153 ceph-mon[36569]: pgmap v7160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:40.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:40 smithi104 ceph-mon[36624]: pgmap v7160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:40.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:40 smithi081 ceph-mon[31453]: pgmap v7160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:42.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:42 smithi153 ceph-mon[36569]: pgmap v7161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:42.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:42 smithi104 ceph-mon[36624]: pgmap v7161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:42.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:42 smithi081 ceph-mon[31453]: pgmap v7161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:44.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:44 smithi153 ceph-mon[36569]: pgmap v7162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:44.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:44 smithi104 ceph-mon[36624]: pgmap v7162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:44.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:44 smithi081 ceph-mon[31453]: pgmap v7162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:44.827+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:50:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:44.827+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:50:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:44.828+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:50:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:44.828+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:50:46.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:46 smithi153 ceph-mon[36569]: pgmap v7163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:46.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:46 smithi104 ceph-mon[36624]: pgmap v7163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:46.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:46 smithi081 ceph-mon[31453]: pgmap v7163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:47.568+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:50:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:47.568+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:50:47.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:50:47.568+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:50:48.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:48 smithi081 ceph-mon[31453]: pgmap v7164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:48.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:48 smithi153 ceph-mon[36569]: pgmap v7164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:48.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:48 smithi104 ceph-mon[36624]: pgmap v7164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:50.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:50 smithi153 ceph-mon[36569]: pgmap v7165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:50.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:50 smithi104 ceph-mon[36624]: pgmap v7165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:50.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:50 smithi081 ceph-mon[31453]: pgmap v7165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:52.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:52 smithi153 ceph-mon[36569]: pgmap v7166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:52.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:52 smithi104 ceph-mon[36624]: pgmap v7166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:52.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:52 smithi081 ceph-mon[31453]: pgmap v7166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:54.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:54 smithi153 ceph-mon[36569]: pgmap v7167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:54.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:54 smithi104 ceph-mon[36624]: pgmap v7167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:54.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:54 smithi081 ceph-mon[31453]: pgmap v7167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:56.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:56 smithi153 ceph-mon[36569]: pgmap v7168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:56.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:56 smithi104 ceph-mon[36624]: pgmap v7168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:56.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:56 smithi081 ceph-mon[31453]: pgmap v7168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:58.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:50:58 smithi153 ceph-mon[36569]: pgmap v7169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:58.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:50:58 smithi104 ceph-mon[36624]: pgmap v7169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:50:58.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:50:58 smithi081 ceph-mon[31453]: pgmap v7169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:00.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:00 smithi153 ceph-mon[36569]: pgmap v7170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:00.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:00 smithi104 ceph-mon[36624]: pgmap v7170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:00.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:00 smithi081 ceph-mon[31453]: pgmap v7170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:02.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:02 smithi153 ceph-mon[36569]: pgmap v7171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:02.581+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:51:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:02.581+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:51:02.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:02.581+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:51:02.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:02 smithi104 ceph-mon[36624]: pgmap v7171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:02.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:02 smithi081 ceph-mon[31453]: pgmap v7171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:04.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:04 smithi153 ceph-mon[36569]: pgmap v7172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:04.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:04 smithi104 ceph-mon[36624]: pgmap v7172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:04.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:04 smithi081 ceph-mon[31453]: pgmap v7172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:06.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:06 smithi153 ceph-mon[36569]: pgmap v7173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:06.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:06 smithi104 ceph-mon[36624]: pgmap v7173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:06.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:06 smithi081 ceph-mon[31453]: pgmap v7173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:08.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:08 smithi153 ceph-mon[36569]: pgmap v7174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:08.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:08 smithi104 ceph-mon[36624]: pgmap v7174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:08.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:08 smithi081 ceph-mon[31453]: pgmap v7174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:10.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:10 smithi153 ceph-mon[36569]: pgmap v7175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:10.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:10 smithi104 ceph-mon[36624]: pgmap v7175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:10.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:10 smithi081 ceph-mon[31453]: pgmap v7175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:12.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:12 smithi153 ceph-mon[36569]: pgmap v7176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:12.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:12 smithi104 ceph-mon[36624]: pgmap v7176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:12.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:12 smithi081 ceph-mon[31453]: pgmap v7176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:14.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:14 smithi153 ceph-mon[36569]: pgmap v7177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:14.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:14 smithi104 ceph-mon[36624]: pgmap v7177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:14.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:14 smithi081 ceph-mon[31453]: pgmap v7177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:16.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:16 smithi153 ceph-mon[36569]: pgmap v7178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:16.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:16 smithi104 ceph-mon[36624]: pgmap v7178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:16.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:16 smithi081 ceph-mon[31453]: pgmap v7178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:17.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:17.597+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:51:17.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:17.597+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:51:17.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:17.597+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:51:18.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:18 smithi153 ceph-mon[36569]: pgmap v7179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:18.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:18 smithi104 ceph-mon[36624]: pgmap v7179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:18.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:18 smithi081 ceph-mon[31453]: pgmap v7179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:20.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:20 smithi153 ceph-mon[36569]: pgmap v7180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:20.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:20 smithi104 ceph-mon[36624]: pgmap v7180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:20.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:20 smithi081 ceph-mon[31453]: pgmap v7180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:22.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:22 smithi153 ceph-mon[36569]: pgmap v7181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:22.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:22 smithi104 ceph-mon[36624]: pgmap v7181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:22.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:22 smithi081 ceph-mon[31453]: pgmap v7181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:24.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:24 smithi153 ceph-mon[36569]: pgmap v7182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:24.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:24 smithi104 ceph-mon[36624]: pgmap v7182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:24.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:24 smithi081 ceph-mon[31453]: pgmap v7182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:26.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:26 smithi153 ceph-mon[36569]: pgmap v7183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:26.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:26 smithi104 ceph-mon[36624]: pgmap v7183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:26.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:26 smithi081 ceph-mon[31453]: pgmap v7183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:28.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:28 smithi153 ceph-mon[36569]: pgmap v7184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:28.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:28 smithi104 ceph-mon[36624]: pgmap v7184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:28.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:28 smithi081 ceph-mon[31453]: pgmap v7184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:30.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:30 smithi153 ceph-mon[36569]: pgmap v7185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:30.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:30 smithi104 ceph-mon[36624]: pgmap v7185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:30.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:30 smithi081 ceph-mon[31453]: pgmap v7185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:32.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:32 smithi153 ceph-mon[36569]: pgmap v7186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:32.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:32 smithi104 ceph-mon[36624]: pgmap v7186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:32.613+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:51:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:32.613+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:51:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:32.613+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:51:32.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:32 smithi081 ceph-mon[31453]: pgmap v7186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:34 smithi081 ceph-mon[31453]: pgmap v7187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:34.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:34 smithi153 ceph-mon[36569]: pgmap v7187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:34.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:34 smithi104 ceph-mon[36624]: pgmap v7187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:35.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:51:35.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:51:35.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:51:36.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:51:36.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:51:36.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:36 smithi153 ceph-mon[36569]: pgmap v7188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:36.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:36 smithi153 ceph-mon[36569]: pgmap v7189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:36.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:51:36.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:51:36.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:51:36.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:36 smithi104 ceph-mon[36624]: pgmap v7188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:36.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:36 smithi104 ceph-mon[36624]: pgmap v7189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:36.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:51:36.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:51:36.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:51:36.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:36 smithi081 ceph-mon[31453]: pgmap v7188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:36.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:36 smithi081 ceph-mon[31453]: pgmap v7189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:36.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:51:39.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:38 smithi153 ceph-mon[36569]: pgmap v7190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:39.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:38 smithi104 ceph-mon[36624]: pgmap v7190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:39.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:38 smithi081 ceph-mon[31453]: pgmap v7190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:41.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:40 smithi153 ceph-mon[36569]: pgmap v7191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:41.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:40 smithi104 ceph-mon[36624]: pgmap v7191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:41.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:40 smithi081 ceph-mon[31453]: pgmap v7191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:43.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:42 smithi153 ceph-mon[36569]: pgmap v7192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:43.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:42 smithi104 ceph-mon[36624]: pgmap v7192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:43.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:42 smithi081 ceph-mon[31453]: pgmap v7192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:45.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:44 smithi153 ceph-mon[36569]: pgmap v7193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:45.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:44 smithi104 ceph-mon[36624]: pgmap v7193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:45.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:44 smithi081 ceph-mon[31453]: pgmap v7193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:47.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:46 smithi104 ceph-mon[36624]: pgmap v7194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:47.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:46 smithi081 ceph-mon[31453]: pgmap v7194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:47.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:46 smithi153 ceph-mon[36569]: pgmap v7194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:47.632 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:47.628+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:51:47.632 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:47.628+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:51:47.632 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:51:47.628+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:51:49.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:48 smithi104 ceph-mon[36624]: pgmap v7195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:49.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:48 smithi081 ceph-mon[31453]: pgmap v7195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:49.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:48 smithi153 ceph-mon[36569]: pgmap v7195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:51.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:50 smithi104 ceph-mon[36624]: pgmap v7196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:51.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:50 smithi081 ceph-mon[31453]: pgmap v7196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:51.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:50 smithi153 ceph-mon[36569]: pgmap v7196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:53.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:52 smithi104 ceph-mon[36624]: pgmap v7197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:53.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:52 smithi081 ceph-mon[31453]: pgmap v7197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:53.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:52 smithi153 ceph-mon[36569]: pgmap v7197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:55.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:54 smithi104 ceph-mon[36624]: pgmap v7198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:55.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:54 smithi081 ceph-mon[31453]: pgmap v7198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:55.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:54 smithi153 ceph-mon[36569]: pgmap v7198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:57.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:56 smithi081 ceph-mon[31453]: pgmap v7199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:57.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:56 smithi153 ceph-mon[36569]: pgmap v7199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:57.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:56 smithi104 ceph-mon[36624]: pgmap v7199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:59.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:51:58 smithi081 ceph-mon[31453]: pgmap v7200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:59.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:51:58 smithi153 ceph-mon[36569]: pgmap v7200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:51:59.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:51:58 smithi104 ceph-mon[36624]: pgmap v7200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:01.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:00 smithi153 ceph-mon[36569]: pgmap v7201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:01.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:00 smithi081 ceph-mon[31453]: pgmap v7201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:01.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:00 smithi104 ceph-mon[36624]: pgmap v7201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:02.634+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:52:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:02.634+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:52:02.639 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:02.634+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:52:03.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:02 smithi081 ceph-mon[31453]: pgmap v7202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:03.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:02 smithi153 ceph-mon[36569]: pgmap v7202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:03.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:02 smithi104 ceph-mon[36624]: pgmap v7202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:05.138 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:04 smithi104 ceph-mon[36624]: pgmap v7203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:05.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:04 smithi081 ceph-mon[31453]: pgmap v7203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:05.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:04 smithi153 ceph-mon[36569]: pgmap v7203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:07.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:06 smithi081 ceph-mon[31453]: pgmap v7204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:07.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:06 smithi153 ceph-mon[36569]: pgmap v7204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:07.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:06 smithi104 ceph-mon[36624]: pgmap v7204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:09.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:08 smithi081 ceph-mon[31453]: pgmap v7205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:09.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:08 smithi153 ceph-mon[36569]: pgmap v7205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:09.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:08 smithi104 ceph-mon[36624]: pgmap v7205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:11.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:10 smithi081 ceph-mon[31453]: pgmap v7206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:11.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:10 smithi153 ceph-mon[36569]: pgmap v7206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:11.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:10 smithi104 ceph-mon[36624]: pgmap v7206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:13.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:12 smithi081 ceph-mon[31453]: pgmap v7207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:13.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:12 smithi153 ceph-mon[36569]: pgmap v7207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:13.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:12 smithi104 ceph-mon[36624]: pgmap v7207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:14.841+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:52:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:14.841+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:52:14.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:14.842+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:52:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:14.842+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:52:15.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:14 smithi081 ceph-mon[31453]: pgmap v7208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:15.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:14 smithi153 ceph-mon[36569]: pgmap v7208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:15.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:14 smithi104 ceph-mon[36624]: pgmap v7208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:17.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:16 smithi081 ceph-mon[31453]: pgmap v7209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:17.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:16 smithi153 ceph-mon[36569]: pgmap v7209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:17.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:16 smithi104 ceph-mon[36624]: pgmap v7209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:17.649+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:52:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:17.649+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:52:17.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:17.649+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:52:19.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:18 smithi081 ceph-mon[31453]: pgmap v7210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:19.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:18 smithi153 ceph-mon[36569]: pgmap v7210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:19.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:18 smithi104 ceph-mon[36624]: pgmap v7210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:21.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:20 smithi081 ceph-mon[31453]: pgmap v7211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:21.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:20 smithi153 ceph-mon[36569]: pgmap v7211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:21.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:20 smithi104 ceph-mon[36624]: pgmap v7211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:23 smithi104 ceph-mon[36624]: pgmap v7212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:23.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:23 smithi081 ceph-mon[31453]: pgmap v7212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:23.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:23 smithi153 ceph-mon[36569]: pgmap v7212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:25.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:25 smithi104 ceph-mon[36624]: pgmap v7213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:25.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:25 smithi081 ceph-mon[31453]: pgmap v7213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:25.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:25 smithi153 ceph-mon[36569]: pgmap v7213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:27.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:27 smithi081 ceph-mon[31453]: pgmap v7214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:27.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:27 smithi153 ceph-mon[36569]: pgmap v7214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:27.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:27 smithi104 ceph-mon[36624]: pgmap v7214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:29.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:29 smithi081 ceph-mon[31453]: pgmap v7215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:29.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:29 smithi153 ceph-mon[36569]: pgmap v7215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:29.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:29 smithi104 ceph-mon[36624]: pgmap v7215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:31.362 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:31 smithi153 ceph-mon[36569]: pgmap v7216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:31.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:31 smithi081 ceph-mon[31453]: pgmap v7216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:31.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:31 smithi104 ceph-mon[36624]: pgmap v7216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:32.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:32.662+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:52:32.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:32.662+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:52:32.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:32.662+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:52:33.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:33 smithi081 ceph-mon[31453]: pgmap v7217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:33.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:33 smithi153 ceph-mon[36569]: pgmap v7217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:33.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:33 smithi104 ceph-mon[36624]: pgmap v7217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:35.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:35 smithi104 ceph-mon[36624]: pgmap v7218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:35 smithi081 ceph-mon[31453]: pgmap v7218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:35.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:35 smithi153 ceph-mon[36569]: pgmap v7218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:36.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:52:36.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:52:36.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:52:37.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:37 smithi081 ceph-mon[31453]: pgmap v7219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:37.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:37 smithi153 ceph-mon[36569]: pgmap v7219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:37.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:37 smithi104 ceph-mon[36624]: pgmap v7219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:38.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:52:38.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:52:38.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:52:38.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:52:38.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:52:38.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:52:38.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:52:38.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:52:38.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:52:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:39 smithi081 ceph-mon[31453]: pgmap v7220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:39 smithi081 ceph-mon[31453]: pgmap v7221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:39.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:39 smithi153 ceph-mon[36569]: pgmap v7220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:39.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:39 smithi153 ceph-mon[36569]: pgmap v7221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:39.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:39 smithi104 ceph-mon[36624]: pgmap v7220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:39.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:39 smithi104 ceph-mon[36624]: pgmap v7221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:41.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:41 smithi081 ceph-mon[31453]: pgmap v7222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:41.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:41 smithi153 ceph-mon[36569]: pgmap v7222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:41.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:41 smithi104 ceph-mon[36624]: pgmap v7222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:43 smithi081 ceph-mon[31453]: pgmap v7223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:43.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:43 smithi153 ceph-mon[36569]: pgmap v7223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:43.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:43 smithi104 ceph-mon[36624]: pgmap v7223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:45 smithi081 ceph-mon[31453]: pgmap v7224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:45.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:45 smithi153 ceph-mon[36569]: pgmap v7224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:45.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:45 smithi104 ceph-mon[36624]: pgmap v7224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:47.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:47 smithi081 ceph-mon[31453]: pgmap v7225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:47 smithi153 ceph-mon[36569]: pgmap v7225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:47.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:47 smithi104 ceph-mon[36624]: pgmap v7225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:47.677 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:47.674+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:52:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:47.674+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:52:47.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:52:47.674+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:52:49.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:49 smithi081 ceph-mon[31453]: pgmap v7226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:49.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:49 smithi153 ceph-mon[36569]: pgmap v7226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:49.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:49 smithi104 ceph-mon[36624]: pgmap v7226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:51.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:51 smithi153 ceph-mon[36569]: pgmap v7227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:51.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:51 smithi104 ceph-mon[36624]: pgmap v7227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:51.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:51 smithi081 ceph-mon[31453]: pgmap v7227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:53.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:53 smithi153 ceph-mon[36569]: pgmap v7228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:53.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:53 smithi104 ceph-mon[36624]: pgmap v7228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:53.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:53 smithi081 ceph-mon[31453]: pgmap v7228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:55.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:55 smithi153 ceph-mon[36569]: pgmap v7229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:55.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:55 smithi104 ceph-mon[36624]: pgmap v7229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:55.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:55 smithi081 ceph-mon[31453]: pgmap v7229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:57.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:57 smithi153 ceph-mon[36569]: pgmap v7230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:57.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:57 smithi104 ceph-mon[36624]: pgmap v7230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:57.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:57 smithi081 ceph-mon[31453]: pgmap v7230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:59.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:52:59 smithi153 ceph-mon[36569]: pgmap v7231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:59.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:52:59 smithi104 ceph-mon[36624]: pgmap v7231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:52:59.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:52:59 smithi081 ceph-mon[31453]: pgmap v7231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:01.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:01 smithi153 ceph-mon[36569]: pgmap v7232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:01.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:01 smithi104 ceph-mon[36624]: pgmap v7232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:01.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:01 smithi081 ceph-mon[31453]: pgmap v7232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:02.685+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:53:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:02.685+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:53:02.688 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:02.685+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:53:03.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:03 smithi153 ceph-mon[36569]: pgmap v7233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:03.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:03 smithi104 ceph-mon[36624]: pgmap v7233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:03.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:03 smithi081 ceph-mon[31453]: pgmap v7233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:05.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:05 smithi153 ceph-mon[36569]: pgmap v7234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:05.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:05 smithi104 ceph-mon[36624]: pgmap v7234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:05.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:05 smithi081 ceph-mon[31453]: pgmap v7234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:07 smithi153 ceph-mon[36569]: pgmap v7235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:07.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:07 smithi104 ceph-mon[36624]: pgmap v7235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:07.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:07 smithi081 ceph-mon[31453]: pgmap v7235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:09.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:09 smithi153 ceph-mon[36569]: pgmap v7236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:09.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:09 smithi104 ceph-mon[36624]: pgmap v7236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:09.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:09 smithi081 ceph-mon[31453]: pgmap v7236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:11.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:11 smithi153 ceph-mon[36569]: pgmap v7237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:11.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:11 smithi104 ceph-mon[36624]: pgmap v7237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:11.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:11 smithi081 ceph-mon[31453]: pgmap v7237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:13.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:13 smithi153 ceph-mon[36569]: pgmap v7238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:13.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:13 smithi104 ceph-mon[36624]: pgmap v7238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:13.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:13 smithi081 ceph-mon[31453]: pgmap v7238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:15.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:15 smithi153 ceph-mon[36569]: pgmap v7239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:15.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:15 smithi104 ceph-mon[36624]: pgmap v7239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:15.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:15 smithi081 ceph-mon[31453]: pgmap v7239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:17.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:17 smithi153 ceph-mon[36569]: pgmap v7240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:17.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:17 smithi104 ceph-mon[36624]: pgmap v7240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:17.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:17 smithi081 ceph-mon[31453]: pgmap v7240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:17.699+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:53:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:17.700+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:53:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:17.700+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:53:19.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:19 smithi153 ceph-mon[36569]: pgmap v7241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:19.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:19 smithi104 ceph-mon[36624]: pgmap v7241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:19.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:19 smithi081 ceph-mon[31453]: pgmap v7241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:21.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:21 smithi153 ceph-mon[36569]: pgmap v7242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:21.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:21 smithi104 ceph-mon[36624]: pgmap v7242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:21.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:21 smithi081 ceph-mon[31453]: pgmap v7242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:23.462 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:23 smithi081 ceph-mon[31453]: pgmap v7243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:23.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:23 smithi153 ceph-mon[36569]: pgmap v7243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:23.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:23 smithi104 ceph-mon[36624]: pgmap v7243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:25.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:25 smithi153 ceph-mon[36569]: pgmap v7244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:25.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:25 smithi104 ceph-mon[36624]: pgmap v7244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:25.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:25 smithi081 ceph-mon[31453]: pgmap v7244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:26.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:26 smithi153 ceph-mon[36569]: pgmap v7245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:26.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:26 smithi104 ceph-mon[36624]: pgmap v7245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:26.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:26 smithi081 ceph-mon[31453]: pgmap v7245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:28.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:28 smithi081 ceph-mon[31453]: pgmap v7246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:28.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:28 smithi153 ceph-mon[36569]: pgmap v7246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:28 smithi104 ceph-mon[36624]: pgmap v7246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:30.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:30 smithi081 ceph-mon[31453]: pgmap v7247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:30.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:30 smithi153 ceph-mon[36569]: pgmap v7247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:30.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:30 smithi104 ceph-mon[36624]: pgmap v7247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:32.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:32 smithi081 ceph-mon[31453]: pgmap v7248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:32.712+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:53:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:32.712+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:53:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:32.712+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:53:32.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:32 smithi153 ceph-mon[36569]: pgmap v7248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:32.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:32 smithi104 ceph-mon[36624]: pgmap v7248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:34.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:34 smithi081 ceph-mon[31453]: pgmap v7249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:34.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:34 smithi153 ceph-mon[36569]: pgmap v7249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:34 smithi104 ceph-mon[36624]: pgmap v7249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:36.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:36 smithi153 ceph-mon[36569]: pgmap v7250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:36.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:36 smithi104 ceph-mon[36624]: pgmap v7250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:36.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:36 smithi081 ceph-mon[31453]: pgmap v7250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:38.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:38 smithi153 ceph-mon[36569]: pgmap v7251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:38.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:53:38.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:38 smithi104 ceph-mon[36624]: pgmap v7251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:38.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:53:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:38 smithi081 ceph-mon[31453]: pgmap v7251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:38.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:53:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:53:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:53:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:39 smithi153 ceph-mon[36569]: pgmap v7252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:39.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:39 smithi153 ceph-mon[36569]: pgmap v7253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:39.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:53:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:53:39.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:53:39.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:39 smithi104 ceph-mon[36624]: pgmap v7252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:39.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:39 smithi104 ceph-mon[36624]: pgmap v7253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:39.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:53:39.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:53:39.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:53:39.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:39 smithi081 ceph-mon[31453]: pgmap v7252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:39.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:39 smithi081 ceph-mon[31453]: pgmap v7253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:39.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:53:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:41 smithi153 ceph-mon[36569]: pgmap v7254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:41 smithi104 ceph-mon[36624]: pgmap v7254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:41 smithi081 ceph-mon[31453]: pgmap v7254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:43 smithi153 ceph-mon[36569]: pgmap v7255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:43 smithi104 ceph-mon[36624]: pgmap v7255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:44.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:43 smithi081 ceph-mon[31453]: pgmap v7255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:45 smithi153 ceph-mon[36569]: pgmap v7256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:45 smithi104 ceph-mon[36624]: pgmap v7256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:45 smithi081 ceph-mon[31453]: pgmap v7256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:47.727+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:53:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:47.727+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:53:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:53:47.727+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:53:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:47 smithi153 ceph-mon[36569]: pgmap v7257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:48.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:47 smithi104 ceph-mon[36624]: pgmap v7257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:47 smithi081 ceph-mon[31453]: pgmap v7257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:49 smithi153 ceph-mon[36569]: pgmap v7258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:49 smithi104 ceph-mon[36624]: pgmap v7258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:49 smithi081 ceph-mon[31453]: pgmap v7258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:52.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:51 smithi153 ceph-mon[36569]: pgmap v7259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:51 smithi104 ceph-mon[36624]: pgmap v7259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:51 smithi081 ceph-mon[31453]: pgmap v7259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:54.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:53 smithi081 ceph-mon[31453]: pgmap v7260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:53 smithi153 ceph-mon[36569]: pgmap v7260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:53 smithi104 ceph-mon[36624]: pgmap v7260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:56.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:55 smithi153 ceph-mon[36569]: pgmap v7261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:55 smithi104 ceph-mon[36624]: pgmap v7261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:55 smithi081 ceph-mon[31453]: pgmap v7261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:57 smithi153 ceph-mon[36569]: pgmap v7262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:57 smithi104 ceph-mon[36624]: pgmap v7262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:53:58.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:57 smithi081 ceph-mon[31453]: pgmap v7262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:53:59 smithi153 ceph-mon[36569]: pgmap v7263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:53:59 smithi104 ceph-mon[36624]: pgmap v7263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:53:59 smithi081 ceph-mon[31453]: pgmap v7263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:01 smithi153 ceph-mon[36569]: pgmap v7264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:01 smithi104 ceph-mon[36624]: pgmap v7264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:01 smithi081 ceph-mon[31453]: pgmap v7264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:02.739+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:54:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:02.739+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:54:02.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:02.739+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:54:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:03 smithi153 ceph-mon[36569]: pgmap v7265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:03 smithi104 ceph-mon[36624]: pgmap v7265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:03 smithi081 ceph-mon[31453]: pgmap v7265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:05 smithi153 ceph-mon[36569]: pgmap v7266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:05 smithi104 ceph-mon[36624]: pgmap v7266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:05 smithi081 ceph-mon[31453]: pgmap v7266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:08 smithi153 ceph-mon[36569]: pgmap v7267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:08 smithi104 ceph-mon[36624]: pgmap v7267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:08 smithi081 ceph-mon[31453]: pgmap v7267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:10.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:10 smithi153 ceph-mon[36569]: pgmap v7268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:10 smithi104 ceph-mon[36624]: pgmap v7268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:10 smithi081 ceph-mon[31453]: pgmap v7268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:12.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:12 smithi153 ceph-mon[36569]: pgmap v7269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:12 smithi104 ceph-mon[36624]: pgmap v7269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:12 smithi081 ceph-mon[31453]: pgmap v7269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:14 smithi153 ceph-mon[36569]: pgmap v7270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:14 smithi104 ceph-mon[36624]: pgmap v7270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:14 smithi081 ceph-mon[31453]: pgmap v7270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:16.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:16 smithi153 ceph-mon[36569]: pgmap v7271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:16 smithi104 ceph-mon[36624]: pgmap v7271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:16 smithi081 ceph-mon[31453]: pgmap v7271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:17.744+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:54:17.746 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:17.744+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:54:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:17.744+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:54:18.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:18 smithi153 ceph-mon[36569]: pgmap v7272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:18 smithi104 ceph-mon[36624]: pgmap v7272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:18.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:18 smithi081 ceph-mon[31453]: pgmap v7272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:20 smithi153 ceph-mon[36569]: pgmap v7273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:20 smithi104 ceph-mon[36624]: pgmap v7273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:20.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:20 smithi081 ceph-mon[31453]: pgmap v7273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:22 smithi153 ceph-mon[36569]: pgmap v7274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:22 smithi104 ceph-mon[36624]: pgmap v7274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:22 smithi081 ceph-mon[31453]: pgmap v7274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:24.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:24 smithi153 ceph-mon[36569]: pgmap v7275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:24 smithi104 ceph-mon[36624]: pgmap v7275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:24.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:24 smithi081 ceph-mon[31453]: pgmap v7275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:26 smithi153 ceph-mon[36569]: pgmap v7276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:26 smithi104 ceph-mon[36624]: pgmap v7276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:26 smithi081 ceph-mon[31453]: pgmap v7276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:28 smithi153 ceph-mon[36569]: pgmap v7277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:28 smithi104 ceph-mon[36624]: pgmap v7277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:28.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:28 smithi081 ceph-mon[31453]: pgmap v7277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:30 smithi153 ceph-mon[36569]: pgmap v7278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:30 smithi104 ceph-mon[36624]: pgmap v7278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:30 smithi081 ceph-mon[31453]: pgmap v7278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:32 smithi104 ceph-mon[36624]: pgmap v7279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:32.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:32 smithi081 ceph-mon[31453]: pgmap v7279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:32 smithi153 ceph-mon[36569]: pgmap v7279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:32.750+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:54:32.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:32.750+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:54:32.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:32.750+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:54:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:34 smithi104 ceph-mon[36624]: pgmap v7280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:34 smithi081 ceph-mon[31453]: pgmap v7280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:34 smithi153 ceph-mon[36569]: pgmap v7280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:36 smithi104 ceph-mon[36624]: pgmap v7281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:36.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:36 smithi081 ceph-mon[31453]: pgmap v7281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:36 smithi153 ceph-mon[36569]: pgmap v7281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:38 smithi104 ceph-mon[36624]: pgmap v7282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:38 smithi081 ceph-mon[31453]: pgmap v7282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:38 smithi153 ceph-mon[36569]: pgmap v7282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:40 smithi153 ceph-mon[36569]: pgmap v7283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:54:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:40 smithi081 ceph-mon[31453]: pgmap v7283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:54:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:40.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:40 smithi104 ceph-mon[36624]: pgmap v7283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:40.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:54:40.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:40.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:40.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:41.327 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:41.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:42 smithi081 ceph-mon[31453]: pgmap v7284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:42 smithi153 ceph-mon[36569]: pgmap v7284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:42.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:42 smithi104 ceph-mon[36624]: pgmap v7284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:44.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:44 smithi081 ceph-mon[31453]: pgmap v7285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:44 smithi153 ceph-mon[36569]: pgmap v7285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:44 smithi104 ceph-mon[36624]: pgmap v7285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:45 smithi104 ceph-mon[36624]: pgmap v7286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:45.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:45.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:45.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:45 smithi081 ceph-mon[31453]: pgmap v7286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:45 smithi153 ceph-mon[36569]: pgmap v7286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:54:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:54:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:46 smithi081 ceph-mon[31453]: pgmap v7287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:46.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:46 smithi081 ceph-mon[31453]: pgmap v7288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:46.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:54:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:54:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:46 smithi153 ceph-mon[36569]: pgmap v7287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:46 smithi153 ceph-mon[36569]: pgmap v7288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:54:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:54:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:46 smithi104 ceph-mon[36624]: pgmap v7287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:46 smithi104 ceph-mon[36624]: pgmap v7288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:46.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:54:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:47.766+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:54:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:47.766+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:54:47.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:54:47.766+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:54:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:48 smithi081 ceph-mon[31453]: pgmap v7289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:48 smithi153 ceph-mon[36569]: pgmap v7289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:48 smithi104 ceph-mon[36624]: pgmap v7289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:50 smithi081 ceph-mon[31453]: pgmap v7290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:50 smithi153 ceph-mon[36569]: pgmap v7290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:50 smithi104 ceph-mon[36624]: pgmap v7290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:52.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:52 smithi081 ceph-mon[31453]: pgmap v7291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:52 smithi153 ceph-mon[36569]: pgmap v7291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:52 smithi104 ceph-mon[36624]: pgmap v7291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:54 smithi081 ceph-mon[31453]: pgmap v7292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:54 smithi153 ceph-mon[36569]: pgmap v7292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:54 smithi104 ceph-mon[36624]: pgmap v7292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:56 smithi153 ceph-mon[36569]: pgmap v7293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:56 smithi104 ceph-mon[36624]: pgmap v7293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:56 smithi081 ceph-mon[31453]: pgmap v7293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:54:58 smithi153 ceph-mon[36569]: pgmap v7294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:54:58 smithi104 ceph-mon[36624]: pgmap v7294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:54:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:54:58 smithi081 ceph-mon[31453]: pgmap v7294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:00 smithi153 ceph-mon[36569]: pgmap v7295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:00 smithi104 ceph-mon[36624]: pgmap v7295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:00 smithi081 ceph-mon[31453]: pgmap v7295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:02.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:02.781+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:55:02.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:02.781+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:55:02.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:02.781+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:55:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:02 smithi153 ceph-mon[36569]: pgmap v7296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:02 smithi104 ceph-mon[36624]: pgmap v7296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:02 smithi081 ceph-mon[31453]: pgmap v7296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:04 smithi153 ceph-mon[36569]: pgmap v7297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:04 smithi104 ceph-mon[36624]: pgmap v7297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:04 smithi081 ceph-mon[31453]: pgmap v7297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:06 smithi153 ceph-mon[36569]: pgmap v7298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:06 smithi104 ceph-mon[36624]: pgmap v7298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:06 smithi081 ceph-mon[31453]: pgmap v7298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:08 smithi153 ceph-mon[36569]: pgmap v7299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:08 smithi104 ceph-mon[36624]: pgmap v7299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:08 smithi081 ceph-mon[31453]: pgmap v7299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:10 smithi153 ceph-mon[36569]: pgmap v7300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:10 smithi104 ceph-mon[36624]: pgmap v7300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:10 smithi081 ceph-mon[31453]: pgmap v7300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:12 smithi153 ceph-mon[36569]: pgmap v7301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:12 smithi104 ceph-mon[36624]: pgmap v7301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:12 smithi081 ceph-mon[31453]: pgmap v7301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:14.872+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:55:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:14.872+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:55:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:14.872+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:55:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:14.872+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:55:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:14 smithi153 ceph-mon[36569]: pgmap v7302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:14 smithi104 ceph-mon[36624]: pgmap v7302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:14 smithi081 ceph-mon[31453]: pgmap v7302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:16 smithi153 ceph-mon[36569]: pgmap v7303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:16 smithi104 ceph-mon[36624]: pgmap v7303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:16 smithi081 ceph-mon[31453]: pgmap v7303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:17.791+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:55:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:17.791+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:55:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:17.791+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:55:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:18 smithi153 ceph-mon[36569]: pgmap v7304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:18 smithi104 ceph-mon[36624]: pgmap v7304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:18 smithi081 ceph-mon[31453]: pgmap v7304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:20 smithi153 ceph-mon[36569]: pgmap v7305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:20 smithi104 ceph-mon[36624]: pgmap v7305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:20 smithi081 ceph-mon[31453]: pgmap v7305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:23 smithi081 ceph-mon[31453]: pgmap v7306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:23 smithi153 ceph-mon[36569]: pgmap v7306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:23 smithi104 ceph-mon[36624]: pgmap v7306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:25 smithi081 ceph-mon[31453]: pgmap v7307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:25 smithi153 ceph-mon[36569]: pgmap v7307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:25.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:25 smithi104 ceph-mon[36624]: pgmap v7307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:27 smithi081 ceph-mon[31453]: pgmap v7308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:27.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:27 smithi153 ceph-mon[36569]: pgmap v7308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:27 smithi104 ceph-mon[36624]: pgmap v7308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:29 smithi081 ceph-mon[31453]: pgmap v7309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:29 smithi153 ceph-mon[36569]: pgmap v7309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:29.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:29 smithi104 ceph-mon[36624]: pgmap v7309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:31 smithi081 ceph-mon[31453]: pgmap v7310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:31 smithi153 ceph-mon[36569]: pgmap v7310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:31 smithi104 ceph-mon[36624]: pgmap v7310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:32.806+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:55:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:32.806+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:55:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:32.806+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:55:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:33 smithi153 ceph-mon[36569]: pgmap v7311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:33 smithi104 ceph-mon[36624]: pgmap v7311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:33 smithi081 ceph-mon[31453]: pgmap v7311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:35 smithi153 ceph-mon[36569]: pgmap v7312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:35 smithi104 ceph-mon[36624]: pgmap v7312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:35 smithi081 ceph-mon[31453]: pgmap v7312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:37.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:37 smithi153 ceph-mon[36569]: pgmap v7313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:37 smithi104 ceph-mon[36624]: pgmap v7313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:37 smithi081 ceph-mon[31453]: pgmap v7313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:39.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:39 smithi153 ceph-mon[36569]: pgmap v7314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:39.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:39 smithi104 ceph-mon[36624]: pgmap v7314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:39 smithi081 ceph-mon[31453]: pgmap v7314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:41 smithi153 ceph-mon[36569]: pgmap v7315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:41.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:41 smithi104 ceph-mon[36624]: pgmap v7315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:41.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:41 smithi081 ceph-mon[31453]: pgmap v7315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:43 smithi081 ceph-mon[31453]: pgmap v7316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:43 smithi153 ceph-mon[36569]: pgmap v7316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:43 smithi104 ceph-mon[36624]: pgmap v7316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:45.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:45 smithi153 ceph-mon[36569]: pgmap v7317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:45 smithi104 ceph-mon[36624]: pgmap v7317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:45 smithi081 ceph-mon[31453]: pgmap v7317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:55:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:55:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:55:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:47 smithi153 ceph-mon[36569]: pgmap v7318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:55:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:55:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:47 smithi104 ceph-mon[36624]: pgmap v7318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:55:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:55:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:47 smithi081 ceph-mon[31453]: pgmap v7318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:55:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:55:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:55:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:47.815+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:55:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:47.815+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:55:47.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:55:47.815+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:55:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:48 smithi153 ceph-mon[36569]: pgmap v7319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:48 smithi153 ceph-mon[36569]: pgmap v7320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:48 smithi104 ceph-mon[36624]: pgmap v7319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:48 smithi104 ceph-mon[36624]: pgmap v7320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:48 smithi081 ceph-mon[31453]: pgmap v7319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:48 smithi081 ceph-mon[31453]: pgmap v7320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:50 smithi153 ceph-mon[36569]: pgmap v7321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:50 smithi104 ceph-mon[36624]: pgmap v7321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:50 smithi081 ceph-mon[31453]: pgmap v7321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:52 smithi153 ceph-mon[36569]: pgmap v7322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:52 smithi104 ceph-mon[36624]: pgmap v7322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:52 smithi081 ceph-mon[31453]: pgmap v7322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:54.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:54 smithi153 ceph-mon[36569]: pgmap v7323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:54 smithi104 ceph-mon[36624]: pgmap v7323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:54 smithi081 ceph-mon[31453]: pgmap v7323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:56 smithi153 ceph-mon[36569]: pgmap v7324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:56 smithi104 ceph-mon[36624]: pgmap v7324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:56 smithi081 ceph-mon[31453]: pgmap v7324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:55:58 smithi153 ceph-mon[36569]: pgmap v7325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:55:58 smithi104 ceph-mon[36624]: pgmap v7325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:55:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:55:58 smithi081 ceph-mon[31453]: pgmap v7325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:00 smithi153 ceph-mon[36569]: pgmap v7326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:00 smithi104 ceph-mon[36624]: pgmap v7326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:00 smithi081 ceph-mon[31453]: pgmap v7326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:02 smithi153 ceph-mon[36569]: pgmap v7327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:02 smithi104 ceph-mon[36624]: pgmap v7327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:02 smithi081 ceph-mon[31453]: pgmap v7327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:02.831+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:56:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:02.831+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:56:02.833 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:02.831+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:56:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:04 smithi153 ceph-mon[36569]: pgmap v7328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:04 smithi104 ceph-mon[36624]: pgmap v7328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:04 smithi081 ceph-mon[31453]: pgmap v7328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:06 smithi153 ceph-mon[36569]: pgmap v7329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:06 smithi104 ceph-mon[36624]: pgmap v7329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:06 smithi081 ceph-mon[31453]: pgmap v7329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:08 smithi153 ceph-mon[36569]: pgmap v7330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:08 smithi104 ceph-mon[36624]: pgmap v7330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:08 smithi081 ceph-mon[31453]: pgmap v7330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:10 smithi153 ceph-mon[36569]: pgmap v7331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:10 smithi104 ceph-mon[36624]: pgmap v7331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:10 smithi081 ceph-mon[31453]: pgmap v7331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:12 smithi153 ceph-mon[36569]: pgmap v7332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:12 smithi104 ceph-mon[36624]: pgmap v7332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:12 smithi081 ceph-mon[31453]: pgmap v7332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:14 smithi153 ceph-mon[36569]: pgmap v7333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:14 smithi104 ceph-mon[36624]: pgmap v7333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:14 smithi081 ceph-mon[31453]: pgmap v7333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:16 smithi153 ceph-mon[36569]: pgmap v7334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:16 smithi104 ceph-mon[36624]: pgmap v7334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:16 smithi081 ceph-mon[31453]: pgmap v7334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:17 smithi081 ceph-mon[31453]: pgmap v7335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:17 smithi153 ceph-mon[36569]: pgmap v7335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:17 smithi104 ceph-mon[36624]: pgmap v7335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:17.845+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:56:17.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:17.845+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:56:17.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:17.845+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:56:20.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:20 smithi153 ceph-mon[36569]: pgmap v7336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:20.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:20 smithi104 ceph-mon[36624]: pgmap v7336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:20 smithi081 ceph-mon[31453]: pgmap v7336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:22 smithi153 ceph-mon[36569]: pgmap v7337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:22 smithi104 ceph-mon[36624]: pgmap v7337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:22.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:22 smithi081 ceph-mon[31453]: pgmap v7337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:24 smithi153 ceph-mon[36569]: pgmap v7338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:24 smithi104 ceph-mon[36624]: pgmap v7338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:24.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:24 smithi081 ceph-mon[31453]: pgmap v7338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:26 smithi153 ceph-mon[36569]: pgmap v7339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:26 smithi104 ceph-mon[36624]: pgmap v7339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:26 smithi081 ceph-mon[31453]: pgmap v7339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:28 smithi153 ceph-mon[36569]: pgmap v7340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:28 smithi104 ceph-mon[36624]: pgmap v7340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:28 smithi081 ceph-mon[31453]: pgmap v7340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:30 smithi081 ceph-mon[31453]: pgmap v7341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:30 smithi153 ceph-mon[36569]: pgmap v7341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:30 smithi104 ceph-mon[36624]: pgmap v7341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:32 smithi153 ceph-mon[36569]: pgmap v7342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:32 smithi104 ceph-mon[36624]: pgmap v7342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:32.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:32 smithi081 ceph-mon[31453]: pgmap v7342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:32.858+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:56:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:32.858+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:56:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:32.858+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:56:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:34 smithi153 ceph-mon[36569]: pgmap v7343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:34 smithi104 ceph-mon[36624]: pgmap v7343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:34 smithi081 ceph-mon[31453]: pgmap v7343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:36 smithi153 ceph-mon[36569]: pgmap v7344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:36 smithi104 ceph-mon[36624]: pgmap v7344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:36 smithi081 ceph-mon[31453]: pgmap v7344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:38 smithi153 ceph-mon[36569]: pgmap v7345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:38 smithi104 ceph-mon[36624]: pgmap v7345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:38.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:38 smithi081 ceph-mon[31453]: pgmap v7345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:40 smithi153 ceph-mon[36569]: pgmap v7346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:40 smithi104 ceph-mon[36624]: pgmap v7346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:40.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:40 smithi081 ceph-mon[31453]: pgmap v7346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:42 smithi153 ceph-mon[36569]: pgmap v7347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:42 smithi104 ceph-mon[36624]: pgmap v7347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:42 smithi081 ceph-mon[31453]: pgmap v7347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:44 smithi153 ceph-mon[36569]: pgmap v7348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:44 smithi104 ceph-mon[36624]: pgmap v7348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:44 smithi081 ceph-mon[31453]: pgmap v7348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:46 smithi153 ceph-mon[36569]: pgmap v7349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:46 smithi104 ceph-mon[36624]: pgmap v7349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:46 smithi081 ceph-mon[31453]: pgmap v7349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:47 smithi153 ceph-mon[36569]: pgmap v7350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:47 smithi104 ceph-mon[36624]: pgmap v7350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:47 smithi081 ceph-mon[31453]: pgmap v7350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:47.873+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:56:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:47.873+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:56:47.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:56:47.873+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:56:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:56:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:56:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:56:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:56:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:56:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:49 smithi104 ceph-mon[36624]: pgmap v7351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:49 smithi104 ceph-mon[36624]: pgmap v7352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:56:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:56:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:56:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:49 smithi081 ceph-mon[31453]: pgmap v7351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:49 smithi081 ceph-mon[31453]: pgmap v7352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:56:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:56:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:56:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:49 smithi153 ceph-mon[36569]: pgmap v7351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:49 smithi153 ceph-mon[36569]: pgmap v7352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:56:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:51 smithi153 ceph-mon[36569]: pgmap v7353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:51 smithi104 ceph-mon[36624]: pgmap v7353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:51 smithi081 ceph-mon[31453]: pgmap v7353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:53 smithi153 ceph-mon[36569]: pgmap v7354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:53 smithi104 ceph-mon[36624]: pgmap v7354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:53 smithi081 ceph-mon[31453]: pgmap v7354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:55 smithi153 ceph-mon[36569]: pgmap v7355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:55 smithi104 ceph-mon[36624]: pgmap v7355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:55 smithi081 ceph-mon[31453]: pgmap v7355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:57 smithi153 ceph-mon[36569]: pgmap v7356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:57 smithi104 ceph-mon[36624]: pgmap v7356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:56:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:57 smithi081 ceph-mon[31453]: pgmap v7356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:56:59 smithi153 ceph-mon[36569]: pgmap v7357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:56:59 smithi104 ceph-mon[36624]: pgmap v7357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:56:59 smithi081 ceph-mon[31453]: pgmap v7357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:01 smithi153 ceph-mon[36569]: pgmap v7358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:01 smithi104 ceph-mon[36624]: pgmap v7358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:01 smithi081 ceph-mon[31453]: pgmap v7358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:02.889+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:57:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:02.889+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:57:02.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:02.889+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:57:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:03 smithi153 ceph-mon[36569]: pgmap v7359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:03 smithi104 ceph-mon[36624]: pgmap v7359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:03 smithi081 ceph-mon[31453]: pgmap v7359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:05 smithi153 ceph-mon[36569]: pgmap v7360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:05 smithi104 ceph-mon[36624]: pgmap v7360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:05 smithi081 ceph-mon[31453]: pgmap v7360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:07 smithi153 ceph-mon[36569]: pgmap v7361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:07 smithi104 ceph-mon[36624]: pgmap v7361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:07 smithi081 ceph-mon[31453]: pgmap v7361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:09 smithi153 ceph-mon[36569]: pgmap v7362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:09 smithi104 ceph-mon[36624]: pgmap v7362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:09 smithi081 ceph-mon[31453]: pgmap v7362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:11 smithi153 ceph-mon[36569]: pgmap v7363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:11 smithi104 ceph-mon[36624]: pgmap v7363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:11 smithi081 ceph-mon[31453]: pgmap v7363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:13 smithi153 ceph-mon[36569]: pgmap v7364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:13 smithi104 ceph-mon[36624]: pgmap v7364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:13 smithi081 ceph-mon[31453]: pgmap v7364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:15 smithi153 ceph-mon[36569]: pgmap v7365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:15 smithi104 ceph-mon[36624]: pgmap v7365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:15 smithi081 ceph-mon[31453]: pgmap v7365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:17.904+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:57:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:17.904+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:57:17.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:17.904+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:57:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:17 smithi153 ceph-mon[36569]: pgmap v7366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:17 smithi104 ceph-mon[36624]: pgmap v7366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:17 smithi081 ceph-mon[31453]: pgmap v7366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:19 smithi153 ceph-mon[36569]: pgmap v7367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:19 smithi104 ceph-mon[36624]: pgmap v7367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:19 smithi081 ceph-mon[31453]: pgmap v7367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:21 smithi104 ceph-mon[36624]: pgmap v7368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:21 smithi081 ceph-mon[31453]: pgmap v7368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:21 smithi153 ceph-mon[36569]: pgmap v7368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:23 smithi104 ceph-mon[36624]: pgmap v7369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:23 smithi081 ceph-mon[31453]: pgmap v7369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:23 smithi153 ceph-mon[36569]: pgmap v7369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:25 smithi104 ceph-mon[36624]: pgmap v7370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:25 smithi081 ceph-mon[31453]: pgmap v7370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:26.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:25 smithi153 ceph-mon[36569]: pgmap v7370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:27 smithi104 ceph-mon[36624]: pgmap v7371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:27 smithi081 ceph-mon[31453]: pgmap v7371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:27 smithi153 ceph-mon[36569]: pgmap v7371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:29 smithi104 ceph-mon[36624]: pgmap v7372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:29 smithi081 ceph-mon[31453]: pgmap v7372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:29 smithi153 ceph-mon[36569]: pgmap v7372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:31 smithi081 ceph-mon[31453]: pgmap v7373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:31 smithi153 ceph-mon[36569]: pgmap v7373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:31 smithi104 ceph-mon[36624]: pgmap v7373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:32.920+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:57:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:32.920+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:57:32.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:32.920+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:57:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:33 smithi081 ceph-mon[31453]: pgmap v7374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:33 smithi153 ceph-mon[36569]: pgmap v7374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:33 smithi104 ceph-mon[36624]: pgmap v7374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:36.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:35 smithi153 ceph-mon[36569]: pgmap v7375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:35 smithi081 ceph-mon[31453]: pgmap v7375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:35 smithi104 ceph-mon[36624]: pgmap v7375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:38 smithi153 ceph-mon[36569]: pgmap v7376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:38 smithi104 ceph-mon[36624]: pgmap v7376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:38.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:38 smithi081 ceph-mon[31453]: pgmap v7376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:40 smithi153 ceph-mon[36569]: pgmap v7377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:40 smithi104 ceph-mon[36624]: pgmap v7377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:40.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:40 smithi081 ceph-mon[31453]: pgmap v7377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:42 smithi153 ceph-mon[36569]: pgmap v7378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:42 smithi104 ceph-mon[36624]: pgmap v7378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:42 smithi081 ceph-mon[31453]: pgmap v7378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:44 smithi153 ceph-mon[36569]: pgmap v7379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:44 smithi104 ceph-mon[36624]: pgmap v7379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:44 smithi081 ceph-mon[31453]: pgmap v7379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:46 smithi153 ceph-mon[36569]: pgmap v7380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:46 smithi104 ceph-mon[36624]: pgmap v7380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:46 smithi081 ceph-mon[31453]: pgmap v7380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:47.933+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:57:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:47.933+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:57:47.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:57:47.933+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:57:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:48 smithi153 ceph-mon[36569]: pgmap v7381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:48 smithi104 ceph-mon[36624]: pgmap v7381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:48 smithi081 ceph-mon[31453]: pgmap v7381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:57:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:57:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:57:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:50 smithi153 ceph-mon[36569]: pgmap v7382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:50 smithi104 ceph-mon[36624]: pgmap v7382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:50 smithi081 ceph-mon[31453]: pgmap v7382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:57:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:57:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:51 smithi153 ceph-mon[36569]: pgmap v7383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:51.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:51 smithi153 ceph-mon[36569]: pgmap v7384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:51.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:57:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:57:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:57:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:51 smithi104 ceph-mon[36624]: pgmap v7383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:51 smithi104 ceph-mon[36624]: pgmap v7384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:57:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:57:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:57:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:51 smithi081 ceph-mon[31453]: pgmap v7383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:51 smithi081 ceph-mon[31453]: pgmap v7384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:57:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:52 smithi081 ceph-mon[31453]: pgmap v7385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:52 smithi153 ceph-mon[36569]: pgmap v7385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:52 smithi104 ceph-mon[36624]: pgmap v7385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:54 smithi081 ceph-mon[31453]: pgmap v7386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:54 smithi153 ceph-mon[36569]: pgmap v7386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:54 smithi104 ceph-mon[36624]: pgmap v7386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:56 smithi081 ceph-mon[31453]: pgmap v7387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:56 smithi153 ceph-mon[36569]: pgmap v7387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:56 smithi104 ceph-mon[36624]: pgmap v7387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:57:58 smithi081 ceph-mon[31453]: pgmap v7388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:57:58 smithi153 ceph-mon[36569]: pgmap v7388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:57:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:57:58 smithi104 ceph-mon[36624]: pgmap v7388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:00 smithi081 ceph-mon[31453]: pgmap v7389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:00 smithi153 ceph-mon[36569]: pgmap v7389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:00 smithi104 ceph-mon[36624]: pgmap v7389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:02 smithi081 ceph-mon[31453]: pgmap v7390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:02 smithi153 ceph-mon[36569]: pgmap v7390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:02 smithi104 ceph-mon[36624]: pgmap v7390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:02.945+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:58:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:02.945+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:58:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:02.945+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:58:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:04 smithi081 ceph-mon[31453]: pgmap v7391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:04 smithi153 ceph-mon[36569]: pgmap v7391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:04 smithi104 ceph-mon[36624]: pgmap v7391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:06 smithi081 ceph-mon[31453]: pgmap v7392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:06 smithi153 ceph-mon[36569]: pgmap v7392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:06 smithi104 ceph-mon[36624]: pgmap v7392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:08 smithi081 ceph-mon[31453]: pgmap v7393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:08 smithi153 ceph-mon[36569]: pgmap v7393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:08 smithi104 ceph-mon[36624]: pgmap v7393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:10.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:10 smithi104 ceph-mon[36624]: pgmap v7394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:10 smithi081 ceph-mon[31453]: pgmap v7394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:10 smithi153 ceph-mon[36569]: pgmap v7394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:12 smithi081 ceph-mon[31453]: pgmap v7395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:12 smithi153 ceph-mon[36569]: pgmap v7395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:12 smithi104 ceph-mon[36624]: pgmap v7395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:14 smithi081 ceph-mon[31453]: pgmap v7396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:14 smithi153 ceph-mon[36569]: pgmap v7396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:14 smithi104 ceph-mon[36624]: pgmap v7396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:14.902+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:58:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:14.902+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:58:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:14.903+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:58:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:14.903+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:58:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:16 smithi081 ceph-mon[31453]: pgmap v7397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:16 smithi153 ceph-mon[36569]: pgmap v7397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:16 smithi104 ceph-mon[36624]: pgmap v7397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:17.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:58:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:17.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:58:17.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:17.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:58:18.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:18 smithi081 ceph-mon[31453]: pgmap v7398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:18.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:18 smithi153 ceph-mon[36569]: pgmap v7398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:18.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:18 smithi104 ceph-mon[36624]: pgmap v7398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:20 smithi081 ceph-mon[31453]: pgmap v7399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:20 smithi153 ceph-mon[36569]: pgmap v7399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:20 smithi104 ceph-mon[36624]: pgmap v7399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:23 smithi081 ceph-mon[31453]: pgmap v7400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:23 smithi153 ceph-mon[36569]: pgmap v7400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:23 smithi104 ceph-mon[36624]: pgmap v7400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:25 smithi081 ceph-mon[31453]: pgmap v7401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:25 smithi153 ceph-mon[36569]: pgmap v7401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:25 smithi104 ceph-mon[36624]: pgmap v7401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:27 smithi081 ceph-mon[31453]: pgmap v7402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:27 smithi153 ceph-mon[36569]: pgmap v7402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:27 smithi104 ceph-mon[36624]: pgmap v7402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:29 smithi081 ceph-mon[31453]: pgmap v7403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:29 smithi153 ceph-mon[36569]: pgmap v7403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:29 smithi104 ceph-mon[36624]: pgmap v7403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:31 smithi081 ceph-mon[31453]: pgmap v7404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:31 smithi153 ceph-mon[36569]: pgmap v7404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:31 smithi104 ceph-mon[36624]: pgmap v7404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:32.970+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:58:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:32.970+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:58:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:32.970+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:58:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:33 smithi081 ceph-mon[31453]: pgmap v7405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:33 smithi153 ceph-mon[36569]: pgmap v7405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:33 smithi104 ceph-mon[36624]: pgmap v7405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:35 smithi081 ceph-mon[31453]: pgmap v7406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:35 smithi153 ceph-mon[36569]: pgmap v7406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:35 smithi104 ceph-mon[36624]: pgmap v7406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:37 smithi081 ceph-mon[31453]: pgmap v7407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:37 smithi153 ceph-mon[36569]: pgmap v7407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:37 smithi104 ceph-mon[36624]: pgmap v7407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:39 smithi153 ceph-mon[36569]: pgmap v7408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:39 smithi104 ceph-mon[36624]: pgmap v7408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:39 smithi081 ceph-mon[31453]: pgmap v7408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:41 smithi153 ceph-mon[36569]: pgmap v7409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:41 smithi104 ceph-mon[36624]: pgmap v7409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:41 smithi081 ceph-mon[31453]: pgmap v7409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:43 smithi153 ceph-mon[36569]: pgmap v7410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:43 smithi104 ceph-mon[36624]: pgmap v7410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:43 smithi081 ceph-mon[31453]: pgmap v7410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:45 smithi153 ceph-mon[36569]: pgmap v7411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:45 smithi104 ceph-mon[36624]: pgmap v7411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:45 smithi081 ceph-mon[31453]: pgmap v7411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:47.985+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:58:47.986 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:47.985+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:58:47.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:47.985+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:58:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:48 smithi153 ceph-mon[36569]: pgmap v7412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:48 smithi104 ceph-mon[36624]: pgmap v7412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:48.610 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:48 smithi081 ceph-mon[31453]: pgmap v7412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:50 smithi153 ceph-mon[36569]: pgmap v7413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:50 smithi104 ceph-mon[36624]: pgmap v7413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:50 smithi081 ceph-mon[31453]: pgmap v7413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:58:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:58:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:58:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:52 smithi153 ceph-mon[36569]: pgmap v7414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:58:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:58:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:58:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:52 smithi104 ceph-mon[36624]: pgmap v7414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:58:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:58:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:58:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:52 smithi081 ceph-mon[31453]: pgmap v7414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:58:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:58:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:58:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:53 smithi153 ceph-mon[36569]: pgmap v7415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:53 smithi153 ceph-mon[36569]: pgmap v7416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:53 smithi104 ceph-mon[36624]: pgmap v7415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:53 smithi104 ceph-mon[36624]: pgmap v7416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:53 smithi081 ceph-mon[31453]: pgmap v7415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:53 smithi081 ceph-mon[31453]: pgmap v7416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:55 smithi153 ceph-mon[36569]: pgmap v7417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:55 smithi104 ceph-mon[36624]: pgmap v7417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:55 smithi081 ceph-mon[31453]: pgmap v7417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:57 smithi153 ceph-mon[36569]: pgmap v7418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:57 smithi104 ceph-mon[36624]: pgmap v7418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:57 smithi081 ceph-mon[31453]: pgmap v7418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:58:59 smithi153 ceph-mon[36569]: pgmap v7419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:58:59 smithi104 ceph-mon[36624]: pgmap v7419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:58:59 smithi081 ceph-mon[31453]: pgmap v7419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:58:59.910 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:59.910+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:58:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:59.910+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:58:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:59.910+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T00:58:59.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:58:59.910+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T00:59:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:01 smithi153 ceph-mon[36569]: pgmap v7420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:01 smithi104 ceph-mon[36624]: pgmap v7420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:01 smithi081 ceph-mon[31453]: pgmap v7420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:03.000+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:59:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:03.000+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:59:03.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:03.000+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:59:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:03 smithi153 ceph-mon[36569]: pgmap v7421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:03 smithi104 ceph-mon[36624]: pgmap v7421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:03 smithi081 ceph-mon[31453]: pgmap v7421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:05 smithi153 ceph-mon[36569]: pgmap v7422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:05 smithi104 ceph-mon[36624]: pgmap v7422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:05 smithi081 ceph-mon[31453]: pgmap v7422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:07 smithi153 ceph-mon[36569]: pgmap v7423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:07 smithi104 ceph-mon[36624]: pgmap v7423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:07 smithi081 ceph-mon[31453]: pgmap v7423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:09 smithi153 ceph-mon[36569]: pgmap v7424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:09 smithi104 ceph-mon[36624]: pgmap v7424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:09 smithi081 ceph-mon[31453]: pgmap v7424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:11 smithi153 ceph-mon[36569]: pgmap v7425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:11 smithi104 ceph-mon[36624]: pgmap v7425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:11 smithi081 ceph-mon[31453]: pgmap v7425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:13 smithi153 ceph-mon[36569]: pgmap v7426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:13 smithi104 ceph-mon[36624]: pgmap v7426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:13 smithi081 ceph-mon[31453]: pgmap v7426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:15 smithi104 ceph-mon[36624]: pgmap v7427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:15 smithi081 ceph-mon[31453]: pgmap v7427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:15 smithi153 ceph-mon[36569]: pgmap v7427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:17 smithi104 ceph-mon[36624]: pgmap v7428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:17 smithi081 ceph-mon[31453]: pgmap v7428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:17 smithi153 ceph-mon[36569]: pgmap v7428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:18.015+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:59:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:18.015+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:59:18.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:18.015+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:59:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:19 smithi104 ceph-mon[36624]: pgmap v7429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:19 smithi081 ceph-mon[31453]: pgmap v7429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:19 smithi153 ceph-mon[36569]: pgmap v7429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:20 smithi104 ceph-mon[36624]: pgmap v7430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:20 smithi081 ceph-mon[31453]: pgmap v7430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:20 smithi153 ceph-mon[36569]: pgmap v7430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:22 smithi081 ceph-mon[31453]: pgmap v7431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:22 smithi153 ceph-mon[36569]: pgmap v7431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:22 smithi104 ceph-mon[36624]: pgmap v7431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:24 smithi081 ceph-mon[31453]: pgmap v7432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:24 smithi153 ceph-mon[36569]: pgmap v7432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:24 smithi104 ceph-mon[36624]: pgmap v7432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:26 smithi081 ceph-mon[31453]: pgmap v7433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:26 smithi153 ceph-mon[36569]: pgmap v7433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:26 smithi104 ceph-mon[36624]: pgmap v7433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:28 smithi081 ceph-mon[31453]: pgmap v7434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:28 smithi153 ceph-mon[36569]: pgmap v7434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:28 smithi104 ceph-mon[36624]: pgmap v7434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:30 smithi081 ceph-mon[31453]: pgmap v7435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:30 smithi153 ceph-mon[36569]: pgmap v7435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:30 smithi104 ceph-mon[36624]: pgmap v7435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:33.023+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:59:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:33.024+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:59:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:33.024+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:59:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:32 smithi081 ceph-mon[31453]: pgmap v7436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:32 smithi153 ceph-mon[36569]: pgmap v7436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:32 smithi104 ceph-mon[36624]: pgmap v7436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:34 smithi081 ceph-mon[31453]: pgmap v7437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:34 smithi153 ceph-mon[36569]: pgmap v7437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:34 smithi104 ceph-mon[36624]: pgmap v7437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:36 smithi081 ceph-mon[31453]: pgmap v7438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:36 smithi153 ceph-mon[36569]: pgmap v7438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:36 smithi104 ceph-mon[36624]: pgmap v7438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:38 smithi081 ceph-mon[31453]: pgmap v7439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:38 smithi153 ceph-mon[36569]: pgmap v7439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:38 smithi104 ceph-mon[36624]: pgmap v7439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:40 smithi153 ceph-mon[36569]: pgmap v7440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:40 smithi104 ceph-mon[36624]: pgmap v7440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:40 smithi081 ceph-mon[31453]: pgmap v7440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:42 smithi153 ceph-mon[36569]: pgmap v7441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:42 smithi104 ceph-mon[36624]: pgmap v7441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:42 smithi081 ceph-mon[31453]: pgmap v7441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:44 smithi153 ceph-mon[36569]: pgmap v7442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:44 smithi104 ceph-mon[36624]: pgmap v7442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:44 smithi081 ceph-mon[31453]: pgmap v7442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:46 smithi153 ceph-mon[36569]: pgmap v7443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:46 smithi104 ceph-mon[36624]: pgmap v7443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:46 smithi081 ceph-mon[31453]: pgmap v7443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:48.039+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T00:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:48.039+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T00:59:48.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T00:59:48.039+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T00:59:49.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:48 smithi081 ceph-mon[31453]: pgmap v7444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:48 smithi153 ceph-mon[36569]: pgmap v7444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:48 smithi104 ceph-mon[36624]: pgmap v7444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:50 smithi153 ceph-mon[36569]: pgmap v7445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:50 smithi104 ceph-mon[36624]: pgmap v7445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:50 smithi081 ceph-mon[31453]: pgmap v7445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:52 smithi153 ceph-mon[36569]: pgmap v7446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:59:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:52 smithi104 ceph-mon[36624]: pgmap v7446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:59:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:52 smithi081 ceph-mon[31453]: pgmap v7446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T00:59:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:59:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:59:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:59:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:59:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:59:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:59:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T00:59:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T00:59:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T00:59:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:54 smithi153 ceph-mon[36569]: pgmap v7447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:54 smithi153 ceph-mon[36569]: pgmap v7448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:54 smithi104 ceph-mon[36624]: pgmap v7447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:54 smithi104 ceph-mon[36624]: pgmap v7448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:54 smithi081 ceph-mon[31453]: pgmap v7447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:54 smithi081 ceph-mon[31453]: pgmap v7448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:56 smithi153 ceph-mon[36569]: pgmap v7449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:56 smithi104 ceph-mon[36624]: pgmap v7449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:56 smithi081 ceph-mon[31453]: pgmap v7449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 00:59:58 smithi153 ceph-mon[36569]: pgmap v7450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 00:59:58 smithi104 ceph-mon[36624]: pgmap v7450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T00:59:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 00:59:58 smithi081 ceph-mon[31453]: pgmap v7450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:00 smithi153 ceph-mon[36569]: pgmap v7451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:00:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:00 smithi104 ceph-mon[36624]: pgmap v7451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:00:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:00 smithi081 ceph-mon[31453]: pgmap v7451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:00:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:03.050+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:00:03.050 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:03.050+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:00:03.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:03.050+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:00:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:02 smithi153 ceph-mon[36569]: pgmap v7452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:02 smithi104 ceph-mon[36624]: pgmap v7452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:02 smithi081 ceph-mon[31453]: pgmap v7452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:04 smithi153 ceph-mon[36569]: pgmap v7453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:04 smithi104 ceph-mon[36624]: pgmap v7453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:04 smithi081 ceph-mon[31453]: pgmap v7453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:06 smithi153 ceph-mon[36569]: pgmap v7454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:06 smithi104 ceph-mon[36624]: pgmap v7454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:06 smithi081 ceph-mon[31453]: pgmap v7454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:08 smithi153 ceph-mon[36569]: pgmap v7455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:08 smithi104 ceph-mon[36624]: pgmap v7455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:08 smithi081 ceph-mon[31453]: pgmap v7455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:10 smithi153 ceph-mon[36569]: pgmap v7456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:11.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:10 smithi104 ceph-mon[36624]: pgmap v7456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:10 smithi081 ceph-mon[31453]: pgmap v7456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:13 smithi153 ceph-mon[36569]: pgmap v7457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:13 smithi104 ceph-mon[36624]: pgmap v7457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:13 smithi081 ceph-mon[31453]: pgmap v7457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:15 smithi153 ceph-mon[36569]: pgmap v7458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:15 smithi104 ceph-mon[36624]: pgmap v7458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:15 smithi081 ceph-mon[31453]: pgmap v7458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:17 smithi153 ceph-mon[36569]: pgmap v7459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:17 smithi104 ceph-mon[36624]: pgmap v7459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:17 smithi081 ceph-mon[31453]: pgmap v7459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:18.053+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:00:18.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:18.053+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:00:18.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:18.053+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:00:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:19 smithi153 ceph-mon[36569]: pgmap v7460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:19 smithi104 ceph-mon[36624]: pgmap v7460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:19 smithi081 ceph-mon[31453]: pgmap v7460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:21 smithi153 ceph-mon[36569]: pgmap v7461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:21 smithi104 ceph-mon[36624]: pgmap v7461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:21 smithi081 ceph-mon[31453]: pgmap v7461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:23 smithi153 ceph-mon[36569]: pgmap v7462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:23 smithi104 ceph-mon[36624]: pgmap v7462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:23 smithi081 ceph-mon[31453]: pgmap v7462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:25 smithi153 ceph-mon[36569]: pgmap v7463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:25 smithi104 ceph-mon[36624]: pgmap v7463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:25 smithi081 ceph-mon[31453]: pgmap v7463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:27 smithi153 ceph-mon[36569]: pgmap v7464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:27 smithi104 ceph-mon[36624]: pgmap v7464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:27 smithi081 ceph-mon[31453]: pgmap v7464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:29 smithi153 ceph-mon[36569]: pgmap v7465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:29 smithi104 ceph-mon[36624]: pgmap v7465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:29 smithi081 ceph-mon[31453]: pgmap v7465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:31 smithi153 ceph-mon[36569]: pgmap v7466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:31 smithi104 ceph-mon[36624]: pgmap v7466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:31 smithi081 ceph-mon[31453]: pgmap v7466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:33.055+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:00:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:33.055+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:00:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:33.055+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:00:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:33 smithi153 ceph-mon[36569]: pgmap v7467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:33 smithi104 ceph-mon[36624]: pgmap v7467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:33 smithi081 ceph-mon[31453]: pgmap v7467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:35 smithi153 ceph-mon[36569]: pgmap v7468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:35 smithi104 ceph-mon[36624]: pgmap v7468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:35 smithi081 ceph-mon[31453]: pgmap v7468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:37 smithi153 ceph-mon[36569]: pgmap v7469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:37 smithi104 ceph-mon[36624]: pgmap v7469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:37 smithi081 ceph-mon[31453]: pgmap v7469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:39 smithi153 ceph-mon[36569]: pgmap v7470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:39 smithi104 ceph-mon[36624]: pgmap v7470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:39 smithi081 ceph-mon[31453]: pgmap v7470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:41 smithi153 ceph-mon[36569]: pgmap v7471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:41 smithi104 ceph-mon[36624]: pgmap v7471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:41 smithi081 ceph-mon[31453]: pgmap v7471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:43 smithi104 ceph-mon[36624]: pgmap v7472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:43 smithi081 ceph-mon[31453]: pgmap v7472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:43 smithi153 ceph-mon[36569]: pgmap v7472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:45 smithi104 ceph-mon[36624]: pgmap v7473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:45 smithi081 ceph-mon[31453]: pgmap v7473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:45 smithi153 ceph-mon[36569]: pgmap v7473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:47 smithi104 ceph-mon[36624]: pgmap v7474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:47 smithi081 ceph-mon[31453]: pgmap v7474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:47 smithi153 ceph-mon[36569]: pgmap v7474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:48.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:48.061+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:00:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:48.061+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:00:48.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:00:48.061+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:00:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:49 smithi104 ceph-mon[36624]: pgmap v7475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:49 smithi081 ceph-mon[31453]: pgmap v7475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:49 smithi153 ceph-mon[36569]: pgmap v7475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:51 smithi104 ceph-mon[36624]: pgmap v7476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:51 smithi081 ceph-mon[31453]: pgmap v7476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:51 smithi153 ceph-mon[36569]: pgmap v7476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:53 smithi081 ceph-mon[31453]: pgmap v7477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:53 smithi153 ceph-mon[36569]: pgmap v7477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:53 smithi104 ceph-mon[36624]: pgmap v7477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:00:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:00:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:00:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: pgmap v7478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:00:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:00:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: pgmap v7478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:00:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:00:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:00:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: pgmap v7478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:00:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:00:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:56 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:00:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:56 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:00:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:56 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:00:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:56 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:00:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:56 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:00:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:56 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:00:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:56 smithi081 ceph-mon[31453]: pgmap v7479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:56 smithi081 ceph-mon[31453]: pgmap v7480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:56 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:00:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:56 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:00:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:56 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:00:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:56 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:00:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:56 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:00:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:56 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:00:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:56 smithi153 ceph-mon[36569]: pgmap v7479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:56 smithi153 ceph-mon[36569]: pgmap v7480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:56 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:56 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:56 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:56 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:56 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:56 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:56 smithi104 ceph-mon[36624]: pgmap v7479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:56 smithi104 ceph-mon[36624]: pgmap v7480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:00:57 smithi081 ceph-mon[31453]: pgmap v7481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:00:57 smithi153 ceph-mon[36569]: pgmap v7481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:00:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:00:57 smithi104 ceph-mon[36624]: pgmap v7481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:00.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:00 smithi153 ceph-mon[36569]: pgmap v7482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:00 smithi104 ceph-mon[36624]: pgmap v7482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:00 smithi081 ceph-mon[31453]: pgmap v7482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:02 smithi153 ceph-mon[36569]: pgmap v7483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:02 smithi104 ceph-mon[36624]: pgmap v7483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:02 smithi081 ceph-mon[31453]: pgmap v7483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:03.078+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:01:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:03.078+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:01:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:03.078+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:01:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:04 smithi153 ceph-mon[36569]: pgmap v7484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:04 smithi104 ceph-mon[36624]: pgmap v7484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:04 smithi081 ceph-mon[31453]: pgmap v7484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:06 smithi153 ceph-mon[36569]: pgmap v7485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:06 smithi104 ceph-mon[36624]: pgmap v7485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:06 smithi081 ceph-mon[31453]: pgmap v7485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:08 smithi153 ceph-mon[36569]: pgmap v7486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:08 smithi104 ceph-mon[36624]: pgmap v7486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:08 smithi081 ceph-mon[31453]: pgmap v7486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:10 smithi153 ceph-mon[36569]: pgmap v7487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:10 smithi104 ceph-mon[36624]: pgmap v7487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:10 smithi081 ceph-mon[31453]: pgmap v7487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:12.125+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000746) for more than 900000000 us, fault. 2024-04-13T01:01:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:12.125+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c9f0 tx=0x7f16f4039000 comp rx=0 tx=0).stop 2024-04-13T01:01:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:12.125+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T01:01:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:12.125+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ec0 0x7f16e80b3a70 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T01:01:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:12.125+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T01:01:12.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:12.126+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T01:01:12.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:12.127+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006330 tx=0x7f16f40c4000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T01:01:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:12 smithi153 ceph-mon[36569]: pgmap v7488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:12 smithi104 ceph-mon[36624]: pgmap v7488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:12 smithi081 ceph-mon[31453]: pgmap v7488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:14 smithi153 ceph-mon[36569]: pgmap v7489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:14 smithi104 ceph-mon[36624]: pgmap v7489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:14 smithi081 ceph-mon[31453]: pgmap v7489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:14.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:14.932+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:01:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:14.932+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:01:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:14.933+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:01:14.933 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:14.933+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:01:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:16 smithi104 ceph-mon[36624]: pgmap v7490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:16 smithi081 ceph-mon[31453]: pgmap v7490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:16 smithi153 ceph-mon[36569]: pgmap v7490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:18.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:18.092+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:01:18.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:18.092+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:01:18.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:18.092+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:01:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:18 smithi104 ceph-mon[36624]: pgmap v7491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:18 smithi081 ceph-mon[31453]: pgmap v7491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:18 smithi153 ceph-mon[36569]: pgmap v7491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:20 smithi104 ceph-mon[36624]: pgmap v7492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:20 smithi081 ceph-mon[31453]: pgmap v7492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:20 smithi153 ceph-mon[36569]: pgmap v7492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:22 smithi104 ceph-mon[36624]: pgmap v7493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:22 smithi081 ceph-mon[31453]: pgmap v7493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:22 smithi153 ceph-mon[36569]: pgmap v7493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:24 smithi104 ceph-mon[36624]: pgmap v7494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:24 smithi081 ceph-mon[31453]: pgmap v7494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:24 smithi153 ceph-mon[36569]: pgmap v7494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:26 smithi081 ceph-mon[31453]: pgmap v7495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:26 smithi153 ceph-mon[36569]: pgmap v7495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:26 smithi104 ceph-mon[36624]: pgmap v7495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:28 smithi081 ceph-mon[31453]: pgmap v7496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:28 smithi153 ceph-mon[36569]: pgmap v7496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:28 smithi104 ceph-mon[36624]: pgmap v7496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:30 smithi081 ceph-mon[31453]: pgmap v7497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:30 smithi153 ceph-mon[36569]: pgmap v7497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:30 smithi104 ceph-mon[36624]: pgmap v7497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:32 smithi081 ceph-mon[31453]: pgmap v7498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:32 smithi153 ceph-mon[36569]: pgmap v7498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:32 smithi104 ceph-mon[36624]: pgmap v7498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:33.108+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:01:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:33.108+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:01:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:33.108+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:01:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:34 smithi081 ceph-mon[31453]: pgmap v7499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:34 smithi153 ceph-mon[36569]: pgmap v7499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:34 smithi104 ceph-mon[36624]: pgmap v7499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:36 smithi081 ceph-mon[31453]: pgmap v7500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:36 smithi153 ceph-mon[36569]: pgmap v7500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:36 smithi104 ceph-mon[36624]: pgmap v7500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:38 smithi081 ceph-mon[31453]: pgmap v7501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:38 smithi153 ceph-mon[36569]: pgmap v7501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:38 smithi104 ceph-mon[36624]: pgmap v7501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:40 smithi081 ceph-mon[31453]: pgmap v7502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:40 smithi153 ceph-mon[36569]: pgmap v7502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:40 smithi104 ceph-mon[36624]: pgmap v7502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:42.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:42 smithi104 ceph-mon[36624]: pgmap v7503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:42 smithi081 ceph-mon[31453]: pgmap v7503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:42 smithi153 ceph-mon[36569]: pgmap v7503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:44 smithi081 ceph-mon[31453]: pgmap v7504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:44 smithi153 ceph-mon[36569]: pgmap v7504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:44 smithi104 ceph-mon[36624]: pgmap v7504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:46 smithi081 ceph-mon[31453]: pgmap v7505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:46 smithi153 ceph-mon[36569]: pgmap v7505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:46 smithi104 ceph-mon[36624]: pgmap v7505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:48.109+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:01:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:48.109+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:01:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:01:48.109+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:01:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:48 smithi081 ceph-mon[31453]: pgmap v7506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:48 smithi153 ceph-mon[36569]: pgmap v7506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:48 smithi104 ceph-mon[36624]: pgmap v7506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:50 smithi081 ceph-mon[31453]: pgmap v7507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:50 smithi153 ceph-mon[36569]: pgmap v7507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:50 smithi104 ceph-mon[36624]: pgmap v7507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:52 smithi153 ceph-mon[36569]: pgmap v7508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:52 smithi104 ceph-mon[36624]: pgmap v7508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:52 smithi081 ceph-mon[31453]: pgmap v7508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:54 smithi153 ceph-mon[36569]: pgmap v7509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:54 smithi104 ceph-mon[36624]: pgmap v7509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:54 smithi081 ceph-mon[31453]: pgmap v7509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:56 smithi153 ceph-mon[36569]: pgmap v7510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:01:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:56 smithi104 ceph-mon[36624]: pgmap v7510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:01:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:56 smithi081 ceph-mon[31453]: pgmap v7510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:01:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:01:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:01:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:01:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:01:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:01:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:01:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:01:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:01:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:01:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:58 smithi153 ceph-mon[36569]: pgmap v7511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:01:58 smithi153 ceph-mon[36569]: pgmap v7512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:58 smithi104 ceph-mon[36624]: pgmap v7511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:01:58 smithi104 ceph-mon[36624]: pgmap v7512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:58 smithi081 ceph-mon[31453]: pgmap v7511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:01:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:01:58 smithi081 ceph-mon[31453]: pgmap v7512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:00 smithi153 ceph-mon[36569]: pgmap v7513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:00 smithi104 ceph-mon[36624]: pgmap v7513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:00 smithi081 ceph-mon[31453]: pgmap v7513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:02 smithi153 ceph-mon[36569]: pgmap v7514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:02 smithi104 ceph-mon[36624]: pgmap v7514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:02 smithi081 ceph-mon[31453]: pgmap v7514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:03.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:03.115+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:02:03.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:03.115+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:02:03.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:03.115+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:02:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:04 smithi153 ceph-mon[36569]: pgmap v7515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:04.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:04 smithi104 ceph-mon[36624]: pgmap v7515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:04 smithi081 ceph-mon[31453]: pgmap v7515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:06 smithi153 ceph-mon[36569]: pgmap v7516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:06 smithi104 ceph-mon[36624]: pgmap v7516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:06 smithi081 ceph-mon[31453]: pgmap v7516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:08 smithi153 ceph-mon[36569]: pgmap v7517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:08 smithi104 ceph-mon[36624]: pgmap v7517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:08 smithi081 ceph-mon[31453]: pgmap v7517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:10 smithi153 ceph-mon[36569]: pgmap v7518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:10 smithi104 ceph-mon[36624]: pgmap v7518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:10 smithi081 ceph-mon[31453]: pgmap v7518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:12 smithi081 ceph-mon[31453]: pgmap v7519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:12 smithi153 ceph-mon[36569]: pgmap v7519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:12 smithi104 ceph-mon[36624]: pgmap v7519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:14 smithi153 ceph-mon[36569]: pgmap v7520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:14 smithi104 ceph-mon[36624]: pgmap v7520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:14 smithi081 ceph-mon[31453]: pgmap v7520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:16 smithi153 ceph-mon[36569]: pgmap v7521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:16 smithi104 ceph-mon[36624]: pgmap v7521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:16 smithi081 ceph-mon[31453]: pgmap v7521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:18.130 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:18.130+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:02:18.130 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:18.130+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:02:18.131 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:18.130+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:02:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:18 smithi153 ceph-mon[36569]: pgmap v7522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:18 smithi104 ceph-mon[36624]: pgmap v7522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:18 smithi081 ceph-mon[31453]: pgmap v7522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:20 smithi153 ceph-mon[36569]: pgmap v7523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:20 smithi104 ceph-mon[36624]: pgmap v7523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:20 smithi081 ceph-mon[31453]: pgmap v7523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:22 smithi153 ceph-mon[36569]: pgmap v7524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:22 smithi104 ceph-mon[36624]: pgmap v7524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:22 smithi081 ceph-mon[31453]: pgmap v7524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:24 smithi153 ceph-mon[36569]: pgmap v7525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:24 smithi104 ceph-mon[36624]: pgmap v7525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:24 smithi081 ceph-mon[31453]: pgmap v7525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:26 smithi153 ceph-mon[36569]: pgmap v7526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:26 smithi104 ceph-mon[36624]: pgmap v7526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:26 smithi081 ceph-mon[31453]: pgmap v7526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:28 smithi153 ceph-mon[36569]: pgmap v7527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:28 smithi104 ceph-mon[36624]: pgmap v7527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:28 smithi081 ceph-mon[31453]: pgmap v7527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:30 smithi153 ceph-mon[36569]: pgmap v7528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:30 smithi104 ceph-mon[36624]: pgmap v7528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:30 smithi081 ceph-mon[31453]: pgmap v7528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:32 smithi153 ceph-mon[36569]: pgmap v7529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:32 smithi104 ceph-mon[36624]: pgmap v7529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:32 smithi081 ceph-mon[31453]: pgmap v7529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:33.145+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:02:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:33.145+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:02:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:33.145+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:02:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:34 smithi153 ceph-mon[36569]: pgmap v7530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:34 smithi104 ceph-mon[36624]: pgmap v7530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:34 smithi081 ceph-mon[31453]: pgmap v7530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:36 smithi153 ceph-mon[36569]: pgmap v7531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:36 smithi104 ceph-mon[36624]: pgmap v7531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:36 smithi081 ceph-mon[31453]: pgmap v7531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:38 smithi153 ceph-mon[36569]: pgmap v7532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:38 smithi104 ceph-mon[36624]: pgmap v7532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:38 smithi081 ceph-mon[31453]: pgmap v7532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:40 smithi153 ceph-mon[36569]: pgmap v7533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:40 smithi104 ceph-mon[36624]: pgmap v7533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:40 smithi081 ceph-mon[31453]: pgmap v7533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:42 smithi153 ceph-mon[36569]: pgmap v7534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:42 smithi104 ceph-mon[36624]: pgmap v7534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:42 smithi081 ceph-mon[31453]: pgmap v7534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:44 smithi153 ceph-mon[36569]: pgmap v7535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:44 smithi104 ceph-mon[36624]: pgmap v7535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:44 smithi081 ceph-mon[31453]: pgmap v7535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:46 smithi104 ceph-mon[36624]: pgmap v7536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:46 smithi081 ceph-mon[31453]: pgmap v7536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:46 smithi153 ceph-mon[36569]: pgmap v7536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:48.157+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:02:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:48.157+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:02:48.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:02:48.157+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:02:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:48 smithi104 ceph-mon[36624]: pgmap v7537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:48 smithi081 ceph-mon[31453]: pgmap v7537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:48 smithi153 ceph-mon[36569]: pgmap v7537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:50 smithi104 ceph-mon[36624]: pgmap v7538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:50 smithi081 ceph-mon[31453]: pgmap v7538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:50 smithi153 ceph-mon[36569]: pgmap v7538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:52 smithi104 ceph-mon[36624]: pgmap v7539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:52 smithi081 ceph-mon[31453]: pgmap v7539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:52 smithi153 ceph-mon[36569]: pgmap v7539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:54 smithi104 ceph-mon[36624]: pgmap v7540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:54 smithi081 ceph-mon[31453]: pgmap v7540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:54 smithi153 ceph-mon[36569]: pgmap v7540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:56 smithi081 ceph-mon[31453]: pgmap v7541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:56 smithi153 ceph-mon[36569]: pgmap v7541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:56 smithi104 ceph-mon[36624]: pgmap v7541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:02:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:02:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:02:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:58 smithi081 ceph-mon[31453]: pgmap v7542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:02:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:02:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:02:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:58 smithi153 ceph-mon[36569]: pgmap v7542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:02:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:02:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:02:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:58 smithi104 ceph-mon[36624]: pgmap v7542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:02:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:02:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:02:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:59 smithi081 ceph-mon[31453]: pgmap v7543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:02:59 smithi081 ceph-mon[31453]: pgmap v7544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:59 smithi153 ceph-mon[36569]: pgmap v7543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:02:59 smithi153 ceph-mon[36569]: pgmap v7544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:59.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:59 smithi104 ceph-mon[36624]: pgmap v7543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:02:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:02:59 smithi104 ceph-mon[36624]: pgmap v7544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:01 smithi081 ceph-mon[31453]: pgmap v7545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:01 smithi153 ceph-mon[36569]: pgmap v7545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:01 smithi104 ceph-mon[36624]: pgmap v7545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:02 smithi153 ceph-mon[36569]: pgmap v7546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:02 smithi104 ceph-mon[36624]: pgmap v7546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:02 smithi081 ceph-mon[31453]: pgmap v7546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:03.172+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:03:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:03.173+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:03:03.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:03.173+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:03:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:05 smithi104 ceph-mon[36624]: pgmap v7547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:05 smithi081 ceph-mon[31453]: pgmap v7547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:05 smithi153 ceph-mon[36569]: pgmap v7547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:07 smithi104 ceph-mon[36624]: pgmap v7548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:07 smithi081 ceph-mon[31453]: pgmap v7548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:07 smithi153 ceph-mon[36569]: pgmap v7548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:09 smithi104 ceph-mon[36624]: pgmap v7549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:09.358 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:09 smithi153 ceph-mon[36569]: pgmap v7549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:09 smithi081 ceph-mon[31453]: pgmap v7549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:11 smithi104 ceph-mon[36624]: pgmap v7550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:11 smithi081 ceph-mon[31453]: pgmap v7550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:11 smithi153 ceph-mon[36569]: pgmap v7550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:13 smithi104 ceph-mon[36624]: pgmap v7551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:13 smithi081 ceph-mon[31453]: pgmap v7551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:13 smithi153 ceph-mon[36569]: pgmap v7551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:15 smithi081 ceph-mon[31453]: pgmap v7552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:15 smithi153 ceph-mon[36569]: pgmap v7552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:15 smithi104 ceph-mon[36624]: pgmap v7552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:17 smithi081 ceph-mon[31453]: pgmap v7553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:17 smithi153 ceph-mon[36569]: pgmap v7553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:17 smithi104 ceph-mon[36624]: pgmap v7553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:18.188+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:03:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:18.188+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:03:18.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:18.188+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:03:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:19 smithi081 ceph-mon[31453]: pgmap v7554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:19 smithi153 ceph-mon[36569]: pgmap v7554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:19 smithi104 ceph-mon[36624]: pgmap v7554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:21 smithi081 ceph-mon[31453]: pgmap v7555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:21 smithi153 ceph-mon[36569]: pgmap v7555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:21 smithi104 ceph-mon[36624]: pgmap v7555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:23.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:23 smithi081 ceph-mon[31453]: pgmap v7556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:23 smithi153 ceph-mon[36569]: pgmap v7556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:23 smithi104 ceph-mon[36624]: pgmap v7556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:25 smithi081 ceph-mon[31453]: pgmap v7557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:25 smithi153 ceph-mon[36569]: pgmap v7557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:25 smithi104 ceph-mon[36624]: pgmap v7557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:27 smithi081 ceph-mon[31453]: pgmap v7558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:27 smithi153 ceph-mon[36569]: pgmap v7558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:27 smithi104 ceph-mon[36624]: pgmap v7558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:29 smithi081 ceph-mon[31453]: pgmap v7559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:29 smithi153 ceph-mon[36569]: pgmap v7559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:29 smithi104 ceph-mon[36624]: pgmap v7559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:31 smithi081 ceph-mon[31453]: pgmap v7560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:31 smithi153 ceph-mon[36569]: pgmap v7560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:31 smithi104 ceph-mon[36624]: pgmap v7560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:33.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:33.203+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:03:33.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:33.203+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:03:33.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:33.203+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:03:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:33 smithi081 ceph-mon[31453]: pgmap v7561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:33 smithi153 ceph-mon[36569]: pgmap v7561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:33 smithi104 ceph-mon[36624]: pgmap v7561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:35 smithi081 ceph-mon[31453]: pgmap v7562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:35 smithi153 ceph-mon[36569]: pgmap v7562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:35 smithi104 ceph-mon[36624]: pgmap v7562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:37 smithi081 ceph-mon[31453]: pgmap v7563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:37 smithi153 ceph-mon[36569]: pgmap v7563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:37.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:37 smithi104 ceph-mon[36624]: pgmap v7563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:39 smithi081 ceph-mon[31453]: pgmap v7564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:39 smithi153 ceph-mon[36569]: pgmap v7564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:39 smithi104 ceph-mon[36624]: pgmap v7564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:41 smithi081 ceph-mon[31453]: pgmap v7565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:41 smithi153 ceph-mon[36569]: pgmap v7565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:41 smithi104 ceph-mon[36624]: pgmap v7565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:43 smithi081 ceph-mon[31453]: pgmap v7566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:43 smithi153 ceph-mon[36569]: pgmap v7566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:43.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:43 smithi104 ceph-mon[36624]: pgmap v7566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:45 smithi153 ceph-mon[36569]: pgmap v7567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:45 smithi104 ceph-mon[36624]: pgmap v7567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:45 smithi081 ceph-mon[31453]: pgmap v7567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:47 smithi153 ceph-mon[36569]: pgmap v7568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:47 smithi104 ceph-mon[36624]: pgmap v7568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:47 smithi081 ceph-mon[31453]: pgmap v7568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:48.209+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:03:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:48.209+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:03:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:03:48.209+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:03:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:49 smithi153 ceph-mon[36569]: pgmap v7569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:49 smithi104 ceph-mon[36624]: pgmap v7569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:49 smithi081 ceph-mon[31453]: pgmap v7569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:51.458 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:51 smithi081 ceph-mon[31453]: pgmap v7570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:51 smithi153 ceph-mon[36569]: pgmap v7570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:51.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:51 smithi104 ceph-mon[36624]: pgmap v7570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:53 smithi153 ceph-mon[36569]: pgmap v7571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:53.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:53 smithi104 ceph-mon[36624]: pgmap v7571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:53 smithi081 ceph-mon[31453]: pgmap v7571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:55 smithi153 ceph-mon[36569]: pgmap v7572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:55.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:55 smithi104 ceph-mon[36624]: pgmap v7572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:55.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:55 smithi081 ceph-mon[31453]: pgmap v7572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:57 smithi153 ceph-mon[36569]: pgmap v7573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:57 smithi104 ceph-mon[36624]: pgmap v7573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:57 smithi081 ceph-mon[31453]: pgmap v7573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:59 smithi153 ceph-mon[36569]: pgmap v7574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:03:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:03:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:59 smithi104 ceph-mon[36624]: pgmap v7574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:03:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:03:59.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:59 smithi081 ceph-mon[31453]: pgmap v7574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:03:59.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:03:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:04:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:04:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:04:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:04:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:04:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:04:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:04:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:04:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:04:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:04:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:01 smithi153 ceph-mon[36569]: pgmap v7575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:01 smithi153 ceph-mon[36569]: pgmap v7576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:01 smithi104 ceph-mon[36624]: pgmap v7575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:01 smithi104 ceph-mon[36624]: pgmap v7576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:01 smithi081 ceph-mon[31453]: pgmap v7575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:01 smithi081 ceph-mon[31453]: pgmap v7576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:03.225+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:04:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:03.225+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:04:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:03.225+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:04:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:03 smithi153 ceph-mon[36569]: pgmap v7577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:03 smithi104 ceph-mon[36624]: pgmap v7577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:03.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:03 smithi081 ceph-mon[31453]: pgmap v7577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:05 smithi153 ceph-mon[36569]: pgmap v7578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:05 smithi104 ceph-mon[36624]: pgmap v7578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:05 smithi081 ceph-mon[31453]: pgmap v7578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:07 smithi153 ceph-mon[36569]: pgmap v7579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:07 smithi104 ceph-mon[36624]: pgmap v7579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:07 smithi081 ceph-mon[31453]: pgmap v7579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:09 smithi153 ceph-mon[36569]: pgmap v7580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:09 smithi104 ceph-mon[36624]: pgmap v7580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:09 smithi081 ceph-mon[31453]: pgmap v7580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:11 smithi153 ceph-mon[36569]: pgmap v7581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:11 smithi104 ceph-mon[36624]: pgmap v7581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:11 smithi081 ceph-mon[31453]: pgmap v7581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:13 smithi153 ceph-mon[36569]: pgmap v7582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:13 smithi104 ceph-mon[36624]: pgmap v7582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:13 smithi081 ceph-mon[31453]: pgmap v7582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:14.961+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:04:14.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:14.961+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:04:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:14.962+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:04:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:14.962+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:04:15.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:15 smithi153 ceph-mon[36569]: pgmap v7583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:15 smithi104 ceph-mon[36624]: pgmap v7583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:15 smithi081 ceph-mon[31453]: pgmap v7583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:17 smithi081 ceph-mon[31453]: pgmap v7584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:17 smithi153 ceph-mon[36569]: pgmap v7584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:17 smithi104 ceph-mon[36624]: pgmap v7584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:18.237 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:18.237+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:04:18.237 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:18.237+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:04:18.237 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:18.237+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:04:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:19 smithi153 ceph-mon[36569]: pgmap v7585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:19 smithi104 ceph-mon[36624]: pgmap v7585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:19 smithi081 ceph-mon[31453]: pgmap v7585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:21 smithi153 ceph-mon[36569]: pgmap v7586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:21 smithi104 ceph-mon[36624]: pgmap v7586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:21 smithi081 ceph-mon[31453]: pgmap v7586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:23 smithi081 ceph-mon[31453]: pgmap v7587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:23 smithi153 ceph-mon[36569]: pgmap v7587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:23.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:23 smithi104 ceph-mon[36624]: pgmap v7587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:25 smithi081 ceph-mon[31453]: pgmap v7588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:25 smithi153 ceph-mon[36569]: pgmap v7588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:25 smithi104 ceph-mon[36624]: pgmap v7588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:27 smithi081 ceph-mon[31453]: pgmap v7589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:27 smithi153 ceph-mon[36569]: pgmap v7589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:27.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:27 smithi104 ceph-mon[36624]: pgmap v7589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:29 smithi081 ceph-mon[31453]: pgmap v7590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:29 smithi153 ceph-mon[36569]: pgmap v7590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:29 smithi104 ceph-mon[36624]: pgmap v7590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:31 smithi081 ceph-mon[31453]: pgmap v7591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:31 smithi153 ceph-mon[36569]: pgmap v7591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:31 smithi104 ceph-mon[36624]: pgmap v7591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:33.240+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:04:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:33.240+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:04:33.240 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:33.240+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:04:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:33 smithi081 ceph-mon[31453]: pgmap v7592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:33 smithi153 ceph-mon[36569]: pgmap v7592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:33.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:33 smithi104 ceph-mon[36624]: pgmap v7592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:35 smithi081 ceph-mon[31453]: pgmap v7593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:35 smithi153 ceph-mon[36569]: pgmap v7593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:35 smithi104 ceph-mon[36624]: pgmap v7593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:37.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:37 smithi081 ceph-mon[31453]: pgmap v7594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:37 smithi153 ceph-mon[36569]: pgmap v7594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:37 smithi104 ceph-mon[36624]: pgmap v7594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:39.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:39 smithi081 ceph-mon[31453]: pgmap v7595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:39 smithi153 ceph-mon[36569]: pgmap v7595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:39 smithi104 ceph-mon[36624]: pgmap v7595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:41 smithi081 ceph-mon[31453]: pgmap v7596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:41 smithi153 ceph-mon[36569]: pgmap v7596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:41.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:41 smithi104 ceph-mon[36624]: pgmap v7596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:42 smithi153 ceph-mon[36569]: pgmap v7597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:42 smithi104 ceph-mon[36624]: pgmap v7597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:42 smithi081 ceph-mon[31453]: pgmap v7597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:44 smithi081 ceph-mon[31453]: pgmap v7598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:44 smithi153 ceph-mon[36569]: pgmap v7598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:44 smithi104 ceph-mon[36624]: pgmap v7598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:46 smithi081 ceph-mon[31453]: pgmap v7599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:46 smithi153 ceph-mon[36569]: pgmap v7599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:46 smithi104 ceph-mon[36624]: pgmap v7599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:48.256+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:04:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:48.256+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:04:48.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:04:48.256+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:04:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:48 smithi081 ceph-mon[31453]: pgmap v7600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:48 smithi153 ceph-mon[36569]: pgmap v7600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:48 smithi104 ceph-mon[36624]: pgmap v7600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:50 smithi153 ceph-mon[36569]: pgmap v7601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:50 smithi104 ceph-mon[36624]: pgmap v7601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:50 smithi081 ceph-mon[31453]: pgmap v7601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:52.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:52 smithi081 ceph-mon[31453]: pgmap v7602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:52 smithi153 ceph-mon[36569]: pgmap v7602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:52 smithi104 ceph-mon[36624]: pgmap v7602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:54 smithi153 ceph-mon[36569]: pgmap v7603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:54 smithi104 ceph-mon[36624]: pgmap v7603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:54 smithi081 ceph-mon[31453]: pgmap v7603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:56 smithi153 ceph-mon[36569]: pgmap v7604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:56 smithi104 ceph-mon[36624]: pgmap v7604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:56 smithi081 ceph-mon[31453]: pgmap v7604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:04:58 smithi153 ceph-mon[36569]: pgmap v7605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:04:58 smithi104 ceph-mon[36624]: pgmap v7605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:04:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:04:58 smithi081 ceph-mon[31453]: pgmap v7605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:00.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:00 smithi153 ceph-mon[36569]: pgmap v7606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:00.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:05:00.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:00.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:00.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:00.912 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:00 smithi104 ceph-mon[36624]: pgmap v7606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:05:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:00 smithi081 ceph-mon[31453]: pgmap v7606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:05:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:03.271+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:05:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:03.271+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:05:03.271 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:03.271+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:05:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:03 smithi081 ceph-mon[31453]: pgmap v7607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:03 smithi153 ceph-mon[36569]: pgmap v7607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:03 smithi104 ceph-mon[36624]: pgmap v7607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:05 smithi153 ceph-mon[36569]: pgmap v7608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:05 smithi104 ceph-mon[36624]: pgmap v7608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:05 smithi081 ceph-mon[31453]: pgmap v7608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:05:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:05:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:06.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:05:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:05:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:05:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:05:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:05:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:07 smithi153 ceph-mon[36569]: pgmap v7609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:07 smithi153 ceph-mon[36569]: pgmap v7610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:07 smithi153 ceph-mon[36569]: pgmap v7611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:07 smithi104 ceph-mon[36624]: pgmap v7609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:07 smithi104 ceph-mon[36624]: pgmap v7610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:07 smithi104 ceph-mon[36624]: pgmap v7611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:07 smithi081 ceph-mon[31453]: pgmap v7609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:07 smithi081 ceph-mon[31453]: pgmap v7610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:07 smithi081 ceph-mon[31453]: pgmap v7611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:09 smithi153 ceph-mon[36569]: pgmap v7612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:09 smithi104 ceph-mon[36624]: pgmap v7612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:09 smithi081 ceph-mon[31453]: pgmap v7612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:11 smithi153 ceph-mon[36569]: pgmap v7613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:11 smithi104 ceph-mon[36624]: pgmap v7613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:11 smithi081 ceph-mon[31453]: pgmap v7613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:13 smithi153 ceph-mon[36569]: pgmap v7614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:13 smithi104 ceph-mon[36624]: pgmap v7614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:13 smithi081 ceph-mon[31453]: pgmap v7614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:15 smithi153 ceph-mon[36569]: pgmap v7615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:15 smithi104 ceph-mon[36624]: pgmap v7615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:15 smithi081 ceph-mon[31453]: pgmap v7615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:17 smithi153 ceph-mon[36569]: pgmap v7616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:17 smithi104 ceph-mon[36624]: pgmap v7616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:17.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:17 smithi081 ceph-mon[31453]: pgmap v7616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:18.286+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:05:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:18.287+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:05:18.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:18.287+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:05:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:19 smithi153 ceph-mon[36569]: pgmap v7617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:19 smithi104 ceph-mon[36624]: pgmap v7617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:19 smithi081 ceph-mon[31453]: pgmap v7617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:21 smithi153 ceph-mon[36569]: pgmap v7618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:21 smithi104 ceph-mon[36624]: pgmap v7618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:21 smithi081 ceph-mon[31453]: pgmap v7618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:23 smithi153 ceph-mon[36569]: pgmap v7619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:23 smithi104 ceph-mon[36624]: pgmap v7619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:23.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:23 smithi081 ceph-mon[31453]: pgmap v7619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:25 smithi153 ceph-mon[36569]: pgmap v7620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:25 smithi104 ceph-mon[36624]: pgmap v7620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:25 smithi081 ceph-mon[31453]: pgmap v7620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:27 smithi153 ceph-mon[36569]: pgmap v7621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:27 smithi104 ceph-mon[36624]: pgmap v7621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:27 smithi081 ceph-mon[31453]: pgmap v7621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:29 smithi153 ceph-mon[36569]: pgmap v7622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:29 smithi104 ceph-mon[36624]: pgmap v7622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:29 smithi081 ceph-mon[31453]: pgmap v7622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:31 smithi153 ceph-mon[36569]: pgmap v7623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:31.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:31 smithi104 ceph-mon[36624]: pgmap v7623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:31 smithi081 ceph-mon[31453]: pgmap v7623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:33.302+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:05:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:33.302+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:05:33.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:33.302+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:05:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:33 smithi153 ceph-mon[36569]: pgmap v7624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:33 smithi104 ceph-mon[36624]: pgmap v7624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:33 smithi081 ceph-mon[31453]: pgmap v7624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:35 smithi153 ceph-mon[36569]: pgmap v7625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:35 smithi104 ceph-mon[36624]: pgmap v7625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:35 smithi081 ceph-mon[31453]: pgmap v7625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:37.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:37 smithi104 ceph-mon[36624]: pgmap v7626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:37.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:37 smithi081 ceph-mon[31453]: pgmap v7626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:37 smithi153 ceph-mon[36569]: pgmap v7626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:39.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:39 smithi081 ceph-mon[31453]: pgmap v7627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:39 smithi153 ceph-mon[36569]: pgmap v7627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:39 smithi104 ceph-mon[36624]: pgmap v7627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:41 smithi081 ceph-mon[31453]: pgmap v7628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:41 smithi153 ceph-mon[36569]: pgmap v7628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:41 smithi104 ceph-mon[36624]: pgmap v7628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:43.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:43 smithi081 ceph-mon[31453]: pgmap v7629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:43 smithi153 ceph-mon[36569]: pgmap v7629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:43.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:43 smithi104 ceph-mon[36624]: pgmap v7629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:45 smithi081 ceph-mon[31453]: pgmap v7630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:45 smithi153 ceph-mon[36569]: pgmap v7630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:45 smithi104 ceph-mon[36624]: pgmap v7630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:47.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:47 smithi081 ceph-mon[31453]: pgmap v7631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:47 smithi153 ceph-mon[36569]: pgmap v7631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:47.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:47 smithi104 ceph-mon[36624]: pgmap v7631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:48.318+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:05:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:48.318+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:05:48.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:05:48.318+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:05:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:48 smithi153 ceph-mon[36569]: pgmap v7632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:48.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:48 smithi104 ceph-mon[36624]: pgmap v7632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:48 smithi081 ceph-mon[31453]: pgmap v7632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:51 smithi153 ceph-mon[36569]: pgmap v7633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:51.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:51 smithi104 ceph-mon[36624]: pgmap v7633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:51.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:51 smithi081 ceph-mon[31453]: pgmap v7633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:53 smithi153 ceph-mon[36569]: pgmap v7634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:53.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:53 smithi104 ceph-mon[36624]: pgmap v7634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:53.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:53 smithi081 ceph-mon[31453]: pgmap v7634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:55 smithi153 ceph-mon[36569]: pgmap v7635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:55.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:55 smithi104 ceph-mon[36624]: pgmap v7635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:55.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:55 smithi081 ceph-mon[31453]: pgmap v7635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:57 smithi153 ceph-mon[36569]: pgmap v7636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:57.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:57 smithi104 ceph-mon[36624]: pgmap v7636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:57 smithi081 ceph-mon[31453]: pgmap v7636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:05:59 smithi153 ceph-mon[36569]: pgmap v7637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:59.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:05:59 smithi104 ceph-mon[36624]: pgmap v7637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:05:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:05:59 smithi081 ceph-mon[31453]: pgmap v7637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:01 smithi153 ceph-mon[36569]: pgmap v7638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:01.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:01 smithi104 ceph-mon[36624]: pgmap v7638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:01.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:01 smithi081 ceph-mon[31453]: pgmap v7638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:03.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:03.333+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:06:03.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:03.333+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:06:03.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:03.333+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:06:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:03 smithi153 ceph-mon[36569]: pgmap v7639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:03 smithi104 ceph-mon[36624]: pgmap v7639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:03.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:03 smithi081 ceph-mon[31453]: pgmap v7639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:05 smithi153 ceph-mon[36569]: pgmap v7640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:05 smithi104 ceph-mon[36624]: pgmap v7640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:05 smithi081 ceph-mon[31453]: pgmap v7640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:07 smithi153 ceph-mon[36569]: pgmap v7641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:06:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:07 smithi104 ceph-mon[36624]: pgmap v7641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:06:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:07 smithi081 ceph-mon[31453]: pgmap v7641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:06:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:06:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:06:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:06:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:06:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:08.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:06:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:06:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:06:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:09 smithi153 ceph-mon[36569]: pgmap v7642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:09 smithi153 ceph-mon[36569]: pgmap v7643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:09 smithi104 ceph-mon[36624]: pgmap v7642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:09 smithi104 ceph-mon[36624]: pgmap v7643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:09 smithi081 ceph-mon[31453]: pgmap v7642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:09 smithi081 ceph-mon[31453]: pgmap v7643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:11 smithi104 ceph-mon[36624]: pgmap v7644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:11 smithi081 ceph-mon[31453]: pgmap v7644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:11 smithi153 ceph-mon[36569]: pgmap v7644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:13 smithi104 ceph-mon[36624]: pgmap v7645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:13 smithi081 ceph-mon[31453]: pgmap v7645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:13 smithi153 ceph-mon[36569]: pgmap v7645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:15 smithi104 ceph-mon[36624]: pgmap v7646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:15 smithi081 ceph-mon[31453]: pgmap v7646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:15 smithi153 ceph-mon[36569]: pgmap v7646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:17 smithi104 ceph-mon[36624]: pgmap v7647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:17.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:17 smithi081 ceph-mon[31453]: pgmap v7647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:17 smithi153 ceph-mon[36569]: pgmap v7647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:18.343+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:06:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:18.344+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:06:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:18.344+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:06:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:19 smithi104 ceph-mon[36624]: pgmap v7648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:19 smithi081 ceph-mon[31453]: pgmap v7648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:19 smithi153 ceph-mon[36569]: pgmap v7648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:21 smithi081 ceph-mon[31453]: pgmap v7649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:21 smithi153 ceph-mon[36569]: pgmap v7649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:21.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:21 smithi104 ceph-mon[36624]: pgmap v7649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:23.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:23 smithi081 ceph-mon[31453]: pgmap v7650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:23 smithi153 ceph-mon[36569]: pgmap v7650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:23.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:23 smithi104 ceph-mon[36624]: pgmap v7650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:25 smithi081 ceph-mon[31453]: pgmap v7651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:25 smithi153 ceph-mon[36569]: pgmap v7651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:25 smithi104 ceph-mon[36624]: pgmap v7651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:27 smithi081 ceph-mon[31453]: pgmap v7652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:27 smithi153 ceph-mon[36569]: pgmap v7652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:27.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:27 smithi104 ceph-mon[36624]: pgmap v7652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:29 smithi081 ceph-mon[31453]: pgmap v7653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:29 smithi153 ceph-mon[36569]: pgmap v7653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:29.874 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:29 smithi104 ceph-mon[36624]: pgmap v7653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:31 smithi153 ceph-mon[36569]: pgmap v7654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:31 smithi104 ceph-mon[36624]: pgmap v7654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:31.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:31 smithi081 ceph-mon[31453]: pgmap v7654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:33.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:33.359+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:06:33.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:33.359+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:06:33.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:33.359+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:06:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:33 smithi153 ceph-mon[36569]: pgmap v7655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:33.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:33 smithi104 ceph-mon[36624]: pgmap v7655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:33.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:33 smithi081 ceph-mon[31453]: pgmap v7655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:34 smithi153 ceph-mon[36569]: pgmap v7656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:34 smithi104 ceph-mon[36624]: pgmap v7656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:34 smithi081 ceph-mon[31453]: pgmap v7656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:36 smithi104 ceph-mon[36624]: pgmap v7657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:36 smithi081 ceph-mon[31453]: pgmap v7657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:36 smithi153 ceph-mon[36569]: pgmap v7657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:38 smithi104 ceph-mon[36624]: pgmap v7658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:39.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:38 smithi153 ceph-mon[36569]: pgmap v7658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:38 smithi081 ceph-mon[31453]: pgmap v7658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:40 smithi104 ceph-mon[36624]: pgmap v7659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:40 smithi081 ceph-mon[31453]: pgmap v7659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:40 smithi153 ceph-mon[36569]: pgmap v7659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:42 smithi104 ceph-mon[36624]: pgmap v7660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:42 smithi081 ceph-mon[31453]: pgmap v7660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:42 smithi153 ceph-mon[36569]: pgmap v7660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:44 smithi081 ceph-mon[31453]: pgmap v7661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:44 smithi153 ceph-mon[36569]: pgmap v7661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:44 smithi104 ceph-mon[36624]: pgmap v7661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:47.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:46 smithi104 ceph-mon[36624]: pgmap v7662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:46 smithi081 ceph-mon[31453]: pgmap v7662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:46 smithi153 ceph-mon[36569]: pgmap v7662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:48.374+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:06:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:48.374+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:06:48.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:06:48.374+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:06:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:48 smithi081 ceph-mon[31453]: pgmap v7663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:49.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:48 smithi153 ceph-mon[36569]: pgmap v7663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:48 smithi104 ceph-mon[36624]: pgmap v7663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:50 smithi081 ceph-mon[31453]: pgmap v7664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:50 smithi153 ceph-mon[36569]: pgmap v7664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:50 smithi104 ceph-mon[36624]: pgmap v7664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:52 smithi081 ceph-mon[31453]: pgmap v7665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:52 smithi153 ceph-mon[36569]: pgmap v7665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:52 smithi104 ceph-mon[36624]: pgmap v7665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:54 smithi081 ceph-mon[31453]: pgmap v7666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:54 smithi153 ceph-mon[36569]: pgmap v7666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:54 smithi104 ceph-mon[36624]: pgmap v7666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:56 smithi081 ceph-mon[31453]: pgmap v7667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:56 smithi153 ceph-mon[36569]: pgmap v7667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:57.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:56 smithi104 ceph-mon[36624]: pgmap v7667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:06:58 smithi081 ceph-mon[31453]: pgmap v7668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:06:58 smithi153 ceph-mon[36569]: pgmap v7668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:06:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:06:58 smithi104 ceph-mon[36624]: pgmap v7668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:00 smithi081 ceph-mon[31453]: pgmap v7669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:00 smithi153 ceph-mon[36569]: pgmap v7669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:01.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:00 smithi104 ceph-mon[36624]: pgmap v7669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:02 smithi081 ceph-mon[31453]: pgmap v7670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:03.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:02 smithi153 ceph-mon[36569]: pgmap v7670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:02 smithi104 ceph-mon[36624]: pgmap v7670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:03.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:03.390+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:07:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:03.390+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:07:03.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:03.390+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:07:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:04 smithi081 ceph-mon[31453]: pgmap v7671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:04 smithi153 ceph-mon[36569]: pgmap v7671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:04 smithi104 ceph-mon[36624]: pgmap v7671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:06 smithi081 ceph-mon[31453]: pgmap v7672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:06 smithi153 ceph-mon[36569]: pgmap v7672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:07.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:06 smithi104 ceph-mon[36624]: pgmap v7672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:08.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:07:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:07:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:07:09.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:09 smithi153 ceph-mon[36569]: pgmap v7673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:09 smithi104 ceph-mon[36624]: pgmap v7673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:09.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:09 smithi081 ceph-mon[31453]: pgmap v7673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:11 smithi104 ceph-mon[36624]: pgmap v7674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:07:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:07:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:11 smithi104 ceph-mon[36624]: pgmap v7675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:11 smithi104 ceph-mon[36624]: pgmap v7676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:11 smithi081 ceph-mon[31453]: pgmap v7674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:07:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:07:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:11 smithi081 ceph-mon[31453]: pgmap v7675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:11 smithi081 ceph-mon[31453]: pgmap v7676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:11 smithi153 ceph-mon[36569]: pgmap v7674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:07:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:07:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:11 smithi153 ceph-mon[36569]: pgmap v7675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:11 smithi153 ceph-mon[36569]: pgmap v7676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:07:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:13 smithi153 ceph-mon[36569]: pgmap v7677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:13 smithi104 ceph-mon[36624]: pgmap v7677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:13 smithi081 ceph-mon[31453]: pgmap v7677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:14.993+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:07:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:14.993+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:07:14.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:14.993+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:07:14.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:14.993+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:07:14.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:14.993+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:07:14.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:14.994+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:07:14.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:14.994+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:07:14.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:14.994+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:07:15.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:15 smithi153 ceph-mon[36569]: pgmap v7678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:15 smithi104 ceph-mon[36624]: pgmap v7678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:15 smithi081 ceph-mon[31453]: pgmap v7678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:17 smithi153 ceph-mon[36569]: pgmap v7679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:17 smithi104 ceph-mon[36624]: pgmap v7679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:17.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:17 smithi081 ceph-mon[31453]: pgmap v7679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:18.392+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:07:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:18.392+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:07:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:18.392+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:07:19.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:19 smithi153 ceph-mon[36569]: pgmap v7680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:19 smithi104 ceph-mon[36624]: pgmap v7680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:19 smithi081 ceph-mon[31453]: pgmap v7680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:21 smithi153 ceph-mon[36569]: pgmap v7681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:21 smithi104 ceph-mon[36624]: pgmap v7681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:21 smithi081 ceph-mon[31453]: pgmap v7681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:23 smithi153 ceph-mon[36569]: pgmap v7682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:23 smithi104 ceph-mon[36624]: pgmap v7682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:23.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:23 smithi081 ceph-mon[31453]: pgmap v7682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:25 smithi153 ceph-mon[36569]: pgmap v7683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:25 smithi104 ceph-mon[36624]: pgmap v7683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:25 smithi081 ceph-mon[31453]: pgmap v7683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:27 smithi153 ceph-mon[36569]: pgmap v7684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:27 smithi104 ceph-mon[36624]: pgmap v7684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:27 smithi081 ceph-mon[31453]: pgmap v7684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:29 smithi153 ceph-mon[36569]: pgmap v7685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:29 smithi104 ceph-mon[36624]: pgmap v7685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:29 smithi081 ceph-mon[31453]: pgmap v7685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:31 smithi153 ceph-mon[36569]: pgmap v7686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:31.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:31 smithi104 ceph-mon[36624]: pgmap v7686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:31.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:31 smithi081 ceph-mon[31453]: pgmap v7686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:33.406+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:07:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:33.406+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:07:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:33.406+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:07:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:33 smithi153 ceph-mon[36569]: pgmap v7687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:33 smithi104 ceph-mon[36624]: pgmap v7687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:33 smithi081 ceph-mon[31453]: pgmap v7687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:35 smithi153 ceph-mon[36569]: pgmap v7688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:35 smithi104 ceph-mon[36624]: pgmap v7688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:35 smithi081 ceph-mon[31453]: pgmap v7688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:37.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:37 smithi153 ceph-mon[36569]: pgmap v7689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:37.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:37 smithi104 ceph-mon[36624]: pgmap v7689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:37.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:37 smithi081 ceph-mon[31453]: pgmap v7689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:39.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:39 smithi153 ceph-mon[36569]: pgmap v7690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:39.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:39 smithi104 ceph-mon[36624]: pgmap v7690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:39.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:39 smithi081 ceph-mon[31453]: pgmap v7690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:41.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:41 smithi153 ceph-mon[36569]: pgmap v7691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:41 smithi104 ceph-mon[36624]: pgmap v7691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:41 smithi081 ceph-mon[31453]: pgmap v7691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:43.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:43 smithi153 ceph-mon[36569]: pgmap v7692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:43.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:43 smithi104 ceph-mon[36624]: pgmap v7692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:43.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:43 smithi081 ceph-mon[31453]: pgmap v7692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:45 smithi153 ceph-mon[36569]: pgmap v7693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:45.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:45 smithi104 ceph-mon[36624]: pgmap v7693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:45 smithi081 ceph-mon[31453]: pgmap v7693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:47 smithi153 ceph-mon[36569]: pgmap v7694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:47 smithi104 ceph-mon[36624]: pgmap v7694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:47.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:47 smithi081 ceph-mon[31453]: pgmap v7694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:48.410+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:07:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:48.410+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:07:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:07:48.410+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:07:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:49 smithi153 ceph-mon[36569]: pgmap v7695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:49 smithi104 ceph-mon[36624]: pgmap v7695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:49.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:49 smithi081 ceph-mon[31453]: pgmap v7695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:51 smithi153 ceph-mon[36569]: pgmap v7696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:51.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:51 smithi104 ceph-mon[36624]: pgmap v7696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:51.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:51 smithi081 ceph-mon[31453]: pgmap v7696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:53 smithi153 ceph-mon[36569]: pgmap v7697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:53.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:53 smithi104 ceph-mon[36624]: pgmap v7697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:53.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:53 smithi081 ceph-mon[31453]: pgmap v7697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:55.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:55 smithi081 ceph-mon[31453]: pgmap v7698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:55.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:55 smithi153 ceph-mon[36569]: pgmap v7698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:55.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:55 smithi104 ceph-mon[36624]: pgmap v7698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:57 smithi153 ceph-mon[36569]: pgmap v7699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:57.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:57 smithi104 ceph-mon[36624]: pgmap v7699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:57 smithi081 ceph-mon[31453]: pgmap v7699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:07:59 smithi153 ceph-mon[36569]: pgmap v7700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:59.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:07:59 smithi104 ceph-mon[36624]: pgmap v7700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:07:59 smithi081 ceph-mon[31453]: pgmap v7700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:01 smithi153 ceph-mon[36569]: pgmap v7701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:01.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:01 smithi104 ceph-mon[36624]: pgmap v7701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:01.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:01 smithi081 ceph-mon[31453]: pgmap v7701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:03.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:03.422+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:08:03.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:03.422+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:08:03.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:03.422+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:08:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:03 smithi153 ceph-mon[36569]: pgmap v7702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:03 smithi104 ceph-mon[36624]: pgmap v7702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:03.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:03 smithi081 ceph-mon[31453]: pgmap v7702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:05.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:05 smithi153 ceph-mon[36569]: pgmap v7703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:05 smithi104 ceph-mon[36624]: pgmap v7703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:05 smithi081 ceph-mon[31453]: pgmap v7703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:07 smithi153 ceph-mon[36569]: pgmap v7704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:07 smithi104 ceph-mon[36624]: pgmap v7704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:07 smithi081 ceph-mon[31453]: pgmap v7704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:09 smithi153 ceph-mon[36569]: pgmap v7705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:09 smithi104 ceph-mon[36624]: pgmap v7705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:09 smithi081 ceph-mon[31453]: pgmap v7705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:11 smithi104 ceph-mon[36624]: pgmap v7706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:08:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:11 smithi081 ceph-mon[31453]: pgmap v7706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:08:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:11 smithi153 ceph-mon[36569]: pgmap v7706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:08:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:08:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:08:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: pgmap v7707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: pgmap v7708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:13.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:08:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:08:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: pgmap v7707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: pgmap v7708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: pgmap v7707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: pgmap v7708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:08:15.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:14 smithi153 ceph-mon[36569]: pgmap v7709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:15.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:14 smithi104 ceph-mon[36624]: pgmap v7709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:14 smithi081 ceph-mon[31453]: pgmap v7709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:16 smithi153 ceph-mon[36569]: pgmap v7710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:17.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:16 smithi104 ceph-mon[36624]: pgmap v7710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:16 smithi081 ceph-mon[31453]: pgmap v7710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:18.433+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:08:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:18.433+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:08:18.432 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:18.433+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:08:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:18 smithi153 ceph-mon[36569]: pgmap v7711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:19.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:18 smithi104 ceph-mon[36624]: pgmap v7711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:18 smithi081 ceph-mon[31453]: pgmap v7711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:20 smithi153 ceph-mon[36569]: pgmap v7712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:20 smithi104 ceph-mon[36624]: pgmap v7712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:20 smithi081 ceph-mon[31453]: pgmap v7712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:23.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:23 smithi153 ceph-mon[36569]: pgmap v7713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:23.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:23 smithi104 ceph-mon[36624]: pgmap v7713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:23 smithi081 ceph-mon[31453]: pgmap v7713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:24 smithi153 ceph-mon[36569]: pgmap v7714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:24 smithi104 ceph-mon[36624]: pgmap v7714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:24 smithi081 ceph-mon[31453]: pgmap v7714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:26 smithi153 ceph-mon[36569]: pgmap v7715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:26 smithi104 ceph-mon[36624]: pgmap v7715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:26 smithi081 ceph-mon[31453]: pgmap v7715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:28 smithi153 ceph-mon[36569]: pgmap v7716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:28 smithi104 ceph-mon[36624]: pgmap v7716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:28 smithi081 ceph-mon[31453]: pgmap v7716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:30 smithi153 ceph-mon[36569]: pgmap v7717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:30 smithi104 ceph-mon[36624]: pgmap v7717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:30 smithi081 ceph-mon[31453]: pgmap v7717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:32 smithi104 ceph-mon[36624]: pgmap v7718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:32 smithi081 ceph-mon[31453]: pgmap v7718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:32 smithi153 ceph-mon[36569]: pgmap v7718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:33.445 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:33.445+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:08:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:33.445+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:08:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:33.445+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:08:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:34 smithi104 ceph-mon[36624]: pgmap v7719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:34 smithi081 ceph-mon[31453]: pgmap v7719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:35.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:34 smithi153 ceph-mon[36569]: pgmap v7719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:36 smithi104 ceph-mon[36624]: pgmap v7720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:36 smithi081 ceph-mon[31453]: pgmap v7720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:36 smithi153 ceph-mon[36569]: pgmap v7720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:38 smithi104 ceph-mon[36624]: pgmap v7721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:39.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:38 smithi153 ceph-mon[36569]: pgmap v7721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:38 smithi081 ceph-mon[31453]: pgmap v7721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:40 smithi104 ceph-mon[36624]: pgmap v7722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:40 smithi081 ceph-mon[31453]: pgmap v7722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:40 smithi153 ceph-mon[36569]: pgmap v7722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:42 smithi081 ceph-mon[31453]: pgmap v7723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:43.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:42 smithi153 ceph-mon[36569]: pgmap v7723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:42 smithi104 ceph-mon[36624]: pgmap v7723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:44 smithi081 ceph-mon[31453]: pgmap v7724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:44 smithi153 ceph-mon[36569]: pgmap v7724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:44 smithi104 ceph-mon[36624]: pgmap v7724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:46 smithi081 ceph-mon[31453]: pgmap v7725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:47.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:46 smithi153 ceph-mon[36569]: pgmap v7725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:46 smithi104 ceph-mon[36624]: pgmap v7725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:48.462+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:08:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:48.462+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:08:48.461 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:08:48.462+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:08:49.133 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:48 smithi104 ceph-mon[36624]: pgmap v7726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:48 smithi081 ceph-mon[31453]: pgmap v7726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:48 smithi153 ceph-mon[36569]: pgmap v7726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:50 smithi081 ceph-mon[31453]: pgmap v7727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:50 smithi153 ceph-mon[36569]: pgmap v7727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:50 smithi104 ceph-mon[36624]: pgmap v7727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:52 smithi081 ceph-mon[31453]: pgmap v7728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:53.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:52 smithi153 ceph-mon[36569]: pgmap v7728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:52 smithi104 ceph-mon[36624]: pgmap v7728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:54 smithi081 ceph-mon[31453]: pgmap v7729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:54 smithi153 ceph-mon[36569]: pgmap v7729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:54 smithi104 ceph-mon[36624]: pgmap v7729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:56 smithi081 ceph-mon[31453]: pgmap v7730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:57.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:56 smithi153 ceph-mon[36569]: pgmap v7730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:57.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:56 smithi104 ceph-mon[36624]: pgmap v7730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:08:58 smithi081 ceph-mon[31453]: pgmap v7731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:08:58 smithi153 ceph-mon[36569]: pgmap v7731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:08:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:08:58 smithi104 ceph-mon[36624]: pgmap v7731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:00 smithi081 ceph-mon[31453]: pgmap v7732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:00 smithi153 ceph-mon[36569]: pgmap v7732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:01.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:00 smithi104 ceph-mon[36624]: pgmap v7732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:02 smithi081 ceph-mon[31453]: pgmap v7733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:03.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:02 smithi153 ceph-mon[36569]: pgmap v7733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:02 smithi104 ceph-mon[36624]: pgmap v7733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:03.477+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:09:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:03.477+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:09:03.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:03.477+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:09:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:04 smithi081 ceph-mon[31453]: pgmap v7734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:05.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:04 smithi153 ceph-mon[36569]: pgmap v7734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:04 smithi104 ceph-mon[36624]: pgmap v7734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:06 smithi081 ceph-mon[31453]: pgmap v7735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:06 smithi153 ceph-mon[36569]: pgmap v7735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:07.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:06 smithi104 ceph-mon[36624]: pgmap v7735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:08 smithi081 ceph-mon[31453]: pgmap v7736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:08 smithi153 ceph-mon[36569]: pgmap v7736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:08 smithi104 ceph-mon[36624]: pgmap v7736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:10 smithi153 ceph-mon[36569]: pgmap v7737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:10 smithi104 ceph-mon[36624]: pgmap v7737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:10 smithi081 ceph-mon[31453]: pgmap v7737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:13.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:12 smithi153 ceph-mon[36569]: pgmap v7738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:09:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:12 smithi104 ceph-mon[36624]: pgmap v7738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:09:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:12 smithi081 ceph-mon[31453]: pgmap v7738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:09:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:09:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:09:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:13 smithi153 ceph-mon[36569]: pgmap v7739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:13 smithi153 ceph-mon[36569]: pgmap v7740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:09:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:09:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:09:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:13 smithi104 ceph-mon[36624]: pgmap v7739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:13 smithi104 ceph-mon[36624]: pgmap v7740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:09:14.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:09:14.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:09:14.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:13 smithi081 ceph-mon[31453]: pgmap v7739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:14.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:13 smithi081 ceph-mon[31453]: pgmap v7740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:09:16.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:16 smithi081 ceph-mon[31453]: pgmap v7741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:16 smithi153 ceph-mon[36569]: pgmap v7741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:16 smithi104 ceph-mon[36624]: pgmap v7741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:18.491+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:09:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:18.491+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:09:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:18.491+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:09:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:18 smithi081 ceph-mon[31453]: pgmap v7742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:18.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:18 smithi153 ceph-mon[36569]: pgmap v7742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:18 smithi104 ceph-mon[36624]: pgmap v7742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:20.633 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:20 smithi104 ceph-mon[36624]: pgmap v7743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:20 smithi081 ceph-mon[31453]: pgmap v7743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:20.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:20 smithi153 ceph-mon[36569]: pgmap v7743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:22.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:22 smithi081 ceph-mon[31453]: pgmap v7744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:22 smithi153 ceph-mon[36569]: pgmap v7744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:22 smithi104 ceph-mon[36624]: pgmap v7744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:24 smithi081 ceph-mon[31453]: pgmap v7745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:24.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:24 smithi153 ceph-mon[36569]: pgmap v7745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:24 smithi104 ceph-mon[36624]: pgmap v7745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:26 smithi081 ceph-mon[31453]: pgmap v7746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:26.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:26 smithi153 ceph-mon[36569]: pgmap v7746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:26 smithi104 ceph-mon[36624]: pgmap v7746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:28 smithi081 ceph-mon[31453]: pgmap v7747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:28.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:28 smithi153 ceph-mon[36569]: pgmap v7747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:28 smithi104 ceph-mon[36624]: pgmap v7747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:30 smithi081 ceph-mon[31453]: pgmap v7748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:30.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:30 smithi153 ceph-mon[36569]: pgmap v7748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:30 smithi104 ceph-mon[36624]: pgmap v7748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:32 smithi153 ceph-mon[36569]: pgmap v7749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:32 smithi104 ceph-mon[36624]: pgmap v7749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:32.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:32 smithi081 ceph-mon[31453]: pgmap v7749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:33.506+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:09:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:33.506+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:09:33.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:33.506+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:09:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:34 smithi153 ceph-mon[36569]: pgmap v7750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:34 smithi104 ceph-mon[36624]: pgmap v7750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:34 smithi081 ceph-mon[31453]: pgmap v7750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:36 smithi153 ceph-mon[36569]: pgmap v7751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:36.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:36 smithi104 ceph-mon[36624]: pgmap v7751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:36.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:36 smithi081 ceph-mon[31453]: pgmap v7751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:38.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:38 smithi153 ceph-mon[36569]: pgmap v7752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:38 smithi104 ceph-mon[36624]: pgmap v7752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:38 smithi081 ceph-mon[31453]: pgmap v7752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:40.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:40 smithi153 ceph-mon[36569]: pgmap v7753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:40 smithi104 ceph-mon[36624]: pgmap v7753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:40 smithi081 ceph-mon[31453]: pgmap v7753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:42 smithi153 ceph-mon[36569]: pgmap v7754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:42 smithi104 ceph-mon[36624]: pgmap v7754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:42 smithi081 ceph-mon[31453]: pgmap v7754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:44 smithi081 ceph-mon[31453]: pgmap v7755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:44.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:44 smithi153 ceph-mon[36569]: pgmap v7755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:44 smithi104 ceph-mon[36624]: pgmap v7755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:46.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:46 smithi153 ceph-mon[36569]: pgmap v7756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:46.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:46 smithi104 ceph-mon[36624]: pgmap v7756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:46 smithi081 ceph-mon[31453]: pgmap v7756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:47 smithi153 ceph-mon[36569]: pgmap v7757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:47.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:47 smithi104 ceph-mon[36624]: pgmap v7757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:47.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:47 smithi081 ceph-mon[31453]: pgmap v7757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:48.510+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:09:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:48.510+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:09:48.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:09:48.510+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:09:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:50 smithi081 ceph-mon[31453]: pgmap v7758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:50 smithi153 ceph-mon[36569]: pgmap v7758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:50 smithi104 ceph-mon[36624]: pgmap v7758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:52 smithi081 ceph-mon[31453]: pgmap v7759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:52.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:52 smithi153 ceph-mon[36569]: pgmap v7759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:52 smithi104 ceph-mon[36624]: pgmap v7759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:54.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:54 smithi081 ceph-mon[31453]: pgmap v7760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:54 smithi153 ceph-mon[36569]: pgmap v7760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:54 smithi104 ceph-mon[36624]: pgmap v7760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:56.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:56 smithi081 ceph-mon[31453]: pgmap v7761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:56.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:56 smithi153 ceph-mon[36569]: pgmap v7761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:56 smithi104 ceph-mon[36624]: pgmap v7761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:09:58 smithi081 ceph-mon[31453]: pgmap v7762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:58.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:09:58 smithi153 ceph-mon[36569]: pgmap v7762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:09:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:09:58 smithi104 ceph-mon[36624]: pgmap v7762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:00 smithi081 ceph-mon[31453]: pgmap v7763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:10:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:00 smithi153 ceph-mon[36569]: pgmap v7763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:10:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:00 smithi104 ceph-mon[36624]: pgmap v7763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:10:02.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:02 smithi153 ceph-mon[36569]: pgmap v7764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:02 smithi104 ceph-mon[36624]: pgmap v7764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:02 smithi081 ceph-mon[31453]: pgmap v7764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:03.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:03.521+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:10:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:03.521+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:10:03.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:03.521+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:10:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:04 smithi153 ceph-mon[36569]: pgmap v7765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:04 smithi104 ceph-mon[36624]: pgmap v7765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:04 smithi081 ceph-mon[31453]: pgmap v7765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:06.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:06 smithi153 ceph-mon[36569]: pgmap v7766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:06 smithi104 ceph-mon[36624]: pgmap v7766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:06 smithi081 ceph-mon[31453]: pgmap v7766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:08.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:08 smithi153 ceph-mon[36569]: pgmap v7767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:08 smithi104 ceph-mon[36624]: pgmap v7767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:08 smithi081 ceph-mon[31453]: pgmap v7767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:10.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:10 smithi153 ceph-mon[36569]: pgmap v7768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:10 smithi104 ceph-mon[36624]: pgmap v7768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:10 smithi081 ceph-mon[31453]: pgmap v7768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:12 smithi153 ceph-mon[36569]: pgmap v7769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:12 smithi104 ceph-mon[36624]: pgmap v7769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:12 smithi081 ceph-mon[31453]: pgmap v7769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:14 smithi153 ceph-mon[36569]: pgmap v7770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:10:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:14 smithi104 ceph-mon[36624]: pgmap v7770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:10:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:14 smithi081 ceph-mon[31453]: pgmap v7770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:10:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:15.023+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:10:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:15.023+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:10:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:15.024+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:10:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:15.024+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:10:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:10:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:10:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:10:15.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:10:15.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:10:15.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:10:15.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:10:15.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:10:15.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:10:16.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:16 smithi153 ceph-mon[36569]: pgmap v7771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:16 smithi153 ceph-mon[36569]: pgmap v7772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:16 smithi104 ceph-mon[36624]: pgmap v7771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:16 smithi104 ceph-mon[36624]: pgmap v7772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:16 smithi081 ceph-mon[31453]: pgmap v7771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:16 smithi081 ceph-mon[31453]: pgmap v7772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:18.528+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:10:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:18.528+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:10:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:18.528+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:10:18.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:18 smithi153 ceph-mon[36569]: pgmap v7773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:18 smithi104 ceph-mon[36624]: pgmap v7773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:18 smithi081 ceph-mon[31453]: pgmap v7773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:19 smithi153 ceph-mon[36569]: pgmap v7774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:19.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:19 smithi104 ceph-mon[36624]: pgmap v7774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:19 smithi081 ceph-mon[31453]: pgmap v7774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:22.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:21 smithi153 ceph-mon[36569]: pgmap v7775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:22.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:21 smithi104 ceph-mon[36624]: pgmap v7775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:22.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:21 smithi081 ceph-mon[31453]: pgmap v7775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:24.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:23 smithi153 ceph-mon[36569]: pgmap v7776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:24.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:23 smithi104 ceph-mon[36624]: pgmap v7776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:24.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:23 smithi081 ceph-mon[31453]: pgmap v7776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:26.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:25 smithi081 ceph-mon[31453]: pgmap v7777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:26.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:25 smithi153 ceph-mon[36569]: pgmap v7777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:26.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:25 smithi104 ceph-mon[36624]: pgmap v7777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:27 smithi153 ceph-mon[36569]: pgmap v7778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:28.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:27 smithi104 ceph-mon[36624]: pgmap v7778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:28.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:27 smithi081 ceph-mon[31453]: pgmap v7778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:30.278 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:29 smithi081 ceph-mon[31453]: pgmap v7779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:30.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:29 smithi153 ceph-mon[36569]: pgmap v7779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:29 smithi104 ceph-mon[36624]: pgmap v7779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:32.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:31 smithi153 ceph-mon[36569]: pgmap v7780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:31 smithi104 ceph-mon[36624]: pgmap v7780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:31 smithi081 ceph-mon[31453]: pgmap v7780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:33.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:33.530+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:10:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:33.530+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:10:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:33.530+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:10:34.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:33 smithi153 ceph-mon[36569]: pgmap v7781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:33 smithi104 ceph-mon[36624]: pgmap v7781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:34.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:33 smithi081 ceph-mon[31453]: pgmap v7781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:36.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:35 smithi153 ceph-mon[36569]: pgmap v7782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:36.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:35 smithi104 ceph-mon[36624]: pgmap v7782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:36.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:35 smithi081 ceph-mon[31453]: pgmap v7782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:38.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:37 smithi153 ceph-mon[36569]: pgmap v7783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:37 smithi104 ceph-mon[36624]: pgmap v7783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:37 smithi081 ceph-mon[31453]: pgmap v7783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:40.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:39 smithi153 ceph-mon[36569]: pgmap v7784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:40.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:39 smithi104 ceph-mon[36624]: pgmap v7784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:40.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:39 smithi081 ceph-mon[31453]: pgmap v7784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:41 smithi153 ceph-mon[36569]: pgmap v7785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:42.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:41 smithi104 ceph-mon[36624]: pgmap v7785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:42.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:41 smithi081 ceph-mon[31453]: pgmap v7785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:43 smithi153 ceph-mon[36569]: pgmap v7786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:44.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:43 smithi104 ceph-mon[36624]: pgmap v7786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:43 smithi081 ceph-mon[31453]: pgmap v7786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:46.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:45 smithi153 ceph-mon[36569]: pgmap v7787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:45 smithi104 ceph-mon[36624]: pgmap v7787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:46.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:45 smithi081 ceph-mon[31453]: pgmap v7787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:48.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:47 smithi153 ceph-mon[36569]: pgmap v7788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:48.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:47 smithi104 ceph-mon[36624]: pgmap v7788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:48.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:47 smithi081 ceph-mon[31453]: pgmap v7788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:48.546+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:10:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:48.546+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:10:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:10:48.546+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:10:50.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:49 smithi153 ceph-mon[36569]: pgmap v7789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:50.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:49 smithi104 ceph-mon[36624]: pgmap v7789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:49 smithi081 ceph-mon[31453]: pgmap v7789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:51 smithi153 ceph-mon[36569]: pgmap v7790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:52.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:51 smithi104 ceph-mon[36624]: pgmap v7790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:51 smithi081 ceph-mon[31453]: pgmap v7790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:54.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:53 smithi153 ceph-mon[36569]: pgmap v7791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:54.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:53 smithi104 ceph-mon[36624]: pgmap v7791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:53 smithi081 ceph-mon[31453]: pgmap v7791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:56.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:56 smithi153 ceph-mon[36569]: pgmap v7792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:56.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:56 smithi104 ceph-mon[36624]: pgmap v7792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:56 smithi081 ceph-mon[31453]: pgmap v7792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:58.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:10:58 smithi153 ceph-mon[36569]: pgmap v7793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:58.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:10:58 smithi104 ceph-mon[36624]: pgmap v7793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:10:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:10:58 smithi081 ceph-mon[31453]: pgmap v7793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:00.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:00 smithi153 ceph-mon[36569]: pgmap v7794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:00.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:00 smithi104 ceph-mon[36624]: pgmap v7794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:00.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:00 smithi081 ceph-mon[31453]: pgmap v7794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:02.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:02 smithi153 ceph-mon[36569]: pgmap v7795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:02.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:02 smithi104 ceph-mon[36624]: pgmap v7795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:02 smithi081 ceph-mon[31453]: pgmap v7795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:03.562+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:11:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:03.562+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:11:03.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:03.562+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:11:04.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:04 smithi153 ceph-mon[36569]: pgmap v7796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:04.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:04 smithi104 ceph-mon[36624]: pgmap v7796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:04.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:04 smithi081 ceph-mon[31453]: pgmap v7796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:06.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:06 smithi153 ceph-mon[36569]: pgmap v7797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:06.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:06 smithi104 ceph-mon[36624]: pgmap v7797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:06.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:06 smithi081 ceph-mon[31453]: pgmap v7797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:08.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:08 smithi153 ceph-mon[36569]: pgmap v7798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:08.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:08 smithi104 ceph-mon[36624]: pgmap v7798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:08.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:08 smithi081 ceph-mon[31453]: pgmap v7798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:10.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:10 smithi153 ceph-mon[36569]: pgmap v7799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:10 smithi104 ceph-mon[36624]: pgmap v7799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:10 smithi081 ceph-mon[31453]: pgmap v7799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:12.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:12 smithi153 ceph-mon[36569]: pgmap v7800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:12 smithi104 ceph-mon[36624]: pgmap v7800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:12.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:12 smithi081 ceph-mon[31453]: pgmap v7800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:14.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:14 smithi153 ceph-mon[36569]: pgmap v7801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:14 smithi104 ceph-mon[36624]: pgmap v7801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:14.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:14 smithi081 ceph-mon[31453]: pgmap v7801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: pgmap v7802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: pgmap v7802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:16.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:11:16.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: pgmap v7802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:11:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:11:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:11:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:11:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: pgmap v7803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: pgmap v7804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:11:17.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:11:17.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: pgmap v7803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: pgmap v7804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:11:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:11:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:11:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: pgmap v7803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: pgmap v7804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:11:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:18.576+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:11:18.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:18.576+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:11:18.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:18.576+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:11:18.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:18 smithi153 ceph-mon[36569]: pgmap v7805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:18 smithi104 ceph-mon[36624]: pgmap v7805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:18 smithi081 ceph-mon[31453]: pgmap v7805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:20.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:20 smithi153 ceph-mon[36569]: pgmap v7806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:20 smithi104 ceph-mon[36624]: pgmap v7806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:20 smithi081 ceph-mon[31453]: pgmap v7806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:22 smithi153 ceph-mon[36569]: pgmap v7807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:22 smithi104 ceph-mon[36624]: pgmap v7807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:22 smithi081 ceph-mon[31453]: pgmap v7807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:24.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:24 smithi153 ceph-mon[36569]: pgmap v7808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:24 smithi104 ceph-mon[36624]: pgmap v7808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:24 smithi081 ceph-mon[31453]: pgmap v7808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:26.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:26 smithi153 ceph-mon[36569]: pgmap v7809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:26 smithi104 ceph-mon[36624]: pgmap v7809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:26.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:26 smithi081 ceph-mon[31453]: pgmap v7809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:28.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:28 smithi153 ceph-mon[36569]: pgmap v7810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:28 smithi104 ceph-mon[36624]: pgmap v7810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:28.893 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:28 smithi081 ceph-mon[31453]: pgmap v7810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:30.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:30 smithi153 ceph-mon[36569]: pgmap v7811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:30 smithi104 ceph-mon[36624]: pgmap v7811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:30 smithi081 ceph-mon[31453]: pgmap v7811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:32.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:32 smithi153 ceph-mon[36569]: pgmap v7812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:32 smithi104 ceph-mon[36624]: pgmap v7812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:32.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:32 smithi081 ceph-mon[31453]: pgmap v7812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:33.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:33.586+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:11:33.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:33.586+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:11:33.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:33.586+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:11:34.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:34 smithi153 ceph-mon[36569]: pgmap v7813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:34 smithi104 ceph-mon[36624]: pgmap v7813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:34 smithi081 ceph-mon[31453]: pgmap v7813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:36.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:36 smithi153 ceph-mon[36569]: pgmap v7814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:36.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:36 smithi104 ceph-mon[36624]: pgmap v7814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:36.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:36 smithi081 ceph-mon[31453]: pgmap v7814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:38 smithi153 ceph-mon[36569]: pgmap v7815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:38 smithi104 ceph-mon[36624]: pgmap v7815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:38 smithi081 ceph-mon[31453]: pgmap v7815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:40.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:40 smithi153 ceph-mon[36569]: pgmap v7816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:40 smithi104 ceph-mon[36624]: pgmap v7816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:40 smithi081 ceph-mon[31453]: pgmap v7816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:42 smithi153 ceph-mon[36569]: pgmap v7817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:42 smithi104 ceph-mon[36624]: pgmap v7817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:42 smithi081 ceph-mon[31453]: pgmap v7817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:44.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:44 smithi153 ceph-mon[36569]: pgmap v7818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:44 smithi104 ceph-mon[36624]: pgmap v7818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:44 smithi081 ceph-mon[31453]: pgmap v7818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:46.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:46 smithi153 ceph-mon[36569]: pgmap v7819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:46.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:46 smithi104 ceph-mon[36624]: pgmap v7819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:46 smithi081 ceph-mon[31453]: pgmap v7819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:48.587 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:48.588+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:48.588+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:11:48.588+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:11:48.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:48 smithi153 ceph-mon[36569]: pgmap v7820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:48.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:48 smithi104 ceph-mon[36624]: pgmap v7820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:48 smithi081 ceph-mon[31453]: pgmap v7820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:50.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:50 smithi153 ceph-mon[36569]: pgmap v7821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:50 smithi104 ceph-mon[36624]: pgmap v7821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:50.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:50 smithi081 ceph-mon[31453]: pgmap v7821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:52.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:52 smithi153 ceph-mon[36569]: pgmap v7822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:52 smithi104 ceph-mon[36624]: pgmap v7822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:52 smithi081 ceph-mon[31453]: pgmap v7822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:54 smithi153 ceph-mon[36569]: pgmap v7823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:54 smithi104 ceph-mon[36624]: pgmap v7823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:54 smithi081 ceph-mon[31453]: pgmap v7823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:56.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:56 smithi153 ceph-mon[36569]: pgmap v7824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:56 smithi104 ceph-mon[36624]: pgmap v7824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:56.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:56 smithi081 ceph-mon[31453]: pgmap v7824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:58.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:11:58 smithi153 ceph-mon[36569]: pgmap v7825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:11:58 smithi104 ceph-mon[36624]: pgmap v7825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:11:58.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:11:58 smithi081 ceph-mon[31453]: pgmap v7825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:00 smithi153 ceph-mon[36569]: pgmap v7826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:00 smithi104 ceph-mon[36624]: pgmap v7826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:00 smithi081 ceph-mon[31453]: pgmap v7826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:02.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:02 smithi153 ceph-mon[36569]: pgmap v7827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:02 smithi104 ceph-mon[36624]: pgmap v7827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:02 smithi081 ceph-mon[31453]: pgmap v7827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:03.603+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:12:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:03.603+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:12:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:03.603+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:12:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:04 smithi153 ceph-mon[36569]: pgmap v7828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:04 smithi104 ceph-mon[36624]: pgmap v7828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:04 smithi081 ceph-mon[31453]: pgmap v7828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:06.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:06 smithi153 ceph-mon[36569]: pgmap v7829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:06 smithi104 ceph-mon[36624]: pgmap v7829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:06 smithi081 ceph-mon[31453]: pgmap v7829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:08.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:08 smithi153 ceph-mon[36569]: pgmap v7830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:08 smithi104 ceph-mon[36624]: pgmap v7830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:08 smithi081 ceph-mon[31453]: pgmap v7830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:10 smithi153 ceph-mon[36569]: pgmap v7831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:10 smithi104 ceph-mon[36624]: pgmap v7831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:10 smithi081 ceph-mon[31453]: pgmap v7831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:12.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:12 smithi153 ceph-mon[36569]: pgmap v7832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:12 smithi104 ceph-mon[36624]: pgmap v7832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:12 smithi081 ceph-mon[31453]: pgmap v7832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:14.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:14 smithi153 ceph-mon[36569]: pgmap v7833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:14 smithi104 ceph-mon[36624]: pgmap v7833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:14 smithi081 ceph-mon[31453]: pgmap v7833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:16.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:16 smithi153 ceph-mon[36569]: pgmap v7834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:16 smithi104 ceph-mon[36624]: pgmap v7834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:16 smithi081 ceph-mon[31453]: pgmap v7834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:17.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:12:17.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:12:17.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:12:18.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:18.618+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:12:18.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:18.618+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:12:18.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:18.618+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:12:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:12:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:12:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:18 smithi153 ceph-mon[36569]: pgmap v7835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:18 smithi153 ceph-mon[36569]: pgmap v7836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:12:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:12:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:12:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:18 smithi104 ceph-mon[36624]: pgmap v7835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:18 smithi104 ceph-mon[36624]: pgmap v7836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:12:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:12:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:12:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:18 smithi081 ceph-mon[31453]: pgmap v7835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:18 smithi081 ceph-mon[31453]: pgmap v7836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:12:21.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:21 smithi153 ceph-mon[36569]: pgmap v7837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:21.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:21 smithi104 ceph-mon[36624]: pgmap v7837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:21 smithi081 ceph-mon[31453]: pgmap v7837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:23.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:23 smithi153 ceph-mon[36569]: pgmap v7838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:23.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:23 smithi104 ceph-mon[36624]: pgmap v7838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:23.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:23 smithi081 ceph-mon[31453]: pgmap v7838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:25.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:25 smithi153 ceph-mon[36569]: pgmap v7839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:25 smithi104 ceph-mon[36624]: pgmap v7839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:25 smithi081 ceph-mon[31453]: pgmap v7839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:27 smithi153 ceph-mon[36569]: pgmap v7840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:27 smithi104 ceph-mon[36624]: pgmap v7840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:27 smithi081 ceph-mon[31453]: pgmap v7840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:29.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:29 smithi153 ceph-mon[36569]: pgmap v7841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:29 smithi104 ceph-mon[36624]: pgmap v7841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:29 smithi081 ceph-mon[31453]: pgmap v7841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:31.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:31 smithi153 ceph-mon[36569]: pgmap v7842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:31 smithi104 ceph-mon[36624]: pgmap v7842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:31 smithi081 ceph-mon[31453]: pgmap v7842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:33.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:33 smithi153 ceph-mon[36569]: pgmap v7843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:33.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:33 smithi104 ceph-mon[36624]: pgmap v7843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:33 smithi081 ceph-mon[31453]: pgmap v7843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:33.633 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:33.634+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:12:33.633 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:33.634+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:12:33.633 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:33.634+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:12:35.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:35 smithi153 ceph-mon[36569]: pgmap v7844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:35.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:35 smithi104 ceph-mon[36624]: pgmap v7844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:35 smithi081 ceph-mon[31453]: pgmap v7844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:37.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:37 smithi153 ceph-mon[36569]: pgmap v7845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:37 smithi104 ceph-mon[36624]: pgmap v7845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:37 smithi081 ceph-mon[31453]: pgmap v7845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:39 smithi153 ceph-mon[36569]: pgmap v7846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:39 smithi104 ceph-mon[36624]: pgmap v7846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:39 smithi081 ceph-mon[31453]: pgmap v7846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:41.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:41 smithi104 ceph-mon[36624]: pgmap v7847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:41 smithi081 ceph-mon[31453]: pgmap v7847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:41.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:41 smithi153 ceph-mon[36569]: pgmap v7847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:43 smithi104 ceph-mon[36624]: pgmap v7848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:43 smithi081 ceph-mon[31453]: pgmap v7848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:43.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:43 smithi153 ceph-mon[36569]: pgmap v7848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:45 smithi104 ceph-mon[36624]: pgmap v7849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:45 smithi081 ceph-mon[31453]: pgmap v7849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:45.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:45 smithi153 ceph-mon[36569]: pgmap v7849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:47 smithi104 ceph-mon[36624]: pgmap v7850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:47 smithi081 ceph-mon[31453]: pgmap v7850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:47 smithi153 ceph-mon[36569]: pgmap v7850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:48.650+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:12:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:48.650+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:12:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:12:48.650+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:12:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:49 smithi104 ceph-mon[36624]: pgmap v7851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:49.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:49 smithi081 ceph-mon[31453]: pgmap v7851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:49.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:49 smithi153 ceph-mon[36569]: pgmap v7851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:51.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:51 smithi081 ceph-mon[31453]: pgmap v7852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:51.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:51 smithi153 ceph-mon[36569]: pgmap v7852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:51.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:51 smithi104 ceph-mon[36624]: pgmap v7852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:53.383 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:53 smithi104 ceph-mon[36624]: pgmap v7853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:53 smithi081 ceph-mon[31453]: pgmap v7853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:53.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:53 smithi153 ceph-mon[36569]: pgmap v7853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:55.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:55 smithi081 ceph-mon[31453]: pgmap v7854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:55.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:55 smithi153 ceph-mon[36569]: pgmap v7854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:55.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:55 smithi104 ceph-mon[36624]: pgmap v7854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:57.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:57 smithi081 ceph-mon[31453]: pgmap v7855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:57.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:57 smithi153 ceph-mon[36569]: pgmap v7855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:57.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:57 smithi104 ceph-mon[36624]: pgmap v7855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:59.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:12:59 smithi081 ceph-mon[31453]: pgmap v7856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:59.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:12:59 smithi153 ceph-mon[36569]: pgmap v7856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:12:59.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:12:59 smithi104 ceph-mon[36624]: pgmap v7856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:01.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:01 smithi081 ceph-mon[31453]: pgmap v7857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:01.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:01 smithi153 ceph-mon[36569]: pgmap v7857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:01.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:01 smithi104 ceph-mon[36624]: pgmap v7857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:03 smithi081 ceph-mon[31453]: pgmap v7858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:03.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:03 smithi153 ceph-mon[36569]: pgmap v7858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:03 smithi104 ceph-mon[36624]: pgmap v7858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:03.665+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:13:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:03.665+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:13:03.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:03.665+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:13:05.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:05 smithi081 ceph-mon[31453]: pgmap v7859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:05.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:05 smithi153 ceph-mon[36569]: pgmap v7859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:05 smithi104 ceph-mon[36624]: pgmap v7859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:07.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:07 smithi081 ceph-mon[31453]: pgmap v7860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:07.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:07 smithi153 ceph-mon[36569]: pgmap v7860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:07 smithi104 ceph-mon[36624]: pgmap v7860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:09 smithi081 ceph-mon[31453]: pgmap v7861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:09.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:09 smithi153 ceph-mon[36569]: pgmap v7861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:09 smithi104 ceph-mon[36624]: pgmap v7861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:11 smithi081 ceph-mon[31453]: pgmap v7862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:11.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:11 smithi153 ceph-mon[36569]: pgmap v7862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:11 smithi104 ceph-mon[36624]: pgmap v7862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:13 smithi081 ceph-mon[31453]: pgmap v7863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:13.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:13 smithi153 ceph-mon[36569]: pgmap v7863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:13 smithi104 ceph-mon[36624]: pgmap v7863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:15.053+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:13:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:15.053+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:13:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:15.053+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:13:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:15.053+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:13:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:15 smithi081 ceph-mon[31453]: pgmap v7864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:15.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:15 smithi153 ceph-mon[36569]: pgmap v7864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:15 smithi104 ceph-mon[36624]: pgmap v7864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:17 smithi081 ceph-mon[31453]: pgmap v7865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:17.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:17 smithi153 ceph-mon[36569]: pgmap v7865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:17 smithi104 ceph-mon[36624]: pgmap v7865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:18.669+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:13:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:18.669+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:13:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:18.669+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:13:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:19 smithi081 ceph-mon[31453]: pgmap v7866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:13:19.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:19 smithi104 ceph-mon[36624]: pgmap v7866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:19.549 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:13:19.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:19 smithi153 ceph-mon[36569]: pgmap v7866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:19.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:13:21.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:21 smithi153 ceph-mon[36569]: pgmap v7867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:21 smithi104 ceph-mon[36624]: pgmap v7867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:21.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:21 smithi081 ceph-mon[31453]: pgmap v7867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:23.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:23 smithi153 ceph-mon[36569]: pgmap v7868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:23 smithi104 ceph-mon[36624]: pgmap v7868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:23.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:23 smithi081 ceph-mon[31453]: pgmap v7868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:24.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:25 smithi081 ceph-mon[31453]: pgmap v7869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:13:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:13:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:25 smithi153 ceph-mon[36569]: pgmap v7869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:13:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:13:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:25 smithi104 ceph-mon[36624]: pgmap v7869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:13:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:13:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:13:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:26 smithi081 ceph-mon[31453]: pgmap v7870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:26 smithi081 ceph-mon[31453]: pgmap v7871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:26.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:26 smithi153 ceph-mon[36569]: pgmap v7870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:26 smithi153 ceph-mon[36569]: pgmap v7871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:26 smithi104 ceph-mon[36624]: pgmap v7870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:26 smithi104 ceph-mon[36624]: pgmap v7871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:28.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:28 smithi153 ceph-mon[36569]: pgmap v7872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:28 smithi104 ceph-mon[36624]: pgmap v7872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:28.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:28 smithi081 ceph-mon[31453]: pgmap v7872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:30.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:30 smithi153 ceph-mon[36569]: pgmap v7873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:30 smithi104 ceph-mon[36624]: pgmap v7873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:30 smithi081 ceph-mon[31453]: pgmap v7873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:32.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:32 smithi153 ceph-mon[36569]: pgmap v7874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:32 smithi104 ceph-mon[36624]: pgmap v7874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:32.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:32 smithi081 ceph-mon[31453]: pgmap v7874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:33.676+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:13:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:33.676+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:13:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:33.676+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:13:34.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:34 smithi153 ceph-mon[36569]: pgmap v7875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:34 smithi104 ceph-mon[36624]: pgmap v7875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:34 smithi081 ceph-mon[31453]: pgmap v7875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:36 smithi153 ceph-mon[36569]: pgmap v7876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:36.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:36 smithi104 ceph-mon[36624]: pgmap v7876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:36.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:36 smithi081 ceph-mon[31453]: pgmap v7876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:38.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:38 smithi153 ceph-mon[36569]: pgmap v7877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:38 smithi104 ceph-mon[36624]: pgmap v7877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:38 smithi081 ceph-mon[31453]: pgmap v7877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:40.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:40 smithi153 ceph-mon[36569]: pgmap v7878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:40 smithi104 ceph-mon[36624]: pgmap v7878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:40 smithi081 ceph-mon[31453]: pgmap v7878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:42 smithi153 ceph-mon[36569]: pgmap v7879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:42 smithi104 ceph-mon[36624]: pgmap v7879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:42 smithi081 ceph-mon[31453]: pgmap v7879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:44.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:44 smithi153 ceph-mon[36569]: pgmap v7880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:44 smithi104 ceph-mon[36624]: pgmap v7880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:44 smithi081 ceph-mon[31453]: pgmap v7880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:46.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:46 smithi153 ceph-mon[36569]: pgmap v7881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:46 smithi104 ceph-mon[36624]: pgmap v7881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:46 smithi081 ceph-mon[31453]: pgmap v7881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:48.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:48.678+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:13:48.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:48.678+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:13:48.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:13:48.678+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:13:48.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:48 smithi153 ceph-mon[36569]: pgmap v7882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:48.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:48 smithi104 ceph-mon[36624]: pgmap v7882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:48 smithi081 ceph-mon[31453]: pgmap v7882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:50.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:50 smithi153 ceph-mon[36569]: pgmap v7883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:50 smithi104 ceph-mon[36624]: pgmap v7883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:50.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:50 smithi081 ceph-mon[31453]: pgmap v7883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:52 smithi153 ceph-mon[36569]: pgmap v7884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:52 smithi104 ceph-mon[36624]: pgmap v7884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:52 smithi081 ceph-mon[31453]: pgmap v7884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:54.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:54 smithi153 ceph-mon[36569]: pgmap v7885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:54 smithi104 ceph-mon[36624]: pgmap v7885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:54 smithi081 ceph-mon[31453]: pgmap v7885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:56.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:56 smithi153 ceph-mon[36569]: pgmap v7886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:56 smithi104 ceph-mon[36624]: pgmap v7886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:56.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:56 smithi081 ceph-mon[31453]: pgmap v7886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:13:58 smithi153 ceph-mon[36569]: pgmap v7887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:13:58 smithi104 ceph-mon[36624]: pgmap v7887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:13:58.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:13:58 smithi081 ceph-mon[31453]: pgmap v7887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:00 smithi153 ceph-mon[36569]: pgmap v7888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:00 smithi104 ceph-mon[36624]: pgmap v7888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:00 smithi081 ceph-mon[31453]: pgmap v7888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:02.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:02 smithi153 ceph-mon[36569]: pgmap v7889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:02 smithi104 ceph-mon[36624]: pgmap v7889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:02 smithi081 ceph-mon[31453]: pgmap v7889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:03.694+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:14:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:03.694+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:14:03.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:03.694+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:14:03.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:03 smithi153 ceph-mon[36569]: pgmap v7890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:03 smithi104 ceph-mon[36624]: pgmap v7890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:03 smithi081 ceph-mon[31453]: pgmap v7890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:06 smithi081 ceph-mon[31453]: pgmap v7891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:06.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:06 smithi153 ceph-mon[36569]: pgmap v7891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:06 smithi104 ceph-mon[36624]: pgmap v7891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:08.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:08 smithi081 ceph-mon[31453]: pgmap v7892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:08.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:08 smithi153 ceph-mon[36569]: pgmap v7892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:08 smithi104 ceph-mon[36624]: pgmap v7892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:10.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:10 smithi081 ceph-mon[31453]: pgmap v7893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:10.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:10 smithi153 ceph-mon[36569]: pgmap v7893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:10 smithi104 ceph-mon[36624]: pgmap v7893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:12.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:12 smithi153 ceph-mon[36569]: pgmap v7894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:12 smithi104 ceph-mon[36624]: pgmap v7894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:12 smithi081 ceph-mon[31453]: pgmap v7894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:14.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:14 smithi153 ceph-mon[36569]: pgmap v7895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:14 smithi104 ceph-mon[36624]: pgmap v7895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:14 smithi081 ceph-mon[31453]: pgmap v7895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:16 smithi153 ceph-mon[36569]: pgmap v7896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:16 smithi104 ceph-mon[36624]: pgmap v7896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:16 smithi081 ceph-mon[31453]: pgmap v7896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:18.708+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:14:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:18.708+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:14:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:18.708+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:14:18.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:18 smithi153 ceph-mon[36569]: pgmap v7897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:18 smithi104 ceph-mon[36624]: pgmap v7897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:18 smithi081 ceph-mon[31453]: pgmap v7897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:20.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:20 smithi153 ceph-mon[36569]: pgmap v7898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:20 smithi104 ceph-mon[36624]: pgmap v7898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:20 smithi081 ceph-mon[31453]: pgmap v7898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:22 smithi153 ceph-mon[36569]: pgmap v7899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:22 smithi104 ceph-mon[36624]: pgmap v7899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:22 smithi081 ceph-mon[31453]: pgmap v7899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:24.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:24 smithi153 ceph-mon[36569]: pgmap v7900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:24 smithi104 ceph-mon[36624]: pgmap v7900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:24 smithi081 ceph-mon[31453]: pgmap v7900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:26.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:26 smithi153 ceph-mon[36569]: pgmap v7901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:14:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:26 smithi104 ceph-mon[36624]: pgmap v7901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:14:26.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:26 smithi081 ceph-mon[31453]: pgmap v7901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:26.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:14:28.673 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:28 smithi153 ceph-mon[36569]: pgmap v7902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:28.707 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:28 smithi081 ceph-mon[31453]: pgmap v7902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:28 smithi104 ceph-mon[36624]: pgmap v7902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:30.534 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:30 smithi081 ceph-mon[31453]: pgmap v7903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:30.535 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:30.535 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:30.535 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:14:30.535 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:14:30.535 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:30.762 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:30 smithi153 ceph-mon[36569]: pgmap v7903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:30.762 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:30.762 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:30.763 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:14:30.763 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:14:30.763 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:30 smithi104 ceph-mon[36624]: pgmap v7903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:14:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:14:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:14:31.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:31 smithi153 ceph-mon[36569]: pgmap v7904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:31.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:31 smithi153 ceph-mon[36569]: pgmap v7905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:31 smithi104 ceph-mon[36624]: pgmap v7904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:31 smithi104 ceph-mon[36624]: pgmap v7905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:31.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:31 smithi081 ceph-mon[31453]: pgmap v7904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:31.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:31 smithi081 ceph-mon[31453]: pgmap v7905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:33.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:33.724+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:14:33.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:33.724+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:14:33.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:33.724+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:14:33.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:33 smithi153 ceph-mon[36569]: pgmap v7906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:33.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:33 smithi104 ceph-mon[36624]: pgmap v7906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:33.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:33 smithi081 ceph-mon[31453]: pgmap v7906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:35.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:35 smithi153 ceph-mon[36569]: pgmap v7907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:35.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:35 smithi104 ceph-mon[36624]: pgmap v7907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:35 smithi081 ceph-mon[31453]: pgmap v7907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:37.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:37 smithi153 ceph-mon[36569]: pgmap v7908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:37 smithi104 ceph-mon[36624]: pgmap v7908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:37.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:37 smithi081 ceph-mon[31453]: pgmap v7908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:39.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:39 smithi153 ceph-mon[36569]: pgmap v7909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:39 smithi104 ceph-mon[36624]: pgmap v7909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:39 smithi081 ceph-mon[31453]: pgmap v7909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:41.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:41 smithi153 ceph-mon[36569]: pgmap v7910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:41.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:41 smithi104 ceph-mon[36624]: pgmap v7910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:41.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:41 smithi081 ceph-mon[31453]: pgmap v7910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:43.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:43 smithi153 ceph-mon[36569]: pgmap v7911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:43.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:43 smithi104 ceph-mon[36624]: pgmap v7911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:43.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:43 smithi081 ceph-mon[31453]: pgmap v7911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:45.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:45 smithi153 ceph-mon[36569]: pgmap v7912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:45.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:45 smithi104 ceph-mon[36624]: pgmap v7912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:45.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:45 smithi081 ceph-mon[31453]: pgmap v7912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:47.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:47 smithi153 ceph-mon[36569]: pgmap v7913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:47.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:47 smithi104 ceph-mon[36624]: pgmap v7913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:47.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:47 smithi081 ceph-mon[31453]: pgmap v7913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:48.739+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:14:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:48.739+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:14:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:14:48.739+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:14:49.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:49 smithi153 ceph-mon[36569]: pgmap v7914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:49.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:49 smithi104 ceph-mon[36624]: pgmap v7914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:49.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:49 smithi081 ceph-mon[31453]: pgmap v7914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:51.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:51 smithi153 ceph-mon[36569]: pgmap v7915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:51.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:51 smithi104 ceph-mon[36624]: pgmap v7915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:51.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:51 smithi081 ceph-mon[31453]: pgmap v7915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:52 smithi104 ceph-mon[36624]: pgmap v7916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:52 smithi081 ceph-mon[31453]: pgmap v7916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:52 smithi153 ceph-mon[36569]: pgmap v7916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:55.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:55 smithi153 ceph-mon[36569]: pgmap v7917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:55 smithi104 ceph-mon[36624]: pgmap v7917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:55.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:55 smithi081 ceph-mon[31453]: pgmap v7917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:57.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:57 smithi153 ceph-mon[36569]: pgmap v7918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:57.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:57 smithi104 ceph-mon[36624]: pgmap v7918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:57.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:57 smithi081 ceph-mon[31453]: pgmap v7918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:14:59 smithi153 ceph-mon[36569]: pgmap v7919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:59.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:14:59 smithi104 ceph-mon[36624]: pgmap v7919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:14:59.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:14:59 smithi081 ceph-mon[31453]: pgmap v7919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:00 smithi153 ceph-mon[36569]: pgmap v7920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:00 smithi104 ceph-mon[36624]: pgmap v7920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:00 smithi081 ceph-mon[31453]: pgmap v7920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:03.745+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:15:03.744 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:03.745+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:15:03.745 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:03.745+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:15:03.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:03 smithi153 ceph-mon[36569]: pgmap v7921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:03 smithi104 ceph-mon[36624]: pgmap v7921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:03 smithi081 ceph-mon[31453]: pgmap v7921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:05.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:05 smithi153 ceph-mon[36569]: pgmap v7922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:05.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:05 smithi104 ceph-mon[36624]: pgmap v7922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:05.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:05 smithi081 ceph-mon[31453]: pgmap v7922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:06 smithi104 ceph-mon[36624]: pgmap v7923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:06 smithi081 ceph-mon[31453]: pgmap v7923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:07.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:06 smithi153 ceph-mon[36569]: pgmap v7923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:09.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:09 smithi153 ceph-mon[36569]: pgmap v7924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:09 smithi104 ceph-mon[36624]: pgmap v7924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:09.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:09 smithi081 ceph-mon[31453]: pgmap v7924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:11.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:11 smithi153 ceph-mon[36569]: pgmap v7925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:11.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:11 smithi104 ceph-mon[36624]: pgmap v7925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:11.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:11 smithi081 ceph-mon[31453]: pgmap v7925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:12.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:12 smithi153 ceph-mon[36569]: pgmap v7926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:12 smithi104 ceph-mon[36624]: pgmap v7926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:12 smithi081 ceph-mon[31453]: pgmap v7926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:15.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:15 smithi153 ceph-mon[36569]: pgmap v7927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:15.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:15 smithi104 ceph-mon[36624]: pgmap v7927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:15.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:15 smithi081 ceph-mon[31453]: pgmap v7927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:16 smithi104 ceph-mon[36624]: pgmap v7928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:16 smithi081 ceph-mon[31453]: pgmap v7928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:16 smithi153 ceph-mon[36569]: pgmap v7928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:18.752+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:15:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:18.752+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:15:18.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:18.752+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:15:19.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:19 smithi153 ceph-mon[36569]: pgmap v7929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:19.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:19 smithi104 ceph-mon[36624]: pgmap v7929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:19 smithi081 ceph-mon[31453]: pgmap v7929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:20.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:20 smithi153 ceph-mon[36569]: pgmap v7930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:20 smithi104 ceph-mon[36624]: pgmap v7930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:20 smithi081 ceph-mon[31453]: pgmap v7930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:23.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:23 smithi153 ceph-mon[36569]: pgmap v7931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:23.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:23 smithi104 ceph-mon[36624]: pgmap v7931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:23 smithi081 ceph-mon[31453]: pgmap v7931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:24.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:24 smithi153 ceph-mon[36569]: pgmap v7932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:24 smithi104 ceph-mon[36624]: pgmap v7932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:24 smithi081 ceph-mon[31453]: pgmap v7932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:26 smithi104 ceph-mon[36624]: pgmap v7933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:26.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:26 smithi081 ceph-mon[31453]: pgmap v7933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:27.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:26 smithi153 ceph-mon[36569]: pgmap v7933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:28 smithi104 ceph-mon[36624]: pgmap v7934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:28.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:28 smithi081 ceph-mon[31453]: pgmap v7934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:29.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:28 smithi153 ceph-mon[36569]: pgmap v7934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:30.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:30.074+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:15:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:30.075+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:15:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:30.075+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:15:30.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:30.076+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:15:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:30 smithi104 ceph-mon[36624]: pgmap v7935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:30 smithi081 ceph-mon[31453]: pgmap v7935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:31.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:30 smithi153 ceph-mon[36569]: pgmap v7935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:15:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:31.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:15:31.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:31.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:32.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:15:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:32.876 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:32.876 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:32.876 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:32 smithi081 ceph-mon[31453]: pgmap v7936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:33.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:32 smithi153 ceph-mon[36569]: pgmap v7936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:32 smithi104 ceph-mon[36624]: pgmap v7936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:33.766+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:15:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:33.766+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:15:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:33.766+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:15:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:35.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:35 smithi153 ceph-mon[36569]: pgmap v7937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:35 smithi104 ceph-mon[36624]: pgmap v7937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:35 smithi081 ceph-mon[31453]: pgmap v7937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:37.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:37 smithi153 ceph-mon[36569]: pgmap v7938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:37 smithi104 ceph-mon[36624]: pgmap v7938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:37.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:37 smithi081 ceph-mon[31453]: pgmap v7938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:15:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:15:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:38 smithi104 ceph-mon[36624]: pgmap v7939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:38 smithi104 ceph-mon[36624]: pgmap v7940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:15:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:15:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:38 smithi081 ceph-mon[31453]: pgmap v7939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:38 smithi081 ceph-mon[31453]: pgmap v7940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:38.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:15:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:15:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:38 smithi153 ceph-mon[36569]: pgmap v7939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:38 smithi153 ceph-mon[36569]: pgmap v7940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:15:40.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:40 smithi081 ceph-mon[31453]: pgmap v7941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:40 smithi153 ceph-mon[36569]: pgmap v7941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:40 smithi104 ceph-mon[36624]: pgmap v7941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:42 smithi081 ceph-mon[31453]: pgmap v7942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:42 smithi153 ceph-mon[36569]: pgmap v7942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:42 smithi104 ceph-mon[36624]: pgmap v7942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:44.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:44 smithi081 ceph-mon[31453]: pgmap v7943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:44.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:44 smithi153 ceph-mon[36569]: pgmap v7943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:44 smithi104 ceph-mon[36624]: pgmap v7943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:46 smithi081 ceph-mon[31453]: pgmap v7944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:46.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:46 smithi153 ceph-mon[36569]: pgmap v7944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:46.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:46 smithi104 ceph-mon[36624]: pgmap v7944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:48 smithi081 ceph-mon[31453]: pgmap v7945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:48.770+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:15:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:48.770+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:15:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:15:48.770+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:15:48.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:48 smithi153 ceph-mon[36569]: pgmap v7945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:48.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:48 smithi104 ceph-mon[36624]: pgmap v7945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:50 smithi081 ceph-mon[31453]: pgmap v7946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:50 smithi153 ceph-mon[36569]: pgmap v7946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:50 smithi104 ceph-mon[36624]: pgmap v7946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:52 smithi081 ceph-mon[31453]: pgmap v7947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:52.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:52 smithi153 ceph-mon[36569]: pgmap v7947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:52 smithi104 ceph-mon[36624]: pgmap v7947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:54.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:54 smithi081 ceph-mon[31453]: pgmap v7948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:54 smithi153 ceph-mon[36569]: pgmap v7948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:54 smithi104 ceph-mon[36624]: pgmap v7948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:56.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:56 smithi081 ceph-mon[31453]: pgmap v7949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:56.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:56 smithi153 ceph-mon[36569]: pgmap v7949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:56 smithi104 ceph-mon[36624]: pgmap v7949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:15:58 smithi081 ceph-mon[31453]: pgmap v7950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:58.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:15:58 smithi153 ceph-mon[36569]: pgmap v7950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:15:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:15:58 smithi104 ceph-mon[36624]: pgmap v7950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:00.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:00 smithi081 ceph-mon[31453]: pgmap v7951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:00 smithi153 ceph-mon[36569]: pgmap v7951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:00 smithi104 ceph-mon[36624]: pgmap v7951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:02.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:02 smithi081 ceph-mon[31453]: pgmap v7952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:02.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:02 smithi153 ceph-mon[36569]: pgmap v7952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:02 smithi104 ceph-mon[36624]: pgmap v7952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:03.785+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:16:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:03.785+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:16:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:03.785+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:16:04.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:04 smithi153 ceph-mon[36569]: pgmap v7953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:04 smithi104 ceph-mon[36624]: pgmap v7953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:04 smithi081 ceph-mon[31453]: pgmap v7953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:06.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:06 smithi153 ceph-mon[36569]: pgmap v7954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:06 smithi104 ceph-mon[36624]: pgmap v7954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:06 smithi081 ceph-mon[31453]: pgmap v7954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:08.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:08 smithi153 ceph-mon[36569]: pgmap v7955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:08 smithi104 ceph-mon[36624]: pgmap v7955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:08 smithi081 ceph-mon[31453]: pgmap v7955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:10.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:10 smithi153 ceph-mon[36569]: pgmap v7956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:10 smithi104 ceph-mon[36624]: pgmap v7956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:10 smithi081 ceph-mon[31453]: pgmap v7956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:12.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.128+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-13T01:16:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.128+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-13T01:16:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.128+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006330 tx=0x7f16f40c4000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-13T01:16:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.128+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006330 tx=0x7f16f40c4000 comp rx=0 tx=0).stop 2024-04-13T01:16:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.128+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-13T01:16:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.128+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T01:16:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.128+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5420 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T01:16:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.128+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5d00 0x7f16e80a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T01:16:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.129+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5d00 0x7f16e80a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T01:16:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:12.130+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5d00 0x7f16e80a4ea0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f40ba000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T01:16:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:12 smithi081 ceph-mon[31453]: pgmap v7957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:12.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:12 smithi153 ceph-mon[36569]: pgmap v7957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:12 smithi104 ceph-mon[36624]: pgmap v7957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:14.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:14 smithi153 ceph-mon[36569]: pgmap v7958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:14 smithi104 ceph-mon[36624]: pgmap v7958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:14 smithi081 ceph-mon[31453]: pgmap v7958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:15.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:15.081+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:16:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:15.081+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:16:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:15.082+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:16:15.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:15.082+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:16:16.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:16 smithi153 ceph-mon[36569]: pgmap v7959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:16 smithi104 ceph-mon[36624]: pgmap v7959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:16 smithi081 ceph-mon[31453]: pgmap v7959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:18.794+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:16:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:18.794+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:16:18.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:18.794+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:16:18.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:18 smithi153 ceph-mon[36569]: pgmap v7960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:18 smithi104 ceph-mon[36624]: pgmap v7960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:18 smithi081 ceph-mon[31453]: pgmap v7960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:20.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:20 smithi153 ceph-mon[36569]: pgmap v7961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:20 smithi104 ceph-mon[36624]: pgmap v7961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:20 smithi081 ceph-mon[31453]: pgmap v7961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:22 smithi153 ceph-mon[36569]: pgmap v7962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:22 smithi104 ceph-mon[36624]: pgmap v7962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:22 smithi081 ceph-mon[31453]: pgmap v7962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:23 smithi081 ceph-mon[31453]: pgmap v7963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:24.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:23 smithi153 ceph-mon[36569]: pgmap v7963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:23 smithi104 ceph-mon[36624]: pgmap v7963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:26 smithi081 ceph-mon[31453]: pgmap v7964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:26.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:26 smithi104 ceph-mon[36624]: pgmap v7964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:26.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:26 smithi153 ceph-mon[36569]: pgmap v7964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:28 smithi081 ceph-mon[31453]: pgmap v7965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:28.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:28 smithi153 ceph-mon[36569]: pgmap v7965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:28 smithi104 ceph-mon[36624]: pgmap v7965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:30 smithi081 ceph-mon[31453]: pgmap v7966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:30.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:30 smithi153 ceph-mon[36569]: pgmap v7966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:30 smithi104 ceph-mon[36624]: pgmap v7966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:32 smithi081 ceph-mon[31453]: pgmap v7967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:32 smithi153 ceph-mon[36569]: pgmap v7967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:32 smithi104 ceph-mon[36624]: pgmap v7967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:33.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:33.808+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:16:33.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:33.808+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:16:33.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:33.808+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:16:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:34 smithi081 ceph-mon[31453]: pgmap v7968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:34.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:34 smithi153 ceph-mon[36569]: pgmap v7968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:34 smithi104 ceph-mon[36624]: pgmap v7968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:36 smithi081 ceph-mon[31453]: pgmap v7969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:36.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:36 smithi153 ceph-mon[36569]: pgmap v7969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:36.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:36 smithi104 ceph-mon[36624]: pgmap v7969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:38 smithi081 ceph-mon[31453]: pgmap v7970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:16:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:38.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:38 smithi153 ceph-mon[36569]: pgmap v7970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:38.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:16:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:38 smithi104 ceph-mon[36624]: pgmap v7970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:16:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:16:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:16:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:16:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:16:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:16:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:16:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:16:40.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:40 smithi153 ceph-mon[36569]: pgmap v7971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:40 smithi153 ceph-mon[36569]: pgmap v7972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:40 smithi104 ceph-mon[36624]: pgmap v7971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:40 smithi104 ceph-mon[36624]: pgmap v7972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:40 smithi081 ceph-mon[31453]: pgmap v7971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:40 smithi081 ceph-mon[31453]: pgmap v7972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:42 smithi153 ceph-mon[36569]: pgmap v7973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:42 smithi104 ceph-mon[36624]: pgmap v7973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:42 smithi081 ceph-mon[31453]: pgmap v7973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:44.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:44 smithi153 ceph-mon[36569]: pgmap v7974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:44 smithi104 ceph-mon[36624]: pgmap v7974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:44 smithi081 ceph-mon[31453]: pgmap v7974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:46.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:46 smithi153 ceph-mon[36569]: pgmap v7975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:46 smithi104 ceph-mon[36624]: pgmap v7975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:46 smithi081 ceph-mon[31453]: pgmap v7975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:48.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:48 smithi153 ceph-mon[36569]: pgmap v7976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:48.823+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:16:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:48.823+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:16:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:16:48.823+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:16:48.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:48 smithi104 ceph-mon[36624]: pgmap v7976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:48 smithi081 ceph-mon[31453]: pgmap v7976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:50.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:50 smithi153 ceph-mon[36569]: pgmap v7977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:50 smithi104 ceph-mon[36624]: pgmap v7977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:50.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:50 smithi081 ceph-mon[31453]: pgmap v7977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:52.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:52 smithi153 ceph-mon[36569]: pgmap v7978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:52 smithi104 ceph-mon[36624]: pgmap v7978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:52 smithi081 ceph-mon[31453]: pgmap v7978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:54.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:54 smithi153 ceph-mon[36569]: pgmap v7979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:54 smithi104 ceph-mon[36624]: pgmap v7979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:54 smithi081 ceph-mon[31453]: pgmap v7979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:56 smithi153 ceph-mon[36569]: pgmap v7980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:56 smithi104 ceph-mon[36624]: pgmap v7980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:56.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:56 smithi081 ceph-mon[31453]: pgmap v7980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:16:58 smithi153 ceph-mon[36569]: pgmap v7981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:16:58 smithi104 ceph-mon[36624]: pgmap v7981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:16:58.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:16:58 smithi081 ceph-mon[31453]: pgmap v7981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:00 smithi153 ceph-mon[36569]: pgmap v7982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:00 smithi104 ceph-mon[36624]: pgmap v7982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:00 smithi081 ceph-mon[31453]: pgmap v7982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:02 smithi153 ceph-mon[36569]: pgmap v7983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:02 smithi104 ceph-mon[36624]: pgmap v7983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:02 smithi081 ceph-mon[31453]: pgmap v7983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:03.831+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:17:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:03.831+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:17:03.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:03.831+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:17:04.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:04 smithi153 ceph-mon[36569]: pgmap v7984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:04 smithi104 ceph-mon[36624]: pgmap v7984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:04 smithi081 ceph-mon[31453]: pgmap v7984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:06.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:06 smithi153 ceph-mon[36569]: pgmap v7985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:06 smithi104 ceph-mon[36624]: pgmap v7985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:06 smithi081 ceph-mon[31453]: pgmap v7985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:08 smithi153 ceph-mon[36569]: pgmap v7986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:08 smithi104 ceph-mon[36624]: pgmap v7986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:08 smithi081 ceph-mon[31453]: pgmap v7986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:10 smithi153 ceph-mon[36569]: pgmap v7987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:10 smithi104 ceph-mon[36624]: pgmap v7987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:10 smithi081 ceph-mon[31453]: pgmap v7987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:12.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:12 smithi153 ceph-mon[36569]: pgmap v7988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:12 smithi104 ceph-mon[36624]: pgmap v7988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:12 smithi081 ceph-mon[31453]: pgmap v7988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:14.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:14 smithi153 ceph-mon[36569]: pgmap v7989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:14 smithi104 ceph-mon[36624]: pgmap v7989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:14 smithi081 ceph-mon[31453]: pgmap v7989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:16 smithi153 ceph-mon[36569]: pgmap v7990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:16 smithi104 ceph-mon[36624]: pgmap v7990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:16 smithi081 ceph-mon[31453]: pgmap v7990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:18.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:18 smithi153 ceph-mon[36569]: pgmap v7991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:18.831 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:18.831+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:17:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:18.831+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:17:18.832 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:18.831+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:17:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:18 smithi104 ceph-mon[36624]: pgmap v7991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:18 smithi081 ceph-mon[31453]: pgmap v7991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:20.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:20 smithi153 ceph-mon[36569]: pgmap v7992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:20 smithi104 ceph-mon[36624]: pgmap v7992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:20 smithi081 ceph-mon[31453]: pgmap v7992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:22 smithi153 ceph-mon[36569]: pgmap v7993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:22 smithi104 ceph-mon[36624]: pgmap v7993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:22 smithi081 ceph-mon[31453]: pgmap v7993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:23 smithi081 ceph-mon[31453]: pgmap v7994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:24.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:23 smithi153 ceph-mon[36569]: pgmap v7994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:23 smithi104 ceph-mon[36624]: pgmap v7994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:26.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:26 smithi081 ceph-mon[31453]: pgmap v7995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:26.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:26 smithi153 ceph-mon[36569]: pgmap v7995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:26 smithi104 ceph-mon[36624]: pgmap v7995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:28.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:28 smithi081 ceph-mon[31453]: pgmap v7996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:28.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:28 smithi153 ceph-mon[36569]: pgmap v7996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:28 smithi104 ceph-mon[36624]: pgmap v7996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:30 smithi081 ceph-mon[31453]: pgmap v7997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:30 smithi153 ceph-mon[36569]: pgmap v7997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:30 smithi104 ceph-mon[36624]: pgmap v7997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:32.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:32 smithi081 ceph-mon[31453]: pgmap v7998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:32.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:32 smithi153 ceph-mon[36569]: pgmap v7998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:32 smithi104 ceph-mon[36624]: pgmap v7998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:33.847+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:17:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:33.847+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:17:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:33.847+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:17:34.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:34 smithi081 ceph-mon[31453]: pgmap v7999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:34 smithi153 ceph-mon[36569]: pgmap v7999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:34 smithi104 ceph-mon[36624]: pgmap v7999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:36.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:36 smithi081 ceph-mon[31453]: pgmap v8000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:36.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:36 smithi153 ceph-mon[36569]: pgmap v8000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:36 smithi104 ceph-mon[36624]: pgmap v8000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:38 smithi081 ceph-mon[31453]: pgmap v8001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:38.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:38 smithi153 ceph-mon[36569]: pgmap v8001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:38 smithi104 ceph-mon[36624]: pgmap v8001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:40.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:40 smithi081 ceph-mon[31453]: pgmap v8002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:40.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:17:40.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:40 smithi153 ceph-mon[36569]: pgmap v8002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:17:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:40 smithi104 ceph-mon[36624]: pgmap v8002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:17:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:17:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:17:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:17:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:17:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:17:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:17:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:17:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:17:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:17:42.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:42 smithi081 ceph-mon[31453]: pgmap v8003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:42.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:42 smithi081 ceph-mon[31453]: pgmap v8004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:42.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:42 smithi153 ceph-mon[36569]: pgmap v8003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:42 smithi153 ceph-mon[36569]: pgmap v8004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:42 smithi104 ceph-mon[36624]: pgmap v8003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:42 smithi104 ceph-mon[36624]: pgmap v8004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:44 smithi081 ceph-mon[31453]: pgmap v8005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:44.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:44 smithi153 ceph-mon[36569]: pgmap v8005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:44 smithi104 ceph-mon[36624]: pgmap v8005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:46.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:46 smithi081 ceph-mon[31453]: pgmap v8006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:46.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:46 smithi153 ceph-mon[36569]: pgmap v8006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:46 smithi104 ceph-mon[36624]: pgmap v8006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:48.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:48 smithi153 ceph-mon[36569]: pgmap v8007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:48 smithi104 ceph-mon[36624]: pgmap v8007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:48 smithi081 ceph-mon[31453]: pgmap v8007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:48.853+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:17:48.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:48.853+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:17:48.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:17:48.853+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:17:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:50 smithi153 ceph-mon[36569]: pgmap v8008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:50.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:50 smithi104 ceph-mon[36624]: pgmap v8008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:50 smithi081 ceph-mon[31453]: pgmap v8008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:52.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:52 smithi153 ceph-mon[36569]: pgmap v8009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:52.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:52 smithi104 ceph-mon[36624]: pgmap v8009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:52 smithi081 ceph-mon[31453]: pgmap v8009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:54 smithi153 ceph-mon[36569]: pgmap v8010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:54.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:54 smithi104 ceph-mon[36624]: pgmap v8010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:54.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:54 smithi081 ceph-mon[31453]: pgmap v8010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:56.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:56 smithi153 ceph-mon[36569]: pgmap v8011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:56.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:56 smithi104 ceph-mon[36624]: pgmap v8011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:56.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:56 smithi081 ceph-mon[31453]: pgmap v8011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:58.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:17:58 smithi153 ceph-mon[36569]: pgmap v8012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:58.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:17:58 smithi104 ceph-mon[36624]: pgmap v8012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:17:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:17:58 smithi081 ceph-mon[31453]: pgmap v8012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:00.458 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:00 smithi081 ceph-mon[31453]: pgmap v8013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:00.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:00 smithi153 ceph-mon[36569]: pgmap v8013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:00 smithi104 ceph-mon[36624]: pgmap v8013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:02.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:02 smithi153 ceph-mon[36569]: pgmap v8014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:02.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:02 smithi104 ceph-mon[36624]: pgmap v8014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:02.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:02 smithi081 ceph-mon[31453]: pgmap v8014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:03.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:03.856+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:18:03.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:03.856+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:18:03.855 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:03.856+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:18:04.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:04 smithi153 ceph-mon[36569]: pgmap v8015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:04.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:04 smithi104 ceph-mon[36624]: pgmap v8015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:04.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:04 smithi081 ceph-mon[31453]: pgmap v8015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:06 smithi153 ceph-mon[36569]: pgmap v8016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:06.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:06 smithi104 ceph-mon[36624]: pgmap v8016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:06 smithi081 ceph-mon[31453]: pgmap v8016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:08.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:08 smithi153 ceph-mon[36569]: pgmap v8017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:08 smithi104 ceph-mon[36624]: pgmap v8017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:08.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:08 smithi081 ceph-mon[31453]: pgmap v8017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:10.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:10 smithi153 ceph-mon[36569]: pgmap v8018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:10.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:10 smithi104 ceph-mon[36624]: pgmap v8018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:10.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:10 smithi081 ceph-mon[31453]: pgmap v8018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:12.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:12 smithi153 ceph-mon[36569]: pgmap v8019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:12.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:12 smithi104 ceph-mon[36624]: pgmap v8019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:12 smithi081 ceph-mon[31453]: pgmap v8019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:14.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:14 smithi153 ceph-mon[36569]: pgmap v8020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:14 smithi104 ceph-mon[36624]: pgmap v8020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:14.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:14 smithi081 ceph-mon[31453]: pgmap v8020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:16.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:16 smithi153 ceph-mon[36569]: pgmap v8021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:16 smithi104 ceph-mon[36624]: pgmap v8021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:16.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:16 smithi081 ceph-mon[31453]: pgmap v8021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:18 smithi153 ceph-mon[36569]: pgmap v8022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:18 smithi104 ceph-mon[36624]: pgmap v8022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:18 smithi081 ceph-mon[31453]: pgmap v8022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:18.869+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:18:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:18.869+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:18:18.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:18.869+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:18:20.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:20 smithi153 ceph-mon[36569]: pgmap v8023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:20 smithi104 ceph-mon[36624]: pgmap v8023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:20 smithi081 ceph-mon[31453]: pgmap v8023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:22 smithi153 ceph-mon[36569]: pgmap v8024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:22.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:22 smithi104 ceph-mon[36624]: pgmap v8024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:22.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:22 smithi081 ceph-mon[31453]: pgmap v8024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:24.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:24 smithi153 ceph-mon[36569]: pgmap v8025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:24 smithi104 ceph-mon[36624]: pgmap v8025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:24 smithi081 ceph-mon[31453]: pgmap v8025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:26.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:26 smithi153 ceph-mon[36569]: pgmap v8026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:26 smithi104 ceph-mon[36624]: pgmap v8026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:26 smithi081 ceph-mon[31453]: pgmap v8026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:28.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:28 smithi153 ceph-mon[36569]: pgmap v8027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:28 smithi104 ceph-mon[36624]: pgmap v8027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:28 smithi081 ceph-mon[31453]: pgmap v8027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:30.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:30 smithi153 ceph-mon[36569]: pgmap v8028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:30 smithi104 ceph-mon[36624]: pgmap v8028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:30 smithi081 ceph-mon[31453]: pgmap v8028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:32.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:32 smithi153 ceph-mon[36569]: pgmap v8029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:32 smithi104 ceph-mon[36624]: pgmap v8029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:32 smithi081 ceph-mon[31453]: pgmap v8029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:33.885+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:18:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:33.885+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:18:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:33.885+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:18:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:34 smithi153 ceph-mon[36569]: pgmap v8030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:34 smithi104 ceph-mon[36624]: pgmap v8030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:34 smithi081 ceph-mon[31453]: pgmap v8030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:36.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:36 smithi153 ceph-mon[36569]: pgmap v8031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:36 smithi104 ceph-mon[36624]: pgmap v8031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:36 smithi081 ceph-mon[31453]: pgmap v8031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:38.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:38 smithi153 ceph-mon[36569]: pgmap v8032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:38 smithi104 ceph-mon[36624]: pgmap v8032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:38 smithi081 ceph-mon[31453]: pgmap v8032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:40.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:40 smithi153 ceph-mon[36569]: pgmap v8033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:40 smithi104 ceph-mon[36624]: pgmap v8033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:40.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:40 smithi081 ceph-mon[31453]: pgmap v8033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:41.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:18:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:18:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:18:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:42 smithi104 ceph-mon[36624]: pgmap v8034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:18:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:18:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:18:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:42 smithi081 ceph-mon[31453]: pgmap v8034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:18:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:18:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:18:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:42 smithi153 ceph-mon[36569]: pgmap v8034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:18:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:18:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:18:43.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:43 smithi104 ceph-mon[36624]: pgmap v8035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:43.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:43 smithi104 ceph-mon[36624]: pgmap v8036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:43.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:43 smithi081 ceph-mon[31453]: pgmap v8035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:43.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:43 smithi081 ceph-mon[31453]: pgmap v8036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:43.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:43 smithi153 ceph-mon[36569]: pgmap v8035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:43 smithi153 ceph-mon[36569]: pgmap v8036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:45.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:45 smithi104 ceph-mon[36624]: pgmap v8037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:45 smithi081 ceph-mon[31453]: pgmap v8037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:45 smithi153 ceph-mon[36569]: pgmap v8037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:47 smithi104 ceph-mon[36624]: pgmap v8038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:47.607 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:47 smithi153 ceph-mon[36569]: pgmap v8038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:47.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:47 smithi081 ceph-mon[31453]: pgmap v8038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:48.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:48.900+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:18:48.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:48.900+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:18:48.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:18:48.900+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:18:49.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:49 smithi081 ceph-mon[31453]: pgmap v8039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:49.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:49 smithi153 ceph-mon[36569]: pgmap v8039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:49.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:49 smithi104 ceph-mon[36624]: pgmap v8039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:51.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:51 smithi081 ceph-mon[31453]: pgmap v8040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:51.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:51 smithi153 ceph-mon[36569]: pgmap v8040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:51.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:51 smithi104 ceph-mon[36624]: pgmap v8040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:53.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:53 smithi081 ceph-mon[31453]: pgmap v8041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:53.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:53 smithi153 ceph-mon[36569]: pgmap v8041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:53.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:53 smithi104 ceph-mon[36624]: pgmap v8041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:55.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:55 smithi081 ceph-mon[31453]: pgmap v8042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:55.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:55 smithi153 ceph-mon[36569]: pgmap v8042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:55 smithi104 ceph-mon[36624]: pgmap v8042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:57.633 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:57 smithi104 ceph-mon[36624]: pgmap v8043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:57 smithi081 ceph-mon[31453]: pgmap v8043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:57.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:57 smithi153 ceph-mon[36569]: pgmap v8043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:59.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:18:59 smithi081 ceph-mon[31453]: pgmap v8044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:59.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:18:59 smithi153 ceph-mon[36569]: pgmap v8044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:18:59.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:18:59 smithi104 ceph-mon[36624]: pgmap v8044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:01.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:01 smithi081 ceph-mon[31453]: pgmap v8045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:01.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:01 smithi153 ceph-mon[36569]: pgmap v8045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:01.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:01 smithi104 ceph-mon[36624]: pgmap v8045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:03.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:03 smithi081 ceph-mon[31453]: pgmap v8046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:03.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:03 smithi153 ceph-mon[36569]: pgmap v8046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:03 smithi104 ceph-mon[36624]: pgmap v8046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:03.912+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:19:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:03.912+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:19:03.912 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:03.912+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:19:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:05 smithi081 ceph-mon[31453]: pgmap v8047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:05.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:05 smithi153 ceph-mon[36569]: pgmap v8047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:05.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:05 smithi104 ceph-mon[36624]: pgmap v8047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:07 smithi081 ceph-mon[31453]: pgmap v8048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:07.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:07 smithi153 ceph-mon[36569]: pgmap v8048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:07.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:07 smithi104 ceph-mon[36624]: pgmap v8048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:09 smithi081 ceph-mon[31453]: pgmap v8049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:09.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:09 smithi153 ceph-mon[36569]: pgmap v8049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:09 smithi104 ceph-mon[36624]: pgmap v8049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:11 smithi081 ceph-mon[31453]: pgmap v8050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:11.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:11 smithi153 ceph-mon[36569]: pgmap v8050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:11.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:11 smithi104 ceph-mon[36624]: pgmap v8050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:13 smithi081 ceph-mon[31453]: pgmap v8051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:13.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:13 smithi153 ceph-mon[36569]: pgmap v8051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:13.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:13 smithi104 ceph-mon[36624]: pgmap v8051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:15.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:15.110+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:19:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:15.110+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:19:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:15.111+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:19:15.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:15.111+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:19:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:15 smithi081 ceph-mon[31453]: pgmap v8052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:15.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:15 smithi153 ceph-mon[36569]: pgmap v8052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:15.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:15 smithi104 ceph-mon[36624]: pgmap v8052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:17.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:17 smithi081 ceph-mon[31453]: pgmap v8053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:17.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:17 smithi153 ceph-mon[36569]: pgmap v8053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:17.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:17 smithi104 ceph-mon[36624]: pgmap v8053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:18.914+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:19:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:18.914+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:19:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:18.914+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:19:19.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:19 smithi153 ceph-mon[36569]: pgmap v8054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:19.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:19 smithi104 ceph-mon[36624]: pgmap v8054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:19 smithi081 ceph-mon[31453]: pgmap v8054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:21 smithi153 ceph-mon[36569]: pgmap v8055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:21.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:21 smithi104 ceph-mon[36624]: pgmap v8055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:21.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:21 smithi081 ceph-mon[31453]: pgmap v8055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:23.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:23 smithi153 ceph-mon[36569]: pgmap v8056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:23.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:23 smithi104 ceph-mon[36624]: pgmap v8056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:23 smithi081 ceph-mon[31453]: pgmap v8056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:25.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:25 smithi153 ceph-mon[36569]: pgmap v8057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:25 smithi104 ceph-mon[36624]: pgmap v8057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:25.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:25 smithi081 ceph-mon[31453]: pgmap v8057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:27.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:27 smithi153 ceph-mon[36569]: pgmap v8058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:27.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:27 smithi104 ceph-mon[36624]: pgmap v8058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:27.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:27 smithi081 ceph-mon[31453]: pgmap v8058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:29.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:29 smithi153 ceph-mon[36569]: pgmap v8059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:29.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:29 smithi104 ceph-mon[36624]: pgmap v8059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:29.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:29 smithi081 ceph-mon[31453]: pgmap v8059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:31.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:31 smithi081 ceph-mon[31453]: pgmap v8060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:31.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:31 smithi153 ceph-mon[36569]: pgmap v8060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:31 smithi104 ceph-mon[36624]: pgmap v8060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:33 smithi153 ceph-mon[36569]: pgmap v8061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:33.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:33 smithi104 ceph-mon[36624]: pgmap v8061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:33.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:33 smithi081 ceph-mon[31453]: pgmap v8061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:33.926 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:33.927+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:19:33.927 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:33.927+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:19:33.927 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:33.927+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:19:35.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:35 smithi153 ceph-mon[36569]: pgmap v8062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:35.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:35 smithi104 ceph-mon[36624]: pgmap v8062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:35 smithi081 ceph-mon[31453]: pgmap v8062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:37.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:37 smithi153 ceph-mon[36569]: pgmap v8063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:37 smithi104 ceph-mon[36624]: pgmap v8063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:37.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:37 smithi081 ceph-mon[31453]: pgmap v8063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:39.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:39 smithi153 ceph-mon[36569]: pgmap v8064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:39 smithi104 ceph-mon[36624]: pgmap v8064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:39 smithi081 ceph-mon[31453]: pgmap v8064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:41.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:41 smithi153 ceph-mon[36569]: pgmap v8065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:41.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:41 smithi104 ceph-mon[36624]: pgmap v8065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:41.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:41 smithi081 ceph-mon[31453]: pgmap v8065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:43.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:43 smithi153 ceph-mon[36569]: pgmap v8066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:19:43.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:43 smithi104 ceph-mon[36624]: pgmap v8066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:43.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:19:43.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:43 smithi081 ceph-mon[31453]: pgmap v8066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:43.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:19:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:19:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:19:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:19:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:19:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:19:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:19:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:19:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:19:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:19:45.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:45 smithi153 ceph-mon[36569]: pgmap v8067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:45 smithi153 ceph-mon[36569]: pgmap v8068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:45.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:45 smithi104 ceph-mon[36624]: pgmap v8067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:45.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:45 smithi104 ceph-mon[36624]: pgmap v8068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:45.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:45 smithi081 ceph-mon[31453]: pgmap v8067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:45.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:45 smithi081 ceph-mon[31453]: pgmap v8068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:47.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:47 smithi153 ceph-mon[36569]: pgmap v8069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:47.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:47 smithi104 ceph-mon[36624]: pgmap v8069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:47.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:47 smithi081 ceph-mon[31453]: pgmap v8069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:48.943+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:19:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:48.943+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:19:48.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:19:48.943+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:19:49.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:49 smithi153 ceph-mon[36569]: pgmap v8070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:49.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:49 smithi104 ceph-mon[36624]: pgmap v8070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:49.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:49 smithi081 ceph-mon[31453]: pgmap v8070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:51.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:51 smithi153 ceph-mon[36569]: pgmap v8071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:51.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:51 smithi104 ceph-mon[36624]: pgmap v8071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:51.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:51 smithi081 ceph-mon[31453]: pgmap v8071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:53 smithi153 ceph-mon[36569]: pgmap v8072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:53.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:53 smithi104 ceph-mon[36624]: pgmap v8072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:53.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:53 smithi081 ceph-mon[31453]: pgmap v8072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:55.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:55 smithi153 ceph-mon[36569]: pgmap v8073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:55 smithi104 ceph-mon[36624]: pgmap v8073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:55.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:55 smithi081 ceph-mon[31453]: pgmap v8073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:57.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:57 smithi153 ceph-mon[36569]: pgmap v8074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:57.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:57 smithi104 ceph-mon[36624]: pgmap v8074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:57.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:57 smithi081 ceph-mon[31453]: pgmap v8074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:59.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:19:59 smithi153 ceph-mon[36569]: pgmap v8075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:59.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:19:59 smithi104 ceph-mon[36624]: pgmap v8075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:19:59.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:19:59 smithi081 ceph-mon[31453]: pgmap v8075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:20:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:20:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:20:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:01 smithi153 ceph-mon[36569]: pgmap v8076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:01.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:01 smithi081 ceph-mon[31453]: pgmap v8076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:01.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:01 smithi104 ceph-mon[36624]: pgmap v8076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:03.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:03 smithi153 ceph-mon[36569]: pgmap v8077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:03 smithi104 ceph-mon[36624]: pgmap v8077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:03 smithi081 ceph-mon[31453]: pgmap v8077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:03.958+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:03.958+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:03.959+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:20:05.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:05 smithi153 ceph-mon[36569]: pgmap v8078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:05.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:05 smithi104 ceph-mon[36624]: pgmap v8078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:05.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:05 smithi081 ceph-mon[31453]: pgmap v8078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:07.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:07 smithi153 ceph-mon[36569]: pgmap v8079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:07.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:07 smithi104 ceph-mon[36624]: pgmap v8079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:07.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:07 smithi081 ceph-mon[31453]: pgmap v8079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:09 smithi104 ceph-mon[36624]: pgmap v8080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:09.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:09 smithi081 ceph-mon[31453]: pgmap v8080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:10.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:09 smithi153 ceph-mon[36569]: pgmap v8080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:11.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:11 smithi104 ceph-mon[36624]: pgmap v8081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:11.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:11 smithi081 ceph-mon[31453]: pgmap v8081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:11 smithi153 ceph-mon[36569]: pgmap v8081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:13.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:13 smithi104 ceph-mon[36624]: pgmap v8082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:13.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:13 smithi081 ceph-mon[31453]: pgmap v8082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:14.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:13 smithi153 ceph-mon[36569]: pgmap v8082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:15.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:15 smithi104 ceph-mon[36624]: pgmap v8083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:15.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:15 smithi081 ceph-mon[31453]: pgmap v8083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:16.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:15 smithi153 ceph-mon[36569]: pgmap v8083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:17.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:17 smithi104 ceph-mon[36624]: pgmap v8084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:17.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:17 smithi153 ceph-mon[36569]: pgmap v8084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:17.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:17 smithi081 ceph-mon[31453]: pgmap v8084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:18 smithi081 ceph-mon[31453]: pgmap v8085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:18.971+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:20:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:18.971+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:20:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:18.971+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:20:19.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:18 smithi153 ceph-mon[36569]: pgmap v8085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:19.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:18 smithi104 ceph-mon[36624]: pgmap v8085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:21.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:20 smithi153 ceph-mon[36569]: pgmap v8086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:21.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:20 smithi104 ceph-mon[36624]: pgmap v8086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:20 smithi081 ceph-mon[31453]: pgmap v8086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:22 smithi153 ceph-mon[36569]: pgmap v8087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:23.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:22 smithi104 ceph-mon[36624]: pgmap v8087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:23.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:22 smithi081 ceph-mon[31453]: pgmap v8087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:25.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:24 smithi153 ceph-mon[36569]: pgmap v8088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:24 smithi104 ceph-mon[36624]: pgmap v8088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:24 smithi081 ceph-mon[31453]: pgmap v8088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:27.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:26 smithi153 ceph-mon[36569]: pgmap v8089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:26 smithi104 ceph-mon[36624]: pgmap v8089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:26 smithi081 ceph-mon[31453]: pgmap v8089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:29.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:28 smithi153 ceph-mon[36569]: pgmap v8090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:28 smithi104 ceph-mon[36624]: pgmap v8090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:28 smithi081 ceph-mon[31453]: pgmap v8090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:31.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:30 smithi081 ceph-mon[31453]: pgmap v8091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:31.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:30 smithi153 ceph-mon[36569]: pgmap v8091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:30 smithi104 ceph-mon[36624]: pgmap v8091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:33.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:32 smithi153 ceph-mon[36569]: pgmap v8092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:33.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:32 smithi104 ceph-mon[36624]: pgmap v8092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:32 smithi081 ceph-mon[31453]: pgmap v8092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:33.987+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:20:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:33.987+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:20:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:33.987+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:20:35.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:34 smithi153 ceph-mon[36569]: pgmap v8093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:34 smithi104 ceph-mon[36624]: pgmap v8093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:34 smithi081 ceph-mon[31453]: pgmap v8093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:37.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:36 smithi153 ceph-mon[36569]: pgmap v8094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:36 smithi104 ceph-mon[36624]: pgmap v8094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:36 smithi081 ceph-mon[31453]: pgmap v8094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:39.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:38 smithi153 ceph-mon[36569]: pgmap v8095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:38 smithi104 ceph-mon[36624]: pgmap v8095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:38 smithi081 ceph-mon[31453]: pgmap v8095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:41.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:40 smithi153 ceph-mon[36569]: pgmap v8096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:41.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:40 smithi104 ceph-mon[36624]: pgmap v8096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:40 smithi081 ceph-mon[31453]: pgmap v8096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:42 smithi153 ceph-mon[36569]: pgmap v8097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:42 smithi104 ceph-mon[36624]: pgmap v8097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:42 smithi081 ceph-mon[31453]: pgmap v8097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:45.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:44 smithi153 ceph-mon[36569]: pgmap v8098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:20:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:44 smithi104 ceph-mon[36624]: pgmap v8098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:20:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:44 smithi081 ceph-mon[31453]: pgmap v8098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:20:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:20:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:20:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:45 smithi153 ceph-mon[36569]: pgmap v8099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:45 smithi153 ceph-mon[36569]: pgmap v8100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:20:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:20:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:20:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:45 smithi104 ceph-mon[36624]: pgmap v8099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:45 smithi104 ceph-mon[36624]: pgmap v8100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:20:46.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:20:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:20:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:45 smithi081 ceph-mon[31453]: pgmap v8099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:45 smithi081 ceph-mon[31453]: pgmap v8100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:20:48.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:48 smithi153 ceph-mon[36569]: pgmap v8101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:48.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:48 smithi104 ceph-mon[36624]: pgmap v8101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:48 smithi081 ceph-mon[31453]: pgmap v8101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:48.988+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:20:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:48.988+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:20:48.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:20:48.988+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:20:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:50 smithi153 ceph-mon[36569]: pgmap v8102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:50 smithi104 ceph-mon[36624]: pgmap v8102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:50.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:50 smithi081 ceph-mon[31453]: pgmap v8102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:52.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:52 smithi153 ceph-mon[36569]: pgmap v8103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:52 smithi104 ceph-mon[36624]: pgmap v8103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:52.870 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:52 smithi081 ceph-mon[31453]: pgmap v8103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:54.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:54 smithi153 ceph-mon[36569]: pgmap v8104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:54 smithi104 ceph-mon[36624]: pgmap v8104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:54 smithi081 ceph-mon[31453]: pgmap v8104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:56.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:56 smithi153 ceph-mon[36569]: pgmap v8105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:56 smithi104 ceph-mon[36624]: pgmap v8105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:56.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:56 smithi081 ceph-mon[31453]: pgmap v8105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:20:58 smithi153 ceph-mon[36569]: pgmap v8106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:20:58 smithi104 ceph-mon[36624]: pgmap v8106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:20:58.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:20:58 smithi081 ceph-mon[31453]: pgmap v8106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:00.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:00 smithi153 ceph-mon[36569]: pgmap v8107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:00 smithi104 ceph-mon[36624]: pgmap v8107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:00 smithi081 ceph-mon[31453]: pgmap v8107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:02.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:02 smithi153 ceph-mon[36569]: pgmap v8108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:02 smithi104 ceph-mon[36624]: pgmap v8108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:02 smithi081 ceph-mon[31453]: pgmap v8108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:04.005+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:21:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:04.005+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:21:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:04.005+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:21:04.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:04 smithi153 ceph-mon[36569]: pgmap v8109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:04 smithi104 ceph-mon[36624]: pgmap v8109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:04 smithi081 ceph-mon[31453]: pgmap v8109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:06.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:06 smithi153 ceph-mon[36569]: pgmap v8110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:06 smithi104 ceph-mon[36624]: pgmap v8110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:06 smithi081 ceph-mon[31453]: pgmap v8110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:08.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:08 smithi153 ceph-mon[36569]: pgmap v8111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:08 smithi104 ceph-mon[36624]: pgmap v8111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:08 smithi081 ceph-mon[31453]: pgmap v8111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:10 smithi153 ceph-mon[36569]: pgmap v8112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:10 smithi104 ceph-mon[36624]: pgmap v8112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:10 smithi081 ceph-mon[31453]: pgmap v8112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:12.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:12 smithi153 ceph-mon[36569]: pgmap v8113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:12 smithi104 ceph-mon[36624]: pgmap v8113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:12 smithi081 ceph-mon[31453]: pgmap v8113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:14.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:14 smithi153 ceph-mon[36569]: pgmap v8114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:14 smithi104 ceph-mon[36624]: pgmap v8114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:14 smithi081 ceph-mon[31453]: pgmap v8114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:16.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:16 smithi153 ceph-mon[36569]: pgmap v8115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:16 smithi104 ceph-mon[36624]: pgmap v8115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:16 smithi081 ceph-mon[31453]: pgmap v8115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:18 smithi153 ceph-mon[36569]: pgmap v8116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:18 smithi104 ceph-mon[36624]: pgmap v8116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:18 smithi081 ceph-mon[31453]: pgmap v8116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:19.020 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:19.020+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:21:19.020 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:19.021+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:21:19.020 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:19.021+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:21:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:20 smithi153 ceph-mon[36569]: pgmap v8117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:20 smithi104 ceph-mon[36624]: pgmap v8117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:20 smithi081 ceph-mon[31453]: pgmap v8117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:22.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:22 smithi153 ceph-mon[36569]: pgmap v8118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:22 smithi104 ceph-mon[36624]: pgmap v8118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:22 smithi081 ceph-mon[31453]: pgmap v8118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:24.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:24 smithi153 ceph-mon[36569]: pgmap v8119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:24 smithi104 ceph-mon[36624]: pgmap v8119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:24 smithi081 ceph-mon[31453]: pgmap v8119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:26.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:26 smithi153 ceph-mon[36569]: pgmap v8120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:26 smithi104 ceph-mon[36624]: pgmap v8120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:26.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:26 smithi081 ceph-mon[31453]: pgmap v8120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:28.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:28 smithi153 ceph-mon[36569]: pgmap v8121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:28 smithi104 ceph-mon[36624]: pgmap v8121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:28.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:28 smithi081 ceph-mon[31453]: pgmap v8121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:30.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:30 smithi153 ceph-mon[36569]: pgmap v8122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:30 smithi104 ceph-mon[36624]: pgmap v8122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:30 smithi081 ceph-mon[31453]: pgmap v8122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:32 smithi104 ceph-mon[36624]: pgmap v8123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:32.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:32 smithi081 ceph-mon[31453]: pgmap v8123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:33.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:32 smithi153 ceph-mon[36569]: pgmap v8123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:34.036+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:21:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:34.036+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:21:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:34.036+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:21:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:34 smithi104 ceph-mon[36624]: pgmap v8124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:34 smithi081 ceph-mon[31453]: pgmap v8124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:35.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:34 smithi153 ceph-mon[36569]: pgmap v8124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:36.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:36 smithi104 ceph-mon[36624]: pgmap v8125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:36.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:36 smithi081 ceph-mon[31453]: pgmap v8125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:37.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:36 smithi153 ceph-mon[36569]: pgmap v8125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:38 smithi104 ceph-mon[36624]: pgmap v8126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:38 smithi081 ceph-mon[31453]: pgmap v8126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:39.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:38 smithi153 ceph-mon[36569]: pgmap v8126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:40 smithi104 ceph-mon[36624]: pgmap v8127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:40 smithi081 ceph-mon[31453]: pgmap v8127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:40 smithi153 ceph-mon[36569]: pgmap v8127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:42 smithi081 ceph-mon[31453]: pgmap v8128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:43.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:42 smithi153 ceph-mon[36569]: pgmap v8128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:42 smithi104 ceph-mon[36624]: pgmap v8128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:44 smithi081 ceph-mon[31453]: pgmap v8129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:45.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:44 smithi153 ceph-mon[36569]: pgmap v8129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:44 smithi104 ceph-mon[36624]: pgmap v8129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: pgmap v8130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: pgmap v8130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: pgmap v8130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:21:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:21:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:21:48.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:21:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:21:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:21:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:21:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:21:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:21:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:21:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:21:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:21:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:21:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:48 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:21:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:48 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:21:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:48 smithi081 ceph-mon[31453]: pgmap v8131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:48 smithi081 ceph-mon[31453]: pgmap v8132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:48.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:48 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:21:48.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:48 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:21:48.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:48 smithi153 ceph-mon[36569]: pgmap v8131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:48.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:48 smithi153 ceph-mon[36569]: pgmap v8132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:49.052+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:21:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:49.052+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:21:49.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:21:49.052+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:21:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:48 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:21:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:48 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:21:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:48 smithi104 ceph-mon[36624]: pgmap v8131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:48 smithi104 ceph-mon[36624]: pgmap v8132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:49.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:49 smithi081 ceph-mon[31453]: pgmap v8133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:50.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:49 smithi153 ceph-mon[36569]: pgmap v8133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:49 smithi104 ceph-mon[36624]: pgmap v8133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:52.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:52 smithi104 ceph-mon[36624]: pgmap v8134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:52 smithi081 ceph-mon[31453]: pgmap v8134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:52.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:52 smithi153 ceph-mon[36569]: pgmap v8134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:54.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:54 smithi104 ceph-mon[36624]: pgmap v8135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:54 smithi081 ceph-mon[31453]: pgmap v8135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:54.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:54 smithi153 ceph-mon[36569]: pgmap v8135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:56.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:56 smithi104 ceph-mon[36624]: pgmap v8136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:56 smithi081 ceph-mon[31453]: pgmap v8136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:56.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:56 smithi153 ceph-mon[36569]: pgmap v8136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:58.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:21:58 smithi104 ceph-mon[36624]: pgmap v8137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:58.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:21:58 smithi081 ceph-mon[31453]: pgmap v8137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:21:58.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:21:58 smithi153 ceph-mon[36569]: pgmap v8137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:00.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:00 smithi081 ceph-mon[31453]: pgmap v8138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:00.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:00 smithi153 ceph-mon[36569]: pgmap v8138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:00 smithi104 ceph-mon[36624]: pgmap v8138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:02 smithi081 ceph-mon[31453]: pgmap v8139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:02 smithi153 ceph-mon[36569]: pgmap v8139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:02.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:02 smithi104 ceph-mon[36624]: pgmap v8139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:04.065+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:22:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:04.065+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:22:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:04.065+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:22:04.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:04 smithi081 ceph-mon[31453]: pgmap v8140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:04 smithi153 ceph-mon[36569]: pgmap v8140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:04.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:04 smithi104 ceph-mon[36624]: pgmap v8140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:06.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:06 smithi081 ceph-mon[31453]: pgmap v8141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:06 smithi153 ceph-mon[36569]: pgmap v8141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:06.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:06 smithi104 ceph-mon[36624]: pgmap v8141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:08.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:08 smithi081 ceph-mon[31453]: pgmap v8142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:08.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:08 smithi153 ceph-mon[36569]: pgmap v8142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:08 smithi104 ceph-mon[36624]: pgmap v8142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:10 smithi081 ceph-mon[31453]: pgmap v8143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:10 smithi153 ceph-mon[36569]: pgmap v8143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:10.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:10 smithi104 ceph-mon[36624]: pgmap v8143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:12.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:12 smithi081 ceph-mon[31453]: pgmap v8144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:12.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:12 smithi153 ceph-mon[36569]: pgmap v8144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:12.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:12 smithi104 ceph-mon[36624]: pgmap v8144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:14.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:14 smithi081 ceph-mon[31453]: pgmap v8145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:14.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:14 smithi153 ceph-mon[36569]: pgmap v8145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:14 smithi104 ceph-mon[36624]: pgmap v8145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:15.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:15.140+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:22:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:15.140+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:22:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:15.141+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:22:15.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:15.141+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:22:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:16 smithi081 ceph-mon[31453]: pgmap v8146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:16 smithi153 ceph-mon[36569]: pgmap v8146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:16 smithi104 ceph-mon[36624]: pgmap v8146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:18.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:18 smithi081 ceph-mon[31453]: pgmap v8147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:18 smithi153 ceph-mon[36569]: pgmap v8147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:18 smithi104 ceph-mon[36624]: pgmap v8147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:19.080+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:22:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:19.080+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:22:19.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:19.080+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:22:20.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:20 smithi081 ceph-mon[31453]: pgmap v8148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:20.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:20 smithi153 ceph-mon[36569]: pgmap v8148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:20 smithi104 ceph-mon[36624]: pgmap v8148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:22.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:22 smithi081 ceph-mon[31453]: pgmap v8149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:22.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:22 smithi153 ceph-mon[36569]: pgmap v8149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:22.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:22 smithi104 ceph-mon[36624]: pgmap v8149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:24.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:24 smithi081 ceph-mon[31453]: pgmap v8150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:24.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:24 smithi153 ceph-mon[36569]: pgmap v8150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:24 smithi104 ceph-mon[36624]: pgmap v8150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:26.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:26 smithi081 ceph-mon[31453]: pgmap v8151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:26.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:26 smithi153 ceph-mon[36569]: pgmap v8151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:26 smithi104 ceph-mon[36624]: pgmap v8151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:28 smithi153 ceph-mon[36569]: pgmap v8152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:28 smithi104 ceph-mon[36624]: pgmap v8152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:28 smithi081 ceph-mon[31453]: pgmap v8152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:30 smithi153 ceph-mon[36569]: pgmap v8153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:30 smithi104 ceph-mon[36624]: pgmap v8153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:30 smithi081 ceph-mon[31453]: pgmap v8153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:32.458 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:32 smithi081 ceph-mon[31453]: pgmap v8154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:32.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:32 smithi153 ceph-mon[36569]: pgmap v8154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:32 smithi104 ceph-mon[36624]: pgmap v8154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:34.095+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:22:34.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:34.095+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:22:34.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:34.095+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:22:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:34 smithi153 ceph-mon[36569]: pgmap v8155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:34 smithi104 ceph-mon[36624]: pgmap v8155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:34 smithi081 ceph-mon[31453]: pgmap v8155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:36 smithi153 ceph-mon[36569]: pgmap v8156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:36 smithi104 ceph-mon[36624]: pgmap v8156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:36 smithi081 ceph-mon[31453]: pgmap v8156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:38.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:38 smithi153 ceph-mon[36569]: pgmap v8157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:38 smithi104 ceph-mon[36624]: pgmap v8157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:38 smithi081 ceph-mon[31453]: pgmap v8157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:40 smithi153 ceph-mon[36569]: pgmap v8158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:40 smithi104 ceph-mon[36624]: pgmap v8158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:40.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:40 smithi081 ceph-mon[31453]: pgmap v8158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:42.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:42 smithi153 ceph-mon[36569]: pgmap v8159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:42 smithi104 ceph-mon[36624]: pgmap v8159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:42 smithi081 ceph-mon[31453]: pgmap v8159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:44.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:44 smithi153 ceph-mon[36569]: pgmap v8160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:44 smithi104 ceph-mon[36624]: pgmap v8160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:44.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:44 smithi081 ceph-mon[31453]: pgmap v8160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:46.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:46 smithi153 ceph-mon[36569]: pgmap v8161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:46 smithi104 ceph-mon[36624]: pgmap v8161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:46 smithi081 ceph-mon[31453]: pgmap v8161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:48.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:48 smithi153 ceph-mon[36569]: pgmap v8162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:22:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:48 smithi104 ceph-mon[36624]: pgmap v8162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:22:48.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:48 smithi081 ceph-mon[31453]: pgmap v8162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:22:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:49.101+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:22:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:49.101+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:22:49.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:22:49.101+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:22:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:22:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:22:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:49 smithi153 ceph-mon[36569]: pgmap v8163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:49 smithi153 ceph-mon[36569]: pgmap v8164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:22:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:22:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:22:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:49 smithi104 ceph-mon[36624]: pgmap v8163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:49 smithi104 ceph-mon[36624]: pgmap v8164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:22:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:22:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:22:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:49 smithi081 ceph-mon[31453]: pgmap v8163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:49 smithi081 ceph-mon[31453]: pgmap v8164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:22:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:50 smithi153 ceph-mon[36569]: pgmap v8165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:50 smithi104 ceph-mon[36624]: pgmap v8165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:50 smithi081 ceph-mon[31453]: pgmap v8165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:52 smithi153 ceph-mon[36569]: pgmap v8166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:52 smithi104 ceph-mon[36624]: pgmap v8166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:52 smithi081 ceph-mon[31453]: pgmap v8166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:55.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:54 smithi153 ceph-mon[36569]: pgmap v8167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:54 smithi104 ceph-mon[36624]: pgmap v8167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:54 smithi081 ceph-mon[31453]: pgmap v8167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:57.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:56 smithi153 ceph-mon[36569]: pgmap v8168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:56 smithi104 ceph-mon[36624]: pgmap v8168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:56 smithi081 ceph-mon[31453]: pgmap v8168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:22:58 smithi153 ceph-mon[36569]: pgmap v8169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:22:58 smithi104 ceph-mon[36624]: pgmap v8169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:22:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:22:58 smithi081 ceph-mon[31453]: pgmap v8169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:00 smithi153 ceph-mon[36569]: pgmap v8170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:00 smithi104 ceph-mon[36624]: pgmap v8170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:00 smithi081 ceph-mon[31453]: pgmap v8170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:02.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:02 smithi081 ceph-mon[31453]: pgmap v8171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:02 smithi153 ceph-mon[36569]: pgmap v8171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:02 smithi104 ceph-mon[36624]: pgmap v8171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:04.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:04.102+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:23:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:04.102+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:23:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:04.102+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:23:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:04 smithi153 ceph-mon[36569]: pgmap v8172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:04 smithi104 ceph-mon[36624]: pgmap v8172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:04 smithi081 ceph-mon[31453]: pgmap v8172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:06 smithi153 ceph-mon[36569]: pgmap v8173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:07.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:06 smithi104 ceph-mon[36624]: pgmap v8173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:06 smithi081 ceph-mon[31453]: pgmap v8173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:08 smithi153 ceph-mon[36569]: pgmap v8174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:09.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:08 smithi104 ceph-mon[36624]: pgmap v8174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:08 smithi081 ceph-mon[31453]: pgmap v8174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:10 smithi153 ceph-mon[36569]: pgmap v8175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:10 smithi104 ceph-mon[36624]: pgmap v8175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:10 smithi081 ceph-mon[31453]: pgmap v8175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:12 smithi153 ceph-mon[36569]: pgmap v8176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:12 smithi104 ceph-mon[36624]: pgmap v8176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:12 smithi081 ceph-mon[31453]: pgmap v8176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:14 smithi153 ceph-mon[36569]: pgmap v8177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:15.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:14 smithi104 ceph-mon[36624]: pgmap v8177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:14 smithi081 ceph-mon[31453]: pgmap v8177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:17.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:16 smithi153 ceph-mon[36569]: pgmap v8178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:17.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:16 smithi104 ceph-mon[36624]: pgmap v8178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:16 smithi081 ceph-mon[31453]: pgmap v8178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:18 smithi153 ceph-mon[36569]: pgmap v8179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:19.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:18 smithi104 ceph-mon[36624]: pgmap v8179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:19.116+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:19.116+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:23:19.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:19.116+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:23:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:18 smithi081 ceph-mon[31453]: pgmap v8179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:20 smithi153 ceph-mon[36569]: pgmap v8180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:20 smithi104 ceph-mon[36624]: pgmap v8180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:20 smithi081 ceph-mon[31453]: pgmap v8180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:22 smithi153 ceph-mon[36569]: pgmap v8181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:22 smithi104 ceph-mon[36624]: pgmap v8181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:22 smithi081 ceph-mon[31453]: pgmap v8181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:24 smithi153 ceph-mon[36569]: pgmap v8182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:24 smithi104 ceph-mon[36624]: pgmap v8182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:24 smithi081 ceph-mon[31453]: pgmap v8182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:26 smithi153 ceph-mon[36569]: pgmap v8183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:26 smithi104 ceph-mon[36624]: pgmap v8183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:26 smithi081 ceph-mon[31453]: pgmap v8183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:29.031 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:28 smithi081 ceph-mon[31453]: pgmap v8184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:29.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:28 smithi153 ceph-mon[36569]: pgmap v8184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:28 smithi104 ceph-mon[36624]: pgmap v8184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:30 smithi153 ceph-mon[36569]: pgmap v8185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:30 smithi104 ceph-mon[36624]: pgmap v8185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:30 smithi081 ceph-mon[31453]: pgmap v8185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:32.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:32 smithi081 ceph-mon[31453]: pgmap v8186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:33.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:32 smithi153 ceph-mon[36569]: pgmap v8186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:32 smithi104 ceph-mon[36624]: pgmap v8186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:34.121+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:23:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:34.121+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:23:34.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:34.122+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:23:35.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:34 smithi153 ceph-mon[36569]: pgmap v8187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:34 smithi104 ceph-mon[36624]: pgmap v8187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:34 smithi081 ceph-mon[31453]: pgmap v8187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:36 smithi153 ceph-mon[36569]: pgmap v8188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:36 smithi104 ceph-mon[36624]: pgmap v8188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:36 smithi081 ceph-mon[31453]: pgmap v8188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:38 smithi153 ceph-mon[36569]: pgmap v8189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:38 smithi104 ceph-mon[36624]: pgmap v8189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:38 smithi081 ceph-mon[31453]: pgmap v8189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:40 smithi153 ceph-mon[36569]: pgmap v8190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:40 smithi104 ceph-mon[36624]: pgmap v8190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:40 smithi081 ceph-mon[31453]: pgmap v8190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:42 smithi153 ceph-mon[36569]: pgmap v8191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:42 smithi104 ceph-mon[36624]: pgmap v8191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:42 smithi081 ceph-mon[31453]: pgmap v8191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:45.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:44 smithi153 ceph-mon[36569]: pgmap v8192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:44 smithi104 ceph-mon[36624]: pgmap v8192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:45.155+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:23:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:45.155+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:23:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:45.155+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:23:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:45.155+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:23:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:44 smithi081 ceph-mon[31453]: pgmap v8192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:46 smithi153 ceph-mon[36569]: pgmap v8193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:46 smithi104 ceph-mon[36624]: pgmap v8193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:46 smithi081 ceph-mon[31453]: pgmap v8193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:48 smithi153 ceph-mon[36569]: pgmap v8194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:48 smithi104 ceph-mon[36624]: pgmap v8194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:49.137+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:23:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:49.137+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:23:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:23:49.137+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:23:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:48 smithi081 ceph-mon[31453]: pgmap v8194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:50.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:23:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:23:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:23:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:23:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:23:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:50 smithi153 ceph-mon[36569]: pgmap v8195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:50 smithi153 ceph-mon[36569]: pgmap v8196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:23:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:23:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:23:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:50 smithi104 ceph-mon[36624]: pgmap v8195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:50 smithi104 ceph-mon[36624]: pgmap v8196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:23:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:23:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:23:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:50 smithi081 ceph-mon[31453]: pgmap v8195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:50 smithi081 ceph-mon[31453]: pgmap v8196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:23:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:53 smithi153 ceph-mon[36569]: pgmap v8197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:53.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:53 smithi104 ceph-mon[36624]: pgmap v8197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:53.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:53 smithi081 ceph-mon[31453]: pgmap v8197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:55 smithi153 ceph-mon[36569]: pgmap v8198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:55.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:55 smithi104 ceph-mon[36624]: pgmap v8198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:55.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:55 smithi081 ceph-mon[31453]: pgmap v8198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:57 smithi153 ceph-mon[36569]: pgmap v8199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:57.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:57 smithi104 ceph-mon[36624]: pgmap v8199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:57 smithi081 ceph-mon[31453]: pgmap v8199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:23:59 smithi153 ceph-mon[36569]: pgmap v8200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:59.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:23:59 smithi104 ceph-mon[36624]: pgmap v8200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:23:59.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:23:59 smithi081 ceph-mon[31453]: pgmap v8200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:01 smithi153 ceph-mon[36569]: pgmap v8201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:01.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:01 smithi104 ceph-mon[36624]: pgmap v8201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:01.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:01 smithi081 ceph-mon[31453]: pgmap v8201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:03 smithi153 ceph-mon[36569]: pgmap v8202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:03 smithi104 ceph-mon[36624]: pgmap v8202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:03.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:03 smithi081 ceph-mon[31453]: pgmap v8202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:04.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:04.150+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:24:04.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:04.150+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:24:04.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:04.150+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:24:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:05 smithi153 ceph-mon[36569]: pgmap v8203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:05 smithi104 ceph-mon[36624]: pgmap v8203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:05.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:05 smithi081 ceph-mon[31453]: pgmap v8203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:07 smithi153 ceph-mon[36569]: pgmap v8204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:07 smithi104 ceph-mon[36624]: pgmap v8204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:07 smithi081 ceph-mon[31453]: pgmap v8204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:09 smithi153 ceph-mon[36569]: pgmap v8205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:09 smithi104 ceph-mon[36624]: pgmap v8205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:09 smithi081 ceph-mon[31453]: pgmap v8205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:11 smithi153 ceph-mon[36569]: pgmap v8206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:11 smithi104 ceph-mon[36624]: pgmap v8206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:11 smithi081 ceph-mon[31453]: pgmap v8206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:13 smithi153 ceph-mon[36569]: pgmap v8207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:13 smithi104 ceph-mon[36624]: pgmap v8207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:13 smithi081 ceph-mon[31453]: pgmap v8207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:15 smithi081 ceph-mon[31453]: pgmap v8208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:15 smithi153 ceph-mon[36569]: pgmap v8208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:15 smithi104 ceph-mon[36624]: pgmap v8208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:17 smithi153 ceph-mon[36569]: pgmap v8209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:17 smithi104 ceph-mon[36624]: pgmap v8209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:17.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:17 smithi081 ceph-mon[31453]: pgmap v8209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:19.165+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:24:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:19.165+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:24:19.165 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:19.165+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:24:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:19 smithi104 ceph-mon[36624]: pgmap v8210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:19 smithi081 ceph-mon[31453]: pgmap v8210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:19 smithi153 ceph-mon[36569]: pgmap v8210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:21 smithi104 ceph-mon[36624]: pgmap v8211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:21 smithi081 ceph-mon[31453]: pgmap v8211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:21 smithi153 ceph-mon[36569]: pgmap v8211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:23 smithi104 ceph-mon[36624]: pgmap v8212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:23.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:23 smithi081 ceph-mon[31453]: pgmap v8212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:23 smithi153 ceph-mon[36569]: pgmap v8212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:25 smithi104 ceph-mon[36624]: pgmap v8213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:25 smithi081 ceph-mon[31453]: pgmap v8213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:25 smithi153 ceph-mon[36569]: pgmap v8213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:27 smithi104 ceph-mon[36624]: pgmap v8214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:27 smithi081 ceph-mon[31453]: pgmap v8214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:27 smithi153 ceph-mon[36569]: pgmap v8214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:29 smithi081 ceph-mon[31453]: pgmap v8215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:29 smithi153 ceph-mon[36569]: pgmap v8215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:29.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:29 smithi104 ceph-mon[36624]: pgmap v8215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:31.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:31 smithi081 ceph-mon[31453]: pgmap v8216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:31 smithi153 ceph-mon[36569]: pgmap v8216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:31 smithi104 ceph-mon[36624]: pgmap v8216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:33 smithi081 ceph-mon[31453]: pgmap v8217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:33 smithi153 ceph-mon[36569]: pgmap v8217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:33.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:33 smithi104 ceph-mon[36624]: pgmap v8217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:34.173+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:24:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:34.173+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:24:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:34.173+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:24:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:35 smithi081 ceph-mon[31453]: pgmap v8218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:35 smithi153 ceph-mon[36569]: pgmap v8218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:35 smithi104 ceph-mon[36624]: pgmap v8218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:37.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:37 smithi081 ceph-mon[31453]: pgmap v8219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:37 smithi153 ceph-mon[36569]: pgmap v8219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:37 smithi104 ceph-mon[36624]: pgmap v8219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:39.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:39 smithi081 ceph-mon[31453]: pgmap v8220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:39 smithi153 ceph-mon[36569]: pgmap v8220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:39 smithi104 ceph-mon[36624]: pgmap v8220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:41 smithi081 ceph-mon[31453]: pgmap v8221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:41 smithi153 ceph-mon[36569]: pgmap v8221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:41.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:41 smithi104 ceph-mon[36624]: pgmap v8221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:43.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:43 smithi081 ceph-mon[31453]: pgmap v8222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:43 smithi153 ceph-mon[36569]: pgmap v8222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:43.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:43 smithi104 ceph-mon[36624]: pgmap v8222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:45.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:45 smithi081 ceph-mon[31453]: pgmap v8223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:45 smithi153 ceph-mon[36569]: pgmap v8223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:45.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:45 smithi104 ceph-mon[36624]: pgmap v8223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:47.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:47 smithi081 ceph-mon[31453]: pgmap v8224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:47 smithi153 ceph-mon[36569]: pgmap v8224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:47.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:47 smithi104 ceph-mon[36624]: pgmap v8224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:49.189+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:24:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:49.189+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:24:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:24:49.189+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:24:49.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:49 smithi081 ceph-mon[31453]: pgmap v8225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:49 smithi153 ceph-mon[36569]: pgmap v8225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:49.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:49 smithi104 ceph-mon[36624]: pgmap v8225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:51.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:51 smithi081 ceph-mon[31453]: pgmap v8226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:24:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:51 smithi153 ceph-mon[36569]: pgmap v8226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:24:51.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:51 smithi104 ceph-mon[36624]: pgmap v8226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:51.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:24:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:24:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:24:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:24:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:24:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:24:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:24:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:24:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:24:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:24:53.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:53 smithi081 ceph-mon[31453]: pgmap v8227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:53 smithi081 ceph-mon[31453]: pgmap v8228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:53 smithi153 ceph-mon[36569]: pgmap v8227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:53 smithi153 ceph-mon[36569]: pgmap v8228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:53.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:53 smithi104 ceph-mon[36624]: pgmap v8227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:53.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:53 smithi104 ceph-mon[36624]: pgmap v8228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:55 smithi153 ceph-mon[36569]: pgmap v8229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:55 smithi104 ceph-mon[36624]: pgmap v8229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:55.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:55 smithi081 ceph-mon[31453]: pgmap v8229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:57 smithi153 ceph-mon[36569]: pgmap v8230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:57.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:57 smithi104 ceph-mon[36624]: pgmap v8230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:57.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:57 smithi081 ceph-mon[31453]: pgmap v8230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:24:59 smithi153 ceph-mon[36569]: pgmap v8231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:59.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:24:59 smithi104 ceph-mon[36624]: pgmap v8231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:24:59.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:24:59 smithi081 ceph-mon[31453]: pgmap v8231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:01 smithi153 ceph-mon[36569]: pgmap v8232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:01.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:01 smithi104 ceph-mon[36624]: pgmap v8232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:01.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:01 smithi081 ceph-mon[31453]: pgmap v8232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:03 smithi153 ceph-mon[36569]: pgmap v8233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:03 smithi104 ceph-mon[36624]: pgmap v8233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:03 smithi081 ceph-mon[31453]: pgmap v8233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:04.205+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:25:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:04.205+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:25:04.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:04.205+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:25:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:05 smithi153 ceph-mon[36569]: pgmap v8234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:05.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:05 smithi104 ceph-mon[36624]: pgmap v8234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:05 smithi081 ceph-mon[31453]: pgmap v8234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:07 smithi153 ceph-mon[36569]: pgmap v8235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:07.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:07 smithi104 ceph-mon[36624]: pgmap v8235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:07.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:07 smithi081 ceph-mon[31453]: pgmap v8235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:09 smithi153 ceph-mon[36569]: pgmap v8236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:09 smithi104 ceph-mon[36624]: pgmap v8236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:09.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:09 smithi081 ceph-mon[31453]: pgmap v8236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:11 smithi153 ceph-mon[36569]: pgmap v8237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:11.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:11 smithi104 ceph-mon[36624]: pgmap v8237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:11.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:11 smithi081 ceph-mon[31453]: pgmap v8237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:13 smithi153 ceph-mon[36569]: pgmap v8238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:13.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:13 smithi104 ceph-mon[36624]: pgmap v8238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:13.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:13 smithi081 ceph-mon[31453]: pgmap v8238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:15.169+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:25:15.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:15.169+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:25:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:15.169+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:25:15.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:15.169+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:25:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:15 smithi153 ceph-mon[36569]: pgmap v8239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:15.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:15 smithi104 ceph-mon[36624]: pgmap v8239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:15.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:15 smithi081 ceph-mon[31453]: pgmap v8239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:17 smithi153 ceph-mon[36569]: pgmap v8240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:17.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:17 smithi104 ceph-mon[36624]: pgmap v8240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:17.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:17 smithi081 ceph-mon[31453]: pgmap v8240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:19.215+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:25:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:19.215+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:25:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:19.215+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:25:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:19 smithi153 ceph-mon[36569]: pgmap v8241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:19.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:19 smithi104 ceph-mon[36624]: pgmap v8241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:19 smithi081 ceph-mon[31453]: pgmap v8241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:21 smithi153 ceph-mon[36569]: pgmap v8242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:21.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:21 smithi104 ceph-mon[36624]: pgmap v8242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:21.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:21 smithi081 ceph-mon[31453]: pgmap v8242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:23 smithi153 ceph-mon[36569]: pgmap v8243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:23.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:23 smithi104 ceph-mon[36624]: pgmap v8243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:23 smithi081 ceph-mon[31453]: pgmap v8243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:25 smithi153 ceph-mon[36569]: pgmap v8244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:25 smithi104 ceph-mon[36624]: pgmap v8244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:25.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:25 smithi081 ceph-mon[31453]: pgmap v8244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:27 smithi153 ceph-mon[36569]: pgmap v8245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:27.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:27 smithi104 ceph-mon[36624]: pgmap v8245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:27.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:27 smithi081 ceph-mon[31453]: pgmap v8245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:29 smithi153 ceph-mon[36569]: pgmap v8246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:29.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:29 smithi104 ceph-mon[36624]: pgmap v8246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:29.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:29 smithi081 ceph-mon[31453]: pgmap v8246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:31 smithi153 ceph-mon[36569]: pgmap v8247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:31.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:31 smithi104 ceph-mon[36624]: pgmap v8247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:31.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:31 smithi081 ceph-mon[31453]: pgmap v8247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:33 smithi153 ceph-mon[36569]: pgmap v8248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:33.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:33 smithi104 ceph-mon[36624]: pgmap v8248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:33.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:33 smithi081 ceph-mon[31453]: pgmap v8248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:34.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:34.226+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:25:34.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:34.226+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:25:34.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:34.226+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:25:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:35 smithi153 ceph-mon[36569]: pgmap v8249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:35.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:35 smithi104 ceph-mon[36624]: pgmap v8249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:35 smithi081 ceph-mon[31453]: pgmap v8249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:37 smithi153 ceph-mon[36569]: pgmap v8250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:37 smithi104 ceph-mon[36624]: pgmap v8250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:37.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:37 smithi081 ceph-mon[31453]: pgmap v8250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:39 smithi153 ceph-mon[36569]: pgmap v8251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:39 smithi104 ceph-mon[36624]: pgmap v8251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:39 smithi081 ceph-mon[31453]: pgmap v8251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:41 smithi153 ceph-mon[36569]: pgmap v8252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:41.904 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:41 smithi104 ceph-mon[36624]: pgmap v8252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:41.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:41 smithi081 ceph-mon[31453]: pgmap v8252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:43 smithi153 ceph-mon[36569]: pgmap v8253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:43.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:43 smithi104 ceph-mon[36624]: pgmap v8253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:43.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:43 smithi081 ceph-mon[31453]: pgmap v8253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:45 smithi153 ceph-mon[36569]: pgmap v8254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:45.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:45 smithi104 ceph-mon[36624]: pgmap v8254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:45.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:45 smithi081 ceph-mon[31453]: pgmap v8254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:47 smithi153 ceph-mon[36569]: pgmap v8255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:47.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:47 smithi104 ceph-mon[36624]: pgmap v8255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:47.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:47 smithi081 ceph-mon[31453]: pgmap v8255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:49.230+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:25:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:49.230+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:25:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:25:49.230+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:25:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:49 smithi104 ceph-mon[36624]: pgmap v8256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:49.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:49 smithi081 ceph-mon[31453]: pgmap v8256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:49 smithi153 ceph-mon[36569]: pgmap v8256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:51.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:51 smithi104 ceph-mon[36624]: pgmap v8257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:51 smithi081 ceph-mon[31453]: pgmap v8257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:51 smithi153 ceph-mon[36569]: pgmap v8257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:25:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:25:52.918 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:25:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:53 smithi104 ceph-mon[36624]: pgmap v8258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:53.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:53 smithi081 ceph-mon[31453]: pgmap v8258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:53.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:53.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:53.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:53.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:53 smithi153 ceph-mon[36569]: pgmap v8258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:55 smithi081 ceph-mon[31453]: pgmap v8259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:55 smithi153 ceph-mon[36569]: pgmap v8259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:55 smithi104 ceph-mon[36624]: pgmap v8259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:57 smithi081 ceph-mon[31453]: pgmap v8260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:57 smithi153 ceph-mon[36569]: pgmap v8260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:57.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:57 smithi104 ceph-mon[36624]: pgmap v8260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:59.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:59 smithi104 ceph-mon[36624]: pgmap v8261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:59.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:25:59.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:25:59.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:25:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:59 smithi153 ceph-mon[36569]: pgmap v8261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:25:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:25:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:25:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:25:59.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:59 smithi081 ceph-mon[31453]: pgmap v8261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:25:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:25:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:25:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:25:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:26:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:00 smithi153 ceph-mon[36569]: pgmap v8262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:00 smithi153 ceph-mon[36569]: pgmap v8263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:00 smithi104 ceph-mon[36624]: pgmap v8262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:00 smithi104 ceph-mon[36624]: pgmap v8263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:00 smithi081 ceph-mon[31453]: pgmap v8262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:00 smithi081 ceph-mon[31453]: pgmap v8263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:02 smithi153 ceph-mon[36569]: pgmap v8264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:02 smithi104 ceph-mon[36624]: pgmap v8264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:02 smithi081 ceph-mon[31453]: pgmap v8264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:04.243+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:26:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:04.243+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:26:04.243 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:04.243+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:26:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:04 smithi153 ceph-mon[36569]: pgmap v8265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:04 smithi104 ceph-mon[36624]: pgmap v8265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:04 smithi081 ceph-mon[31453]: pgmap v8265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:06 smithi081 ceph-mon[31453]: pgmap v8266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:06 smithi153 ceph-mon[36569]: pgmap v8266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:06 smithi104 ceph-mon[36624]: pgmap v8266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:08 smithi153 ceph-mon[36569]: pgmap v8267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:08.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:08 smithi104 ceph-mon[36624]: pgmap v8267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:08 smithi081 ceph-mon[31453]: pgmap v8267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:10 smithi153 ceph-mon[36569]: pgmap v8268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:10.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:10 smithi104 ceph-mon[36624]: pgmap v8268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:10 smithi081 ceph-mon[31453]: pgmap v8268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:12 smithi153 ceph-mon[36569]: pgmap v8269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:12 smithi104 ceph-mon[36624]: pgmap v8269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:12 smithi081 ceph-mon[31453]: pgmap v8269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:14 smithi153 ceph-mon[36569]: pgmap v8270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:14 smithi104 ceph-mon[36624]: pgmap v8270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:14 smithi081 ceph-mon[31453]: pgmap v8270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:16 smithi153 ceph-mon[36569]: pgmap v8271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:16 smithi104 ceph-mon[36624]: pgmap v8271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:16 smithi081 ceph-mon[31453]: pgmap v8271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:18 smithi153 ceph-mon[36569]: pgmap v8272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:18 smithi104 ceph-mon[36624]: pgmap v8272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:18 smithi081 ceph-mon[31453]: pgmap v8272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:19.259+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:26:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:19.259+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:26:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:19.259+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:26:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:20 smithi153 ceph-mon[36569]: pgmap v8273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:20 smithi104 ceph-mon[36624]: pgmap v8273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:20 smithi081 ceph-mon[31453]: pgmap v8273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:22 smithi153 ceph-mon[36569]: pgmap v8274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:22 smithi104 ceph-mon[36624]: pgmap v8274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:22 smithi081 ceph-mon[31453]: pgmap v8274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:24 smithi153 ceph-mon[36569]: pgmap v8275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:24 smithi104 ceph-mon[36624]: pgmap v8275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:24 smithi081 ceph-mon[31453]: pgmap v8275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:26 smithi153 ceph-mon[36569]: pgmap v8276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:26 smithi104 ceph-mon[36624]: pgmap v8276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:26 smithi081 ceph-mon[31453]: pgmap v8276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:28 smithi153 ceph-mon[36569]: pgmap v8277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:28 smithi104 ceph-mon[36624]: pgmap v8277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:28.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:28 smithi081 ceph-mon[31453]: pgmap v8277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:30 smithi104 ceph-mon[36624]: pgmap v8278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:30 smithi081 ceph-mon[31453]: pgmap v8278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:30 smithi153 ceph-mon[36569]: pgmap v8278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:32 smithi104 ceph-mon[36624]: pgmap v8279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:32 smithi081 ceph-mon[31453]: pgmap v8279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:32.910 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:32 smithi153 ceph-mon[36569]: pgmap v8279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:34.266+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:26:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:34.266+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:26:34.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:34.266+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:26:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:34 smithi104 ceph-mon[36624]: pgmap v8280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:34.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:34 smithi081 ceph-mon[31453]: pgmap v8280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:34 smithi153 ceph-mon[36569]: pgmap v8280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:36 smithi104 ceph-mon[36624]: pgmap v8281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:36 smithi081 ceph-mon[31453]: pgmap v8281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:36 smithi153 ceph-mon[36569]: pgmap v8281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:38 smithi104 ceph-mon[36624]: pgmap v8282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:38 smithi081 ceph-mon[31453]: pgmap v8282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:38 smithi153 ceph-mon[36569]: pgmap v8282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:40 smithi081 ceph-mon[31453]: pgmap v8283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:40 smithi153 ceph-mon[36569]: pgmap v8283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:40 smithi104 ceph-mon[36624]: pgmap v8283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:42 smithi081 ceph-mon[31453]: pgmap v8284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:42 smithi153 ceph-mon[36569]: pgmap v8284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:42 smithi104 ceph-mon[36624]: pgmap v8284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:44 smithi081 ceph-mon[31453]: pgmap v8285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:44 smithi153 ceph-mon[36569]: pgmap v8285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:44 smithi104 ceph-mon[36624]: pgmap v8285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:46 smithi081 ceph-mon[31453]: pgmap v8286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:46 smithi153 ceph-mon[36569]: pgmap v8286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:46 smithi104 ceph-mon[36624]: pgmap v8286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:48 smithi081 ceph-mon[31453]: pgmap v8287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:48 smithi153 ceph-mon[36569]: pgmap v8287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:48 smithi104 ceph-mon[36624]: pgmap v8287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:49.280+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:26:49.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:49.280+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:26:49.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:26:49.280+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:26:50.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:50 smithi081 ceph-mon[31453]: pgmap v8288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:50 smithi153 ceph-mon[36569]: pgmap v8288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:50 smithi104 ceph-mon[36624]: pgmap v8288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:52 smithi081 ceph-mon[31453]: pgmap v8289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:52 smithi153 ceph-mon[36569]: pgmap v8289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:52 smithi104 ceph-mon[36624]: pgmap v8289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:54 smithi081 ceph-mon[31453]: pgmap v8290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:54 smithi153 ceph-mon[36569]: pgmap v8290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:54 smithi104 ceph-mon[36624]: pgmap v8290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:56.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:56 smithi081 ceph-mon[31453]: pgmap v8291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:56 smithi153 ceph-mon[36569]: pgmap v8291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:56 smithi104 ceph-mon[36624]: pgmap v8291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:58 smithi081 ceph-mon[31453]: pgmap v8292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:58 smithi153 ceph-mon[36569]: pgmap v8292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:58 smithi104 ceph-mon[36624]: pgmap v8292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:26:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:26:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:26:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:26:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:27:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:27:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:27:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:26:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:27:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:27:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:27:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:26:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:27:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:00 smithi081 ceph-mon[31453]: pgmap v8293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:27:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:27:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:27:00.983 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:00 smithi104 ceph-mon[36624]: pgmap v8293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:00.983 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:27:00.983 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:27:00.983 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:27:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:00 smithi153 ceph-mon[36569]: pgmap v8293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:27:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:27:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:27:01.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:01 smithi081 ceph-mon[31453]: pgmap v8294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:01 smithi081 ceph-mon[31453]: pgmap v8295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:02.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:01 smithi153 ceph-mon[36569]: pgmap v8294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:02.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:01 smithi153 ceph-mon[36569]: pgmap v8295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:01 smithi104 ceph-mon[36624]: pgmap v8294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:01 smithi104 ceph-mon[36624]: pgmap v8295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:03 smithi153 ceph-mon[36569]: pgmap v8296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:04.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:03 smithi104 ceph-mon[36624]: pgmap v8296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:04.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:03 smithi081 ceph-mon[31453]: pgmap v8296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:04.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:04.295+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:27:04.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:04.295+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:27:04.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:04.295+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:27:05.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:05 smithi081 ceph-mon[31453]: pgmap v8297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:05 smithi153 ceph-mon[36569]: pgmap v8297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:06.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:05 smithi104 ceph-mon[36624]: pgmap v8297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:06 smithi081 ceph-mon[31453]: pgmap v8298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:06 smithi153 ceph-mon[36569]: pgmap v8298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:06 smithi104 ceph-mon[36624]: pgmap v8298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:09 smithi153 ceph-mon[36569]: pgmap v8299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:09 smithi104 ceph-mon[36624]: pgmap v8299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:09 smithi081 ceph-mon[31453]: pgmap v8299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:11 smithi153 ceph-mon[36569]: pgmap v8300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:11 smithi104 ceph-mon[36624]: pgmap v8300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:11 smithi081 ceph-mon[31453]: pgmap v8300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:13 smithi153 ceph-mon[36569]: pgmap v8301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:13.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:13 smithi104 ceph-mon[36624]: pgmap v8301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:13 smithi081 ceph-mon[31453]: pgmap v8301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:15 smithi153 ceph-mon[36569]: pgmap v8302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:15 smithi104 ceph-mon[36624]: pgmap v8302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:15 smithi081 ceph-mon[31453]: pgmap v8302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:17 smithi153 ceph-mon[36569]: pgmap v8303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:17.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:17 smithi104 ceph-mon[36624]: pgmap v8303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:17.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:17 smithi081 ceph-mon[31453]: pgmap v8303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:19.311+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:27:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:19.311+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:27:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:19.311+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:27:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:19 smithi153 ceph-mon[36569]: pgmap v8304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:19.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:19 smithi104 ceph-mon[36624]: pgmap v8304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:19 smithi081 ceph-mon[31453]: pgmap v8304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:21 smithi153 ceph-mon[36569]: pgmap v8305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:21.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:21 smithi104 ceph-mon[36624]: pgmap v8305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:21 smithi081 ceph-mon[31453]: pgmap v8305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:23 smithi153 ceph-mon[36569]: pgmap v8306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:23.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:23 smithi104 ceph-mon[36624]: pgmap v8306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:23 smithi081 ceph-mon[31453]: pgmap v8306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:25 smithi153 ceph-mon[36569]: pgmap v8307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:25 smithi104 ceph-mon[36624]: pgmap v8307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:25 smithi081 ceph-mon[31453]: pgmap v8307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:27 smithi153 ceph-mon[36569]: pgmap v8308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:27 smithi104 ceph-mon[36624]: pgmap v8308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:27.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:27 smithi081 ceph-mon[31453]: pgmap v8308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:29 smithi153 ceph-mon[36569]: pgmap v8309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:29.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:29 smithi104 ceph-mon[36624]: pgmap v8309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:29 smithi081 ceph-mon[31453]: pgmap v8309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:31 smithi153 ceph-mon[36569]: pgmap v8310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:31 smithi104 ceph-mon[36624]: pgmap v8310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:31 smithi081 ceph-mon[31453]: pgmap v8310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:33 smithi153 ceph-mon[36569]: pgmap v8311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:33 smithi104 ceph-mon[36624]: pgmap v8311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:33 smithi081 ceph-mon[31453]: pgmap v8311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:34.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:34.326+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:27:34.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:34.326+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:27:34.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:34.326+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:27:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:35 smithi153 ceph-mon[36569]: pgmap v8312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:35.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:35 smithi104 ceph-mon[36624]: pgmap v8312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:35 smithi081 ceph-mon[31453]: pgmap v8312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:37 smithi153 ceph-mon[36569]: pgmap v8313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:37 smithi104 ceph-mon[36624]: pgmap v8313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:37 smithi081 ceph-mon[31453]: pgmap v8313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:39 smithi153 ceph-mon[36569]: pgmap v8314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:39.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:39 smithi104 ceph-mon[36624]: pgmap v8314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:39 smithi081 ceph-mon[31453]: pgmap v8314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:41 smithi153 ceph-mon[36569]: pgmap v8315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:41 smithi104 ceph-mon[36624]: pgmap v8315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:41 smithi081 ceph-mon[31453]: pgmap v8315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:43 smithi153 ceph-mon[36569]: pgmap v8316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:43 smithi104 ceph-mon[36624]: pgmap v8316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:43 smithi081 ceph-mon[31453]: pgmap v8316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:45 smithi153 ceph-mon[36569]: pgmap v8317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:45 smithi104 ceph-mon[36624]: pgmap v8317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:45 smithi081 ceph-mon[31453]: pgmap v8317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:47 smithi153 ceph-mon[36569]: pgmap v8318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:47.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:47 smithi104 ceph-mon[36624]: pgmap v8318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:47 smithi081 ceph-mon[31453]: pgmap v8318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:49.341+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:27:49.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:49.341+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:27:49.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:27:49.341+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:27:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:49 smithi153 ceph-mon[36569]: pgmap v8319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:49.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:49 smithi104 ceph-mon[36624]: pgmap v8319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:49 smithi081 ceph-mon[31453]: pgmap v8319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:51 smithi153 ceph-mon[36569]: pgmap v8320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:51 smithi104 ceph-mon[36624]: pgmap v8320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:51 smithi081 ceph-mon[31453]: pgmap v8320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:53 smithi104 ceph-mon[36624]: pgmap v8321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:53 smithi081 ceph-mon[31453]: pgmap v8321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:53 smithi153 ceph-mon[36569]: pgmap v8321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:55 smithi104 ceph-mon[36624]: pgmap v8322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:55 smithi081 ceph-mon[31453]: pgmap v8322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:55 smithi153 ceph-mon[36569]: pgmap v8322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:57 smithi104 ceph-mon[36624]: pgmap v8323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:57 smithi081 ceph-mon[31453]: pgmap v8323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:57 smithi153 ceph-mon[36569]: pgmap v8323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:27:59 smithi104 ceph-mon[36624]: pgmap v8324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:27:59.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:27:59 smithi081 ceph-mon[31453]: pgmap v8324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:27:59 smithi153 ceph-mon[36569]: pgmap v8324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:01 smithi104 ceph-mon[36624]: pgmap v8325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:28:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:01 smithi081 ceph-mon[31453]: pgmap v8325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:28:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:01 smithi153 ceph-mon[36569]: pgmap v8325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:28:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:28:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:28:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:28:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:28:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:28:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:28:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:28:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:28:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:28:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:03 smithi081 ceph-mon[31453]: pgmap v8326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:03 smithi081 ceph-mon[31453]: pgmap v8327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:03 smithi153 ceph-mon[36569]: pgmap v8326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:03 smithi153 ceph-mon[36569]: pgmap v8327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:03 smithi104 ceph-mon[36624]: pgmap v8326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:03 smithi104 ceph-mon[36624]: pgmap v8327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:04.352+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:28:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:04.352+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:28:04.352 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:04.352+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:28:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:05 smithi081 ceph-mon[31453]: pgmap v8328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:05 smithi153 ceph-mon[36569]: pgmap v8328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:05 smithi104 ceph-mon[36624]: pgmap v8328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:07 smithi081 ceph-mon[31453]: pgmap v8329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:07 smithi153 ceph-mon[36569]: pgmap v8329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:07 smithi104 ceph-mon[36624]: pgmap v8329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:09 smithi081 ceph-mon[31453]: pgmap v8330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:09 smithi153 ceph-mon[36569]: pgmap v8330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:09 smithi104 ceph-mon[36624]: pgmap v8330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:11 smithi081 ceph-mon[31453]: pgmap v8331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:11 smithi153 ceph-mon[36569]: pgmap v8331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:11 smithi104 ceph-mon[36624]: pgmap v8331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:13 smithi081 ceph-mon[31453]: pgmap v8332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:13 smithi153 ceph-mon[36569]: pgmap v8332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:13 smithi104 ceph-mon[36624]: pgmap v8332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:15.200+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:28:15.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:15.201+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:28:15.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:15.203+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:28:15.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:15.203+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:28:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:15 smithi081 ceph-mon[31453]: pgmap v8333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:15 smithi153 ceph-mon[36569]: pgmap v8333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:15 smithi104 ceph-mon[36624]: pgmap v8333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:17 smithi081 ceph-mon[31453]: pgmap v8334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:17 smithi153 ceph-mon[36569]: pgmap v8334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:17 smithi104 ceph-mon[36624]: pgmap v8334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:19.364+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:28:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:19.364+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:28:19.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:19.364+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:28:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:19 smithi081 ceph-mon[31453]: pgmap v8335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:19 smithi153 ceph-mon[36569]: pgmap v8335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:19 smithi104 ceph-mon[36624]: pgmap v8335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:21 smithi081 ceph-mon[31453]: pgmap v8336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:21 smithi153 ceph-mon[36569]: pgmap v8336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:21 smithi104 ceph-mon[36624]: pgmap v8336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:23 smithi081 ceph-mon[31453]: pgmap v8337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:23 smithi153 ceph-mon[36569]: pgmap v8337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:23 smithi104 ceph-mon[36624]: pgmap v8337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:25 smithi081 ceph-mon[31453]: pgmap v8338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:25 smithi153 ceph-mon[36569]: pgmap v8338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:25 smithi104 ceph-mon[36624]: pgmap v8338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:27 smithi081 ceph-mon[31453]: pgmap v8339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:27 smithi153 ceph-mon[36569]: pgmap v8339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:27 smithi104 ceph-mon[36624]: pgmap v8339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:29 smithi153 ceph-mon[36569]: pgmap v8340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:29 smithi104 ceph-mon[36624]: pgmap v8340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:29 smithi081 ceph-mon[31453]: pgmap v8340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:31 smithi153 ceph-mon[36569]: pgmap v8341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:31 smithi104 ceph-mon[36624]: pgmap v8341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:31 smithi081 ceph-mon[31453]: pgmap v8341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:33 smithi153 ceph-mon[36569]: pgmap v8342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:33 smithi104 ceph-mon[36624]: pgmap v8342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:33 smithi081 ceph-mon[31453]: pgmap v8342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:34.380+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:28:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:34.380+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:28:34.380 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:34.380+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:28:35.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:35 smithi081 ceph-mon[31453]: pgmap v8343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:35 smithi153 ceph-mon[36569]: pgmap v8343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:35 smithi104 ceph-mon[36624]: pgmap v8343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:37 smithi153 ceph-mon[36569]: pgmap v8344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:37 smithi104 ceph-mon[36624]: pgmap v8344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:37 smithi081 ceph-mon[31453]: pgmap v8344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:38 smithi153 ceph-mon[36569]: pgmap v8345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:38 smithi104 ceph-mon[36624]: pgmap v8345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:38 smithi081 ceph-mon[31453]: pgmap v8345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:41 smithi153 ceph-mon[36569]: pgmap v8346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:41 smithi104 ceph-mon[36624]: pgmap v8346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:41 smithi081 ceph-mon[31453]: pgmap v8346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:43 smithi153 ceph-mon[36569]: pgmap v8347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:43 smithi104 ceph-mon[36624]: pgmap v8347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:43 smithi081 ceph-mon[31453]: pgmap v8347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:45 smithi153 ceph-mon[36569]: pgmap v8348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:45 smithi104 ceph-mon[36624]: pgmap v8348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:45 smithi081 ceph-mon[31453]: pgmap v8348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:47 smithi153 ceph-mon[36569]: pgmap v8349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:47 smithi104 ceph-mon[36624]: pgmap v8349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:47 smithi081 ceph-mon[31453]: pgmap v8349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:49 smithi153 ceph-mon[36569]: pgmap v8350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:49 smithi104 ceph-mon[36624]: pgmap v8350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:49.381+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:49.381+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:28:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:28:49.381+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:28:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:49 smithi081 ceph-mon[31453]: pgmap v8350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:51 smithi104 ceph-mon[36624]: pgmap v8351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:51 smithi081 ceph-mon[31453]: pgmap v8351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:51 smithi153 ceph-mon[36569]: pgmap v8351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:53 smithi104 ceph-mon[36624]: pgmap v8352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:53 smithi081 ceph-mon[31453]: pgmap v8352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:53 smithi153 ceph-mon[36569]: pgmap v8352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:55 smithi104 ceph-mon[36624]: pgmap v8353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:55 smithi081 ceph-mon[31453]: pgmap v8353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:55 smithi153 ceph-mon[36569]: pgmap v8353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:57 smithi104 ceph-mon[36624]: pgmap v8354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:57 smithi081 ceph-mon[31453]: pgmap v8354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:57 smithi153 ceph-mon[36569]: pgmap v8354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:28:59 smithi081 ceph-mon[31453]: pgmap v8355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:28:59 smithi153 ceph-mon[36569]: pgmap v8355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:28:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:28:59 smithi104 ceph-mon[36624]: pgmap v8355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:01.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:01 smithi104 ceph-mon[36624]: pgmap v8356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:01 smithi081 ceph-mon[31453]: pgmap v8356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:01 smithi153 ceph-mon[36569]: pgmap v8356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:03 smithi081 ceph-mon[31453]: pgmap v8357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:29:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:03 smithi153 ceph-mon[36569]: pgmap v8357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:29:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:03 smithi104 ceph-mon[36624]: pgmap v8357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:29:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:04.397+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:29:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:04.397+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:29:04.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:04.397+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:29:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:29:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:29:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:04 smithi081 ceph-mon[31453]: pgmap v8358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:04 smithi081 ceph-mon[31453]: pgmap v8359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:29:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:29:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:29:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:04 smithi153 ceph-mon[36569]: pgmap v8358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:04 smithi153 ceph-mon[36569]: pgmap v8359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:29:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:29:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:29:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:04 smithi104 ceph-mon[36624]: pgmap v8358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:04 smithi104 ceph-mon[36624]: pgmap v8359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:29:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:06 smithi153 ceph-mon[36569]: pgmap v8360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:06 smithi104 ceph-mon[36624]: pgmap v8360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:06 smithi081 ceph-mon[31453]: pgmap v8360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:08 smithi153 ceph-mon[36569]: pgmap v8361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:08 smithi104 ceph-mon[36624]: pgmap v8361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:08 smithi081 ceph-mon[31453]: pgmap v8361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:10 smithi153 ceph-mon[36569]: pgmap v8362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:10 smithi104 ceph-mon[36624]: pgmap v8362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:10 smithi081 ceph-mon[31453]: pgmap v8362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:12 smithi153 ceph-mon[36569]: pgmap v8363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:12 smithi104 ceph-mon[36624]: pgmap v8363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:12 smithi081 ceph-mon[31453]: pgmap v8363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:14 smithi153 ceph-mon[36569]: pgmap v8364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:14 smithi104 ceph-mon[36624]: pgmap v8364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:14 smithi081 ceph-mon[31453]: pgmap v8364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:16 smithi153 ceph-mon[36569]: pgmap v8365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:16 smithi104 ceph-mon[36624]: pgmap v8365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:16 smithi081 ceph-mon[31453]: pgmap v8365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:18 smithi104 ceph-mon[36624]: pgmap v8366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:18 smithi081 ceph-mon[31453]: pgmap v8366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:18 smithi153 ceph-mon[36569]: pgmap v8366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:19.412+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:29:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:19.412+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:29:19.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:19.412+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:29:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:20 smithi104 ceph-mon[36624]: pgmap v8367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:20 smithi081 ceph-mon[31453]: pgmap v8367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:20 smithi153 ceph-mon[36569]: pgmap v8367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:22 smithi104 ceph-mon[36624]: pgmap v8368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:22.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:22 smithi153 ceph-mon[36569]: pgmap v8368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:22 smithi081 ceph-mon[31453]: pgmap v8368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:24 smithi104 ceph-mon[36624]: pgmap v8369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:24.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:24 smithi081 ceph-mon[31453]: pgmap v8369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:24 smithi153 ceph-mon[36569]: pgmap v8369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:26 smithi104 ceph-mon[36624]: pgmap v8370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:26 smithi081 ceph-mon[31453]: pgmap v8370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:26 smithi153 ceph-mon[36569]: pgmap v8370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:28 smithi081 ceph-mon[31453]: pgmap v8371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:28 smithi153 ceph-mon[36569]: pgmap v8371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:28 smithi104 ceph-mon[36624]: pgmap v8371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:30 smithi081 ceph-mon[31453]: pgmap v8372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:30 smithi153 ceph-mon[36569]: pgmap v8372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:30 smithi104 ceph-mon[36624]: pgmap v8372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:32.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:32 smithi104 ceph-mon[36624]: pgmap v8373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:32 smithi081 ceph-mon[31453]: pgmap v8373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:32 smithi153 ceph-mon[36569]: pgmap v8373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:34.421+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:29:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:34.421+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:29:34.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:34.421+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:29:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:34 smithi081 ceph-mon[31453]: pgmap v8374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:34 smithi153 ceph-mon[36569]: pgmap v8374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:34 smithi104 ceph-mon[36624]: pgmap v8374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:36 smithi081 ceph-mon[31453]: pgmap v8375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:36 smithi153 ceph-mon[36569]: pgmap v8375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:36 smithi104 ceph-mon[36624]: pgmap v8375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:38 smithi081 ceph-mon[31453]: pgmap v8376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:38 smithi153 ceph-mon[36569]: pgmap v8376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:38 smithi104 ceph-mon[36624]: pgmap v8376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:40 smithi081 ceph-mon[31453]: pgmap v8377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:40 smithi153 ceph-mon[36569]: pgmap v8377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:40 smithi104 ceph-mon[36624]: pgmap v8377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:42 smithi081 ceph-mon[31453]: pgmap v8378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:42 smithi153 ceph-mon[36569]: pgmap v8378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:42 smithi104 ceph-mon[36624]: pgmap v8378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:44 smithi081 ceph-mon[31453]: pgmap v8379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:44 smithi153 ceph-mon[36569]: pgmap v8379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:44 smithi104 ceph-mon[36624]: pgmap v8379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:46 smithi081 ceph-mon[31453]: pgmap v8380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:46 smithi153 ceph-mon[36569]: pgmap v8380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:46 smithi104 ceph-mon[36624]: pgmap v8380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:48 smithi081 ceph-mon[31453]: pgmap v8381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:48 smithi153 ceph-mon[36569]: pgmap v8381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:48 smithi104 ceph-mon[36624]: pgmap v8381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:49.435+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:29:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:49.435+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:29:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:29:49.435+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:29:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:50 smithi081 ceph-mon[31453]: pgmap v8382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:50 smithi153 ceph-mon[36569]: pgmap v8382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:50 smithi104 ceph-mon[36624]: pgmap v8382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:52 smithi081 ceph-mon[31453]: pgmap v8383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:52 smithi153 ceph-mon[36569]: pgmap v8383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:52 smithi104 ceph-mon[36624]: pgmap v8383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:53 smithi081 ceph-mon[31453]: pgmap v8384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:53 smithi153 ceph-mon[36569]: pgmap v8384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:53 smithi104 ceph-mon[36624]: pgmap v8384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:56 smithi153 ceph-mon[36569]: pgmap v8385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:56 smithi104 ceph-mon[36624]: pgmap v8385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:56 smithi081 ceph-mon[31453]: pgmap v8385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:29:58 smithi104 ceph-mon[36624]: pgmap v8386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:29:58 smithi081 ceph-mon[31453]: pgmap v8386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:29:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:29:58 smithi153 ceph-mon[36569]: pgmap v8386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:00 smithi104 ceph-mon[36624]: pgmap v8387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:30:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:00 smithi081 ceph-mon[31453]: pgmap v8387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:30:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:00 smithi153 ceph-mon[36569]: pgmap v8387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:30:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:02 smithi104 ceph-mon[36624]: pgmap v8388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:02 smithi081 ceph-mon[31453]: pgmap v8388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:02 smithi153 ceph-mon[36569]: pgmap v8388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:04.450+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:30:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:04.450+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:30:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:04.450+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:30:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:04 smithi104 ceph-mon[36624]: pgmap v8389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:30:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:04 smithi081 ceph-mon[31453]: pgmap v8389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:30:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:04 smithi153 ceph-mon[36569]: pgmap v8389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:30:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:30:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:30:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:30:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:30:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:30:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:30:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:30:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:30:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:30:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:06 smithi081 ceph-mon[31453]: pgmap v8390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:06 smithi081 ceph-mon[31453]: pgmap v8391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:06 smithi153 ceph-mon[36569]: pgmap v8390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:06 smithi153 ceph-mon[36569]: pgmap v8391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:06 smithi104 ceph-mon[36624]: pgmap v8390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:06 smithi104 ceph-mon[36624]: pgmap v8391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:08 smithi081 ceph-mon[31453]: pgmap v8392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:08 smithi153 ceph-mon[36569]: pgmap v8392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:08 smithi104 ceph-mon[36624]: pgmap v8392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:10 smithi081 ceph-mon[31453]: pgmap v8393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:10 smithi153 ceph-mon[36569]: pgmap v8393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:10 smithi104 ceph-mon[36624]: pgmap v8393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:12 smithi081 ceph-mon[31453]: pgmap v8394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:12 smithi153 ceph-mon[36569]: pgmap v8394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:12 smithi104 ceph-mon[36624]: pgmap v8394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:14 smithi081 ceph-mon[31453]: pgmap v8395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:14 smithi153 ceph-mon[36569]: pgmap v8395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:14 smithi104 ceph-mon[36624]: pgmap v8395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:16 smithi081 ceph-mon[31453]: pgmap v8396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:16 smithi153 ceph-mon[36569]: pgmap v8396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:16 smithi104 ceph-mon[36624]: pgmap v8396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:18 smithi081 ceph-mon[31453]: pgmap v8397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:18 smithi153 ceph-mon[36569]: pgmap v8397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:18 smithi104 ceph-mon[36624]: pgmap v8397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:19.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:19.461+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:30:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:19.461+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:30:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:19.461+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:30:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:20 smithi081 ceph-mon[31453]: pgmap v8398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:20 smithi153 ceph-mon[36569]: pgmap v8398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:20 smithi104 ceph-mon[36624]: pgmap v8398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:22 smithi081 ceph-mon[31453]: pgmap v8399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:22 smithi153 ceph-mon[36569]: pgmap v8399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:22 smithi104 ceph-mon[36624]: pgmap v8399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:24 smithi081 ceph-mon[31453]: pgmap v8400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:24 smithi153 ceph-mon[36569]: pgmap v8400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:24 smithi104 ceph-mon[36624]: pgmap v8400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:26 smithi081 ceph-mon[31453]: pgmap v8401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:26 smithi153 ceph-mon[36569]: pgmap v8401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:26 smithi104 ceph-mon[36624]: pgmap v8401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:28 smithi081 ceph-mon[31453]: pgmap v8402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:28 smithi153 ceph-mon[36569]: pgmap v8402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:28 smithi104 ceph-mon[36624]: pgmap v8402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:30 smithi081 ceph-mon[31453]: pgmap v8403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:30 smithi153 ceph-mon[36569]: pgmap v8403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:30 smithi104 ceph-mon[36624]: pgmap v8403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:32 smithi081 ceph-mon[31453]: pgmap v8404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:32 smithi153 ceph-mon[36569]: pgmap v8404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:32 smithi104 ceph-mon[36624]: pgmap v8404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:34.473+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:30:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:34.473+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:30:34.473 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:34.473+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:30:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:34 smithi153 ceph-mon[36569]: pgmap v8405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:34 smithi104 ceph-mon[36624]: pgmap v8405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:34 smithi081 ceph-mon[31453]: pgmap v8405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:36.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:36 smithi081 ceph-mon[31453]: pgmap v8406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:36 smithi153 ceph-mon[36569]: pgmap v8406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:36 smithi104 ceph-mon[36624]: pgmap v8406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:38 smithi153 ceph-mon[36569]: pgmap v8407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:38 smithi104 ceph-mon[36624]: pgmap v8407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:38 smithi081 ceph-mon[31453]: pgmap v8407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:40 smithi153 ceph-mon[36569]: pgmap v8408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:40 smithi104 ceph-mon[36624]: pgmap v8408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:40 smithi081 ceph-mon[31453]: pgmap v8408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:42 smithi153 ceph-mon[36569]: pgmap v8409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:42 smithi104 ceph-mon[36624]: pgmap v8409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:42 smithi081 ceph-mon[31453]: pgmap v8409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:44 smithi153 ceph-mon[36569]: pgmap v8410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:44 smithi104 ceph-mon[36624]: pgmap v8410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:44 smithi081 ceph-mon[31453]: pgmap v8410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:46 smithi153 ceph-mon[36569]: pgmap v8411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:46 smithi104 ceph-mon[36624]: pgmap v8411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:46 smithi081 ceph-mon[31453]: pgmap v8411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:48 smithi153 ceph-mon[36569]: pgmap v8412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:48 smithi104 ceph-mon[36624]: pgmap v8412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:48 smithi081 ceph-mon[31453]: pgmap v8412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:49.488+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:30:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:49.489+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:30:49.489 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:30:49.489+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:30:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:50 smithi153 ceph-mon[36569]: pgmap v8413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:50 smithi104 ceph-mon[36624]: pgmap v8413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:50 smithi081 ceph-mon[31453]: pgmap v8413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:52 smithi153 ceph-mon[36569]: pgmap v8414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:52 smithi104 ceph-mon[36624]: pgmap v8414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:52 smithi081 ceph-mon[31453]: pgmap v8414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:53 smithi153 ceph-mon[36569]: pgmap v8415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:53 smithi104 ceph-mon[36624]: pgmap v8415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:53 smithi081 ceph-mon[31453]: pgmap v8415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:56 smithi081 ceph-mon[31453]: pgmap v8416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:56 smithi153 ceph-mon[36569]: pgmap v8416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:56 smithi104 ceph-mon[36624]: pgmap v8416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:30:58 smithi081 ceph-mon[31453]: pgmap v8417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:30:58 smithi153 ceph-mon[36569]: pgmap v8417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:30:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:30:58 smithi104 ceph-mon[36624]: pgmap v8417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:00 smithi081 ceph-mon[31453]: pgmap v8418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:00 smithi153 ceph-mon[36569]: pgmap v8418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:00 smithi104 ceph-mon[36624]: pgmap v8418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:02.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:02 smithi104 ceph-mon[36624]: pgmap v8419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:02 smithi081 ceph-mon[31453]: pgmap v8419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:02 smithi153 ceph-mon[36569]: pgmap v8419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:04 smithi081 ceph-mon[31453]: pgmap v8420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:04.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:04.498+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:31:04.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:04.498+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:31:04.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:04.498+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:31:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:04 smithi153 ceph-mon[36569]: pgmap v8420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:04 smithi104 ceph-mon[36624]: pgmap v8420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:06 smithi081 ceph-mon[31453]: pgmap v8421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:31:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:06 smithi153 ceph-mon[36569]: pgmap v8421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:31:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:06 smithi104 ceph-mon[36624]: pgmap v8421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:31:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:31:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:31:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:07 smithi081 ceph-mon[31453]: pgmap v8422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:07 smithi081 ceph-mon[31453]: pgmap v8423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:31:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:31:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:31:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:07 smithi153 ceph-mon[36569]: pgmap v8422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:07 smithi153 ceph-mon[36569]: pgmap v8423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:31:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:31:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:31:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:07 smithi104 ceph-mon[36624]: pgmap v8422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:07 smithi104 ceph-mon[36624]: pgmap v8423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:31:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:09 smithi081 ceph-mon[31453]: pgmap v8424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:09 smithi153 ceph-mon[36569]: pgmap v8424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:09 smithi104 ceph-mon[36624]: pgmap v8424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:11 smithi081 ceph-mon[31453]: pgmap v8425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:11 smithi153 ceph-mon[36569]: pgmap v8425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:11 smithi104 ceph-mon[36624]: pgmap v8425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:12.130+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5d00 msgr2=0x7f16e80a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000772) for more than 900000000 us, fault. 2024-04-13T01:31:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:12.130+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5d00 0x7f16e80a4ea0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f40ba000 comp rx=0 tx=0).stop 2024-04-13T01:31:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:12.130+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5d00 msgr2=0x7f16e80a4ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T01:31:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:12.130+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5d00 0x7f16e80a4ea0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T01:31:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:12.130+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 0x7f16e80b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T01:31:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:12.131+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 0x7f16e80b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T01:31:12.133 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:12.132+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 0x7f16e80b3a70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f4039000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T01:31:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:13 smithi081 ceph-mon[31453]: pgmap v8426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:13 smithi153 ceph-mon[36569]: pgmap v8426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:13 smithi104 ceph-mon[36624]: pgmap v8426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:15.230+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:31:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:15.230+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:31:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:15.230+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:31:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:15.230+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:31:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:15 smithi081 ceph-mon[31453]: pgmap v8427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:15 smithi153 ceph-mon[36569]: pgmap v8427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:15 smithi104 ceph-mon[36624]: pgmap v8427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:17 smithi081 ceph-mon[31453]: pgmap v8428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:17 smithi153 ceph-mon[36569]: pgmap v8428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:17 smithi104 ceph-mon[36624]: pgmap v8428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:19.506+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:31:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:19.506+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:31:19.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:19.506+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:31:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:19 smithi081 ceph-mon[31453]: pgmap v8429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:19 smithi153 ceph-mon[36569]: pgmap v8429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:19 smithi104 ceph-mon[36624]: pgmap v8429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:21 smithi081 ceph-mon[31453]: pgmap v8430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:21 smithi153 ceph-mon[36569]: pgmap v8430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:21 smithi104 ceph-mon[36624]: pgmap v8430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:23 smithi081 ceph-mon[31453]: pgmap v8431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:23 smithi153 ceph-mon[36569]: pgmap v8431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:23 smithi104 ceph-mon[36624]: pgmap v8431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:25 smithi081 ceph-mon[31453]: pgmap v8432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:25 smithi153 ceph-mon[36569]: pgmap v8432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:25 smithi104 ceph-mon[36624]: pgmap v8432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:27 smithi081 ceph-mon[31453]: pgmap v8433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:27 smithi153 ceph-mon[36569]: pgmap v8433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:27 smithi104 ceph-mon[36624]: pgmap v8433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:29 smithi153 ceph-mon[36569]: pgmap v8434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:29 smithi104 ceph-mon[36624]: pgmap v8434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:29 smithi081 ceph-mon[31453]: pgmap v8434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:31 smithi153 ceph-mon[36569]: pgmap v8435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:31 smithi104 ceph-mon[36624]: pgmap v8435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:31 smithi081 ceph-mon[31453]: pgmap v8435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:33 smithi153 ceph-mon[36569]: pgmap v8436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:33 smithi104 ceph-mon[36624]: pgmap v8436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:33 smithi081 ceph-mon[31453]: pgmap v8436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:34.522+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:31:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:34.522+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:31:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:34.522+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:31:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:35 smithi153 ceph-mon[36569]: pgmap v8437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:35 smithi104 ceph-mon[36624]: pgmap v8437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:35 smithi081 ceph-mon[31453]: pgmap v8437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:37.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:37 smithi081 ceph-mon[31453]: pgmap v8438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:37 smithi153 ceph-mon[36569]: pgmap v8438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:37 smithi104 ceph-mon[36624]: pgmap v8438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:39 smithi153 ceph-mon[36569]: pgmap v8439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:39 smithi104 ceph-mon[36624]: pgmap v8439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:39 smithi081 ceph-mon[31453]: pgmap v8439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:41 smithi153 ceph-mon[36569]: pgmap v8440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:41 smithi104 ceph-mon[36624]: pgmap v8440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:41 smithi081 ceph-mon[31453]: pgmap v8440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:43 smithi153 ceph-mon[36569]: pgmap v8441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:43 smithi104 ceph-mon[36624]: pgmap v8441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:43 smithi081 ceph-mon[31453]: pgmap v8441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:45 smithi153 ceph-mon[36569]: pgmap v8442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:45 smithi104 ceph-mon[36624]: pgmap v8442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:45 smithi081 ceph-mon[31453]: pgmap v8442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:47 smithi153 ceph-mon[36569]: pgmap v8443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:47 smithi104 ceph-mon[36624]: pgmap v8443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:47 smithi081 ceph-mon[31453]: pgmap v8443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:49.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:49.538+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:31:49.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:49.538+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:31:49.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:31:49.538+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:31:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:49 smithi153 ceph-mon[36569]: pgmap v8444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:49 smithi104 ceph-mon[36624]: pgmap v8444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:49 smithi081 ceph-mon[31453]: pgmap v8444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:51 smithi153 ceph-mon[36569]: pgmap v8445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:51 smithi104 ceph-mon[36624]: pgmap v8445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:51 smithi081 ceph-mon[31453]: pgmap v8445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:53 smithi153 ceph-mon[36569]: pgmap v8446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:53 smithi104 ceph-mon[36624]: pgmap v8446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:53 smithi081 ceph-mon[31453]: pgmap v8446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:54 smithi153 ceph-mon[36569]: pgmap v8447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:54 smithi104 ceph-mon[36624]: pgmap v8447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:54 smithi081 ceph-mon[31453]: pgmap v8447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:57 smithi081 ceph-mon[31453]: pgmap v8448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:57 smithi153 ceph-mon[36569]: pgmap v8448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:57 smithi104 ceph-mon[36624]: pgmap v8448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:31:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:31:59 smithi081 ceph-mon[31453]: pgmap v8449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:31:59 smithi153 ceph-mon[36569]: pgmap v8449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:31:59 smithi104 ceph-mon[36624]: pgmap v8449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:00.238+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:32:00.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:00.238+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:32:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:00.238+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:32:00.239 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:00.238+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:32:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:01 smithi081 ceph-mon[31453]: pgmap v8450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:01 smithi153 ceph-mon[36569]: pgmap v8450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:01 smithi104 ceph-mon[36624]: pgmap v8450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:03 smithi081 ceph-mon[31453]: pgmap v8451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:03 smithi153 ceph-mon[36569]: pgmap v8451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:03 smithi104 ceph-mon[36624]: pgmap v8451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:04.542+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:32:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:04.542+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:32:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:04.542+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:32:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:05 smithi081 ceph-mon[31453]: pgmap v8452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:05 smithi153 ceph-mon[36569]: pgmap v8452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:05 smithi104 ceph-mon[36624]: pgmap v8452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: pgmap v8453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: pgmap v8453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:32:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: pgmap v8453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:32:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:32:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:32:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:32:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:32:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:32:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:32:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:32:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:32:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:32:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:32:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:32:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:32:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:32:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:32:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:32:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:09 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:32:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:09 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:32:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:09 smithi153 ceph-mon[36569]: pgmap v8454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:09 smithi153 ceph-mon[36569]: pgmap v8455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:09 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:32:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:09 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:32:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:09 smithi104 ceph-mon[36624]: pgmap v8454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:09 smithi104 ceph-mon[36624]: pgmap v8455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:09 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:32:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:09 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:32:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:09 smithi081 ceph-mon[31453]: pgmap v8454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:09 smithi081 ceph-mon[31453]: pgmap v8455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:11 smithi153 ceph-mon[36569]: pgmap v8456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:11 smithi104 ceph-mon[36624]: pgmap v8456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:11 smithi081 ceph-mon[31453]: pgmap v8456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:13 smithi153 ceph-mon[36569]: pgmap v8457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:13 smithi104 ceph-mon[36624]: pgmap v8457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:13 smithi081 ceph-mon[31453]: pgmap v8457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:15 smithi153 ceph-mon[36569]: pgmap v8458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:15 smithi104 ceph-mon[36624]: pgmap v8458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:15 smithi081 ceph-mon[31453]: pgmap v8458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:17 smithi153 ceph-mon[36569]: pgmap v8459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:17 smithi104 ceph-mon[36624]: pgmap v8459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:17 smithi081 ceph-mon[31453]: pgmap v8459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:19.559+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:32:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:19.559+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:32:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:19.559+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:32:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:19 smithi153 ceph-mon[36569]: pgmap v8460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:19 smithi104 ceph-mon[36624]: pgmap v8460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:19 smithi081 ceph-mon[31453]: pgmap v8460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:21 smithi153 ceph-mon[36569]: pgmap v8461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:21 smithi104 ceph-mon[36624]: pgmap v8461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:21 smithi081 ceph-mon[31453]: pgmap v8461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:23 smithi153 ceph-mon[36569]: pgmap v8462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:23 smithi104 ceph-mon[36624]: pgmap v8462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:23 smithi081 ceph-mon[31453]: pgmap v8462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:25 smithi153 ceph-mon[36569]: pgmap v8463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:25 smithi104 ceph-mon[36624]: pgmap v8463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:25 smithi081 ceph-mon[31453]: pgmap v8463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:27 smithi153 ceph-mon[36569]: pgmap v8464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:27 smithi104 ceph-mon[36624]: pgmap v8464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:27 smithi081 ceph-mon[31453]: pgmap v8464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:29 smithi153 ceph-mon[36569]: pgmap v8465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:29 smithi104 ceph-mon[36624]: pgmap v8465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:29 smithi081 ceph-mon[31453]: pgmap v8465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:30 smithi153 ceph-mon[36569]: pgmap v8466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:30 smithi104 ceph-mon[36624]: pgmap v8466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:30 smithi081 ceph-mon[31453]: pgmap v8466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:33 smithi153 ceph-mon[36569]: pgmap v8467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:33 smithi104 ceph-mon[36624]: pgmap v8467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:33 smithi081 ceph-mon[31453]: pgmap v8467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:34.574+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:32:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:34.574+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:32:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:34.574+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:32:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:35 smithi153 ceph-mon[36569]: pgmap v8468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:35 smithi104 ceph-mon[36624]: pgmap v8468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:35 smithi081 ceph-mon[31453]: pgmap v8468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:37.458 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:37 smithi081 ceph-mon[31453]: pgmap v8469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:37 smithi153 ceph-mon[36569]: pgmap v8469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:37 smithi104 ceph-mon[36624]: pgmap v8469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:39 smithi153 ceph-mon[36569]: pgmap v8470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:39 smithi104 ceph-mon[36624]: pgmap v8470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:39 smithi081 ceph-mon[31453]: pgmap v8470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:41 smithi153 ceph-mon[36569]: pgmap v8471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:41 smithi104 ceph-mon[36624]: pgmap v8471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:41 smithi081 ceph-mon[31453]: pgmap v8471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:43 smithi153 ceph-mon[36569]: pgmap v8472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:43 smithi104 ceph-mon[36624]: pgmap v8472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:43 smithi081 ceph-mon[31453]: pgmap v8472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:45 smithi153 ceph-mon[36569]: pgmap v8473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:45 smithi104 ceph-mon[36624]: pgmap v8473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:45 smithi081 ceph-mon[31453]: pgmap v8473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:47 smithi153 ceph-mon[36569]: pgmap v8474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:47 smithi104 ceph-mon[36624]: pgmap v8474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:47 smithi081 ceph-mon[31453]: pgmap v8474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:49 smithi153 ceph-mon[36569]: pgmap v8475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:49 smithi104 ceph-mon[36624]: pgmap v8475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:49.590 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:49.590+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:32:49.590 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:49.590+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:32:49.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:32:49.590+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:32:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:49 smithi081 ceph-mon[31453]: pgmap v8475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:51 smithi153 ceph-mon[36569]: pgmap v8476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:51 smithi104 ceph-mon[36624]: pgmap v8476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:51 smithi081 ceph-mon[31453]: pgmap v8476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:53 smithi153 ceph-mon[36569]: pgmap v8477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:53 smithi104 ceph-mon[36624]: pgmap v8477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:53 smithi081 ceph-mon[31453]: pgmap v8477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:55 smithi153 ceph-mon[36569]: pgmap v8478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:55 smithi104 ceph-mon[36624]: pgmap v8478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:55 smithi081 ceph-mon[31453]: pgmap v8478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:57 smithi153 ceph-mon[36569]: pgmap v8479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:57 smithi104 ceph-mon[36624]: pgmap v8479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:57 smithi081 ceph-mon[31453]: pgmap v8479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:32:59 smithi153 ceph-mon[36569]: pgmap v8480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:32:59 smithi104 ceph-mon[36624]: pgmap v8480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:32:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:32:59 smithi081 ceph-mon[31453]: pgmap v8480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:01 smithi153 ceph-mon[36569]: pgmap v8481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:01 smithi104 ceph-mon[36624]: pgmap v8481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:01 smithi081 ceph-mon[31453]: pgmap v8481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:03 smithi153 ceph-mon[36569]: pgmap v8482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:03 smithi104 ceph-mon[36624]: pgmap v8482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:03 smithi081 ceph-mon[31453]: pgmap v8482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:04.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:04.590+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:33:04.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:04.590+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:33:04.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:04.590+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:33:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:05 smithi153 ceph-mon[36569]: pgmap v8483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:05 smithi104 ceph-mon[36624]: pgmap v8483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:05 smithi081 ceph-mon[31453]: pgmap v8483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:07 smithi153 ceph-mon[36569]: pgmap v8484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:07 smithi104 ceph-mon[36624]: pgmap v8484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:07 smithi081 ceph-mon[31453]: pgmap v8484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:09 smithi153 ceph-mon[36569]: pgmap v8485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:33:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:09 smithi104 ceph-mon[36624]: pgmap v8485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:33:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:09 smithi081 ceph-mon[31453]: pgmap v8485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:33:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:33:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:33:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:10 smithi153 ceph-mon[36569]: pgmap v8486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:10 smithi153 ceph-mon[36569]: pgmap v8487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:33:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:33:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:33:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:10 smithi104 ceph-mon[36624]: pgmap v8486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:10 smithi104 ceph-mon[36624]: pgmap v8487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:33:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:33:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:33:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:10 smithi081 ceph-mon[31453]: pgmap v8486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:10 smithi081 ceph-mon[31453]: pgmap v8487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:33:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:12 smithi153 ceph-mon[36569]: pgmap v8488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:12 smithi104 ceph-mon[36624]: pgmap v8488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:12 smithi081 ceph-mon[31453]: pgmap v8488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:14 smithi153 ceph-mon[36569]: pgmap v8489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:14 smithi104 ceph-mon[36624]: pgmap v8489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:14 smithi081 ceph-mon[31453]: pgmap v8489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:16 smithi153 ceph-mon[36569]: pgmap v8490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:16 smithi104 ceph-mon[36624]: pgmap v8490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:16 smithi081 ceph-mon[31453]: pgmap v8490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:18 smithi153 ceph-mon[36569]: pgmap v8491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:18 smithi104 ceph-mon[36624]: pgmap v8491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:18 smithi081 ceph-mon[31453]: pgmap v8491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:19.602 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:19.602+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:33:19.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:19.602+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:33:19.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:19.602+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:33:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:20 smithi153 ceph-mon[36569]: pgmap v8492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:20 smithi104 ceph-mon[36624]: pgmap v8492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:20 smithi081 ceph-mon[31453]: pgmap v8492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:22 smithi153 ceph-mon[36569]: pgmap v8493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:22 smithi104 ceph-mon[36624]: pgmap v8493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:22 smithi081 ceph-mon[31453]: pgmap v8493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:24 smithi153 ceph-mon[36569]: pgmap v8494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:24 smithi104 ceph-mon[36624]: pgmap v8494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:24 smithi081 ceph-mon[31453]: pgmap v8494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:25 smithi153 ceph-mon[36569]: pgmap v8495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:25 smithi104 ceph-mon[36624]: pgmap v8495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:25 smithi081 ceph-mon[31453]: pgmap v8495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:28 smithi153 ceph-mon[36569]: pgmap v8496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:28 smithi104 ceph-mon[36624]: pgmap v8496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:28 smithi081 ceph-mon[31453]: pgmap v8496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:30 smithi153 ceph-mon[36569]: pgmap v8497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:30 smithi104 ceph-mon[36624]: pgmap v8497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:30 smithi081 ceph-mon[31453]: pgmap v8497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:32 smithi153 ceph-mon[36569]: pgmap v8498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:32 smithi104 ceph-mon[36624]: pgmap v8498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:32 smithi081 ceph-mon[31453]: pgmap v8498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:34.618+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:33:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:34.618+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:33:34.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:34.618+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:33:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:34 smithi153 ceph-mon[36569]: pgmap v8499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:34 smithi104 ceph-mon[36624]: pgmap v8499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:34 smithi081 ceph-mon[31453]: pgmap v8499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:36 smithi153 ceph-mon[36569]: pgmap v8500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:36 smithi104 ceph-mon[36624]: pgmap v8500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:36 smithi081 ceph-mon[31453]: pgmap v8500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:38 smithi153 ceph-mon[36569]: pgmap v8501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:38 smithi104 ceph-mon[36624]: pgmap v8501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:38 smithi081 ceph-mon[31453]: pgmap v8501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:39 smithi153 ceph-mon[36569]: pgmap v8502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:39 smithi104 ceph-mon[36624]: pgmap v8502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:39 smithi081 ceph-mon[31453]: pgmap v8502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:42 smithi153 ceph-mon[36569]: pgmap v8503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:42 smithi104 ceph-mon[36624]: pgmap v8503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:42 smithi081 ceph-mon[31453]: pgmap v8503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:44 smithi153 ceph-mon[36569]: pgmap v8504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:44 smithi104 ceph-mon[36624]: pgmap v8504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:44 smithi081 ceph-mon[31453]: pgmap v8504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:46 smithi153 ceph-mon[36569]: pgmap v8505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:46 smithi104 ceph-mon[36624]: pgmap v8505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:46 smithi081 ceph-mon[31453]: pgmap v8505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:48 smithi153 ceph-mon[36569]: pgmap v8506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:48 smithi104 ceph-mon[36624]: pgmap v8506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:48 smithi081 ceph-mon[31453]: pgmap v8506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:49.633+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:33:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:49.633+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:33:49.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:33:49.633+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:33:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:49 smithi153 ceph-mon[36569]: pgmap v8507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:49 smithi104 ceph-mon[36624]: pgmap v8507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:50.087 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:49 smithi081 ceph-mon[31453]: pgmap v8507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:52 smithi153 ceph-mon[36569]: pgmap v8508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:52 smithi104 ceph-mon[36624]: pgmap v8508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:52 smithi081 ceph-mon[31453]: pgmap v8508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:54 smithi153 ceph-mon[36569]: pgmap v8509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:54 smithi104 ceph-mon[36624]: pgmap v8509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:54 smithi081 ceph-mon[31453]: pgmap v8509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:56 smithi153 ceph-mon[36569]: pgmap v8510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:56 smithi104 ceph-mon[36624]: pgmap v8510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:56 smithi081 ceph-mon[31453]: pgmap v8510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:58 smithi153 ceph-mon[36569]: pgmap v8511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:58 smithi104 ceph-mon[36624]: pgmap v8511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:33:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:58 smithi081 ceph-mon[31453]: pgmap v8511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:33:59 smithi153 ceph-mon[36569]: pgmap v8512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:33:59 smithi104 ceph-mon[36624]: pgmap v8512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:33:59 smithi081 ceph-mon[31453]: pgmap v8512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:02 smithi153 ceph-mon[36569]: pgmap v8513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:02 smithi104 ceph-mon[36624]: pgmap v8513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:02 smithi081 ceph-mon[31453]: pgmap v8513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:04.646+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:34:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:04.647+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:34:04.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:04.647+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:34:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:04 smithi153 ceph-mon[36569]: pgmap v8514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:04 smithi104 ceph-mon[36624]: pgmap v8514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:04 smithi081 ceph-mon[31453]: pgmap v8514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:06 smithi153 ceph-mon[36569]: pgmap v8515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:06 smithi104 ceph-mon[36624]: pgmap v8515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:06 smithi081 ceph-mon[31453]: pgmap v8515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:07 smithi153 ceph-mon[36569]: pgmap v8516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:08.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:07 smithi081 ceph-mon[31453]: pgmap v8516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:07 smithi104 ceph-mon[36624]: pgmap v8516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:10 smithi153 ceph-mon[36569]: pgmap v8517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:34:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:10 smithi104 ceph-mon[36624]: pgmap v8517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:34:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:10 smithi081 ceph-mon[31453]: pgmap v8517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:34:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:34:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:34:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:34:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:34:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:34:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:34:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:34:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:34:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:34:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:12 smithi153 ceph-mon[36569]: pgmap v8518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:12 smithi153 ceph-mon[36569]: pgmap v8519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:12 smithi104 ceph-mon[36624]: pgmap v8518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:12 smithi104 ceph-mon[36624]: pgmap v8519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:12 smithi081 ceph-mon[31453]: pgmap v8518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:12 smithi081 ceph-mon[31453]: pgmap v8519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:13 smithi153 ceph-mon[36569]: pgmap v8520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:13 smithi104 ceph-mon[36624]: pgmap v8520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:13 smithi081 ceph-mon[31453]: pgmap v8520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:15.260+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:34:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:15.260+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:34:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:15.261+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:34:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:15.261+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:34:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:16 smithi153 ceph-mon[36569]: pgmap v8521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:16 smithi104 ceph-mon[36624]: pgmap v8521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:16 smithi081 ceph-mon[31453]: pgmap v8521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:18 smithi153 ceph-mon[36569]: pgmap v8522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:18 smithi104 ceph-mon[36624]: pgmap v8522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:18 smithi081 ceph-mon[31453]: pgmap v8522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:19.657+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:34:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:19.657+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:34:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:19.657+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:34:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:20 smithi153 ceph-mon[36569]: pgmap v8523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:20 smithi104 ceph-mon[36624]: pgmap v8523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:20 smithi081 ceph-mon[31453]: pgmap v8523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:22 smithi153 ceph-mon[36569]: pgmap v8524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:22 smithi104 ceph-mon[36624]: pgmap v8524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:22 smithi081 ceph-mon[31453]: pgmap v8524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:24 smithi153 ceph-mon[36569]: pgmap v8525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:24 smithi104 ceph-mon[36624]: pgmap v8525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:24 smithi081 ceph-mon[31453]: pgmap v8525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:26 smithi153 ceph-mon[36569]: pgmap v8526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:26 smithi104 ceph-mon[36624]: pgmap v8526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:26 smithi081 ceph-mon[31453]: pgmap v8526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:28 smithi153 ceph-mon[36569]: pgmap v8527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:28 smithi104 ceph-mon[36624]: pgmap v8527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:28 smithi081 ceph-mon[31453]: pgmap v8527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:30 smithi153 ceph-mon[36569]: pgmap v8528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:30 smithi104 ceph-mon[36624]: pgmap v8528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:30 smithi081 ceph-mon[31453]: pgmap v8528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:32 smithi153 ceph-mon[36569]: pgmap v8529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:32 smithi104 ceph-mon[36624]: pgmap v8529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:32 smithi081 ceph-mon[31453]: pgmap v8529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:34 smithi153 ceph-mon[36569]: pgmap v8530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:34 smithi104 ceph-mon[36624]: pgmap v8530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:34 smithi081 ceph-mon[31453]: pgmap v8530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:34.666+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:34:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:34.666+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:34:34.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:34.666+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:34:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:36 smithi153 ceph-mon[36569]: pgmap v8531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:36 smithi104 ceph-mon[36624]: pgmap v8531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:36 smithi081 ceph-mon[31453]: pgmap v8531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:38 smithi153 ceph-mon[36569]: pgmap v8532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:38.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:38 smithi081 ceph-mon[31453]: pgmap v8532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:38 smithi104 ceph-mon[36624]: pgmap v8532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:40 smithi153 ceph-mon[36569]: pgmap v8533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:40 smithi104 ceph-mon[36624]: pgmap v8533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:40 smithi081 ceph-mon[31453]: pgmap v8533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:42 smithi153 ceph-mon[36569]: pgmap v8534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:42 smithi104 ceph-mon[36624]: pgmap v8534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:42 smithi081 ceph-mon[31453]: pgmap v8534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:44 smithi104 ceph-mon[36624]: pgmap v8535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:44 smithi081 ceph-mon[31453]: pgmap v8535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:44 smithi153 ceph-mon[36569]: pgmap v8535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:46 smithi104 ceph-mon[36624]: pgmap v8536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:46 smithi081 ceph-mon[31453]: pgmap v8536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:46 smithi153 ceph-mon[36569]: pgmap v8536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:48 smithi104 ceph-mon[36624]: pgmap v8537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:48 smithi081 ceph-mon[31453]: pgmap v8537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:48 smithi153 ceph-mon[36569]: pgmap v8537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:49.678+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:34:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:49.678+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:34:49.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:34:49.678+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:34:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:50 smithi104 ceph-mon[36624]: pgmap v8538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:50 smithi081 ceph-mon[31453]: pgmap v8538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:50 smithi153 ceph-mon[36569]: pgmap v8538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:52 smithi104 ceph-mon[36624]: pgmap v8539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:52 smithi081 ceph-mon[31453]: pgmap v8539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:52 smithi153 ceph-mon[36569]: pgmap v8539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:54.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:54 smithi153 ceph-mon[36569]: pgmap v8540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:54 smithi081 ceph-mon[31453]: pgmap v8540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:54 smithi104 ceph-mon[36624]: pgmap v8540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:56 smithi081 ceph-mon[31453]: pgmap v8541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:56 smithi153 ceph-mon[36569]: pgmap v8541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:56 smithi104 ceph-mon[36624]: pgmap v8541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:34:58 smithi081 ceph-mon[31453]: pgmap v8542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:34:58 smithi153 ceph-mon[36569]: pgmap v8542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:34:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:34:58 smithi104 ceph-mon[36624]: pgmap v8542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:00 smithi081 ceph-mon[31453]: pgmap v8543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:00 smithi153 ceph-mon[36569]: pgmap v8543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:00 smithi104 ceph-mon[36624]: pgmap v8543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:02 smithi081 ceph-mon[31453]: pgmap v8544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:02 smithi153 ceph-mon[36569]: pgmap v8544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:02 smithi104 ceph-mon[36624]: pgmap v8544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:04.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:04 smithi104 ceph-mon[36624]: pgmap v8545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:04 smithi081 ceph-mon[31453]: pgmap v8545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:04.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:04.693+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:35:04.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:04.694+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:35:04.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:04.694+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:35:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:04 smithi153 ceph-mon[36569]: pgmap v8545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:06 smithi081 ceph-mon[31453]: pgmap v8546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:06 smithi153 ceph-mon[36569]: pgmap v8546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:06 smithi104 ceph-mon[36624]: pgmap v8546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:08 smithi081 ceph-mon[31453]: pgmap v8547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:08 smithi153 ceph-mon[36569]: pgmap v8547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:08 smithi104 ceph-mon[36624]: pgmap v8547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:10 smithi081 ceph-mon[31453]: pgmap v8548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:10 smithi153 ceph-mon[36569]: pgmap v8548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:10 smithi104 ceph-mon[36624]: pgmap v8548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:12 smithi081 ceph-mon[31453]: pgmap v8549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:35:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:12 smithi153 ceph-mon[36569]: pgmap v8549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:35:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:12 smithi104 ceph-mon[36624]: pgmap v8549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:35:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:35:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:35:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:35:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:35:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:35:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:35:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:35:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:35:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:35:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:14 smithi081 ceph-mon[31453]: pgmap v8550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:14 smithi081 ceph-mon[31453]: pgmap v8551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:14 smithi153 ceph-mon[36569]: pgmap v8550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:14 smithi153 ceph-mon[36569]: pgmap v8551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:14 smithi104 ceph-mon[36624]: pgmap v8550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:14 smithi104 ceph-mon[36624]: pgmap v8551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:16 smithi081 ceph-mon[31453]: pgmap v8552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:16 smithi153 ceph-mon[36569]: pgmap v8552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:16 smithi104 ceph-mon[36624]: pgmap v8552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:18 smithi153 ceph-mon[36569]: pgmap v8553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:18 smithi104 ceph-mon[36624]: pgmap v8553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:18 smithi081 ceph-mon[31453]: pgmap v8553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:19.702+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:35:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:19.702+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:35:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:19.702+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:35:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:20 smithi153 ceph-mon[36569]: pgmap v8554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:20 smithi104 ceph-mon[36624]: pgmap v8554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:20 smithi081 ceph-mon[31453]: pgmap v8554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:22 smithi153 ceph-mon[36569]: pgmap v8555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:22 smithi104 ceph-mon[36624]: pgmap v8555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:22 smithi081 ceph-mon[31453]: pgmap v8555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:24 smithi153 ceph-mon[36569]: pgmap v8556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:24 smithi104 ceph-mon[36624]: pgmap v8556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:24 smithi081 ceph-mon[31453]: pgmap v8556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:26 smithi153 ceph-mon[36569]: pgmap v8557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:26 smithi104 ceph-mon[36624]: pgmap v8557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:26 smithi081 ceph-mon[31453]: pgmap v8557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:28 smithi153 ceph-mon[36569]: pgmap v8558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:28 smithi104 ceph-mon[36624]: pgmap v8558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:28 smithi081 ceph-mon[31453]: pgmap v8558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:30 smithi153 ceph-mon[36569]: pgmap v8559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:30 smithi104 ceph-mon[36624]: pgmap v8559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:30 smithi081 ceph-mon[31453]: pgmap v8559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:32 smithi153 ceph-mon[36569]: pgmap v8560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:32 smithi104 ceph-mon[36624]: pgmap v8560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:32.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:32 smithi081 ceph-mon[31453]: pgmap v8560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:34.711+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:35:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:34.711+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:35:34.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:34.711+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:35:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:34 smithi153 ceph-mon[36569]: pgmap v8561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:34 smithi104 ceph-mon[36624]: pgmap v8561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:34 smithi081 ceph-mon[31453]: pgmap v8561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:36 smithi153 ceph-mon[36569]: pgmap v8562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:36 smithi104 ceph-mon[36624]: pgmap v8562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:36 smithi081 ceph-mon[31453]: pgmap v8562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:38 smithi153 ceph-mon[36569]: pgmap v8563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:38 smithi104 ceph-mon[36624]: pgmap v8563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:38 smithi081 ceph-mon[31453]: pgmap v8563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:40 smithi153 ceph-mon[36569]: pgmap v8564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:40 smithi104 ceph-mon[36624]: pgmap v8564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:40 smithi081 ceph-mon[31453]: pgmap v8564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:42 smithi153 ceph-mon[36569]: pgmap v8565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:42 smithi104 ceph-mon[36624]: pgmap v8565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:42 smithi081 ceph-mon[31453]: pgmap v8565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:44 smithi153 ceph-mon[36569]: pgmap v8566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:44 smithi104 ceph-mon[36624]: pgmap v8566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:44 smithi081 ceph-mon[31453]: pgmap v8566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:46 smithi153 ceph-mon[36569]: pgmap v8567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:46 smithi104 ceph-mon[36624]: pgmap v8567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:46 smithi081 ceph-mon[31453]: pgmap v8567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:48 smithi153 ceph-mon[36569]: pgmap v8568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:48 smithi104 ceph-mon[36624]: pgmap v8568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:48 smithi081 ceph-mon[31453]: pgmap v8568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:49.717 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:49.716+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:35:49.717 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:49.716+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:35:49.717 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:35:49.716+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:35:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:50 smithi153 ceph-mon[36569]: pgmap v8569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:50 smithi104 ceph-mon[36624]: pgmap v8569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:50 smithi081 ceph-mon[31453]: pgmap v8569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:52 smithi153 ceph-mon[36569]: pgmap v8570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:52 smithi104 ceph-mon[36624]: pgmap v8570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:52 smithi081 ceph-mon[31453]: pgmap v8570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:54.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:54 smithi153 ceph-mon[36569]: pgmap v8571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:54 smithi104 ceph-mon[36624]: pgmap v8571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:54 smithi081 ceph-mon[31453]: pgmap v8571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:56 smithi153 ceph-mon[36569]: pgmap v8572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:56 smithi104 ceph-mon[36624]: pgmap v8572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:56 smithi081 ceph-mon[31453]: pgmap v8572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:35:58 smithi153 ceph-mon[36569]: pgmap v8573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:35:58 smithi104 ceph-mon[36624]: pgmap v8573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:35:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:35:58 smithi081 ceph-mon[31453]: pgmap v8573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:00 smithi153 ceph-mon[36569]: pgmap v8574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:00 smithi104 ceph-mon[36624]: pgmap v8574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:00 smithi081 ceph-mon[31453]: pgmap v8574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:02 smithi153 ceph-mon[36569]: pgmap v8575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:02 smithi104 ceph-mon[36624]: pgmap v8575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:02 smithi081 ceph-mon[31453]: pgmap v8575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:04.725+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:36:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:04.725+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:36:04.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:04.725+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:36:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:04 smithi153 ceph-mon[36569]: pgmap v8576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:04 smithi104 ceph-mon[36624]: pgmap v8576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:04 smithi081 ceph-mon[31453]: pgmap v8576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:06 smithi153 ceph-mon[36569]: pgmap v8577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:06 smithi104 ceph-mon[36624]: pgmap v8577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:06 smithi081 ceph-mon[31453]: pgmap v8577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:08 smithi153 ceph-mon[36569]: pgmap v8578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:08 smithi104 ceph-mon[36624]: pgmap v8578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:08 smithi081 ceph-mon[31453]: pgmap v8578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:10 smithi153 ceph-mon[36569]: pgmap v8579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:10 smithi104 ceph-mon[36624]: pgmap v8579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:10 smithi081 ceph-mon[31453]: pgmap v8579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:12 smithi153 ceph-mon[36569]: pgmap v8580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:12 smithi104 ceph-mon[36624]: pgmap v8580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:12 smithi081 ceph-mon[31453]: pgmap v8580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:36:13.885 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:36:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:36:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:14 smithi081 ceph-mon[31453]: pgmap v8581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:14.949 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:14 smithi104 ceph-mon[36624]: pgmap v8581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:14.949 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:14.949 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:14.949 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:14.949 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:14 smithi153 ceph-mon[36569]: pgmap v8581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:16 smithi153 ceph-mon[36569]: pgmap v8582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:16 smithi104 ceph-mon[36624]: pgmap v8582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:16 smithi081 ceph-mon[31453]: pgmap v8582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:18 smithi153 ceph-mon[36569]: pgmap v8583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:18 smithi104 ceph-mon[36624]: pgmap v8583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:18 smithi081 ceph-mon[31453]: pgmap v8583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:19.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:19.740+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:36:19.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:19.740+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:36:19.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:19.740+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:36:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:20 smithi153 ceph-mon[36569]: pgmap v8584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:36:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:36:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:20 smithi104 ceph-mon[36624]: pgmap v8584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:36:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:36:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:20 smithi081 ceph-mon[31453]: pgmap v8584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:36:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:36:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:36:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:21 smithi153 ceph-mon[36569]: pgmap v8585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:21 smithi153 ceph-mon[36569]: pgmap v8586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:21 smithi104 ceph-mon[36624]: pgmap v8585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:21 smithi104 ceph-mon[36624]: pgmap v8586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:21 smithi081 ceph-mon[31453]: pgmap v8585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:21 smithi081 ceph-mon[31453]: pgmap v8586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:23 smithi153 ceph-mon[36569]: pgmap v8587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:23 smithi104 ceph-mon[36624]: pgmap v8587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:23 smithi081 ceph-mon[31453]: pgmap v8587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:25 smithi153 ceph-mon[36569]: pgmap v8588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:25 smithi104 ceph-mon[36624]: pgmap v8588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:25 smithi081 ceph-mon[31453]: pgmap v8588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:27 smithi153 ceph-mon[36569]: pgmap v8589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:27 smithi104 ceph-mon[36624]: pgmap v8589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:27 smithi081 ceph-mon[31453]: pgmap v8589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:29 smithi153 ceph-mon[36569]: pgmap v8590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:29 smithi104 ceph-mon[36624]: pgmap v8590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:29 smithi081 ceph-mon[31453]: pgmap v8590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:31 smithi153 ceph-mon[36569]: pgmap v8591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:31 smithi104 ceph-mon[36624]: pgmap v8591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:31 smithi081 ceph-mon[31453]: pgmap v8591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:33 smithi153 ceph-mon[36569]: pgmap v8592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:33 smithi104 ceph-mon[36624]: pgmap v8592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:33 smithi081 ceph-mon[31453]: pgmap v8592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:34.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:34.755+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:36:34.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:34.755+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:36:34.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:34.755+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:36:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:35 smithi153 ceph-mon[36569]: pgmap v8593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:35 smithi104 ceph-mon[36624]: pgmap v8593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:35 smithi081 ceph-mon[31453]: pgmap v8593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:37 smithi081 ceph-mon[31453]: pgmap v8594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:37 smithi153 ceph-mon[36569]: pgmap v8594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:37 smithi104 ceph-mon[36624]: pgmap v8594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:39 smithi153 ceph-mon[36569]: pgmap v8595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:39 smithi104 ceph-mon[36624]: pgmap v8595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:39 smithi081 ceph-mon[31453]: pgmap v8595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:41 smithi153 ceph-mon[36569]: pgmap v8596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:41 smithi104 ceph-mon[36624]: pgmap v8596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:41 smithi081 ceph-mon[31453]: pgmap v8596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:43 smithi153 ceph-mon[36569]: pgmap v8597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:43 smithi104 ceph-mon[36624]: pgmap v8597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:43 smithi081 ceph-mon[31453]: pgmap v8597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:45 smithi153 ceph-mon[36569]: pgmap v8598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:45 smithi104 ceph-mon[36624]: pgmap v8598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:45 smithi081 ceph-mon[31453]: pgmap v8598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:47 smithi104 ceph-mon[36624]: pgmap v8599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:47 smithi153 ceph-mon[36569]: pgmap v8599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:47 smithi081 ceph-mon[31453]: pgmap v8599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:49.771+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:36:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:49.771+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:36:49.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:36:49.771+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:36:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:49 smithi153 ceph-mon[36569]: pgmap v8600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:49 smithi104 ceph-mon[36624]: pgmap v8600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:49 smithi081 ceph-mon[31453]: pgmap v8600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:51 smithi153 ceph-mon[36569]: pgmap v8601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:51 smithi104 ceph-mon[36624]: pgmap v8601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:51 smithi081 ceph-mon[31453]: pgmap v8601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:52 smithi153 ceph-mon[36569]: pgmap v8602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:52 smithi104 ceph-mon[36624]: pgmap v8602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:52 smithi081 ceph-mon[31453]: pgmap v8602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:54 smithi153 ceph-mon[36569]: pgmap v8603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:54 smithi104 ceph-mon[36624]: pgmap v8603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:54 smithi081 ceph-mon[31453]: pgmap v8603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:56 smithi153 ceph-mon[36569]: pgmap v8604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:56 smithi104 ceph-mon[36624]: pgmap v8604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:56 smithi081 ceph-mon[31453]: pgmap v8604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:36:58 smithi153 ceph-mon[36569]: pgmap v8605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:36:58 smithi104 ceph-mon[36624]: pgmap v8605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:36:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:36:58 smithi081 ceph-mon[31453]: pgmap v8605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:01 smithi153 ceph-mon[36569]: pgmap v8606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:01 smithi104 ceph-mon[36624]: pgmap v8606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:01 smithi081 ceph-mon[31453]: pgmap v8606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:03 smithi153 ceph-mon[36569]: pgmap v8607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:03 smithi104 ceph-mon[36624]: pgmap v8607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:03 smithi081 ceph-mon[31453]: pgmap v8607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:04.786+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:37:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:04.786+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:37:04.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:04.786+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:37:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:05 smithi153 ceph-mon[36569]: pgmap v8608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:05 smithi104 ceph-mon[36624]: pgmap v8608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:05 smithi081 ceph-mon[31453]: pgmap v8608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:07 smithi153 ceph-mon[36569]: pgmap v8609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:07 smithi104 ceph-mon[36624]: pgmap v8609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:07 smithi081 ceph-mon[31453]: pgmap v8609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:09 smithi153 ceph-mon[36569]: pgmap v8610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:09 smithi104 ceph-mon[36624]: pgmap v8610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:09 smithi081 ceph-mon[31453]: pgmap v8610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:11.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:11 smithi081 ceph-mon[31453]: pgmap v8611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:11 smithi153 ceph-mon[36569]: pgmap v8611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:11 smithi104 ceph-mon[36624]: pgmap v8611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:13 smithi153 ceph-mon[36569]: pgmap v8612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:13 smithi104 ceph-mon[36624]: pgmap v8612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:13 smithi081 ceph-mon[31453]: pgmap v8612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:15.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:15.289+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:37:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:15.289+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:37:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:15.290+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:37:15.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:15.290+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:37:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:15 smithi153 ceph-mon[36569]: pgmap v8613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:15 smithi104 ceph-mon[36624]: pgmap v8613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:15 smithi081 ceph-mon[31453]: pgmap v8613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:17 smithi153 ceph-mon[36569]: pgmap v8614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:17 smithi104 ceph-mon[36624]: pgmap v8614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:17 smithi081 ceph-mon[31453]: pgmap v8614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:19 smithi153 ceph-mon[36569]: pgmap v8615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:19 smithi104 ceph-mon[36624]: pgmap v8615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:19 smithi081 ceph-mon[31453]: pgmap v8615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:19.795+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:37:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:19.795+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:37:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:19.795+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:37:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:21 smithi153 ceph-mon[36569]: pgmap v8616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:37:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:21 smithi104 ceph-mon[36624]: pgmap v8616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:37:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:21 smithi081 ceph-mon[31453]: pgmap v8616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:37:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:23 smithi153 ceph-mon[36569]: pgmap v8617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:37:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:37:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:23 smithi153 ceph-mon[36569]: pgmap v8618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:23 smithi153 ceph-mon[36569]: pgmap v8619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:23 smithi104 ceph-mon[36624]: pgmap v8617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:37:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:37:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:23 smithi104 ceph-mon[36624]: pgmap v8618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:23 smithi104 ceph-mon[36624]: pgmap v8619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:23 smithi081 ceph-mon[31453]: pgmap v8617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:23 smithi081 ceph-mon[31453]: pgmap v8618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:23 smithi081 ceph-mon[31453]: pgmap v8619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:37:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:25 smithi081 ceph-mon[31453]: pgmap v8620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:25 smithi153 ceph-mon[36569]: pgmap v8620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:25 smithi104 ceph-mon[36624]: pgmap v8620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:27.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:27 smithi081 ceph-mon[31453]: pgmap v8621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:27 smithi153 ceph-mon[36569]: pgmap v8621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:27 smithi104 ceph-mon[36624]: pgmap v8621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:29 smithi081 ceph-mon[31453]: pgmap v8622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:29 smithi153 ceph-mon[36569]: pgmap v8622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:29 smithi104 ceph-mon[36624]: pgmap v8622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:31 smithi081 ceph-mon[31453]: pgmap v8623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:31 smithi153 ceph-mon[36569]: pgmap v8623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:31 smithi104 ceph-mon[36624]: pgmap v8623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:33 smithi081 ceph-mon[31453]: pgmap v8624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:33 smithi153 ceph-mon[36569]: pgmap v8624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:33 smithi104 ceph-mon[36624]: pgmap v8624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:34.809+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:37:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:34.809+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:37:34.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:34.810+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:37:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:35 smithi081 ceph-mon[31453]: pgmap v8625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:35 smithi153 ceph-mon[36569]: pgmap v8625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:35 smithi104 ceph-mon[36624]: pgmap v8625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:37 smithi081 ceph-mon[31453]: pgmap v8626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:37 smithi153 ceph-mon[36569]: pgmap v8626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:37 smithi104 ceph-mon[36624]: pgmap v8626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:39 smithi153 ceph-mon[36569]: pgmap v8627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:39 smithi104 ceph-mon[36624]: pgmap v8627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:39 smithi081 ceph-mon[31453]: pgmap v8627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:41 smithi153 ceph-mon[36569]: pgmap v8628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:41 smithi104 ceph-mon[36624]: pgmap v8628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:41.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:41 smithi081 ceph-mon[31453]: pgmap v8628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:43 smithi153 ceph-mon[36569]: pgmap v8629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:43 smithi104 ceph-mon[36624]: pgmap v8629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:43 smithi081 ceph-mon[31453]: pgmap v8629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:45.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:45 smithi153 ceph-mon[36569]: pgmap v8630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:45 smithi104 ceph-mon[36624]: pgmap v8630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:45 smithi081 ceph-mon[31453]: pgmap v8630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:47 smithi153 ceph-mon[36569]: pgmap v8631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:47 smithi104 ceph-mon[36624]: pgmap v8631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:47 smithi081 ceph-mon[31453]: pgmap v8631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:49 smithi153 ceph-mon[36569]: pgmap v8632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:49.825+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:37:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:49.825+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:37:49.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:37:49.825+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:37:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:49 smithi104 ceph-mon[36624]: pgmap v8632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:49 smithi081 ceph-mon[31453]: pgmap v8632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:51 smithi153 ceph-mon[36569]: pgmap v8633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:51 smithi104 ceph-mon[36624]: pgmap v8633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:51 smithi081 ceph-mon[31453]: pgmap v8633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:53 smithi153 ceph-mon[36569]: pgmap v8634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:53 smithi104 ceph-mon[36624]: pgmap v8634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:53 smithi081 ceph-mon[31453]: pgmap v8634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:55 smithi153 ceph-mon[36569]: pgmap v8635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:55 smithi104 ceph-mon[36624]: pgmap v8635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:55 smithi081 ceph-mon[31453]: pgmap v8635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:57 smithi153 ceph-mon[36569]: pgmap v8636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:57 smithi104 ceph-mon[36624]: pgmap v8636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:57.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:57 smithi081 ceph-mon[31453]: pgmap v8636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:37:59 smithi081 ceph-mon[31453]: pgmap v8637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:37:59 smithi153 ceph-mon[36569]: pgmap v8637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:37:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:37:59 smithi104 ceph-mon[36624]: pgmap v8637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:01 smithi081 ceph-mon[31453]: pgmap v8638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:01 smithi153 ceph-mon[36569]: pgmap v8638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:01 smithi104 ceph-mon[36624]: pgmap v8638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:03 smithi081 ceph-mon[31453]: pgmap v8639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:03 smithi153 ceph-mon[36569]: pgmap v8639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:03.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:03 smithi104 ceph-mon[36624]: pgmap v8639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:04.834+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:38:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:04.834+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:38:04.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:04.834+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:38:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:05 smithi081 ceph-mon[31453]: pgmap v8640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:05 smithi153 ceph-mon[36569]: pgmap v8640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:05 smithi104 ceph-mon[36624]: pgmap v8640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:07 smithi153 ceph-mon[36569]: pgmap v8641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:07 smithi104 ceph-mon[36624]: pgmap v8641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:07 smithi081 ceph-mon[31453]: pgmap v8641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:09.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:09 smithi081 ceph-mon[31453]: pgmap v8642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:09 smithi153 ceph-mon[36569]: pgmap v8642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:09 smithi104 ceph-mon[36624]: pgmap v8642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:11 smithi153 ceph-mon[36569]: pgmap v8643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:11 smithi104 ceph-mon[36624]: pgmap v8643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:11 smithi081 ceph-mon[31453]: pgmap v8643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:13 smithi081 ceph-mon[31453]: pgmap v8644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:13 smithi153 ceph-mon[36569]: pgmap v8644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:13 smithi104 ceph-mon[36624]: pgmap v8644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:15 smithi153 ceph-mon[36569]: pgmap v8645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:15 smithi104 ceph-mon[36624]: pgmap v8645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:15 smithi081 ceph-mon[31453]: pgmap v8645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:17 smithi153 ceph-mon[36569]: pgmap v8646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:17 smithi104 ceph-mon[36624]: pgmap v8646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:17 smithi081 ceph-mon[31453]: pgmap v8646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:19 smithi153 ceph-mon[36569]: pgmap v8647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:19 smithi104 ceph-mon[36624]: pgmap v8647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:19.835+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:19.835+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:38:19.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:19.835+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:38:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:19 smithi081 ceph-mon[31453]: pgmap v8647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:21 smithi153 ceph-mon[36569]: pgmap v8648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:21 smithi104 ceph-mon[36624]: pgmap v8648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:21 smithi081 ceph-mon[31453]: pgmap v8648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:23.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:23 smithi153 ceph-mon[36569]: pgmap v8649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:23.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:38:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:23 smithi104 ceph-mon[36624]: pgmap v8649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:38:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:23 smithi081 ceph-mon[31453]: pgmap v8649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: pgmap v8650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: pgmap v8651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:38:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:38:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: pgmap v8650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: pgmap v8651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: pgmap v8650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: pgmap v8651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:38:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:26 smithi153 ceph-mon[36569]: pgmap v8652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:26 smithi104 ceph-mon[36624]: pgmap v8652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:26 smithi081 ceph-mon[31453]: pgmap v8652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:29.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:28 smithi081 ceph-mon[31453]: pgmap v8653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:28 smithi153 ceph-mon[36569]: pgmap v8653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:28 smithi104 ceph-mon[36624]: pgmap v8653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:30 smithi153 ceph-mon[36569]: pgmap v8654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:30 smithi104 ceph-mon[36624]: pgmap v8654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:30 smithi081 ceph-mon[31453]: pgmap v8654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:32 smithi153 ceph-mon[36569]: pgmap v8655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:32 smithi104 ceph-mon[36624]: pgmap v8655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:32 smithi081 ceph-mon[31453]: pgmap v8655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:34.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:34.837+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:38:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:34.837+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:38:34.838 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:34.837+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:38:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:34 smithi153 ceph-mon[36569]: pgmap v8656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:34 smithi104 ceph-mon[36624]: pgmap v8656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:34 smithi081 ceph-mon[31453]: pgmap v8656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:36 smithi153 ceph-mon[36569]: pgmap v8657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:36 smithi104 ceph-mon[36624]: pgmap v8657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:36 smithi081 ceph-mon[31453]: pgmap v8657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:38 smithi153 ceph-mon[36569]: pgmap v8658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:38 smithi104 ceph-mon[36624]: pgmap v8658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:38 smithi081 ceph-mon[31453]: pgmap v8658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:40 smithi153 ceph-mon[36569]: pgmap v8659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:40 smithi104 ceph-mon[36624]: pgmap v8659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:40 smithi081 ceph-mon[31453]: pgmap v8659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:42 smithi153 ceph-mon[36569]: pgmap v8660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:42 smithi104 ceph-mon[36624]: pgmap v8660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:42 smithi081 ceph-mon[31453]: pgmap v8660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:44 smithi153 ceph-mon[36569]: pgmap v8661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:44 smithi104 ceph-mon[36624]: pgmap v8661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:44 smithi081 ceph-mon[31453]: pgmap v8661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:46 smithi153 ceph-mon[36569]: pgmap v8662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:46 smithi104 ceph-mon[36624]: pgmap v8662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:46 smithi081 ceph-mon[31453]: pgmap v8662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:48 smithi153 ceph-mon[36569]: pgmap v8663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:48 smithi104 ceph-mon[36624]: pgmap v8663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:48 smithi081 ceph-mon[31453]: pgmap v8663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:49.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:49.852+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:38:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:49.852+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:38:49.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:38:49.852+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:38:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:50 smithi153 ceph-mon[36569]: pgmap v8664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:50 smithi104 ceph-mon[36624]: pgmap v8664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:50 smithi081 ceph-mon[31453]: pgmap v8664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:53 smithi153 ceph-mon[36569]: pgmap v8665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:53 smithi104 ceph-mon[36624]: pgmap v8665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:53 smithi081 ceph-mon[31453]: pgmap v8665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:55 smithi153 ceph-mon[36569]: pgmap v8666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:55 smithi104 ceph-mon[36624]: pgmap v8666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:55 smithi081 ceph-mon[31453]: pgmap v8666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:57 smithi153 ceph-mon[36569]: pgmap v8667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:57 smithi104 ceph-mon[36624]: pgmap v8667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:57 smithi081 ceph-mon[31453]: pgmap v8667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:38:59 smithi153 ceph-mon[36569]: pgmap v8668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:38:59 smithi104 ceph-mon[36624]: pgmap v8668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:38:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:38:59 smithi081 ceph-mon[31453]: pgmap v8668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:01 smithi153 ceph-mon[36569]: pgmap v8669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:01 smithi104 ceph-mon[36624]: pgmap v8669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:01 smithi081 ceph-mon[31453]: pgmap v8669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:03 smithi153 ceph-mon[36569]: pgmap v8670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:03 smithi104 ceph-mon[36624]: pgmap v8670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:03 smithi081 ceph-mon[31453]: pgmap v8670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:04.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:04.867+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:39:04.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:04.867+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:39:04.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:04.867+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:39:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:05 smithi153 ceph-mon[36569]: pgmap v8671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:05 smithi104 ceph-mon[36624]: pgmap v8671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:05 smithi081 ceph-mon[31453]: pgmap v8671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:07 smithi153 ceph-mon[36569]: pgmap v8672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:07 smithi104 ceph-mon[36624]: pgmap v8672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:07 smithi081 ceph-mon[31453]: pgmap v8672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:09 smithi153 ceph-mon[36569]: pgmap v8673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:09 smithi104 ceph-mon[36624]: pgmap v8673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:09 smithi081 ceph-mon[31453]: pgmap v8673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:11 smithi153 ceph-mon[36569]: pgmap v8674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:11 smithi104 ceph-mon[36624]: pgmap v8674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:11 smithi081 ceph-mon[31453]: pgmap v8674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:13 smithi153 ceph-mon[36569]: pgmap v8675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:13 smithi104 ceph-mon[36624]: pgmap v8675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:13 smithi081 ceph-mon[31453]: pgmap v8675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:15 smithi104 ceph-mon[36624]: pgmap v8676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:15 smithi081 ceph-mon[31453]: pgmap v8676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:15 smithi153 ceph-mon[36569]: pgmap v8676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:17 smithi104 ceph-mon[36624]: pgmap v8677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:17 smithi081 ceph-mon[31453]: pgmap v8677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:17 smithi153 ceph-mon[36569]: pgmap v8677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:19 smithi104 ceph-mon[36624]: pgmap v8678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:19 smithi081 ceph-mon[31453]: pgmap v8678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:19 smithi153 ceph-mon[36569]: pgmap v8678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:19.872+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:39:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:19.872+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:39:19.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:19.872+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:39:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:21 smithi104 ceph-mon[36624]: pgmap v8679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:21 smithi081 ceph-mon[31453]: pgmap v8679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:21 smithi153 ceph-mon[36569]: pgmap v8679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:23 smithi104 ceph-mon[36624]: pgmap v8680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:23 smithi081 ceph-mon[31453]: pgmap v8680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:23 smithi153 ceph-mon[36569]: pgmap v8680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:25.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:25 smithi153 ceph-mon[36569]: pgmap v8681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:25.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:39:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:25 smithi081 ceph-mon[31453]: pgmap v8681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:39:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:25 smithi104 ceph-mon[36624]: pgmap v8681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:39:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:39:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:39:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:26 smithi081 ceph-mon[31453]: pgmap v8682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:26 smithi081 ceph-mon[31453]: pgmap v8683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:39:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:39:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:39:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:26 smithi153 ceph-mon[36569]: pgmap v8682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:26 smithi153 ceph-mon[36569]: pgmap v8683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:39:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:39:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:39:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:26 smithi104 ceph-mon[36624]: pgmap v8682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:26 smithi104 ceph-mon[36624]: pgmap v8683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:39:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:28 smithi153 ceph-mon[36569]: pgmap v8684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:28 smithi104 ceph-mon[36624]: pgmap v8684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:28 smithi081 ceph-mon[31453]: pgmap v8684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:30 smithi153 ceph-mon[36569]: pgmap v8685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:30 smithi104 ceph-mon[36624]: pgmap v8685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:30 smithi081 ceph-mon[31453]: pgmap v8685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:32 smithi153 ceph-mon[36569]: pgmap v8686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:32 smithi104 ceph-mon[36624]: pgmap v8686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:32 smithi081 ceph-mon[31453]: pgmap v8686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:34 smithi153 ceph-mon[36569]: pgmap v8687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:34 smithi104 ceph-mon[36624]: pgmap v8687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:34.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:34.884+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:39:34.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:34.884+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:39:34.885 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:34.884+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:39:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:34 smithi081 ceph-mon[31453]: pgmap v8687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:36 smithi153 ceph-mon[36569]: pgmap v8688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:36 smithi104 ceph-mon[36624]: pgmap v8688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:36 smithi081 ceph-mon[31453]: pgmap v8688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:38 smithi153 ceph-mon[36569]: pgmap v8689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:38 smithi104 ceph-mon[36624]: pgmap v8689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:38 smithi081 ceph-mon[31453]: pgmap v8689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:40 smithi153 ceph-mon[36569]: pgmap v8690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:40 smithi104 ceph-mon[36624]: pgmap v8690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:40 smithi081 ceph-mon[31453]: pgmap v8690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:42 smithi153 ceph-mon[36569]: pgmap v8691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:42 smithi104 ceph-mon[36624]: pgmap v8691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:42 smithi081 ceph-mon[31453]: pgmap v8691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:44 smithi153 ceph-mon[36569]: pgmap v8692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:44 smithi104 ceph-mon[36624]: pgmap v8692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:44 smithi081 ceph-mon[31453]: pgmap v8692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:46 smithi153 ceph-mon[36569]: pgmap v8693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:46 smithi104 ceph-mon[36624]: pgmap v8693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:46 smithi081 ceph-mon[31453]: pgmap v8693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:48 smithi104 ceph-mon[36624]: pgmap v8694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:48 smithi081 ceph-mon[31453]: pgmap v8694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:48 smithi153 ceph-mon[36569]: pgmap v8694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:49.887+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:39:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:49.887+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:39:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:39:49.887+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:39:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:50 smithi104 ceph-mon[36624]: pgmap v8695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:50 smithi081 ceph-mon[31453]: pgmap v8695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:50 smithi153 ceph-mon[36569]: pgmap v8695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:52 smithi104 ceph-mon[36624]: pgmap v8696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:52 smithi081 ceph-mon[31453]: pgmap v8696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:52 smithi153 ceph-mon[36569]: pgmap v8696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:54 smithi104 ceph-mon[36624]: pgmap v8697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:54 smithi081 ceph-mon[31453]: pgmap v8697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:54 smithi153 ceph-mon[36569]: pgmap v8697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:56 smithi104 ceph-mon[36624]: pgmap v8698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:56.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:56 smithi153 ceph-mon[36569]: pgmap v8698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:56 smithi081 ceph-mon[31453]: pgmap v8698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:39:58 smithi081 ceph-mon[31453]: pgmap v8699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:39:58 smithi153 ceph-mon[36569]: pgmap v8699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:39:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:39:58 smithi104 ceph-mon[36624]: pgmap v8699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:00 smithi081 ceph-mon[31453]: pgmap v8700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:00 smithi153 ceph-mon[36569]: pgmap v8700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:40:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:00 smithi104 ceph-mon[36624]: pgmap v8700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:40:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:02 smithi081 ceph-mon[31453]: pgmap v8701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:02 smithi153 ceph-mon[36569]: pgmap v8701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:02 smithi104 ceph-mon[36624]: pgmap v8701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:04.902+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:40:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:04.902+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:40:04.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:04.902+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:40:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:04 smithi081 ceph-mon[31453]: pgmap v8702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:04 smithi153 ceph-mon[36569]: pgmap v8702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:04 smithi104 ceph-mon[36624]: pgmap v8702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:06.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:06 smithi104 ceph-mon[36624]: pgmap v8703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:06 smithi081 ceph-mon[31453]: pgmap v8703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:06 smithi153 ceph-mon[36569]: pgmap v8703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:08 smithi081 ceph-mon[31453]: pgmap v8704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:08 smithi153 ceph-mon[36569]: pgmap v8704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:08 smithi104 ceph-mon[36624]: pgmap v8704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:10 smithi081 ceph-mon[31453]: pgmap v8705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:10 smithi153 ceph-mon[36569]: pgmap v8705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:10 smithi104 ceph-mon[36624]: pgmap v8705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:12 smithi081 ceph-mon[31453]: pgmap v8706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:12 smithi153 ceph-mon[36569]: pgmap v8706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:12 smithi104 ceph-mon[36624]: pgmap v8706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:14 smithi081 ceph-mon[31453]: pgmap v8707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:14 smithi153 ceph-mon[36569]: pgmap v8707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:14 smithi104 ceph-mon[36624]: pgmap v8707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:15.319+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:40:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:15.319+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:40:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:15.319+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T01:40:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:15.319+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T01:40:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:15.319+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:40:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:15.319+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:40:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:15.320+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:40:15.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:15.320+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:40:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:16 smithi081 ceph-mon[31453]: pgmap v8708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:16 smithi153 ceph-mon[36569]: pgmap v8708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:16 smithi104 ceph-mon[36624]: pgmap v8708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:18 smithi081 ceph-mon[31453]: pgmap v8709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:18 smithi153 ceph-mon[36569]: pgmap v8709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:18 smithi104 ceph-mon[36624]: pgmap v8709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:19.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:19.917+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:40:19.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:19.917+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:40:19.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:19.917+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:40:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:20 smithi081 ceph-mon[31453]: pgmap v8710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:20 smithi153 ceph-mon[36569]: pgmap v8710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:20 smithi104 ceph-mon[36624]: pgmap v8710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:22 smithi081 ceph-mon[31453]: pgmap v8711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:22 smithi153 ceph-mon[36569]: pgmap v8711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:22 smithi104 ceph-mon[36624]: pgmap v8711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:24 smithi081 ceph-mon[31453]: pgmap v8712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:24 smithi153 ceph-mon[36569]: pgmap v8712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:24 smithi104 ceph-mon[36624]: pgmap v8712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:26 smithi153 ceph-mon[36569]: pgmap v8713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:40:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:26 smithi104 ceph-mon[36624]: pgmap v8713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:40:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:26 smithi081 ceph-mon[31453]: pgmap v8713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:40:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:40:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:40:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:40:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:40:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:40:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:40:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:40:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:40:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:40:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:28 smithi153 ceph-mon[36569]: pgmap v8714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:28 smithi153 ceph-mon[36569]: pgmap v8715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:28 smithi104 ceph-mon[36624]: pgmap v8714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:28 smithi104 ceph-mon[36624]: pgmap v8715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:28 smithi081 ceph-mon[31453]: pgmap v8714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:28 smithi081 ceph-mon[31453]: pgmap v8715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:30 smithi104 ceph-mon[36624]: pgmap v8716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:30 smithi081 ceph-mon[31453]: pgmap v8716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:30 smithi153 ceph-mon[36569]: pgmap v8716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:32 smithi104 ceph-mon[36624]: pgmap v8717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:32 smithi081 ceph-mon[31453]: pgmap v8717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:32 smithi153 ceph-mon[36569]: pgmap v8717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:34 smithi104 ceph-mon[36624]: pgmap v8718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:34 smithi081 ceph-mon[31453]: pgmap v8718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:34.931+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:34.931+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:40:34.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:34.931+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:40:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:34 smithi153 ceph-mon[36569]: pgmap v8718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:36 smithi104 ceph-mon[36624]: pgmap v8719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:36 smithi081 ceph-mon[31453]: pgmap v8719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:36 smithi153 ceph-mon[36569]: pgmap v8719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:38 smithi081 ceph-mon[31453]: pgmap v8720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:38 smithi153 ceph-mon[36569]: pgmap v8720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:38 smithi104 ceph-mon[36624]: pgmap v8720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:40 smithi081 ceph-mon[31453]: pgmap v8721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:40 smithi153 ceph-mon[36569]: pgmap v8721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:40 smithi104 ceph-mon[36624]: pgmap v8721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:42 smithi081 ceph-mon[31453]: pgmap v8722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:42 smithi153 ceph-mon[36569]: pgmap v8722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:42 smithi104 ceph-mon[36624]: pgmap v8722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:44 smithi081 ceph-mon[31453]: pgmap v8723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:44 smithi153 ceph-mon[36569]: pgmap v8723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:44 smithi104 ceph-mon[36624]: pgmap v8723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:46 smithi081 ceph-mon[31453]: pgmap v8724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:46 smithi153 ceph-mon[36569]: pgmap v8724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:46 smithi104 ceph-mon[36624]: pgmap v8724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:48 smithi081 ceph-mon[31453]: pgmap v8725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:48 smithi153 ceph-mon[36569]: pgmap v8725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:48 smithi104 ceph-mon[36624]: pgmap v8725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:49.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:49.946+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:40:49.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:49.946+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:40:49.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:40:49.946+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:40:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:50 smithi081 ceph-mon[31453]: pgmap v8726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:50 smithi153 ceph-mon[36569]: pgmap v8726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:50 smithi104 ceph-mon[36624]: pgmap v8726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:52 smithi081 ceph-mon[31453]: pgmap v8727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:52 smithi153 ceph-mon[36569]: pgmap v8727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:52 smithi104 ceph-mon[36624]: pgmap v8727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:54 smithi081 ceph-mon[31453]: pgmap v8728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:54 smithi153 ceph-mon[36569]: pgmap v8728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:54 smithi104 ceph-mon[36624]: pgmap v8728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:56 smithi081 ceph-mon[31453]: pgmap v8729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:56 smithi153 ceph-mon[36569]: pgmap v8729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:56 smithi104 ceph-mon[36624]: pgmap v8729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:40:58 smithi081 ceph-mon[31453]: pgmap v8730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:40:58 smithi153 ceph-mon[36569]: pgmap v8730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:40:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:40:58 smithi104 ceph-mon[36624]: pgmap v8730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:00 smithi081 ceph-mon[31453]: pgmap v8731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:00 smithi153 ceph-mon[36569]: pgmap v8731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:00 smithi104 ceph-mon[36624]: pgmap v8731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:02 smithi081 ceph-mon[31453]: pgmap v8732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:02 smithi153 ceph-mon[36569]: pgmap v8732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:02 smithi104 ceph-mon[36624]: pgmap v8732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:04.956+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:41:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:04.956+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:41:04.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:04.956+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:41:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:04 smithi153 ceph-mon[36569]: pgmap v8733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:04 smithi104 ceph-mon[36624]: pgmap v8733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:04 smithi081 ceph-mon[31453]: pgmap v8733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:06 smithi153 ceph-mon[36569]: pgmap v8734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:06 smithi104 ceph-mon[36624]: pgmap v8734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:06 smithi081 ceph-mon[31453]: pgmap v8734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:08 smithi153 ceph-mon[36569]: pgmap v8735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:08 smithi104 ceph-mon[36624]: pgmap v8735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:08 smithi081 ceph-mon[31453]: pgmap v8735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:10.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:10 smithi081 ceph-mon[31453]: pgmap v8736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:10 smithi153 ceph-mon[36569]: pgmap v8736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:10 smithi104 ceph-mon[36624]: pgmap v8736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:12 smithi153 ceph-mon[36569]: pgmap v8737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:12 smithi104 ceph-mon[36624]: pgmap v8737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:12 smithi081 ceph-mon[31453]: pgmap v8737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:14 smithi153 ceph-mon[36569]: pgmap v8738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:14 smithi104 ceph-mon[36624]: pgmap v8738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:14 smithi081 ceph-mon[31453]: pgmap v8738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:16 smithi153 ceph-mon[36569]: pgmap v8739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:16 smithi104 ceph-mon[36624]: pgmap v8739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:16 smithi081 ceph-mon[31453]: pgmap v8739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:18 smithi153 ceph-mon[36569]: pgmap v8740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:18 smithi104 ceph-mon[36624]: pgmap v8740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:19.112 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:18 smithi081 ceph-mon[31453]: pgmap v8740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:19.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:19.969+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:41:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:19.969+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:41:19.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:19.969+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:41:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:20 smithi153 ceph-mon[36569]: pgmap v8741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:20 smithi104 ceph-mon[36624]: pgmap v8741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:20 smithi081 ceph-mon[31453]: pgmap v8741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:22 smithi153 ceph-mon[36569]: pgmap v8742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:22 smithi104 ceph-mon[36624]: pgmap v8742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:22 smithi081 ceph-mon[31453]: pgmap v8742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:24 smithi153 ceph-mon[36569]: pgmap v8743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:24 smithi104 ceph-mon[36624]: pgmap v8743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:24 smithi081 ceph-mon[31453]: pgmap v8743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:26 smithi153 ceph-mon[36569]: pgmap v8744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:26 smithi104 ceph-mon[36624]: pgmap v8744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:26 smithi081 ceph-mon[31453]: pgmap v8744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:41:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:41:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:41:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:28 smithi153 ceph-mon[36569]: pgmap v8745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:41:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:41:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:41:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:28 smithi104 ceph-mon[36624]: pgmap v8745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:41:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:41:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:41:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:28 smithi081 ceph-mon[31453]: pgmap v8745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:41:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:41:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:41:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:29 smithi153 ceph-mon[36569]: pgmap v8746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:29 smithi153 ceph-mon[36569]: pgmap v8747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:29 smithi104 ceph-mon[36624]: pgmap v8746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:29 smithi104 ceph-mon[36624]: pgmap v8747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:29 smithi081 ceph-mon[31453]: pgmap v8746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:29 smithi081 ceph-mon[31453]: pgmap v8747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:31 smithi153 ceph-mon[36569]: pgmap v8748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:31 smithi104 ceph-mon[36624]: pgmap v8748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:31 smithi081 ceph-mon[31453]: pgmap v8748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:33 smithi153 ceph-mon[36569]: pgmap v8749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:33 smithi104 ceph-mon[36624]: pgmap v8749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:33 smithi081 ceph-mon[31453]: pgmap v8749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:34.976+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:34.976+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:41:34.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:34.976+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:41:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:35 smithi153 ceph-mon[36569]: pgmap v8750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:35 smithi104 ceph-mon[36624]: pgmap v8750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:35 smithi081 ceph-mon[31453]: pgmap v8750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:37 smithi153 ceph-mon[36569]: pgmap v8751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:37 smithi104 ceph-mon[36624]: pgmap v8751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:37 smithi081 ceph-mon[31453]: pgmap v8751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:39 smithi153 ceph-mon[36569]: pgmap v8752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:39 smithi104 ceph-mon[36624]: pgmap v8752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:39 smithi081 ceph-mon[31453]: pgmap v8752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:41 smithi153 ceph-mon[36569]: pgmap v8753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:41 smithi104 ceph-mon[36624]: pgmap v8753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:41 smithi081 ceph-mon[31453]: pgmap v8753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:43 smithi153 ceph-mon[36569]: pgmap v8754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:43 smithi104 ceph-mon[36624]: pgmap v8754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:43 smithi081 ceph-mon[31453]: pgmap v8754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:45 smithi153 ceph-mon[36569]: pgmap v8755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:45 smithi104 ceph-mon[36624]: pgmap v8755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:45 smithi081 ceph-mon[31453]: pgmap v8755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:47 smithi153 ceph-mon[36569]: pgmap v8756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:47 smithi104 ceph-mon[36624]: pgmap v8756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:47 smithi081 ceph-mon[31453]: pgmap v8756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:49.983+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:41:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:49.983+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:41:49.984 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:41:49.983+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:41:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:49 smithi153 ceph-mon[36569]: pgmap v8757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:49 smithi104 ceph-mon[36624]: pgmap v8757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:49 smithi081 ceph-mon[31453]: pgmap v8757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:51 smithi153 ceph-mon[36569]: pgmap v8758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:51 smithi104 ceph-mon[36624]: pgmap v8758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:51 smithi081 ceph-mon[31453]: pgmap v8758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:53 smithi153 ceph-mon[36569]: pgmap v8759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:53 smithi104 ceph-mon[36624]: pgmap v8759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:53 smithi081 ceph-mon[31453]: pgmap v8759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:55 smithi153 ceph-mon[36569]: pgmap v8760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:55 smithi104 ceph-mon[36624]: pgmap v8760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:55 smithi081 ceph-mon[31453]: pgmap v8760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:57 smithi104 ceph-mon[36624]: pgmap v8761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:57 smithi081 ceph-mon[31453]: pgmap v8761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:41:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:57 smithi153 ceph-mon[36569]: pgmap v8761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:41:59 smithi104 ceph-mon[36624]: pgmap v8762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:41:59 smithi081 ceph-mon[31453]: pgmap v8762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:41:59 smithi153 ceph-mon[36569]: pgmap v8762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:01 smithi104 ceph-mon[36624]: pgmap v8763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:01 smithi081 ceph-mon[31453]: pgmap v8763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:01 smithi153 ceph-mon[36569]: pgmap v8763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:03 smithi104 ceph-mon[36624]: pgmap v8764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:03 smithi081 ceph-mon[31453]: pgmap v8764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:03 smithi153 ceph-mon[36569]: pgmap v8764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:04.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:04.988+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:42:04.990 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:04.988+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:42:04.990 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:04.988+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:42:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:05 smithi081 ceph-mon[31453]: pgmap v8765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:05 smithi153 ceph-mon[36569]: pgmap v8765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:05 smithi104 ceph-mon[36624]: pgmap v8765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:08.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:07 smithi104 ceph-mon[36624]: pgmap v8766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:07 smithi081 ceph-mon[31453]: pgmap v8766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:07 smithi153 ceph-mon[36569]: pgmap v8766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:09 smithi081 ceph-mon[31453]: pgmap v8767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:09 smithi153 ceph-mon[36569]: pgmap v8767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:09 smithi104 ceph-mon[36624]: pgmap v8767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:11 smithi081 ceph-mon[31453]: pgmap v8768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:11 smithi153 ceph-mon[36569]: pgmap v8768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:11 smithi104 ceph-mon[36624]: pgmap v8768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:12 smithi081 ceph-mon[31453]: pgmap v8769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:12 smithi153 ceph-mon[36569]: pgmap v8769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:12 smithi104 ceph-mon[36624]: pgmap v8769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:15 smithi153 ceph-mon[36569]: pgmap v8770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:15 smithi104 ceph-mon[36624]: pgmap v8770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:15 smithi081 ceph-mon[31453]: pgmap v8770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:17 smithi153 ceph-mon[36569]: pgmap v8771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:17 smithi104 ceph-mon[36624]: pgmap v8771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:17 smithi081 ceph-mon[31453]: pgmap v8771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:19.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:19.994+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:42:19.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:19.994+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:42:19.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:19.994+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:42:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:19 smithi153 ceph-mon[36569]: pgmap v8772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:19 smithi104 ceph-mon[36624]: pgmap v8772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:19 smithi081 ceph-mon[31453]: pgmap v8772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:21 smithi153 ceph-mon[36569]: pgmap v8773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:21 smithi104 ceph-mon[36624]: pgmap v8773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:22.088 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:21 smithi081 ceph-mon[31453]: pgmap v8773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:23 smithi153 ceph-mon[36569]: pgmap v8774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:23 smithi104 ceph-mon[36624]: pgmap v8774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:23 smithi081 ceph-mon[31453]: pgmap v8774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:25 smithi153 ceph-mon[36569]: pgmap v8775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:25 smithi104 ceph-mon[36624]: pgmap v8775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:25 smithi081 ceph-mon[31453]: pgmap v8775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:27 smithi153 ceph-mon[36569]: pgmap v8776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:27 smithi104 ceph-mon[36624]: pgmap v8776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:27 smithi081 ceph-mon[31453]: pgmap v8776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: pgmap v8777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:42:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: pgmap v8777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:30.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:42:30.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: pgmap v8777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:42:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:42:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:42:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:42:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:42:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:42:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:42:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:42:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:31 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:42:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:31 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:42:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:31 smithi153 ceph-mon[36569]: pgmap v8778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:31 smithi153 ceph-mon[36569]: pgmap v8779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:31 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:42:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:31 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:42:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:31 smithi104 ceph-mon[36624]: pgmap v8778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:31 smithi104 ceph-mon[36624]: pgmap v8779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:31 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:42:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:31 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:42:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:31 smithi081 ceph-mon[31453]: pgmap v8778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:31 smithi081 ceph-mon[31453]: pgmap v8779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:33 smithi153 ceph-mon[36569]: pgmap v8780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:33 smithi104 ceph-mon[36624]: pgmap v8780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:33 smithi081 ceph-mon[31453]: pgmap v8780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:35.007+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:42:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:35.007+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:42:35.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:35.007+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:42:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:35 smithi153 ceph-mon[36569]: pgmap v8781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:35 smithi104 ceph-mon[36624]: pgmap v8781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:35 smithi081 ceph-mon[31453]: pgmap v8781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:37 smithi153 ceph-mon[36569]: pgmap v8782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:37 smithi104 ceph-mon[36624]: pgmap v8782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:37 smithi081 ceph-mon[31453]: pgmap v8782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:39 smithi153 ceph-mon[36569]: pgmap v8783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:39 smithi104 ceph-mon[36624]: pgmap v8783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:39 smithi081 ceph-mon[31453]: pgmap v8783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:41 smithi153 ceph-mon[36569]: pgmap v8784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:41 smithi104 ceph-mon[36624]: pgmap v8784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:41 smithi081 ceph-mon[31453]: pgmap v8784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:43 smithi153 ceph-mon[36569]: pgmap v8785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:43 smithi104 ceph-mon[36624]: pgmap v8785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:43 smithi081 ceph-mon[31453]: pgmap v8785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:45 smithi153 ceph-mon[36569]: pgmap v8786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:45 smithi104 ceph-mon[36624]: pgmap v8786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:45 smithi081 ceph-mon[31453]: pgmap v8786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:47 smithi153 ceph-mon[36569]: pgmap v8787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:47 smithi104 ceph-mon[36624]: pgmap v8787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:47 smithi081 ceph-mon[31453]: pgmap v8787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:50.011+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:42:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:50.011+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:42:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:42:50.011+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:42:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:49 smithi153 ceph-mon[36569]: pgmap v8788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:49 smithi104 ceph-mon[36624]: pgmap v8788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:49 smithi081 ceph-mon[31453]: pgmap v8788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:51 smithi153 ceph-mon[36569]: pgmap v8789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:51 smithi104 ceph-mon[36624]: pgmap v8789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:51 smithi081 ceph-mon[31453]: pgmap v8789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:53 smithi153 ceph-mon[36569]: pgmap v8790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:53 smithi104 ceph-mon[36624]: pgmap v8790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:53 smithi081 ceph-mon[31453]: pgmap v8790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:56.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:55 smithi153 ceph-mon[36569]: pgmap v8791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:55 smithi104 ceph-mon[36624]: pgmap v8791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:55 smithi081 ceph-mon[31453]: pgmap v8791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:57 smithi153 ceph-mon[36569]: pgmap v8792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:57 smithi104 ceph-mon[36624]: pgmap v8792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:42:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:57 smithi081 ceph-mon[31453]: pgmap v8792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:42:59 smithi153 ceph-mon[36569]: pgmap v8793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:42:59 smithi104 ceph-mon[36624]: pgmap v8793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:42:59 smithi081 ceph-mon[31453]: pgmap v8793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:01 smithi153 ceph-mon[36569]: pgmap v8794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:01 smithi104 ceph-mon[36624]: pgmap v8794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:01 smithi081 ceph-mon[31453]: pgmap v8794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:03 smithi104 ceph-mon[36624]: pgmap v8795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:03 smithi081 ceph-mon[31453]: pgmap v8795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:03 smithi153 ceph-mon[36569]: pgmap v8795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:05.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:05.016+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:43:05.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:05.016+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:43:05.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:05.016+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:43:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:05 smithi104 ceph-mon[36624]: pgmap v8796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:05 smithi081 ceph-mon[31453]: pgmap v8796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:05 smithi153 ceph-mon[36569]: pgmap v8796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:07 smithi104 ceph-mon[36624]: pgmap v8797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:07 smithi081 ceph-mon[31453]: pgmap v8797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:07 smithi153 ceph-mon[36569]: pgmap v8797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:09 smithi104 ceph-mon[36624]: pgmap v8798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:09 smithi081 ceph-mon[31453]: pgmap v8798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:09 smithi153 ceph-mon[36569]: pgmap v8798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:11 smithi104 ceph-mon[36624]: pgmap v8799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:11 smithi081 ceph-mon[31453]: pgmap v8799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:11 smithi153 ceph-mon[36569]: pgmap v8799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:13 smithi081 ceph-mon[31453]: pgmap v8800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:13 smithi153 ceph-mon[36569]: pgmap v8800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:13 smithi104 ceph-mon[36624]: pgmap v8800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:15.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:15.349+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:43:15.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:15.349+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:43:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:15.349+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:43:15.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:15.349+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:43:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:15 smithi081 ceph-mon[31453]: pgmap v8801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:15 smithi153 ceph-mon[36569]: pgmap v8801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:15 smithi104 ceph-mon[36624]: pgmap v8801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:17 smithi081 ceph-mon[31453]: pgmap v8802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:17 smithi153 ceph-mon[36569]: pgmap v8802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:17 smithi104 ceph-mon[36624]: pgmap v8802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:20.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:20.028+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:43:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:20.028+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:43:20.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:20.028+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:43:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:19 smithi081 ceph-mon[31453]: pgmap v8803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:19 smithi153 ceph-mon[36569]: pgmap v8803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:19 smithi104 ceph-mon[36624]: pgmap v8803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:21 smithi081 ceph-mon[31453]: pgmap v8804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:21 smithi153 ceph-mon[36569]: pgmap v8804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:21 smithi104 ceph-mon[36624]: pgmap v8804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:22 smithi153 ceph-mon[36569]: pgmap v8805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:22 smithi104 ceph-mon[36624]: pgmap v8805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:22 smithi081 ceph-mon[31453]: pgmap v8805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:25 smithi081 ceph-mon[31453]: pgmap v8806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:25 smithi153 ceph-mon[36569]: pgmap v8806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:25 smithi104 ceph-mon[36624]: pgmap v8806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:27 smithi153 ceph-mon[36569]: pgmap v8807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:27 smithi104 ceph-mon[36624]: pgmap v8807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:27 smithi081 ceph-mon[31453]: pgmap v8807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:29 smithi153 ceph-mon[36569]: pgmap v8808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:29 smithi104 ceph-mon[36624]: pgmap v8808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:29 smithi081 ceph-mon[31453]: pgmap v8808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:31 smithi104 ceph-mon[36624]: pgmap v8809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:43:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:31 smithi081 ceph-mon[31453]: pgmap v8809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:43:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:31 smithi153 ceph-mon[36569]: pgmap v8809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:43:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:33 smithi104 ceph-mon[36624]: pgmap v8810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:33 smithi081 ceph-mon[31453]: pgmap v8810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:33 smithi153 ceph-mon[36569]: pgmap v8810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:35.043+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:43:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:35.043+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:43:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:35.044+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:43:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:35 smithi104 ceph-mon[36624]: pgmap v8811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:35 smithi081 ceph-mon[31453]: pgmap v8811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:35 smithi153 ceph-mon[36569]: pgmap v8811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:37 smithi081 ceph-mon[31453]: pgmap v8812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:37 smithi104 ceph-mon[36624]: pgmap v8812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:37 smithi153 ceph-mon[36569]: pgmap v8812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:43:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:43:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:38 smithi081 ceph-mon[31453]: pgmap v8813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:38 smithi081 ceph-mon[31453]: pgmap v8814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:43:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:43:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:38 smithi153 ceph-mon[36569]: pgmap v8813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:38 smithi153 ceph-mon[36569]: pgmap v8814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:39.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:39.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:39.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:43:39.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:43:39.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:38 smithi104 ceph-mon[36624]: pgmap v8813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:39.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:38 smithi104 ceph-mon[36624]: pgmap v8814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:39.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:43:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:40 smithi153 ceph-mon[36569]: pgmap v8815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:40 smithi104 ceph-mon[36624]: pgmap v8815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:40 smithi081 ceph-mon[31453]: pgmap v8815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:42 smithi153 ceph-mon[36569]: pgmap v8816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:42 smithi104 ceph-mon[36624]: pgmap v8816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:42 smithi081 ceph-mon[31453]: pgmap v8816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:44 smithi153 ceph-mon[36569]: pgmap v8817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:44 smithi104 ceph-mon[36624]: pgmap v8817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:44 smithi081 ceph-mon[31453]: pgmap v8817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:46 smithi153 ceph-mon[36569]: pgmap v8818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:46 smithi104 ceph-mon[36624]: pgmap v8818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:46 smithi081 ceph-mon[31453]: pgmap v8818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:48 smithi153 ceph-mon[36569]: pgmap v8819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:48 smithi104 ceph-mon[36624]: pgmap v8819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:48 smithi081 ceph-mon[31453]: pgmap v8819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:50.059+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:43:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:50.059+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:43:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:43:50.059+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:43:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:50 smithi153 ceph-mon[36569]: pgmap v8820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:50 smithi104 ceph-mon[36624]: pgmap v8820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:50 smithi081 ceph-mon[31453]: pgmap v8820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:52 smithi153 ceph-mon[36569]: pgmap v8821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:52 smithi104 ceph-mon[36624]: pgmap v8821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:52 smithi081 ceph-mon[31453]: pgmap v8821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:54 smithi153 ceph-mon[36569]: pgmap v8822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:54 smithi104 ceph-mon[36624]: pgmap v8822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:54 smithi081 ceph-mon[31453]: pgmap v8822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:56 smithi153 ceph-mon[36569]: pgmap v8823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:56 smithi104 ceph-mon[36624]: pgmap v8823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:56 smithi081 ceph-mon[31453]: pgmap v8823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:43:58 smithi153 ceph-mon[36569]: pgmap v8824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:43:58 smithi104 ceph-mon[36624]: pgmap v8824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:43:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:43:58 smithi081 ceph-mon[31453]: pgmap v8824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:00 smithi153 ceph-mon[36569]: pgmap v8825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:00 smithi104 ceph-mon[36624]: pgmap v8825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:00 smithi081 ceph-mon[31453]: pgmap v8825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:02 smithi153 ceph-mon[36569]: pgmap v8826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:02 smithi104 ceph-mon[36624]: pgmap v8826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:02 smithi081 ceph-mon[31453]: pgmap v8826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:04 smithi153 ceph-mon[36569]: pgmap v8827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:05.069+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:44:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:05.069+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:44:05.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:05.069+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:44:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:04 smithi104 ceph-mon[36624]: pgmap v8827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:04 smithi081 ceph-mon[31453]: pgmap v8827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:06 smithi153 ceph-mon[36569]: pgmap v8828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:06 smithi104 ceph-mon[36624]: pgmap v8828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:06 smithi081 ceph-mon[31453]: pgmap v8828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:08 smithi153 ceph-mon[36569]: pgmap v8829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:08 smithi104 ceph-mon[36624]: pgmap v8829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:08 smithi081 ceph-mon[31453]: pgmap v8829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:10 smithi153 ceph-mon[36569]: pgmap v8830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:10 smithi104 ceph-mon[36624]: pgmap v8830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:11.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:10 smithi081 ceph-mon[31453]: pgmap v8830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:12 smithi153 ceph-mon[36569]: pgmap v8831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:12 smithi104 ceph-mon[36624]: pgmap v8831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:12 smithi081 ceph-mon[31453]: pgmap v8831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:14 smithi153 ceph-mon[36569]: pgmap v8832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:14 smithi104 ceph-mon[36624]: pgmap v8832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:14 smithi081 ceph-mon[31453]: pgmap v8832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:16 smithi153 ceph-mon[36569]: pgmap v8833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:16 smithi104 ceph-mon[36624]: pgmap v8833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:16 smithi081 ceph-mon[31453]: pgmap v8833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:18 smithi153 ceph-mon[36569]: pgmap v8834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:18 smithi104 ceph-mon[36624]: pgmap v8834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:18 smithi081 ceph-mon[31453]: pgmap v8834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:20.083+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:44:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:20.083+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:44:20.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:20.083+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:44:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:20 smithi153 ceph-mon[36569]: pgmap v8835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:20 smithi104 ceph-mon[36624]: pgmap v8835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:20 smithi081 ceph-mon[31453]: pgmap v8835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:23.079 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:22 smithi081 ceph-mon[31453]: pgmap v8836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:23.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:22 smithi104 ceph-mon[36624]: pgmap v8836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:22 smithi153 ceph-mon[36569]: pgmap v8836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:24 smithi153 ceph-mon[36569]: pgmap v8837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:24 smithi104 ceph-mon[36624]: pgmap v8837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:24.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:24 smithi081 ceph-mon[31453]: pgmap v8837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:27.016 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:26 smithi081 ceph-mon[31453]: pgmap v8838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:26 smithi153 ceph-mon[36569]: pgmap v8838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:26 smithi104 ceph-mon[36624]: pgmap v8838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:28 smithi153 ceph-mon[36569]: pgmap v8839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:28 smithi104 ceph-mon[36624]: pgmap v8839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:28 smithi081 ceph-mon[31453]: pgmap v8839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:30 smithi153 ceph-mon[36569]: pgmap v8840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:30 smithi104 ceph-mon[36624]: pgmap v8840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:30 smithi081 ceph-mon[31453]: pgmap v8840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:32 smithi153 ceph-mon[36569]: pgmap v8841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:32 smithi104 ceph-mon[36624]: pgmap v8841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:32 smithi081 ceph-mon[31453]: pgmap v8841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:34 smithi153 ceph-mon[36569]: pgmap v8842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:34 smithi104 ceph-mon[36624]: pgmap v8842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:35.096+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:44:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:35.096+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:44:35.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:35.096+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:44:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:34 smithi081 ceph-mon[31453]: pgmap v8842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:36 smithi153 ceph-mon[36569]: pgmap v8843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:36 smithi104 ceph-mon[36624]: pgmap v8843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:36 smithi081 ceph-mon[31453]: pgmap v8843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:38 smithi153 ceph-mon[36569]: pgmap v8844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:44:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:38 smithi104 ceph-mon[36624]: pgmap v8844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:44:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:38 smithi081 ceph-mon[31453]: pgmap v8844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:44:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:40 smithi153 ceph-mon[36569]: pgmap v8845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:40 smithi104 ceph-mon[36624]: pgmap v8845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:40 smithi081 ceph-mon[31453]: pgmap v8845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:42 smithi153 ceph-mon[36569]: pgmap v8846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:44:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:44:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:42 smithi104 ceph-mon[36624]: pgmap v8846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:44:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:44:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:42 smithi081 ceph-mon[31453]: pgmap v8846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:44:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:44:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:44:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:43 smithi153 ceph-mon[36569]: pgmap v8847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:44.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:43 smithi153 ceph-mon[36569]: pgmap v8848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:43 smithi104 ceph-mon[36624]: pgmap v8847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:43 smithi104 ceph-mon[36624]: pgmap v8848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:43 smithi081 ceph-mon[31453]: pgmap v8847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:43 smithi081 ceph-mon[31453]: pgmap v8848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:45 smithi153 ceph-mon[36569]: pgmap v8849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:45 smithi104 ceph-mon[36624]: pgmap v8849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:45 smithi081 ceph-mon[31453]: pgmap v8849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:47 smithi153 ceph-mon[36569]: pgmap v8850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:47 smithi104 ceph-mon[36624]: pgmap v8850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:47 smithi081 ceph-mon[31453]: pgmap v8850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:49 smithi153 ceph-mon[36569]: pgmap v8851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:49 smithi104 ceph-mon[36624]: pgmap v8851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:50.112+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:44:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:50.112+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:44:50.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:44:50.112+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:44:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:49 smithi081 ceph-mon[31453]: pgmap v8851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:51 smithi153 ceph-mon[36569]: pgmap v8852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:51 smithi104 ceph-mon[36624]: pgmap v8852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:51 smithi081 ceph-mon[31453]: pgmap v8852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:53 smithi104 ceph-mon[36624]: pgmap v8853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:53 smithi081 ceph-mon[31453]: pgmap v8853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:53 smithi153 ceph-mon[36569]: pgmap v8853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:55 smithi104 ceph-mon[36624]: pgmap v8854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:55 smithi081 ceph-mon[31453]: pgmap v8854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:55 smithi153 ceph-mon[36569]: pgmap v8854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:57 smithi104 ceph-mon[36624]: pgmap v8855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:58.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:57 smithi153 ceph-mon[36569]: pgmap v8855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:44:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:57 smithi081 ceph-mon[31453]: pgmap v8855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:44:59 smithi104 ceph-mon[36624]: pgmap v8856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:44:59 smithi081 ceph-mon[31453]: pgmap v8856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:44:59 smithi153 ceph-mon[36569]: pgmap v8856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:01 smithi104 ceph-mon[36624]: pgmap v8857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:01 smithi081 ceph-mon[31453]: pgmap v8857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:01 smithi153 ceph-mon[36569]: pgmap v8857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:03 smithi081 ceph-mon[31453]: pgmap v8858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:03 smithi153 ceph-mon[36569]: pgmap v8858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:03 smithi104 ceph-mon[36624]: pgmap v8858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:05.127+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:45:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:05.127+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:45:05.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:05.127+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:45:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:05 smithi081 ceph-mon[31453]: pgmap v8859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:05 smithi153 ceph-mon[36569]: pgmap v8859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:05 smithi104 ceph-mon[36624]: pgmap v8859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:07 smithi081 ceph-mon[31453]: pgmap v8860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:07 smithi153 ceph-mon[36569]: pgmap v8860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:07 smithi104 ceph-mon[36624]: pgmap v8860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:10.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:09 smithi104 ceph-mon[36624]: pgmap v8861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:09 smithi081 ceph-mon[31453]: pgmap v8861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:09 smithi153 ceph-mon[36569]: pgmap v8861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:11 smithi081 ceph-mon[31453]: pgmap v8862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:11 smithi153 ceph-mon[36569]: pgmap v8862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:11 smithi104 ceph-mon[36624]: pgmap v8862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:13 smithi081 ceph-mon[31453]: pgmap v8863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:13 smithi153 ceph-mon[36569]: pgmap v8863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:13 smithi104 ceph-mon[36624]: pgmap v8863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:15 smithi081 ceph-mon[31453]: pgmap v8864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:15 smithi153 ceph-mon[36569]: pgmap v8864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:15 smithi104 ceph-mon[36624]: pgmap v8864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:17 smithi081 ceph-mon[31453]: pgmap v8865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:17 smithi153 ceph-mon[36569]: pgmap v8865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:17 smithi104 ceph-mon[36624]: pgmap v8865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:20.137+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:45:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:20.137+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:45:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:20.137+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:45:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:19 smithi081 ceph-mon[31453]: pgmap v8866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:19 smithi153 ceph-mon[36569]: pgmap v8866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:19 smithi104 ceph-mon[36624]: pgmap v8866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:21 smithi081 ceph-mon[31453]: pgmap v8867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:21 smithi153 ceph-mon[36569]: pgmap v8867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:21 smithi104 ceph-mon[36624]: pgmap v8867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:23 smithi081 ceph-mon[31453]: pgmap v8868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:23 smithi153 ceph-mon[36569]: pgmap v8868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:23 smithi104 ceph-mon[36624]: pgmap v8868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:25 smithi081 ceph-mon[31453]: pgmap v8869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:25 smithi153 ceph-mon[36569]: pgmap v8869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:25 smithi104 ceph-mon[36624]: pgmap v8869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:26 smithi081 ceph-mon[31453]: pgmap v8870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:26 smithi153 ceph-mon[36569]: pgmap v8870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:26 smithi104 ceph-mon[36624]: pgmap v8870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:30.010 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:29 smithi153 ceph-mon[36569]: pgmap v8871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:29 smithi104 ceph-mon[36624]: pgmap v8871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:29 smithi081 ceph-mon[31453]: pgmap v8871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:31 smithi153 ceph-mon[36569]: pgmap v8872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:31 smithi104 ceph-mon[36624]: pgmap v8872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:31 smithi081 ceph-mon[31453]: pgmap v8872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:33 smithi153 ceph-mon[36569]: pgmap v8873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:33 smithi104 ceph-mon[36624]: pgmap v8873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:33 smithi081 ceph-mon[31453]: pgmap v8873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:35.146+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:45:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:35.146+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:45:35.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:35.146+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:45:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:35 smithi153 ceph-mon[36569]: pgmap v8874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:35 smithi104 ceph-mon[36624]: pgmap v8874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:35 smithi081 ceph-mon[31453]: pgmap v8874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:37 smithi153 ceph-mon[36569]: pgmap v8875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:37 smithi104 ceph-mon[36624]: pgmap v8875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:37 smithi081 ceph-mon[31453]: pgmap v8875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:39 smithi153 ceph-mon[36569]: pgmap v8876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:39 smithi104 ceph-mon[36624]: pgmap v8876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:39 smithi081 ceph-mon[31453]: pgmap v8876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:41 smithi153 ceph-mon[36569]: pgmap v8877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:41 smithi104 ceph-mon[36624]: pgmap v8877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:41 smithi081 ceph-mon[31453]: pgmap v8877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:43 smithi153 ceph-mon[36569]: pgmap v8878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:44.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:45:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:43 smithi104 ceph-mon[36624]: pgmap v8878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:45:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:43 smithi081 ceph-mon[31453]: pgmap v8878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:45:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:45:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:45:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:45:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:45:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:45:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:45:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:45:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:45:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:45:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:45 smithi153 ceph-mon[36569]: pgmap v8879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:45 smithi153 ceph-mon[36569]: pgmap v8880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:45 smithi104 ceph-mon[36624]: pgmap v8879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:45 smithi104 ceph-mon[36624]: pgmap v8880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:45 smithi081 ceph-mon[31453]: pgmap v8879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:45 smithi081 ceph-mon[31453]: pgmap v8880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:47 smithi153 ceph-mon[36569]: pgmap v8881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:47 smithi104 ceph-mon[36624]: pgmap v8881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:47 smithi081 ceph-mon[31453]: pgmap v8881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:49 smithi153 ceph-mon[36569]: pgmap v8882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:49 smithi104 ceph-mon[36624]: pgmap v8882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:49 smithi081 ceph-mon[31453]: pgmap v8882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:50.162+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:45:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:50.162+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:45:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:45:50.162+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:45:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:51 smithi153 ceph-mon[36569]: pgmap v8883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:51 smithi104 ceph-mon[36624]: pgmap v8883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:51 smithi081 ceph-mon[31453]: pgmap v8883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:53 smithi153 ceph-mon[36569]: pgmap v8884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:53 smithi104 ceph-mon[36624]: pgmap v8884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:53 smithi081 ceph-mon[31453]: pgmap v8884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:55 smithi153 ceph-mon[36569]: pgmap v8885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:55 smithi104 ceph-mon[36624]: pgmap v8885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:55 smithi081 ceph-mon[31453]: pgmap v8885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:57 smithi153 ceph-mon[36569]: pgmap v8886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:57 smithi104 ceph-mon[36624]: pgmap v8886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:45:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:57 smithi081 ceph-mon[31453]: pgmap v8886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:45:59 smithi153 ceph-mon[36569]: pgmap v8887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:45:59 smithi104 ceph-mon[36624]: pgmap v8887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:45:59 smithi081 ceph-mon[31453]: pgmap v8887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:01 smithi153 ceph-mon[36569]: pgmap v8888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:01 smithi104 ceph-mon[36624]: pgmap v8888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:01 smithi081 ceph-mon[31453]: pgmap v8888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:04.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:03 smithi153 ceph-mon[36569]: pgmap v8889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:03 smithi104 ceph-mon[36624]: pgmap v8889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:03 smithi081 ceph-mon[31453]: pgmap v8889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:05.177+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:46:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:05.177+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:46:05.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:05.177+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:46:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:05 smithi153 ceph-mon[36569]: pgmap v8890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:05 smithi104 ceph-mon[36624]: pgmap v8890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:05 smithi081 ceph-mon[31453]: pgmap v8890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:07 smithi104 ceph-mon[36624]: pgmap v8891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:08.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:07 smithi081 ceph-mon[31453]: pgmap v8891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:07 smithi153 ceph-mon[36569]: pgmap v8891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:09 smithi104 ceph-mon[36624]: pgmap v8892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:09 smithi081 ceph-mon[31453]: pgmap v8892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:09 smithi153 ceph-mon[36569]: pgmap v8892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:11 smithi104 ceph-mon[36624]: pgmap v8893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:12.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.133+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-13T01:46:12.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.133+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 msgr2=0x7f16e80b3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-13T01:46:12.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.133+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 0x7f16e80b3a70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f4039000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-13T01:46:12.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.133+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 0x7f16e80b3a70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005aa0 tx=0x7f16f4039000 comp rx=0 tx=0).stop 2024-04-13T01:46:12.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.133+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-13T01:46:12.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.133+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 msgr2=0x7f16e80b3a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T01:46:12.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.133+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80af510 0x7f16e80b3a70 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T01:46:12.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.133+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T01:46:12.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.134+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T01:46:12.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:12.135+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3fb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f16f402b000 tx=0x7f16f40c4000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T01:46:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:11 smithi081 ceph-mon[31453]: pgmap v8893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:11 smithi153 ceph-mon[36569]: pgmap v8893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:13 smithi104 ceph-mon[36624]: pgmap v8894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:13 smithi081 ceph-mon[31453]: pgmap v8894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:13 smithi153 ceph-mon[36569]: pgmap v8894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:15.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:15.377+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:46:15.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:15.377+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:46:15.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:15.378+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:46:15.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:15.378+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:46:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:15 smithi104 ceph-mon[36624]: pgmap v8895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:15 smithi081 ceph-mon[31453]: pgmap v8895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:15 smithi153 ceph-mon[36569]: pgmap v8895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:17 smithi081 ceph-mon[31453]: pgmap v8896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:17 smithi153 ceph-mon[36569]: pgmap v8896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:17 smithi104 ceph-mon[36624]: pgmap v8896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:19 smithi081 ceph-mon[31453]: pgmap v8897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:20.193+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:46:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:20.193+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:46:20.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:20.193+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:46:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:19 smithi153 ceph-mon[36569]: pgmap v8897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:19 smithi104 ceph-mon[36624]: pgmap v8897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:21 smithi081 ceph-mon[31453]: pgmap v8898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:21 smithi153 ceph-mon[36569]: pgmap v8898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:21 smithi104 ceph-mon[36624]: pgmap v8898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:23 smithi081 ceph-mon[31453]: pgmap v8899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:23 smithi153 ceph-mon[36569]: pgmap v8899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:23 smithi104 ceph-mon[36624]: pgmap v8899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:25 smithi081 ceph-mon[31453]: pgmap v8900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:25 smithi153 ceph-mon[36569]: pgmap v8900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:25 smithi104 ceph-mon[36624]: pgmap v8900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:27 smithi081 ceph-mon[31453]: pgmap v8901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:27 smithi153 ceph-mon[36569]: pgmap v8901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:27 smithi104 ceph-mon[36624]: pgmap v8901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:29 smithi081 ceph-mon[31453]: pgmap v8902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:29 smithi153 ceph-mon[36569]: pgmap v8902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:29 smithi104 ceph-mon[36624]: pgmap v8902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:31 smithi081 ceph-mon[31453]: pgmap v8903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:31 smithi153 ceph-mon[36569]: pgmap v8903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:31 smithi104 ceph-mon[36624]: pgmap v8903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:33 smithi081 ceph-mon[31453]: pgmap v8904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:33 smithi153 ceph-mon[36569]: pgmap v8904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:33 smithi104 ceph-mon[36624]: pgmap v8904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:35.207 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:35.206+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:46:35.207 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:35.206+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:46:35.207 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:35.206+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:46:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:35 smithi081 ceph-mon[31453]: pgmap v8905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:35 smithi153 ceph-mon[36569]: pgmap v8905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:35 smithi104 ceph-mon[36624]: pgmap v8905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:37 smithi081 ceph-mon[31453]: pgmap v8906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:37 smithi153 ceph-mon[36569]: pgmap v8906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:37 smithi104 ceph-mon[36624]: pgmap v8906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:39 smithi081 ceph-mon[31453]: pgmap v8907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:39 smithi153 ceph-mon[36569]: pgmap v8907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:39 smithi104 ceph-mon[36624]: pgmap v8907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:41 smithi081 ceph-mon[31453]: pgmap v8908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:41 smithi153 ceph-mon[36569]: pgmap v8908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:41 smithi104 ceph-mon[36624]: pgmap v8908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:43 smithi081 ceph-mon[31453]: pgmap v8909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:43 smithi153 ceph-mon[36569]: pgmap v8909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:43 smithi104 ceph-mon[36624]: pgmap v8909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:46:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:46:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:46:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:45 smithi153 ceph-mon[36569]: pgmap v8910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.281 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:45 smithi081 ceph-mon[31453]: pgmap v8910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:46.281 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.281 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.281 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.281 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:45 smithi104 ceph-mon[36624]: pgmap v8910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:46 smithi153 ceph-mon[36569]: pgmap v8911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:46 smithi104 ceph-mon[36624]: pgmap v8911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:46 smithi081 ceph-mon[31453]: pgmap v8911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:49 smithi153 ceph-mon[36569]: pgmap v8912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:49 smithi104 ceph-mon[36624]: pgmap v8912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:49 smithi081 ceph-mon[31453]: pgmap v8912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:50.211 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:50.208+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:46:50.211 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:50.208+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:46:50.211 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:46:50.208+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:46:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:51 smithi153 ceph-mon[36569]: pgmap v8913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:46:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:46:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:51 smithi104 ceph-mon[36624]: pgmap v8913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:46:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:46:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:51 smithi081 ceph-mon[31453]: pgmap v8913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:46:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:46:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:46:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:52 smithi153 ceph-mon[36569]: pgmap v8914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:52 smithi153 ceph-mon[36569]: pgmap v8915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:52 smithi104 ceph-mon[36624]: pgmap v8914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:52 smithi104 ceph-mon[36624]: pgmap v8915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:52 smithi081 ceph-mon[31453]: pgmap v8914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:52 smithi081 ceph-mon[31453]: pgmap v8915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:54 smithi153 ceph-mon[36569]: pgmap v8916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:54 smithi104 ceph-mon[36624]: pgmap v8916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:54 smithi081 ceph-mon[31453]: pgmap v8916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:56 smithi153 ceph-mon[36569]: pgmap v8917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:56 smithi104 ceph-mon[36624]: pgmap v8917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:56 smithi081 ceph-mon[31453]: pgmap v8917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:46:58 smithi153 ceph-mon[36569]: pgmap v8918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:46:58 smithi104 ceph-mon[36624]: pgmap v8918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:46:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:46:58 smithi081 ceph-mon[31453]: pgmap v8918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:00 smithi153 ceph-mon[36569]: pgmap v8919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:00 smithi104 ceph-mon[36624]: pgmap v8919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:00 smithi081 ceph-mon[31453]: pgmap v8919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:02 smithi153 ceph-mon[36569]: pgmap v8920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:02 smithi104 ceph-mon[36624]: pgmap v8920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:02 smithi081 ceph-mon[31453]: pgmap v8920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:04 smithi153 ceph-mon[36569]: pgmap v8921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:04 smithi104 ceph-mon[36624]: pgmap v8921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:04 smithi081 ceph-mon[31453]: pgmap v8921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:05.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:05.215+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:47:05.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:05.215+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:47:05.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:05.215+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:47:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:06 smithi104 ceph-mon[36624]: pgmap v8922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:06 smithi081 ceph-mon[31453]: pgmap v8922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:07.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:06 smithi153 ceph-mon[36569]: pgmap v8922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:08 smithi104 ceph-mon[36624]: pgmap v8923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:08 smithi081 ceph-mon[31453]: pgmap v8923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:08 smithi153 ceph-mon[36569]: pgmap v8923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:10 smithi104 ceph-mon[36624]: pgmap v8924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:10 smithi081 ceph-mon[31453]: pgmap v8924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:10 smithi153 ceph-mon[36569]: pgmap v8924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:12 smithi104 ceph-mon[36624]: pgmap v8925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:12 smithi081 ceph-mon[31453]: pgmap v8925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:12 smithi153 ceph-mon[36569]: pgmap v8925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:14 smithi104 ceph-mon[36624]: pgmap v8926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:14 smithi081 ceph-mon[31453]: pgmap v8926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:14 smithi153 ceph-mon[36569]: pgmap v8926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:16 smithi081 ceph-mon[31453]: pgmap v8927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:16 smithi153 ceph-mon[36569]: pgmap v8927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:16 smithi104 ceph-mon[36624]: pgmap v8927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:18 smithi081 ceph-mon[31453]: pgmap v8928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:18 smithi153 ceph-mon[36569]: pgmap v8928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:18 smithi104 ceph-mon[36624]: pgmap v8928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:20.217+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:47:20.218 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:20.217+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:47:20.219 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:20.217+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:47:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:20 smithi081 ceph-mon[31453]: pgmap v8929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:20 smithi153 ceph-mon[36569]: pgmap v8929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:20 smithi104 ceph-mon[36624]: pgmap v8929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:22 smithi081 ceph-mon[31453]: pgmap v8930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:22 smithi153 ceph-mon[36569]: pgmap v8930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:22 smithi104 ceph-mon[36624]: pgmap v8930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:24 smithi081 ceph-mon[31453]: pgmap v8931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:24 smithi153 ceph-mon[36569]: pgmap v8931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:24 smithi104 ceph-mon[36624]: pgmap v8931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:26 smithi081 ceph-mon[31453]: pgmap v8932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:26 smithi153 ceph-mon[36569]: pgmap v8932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:26 smithi104 ceph-mon[36624]: pgmap v8932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:28 smithi081 ceph-mon[31453]: pgmap v8933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:28 smithi153 ceph-mon[36569]: pgmap v8933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:28 smithi104 ceph-mon[36624]: pgmap v8933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:30 smithi081 ceph-mon[31453]: pgmap v8934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:30 smithi153 ceph-mon[36569]: pgmap v8934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:30 smithi104 ceph-mon[36624]: pgmap v8934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:32 smithi081 ceph-mon[31453]: pgmap v8935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:32 smithi153 ceph-mon[36569]: pgmap v8935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:32 smithi104 ceph-mon[36624]: pgmap v8935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:34 smithi081 ceph-mon[31453]: pgmap v8936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:35.233+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:47:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:35.233+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:47:35.234 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:35.233+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:47:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:34 smithi153 ceph-mon[36569]: pgmap v8936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:34 smithi104 ceph-mon[36624]: pgmap v8936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:36 smithi081 ceph-mon[31453]: pgmap v8937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:36 smithi153 ceph-mon[36569]: pgmap v8937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:36 smithi104 ceph-mon[36624]: pgmap v8937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:38 smithi081 ceph-mon[31453]: pgmap v8938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:38 smithi153 ceph-mon[36569]: pgmap v8938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:38 smithi104 ceph-mon[36624]: pgmap v8938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:40 smithi081 ceph-mon[31453]: pgmap v8939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:40 smithi153 ceph-mon[36569]: pgmap v8939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:40 smithi104 ceph-mon[36624]: pgmap v8939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:42 smithi081 ceph-mon[31453]: pgmap v8940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:42 smithi153 ceph-mon[36569]: pgmap v8940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:42 smithi104 ceph-mon[36624]: pgmap v8940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:44 smithi153 ceph-mon[36569]: pgmap v8941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:44 smithi104 ceph-mon[36624]: pgmap v8941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:44 smithi081 ceph-mon[31453]: pgmap v8941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:46 smithi153 ceph-mon[36569]: pgmap v8942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:46 smithi104 ceph-mon[36624]: pgmap v8942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:46 smithi081 ceph-mon[31453]: pgmap v8942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:48 smithi153 ceph-mon[36569]: pgmap v8943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:48 smithi104 ceph-mon[36624]: pgmap v8943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:48 smithi081 ceph-mon[31453]: pgmap v8943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:50.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:50.248+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:47:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:50.248+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:47:50.250 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:47:50.248+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:47:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:49 smithi153 ceph-mon[36569]: pgmap v8944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:49 smithi104 ceph-mon[36624]: pgmap v8944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:49 smithi081 ceph-mon[31453]: pgmap v8944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:52 smithi081 ceph-mon[31453]: pgmap v8945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:47:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:52 smithi153 ceph-mon[36569]: pgmap v8945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:47:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:52 smithi104 ceph-mon[36624]: pgmap v8945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:47:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:47:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:47:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:53 smithi081 ceph-mon[31453]: pgmap v8946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:53 smithi081 ceph-mon[31453]: pgmap v8947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:47:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:47:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:53 smithi153 ceph-mon[36569]: pgmap v8946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:53 smithi153 ceph-mon[36569]: pgmap v8947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:47:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:47:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:53 smithi104 ceph-mon[36624]: pgmap v8946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:53 smithi104 ceph-mon[36624]: pgmap v8947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:47:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:55 smithi153 ceph-mon[36569]: pgmap v8948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:55 smithi104 ceph-mon[36624]: pgmap v8948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:55 smithi081 ceph-mon[31453]: pgmap v8948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:57 smithi153 ceph-mon[36569]: pgmap v8949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:57 smithi104 ceph-mon[36624]: pgmap v8949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:47:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:57 smithi081 ceph-mon[31453]: pgmap v8949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:47:59 smithi153 ceph-mon[36569]: pgmap v8950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:47:59 smithi104 ceph-mon[36624]: pgmap v8950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:47:59 smithi081 ceph-mon[31453]: pgmap v8950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:01 smithi153 ceph-mon[36569]: pgmap v8951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:01 smithi104 ceph-mon[36624]: pgmap v8951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:01 smithi081 ceph-mon[31453]: pgmap v8951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:03 smithi153 ceph-mon[36569]: pgmap v8952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:03 smithi104 ceph-mon[36624]: pgmap v8952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:04.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:03 smithi081 ceph-mon[31453]: pgmap v8952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:05.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:05.264+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:48:05.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:05.264+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:48:05.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:05.264+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:48:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:05 smithi153 ceph-mon[36569]: pgmap v8953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:05 smithi104 ceph-mon[36624]: pgmap v8953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:05 smithi081 ceph-mon[31453]: pgmap v8953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:07 smithi153 ceph-mon[36569]: pgmap v8954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:07 smithi104 ceph-mon[36624]: pgmap v8954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:07 smithi081 ceph-mon[31453]: pgmap v8954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:09 smithi153 ceph-mon[36569]: pgmap v8955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:09 smithi104 ceph-mon[36624]: pgmap v8955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:09 smithi081 ceph-mon[31453]: pgmap v8955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:11 smithi153 ceph-mon[36569]: pgmap v8956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:11 smithi104 ceph-mon[36624]: pgmap v8956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:11 smithi081 ceph-mon[31453]: pgmap v8956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:14.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:13 smithi153 ceph-mon[36569]: pgmap v8957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:13 smithi104 ceph-mon[36624]: pgmap v8957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:13 smithi081 ceph-mon[31453]: pgmap v8957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:15 smithi153 ceph-mon[36569]: pgmap v8958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:15 smithi104 ceph-mon[36624]: pgmap v8958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:15 smithi081 ceph-mon[31453]: pgmap v8958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:17 smithi153 ceph-mon[36569]: pgmap v8959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:17 smithi104 ceph-mon[36624]: pgmap v8959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:17 smithi081 ceph-mon[31453]: pgmap v8959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:19 smithi153 ceph-mon[36569]: pgmap v8960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:19 smithi104 ceph-mon[36624]: pgmap v8960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:19 smithi081 ceph-mon[31453]: pgmap v8960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:20.279+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:48:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:20.279+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:48:20.281 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:20.279+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:48:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:21 smithi153 ceph-mon[36569]: pgmap v8961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:21 smithi104 ceph-mon[36624]: pgmap v8961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:21 smithi081 ceph-mon[31453]: pgmap v8961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:23 smithi153 ceph-mon[36569]: pgmap v8962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:23 smithi104 ceph-mon[36624]: pgmap v8962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:23 smithi081 ceph-mon[31453]: pgmap v8962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:25 smithi104 ceph-mon[36624]: pgmap v8963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:25 smithi081 ceph-mon[31453]: pgmap v8963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:25 smithi153 ceph-mon[36569]: pgmap v8963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:27 smithi104 ceph-mon[36624]: pgmap v8964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:27 smithi081 ceph-mon[31453]: pgmap v8964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:27 smithi153 ceph-mon[36569]: pgmap v8964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:29 smithi104 ceph-mon[36624]: pgmap v8965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:29 smithi081 ceph-mon[31453]: pgmap v8965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:29 smithi153 ceph-mon[36569]: pgmap v8965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:30.400+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:48:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:30.400+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:48:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:30.400+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:48:30.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:30.400+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:48:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:31 smithi104 ceph-mon[36624]: pgmap v8966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:31 smithi081 ceph-mon[31453]: pgmap v8966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:31 smithi153 ceph-mon[36569]: pgmap v8966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:33 smithi104 ceph-mon[36624]: pgmap v8967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:34.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:33 smithi153 ceph-mon[36569]: pgmap v8967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:33 smithi081 ceph-mon[31453]: pgmap v8967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:35.284+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:48:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:35.284+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:48:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:35.284+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:48:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:35 smithi081 ceph-mon[31453]: pgmap v8968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:35 smithi153 ceph-mon[36569]: pgmap v8968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:35 smithi104 ceph-mon[36624]: pgmap v8968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:37 smithi081 ceph-mon[31453]: pgmap v8969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:37 smithi153 ceph-mon[36569]: pgmap v8969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:37 smithi104 ceph-mon[36624]: pgmap v8969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:39 smithi081 ceph-mon[31453]: pgmap v8970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:39 smithi153 ceph-mon[36569]: pgmap v8970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:39 smithi104 ceph-mon[36624]: pgmap v8970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:41 smithi081 ceph-mon[31453]: pgmap v8971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:41 smithi153 ceph-mon[36569]: pgmap v8971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:41 smithi104 ceph-mon[36624]: pgmap v8971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:44.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:43 smithi104 ceph-mon[36624]: pgmap v8972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:43 smithi081 ceph-mon[31453]: pgmap v8972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:43 smithi153 ceph-mon[36569]: pgmap v8972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:45 smithi081 ceph-mon[31453]: pgmap v8973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:45 smithi153 ceph-mon[36569]: pgmap v8973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:45 smithi104 ceph-mon[36624]: pgmap v8973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:47 smithi081 ceph-mon[31453]: pgmap v8974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:47 smithi153 ceph-mon[36569]: pgmap v8974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:47 smithi104 ceph-mon[36624]: pgmap v8974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:49 smithi081 ceph-mon[31453]: pgmap v8975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:50.300+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:48:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:50.300+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:48:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:48:50.300+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:48:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:49 smithi153 ceph-mon[36569]: pgmap v8975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:49 smithi104 ceph-mon[36624]: pgmap v8975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:51 smithi081 ceph-mon[31453]: pgmap v8976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:51 smithi153 ceph-mon[36569]: pgmap v8976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:51 smithi104 ceph-mon[36624]: pgmap v8976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:53 smithi081 ceph-mon[31453]: pgmap v8977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:48:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:53 smithi153 ceph-mon[36569]: pgmap v8977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:48:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:53 smithi104 ceph-mon[36624]: pgmap v8977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:48:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:48:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:48:55.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:48:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:48:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:48:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:48:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:48:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:48:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:48:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:55 smithi081 ceph-mon[31453]: pgmap v8978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:55 smithi081 ceph-mon[31453]: pgmap v8979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:55 smithi153 ceph-mon[36569]: pgmap v8978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:55 smithi153 ceph-mon[36569]: pgmap v8979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:55 smithi104 ceph-mon[36624]: pgmap v8978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:55 smithi104 ceph-mon[36624]: pgmap v8979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:57 smithi081 ceph-mon[31453]: pgmap v8980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:57 smithi153 ceph-mon[36569]: pgmap v8980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:48:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:57 smithi104 ceph-mon[36624]: pgmap v8980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:48:59 smithi081 ceph-mon[31453]: pgmap v8981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:48:59 smithi153 ceph-mon[36569]: pgmap v8981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:48:59 smithi104 ceph-mon[36624]: pgmap v8981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:01 smithi153 ceph-mon[36569]: pgmap v8982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:01 smithi104 ceph-mon[36624]: pgmap v8982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:01 smithi081 ceph-mon[31453]: pgmap v8982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:03 smithi153 ceph-mon[36569]: pgmap v8983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:03 smithi104 ceph-mon[36624]: pgmap v8983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:03 smithi081 ceph-mon[31453]: pgmap v8983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:05.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:05.307+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:49:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:05.307+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:49:05.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:05.307+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:49:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:05 smithi153 ceph-mon[36569]: pgmap v8984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:05 smithi104 ceph-mon[36624]: pgmap v8984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:05 smithi081 ceph-mon[31453]: pgmap v8984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:07 smithi153 ceph-mon[36569]: pgmap v8985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:07 smithi104 ceph-mon[36624]: pgmap v8985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:07 smithi081 ceph-mon[31453]: pgmap v8985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:09 smithi153 ceph-mon[36569]: pgmap v8986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:09 smithi104 ceph-mon[36624]: pgmap v8986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:09 smithi081 ceph-mon[31453]: pgmap v8986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:11 smithi153 ceph-mon[36569]: pgmap v8987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:11 smithi104 ceph-mon[36624]: pgmap v8987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:11 smithi081 ceph-mon[31453]: pgmap v8987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:12 smithi153 ceph-mon[36569]: pgmap v8988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:12 smithi104 ceph-mon[36624]: pgmap v8988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:12 smithi081 ceph-mon[31453]: pgmap v8988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:15.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:15.408+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:49:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:15.408+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:49:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:15.408+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:49:15.409 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:15.408+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:49:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:15 smithi153 ceph-mon[36569]: pgmap v8989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:15 smithi104 ceph-mon[36624]: pgmap v8989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:15 smithi081 ceph-mon[31453]: pgmap v8989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:17 smithi153 ceph-mon[36569]: pgmap v8990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:17 smithi104 ceph-mon[36624]: pgmap v8990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:17 smithi081 ceph-mon[31453]: pgmap v8990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:19 smithi153 ceph-mon[36569]: pgmap v8991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:19 smithi104 ceph-mon[36624]: pgmap v8991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:19 smithi081 ceph-mon[31453]: pgmap v8991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:20.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:20.312+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:49:20.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:20.312+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:49:20.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:20.312+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:49:21.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:21 smithi153 ceph-mon[36569]: pgmap v8992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:21 smithi104 ceph-mon[36624]: pgmap v8992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:21 smithi081 ceph-mon[31453]: pgmap v8992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:23 smithi153 ceph-mon[36569]: pgmap v8993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:23 smithi104 ceph-mon[36624]: pgmap v8993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:23 smithi081 ceph-mon[31453]: pgmap v8993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:25 smithi153 ceph-mon[36569]: pgmap v8994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:25 smithi104 ceph-mon[36624]: pgmap v8994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:25 smithi081 ceph-mon[31453]: pgmap v8994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:27.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:27 smithi153 ceph-mon[36569]: pgmap v8995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:27 smithi104 ceph-mon[36624]: pgmap v8995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:27 smithi081 ceph-mon[31453]: pgmap v8995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:29 smithi153 ceph-mon[36569]: pgmap v8996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:29 smithi104 ceph-mon[36624]: pgmap v8996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:29 smithi081 ceph-mon[31453]: pgmap v8996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:31 smithi153 ceph-mon[36569]: pgmap v8997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:31 smithi104 ceph-mon[36624]: pgmap v8997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:31 smithi081 ceph-mon[31453]: pgmap v8997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:33 smithi153 ceph-mon[36569]: pgmap v8998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:33 smithi104 ceph-mon[36624]: pgmap v8998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:33 smithi081 ceph-mon[31453]: pgmap v8998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:35.325+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:49:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:35.325+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:49:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:35.325+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:49:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:35 smithi153 ceph-mon[36569]: pgmap v8999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:35 smithi104 ceph-mon[36624]: pgmap v8999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:35 smithi081 ceph-mon[31453]: pgmap v8999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:37.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:37 smithi153 ceph-mon[36569]: pgmap v9000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:37 smithi104 ceph-mon[36624]: pgmap v9000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:37 smithi081 ceph-mon[31453]: pgmap v9000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:39.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:39 smithi153 ceph-mon[36569]: pgmap v9001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:39 smithi104 ceph-mon[36624]: pgmap v9001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:39 smithi081 ceph-mon[31453]: pgmap v9001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:41 smithi153 ceph-mon[36569]: pgmap v9002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:41 smithi104 ceph-mon[36624]: pgmap v9002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:41 smithi081 ceph-mon[31453]: pgmap v9002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:43 smithi104 ceph-mon[36624]: pgmap v9003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:43 smithi081 ceph-mon[31453]: pgmap v9003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:43 smithi153 ceph-mon[36569]: pgmap v9003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:45 smithi104 ceph-mon[36624]: pgmap v9004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:45 smithi081 ceph-mon[31453]: pgmap v9004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:45 smithi153 ceph-mon[36569]: pgmap v9004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:47 smithi104 ceph-mon[36624]: pgmap v9005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:47 smithi081 ceph-mon[31453]: pgmap v9005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:47 smithi153 ceph-mon[36569]: pgmap v9005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:49 smithi104 ceph-mon[36624]: pgmap v9006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:49 smithi081 ceph-mon[31453]: pgmap v9006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:49 smithi153 ceph-mon[36569]: pgmap v9006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:50.331+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:49:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:50.331+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:49:50.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:49:50.331+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:49:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:51 smithi104 ceph-mon[36624]: pgmap v9007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:51 smithi081 ceph-mon[31453]: pgmap v9007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:51 smithi153 ceph-mon[36569]: pgmap v9007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:53.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:53 smithi081 ceph-mon[31453]: pgmap v9008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:53 smithi153 ceph-mon[36569]: pgmap v9008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:53 smithi104 ceph-mon[36624]: pgmap v9008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:55 smithi081 ceph-mon[31453]: pgmap v9009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:49:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:55 smithi153 ceph-mon[36569]: pgmap v9009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:49:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:55 smithi104 ceph-mon[36624]: pgmap v9009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:49:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:49:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:49:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:56 smithi081 ceph-mon[31453]: pgmap v9010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:56 smithi081 ceph-mon[31453]: pgmap v9011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:49:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:49:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:49:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:56 smithi153 ceph-mon[36569]: pgmap v9010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:56 smithi153 ceph-mon[36569]: pgmap v9011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:49:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:49:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:49:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:56 smithi104 ceph-mon[36624]: pgmap v9010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:56 smithi104 ceph-mon[36624]: pgmap v9011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:49:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:49:58 smithi153 ceph-mon[36569]: pgmap v9012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:49:58 smithi104 ceph-mon[36624]: pgmap v9012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:49:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:49:58 smithi081 ceph-mon[31453]: pgmap v9012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:00 smithi104 ceph-mon[36624]: pgmap v9013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:50:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:00 smithi081 ceph-mon[31453]: pgmap v9013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:50:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:00 smithi153 ceph-mon[36569]: pgmap v9013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T01:50:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:02 smithi104 ceph-mon[36624]: pgmap v9014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:02 smithi081 ceph-mon[31453]: pgmap v9014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:02 smithi153 ceph-mon[36569]: pgmap v9014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:04 smithi104 ceph-mon[36624]: pgmap v9015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:05.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:04 smithi153 ceph-mon[36569]: pgmap v9015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:04 smithi081 ceph-mon[31453]: pgmap v9015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:05.342+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:50:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:05.342+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:50:05.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:05.342+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:50:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:06 smithi104 ceph-mon[36624]: pgmap v9016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:06 smithi081 ceph-mon[31453]: pgmap v9016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:06 smithi153 ceph-mon[36569]: pgmap v9016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:08 smithi104 ceph-mon[36624]: pgmap v9017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:08 smithi081 ceph-mon[31453]: pgmap v9017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:08 smithi153 ceph-mon[36569]: pgmap v9017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:10 smithi081 ceph-mon[31453]: pgmap v9018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:11.173 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:10 smithi104 ceph-mon[36624]: pgmap v9018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:10 smithi153 ceph-mon[36569]: pgmap v9018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:12 smithi081 ceph-mon[31453]: pgmap v9019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:12 smithi153 ceph-mon[36569]: pgmap v9019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:12 smithi104 ceph-mon[36624]: pgmap v9019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:15.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:14 smithi104 ceph-mon[36624]: pgmap v9020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:14 smithi081 ceph-mon[31453]: pgmap v9020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:14 smithi153 ceph-mon[36569]: pgmap v9020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:16 smithi081 ceph-mon[31453]: pgmap v9021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:16 smithi153 ceph-mon[36569]: pgmap v9021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:16 smithi104 ceph-mon[36624]: pgmap v9021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:18 smithi081 ceph-mon[31453]: pgmap v9022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:18 smithi153 ceph-mon[36569]: pgmap v9022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:18 smithi104 ceph-mon[36624]: pgmap v9022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:20.357+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:50:20.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:20.357+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:50:20.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:20.357+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:50:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:20 smithi081 ceph-mon[31453]: pgmap v9023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:20 smithi153 ceph-mon[36569]: pgmap v9023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:20 smithi104 ceph-mon[36624]: pgmap v9023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:22 smithi081 ceph-mon[31453]: pgmap v9024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:22 smithi153 ceph-mon[36569]: pgmap v9024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:22 smithi104 ceph-mon[36624]: pgmap v9024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:24.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:24 smithi081 ceph-mon[31453]: pgmap v9025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:24.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:24 smithi153 ceph-mon[36569]: pgmap v9025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:24 smithi104 ceph-mon[36624]: pgmap v9025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:26 smithi104 ceph-mon[36624]: pgmap v9026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:26 smithi081 ceph-mon[31453]: pgmap v9026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:26 smithi153 ceph-mon[36569]: pgmap v9026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:28 smithi104 ceph-mon[36624]: pgmap v9027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:28 smithi081 ceph-mon[31453]: pgmap v9027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:28 smithi153 ceph-mon[36569]: pgmap v9027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:30 smithi104 ceph-mon[36624]: pgmap v9028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:30 smithi081 ceph-mon[31453]: pgmap v9028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:30 smithi153 ceph-mon[36569]: pgmap v9028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:32 smithi081 ceph-mon[31453]: pgmap v9029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:32 smithi153 ceph-mon[36569]: pgmap v9029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:32 smithi104 ceph-mon[36624]: pgmap v9029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:35.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:34 smithi153 ceph-mon[36569]: pgmap v9030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:34 smithi081 ceph-mon[31453]: pgmap v9030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:34 smithi104 ceph-mon[36624]: pgmap v9030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:35.373+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:50:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:35.373+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:50:35.375 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:35.373+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:50:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:36 smithi081 ceph-mon[31453]: pgmap v9031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:36 smithi153 ceph-mon[36569]: pgmap v9031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:36 smithi104 ceph-mon[36624]: pgmap v9031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:38 smithi081 ceph-mon[31453]: pgmap v9032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:38 smithi153 ceph-mon[36569]: pgmap v9032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:38 smithi104 ceph-mon[36624]: pgmap v9032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:40 smithi081 ceph-mon[31453]: pgmap v9033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:40 smithi153 ceph-mon[36569]: pgmap v9033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:40 smithi104 ceph-mon[36624]: pgmap v9033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:42 smithi081 ceph-mon[31453]: pgmap v9034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:42 smithi153 ceph-mon[36569]: pgmap v9034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:42 smithi104 ceph-mon[36624]: pgmap v9034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:45.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:44 smithi104 ceph-mon[36624]: pgmap v9035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:44 smithi081 ceph-mon[31453]: pgmap v9035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:44 smithi153 ceph-mon[36569]: pgmap v9035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:46 smithi081 ceph-mon[31453]: pgmap v9036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:46 smithi153 ceph-mon[36569]: pgmap v9036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:46 smithi104 ceph-mon[36624]: pgmap v9036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:48 smithi081 ceph-mon[31453]: pgmap v9037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:48 smithi153 ceph-mon[36569]: pgmap v9037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:48 smithi104 ceph-mon[36624]: pgmap v9037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:50.388+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:50:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:50.388+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:50:50.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:50:50.388+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:50:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:50 smithi081 ceph-mon[31453]: pgmap v9038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:50 smithi153 ceph-mon[36569]: pgmap v9038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:50 smithi104 ceph-mon[36624]: pgmap v9038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:52 smithi081 ceph-mon[31453]: pgmap v9039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:52 smithi153 ceph-mon[36569]: pgmap v9039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:52 smithi104 ceph-mon[36624]: pgmap v9039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:54 smithi081 ceph-mon[31453]: pgmap v9040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:54 smithi153 ceph-mon[36569]: pgmap v9040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:54 smithi104 ceph-mon[36624]: pgmap v9040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:56 smithi081 ceph-mon[31453]: pgmap v9041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:50:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:56 smithi153 ceph-mon[36569]: pgmap v9041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:50:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:56 smithi104 ceph-mon[36624]: pgmap v9041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:50:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:50:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:50:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:50:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:50:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:50:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:50:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:50:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:50:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:50:59.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:58 smithi081 ceph-mon[31453]: pgmap v9042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:59.205 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:50:58 smithi081 ceph-mon[31453]: pgmap v9043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:58 smithi153 ceph-mon[36569]: pgmap v9042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:50:58 smithi153 ceph-mon[36569]: pgmap v9043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:58 smithi104 ceph-mon[36624]: pgmap v9042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:50:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:50:58 smithi104 ceph-mon[36624]: pgmap v9043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:00 smithi153 ceph-mon[36569]: pgmap v9044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:00 smithi104 ceph-mon[36624]: pgmap v9044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:00 smithi081 ceph-mon[31453]: pgmap v9044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:02 smithi153 ceph-mon[36569]: pgmap v9045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:02 smithi104 ceph-mon[36624]: pgmap v9045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:02 smithi081 ceph-mon[31453]: pgmap v9045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:04 smithi153 ceph-mon[36569]: pgmap v9046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:04 smithi104 ceph-mon[36624]: pgmap v9046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:05.403+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:51:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:05.403+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:51:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:05.403+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:51:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:04 smithi081 ceph-mon[31453]: pgmap v9046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:06 smithi153 ceph-mon[36569]: pgmap v9047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:06 smithi104 ceph-mon[36624]: pgmap v9047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:06 smithi081 ceph-mon[31453]: pgmap v9047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:08 smithi153 ceph-mon[36569]: pgmap v9048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:08 smithi104 ceph-mon[36624]: pgmap v9048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:08 smithi081 ceph-mon[31453]: pgmap v9048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:10 smithi153 ceph-mon[36569]: pgmap v9049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:10 smithi104 ceph-mon[36624]: pgmap v9049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:10 smithi081 ceph-mon[31453]: pgmap v9049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:12 smithi153 ceph-mon[36569]: pgmap v9050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:12 smithi104 ceph-mon[36624]: pgmap v9050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:12 smithi081 ceph-mon[31453]: pgmap v9050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:14 smithi153 ceph-mon[36569]: pgmap v9051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:14 smithi104 ceph-mon[36624]: pgmap v9051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:14 smithi081 ceph-mon[31453]: pgmap v9051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:16 smithi153 ceph-mon[36569]: pgmap v9052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:16 smithi104 ceph-mon[36624]: pgmap v9052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:16 smithi081 ceph-mon[31453]: pgmap v9052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:18 smithi153 ceph-mon[36569]: pgmap v9053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:18 smithi104 ceph-mon[36624]: pgmap v9053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:18 smithi081 ceph-mon[31453]: pgmap v9053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:20.420 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:20.418+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:51:20.420 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:20.418+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:51:20.420 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:20.418+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:51:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:20 smithi081 ceph-mon[31453]: pgmap v9054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:20.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:20 smithi153 ceph-mon[36569]: pgmap v9054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:20 smithi104 ceph-mon[36624]: pgmap v9054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:22.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:22 smithi081 ceph-mon[31453]: pgmap v9055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:22.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:22 smithi153 ceph-mon[36569]: pgmap v9055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:22 smithi104 ceph-mon[36624]: pgmap v9055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:24.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:24 smithi081 ceph-mon[31453]: pgmap v9056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:24.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:24 smithi153 ceph-mon[36569]: pgmap v9056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:24 smithi104 ceph-mon[36624]: pgmap v9056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:26 smithi104 ceph-mon[36624]: pgmap v9057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:26 smithi081 ceph-mon[31453]: pgmap v9057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:26.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:26 smithi153 ceph-mon[36569]: pgmap v9057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:28 smithi153 ceph-mon[36569]: pgmap v9058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:28 smithi104 ceph-mon[36624]: pgmap v9058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:28 smithi081 ceph-mon[31453]: pgmap v9058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:30.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:30 smithi153 ceph-mon[36569]: pgmap v9059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:30 smithi104 ceph-mon[36624]: pgmap v9059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:30 smithi081 ceph-mon[31453]: pgmap v9059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:32.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:32 smithi153 ceph-mon[36569]: pgmap v9060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:32 smithi104 ceph-mon[36624]: pgmap v9060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:32 smithi081 ceph-mon[31453]: pgmap v9060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:34.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:34 smithi153 ceph-mon[36569]: pgmap v9061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:34 smithi104 ceph-mon[36624]: pgmap v9061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:34 smithi081 ceph-mon[31453]: pgmap v9061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:35.426+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:51:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:35.426+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:51:35.428 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:35.426+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:51:36.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:36 smithi153 ceph-mon[36569]: pgmap v9062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:36 smithi104 ceph-mon[36624]: pgmap v9062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:36 smithi081 ceph-mon[31453]: pgmap v9062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:38 smithi153 ceph-mon[36569]: pgmap v9063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:38 smithi104 ceph-mon[36624]: pgmap v9063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:38 smithi081 ceph-mon[31453]: pgmap v9063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:40.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:40 smithi153 ceph-mon[36569]: pgmap v9064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:40 smithi104 ceph-mon[36624]: pgmap v9064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:40 smithi081 ceph-mon[31453]: pgmap v9064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:42 smithi081 ceph-mon[31453]: pgmap v9065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:42 smithi153 ceph-mon[36569]: pgmap v9065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:42 smithi104 ceph-mon[36624]: pgmap v9065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:44 smithi153 ceph-mon[36569]: pgmap v9066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:44 smithi104 ceph-mon[36624]: pgmap v9066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:44 smithi081 ceph-mon[31453]: pgmap v9066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:46 smithi153 ceph-mon[36569]: pgmap v9067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:46 smithi104 ceph-mon[36624]: pgmap v9067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:46 smithi081 ceph-mon[31453]: pgmap v9067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:48 smithi153 ceph-mon[36569]: pgmap v9068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:48 smithi104 ceph-mon[36624]: pgmap v9068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:48 smithi081 ceph-mon[31453]: pgmap v9068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:50.439+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:51:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:50.439+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:51:50.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:51:50.439+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:51:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:50 smithi153 ceph-mon[36569]: pgmap v9069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:50.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:50 smithi104 ceph-mon[36624]: pgmap v9069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:50 smithi081 ceph-mon[31453]: pgmap v9069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:52 smithi153 ceph-mon[36569]: pgmap v9070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:52 smithi104 ceph-mon[36624]: pgmap v9070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:52 smithi081 ceph-mon[31453]: pgmap v9070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:54 smithi153 ceph-mon[36569]: pgmap v9071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:54 smithi104 ceph-mon[36624]: pgmap v9071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:54 smithi081 ceph-mon[31453]: pgmap v9071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:56 smithi153 ceph-mon[36569]: pgmap v9072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:56 smithi104 ceph-mon[36624]: pgmap v9072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:56 smithi081 ceph-mon[31453]: pgmap v9072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:58.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:58 smithi153 ceph-mon[36569]: pgmap v9073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:58.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:51:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:58 smithi104 ceph-mon[36624]: pgmap v9073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:51:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:58 smithi081 ceph-mon[31453]: pgmap v9073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:51:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:51:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:51:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:59 smithi153 ceph-mon[36569]: pgmap v9074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:59 smithi153 ceph-mon[36569]: pgmap v9075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:51:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:51:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:51:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:51:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:59 smithi104 ceph-mon[36624]: pgmap v9074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:59 smithi104 ceph-mon[36624]: pgmap v9075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:51:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:51:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:51:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:51:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:59 smithi081 ceph-mon[31453]: pgmap v9074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:59 smithi081 ceph-mon[31453]: pgmap v9075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:51:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:51:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:52:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:01 smithi153 ceph-mon[36569]: pgmap v9076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:01 smithi104 ceph-mon[36624]: pgmap v9076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:01 smithi081 ceph-mon[31453]: pgmap v9076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:03 smithi153 ceph-mon[36569]: pgmap v9077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:03 smithi104 ceph-mon[36624]: pgmap v9077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:03 smithi081 ceph-mon[31453]: pgmap v9077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:05 smithi153 ceph-mon[36569]: pgmap v9078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:05 smithi104 ceph-mon[36624]: pgmap v9078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:05 smithi081 ceph-mon[31453]: pgmap v9078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:05.451+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:52:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:05.451+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:52:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:05.451+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:52:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:07 smithi153 ceph-mon[36569]: pgmap v9079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:07 smithi104 ceph-mon[36624]: pgmap v9079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:07 smithi081 ceph-mon[31453]: pgmap v9079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:09 smithi153 ceph-mon[36569]: pgmap v9080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:09 smithi104 ceph-mon[36624]: pgmap v9080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:09 smithi081 ceph-mon[31453]: pgmap v9080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:11 smithi153 ceph-mon[36569]: pgmap v9081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:11 smithi104 ceph-mon[36624]: pgmap v9081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:11 smithi081 ceph-mon[31453]: pgmap v9081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:13 smithi153 ceph-mon[36569]: pgmap v9082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:13 smithi104 ceph-mon[36624]: pgmap v9082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:13 smithi081 ceph-mon[31453]: pgmap v9082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:15 smithi153 ceph-mon[36569]: pgmap v9083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:15 smithi104 ceph-mon[36624]: pgmap v9083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:15 smithi081 ceph-mon[31453]: pgmap v9083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:15.439 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:15.437+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:52:15.439 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:15.437+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:52:15.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:15.438+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:52:15.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:15.438+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:52:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:17 smithi153 ceph-mon[36569]: pgmap v9084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:17 smithi104 ceph-mon[36624]: pgmap v9084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:17 smithi081 ceph-mon[31453]: pgmap v9084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:19.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:19 smithi153 ceph-mon[36569]: pgmap v9085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:19 smithi104 ceph-mon[36624]: pgmap v9085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:19 smithi081 ceph-mon[31453]: pgmap v9085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:20.466+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:52:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:20.466+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:52:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:20.466+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:52:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:21 smithi153 ceph-mon[36569]: pgmap v9086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:21 smithi104 ceph-mon[36624]: pgmap v9086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:21 smithi081 ceph-mon[31453]: pgmap v9086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:23 smithi153 ceph-mon[36569]: pgmap v9087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:23 smithi104 ceph-mon[36624]: pgmap v9087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:23 smithi081 ceph-mon[31453]: pgmap v9087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:25 smithi153 ceph-mon[36569]: pgmap v9088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:25 smithi104 ceph-mon[36624]: pgmap v9088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:25 smithi081 ceph-mon[31453]: pgmap v9088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:27 smithi153 ceph-mon[36569]: pgmap v9089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:27 smithi104 ceph-mon[36624]: pgmap v9089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:27 smithi081 ceph-mon[31453]: pgmap v9089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:29 smithi153 ceph-mon[36569]: pgmap v9090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:29.320 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:29 smithi081 ceph-mon[31453]: pgmap v9090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:29 smithi104 ceph-mon[36624]: pgmap v9090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:31 smithi153 ceph-mon[36569]: pgmap v9091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:31 smithi104 ceph-mon[36624]: pgmap v9091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:31 smithi081 ceph-mon[31453]: pgmap v9091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:33 smithi153 ceph-mon[36569]: pgmap v9092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:33 smithi104 ceph-mon[36624]: pgmap v9092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:33 smithi081 ceph-mon[31453]: pgmap v9092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:35 smithi153 ceph-mon[36569]: pgmap v9093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:35 smithi104 ceph-mon[36624]: pgmap v9093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:35 smithi081 ceph-mon[31453]: pgmap v9093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:35.469+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:52:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:35.469+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:52:35.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:35.469+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:52:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:37 smithi153 ceph-mon[36569]: pgmap v9094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:37 smithi104 ceph-mon[36624]: pgmap v9094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:37 smithi081 ceph-mon[31453]: pgmap v9094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:39 smithi153 ceph-mon[36569]: pgmap v9095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:39 smithi104 ceph-mon[36624]: pgmap v9095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:39 smithi081 ceph-mon[31453]: pgmap v9095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:41 smithi153 ceph-mon[36569]: pgmap v9096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:41 smithi104 ceph-mon[36624]: pgmap v9096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:41 smithi081 ceph-mon[31453]: pgmap v9096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:43 smithi153 ceph-mon[36569]: pgmap v9097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:43 smithi104 ceph-mon[36624]: pgmap v9097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:43 smithi081 ceph-mon[31453]: pgmap v9097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:45 smithi153 ceph-mon[36569]: pgmap v9098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:45 smithi104 ceph-mon[36624]: pgmap v9098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:45 smithi081 ceph-mon[31453]: pgmap v9098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:47 smithi153 ceph-mon[36569]: pgmap v9099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:47 smithi104 ceph-mon[36624]: pgmap v9099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:47 smithi081 ceph-mon[31453]: pgmap v9099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:49 smithi153 ceph-mon[36569]: pgmap v9100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:49 smithi104 ceph-mon[36624]: pgmap v9100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:49 smithi081 ceph-mon[31453]: pgmap v9100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:50.478+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:52:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:50.478+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:52:50.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:52:50.478+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:52:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:51 smithi153 ceph-mon[36569]: pgmap v9101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:51 smithi104 ceph-mon[36624]: pgmap v9101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:51 smithi081 ceph-mon[31453]: pgmap v9101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:53 smithi153 ceph-mon[36569]: pgmap v9102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:53 smithi104 ceph-mon[36624]: pgmap v9102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:53 smithi081 ceph-mon[31453]: pgmap v9102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:55 smithi153 ceph-mon[36569]: pgmap v9103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:55 smithi104 ceph-mon[36624]: pgmap v9103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:55 smithi081 ceph-mon[31453]: pgmap v9103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:57 smithi153 ceph-mon[36569]: pgmap v9104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:57 smithi104 ceph-mon[36624]: pgmap v9104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:57 smithi081 ceph-mon[31453]: pgmap v9104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:52:59 smithi153 ceph-mon[36569]: pgmap v9105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:52:59 smithi104 ceph-mon[36624]: pgmap v9105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:52:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:52:59 smithi081 ceph-mon[31453]: pgmap v9105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:53:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:53:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:53:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:53:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: pgmap v9106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:01.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: pgmap v9107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:01.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:53:01.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:53:01.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:53:01.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:53:01.313 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: pgmap v9106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: pgmap v9107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:01.314 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:53:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T01:53:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T01:53:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T01:53:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T01:53:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: pgmap v9106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: pgmap v9107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:53:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:03 smithi081 ceph-mon[31453]: pgmap v9108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:03 smithi153 ceph-mon[36569]: pgmap v9108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:03 smithi104 ceph-mon[36624]: pgmap v9108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:05.488+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:53:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:05.488+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:53:05.490 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:05.488+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:53:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:05 smithi081 ceph-mon[31453]: pgmap v9109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:05 smithi153 ceph-mon[36569]: pgmap v9109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:05 smithi104 ceph-mon[36624]: pgmap v9109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:07 smithi081 ceph-mon[31453]: pgmap v9110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:07 smithi153 ceph-mon[36569]: pgmap v9110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:07 smithi104 ceph-mon[36624]: pgmap v9110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:09 smithi081 ceph-mon[31453]: pgmap v9111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:09 smithi153 ceph-mon[36569]: pgmap v9111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:09 smithi104 ceph-mon[36624]: pgmap v9111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:11 smithi081 ceph-mon[31453]: pgmap v9112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:11 smithi153 ceph-mon[36569]: pgmap v9112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:11 smithi104 ceph-mon[36624]: pgmap v9112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:13 smithi081 ceph-mon[31453]: pgmap v9113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:14.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:13 smithi153 ceph-mon[36569]: pgmap v9113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:13 smithi104 ceph-mon[36624]: pgmap v9113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:15 smithi081 ceph-mon[31453]: pgmap v9114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:15 smithi153 ceph-mon[36569]: pgmap v9114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:15 smithi104 ceph-mon[36624]: pgmap v9114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:17 smithi153 ceph-mon[36569]: pgmap v9115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:17 smithi104 ceph-mon[36624]: pgmap v9115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:17 smithi081 ceph-mon[31453]: pgmap v9115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:19.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:19 smithi081 ceph-mon[31453]: pgmap v9116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:19 smithi153 ceph-mon[36569]: pgmap v9116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:19 smithi104 ceph-mon[36624]: pgmap v9116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:20.502+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:53:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:20.502+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:53:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:20.502+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:53:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:21 smithi153 ceph-mon[36569]: pgmap v9117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:21 smithi104 ceph-mon[36624]: pgmap v9117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:21 smithi081 ceph-mon[31453]: pgmap v9117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:23 smithi153 ceph-mon[36569]: pgmap v9118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:23 smithi104 ceph-mon[36624]: pgmap v9118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:23 smithi081 ceph-mon[31453]: pgmap v9118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:25 smithi153 ceph-mon[36569]: pgmap v9119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:25 smithi104 ceph-mon[36624]: pgmap v9119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:25 smithi081 ceph-mon[31453]: pgmap v9119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:27 smithi153 ceph-mon[36569]: pgmap v9120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:27 smithi104 ceph-mon[36624]: pgmap v9120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:27 smithi081 ceph-mon[31453]: pgmap v9120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:30.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:29 smithi153 ceph-mon[36569]: pgmap v9121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:29 smithi104 ceph-mon[36624]: pgmap v9121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:29 smithi081 ceph-mon[31453]: pgmap v9121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:31 smithi153 ceph-mon[36569]: pgmap v9122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:31 smithi104 ceph-mon[36624]: pgmap v9122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:31 smithi081 ceph-mon[31453]: pgmap v9122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:33 smithi153 ceph-mon[36569]: pgmap v9123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:33 smithi104 ceph-mon[36624]: pgmap v9123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:33 smithi081 ceph-mon[31453]: pgmap v9123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:35.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:35.517+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:53:35.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:35.517+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:53:35.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:35.517+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:53:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:35 smithi153 ceph-mon[36569]: pgmap v9124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:35 smithi104 ceph-mon[36624]: pgmap v9124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:35 smithi081 ceph-mon[31453]: pgmap v9124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:37 smithi153 ceph-mon[36569]: pgmap v9125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:37 smithi104 ceph-mon[36624]: pgmap v9125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:37 smithi081 ceph-mon[31453]: pgmap v9125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:39 smithi153 ceph-mon[36569]: pgmap v9126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:39 smithi104 ceph-mon[36624]: pgmap v9126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:39 smithi081 ceph-mon[31453]: pgmap v9126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:41 smithi153 ceph-mon[36569]: pgmap v9127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:41 smithi104 ceph-mon[36624]: pgmap v9127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:42.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:41 smithi081 ceph-mon[31453]: pgmap v9127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:44.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:43 smithi153 ceph-mon[36569]: pgmap v9128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:44.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:43 smithi104 ceph-mon[36624]: pgmap v9128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:43 smithi081 ceph-mon[31453]: pgmap v9128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:45 smithi153 ceph-mon[36569]: pgmap v9129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:46.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:45 smithi104 ceph-mon[36624]: pgmap v9129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:46.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:45 smithi081 ceph-mon[31453]: pgmap v9129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:47 smithi153 ceph-mon[36569]: pgmap v9130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:47 smithi104 ceph-mon[36624]: pgmap v9130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:47 smithi081 ceph-mon[31453]: pgmap v9130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:49 smithi153 ceph-mon[36569]: pgmap v9131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:49 smithi104 ceph-mon[36624]: pgmap v9131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:49 smithi081 ceph-mon[31453]: pgmap v9131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:50.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:50.531+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:53:50.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:50.531+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:53:50.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:53:50.531+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:53:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:51 smithi153 ceph-mon[36569]: pgmap v9132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:51 smithi104 ceph-mon[36624]: pgmap v9132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:51 smithi081 ceph-mon[31453]: pgmap v9132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:53 smithi153 ceph-mon[36569]: pgmap v9133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:53 smithi104 ceph-mon[36624]: pgmap v9133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:53 smithi081 ceph-mon[31453]: pgmap v9133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:55 smithi153 ceph-mon[36569]: pgmap v9134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:55 smithi104 ceph-mon[36624]: pgmap v9134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:55 smithi081 ceph-mon[31453]: pgmap v9134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:57 smithi153 ceph-mon[36569]: pgmap v9135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:57 smithi104 ceph-mon[36624]: pgmap v9135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:53:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:57 smithi081 ceph-mon[31453]: pgmap v9135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:53:59 smithi153 ceph-mon[36569]: pgmap v9136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:53:59 smithi104 ceph-mon[36624]: pgmap v9136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:53:59 smithi081 ceph-mon[31453]: pgmap v9136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:01 smithi153 ceph-mon[36569]: pgmap v9137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:54:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:01 smithi104 ceph-mon[36624]: pgmap v9137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:54:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:01 smithi081 ceph-mon[31453]: pgmap v9137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:02.191 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:54:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:54:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:54:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:54:03.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:54:03.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:54:03.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:54:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:54:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:54:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:54:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:03 smithi153 ceph-mon[36569]: pgmap v9138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:03 smithi153 ceph-mon[36569]: pgmap v9139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:03 smithi104 ceph-mon[36624]: pgmap v9138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:04.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:03 smithi104 ceph-mon[36624]: pgmap v9139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:03 smithi081 ceph-mon[31453]: pgmap v9138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:03 smithi081 ceph-mon[31453]: pgmap v9139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:05.545+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:54:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:05.545+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:54:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:05.545+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:54:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:05 smithi153 ceph-mon[36569]: pgmap v9140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:05 smithi104 ceph-mon[36624]: pgmap v9140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:05 smithi081 ceph-mon[31453]: pgmap v9140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:07 smithi153 ceph-mon[36569]: pgmap v9141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:07 smithi104 ceph-mon[36624]: pgmap v9141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:08.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:07 smithi081 ceph-mon[31453]: pgmap v9141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:09 smithi153 ceph-mon[36569]: pgmap v9142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:10.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:09 smithi104 ceph-mon[36624]: pgmap v9142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:09 smithi081 ceph-mon[31453]: pgmap v9142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:11 smithi153 ceph-mon[36569]: pgmap v9143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:12.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:11 smithi104 ceph-mon[36624]: pgmap v9143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:11 smithi081 ceph-mon[31453]: pgmap v9143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:13 smithi104 ceph-mon[36624]: pgmap v9144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:13 smithi081 ceph-mon[31453]: pgmap v9144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:13 smithi153 ceph-mon[36569]: pgmap v9144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:15 smithi104 ceph-mon[36624]: pgmap v9145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:15 smithi081 ceph-mon[31453]: pgmap v9145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:15 smithi153 ceph-mon[36569]: pgmap v9145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:17 smithi104 ceph-mon[36624]: pgmap v9146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:17 smithi081 ceph-mon[31453]: pgmap v9146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:17 smithi153 ceph-mon[36569]: pgmap v9146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:20.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:19 smithi104 ceph-mon[36624]: pgmap v9147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:19 smithi081 ceph-mon[31453]: pgmap v9147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:19 smithi153 ceph-mon[36569]: pgmap v9147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:20.553+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:54:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:20.553+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:54:20.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:20.553+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:54:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:21 smithi104 ceph-mon[36624]: pgmap v9148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:21 smithi081 ceph-mon[31453]: pgmap v9148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:21 smithi153 ceph-mon[36569]: pgmap v9148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:23 smithi081 ceph-mon[31453]: pgmap v9149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:23 smithi153 ceph-mon[36569]: pgmap v9149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:23 smithi104 ceph-mon[36624]: pgmap v9149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:25 smithi081 ceph-mon[31453]: pgmap v9150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:25 smithi153 ceph-mon[36569]: pgmap v9150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:25 smithi104 ceph-mon[36624]: pgmap v9150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:27 smithi081 ceph-mon[31453]: pgmap v9151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:28.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:27 smithi153 ceph-mon[36569]: pgmap v9151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:27 smithi104 ceph-mon[36624]: pgmap v9151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:29 smithi081 ceph-mon[31453]: pgmap v9152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:29 smithi153 ceph-mon[36569]: pgmap v9152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:29 smithi104 ceph-mon[36624]: pgmap v9152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:31 smithi081 ceph-mon[31453]: pgmap v9153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:31 smithi153 ceph-mon[36569]: pgmap v9153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:31 smithi104 ceph-mon[36624]: pgmap v9153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:33 smithi081 ceph-mon[31453]: pgmap v9154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:33 smithi153 ceph-mon[36569]: pgmap v9154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:33 smithi104 ceph-mon[36624]: pgmap v9154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:35.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:54:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:35.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:54:35.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:35.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:54:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:35 smithi081 ceph-mon[31453]: pgmap v9155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:35 smithi153 ceph-mon[36569]: pgmap v9155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:35 smithi104 ceph-mon[36624]: pgmap v9155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:37 smithi081 ceph-mon[31453]: pgmap v9156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:37 smithi153 ceph-mon[36569]: pgmap v9156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:37 smithi104 ceph-mon[36624]: pgmap v9156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:39 smithi081 ceph-mon[31453]: pgmap v9157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:39 smithi153 ceph-mon[36569]: pgmap v9157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:39 smithi104 ceph-mon[36624]: pgmap v9157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:42.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:41 smithi081 ceph-mon[31453]: pgmap v9158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:41 smithi153 ceph-mon[36569]: pgmap v9158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:41 smithi104 ceph-mon[36624]: pgmap v9158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:43 smithi081 ceph-mon[31453]: pgmap v9159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:43 smithi153 ceph-mon[36569]: pgmap v9159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:43 smithi104 ceph-mon[36624]: pgmap v9159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:45 smithi081 ceph-mon[31453]: pgmap v9160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:45 smithi153 ceph-mon[36569]: pgmap v9160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:45 smithi104 ceph-mon[36624]: pgmap v9160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:47 smithi081 ceph-mon[31453]: pgmap v9161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:47 smithi153 ceph-mon[36569]: pgmap v9161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:47 smithi104 ceph-mon[36624]: pgmap v9161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:49 smithi081 ceph-mon[31453]: pgmap v9162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:49 smithi153 ceph-mon[36569]: pgmap v9162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:49 smithi104 ceph-mon[36624]: pgmap v9162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:50.573+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:54:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:50.573+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:54:50.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:54:50.573+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:54:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:51 smithi153 ceph-mon[36569]: pgmap v9163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:51 smithi104 ceph-mon[36624]: pgmap v9163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:51 smithi081 ceph-mon[31453]: pgmap v9163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:53 smithi153 ceph-mon[36569]: pgmap v9164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:53 smithi104 ceph-mon[36624]: pgmap v9164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:53 smithi081 ceph-mon[31453]: pgmap v9164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:55 smithi153 ceph-mon[36569]: pgmap v9165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:55 smithi104 ceph-mon[36624]: pgmap v9165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:55 smithi081 ceph-mon[31453]: pgmap v9165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:57 smithi153 ceph-mon[36569]: pgmap v9166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:57 smithi104 ceph-mon[36624]: pgmap v9166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:54:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:57 smithi081 ceph-mon[31453]: pgmap v9166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:54:59 smithi153 ceph-mon[36569]: pgmap v9167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:54:59 smithi104 ceph-mon[36624]: pgmap v9167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:54:59 smithi081 ceph-mon[31453]: pgmap v9167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:01 smithi153 ceph-mon[36569]: pgmap v9168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:01 smithi104 ceph-mon[36624]: pgmap v9168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:01 smithi081 ceph-mon[31453]: pgmap v9168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:55:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:55:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:55:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:03 smithi153 ceph-mon[36569]: pgmap v9169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:55:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:55:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:55:04.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:03 smithi081 ceph-mon[31453]: pgmap v9169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:04.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:55:04.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:55:04.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:55:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:03 smithi104 ceph-mon[36624]: pgmap v9169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:55:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:55:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:55:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:04 smithi153 ceph-mon[36569]: pgmap v9170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:04 smithi153 ceph-mon[36569]: pgmap v9171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:04 smithi104 ceph-mon[36624]: pgmap v9170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:04 smithi104 ceph-mon[36624]: pgmap v9171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:04 smithi081 ceph-mon[31453]: pgmap v9170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:04 smithi081 ceph-mon[31453]: pgmap v9171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:05.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:05.589+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:55:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:05.589+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:55:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:05.589+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:55:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:06 smithi153 ceph-mon[36569]: pgmap v9172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:06 smithi104 ceph-mon[36624]: pgmap v9172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:06 smithi081 ceph-mon[31453]: pgmap v9172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:08 smithi153 ceph-mon[36569]: pgmap v9173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:08 smithi104 ceph-mon[36624]: pgmap v9173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:08 smithi081 ceph-mon[31453]: pgmap v9173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:10 smithi153 ceph-mon[36569]: pgmap v9174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:10 smithi104 ceph-mon[36624]: pgmap v9174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:10 smithi081 ceph-mon[31453]: pgmap v9174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:12 smithi153 ceph-mon[36569]: pgmap v9175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:12 smithi104 ceph-mon[36624]: pgmap v9175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:12 smithi081 ceph-mon[31453]: pgmap v9175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:14 smithi153 ceph-mon[36569]: pgmap v9176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:14 smithi104 ceph-mon[36624]: pgmap v9176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:14 smithi081 ceph-mon[31453]: pgmap v9176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:15.467+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:55:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:15.467+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:55:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:15.468+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:55:15.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:15.468+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:55:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:16 smithi153 ceph-mon[36569]: pgmap v9177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:16 smithi104 ceph-mon[36624]: pgmap v9177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:16 smithi081 ceph-mon[31453]: pgmap v9177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:19.306 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:18 smithi081 ceph-mon[31453]: pgmap v9178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:18 smithi153 ceph-mon[36569]: pgmap v9178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:18 smithi104 ceph-mon[36624]: pgmap v9178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:20.603+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:55:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:20.603+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:55:20.605 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:20.603+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:55:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:20 smithi153 ceph-mon[36569]: pgmap v9179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:20 smithi104 ceph-mon[36624]: pgmap v9179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:20 smithi081 ceph-mon[31453]: pgmap v9179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:23 smithi153 ceph-mon[36569]: pgmap v9180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:23 smithi104 ceph-mon[36624]: pgmap v9180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:23 smithi081 ceph-mon[31453]: pgmap v9180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:25 smithi153 ceph-mon[36569]: pgmap v9181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:25 smithi104 ceph-mon[36624]: pgmap v9181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:25 smithi081 ceph-mon[31453]: pgmap v9181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:27 smithi153 ceph-mon[36569]: pgmap v9182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:27 smithi104 ceph-mon[36624]: pgmap v9182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:27 smithi081 ceph-mon[31453]: pgmap v9182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:28 smithi153 ceph-mon[36569]: pgmap v9183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:28 smithi104 ceph-mon[36624]: pgmap v9183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:28 smithi081 ceph-mon[31453]: pgmap v9183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:30 smithi153 ceph-mon[36569]: pgmap v9184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:30 smithi104 ceph-mon[36624]: pgmap v9184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:30 smithi081 ceph-mon[31453]: pgmap v9184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:32 smithi153 ceph-mon[36569]: pgmap v9185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:32 smithi104 ceph-mon[36624]: pgmap v9185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:32 smithi081 ceph-mon[31453]: pgmap v9185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:34 smithi153 ceph-mon[36569]: pgmap v9186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:34 smithi104 ceph-mon[36624]: pgmap v9186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:34 smithi081 ceph-mon[31453]: pgmap v9186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:35.615+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:55:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:35.615+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:55:35.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:35.615+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:55:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:36 smithi153 ceph-mon[36569]: pgmap v9187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:36 smithi104 ceph-mon[36624]: pgmap v9187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:36 smithi081 ceph-mon[31453]: pgmap v9187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:38 smithi153 ceph-mon[36569]: pgmap v9188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:38 smithi104 ceph-mon[36624]: pgmap v9188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:38 smithi081 ceph-mon[31453]: pgmap v9188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:40 smithi153 ceph-mon[36569]: pgmap v9189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:40 smithi104 ceph-mon[36624]: pgmap v9189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:40 smithi081 ceph-mon[31453]: pgmap v9189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:42 smithi153 ceph-mon[36569]: pgmap v9190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:43.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:42 smithi104 ceph-mon[36624]: pgmap v9190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:42 smithi081 ceph-mon[31453]: pgmap v9190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:44 smithi153 ceph-mon[36569]: pgmap v9191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:44 smithi104 ceph-mon[36624]: pgmap v9191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:45.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:44 smithi081 ceph-mon[31453]: pgmap v9191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:46 smithi153 ceph-mon[36569]: pgmap v9192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:47.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:46 smithi104 ceph-mon[36624]: pgmap v9192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:46 smithi081 ceph-mon[31453]: pgmap v9192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:48 smithi153 ceph-mon[36569]: pgmap v9193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:48 smithi104 ceph-mon[36624]: pgmap v9193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:48 smithi081 ceph-mon[31453]: pgmap v9193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:50.617+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:50.617+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:55:50.617+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:55:51.049 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:50 smithi081 ceph-mon[31453]: pgmap v9194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:50 smithi153 ceph-mon[36569]: pgmap v9194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:50 smithi104 ceph-mon[36624]: pgmap v9194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:52 smithi153 ceph-mon[36569]: pgmap v9195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:52 smithi104 ceph-mon[36624]: pgmap v9195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:52 smithi081 ceph-mon[31453]: pgmap v9195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:54 smithi153 ceph-mon[36569]: pgmap v9196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:54 smithi104 ceph-mon[36624]: pgmap v9196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:54 smithi081 ceph-mon[31453]: pgmap v9196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:56 smithi153 ceph-mon[36569]: pgmap v9197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:56 smithi104 ceph-mon[36624]: pgmap v9197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:56 smithi081 ceph-mon[31453]: pgmap v9197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:55:58 smithi153 ceph-mon[36569]: pgmap v9198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:55:58 smithi104 ceph-mon[36624]: pgmap v9198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:55:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:55:58 smithi081 ceph-mon[31453]: pgmap v9198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:00 smithi153 ceph-mon[36569]: pgmap v9199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:00 smithi104 ceph-mon[36624]: pgmap v9199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:00 smithi081 ceph-mon[31453]: pgmap v9199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:02 smithi153 ceph-mon[36569]: pgmap v9200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:02 smithi104 ceph-mon[36624]: pgmap v9200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:02 smithi081 ceph-mon[31453]: pgmap v9200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:04 smithi104 ceph-mon[36624]: pgmap v9201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:56:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:04 smithi081 ceph-mon[31453]: pgmap v9201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:56:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:04 smithi153 ceph-mon[36569]: pgmap v9201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:56:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:05.632+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:56:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:05.632+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:56:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:05.632+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:56:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:56:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:56:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:56:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:56:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:56:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:56:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:56:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:56:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:56:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:06 smithi104 ceph-mon[36624]: pgmap v9202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:06 smithi104 ceph-mon[36624]: pgmap v9203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:06 smithi081 ceph-mon[31453]: pgmap v9202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:06 smithi081 ceph-mon[31453]: pgmap v9203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:06 smithi153 ceph-mon[36569]: pgmap v9202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:06 smithi153 ceph-mon[36569]: pgmap v9203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:08 smithi104 ceph-mon[36624]: pgmap v9204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:08 smithi081 ceph-mon[31453]: pgmap v9204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:08 smithi153 ceph-mon[36569]: pgmap v9204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:10 smithi104 ceph-mon[36624]: pgmap v9205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:11.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:10 smithi153 ceph-mon[36569]: pgmap v9205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:10 smithi081 ceph-mon[31453]: pgmap v9205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:12 smithi081 ceph-mon[31453]: pgmap v9206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:12 smithi153 ceph-mon[36569]: pgmap v9206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:12 smithi104 ceph-mon[36624]: pgmap v9206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:14 smithi081 ceph-mon[31453]: pgmap v9207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:14 smithi153 ceph-mon[36569]: pgmap v9207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:15.374 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:14 smithi104 ceph-mon[36624]: pgmap v9207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:16 smithi081 ceph-mon[31453]: pgmap v9208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:16 smithi153 ceph-mon[36569]: pgmap v9208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:16 smithi104 ceph-mon[36624]: pgmap v9208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:18 smithi081 ceph-mon[31453]: pgmap v9209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:18 smithi153 ceph-mon[36569]: pgmap v9209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:18 smithi104 ceph-mon[36624]: pgmap v9209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:20.645+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:56:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:20.645+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:56:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:20.645+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:56:21.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:20 smithi104 ceph-mon[36624]: pgmap v9210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:20 smithi081 ceph-mon[31453]: pgmap v9210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:20 smithi153 ceph-mon[36569]: pgmap v9210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:22 smithi081 ceph-mon[31453]: pgmap v9211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:22 smithi153 ceph-mon[36569]: pgmap v9211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:22 smithi104 ceph-mon[36624]: pgmap v9211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:24 smithi081 ceph-mon[31453]: pgmap v9212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:24 smithi153 ceph-mon[36569]: pgmap v9212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:24 smithi104 ceph-mon[36624]: pgmap v9212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:26 smithi081 ceph-mon[31453]: pgmap v9213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:26 smithi153 ceph-mon[36569]: pgmap v9213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:26 smithi104 ceph-mon[36624]: pgmap v9213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:28 smithi081 ceph-mon[31453]: pgmap v9214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:28 smithi153 ceph-mon[36569]: pgmap v9214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:28 smithi104 ceph-mon[36624]: pgmap v9214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:30 smithi081 ceph-mon[31453]: pgmap v9215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:30 smithi153 ceph-mon[36569]: pgmap v9215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:30 smithi104 ceph-mon[36624]: pgmap v9215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:32 smithi081 ceph-mon[31453]: pgmap v9216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:32 smithi153 ceph-mon[36569]: pgmap v9216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:33.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:32 smithi104 ceph-mon[36624]: pgmap v9216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:34 smithi081 ceph-mon[31453]: pgmap v9217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:34 smithi153 ceph-mon[36569]: pgmap v9217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:34 smithi104 ceph-mon[36624]: pgmap v9217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:35.649+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:56:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:35.649+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:56:35.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:35.649+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:56:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:36 smithi153 ceph-mon[36569]: pgmap v9218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:36 smithi104 ceph-mon[36624]: pgmap v9218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:37.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:36 smithi081 ceph-mon[31453]: pgmap v9218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:38 smithi153 ceph-mon[36569]: pgmap v9219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:38 smithi104 ceph-mon[36624]: pgmap v9219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:38 smithi081 ceph-mon[31453]: pgmap v9219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:40 smithi153 ceph-mon[36569]: pgmap v9220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:40 smithi104 ceph-mon[36624]: pgmap v9220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:40 smithi081 ceph-mon[31453]: pgmap v9220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:42 smithi153 ceph-mon[36569]: pgmap v9221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:42 smithi104 ceph-mon[36624]: pgmap v9221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:43.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:42 smithi081 ceph-mon[31453]: pgmap v9221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:44 smithi153 ceph-mon[36569]: pgmap v9222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:44 smithi104 ceph-mon[36624]: pgmap v9222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:44 smithi081 ceph-mon[31453]: pgmap v9222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:45.482+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:56:45.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:45.482+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:56:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:45.482+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:56:45.484 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:45.482+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:56:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:46 smithi153 ceph-mon[36569]: pgmap v9223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:46 smithi104 ceph-mon[36624]: pgmap v9223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:46 smithi081 ceph-mon[31453]: pgmap v9223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:48 smithi153 ceph-mon[36569]: pgmap v9224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:48 smithi104 ceph-mon[36624]: pgmap v9224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:48 smithi081 ceph-mon[31453]: pgmap v9224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:50.665+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:56:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:50.665+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:56:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:56:50.665+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:56:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:50 smithi153 ceph-mon[36569]: pgmap v9225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:50 smithi104 ceph-mon[36624]: pgmap v9225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:50 smithi081 ceph-mon[31453]: pgmap v9225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:53.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:52 smithi081 ceph-mon[31453]: pgmap v9226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:52 smithi153 ceph-mon[36569]: pgmap v9226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:52 smithi104 ceph-mon[36624]: pgmap v9226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:54 smithi153 ceph-mon[36569]: pgmap v9227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:54 smithi104 ceph-mon[36624]: pgmap v9227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:54 smithi081 ceph-mon[31453]: pgmap v9227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:56 smithi153 ceph-mon[36569]: pgmap v9228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:56 smithi104 ceph-mon[36624]: pgmap v9228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:56 smithi081 ceph-mon[31453]: pgmap v9228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:56:58 smithi153 ceph-mon[36569]: pgmap v9229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:56:58 smithi104 ceph-mon[36624]: pgmap v9229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:56:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:56:58 smithi081 ceph-mon[31453]: pgmap v9229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:00 smithi153 ceph-mon[36569]: pgmap v9230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:00 smithi104 ceph-mon[36624]: pgmap v9230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:00 smithi081 ceph-mon[31453]: pgmap v9230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:02 smithi153 ceph-mon[36569]: pgmap v9231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:02 smithi104 ceph-mon[36624]: pgmap v9231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:02 smithi081 ceph-mon[31453]: pgmap v9231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:04 smithi153 ceph-mon[36569]: pgmap v9232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:04 smithi104 ceph-mon[36624]: pgmap v9232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:04 smithi081 ceph-mon[31453]: pgmap v9232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:05.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:05.680+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:57:05.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:05.680+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:57:05.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:05.680+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:57:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:57:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:57:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:57:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:06 smithi153 ceph-mon[36569]: pgmap v9233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:06 smithi104 ceph-mon[36624]: pgmap v9233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:06 smithi081 ceph-mon[31453]: pgmap v9233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:07.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:07.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:08 smithi153 ceph-mon[36569]: pgmap v9234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:08 smithi104 ceph-mon[36624]: pgmap v9234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:08 smithi081 ceph-mon[31453]: pgmap v9234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:10 smithi153 ceph-mon[36569]: pgmap v9235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:10 smithi104 ceph-mon[36624]: pgmap v9235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:10 smithi081 ceph-mon[31453]: pgmap v9235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:57:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:57:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:57:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:57:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:57:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:57:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:57:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:12 smithi153 ceph-mon[36569]: pgmap v9236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:12 smithi153 ceph-mon[36569]: pgmap v9237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:12 smithi153 ceph-mon[36569]: pgmap v9238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:12 smithi104 ceph-mon[36624]: pgmap v9236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:12 smithi104 ceph-mon[36624]: pgmap v9237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:12 smithi104 ceph-mon[36624]: pgmap v9238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:12 smithi081 ceph-mon[31453]: pgmap v9236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:12 smithi081 ceph-mon[31453]: pgmap v9237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:12 smithi081 ceph-mon[31453]: pgmap v9238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:14 smithi153 ceph-mon[36569]: pgmap v9239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:14 smithi104 ceph-mon[36624]: pgmap v9239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:14 smithi081 ceph-mon[31453]: pgmap v9239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:16 smithi153 ceph-mon[36569]: pgmap v9240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:16 smithi104 ceph-mon[36624]: pgmap v9240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:16 smithi081 ceph-mon[31453]: pgmap v9240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:19 smithi153 ceph-mon[36569]: pgmap v9241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:19 smithi104 ceph-mon[36624]: pgmap v9241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:19 smithi081 ceph-mon[31453]: pgmap v9241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:20.681+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:57:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:20.681+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:57:20.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:20.681+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:57:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:21 smithi153 ceph-mon[36569]: pgmap v9242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:21 smithi104 ceph-mon[36624]: pgmap v9242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:21 smithi081 ceph-mon[31453]: pgmap v9242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:23 smithi153 ceph-mon[36569]: pgmap v9243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:23 smithi104 ceph-mon[36624]: pgmap v9243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:23 smithi081 ceph-mon[31453]: pgmap v9243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:25 smithi153 ceph-mon[36569]: pgmap v9244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:25 smithi104 ceph-mon[36624]: pgmap v9244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:25 smithi081 ceph-mon[31453]: pgmap v9244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:27 smithi153 ceph-mon[36569]: pgmap v9245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:27 smithi104 ceph-mon[36624]: pgmap v9245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:27 smithi081 ceph-mon[31453]: pgmap v9245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:29 smithi153 ceph-mon[36569]: pgmap v9246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:29 smithi104 ceph-mon[36624]: pgmap v9246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:29 smithi081 ceph-mon[31453]: pgmap v9246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:31 smithi153 ceph-mon[36569]: pgmap v9247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:31 smithi104 ceph-mon[36624]: pgmap v9247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:31 smithi081 ceph-mon[31453]: pgmap v9247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:33 smithi153 ceph-mon[36569]: pgmap v9248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:33 smithi104 ceph-mon[36624]: pgmap v9248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:33 smithi081 ceph-mon[31453]: pgmap v9248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:34 smithi081 ceph-mon[31453]: pgmap v9249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:34 smithi153 ceph-mon[36569]: pgmap v9249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:34 smithi104 ceph-mon[36624]: pgmap v9249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:35.696+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:57:35.698 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:35.697+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:57:35.699 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:35.697+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:57:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:36 smithi153 ceph-mon[36569]: pgmap v9250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:36 smithi104 ceph-mon[36624]: pgmap v9250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:36 smithi081 ceph-mon[31453]: pgmap v9250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:39 smithi153 ceph-mon[36569]: pgmap v9251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:39 smithi104 ceph-mon[36624]: pgmap v9251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:39 smithi081 ceph-mon[31453]: pgmap v9251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:41 smithi153 ceph-mon[36569]: pgmap v9252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:41 smithi104 ceph-mon[36624]: pgmap v9252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:41 smithi081 ceph-mon[31453]: pgmap v9252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:43 smithi153 ceph-mon[36569]: pgmap v9253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:43 smithi104 ceph-mon[36624]: pgmap v9253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:43 smithi081 ceph-mon[31453]: pgmap v9253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:45 smithi153 ceph-mon[36569]: pgmap v9254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:45 smithi104 ceph-mon[36624]: pgmap v9254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:45 smithi081 ceph-mon[31453]: pgmap v9254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:47 smithi153 ceph-mon[36569]: pgmap v9255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:47 smithi104 ceph-mon[36624]: pgmap v9255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:47 smithi081 ceph-mon[31453]: pgmap v9255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:49 smithi153 ceph-mon[36569]: pgmap v9256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:49 smithi104 ceph-mon[36624]: pgmap v9256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:49 smithi081 ceph-mon[31453]: pgmap v9256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:50.712+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:57:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:50.712+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:57:50.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:57:50.712+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:57:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:51 smithi153 ceph-mon[36569]: pgmap v9257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:51 smithi104 ceph-mon[36624]: pgmap v9257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:51 smithi081 ceph-mon[31453]: pgmap v9257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:53 smithi153 ceph-mon[36569]: pgmap v9258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:53 smithi104 ceph-mon[36624]: pgmap v9258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:53 smithi081 ceph-mon[31453]: pgmap v9258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:55 smithi153 ceph-mon[36569]: pgmap v9259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:55 smithi104 ceph-mon[36624]: pgmap v9259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:55 smithi081 ceph-mon[31453]: pgmap v9259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:57 smithi153 ceph-mon[36569]: pgmap v9260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:57 smithi104 ceph-mon[36624]: pgmap v9260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:57 smithi081 ceph-mon[31453]: pgmap v9260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:57:58 smithi081 ceph-mon[31453]: pgmap v9261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:57:58 smithi153 ceph-mon[36569]: pgmap v9261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:57:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:57:58 smithi104 ceph-mon[36624]: pgmap v9261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:01 smithi153 ceph-mon[36569]: pgmap v9262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:01 smithi104 ceph-mon[36624]: pgmap v9262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:01 smithi081 ceph-mon[31453]: pgmap v9262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:03 smithi153 ceph-mon[36569]: pgmap v9263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:03 smithi104 ceph-mon[36624]: pgmap v9263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:03 smithi081 ceph-mon[31453]: pgmap v9263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:05 smithi153 ceph-mon[36569]: pgmap v9264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:05 smithi104 ceph-mon[36624]: pgmap v9264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:05 smithi081 ceph-mon[31453]: pgmap v9264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:05.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:05.727+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:58:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:05.727+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:58:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:05.727+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:58:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:07 smithi153 ceph-mon[36569]: pgmap v9265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:07 smithi104 ceph-mon[36624]: pgmap v9265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:07 smithi081 ceph-mon[31453]: pgmap v9265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:09 smithi153 ceph-mon[36569]: pgmap v9266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:09 smithi104 ceph-mon[36624]: pgmap v9266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:09 smithi081 ceph-mon[31453]: pgmap v9266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:11 smithi153 ceph-mon[36569]: pgmap v9267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:11 smithi104 ceph-mon[36624]: pgmap v9267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:11.336 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:11 smithi081 ceph-mon[31453]: pgmap v9267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:13 smithi153 ceph-mon[36569]: pgmap v9268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:58:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:13 smithi104 ceph-mon[36624]: pgmap v9268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:58:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:13 smithi081 ceph-mon[31453]: pgmap v9268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:58:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:58:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:58:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:58:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:58:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:58:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:58:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:58:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:15 smithi153 ceph-mon[36569]: pgmap v9269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:15 smithi153 ceph-mon[36569]: pgmap v9270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:15 smithi104 ceph-mon[36624]: pgmap v9269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:15 smithi104 ceph-mon[36624]: pgmap v9270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:15 smithi081 ceph-mon[31453]: pgmap v9269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:15 smithi081 ceph-mon[31453]: pgmap v9270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:15.496+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:58:15.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:15.496+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:58:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:15.497+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T01:58:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:15.497+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T01:58:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:16 smithi104 ceph-mon[36624]: pgmap v9271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:16 smithi081 ceph-mon[31453]: pgmap v9271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:16 smithi153 ceph-mon[36569]: pgmap v9271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:18 smithi081 ceph-mon[31453]: pgmap v9272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:18 smithi153 ceph-mon[36569]: pgmap v9272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:18 smithi104 ceph-mon[36624]: pgmap v9272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:20.742+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:58:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:20.742+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:58:20.744 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:20.742+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:58:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:20 smithi081 ceph-mon[31453]: pgmap v9273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:20 smithi153 ceph-mon[36569]: pgmap v9273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:20 smithi104 ceph-mon[36624]: pgmap v9273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:22.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:22 smithi104 ceph-mon[36624]: pgmap v9274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:22 smithi081 ceph-mon[31453]: pgmap v9274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:22 smithi153 ceph-mon[36569]: pgmap v9274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:24.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:24 smithi081 ceph-mon[31453]: pgmap v9275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:24 smithi153 ceph-mon[36569]: pgmap v9275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:24 smithi104 ceph-mon[36624]: pgmap v9275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:26 smithi081 ceph-mon[31453]: pgmap v9276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:26 smithi153 ceph-mon[36569]: pgmap v9276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:26 smithi104 ceph-mon[36624]: pgmap v9276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:28 smithi081 ceph-mon[31453]: pgmap v9277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:28 smithi153 ceph-mon[36569]: pgmap v9277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:28 smithi104 ceph-mon[36624]: pgmap v9277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:30 smithi081 ceph-mon[31453]: pgmap v9278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:30 smithi153 ceph-mon[36569]: pgmap v9278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:30 smithi104 ceph-mon[36624]: pgmap v9278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:32 smithi081 ceph-mon[31453]: pgmap v9279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:32 smithi153 ceph-mon[36569]: pgmap v9279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:32 smithi104 ceph-mon[36624]: pgmap v9279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:34 smithi081 ceph-mon[31453]: pgmap v9280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:34 smithi153 ceph-mon[36569]: pgmap v9280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:34 smithi104 ceph-mon[36624]: pgmap v9280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:35.747+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:58:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:35.747+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:58:35.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:35.747+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:58:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:36 smithi081 ceph-mon[31453]: pgmap v9281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:36 smithi153 ceph-mon[36569]: pgmap v9281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:36 smithi104 ceph-mon[36624]: pgmap v9281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:38 smithi081 ceph-mon[31453]: pgmap v9282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:38 smithi153 ceph-mon[36569]: pgmap v9282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:38 smithi104 ceph-mon[36624]: pgmap v9282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:40 smithi153 ceph-mon[36569]: pgmap v9283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:40 smithi104 ceph-mon[36624]: pgmap v9283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:40 smithi081 ceph-mon[31453]: pgmap v9283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:42 smithi153 ceph-mon[36569]: pgmap v9284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:43.081 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:42 smithi081 ceph-mon[31453]: pgmap v9284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:42 smithi104 ceph-mon[36624]: pgmap v9284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:44 smithi153 ceph-mon[36569]: pgmap v9285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:44 smithi104 ceph-mon[36624]: pgmap v9285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:44 smithi081 ceph-mon[31453]: pgmap v9285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:46 smithi153 ceph-mon[36569]: pgmap v9286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:46 smithi104 ceph-mon[36624]: pgmap v9286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:46 smithi081 ceph-mon[31453]: pgmap v9286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:48 smithi153 ceph-mon[36569]: pgmap v9287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:48 smithi104 ceph-mon[36624]: pgmap v9287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:48 smithi081 ceph-mon[31453]: pgmap v9287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:50.761+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:58:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:50.762+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:58:50.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:58:50.762+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:58:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:50 smithi153 ceph-mon[36569]: pgmap v9288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:50 smithi104 ceph-mon[36624]: pgmap v9288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:50 smithi081 ceph-mon[31453]: pgmap v9288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:52 smithi153 ceph-mon[36569]: pgmap v9289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:52 smithi104 ceph-mon[36624]: pgmap v9289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:52 smithi081 ceph-mon[31453]: pgmap v9289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:54.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:54 smithi081 ceph-mon[31453]: pgmap v9290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:54 smithi153 ceph-mon[36569]: pgmap v9290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:54 smithi104 ceph-mon[36624]: pgmap v9290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:56 smithi153 ceph-mon[36569]: pgmap v9291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:56 smithi104 ceph-mon[36624]: pgmap v9291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:56 smithi081 ceph-mon[31453]: pgmap v9291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:58:58 smithi153 ceph-mon[36569]: pgmap v9292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:58:58 smithi104 ceph-mon[36624]: pgmap v9292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:58:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:58:58 smithi081 ceph-mon[31453]: pgmap v9292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:00 smithi153 ceph-mon[36569]: pgmap v9293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:00 smithi104 ceph-mon[36624]: pgmap v9293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:00 smithi081 ceph-mon[31453]: pgmap v9293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:02 smithi153 ceph-mon[36569]: pgmap v9294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:02 smithi104 ceph-mon[36624]: pgmap v9294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:02 smithi081 ceph-mon[31453]: pgmap v9294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:04 smithi153 ceph-mon[36569]: pgmap v9295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:04 smithi104 ceph-mon[36624]: pgmap v9295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:04 smithi081 ceph-mon[31453]: pgmap v9295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:05.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:05.777+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:59:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:05.777+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:59:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:05.777+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:59:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:06 smithi153 ceph-mon[36569]: pgmap v9296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:06 smithi104 ceph-mon[36624]: pgmap v9296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:06 smithi081 ceph-mon[31453]: pgmap v9296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:08 smithi153 ceph-mon[36569]: pgmap v9297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:08 smithi104 ceph-mon[36624]: pgmap v9297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:08 smithi081 ceph-mon[31453]: pgmap v9297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:10 smithi153 ceph-mon[36569]: pgmap v9298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:10 smithi104 ceph-mon[36624]: pgmap v9298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:10 smithi081 ceph-mon[31453]: pgmap v9298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:12 smithi153 ceph-mon[36569]: pgmap v9299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:13.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:12 smithi104 ceph-mon[36624]: pgmap v9299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:12 smithi081 ceph-mon[31453]: pgmap v9299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:14 smithi153 ceph-mon[36569]: pgmap v9300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:59:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:14 smithi104 ceph-mon[36624]: pgmap v9300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:59:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:14 smithi081 ceph-mon[31453]: pgmap v9300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T01:59:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:59:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:59:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:15 smithi153 ceph-mon[36569]: pgmap v9301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:15 smithi153 ceph-mon[36569]: pgmap v9302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:16.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:59:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:59:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:59:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:15 smithi104 ceph-mon[36624]: pgmap v9301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:15 smithi104 ceph-mon[36624]: pgmap v9302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:59:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T01:59:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T01:59:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:15 smithi081 ceph-mon[31453]: pgmap v9301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:15 smithi081 ceph-mon[31453]: pgmap v9302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T01:59:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:17 smithi081 ceph-mon[31453]: pgmap v9303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:17 smithi153 ceph-mon[36569]: pgmap v9303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:17 smithi104 ceph-mon[36624]: pgmap v9303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:19 smithi081 ceph-mon[31453]: pgmap v9304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:19 smithi153 ceph-mon[36569]: pgmap v9304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:19 smithi104 ceph-mon[36624]: pgmap v9304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:20.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:20.787+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:59:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:20.787+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:59:20.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:20.787+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:59:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:21 smithi081 ceph-mon[31453]: pgmap v9305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:21.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:21 smithi153 ceph-mon[36569]: pgmap v9305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:21 smithi104 ceph-mon[36624]: pgmap v9305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:23.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:23 smithi104 ceph-mon[36624]: pgmap v9306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:23 smithi081 ceph-mon[31453]: pgmap v9306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:23 smithi153 ceph-mon[36569]: pgmap v9306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:25 smithi081 ceph-mon[31453]: pgmap v9307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:25 smithi153 ceph-mon[36569]: pgmap v9307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:25 smithi104 ceph-mon[36624]: pgmap v9307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:27 smithi081 ceph-mon[31453]: pgmap v9308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:27 smithi153 ceph-mon[36569]: pgmap v9308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:27 smithi104 ceph-mon[36624]: pgmap v9308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:29 smithi081 ceph-mon[31453]: pgmap v9309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:29 smithi153 ceph-mon[36569]: pgmap v9309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:29 smithi104 ceph-mon[36624]: pgmap v9309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:31 smithi081 ceph-mon[31453]: pgmap v9310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:31 smithi153 ceph-mon[36569]: pgmap v9310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:31 smithi104 ceph-mon[36624]: pgmap v9310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:34 smithi081 ceph-mon[31453]: pgmap v9311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:34 smithi153 ceph-mon[36569]: pgmap v9311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:34 smithi104 ceph-mon[36624]: pgmap v9311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:35.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:35.802+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:59:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:35.802+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:59:35.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:35.802+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:59:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:36 smithi081 ceph-mon[31453]: pgmap v9312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:36 smithi153 ceph-mon[36569]: pgmap v9312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:36 smithi104 ceph-mon[36624]: pgmap v9312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:38 smithi081 ceph-mon[31453]: pgmap v9313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:38 smithi153 ceph-mon[36569]: pgmap v9313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:38 smithi104 ceph-mon[36624]: pgmap v9313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:40 smithi081 ceph-mon[31453]: pgmap v9314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:40 smithi153 ceph-mon[36569]: pgmap v9314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:40 smithi104 ceph-mon[36624]: pgmap v9314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:42 smithi081 ceph-mon[31453]: pgmap v9315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:42 smithi153 ceph-mon[36569]: pgmap v9315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:42 smithi104 ceph-mon[36624]: pgmap v9315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:44 smithi081 ceph-mon[31453]: pgmap v9316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:44 smithi153 ceph-mon[36569]: pgmap v9316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:44 smithi104 ceph-mon[36624]: pgmap v9316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:46 smithi081 ceph-mon[31453]: pgmap v9317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:46 smithi153 ceph-mon[36569]: pgmap v9317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:46 smithi104 ceph-mon[36624]: pgmap v9317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:48 smithi153 ceph-mon[36569]: pgmap v9318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:48 smithi104 ceph-mon[36624]: pgmap v9318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:48 smithi081 ceph-mon[31453]: pgmap v9318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:50 smithi153 ceph-mon[36569]: pgmap v9319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:50 smithi104 ceph-mon[36624]: pgmap v9319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:50 smithi081 ceph-mon[31453]: pgmap v9319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:50.815+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T01:59:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:50.815+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T01:59:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T01:59:50.815+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T01:59:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:52 smithi153 ceph-mon[36569]: pgmap v9320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:52 smithi104 ceph-mon[36624]: pgmap v9320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:52 smithi081 ceph-mon[31453]: pgmap v9320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:54.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:54 smithi153 ceph-mon[36569]: pgmap v9321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:54 smithi104 ceph-mon[36624]: pgmap v9321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:54 smithi081 ceph-mon[31453]: pgmap v9321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:56.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:56 smithi081 ceph-mon[31453]: pgmap v9322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:56 smithi153 ceph-mon[36569]: pgmap v9322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:56 smithi104 ceph-mon[36624]: pgmap v9322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 01:59:58 smithi153 ceph-mon[36569]: pgmap v9323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 01:59:58 smithi104 ceph-mon[36624]: pgmap v9323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T01:59:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 01:59:58 smithi081 ceph-mon[31453]: pgmap v9323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:00 smithi153 ceph-mon[36569]: pgmap v9324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:00.644 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:00:00.645 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:00 smithi104 ceph-mon[36624]: pgmap v9324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:00.645 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:00:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:00 smithi081 ceph-mon[31453]: pgmap v9324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:00:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:02 smithi153 ceph-mon[36569]: pgmap v9325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:02 smithi104 ceph-mon[36624]: pgmap v9325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:02 smithi081 ceph-mon[31453]: pgmap v9325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:04 smithi153 ceph-mon[36569]: pgmap v9326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:04 smithi104 ceph-mon[36624]: pgmap v9326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:04 smithi081 ceph-mon[31453]: pgmap v9326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:05.818+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:00:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:05.818+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:00:05.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:05.818+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:00:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:06 smithi153 ceph-mon[36569]: pgmap v9327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:06 smithi104 ceph-mon[36624]: pgmap v9327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:06 smithi081 ceph-mon[31453]: pgmap v9327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:08 smithi153 ceph-mon[36569]: pgmap v9328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:08 smithi104 ceph-mon[36624]: pgmap v9328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:08 smithi081 ceph-mon[31453]: pgmap v9328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:10 smithi153 ceph-mon[36569]: pgmap v9329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:10 smithi104 ceph-mon[36624]: pgmap v9329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:10 smithi081 ceph-mon[31453]: pgmap v9329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:12.504 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:12 smithi081 ceph-mon[31453]: pgmap v9330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:12 smithi153 ceph-mon[36569]: pgmap v9330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:12 smithi104 ceph-mon[36624]: pgmap v9330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:14 smithi153 ceph-mon[36569]: pgmap v9331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:14.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:14 smithi104 ceph-mon[36624]: pgmap v9331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:14 smithi081 ceph-mon[31453]: pgmap v9331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:16.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:16 smithi081 ceph-mon[31453]: pgmap v9332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:16.545 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:00:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:16 smithi153 ceph-mon[36569]: pgmap v9332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:00:16.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:16 smithi104 ceph-mon[36624]: pgmap v9332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:16.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:00:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:00:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:00:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:17 smithi153 ceph-mon[36569]: pgmap v9333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:17 smithi153 ceph-mon[36569]: pgmap v9334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:00:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:00:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:00:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:17 smithi104 ceph-mon[36624]: pgmap v9333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:17 smithi104 ceph-mon[36624]: pgmap v9334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:17.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:00:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:00:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:00:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:17 smithi081 ceph-mon[31453]: pgmap v9333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:17 smithi081 ceph-mon[31453]: pgmap v9334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:00:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:19 smithi081 ceph-mon[31453]: pgmap v9335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:19 smithi153 ceph-mon[36569]: pgmap v9335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:19 smithi104 ceph-mon[36624]: pgmap v9335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:20.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:20.827+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:00:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:20.827+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:00:20.829 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:20.827+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:00:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:21 smithi081 ceph-mon[31453]: pgmap v9336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:21 smithi153 ceph-mon[36569]: pgmap v9336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:21 smithi104 ceph-mon[36624]: pgmap v9336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:23 smithi081 ceph-mon[31453]: pgmap v9337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:23 smithi153 ceph-mon[36569]: pgmap v9337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:23 smithi104 ceph-mon[36624]: pgmap v9337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:25.904 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:25 smithi104 ceph-mon[36624]: pgmap v9338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:25 smithi081 ceph-mon[31453]: pgmap v9338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:25 smithi153 ceph-mon[36569]: pgmap v9338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:27 smithi104 ceph-mon[36624]: pgmap v9339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:27 smithi081 ceph-mon[31453]: pgmap v9339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:27 smithi153 ceph-mon[36569]: pgmap v9339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:29 smithi081 ceph-mon[31453]: pgmap v9340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:29 smithi153 ceph-mon[36569]: pgmap v9340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:29 smithi104 ceph-mon[36624]: pgmap v9340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:31 smithi153 ceph-mon[36569]: pgmap v9341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:31 smithi104 ceph-mon[36624]: pgmap v9341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:31 smithi081 ceph-mon[31453]: pgmap v9341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:33 smithi153 ceph-mon[36569]: pgmap v9342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:33 smithi104 ceph-mon[36624]: pgmap v9342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:33 smithi081 ceph-mon[31453]: pgmap v9342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:35.843+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:00:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:35.843+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:00:35.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:35.843+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:00:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:35 smithi153 ceph-mon[36569]: pgmap v9343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:35 smithi104 ceph-mon[36624]: pgmap v9343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:35 smithi081 ceph-mon[31453]: pgmap v9343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:37 smithi153 ceph-mon[36569]: pgmap v9344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:37 smithi104 ceph-mon[36624]: pgmap v9344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:37 smithi081 ceph-mon[31453]: pgmap v9344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:39 smithi153 ceph-mon[36569]: pgmap v9345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:39 smithi104 ceph-mon[36624]: pgmap v9345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:39 smithi081 ceph-mon[31453]: pgmap v9345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:41 smithi153 ceph-mon[36569]: pgmap v9346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:41 smithi104 ceph-mon[36624]: pgmap v9346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:41 smithi081 ceph-mon[31453]: pgmap v9346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:43 smithi153 ceph-mon[36569]: pgmap v9347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:43 smithi104 ceph-mon[36624]: pgmap v9347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:43 smithi081 ceph-mon[31453]: pgmap v9347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:45 smithi153 ceph-mon[36569]: pgmap v9348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:45 smithi104 ceph-mon[36624]: pgmap v9348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:45 smithi081 ceph-mon[31453]: pgmap v9348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:47 smithi153 ceph-mon[36569]: pgmap v9349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:47 smithi104 ceph-mon[36624]: pgmap v9349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:47 smithi081 ceph-mon[31453]: pgmap v9349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:49 smithi153 ceph-mon[36569]: pgmap v9350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:49 smithi104 ceph-mon[36624]: pgmap v9350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:49 smithi081 ceph-mon[31453]: pgmap v9350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:50.851+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:00:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:50.851+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:00:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:00:50.851+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:00:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:51 smithi153 ceph-mon[36569]: pgmap v9351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:51 smithi104 ceph-mon[36624]: pgmap v9351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:51 smithi081 ceph-mon[31453]: pgmap v9351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:53 smithi153 ceph-mon[36569]: pgmap v9352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:53 smithi104 ceph-mon[36624]: pgmap v9352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:53 smithi081 ceph-mon[31453]: pgmap v9352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:55 smithi153 ceph-mon[36569]: pgmap v9353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:55 smithi104 ceph-mon[36624]: pgmap v9353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:55 smithi081 ceph-mon[31453]: pgmap v9353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:57 smithi153 ceph-mon[36569]: pgmap v9354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:57 smithi104 ceph-mon[36624]: pgmap v9354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:00:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:57 smithi081 ceph-mon[31453]: pgmap v9354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:00:59 smithi153 ceph-mon[36569]: pgmap v9355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:00:59 smithi104 ceph-mon[36624]: pgmap v9355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:00:59 smithi081 ceph-mon[31453]: pgmap v9355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:01 smithi153 ceph-mon[36569]: pgmap v9356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:01 smithi104 ceph-mon[36624]: pgmap v9356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:01 smithi081 ceph-mon[31453]: pgmap v9356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:03 smithi153 ceph-mon[36569]: pgmap v9357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:03 smithi104 ceph-mon[36624]: pgmap v9357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:03 smithi081 ceph-mon[31453]: pgmap v9357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:05.860+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:01:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:05.860+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:01:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:05.860+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:01:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:05 smithi153 ceph-mon[36569]: pgmap v9358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:05 smithi104 ceph-mon[36624]: pgmap v9358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:05 smithi081 ceph-mon[31453]: pgmap v9358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:07 smithi153 ceph-mon[36569]: pgmap v9359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:07 smithi104 ceph-mon[36624]: pgmap v9359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:07 smithi081 ceph-mon[31453]: pgmap v9359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:09 smithi153 ceph-mon[36569]: pgmap v9360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:09 smithi104 ceph-mon[36624]: pgmap v9360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:09 smithi081 ceph-mon[31453]: pgmap v9360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:11 smithi153 ceph-mon[36569]: pgmap v9361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:11 smithi104 ceph-mon[36624]: pgmap v9361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:12.136+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 msgr2=0x7f16e80b3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000664) for more than 900000000 us, fault. 2024-04-13T02:01:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:12.136+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3fb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f16f402b000 tx=0x7f16f40c4000 comp rx=0 tx=0).stop 2024-04-13T02:01:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:12.136+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 msgr2=0x7f16e80b3fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T02:01:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:12.136+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3fb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T02:01:12.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:12.136+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T02:01:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:12.137+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T02:01:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:12.138+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b44f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f4032000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T02:01:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:11 smithi081 ceph-mon[31453]: pgmap v9361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:13 smithi153 ceph-mon[36569]: pgmap v9362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:13 smithi104 ceph-mon[36624]: pgmap v9362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:13 smithi081 ceph-mon[31453]: pgmap v9362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:15.528 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:15.527+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:01:15.528 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:15.527+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:01:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:15.528+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:01:15.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:15.528+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:01:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:15 smithi153 ceph-mon[36569]: pgmap v9363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:15 smithi104 ceph-mon[36624]: pgmap v9363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:15 smithi081 ceph-mon[31453]: pgmap v9363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:17 smithi153 ceph-mon[36569]: pgmap v9364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:01:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:17 smithi104 ceph-mon[36624]: pgmap v9364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:01:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:17 smithi081 ceph-mon[31453]: pgmap v9364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:01:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:01:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:01:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:01:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:01:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:01:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:01:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:01:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:01:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:01:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:19 smithi153 ceph-mon[36569]: pgmap v9365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:19 smithi153 ceph-mon[36569]: pgmap v9366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:19 smithi104 ceph-mon[36624]: pgmap v9365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:19 smithi104 ceph-mon[36624]: pgmap v9366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:19 smithi081 ceph-mon[31453]: pgmap v9365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:19 smithi081 ceph-mon[31453]: pgmap v9366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:20.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:20.862+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:01:20.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:20.862+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:01:20.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:20.862+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:01:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:21 smithi153 ceph-mon[36569]: pgmap v9367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:21 smithi104 ceph-mon[36624]: pgmap v9367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:21 smithi081 ceph-mon[31453]: pgmap v9367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:23.802 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:23 smithi104 ceph-mon[36624]: pgmap v9368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:23 smithi153 ceph-mon[36569]: pgmap v9368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:23 smithi081 ceph-mon[31453]: pgmap v9368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:25 smithi153 ceph-mon[36569]: pgmap v9369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:25 smithi104 ceph-mon[36624]: pgmap v9369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:25 smithi081 ceph-mon[31453]: pgmap v9369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:27 smithi104 ceph-mon[36624]: pgmap v9370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:27 smithi081 ceph-mon[31453]: pgmap v9370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:27 smithi153 ceph-mon[36569]: pgmap v9370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:29 smithi104 ceph-mon[36624]: pgmap v9371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:29 smithi081 ceph-mon[31453]: pgmap v9371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:29 smithi153 ceph-mon[36569]: pgmap v9371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:31 smithi104 ceph-mon[36624]: pgmap v9372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:31 smithi081 ceph-mon[31453]: pgmap v9372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:31 smithi153 ceph-mon[36569]: pgmap v9372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:33 smithi104 ceph-mon[36624]: pgmap v9373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:34.088 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:33 smithi081 ceph-mon[31453]: pgmap v9373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:33 smithi153 ceph-mon[36569]: pgmap v9373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:35.873+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:01:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:35.873+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:01:35.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:35.873+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:01:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:35 smithi081 ceph-mon[31453]: pgmap v9374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:35 smithi153 ceph-mon[36569]: pgmap v9374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:35 smithi104 ceph-mon[36624]: pgmap v9374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:37 smithi081 ceph-mon[31453]: pgmap v9375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:37 smithi153 ceph-mon[36569]: pgmap v9375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:37 smithi104 ceph-mon[36624]: pgmap v9375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:39 smithi081 ceph-mon[31453]: pgmap v9376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:39 smithi153 ceph-mon[36569]: pgmap v9376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:39 smithi104 ceph-mon[36624]: pgmap v9376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:41 smithi081 ceph-mon[31453]: pgmap v9377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:41 smithi153 ceph-mon[36569]: pgmap v9377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:41 smithi104 ceph-mon[36624]: pgmap v9377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:44.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:43 smithi153 ceph-mon[36569]: pgmap v9378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:43 smithi081 ceph-mon[31453]: pgmap v9378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:43 smithi104 ceph-mon[36624]: pgmap v9378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:45 smithi081 ceph-mon[31453]: pgmap v9379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:45 smithi153 ceph-mon[36569]: pgmap v9379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:45 smithi104 ceph-mon[36624]: pgmap v9379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:47 smithi081 ceph-mon[31453]: pgmap v9380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:47 smithi153 ceph-mon[36569]: pgmap v9380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:47 smithi104 ceph-mon[36624]: pgmap v9380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:49 smithi081 ceph-mon[31453]: pgmap v9381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:49 smithi153 ceph-mon[36569]: pgmap v9381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:49 smithi104 ceph-mon[36624]: pgmap v9381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:50.890+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:01:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:50.890+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:01:50.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:01:50.890+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:01:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:51 smithi081 ceph-mon[31453]: pgmap v9382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:51 smithi153 ceph-mon[36569]: pgmap v9382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:51 smithi104 ceph-mon[36624]: pgmap v9382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:53 smithi081 ceph-mon[31453]: pgmap v9383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:53 smithi153 ceph-mon[36569]: pgmap v9383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:53 smithi104 ceph-mon[36624]: pgmap v9383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:56.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:55 smithi104 ceph-mon[36624]: pgmap v9384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:55 smithi081 ceph-mon[31453]: pgmap v9384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:55 smithi153 ceph-mon[36569]: pgmap v9384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:57 smithi081 ceph-mon[31453]: pgmap v9385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:57 smithi153 ceph-mon[36569]: pgmap v9385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:01:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:57 smithi104 ceph-mon[36624]: pgmap v9385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:01:59 smithi081 ceph-mon[31453]: pgmap v9386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:01:59 smithi153 ceph-mon[36569]: pgmap v9386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:01:59 smithi104 ceph-mon[36624]: pgmap v9386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:01 smithi081 ceph-mon[31453]: pgmap v9387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:01 smithi153 ceph-mon[36569]: pgmap v9387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:01 smithi104 ceph-mon[36624]: pgmap v9387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:04.296 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:03 smithi081 ceph-mon[31453]: pgmap v9388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:03 smithi153 ceph-mon[36569]: pgmap v9388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:03 smithi104 ceph-mon[36624]: pgmap v9388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:05.905+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:02:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:05.905+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:02:05.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:05.905+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:02:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:05 smithi153 ceph-mon[36569]: pgmap v9389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:05 smithi104 ceph-mon[36624]: pgmap v9389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:05 smithi081 ceph-mon[31453]: pgmap v9389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:07 smithi153 ceph-mon[36569]: pgmap v9390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:07 smithi104 ceph-mon[36624]: pgmap v9390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:07 smithi081 ceph-mon[31453]: pgmap v9390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:09 smithi153 ceph-mon[36569]: pgmap v9391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:09 smithi104 ceph-mon[36624]: pgmap v9391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:09 smithi081 ceph-mon[31453]: pgmap v9391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:11 smithi153 ceph-mon[36569]: pgmap v9392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:11 smithi104 ceph-mon[36624]: pgmap v9392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:11 smithi081 ceph-mon[31453]: pgmap v9392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:13 smithi153 ceph-mon[36569]: pgmap v9393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:13 smithi104 ceph-mon[36624]: pgmap v9393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:13 smithi081 ceph-mon[31453]: pgmap v9393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:15 smithi153 ceph-mon[36569]: pgmap v9394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:15 smithi104 ceph-mon[36624]: pgmap v9394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:15 smithi081 ceph-mon[31453]: pgmap v9394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:17 smithi153 ceph-mon[36569]: pgmap v9395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:17 smithi104 ceph-mon[36624]: pgmap v9395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:17 smithi081 ceph-mon[31453]: pgmap v9395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:02:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:02:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:02:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:19 smithi153 ceph-mon[36569]: pgmap v9396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:02:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:02:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:02:20.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:19 smithi081 ceph-mon[31453]: pgmap v9396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:20.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:02:20.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:02:20.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:02:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:19 smithi104 ceph-mon[36624]: pgmap v9396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:02:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:02:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:02:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:20.913+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:02:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:20.913+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:02:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:20.913+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:02:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:20 smithi153 ceph-mon[36569]: pgmap v9397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:20 smithi153 ceph-mon[36569]: pgmap v9398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:20 smithi104 ceph-mon[36624]: pgmap v9397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:20 smithi104 ceph-mon[36624]: pgmap v9398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:20 smithi081 ceph-mon[31453]: pgmap v9397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:20 smithi081 ceph-mon[31453]: pgmap v9398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:22 smithi153 ceph-mon[36569]: pgmap v9399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:22 smithi104 ceph-mon[36624]: pgmap v9399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:22 smithi081 ceph-mon[31453]: pgmap v9399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:24 smithi153 ceph-mon[36569]: pgmap v9400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:24 smithi104 ceph-mon[36624]: pgmap v9400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:24 smithi081 ceph-mon[31453]: pgmap v9400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:26 smithi153 ceph-mon[36569]: pgmap v9401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:26 smithi104 ceph-mon[36624]: pgmap v9401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:26 smithi081 ceph-mon[31453]: pgmap v9401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:28 smithi153 ceph-mon[36569]: pgmap v9402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:28 smithi104 ceph-mon[36624]: pgmap v9402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:28 smithi081 ceph-mon[31453]: pgmap v9402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:30 smithi153 ceph-mon[36569]: pgmap v9403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:30 smithi104 ceph-mon[36624]: pgmap v9403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:30 smithi081 ceph-mon[31453]: pgmap v9403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:33 smithi153 ceph-mon[36569]: pgmap v9404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:33 smithi104 ceph-mon[36624]: pgmap v9404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:33 smithi081 ceph-mon[31453]: pgmap v9404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:35 smithi153 ceph-mon[36569]: pgmap v9405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:35 smithi104 ceph-mon[36624]: pgmap v9405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:35 smithi081 ceph-mon[31453]: pgmap v9405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:35.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:35.920+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:02:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:35.920+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:02:35.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:35.920+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:02:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:37 smithi153 ceph-mon[36569]: pgmap v9406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:37 smithi104 ceph-mon[36624]: pgmap v9406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:37 smithi081 ceph-mon[31453]: pgmap v9406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:39 smithi153 ceph-mon[36569]: pgmap v9407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:39 smithi104 ceph-mon[36624]: pgmap v9407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:39 smithi081 ceph-mon[31453]: pgmap v9407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:41 smithi153 ceph-mon[36569]: pgmap v9408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:41 smithi104 ceph-mon[36624]: pgmap v9408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:41 smithi081 ceph-mon[31453]: pgmap v9408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:43 smithi153 ceph-mon[36569]: pgmap v9409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:43 smithi104 ceph-mon[36624]: pgmap v9409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:43 smithi081 ceph-mon[31453]: pgmap v9409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:45 smithi153 ceph-mon[36569]: pgmap v9410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:45 smithi104 ceph-mon[36624]: pgmap v9410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:45 smithi081 ceph-mon[31453]: pgmap v9410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:47 smithi153 ceph-mon[36569]: pgmap v9411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:47 smithi104 ceph-mon[36624]: pgmap v9411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:47 smithi081 ceph-mon[31453]: pgmap v9411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:49 smithi153 ceph-mon[36569]: pgmap v9412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:49 smithi104 ceph-mon[36624]: pgmap v9412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:49 smithi081 ceph-mon[31453]: pgmap v9412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:50.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:50.930+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:02:50.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:50.931+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:02:50.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:02:50.931+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:02:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:51 smithi081 ceph-mon[31453]: pgmap v9413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:51 smithi153 ceph-mon[36569]: pgmap v9413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:51 smithi104 ceph-mon[36624]: pgmap v9413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:53 smithi153 ceph-mon[36569]: pgmap v9414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:53 smithi104 ceph-mon[36624]: pgmap v9414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:53 smithi081 ceph-mon[31453]: pgmap v9414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:55 smithi104 ceph-mon[36624]: pgmap v9415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:55 smithi081 ceph-mon[31453]: pgmap v9415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:55 smithi153 ceph-mon[36569]: pgmap v9415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:57 smithi104 ceph-mon[36624]: pgmap v9416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:57 smithi081 ceph-mon[31453]: pgmap v9416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:57 smithi153 ceph-mon[36569]: pgmap v9416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:02:59 smithi104 ceph-mon[36624]: pgmap v9417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:02:59 smithi081 ceph-mon[31453]: pgmap v9417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:02:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:02:59 smithi153 ceph-mon[36569]: pgmap v9417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:01 smithi104 ceph-mon[36624]: pgmap v9418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:01 smithi081 ceph-mon[31453]: pgmap v9418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:01 smithi153 ceph-mon[36569]: pgmap v9418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:03 smithi104 ceph-mon[36624]: pgmap v9419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:03 smithi081 ceph-mon[31453]: pgmap v9419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:03 smithi153 ceph-mon[36569]: pgmap v9419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:05 smithi081 ceph-mon[31453]: pgmap v9420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:05 smithi153 ceph-mon[36569]: pgmap v9420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:05 smithi104 ceph-mon[36624]: pgmap v9420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:05.936+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:03:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:05.936+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:03:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:05.936+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:03:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:07 smithi081 ceph-mon[31453]: pgmap v9421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:07 smithi153 ceph-mon[36569]: pgmap v9421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:07 smithi104 ceph-mon[36624]: pgmap v9421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:09 smithi081 ceph-mon[31453]: pgmap v9422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:09 smithi153 ceph-mon[36569]: pgmap v9422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:09 smithi104 ceph-mon[36624]: pgmap v9422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:11 smithi081 ceph-mon[31453]: pgmap v9423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:11 smithi153 ceph-mon[36569]: pgmap v9423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:11 smithi104 ceph-mon[36624]: pgmap v9423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:13 smithi081 ceph-mon[31453]: pgmap v9424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:13 smithi153 ceph-mon[36569]: pgmap v9424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:13 smithi104 ceph-mon[36624]: pgmap v9424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:15 smithi081 ceph-mon[31453]: pgmap v9425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:15 smithi153 ceph-mon[36569]: pgmap v9425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:15 smithi104 ceph-mon[36624]: pgmap v9425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:16 smithi153 ceph-mon[36569]: pgmap v9426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:16 smithi104 ceph-mon[36624]: pgmap v9426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:16 smithi081 ceph-mon[31453]: pgmap v9426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:18 smithi153 ceph-mon[36569]: pgmap v9427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:18 smithi104 ceph-mon[36624]: pgmap v9427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:18 smithi081 ceph-mon[31453]: pgmap v9427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:20.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:20.950+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:03:20.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:20.950+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:03:20.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:20.950+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:03:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: pgmap v9428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:03:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: pgmap v9428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:03:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: pgmap v9428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:03:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:03:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:03:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:03:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:03:22.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:03:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:22 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:03:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:22 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:03:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:22 smithi153 ceph-mon[36569]: pgmap v9429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:22 smithi153 ceph-mon[36569]: pgmap v9430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:22 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:03:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:22 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:03:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:22 smithi104 ceph-mon[36624]: pgmap v9429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:22 smithi104 ceph-mon[36624]: pgmap v9430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:22 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:03:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:22 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:03:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:22 smithi081 ceph-mon[31453]: pgmap v9429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:22 smithi081 ceph-mon[31453]: pgmap v9430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:24 smithi153 ceph-mon[36569]: pgmap v9431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:24 smithi104 ceph-mon[36624]: pgmap v9431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:24 smithi081 ceph-mon[31453]: pgmap v9431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:26 smithi153 ceph-mon[36569]: pgmap v9432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:26 smithi104 ceph-mon[36624]: pgmap v9432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:26 smithi081 ceph-mon[31453]: pgmap v9432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:28 smithi153 ceph-mon[36569]: pgmap v9433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:28 smithi104 ceph-mon[36624]: pgmap v9433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:28 smithi081 ceph-mon[31453]: pgmap v9433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:30 smithi153 ceph-mon[36569]: pgmap v9434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:30 smithi104 ceph-mon[36624]: pgmap v9434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:30 smithi081 ceph-mon[31453]: pgmap v9434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:32 smithi153 ceph-mon[36569]: pgmap v9435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:32 smithi104 ceph-mon[36624]: pgmap v9435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:32 smithi081 ceph-mon[31453]: pgmap v9435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:34 smithi153 ceph-mon[36569]: pgmap v9436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:34 smithi104 ceph-mon[36624]: pgmap v9436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:34 smithi081 ceph-mon[31453]: pgmap v9436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:35.952+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:03:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:35.952+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:03:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:35.952+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:03:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:36 smithi153 ceph-mon[36569]: pgmap v9437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:36 smithi104 ceph-mon[36624]: pgmap v9437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:36 smithi081 ceph-mon[31453]: pgmap v9437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:38 smithi104 ceph-mon[36624]: pgmap v9438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:38 smithi081 ceph-mon[31453]: pgmap v9438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:38 smithi153 ceph-mon[36569]: pgmap v9438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:40 smithi104 ceph-mon[36624]: pgmap v9439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:40 smithi081 ceph-mon[31453]: pgmap v9439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:40 smithi153 ceph-mon[36569]: pgmap v9439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:42 smithi104 ceph-mon[36624]: pgmap v9440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:42 smithi081 ceph-mon[31453]: pgmap v9440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:42 smithi153 ceph-mon[36569]: pgmap v9440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:44 smithi104 ceph-mon[36624]: pgmap v9441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:45.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:44 smithi153 ceph-mon[36569]: pgmap v9441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:44 smithi081 ceph-mon[31453]: pgmap v9441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:46 smithi104 ceph-mon[36624]: pgmap v9442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:46 smithi081 ceph-mon[31453]: pgmap v9442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:46 smithi153 ceph-mon[36569]: pgmap v9442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:48 smithi081 ceph-mon[31453]: pgmap v9443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:48 smithi153 ceph-mon[36569]: pgmap v9443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:48 smithi104 ceph-mon[36624]: pgmap v9443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:50.957+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:03:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:50.957+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:03:50.958 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:03:50.957+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:03:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:50 smithi081 ceph-mon[31453]: pgmap v9444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:50 smithi153 ceph-mon[36569]: pgmap v9444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:50 smithi104 ceph-mon[36624]: pgmap v9444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:52 smithi081 ceph-mon[31453]: pgmap v9445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:52 smithi153 ceph-mon[36569]: pgmap v9445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:52 smithi104 ceph-mon[36624]: pgmap v9445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:54 smithi081 ceph-mon[31453]: pgmap v9446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:54 smithi153 ceph-mon[36569]: pgmap v9446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:54 smithi104 ceph-mon[36624]: pgmap v9446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:56 smithi081 ceph-mon[31453]: pgmap v9447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:56 smithi153 ceph-mon[36569]: pgmap v9447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:56 smithi104 ceph-mon[36624]: pgmap v9447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:59.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:03:58 smithi104 ceph-mon[36624]: pgmap v9448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:03:58 smithi081 ceph-mon[31453]: pgmap v9448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:03:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:03:58 smithi153 ceph-mon[36569]: pgmap v9448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:00 smithi081 ceph-mon[31453]: pgmap v9449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:00 smithi153 ceph-mon[36569]: pgmap v9449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:00 smithi104 ceph-mon[36624]: pgmap v9449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:02 smithi081 ceph-mon[31453]: pgmap v9450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:02 smithi153 ceph-mon[36569]: pgmap v9450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:02 smithi104 ceph-mon[36624]: pgmap v9450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:04 smithi081 ceph-mon[31453]: pgmap v9451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:04 smithi153 ceph-mon[36569]: pgmap v9451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:04 smithi104 ceph-mon[36624]: pgmap v9451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:05.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:05.969+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:05.969+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:04:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:05.969+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:04:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:06 smithi081 ceph-mon[31453]: pgmap v9452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:06 smithi153 ceph-mon[36569]: pgmap v9452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:06 smithi104 ceph-mon[36624]: pgmap v9452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:08 smithi081 ceph-mon[31453]: pgmap v9453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:08 smithi153 ceph-mon[36569]: pgmap v9453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:08 smithi104 ceph-mon[36624]: pgmap v9453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:10 smithi081 ceph-mon[31453]: pgmap v9454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:10 smithi153 ceph-mon[36569]: pgmap v9454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:10 smithi104 ceph-mon[36624]: pgmap v9454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:12 smithi153 ceph-mon[36569]: pgmap v9455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:12 smithi104 ceph-mon[36624]: pgmap v9455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:12 smithi081 ceph-mon[31453]: pgmap v9455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:14 smithi153 ceph-mon[36569]: pgmap v9456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:14 smithi104 ceph-mon[36624]: pgmap v9456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:14 smithi081 ceph-mon[31453]: pgmap v9456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:15.557+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:04:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:15.557+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:04:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:15.557+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:04:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:15.557+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:04:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:16 smithi153 ceph-mon[36569]: pgmap v9457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:16 smithi104 ceph-mon[36624]: pgmap v9457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:16 smithi081 ceph-mon[31453]: pgmap v9457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:18 smithi153 ceph-mon[36569]: pgmap v9458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:18 smithi104 ceph-mon[36624]: pgmap v9458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:18 smithi081 ceph-mon[31453]: pgmap v9458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:20.977+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:04:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:20.977+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:04:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:20.977+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:04:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:20 smithi153 ceph-mon[36569]: pgmap v9459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:20 smithi104 ceph-mon[36624]: pgmap v9459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:20 smithi081 ceph-mon[31453]: pgmap v9459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:04:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:04:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:04:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:22 smithi153 ceph-mon[36569]: pgmap v9460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:04:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:04:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:04:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:22 smithi104 ceph-mon[36624]: pgmap v9460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:04:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:04:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:04:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:22 smithi081 ceph-mon[31453]: pgmap v9460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:04:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:04:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:04:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:23 smithi153 ceph-mon[36569]: pgmap v9461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:23 smithi153 ceph-mon[36569]: pgmap v9462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:24.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:23 smithi081 ceph-mon[31453]: pgmap v9461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:24.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:23 smithi081 ceph-mon[31453]: pgmap v9462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:23 smithi104 ceph-mon[36624]: pgmap v9461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:23 smithi104 ceph-mon[36624]: pgmap v9462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:25 smithi153 ceph-mon[36569]: pgmap v9463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:25 smithi104 ceph-mon[36624]: pgmap v9463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:25 smithi081 ceph-mon[31453]: pgmap v9463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:28.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:27 smithi081 ceph-mon[31453]: pgmap v9464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:27 smithi153 ceph-mon[36569]: pgmap v9464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:27 smithi104 ceph-mon[36624]: pgmap v9464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:29 smithi153 ceph-mon[36569]: pgmap v9465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:29 smithi104 ceph-mon[36624]: pgmap v9465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:29 smithi081 ceph-mon[31453]: pgmap v9465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:31 smithi153 ceph-mon[36569]: pgmap v9466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:31 smithi104 ceph-mon[36624]: pgmap v9466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:31 smithi081 ceph-mon[31453]: pgmap v9466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:33 smithi153 ceph-mon[36569]: pgmap v9467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:33 smithi104 ceph-mon[36624]: pgmap v9467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:33 smithi081 ceph-mon[31453]: pgmap v9467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:35.992+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:04:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:35.992+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:04:35.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:35.992+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:04:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:35 smithi153 ceph-mon[36569]: pgmap v9468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:35 smithi104 ceph-mon[36624]: pgmap v9468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:35 smithi081 ceph-mon[31453]: pgmap v9468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:37 smithi153 ceph-mon[36569]: pgmap v9469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:37 smithi104 ceph-mon[36624]: pgmap v9469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:37 smithi081 ceph-mon[31453]: pgmap v9469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:39 smithi153 ceph-mon[36569]: pgmap v9470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:39 smithi104 ceph-mon[36624]: pgmap v9470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:39 smithi081 ceph-mon[31453]: pgmap v9470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:41 smithi153 ceph-mon[36569]: pgmap v9471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:41 smithi104 ceph-mon[36624]: pgmap v9471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:41 smithi081 ceph-mon[31453]: pgmap v9471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:44 smithi153 ceph-mon[36569]: pgmap v9472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:44 smithi104 ceph-mon[36624]: pgmap v9472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:44 smithi081 ceph-mon[31453]: pgmap v9472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:46 smithi153 ceph-mon[36569]: pgmap v9473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:46 smithi104 ceph-mon[36624]: pgmap v9473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:46 smithi081 ceph-mon[31453]: pgmap v9473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:48 smithi153 ceph-mon[36569]: pgmap v9474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:48 smithi104 ceph-mon[36624]: pgmap v9474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:48 smithi081 ceph-mon[31453]: pgmap v9474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:50 smithi153 ceph-mon[36569]: pgmap v9475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:50 smithi104 ceph-mon[36624]: pgmap v9475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:50 smithi081 ceph-mon[31453]: pgmap v9475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:50.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:50.996+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:04:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:50.996+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:04:50.997 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:04:50.996+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:04:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:52 smithi153 ceph-mon[36569]: pgmap v9476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:52 smithi104 ceph-mon[36624]: pgmap v9476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:52 smithi081 ceph-mon[31453]: pgmap v9476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:54.280 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:54 smithi081 ceph-mon[31453]: pgmap v9477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:54 smithi153 ceph-mon[36569]: pgmap v9477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:54 smithi104 ceph-mon[36624]: pgmap v9477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:56 smithi153 ceph-mon[36569]: pgmap v9478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:56 smithi104 ceph-mon[36624]: pgmap v9478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:56 smithi081 ceph-mon[31453]: pgmap v9478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:04:58 smithi153 ceph-mon[36569]: pgmap v9479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:04:58 smithi104 ceph-mon[36624]: pgmap v9479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:04:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:04:58 smithi081 ceph-mon[31453]: pgmap v9479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:00 smithi153 ceph-mon[36569]: pgmap v9480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:00 smithi104 ceph-mon[36624]: pgmap v9480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:00 smithi081 ceph-mon[31453]: pgmap v9480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:00.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:00.564+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:05:00.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:00.564+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:05:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:00.564+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:05:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:00.564+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:05:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:02 smithi153 ceph-mon[36569]: pgmap v9481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:02 smithi104 ceph-mon[36624]: pgmap v9481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:02 smithi081 ceph-mon[31453]: pgmap v9481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:04 smithi153 ceph-mon[36569]: pgmap v9482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:04 smithi104 ceph-mon[36624]: pgmap v9482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:04 smithi081 ceph-mon[31453]: pgmap v9482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:06.000+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:05:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:06.000+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:05:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:06.000+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:05:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:06 smithi104 ceph-mon[36624]: pgmap v9483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:06 smithi081 ceph-mon[31453]: pgmap v9483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:06 smithi153 ceph-mon[36569]: pgmap v9483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:08 smithi104 ceph-mon[36624]: pgmap v9484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:08 smithi081 ceph-mon[31453]: pgmap v9484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:08 smithi153 ceph-mon[36569]: pgmap v9484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:10 smithi104 ceph-mon[36624]: pgmap v9485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:10 smithi081 ceph-mon[31453]: pgmap v9485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:10 smithi153 ceph-mon[36569]: pgmap v9485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:12 smithi104 ceph-mon[36624]: pgmap v9486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:12 smithi081 ceph-mon[31453]: pgmap v9486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:12 smithi153 ceph-mon[36569]: pgmap v9486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:14 smithi104 ceph-mon[36624]: pgmap v9487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:14 smithi081 ceph-mon[31453]: pgmap v9487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:14 smithi153 ceph-mon[36569]: pgmap v9487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:16 smithi153 ceph-mon[36569]: pgmap v9488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:16 smithi081 ceph-mon[31453]: pgmap v9488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:16 smithi104 ceph-mon[36624]: pgmap v9488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:18 smithi081 ceph-mon[31453]: pgmap v9489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:18 smithi153 ceph-mon[36569]: pgmap v9489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:18 smithi104 ceph-mon[36624]: pgmap v9489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:20 smithi081 ceph-mon[31453]: pgmap v9490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:20 smithi153 ceph-mon[36569]: pgmap v9490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:20 smithi104 ceph-mon[36624]: pgmap v9490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:21.007+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:05:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:21.007+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:05:21.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:21.007+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:05:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:22 smithi081 ceph-mon[31453]: pgmap v9491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:22 smithi153 ceph-mon[36569]: pgmap v9491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:22 smithi104 ceph-mon[36624]: pgmap v9491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:05:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:05:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:05:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:24 smithi081 ceph-mon[31453]: pgmap v9492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:24 smithi153 ceph-mon[36569]: pgmap v9492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:24 smithi104 ceph-mon[36624]: pgmap v9492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:05:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:05:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:05:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:05:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:05:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:05:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:05:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:05:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:05:26.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:26 smithi081 ceph-mon[31453]: pgmap v9493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:26.309 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:26 smithi081 ceph-mon[31453]: pgmap v9494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:26 smithi153 ceph-mon[36569]: pgmap v9493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:26 smithi153 ceph-mon[36569]: pgmap v9494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:26 smithi104 ceph-mon[36624]: pgmap v9493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:26 smithi104 ceph-mon[36624]: pgmap v9494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:28 smithi081 ceph-mon[31453]: pgmap v9495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:28 smithi153 ceph-mon[36569]: pgmap v9495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:28 smithi104 ceph-mon[36624]: pgmap v9495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:30 smithi081 ceph-mon[31453]: pgmap v9496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:30 smithi153 ceph-mon[36569]: pgmap v9496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:30 smithi104 ceph-mon[36624]: pgmap v9496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:32 smithi081 ceph-mon[31453]: pgmap v9497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:32 smithi153 ceph-mon[36569]: pgmap v9497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:32 smithi104 ceph-mon[36624]: pgmap v9497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:33 smithi081 ceph-mon[31453]: pgmap v9498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:33 smithi153 ceph-mon[36569]: pgmap v9498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:33 smithi104 ceph-mon[36624]: pgmap v9498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:35 smithi081 ceph-mon[31453]: pgmap v9499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:35 smithi153 ceph-mon[36569]: pgmap v9499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:35 smithi104 ceph-mon[36624]: pgmap v9499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:36.020+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:05:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:36.020+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:05:36.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:36.020+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:05:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:37 smithi081 ceph-mon[31453]: pgmap v9500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:37 smithi153 ceph-mon[36569]: pgmap v9500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:37 smithi104 ceph-mon[36624]: pgmap v9500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:39 smithi081 ceph-mon[31453]: pgmap v9501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:39 smithi153 ceph-mon[36569]: pgmap v9501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:39 smithi104 ceph-mon[36624]: pgmap v9501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:41 smithi081 ceph-mon[31453]: pgmap v9502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:41 smithi153 ceph-mon[36569]: pgmap v9502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:41 smithi104 ceph-mon[36624]: pgmap v9502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:43 smithi081 ceph-mon[31453]: pgmap v9503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:43 smithi153 ceph-mon[36569]: pgmap v9503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:43 smithi104 ceph-mon[36624]: pgmap v9503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:45 smithi153 ceph-mon[36569]: pgmap v9504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:45 smithi104 ceph-mon[36624]: pgmap v9504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:45 smithi081 ceph-mon[31453]: pgmap v9504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:47 smithi153 ceph-mon[36569]: pgmap v9505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:47 smithi104 ceph-mon[36624]: pgmap v9505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:47 smithi081 ceph-mon[31453]: pgmap v9505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:49 smithi153 ceph-mon[36569]: pgmap v9506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:49 smithi104 ceph-mon[36624]: pgmap v9506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:49 smithi081 ceph-mon[31453]: pgmap v9506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:51.030+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:05:51.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:51.030+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:05:51.032 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:05:51.030+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:05:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:51 smithi153 ceph-mon[36569]: pgmap v9507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:51 smithi104 ceph-mon[36624]: pgmap v9507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:51 smithi081 ceph-mon[31453]: pgmap v9507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:53 smithi153 ceph-mon[36569]: pgmap v9508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:53 smithi104 ceph-mon[36624]: pgmap v9508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:53 smithi081 ceph-mon[31453]: pgmap v9508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:55 smithi153 ceph-mon[36569]: pgmap v9509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:55 smithi104 ceph-mon[36624]: pgmap v9509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:55 smithi081 ceph-mon[31453]: pgmap v9509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:57 smithi153 ceph-mon[36569]: pgmap v9510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:57 smithi104 ceph-mon[36624]: pgmap v9510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:57 smithi081 ceph-mon[31453]: pgmap v9510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:59.710 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:05:59 smithi081 ceph-mon[31453]: pgmap v9511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:05:59 smithi153 ceph-mon[36569]: pgmap v9511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:05:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:05:59 smithi104 ceph-mon[36624]: pgmap v9511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:01 smithi153 ceph-mon[36569]: pgmap v9512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:01 smithi104 ceph-mon[36624]: pgmap v9512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:01 smithi081 ceph-mon[31453]: pgmap v9512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:03 smithi153 ceph-mon[36569]: pgmap v9513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:03 smithi104 ceph-mon[36624]: pgmap v9513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:03 smithi081 ceph-mon[31453]: pgmap v9513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:05 smithi153 ceph-mon[36569]: pgmap v9514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:05 smithi104 ceph-mon[36624]: pgmap v9514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:05 smithi081 ceph-mon[31453]: pgmap v9514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:06.046+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:06:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:06.046+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:06:06.047 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:06.046+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:06:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:07 smithi153 ceph-mon[36569]: pgmap v9515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:07 smithi104 ceph-mon[36624]: pgmap v9515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:07 smithi081 ceph-mon[31453]: pgmap v9515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:09 smithi153 ceph-mon[36569]: pgmap v9516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:09 smithi104 ceph-mon[36624]: pgmap v9516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:09 smithi081 ceph-mon[31453]: pgmap v9516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:11 smithi153 ceph-mon[36569]: pgmap v9517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:11 smithi104 ceph-mon[36624]: pgmap v9517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:11 smithi081 ceph-mon[31453]: pgmap v9517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:13 smithi153 ceph-mon[36569]: pgmap v9518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:13 smithi104 ceph-mon[36624]: pgmap v9518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:13 smithi081 ceph-mon[31453]: pgmap v9518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:15 smithi153 ceph-mon[36569]: pgmap v9519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:15 smithi104 ceph-mon[36624]: pgmap v9519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:15 smithi081 ceph-mon[31453]: pgmap v9519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:17 smithi153 ceph-mon[36569]: pgmap v9520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:17 smithi104 ceph-mon[36624]: pgmap v9520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:17 smithi081 ceph-mon[31453]: pgmap v9520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:19 smithi153 ceph-mon[36569]: pgmap v9521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:19 smithi104 ceph-mon[36624]: pgmap v9521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:19 smithi081 ceph-mon[31453]: pgmap v9521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:21.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:21.062+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:06:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:21.062+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:06:21.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:21.062+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:06:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:21 smithi153 ceph-mon[36569]: pgmap v9522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:21 smithi104 ceph-mon[36624]: pgmap v9522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:21 smithi081 ceph-mon[31453]: pgmap v9522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:23 smithi153 ceph-mon[36569]: pgmap v9523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:23 smithi104 ceph-mon[36624]: pgmap v9523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:23 smithi081 ceph-mon[31453]: pgmap v9523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:25 smithi153 ceph-mon[36569]: pgmap v9524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:06:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:25 smithi104 ceph-mon[36624]: pgmap v9524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:06:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:25 smithi081 ceph-mon[31453]: pgmap v9524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:06:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:06:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:06:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:26 smithi153 ceph-mon[36569]: pgmap v9525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:26 smithi153 ceph-mon[36569]: pgmap v9526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:06:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:06:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:06:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:26 smithi104 ceph-mon[36624]: pgmap v9525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:26 smithi104 ceph-mon[36624]: pgmap v9526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:06:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:06:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:06:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:26 smithi081 ceph-mon[31453]: pgmap v9525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:26 smithi081 ceph-mon[31453]: pgmap v9526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:06:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:28 smithi153 ceph-mon[36569]: pgmap v9527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:28 smithi104 ceph-mon[36624]: pgmap v9527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:28 smithi081 ceph-mon[31453]: pgmap v9527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:31 smithi153 ceph-mon[36569]: pgmap v9528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:31 smithi104 ceph-mon[36624]: pgmap v9528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:31 smithi081 ceph-mon[31453]: pgmap v9528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:33 smithi153 ceph-mon[36569]: pgmap v9529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:33 smithi104 ceph-mon[36624]: pgmap v9529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:33 smithi081 ceph-mon[31453]: pgmap v9529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:35 smithi153 ceph-mon[36569]: pgmap v9530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:35 smithi104 ceph-mon[36624]: pgmap v9530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:35 smithi081 ceph-mon[31453]: pgmap v9530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:36.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:36.078+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:06:36.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:36.078+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:06:36.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:36.078+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:06:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:37 smithi153 ceph-mon[36569]: pgmap v9531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:37 smithi104 ceph-mon[36624]: pgmap v9531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:37 smithi081 ceph-mon[31453]: pgmap v9531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:39 smithi153 ceph-mon[36569]: pgmap v9532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:39 smithi104 ceph-mon[36624]: pgmap v9532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:39 smithi081 ceph-mon[31453]: pgmap v9532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:41 smithi153 ceph-mon[36569]: pgmap v9533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:41 smithi104 ceph-mon[36624]: pgmap v9533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:41 smithi081 ceph-mon[31453]: pgmap v9533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:43 smithi153 ceph-mon[36569]: pgmap v9534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:43 smithi104 ceph-mon[36624]: pgmap v9534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:43.336 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:43 smithi081 ceph-mon[31453]: pgmap v9534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:45 smithi153 ceph-mon[36569]: pgmap v9535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:45 smithi104 ceph-mon[36624]: pgmap v9535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:45 smithi081 ceph-mon[31453]: pgmap v9535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:47 smithi153 ceph-mon[36569]: pgmap v9536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:47 smithi104 ceph-mon[36624]: pgmap v9536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:47 smithi081 ceph-mon[31453]: pgmap v9536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:49 smithi153 ceph-mon[36569]: pgmap v9537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:49 smithi104 ceph-mon[36624]: pgmap v9537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:49 smithi081 ceph-mon[31453]: pgmap v9537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:51.091+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:06:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:51.092+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:06:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:06:51.092+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:06:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:51 smithi153 ceph-mon[36569]: pgmap v9538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:51 smithi104 ceph-mon[36624]: pgmap v9538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:51 smithi081 ceph-mon[31453]: pgmap v9538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:53 smithi104 ceph-mon[36624]: pgmap v9539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:53 smithi081 ceph-mon[31453]: pgmap v9539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:53 smithi153 ceph-mon[36569]: pgmap v9539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:55 smithi104 ceph-mon[36624]: pgmap v9540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:55 smithi081 ceph-mon[31453]: pgmap v9540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:55 smithi153 ceph-mon[36569]: pgmap v9540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:57 smithi104 ceph-mon[36624]: pgmap v9541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:57 smithi081 ceph-mon[31453]: pgmap v9541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:57 smithi153 ceph-mon[36569]: pgmap v9541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:06:59 smithi104 ceph-mon[36624]: pgmap v9542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:06:59 smithi081 ceph-mon[31453]: pgmap v9542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:06:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:06:59 smithi153 ceph-mon[36569]: pgmap v9542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:01 smithi104 ceph-mon[36624]: pgmap v9543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:01 smithi081 ceph-mon[31453]: pgmap v9543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:01 smithi153 ceph-mon[36569]: pgmap v9543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:03 smithi081 ceph-mon[31453]: pgmap v9544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:03 smithi153 ceph-mon[36569]: pgmap v9544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:03 smithi104 ceph-mon[36624]: pgmap v9544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:05 smithi081 ceph-mon[31453]: pgmap v9545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:05 smithi153 ceph-mon[36569]: pgmap v9545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:05 smithi104 ceph-mon[36624]: pgmap v9545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:06.107+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:07:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:06.107+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:07:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:06.107+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:07:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:07 smithi081 ceph-mon[31453]: pgmap v9546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:07 smithi153 ceph-mon[36569]: pgmap v9546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:07 smithi104 ceph-mon[36624]: pgmap v9546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:09 smithi081 ceph-mon[31453]: pgmap v9547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:09 smithi153 ceph-mon[36569]: pgmap v9547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:09 smithi104 ceph-mon[36624]: pgmap v9547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:11 smithi081 ceph-mon[31453]: pgmap v9548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:11 smithi153 ceph-mon[36569]: pgmap v9548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:11 smithi104 ceph-mon[36624]: pgmap v9548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:13 smithi081 ceph-mon[31453]: pgmap v9549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:13 smithi153 ceph-mon[36569]: pgmap v9549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:13 smithi104 ceph-mon[36624]: pgmap v9549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:15 smithi081 ceph-mon[31453]: pgmap v9550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:15 smithi153 ceph-mon[36569]: pgmap v9550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:15 smithi104 ceph-mon[36624]: pgmap v9550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:15.587 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:15.587+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:07:15.587 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:15.587+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:07:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:15.588+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:07:15.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:15.588+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:07:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:17 smithi081 ceph-mon[31453]: pgmap v9551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:17 smithi153 ceph-mon[36569]: pgmap v9551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:17 smithi104 ceph-mon[36624]: pgmap v9551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:19 smithi081 ceph-mon[31453]: pgmap v9552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:19 smithi153 ceph-mon[36569]: pgmap v9552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:19 smithi104 ceph-mon[36624]: pgmap v9552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:21.121+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:07:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:21.121+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:07:21.121 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:21.121+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:07:21.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:21 smithi081 ceph-mon[31453]: pgmap v9553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:21 smithi153 ceph-mon[36569]: pgmap v9553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:21 smithi104 ceph-mon[36624]: pgmap v9553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:23 smithi081 ceph-mon[31453]: pgmap v9554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:23 smithi153 ceph-mon[36569]: pgmap v9554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:23 smithi104 ceph-mon[36624]: pgmap v9554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:25 smithi081 ceph-mon[31453]: pgmap v9555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:25 smithi153 ceph-mon[36569]: pgmap v9555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:25 smithi104 ceph-mon[36624]: pgmap v9555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:27.334 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:27 smithi153 ceph-mon[36569]: pgmap v9556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:27.334 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:07:27.335 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.335 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.335 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.335 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:27 smithi081 ceph-mon[31453]: pgmap v9556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:07:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:27 smithi104 ceph-mon[36624]: pgmap v9556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:27.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:07:27.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:27.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:29 smithi081 ceph-mon[31453]: pgmap v9557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:29 smithi153 ceph-mon[36569]: pgmap v9557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:29 smithi104 ceph-mon[36624]: pgmap v9557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:31 smithi081 ceph-mon[31453]: pgmap v9558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:31 smithi153 ceph-mon[36569]: pgmap v9558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:31 smithi104 ceph-mon[36624]: pgmap v9558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:32 smithi081 ceph-mon[31453]: pgmap v9559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:32 smithi153 ceph-mon[36569]: pgmap v9559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:32 smithi104 ceph-mon[36624]: pgmap v9559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:07:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:07:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:34 smithi081 ceph-mon[31453]: pgmap v9560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:34 smithi081 ceph-mon[31453]: pgmap v9561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:07:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:07:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:34 smithi153 ceph-mon[36569]: pgmap v9560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:34.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:34 smithi153 ceph-mon[36569]: pgmap v9561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:34.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:07:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:07:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:34 smithi104 ceph-mon[36624]: pgmap v9560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:34 smithi104 ceph-mon[36624]: pgmap v9561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:07:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:36.127+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:07:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:36.127+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:07:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:36.127+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:07:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:36 smithi153 ceph-mon[36569]: pgmap v9562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:36 smithi104 ceph-mon[36624]: pgmap v9562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:36 smithi081 ceph-mon[31453]: pgmap v9562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:38 smithi153 ceph-mon[36569]: pgmap v9563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:38 smithi104 ceph-mon[36624]: pgmap v9563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:38 smithi081 ceph-mon[31453]: pgmap v9563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:40 smithi153 ceph-mon[36569]: pgmap v9564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:40 smithi104 ceph-mon[36624]: pgmap v9564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:40 smithi081 ceph-mon[31453]: pgmap v9564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:42 smithi153 ceph-mon[36569]: pgmap v9565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:42 smithi104 ceph-mon[36624]: pgmap v9565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:42 smithi081 ceph-mon[31453]: pgmap v9565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:44 smithi153 ceph-mon[36569]: pgmap v9566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:44 smithi104 ceph-mon[36624]: pgmap v9566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:44 smithi081 ceph-mon[31453]: pgmap v9566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:46 smithi153 ceph-mon[36569]: pgmap v9567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:46 smithi104 ceph-mon[36624]: pgmap v9567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:46 smithi081 ceph-mon[31453]: pgmap v9567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:48 smithi153 ceph-mon[36569]: pgmap v9568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:48 smithi104 ceph-mon[36624]: pgmap v9568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:48 smithi081 ceph-mon[31453]: pgmap v9568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:50 smithi153 ceph-mon[36569]: pgmap v9569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:50 smithi104 ceph-mon[36624]: pgmap v9569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:50 smithi081 ceph-mon[31453]: pgmap v9569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:51.143+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:07:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:51.143+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:07:51.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:07:51.143+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:07:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:52 smithi153 ceph-mon[36569]: pgmap v9570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:52 smithi104 ceph-mon[36624]: pgmap v9570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:52 smithi081 ceph-mon[31453]: pgmap v9570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:54 smithi153 ceph-mon[36569]: pgmap v9571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:54 smithi104 ceph-mon[36624]: pgmap v9571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:54 smithi081 ceph-mon[31453]: pgmap v9571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:56 smithi153 ceph-mon[36569]: pgmap v9572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:56 smithi104 ceph-mon[36624]: pgmap v9572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:56 smithi081 ceph-mon[31453]: pgmap v9572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:07:58 smithi153 ceph-mon[36569]: pgmap v9573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:07:58 smithi104 ceph-mon[36624]: pgmap v9573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:07:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:07:58 smithi081 ceph-mon[31453]: pgmap v9573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:00 smithi153 ceph-mon[36569]: pgmap v9574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:00 smithi104 ceph-mon[36624]: pgmap v9574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:00 smithi081 ceph-mon[31453]: pgmap v9574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:02 smithi081 ceph-mon[31453]: pgmap v9575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:02 smithi153 ceph-mon[36569]: pgmap v9575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:02 smithi104 ceph-mon[36624]: pgmap v9575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:04 smithi153 ceph-mon[36569]: pgmap v9576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:04 smithi104 ceph-mon[36624]: pgmap v9576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:04 smithi081 ceph-mon[31453]: pgmap v9576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:06.147+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:08:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:06.147+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:08:06.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:06.147+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:08:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:06 smithi153 ceph-mon[36569]: pgmap v9577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:06 smithi104 ceph-mon[36624]: pgmap v9577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:06 smithi081 ceph-mon[31453]: pgmap v9577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:08 smithi153 ceph-mon[36569]: pgmap v9578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:08 smithi104 ceph-mon[36624]: pgmap v9578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:08 smithi081 ceph-mon[31453]: pgmap v9578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:10 smithi153 ceph-mon[36569]: pgmap v9579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:10 smithi104 ceph-mon[36624]: pgmap v9579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:10 smithi081 ceph-mon[31453]: pgmap v9579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:12 smithi153 ceph-mon[36569]: pgmap v9580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:12 smithi104 ceph-mon[36624]: pgmap v9580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:12 smithi081 ceph-mon[31453]: pgmap v9580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:14 smithi153 ceph-mon[36569]: pgmap v9581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:14 smithi104 ceph-mon[36624]: pgmap v9581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:14 smithi081 ceph-mon[31453]: pgmap v9581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:16 smithi153 ceph-mon[36569]: pgmap v9582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:16 smithi104 ceph-mon[36624]: pgmap v9582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:16 smithi081 ceph-mon[31453]: pgmap v9582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:18 smithi104 ceph-mon[36624]: pgmap v9583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:18 smithi081 ceph-mon[31453]: pgmap v9583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:18 smithi153 ceph-mon[36569]: pgmap v9583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:20 smithi104 ceph-mon[36624]: pgmap v9584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:20 smithi081 ceph-mon[31453]: pgmap v9584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:20 smithi153 ceph-mon[36569]: pgmap v9584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:21.162 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:21.161+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:08:21.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:21.162+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:08:21.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:21.162+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:08:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:22 smithi104 ceph-mon[36624]: pgmap v9585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:22 smithi081 ceph-mon[31453]: pgmap v9585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:22 smithi153 ceph-mon[36569]: pgmap v9585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:24 smithi153 ceph-mon[36569]: pgmap v9586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:24 smithi104 ceph-mon[36624]: pgmap v9586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:24 smithi081 ceph-mon[31453]: pgmap v9586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:26 smithi153 ceph-mon[36569]: pgmap v9587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:26 smithi104 ceph-mon[36624]: pgmap v9587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:26 smithi081 ceph-mon[31453]: pgmap v9587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:28 smithi153 ceph-mon[36569]: pgmap v9588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:28 smithi104 ceph-mon[36624]: pgmap v9588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:28 smithi081 ceph-mon[31453]: pgmap v9588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:30 smithi153 ceph-mon[36569]: pgmap v9589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:30 smithi104 ceph-mon[36624]: pgmap v9589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:30 smithi081 ceph-mon[31453]: pgmap v9589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:32 smithi153 ceph-mon[36569]: pgmap v9590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:32 smithi104 ceph-mon[36624]: pgmap v9590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:32 smithi081 ceph-mon[31453]: pgmap v9590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:08:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:08:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:08:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:34 smithi153 ceph-mon[36569]: pgmap v9591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:34 smithi104 ceph-mon[36624]: pgmap v9591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:34 smithi081 ceph-mon[31453]: pgmap v9591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:36.177+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:08:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:36.177+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:08:36.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:36.177+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:08:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:08:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:08:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:08:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:08:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:08:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:08:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:08:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:37 smithi153 ceph-mon[36569]: pgmap v9592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:37 smithi153 ceph-mon[36569]: pgmap v9593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:37 smithi153 ceph-mon[36569]: pgmap v9594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:37 smithi104 ceph-mon[36624]: pgmap v9592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:37 smithi104 ceph-mon[36624]: pgmap v9593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:37 smithi104 ceph-mon[36624]: pgmap v9594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:37 smithi081 ceph-mon[31453]: pgmap v9592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:37 smithi081 ceph-mon[31453]: pgmap v9593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:37 smithi081 ceph-mon[31453]: pgmap v9594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:39 smithi153 ceph-mon[36569]: pgmap v9595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:39 smithi104 ceph-mon[36624]: pgmap v9595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:39 smithi081 ceph-mon[31453]: pgmap v9595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:41 smithi153 ceph-mon[36569]: pgmap v9596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:41 smithi104 ceph-mon[36624]: pgmap v9596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:41 smithi081 ceph-mon[31453]: pgmap v9596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:43 smithi153 ceph-mon[36569]: pgmap v9597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:43 smithi104 ceph-mon[36624]: pgmap v9597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:43 smithi081 ceph-mon[31453]: pgmap v9597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:45 smithi153 ceph-mon[36569]: pgmap v9598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:45 smithi104 ceph-mon[36624]: pgmap v9598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:45 smithi081 ceph-mon[31453]: pgmap v9598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:47 smithi153 ceph-mon[36569]: pgmap v9599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:47 smithi104 ceph-mon[36624]: pgmap v9599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:47 smithi081 ceph-mon[31453]: pgmap v9599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:49 smithi104 ceph-mon[36624]: pgmap v9600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:49 smithi081 ceph-mon[31453]: pgmap v9600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:49 smithi153 ceph-mon[36569]: pgmap v9600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:51.192+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:08:51.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:51.192+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:08:51.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:08:51.193+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:08:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:51 smithi104 ceph-mon[36624]: pgmap v9601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:51 smithi081 ceph-mon[31453]: pgmap v9601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:51 smithi153 ceph-mon[36569]: pgmap v9601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:53 smithi104 ceph-mon[36624]: pgmap v9602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:53 smithi081 ceph-mon[31453]: pgmap v9602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:53 smithi153 ceph-mon[36569]: pgmap v9602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:55 smithi104 ceph-mon[36624]: pgmap v9603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:55 smithi081 ceph-mon[31453]: pgmap v9603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:55 smithi153 ceph-mon[36569]: pgmap v9603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:57 smithi104 ceph-mon[36624]: pgmap v9604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:57 smithi081 ceph-mon[31453]: pgmap v9604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:57 smithi153 ceph-mon[36569]: pgmap v9604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:08:59 smithi081 ceph-mon[31453]: pgmap v9605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:08:59 smithi153 ceph-mon[36569]: pgmap v9605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:08:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:08:59 smithi104 ceph-mon[36624]: pgmap v9605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:01 smithi081 ceph-mon[31453]: pgmap v9606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:01 smithi153 ceph-mon[36569]: pgmap v9606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:01 smithi104 ceph-mon[36624]: pgmap v9606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:03.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:03 smithi104 ceph-mon[36624]: pgmap v9607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:03 smithi081 ceph-mon[31453]: pgmap v9607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:03 smithi153 ceph-mon[36569]: pgmap v9607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:05 smithi081 ceph-mon[31453]: pgmap v9608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:05 smithi153 ceph-mon[36569]: pgmap v9608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:05 smithi104 ceph-mon[36624]: pgmap v9608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:06.208+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:09:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:06.208+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:09:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:06.208+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:09:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:07 smithi081 ceph-mon[31453]: pgmap v9609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:07 smithi153 ceph-mon[36569]: pgmap v9609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:07 smithi104 ceph-mon[36624]: pgmap v9609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:09 smithi081 ceph-mon[31453]: pgmap v9610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:09 smithi153 ceph-mon[36569]: pgmap v9610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:09 smithi104 ceph-mon[36624]: pgmap v9610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:11 smithi081 ceph-mon[31453]: pgmap v9611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:11 smithi153 ceph-mon[36569]: pgmap v9611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:11 smithi104 ceph-mon[36624]: pgmap v9611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:13 smithi081 ceph-mon[31453]: pgmap v9612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:13 smithi153 ceph-mon[36569]: pgmap v9612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:13 smithi104 ceph-mon[36624]: pgmap v9612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:15 smithi081 ceph-mon[31453]: pgmap v9613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:15 smithi153 ceph-mon[36569]: pgmap v9613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:15 smithi104 ceph-mon[36624]: pgmap v9613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:17 smithi081 ceph-mon[31453]: pgmap v9614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:17 smithi153 ceph-mon[36569]: pgmap v9614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:17 smithi104 ceph-mon[36624]: pgmap v9614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:19 smithi081 ceph-mon[31453]: pgmap v9615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:19 smithi153 ceph-mon[36569]: pgmap v9615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:19 smithi104 ceph-mon[36624]: pgmap v9615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:21.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:21.223+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:09:21.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:21.223+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:09:21.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:21.223+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:09:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:21 smithi081 ceph-mon[31453]: pgmap v9616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:21 smithi153 ceph-mon[36569]: pgmap v9616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:21 smithi104 ceph-mon[36624]: pgmap v9616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:23 smithi081 ceph-mon[31453]: pgmap v9617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:23 smithi153 ceph-mon[36569]: pgmap v9617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:23 smithi104 ceph-mon[36624]: pgmap v9617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:25 smithi153 ceph-mon[36569]: pgmap v9618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:25 smithi104 ceph-mon[36624]: pgmap v9618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:25 smithi081 ceph-mon[31453]: pgmap v9618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:27 smithi153 ceph-mon[36569]: pgmap v9619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:27 smithi104 ceph-mon[36624]: pgmap v9619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:27 smithi081 ceph-mon[31453]: pgmap v9619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:29 smithi153 ceph-mon[36569]: pgmap v9620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:29 smithi104 ceph-mon[36624]: pgmap v9620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:29 smithi081 ceph-mon[31453]: pgmap v9620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:31.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:31 smithi081 ceph-mon[31453]: pgmap v9621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:31 smithi153 ceph-mon[36569]: pgmap v9621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:31 smithi104 ceph-mon[36624]: pgmap v9621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:33 smithi153 ceph-mon[36569]: pgmap v9622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:33 smithi104 ceph-mon[36624]: pgmap v9622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:33 smithi081 ceph-mon[31453]: pgmap v9622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:35 smithi153 ceph-mon[36569]: pgmap v9623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:35 smithi104 ceph-mon[36624]: pgmap v9623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:35 smithi081 ceph-mon[31453]: pgmap v9623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:36.239+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:09:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:36.239+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:09:36.240 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:36.239+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:09:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:09:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:09:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:09:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:37 smithi153 ceph-mon[36569]: pgmap v9624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:09:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:09:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:37 smithi153 ceph-mon[36569]: pgmap v9625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:37 smithi153 ceph-mon[36569]: pgmap v9626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:09:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:37 smithi104 ceph-mon[36624]: pgmap v9624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:09:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:09:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:37 smithi104 ceph-mon[36624]: pgmap v9625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:37 smithi104 ceph-mon[36624]: pgmap v9626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:09:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:37 smithi081 ceph-mon[31453]: pgmap v9624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:09:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:09:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:37 smithi081 ceph-mon[31453]: pgmap v9625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:37 smithi081 ceph-mon[31453]: pgmap v9626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:09:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:40 smithi153 ceph-mon[36569]: pgmap v9627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:40.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:40 smithi104 ceph-mon[36624]: pgmap v9627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:40 smithi081 ceph-mon[31453]: pgmap v9627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:42 smithi153 ceph-mon[36569]: pgmap v9628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:42 smithi104 ceph-mon[36624]: pgmap v9628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:42 smithi081 ceph-mon[31453]: pgmap v9628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:44 smithi153 ceph-mon[36569]: pgmap v9629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:44 smithi104 ceph-mon[36624]: pgmap v9629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:44 smithi081 ceph-mon[31453]: pgmap v9629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:45 smithi153 ceph-mon[36569]: pgmap v9630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:45 smithi104 ceph-mon[36624]: pgmap v9630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:45 smithi081 ceph-mon[31453]: pgmap v9630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:48 smithi153 ceph-mon[36569]: pgmap v9631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:48 smithi104 ceph-mon[36624]: pgmap v9631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:48 smithi081 ceph-mon[31453]: pgmap v9631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:50 smithi153 ceph-mon[36569]: pgmap v9632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:50 smithi104 ceph-mon[36624]: pgmap v9632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:50 smithi081 ceph-mon[31453]: pgmap v9632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:51.250+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:09:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:51.250+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:09:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:09:51.250+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:09:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:52 smithi153 ceph-mon[36569]: pgmap v9633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:52 smithi104 ceph-mon[36624]: pgmap v9633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:52 smithi081 ceph-mon[31453]: pgmap v9633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:53 smithi153 ceph-mon[36569]: pgmap v9634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:53 smithi104 ceph-mon[36624]: pgmap v9634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:53 smithi081 ceph-mon[31453]: pgmap v9634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:56 smithi153 ceph-mon[36569]: pgmap v9635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:56 smithi104 ceph-mon[36624]: pgmap v9635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:56 smithi081 ceph-mon[31453]: pgmap v9635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:58 smithi153 ceph-mon[36569]: pgmap v9636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:58 smithi104 ceph-mon[36624]: pgmap v9636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:58 smithi081 ceph-mon[31453]: pgmap v9636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:09:59 smithi153 ceph-mon[36569]: pgmap v9637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:09:59 smithi104 ceph-mon[36624]: pgmap v9637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:09:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:09:59 smithi081 ceph-mon[31453]: pgmap v9637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:10:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:10:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:10:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:01 smithi153 ceph-mon[36569]: pgmap v9638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:01.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:01 smithi081 ceph-mon[31453]: pgmap v9638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:01 smithi104 ceph-mon[36624]: pgmap v9638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:04 smithi153 ceph-mon[36569]: pgmap v9639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:04 smithi104 ceph-mon[36624]: pgmap v9639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:04 smithi081 ceph-mon[31453]: pgmap v9639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:06.253+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:10:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:06.253+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:10:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:06.253+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:10:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:06 smithi081 ceph-mon[31453]: pgmap v9640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:06 smithi153 ceph-mon[36569]: pgmap v9640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:06 smithi104 ceph-mon[36624]: pgmap v9640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:07 smithi153 ceph-mon[36569]: pgmap v9641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:07 smithi104 ceph-mon[36624]: pgmap v9641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:07 smithi081 ceph-mon[31453]: pgmap v9641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:10 smithi153 ceph-mon[36569]: pgmap v9642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:10 smithi104 ceph-mon[36624]: pgmap v9642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:10 smithi081 ceph-mon[31453]: pgmap v9642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:11 smithi153 ceph-mon[36569]: pgmap v9643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:11 smithi104 ceph-mon[36624]: pgmap v9643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:11 smithi081 ceph-mon[31453]: pgmap v9643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:14 smithi153 ceph-mon[36569]: pgmap v9644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:14 smithi104 ceph-mon[36624]: pgmap v9644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:14 smithi081 ceph-mon[31453]: pgmap v9644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:15 smithi153 ceph-mon[36569]: pgmap v9645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:15 smithi104 ceph-mon[36624]: pgmap v9645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:15.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:15.616+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:10:15.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:15.616+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:10:15.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:15.616+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:10:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:15.616+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:10:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:15 smithi081 ceph-mon[31453]: pgmap v9645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:18 smithi153 ceph-mon[36569]: pgmap v9646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:18 smithi104 ceph-mon[36624]: pgmap v9646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:18 smithi081 ceph-mon[31453]: pgmap v9646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:19 smithi104 ceph-mon[36624]: pgmap v9647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:19 smithi081 ceph-mon[31453]: pgmap v9647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:19 smithi153 ceph-mon[36569]: pgmap v9647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:21.269+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:10:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:21.269+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:10:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:21.269+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:10:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:21 smithi153 ceph-mon[36569]: pgmap v9648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:21 smithi104 ceph-mon[36624]: pgmap v9648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:21 smithi081 ceph-mon[31453]: pgmap v9648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:23 smithi153 ceph-mon[36569]: pgmap v9649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:23 smithi104 ceph-mon[36624]: pgmap v9649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:23 smithi081 ceph-mon[31453]: pgmap v9649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:25 smithi153 ceph-mon[36569]: pgmap v9650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:25 smithi104 ceph-mon[36624]: pgmap v9650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:25 smithi081 ceph-mon[31453]: pgmap v9650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:27 smithi153 ceph-mon[36569]: pgmap v9651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:27 smithi104 ceph-mon[36624]: pgmap v9651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:27 smithi081 ceph-mon[31453]: pgmap v9651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:29 smithi153 ceph-mon[36569]: pgmap v9652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:29 smithi104 ceph-mon[36624]: pgmap v9652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:29 smithi081 ceph-mon[31453]: pgmap v9652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:31 smithi153 ceph-mon[36569]: pgmap v9653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:31 smithi104 ceph-mon[36624]: pgmap v9653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:31 smithi081 ceph-mon[31453]: pgmap v9653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:33 smithi153 ceph-mon[36569]: pgmap v9654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:33 smithi104 ceph-mon[36624]: pgmap v9654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:33 smithi081 ceph-mon[31453]: pgmap v9654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:35 smithi153 ceph-mon[36569]: pgmap v9655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:35 smithi104 ceph-mon[36624]: pgmap v9655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:35 smithi081 ceph-mon[31453]: pgmap v9655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:36.282+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:10:36.283 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:36.282+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:10:36.283 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:36.282+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:10:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:37 smithi153 ceph-mon[36569]: pgmap v9656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:37 smithi104 ceph-mon[36624]: pgmap v9656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:37 smithi081 ceph-mon[31453]: pgmap v9656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:10:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:10:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:10:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:10:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:10:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:39 smithi153 ceph-mon[36569]: pgmap v9657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:39 smithi153 ceph-mon[36569]: pgmap v9658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:10:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:10:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:10:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:39 smithi104 ceph-mon[36624]: pgmap v9657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:39 smithi104 ceph-mon[36624]: pgmap v9658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:10:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:10:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:10:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:39 smithi081 ceph-mon[31453]: pgmap v9657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:39 smithi081 ceph-mon[31453]: pgmap v9658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:10:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:41 smithi153 ceph-mon[36569]: pgmap v9659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:41 smithi104 ceph-mon[36624]: pgmap v9659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:41 smithi081 ceph-mon[31453]: pgmap v9659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:43 smithi153 ceph-mon[36569]: pgmap v9660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:43 smithi104 ceph-mon[36624]: pgmap v9660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:43 smithi081 ceph-mon[31453]: pgmap v9660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:45 smithi153 ceph-mon[36569]: pgmap v9661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:45 smithi104 ceph-mon[36624]: pgmap v9661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:45 smithi081 ceph-mon[31453]: pgmap v9661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:47 smithi153 ceph-mon[36569]: pgmap v9662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:47 smithi104 ceph-mon[36624]: pgmap v9662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:47 smithi081 ceph-mon[31453]: pgmap v9662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:49 smithi104 ceph-mon[36624]: pgmap v9663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:49 smithi081 ceph-mon[31453]: pgmap v9663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:49 smithi153 ceph-mon[36569]: pgmap v9663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:51.287+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:10:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:51.287+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:10:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:10:51.287+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:10:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:51 smithi104 ceph-mon[36624]: pgmap v9664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:52.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:51 smithi153 ceph-mon[36569]: pgmap v9664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:51 smithi081 ceph-mon[31453]: pgmap v9664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:53 smithi104 ceph-mon[36624]: pgmap v9665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:53 smithi081 ceph-mon[31453]: pgmap v9665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:53 smithi153 ceph-mon[36569]: pgmap v9665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:55 smithi104 ceph-mon[36624]: pgmap v9666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:55 smithi081 ceph-mon[31453]: pgmap v9666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:55 smithi153 ceph-mon[36569]: pgmap v9666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:57 smithi104 ceph-mon[36624]: pgmap v9667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:57 smithi081 ceph-mon[31453]: pgmap v9667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:10:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:57 smithi153 ceph-mon[36569]: pgmap v9667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:10:59 smithi081 ceph-mon[31453]: pgmap v9668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:10:59 smithi153 ceph-mon[36569]: pgmap v9668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:10:59 smithi104 ceph-mon[36624]: pgmap v9668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:01 smithi081 ceph-mon[31453]: pgmap v9669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:01 smithi153 ceph-mon[36569]: pgmap v9669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:01 smithi104 ceph-mon[36624]: pgmap v9669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:03 smithi081 ceph-mon[31453]: pgmap v9670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:03 smithi153 ceph-mon[36569]: pgmap v9670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:03 smithi104 ceph-mon[36624]: pgmap v9670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:06.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:05 smithi104 ceph-mon[36624]: pgmap v9671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:05 smithi081 ceph-mon[31453]: pgmap v9671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:06.289+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:11:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:06.289+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:11:06.291 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:06.289+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:11:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:05 smithi153 ceph-mon[36569]: pgmap v9671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:07 smithi081 ceph-mon[31453]: pgmap v9672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:07 smithi153 ceph-mon[36569]: pgmap v9672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:07 smithi104 ceph-mon[36624]: pgmap v9672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:09 smithi081 ceph-mon[31453]: pgmap v9673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:09 smithi153 ceph-mon[36569]: pgmap v9673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:09 smithi104 ceph-mon[36624]: pgmap v9673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:11 smithi081 ceph-mon[31453]: pgmap v9674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:11 smithi153 ceph-mon[36569]: pgmap v9674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:11 smithi104 ceph-mon[36624]: pgmap v9674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:13 smithi081 ceph-mon[31453]: pgmap v9675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:13 smithi153 ceph-mon[36569]: pgmap v9675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:13 smithi104 ceph-mon[36624]: pgmap v9675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:15 smithi081 ceph-mon[31453]: pgmap v9676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:15 smithi153 ceph-mon[36569]: pgmap v9676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:15 smithi104 ceph-mon[36624]: pgmap v9676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:17 smithi081 ceph-mon[31453]: pgmap v9677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:17 smithi153 ceph-mon[36569]: pgmap v9677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:17 smithi104 ceph-mon[36624]: pgmap v9677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:19 smithi081 ceph-mon[31453]: pgmap v9678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:19 smithi153 ceph-mon[36569]: pgmap v9678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:19 smithi104 ceph-mon[36624]: pgmap v9678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:21.305+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:11:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:21.305+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:11:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:21.305+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:11:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:21 smithi081 ceph-mon[31453]: pgmap v9679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:21 smithi153 ceph-mon[36569]: pgmap v9679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:21 smithi104 ceph-mon[36624]: pgmap v9679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:23 smithi081 ceph-mon[31453]: pgmap v9680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:23 smithi153 ceph-mon[36569]: pgmap v9680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:23 smithi104 ceph-mon[36624]: pgmap v9680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:25 smithi081 ceph-mon[31453]: pgmap v9681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:25 smithi153 ceph-mon[36569]: pgmap v9681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:25 smithi104 ceph-mon[36624]: pgmap v9681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:27 smithi153 ceph-mon[36569]: pgmap v9682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:27 smithi104 ceph-mon[36624]: pgmap v9682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:27 smithi081 ceph-mon[31453]: pgmap v9682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:29 smithi153 ceph-mon[36569]: pgmap v9683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:29 smithi104 ceph-mon[36624]: pgmap v9683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:29 smithi081 ceph-mon[31453]: pgmap v9683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:32.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:31 smithi081 ceph-mon[31453]: pgmap v9684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:31 smithi153 ceph-mon[36569]: pgmap v9684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:31 smithi104 ceph-mon[36624]: pgmap v9684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:33 smithi153 ceph-mon[36569]: pgmap v9685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:33 smithi104 ceph-mon[36624]: pgmap v9685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:33 smithi081 ceph-mon[31453]: pgmap v9685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:35 smithi153 ceph-mon[36569]: pgmap v9686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:36.307+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:11:36.307 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:36.307+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:11:36.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:36.307+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:11:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:35 smithi104 ceph-mon[36624]: pgmap v9686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:35 smithi081 ceph-mon[31453]: pgmap v9686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:38.248 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:37 smithi081 ceph-mon[31453]: pgmap v9687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:37 smithi153 ceph-mon[36569]: pgmap v9687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:37 smithi104 ceph-mon[36624]: pgmap v9687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:39 smithi153 ceph-mon[36569]: pgmap v9688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:11:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:39 smithi104 ceph-mon[36624]: pgmap v9688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:11:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:39 smithi081 ceph-mon[31453]: pgmap v9688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:11:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:11:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:11:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:11:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:11:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:11:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:11:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:11:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:11:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:11:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:41 smithi153 ceph-mon[36569]: pgmap v9689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:41 smithi153 ceph-mon[36569]: pgmap v9690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:41 smithi104 ceph-mon[36624]: pgmap v9689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:41 smithi104 ceph-mon[36624]: pgmap v9690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:41 smithi081 ceph-mon[31453]: pgmap v9689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:41 smithi081 ceph-mon[31453]: pgmap v9690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:43 smithi153 ceph-mon[36569]: pgmap v9691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:43 smithi104 ceph-mon[36624]: pgmap v9691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:43 smithi081 ceph-mon[31453]: pgmap v9691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:45 smithi153 ceph-mon[36569]: pgmap v9692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:45 smithi104 ceph-mon[36624]: pgmap v9692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:45 smithi081 ceph-mon[31453]: pgmap v9692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:47 smithi153 ceph-mon[36569]: pgmap v9693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:47 smithi104 ceph-mon[36624]: pgmap v9693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:47 smithi081 ceph-mon[31453]: pgmap v9693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:49 smithi153 ceph-mon[36569]: pgmap v9694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:49 smithi104 ceph-mon[36624]: pgmap v9694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:49 smithi081 ceph-mon[31453]: pgmap v9694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:51.309+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:11:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:51.309+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:11:51.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:11:51.309+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:11:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:51 smithi153 ceph-mon[36569]: pgmap v9695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:51 smithi104 ceph-mon[36624]: pgmap v9695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:51 smithi081 ceph-mon[31453]: pgmap v9695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:53 smithi153 ceph-mon[36569]: pgmap v9696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:53 smithi104 ceph-mon[36624]: pgmap v9696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:53 smithi081 ceph-mon[31453]: pgmap v9696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:55 smithi153 ceph-mon[36569]: pgmap v9697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:55 smithi104 ceph-mon[36624]: pgmap v9697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:55 smithi081 ceph-mon[31453]: pgmap v9697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:11:58 smithi153 ceph-mon[36569]: pgmap v9698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:11:58 smithi104 ceph-mon[36624]: pgmap v9698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:11:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:11:58 smithi081 ceph-mon[31453]: pgmap v9698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:00 smithi153 ceph-mon[36569]: pgmap v9699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:00 smithi104 ceph-mon[36624]: pgmap v9699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:00 smithi081 ceph-mon[31453]: pgmap v9699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:02 smithi153 ceph-mon[36569]: pgmap v9700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:02 smithi104 ceph-mon[36624]: pgmap v9700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:02 smithi081 ceph-mon[31453]: pgmap v9700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:04 smithi153 ceph-mon[36569]: pgmap v9701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:04 smithi104 ceph-mon[36624]: pgmap v9701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:04 smithi081 ceph-mon[31453]: pgmap v9701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:06 smithi153 ceph-mon[36569]: pgmap v9702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:06.325+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:12:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:06.325+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:12:06.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:06.325+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:12:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:06 smithi104 ceph-mon[36624]: pgmap v9702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:06 smithi081 ceph-mon[31453]: pgmap v9702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:08 smithi153 ceph-mon[36569]: pgmap v9703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:08 smithi104 ceph-mon[36624]: pgmap v9703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:08 smithi081 ceph-mon[31453]: pgmap v9703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:10 smithi153 ceph-mon[36569]: pgmap v9704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:10 smithi104 ceph-mon[36624]: pgmap v9704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:10 smithi081 ceph-mon[31453]: pgmap v9704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:12 smithi153 ceph-mon[36569]: pgmap v9705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:12 smithi104 ceph-mon[36624]: pgmap v9705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:12 smithi081 ceph-mon[31453]: pgmap v9705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:14 smithi153 ceph-mon[36569]: pgmap v9706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:14 smithi104 ceph-mon[36624]: pgmap v9706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:14 smithi081 ceph-mon[31453]: pgmap v9706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:16 smithi153 ceph-mon[36569]: pgmap v9707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:16 smithi104 ceph-mon[36624]: pgmap v9707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:16 smithi081 ceph-mon[31453]: pgmap v9707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:18 smithi153 ceph-mon[36569]: pgmap v9708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:18 smithi104 ceph-mon[36624]: pgmap v9708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:18 smithi081 ceph-mon[31453]: pgmap v9708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:20 smithi104 ceph-mon[36624]: pgmap v9709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:20 smithi081 ceph-mon[31453]: pgmap v9709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:20 smithi153 ceph-mon[36569]: pgmap v9709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:21.333+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:12:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:21.333+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:12:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:21.333+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:12:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:22 smithi104 ceph-mon[36624]: pgmap v9710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:22.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:22 smithi153 ceph-mon[36569]: pgmap v9710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:22 smithi081 ceph-mon[31453]: pgmap v9710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:24 smithi104 ceph-mon[36624]: pgmap v9711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:24 smithi081 ceph-mon[31453]: pgmap v9711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:24 smithi153 ceph-mon[36569]: pgmap v9711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:26 smithi104 ceph-mon[36624]: pgmap v9712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:26 smithi081 ceph-mon[31453]: pgmap v9712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:26 smithi153 ceph-mon[36569]: pgmap v9712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:28 smithi104 ceph-mon[36624]: pgmap v9713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:28 smithi081 ceph-mon[31453]: pgmap v9713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:28 smithi153 ceph-mon[36569]: pgmap v9713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:30 smithi081 ceph-mon[31453]: pgmap v9714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:30 smithi153 ceph-mon[36569]: pgmap v9714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:30 smithi104 ceph-mon[36624]: pgmap v9714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:32 smithi081 ceph-mon[31453]: pgmap v9715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:32 smithi153 ceph-mon[36569]: pgmap v9715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:32 smithi104 ceph-mon[36624]: pgmap v9715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:34 smithi081 ceph-mon[31453]: pgmap v9716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:34 smithi153 ceph-mon[36569]: pgmap v9716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:34 smithi104 ceph-mon[36624]: pgmap v9716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:36.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:36.343+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:12:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:36.343+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:12:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:36.343+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:12:36.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:36 smithi104 ceph-mon[36624]: pgmap v9717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:36 smithi081 ceph-mon[31453]: pgmap v9717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:36 smithi153 ceph-mon[36569]: pgmap v9717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:38 smithi081 ceph-mon[31453]: pgmap v9718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:38 smithi153 ceph-mon[36569]: pgmap v9718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:38 smithi104 ceph-mon[36624]: pgmap v9718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:40 smithi081 ceph-mon[31453]: pgmap v9719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:40 smithi153 ceph-mon[36569]: pgmap v9719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:40 smithi104 ceph-mon[36624]: pgmap v9719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:12:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:12:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:12:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:42 smithi081 ceph-mon[31453]: pgmap v9720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:12:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:12:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:12:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:42 smithi153 ceph-mon[36569]: pgmap v9720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:12:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:12:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:12:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:42 smithi104 ceph-mon[36624]: pgmap v9720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:12:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:12:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:12:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:43 smithi081 ceph-mon[31453]: pgmap v9721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:43 smithi081 ceph-mon[31453]: pgmap v9722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:43 smithi153 ceph-mon[36569]: pgmap v9721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:43 smithi153 ceph-mon[36569]: pgmap v9722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:43 smithi104 ceph-mon[36624]: pgmap v9721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:43 smithi104 ceph-mon[36624]: pgmap v9722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:45 smithi081 ceph-mon[31453]: pgmap v9723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:45 smithi153 ceph-mon[36569]: pgmap v9723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:45 smithi104 ceph-mon[36624]: pgmap v9723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:47 smithi081 ceph-mon[31453]: pgmap v9724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:47 smithi153 ceph-mon[36569]: pgmap v9724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:47 smithi104 ceph-mon[36624]: pgmap v9724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:49 smithi081 ceph-mon[31453]: pgmap v9725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:49 smithi153 ceph-mon[36569]: pgmap v9725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:49 smithi104 ceph-mon[36624]: pgmap v9725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:51.348+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:12:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:51.348+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:12:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:12:51.348+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:12:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:51 smithi081 ceph-mon[31453]: pgmap v9726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:51 smithi153 ceph-mon[36569]: pgmap v9726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:51 smithi104 ceph-mon[36624]: pgmap v9726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:53 smithi153 ceph-mon[36569]: pgmap v9727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:53 smithi104 ceph-mon[36624]: pgmap v9727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:53 smithi081 ceph-mon[31453]: pgmap v9727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:55 smithi153 ceph-mon[36569]: pgmap v9728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:55 smithi104 ceph-mon[36624]: pgmap v9728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:55 smithi081 ceph-mon[31453]: pgmap v9728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:57 smithi081 ceph-mon[31453]: pgmap v9729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:57 smithi153 ceph-mon[36569]: pgmap v9729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:57 smithi104 ceph-mon[36624]: pgmap v9729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:12:59 smithi153 ceph-mon[36569]: pgmap v9730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:12:59 smithi104 ceph-mon[36624]: pgmap v9730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:12:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:12:59 smithi081 ceph-mon[31453]: pgmap v9730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:01 smithi153 ceph-mon[36569]: pgmap v9731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:01 smithi104 ceph-mon[36624]: pgmap v9731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:01 smithi081 ceph-mon[31453]: pgmap v9731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:03.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:03 smithi081 ceph-mon[31453]: pgmap v9732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:03 smithi153 ceph-mon[36569]: pgmap v9732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:03 smithi104 ceph-mon[36624]: pgmap v9732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:05 smithi153 ceph-mon[36569]: pgmap v9733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:05 smithi104 ceph-mon[36624]: pgmap v9733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:05 smithi081 ceph-mon[31453]: pgmap v9733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:06.360+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:13:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:06.360+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:13:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:06.360+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:13:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:07 smithi153 ceph-mon[36569]: pgmap v9734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:07 smithi104 ceph-mon[36624]: pgmap v9734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:07 smithi081 ceph-mon[31453]: pgmap v9734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:09 smithi153 ceph-mon[36569]: pgmap v9735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:09 smithi104 ceph-mon[36624]: pgmap v9735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:09 smithi081 ceph-mon[31453]: pgmap v9735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:11 smithi153 ceph-mon[36569]: pgmap v9736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:11 smithi104 ceph-mon[36624]: pgmap v9736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:11 smithi081 ceph-mon[31453]: pgmap v9736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:13 smithi153 ceph-mon[36569]: pgmap v9737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:13 smithi104 ceph-mon[36624]: pgmap v9737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:13 smithi081 ceph-mon[31453]: pgmap v9737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:15 smithi153 ceph-mon[36569]: pgmap v9738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:15 smithi104 ceph-mon[36624]: pgmap v9738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:15.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:15.646+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:13:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:15.646+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:13:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:15.647+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:13:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:15.647+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:13:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:15.647+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:13:15.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:15.647+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:13:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:15.647+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:13:15.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:15.648+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:13:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:15 smithi081 ceph-mon[31453]: pgmap v9738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:17 smithi153 ceph-mon[36569]: pgmap v9739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:17 smithi104 ceph-mon[36624]: pgmap v9739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:17 smithi081 ceph-mon[31453]: pgmap v9739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:19 smithi153 ceph-mon[36569]: pgmap v9740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:19 smithi104 ceph-mon[36624]: pgmap v9740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:19 smithi081 ceph-mon[31453]: pgmap v9740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:21.376+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:13:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:21.376+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:13:21.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:21.376+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:13:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:21 smithi153 ceph-mon[36569]: pgmap v9741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:21 smithi104 ceph-mon[36624]: pgmap v9741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:21 smithi081 ceph-mon[31453]: pgmap v9741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:23 smithi153 ceph-mon[36569]: pgmap v9742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:23 smithi104 ceph-mon[36624]: pgmap v9742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:23 smithi081 ceph-mon[31453]: pgmap v9742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:24 smithi153 ceph-mon[36569]: pgmap v9743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:24 smithi104 ceph-mon[36624]: pgmap v9743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:24 smithi081 ceph-mon[31453]: pgmap v9743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:26 smithi081 ceph-mon[31453]: pgmap v9744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:26 smithi153 ceph-mon[36569]: pgmap v9744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:26 smithi104 ceph-mon[36624]: pgmap v9744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:28 smithi081 ceph-mon[31453]: pgmap v9745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:28 smithi153 ceph-mon[36569]: pgmap v9745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:28 smithi104 ceph-mon[36624]: pgmap v9745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:30 smithi081 ceph-mon[31453]: pgmap v9746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:30 smithi153 ceph-mon[36569]: pgmap v9746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:30 smithi104 ceph-mon[36624]: pgmap v9746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:32 smithi081 ceph-mon[31453]: pgmap v9747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:32 smithi153 ceph-mon[36569]: pgmap v9747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:32 smithi104 ceph-mon[36624]: pgmap v9747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:34 smithi153 ceph-mon[36569]: pgmap v9748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:34 smithi104 ceph-mon[36624]: pgmap v9748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:34 smithi081 ceph-mon[31453]: pgmap v9748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:36.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:36.391+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:13:36.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:36.391+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:13:36.393 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:36.391+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:13:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:36 smithi153 ceph-mon[36569]: pgmap v9749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:36 smithi104 ceph-mon[36624]: pgmap v9749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:36 smithi081 ceph-mon[31453]: pgmap v9749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:38 smithi153 ceph-mon[36569]: pgmap v9750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:38 smithi104 ceph-mon[36624]: pgmap v9750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:38 smithi081 ceph-mon[31453]: pgmap v9750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:40 smithi153 ceph-mon[36569]: pgmap v9751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:40 smithi104 ceph-mon[36624]: pgmap v9751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:40 smithi081 ceph-mon[31453]: pgmap v9751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:43.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:42 smithi081 ceph-mon[31453]: pgmap v9752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:43.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:13:43.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:42 smithi153 ceph-mon[36569]: pgmap v9752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:13:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:42 smithi104 ceph-mon[36624]: pgmap v9752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:13:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:43 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:13:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:43 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:13:44.325 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:43 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:13:44.325 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:43 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:13:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:43 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:13:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:43 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:13:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:44 smithi153 ceph-mon[36569]: pgmap v9753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:44 smithi104 ceph-mon[36624]: pgmap v9753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:45.376 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:44 smithi081 ceph-mon[31453]: pgmap v9753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:46 smithi153 ceph-mon[36569]: pgmap v9754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:46 smithi104 ceph-mon[36624]: pgmap v9754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:46 smithi081 ceph-mon[31453]: pgmap v9754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:48.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:48.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:48.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.430 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:48 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:13:48.431 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:48 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:13:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:48 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:13:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:48 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:13:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:48 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:13:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:48 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:13:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:49 smithi153 ceph-mon[36569]: pgmap v9755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:49 smithi104 ceph-mon[36624]: pgmap v9755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:49 smithi081 ceph-mon[31453]: pgmap v9755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:13:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:13:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:13:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: pgmap v9756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: pgmap v9757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:13:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:13:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:13:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:13:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: pgmap v9756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: pgmap v9757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:13:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: pgmap v9756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: pgmap v9757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:13:51.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:51.407+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:13:51.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:51.407+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:13:51.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:13:51.407+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:13:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:52 smithi153 ceph-mon[36569]: pgmap v9758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:52 smithi104 ceph-mon[36624]: pgmap v9758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:52 smithi081 ceph-mon[31453]: pgmap v9758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:54 smithi153 ceph-mon[36569]: pgmap v9759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:54 smithi104 ceph-mon[36624]: pgmap v9759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:54 smithi081 ceph-mon[31453]: pgmap v9759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:56 smithi153 ceph-mon[36569]: pgmap v9760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:56 smithi104 ceph-mon[36624]: pgmap v9760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:56 smithi081 ceph-mon[31453]: pgmap v9760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:13:58 smithi153 ceph-mon[36569]: pgmap v9761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:13:58 smithi104 ceph-mon[36624]: pgmap v9761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:13:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:13:58 smithi081 ceph-mon[31453]: pgmap v9761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:00 smithi153 ceph-mon[36569]: pgmap v9762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:00 smithi104 ceph-mon[36624]: pgmap v9762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:00 smithi081 ceph-mon[31453]: pgmap v9762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:02 smithi153 ceph-mon[36569]: pgmap v9763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:02 smithi104 ceph-mon[36624]: pgmap v9763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:02 smithi081 ceph-mon[31453]: pgmap v9763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:04.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:04 smithi081 ceph-mon[31453]: pgmap v9764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:04 smithi153 ceph-mon[36569]: pgmap v9764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:04 smithi104 ceph-mon[36624]: pgmap v9764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:06.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:06.422+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:14:06.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:06.422+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:14:06.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:06.422+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:14:06.790 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:06 smithi081 ceph-mon[31453]: pgmap v9765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:06 smithi153 ceph-mon[36569]: pgmap v9765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:06 smithi104 ceph-mon[36624]: pgmap v9765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:08 smithi153 ceph-mon[36569]: pgmap v9766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:08 smithi104 ceph-mon[36624]: pgmap v9766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:08 smithi081 ceph-mon[31453]: pgmap v9766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:10 smithi153 ceph-mon[36569]: pgmap v9767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:10 smithi104 ceph-mon[36624]: pgmap v9767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:10 smithi081 ceph-mon[31453]: pgmap v9767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:12 smithi153 ceph-mon[36569]: pgmap v9768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:12 smithi104 ceph-mon[36624]: pgmap v9768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:12 smithi081 ceph-mon[31453]: pgmap v9768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:14 smithi153 ceph-mon[36569]: pgmap v9769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:14 smithi104 ceph-mon[36624]: pgmap v9769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:14 smithi081 ceph-mon[31453]: pgmap v9769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:16 smithi153 ceph-mon[36569]: pgmap v9770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:16 smithi104 ceph-mon[36624]: pgmap v9770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:16 smithi081 ceph-mon[31453]: pgmap v9770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:18 smithi153 ceph-mon[36569]: pgmap v9771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:18 smithi104 ceph-mon[36624]: pgmap v9771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:18 smithi081 ceph-mon[31453]: pgmap v9771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:20 smithi153 ceph-mon[36569]: pgmap v9772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:20 smithi104 ceph-mon[36624]: pgmap v9772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:20 smithi081 ceph-mon[31453]: pgmap v9772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:21.424+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:14:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:21.424+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:14:21.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:21.424+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:14:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:22 smithi153 ceph-mon[36569]: pgmap v9773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:22 smithi104 ceph-mon[36624]: pgmap v9773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:22 smithi081 ceph-mon[31453]: pgmap v9773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:24 smithi153 ceph-mon[36569]: pgmap v9774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:24 smithi104 ceph-mon[36624]: pgmap v9774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:24 smithi081 ceph-mon[31453]: pgmap v9774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:26 smithi153 ceph-mon[36569]: pgmap v9775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:26 smithi104 ceph-mon[36624]: pgmap v9775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:26 smithi081 ceph-mon[31453]: pgmap v9775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:28 smithi153 ceph-mon[36569]: pgmap v9776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:28 smithi104 ceph-mon[36624]: pgmap v9776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:28 smithi081 ceph-mon[31453]: pgmap v9776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:30 smithi153 ceph-mon[36569]: pgmap v9777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:30 smithi104 ceph-mon[36624]: pgmap v9777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:30 smithi081 ceph-mon[31453]: pgmap v9777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:32 smithi153 ceph-mon[36569]: pgmap v9778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:32 smithi104 ceph-mon[36624]: pgmap v9778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:32 smithi081 ceph-mon[31453]: pgmap v9778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:34 smithi153 ceph-mon[36569]: pgmap v9779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:34 smithi104 ceph-mon[36624]: pgmap v9779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:34 smithi081 ceph-mon[31453]: pgmap v9779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:36.440+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:14:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:36.440+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:14:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:36.440+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:14:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:36 smithi153 ceph-mon[36569]: pgmap v9780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:36 smithi104 ceph-mon[36624]: pgmap v9780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:36 smithi081 ceph-mon[31453]: pgmap v9780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:38 smithi153 ceph-mon[36569]: pgmap v9781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:38 smithi104 ceph-mon[36624]: pgmap v9781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:38 smithi081 ceph-mon[31453]: pgmap v9781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:40 smithi153 ceph-mon[36569]: pgmap v9782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:40 smithi104 ceph-mon[36624]: pgmap v9782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:40 smithi081 ceph-mon[31453]: pgmap v9782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:42 smithi153 ceph-mon[36569]: pgmap v9783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:42 smithi104 ceph-mon[36624]: pgmap v9783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:42 smithi081 ceph-mon[31453]: pgmap v9783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:44 smithi104 ceph-mon[36624]: pgmap v9784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:44 smithi081 ceph-mon[31453]: pgmap v9784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:44 smithi153 ceph-mon[36569]: pgmap v9784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:46 smithi104 ceph-mon[36624]: pgmap v9785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:46 smithi081 ceph-mon[31453]: pgmap v9785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:46 smithi153 ceph-mon[36569]: pgmap v9785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:48 smithi104 ceph-mon[36624]: pgmap v9786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:48 smithi081 ceph-mon[31453]: pgmap v9786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:48 smithi153 ceph-mon[36569]: pgmap v9786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:50 smithi104 ceph-mon[36624]: pgmap v9787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:14:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:50 smithi081 ceph-mon[31453]: pgmap v9787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:14:51.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:50 smithi153 ceph-mon[36569]: pgmap v9787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:51.029 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:14:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:51.455+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:14:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:51.455+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:14:51.456 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:14:51.455+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:14:52.684 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:52 smithi153 ceph-mon[36569]: pgmap v9788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:52 smithi104 ceph-mon[36624]: pgmap v9788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:52 smithi081 ceph-mon[31453]: pgmap v9788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:54 smithi104 ceph-mon[36624]: pgmap v9789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:54 smithi081 ceph-mon[31453]: pgmap v9789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:54 smithi153 ceph-mon[36569]: pgmap v9789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:14:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:14:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:55 smithi081 ceph-mon[31453]: pgmap v9790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:55 smithi081 ceph-mon[31453]: pgmap v9791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:14:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:14:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:55 smithi153 ceph-mon[36569]: pgmap v9790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:55 smithi153 ceph-mon[36569]: pgmap v9791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:14:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:14:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:55 smithi104 ceph-mon[36624]: pgmap v9790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:55 smithi104 ceph-mon[36624]: pgmap v9791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:14:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:57 smithi153 ceph-mon[36569]: pgmap v9792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:57 smithi104 ceph-mon[36624]: pgmap v9792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:14:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:57 smithi081 ceph-mon[31453]: pgmap v9792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:14:59 smithi153 ceph-mon[36569]: pgmap v9793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:14:59 smithi104 ceph-mon[36624]: pgmap v9793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:14:59 smithi081 ceph-mon[31453]: pgmap v9793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:01 smithi153 ceph-mon[36569]: pgmap v9794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:01 smithi104 ceph-mon[36624]: pgmap v9794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:01 smithi081 ceph-mon[31453]: pgmap v9794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:03 smithi153 ceph-mon[36569]: pgmap v9795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:03 smithi104 ceph-mon[36624]: pgmap v9795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:03 smithi081 ceph-mon[31453]: pgmap v9795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:05 smithi153 ceph-mon[36569]: pgmap v9796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:05 smithi104 ceph-mon[36624]: pgmap v9796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:05 smithi081 ceph-mon[31453]: pgmap v9796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:06.470+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:15:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:06.470+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:15:06.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:06.470+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:15:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:07 smithi153 ceph-mon[36569]: pgmap v9797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:07 smithi104 ceph-mon[36624]: pgmap v9797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:07 smithi081 ceph-mon[31453]: pgmap v9797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:09 smithi153 ceph-mon[36569]: pgmap v9798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:09 smithi104 ceph-mon[36624]: pgmap v9798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:09 smithi081 ceph-mon[31453]: pgmap v9798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:11 smithi153 ceph-mon[36569]: pgmap v9799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:11 smithi104 ceph-mon[36624]: pgmap v9799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:11 smithi081 ceph-mon[31453]: pgmap v9799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:13 smithi153 ceph-mon[36569]: pgmap v9800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:13 smithi104 ceph-mon[36624]: pgmap v9800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:13 smithi081 ceph-mon[31453]: pgmap v9800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:15 smithi153 ceph-mon[36569]: pgmap v9801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:15 smithi104 ceph-mon[36624]: pgmap v9801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:15 smithi081 ceph-mon[31453]: pgmap v9801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:17 smithi153 ceph-mon[36569]: pgmap v9802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:17 smithi104 ceph-mon[36624]: pgmap v9802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:17 smithi081 ceph-mon[31453]: pgmap v9802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:19 smithi153 ceph-mon[36569]: pgmap v9803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:19 smithi104 ceph-mon[36624]: pgmap v9803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:19 smithi081 ceph-mon[31453]: pgmap v9803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:21.471+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:15:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:21.471+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:15:21.472 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:21.471+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:15:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:21 smithi153 ceph-mon[36569]: pgmap v9804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:21 smithi104 ceph-mon[36624]: pgmap v9804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:21 smithi081 ceph-mon[31453]: pgmap v9804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:23 smithi153 ceph-mon[36569]: pgmap v9805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:23 smithi104 ceph-mon[36624]: pgmap v9805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:23 smithi081 ceph-mon[31453]: pgmap v9805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:25 smithi153 ceph-mon[36569]: pgmap v9806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:25 smithi104 ceph-mon[36624]: pgmap v9806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:25 smithi081 ceph-mon[31453]: pgmap v9806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:27 smithi153 ceph-mon[36569]: pgmap v9807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:27 smithi104 ceph-mon[36624]: pgmap v9807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:27 smithi081 ceph-mon[31453]: pgmap v9807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:29 smithi153 ceph-mon[36569]: pgmap v9808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:29 smithi104 ceph-mon[36624]: pgmap v9808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:29 smithi081 ceph-mon[31453]: pgmap v9808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:31 smithi153 ceph-mon[36569]: pgmap v9809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:31 smithi104 ceph-mon[36624]: pgmap v9809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:31 smithi081 ceph-mon[31453]: pgmap v9809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:33 smithi153 ceph-mon[36569]: pgmap v9810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:33 smithi104 ceph-mon[36624]: pgmap v9810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:33 smithi081 ceph-mon[31453]: pgmap v9810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:35 smithi153 ceph-mon[36569]: pgmap v9811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:35 smithi104 ceph-mon[36624]: pgmap v9811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:35 smithi081 ceph-mon[31453]: pgmap v9811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:36.480+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:15:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:36.480+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:15:36.481 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:36.480+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:15:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:37 smithi153 ceph-mon[36569]: pgmap v9812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:37 smithi104 ceph-mon[36624]: pgmap v9812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:37 smithi081 ceph-mon[31453]: pgmap v9812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:39 smithi104 ceph-mon[36624]: pgmap v9813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:39 smithi081 ceph-mon[31453]: pgmap v9813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:39 smithi153 ceph-mon[36569]: pgmap v9813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:41 smithi104 ceph-mon[36624]: pgmap v9814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:41 smithi081 ceph-mon[31453]: pgmap v9814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:41 smithi153 ceph-mon[36569]: pgmap v9814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:43 smithi104 ceph-mon[36624]: pgmap v9815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:43 smithi081 ceph-mon[31453]: pgmap v9815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:43 smithi153 ceph-mon[36569]: pgmap v9815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:45 smithi104 ceph-mon[36624]: pgmap v9816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:45 smithi081 ceph-mon[31453]: pgmap v9816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:45 smithi153 ceph-mon[36569]: pgmap v9816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:48.104 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:47 smithi081 ceph-mon[31453]: pgmap v9817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:47 smithi153 ceph-mon[36569]: pgmap v9817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:47 smithi104 ceph-mon[36624]: pgmap v9817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:49 smithi081 ceph-mon[31453]: pgmap v9818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:50.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:49 smithi153 ceph-mon[36569]: pgmap v9818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:49 smithi104 ceph-mon[36624]: pgmap v9818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:51.482+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:15:51.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:51.482+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:15:51.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:15:51.482+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:15:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:51 smithi081 ceph-mon[31453]: pgmap v9819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:51 smithi153 ceph-mon[36569]: pgmap v9819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:51 smithi104 ceph-mon[36624]: pgmap v9819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:53 smithi081 ceph-mon[31453]: pgmap v9820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:53 smithi153 ceph-mon[36569]: pgmap v9820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:53 smithi104 ceph-mon[36624]: pgmap v9820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:56.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:55 smithi153 ceph-mon[36569]: pgmap v9821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:56.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:15:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:55 smithi081 ceph-mon[31453]: pgmap v9821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:15:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:55 smithi104 ceph-mon[36624]: pgmap v9821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:15:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:15:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:15:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:56 smithi081 ceph-mon[31453]: pgmap v9822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:56 smithi081 ceph-mon[31453]: pgmap v9823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:15:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:15:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:15:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:56 smithi153 ceph-mon[36569]: pgmap v9822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:56 smithi153 ceph-mon[36569]: pgmap v9823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:15:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:15:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:15:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:56 smithi104 ceph-mon[36624]: pgmap v9822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:56 smithi104 ceph-mon[36624]: pgmap v9823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:15:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:15:59 smithi153 ceph-mon[36569]: pgmap v9824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:15:59 smithi104 ceph-mon[36624]: pgmap v9824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:15:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:15:59 smithi081 ceph-mon[31453]: pgmap v9824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:01 smithi153 ceph-mon[36569]: pgmap v9825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:01 smithi104 ceph-mon[36624]: pgmap v9825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:01 smithi081 ceph-mon[31453]: pgmap v9825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:03 smithi153 ceph-mon[36569]: pgmap v9826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:03 smithi104 ceph-mon[36624]: pgmap v9826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:03 smithi081 ceph-mon[31453]: pgmap v9826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:05 smithi153 ceph-mon[36569]: pgmap v9827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:05 smithi104 ceph-mon[36624]: pgmap v9827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:05 smithi081 ceph-mon[31453]: pgmap v9827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:06.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:06.484+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:16:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:06.484+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:16:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:06.484+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:16:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:07 smithi153 ceph-mon[36569]: pgmap v9828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:07 smithi104 ceph-mon[36624]: pgmap v9828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:07 smithi081 ceph-mon[31453]: pgmap v9828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:09 smithi153 ceph-mon[36569]: pgmap v9829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:09 smithi104 ceph-mon[36624]: pgmap v9829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:09 smithi081 ceph-mon[31453]: pgmap v9829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:11 smithi153 ceph-mon[36569]: pgmap v9830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:11 smithi104 ceph-mon[36624]: pgmap v9830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:11 smithi081 ceph-mon[31453]: pgmap v9830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:12.138+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 msgr2=0x7f16e80b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000593) for more than 900000000 us, fault. 2024-04-13T02:16:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:12.138+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b44f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f4032000 comp rx=0 tx=0).stop 2024-04-13T02:16:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:12.138+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 msgr2=0x7f16e80b44f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T02:16:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:12.138+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b44f0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T02:16:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:12.138+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T02:16:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:12.138+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T02:16:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:12.140+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3cf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c6a0 tx=0x7f16f4039000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T02:16:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:13 smithi153 ceph-mon[36569]: pgmap v9831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:13 smithi104 ceph-mon[36624]: pgmap v9831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:13 smithi081 ceph-mon[31453]: pgmap v9831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:15 smithi153 ceph-mon[36569]: pgmap v9832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:15 smithi104 ceph-mon[36624]: pgmap v9832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:15 smithi081 ceph-mon[31453]: pgmap v9832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:15.674+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:16:15.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:15.674+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:16:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:15.674+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:16:15.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:15.675+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:16:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:17 smithi153 ceph-mon[36569]: pgmap v9833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:17 smithi104 ceph-mon[36624]: pgmap v9833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:17 smithi081 ceph-mon[31453]: pgmap v9833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:19 smithi153 ceph-mon[36569]: pgmap v9834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:19 smithi104 ceph-mon[36624]: pgmap v9834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:19 smithi081 ceph-mon[31453]: pgmap v9834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:21.496+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:16:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:21.496+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:16:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:21.496+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:16:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:21 smithi153 ceph-mon[36569]: pgmap v9835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:21 smithi104 ceph-mon[36624]: pgmap v9835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:21 smithi081 ceph-mon[31453]: pgmap v9835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:23 smithi153 ceph-mon[36569]: pgmap v9836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:23 smithi104 ceph-mon[36624]: pgmap v9836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:23 smithi081 ceph-mon[31453]: pgmap v9836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:24 smithi153 ceph-mon[36569]: pgmap v9837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:24 smithi104 ceph-mon[36624]: pgmap v9837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:24 smithi081 ceph-mon[31453]: pgmap v9837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:27 smithi153 ceph-mon[36569]: pgmap v9838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:27 smithi104 ceph-mon[36624]: pgmap v9838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:27 smithi081 ceph-mon[31453]: pgmap v9838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:29 smithi153 ceph-mon[36569]: pgmap v9839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:29 smithi104 ceph-mon[36624]: pgmap v9839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:29 smithi081 ceph-mon[31453]: pgmap v9839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:31 smithi153 ceph-mon[36569]: pgmap v9840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:31 smithi104 ceph-mon[36624]: pgmap v9840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:31 smithi081 ceph-mon[31453]: pgmap v9840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:33 smithi153 ceph-mon[36569]: pgmap v9841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:33 smithi104 ceph-mon[36624]: pgmap v9841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:33 smithi081 ceph-mon[31453]: pgmap v9841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:35.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:35 smithi081 ceph-mon[31453]: pgmap v9842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:35 smithi153 ceph-mon[36569]: pgmap v9842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:35 smithi104 ceph-mon[36624]: pgmap v9842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:36.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:36.511+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:16:36.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:36.511+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:16:36.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:36.511+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:16:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:37 smithi153 ceph-mon[36569]: pgmap v9843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:37 smithi104 ceph-mon[36624]: pgmap v9843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:37 smithi081 ceph-mon[31453]: pgmap v9843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:39 smithi153 ceph-mon[36569]: pgmap v9844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:39 smithi104 ceph-mon[36624]: pgmap v9844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:39 smithi081 ceph-mon[31453]: pgmap v9844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:41 smithi153 ceph-mon[36569]: pgmap v9845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:41 smithi104 ceph-mon[36624]: pgmap v9845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:41 smithi081 ceph-mon[31453]: pgmap v9845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:43 smithi153 ceph-mon[36569]: pgmap v9846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:43 smithi104 ceph-mon[36624]: pgmap v9846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:43 smithi081 ceph-mon[31453]: pgmap v9846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:45 smithi153 ceph-mon[36569]: pgmap v9847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:45 smithi104 ceph-mon[36624]: pgmap v9847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:45 smithi081 ceph-mon[31453]: pgmap v9847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:47 smithi153 ceph-mon[36569]: pgmap v9848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:47 smithi104 ceph-mon[36624]: pgmap v9848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:47 smithi081 ceph-mon[31453]: pgmap v9848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:49 smithi153 ceph-mon[36569]: pgmap v9849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:49 smithi104 ceph-mon[36624]: pgmap v9849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:49 smithi081 ceph-mon[31453]: pgmap v9849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:50 smithi153 ceph-mon[36569]: pgmap v9850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:50 smithi104 ceph-mon[36624]: pgmap v9850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:50 smithi081 ceph-mon[31453]: pgmap v9850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:51.522+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:16:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:51.522+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:16:51.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:16:51.522+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:16:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:53 smithi153 ceph-mon[36569]: pgmap v9851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:53 smithi104 ceph-mon[36624]: pgmap v9851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:53 smithi081 ceph-mon[31453]: pgmap v9851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:55 smithi153 ceph-mon[36569]: pgmap v9852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:55 smithi104 ceph-mon[36624]: pgmap v9852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:55 smithi081 ceph-mon[31453]: pgmap v9852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:57 smithi153 ceph-mon[36569]: pgmap v9853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:16:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:57 smithi104 ceph-mon[36624]: pgmap v9853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:16:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:57 smithi081 ceph-mon[31453]: pgmap v9853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:16:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:16:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:16:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:16:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:16:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:16:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:16:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:16:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:16:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:16:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:59 smithi153 ceph-mon[36569]: pgmap v9854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:16:59 smithi153 ceph-mon[36569]: pgmap v9855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:59 smithi104 ceph-mon[36624]: pgmap v9854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:16:59 smithi104 ceph-mon[36624]: pgmap v9855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:59 smithi081 ceph-mon[31453]: pgmap v9854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:16:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:16:59 smithi081 ceph-mon[31453]: pgmap v9855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:01 smithi153 ceph-mon[36569]: pgmap v9856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:01 smithi104 ceph-mon[36624]: pgmap v9856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:01 smithi081 ceph-mon[31453]: pgmap v9856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:03 smithi153 ceph-mon[36569]: pgmap v9857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:03 smithi104 ceph-mon[36624]: pgmap v9857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:03 smithi081 ceph-mon[31453]: pgmap v9857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:05 smithi153 ceph-mon[36569]: pgmap v9858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:05 smithi104 ceph-mon[36624]: pgmap v9858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:05 smithi081 ceph-mon[31453]: pgmap v9858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:06.532+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:17:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:06.532+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:17:06.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:06.532+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:17:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:07 smithi153 ceph-mon[36569]: pgmap v9859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:07 smithi104 ceph-mon[36624]: pgmap v9859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:07 smithi081 ceph-mon[31453]: pgmap v9859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:09 smithi153 ceph-mon[36569]: pgmap v9860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:09 smithi104 ceph-mon[36624]: pgmap v9860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:09 smithi081 ceph-mon[31453]: pgmap v9860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:10 smithi153 ceph-mon[36569]: pgmap v9861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:10 smithi104 ceph-mon[36624]: pgmap v9861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:10 smithi081 ceph-mon[31453]: pgmap v9861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:12 smithi081 ceph-mon[31453]: pgmap v9862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:12 smithi153 ceph-mon[36569]: pgmap v9862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:12 smithi104 ceph-mon[36624]: pgmap v9862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:14 smithi081 ceph-mon[31453]: pgmap v9863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:14 smithi153 ceph-mon[36569]: pgmap v9863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:14 smithi104 ceph-mon[36624]: pgmap v9863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:16 smithi081 ceph-mon[31453]: pgmap v9864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:16 smithi153 ceph-mon[36569]: pgmap v9864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:16 smithi104 ceph-mon[36624]: pgmap v9864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:18 smithi081 ceph-mon[31453]: pgmap v9865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:18 smithi153 ceph-mon[36569]: pgmap v9865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:18 smithi104 ceph-mon[36624]: pgmap v9865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:20 smithi081 ceph-mon[31453]: pgmap v9866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:20 smithi153 ceph-mon[36569]: pgmap v9866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:20 smithi104 ceph-mon[36624]: pgmap v9866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:21.543+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:17:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:21.543+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:17:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:21.543+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:17:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:22 smithi081 ceph-mon[31453]: pgmap v9867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:22 smithi153 ceph-mon[36569]: pgmap v9867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:22 smithi104 ceph-mon[36624]: pgmap v9867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:24 smithi081 ceph-mon[31453]: pgmap v9868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:24 smithi153 ceph-mon[36569]: pgmap v9868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:24 smithi104 ceph-mon[36624]: pgmap v9868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:26 smithi081 ceph-mon[31453]: pgmap v9869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:26 smithi153 ceph-mon[36569]: pgmap v9869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:26 smithi104 ceph-mon[36624]: pgmap v9869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:28 smithi081 ceph-mon[31453]: pgmap v9870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:28 smithi153 ceph-mon[36569]: pgmap v9870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:28 smithi104 ceph-mon[36624]: pgmap v9870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:30 smithi081 ceph-mon[31453]: pgmap v9871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:30 smithi153 ceph-mon[36569]: pgmap v9871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:30 smithi104 ceph-mon[36624]: pgmap v9871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:32 smithi081 ceph-mon[31453]: pgmap v9872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:32 smithi153 ceph-mon[36569]: pgmap v9872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:32 smithi104 ceph-mon[36624]: pgmap v9872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:34 smithi081 ceph-mon[31453]: pgmap v9873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:34 smithi153 ceph-mon[36569]: pgmap v9873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:34 smithi104 ceph-mon[36624]: pgmap v9873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:36.544+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:17:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:36.544+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:17:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:36.544+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:17:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:36 smithi081 ceph-mon[31453]: pgmap v9874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:36 smithi153 ceph-mon[36569]: pgmap v9874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:36 smithi104 ceph-mon[36624]: pgmap v9874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:38 smithi081 ceph-mon[31453]: pgmap v9875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:38 smithi153 ceph-mon[36569]: pgmap v9875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:38 smithi104 ceph-mon[36624]: pgmap v9875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:40 smithi153 ceph-mon[36569]: pgmap v9876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:40 smithi104 ceph-mon[36624]: pgmap v9876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:40 smithi081 ceph-mon[31453]: pgmap v9876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:42 smithi153 ceph-mon[36569]: pgmap v9877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:42 smithi104 ceph-mon[36624]: pgmap v9877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:42 smithi081 ceph-mon[31453]: pgmap v9877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:44 smithi153 ceph-mon[36569]: pgmap v9878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:44 smithi104 ceph-mon[36624]: pgmap v9878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:44 smithi081 ceph-mon[31453]: pgmap v9878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:46 smithi153 ceph-mon[36569]: pgmap v9879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:46 smithi104 ceph-mon[36624]: pgmap v9879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:46 smithi081 ceph-mon[31453]: pgmap v9879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:48 smithi153 ceph-mon[36569]: pgmap v9880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:48 smithi104 ceph-mon[36624]: pgmap v9880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:48 smithi081 ceph-mon[31453]: pgmap v9880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:50 smithi153 ceph-mon[36569]: pgmap v9881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:50 smithi104 ceph-mon[36624]: pgmap v9881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:50 smithi081 ceph-mon[31453]: pgmap v9881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:51.559+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:17:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:51.559+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:17:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:17:51.559+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:17:53.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:52 smithi081 ceph-mon[31453]: pgmap v9882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:52 smithi153 ceph-mon[36569]: pgmap v9882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:52 smithi104 ceph-mon[36624]: pgmap v9882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:54 smithi153 ceph-mon[36569]: pgmap v9883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:54 smithi104 ceph-mon[36624]: pgmap v9883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:54 smithi081 ceph-mon[31453]: pgmap v9883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:56 smithi153 ceph-mon[36569]: pgmap v9884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:56 smithi104 ceph-mon[36624]: pgmap v9884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:56 smithi081 ceph-mon[31453]: pgmap v9884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:59.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:58 smithi153 ceph-mon[36569]: pgmap v9885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:59.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:17:59.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:17:59.191 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:17:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:58 smithi104 ceph-mon[36624]: pgmap v9885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:17:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:17:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:17:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:58 smithi081 ceph-mon[31453]: pgmap v9885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:17:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:17:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:17:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:17:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:00.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:00.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:17:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:17:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:00 smithi153 ceph-mon[36569]: pgmap v9886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:00 smithi104 ceph-mon[36624]: pgmap v9886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:00 smithi081 ceph-mon[31453]: pgmap v9886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:02 smithi153 ceph-mon[36569]: pgmap v9887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:02 smithi104 ceph-mon[36624]: pgmap v9887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:02 smithi081 ceph-mon[31453]: pgmap v9887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:05 smithi153 ceph-mon[36569]: pgmap v9888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:18:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:18:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:05 smithi104 ceph-mon[36624]: pgmap v9888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:18:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:18:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:05 smithi081 ceph-mon[31453]: pgmap v9888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:18:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:18:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:18:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:06 smithi153 ceph-mon[36569]: pgmap v9889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:06 smithi153 ceph-mon[36569]: pgmap v9890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:06.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:06 smithi081 ceph-mon[31453]: pgmap v9889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:06.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:06 smithi081 ceph-mon[31453]: pgmap v9890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:06.575+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:18:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:06.575+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:18:06.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:06.575+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:18:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:06 smithi104 ceph-mon[36624]: pgmap v9889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:06 smithi104 ceph-mon[36624]: pgmap v9890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:08 smithi153 ceph-mon[36569]: pgmap v9891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:08 smithi104 ceph-mon[36624]: pgmap v9891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:08 smithi081 ceph-mon[31453]: pgmap v9891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:10 smithi153 ceph-mon[36569]: pgmap v9892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:10 smithi104 ceph-mon[36624]: pgmap v9892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:10 smithi081 ceph-mon[31453]: pgmap v9892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:12 smithi153 ceph-mon[36569]: pgmap v9893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:12 smithi104 ceph-mon[36624]: pgmap v9893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:12 smithi081 ceph-mon[31453]: pgmap v9893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:14 smithi153 ceph-mon[36569]: pgmap v9894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:14 smithi104 ceph-mon[36624]: pgmap v9894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:14 smithi081 ceph-mon[31453]: pgmap v9894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:16 smithi153 ceph-mon[36569]: pgmap v9895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:16 smithi104 ceph-mon[36624]: pgmap v9895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:16 smithi081 ceph-mon[31453]: pgmap v9895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:18 smithi153 ceph-mon[36569]: pgmap v9896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:18 smithi104 ceph-mon[36624]: pgmap v9896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:18 smithi081 ceph-mon[31453]: pgmap v9896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:20 smithi153 ceph-mon[36569]: pgmap v9897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:20.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:20 smithi104 ceph-mon[36624]: pgmap v9897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:20 smithi081 ceph-mon[31453]: pgmap v9897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:21.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:21.590+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:18:21.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:21.590+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:18:21.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:21.590+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:18:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:22 smithi153 ceph-mon[36569]: pgmap v9898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:22 smithi104 ceph-mon[36624]: pgmap v9898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:22 smithi081 ceph-mon[31453]: pgmap v9898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:24 smithi153 ceph-mon[36569]: pgmap v9899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:24 smithi104 ceph-mon[36624]: pgmap v9899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:24 smithi081 ceph-mon[31453]: pgmap v9899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:26 smithi153 ceph-mon[36569]: pgmap v9900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:26 smithi104 ceph-mon[36624]: pgmap v9900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:26 smithi081 ceph-mon[31453]: pgmap v9900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:28 smithi153 ceph-mon[36569]: pgmap v9901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:28 smithi104 ceph-mon[36624]: pgmap v9901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:28 smithi081 ceph-mon[31453]: pgmap v9901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:30 smithi153 ceph-mon[36569]: pgmap v9902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:30 smithi104 ceph-mon[36624]: pgmap v9902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:30 smithi081 ceph-mon[31453]: pgmap v9902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:32 smithi153 ceph-mon[36569]: pgmap v9903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:32 smithi104 ceph-mon[36624]: pgmap v9903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:32 smithi081 ceph-mon[31453]: pgmap v9903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:34 smithi153 ceph-mon[36569]: pgmap v9904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:34 smithi104 ceph-mon[36624]: pgmap v9904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:34 smithi081 ceph-mon[31453]: pgmap v9904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:36 smithi153 ceph-mon[36569]: pgmap v9905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:36 smithi104 ceph-mon[36624]: pgmap v9905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:36.599+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:18:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:36.599+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:18:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:36.599+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:18:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:36 smithi081 ceph-mon[31453]: pgmap v9905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:38 smithi153 ceph-mon[36569]: pgmap v9906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:38 smithi104 ceph-mon[36624]: pgmap v9906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:38 smithi081 ceph-mon[31453]: pgmap v9906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:39 smithi081 ceph-mon[31453]: pgmap v9907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:39 smithi153 ceph-mon[36569]: pgmap v9907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:39 smithi104 ceph-mon[36624]: pgmap v9907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:41 smithi104 ceph-mon[36624]: pgmap v9908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:41 smithi081 ceph-mon[31453]: pgmap v9908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:41 smithi153 ceph-mon[36569]: pgmap v9908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:43 smithi104 ceph-mon[36624]: pgmap v9909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:43 smithi081 ceph-mon[31453]: pgmap v9909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:43 smithi153 ceph-mon[36569]: pgmap v9909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:45 smithi081 ceph-mon[31453]: pgmap v9910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:45 smithi153 ceph-mon[36569]: pgmap v9910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:45 smithi104 ceph-mon[36624]: pgmap v9910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:47 smithi081 ceph-mon[31453]: pgmap v9911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:47 smithi153 ceph-mon[36569]: pgmap v9911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:47 smithi104 ceph-mon[36624]: pgmap v9911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:49 smithi081 ceph-mon[31453]: pgmap v9912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:49 smithi153 ceph-mon[36569]: pgmap v9912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:49 smithi104 ceph-mon[36624]: pgmap v9912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:51.612+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:18:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:51.612+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:18:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:18:51.612+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:18:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:51 smithi081 ceph-mon[31453]: pgmap v9913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:51 smithi153 ceph-mon[36569]: pgmap v9913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:51 smithi104 ceph-mon[36624]: pgmap v9913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:53 smithi081 ceph-mon[31453]: pgmap v9914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:53 smithi153 ceph-mon[36569]: pgmap v9914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:53 smithi104 ceph-mon[36624]: pgmap v9914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:55 smithi081 ceph-mon[31453]: pgmap v9915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:55 smithi153 ceph-mon[36569]: pgmap v9915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:55 smithi104 ceph-mon[36624]: pgmap v9915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:57 smithi081 ceph-mon[31453]: pgmap v9916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:57 smithi153 ceph-mon[36569]: pgmap v9916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:57 smithi104 ceph-mon[36624]: pgmap v9916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:18:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:18:59 smithi081 ceph-mon[31453]: pgmap v9917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:18:59 smithi153 ceph-mon[36569]: pgmap v9917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:18:59 smithi104 ceph-mon[36624]: pgmap v9917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:01 smithi081 ceph-mon[31453]: pgmap v9918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:01 smithi153 ceph-mon[36569]: pgmap v9918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:01 smithi104 ceph-mon[36624]: pgmap v9918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:03 smithi081 ceph-mon[31453]: pgmap v9919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:03 smithi153 ceph-mon[36569]: pgmap v9919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:03 smithi104 ceph-mon[36624]: pgmap v9919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:05 smithi081 ceph-mon[31453]: pgmap v9920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:19:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:05 smithi153 ceph-mon[36569]: pgmap v9920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:19:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:05 smithi104 ceph-mon[36624]: pgmap v9920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:19:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:06.620+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:19:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:06.620+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:19:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:06.620+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:19:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:19:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:19:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:06 smithi081 ceph-mon[31453]: pgmap v9921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:06 smithi081 ceph-mon[31453]: pgmap v9922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:19:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:19:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:06 smithi153 ceph-mon[36569]: pgmap v9921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:06 smithi153 ceph-mon[36569]: pgmap v9922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:19:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:19:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:06 smithi104 ceph-mon[36624]: pgmap v9921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:06 smithi104 ceph-mon[36624]: pgmap v9922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:19:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:09 smithi081 ceph-mon[31453]: pgmap v9923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:09 smithi153 ceph-mon[36569]: pgmap v9923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:09 smithi104 ceph-mon[36624]: pgmap v9923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:11 smithi081 ceph-mon[31453]: pgmap v9924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:11 smithi153 ceph-mon[36569]: pgmap v9924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:11 smithi104 ceph-mon[36624]: pgmap v9924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:13 smithi081 ceph-mon[31453]: pgmap v9925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:13 smithi153 ceph-mon[36569]: pgmap v9925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:13 smithi104 ceph-mon[36624]: pgmap v9925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:15 smithi081 ceph-mon[31453]: pgmap v9926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:15 smithi153 ceph-mon[36569]: pgmap v9926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:15 smithi104 ceph-mon[36624]: pgmap v9926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:15.704+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:19:15.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:15.704+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:19:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:15.704+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:19:15.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:15.704+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:19:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:17 smithi153 ceph-mon[36569]: pgmap v9927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:17 smithi104 ceph-mon[36624]: pgmap v9927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:17 smithi081 ceph-mon[31453]: pgmap v9927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:19 smithi153 ceph-mon[36569]: pgmap v9928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:19 smithi104 ceph-mon[36624]: pgmap v9928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:19 smithi081 ceph-mon[31453]: pgmap v9928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:21 smithi153 ceph-mon[36569]: pgmap v9929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:21 smithi104 ceph-mon[36624]: pgmap v9929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:21.630+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:19:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:21.630+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:19:21.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:21.630+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:19:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:21 smithi081 ceph-mon[31453]: pgmap v9929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:23 smithi153 ceph-mon[36569]: pgmap v9930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:23 smithi104 ceph-mon[36624]: pgmap v9930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:23 smithi081 ceph-mon[31453]: pgmap v9930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:25 smithi153 ceph-mon[36569]: pgmap v9931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:25 smithi104 ceph-mon[36624]: pgmap v9931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:25 smithi081 ceph-mon[31453]: pgmap v9931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:27 smithi153 ceph-mon[36569]: pgmap v9932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:27 smithi104 ceph-mon[36624]: pgmap v9932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:27 smithi081 ceph-mon[31453]: pgmap v9932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:29 smithi153 ceph-mon[36569]: pgmap v9933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:29 smithi104 ceph-mon[36624]: pgmap v9933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:29 smithi081 ceph-mon[31453]: pgmap v9933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:31 smithi153 ceph-mon[36569]: pgmap v9934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:31 smithi104 ceph-mon[36624]: pgmap v9934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:31 smithi081 ceph-mon[31453]: pgmap v9934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:33 smithi153 ceph-mon[36569]: pgmap v9935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:33 smithi104 ceph-mon[36624]: pgmap v9935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:33 smithi081 ceph-mon[31453]: pgmap v9935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:35 smithi153 ceph-mon[36569]: pgmap v9936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:35 smithi104 ceph-mon[36624]: pgmap v9936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:35 smithi081 ceph-mon[31453]: pgmap v9936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:36.636+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:19:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:36.636+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:19:36.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:36.636+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:19:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:37 smithi153 ceph-mon[36569]: pgmap v9937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:37 smithi104 ceph-mon[36624]: pgmap v9937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:37 smithi081 ceph-mon[31453]: pgmap v9937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:39 smithi153 ceph-mon[36569]: pgmap v9938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:39 smithi104 ceph-mon[36624]: pgmap v9938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:39 smithi081 ceph-mon[31453]: pgmap v9938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:41 smithi153 ceph-mon[36569]: pgmap v9939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:41 smithi104 ceph-mon[36624]: pgmap v9939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:41 smithi081 ceph-mon[31453]: pgmap v9939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:43 smithi153 ceph-mon[36569]: pgmap v9940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:43 smithi104 ceph-mon[36624]: pgmap v9940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:43 smithi081 ceph-mon[31453]: pgmap v9940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:45 smithi153 ceph-mon[36569]: pgmap v9941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:45 smithi104 ceph-mon[36624]: pgmap v9941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:45 smithi081 ceph-mon[31453]: pgmap v9941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:47 smithi153 ceph-mon[36569]: pgmap v9942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:47 smithi104 ceph-mon[36624]: pgmap v9942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:47 smithi081 ceph-mon[31453]: pgmap v9942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:49 smithi153 ceph-mon[36569]: pgmap v9943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:49 smithi104 ceph-mon[36624]: pgmap v9943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:49 smithi081 ceph-mon[31453]: pgmap v9943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:51 smithi153 ceph-mon[36569]: pgmap v9944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:51 smithi104 ceph-mon[36624]: pgmap v9944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:51.652 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:51.652+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:19:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:51.652+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:19:51.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:19:51.652+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:19:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:51 smithi081 ceph-mon[31453]: pgmap v9944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:53 smithi153 ceph-mon[36569]: pgmap v9945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:53 smithi104 ceph-mon[36624]: pgmap v9945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:53 smithi081 ceph-mon[31453]: pgmap v9945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:55 smithi153 ceph-mon[36569]: pgmap v9946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:55 smithi104 ceph-mon[36624]: pgmap v9946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:55 smithi081 ceph-mon[31453]: pgmap v9946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:57 smithi153 ceph-mon[36569]: pgmap v9947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:57 smithi104 ceph-mon[36624]: pgmap v9947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:57 smithi081 ceph-mon[31453]: pgmap v9947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:19:59 smithi153 ceph-mon[36569]: pgmap v9948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:19:59 smithi104 ceph-mon[36624]: pgmap v9948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:19:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:19:59 smithi081 ceph-mon[31453]: pgmap v9948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:20:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:20:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:20:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:01 smithi153 ceph-mon[36569]: pgmap v9949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:01 smithi104 ceph-mon[36624]: pgmap v9949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:01 smithi081 ceph-mon[31453]: pgmap v9949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:03 smithi153 ceph-mon[36569]: pgmap v9950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:03 smithi104 ceph-mon[36624]: pgmap v9950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:03 smithi081 ceph-mon[31453]: pgmap v9950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:05 smithi153 ceph-mon[36569]: pgmap v9951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:05 smithi104 ceph-mon[36624]: pgmap v9951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:05 smithi081 ceph-mon[31453]: pgmap v9951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:06.667+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:20:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:06.667+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:20:06.668 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:06.667+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:20:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:07 smithi153 ceph-mon[36569]: pgmap v9952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:20:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:07 smithi104 ceph-mon[36624]: pgmap v9952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:20:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:07 smithi081 ceph-mon[31453]: pgmap v9952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:20:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:20:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:20:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:20:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:20:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:20:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:20:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:20:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:20:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:20:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:09 smithi104 ceph-mon[36624]: pgmap v9953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:09 smithi104 ceph-mon[36624]: pgmap v9954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:09 smithi081 ceph-mon[31453]: pgmap v9953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:09 smithi081 ceph-mon[31453]: pgmap v9954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:09 smithi153 ceph-mon[36569]: pgmap v9953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:09 smithi153 ceph-mon[36569]: pgmap v9954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:11 smithi104 ceph-mon[36624]: pgmap v9955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:11 smithi081 ceph-mon[31453]: pgmap v9955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:11 smithi153 ceph-mon[36569]: pgmap v9955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:13 smithi104 ceph-mon[36624]: pgmap v9956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:13 smithi081 ceph-mon[31453]: pgmap v9956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:13 smithi153 ceph-mon[36569]: pgmap v9956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:14 smithi081 ceph-mon[31453]: pgmap v9957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:14 smithi153 ceph-mon[36569]: pgmap v9957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:14 smithi104 ceph-mon[36624]: pgmap v9957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:16 smithi153 ceph-mon[36569]: pgmap v9958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:16 smithi104 ceph-mon[36624]: pgmap v9958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:16 smithi081 ceph-mon[31453]: pgmap v9958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:18 smithi153 ceph-mon[36569]: pgmap v9959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:18 smithi104 ceph-mon[36624]: pgmap v9959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:18 smithi081 ceph-mon[31453]: pgmap v9959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:20 smithi153 ceph-mon[36569]: pgmap v9960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:20 smithi104 ceph-mon[36624]: pgmap v9960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:20 smithi081 ceph-mon[31453]: pgmap v9960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:21.679+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:20:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:21.679+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:20:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:21.679+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:20:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:22 smithi153 ceph-mon[36569]: pgmap v9961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:22 smithi104 ceph-mon[36624]: pgmap v9961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:22 smithi081 ceph-mon[31453]: pgmap v9961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:25 smithi153 ceph-mon[36569]: pgmap v9962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:25 smithi104 ceph-mon[36624]: pgmap v9962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:25 smithi081 ceph-mon[31453]: pgmap v9962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:27 smithi153 ceph-mon[36569]: pgmap v9963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:27 smithi104 ceph-mon[36624]: pgmap v9963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:27 smithi081 ceph-mon[31453]: pgmap v9963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:29 smithi153 ceph-mon[36569]: pgmap v9964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:29 smithi104 ceph-mon[36624]: pgmap v9964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:29 smithi081 ceph-mon[31453]: pgmap v9964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:31 smithi153 ceph-mon[36569]: pgmap v9965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:31 smithi104 ceph-mon[36624]: pgmap v9965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:31 smithi081 ceph-mon[31453]: pgmap v9965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:33 smithi153 ceph-mon[36569]: pgmap v9966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:33 smithi104 ceph-mon[36624]: pgmap v9966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:33 smithi081 ceph-mon[31453]: pgmap v9966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:35 smithi153 ceph-mon[36569]: pgmap v9967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:35 smithi104 ceph-mon[36624]: pgmap v9967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:35 smithi081 ceph-mon[31453]: pgmap v9967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:36.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:36.693+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:20:36.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:36.693+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:20:36.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:36.693+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:20:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:37 smithi104 ceph-mon[36624]: pgmap v9968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:37 smithi081 ceph-mon[31453]: pgmap v9968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:37 smithi153 ceph-mon[36569]: pgmap v9968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:39 smithi104 ceph-mon[36624]: pgmap v9969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:39 smithi081 ceph-mon[31453]: pgmap v9969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:39 smithi153 ceph-mon[36569]: pgmap v9969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:41 smithi104 ceph-mon[36624]: pgmap v9970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:41 smithi081 ceph-mon[31453]: pgmap v9970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:41 smithi153 ceph-mon[36569]: pgmap v9970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:43 smithi104 ceph-mon[36624]: pgmap v9971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:43 smithi081 ceph-mon[31453]: pgmap v9971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:43 smithi153 ceph-mon[36569]: pgmap v9971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:45 smithi104 ceph-mon[36624]: pgmap v9972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:45 smithi081 ceph-mon[31453]: pgmap v9972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:45 smithi153 ceph-mon[36569]: pgmap v9972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:47 smithi081 ceph-mon[31453]: pgmap v9973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:47 smithi153 ceph-mon[36569]: pgmap v9973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:47 smithi104 ceph-mon[36624]: pgmap v9973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:49 smithi081 ceph-mon[31453]: pgmap v9974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:49 smithi153 ceph-mon[36569]: pgmap v9974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:49 smithi104 ceph-mon[36624]: pgmap v9974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:51 smithi081 ceph-mon[31453]: pgmap v9975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:51 smithi153 ceph-mon[36569]: pgmap v9975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:51 smithi104 ceph-mon[36624]: pgmap v9975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:51.708+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:20:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:51.708+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:20:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:20:51.708+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:20:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:53 smithi081 ceph-mon[31453]: pgmap v9976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:53 smithi153 ceph-mon[36569]: pgmap v9976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:53 smithi104 ceph-mon[36624]: pgmap v9976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:55 smithi081 ceph-mon[31453]: pgmap v9977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:55 smithi153 ceph-mon[36569]: pgmap v9977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:55 smithi104 ceph-mon[36624]: pgmap v9977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:57 smithi081 ceph-mon[31453]: pgmap v9978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:57 smithi153 ceph-mon[36569]: pgmap v9978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:57 smithi104 ceph-mon[36624]: pgmap v9978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:20:59 smithi081 ceph-mon[31453]: pgmap v9979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:20:59 smithi153 ceph-mon[36569]: pgmap v9979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:20:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:20:59 smithi104 ceph-mon[36624]: pgmap v9979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:01 smithi081 ceph-mon[31453]: pgmap v9980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:01 smithi153 ceph-mon[36569]: pgmap v9980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:01 smithi104 ceph-mon[36624]: pgmap v9980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:03 smithi081 ceph-mon[31453]: pgmap v9981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:03 smithi153 ceph-mon[36569]: pgmap v9981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:03 smithi104 ceph-mon[36624]: pgmap v9981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:05 smithi081 ceph-mon[31453]: pgmap v9982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:05 smithi153 ceph-mon[36569]: pgmap v9982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:05 smithi104 ceph-mon[36624]: pgmap v9982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:06.724+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:21:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:06.724+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:21:06.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:06.724+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:21:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:07 smithi081 ceph-mon[31453]: pgmap v9983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:07 smithi153 ceph-mon[36569]: pgmap v9983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:07 smithi104 ceph-mon[36624]: pgmap v9983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:21:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:21:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:21:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:09 smithi081 ceph-mon[31453]: pgmap v9984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:09 smithi153 ceph-mon[36569]: pgmap v9984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:09 smithi104 ceph-mon[36624]: pgmap v9984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:21:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:21:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:10 smithi153 ceph-mon[36569]: pgmap v9985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:10 smithi153 ceph-mon[36569]: pgmap v9986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:21:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:21:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:21:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:10 smithi104 ceph-mon[36624]: pgmap v9985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:10 smithi104 ceph-mon[36624]: pgmap v9986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:21:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:21:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:21:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:10 smithi081 ceph-mon[31453]: pgmap v9985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:10 smithi081 ceph-mon[31453]: pgmap v9986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:21:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:11 smithi081 ceph-mon[31453]: pgmap v9987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:11 smithi153 ceph-mon[36569]: pgmap v9987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:11 smithi104 ceph-mon[36624]: pgmap v9987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:13 smithi081 ceph-mon[31453]: pgmap v9988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:13 smithi153 ceph-mon[36569]: pgmap v9988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:13 smithi104 ceph-mon[36624]: pgmap v9988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:15 smithi081 ceph-mon[31453]: pgmap v9989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:15 smithi153 ceph-mon[36569]: pgmap v9989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:15 smithi104 ceph-mon[36624]: pgmap v9989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:17 smithi081 ceph-mon[31453]: pgmap v9990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:17 smithi153 ceph-mon[36569]: pgmap v9990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:17 smithi104 ceph-mon[36624]: pgmap v9990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:19 smithi081 ceph-mon[31453]: pgmap v9991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:19 smithi153 ceph-mon[36569]: pgmap v9991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:19 smithi104 ceph-mon[36624]: pgmap v9991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:21 smithi081 ceph-mon[31453]: pgmap v9992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:21.734+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:21:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:21.734+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:21:21.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:21.734+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:21:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:21 smithi153 ceph-mon[36569]: pgmap v9992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:21 smithi104 ceph-mon[36624]: pgmap v9992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:23 smithi081 ceph-mon[31453]: pgmap v9993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:23 smithi153 ceph-mon[36569]: pgmap v9993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:23 smithi104 ceph-mon[36624]: pgmap v9993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:25 smithi081 ceph-mon[31453]: pgmap v9994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:25 smithi153 ceph-mon[36569]: pgmap v9994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:25 smithi104 ceph-mon[36624]: pgmap v9994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:27 smithi081 ceph-mon[31453]: pgmap v9995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:27 smithi153 ceph-mon[36569]: pgmap v9995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:27 smithi104 ceph-mon[36624]: pgmap v9995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:29 smithi081 ceph-mon[31453]: pgmap v9996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:29 smithi153 ceph-mon[36569]: pgmap v9996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:29 smithi104 ceph-mon[36624]: pgmap v9996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:30.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:30.726+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:21:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:30.726+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:21:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:30.727+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:21:30.727 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:30.727+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:21:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:31 smithi081 ceph-mon[31453]: pgmap v9997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:31 smithi153 ceph-mon[36569]: pgmap v9997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:31 smithi104 ceph-mon[36624]: pgmap v9997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:33 smithi081 ceph-mon[31453]: pgmap v9998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:33 smithi153 ceph-mon[36569]: pgmap v9998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:33 smithi104 ceph-mon[36624]: pgmap v9998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:35 smithi081 ceph-mon[31453]: pgmap v9999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:35 smithi153 ceph-mon[36569]: pgmap v9999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:35 smithi104 ceph-mon[36624]: pgmap v9999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:36.751+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:21:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:36.751+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:21:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:36.751+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:21:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:37 smithi081 ceph-mon[31453]: pgmap v10000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:37 smithi153 ceph-mon[36569]: pgmap v10000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:37 smithi104 ceph-mon[36624]: pgmap v10000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:39 smithi081 ceph-mon[31453]: pgmap v10001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:39 smithi153 ceph-mon[36569]: pgmap v10001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:39 smithi104 ceph-mon[36624]: pgmap v10001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:41 smithi081 ceph-mon[31453]: pgmap v10002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:41 smithi153 ceph-mon[36569]: pgmap v10002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:41 smithi104 ceph-mon[36624]: pgmap v10002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:43 smithi081 ceph-mon[31453]: pgmap v10003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:43 smithi153 ceph-mon[36569]: pgmap v10003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:43 smithi104 ceph-mon[36624]: pgmap v10003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:45 smithi081 ceph-mon[31453]: pgmap v10004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:45 smithi153 ceph-mon[36569]: pgmap v10004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:45 smithi104 ceph-mon[36624]: pgmap v10004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:47 smithi081 ceph-mon[31453]: pgmap v10005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:47 smithi153 ceph-mon[36569]: pgmap v10005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:47 smithi104 ceph-mon[36624]: pgmap v10005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:50 smithi104 ceph-mon[36624]: pgmap v10006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:50 smithi081 ceph-mon[31453]: pgmap v10006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:50 smithi153 ceph-mon[36569]: pgmap v10006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:51 smithi081 ceph-mon[31453]: pgmap v10007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:51.754+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:21:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:51.754+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:21:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:21:51.754+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:21:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:51 smithi153 ceph-mon[36569]: pgmap v10007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:51 smithi104 ceph-mon[36624]: pgmap v10007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:53 smithi081 ceph-mon[31453]: pgmap v10008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:53 smithi153 ceph-mon[36569]: pgmap v10008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:53 smithi104 ceph-mon[36624]: pgmap v10008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:55 smithi081 ceph-mon[31453]: pgmap v10009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:55 smithi153 ceph-mon[36569]: pgmap v10009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:55 smithi104 ceph-mon[36624]: pgmap v10009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:57 smithi081 ceph-mon[31453]: pgmap v10010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:57 smithi153 ceph-mon[36569]: pgmap v10010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:57 smithi104 ceph-mon[36624]: pgmap v10010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:21:59 smithi081 ceph-mon[31453]: pgmap v10011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:21:59 smithi153 ceph-mon[36569]: pgmap v10011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:21:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:21:59 smithi104 ceph-mon[36624]: pgmap v10011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:01 smithi081 ceph-mon[31453]: pgmap v10012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:01 smithi153 ceph-mon[36569]: pgmap v10012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:01 smithi104 ceph-mon[36624]: pgmap v10012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:03 smithi081 ceph-mon[31453]: pgmap v10013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:03 smithi153 ceph-mon[36569]: pgmap v10013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:03 smithi104 ceph-mon[36624]: pgmap v10013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:05 smithi081 ceph-mon[31453]: pgmap v10014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:05 smithi153 ceph-mon[36569]: pgmap v10014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:05 smithi104 ceph-mon[36624]: pgmap v10014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:06.770+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:22:06.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:06.770+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:22:06.771 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:06.770+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:22:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:07 smithi153 ceph-mon[36569]: pgmap v10015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:07 smithi104 ceph-mon[36624]: pgmap v10015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:07 smithi081 ceph-mon[31453]: pgmap v10015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:09 smithi153 ceph-mon[36569]: pgmap v10016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:09 smithi104 ceph-mon[36624]: pgmap v10016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:09 smithi081 ceph-mon[31453]: pgmap v10016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:22:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:22:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:22:11.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:22:11.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:22:11.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:11 smithi104 ceph-mon[36624]: pgmap v10017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:11.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:11 smithi104 ceph-mon[36624]: pgmap v10018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:11.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:22:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:22:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:22:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:11 smithi153 ceph-mon[36569]: pgmap v10017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:11 smithi153 ceph-mon[36569]: pgmap v10018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:22:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:22:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:22:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:11 smithi081 ceph-mon[31453]: pgmap v10017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:11 smithi081 ceph-mon[31453]: pgmap v10018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:22:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:14 smithi153 ceph-mon[36569]: pgmap v10019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:14 smithi104 ceph-mon[36624]: pgmap v10019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:14 smithi081 ceph-mon[31453]: pgmap v10019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:15.733+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:22:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:15.733+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:22:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:15.733+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:22:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:15.733+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:22:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:16 smithi153 ceph-mon[36569]: pgmap v10020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:16 smithi104 ceph-mon[36624]: pgmap v10020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:16 smithi081 ceph-mon[31453]: pgmap v10020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:18 smithi153 ceph-mon[36569]: pgmap v10021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:18 smithi104 ceph-mon[36624]: pgmap v10021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:18 smithi081 ceph-mon[31453]: pgmap v10021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:20 smithi153 ceph-mon[36569]: pgmap v10022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:20 smithi104 ceph-mon[36624]: pgmap v10022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:20 smithi081 ceph-mon[31453]: pgmap v10022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:21.780+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:22:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:21.780+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:22:21.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:21.780+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:22:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:22 smithi153 ceph-mon[36569]: pgmap v10023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:22 smithi104 ceph-mon[36624]: pgmap v10023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:22 smithi081 ceph-mon[31453]: pgmap v10023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:24 smithi153 ceph-mon[36569]: pgmap v10024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:24 smithi104 ceph-mon[36624]: pgmap v10024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:24 smithi081 ceph-mon[31453]: pgmap v10024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:26 smithi153 ceph-mon[36569]: pgmap v10025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:26 smithi104 ceph-mon[36624]: pgmap v10025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:26 smithi081 ceph-mon[31453]: pgmap v10025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:28 smithi104 ceph-mon[36624]: pgmap v10026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:28 smithi081 ceph-mon[31453]: pgmap v10026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:28 smithi153 ceph-mon[36569]: pgmap v10026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:30 smithi104 ceph-mon[36624]: pgmap v10027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:30.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:30 smithi153 ceph-mon[36569]: pgmap v10027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:30 smithi081 ceph-mon[31453]: pgmap v10027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:32 smithi104 ceph-mon[36624]: pgmap v10028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:32 smithi081 ceph-mon[31453]: pgmap v10028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:32 smithi153 ceph-mon[36569]: pgmap v10028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:34 smithi104 ceph-mon[36624]: pgmap v10029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:34 smithi081 ceph-mon[31453]: pgmap v10029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:34 smithi153 ceph-mon[36569]: pgmap v10029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:36 smithi104 ceph-mon[36624]: pgmap v10030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:36 smithi081 ceph-mon[31453]: pgmap v10030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:36 smithi153 ceph-mon[36569]: pgmap v10030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:36.781+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:22:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:36.781+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:22:36.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:36.782+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:22:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:38 smithi081 ceph-mon[31453]: pgmap v10031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:38 smithi153 ceph-mon[36569]: pgmap v10031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:38 smithi104 ceph-mon[36624]: pgmap v10031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:40 smithi081 ceph-mon[31453]: pgmap v10032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:40 smithi153 ceph-mon[36569]: pgmap v10032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:40 smithi104 ceph-mon[36624]: pgmap v10032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:42.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:42 smithi104 ceph-mon[36624]: pgmap v10033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:42 smithi081 ceph-mon[31453]: pgmap v10033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:42 smithi153 ceph-mon[36569]: pgmap v10033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:44 smithi081 ceph-mon[31453]: pgmap v10034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:44 smithi153 ceph-mon[36569]: pgmap v10034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:44 smithi104 ceph-mon[36624]: pgmap v10034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:46 smithi081 ceph-mon[31453]: pgmap v10035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:46 smithi153 ceph-mon[36569]: pgmap v10035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:46 smithi104 ceph-mon[36624]: pgmap v10035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:48 smithi081 ceph-mon[31453]: pgmap v10036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:48 smithi153 ceph-mon[36569]: pgmap v10036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:48 smithi104 ceph-mon[36624]: pgmap v10036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:50 smithi081 ceph-mon[31453]: pgmap v10037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:50 smithi153 ceph-mon[36569]: pgmap v10037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:50 smithi104 ceph-mon[36624]: pgmap v10037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:51.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:51.787+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:22:51.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:51.787+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:22:51.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:22:51.787+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:22:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:52 smithi081 ceph-mon[31453]: pgmap v10038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:52 smithi153 ceph-mon[36569]: pgmap v10038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:52 smithi104 ceph-mon[36624]: pgmap v10038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:54 smithi081 ceph-mon[31453]: pgmap v10039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:54 smithi153 ceph-mon[36569]: pgmap v10039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:54 smithi104 ceph-mon[36624]: pgmap v10039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:56 smithi081 ceph-mon[31453]: pgmap v10040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:56 smithi153 ceph-mon[36569]: pgmap v10040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:56 smithi104 ceph-mon[36624]: pgmap v10040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:22:58 smithi081 ceph-mon[31453]: pgmap v10041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:22:58 smithi153 ceph-mon[36569]: pgmap v10041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:22:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:22:58 smithi104 ceph-mon[36624]: pgmap v10041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:00 smithi081 ceph-mon[31453]: pgmap v10042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:00 smithi153 ceph-mon[36569]: pgmap v10042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:00 smithi104 ceph-mon[36624]: pgmap v10042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:02 smithi081 ceph-mon[31453]: pgmap v10043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:02 smithi153 ceph-mon[36569]: pgmap v10043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:02 smithi104 ceph-mon[36624]: pgmap v10043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:04 smithi081 ceph-mon[31453]: pgmap v10044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:04 smithi153 ceph-mon[36569]: pgmap v10044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:04 smithi104 ceph-mon[36624]: pgmap v10044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:06 smithi153 ceph-mon[36569]: pgmap v10045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:06 smithi104 ceph-mon[36624]: pgmap v10045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:06 smithi081 ceph-mon[31453]: pgmap v10045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:06.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:06.801+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:23:06.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:06.801+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:23:06.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:06.801+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:23:08.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:08 smithi081 ceph-mon[31453]: pgmap v10046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:08 smithi153 ceph-mon[36569]: pgmap v10046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:08 smithi104 ceph-mon[36624]: pgmap v10046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:10 smithi153 ceph-mon[36569]: pgmap v10047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:10 smithi104 ceph-mon[36624]: pgmap v10047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:10 smithi081 ceph-mon[31453]: pgmap v10047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:12 smithi153 ceph-mon[36569]: pgmap v10048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:23:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:12 smithi104 ceph-mon[36624]: pgmap v10048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:23:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:12 smithi081 ceph-mon[31453]: pgmap v10048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:23:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:23:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:23:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:23:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:23:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:23:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:23:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:23:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:23:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:23:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:14 smithi153 ceph-mon[36569]: pgmap v10049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:14 smithi153 ceph-mon[36569]: pgmap v10050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:14 smithi104 ceph-mon[36624]: pgmap v10049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:14 smithi104 ceph-mon[36624]: pgmap v10050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:14 smithi081 ceph-mon[31453]: pgmap v10049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:14 smithi081 ceph-mon[31453]: pgmap v10050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:16 smithi153 ceph-mon[36569]: pgmap v10051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:16 smithi104 ceph-mon[36624]: pgmap v10051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:16 smithi081 ceph-mon[31453]: pgmap v10051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:18 smithi153 ceph-mon[36569]: pgmap v10052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:18 smithi104 ceph-mon[36624]: pgmap v10052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:18 smithi081 ceph-mon[31453]: pgmap v10052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:20 smithi153 ceph-mon[36569]: pgmap v10053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:20 smithi104 ceph-mon[36624]: pgmap v10053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:20 smithi081 ceph-mon[31453]: pgmap v10053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:21.803+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:21.803+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:23:21.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:21.803+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:23:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:22 smithi153 ceph-mon[36569]: pgmap v10054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:22 smithi104 ceph-mon[36624]: pgmap v10054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:22 smithi081 ceph-mon[31453]: pgmap v10054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:24 smithi153 ceph-mon[36569]: pgmap v10055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:24 smithi104 ceph-mon[36624]: pgmap v10055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:24 smithi081 ceph-mon[31453]: pgmap v10055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:26 smithi153 ceph-mon[36569]: pgmap v10056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:26 smithi104 ceph-mon[36624]: pgmap v10056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:26 smithi081 ceph-mon[31453]: pgmap v10056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:28 smithi153 ceph-mon[36569]: pgmap v10057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:28 smithi104 ceph-mon[36624]: pgmap v10057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:28 smithi081 ceph-mon[31453]: pgmap v10057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:30 smithi153 ceph-mon[36569]: pgmap v10058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:30 smithi104 ceph-mon[36624]: pgmap v10058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:30 smithi081 ceph-mon[31453]: pgmap v10058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:32 smithi153 ceph-mon[36569]: pgmap v10059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:32 smithi104 ceph-mon[36624]: pgmap v10059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:32 smithi081 ceph-mon[31453]: pgmap v10059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:34 smithi153 ceph-mon[36569]: pgmap v10060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:34 smithi104 ceph-mon[36624]: pgmap v10060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:34 smithi081 ceph-mon[31453]: pgmap v10060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:36 smithi081 ceph-mon[31453]: pgmap v10061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:36 smithi153 ceph-mon[36569]: pgmap v10061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:36 smithi104 ceph-mon[36624]: pgmap v10061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:36.817+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:23:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:36.818+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:23:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:36.818+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:23:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:38 smithi153 ceph-mon[36569]: pgmap v10062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:38 smithi104 ceph-mon[36624]: pgmap v10062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:38 smithi081 ceph-mon[31453]: pgmap v10062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:40 smithi153 ceph-mon[36569]: pgmap v10063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:40 smithi104 ceph-mon[36624]: pgmap v10063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:40 smithi081 ceph-mon[31453]: pgmap v10063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:42 smithi153 ceph-mon[36569]: pgmap v10064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:42 smithi104 ceph-mon[36624]: pgmap v10064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:42 smithi081 ceph-mon[31453]: pgmap v10064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:44 smithi153 ceph-mon[36569]: pgmap v10065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:44 smithi104 ceph-mon[36624]: pgmap v10065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:44 smithi081 ceph-mon[31453]: pgmap v10065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:46 smithi153 ceph-mon[36569]: pgmap v10066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:46 smithi104 ceph-mon[36624]: pgmap v10066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:46 smithi081 ceph-mon[31453]: pgmap v10066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:48 smithi153 ceph-mon[36569]: pgmap v10067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:48 smithi104 ceph-mon[36624]: pgmap v10067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:48 smithi081 ceph-mon[31453]: pgmap v10067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:50 smithi153 ceph-mon[36569]: pgmap v10068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:50 smithi104 ceph-mon[36624]: pgmap v10068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:50 smithi081 ceph-mon[31453]: pgmap v10068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:51.833+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:23:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:51.833+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:23:51.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:23:51.833+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:23:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:52 smithi153 ceph-mon[36569]: pgmap v10069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:52 smithi104 ceph-mon[36624]: pgmap v10069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:52 smithi081 ceph-mon[31453]: pgmap v10069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:54 smithi153 ceph-mon[36569]: pgmap v10070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:54 smithi104 ceph-mon[36624]: pgmap v10070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:54 smithi081 ceph-mon[31453]: pgmap v10070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:56 smithi153 ceph-mon[36569]: pgmap v10071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:56 smithi104 ceph-mon[36624]: pgmap v10071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:56 smithi081 ceph-mon[31453]: pgmap v10071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:23:58 smithi104 ceph-mon[36624]: pgmap v10072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:23:58 smithi081 ceph-mon[31453]: pgmap v10072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:23:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:23:58 smithi153 ceph-mon[36569]: pgmap v10072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:00 smithi104 ceph-mon[36624]: pgmap v10073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:00.608 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:00 smithi153 ceph-mon[36569]: pgmap v10073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:00 smithi081 ceph-mon[31453]: pgmap v10073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:02 smithi104 ceph-mon[36624]: pgmap v10074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:02 smithi081 ceph-mon[31453]: pgmap v10074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:02 smithi153 ceph-mon[36569]: pgmap v10074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:04 smithi104 ceph-mon[36624]: pgmap v10075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:04 smithi081 ceph-mon[31453]: pgmap v10075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:04 smithi153 ceph-mon[36569]: pgmap v10075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:06 smithi104 ceph-mon[36624]: pgmap v10076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:06 smithi081 ceph-mon[31453]: pgmap v10076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:06 smithi153 ceph-mon[36569]: pgmap v10076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:06.848+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:24:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:06.848+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:24:06.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:06.848+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:24:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:08 smithi081 ceph-mon[31453]: pgmap v10077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:08 smithi153 ceph-mon[36569]: pgmap v10077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:08 smithi104 ceph-mon[36624]: pgmap v10077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:10 smithi081 ceph-mon[31453]: pgmap v10078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:10 smithi153 ceph-mon[36569]: pgmap v10078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:10 smithi104 ceph-mon[36624]: pgmap v10078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:12.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:12 smithi104 ceph-mon[36624]: pgmap v10079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:12 smithi081 ceph-mon[31453]: pgmap v10079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:12 smithi153 ceph-mon[36569]: pgmap v10079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:24:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:24:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:24:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: pgmap v10080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:24:14.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:24:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: pgmap v10080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:24:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:24:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:24:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: pgmap v10080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:24:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:24:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:15 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:24:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:15 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:24:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:15 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:24:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:15 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:24:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:15 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:24:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:15 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:24:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:15 smithi081 ceph-mon[31453]: pgmap v10081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:15 smithi081 ceph-mon[31453]: pgmap v10082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:15 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:24:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:15 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:24:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:15 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:24:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:15 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:24:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:15 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:24:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:15 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:24:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:15 smithi153 ceph-mon[36569]: pgmap v10081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:15 smithi153 ceph-mon[36569]: pgmap v10082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:15 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:24:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:15 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:24:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:15 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:24:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:15 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:24:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:15 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:24:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:15 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:24:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:15 smithi104 ceph-mon[36624]: pgmap v10081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:15 smithi104 ceph-mon[36624]: pgmap v10082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:17 smithi081 ceph-mon[31453]: pgmap v10083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:17 smithi153 ceph-mon[36569]: pgmap v10083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:17 smithi104 ceph-mon[36624]: pgmap v10083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:18 smithi153 ceph-mon[36569]: pgmap v10084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:18 smithi104 ceph-mon[36624]: pgmap v10084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:18 smithi081 ceph-mon[31453]: pgmap v10084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:21 smithi081 ceph-mon[31453]: pgmap v10085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:21 smithi153 ceph-mon[36569]: pgmap v10085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:21 smithi104 ceph-mon[36624]: pgmap v10085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:21.864+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:24:21.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:21.864+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:24:21.865 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:21.864+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:24:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:23 smithi081 ceph-mon[31453]: pgmap v10086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:23 smithi153 ceph-mon[36569]: pgmap v10086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:23 smithi104 ceph-mon[36624]: pgmap v10086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:25 smithi081 ceph-mon[31453]: pgmap v10087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:25 smithi153 ceph-mon[36569]: pgmap v10087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:25 smithi104 ceph-mon[36624]: pgmap v10087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:27 smithi081 ceph-mon[31453]: pgmap v10088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:27 smithi153 ceph-mon[36569]: pgmap v10088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:27 smithi104 ceph-mon[36624]: pgmap v10088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:29 smithi081 ceph-mon[31453]: pgmap v10089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:29 smithi153 ceph-mon[36569]: pgmap v10089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:29 smithi104 ceph-mon[36624]: pgmap v10089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:31 smithi081 ceph-mon[31453]: pgmap v10090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:31 smithi153 ceph-mon[36569]: pgmap v10090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:31 smithi104 ceph-mon[36624]: pgmap v10090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:33 smithi081 ceph-mon[31453]: pgmap v10091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:33 smithi153 ceph-mon[36569]: pgmap v10091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:33 smithi104 ceph-mon[36624]: pgmap v10091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:35 smithi081 ceph-mon[31453]: pgmap v10092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:35 smithi153 ceph-mon[36569]: pgmap v10092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:35 smithi104 ceph-mon[36624]: pgmap v10092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:36.879+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:24:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:36.879+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:24:36.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:36.879+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:24:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:37 smithi081 ceph-mon[31453]: pgmap v10093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:37 smithi153 ceph-mon[36569]: pgmap v10093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:37 smithi104 ceph-mon[36624]: pgmap v10093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:39 smithi153 ceph-mon[36569]: pgmap v10094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:39 smithi104 ceph-mon[36624]: pgmap v10094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:39 smithi081 ceph-mon[31453]: pgmap v10094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:41 smithi153 ceph-mon[36569]: pgmap v10095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:41 smithi104 ceph-mon[36624]: pgmap v10095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:41 smithi081 ceph-mon[31453]: pgmap v10095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:43 smithi153 ceph-mon[36569]: pgmap v10096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:43 smithi104 ceph-mon[36624]: pgmap v10096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:43 smithi081 ceph-mon[31453]: pgmap v10096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:45 smithi153 ceph-mon[36569]: pgmap v10097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:45 smithi104 ceph-mon[36624]: pgmap v10097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:45 smithi081 ceph-mon[31453]: pgmap v10097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:47 smithi153 ceph-mon[36569]: pgmap v10098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:47 smithi104 ceph-mon[36624]: pgmap v10098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:47 smithi081 ceph-mon[31453]: pgmap v10098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:49 smithi153 ceph-mon[36569]: pgmap v10099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:49 smithi104 ceph-mon[36624]: pgmap v10099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:49 smithi081 ceph-mon[31453]: pgmap v10099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:51 smithi153 ceph-mon[36569]: pgmap v10100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:51 smithi104 ceph-mon[36624]: pgmap v10100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:51 smithi081 ceph-mon[31453]: pgmap v10100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:51.894+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:24:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:51.894+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:24:51.896 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:24:51.894+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:24:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:53 smithi153 ceph-mon[36569]: pgmap v10101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:53 smithi104 ceph-mon[36624]: pgmap v10101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:53 smithi081 ceph-mon[31453]: pgmap v10101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:55 smithi153 ceph-mon[36569]: pgmap v10102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:55 smithi104 ceph-mon[36624]: pgmap v10102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:55 smithi081 ceph-mon[31453]: pgmap v10102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:57 smithi153 ceph-mon[36569]: pgmap v10103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:57 smithi104 ceph-mon[36624]: pgmap v10103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:57 smithi081 ceph-mon[31453]: pgmap v10103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:24:59 smithi153 ceph-mon[36569]: pgmap v10104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:24:59 smithi104 ceph-mon[36624]: pgmap v10104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:24:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:24:59 smithi081 ceph-mon[31453]: pgmap v10104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:01 smithi153 ceph-mon[36569]: pgmap v10105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:01 smithi104 ceph-mon[36624]: pgmap v10105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:01 smithi081 ceph-mon[31453]: pgmap v10105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:03 smithi153 ceph-mon[36569]: pgmap v10106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:03 smithi104 ceph-mon[36624]: pgmap v10106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:03 smithi081 ceph-mon[31453]: pgmap v10106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:05 smithi153 ceph-mon[36569]: pgmap v10107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:05 smithi104 ceph-mon[36624]: pgmap v10107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:05 smithi081 ceph-mon[31453]: pgmap v10107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:06.904 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:06.904+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:25:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:06.904+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:25:06.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:06.904+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:25:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:07 smithi153 ceph-mon[36569]: pgmap v10108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:07 smithi104 ceph-mon[36624]: pgmap v10108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:07 smithi081 ceph-mon[31453]: pgmap v10108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:09 smithi153 ceph-mon[36569]: pgmap v10109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:09 smithi104 ceph-mon[36624]: pgmap v10109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:09 smithi081 ceph-mon[31453]: pgmap v10109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:11 smithi153 ceph-mon[36569]: pgmap v10110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:11 smithi104 ceph-mon[36624]: pgmap v10110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:11 smithi081 ceph-mon[31453]: pgmap v10110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:13 smithi153 ceph-mon[36569]: pgmap v10111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:13 smithi104 ceph-mon[36624]: pgmap v10111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:13 smithi081 ceph-mon[31453]: pgmap v10111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:15 smithi153 ceph-mon[36569]: pgmap v10112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:25:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:15 smithi104 ceph-mon[36624]: pgmap v10112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:25:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:15 smithi081 ceph-mon[31453]: pgmap v10112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:25:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:15.763+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:25:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:15.763+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:25:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:15.763+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:25:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:15.764+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:25:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:25:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:25:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:25:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:25:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:25:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:25:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:25:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:25:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:25:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:17 smithi153 ceph-mon[36569]: pgmap v10113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:17.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:17 smithi153 ceph-mon[36569]: pgmap v10114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:17 smithi104 ceph-mon[36624]: pgmap v10113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:17 smithi104 ceph-mon[36624]: pgmap v10114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:17 smithi081 ceph-mon[31453]: pgmap v10113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:17 smithi081 ceph-mon[31453]: pgmap v10114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:19 smithi153 ceph-mon[36569]: pgmap v10115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:19 smithi104 ceph-mon[36624]: pgmap v10115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:19 smithi081 ceph-mon[31453]: pgmap v10115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:21 smithi153 ceph-mon[36569]: pgmap v10116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:21 smithi104 ceph-mon[36624]: pgmap v10116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:21 smithi081 ceph-mon[31453]: pgmap v10116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:21.912+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:25:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:21.912+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:25:21.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:21.912+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:25:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:23 smithi153 ceph-mon[36569]: pgmap v10117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:23 smithi104 ceph-mon[36624]: pgmap v10117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:23 smithi081 ceph-mon[31453]: pgmap v10117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:25 smithi153 ceph-mon[36569]: pgmap v10118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:25 smithi104 ceph-mon[36624]: pgmap v10118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:25 smithi081 ceph-mon[31453]: pgmap v10118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:27 smithi153 ceph-mon[36569]: pgmap v10119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:27 smithi104 ceph-mon[36624]: pgmap v10119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:27 smithi081 ceph-mon[31453]: pgmap v10119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:29 smithi153 ceph-mon[36569]: pgmap v10120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:29 smithi104 ceph-mon[36624]: pgmap v10120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:29 smithi081 ceph-mon[31453]: pgmap v10120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:31 smithi153 ceph-mon[36569]: pgmap v10121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:31 smithi104 ceph-mon[36624]: pgmap v10121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:31 smithi081 ceph-mon[31453]: pgmap v10121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:33 smithi104 ceph-mon[36624]: pgmap v10122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:33 smithi081 ceph-mon[31453]: pgmap v10122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:33 smithi153 ceph-mon[36569]: pgmap v10122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:35 smithi104 ceph-mon[36624]: pgmap v10123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:35 smithi081 ceph-mon[31453]: pgmap v10123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:35 smithi153 ceph-mon[36569]: pgmap v10123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:36.928+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:25:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:36.928+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:25:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:36.928+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:25:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:37 smithi104 ceph-mon[36624]: pgmap v10124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:37 smithi081 ceph-mon[31453]: pgmap v10124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:37 smithi153 ceph-mon[36569]: pgmap v10124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:39 smithi104 ceph-mon[36624]: pgmap v10125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:39 smithi081 ceph-mon[31453]: pgmap v10125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:39 smithi153 ceph-mon[36569]: pgmap v10125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:41 smithi104 ceph-mon[36624]: pgmap v10126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:41 smithi081 ceph-mon[31453]: pgmap v10126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:41 smithi153 ceph-mon[36569]: pgmap v10126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:43 smithi081 ceph-mon[31453]: pgmap v10127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:43 smithi153 ceph-mon[36569]: pgmap v10127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:43 smithi104 ceph-mon[36624]: pgmap v10127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:45 smithi081 ceph-mon[31453]: pgmap v10128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:45 smithi153 ceph-mon[36569]: pgmap v10128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:45 smithi104 ceph-mon[36624]: pgmap v10128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:47 smithi081 ceph-mon[31453]: pgmap v10129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:47 smithi153 ceph-mon[36569]: pgmap v10129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:47 smithi104 ceph-mon[36624]: pgmap v10129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:49 smithi081 ceph-mon[31453]: pgmap v10130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:49 smithi153 ceph-mon[36569]: pgmap v10130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:49 smithi104 ceph-mon[36624]: pgmap v10130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:51 smithi081 ceph-mon[31453]: pgmap v10131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:51 smithi153 ceph-mon[36569]: pgmap v10131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:51 smithi104 ceph-mon[36624]: pgmap v10131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:51.942+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:25:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:51.942+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:25:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:25:51.942+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:25:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:53 smithi081 ceph-mon[31453]: pgmap v10132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:53 smithi153 ceph-mon[36569]: pgmap v10132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:53 smithi104 ceph-mon[36624]: pgmap v10132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:55 smithi081 ceph-mon[31453]: pgmap v10133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:55 smithi153 ceph-mon[36569]: pgmap v10133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:55 smithi104 ceph-mon[36624]: pgmap v10133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:57 smithi081 ceph-mon[31453]: pgmap v10134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:57 smithi153 ceph-mon[36569]: pgmap v10134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:57 smithi104 ceph-mon[36624]: pgmap v10134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:25:59 smithi081 ceph-mon[31453]: pgmap v10135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:25:59 smithi153 ceph-mon[36569]: pgmap v10135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:25:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:25:59 smithi104 ceph-mon[36624]: pgmap v10135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:01 smithi081 ceph-mon[31453]: pgmap v10136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:01 smithi153 ceph-mon[36569]: pgmap v10136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:01 smithi104 ceph-mon[36624]: pgmap v10136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:03 smithi081 ceph-mon[31453]: pgmap v10137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:03.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:03 smithi153 ceph-mon[36569]: pgmap v10137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:03 smithi104 ceph-mon[36624]: pgmap v10137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:05 smithi081 ceph-mon[31453]: pgmap v10138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:05 smithi153 ceph-mon[36569]: pgmap v10138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:05 smithi104 ceph-mon[36624]: pgmap v10138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:06.949+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:26:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:06.949+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:26:06.950 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:06.949+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:26:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:07 smithi081 ceph-mon[31453]: pgmap v10139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:07 smithi153 ceph-mon[36569]: pgmap v10139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:07 smithi104 ceph-mon[36624]: pgmap v10139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:08 smithi153 ceph-mon[36569]: pgmap v10140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:08 smithi104 ceph-mon[36624]: pgmap v10140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:08 smithi081 ceph-mon[31453]: pgmap v10140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:10 smithi153 ceph-mon[36569]: pgmap v10141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:10 smithi104 ceph-mon[36624]: pgmap v10141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:10 smithi081 ceph-mon[31453]: pgmap v10141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:12 smithi153 ceph-mon[36569]: pgmap v10142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:12 smithi104 ceph-mon[36624]: pgmap v10142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:12 smithi081 ceph-mon[31453]: pgmap v10142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:14 smithi153 ceph-mon[36569]: pgmap v10143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:14 smithi104 ceph-mon[36624]: pgmap v10143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:14 smithi081 ceph-mon[31453]: pgmap v10143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:16 smithi153 ceph-mon[36569]: pgmap v10144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:26:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:16 smithi104 ceph-mon[36624]: pgmap v10144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:26:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:16 smithi081 ceph-mon[31453]: pgmap v10144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:26:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:26:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:26:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:17 smithi153 ceph-mon[36569]: pgmap v10145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:17 smithi153 ceph-mon[36569]: pgmap v10146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:26:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:26:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:26:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:17 smithi104 ceph-mon[36624]: pgmap v10145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:17 smithi104 ceph-mon[36624]: pgmap v10146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:26:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:26:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:26:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:17 smithi081 ceph-mon[31453]: pgmap v10145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:17 smithi081 ceph-mon[31453]: pgmap v10146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:26:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:20 smithi153 ceph-mon[36569]: pgmap v10147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:20.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:20 smithi104 ceph-mon[36624]: pgmap v10147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:20 smithi081 ceph-mon[31453]: pgmap v10147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:21.958+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:26:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:21.959+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:26:21.959 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:21.959+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:26:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:22 smithi153 ceph-mon[36569]: pgmap v10148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:22 smithi104 ceph-mon[36624]: pgmap v10148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:22.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:22 smithi081 ceph-mon[31453]: pgmap v10148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:24 smithi153 ceph-mon[36569]: pgmap v10149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:24 smithi104 ceph-mon[36624]: pgmap v10149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:24 smithi081 ceph-mon[31453]: pgmap v10149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:26 smithi153 ceph-mon[36569]: pgmap v10150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:26 smithi104 ceph-mon[36624]: pgmap v10150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:26 smithi081 ceph-mon[31453]: pgmap v10150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:28 smithi153 ceph-mon[36569]: pgmap v10151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:28 smithi104 ceph-mon[36624]: pgmap v10151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:28 smithi081 ceph-mon[31453]: pgmap v10151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:30 smithi153 ceph-mon[36569]: pgmap v10152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:30 smithi104 ceph-mon[36624]: pgmap v10152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:30 smithi081 ceph-mon[31453]: pgmap v10152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:32 smithi153 ceph-mon[36569]: pgmap v10153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:32 smithi104 ceph-mon[36624]: pgmap v10153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:32 smithi081 ceph-mon[31453]: pgmap v10153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:34 smithi104 ceph-mon[36624]: pgmap v10154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:34 smithi081 ceph-mon[31453]: pgmap v10154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:34 smithi153 ceph-mon[36569]: pgmap v10154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:36 smithi104 ceph-mon[36624]: pgmap v10155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:36 smithi081 ceph-mon[31453]: pgmap v10155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:36 smithi153 ceph-mon[36569]: pgmap v10155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:36.974+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:26:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:36.974+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:26:36.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:36.974+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:26:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:38 smithi104 ceph-mon[36624]: pgmap v10156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:38 smithi081 ceph-mon[31453]: pgmap v10156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:38 smithi153 ceph-mon[36569]: pgmap v10156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:40 smithi104 ceph-mon[36624]: pgmap v10157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:40 smithi081 ceph-mon[31453]: pgmap v10157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:40 smithi153 ceph-mon[36569]: pgmap v10157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:42 smithi104 ceph-mon[36624]: pgmap v10158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:42 smithi081 ceph-mon[31453]: pgmap v10158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:42 smithi153 ceph-mon[36569]: pgmap v10158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:44 smithi081 ceph-mon[31453]: pgmap v10159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:44 smithi153 ceph-mon[36569]: pgmap v10159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:44 smithi104 ceph-mon[36624]: pgmap v10159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:46 smithi081 ceph-mon[31453]: pgmap v10160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:46 smithi153 ceph-mon[36569]: pgmap v10160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:46 smithi104 ceph-mon[36624]: pgmap v10160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:48 smithi081 ceph-mon[31453]: pgmap v10161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:48 smithi153 ceph-mon[36569]: pgmap v10161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:48 smithi104 ceph-mon[36624]: pgmap v10161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:50 smithi081 ceph-mon[31453]: pgmap v10162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:50 smithi153 ceph-mon[36569]: pgmap v10162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:50 smithi104 ceph-mon[36624]: pgmap v10162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:51.978+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:26:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:51.978+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:26:51.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:26:51.979+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:26:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:52 smithi081 ceph-mon[31453]: pgmap v10163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:52 smithi153 ceph-mon[36569]: pgmap v10163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:52 smithi104 ceph-mon[36624]: pgmap v10163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:54 smithi081 ceph-mon[31453]: pgmap v10164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:54 smithi153 ceph-mon[36569]: pgmap v10164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:54 smithi104 ceph-mon[36624]: pgmap v10164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:56 smithi081 ceph-mon[31453]: pgmap v10165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:56 smithi153 ceph-mon[36569]: pgmap v10165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:56 smithi104 ceph-mon[36624]: pgmap v10165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:26:58 smithi081 ceph-mon[31453]: pgmap v10166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:26:58 smithi153 ceph-mon[36569]: pgmap v10166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:26:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:26:58 smithi104 ceph-mon[36624]: pgmap v10166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:00 smithi081 ceph-mon[31453]: pgmap v10167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:00 smithi153 ceph-mon[36569]: pgmap v10167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:00 smithi104 ceph-mon[36624]: pgmap v10167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:02 smithi081 ceph-mon[31453]: pgmap v10168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:02 smithi153 ceph-mon[36569]: pgmap v10168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:02 smithi104 ceph-mon[36624]: pgmap v10168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:04 smithi081 ceph-mon[31453]: pgmap v10169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:04 smithi153 ceph-mon[36569]: pgmap v10169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:04 smithi104 ceph-mon[36624]: pgmap v10169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:06 smithi081 ceph-mon[31453]: pgmap v10170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:06 smithi153 ceph-mon[36569]: pgmap v10170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:06 smithi104 ceph-mon[36624]: pgmap v10170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:06.987+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:27:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:06.987+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:27:06.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:06.988+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:27:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:08 smithi081 ceph-mon[31453]: pgmap v10171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:08 smithi153 ceph-mon[36569]: pgmap v10171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:08 smithi104 ceph-mon[36624]: pgmap v10171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:10 smithi081 ceph-mon[31453]: pgmap v10172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:10 smithi153 ceph-mon[36569]: pgmap v10172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:10 smithi104 ceph-mon[36624]: pgmap v10172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:12 smithi153 ceph-mon[36569]: pgmap v10173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:12 smithi104 ceph-mon[36624]: pgmap v10173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:12 smithi081 ceph-mon[31453]: pgmap v10173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:13 smithi153 ceph-mon[36569]: pgmap v10174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:13 smithi104 ceph-mon[36624]: pgmap v10174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:13 smithi081 ceph-mon[31453]: pgmap v10174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:16 smithi153 ceph-mon[36569]: pgmap v10175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:16 smithi104 ceph-mon[36624]: pgmap v10175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:16 smithi081 ceph-mon[31453]: pgmap v10175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:18 smithi104 ceph-mon[36624]: pgmap v10176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:27:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:18 smithi081 ceph-mon[31453]: pgmap v10176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:27:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:18 smithi153 ceph-mon[36569]: pgmap v10176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:27:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:27:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:27:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:27:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:27:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:27:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:27:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:27:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:27:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:27:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:20 smithi081 ceph-mon[31453]: pgmap v10177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:20 smithi081 ceph-mon[31453]: pgmap v10178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:20 smithi153 ceph-mon[36569]: pgmap v10177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:20 smithi153 ceph-mon[36569]: pgmap v10178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:20 smithi104 ceph-mon[36624]: pgmap v10177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:20 smithi104 ceph-mon[36624]: pgmap v10178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:21.991+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:27:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:21.991+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:27:21.992 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:21.991+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:27:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:22 smithi081 ceph-mon[31453]: pgmap v10179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:22 smithi153 ceph-mon[36569]: pgmap v10179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:22 smithi104 ceph-mon[36624]: pgmap v10179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:24 smithi081 ceph-mon[31453]: pgmap v10180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:24 smithi153 ceph-mon[36569]: pgmap v10180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:24 smithi104 ceph-mon[36624]: pgmap v10180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:26 smithi081 ceph-mon[31453]: pgmap v10181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:26 smithi153 ceph-mon[36569]: pgmap v10181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:26 smithi104 ceph-mon[36624]: pgmap v10181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:28 smithi081 ceph-mon[31453]: pgmap v10182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:28 smithi153 ceph-mon[36569]: pgmap v10182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:28 smithi104 ceph-mon[36624]: pgmap v10182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:30 smithi081 ceph-mon[31453]: pgmap v10183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:30 smithi153 ceph-mon[36569]: pgmap v10183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:30 smithi104 ceph-mon[36624]: pgmap v10183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:32 smithi081 ceph-mon[31453]: pgmap v10184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:32 smithi153 ceph-mon[36569]: pgmap v10184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:32 smithi104 ceph-mon[36624]: pgmap v10184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:34 smithi081 ceph-mon[31453]: pgmap v10185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:34 smithi153 ceph-mon[36569]: pgmap v10185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:34 smithi104 ceph-mon[36624]: pgmap v10185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:36 smithi081 ceph-mon[31453]: pgmap v10186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:36 smithi153 ceph-mon[36569]: pgmap v10186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:36 smithi104 ceph-mon[36624]: pgmap v10186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:37.000+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:27:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:37.000+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:27:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:37.000+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:27:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:38 smithi081 ceph-mon[31453]: pgmap v10187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:38 smithi153 ceph-mon[36569]: pgmap v10187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:38 smithi104 ceph-mon[36624]: pgmap v10187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:40 smithi081 ceph-mon[31453]: pgmap v10188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:40.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:40 smithi153 ceph-mon[36569]: pgmap v10188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:40 smithi104 ceph-mon[36624]: pgmap v10188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:42 smithi081 ceph-mon[31453]: pgmap v10189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:42 smithi153 ceph-mon[36569]: pgmap v10189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:42 smithi104 ceph-mon[36624]: pgmap v10189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:44 smithi081 ceph-mon[31453]: pgmap v10190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:44 smithi153 ceph-mon[36569]: pgmap v10190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:44 smithi104 ceph-mon[36624]: pgmap v10190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:46 smithi081 ceph-mon[31453]: pgmap v10191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:46 smithi153 ceph-mon[36569]: pgmap v10191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:46 smithi104 ceph-mon[36624]: pgmap v10191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:48 smithi104 ceph-mon[36624]: pgmap v10192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:48 smithi153 ceph-mon[36569]: pgmap v10192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:48 smithi081 ceph-mon[31453]: pgmap v10192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:50 smithi153 ceph-mon[36569]: pgmap v10193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:50 smithi104 ceph-mon[36624]: pgmap v10193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:50 smithi081 ceph-mon[31453]: pgmap v10193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:52.015+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:27:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:52.016+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:27:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:27:52.016+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:27:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:52 smithi153 ceph-mon[36569]: pgmap v10194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:52 smithi104 ceph-mon[36624]: pgmap v10194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:52 smithi081 ceph-mon[31453]: pgmap v10194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:53 smithi153 ceph-mon[36569]: pgmap v10195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:53 smithi104 ceph-mon[36624]: pgmap v10195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:53 smithi081 ceph-mon[31453]: pgmap v10195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:55 smithi081 ceph-mon[31453]: pgmap v10196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:55 smithi153 ceph-mon[36569]: pgmap v10196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:55 smithi104 ceph-mon[36624]: pgmap v10196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:57 smithi104 ceph-mon[36624]: pgmap v10197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:57 smithi081 ceph-mon[31453]: pgmap v10197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:27:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:57 smithi153 ceph-mon[36569]: pgmap v10197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:27:59 smithi081 ceph-mon[31453]: pgmap v10198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:27:59 smithi153 ceph-mon[36569]: pgmap v10198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:27:59 smithi104 ceph-mon[36624]: pgmap v10198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:01 smithi081 ceph-mon[31453]: pgmap v10199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:01 smithi153 ceph-mon[36569]: pgmap v10199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:01 smithi104 ceph-mon[36624]: pgmap v10199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:03 smithi081 ceph-mon[31453]: pgmap v10200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:03 smithi153 ceph-mon[36569]: pgmap v10200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:03 smithi104 ceph-mon[36624]: pgmap v10200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:05 smithi081 ceph-mon[31453]: pgmap v10201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:05 smithi153 ceph-mon[36569]: pgmap v10201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:05 smithi104 ceph-mon[36624]: pgmap v10201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:07.030+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:28:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:07.030+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:28:07.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:07.030+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:28:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:07 smithi081 ceph-mon[31453]: pgmap v10202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:07 smithi153 ceph-mon[36569]: pgmap v10202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:07 smithi104 ceph-mon[36624]: pgmap v10202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:09 smithi081 ceph-mon[31453]: pgmap v10203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:09 smithi153 ceph-mon[36569]: pgmap v10203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:09 smithi104 ceph-mon[36624]: pgmap v10203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:11 smithi081 ceph-mon[31453]: pgmap v10204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:11 smithi153 ceph-mon[36569]: pgmap v10204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:11 smithi104 ceph-mon[36624]: pgmap v10204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:13 smithi081 ceph-mon[31453]: pgmap v10205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:13 smithi153 ceph-mon[36569]: pgmap v10205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:13 smithi104 ceph-mon[36624]: pgmap v10205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:15.793+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:28:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:15.793+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:28:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:15.793+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:28:15.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:15.793+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:28:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:15 smithi081 ceph-mon[31453]: pgmap v10206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:15 smithi153 ceph-mon[36569]: pgmap v10206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:15 smithi104 ceph-mon[36624]: pgmap v10206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:17 smithi153 ceph-mon[36569]: pgmap v10207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:17 smithi104 ceph-mon[36624]: pgmap v10207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:17 smithi081 ceph-mon[31453]: pgmap v10207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:20.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:19 smithi153 ceph-mon[36569]: pgmap v10208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:20.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:28:20.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:19 smithi104 ceph-mon[36624]: pgmap v10208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:28:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:19 smithi081 ceph-mon[31453]: pgmap v10208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:28:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:22.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:22.044+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:28:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:22.044+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:28:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:22.044+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:28:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:21 smithi153 ceph-mon[36569]: pgmap v10209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:21 smithi104 ceph-mon[36624]: pgmap v10209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:21 smithi081 ceph-mon[31453]: pgmap v10209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:24 smithi153 ceph-mon[36569]: pgmap v10210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:24 smithi104 ceph-mon[36624]: pgmap v10210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:24 smithi081 ceph-mon[31453]: pgmap v10210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:26 smithi153 ceph-mon[36569]: pgmap v10211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:28:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:28:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:26 smithi104 ceph-mon[36624]: pgmap v10211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:28:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:28:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:26 smithi081 ceph-mon[31453]: pgmap v10211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:28:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:28:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:28:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:27 smithi153 ceph-mon[36569]: pgmap v10212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:27 smithi153 ceph-mon[36569]: pgmap v10213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:27 smithi104 ceph-mon[36624]: pgmap v10212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:27 smithi104 ceph-mon[36624]: pgmap v10213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:27 smithi081 ceph-mon[31453]: pgmap v10212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:27 smithi081 ceph-mon[31453]: pgmap v10213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:29 smithi153 ceph-mon[36569]: pgmap v10214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:29 smithi104 ceph-mon[36624]: pgmap v10214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:29 smithi081 ceph-mon[31453]: pgmap v10214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:31 smithi153 ceph-mon[36569]: pgmap v10215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:31 smithi104 ceph-mon[36624]: pgmap v10215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:31 smithi081 ceph-mon[31453]: pgmap v10215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:33 smithi153 ceph-mon[36569]: pgmap v10216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:33 smithi104 ceph-mon[36624]: pgmap v10216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:33 smithi081 ceph-mon[31453]: pgmap v10216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:35 smithi153 ceph-mon[36569]: pgmap v10217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:35 smithi104 ceph-mon[36624]: pgmap v10217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:35 smithi081 ceph-mon[31453]: pgmap v10217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:37.060+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:28:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:37.060+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:28:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:37.060+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:28:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:37 smithi153 ceph-mon[36569]: pgmap v10218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:37 smithi104 ceph-mon[36624]: pgmap v10218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:37 smithi081 ceph-mon[31453]: pgmap v10218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:39 smithi153 ceph-mon[36569]: pgmap v10219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:39 smithi104 ceph-mon[36624]: pgmap v10219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:39 smithi081 ceph-mon[31453]: pgmap v10219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:41 smithi153 ceph-mon[36569]: pgmap v10220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:41 smithi104 ceph-mon[36624]: pgmap v10220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:41 smithi081 ceph-mon[31453]: pgmap v10220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:43 smithi153 ceph-mon[36569]: pgmap v10221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:43 smithi104 ceph-mon[36624]: pgmap v10221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:43 smithi081 ceph-mon[31453]: pgmap v10221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:45 smithi153 ceph-mon[36569]: pgmap v10222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:45 smithi104 ceph-mon[36624]: pgmap v10222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:45 smithi081 ceph-mon[31453]: pgmap v10222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:47 smithi153 ceph-mon[36569]: pgmap v10223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:47 smithi104 ceph-mon[36624]: pgmap v10223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:47 smithi081 ceph-mon[31453]: pgmap v10223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:49 smithi153 ceph-mon[36569]: pgmap v10224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:49 smithi104 ceph-mon[36624]: pgmap v10224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:49 smithi081 ceph-mon[31453]: pgmap v10224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:51 smithi153 ceph-mon[36569]: pgmap v10225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:51 smithi104 ceph-mon[36624]: pgmap v10225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:51 smithi081 ceph-mon[31453]: pgmap v10225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:52.073+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:28:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:52.073+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:28:52.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:28:52.073+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:28:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:53 smithi153 ceph-mon[36569]: pgmap v10226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:53 smithi104 ceph-mon[36624]: pgmap v10226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:53 smithi081 ceph-mon[31453]: pgmap v10226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:55 smithi153 ceph-mon[36569]: pgmap v10227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:55 smithi104 ceph-mon[36624]: pgmap v10227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:55 smithi081 ceph-mon[31453]: pgmap v10227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:57 smithi104 ceph-mon[36624]: pgmap v10228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:57 smithi081 ceph-mon[31453]: pgmap v10228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:57 smithi153 ceph-mon[36569]: pgmap v10228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:28:59 smithi104 ceph-mon[36624]: pgmap v10229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:28:59 smithi081 ceph-mon[31453]: pgmap v10229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:28:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:28:59 smithi153 ceph-mon[36569]: pgmap v10229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:01 smithi104 ceph-mon[36624]: pgmap v10230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:01 smithi081 ceph-mon[31453]: pgmap v10230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:01 smithi153 ceph-mon[36569]: pgmap v10230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:03 smithi104 ceph-mon[36624]: pgmap v10231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:03.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:03 smithi153 ceph-mon[36569]: pgmap v10231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:03 smithi081 ceph-mon[31453]: pgmap v10231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:05 smithi104 ceph-mon[36624]: pgmap v10232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:05 smithi081 ceph-mon[31453]: pgmap v10232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:05 smithi153 ceph-mon[36569]: pgmap v10232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:07.079+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:29:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:07.079+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:29:07.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:07.079+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:29:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:07 smithi081 ceph-mon[31453]: pgmap v10233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:07 smithi153 ceph-mon[36569]: pgmap v10233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:07 smithi104 ceph-mon[36624]: pgmap v10233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:09 smithi081 ceph-mon[31453]: pgmap v10234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:09 smithi153 ceph-mon[36569]: pgmap v10234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:09 smithi104 ceph-mon[36624]: pgmap v10234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:11 smithi081 ceph-mon[31453]: pgmap v10235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:11 smithi153 ceph-mon[36569]: pgmap v10235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:11 smithi104 ceph-mon[36624]: pgmap v10235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:13 smithi081 ceph-mon[31453]: pgmap v10236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:13 smithi153 ceph-mon[36569]: pgmap v10236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:13 smithi104 ceph-mon[36624]: pgmap v10236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:15 smithi081 ceph-mon[31453]: pgmap v10237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:15 smithi153 ceph-mon[36569]: pgmap v10237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:15 smithi104 ceph-mon[36624]: pgmap v10237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:17.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:17 smithi104 ceph-mon[36624]: pgmap v10238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:17 smithi081 ceph-mon[31453]: pgmap v10238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:17 smithi153 ceph-mon[36569]: pgmap v10238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:19 smithi081 ceph-mon[31453]: pgmap v10239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:19 smithi153 ceph-mon[36569]: pgmap v10239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:19 smithi104 ceph-mon[36624]: pgmap v10239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:21 smithi081 ceph-mon[31453]: pgmap v10240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:21 smithi153 ceph-mon[36569]: pgmap v10240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:21 smithi104 ceph-mon[36624]: pgmap v10240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:22.096+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:29:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:22.096+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:29:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:22.096+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:29:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:23 smithi081 ceph-mon[31453]: pgmap v10241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:23 smithi153 ceph-mon[36569]: pgmap v10241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:23 smithi104 ceph-mon[36624]: pgmap v10241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:25 smithi081 ceph-mon[31453]: pgmap v10242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:25 smithi153 ceph-mon[36569]: pgmap v10242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:25 smithi104 ceph-mon[36624]: pgmap v10242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:29:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:29:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:29:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:27 smithi081 ceph-mon[31453]: pgmap v10243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:29:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:29:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:27 smithi153 ceph-mon[36569]: pgmap v10243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:29:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:29:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:27 smithi104 ceph-mon[36624]: pgmap v10243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:29:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:29:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:29:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:28 smithi081 ceph-mon[31453]: pgmap v10244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:28 smithi081 ceph-mon[31453]: pgmap v10245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:28 smithi153 ceph-mon[36569]: pgmap v10244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:28 smithi153 ceph-mon[36569]: pgmap v10245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:28 smithi104 ceph-mon[36624]: pgmap v10244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:28 smithi104 ceph-mon[36624]: pgmap v10245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:30 smithi153 ceph-mon[36569]: pgmap v10246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:30 smithi104 ceph-mon[36624]: pgmap v10246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:30 smithi081 ceph-mon[31453]: pgmap v10246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:32 smithi153 ceph-mon[36569]: pgmap v10247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:32 smithi104 ceph-mon[36624]: pgmap v10247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:32 smithi081 ceph-mon[31453]: pgmap v10247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:34 smithi153 ceph-mon[36569]: pgmap v10248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:34 smithi104 ceph-mon[36624]: pgmap v10248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:34 smithi081 ceph-mon[31453]: pgmap v10248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:36 smithi153 ceph-mon[36569]: pgmap v10249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:36 smithi104 ceph-mon[36624]: pgmap v10249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:36 smithi081 ceph-mon[31453]: pgmap v10249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:37.111+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:29:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:37.111+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:29:37.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:37.111+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:29:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:38 smithi153 ceph-mon[36569]: pgmap v10250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:38 smithi104 ceph-mon[36624]: pgmap v10250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:38 smithi081 ceph-mon[31453]: pgmap v10250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:40 smithi153 ceph-mon[36569]: pgmap v10251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:40 smithi104 ceph-mon[36624]: pgmap v10251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:40 smithi081 ceph-mon[31453]: pgmap v10251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:42 smithi153 ceph-mon[36569]: pgmap v10252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:42 smithi104 ceph-mon[36624]: pgmap v10252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:42 smithi081 ceph-mon[31453]: pgmap v10252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:43 smithi153 ceph-mon[36569]: pgmap v10253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:43.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:43 smithi081 ceph-mon[31453]: pgmap v10253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:43 smithi104 ceph-mon[36624]: pgmap v10253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:45.809+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:29:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:45.809+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:29:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:45.809+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:29:45.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:45.809+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:29:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:46 smithi153 ceph-mon[36569]: pgmap v10254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:46 smithi104 ceph-mon[36624]: pgmap v10254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:46 smithi081 ceph-mon[31453]: pgmap v10254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:48 smithi153 ceph-mon[36569]: pgmap v10255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:48 smithi104 ceph-mon[36624]: pgmap v10255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:48 smithi081 ceph-mon[31453]: pgmap v10255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:50 smithi153 ceph-mon[36569]: pgmap v10256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:50 smithi104 ceph-mon[36624]: pgmap v10256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:50 smithi081 ceph-mon[31453]: pgmap v10256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:52.123+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:29:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:52.123+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:29:52.124 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:29:52.123+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:29:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:52 smithi153 ceph-mon[36569]: pgmap v10257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:52 smithi104 ceph-mon[36624]: pgmap v10257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:52 smithi081 ceph-mon[31453]: pgmap v10257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:54 smithi153 ceph-mon[36569]: pgmap v10258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:54 smithi104 ceph-mon[36624]: pgmap v10258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:54 smithi081 ceph-mon[31453]: pgmap v10258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:56 smithi153 ceph-mon[36569]: pgmap v10259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:56 smithi104 ceph-mon[36624]: pgmap v10259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:56 smithi081 ceph-mon[31453]: pgmap v10259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:29:58 smithi153 ceph-mon[36569]: pgmap v10260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:29:58 smithi104 ceph-mon[36624]: pgmap v10260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:29:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:29:58 smithi081 ceph-mon[31453]: pgmap v10260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:00 smithi153 ceph-mon[36569]: pgmap v10261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:30:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:00 smithi104 ceph-mon[36624]: pgmap v10261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:30:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:00 smithi081 ceph-mon[31453]: pgmap v10261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:30:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:02 smithi153 ceph-mon[36569]: pgmap v10262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:02 smithi104 ceph-mon[36624]: pgmap v10262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:02 smithi081 ceph-mon[31453]: pgmap v10262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:04 smithi153 ceph-mon[36569]: pgmap v10263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:04 smithi104 ceph-mon[36624]: pgmap v10263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:04 smithi081 ceph-mon[31453]: pgmap v10263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:06 smithi153 ceph-mon[36569]: pgmap v10264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:06 smithi104 ceph-mon[36624]: pgmap v10264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:06 smithi081 ceph-mon[31453]: pgmap v10264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:07.135+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:30:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:07.135+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:30:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:07.135+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:30:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:08 smithi153 ceph-mon[36569]: pgmap v10265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:08 smithi104 ceph-mon[36624]: pgmap v10265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:08 smithi081 ceph-mon[31453]: pgmap v10265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:10 smithi153 ceph-mon[36569]: pgmap v10266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:10 smithi104 ceph-mon[36624]: pgmap v10266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:10 smithi081 ceph-mon[31453]: pgmap v10266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:12 smithi153 ceph-mon[36569]: pgmap v10267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:12 smithi104 ceph-mon[36624]: pgmap v10267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:12 smithi081 ceph-mon[31453]: pgmap v10267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:14 smithi153 ceph-mon[36569]: pgmap v10268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:14 smithi104 ceph-mon[36624]: pgmap v10268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:14 smithi081 ceph-mon[31453]: pgmap v10268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:16 smithi104 ceph-mon[36624]: pgmap v10269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:16.595 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:16 smithi081 ceph-mon[31453]: pgmap v10269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:16 smithi153 ceph-mon[36569]: pgmap v10269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:18 smithi104 ceph-mon[36624]: pgmap v10270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:18 smithi081 ceph-mon[31453]: pgmap v10270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:18 smithi153 ceph-mon[36569]: pgmap v10270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:20 smithi104 ceph-mon[36624]: pgmap v10271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:20 smithi081 ceph-mon[31453]: pgmap v10271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:20 smithi153 ceph-mon[36569]: pgmap v10271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:22.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:22.141+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:30:22.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:22.142+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:30:22.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:22.142+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:30:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:22 smithi104 ceph-mon[36624]: pgmap v10272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:22 smithi081 ceph-mon[31453]: pgmap v10272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:22 smithi153 ceph-mon[36569]: pgmap v10272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:24 smithi104 ceph-mon[36624]: pgmap v10273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:24 smithi081 ceph-mon[31453]: pgmap v10273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:24 smithi153 ceph-mon[36569]: pgmap v10273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:26 smithi081 ceph-mon[31453]: pgmap v10274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:26 smithi153 ceph-mon[36569]: pgmap v10274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:26 smithi104 ceph-mon[36624]: pgmap v10274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:28 smithi081 ceph-mon[31453]: pgmap v10275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:30:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:28 smithi153 ceph-mon[36569]: pgmap v10275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:30:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:28 smithi104 ceph-mon[36624]: pgmap v10275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:30:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:30:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:30:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:30:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:30:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:30:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:30:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:30:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:30:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:30:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:30 smithi081 ceph-mon[31453]: pgmap v10276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:30 smithi081 ceph-mon[31453]: pgmap v10277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:30 smithi153 ceph-mon[36569]: pgmap v10276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:30 smithi153 ceph-mon[36569]: pgmap v10277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:30 smithi104 ceph-mon[36624]: pgmap v10276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:30 smithi104 ceph-mon[36624]: pgmap v10277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:32 smithi081 ceph-mon[31453]: pgmap v10278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:32 smithi153 ceph-mon[36569]: pgmap v10278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:32 smithi104 ceph-mon[36624]: pgmap v10278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:34.629 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:34 smithi153 ceph-mon[36569]: pgmap v10279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:34 smithi081 ceph-mon[31453]: pgmap v10279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:34 smithi104 ceph-mon[36624]: pgmap v10279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:36 smithi081 ceph-mon[31453]: pgmap v10280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:36 smithi153 ceph-mon[36569]: pgmap v10280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:36 smithi104 ceph-mon[36624]: pgmap v10280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:37.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:37.144+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:30:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:37.144+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:30:37.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:37.144+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:30:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:38 smithi081 ceph-mon[31453]: pgmap v10281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:38 smithi153 ceph-mon[36569]: pgmap v10281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:38 smithi104 ceph-mon[36624]: pgmap v10281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:40 smithi081 ceph-mon[31453]: pgmap v10282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:40 smithi153 ceph-mon[36569]: pgmap v10282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:40 smithi104 ceph-mon[36624]: pgmap v10282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:42 smithi081 ceph-mon[31453]: pgmap v10283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:42 smithi153 ceph-mon[36569]: pgmap v10283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:42 smithi104 ceph-mon[36624]: pgmap v10283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:44 smithi081 ceph-mon[31453]: pgmap v10284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:44 smithi153 ceph-mon[36569]: pgmap v10284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:44 smithi104 ceph-mon[36624]: pgmap v10284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:46 smithi081 ceph-mon[31453]: pgmap v10285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:46 smithi153 ceph-mon[36569]: pgmap v10285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:46 smithi104 ceph-mon[36624]: pgmap v10285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:48 smithi081 ceph-mon[31453]: pgmap v10286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:48 smithi153 ceph-mon[36569]: pgmap v10286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:48 smithi104 ceph-mon[36624]: pgmap v10286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:50 smithi081 ceph-mon[31453]: pgmap v10287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:50 smithi153 ceph-mon[36569]: pgmap v10287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:50 smithi104 ceph-mon[36624]: pgmap v10287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:52.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:52.146+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:30:52.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:52.146+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:30:52.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:30:52.146+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:30:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:52 smithi153 ceph-mon[36569]: pgmap v10288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:52 smithi104 ceph-mon[36624]: pgmap v10288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:52 smithi081 ceph-mon[31453]: pgmap v10288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:54 smithi153 ceph-mon[36569]: pgmap v10289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:54 smithi104 ceph-mon[36624]: pgmap v10289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:54 smithi081 ceph-mon[31453]: pgmap v10289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:56 smithi153 ceph-mon[36569]: pgmap v10290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:56 smithi104 ceph-mon[36624]: pgmap v10290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:56 smithi081 ceph-mon[31453]: pgmap v10290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:30:58 smithi153 ceph-mon[36569]: pgmap v10291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:30:58 smithi104 ceph-mon[36624]: pgmap v10291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:30:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:30:58 smithi081 ceph-mon[31453]: pgmap v10291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:00 smithi153 ceph-mon[36569]: pgmap v10292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:00 smithi104 ceph-mon[36624]: pgmap v10292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:00 smithi081 ceph-mon[31453]: pgmap v10292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:02 smithi153 ceph-mon[36569]: pgmap v10293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:02 smithi104 ceph-mon[36624]: pgmap v10293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:02 smithi081 ceph-mon[31453]: pgmap v10293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:04 smithi153 ceph-mon[36569]: pgmap v10294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:04 smithi104 ceph-mon[36624]: pgmap v10294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:04 smithi081 ceph-mon[31453]: pgmap v10294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:06 smithi153 ceph-mon[36569]: pgmap v10295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:06 smithi104 ceph-mon[36624]: pgmap v10295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:06 smithi081 ceph-mon[31453]: pgmap v10295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:07.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:07.162+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:31:07.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:07.162+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:31:07.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:07.162+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:31:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:07 smithi153 ceph-mon[36569]: pgmap v10296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:07 smithi104 ceph-mon[36624]: pgmap v10296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:07 smithi081 ceph-mon[31453]: pgmap v10296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:09 smithi081 ceph-mon[31453]: pgmap v10297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:09 smithi153 ceph-mon[36569]: pgmap v10297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:09 smithi104 ceph-mon[36624]: pgmap v10297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:12.141+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 msgr2=0x7f16e80b3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000596) for more than 900000000 us, fault. 2024-04-13T02:31:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:12.141+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3cf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c6a0 tx=0x7f16f4039000 comp rx=0 tx=0).stop 2024-04-13T02:31:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:12.141+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 msgr2=0x7f16e80b3cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T02:31:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:12.141+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4ea0 0x7f16e80b3cf0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T02:31:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:12.141+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T02:31:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:12.142+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T02:31:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:12.143+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b4230 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006580 tx=0x7f16f40ac000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T02:31:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:11 smithi081 ceph-mon[31453]: pgmap v10298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:11 smithi153 ceph-mon[36569]: pgmap v10298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:11 smithi104 ceph-mon[36624]: pgmap v10298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:13 smithi081 ceph-mon[31453]: pgmap v10299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:13 smithi153 ceph-mon[36569]: pgmap v10299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:13 smithi104 ceph-mon[36624]: pgmap v10299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:15.823+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:31:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:15.823+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:31:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:15.824+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:31:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:15.824+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4006650 con 0x7f1704076980 2024-04-13T02:31:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:15 smithi081 ceph-mon[31453]: pgmap v10300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:15 smithi153 ceph-mon[36569]: pgmap v10300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:15 smithi104 ceph-mon[36624]: pgmap v10300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:18.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:17 smithi104 ceph-mon[36624]: pgmap v10301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:17 smithi081 ceph-mon[31453]: pgmap v10301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:17 smithi153 ceph-mon[36569]: pgmap v10301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:19 smithi081 ceph-mon[31453]: pgmap v10302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:19 smithi153 ceph-mon[36569]: pgmap v10302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:19 smithi104 ceph-mon[36624]: pgmap v10302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:21 smithi081 ceph-mon[31453]: pgmap v10303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:22.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:22.177+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:31:22.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:22.177+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:31:22.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:22.177+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:31:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:21 smithi153 ceph-mon[36569]: pgmap v10303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:21 smithi104 ceph-mon[36624]: pgmap v10303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:23 smithi081 ceph-mon[31453]: pgmap v10304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:23 smithi153 ceph-mon[36569]: pgmap v10304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:23 smithi104 ceph-mon[36624]: pgmap v10304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:25 smithi081 ceph-mon[31453]: pgmap v10305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:25 smithi153 ceph-mon[36569]: pgmap v10305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:25 smithi104 ceph-mon[36624]: pgmap v10305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:27 smithi153 ceph-mon[36569]: pgmap v10306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:27 smithi104 ceph-mon[36624]: pgmap v10306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:27 smithi081 ceph-mon[31453]: pgmap v10306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:29 smithi153 ceph-mon[36569]: pgmap v10307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:31:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:29 smithi104 ceph-mon[36624]: pgmap v10307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:31:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:29 smithi081 ceph-mon[31453]: pgmap v10307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:31:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:31:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:31:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:30 smithi153 ceph-mon[36569]: pgmap v10308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:30 smithi153 ceph-mon[36569]: pgmap v10309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:31:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:31:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:31:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:30 smithi104 ceph-mon[36624]: pgmap v10308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:30 smithi104 ceph-mon[36624]: pgmap v10309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:31:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:31:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:31:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:30 smithi081 ceph-mon[31453]: pgmap v10308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:30 smithi081 ceph-mon[31453]: pgmap v10309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:31:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:33 smithi081 ceph-mon[31453]: pgmap v10310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:33 smithi153 ceph-mon[36569]: pgmap v10310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:33 smithi104 ceph-mon[36624]: pgmap v10310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:35 smithi081 ceph-mon[31453]: pgmap v10311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:35 smithi153 ceph-mon[36569]: pgmap v10311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:35 smithi104 ceph-mon[36624]: pgmap v10311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:37.193+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:31:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:37.193+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:31:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:37.193+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:31:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:37 smithi081 ceph-mon[31453]: pgmap v10312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:37 smithi153 ceph-mon[36569]: pgmap v10312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:37 smithi104 ceph-mon[36624]: pgmap v10312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:39 smithi081 ceph-mon[31453]: pgmap v10313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:39 smithi153 ceph-mon[36569]: pgmap v10313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:39 smithi104 ceph-mon[36624]: pgmap v10313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:41 smithi081 ceph-mon[31453]: pgmap v10314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:41 smithi153 ceph-mon[36569]: pgmap v10314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:41 smithi104 ceph-mon[36624]: pgmap v10314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:43 smithi081 ceph-mon[31453]: pgmap v10315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:43 smithi153 ceph-mon[36569]: pgmap v10315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:43 smithi104 ceph-mon[36624]: pgmap v10315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:45 smithi081 ceph-mon[31453]: pgmap v10316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:45 smithi153 ceph-mon[36569]: pgmap v10316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:45 smithi104 ceph-mon[36624]: pgmap v10316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:47 smithi153 ceph-mon[36569]: pgmap v10317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:47 smithi104 ceph-mon[36624]: pgmap v10317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:47 smithi081 ceph-mon[31453]: pgmap v10317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:49 smithi153 ceph-mon[36569]: pgmap v10318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:49 smithi104 ceph-mon[36624]: pgmap v10318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:49 smithi081 ceph-mon[31453]: pgmap v10318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:51 smithi153 ceph-mon[36569]: pgmap v10319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:51 smithi104 ceph-mon[36624]: pgmap v10319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:51 smithi081 ceph-mon[31453]: pgmap v10319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:52.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:52.208+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:31:52.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:52.208+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:31:52.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:31:52.208+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:31:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:53 smithi081 ceph-mon[31453]: pgmap v10320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:53 smithi153 ceph-mon[36569]: pgmap v10320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:53 smithi104 ceph-mon[36624]: pgmap v10320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:55 smithi153 ceph-mon[36569]: pgmap v10321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:55 smithi104 ceph-mon[36624]: pgmap v10321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:55 smithi081 ceph-mon[31453]: pgmap v10321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:57 smithi153 ceph-mon[36569]: pgmap v10322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:57 smithi104 ceph-mon[36624]: pgmap v10322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:57 smithi081 ceph-mon[31453]: pgmap v10322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:31:59 smithi153 ceph-mon[36569]: pgmap v10323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:31:59 smithi104 ceph-mon[36624]: pgmap v10323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:31:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:31:59 smithi081 ceph-mon[31453]: pgmap v10323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:01 smithi153 ceph-mon[36569]: pgmap v10324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:01 smithi104 ceph-mon[36624]: pgmap v10324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:01 smithi081 ceph-mon[31453]: pgmap v10324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:03 smithi153 ceph-mon[36569]: pgmap v10325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:03 smithi104 ceph-mon[36624]: pgmap v10325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:03 smithi081 ceph-mon[31453]: pgmap v10325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:05 smithi153 ceph-mon[36569]: pgmap v10326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:05 smithi104 ceph-mon[36624]: pgmap v10326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:05 smithi081 ceph-mon[31453]: pgmap v10326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:07.221+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:32:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:07.222+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:32:07.223 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:07.222+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:32:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:07 smithi153 ceph-mon[36569]: pgmap v10327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:07 smithi104 ceph-mon[36624]: pgmap v10327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:07 smithi081 ceph-mon[31453]: pgmap v10327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:09 smithi153 ceph-mon[36569]: pgmap v10328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:09 smithi104 ceph-mon[36624]: pgmap v10328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:09 smithi081 ceph-mon[31453]: pgmap v10328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:11 smithi153 ceph-mon[36569]: pgmap v10329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:11 smithi104 ceph-mon[36624]: pgmap v10329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:11 smithi081 ceph-mon[31453]: pgmap v10329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:12 smithi153 ceph-mon[36569]: pgmap v10330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:12 smithi104 ceph-mon[36624]: pgmap v10330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:12 smithi081 ceph-mon[31453]: pgmap v10330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:15 smithi081 ceph-mon[31453]: pgmap v10331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:15 smithi153 ceph-mon[36569]: pgmap v10331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:15 smithi104 ceph-mon[36624]: pgmap v10331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:17 smithi081 ceph-mon[31453]: pgmap v10332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:17 smithi153 ceph-mon[36569]: pgmap v10332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:17 smithi104 ceph-mon[36624]: pgmap v10332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:19 smithi153 ceph-mon[36569]: pgmap v10333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:19 smithi104 ceph-mon[36624]: pgmap v10333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:19 smithi081 ceph-mon[31453]: pgmap v10333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:21 smithi153 ceph-mon[36569]: pgmap v10334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:21 smithi104 ceph-mon[36624]: pgmap v10334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:21 smithi081 ceph-mon[31453]: pgmap v10334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:22.237+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:32:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:22.237+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:32:22.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:22.237+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:32:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:23 smithi153 ceph-mon[36569]: pgmap v10335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:23 smithi104 ceph-mon[36624]: pgmap v10335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:23 smithi081 ceph-mon[31453]: pgmap v10335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:25 smithi104 ceph-mon[36624]: pgmap v10336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:25 smithi081 ceph-mon[31453]: pgmap v10336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:25 smithi153 ceph-mon[36569]: pgmap v10336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:27 smithi081 ceph-mon[31453]: pgmap v10337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:27 smithi153 ceph-mon[36569]: pgmap v10337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:27 smithi104 ceph-mon[36624]: pgmap v10337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:29 smithi153 ceph-mon[36569]: pgmap v10338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:29 smithi104 ceph-mon[36624]: pgmap v10338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:29 smithi081 ceph-mon[31453]: pgmap v10338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:31 smithi153 ceph-mon[36569]: pgmap v10339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:32:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:31 smithi104 ceph-mon[36624]: pgmap v10339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:32:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:31 smithi081 ceph-mon[31453]: pgmap v10339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:32:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:32:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:32:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:32:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:32:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:32:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:32:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:32:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:32:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:32:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:33 smithi153 ceph-mon[36569]: pgmap v10340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:33 smithi153 ceph-mon[36569]: pgmap v10341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:33 smithi104 ceph-mon[36624]: pgmap v10340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:33 smithi104 ceph-mon[36624]: pgmap v10341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:33 smithi081 ceph-mon[31453]: pgmap v10340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:33 smithi081 ceph-mon[31453]: pgmap v10341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:35 smithi153 ceph-mon[36569]: pgmap v10342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:35 smithi104 ceph-mon[36624]: pgmap v10342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:35 smithi081 ceph-mon[31453]: pgmap v10342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:37.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:37.252+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:32:37.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:37.252+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:32:37.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:37.252+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:32:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:37 smithi153 ceph-mon[36569]: pgmap v10343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:37 smithi104 ceph-mon[36624]: pgmap v10343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:37 smithi081 ceph-mon[31453]: pgmap v10343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:39 smithi153 ceph-mon[36569]: pgmap v10344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:39 smithi104 ceph-mon[36624]: pgmap v10344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:39 smithi081 ceph-mon[31453]: pgmap v10344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:41 smithi153 ceph-mon[36569]: pgmap v10345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:41 smithi104 ceph-mon[36624]: pgmap v10345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:41 smithi081 ceph-mon[31453]: pgmap v10345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:43 smithi153 ceph-mon[36569]: pgmap v10346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:43 smithi104 ceph-mon[36624]: pgmap v10346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:43 smithi081 ceph-mon[31453]: pgmap v10346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:45.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:45 smithi081 ceph-mon[31453]: pgmap v10347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:45 smithi153 ceph-mon[36569]: pgmap v10347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:45 smithi104 ceph-mon[36624]: pgmap v10347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:47 smithi153 ceph-mon[36569]: pgmap v10348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:47 smithi104 ceph-mon[36624]: pgmap v10348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:47 smithi081 ceph-mon[31453]: pgmap v10348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:49 smithi153 ceph-mon[36569]: pgmap v10349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:49 smithi104 ceph-mon[36624]: pgmap v10349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:49 smithi081 ceph-mon[31453]: pgmap v10349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:51 smithi153 ceph-mon[36569]: pgmap v10350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:51 smithi104 ceph-mon[36624]: pgmap v10350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:51 smithi081 ceph-mon[31453]: pgmap v10350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:52.268+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:32:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:52.269+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:32:52.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:32:52.269+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:32:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:53 smithi153 ceph-mon[36569]: pgmap v10351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:53 smithi104 ceph-mon[36624]: pgmap v10351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:53 smithi081 ceph-mon[31453]: pgmap v10351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:55 smithi153 ceph-mon[36569]: pgmap v10352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:55 smithi104 ceph-mon[36624]: pgmap v10352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:55 smithi081 ceph-mon[31453]: pgmap v10352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:56 smithi153 ceph-mon[36569]: pgmap v10353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:56 smithi104 ceph-mon[36624]: pgmap v10353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:56 smithi081 ceph-mon[31453]: pgmap v10353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:32:59 smithi153 ceph-mon[36569]: pgmap v10354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:32:59 smithi104 ceph-mon[36624]: pgmap v10354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:32:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:32:59 smithi081 ceph-mon[31453]: pgmap v10354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:01 smithi153 ceph-mon[36569]: pgmap v10355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:01 smithi104 ceph-mon[36624]: pgmap v10355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:01 smithi081 ceph-mon[31453]: pgmap v10355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:03 smithi153 ceph-mon[36569]: pgmap v10356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:03 smithi104 ceph-mon[36624]: pgmap v10356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:03 smithi081 ceph-mon[31453]: pgmap v10356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:05 smithi153 ceph-mon[36569]: pgmap v10357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:05 smithi104 ceph-mon[36624]: pgmap v10357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:05 smithi081 ceph-mon[31453]: pgmap v10357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:07.284+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:33:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:07.284+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:33:07.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:07.284+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:33:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:07 smithi153 ceph-mon[36569]: pgmap v10358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:07 smithi104 ceph-mon[36624]: pgmap v10358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:07 smithi081 ceph-mon[31453]: pgmap v10358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:09 smithi153 ceph-mon[36569]: pgmap v10359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:09 smithi104 ceph-mon[36624]: pgmap v10359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:09 smithi081 ceph-mon[31453]: pgmap v10359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:11 smithi153 ceph-mon[36569]: pgmap v10360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:11 smithi104 ceph-mon[36624]: pgmap v10360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:11 smithi081 ceph-mon[31453]: pgmap v10360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:13 smithi153 ceph-mon[36569]: pgmap v10361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:13 smithi104 ceph-mon[36624]: pgmap v10361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:13 smithi081 ceph-mon[31453]: pgmap v10361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:15 smithi153 ceph-mon[36569]: pgmap v10362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:15 smithi104 ceph-mon[36624]: pgmap v10362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:15 smithi081 ceph-mon[31453]: pgmap v10362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:17 smithi104 ceph-mon[36624]: pgmap v10363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:17 smithi081 ceph-mon[31453]: pgmap v10363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:17 smithi153 ceph-mon[36569]: pgmap v10363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:19 smithi104 ceph-mon[36624]: pgmap v10364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:19 smithi081 ceph-mon[31453]: pgmap v10364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:19 smithi153 ceph-mon[36569]: pgmap v10364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:21 smithi104 ceph-mon[36624]: pgmap v10365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:21 smithi081 ceph-mon[31453]: pgmap v10365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:21 smithi153 ceph-mon[36569]: pgmap v10365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:22.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:22.299+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:33:22.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:22.299+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:33:22.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:22.299+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:33:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:23 smithi104 ceph-mon[36624]: pgmap v10366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:23 smithi081 ceph-mon[31453]: pgmap v10366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:23 smithi153 ceph-mon[36569]: pgmap v10366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:25 smithi104 ceph-mon[36624]: pgmap v10367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:25 smithi081 ceph-mon[31453]: pgmap v10367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:25 smithi153 ceph-mon[36569]: pgmap v10367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:27.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:27 smithi081 ceph-mon[31453]: pgmap v10368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:27 smithi153 ceph-mon[36569]: pgmap v10368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:27 smithi104 ceph-mon[36624]: pgmap v10368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:29 smithi081 ceph-mon[31453]: pgmap v10369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:29 smithi153 ceph-mon[36569]: pgmap v10369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:29 smithi104 ceph-mon[36624]: pgmap v10369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:31 smithi081 ceph-mon[31453]: pgmap v10370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:31 smithi153 ceph-mon[36569]: pgmap v10370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:31 smithi104 ceph-mon[36624]: pgmap v10370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:33 smithi081 ceph-mon[31453]: pgmap v10371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:33:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:33 smithi153 ceph-mon[36569]: pgmap v10371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:33:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:33 smithi104 ceph-mon[36624]: pgmap v10371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:33:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:33:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:33:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:34 smithi081 ceph-mon[31453]: pgmap v10372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:34 smithi081 ceph-mon[31453]: pgmap v10373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:33:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:33:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:33:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:34 smithi153 ceph-mon[36569]: pgmap v10372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:34 smithi153 ceph-mon[36569]: pgmap v10373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:33:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:33:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:33:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:34 smithi104 ceph-mon[36624]: pgmap v10372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:34 smithi104 ceph-mon[36624]: pgmap v10373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:33:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:35 smithi153 ceph-mon[36569]: pgmap v10374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:35 smithi104 ceph-mon[36624]: pgmap v10374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:35 smithi081 ceph-mon[31453]: pgmap v10374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:37.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:37.309+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:33:37.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:37.309+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:33:37.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:37.309+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:33:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:37 smithi153 ceph-mon[36569]: pgmap v10375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:37 smithi104 ceph-mon[36624]: pgmap v10375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:37 smithi081 ceph-mon[31453]: pgmap v10375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:39 smithi153 ceph-mon[36569]: pgmap v10376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:39 smithi104 ceph-mon[36624]: pgmap v10376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:39 smithi081 ceph-mon[31453]: pgmap v10376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:41 smithi153 ceph-mon[36569]: pgmap v10377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:41 smithi104 ceph-mon[36624]: pgmap v10377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:41 smithi081 ceph-mon[31453]: pgmap v10377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:43 smithi153 ceph-mon[36569]: pgmap v10378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:43 smithi104 ceph-mon[36624]: pgmap v10378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:43 smithi081 ceph-mon[31453]: pgmap v10378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:45 smithi153 ceph-mon[36569]: pgmap v10379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:45 smithi104 ceph-mon[36624]: pgmap v10379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:45 smithi081 ceph-mon[31453]: pgmap v10379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:47 smithi153 ceph-mon[36569]: pgmap v10380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:47 smithi104 ceph-mon[36624]: pgmap v10380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:47 smithi081 ceph-mon[31453]: pgmap v10380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:49 smithi153 ceph-mon[36569]: pgmap v10381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:49 smithi104 ceph-mon[36624]: pgmap v10381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:49 smithi081 ceph-mon[31453]: pgmap v10381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:51 smithi153 ceph-mon[36569]: pgmap v10382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:51 smithi104 ceph-mon[36624]: pgmap v10382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:51 smithi081 ceph-mon[31453]: pgmap v10382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:52.323+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:33:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:52.323+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:33:52.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:33:52.323+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:33:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:53 smithi153 ceph-mon[36569]: pgmap v10383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:53 smithi104 ceph-mon[36624]: pgmap v10383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:53 smithi081 ceph-mon[31453]: pgmap v10383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:55 smithi153 ceph-mon[36569]: pgmap v10384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:55 smithi104 ceph-mon[36624]: pgmap v10384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:55 smithi081 ceph-mon[31453]: pgmap v10384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:57 smithi153 ceph-mon[36569]: pgmap v10385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:57 smithi104 ceph-mon[36624]: pgmap v10385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:57 smithi081 ceph-mon[31453]: pgmap v10385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:33:59 smithi153 ceph-mon[36569]: pgmap v10386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:33:59 smithi104 ceph-mon[36624]: pgmap v10386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:33:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:33:59 smithi081 ceph-mon[31453]: pgmap v10386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:01 smithi153 ceph-mon[36569]: pgmap v10387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:01 smithi104 ceph-mon[36624]: pgmap v10387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:01 smithi081 ceph-mon[31453]: pgmap v10387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:03 smithi153 ceph-mon[36569]: pgmap v10388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:03 smithi104 ceph-mon[36624]: pgmap v10388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:03 smithi081 ceph-mon[31453]: pgmap v10388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:05 smithi153 ceph-mon[36569]: pgmap v10389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:05 smithi104 ceph-mon[36624]: pgmap v10389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:05 smithi081 ceph-mon[31453]: pgmap v10389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:07.330+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:34:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:07.330+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:34:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:07.330+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:34:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:07 smithi153 ceph-mon[36569]: pgmap v10390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:07 smithi104 ceph-mon[36624]: pgmap v10390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:07 smithi081 ceph-mon[31453]: pgmap v10390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:09 smithi153 ceph-mon[36569]: pgmap v10391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:09 smithi104 ceph-mon[36624]: pgmap v10391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:09 smithi081 ceph-mon[31453]: pgmap v10391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:11 smithi153 ceph-mon[36569]: pgmap v10392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:11 smithi104 ceph-mon[36624]: pgmap v10392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:11 smithi081 ceph-mon[31453]: pgmap v10392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:13 smithi153 ceph-mon[36569]: pgmap v10393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:13 smithi104 ceph-mon[36624]: pgmap v10393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:13 smithi081 ceph-mon[31453]: pgmap v10393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:15 smithi153 ceph-mon[36569]: pgmap v10394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:15 smithi104 ceph-mon[36624]: pgmap v10394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:15.851+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:34:15.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:15.851+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:34:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:15.851+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:34:15.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:15.852+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:34:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:15 smithi081 ceph-mon[31453]: pgmap v10394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:17 smithi153 ceph-mon[36569]: pgmap v10395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:17 smithi104 ceph-mon[36624]: pgmap v10395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:17 smithi081 ceph-mon[31453]: pgmap v10395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:19 smithi153 ceph-mon[36569]: pgmap v10396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:19 smithi104 ceph-mon[36624]: pgmap v10396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:19 smithi081 ceph-mon[31453]: pgmap v10396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:21 smithi153 ceph-mon[36569]: pgmap v10397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:21 smithi104 ceph-mon[36624]: pgmap v10397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:21 smithi081 ceph-mon[31453]: pgmap v10397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:22.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:22.345+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:34:22.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:22.345+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:34:22.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:22.345+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:34:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:23 smithi153 ceph-mon[36569]: pgmap v10398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:23 smithi104 ceph-mon[36624]: pgmap v10398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:23 smithi081 ceph-mon[31453]: pgmap v10398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:25 smithi153 ceph-mon[36569]: pgmap v10399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:25 smithi104 ceph-mon[36624]: pgmap v10399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:25 smithi081 ceph-mon[31453]: pgmap v10399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:27 smithi153 ceph-mon[36569]: pgmap v10400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:27 smithi104 ceph-mon[36624]: pgmap v10400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:27 smithi081 ceph-mon[31453]: pgmap v10400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:30 smithi081 ceph-mon[31453]: pgmap v10401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:30 smithi153 ceph-mon[36569]: pgmap v10401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:30 smithi104 ceph-mon[36624]: pgmap v10401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:31 smithi104 ceph-mon[36624]: pgmap v10402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:31 smithi081 ceph-mon[31453]: pgmap v10402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:31 smithi153 ceph-mon[36569]: pgmap v10402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:33 smithi104 ceph-mon[36624]: pgmap v10403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:33 smithi081 ceph-mon[31453]: pgmap v10403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:33 smithi153 ceph-mon[36569]: pgmap v10403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:34:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:34:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:34:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:34:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:34:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:34:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:34:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:34:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:34:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:34:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:34:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: pgmap v10404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: pgmap v10405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: pgmap v10404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: pgmap v10405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:34:35.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:34:35.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:34:35.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:34:35.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:34:35.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:34:35.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: pgmap v10404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:35.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: pgmap v10405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:35.959 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:34:37.361 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:37.361+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:34:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:37.361+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:34:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:37.361+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:34:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:38 smithi081 ceph-mon[31453]: pgmap v10406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:38 smithi153 ceph-mon[36569]: pgmap v10406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:38 smithi104 ceph-mon[36624]: pgmap v10406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:40 smithi081 ceph-mon[31453]: pgmap v10407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:40 smithi153 ceph-mon[36569]: pgmap v10407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:40 smithi104 ceph-mon[36624]: pgmap v10407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:42 smithi081 ceph-mon[31453]: pgmap v10408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:42 smithi153 ceph-mon[36569]: pgmap v10408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:42 smithi104 ceph-mon[36624]: pgmap v10408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:44 smithi081 ceph-mon[31453]: pgmap v10409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:44 smithi153 ceph-mon[36569]: pgmap v10409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:44 smithi104 ceph-mon[36624]: pgmap v10409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:46 smithi081 ceph-mon[31453]: pgmap v10410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:46 smithi153 ceph-mon[36569]: pgmap v10410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:46 smithi104 ceph-mon[36624]: pgmap v10410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:48 smithi081 ceph-mon[31453]: pgmap v10411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:48 smithi153 ceph-mon[36569]: pgmap v10411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:48 smithi104 ceph-mon[36624]: pgmap v10411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:50.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:50 smithi104 ceph-mon[36624]: pgmap v10412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:50 smithi081 ceph-mon[31453]: pgmap v10412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:50 smithi153 ceph-mon[36569]: pgmap v10412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:52.362+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:34:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:52.362+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:34:52.363 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:34:52.362+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:34:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:52 smithi081 ceph-mon[31453]: pgmap v10413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:52 smithi153 ceph-mon[36569]: pgmap v10413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:52 smithi104 ceph-mon[36624]: pgmap v10413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:54 smithi081 ceph-mon[31453]: pgmap v10414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:54 smithi153 ceph-mon[36569]: pgmap v10414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:54 smithi104 ceph-mon[36624]: pgmap v10414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:56 smithi081 ceph-mon[31453]: pgmap v10415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:56 smithi153 ceph-mon[36569]: pgmap v10415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:56 smithi104 ceph-mon[36624]: pgmap v10415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:34:58 smithi081 ceph-mon[31453]: pgmap v10416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:34:58 smithi153 ceph-mon[36569]: pgmap v10416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:34:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:34:58 smithi104 ceph-mon[36624]: pgmap v10416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:00 smithi081 ceph-mon[31453]: pgmap v10417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:00 smithi153 ceph-mon[36569]: pgmap v10417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:00 smithi104 ceph-mon[36624]: pgmap v10417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:02 smithi153 ceph-mon[36569]: pgmap v10418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:02 smithi104 ceph-mon[36624]: pgmap v10418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:02 smithi081 ceph-mon[31453]: pgmap v10418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:04 smithi153 ceph-mon[36569]: pgmap v10419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:04 smithi104 ceph-mon[36624]: pgmap v10419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:04 smithi081 ceph-mon[31453]: pgmap v10419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:06 smithi153 ceph-mon[36569]: pgmap v10420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:06 smithi104 ceph-mon[36624]: pgmap v10420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:06 smithi081 ceph-mon[31453]: pgmap v10420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:07.377+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:35:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:07.377+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:35:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:07.377+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:35:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:08 smithi153 ceph-mon[36569]: pgmap v10421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:08 smithi104 ceph-mon[36624]: pgmap v10421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:08 smithi081 ceph-mon[31453]: pgmap v10421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:10 smithi153 ceph-mon[36569]: pgmap v10422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:10 smithi104 ceph-mon[36624]: pgmap v10422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:10 smithi081 ceph-mon[31453]: pgmap v10422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:12 smithi153 ceph-mon[36569]: pgmap v10423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:12 smithi104 ceph-mon[36624]: pgmap v10423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:12 smithi081 ceph-mon[31453]: pgmap v10423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:14 smithi153 ceph-mon[36569]: pgmap v10424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:14 smithi104 ceph-mon[36624]: pgmap v10424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:14 smithi081 ceph-mon[31453]: pgmap v10424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:16 smithi081 ceph-mon[31453]: pgmap v10425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:16 smithi153 ceph-mon[36569]: pgmap v10425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:16 smithi104 ceph-mon[36624]: pgmap v10425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:18 smithi153 ceph-mon[36569]: pgmap v10426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:18 smithi104 ceph-mon[36624]: pgmap v10426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:18 smithi081 ceph-mon[31453]: pgmap v10426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:20 smithi153 ceph-mon[36569]: pgmap v10427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:20 smithi104 ceph-mon[36624]: pgmap v10427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:20 smithi081 ceph-mon[31453]: pgmap v10427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:22.393+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:35:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:22.393+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:35:22.394 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:22.393+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:35:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:22 smithi153 ceph-mon[36569]: pgmap v10428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:22 smithi104 ceph-mon[36624]: pgmap v10428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:22 smithi081 ceph-mon[31453]: pgmap v10428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:24 smithi153 ceph-mon[36569]: pgmap v10429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:24 smithi104 ceph-mon[36624]: pgmap v10429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:24 smithi081 ceph-mon[31453]: pgmap v10429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:26 smithi153 ceph-mon[36569]: pgmap v10430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:26 smithi104 ceph-mon[36624]: pgmap v10430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:26 smithi081 ceph-mon[31453]: pgmap v10430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:28 smithi153 ceph-mon[36569]: pgmap v10431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:28 smithi104 ceph-mon[36624]: pgmap v10431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:28 smithi081 ceph-mon[31453]: pgmap v10431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:30 smithi153 ceph-mon[36569]: pgmap v10432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:30 smithi104 ceph-mon[36624]: pgmap v10432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:30 smithi081 ceph-mon[31453]: pgmap v10432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:32 smithi153 ceph-mon[36569]: pgmap v10433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:32 smithi104 ceph-mon[36624]: pgmap v10433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:32 smithi081 ceph-mon[31453]: pgmap v10433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:34 smithi153 ceph-mon[36569]: pgmap v10434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:34 smithi104 ceph-mon[36624]: pgmap v10434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:34 smithi081 ceph-mon[31453]: pgmap v10434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:36 smithi153 ceph-mon[36569]: pgmap v10435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:35:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:36 smithi104 ceph-mon[36624]: pgmap v10435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:35:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:36 smithi081 ceph-mon[31453]: pgmap v10435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:35:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:37.405+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:35:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:37.405+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:35:37.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:37.405+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:35:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:35:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:35:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:35:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:35:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:35:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:35:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:35:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:35:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:35:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:38 smithi153 ceph-mon[36569]: pgmap v10436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:38 smithi153 ceph-mon[36569]: pgmap v10437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:38 smithi104 ceph-mon[36624]: pgmap v10436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:38 smithi104 ceph-mon[36624]: pgmap v10437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:38 smithi081 ceph-mon[31453]: pgmap v10436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:38.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:38 smithi081 ceph-mon[31453]: pgmap v10437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:40 smithi153 ceph-mon[36569]: pgmap v10438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:40 smithi104 ceph-mon[36624]: pgmap v10438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:40 smithi081 ceph-mon[31453]: pgmap v10438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:42 smithi153 ceph-mon[36569]: pgmap v10439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:42 smithi104 ceph-mon[36624]: pgmap v10439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:42 smithi081 ceph-mon[31453]: pgmap v10439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:44 smithi153 ceph-mon[36569]: pgmap v10440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:44 smithi104 ceph-mon[36624]: pgmap v10440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:44 smithi081 ceph-mon[31453]: pgmap v10440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:46 smithi153 ceph-mon[36569]: pgmap v10441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:46 smithi104 ceph-mon[36624]: pgmap v10441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:46 smithi081 ceph-mon[31453]: pgmap v10441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:48 smithi153 ceph-mon[36569]: pgmap v10442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:48 smithi104 ceph-mon[36624]: pgmap v10442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:48 smithi081 ceph-mon[31453]: pgmap v10442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:50 smithi153 ceph-mon[36569]: pgmap v10443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:50 smithi104 ceph-mon[36624]: pgmap v10443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:50 smithi081 ceph-mon[31453]: pgmap v10443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:52.421+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:35:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:52.421+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:35:52.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:35:52.421+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:35:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:52 smithi104 ceph-mon[36624]: pgmap v10444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:52 smithi081 ceph-mon[31453]: pgmap v10444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:52 smithi153 ceph-mon[36569]: pgmap v10444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:54 smithi104 ceph-mon[36624]: pgmap v10445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:54 smithi081 ceph-mon[31453]: pgmap v10445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:54 smithi153 ceph-mon[36569]: pgmap v10445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:56 smithi104 ceph-mon[36624]: pgmap v10446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:56 smithi081 ceph-mon[31453]: pgmap v10446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:56 smithi153 ceph-mon[36569]: pgmap v10446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:35:58 smithi104 ceph-mon[36624]: pgmap v10447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:35:58 smithi081 ceph-mon[31453]: pgmap v10447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:35:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:35:58 smithi153 ceph-mon[36569]: pgmap v10447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:00 smithi104 ceph-mon[36624]: pgmap v10448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:00 smithi081 ceph-mon[31453]: pgmap v10448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:00 smithi153 ceph-mon[36569]: pgmap v10448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:02 smithi081 ceph-mon[31453]: pgmap v10449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:02 smithi153 ceph-mon[36569]: pgmap v10449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:02 smithi104 ceph-mon[36624]: pgmap v10449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:04 smithi081 ceph-mon[31453]: pgmap v10450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:04 smithi153 ceph-mon[36569]: pgmap v10450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:04 smithi104 ceph-mon[36624]: pgmap v10450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:06.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:06 smithi153 ceph-mon[36569]: pgmap v10451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:06 smithi081 ceph-mon[31453]: pgmap v10451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:06 smithi104 ceph-mon[36624]: pgmap v10451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:07.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:07.436+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:36:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:07.436+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:36:07.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:07.436+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:36:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:08 smithi081 ceph-mon[31453]: pgmap v10452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:08 smithi153 ceph-mon[36569]: pgmap v10452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:08 smithi104 ceph-mon[36624]: pgmap v10452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:10 smithi081 ceph-mon[31453]: pgmap v10453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:10 smithi153 ceph-mon[36569]: pgmap v10453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:10 smithi104 ceph-mon[36624]: pgmap v10453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:12 smithi081 ceph-mon[31453]: pgmap v10454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:12 smithi153 ceph-mon[36569]: pgmap v10454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:12 smithi104 ceph-mon[36624]: pgmap v10454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:14 smithi081 ceph-mon[31453]: pgmap v10455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:14 smithi153 ceph-mon[36569]: pgmap v10455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:14 smithi104 ceph-mon[36624]: pgmap v10455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:16 smithi081 ceph-mon[31453]: pgmap v10456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:16 smithi153 ceph-mon[36569]: pgmap v10456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:16 smithi104 ceph-mon[36624]: pgmap v10456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:18 smithi081 ceph-mon[31453]: pgmap v10457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:18 smithi153 ceph-mon[36569]: pgmap v10457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:18 smithi104 ceph-mon[36624]: pgmap v10457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:20.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:20 smithi104 ceph-mon[36624]: pgmap v10458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:20 smithi081 ceph-mon[31453]: pgmap v10458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:20 smithi153 ceph-mon[36569]: pgmap v10458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:22.440+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:36:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:22.440+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:36:22.441 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:22.440+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:36:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:22 smithi081 ceph-mon[31453]: pgmap v10459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:22 smithi153 ceph-mon[36569]: pgmap v10459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:22 smithi104 ceph-mon[36624]: pgmap v10459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:24 smithi081 ceph-mon[31453]: pgmap v10460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:24 smithi153 ceph-mon[36569]: pgmap v10460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:24 smithi104 ceph-mon[36624]: pgmap v10460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:26 smithi153 ceph-mon[36569]: pgmap v10461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:26 smithi104 ceph-mon[36624]: pgmap v10461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:26 smithi081 ceph-mon[31453]: pgmap v10461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:28 smithi153 ceph-mon[36569]: pgmap v10462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:28 smithi104 ceph-mon[36624]: pgmap v10462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:28 smithi081 ceph-mon[31453]: pgmap v10462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:30.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:30 smithi153 ceph-mon[36569]: pgmap v10463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:30 smithi104 ceph-mon[36624]: pgmap v10463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:30 smithi081 ceph-mon[31453]: pgmap v10463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:32 smithi153 ceph-mon[36569]: pgmap v10464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:32 smithi104 ceph-mon[36624]: pgmap v10464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:32 smithi081 ceph-mon[31453]: pgmap v10464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:34 smithi153 ceph-mon[36569]: pgmap v10465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:34 smithi104 ceph-mon[36624]: pgmap v10465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:34 smithi081 ceph-mon[31453]: pgmap v10465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:36 smithi153 ceph-mon[36569]: pgmap v10466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:36 smithi104 ceph-mon[36624]: pgmap v10466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:36 smithi081 ceph-mon[31453]: pgmap v10466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:37.455+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:36:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:37.455+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:36:37.456 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:37.455+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:36:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:36:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:36:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:36:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:38 smithi153 ceph-mon[36569]: pgmap v10467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:36:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:36:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:36:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:38 smithi104 ceph-mon[36624]: pgmap v10467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:36:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:36:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:36:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:38 smithi081 ceph-mon[31453]: pgmap v10467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:36:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:36:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:36:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:39 smithi153 ceph-mon[36569]: pgmap v10468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:39 smithi153 ceph-mon[36569]: pgmap v10469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:39 smithi104 ceph-mon[36624]: pgmap v10468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:39 smithi104 ceph-mon[36624]: pgmap v10469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:39 smithi081 ceph-mon[31453]: pgmap v10468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:39 smithi081 ceph-mon[31453]: pgmap v10469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:41 smithi153 ceph-mon[36569]: pgmap v10470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:41 smithi104 ceph-mon[36624]: pgmap v10470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:41 smithi081 ceph-mon[31453]: pgmap v10470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:43 smithi153 ceph-mon[36569]: pgmap v10471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:43 smithi104 ceph-mon[36624]: pgmap v10471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:43 smithi081 ceph-mon[31453]: pgmap v10471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:45 smithi153 ceph-mon[36569]: pgmap v10472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:45 smithi104 ceph-mon[36624]: pgmap v10472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:45 smithi081 ceph-mon[31453]: pgmap v10472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:47.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:47 smithi153 ceph-mon[36569]: pgmap v10473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:47 smithi104 ceph-mon[36624]: pgmap v10473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:47 smithi081 ceph-mon[31453]: pgmap v10473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:49 smithi153 ceph-mon[36569]: pgmap v10474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:49 smithi104 ceph-mon[36624]: pgmap v10474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:49 smithi081 ceph-mon[31453]: pgmap v10474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:51 smithi153 ceph-mon[36569]: pgmap v10475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:51 smithi104 ceph-mon[36624]: pgmap v10475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:51 smithi081 ceph-mon[31453]: pgmap v10475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:52.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:52.466+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:36:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:52.466+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:36:52.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:36:52.466+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:36:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:53 smithi153 ceph-mon[36569]: pgmap v10476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:53 smithi104 ceph-mon[36624]: pgmap v10476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:53 smithi081 ceph-mon[31453]: pgmap v10476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:55 smithi153 ceph-mon[36569]: pgmap v10477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:55 smithi104 ceph-mon[36624]: pgmap v10477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:55 smithi081 ceph-mon[31453]: pgmap v10477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:57 smithi153 ceph-mon[36569]: pgmap v10478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:57 smithi104 ceph-mon[36624]: pgmap v10478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:57 smithi081 ceph-mon[31453]: pgmap v10478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:36:59 smithi153 ceph-mon[36569]: pgmap v10479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:36:59 smithi104 ceph-mon[36624]: pgmap v10479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:36:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:36:59 smithi081 ceph-mon[31453]: pgmap v10479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:01 smithi153 ceph-mon[36569]: pgmap v10480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:01 smithi104 ceph-mon[36624]: pgmap v10480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:01 smithi081 ceph-mon[31453]: pgmap v10480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:03 smithi153 ceph-mon[36569]: pgmap v10481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:03 smithi104 ceph-mon[36624]: pgmap v10481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:03 smithi081 ceph-mon[31453]: pgmap v10481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:05 smithi153 ceph-mon[36569]: pgmap v10482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:05 smithi104 ceph-mon[36624]: pgmap v10482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:05 smithi081 ceph-mon[31453]: pgmap v10482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:07.470 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:07.469+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:37:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:07.469+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:37:07.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:07.469+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:37:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:07 smithi153 ceph-mon[36569]: pgmap v10483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:07 smithi104 ceph-mon[36624]: pgmap v10483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:07 smithi081 ceph-mon[31453]: pgmap v10483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:08.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:08 smithi081 ceph-mon[31453]: pgmap v10484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:08 smithi153 ceph-mon[36569]: pgmap v10484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:08 smithi104 ceph-mon[36624]: pgmap v10484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:11 smithi153 ceph-mon[36569]: pgmap v10485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:11 smithi104 ceph-mon[36624]: pgmap v10485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:11 smithi081 ceph-mon[31453]: pgmap v10485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:13 smithi153 ceph-mon[36569]: pgmap v10486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:13 smithi104 ceph-mon[36624]: pgmap v10486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:13 smithi081 ceph-mon[31453]: pgmap v10486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:15 smithi153 ceph-mon[36569]: pgmap v10487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:15 smithi104 ceph-mon[36624]: pgmap v10487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:15 smithi081 ceph-mon[31453]: pgmap v10487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:15.881+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:37:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:15.881+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:37:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:15.882+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:37:15.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:15.882+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 150 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:37:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:17 smithi153 ceph-mon[36569]: pgmap v10488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:17 smithi104 ceph-mon[36624]: pgmap v10488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:17 smithi081 ceph-mon[31453]: pgmap v10488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:19 smithi153 ceph-mon[36569]: pgmap v10489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:19 smithi104 ceph-mon[36624]: pgmap v10489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:19 smithi081 ceph-mon[31453]: pgmap v10489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:21 smithi153 ceph-mon[36569]: pgmap v10490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:21 smithi104 ceph-mon[36624]: pgmap v10490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:21 smithi081 ceph-mon[31453]: pgmap v10490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:22.485+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:37:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:22.485+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:37:22.486 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:22.485+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:37:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:23 smithi153 ceph-mon[36569]: pgmap v10491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:23 smithi104 ceph-mon[36624]: pgmap v10491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:23 smithi081 ceph-mon[31453]: pgmap v10491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:25 smithi153 ceph-mon[36569]: pgmap v10492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:25 smithi104 ceph-mon[36624]: pgmap v10492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:25 smithi081 ceph-mon[31453]: pgmap v10492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:27 smithi104 ceph-mon[36624]: pgmap v10493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:27 smithi081 ceph-mon[31453]: pgmap v10493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:27 smithi153 ceph-mon[36569]: pgmap v10493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:29 smithi104 ceph-mon[36624]: pgmap v10494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:29 smithi081 ceph-mon[31453]: pgmap v10494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:29 smithi153 ceph-mon[36569]: pgmap v10494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:31 smithi104 ceph-mon[36624]: pgmap v10495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:31 smithi081 ceph-mon[31453]: pgmap v10495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:31 smithi153 ceph-mon[36569]: pgmap v10495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:33 smithi104 ceph-mon[36624]: pgmap v10496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:33 smithi081 ceph-mon[31453]: pgmap v10496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:33 smithi153 ceph-mon[36569]: pgmap v10496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:35 smithi081 ceph-mon[31453]: pgmap v10497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:35 smithi153 ceph-mon[36569]: pgmap v10497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:35 smithi104 ceph-mon[36624]: pgmap v10497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:37.501 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:37.501+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:37:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:37.501+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:37:37.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:37.501+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:37:37.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:37 smithi153 ceph-mon[36569]: pgmap v10498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:37 smithi081 ceph-mon[31453]: pgmap v10498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:37 smithi104 ceph-mon[36624]: pgmap v10498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:39 smithi081 ceph-mon[31453]: pgmap v10499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:37:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:39 smithi153 ceph-mon[36569]: pgmap v10499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:37:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:39 smithi104 ceph-mon[36624]: pgmap v10499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:37:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:41 smithi153 ceph-mon[36569]: pgmap v10500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:37:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:37:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:41 smithi104 ceph-mon[36624]: pgmap v10500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:37:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:37:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:41 smithi081 ceph-mon[31453]: pgmap v10500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:37:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:37:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:37:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:42 smithi153 ceph-mon[36569]: pgmap v10501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:42 smithi153 ceph-mon[36569]: pgmap v10502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:42 smithi104 ceph-mon[36624]: pgmap v10501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:42 smithi104 ceph-mon[36624]: pgmap v10502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:42 smithi081 ceph-mon[31453]: pgmap v10501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:42 smithi081 ceph-mon[31453]: pgmap v10502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:44 smithi153 ceph-mon[36569]: pgmap v10503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:44 smithi104 ceph-mon[36624]: pgmap v10503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:44 smithi081 ceph-mon[31453]: pgmap v10503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:45 smithi081 ceph-mon[31453]: pgmap v10504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:45 smithi153 ceph-mon[36569]: pgmap v10504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:45 smithi104 ceph-mon[36624]: pgmap v10504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:47 smithi081 ceph-mon[31453]: pgmap v10505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:47 smithi153 ceph-mon[36569]: pgmap v10505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:47 smithi104 ceph-mon[36624]: pgmap v10505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:49 smithi081 ceph-mon[31453]: pgmap v10506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:49 smithi153 ceph-mon[36569]: pgmap v10506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:49 smithi104 ceph-mon[36624]: pgmap v10506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:51.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:51 smithi104 ceph-mon[36624]: pgmap v10507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:51 smithi081 ceph-mon[31453]: pgmap v10507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:51 smithi153 ceph-mon[36569]: pgmap v10507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:52.513+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:37:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:52.513+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:37:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:37:52.513+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:37:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:53 smithi081 ceph-mon[31453]: pgmap v10508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:53 smithi153 ceph-mon[36569]: pgmap v10508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:53 smithi104 ceph-mon[36624]: pgmap v10508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:55 smithi081 ceph-mon[31453]: pgmap v10509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:55 smithi153 ceph-mon[36569]: pgmap v10509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:55 smithi104 ceph-mon[36624]: pgmap v10509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:57 smithi081 ceph-mon[31453]: pgmap v10510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:57 smithi153 ceph-mon[36569]: pgmap v10510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:57 smithi104 ceph-mon[36624]: pgmap v10510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:37:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:37:59 smithi081 ceph-mon[31453]: pgmap v10511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:37:59 smithi153 ceph-mon[36569]: pgmap v10511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:37:59 smithi104 ceph-mon[36624]: pgmap v10511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:00.888+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:38:00.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:00.888+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:38:00.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:00.889+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:38:00.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:00.889+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 151 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:38:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:01 smithi081 ceph-mon[31453]: pgmap v10512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:01 smithi153 ceph-mon[36569]: pgmap v10512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:01 smithi104 ceph-mon[36624]: pgmap v10512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:03 smithi081 ceph-mon[31453]: pgmap v10513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:03 smithi153 ceph-mon[36569]: pgmap v10513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:03 smithi104 ceph-mon[36624]: pgmap v10513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:05 smithi081 ceph-mon[31453]: pgmap v10514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:05 smithi153 ceph-mon[36569]: pgmap v10514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:05 smithi104 ceph-mon[36624]: pgmap v10514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:07.529+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:38:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:07.529+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:38:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:07.529+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:38:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:07 smithi081 ceph-mon[31453]: pgmap v10515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:07 smithi153 ceph-mon[36569]: pgmap v10515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:07 smithi104 ceph-mon[36624]: pgmap v10515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:09 smithi153 ceph-mon[36569]: pgmap v10516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:09 smithi104 ceph-mon[36624]: pgmap v10516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:09 smithi081 ceph-mon[31453]: pgmap v10516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:11 smithi153 ceph-mon[36569]: pgmap v10517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:11 smithi104 ceph-mon[36624]: pgmap v10517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:11 smithi081 ceph-mon[31453]: pgmap v10517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:13 smithi153 ceph-mon[36569]: pgmap v10518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:13 smithi104 ceph-mon[36624]: pgmap v10518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:13 smithi081 ceph-mon[31453]: pgmap v10518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:15 smithi153 ceph-mon[36569]: pgmap v10519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:15 smithi104 ceph-mon[36624]: pgmap v10519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:15 smithi081 ceph-mon[31453]: pgmap v10519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:17.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:17 smithi081 ceph-mon[31453]: pgmap v10520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:17 smithi153 ceph-mon[36569]: pgmap v10520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:17 smithi104 ceph-mon[36624]: pgmap v10520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:20 smithi153 ceph-mon[36569]: pgmap v10521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:20 smithi104 ceph-mon[36624]: pgmap v10521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:20 smithi081 ceph-mon[31453]: pgmap v10521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:22.541+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:38:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:22.541+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:38:22.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:22.541+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:38:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:22 smithi081 ceph-mon[31453]: pgmap v10522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:22 smithi153 ceph-mon[36569]: pgmap v10522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:22 smithi104 ceph-mon[36624]: pgmap v10522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:24 smithi153 ceph-mon[36569]: pgmap v10523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:24 smithi104 ceph-mon[36624]: pgmap v10523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:24 smithi081 ceph-mon[31453]: pgmap v10523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:26 smithi153 ceph-mon[36569]: pgmap v10524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:26 smithi104 ceph-mon[36624]: pgmap v10524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:26 smithi081 ceph-mon[31453]: pgmap v10524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:28 smithi153 ceph-mon[36569]: pgmap v10525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:28 smithi104 ceph-mon[36624]: pgmap v10525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:28 smithi081 ceph-mon[31453]: pgmap v10525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:30 smithi153 ceph-mon[36569]: pgmap v10526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:30 smithi104 ceph-mon[36624]: pgmap v10526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:30 smithi081 ceph-mon[31453]: pgmap v10526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:32 smithi153 ceph-mon[36569]: pgmap v10527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:32 smithi104 ceph-mon[36624]: pgmap v10527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:32 smithi081 ceph-mon[31453]: pgmap v10527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:34 smithi153 ceph-mon[36569]: pgmap v10528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:34 smithi104 ceph-mon[36624]: pgmap v10528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:34 smithi081 ceph-mon[31453]: pgmap v10528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:36 smithi153 ceph-mon[36569]: pgmap v10529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:36 smithi104 ceph-mon[36624]: pgmap v10529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:36 smithi081 ceph-mon[31453]: pgmap v10529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:37.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:37.556+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:38:37.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:37.556+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:38:37.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:37.556+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:38:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:38 smithi153 ceph-mon[36569]: pgmap v10530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:38 smithi104 ceph-mon[36624]: pgmap v10530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:38 smithi081 ceph-mon[31453]: pgmap v10530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:40 smithi153 ceph-mon[36569]: pgmap v10531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:40 smithi104 ceph-mon[36624]: pgmap v10531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:40 smithi081 ceph-mon[31453]: pgmap v10531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:41.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:38:41.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:41.776 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:38:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:38:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:42 smithi153 ceph-mon[36569]: pgmap v10532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:42 smithi104 ceph-mon[36624]: pgmap v10532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:42 smithi081 ceph-mon[31453]: pgmap v10532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:44 smithi081 ceph-mon[31453]: pgmap v10533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:44 smithi153 ceph-mon[36569]: pgmap v10533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:44 smithi104 ceph-mon[36624]: pgmap v10533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:45 smithi104 ceph-mon[36624]: pgmap v10534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:45 smithi081 ceph-mon[31453]: pgmap v10534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:45 smithi153 ceph-mon[36569]: pgmap v10534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:48 smithi153 ceph-mon[36569]: pgmap v10535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:38:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:38:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:48 smithi153 ceph-mon[36569]: pgmap v10536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:48 smithi153 ceph-mon[36569]: pgmap v10537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:48 smithi104 ceph-mon[36624]: pgmap v10535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:38:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:38:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:48 smithi104 ceph-mon[36624]: pgmap v10536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:48 smithi104 ceph-mon[36624]: pgmap v10537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:48 smithi081 ceph-mon[31453]: pgmap v10535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:38:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:38:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:48 smithi081 ceph-mon[31453]: pgmap v10536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:48 smithi081 ceph-mon[31453]: pgmap v10537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:38:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:50 smithi081 ceph-mon[31453]: pgmap v10538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:50 smithi153 ceph-mon[36569]: pgmap v10538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:50 smithi104 ceph-mon[36624]: pgmap v10538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:52.569+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:38:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:52.569+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:38:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:38:52.569+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:38:52.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:52 smithi104 ceph-mon[36624]: pgmap v10539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:52 smithi081 ceph-mon[31453]: pgmap v10539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:52 smithi153 ceph-mon[36569]: pgmap v10539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:54 smithi104 ceph-mon[36624]: pgmap v10540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:54 smithi081 ceph-mon[31453]: pgmap v10540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:54 smithi153 ceph-mon[36569]: pgmap v10540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:56 smithi081 ceph-mon[31453]: pgmap v10541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:56 smithi153 ceph-mon[36569]: pgmap v10541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:56 smithi104 ceph-mon[36624]: pgmap v10541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:38:58 smithi081 ceph-mon[31453]: pgmap v10542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:38:58 smithi153 ceph-mon[36569]: pgmap v10542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:38:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:38:58 smithi104 ceph-mon[36624]: pgmap v10542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:00 smithi081 ceph-mon[31453]: pgmap v10543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:00 smithi153 ceph-mon[36569]: pgmap v10543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:00 smithi104 ceph-mon[36624]: pgmap v10543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:02 smithi081 ceph-mon[31453]: pgmap v10544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:02 smithi153 ceph-mon[36569]: pgmap v10544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:02 smithi104 ceph-mon[36624]: pgmap v10544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:04 smithi104 ceph-mon[36624]: pgmap v10545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:04 smithi081 ceph-mon[31453]: pgmap v10545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:04 smithi153 ceph-mon[36569]: pgmap v10545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:06 smithi081 ceph-mon[31453]: pgmap v10546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:06 smithi153 ceph-mon[36569]: pgmap v10546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:06 smithi104 ceph-mon[36624]: pgmap v10546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:07.581+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:39:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:07.581+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:39:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:07.581+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:39:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:08 smithi153 ceph-mon[36569]: pgmap v10547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:08 smithi104 ceph-mon[36624]: pgmap v10547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:08 smithi081 ceph-mon[31453]: pgmap v10547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:10 smithi153 ceph-mon[36569]: pgmap v10548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:10.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:10 smithi104 ceph-mon[36624]: pgmap v10548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:10 smithi081 ceph-mon[31453]: pgmap v10548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:12 smithi153 ceph-mon[36569]: pgmap v10549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:12 smithi104 ceph-mon[36624]: pgmap v10549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:12 smithi081 ceph-mon[31453]: pgmap v10549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:14 smithi153 ceph-mon[36569]: pgmap v10550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:14 smithi104 ceph-mon[36624]: pgmap v10550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:14 smithi081 ceph-mon[31453]: pgmap v10550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:16 smithi153 ceph-mon[36569]: pgmap v10551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:16 smithi104 ceph-mon[36624]: pgmap v10551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:16 smithi081 ceph-mon[31453]: pgmap v10551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:18.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:18 smithi081 ceph-mon[31453]: pgmap v10552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:18 smithi153 ceph-mon[36569]: pgmap v10552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:18 smithi104 ceph-mon[36624]: pgmap v10552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:20 smithi153 ceph-mon[36569]: pgmap v10553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:20 smithi104 ceph-mon[36624]: pgmap v10553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:20 smithi081 ceph-mon[31453]: pgmap v10553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:22.596+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:39:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:22.596+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:39:22.597 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:22.596+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:39:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:22 smithi153 ceph-mon[36569]: pgmap v10554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:22 smithi104 ceph-mon[36624]: pgmap v10554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:22 smithi081 ceph-mon[31453]: pgmap v10554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:24 smithi153 ceph-mon[36569]: pgmap v10555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:24 smithi104 ceph-mon[36624]: pgmap v10555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:24 smithi081 ceph-mon[31453]: pgmap v10555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:26 smithi153 ceph-mon[36569]: pgmap v10556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:26 smithi104 ceph-mon[36624]: pgmap v10556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:26 smithi081 ceph-mon[31453]: pgmap v10556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:28 smithi153 ceph-mon[36569]: pgmap v10557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:28 smithi104 ceph-mon[36624]: pgmap v10557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:28 smithi081 ceph-mon[31453]: pgmap v10557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:30 smithi153 ceph-mon[36569]: pgmap v10558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:30 smithi104 ceph-mon[36624]: pgmap v10558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:30 smithi081 ceph-mon[31453]: pgmap v10558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:32 smithi153 ceph-mon[36569]: pgmap v10559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:32 smithi104 ceph-mon[36624]: pgmap v10559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:32 smithi081 ceph-mon[31453]: pgmap v10559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:34 smithi153 ceph-mon[36569]: pgmap v10560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:34 smithi104 ceph-mon[36624]: pgmap v10560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:34 smithi081 ceph-mon[31453]: pgmap v10560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:36 smithi153 ceph-mon[36569]: pgmap v10561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:36 smithi104 ceph-mon[36624]: pgmap v10561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:36 smithi081 ceph-mon[31453]: pgmap v10561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:37.608+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:39:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:37.608+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:39:37.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:37.608+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:39:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:38 smithi153 ceph-mon[36569]: pgmap v10562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:38 smithi104 ceph-mon[36624]: pgmap v10562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:38 smithi081 ceph-mon[31453]: pgmap v10562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:40 smithi153 ceph-mon[36569]: pgmap v10563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:40 smithi104 ceph-mon[36624]: pgmap v10563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:40 smithi081 ceph-mon[31453]: pgmap v10563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:42.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:42 smithi153 ceph-mon[36569]: pgmap v10564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:42 smithi104 ceph-mon[36624]: pgmap v10564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:42 smithi081 ceph-mon[31453]: pgmap v10564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:44 smithi153 ceph-mon[36569]: pgmap v10565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:44 smithi104 ceph-mon[36624]: pgmap v10565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:44 smithi081 ceph-mon[31453]: pgmap v10565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:46 smithi153 ceph-mon[36569]: pgmap v10566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:46 smithi104 ceph-mon[36624]: pgmap v10566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:46.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:46 smithi081 ceph-mon[31453]: pgmap v10566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:48 smithi153 ceph-mon[36569]: pgmap v10567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:39:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:48 smithi104 ceph-mon[36624]: pgmap v10567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:39:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:48 smithi081 ceph-mon[31453]: pgmap v10567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:39:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:39:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:39:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:39:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:39:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:39:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:39:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:39:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:50 smithi153 ceph-mon[36569]: pgmap v10568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:50 smithi153 ceph-mon[36569]: pgmap v10569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:50 smithi104 ceph-mon[36624]: pgmap v10568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:50 smithi104 ceph-mon[36624]: pgmap v10569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:50 smithi081 ceph-mon[31453]: pgmap v10568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:50 smithi081 ceph-mon[31453]: pgmap v10569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:52.623+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:39:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:52.623+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:39:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:39:52.623+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:39:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:52 smithi153 ceph-mon[36569]: pgmap v10570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:52 smithi104 ceph-mon[36624]: pgmap v10570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:52 smithi081 ceph-mon[31453]: pgmap v10570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:54 smithi153 ceph-mon[36569]: pgmap v10571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:54 smithi104 ceph-mon[36624]: pgmap v10571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:54 smithi081 ceph-mon[31453]: pgmap v10571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:56 smithi153 ceph-mon[36569]: pgmap v10572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:56 smithi104 ceph-mon[36624]: pgmap v10572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:56 smithi081 ceph-mon[31453]: pgmap v10572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:39:58 smithi153 ceph-mon[36569]: pgmap v10573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:39:58 smithi104 ceph-mon[36624]: pgmap v10573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:39:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:39:58 smithi081 ceph-mon[31453]: pgmap v10573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:00 smithi153 ceph-mon[36569]: pgmap v10574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:40:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:00 smithi104 ceph-mon[36624]: pgmap v10574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:40:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:00 smithi081 ceph-mon[31453]: pgmap v10574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:40:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:02 smithi153 ceph-mon[36569]: pgmap v10575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:02 smithi104 ceph-mon[36624]: pgmap v10575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:02 smithi081 ceph-mon[31453]: pgmap v10575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:04 smithi104 ceph-mon[36624]: pgmap v10576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:04 smithi081 ceph-mon[31453]: pgmap v10576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:04 smithi153 ceph-mon[36569]: pgmap v10576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:06 smithi104 ceph-mon[36624]: pgmap v10577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:06 smithi081 ceph-mon[31453]: pgmap v10577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:06 smithi153 ceph-mon[36569]: pgmap v10577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:07.635+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:40:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:07.635+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:40:07.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:07.635+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:40:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:08 smithi104 ceph-mon[36624]: pgmap v10578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:08.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:08 smithi153 ceph-mon[36569]: pgmap v10578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:08 smithi081 ceph-mon[31453]: pgmap v10578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:09 smithi081 ceph-mon[31453]: pgmap v10579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:09 smithi153 ceph-mon[36569]: pgmap v10579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:09 smithi104 ceph-mon[36624]: pgmap v10579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:11 smithi081 ceph-mon[31453]: pgmap v10580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:11 smithi153 ceph-mon[36569]: pgmap v10580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:11 smithi104 ceph-mon[36624]: pgmap v10580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:13 smithi081 ceph-mon[31453]: pgmap v10581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:13 smithi153 ceph-mon[36569]: pgmap v10581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:13 smithi104 ceph-mon[36624]: pgmap v10581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:15.910+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:40:15.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:15.910+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:40:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:15.917+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:40:15.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:15.917+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 152 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:40:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:15 smithi153 ceph-mon[36569]: pgmap v10582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:15 smithi104 ceph-mon[36624]: pgmap v10582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:15 smithi081 ceph-mon[31453]: pgmap v10582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:18.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:17 smithi081 ceph-mon[31453]: pgmap v10583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:17 smithi153 ceph-mon[36569]: pgmap v10583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:17 smithi104 ceph-mon[36624]: pgmap v10583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:19 smithi153 ceph-mon[36569]: pgmap v10584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:19 smithi104 ceph-mon[36624]: pgmap v10584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:19 smithi081 ceph-mon[31453]: pgmap v10584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:21 smithi153 ceph-mon[36569]: pgmap v10585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:21 smithi104 ceph-mon[36624]: pgmap v10585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:21 smithi081 ceph-mon[31453]: pgmap v10585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:22.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:22.644+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:40:22.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:22.644+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:40:22.647 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:22.644+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:40:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:23 smithi153 ceph-mon[36569]: pgmap v10586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:23 smithi104 ceph-mon[36624]: pgmap v10586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:23 smithi081 ceph-mon[31453]: pgmap v10586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:26 smithi153 ceph-mon[36569]: pgmap v10587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:26 smithi104 ceph-mon[36624]: pgmap v10587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:26 smithi081 ceph-mon[31453]: pgmap v10587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:28 smithi153 ceph-mon[36569]: pgmap v10588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:28 smithi104 ceph-mon[36624]: pgmap v10588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:28 smithi081 ceph-mon[31453]: pgmap v10588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:30 smithi153 ceph-mon[36569]: pgmap v10589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:30 smithi104 ceph-mon[36624]: pgmap v10589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:30 smithi081 ceph-mon[31453]: pgmap v10589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:32 smithi153 ceph-mon[36569]: pgmap v10590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:32 smithi104 ceph-mon[36624]: pgmap v10590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:32 smithi081 ceph-mon[31453]: pgmap v10590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:34 smithi153 ceph-mon[36569]: pgmap v10591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:34 smithi104 ceph-mon[36624]: pgmap v10591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:34 smithi081 ceph-mon[31453]: pgmap v10591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:36 smithi153 ceph-mon[36569]: pgmap v10592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:36 smithi104 ceph-mon[36624]: pgmap v10592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:36 smithi081 ceph-mon[31453]: pgmap v10592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:37.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:37.656+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:40:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:37.656+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:40:37.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:37.656+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:40:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:38 smithi153 ceph-mon[36569]: pgmap v10593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:38 smithi104 ceph-mon[36624]: pgmap v10593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:38 smithi081 ceph-mon[31453]: pgmap v10593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:40 smithi153 ceph-mon[36569]: pgmap v10594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:40 smithi104 ceph-mon[36624]: pgmap v10594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:40 smithi081 ceph-mon[31453]: pgmap v10594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:42 smithi153 ceph-mon[36569]: pgmap v10595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:42 smithi104 ceph-mon[36624]: pgmap v10595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:42 smithi081 ceph-mon[31453]: pgmap v10595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:44 smithi153 ceph-mon[36569]: pgmap v10596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:44 smithi104 ceph-mon[36624]: pgmap v10596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:44 smithi081 ceph-mon[31453]: pgmap v10596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:46 smithi153 ceph-mon[36569]: pgmap v10597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:46 smithi104 ceph-mon[36624]: pgmap v10597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:46 smithi081 ceph-mon[31453]: pgmap v10597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:48 smithi153 ceph-mon[36569]: pgmap v10598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:48 smithi104 ceph-mon[36624]: pgmap v10598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:48 smithi081 ceph-mon[31453]: pgmap v10598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:40:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:40:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:40:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:50 smithi104 ceph-mon[36624]: pgmap v10599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:40:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:40:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:40:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:50 smithi081 ceph-mon[31453]: pgmap v10599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:40:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:40:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:40:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:50 smithi153 ceph-mon[36569]: pgmap v10599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:40:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:40:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:40:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:51 smithi104 ceph-mon[36624]: pgmap v10600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:51 smithi104 ceph-mon[36624]: pgmap v10601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:51 smithi081 ceph-mon[31453]: pgmap v10600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:51 smithi081 ceph-mon[31453]: pgmap v10601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:51 smithi153 ceph-mon[36569]: pgmap v10600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:51 smithi153 ceph-mon[36569]: pgmap v10601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:52.663+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:40:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:52.663+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:40:52.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:40:52.663+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:40:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:53 smithi104 ceph-mon[36624]: pgmap v10602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:53 smithi081 ceph-mon[31453]: pgmap v10602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:53 smithi153 ceph-mon[36569]: pgmap v10602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:54 smithi081 ceph-mon[31453]: pgmap v10603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:54 smithi153 ceph-mon[36569]: pgmap v10603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:54 smithi104 ceph-mon[36624]: pgmap v10603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:57 smithi081 ceph-mon[31453]: pgmap v10604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:57 smithi153 ceph-mon[36569]: pgmap v10604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:57 smithi104 ceph-mon[36624]: pgmap v10604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:40:59 smithi081 ceph-mon[31453]: pgmap v10605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:40:59 smithi153 ceph-mon[36569]: pgmap v10605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:40:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:40:59 smithi104 ceph-mon[36624]: pgmap v10605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:01 smithi153 ceph-mon[36569]: pgmap v10606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:01 smithi104 ceph-mon[36624]: pgmap v10606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:01 smithi081 ceph-mon[31453]: pgmap v10606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:03 smithi153 ceph-mon[36569]: pgmap v10607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:03 smithi104 ceph-mon[36624]: pgmap v10607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:03 smithi081 ceph-mon[31453]: pgmap v10607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:05 smithi153 ceph-mon[36569]: pgmap v10608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:05 smithi104 ceph-mon[36624]: pgmap v10608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:05 smithi081 ceph-mon[31453]: pgmap v10608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:07.677+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:07.677+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:41:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:07.677+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:41:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:07 smithi153 ceph-mon[36569]: pgmap v10609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:07 smithi104 ceph-mon[36624]: pgmap v10609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:07 smithi081 ceph-mon[31453]: pgmap v10609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:09 smithi153 ceph-mon[36569]: pgmap v10610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:09 smithi104 ceph-mon[36624]: pgmap v10610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:09 smithi081 ceph-mon[31453]: pgmap v10610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:11 smithi153 ceph-mon[36569]: pgmap v10611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:11 smithi104 ceph-mon[36624]: pgmap v10611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:11 smithi081 ceph-mon[31453]: pgmap v10611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:13 smithi153 ceph-mon[36569]: pgmap v10612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:13 smithi104 ceph-mon[36624]: pgmap v10612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:13.864 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:13 smithi081 ceph-mon[31453]: pgmap v10612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:15 smithi153 ceph-mon[36569]: pgmap v10613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:15 smithi104 ceph-mon[36624]: pgmap v10613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:15 smithi081 ceph-mon[31453]: pgmap v10613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:17 smithi153 ceph-mon[36569]: pgmap v10614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:17 smithi104 ceph-mon[36624]: pgmap v10614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:17 smithi081 ceph-mon[31453]: pgmap v10614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:19.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:19 smithi081 ceph-mon[31453]: pgmap v10615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:19 smithi153 ceph-mon[36569]: pgmap v10615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:19 smithi104 ceph-mon[36624]: pgmap v10615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:21 smithi153 ceph-mon[36569]: pgmap v10616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:21 smithi104 ceph-mon[36624]: pgmap v10616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:21 smithi081 ceph-mon[31453]: pgmap v10616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:22.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:22.690+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:41:22.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:22.691+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:41:22.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:22.691+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:41:23.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:23 smithi153 ceph-mon[36569]: pgmap v10617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:23 smithi104 ceph-mon[36624]: pgmap v10617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:23 smithi081 ceph-mon[31453]: pgmap v10617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:25 smithi153 ceph-mon[36569]: pgmap v10618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:25 smithi104 ceph-mon[36624]: pgmap v10618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:25 smithi081 ceph-mon[31453]: pgmap v10618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:27 smithi153 ceph-mon[36569]: pgmap v10619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:27 smithi104 ceph-mon[36624]: pgmap v10619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:27 smithi081 ceph-mon[31453]: pgmap v10619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:29 smithi153 ceph-mon[36569]: pgmap v10620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:29 smithi104 ceph-mon[36624]: pgmap v10620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:29 smithi081 ceph-mon[31453]: pgmap v10620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:31 smithi153 ceph-mon[36569]: pgmap v10621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:31 smithi104 ceph-mon[36624]: pgmap v10621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:31 smithi081 ceph-mon[31453]: pgmap v10621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:33 smithi153 ceph-mon[36569]: pgmap v10622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:33 smithi104 ceph-mon[36624]: pgmap v10622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:33 smithi081 ceph-mon[31453]: pgmap v10622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:35 smithi153 ceph-mon[36569]: pgmap v10623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:35 smithi104 ceph-mon[36624]: pgmap v10623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:35 smithi081 ceph-mon[31453]: pgmap v10623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:37.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:37.706+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:41:37.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:37.706+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:41:37.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:37.706+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:41:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:37 smithi153 ceph-mon[36569]: pgmap v10624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:37 smithi104 ceph-mon[36624]: pgmap v10624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:37 smithi081 ceph-mon[31453]: pgmap v10624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:39 smithi153 ceph-mon[36569]: pgmap v10625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:39 smithi104 ceph-mon[36624]: pgmap v10625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:39 smithi081 ceph-mon[31453]: pgmap v10625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:41 smithi153 ceph-mon[36569]: pgmap v10626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:41 smithi104 ceph-mon[36624]: pgmap v10626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:41 smithi081 ceph-mon[31453]: pgmap v10626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:43 smithi153 ceph-mon[36569]: pgmap v10627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:43 smithi104 ceph-mon[36624]: pgmap v10627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:43 smithi081 ceph-mon[31453]: pgmap v10627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:45 smithi153 ceph-mon[36569]: pgmap v10628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:45 smithi104 ceph-mon[36624]: pgmap v10628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:45 smithi081 ceph-mon[31453]: pgmap v10628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:47 smithi153 ceph-mon[36569]: pgmap v10629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:47 smithi104 ceph-mon[36624]: pgmap v10629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:47 smithi081 ceph-mon[31453]: pgmap v10629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:49 smithi153 ceph-mon[36569]: pgmap v10630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:49 smithi104 ceph-mon[36624]: pgmap v10630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:49 smithi081 ceph-mon[31453]: pgmap v10630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:51 smithi153 ceph-mon[36569]: pgmap v10631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:41:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:51 smithi104 ceph-mon[36624]: pgmap v10631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:41:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:51 smithi081 ceph-mon[31453]: pgmap v10631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:41:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:52.714+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:41:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:52.714+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:41:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:41:52.714+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:41:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:41:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:41:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:41:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:41:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:41:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:41:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:41:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:41:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:41:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:53 smithi153 ceph-mon[36569]: pgmap v10632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:53 smithi153 ceph-mon[36569]: pgmap v10633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:53 smithi104 ceph-mon[36624]: pgmap v10632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:53 smithi104 ceph-mon[36624]: pgmap v10633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:53 smithi081 ceph-mon[31453]: pgmap v10632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:53 smithi081 ceph-mon[31453]: pgmap v10633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:55 smithi153 ceph-mon[36569]: pgmap v10634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:55 smithi104 ceph-mon[36624]: pgmap v10634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:55 smithi081 ceph-mon[31453]: pgmap v10634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:57 smithi104 ceph-mon[36624]: pgmap v10635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:57 smithi081 ceph-mon[31453]: pgmap v10635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:57 smithi153 ceph-mon[36569]: pgmap v10635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:41:59 smithi104 ceph-mon[36624]: pgmap v10636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:41:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:41:59 smithi081 ceph-mon[31453]: pgmap v10636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:41:59 smithi153 ceph-mon[36569]: pgmap v10636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:01 smithi104 ceph-mon[36624]: pgmap v10637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:01 smithi081 ceph-mon[31453]: pgmap v10637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:01 smithi153 ceph-mon[36569]: pgmap v10637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:03 smithi104 ceph-mon[36624]: pgmap v10638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:03 smithi081 ceph-mon[31453]: pgmap v10638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:03 smithi153 ceph-mon[36569]: pgmap v10638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:05 smithi104 ceph-mon[36624]: pgmap v10639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:05 smithi081 ceph-mon[31453]: pgmap v10639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:05 smithi153 ceph-mon[36569]: pgmap v10639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:07.730 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:07.729+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:42:07.730 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:07.729+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:42:07.731 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:07.729+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:42:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:07 smithi081 ceph-mon[31453]: pgmap v10640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:07 smithi153 ceph-mon[36569]: pgmap v10640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:07 smithi104 ceph-mon[36624]: pgmap v10640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:09.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:09 smithi153 ceph-mon[36569]: pgmap v10641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:09 smithi081 ceph-mon[31453]: pgmap v10641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:09 smithi104 ceph-mon[36624]: pgmap v10641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:11 smithi081 ceph-mon[31453]: pgmap v10642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:11 smithi153 ceph-mon[36569]: pgmap v10642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:11 smithi104 ceph-mon[36624]: pgmap v10642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:12 smithi081 ceph-mon[31453]: pgmap v10643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:12 smithi153 ceph-mon[36569]: pgmap v10643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:12 smithi104 ceph-mon[36624]: pgmap v10643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:15.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:14 smithi081 ceph-mon[31453]: pgmap v10644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:14 smithi153 ceph-mon[36569]: pgmap v10644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:14 smithi104 ceph-mon[36624]: pgmap v10644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:16 smithi153 ceph-mon[36569]: pgmap v10645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:16 smithi104 ceph-mon[36624]: pgmap v10645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:16 smithi081 ceph-mon[31453]: pgmap v10645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:19.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:18 smithi081 ceph-mon[31453]: pgmap v10646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:18 smithi153 ceph-mon[36569]: pgmap v10646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:18 smithi104 ceph-mon[36624]: pgmap v10646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:20 smithi153 ceph-mon[36569]: pgmap v10647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:20 smithi104 ceph-mon[36624]: pgmap v10647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:20 smithi081 ceph-mon[31453]: pgmap v10647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:22.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:22.734+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:42:22.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:22.734+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:42:22.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:22.734+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:42:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:22 smithi153 ceph-mon[36569]: pgmap v10648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:22 smithi104 ceph-mon[36624]: pgmap v10648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:22 smithi081 ceph-mon[31453]: pgmap v10648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:24 smithi153 ceph-mon[36569]: pgmap v10649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:24 smithi104 ceph-mon[36624]: pgmap v10649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:24 smithi081 ceph-mon[31453]: pgmap v10649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:26 smithi153 ceph-mon[36569]: pgmap v10650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:26 smithi104 ceph-mon[36624]: pgmap v10650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:26 smithi081 ceph-mon[31453]: pgmap v10650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:28 smithi153 ceph-mon[36569]: pgmap v10651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:28 smithi104 ceph-mon[36624]: pgmap v10651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:28 smithi081 ceph-mon[31453]: pgmap v10651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:31 smithi153 ceph-mon[36569]: pgmap v10652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:31 smithi104 ceph-mon[36624]: pgmap v10652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:31 smithi081 ceph-mon[31453]: pgmap v10652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:33 smithi153 ceph-mon[36569]: pgmap v10653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:33 smithi104 ceph-mon[36624]: pgmap v10653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:33 smithi081 ceph-mon[31453]: pgmap v10653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:35 smithi153 ceph-mon[36569]: pgmap v10654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:35 smithi104 ceph-mon[36624]: pgmap v10654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:35 smithi081 ceph-mon[31453]: pgmap v10654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:37.749+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:42:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:37.749+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:42:37.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:37.749+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:42:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:37 smithi104 ceph-mon[36624]: pgmap v10655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:37 smithi081 ceph-mon[31453]: pgmap v10655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:37 smithi153 ceph-mon[36569]: pgmap v10655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:39 smithi104 ceph-mon[36624]: pgmap v10656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:39.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:39 smithi153 ceph-mon[36569]: pgmap v10656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:39 smithi081 ceph-mon[31453]: pgmap v10656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:41 smithi104 ceph-mon[36624]: pgmap v10657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:41 smithi081 ceph-mon[31453]: pgmap v10657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:41 smithi153 ceph-mon[36569]: pgmap v10657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:43 smithi104 ceph-mon[36624]: pgmap v10658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:43 smithi081 ceph-mon[31453]: pgmap v10658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:43 smithi153 ceph-mon[36569]: pgmap v10658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:45 smithi104 ceph-mon[36624]: pgmap v10659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:45 smithi081 ceph-mon[31453]: pgmap v10659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:45 smithi153 ceph-mon[36569]: pgmap v10659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:47 smithi081 ceph-mon[31453]: pgmap v10660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:47 smithi153 ceph-mon[36569]: pgmap v10660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:47 smithi104 ceph-mon[36624]: pgmap v10660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:49 smithi081 ceph-mon[31453]: pgmap v10661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:49 smithi153 ceph-mon[36569]: pgmap v10661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:49 smithi104 ceph-mon[36624]: pgmap v10661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:51 smithi081 ceph-mon[31453]: pgmap v10662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:51 smithi153 ceph-mon[36569]: pgmap v10662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:51 smithi104 ceph-mon[36624]: pgmap v10662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:52.763+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:42:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:52.763+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:42:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:42:52.763+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:42:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:52 smithi081 ceph-mon[31453]: pgmap v10663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:42:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:52 smithi153 ceph-mon[36569]: pgmap v10663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:42:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:52 smithi104 ceph-mon[36624]: pgmap v10663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:42:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:42:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:42:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:53 smithi081 ceph-mon[31453]: pgmap v10664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:53 smithi081 ceph-mon[31453]: pgmap v10665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:42:53.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:42:53.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:42:53.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:53 smithi104 ceph-mon[36624]: pgmap v10664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:53.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:53 smithi104 ceph-mon[36624]: pgmap v10665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:53.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:42:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:42:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:42:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:53 smithi153 ceph-mon[36569]: pgmap v10664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:53 smithi153 ceph-mon[36569]: pgmap v10665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:42:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:56 smithi153 ceph-mon[36569]: pgmap v10666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:56 smithi104 ceph-mon[36624]: pgmap v10666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:56 smithi081 ceph-mon[31453]: pgmap v10666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:42:58 smithi153 ceph-mon[36569]: pgmap v10667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:42:58 smithi104 ceph-mon[36624]: pgmap v10667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:42:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:42:58 smithi081 ceph-mon[31453]: pgmap v10667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:00 smithi153 ceph-mon[36569]: pgmap v10668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:00 smithi104 ceph-mon[36624]: pgmap v10668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:00 smithi081 ceph-mon[31453]: pgmap v10668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:02 smithi153 ceph-mon[36569]: pgmap v10669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:02 smithi104 ceph-mon[36624]: pgmap v10669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:02 smithi081 ceph-mon[31453]: pgmap v10669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:04 smithi153 ceph-mon[36569]: pgmap v10670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:04 smithi104 ceph-mon[36624]: pgmap v10670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:04 smithi081 ceph-mon[31453]: pgmap v10670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:06.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:06 smithi153 ceph-mon[36569]: pgmap v10671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:06 smithi104 ceph-mon[36624]: pgmap v10671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:06 smithi081 ceph-mon[31453]: pgmap v10671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:07.777 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:07.777+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:43:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:07.777+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:43:07.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:07.777+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:43:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:08 smithi153 ceph-mon[36569]: pgmap v10672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:08 smithi104 ceph-mon[36624]: pgmap v10672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:08 smithi081 ceph-mon[31453]: pgmap v10672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:10 smithi153 ceph-mon[36569]: pgmap v10673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:10.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:10 smithi104 ceph-mon[36624]: pgmap v10673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:10.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:10 smithi081 ceph-mon[31453]: pgmap v10673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:12 smithi153 ceph-mon[36569]: pgmap v10674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:12 smithi104 ceph-mon[36624]: pgmap v10674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:12 smithi081 ceph-mon[31453]: pgmap v10674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:14 smithi153 ceph-mon[36569]: pgmap v10675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:14 smithi104 ceph-mon[36624]: pgmap v10675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:14 smithi081 ceph-mon[31453]: pgmap v10675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:15.940+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:43:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:15.940+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:43:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:15.941+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:43:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:15.941+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 153 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:43:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:16 smithi153 ceph-mon[36569]: pgmap v10676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:16 smithi104 ceph-mon[36624]: pgmap v10676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:16 smithi081 ceph-mon[31453]: pgmap v10676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:18 smithi153 ceph-mon[36569]: pgmap v10677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:18 smithi104 ceph-mon[36624]: pgmap v10677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:18 smithi081 ceph-mon[31453]: pgmap v10677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:20 smithi153 ceph-mon[36569]: pgmap v10678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:20 smithi104 ceph-mon[36624]: pgmap v10678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:20 smithi081 ceph-mon[31453]: pgmap v10678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:22.786 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:22.785+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:43:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:22.785+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:43:22.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:22.785+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:43:22.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:22 smithi153 ceph-mon[36569]: pgmap v10679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:22 smithi104 ceph-mon[36624]: pgmap v10679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:22 smithi081 ceph-mon[31453]: pgmap v10679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:24 smithi153 ceph-mon[36569]: pgmap v10680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:24 smithi104 ceph-mon[36624]: pgmap v10680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:24.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:24 smithi081 ceph-mon[31453]: pgmap v10680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:26 smithi153 ceph-mon[36569]: pgmap v10681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:26 smithi104 ceph-mon[36624]: pgmap v10681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:26 smithi081 ceph-mon[31453]: pgmap v10681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:28 smithi153 ceph-mon[36569]: pgmap v10682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:28 smithi104 ceph-mon[36624]: pgmap v10682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:28 smithi081 ceph-mon[31453]: pgmap v10682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:30 smithi153 ceph-mon[36569]: pgmap v10683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:30 smithi104 ceph-mon[36624]: pgmap v10683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:30 smithi081 ceph-mon[31453]: pgmap v10683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:32.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:32 smithi153 ceph-mon[36569]: pgmap v10684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:32 smithi104 ceph-mon[36624]: pgmap v10684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:32 smithi081 ceph-mon[31453]: pgmap v10684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:34 smithi153 ceph-mon[36569]: pgmap v10685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:34 smithi104 ceph-mon[36624]: pgmap v10685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:34 smithi081 ceph-mon[31453]: pgmap v10685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:36 smithi153 ceph-mon[36569]: pgmap v10686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:36 smithi104 ceph-mon[36624]: pgmap v10686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:36 smithi081 ceph-mon[31453]: pgmap v10686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:37.794+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:43:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:37.794+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:43:37.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:37.794+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:43:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:38 smithi153 ceph-mon[36569]: pgmap v10687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:38 smithi104 ceph-mon[36624]: pgmap v10687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:38 smithi081 ceph-mon[31453]: pgmap v10687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:40 smithi153 ceph-mon[36569]: pgmap v10688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:40 smithi104 ceph-mon[36624]: pgmap v10688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:40.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:40 smithi081 ceph-mon[31453]: pgmap v10688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:42 smithi153 ceph-mon[36569]: pgmap v10689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:42 smithi104 ceph-mon[36624]: pgmap v10689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:42 smithi081 ceph-mon[31453]: pgmap v10689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:44 smithi153 ceph-mon[36569]: pgmap v10690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:44 smithi104 ceph-mon[36624]: pgmap v10690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:44 smithi081 ceph-mon[31453]: pgmap v10690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:46 smithi153 ceph-mon[36569]: pgmap v10691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:46 smithi104 ceph-mon[36624]: pgmap v10691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:46.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:46 smithi081 ceph-mon[31453]: pgmap v10691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:48 smithi104 ceph-mon[36624]: pgmap v10692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:48 smithi081 ceph-mon[31453]: pgmap v10692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:48 smithi153 ceph-mon[36569]: pgmap v10692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:50.813 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:50 smithi081 ceph-mon[31453]: pgmap v10693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:50 smithi104 ceph-mon[36624]: pgmap v10693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:50 smithi153 ceph-mon[36569]: pgmap v10693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:52.808+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:43:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:52.809+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:43:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:43:52.809+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:43:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:52 smithi104 ceph-mon[36624]: pgmap v10694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:52 smithi081 ceph-mon[31453]: pgmap v10694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:52 smithi153 ceph-mon[36569]: pgmap v10694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:54 smithi104 ceph-mon[36624]: pgmap v10695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:43:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:54 smithi081 ceph-mon[31453]: pgmap v10695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:43:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:54 smithi153 ceph-mon[36569]: pgmap v10695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:43:56.626 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:56 smithi104 ceph-mon[36624]: pgmap v10696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:56 smithi081 ceph-mon[31453]: pgmap v10696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:56 smithi153 ceph-mon[36569]: pgmap v10696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:58 smithi104 ceph-mon[36624]: pgmap v10697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:43:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:43:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:43:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:58 smithi081 ceph-mon[31453]: pgmap v10697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:43:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:43:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:43:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:58 smithi153 ceph-mon[36569]: pgmap v10697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:43:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:43:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:43:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:00 smithi081 ceph-mon[31453]: pgmap v10698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:00 smithi153 ceph-mon[36569]: pgmap v10698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:00 smithi104 ceph-mon[36624]: pgmap v10698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:44:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:44:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:01 smithi081 ceph-mon[31453]: pgmap v10699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:01 smithi081 ceph-mon[31453]: pgmap v10700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:44:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:44:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:01 smithi153 ceph-mon[36569]: pgmap v10699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:01 smithi153 ceph-mon[36569]: pgmap v10700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:44:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:44:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:01 smithi104 ceph-mon[36624]: pgmap v10699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:01 smithi104 ceph-mon[36624]: pgmap v10700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:44:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:03 smithi081 ceph-mon[31453]: pgmap v10701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:03 smithi153 ceph-mon[36569]: pgmap v10701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:03 smithi104 ceph-mon[36624]: pgmap v10701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:05 smithi153 ceph-mon[36569]: pgmap v10702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:05 smithi104 ceph-mon[36624]: pgmap v10702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:05 smithi081 ceph-mon[31453]: pgmap v10702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:07.825+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:44:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:07.825+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:44:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:07.825+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:44:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:07 smithi153 ceph-mon[36569]: pgmap v10703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:07 smithi104 ceph-mon[36624]: pgmap v10703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:07 smithi081 ceph-mon[31453]: pgmap v10703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:10 smithi153 ceph-mon[36569]: pgmap v10704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:10 smithi104 ceph-mon[36624]: pgmap v10704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:10 smithi081 ceph-mon[31453]: pgmap v10704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:12 smithi153 ceph-mon[36569]: pgmap v10705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:12 smithi104 ceph-mon[36624]: pgmap v10705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:12 smithi081 ceph-mon[31453]: pgmap v10705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:14 smithi153 ceph-mon[36569]: pgmap v10706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:14 smithi104 ceph-mon[36624]: pgmap v10706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:14 smithi081 ceph-mon[31453]: pgmap v10706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:16 smithi153 ceph-mon[36569]: pgmap v10707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:16 smithi104 ceph-mon[36624]: pgmap v10707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:16.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:16 smithi081 ceph-mon[31453]: pgmap v10707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:18 smithi153 ceph-mon[36569]: pgmap v10708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:18 smithi104 ceph-mon[36624]: pgmap v10708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:18 smithi081 ceph-mon[31453]: pgmap v10708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:20 smithi153 ceph-mon[36569]: pgmap v10709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:20 smithi104 ceph-mon[36624]: pgmap v10709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:20.337 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:20 smithi081 ceph-mon[31453]: pgmap v10709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:22 smithi153 ceph-mon[36569]: pgmap v10710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:22 smithi104 ceph-mon[36624]: pgmap v10710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:22 smithi081 ceph-mon[31453]: pgmap v10710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:22.835+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:44:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:22.835+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:44:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:22.835+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:44:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:24 smithi153 ceph-mon[36569]: pgmap v10711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:24 smithi104 ceph-mon[36624]: pgmap v10711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:24 smithi081 ceph-mon[31453]: pgmap v10711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:26 smithi081 ceph-mon[31453]: pgmap v10712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:26 smithi153 ceph-mon[36569]: pgmap v10712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:26 smithi104 ceph-mon[36624]: pgmap v10712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:28 smithi081 ceph-mon[31453]: pgmap v10713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:28 smithi153 ceph-mon[36569]: pgmap v10713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:28 smithi104 ceph-mon[36624]: pgmap v10713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:30 smithi081 ceph-mon[31453]: pgmap v10714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:30 smithi153 ceph-mon[36569]: pgmap v10714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:30 smithi104 ceph-mon[36624]: pgmap v10714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:32 smithi081 ceph-mon[31453]: pgmap v10715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:32 smithi153 ceph-mon[36569]: pgmap v10715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:32 smithi104 ceph-mon[36624]: pgmap v10715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:34 smithi081 ceph-mon[31453]: pgmap v10716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:34 smithi153 ceph-mon[36569]: pgmap v10716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:34 smithi104 ceph-mon[36624]: pgmap v10716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:36 smithi081 ceph-mon[31453]: pgmap v10717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:36 smithi153 ceph-mon[36569]: pgmap v10717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:36 smithi104 ceph-mon[36624]: pgmap v10717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:37.844+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:44:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:37.844+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:44:37.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:37.844+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:44:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:38 smithi081 ceph-mon[31453]: pgmap v10718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:38 smithi153 ceph-mon[36569]: pgmap v10718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:38 smithi104 ceph-mon[36624]: pgmap v10718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:40 smithi081 ceph-mon[31453]: pgmap v10719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:40 smithi153 ceph-mon[36569]: pgmap v10719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:40 smithi104 ceph-mon[36624]: pgmap v10719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:42 smithi081 ceph-mon[31453]: pgmap v10720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:42 smithi153 ceph-mon[36569]: pgmap v10720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:42 smithi104 ceph-mon[36624]: pgmap v10720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:43 smithi153 ceph-mon[36569]: pgmap v10721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:43 smithi104 ceph-mon[36624]: pgmap v10721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:43 smithi081 ceph-mon[31453]: pgmap v10721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:46 smithi153 ceph-mon[36569]: pgmap v10722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:46 smithi104 ceph-mon[36624]: pgmap v10722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:46 smithi081 ceph-mon[31453]: pgmap v10722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:48 smithi153 ceph-mon[36569]: pgmap v10723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:48 smithi104 ceph-mon[36624]: pgmap v10723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:48 smithi081 ceph-mon[31453]: pgmap v10723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:50.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:50 smithi081 ceph-mon[31453]: pgmap v10724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:50 smithi153 ceph-mon[36569]: pgmap v10724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:50 smithi104 ceph-mon[36624]: pgmap v10724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:52 smithi153 ceph-mon[36569]: pgmap v10725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:52 smithi104 ceph-mon[36624]: pgmap v10725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:52 smithi081 ceph-mon[31453]: pgmap v10725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:52.860+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:44:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:52.861+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:44:52.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:44:52.861+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:44:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:54 smithi153 ceph-mon[36569]: pgmap v10726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:54 smithi104 ceph-mon[36624]: pgmap v10726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:54 smithi081 ceph-mon[31453]: pgmap v10726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:56 smithi153 ceph-mon[36569]: pgmap v10727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:56 smithi104 ceph-mon[36624]: pgmap v10727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:56 smithi081 ceph-mon[31453]: pgmap v10727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:44:58 smithi153 ceph-mon[36569]: pgmap v10728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:44:58 smithi104 ceph-mon[36624]: pgmap v10728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:44:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:44:58 smithi081 ceph-mon[31453]: pgmap v10728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:00 smithi153 ceph-mon[36569]: pgmap v10729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:00 smithi104 ceph-mon[36624]: pgmap v10729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:00 smithi081 ceph-mon[31453]: pgmap v10729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:02 smithi153 ceph-mon[36569]: pgmap v10730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:45:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:02 smithi104 ceph-mon[36624]: pgmap v10730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:45:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:02 smithi081 ceph-mon[31453]: pgmap v10730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:45:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:03 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:45:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:03 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:45:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:03 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:45:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:03 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:45:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:03 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:45:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:03 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:45:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:03 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:45:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:03 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:45:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:03 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:45:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:03 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:45:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:03 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:45:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:03 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:45:04.224 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:04 smithi153 ceph-mon[36569]: pgmap v10731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:04 smithi104 ceph-mon[36624]: pgmap v10731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:04 smithi081 ceph-mon[31453]: pgmap v10731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:06 smithi104 ceph-mon[36624]: pgmap v10732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:06 smithi081 ceph-mon[31453]: pgmap v10732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:06.472 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:06 smithi153 ceph-mon[36569]: pgmap v10732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: pgmap v10733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: pgmap v10734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:45:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:45:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:45:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: pgmap v10733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: pgmap v10734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: pgmap v10733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: pgmap v10734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:45:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:07.872+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:45:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:07.872+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:45:07.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:07.872+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:45:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:09 smithi104 ceph-mon[36624]: pgmap v10735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:09 smithi081 ceph-mon[31453]: pgmap v10735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:09 smithi153 ceph-mon[36569]: pgmap v10735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:11 smithi104 ceph-mon[36624]: pgmap v10736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:11.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:11 smithi153 ceph-mon[36569]: pgmap v10736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:11 smithi081 ceph-mon[31453]: pgmap v10736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:13 smithi104 ceph-mon[36624]: pgmap v10737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:13 smithi081 ceph-mon[31453]: pgmap v10737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:13 smithi153 ceph-mon[36569]: pgmap v10737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:15 smithi104 ceph-mon[36624]: pgmap v10738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:15 smithi081 ceph-mon[31453]: pgmap v10738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:15 smithi153 ceph-mon[36569]: pgmap v10738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:17 smithi153 ceph-mon[36569]: pgmap v10739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:17 smithi104 ceph-mon[36624]: pgmap v10739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:17 smithi081 ceph-mon[31453]: pgmap v10739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:19 smithi104 ceph-mon[36624]: pgmap v10740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:19 smithi081 ceph-mon[31453]: pgmap v10740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:19 smithi153 ceph-mon[36569]: pgmap v10740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:21.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:21 smithi081 ceph-mon[31453]: pgmap v10741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:21 smithi153 ceph-mon[36569]: pgmap v10741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:21 smithi104 ceph-mon[36624]: pgmap v10741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:22.886+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:45:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:22.886+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:45:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:22.886+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:45:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:23 smithi081 ceph-mon[31453]: pgmap v10742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:23 smithi153 ceph-mon[36569]: pgmap v10742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:23 smithi104 ceph-mon[36624]: pgmap v10742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:25 smithi081 ceph-mon[31453]: pgmap v10743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:25 smithi153 ceph-mon[36569]: pgmap v10743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:25 smithi104 ceph-mon[36624]: pgmap v10743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:27 smithi081 ceph-mon[31453]: pgmap v10744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:27 smithi153 ceph-mon[36569]: pgmap v10744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:27 smithi104 ceph-mon[36624]: pgmap v10744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:29 smithi081 ceph-mon[31453]: pgmap v10745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:29 smithi153 ceph-mon[36569]: pgmap v10745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:29 smithi104 ceph-mon[36624]: pgmap v10745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:31 smithi081 ceph-mon[31453]: pgmap v10746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:31 smithi153 ceph-mon[36569]: pgmap v10746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:31 smithi104 ceph-mon[36624]: pgmap v10746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:33 smithi081 ceph-mon[31453]: pgmap v10747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:33 smithi153 ceph-mon[36569]: pgmap v10747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:33 smithi104 ceph-mon[36624]: pgmap v10747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:35 smithi081 ceph-mon[31453]: pgmap v10748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:35 smithi153 ceph-mon[36569]: pgmap v10748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:35 smithi104 ceph-mon[36624]: pgmap v10748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:37.901+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:45:37.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:37.901+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:45:37.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:37.901+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:45:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:37 smithi081 ceph-mon[31453]: pgmap v10749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:37 smithi153 ceph-mon[36569]: pgmap v10749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:37 smithi104 ceph-mon[36624]: pgmap v10749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:38 smithi153 ceph-mon[36569]: pgmap v10750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:38 smithi104 ceph-mon[36624]: pgmap v10750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:38 smithi081 ceph-mon[31453]: pgmap v10750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:41 smithi104 ceph-mon[36624]: pgmap v10751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:41 smithi081 ceph-mon[31453]: pgmap v10751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:41.760 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:41 smithi153 ceph-mon[36569]: pgmap v10751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:43 smithi104 ceph-mon[36624]: pgmap v10752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:43 smithi081 ceph-mon[31453]: pgmap v10752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:43 smithi153 ceph-mon[36569]: pgmap v10752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:45 smithi081 ceph-mon[31453]: pgmap v10753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:45 smithi153 ceph-mon[36569]: pgmap v10753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:45 smithi104 ceph-mon[36624]: pgmap v10753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:47 smithi081 ceph-mon[31453]: pgmap v10754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:47 smithi153 ceph-mon[36569]: pgmap v10754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:47 smithi104 ceph-mon[36624]: pgmap v10754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:49 smithi081 ceph-mon[31453]: pgmap v10755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:49 smithi153 ceph-mon[36569]: pgmap v10755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:49 smithi104 ceph-mon[36624]: pgmap v10755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:51 smithi081 ceph-mon[31453]: pgmap v10756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:51 smithi153 ceph-mon[36569]: pgmap v10756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:51 smithi104 ceph-mon[36624]: pgmap v10756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:52.916+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:45:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:52.916+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:45:52.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:45:52.916+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:45:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:53 smithi081 ceph-mon[31453]: pgmap v10757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:53 smithi153 ceph-mon[36569]: pgmap v10757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:53 smithi104 ceph-mon[36624]: pgmap v10757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:55.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:55 smithi104 ceph-mon[36624]: pgmap v10758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:55 smithi081 ceph-mon[31453]: pgmap v10758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:55 smithi153 ceph-mon[36569]: pgmap v10758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:57 smithi081 ceph-mon[31453]: pgmap v10759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:57 smithi153 ceph-mon[36569]: pgmap v10759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:57 smithi104 ceph-mon[36624]: pgmap v10759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:45:59 smithi081 ceph-mon[31453]: pgmap v10760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:45:59 smithi153 ceph-mon[36569]: pgmap v10760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:45:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:45:59 smithi104 ceph-mon[36624]: pgmap v10760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:01 smithi081 ceph-mon[31453]: pgmap v10761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:01 smithi153 ceph-mon[36569]: pgmap v10761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:01 smithi104 ceph-mon[36624]: pgmap v10761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:03 smithi081 ceph-mon[31453]: pgmap v10762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:03 smithi153 ceph-mon[36569]: pgmap v10762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:03 smithi104 ceph-mon[36624]: pgmap v10762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:05 smithi081 ceph-mon[31453]: pgmap v10763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:05 smithi153 ceph-mon[36569]: pgmap v10763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:05 smithi104 ceph-mon[36624]: pgmap v10763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:07 smithi081 ceph-mon[31453]: pgmap v10764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:46:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:07 smithi153 ceph-mon[36569]: pgmap v10764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:46:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:07 smithi104 ceph-mon[36624]: pgmap v10764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:46:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:07.931+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:46:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:07.931+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:46:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:07.931+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:46:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:46:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:46:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:46:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:46:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:46:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:46:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:46:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:46:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:46:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:09 smithi153 ceph-mon[36569]: pgmap v10765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:09 smithi153 ceph-mon[36569]: pgmap v10766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:09 smithi104 ceph-mon[36624]: pgmap v10765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:09 smithi104 ceph-mon[36624]: pgmap v10766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:09 smithi081 ceph-mon[31453]: pgmap v10765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:09 smithi081 ceph-mon[31453]: pgmap v10766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:11 smithi153 ceph-mon[36569]: pgmap v10767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:11 smithi104 ceph-mon[36624]: pgmap v10767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:11 smithi081 ceph-mon[31453]: pgmap v10767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 msgr2=0x7f16e80b4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-13T02:46:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 msgr2=0x7f16e80b4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-13T02:46:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b4230 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006580 tx=0x7f16f40ac000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-13T02:46:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b4230 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006580 tx=0x7f16f40ac000 comp rx=0 tx=0).stop 2024-04-13T02:46:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 msgr2=0x7f16e80b4230 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.81:6800/3972789130, fault. 2024-04-13T02:46:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 msgr2=0x7f16e80b4230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T02:46:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3910 0x7f16e80b4230 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T02:46:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3cf0 0x7f16e80a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T02:46:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.144+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3cf0 0x7f16e80a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T02:46:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:12.146+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3cf0 0x7f16e80a4e80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T02:46:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:13 smithi153 ceph-mon[36569]: pgmap v10768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:13 smithi104 ceph-mon[36624]: pgmap v10768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:13 smithi081 ceph-mon[31453]: pgmap v10768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:15 smithi153 ceph-mon[36569]: pgmap v10769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:15 smithi104 ceph-mon[36624]: pgmap v10769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:15 smithi081 ceph-mon[31453]: pgmap v10769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:15.969+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:46:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:15.969+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:46:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:15.970+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 154 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T02:46:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:15.970+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 154 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T02:46:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:15.970+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:46:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:15.970+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:46:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:15.971+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 155 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:46:15.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:15.971+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 155 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036c0 con 0x7f1704076980 2024-04-13T02:46:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:17 smithi153 ceph-mon[36569]: pgmap v10770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:17 smithi104 ceph-mon[36624]: pgmap v10770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:17 smithi081 ceph-mon[31453]: pgmap v10770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:19 smithi153 ceph-mon[36569]: pgmap v10771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:19 smithi104 ceph-mon[36624]: pgmap v10771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:19.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:19 smithi081 ceph-mon[31453]: pgmap v10771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:21.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:21 smithi081 ceph-mon[31453]: pgmap v10772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:21 smithi153 ceph-mon[36569]: pgmap v10772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:21 smithi104 ceph-mon[36624]: pgmap v10772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:22.945+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:46:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:22.945+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:46:22.947 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:22.945+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:46:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:23 smithi153 ceph-mon[36569]: pgmap v10773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:23 smithi104 ceph-mon[36624]: pgmap v10773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:23 smithi081 ceph-mon[31453]: pgmap v10773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:25 smithi153 ceph-mon[36569]: pgmap v10774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:25 smithi104 ceph-mon[36624]: pgmap v10774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:25 smithi081 ceph-mon[31453]: pgmap v10774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:27 smithi153 ceph-mon[36569]: pgmap v10775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:27 smithi104 ceph-mon[36624]: pgmap v10775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:27 smithi081 ceph-mon[31453]: pgmap v10775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:29 smithi153 ceph-mon[36569]: pgmap v10776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:29 smithi104 ceph-mon[36624]: pgmap v10776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:29 smithi081 ceph-mon[31453]: pgmap v10776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:31 smithi153 ceph-mon[36569]: pgmap v10777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:31 smithi104 ceph-mon[36624]: pgmap v10777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:31 smithi081 ceph-mon[31453]: pgmap v10777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:33 smithi153 ceph-mon[36569]: pgmap v10778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:33 smithi104 ceph-mon[36624]: pgmap v10778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:33 smithi081 ceph-mon[31453]: pgmap v10778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:35 smithi081 ceph-mon[31453]: pgmap v10779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:35 smithi153 ceph-mon[36569]: pgmap v10779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:35 smithi104 ceph-mon[36624]: pgmap v10779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:37 smithi081 ceph-mon[31453]: pgmap v10780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:37.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:46:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:37.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:46:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:37.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:46:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:37 smithi153 ceph-mon[36569]: pgmap v10780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:37 smithi104 ceph-mon[36624]: pgmap v10780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:39 smithi153 ceph-mon[36569]: pgmap v10781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:39 smithi104 ceph-mon[36624]: pgmap v10781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:39 smithi081 ceph-mon[31453]: pgmap v10781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:40 smithi153 ceph-mon[36569]: pgmap v10782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:40 smithi104 ceph-mon[36624]: pgmap v10782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:40 smithi081 ceph-mon[31453]: pgmap v10782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:42 smithi081 ceph-mon[31453]: pgmap v10783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:42 smithi153 ceph-mon[36569]: pgmap v10783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:42 smithi104 ceph-mon[36624]: pgmap v10783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:44 smithi081 ceph-mon[31453]: pgmap v10784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:44 smithi153 ceph-mon[36569]: pgmap v10784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:44 smithi104 ceph-mon[36624]: pgmap v10784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:46 smithi081 ceph-mon[31453]: pgmap v10785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:46 smithi153 ceph-mon[36569]: pgmap v10785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:46 smithi104 ceph-mon[36624]: pgmap v10785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:48 smithi153 ceph-mon[36569]: pgmap v10786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:48 smithi104 ceph-mon[36624]: pgmap v10786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:48 smithi081 ceph-mon[31453]: pgmap v10786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:51.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:50 smithi081 ceph-mon[31453]: pgmap v10787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:50 smithi153 ceph-mon[36569]: pgmap v10787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:50 smithi104 ceph-mon[36624]: pgmap v10787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:52.963+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:46:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:52.963+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:46:52.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:46:52.963+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:46:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:52 smithi153 ceph-mon[36569]: pgmap v10788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:52 smithi104 ceph-mon[36624]: pgmap v10788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:52 smithi081 ceph-mon[31453]: pgmap v10788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:54 smithi153 ceph-mon[36569]: pgmap v10789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:54 smithi104 ceph-mon[36624]: pgmap v10789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:54 smithi081 ceph-mon[31453]: pgmap v10789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:56 smithi153 ceph-mon[36569]: pgmap v10790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:56 smithi104 ceph-mon[36624]: pgmap v10790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:56 smithi081 ceph-mon[31453]: pgmap v10790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:46:58 smithi153 ceph-mon[36569]: pgmap v10791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:46:58 smithi104 ceph-mon[36624]: pgmap v10791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:46:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:46:58 smithi081 ceph-mon[31453]: pgmap v10791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:00 smithi153 ceph-mon[36569]: pgmap v10792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:00 smithi104 ceph-mon[36624]: pgmap v10792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:00 smithi081 ceph-mon[31453]: pgmap v10792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:02 smithi153 ceph-mon[36569]: pgmap v10793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:02 smithi104 ceph-mon[36624]: pgmap v10793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:02 smithi081 ceph-mon[31453]: pgmap v10793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:04 smithi153 ceph-mon[36569]: pgmap v10794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:04 smithi104 ceph-mon[36624]: pgmap v10794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:04 smithi081 ceph-mon[31453]: pgmap v10794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:06 smithi153 ceph-mon[36569]: pgmap v10795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:06 smithi104 ceph-mon[36624]: pgmap v10795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:06 smithi081 ceph-mon[31453]: pgmap v10795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:07.979+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:47:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:07.979+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:47:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:07.979+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:47:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:08 smithi153 ceph-mon[36569]: pgmap v10796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:47:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:08 smithi104 ceph-mon[36624]: pgmap v10796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:47:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:08 smithi081 ceph-mon[31453]: pgmap v10796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:47:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:47:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:47:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:09 smithi153 ceph-mon[36569]: pgmap v10797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:09 smithi153 ceph-mon[36569]: pgmap v10798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:47:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:47:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:47:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:09 smithi104 ceph-mon[36624]: pgmap v10797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:09 smithi104 ceph-mon[36624]: pgmap v10798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:47:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:47:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:47:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:09 smithi081 ceph-mon[31453]: pgmap v10797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:09 smithi081 ceph-mon[31453]: pgmap v10798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:47:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:12 smithi153 ceph-mon[36569]: pgmap v10799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:12 smithi104 ceph-mon[36624]: pgmap v10799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:12 smithi081 ceph-mon[31453]: pgmap v10799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:14 smithi153 ceph-mon[36569]: pgmap v10800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:14 smithi104 ceph-mon[36624]: pgmap v10800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:14 smithi081 ceph-mon[31453]: pgmap v10800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:16 smithi153 ceph-mon[36569]: pgmap v10801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:16 smithi104 ceph-mon[36624]: pgmap v10801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:16 smithi081 ceph-mon[31453]: pgmap v10801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:18 smithi153 ceph-mon[36569]: pgmap v10802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:18 smithi104 ceph-mon[36624]: pgmap v10802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:18 smithi081 ceph-mon[31453]: pgmap v10802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:20 smithi153 ceph-mon[36569]: pgmap v10803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:20 smithi104 ceph-mon[36624]: pgmap v10803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:20 smithi081 ceph-mon[31453]: pgmap v10803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:22 smithi153 ceph-mon[36569]: pgmap v10804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:22 smithi104 ceph-mon[36624]: pgmap v10804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:22 smithi081 ceph-mon[31453]: pgmap v10804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:22.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:22.981+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:47:22.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:22.981+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:47:22.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:22.981+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:47:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:24 smithi153 ceph-mon[36569]: pgmap v10805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:24.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:24 smithi104 ceph-mon[36624]: pgmap v10805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:24.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:24 smithi081 ceph-mon[31453]: pgmap v10805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:26 smithi153 ceph-mon[36569]: pgmap v10806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:26 smithi104 ceph-mon[36624]: pgmap v10806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:26 smithi081 ceph-mon[31453]: pgmap v10806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:28 smithi153 ceph-mon[36569]: pgmap v10807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:28 smithi104 ceph-mon[36624]: pgmap v10807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:28 smithi081 ceph-mon[31453]: pgmap v10807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:30 smithi153 ceph-mon[36569]: pgmap v10808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:30 smithi104 ceph-mon[36624]: pgmap v10808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:30 smithi081 ceph-mon[31453]: pgmap v10808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:32 smithi153 ceph-mon[36569]: pgmap v10809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:32 smithi104 ceph-mon[36624]: pgmap v10809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:32 smithi081 ceph-mon[31453]: pgmap v10809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:34 smithi153 ceph-mon[36569]: pgmap v10810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:34 smithi104 ceph-mon[36624]: pgmap v10810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:34 smithi081 ceph-mon[31453]: pgmap v10810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:36 smithi104 ceph-mon[36624]: pgmap v10811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:36 smithi081 ceph-mon[31453]: pgmap v10811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:36 smithi153 ceph-mon[36569]: pgmap v10811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:37.994+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:47:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:37.994+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:47:37.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:37.994+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:47:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:38 smithi104 ceph-mon[36624]: pgmap v10812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:38 smithi081 ceph-mon[31453]: pgmap v10812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:38 smithi153 ceph-mon[36569]: pgmap v10812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:40 smithi104 ceph-mon[36624]: pgmap v10813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:40 smithi081 ceph-mon[31453]: pgmap v10813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:40 smithi153 ceph-mon[36569]: pgmap v10813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:42 smithi104 ceph-mon[36624]: pgmap v10814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:42 smithi081 ceph-mon[31453]: pgmap v10814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:42 smithi153 ceph-mon[36569]: pgmap v10814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:44.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:44 smithi104 ceph-mon[36624]: pgmap v10815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:44.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:44 smithi153 ceph-mon[36569]: pgmap v10815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:44 smithi081 ceph-mon[31453]: pgmap v10815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:46 smithi081 ceph-mon[31453]: pgmap v10816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:46 smithi153 ceph-mon[36569]: pgmap v10816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:46 smithi104 ceph-mon[36624]: pgmap v10816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:48 smithi081 ceph-mon[31453]: pgmap v10817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:48 smithi153 ceph-mon[36569]: pgmap v10817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:48 smithi104 ceph-mon[36624]: pgmap v10817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:50 smithi081 ceph-mon[31453]: pgmap v10818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:50 smithi153 ceph-mon[36569]: pgmap v10818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:50 smithi104 ceph-mon[36624]: pgmap v10818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:52 smithi081 ceph-mon[31453]: pgmap v10819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:53.010+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:47:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:53.010+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:47:53.011 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:47:53.010+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:47:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:52 smithi153 ceph-mon[36569]: pgmap v10819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:52 smithi104 ceph-mon[36624]: pgmap v10819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:54 smithi081 ceph-mon[31453]: pgmap v10820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:54 smithi153 ceph-mon[36569]: pgmap v10820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:54 smithi104 ceph-mon[36624]: pgmap v10820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:56.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:56 smithi104 ceph-mon[36624]: pgmap v10821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:56 smithi081 ceph-mon[31453]: pgmap v10821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:56 smithi153 ceph-mon[36569]: pgmap v10821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:47:58 smithi081 ceph-mon[31453]: pgmap v10822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:47:58 smithi153 ceph-mon[36569]: pgmap v10822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:47:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:47:58 smithi104 ceph-mon[36624]: pgmap v10822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:00 smithi081 ceph-mon[31453]: pgmap v10823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:00 smithi153 ceph-mon[36569]: pgmap v10823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:00 smithi104 ceph-mon[36624]: pgmap v10823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:02 smithi081 ceph-mon[31453]: pgmap v10824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:02 smithi153 ceph-mon[36569]: pgmap v10824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:02 smithi104 ceph-mon[36624]: pgmap v10824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:04.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:04 smithi081 ceph-mon[31453]: pgmap v10825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:04 smithi153 ceph-mon[36569]: pgmap v10825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:04 smithi104 ceph-mon[36624]: pgmap v10825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:06 smithi081 ceph-mon[31453]: pgmap v10826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:06 smithi153 ceph-mon[36569]: pgmap v10826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:06 smithi104 ceph-mon[36624]: pgmap v10826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:08.011+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:48:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:08.011+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:48:08.013 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:08.011+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:48:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:08 smithi081 ceph-mon[31453]: pgmap v10827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:08 smithi153 ceph-mon[36569]: pgmap v10827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:08 smithi104 ceph-mon[36624]: pgmap v10827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:10 smithi081 ceph-mon[31453]: pgmap v10828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:48:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:10 smithi153 ceph-mon[36569]: pgmap v10828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:48:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:10 smithi104 ceph-mon[36624]: pgmap v10828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:48:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:48:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:48:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:48:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:48:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:48:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:48:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:48:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:48:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:48:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:12 smithi153 ceph-mon[36569]: pgmap v10829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:12 smithi153 ceph-mon[36569]: pgmap v10830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:12 smithi104 ceph-mon[36624]: pgmap v10829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:12 smithi104 ceph-mon[36624]: pgmap v10830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:12 smithi081 ceph-mon[31453]: pgmap v10829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:12 smithi081 ceph-mon[31453]: pgmap v10830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:14 smithi153 ceph-mon[36569]: pgmap v10831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:14 smithi104 ceph-mon[36624]: pgmap v10831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:14 smithi081 ceph-mon[31453]: pgmap v10831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:16 smithi153 ceph-mon[36569]: pgmap v10832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:16 smithi104 ceph-mon[36624]: pgmap v10832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:16 smithi081 ceph-mon[31453]: pgmap v10832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:17 smithi153 ceph-mon[36569]: pgmap v10833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:17 smithi104 ceph-mon[36624]: pgmap v10833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:17 smithi081 ceph-mon[31453]: pgmap v10833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:20 smithi153 ceph-mon[36569]: pgmap v10834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:20 smithi104 ceph-mon[36624]: pgmap v10834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:20 smithi081 ceph-mon[31453]: pgmap v10834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:22 smithi153 ceph-mon[36569]: pgmap v10835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:22 smithi104 ceph-mon[36624]: pgmap v10835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:22 smithi081 ceph-mon[31453]: pgmap v10835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:23.025+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:48:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:23.025+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:48:23.026 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:23.025+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:48:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:24 smithi153 ceph-mon[36569]: pgmap v10836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:24 smithi104 ceph-mon[36624]: pgmap v10836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:24 smithi081 ceph-mon[31453]: pgmap v10836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:25 smithi153 ceph-mon[36569]: pgmap v10837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:25 smithi104 ceph-mon[36624]: pgmap v10837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:25 smithi081 ceph-mon[31453]: pgmap v10837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:28 smithi153 ceph-mon[36569]: pgmap v10838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:28 smithi104 ceph-mon[36624]: pgmap v10838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:28 smithi081 ceph-mon[31453]: pgmap v10838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:30 smithi153 ceph-mon[36569]: pgmap v10839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:30 smithi104 ceph-mon[36624]: pgmap v10839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:30 smithi081 ceph-mon[31453]: pgmap v10839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:32 smithi153 ceph-mon[36569]: pgmap v10840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:32 smithi104 ceph-mon[36624]: pgmap v10840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:32 smithi081 ceph-mon[31453]: pgmap v10840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:34 smithi153 ceph-mon[36569]: pgmap v10841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:34 smithi104 ceph-mon[36624]: pgmap v10841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:34 smithi081 ceph-mon[31453]: pgmap v10841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:36 smithi153 ceph-mon[36569]: pgmap v10842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:36 smithi104 ceph-mon[36624]: pgmap v10842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:36 smithi081 ceph-mon[31453]: pgmap v10842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:38.041+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:48:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:38.041+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:48:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:38.041+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:48:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:38 smithi153 ceph-mon[36569]: pgmap v10843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:38 smithi104 ceph-mon[36624]: pgmap v10843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:38 smithi081 ceph-mon[31453]: pgmap v10843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:40 smithi153 ceph-mon[36569]: pgmap v10844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:40 smithi104 ceph-mon[36624]: pgmap v10844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:40 smithi081 ceph-mon[31453]: pgmap v10844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:42 smithi153 ceph-mon[36569]: pgmap v10845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:42 smithi104 ceph-mon[36624]: pgmap v10845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:42 smithi081 ceph-mon[31453]: pgmap v10845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:44 smithi104 ceph-mon[36624]: pgmap v10846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:44 smithi081 ceph-mon[31453]: pgmap v10846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:44 smithi153 ceph-mon[36569]: pgmap v10846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:46 smithi104 ceph-mon[36624]: pgmap v10847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:46 smithi081 ceph-mon[31453]: pgmap v10847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:46 smithi153 ceph-mon[36569]: pgmap v10847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:48 smithi104 ceph-mon[36624]: pgmap v10848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:48 smithi081 ceph-mon[31453]: pgmap v10848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:48 smithi153 ceph-mon[36569]: pgmap v10848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:50 smithi104 ceph-mon[36624]: pgmap v10849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:50 smithi081 ceph-mon[31453]: pgmap v10849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:50 smithi153 ceph-mon[36569]: pgmap v10849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:52 smithi104 ceph-mon[36624]: pgmap v10850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:52 smithi081 ceph-mon[31453]: pgmap v10850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:52 smithi153 ceph-mon[36569]: pgmap v10850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:53.057 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:53.057+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:48:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:53.057+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:48:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:48:53.057+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:48:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:54 smithi081 ceph-mon[31453]: pgmap v10851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:54 smithi153 ceph-mon[36569]: pgmap v10851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:54 smithi104 ceph-mon[36624]: pgmap v10851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:56.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:56 smithi104 ceph-mon[36624]: pgmap v10852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:56 smithi081 ceph-mon[31453]: pgmap v10852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:56 smithi153 ceph-mon[36569]: pgmap v10852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:48:58 smithi081 ceph-mon[31453]: pgmap v10853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:48:58 smithi153 ceph-mon[36569]: pgmap v10853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:48:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:48:58 smithi104 ceph-mon[36624]: pgmap v10853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:00 smithi081 ceph-mon[31453]: pgmap v10854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:00 smithi153 ceph-mon[36569]: pgmap v10854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:00 smithi104 ceph-mon[36624]: pgmap v10854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:02 smithi081 ceph-mon[31453]: pgmap v10855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:02 smithi153 ceph-mon[36569]: pgmap v10855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:02 smithi104 ceph-mon[36624]: pgmap v10855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:04 smithi081 ceph-mon[31453]: pgmap v10856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:04 smithi153 ceph-mon[36569]: pgmap v10856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:04 smithi104 ceph-mon[36624]: pgmap v10856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:06 smithi081 ceph-mon[31453]: pgmap v10857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:06 smithi153 ceph-mon[36569]: pgmap v10857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:06 smithi104 ceph-mon[36624]: pgmap v10857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:08.072+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:49:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:08.072+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:49:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:08.072+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:49:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:08 smithi081 ceph-mon[31453]: pgmap v10858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:08 smithi153 ceph-mon[36569]: pgmap v10858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:08 smithi104 ceph-mon[36624]: pgmap v10858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:10 smithi081 ceph-mon[31453]: pgmap v10859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:10 smithi153 ceph-mon[36569]: pgmap v10859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:10 smithi104 ceph-mon[36624]: pgmap v10859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:12 smithi081 ceph-mon[31453]: pgmap v10860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:49:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:12 smithi153 ceph-mon[36569]: pgmap v10860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:49:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:12 smithi104 ceph-mon[36624]: pgmap v10860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:49:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:14 smithi081 ceph-mon[31453]: pgmap v10861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:14 smithi153 ceph-mon[36569]: pgmap v10861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:14 smithi104 ceph-mon[36624]: pgmap v10861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:15.999+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:49:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:15.999+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:49:16.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:15.999+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 156 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:49:16.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:15.999+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 156 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:49:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:16 smithi153 ceph-mon[36569]: pgmap v10862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:16 smithi104 ceph-mon[36624]: pgmap v10862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:16 smithi081 ceph-mon[31453]: pgmap v10862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:18.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:18 smithi081 ceph-mon[31453]: pgmap v10863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:18.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:49:18.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:49:18.993 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:18 smithi153 ceph-mon[36569]: pgmap v10863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:49:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:49:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:18 smithi104 ceph-mon[36624]: pgmap v10863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:49:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:49:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:49:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:19 smithi081 ceph-mon[31453]: pgmap v10864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:19 smithi081 ceph-mon[31453]: pgmap v10865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:19 smithi153 ceph-mon[36569]: pgmap v10864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:19 smithi153 ceph-mon[36569]: pgmap v10865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:19 smithi104 ceph-mon[36624]: pgmap v10864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:19 smithi104 ceph-mon[36624]: pgmap v10865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:21 smithi081 ceph-mon[31453]: pgmap v10866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:21 smithi153 ceph-mon[36569]: pgmap v10866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:21 smithi104 ceph-mon[36624]: pgmap v10866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:23.079+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:49:23.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:23.079+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:49:23.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:23.079+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:49:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:23 smithi081 ceph-mon[31453]: pgmap v10867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:23 smithi153 ceph-mon[36569]: pgmap v10867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:23 smithi104 ceph-mon[36624]: pgmap v10867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:25 smithi081 ceph-mon[31453]: pgmap v10868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:25 smithi153 ceph-mon[36569]: pgmap v10868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:25 smithi104 ceph-mon[36624]: pgmap v10868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:27.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:27 smithi104 ceph-mon[36624]: pgmap v10869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:27 smithi081 ceph-mon[31453]: pgmap v10869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:27 smithi153 ceph-mon[36569]: pgmap v10869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:29 smithi104 ceph-mon[36624]: pgmap v10870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:29 smithi081 ceph-mon[31453]: pgmap v10870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:29 smithi153 ceph-mon[36569]: pgmap v10870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:31 smithi081 ceph-mon[31453]: pgmap v10871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:31 smithi153 ceph-mon[36569]: pgmap v10871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:31 smithi104 ceph-mon[36624]: pgmap v10871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:33 smithi081 ceph-mon[31453]: pgmap v10872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:33 smithi153 ceph-mon[36569]: pgmap v10872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:33 smithi104 ceph-mon[36624]: pgmap v10872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:35 smithi081 ceph-mon[31453]: pgmap v10873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:35 smithi153 ceph-mon[36569]: pgmap v10873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:35 smithi104 ceph-mon[36624]: pgmap v10873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:37 smithi081 ceph-mon[31453]: pgmap v10874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:37 smithi153 ceph-mon[36569]: pgmap v10874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:37 smithi104 ceph-mon[36624]: pgmap v10874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:38.089+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:49:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:38.089+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:49:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:38.089+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:49:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:39 smithi104 ceph-mon[36624]: pgmap v10875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:39 smithi081 ceph-mon[31453]: pgmap v10875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:39 smithi153 ceph-mon[36569]: pgmap v10875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:41 smithi081 ceph-mon[31453]: pgmap v10876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:41 smithi153 ceph-mon[36569]: pgmap v10876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:41 smithi104 ceph-mon[36624]: pgmap v10876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:43 smithi153 ceph-mon[36569]: pgmap v10877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:43 smithi104 ceph-mon[36624]: pgmap v10877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:43 smithi081 ceph-mon[31453]: pgmap v10877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:45 smithi153 ceph-mon[36569]: pgmap v10878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:45 smithi104 ceph-mon[36624]: pgmap v10878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:45 smithi081 ceph-mon[31453]: pgmap v10878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:47 smithi153 ceph-mon[36569]: pgmap v10879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:47 smithi104 ceph-mon[36624]: pgmap v10879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:47 smithi081 ceph-mon[31453]: pgmap v10879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:49 smithi153 ceph-mon[36569]: pgmap v10880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:49 smithi104 ceph-mon[36624]: pgmap v10880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:49 smithi081 ceph-mon[31453]: pgmap v10880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:51 smithi153 ceph-mon[36569]: pgmap v10881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:51 smithi104 ceph-mon[36624]: pgmap v10881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:51 smithi081 ceph-mon[31453]: pgmap v10881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:53.103+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:49:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:53.103+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:49:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:49:53.103+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:49:53.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:53 smithi081 ceph-mon[31453]: pgmap v10882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:53 smithi153 ceph-mon[36569]: pgmap v10882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:53 smithi104 ceph-mon[36624]: pgmap v10882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:55 smithi153 ceph-mon[36569]: pgmap v10883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:55 smithi104 ceph-mon[36624]: pgmap v10883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:55 smithi081 ceph-mon[31453]: pgmap v10883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:57 smithi153 ceph-mon[36569]: pgmap v10884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:57 smithi104 ceph-mon[36624]: pgmap v10884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:57 smithi081 ceph-mon[31453]: pgmap v10884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:49:58 smithi153 ceph-mon[36569]: pgmap v10885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:49:58 smithi104 ceph-mon[36624]: pgmap v10885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:49:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:49:58 smithi081 ceph-mon[31453]: pgmap v10885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:00 smithi081 ceph-mon[31453]: pgmap v10886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:50:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:00 smithi153 ceph-mon[36569]: pgmap v10886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:50:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:00 smithi104 ceph-mon[36624]: pgmap v10886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T02:50:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:02 smithi081 ceph-mon[31453]: pgmap v10887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:02 smithi153 ceph-mon[36569]: pgmap v10887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:02 smithi104 ceph-mon[36624]: pgmap v10887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:04 smithi153 ceph-mon[36569]: pgmap v10888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:04 smithi104 ceph-mon[36624]: pgmap v10888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:04 smithi081 ceph-mon[31453]: pgmap v10888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:06 smithi153 ceph-mon[36569]: pgmap v10889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:06 smithi104 ceph-mon[36624]: pgmap v10889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:06 smithi081 ceph-mon[31453]: pgmap v10889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:08.118+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:50:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:08.118+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:50:08.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:08.118+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:50:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:08 smithi153 ceph-mon[36569]: pgmap v10890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:08 smithi104 ceph-mon[36624]: pgmap v10890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:08 smithi081 ceph-mon[31453]: pgmap v10890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:11.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:10 smithi104 ceph-mon[36624]: pgmap v10891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:10 smithi153 ceph-mon[36569]: pgmap v10891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:10 smithi081 ceph-mon[31453]: pgmap v10891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:12 smithi153 ceph-mon[36569]: pgmap v10892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:12 smithi104 ceph-mon[36624]: pgmap v10892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:12 smithi081 ceph-mon[31453]: pgmap v10892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:14 smithi153 ceph-mon[36569]: pgmap v10893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:14 smithi104 ceph-mon[36624]: pgmap v10893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:14 smithi081 ceph-mon[31453]: pgmap v10893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:16 smithi153 ceph-mon[36569]: pgmap v10894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:16 smithi104 ceph-mon[36624]: pgmap v10894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:16 smithi081 ceph-mon[31453]: pgmap v10894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:18 smithi153 ceph-mon[36569]: pgmap v10895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:50:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:18 smithi104 ceph-mon[36624]: pgmap v10895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:50:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:18 smithi081 ceph-mon[31453]: pgmap v10895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:50:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:50:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:50:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:19 smithi153 ceph-mon[36569]: pgmap v10896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:19 smithi153 ceph-mon[36569]: pgmap v10897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:50:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:50:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:19 smithi104 ceph-mon[36624]: pgmap v10896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:19 smithi104 ceph-mon[36624]: pgmap v10897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:50:20.523 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:50:20.523 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:19 smithi081 ceph-mon[31453]: pgmap v10896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:20.523 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:19 smithi081 ceph-mon[31453]: pgmap v10897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:20.524 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:50:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:22 smithi153 ceph-mon[36569]: pgmap v10898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:22 smithi104 ceph-mon[36624]: pgmap v10898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:22 smithi081 ceph-mon[31453]: pgmap v10898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:23.134+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:50:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:23.134+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:50:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:23.134+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:50:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:24 smithi153 ceph-mon[36569]: pgmap v10899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:24 smithi104 ceph-mon[36624]: pgmap v10899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:24.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:24 smithi081 ceph-mon[31453]: pgmap v10899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:26 smithi153 ceph-mon[36569]: pgmap v10900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:26 smithi104 ceph-mon[36624]: pgmap v10900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:26 smithi081 ceph-mon[31453]: pgmap v10900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:28 smithi153 ceph-mon[36569]: pgmap v10901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:28 smithi104 ceph-mon[36624]: pgmap v10901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:28 smithi081 ceph-mon[31453]: pgmap v10901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:30 smithi153 ceph-mon[36569]: pgmap v10902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:30 smithi104 ceph-mon[36624]: pgmap v10902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:30 smithi081 ceph-mon[31453]: pgmap v10902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:32 smithi153 ceph-mon[36569]: pgmap v10903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:32 smithi104 ceph-mon[36624]: pgmap v10903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:32 smithi081 ceph-mon[31453]: pgmap v10903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:34 smithi153 ceph-mon[36569]: pgmap v10904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:34 smithi104 ceph-mon[36624]: pgmap v10904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:34 smithi081 ceph-mon[31453]: pgmap v10904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:35 smithi153 ceph-mon[36569]: pgmap v10905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:35 smithi104 ceph-mon[36624]: pgmap v10905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:35 smithi081 ceph-mon[31453]: pgmap v10905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:38.141+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:50:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:38.141+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:50:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:38.141+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:50:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:38 smithi153 ceph-mon[36569]: pgmap v10906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:38 smithi104 ceph-mon[36624]: pgmap v10906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:38 smithi081 ceph-mon[31453]: pgmap v10906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:40 smithi153 ceph-mon[36569]: pgmap v10907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:40 smithi104 ceph-mon[36624]: pgmap v10907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:40 smithi081 ceph-mon[31453]: pgmap v10907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:42 smithi153 ceph-mon[36569]: pgmap v10908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:42 smithi104 ceph-mon[36624]: pgmap v10908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:42 smithi081 ceph-mon[31453]: pgmap v10908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:44 smithi153 ceph-mon[36569]: pgmap v10909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:44 smithi104 ceph-mon[36624]: pgmap v10909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:44 smithi081 ceph-mon[31453]: pgmap v10909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:46 smithi153 ceph-mon[36569]: pgmap v10910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:46 smithi104 ceph-mon[36624]: pgmap v10910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:46 smithi081 ceph-mon[31453]: pgmap v10910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:48 smithi153 ceph-mon[36569]: pgmap v10911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:48 smithi104 ceph-mon[36624]: pgmap v10911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:48 smithi081 ceph-mon[31453]: pgmap v10911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:50 smithi153 ceph-mon[36569]: pgmap v10912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:50 smithi104 ceph-mon[36624]: pgmap v10912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:50 smithi081 ceph-mon[31453]: pgmap v10912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:52 smithi104 ceph-mon[36624]: pgmap v10913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:52 smithi081 ceph-mon[31453]: pgmap v10913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:52 smithi153 ceph-mon[36569]: pgmap v10913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:53.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:53.152+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:50:53.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:53.152+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:50:53.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:50:53.152+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:50:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:54 smithi104 ceph-mon[36624]: pgmap v10914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:54 smithi081 ceph-mon[31453]: pgmap v10914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:54 smithi153 ceph-mon[36569]: pgmap v10914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:56 smithi104 ceph-mon[36624]: pgmap v10915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:56 smithi081 ceph-mon[31453]: pgmap v10915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:56 smithi153 ceph-mon[36569]: pgmap v10915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:50:58 smithi104 ceph-mon[36624]: pgmap v10916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:50:58 smithi081 ceph-mon[31453]: pgmap v10916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:50:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:50:58 smithi153 ceph-mon[36569]: pgmap v10916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:00 smithi104 ceph-mon[36624]: pgmap v10917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:00 smithi081 ceph-mon[31453]: pgmap v10917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:00 smithi153 ceph-mon[36569]: pgmap v10917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:02 smithi081 ceph-mon[31453]: pgmap v10918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:02 smithi153 ceph-mon[36569]: pgmap v10918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:02 smithi104 ceph-mon[36624]: pgmap v10918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:04.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:04 smithi081 ceph-mon[31453]: pgmap v10919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:04 smithi153 ceph-mon[36569]: pgmap v10919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:04 smithi104 ceph-mon[36624]: pgmap v10919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:06 smithi081 ceph-mon[31453]: pgmap v10920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:06 smithi153 ceph-mon[36569]: pgmap v10920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:06 smithi104 ceph-mon[36624]: pgmap v10920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:08.162+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:51:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:08.162+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:51:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:08.162+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:51:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:08 smithi081 ceph-mon[31453]: pgmap v10921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:08 smithi153 ceph-mon[36569]: pgmap v10921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:08 smithi104 ceph-mon[36624]: pgmap v10921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:10 smithi081 ceph-mon[31453]: pgmap v10922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:10 smithi153 ceph-mon[36569]: pgmap v10922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:10 smithi104 ceph-mon[36624]: pgmap v10922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:12 smithi081 ceph-mon[31453]: pgmap v10923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:12 smithi153 ceph-mon[36569]: pgmap v10923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:12 smithi104 ceph-mon[36624]: pgmap v10923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:14 smithi081 ceph-mon[31453]: pgmap v10924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:14 smithi153 ceph-mon[36569]: pgmap v10924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:14 smithi104 ceph-mon[36624]: pgmap v10924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:16 smithi081 ceph-mon[31453]: pgmap v10925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:16 smithi153 ceph-mon[36569]: pgmap v10925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:16 smithi104 ceph-mon[36624]: pgmap v10925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:18 smithi081 ceph-mon[31453]: pgmap v10926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:18 smithi153 ceph-mon[36569]: pgmap v10926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:18 smithi104 ceph-mon[36624]: pgmap v10926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:20 smithi081 ceph-mon[31453]: pgmap v10927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:51:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:20 smithi153 ceph-mon[36569]: pgmap v10927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:51:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:20 smithi104 ceph-mon[36624]: pgmap v10927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:51:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:51:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:51:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:51:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:51:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:51:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:51:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:51:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:51:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:51:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:22 smithi081 ceph-mon[31453]: pgmap v10928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:22 smithi081 ceph-mon[31453]: pgmap v10929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:22 smithi153 ceph-mon[36569]: pgmap v10928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:22 smithi153 ceph-mon[36569]: pgmap v10929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:22 smithi104 ceph-mon[36624]: pgmap v10928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:22 smithi104 ceph-mon[36624]: pgmap v10929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:23.176+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:51:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:23.176+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:51:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:23.176+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:51:24.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:24 smithi081 ceph-mon[31453]: pgmap v10930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:24 smithi153 ceph-mon[36569]: pgmap v10930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:24 smithi104 ceph-mon[36624]: pgmap v10930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:26 smithi081 ceph-mon[31453]: pgmap v10931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:26 smithi153 ceph-mon[36569]: pgmap v10931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:26 smithi104 ceph-mon[36624]: pgmap v10931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:28 smithi153 ceph-mon[36569]: pgmap v10932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:28 smithi104 ceph-mon[36624]: pgmap v10932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:28 smithi081 ceph-mon[31453]: pgmap v10932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:30 smithi153 ceph-mon[36569]: pgmap v10933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:30 smithi104 ceph-mon[36624]: pgmap v10933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:30 smithi081 ceph-mon[31453]: pgmap v10933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:32 smithi153 ceph-mon[36569]: pgmap v10934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:32 smithi104 ceph-mon[36624]: pgmap v10934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:32 smithi081 ceph-mon[31453]: pgmap v10934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:34 smithi153 ceph-mon[36569]: pgmap v10935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:34 smithi104 ceph-mon[36624]: pgmap v10935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:34 smithi081 ceph-mon[31453]: pgmap v10935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:36 smithi153 ceph-mon[36569]: pgmap v10936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:36 smithi104 ceph-mon[36624]: pgmap v10936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:36 smithi081 ceph-mon[31453]: pgmap v10936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:38.179+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:51:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:38.179+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:51:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:38.179+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:51:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:38 smithi153 ceph-mon[36569]: pgmap v10937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:38 smithi104 ceph-mon[36624]: pgmap v10937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:38 smithi081 ceph-mon[31453]: pgmap v10937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:40 smithi153 ceph-mon[36569]: pgmap v10938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:40 smithi104 ceph-mon[36624]: pgmap v10938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:40 smithi081 ceph-mon[31453]: pgmap v10938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:42 smithi153 ceph-mon[36569]: pgmap v10939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:42 smithi104 ceph-mon[36624]: pgmap v10939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:42 smithi081 ceph-mon[31453]: pgmap v10939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:44 smithi153 ceph-mon[36569]: pgmap v10940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:44 smithi104 ceph-mon[36624]: pgmap v10940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:44 smithi081 ceph-mon[31453]: pgmap v10940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:46 smithi153 ceph-mon[36569]: pgmap v10941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:46 smithi104 ceph-mon[36624]: pgmap v10941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:46 smithi081 ceph-mon[31453]: pgmap v10941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:48 smithi153 ceph-mon[36569]: pgmap v10942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:48 smithi104 ceph-mon[36624]: pgmap v10942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:48 smithi081 ceph-mon[31453]: pgmap v10942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:50 smithi153 ceph-mon[36569]: pgmap v10943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:50 smithi104 ceph-mon[36624]: pgmap v10943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:50 smithi081 ceph-mon[31453]: pgmap v10943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:51 smithi153 ceph-mon[36569]: pgmap v10944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:51 smithi104 ceph-mon[36624]: pgmap v10944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:51 smithi081 ceph-mon[31453]: pgmap v10944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:53.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:53.193+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:51:53.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:53.193+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:51:53.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:51:53.193+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:51:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:54 smithi153 ceph-mon[36569]: pgmap v10945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:54 smithi104 ceph-mon[36624]: pgmap v10945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:54 smithi081 ceph-mon[31453]: pgmap v10945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:56 smithi153 ceph-mon[36569]: pgmap v10946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:56 smithi104 ceph-mon[36624]: pgmap v10946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:56 smithi081 ceph-mon[31453]: pgmap v10946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:51:58 smithi153 ceph-mon[36569]: pgmap v10947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:51:58 smithi104 ceph-mon[36624]: pgmap v10947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:51:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:51:58 smithi081 ceph-mon[31453]: pgmap v10947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:00 smithi153 ceph-mon[36569]: pgmap v10948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:00 smithi104 ceph-mon[36624]: pgmap v10948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:00 smithi081 ceph-mon[31453]: pgmap v10948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:02 smithi104 ceph-mon[36624]: pgmap v10949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:02 smithi081 ceph-mon[31453]: pgmap v10949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:02 smithi153 ceph-mon[36569]: pgmap v10949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:04 smithi104 ceph-mon[36624]: pgmap v10950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:04 smithi081 ceph-mon[31453]: pgmap v10950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:04 smithi153 ceph-mon[36569]: pgmap v10950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:06 smithi104 ceph-mon[36624]: pgmap v10951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:06 smithi081 ceph-mon[31453]: pgmap v10951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:06 smithi153 ceph-mon[36569]: pgmap v10951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:08.208+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:52:08.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:08.208+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:52:08.210 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:08.208+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:52:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:08 smithi104 ceph-mon[36624]: pgmap v10952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:08 smithi081 ceph-mon[31453]: pgmap v10952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:08 smithi153 ceph-mon[36569]: pgmap v10952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:10 smithi081 ceph-mon[31453]: pgmap v10953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:10 smithi153 ceph-mon[36569]: pgmap v10953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:10 smithi104 ceph-mon[36624]: pgmap v10953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:12 smithi081 ceph-mon[31453]: pgmap v10954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:12 smithi153 ceph-mon[36569]: pgmap v10954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:12 smithi104 ceph-mon[36624]: pgmap v10954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:14 smithi081 ceph-mon[31453]: pgmap v10955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:14 smithi153 ceph-mon[36569]: pgmap v10955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:14 smithi104 ceph-mon[36624]: pgmap v10955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:16.029+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:52:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:16.029+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:52:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:16.029+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 157 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:52:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:16.029+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 157 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:52:16.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:16 smithi153 ceph-mon[36569]: pgmap v10956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:16 smithi081 ceph-mon[31453]: pgmap v10956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:16 smithi104 ceph-mon[36624]: pgmap v10956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:18 smithi081 ceph-mon[31453]: pgmap v10957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:18 smithi153 ceph-mon[36569]: pgmap v10957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:18 smithi104 ceph-mon[36624]: pgmap v10957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:20 smithi081 ceph-mon[31453]: pgmap v10958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:20 smithi153 ceph-mon[36569]: pgmap v10958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:20.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:20 smithi104 ceph-mon[36624]: pgmap v10958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:22 smithi081 ceph-mon[31453]: pgmap v10959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:52:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:22 smithi153 ceph-mon[36569]: pgmap v10959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:52:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:22 smithi104 ceph-mon[36624]: pgmap v10959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:22.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:52:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:23.210+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:52:23.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:23.210+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:52:23.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:23.210+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:52:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:52:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:52:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:23 smithi081 ceph-mon[31453]: pgmap v10960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:23 smithi081 ceph-mon[31453]: pgmap v10961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:52:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:52:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:52:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:23 smithi153 ceph-mon[36569]: pgmap v10960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:23 smithi153 ceph-mon[36569]: pgmap v10961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:52:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:52:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:52:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:23 smithi104 ceph-mon[36624]: pgmap v10960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:23 smithi104 ceph-mon[36624]: pgmap v10961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:52:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:25 smithi081 ceph-mon[31453]: pgmap v10962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:25 smithi153 ceph-mon[36569]: pgmap v10962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:25 smithi104 ceph-mon[36624]: pgmap v10962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:27 smithi081 ceph-mon[31453]: pgmap v10963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:27 smithi153 ceph-mon[36569]: pgmap v10963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:27 smithi104 ceph-mon[36624]: pgmap v10963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:29.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:29 smithi104 ceph-mon[36624]: pgmap v10964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:29 smithi081 ceph-mon[31453]: pgmap v10964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:29 smithi153 ceph-mon[36569]: pgmap v10964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:31 smithi081 ceph-mon[31453]: pgmap v10965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:31 smithi153 ceph-mon[36569]: pgmap v10965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:31 smithi104 ceph-mon[36624]: pgmap v10965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:33 smithi081 ceph-mon[31453]: pgmap v10966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:33 smithi153 ceph-mon[36569]: pgmap v10966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:33 smithi104 ceph-mon[36624]: pgmap v10966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:35 smithi081 ceph-mon[31453]: pgmap v10967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:35 smithi153 ceph-mon[36569]: pgmap v10967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:35 smithi104 ceph-mon[36624]: pgmap v10967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:37 smithi081 ceph-mon[31453]: pgmap v10968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:37 smithi153 ceph-mon[36569]: pgmap v10968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:37 smithi104 ceph-mon[36624]: pgmap v10968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:38.215+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:52:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:38.215+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:52:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:38.216+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:52:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:39 smithi081 ceph-mon[31453]: pgmap v10969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:39 smithi153 ceph-mon[36569]: pgmap v10969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:39 smithi104 ceph-mon[36624]: pgmap v10969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:41 smithi081 ceph-mon[31453]: pgmap v10970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:41 smithi153 ceph-mon[36569]: pgmap v10970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:41 smithi104 ceph-mon[36624]: pgmap v10970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:43 smithi081 ceph-mon[31453]: pgmap v10971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:43 smithi153 ceph-mon[36569]: pgmap v10971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:43 smithi104 ceph-mon[36624]: pgmap v10971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:45 smithi081 ceph-mon[31453]: pgmap v10972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:45 smithi153 ceph-mon[36569]: pgmap v10972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:45 smithi104 ceph-mon[36624]: pgmap v10972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:47 smithi081 ceph-mon[31453]: pgmap v10973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:47 smithi153 ceph-mon[36569]: pgmap v10973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:47 smithi104 ceph-mon[36624]: pgmap v10973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:49 smithi081 ceph-mon[31453]: pgmap v10974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:49 smithi153 ceph-mon[36569]: pgmap v10974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:49 smithi104 ceph-mon[36624]: pgmap v10974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:51 smithi153 ceph-mon[36569]: pgmap v10975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:51 smithi104 ceph-mon[36624]: pgmap v10975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:51 smithi081 ceph-mon[31453]: pgmap v10975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:53.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:53.228+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:52:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:53.228+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:52:53.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:52:53.228+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:52:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:53 smithi153 ceph-mon[36569]: pgmap v10976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:54.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:53 smithi104 ceph-mon[36624]: pgmap v10976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:53 smithi081 ceph-mon[31453]: pgmap v10976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:55.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:55 smithi081 ceph-mon[31453]: pgmap v10977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:55 smithi153 ceph-mon[36569]: pgmap v10977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:55 smithi104 ceph-mon[36624]: pgmap v10977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:57 smithi153 ceph-mon[36569]: pgmap v10978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:57 smithi104 ceph-mon[36624]: pgmap v10978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:52:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:57 smithi081 ceph-mon[31453]: pgmap v10978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:52:59 smithi153 ceph-mon[36569]: pgmap v10979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:52:59 smithi104 ceph-mon[36624]: pgmap v10979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:52:59 smithi081 ceph-mon[31453]: pgmap v10979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:01 smithi153 ceph-mon[36569]: pgmap v10980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:01 smithi104 ceph-mon[36624]: pgmap v10980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:01 smithi081 ceph-mon[31453]: pgmap v10980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:03 smithi153 ceph-mon[36569]: pgmap v10981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:03 smithi104 ceph-mon[36624]: pgmap v10981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:03 smithi081 ceph-mon[31453]: pgmap v10981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:05 smithi153 ceph-mon[36569]: pgmap v10982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:05 smithi104 ceph-mon[36624]: pgmap v10982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:05 smithi081 ceph-mon[31453]: pgmap v10982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:07 smithi153 ceph-mon[36569]: pgmap v10983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:07 smithi104 ceph-mon[36624]: pgmap v10983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:07 smithi081 ceph-mon[31453]: pgmap v10983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:08.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:08.244+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:53:08.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:08.244+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:53:08.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:08.244+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:53:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:09 smithi153 ceph-mon[36569]: pgmap v10984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:09 smithi104 ceph-mon[36624]: pgmap v10984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:09 smithi081 ceph-mon[31453]: pgmap v10984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:11 smithi153 ceph-mon[36569]: pgmap v10985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:11 smithi104 ceph-mon[36624]: pgmap v10985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:11 smithi081 ceph-mon[31453]: pgmap v10985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:13 smithi153 ceph-mon[36569]: pgmap v10986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:13 smithi104 ceph-mon[36624]: pgmap v10986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:13 smithi081 ceph-mon[31453]: pgmap v10986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:15 smithi153 ceph-mon[36569]: pgmap v10987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:15 smithi104 ceph-mon[36624]: pgmap v10987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:15 smithi081 ceph-mon[31453]: pgmap v10987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:17 smithi153 ceph-mon[36569]: pgmap v10988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:17 smithi104 ceph-mon[36624]: pgmap v10988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:17 smithi081 ceph-mon[31453]: pgmap v10988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:19 smithi153 ceph-mon[36569]: pgmap v10989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:19 smithi104 ceph-mon[36624]: pgmap v10989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:19 smithi081 ceph-mon[31453]: pgmap v10989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:21 smithi153 ceph-mon[36569]: pgmap v10990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:21 smithi104 ceph-mon[36624]: pgmap v10990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:21 smithi081 ceph-mon[31453]: pgmap v10990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:22 smithi153 ceph-mon[36569]: pgmap v10991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:22 smithi104 ceph-mon[36624]: pgmap v10991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:22 smithi081 ceph-mon[31453]: pgmap v10991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:23.259+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:23.259+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:53:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:23.259+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:53:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:53:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:53:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:53:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:53:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:53:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:24 smithi153 ceph-mon[36569]: pgmap v10992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:24 smithi153 ceph-mon[36569]: pgmap v10993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:53:25.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:53:25.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:53:25.059 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:24 smithi081 ceph-mon[31453]: pgmap v10992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:25.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:24 smithi081 ceph-mon[31453]: pgmap v10993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:25.060 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:53:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:53:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:53:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:24 smithi104 ceph-mon[36624]: pgmap v10992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:24 smithi104 ceph-mon[36624]: pgmap v10993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:53:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:27 smithi153 ceph-mon[36569]: pgmap v10994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:27 smithi104 ceph-mon[36624]: pgmap v10994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:27 smithi081 ceph-mon[31453]: pgmap v10994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:29 smithi153 ceph-mon[36569]: pgmap v10995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:29 smithi104 ceph-mon[36624]: pgmap v10995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:29 smithi081 ceph-mon[31453]: pgmap v10995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:31 smithi153 ceph-mon[36569]: pgmap v10996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:31 smithi104 ceph-mon[36624]: pgmap v10996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:31 smithi081 ceph-mon[31453]: pgmap v10996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:33 smithi153 ceph-mon[36569]: pgmap v10997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:33 smithi104 ceph-mon[36624]: pgmap v10997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:33 smithi081 ceph-mon[31453]: pgmap v10997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:35 smithi153 ceph-mon[36569]: pgmap v10998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:35 smithi104 ceph-mon[36624]: pgmap v10998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:35 smithi081 ceph-mon[31453]: pgmap v10998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:37 smithi153 ceph-mon[36569]: pgmap v10999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:37 smithi104 ceph-mon[36624]: pgmap v10999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:37 smithi081 ceph-mon[31453]: pgmap v10999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:38.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:38.273+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:53:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:38.273+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:53:38.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:38.273+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:53:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:39 smithi153 ceph-mon[36569]: pgmap v11000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:39 smithi104 ceph-mon[36624]: pgmap v11000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:39 smithi081 ceph-mon[31453]: pgmap v11000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:41 smithi153 ceph-mon[36569]: pgmap v11001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:41 smithi104 ceph-mon[36624]: pgmap v11001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:41 smithi081 ceph-mon[31453]: pgmap v11001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:43 smithi153 ceph-mon[36569]: pgmap v11002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:43 smithi104 ceph-mon[36624]: pgmap v11002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:43 smithi081 ceph-mon[31453]: pgmap v11002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:45 smithi153 ceph-mon[36569]: pgmap v11003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:45 smithi104 ceph-mon[36624]: pgmap v11003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:45 smithi081 ceph-mon[31453]: pgmap v11003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:47 smithi153 ceph-mon[36569]: pgmap v11004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:47 smithi104 ceph-mon[36624]: pgmap v11004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:47 smithi081 ceph-mon[31453]: pgmap v11004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:49 smithi153 ceph-mon[36569]: pgmap v11005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:49.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:49 smithi104 ceph-mon[36624]: pgmap v11005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:49 smithi081 ceph-mon[31453]: pgmap v11005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:51 smithi153 ceph-mon[36569]: pgmap v11006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:51 smithi104 ceph-mon[36624]: pgmap v11006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:51 smithi081 ceph-mon[31453]: pgmap v11006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:53.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:53.288+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:53:53.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:53.289+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:53:53.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:53:53.289+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:53:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:53 smithi153 ceph-mon[36569]: pgmap v11007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:53 smithi104 ceph-mon[36624]: pgmap v11007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:53 smithi081 ceph-mon[31453]: pgmap v11007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:55 smithi153 ceph-mon[36569]: pgmap v11008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:55 smithi104 ceph-mon[36624]: pgmap v11008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:55 smithi081 ceph-mon[31453]: pgmap v11008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:57 smithi153 ceph-mon[36569]: pgmap v11009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:57 smithi104 ceph-mon[36624]: pgmap v11009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:57 smithi081 ceph-mon[31453]: pgmap v11009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:53:59 smithi153 ceph-mon[36569]: pgmap v11010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:53:59 smithi104 ceph-mon[36624]: pgmap v11010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:53:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:53:59 smithi081 ceph-mon[31453]: pgmap v11010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:01 smithi153 ceph-mon[36569]: pgmap v11011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:01 smithi104 ceph-mon[36624]: pgmap v11011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:01 smithi081 ceph-mon[31453]: pgmap v11011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:03 smithi104 ceph-mon[36624]: pgmap v11012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:03 smithi081 ceph-mon[31453]: pgmap v11012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:03 smithi153 ceph-mon[36569]: pgmap v11012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:05 smithi104 ceph-mon[36624]: pgmap v11013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:05 smithi081 ceph-mon[31453]: pgmap v11013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:05 smithi153 ceph-mon[36569]: pgmap v11013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:07 smithi104 ceph-mon[36624]: pgmap v11014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:07 smithi081 ceph-mon[31453]: pgmap v11014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:07 smithi153 ceph-mon[36569]: pgmap v11014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:08.298+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:54:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:08.298+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:54:08.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:08.298+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:54:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:09 smithi104 ceph-mon[36624]: pgmap v11015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:09 smithi081 ceph-mon[31453]: pgmap v11015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:09 smithi153 ceph-mon[36569]: pgmap v11015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:11 smithi104 ceph-mon[36624]: pgmap v11016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:11 smithi081 ceph-mon[31453]: pgmap v11016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:11 smithi153 ceph-mon[36569]: pgmap v11016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:13 smithi081 ceph-mon[31453]: pgmap v11017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:13 smithi153 ceph-mon[36569]: pgmap v11017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:13 smithi104 ceph-mon[36624]: pgmap v11017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:15 smithi081 ceph-mon[31453]: pgmap v11018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:15 smithi153 ceph-mon[36569]: pgmap v11018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:15 smithi104 ceph-mon[36624]: pgmap v11018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:17.610 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:17 smithi153 ceph-mon[36569]: pgmap v11019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:17 smithi081 ceph-mon[31453]: pgmap v11019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:17 smithi104 ceph-mon[36624]: pgmap v11019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:19 smithi081 ceph-mon[31453]: pgmap v11020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:19 smithi153 ceph-mon[36569]: pgmap v11020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:19 smithi104 ceph-mon[36624]: pgmap v11020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:21 smithi081 ceph-mon[31453]: pgmap v11021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:21 smithi153 ceph-mon[36569]: pgmap v11021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:21 smithi104 ceph-mon[36624]: pgmap v11021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:23.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:23.303+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:54:23.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:23.303+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:54:23.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:23.303+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:54:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:23 smithi081 ceph-mon[31453]: pgmap v11022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:23 smithi153 ceph-mon[36569]: pgmap v11022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:23 smithi104 ceph-mon[36624]: pgmap v11022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:25 smithi081 ceph-mon[31453]: pgmap v11023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:54:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:25 smithi153 ceph-mon[36569]: pgmap v11023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:54:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:25 smithi104 ceph-mon[36624]: pgmap v11023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:54:26.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:54:26.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:54:26.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:54:26.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:54:26.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:54:26.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:54:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:54:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:54:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:54:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:27 smithi081 ceph-mon[31453]: pgmap v11024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:27.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:27 smithi081 ceph-mon[31453]: pgmap v11025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:27 smithi153 ceph-mon[36569]: pgmap v11024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:27 smithi153 ceph-mon[36569]: pgmap v11025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:27 smithi104 ceph-mon[36624]: pgmap v11024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:27 smithi104 ceph-mon[36624]: pgmap v11025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:29.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:29 smithi104 ceph-mon[36624]: pgmap v11026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:29 smithi081 ceph-mon[31453]: pgmap v11026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:29 smithi153 ceph-mon[36569]: pgmap v11026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:31.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:31.051+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:54:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:31.051+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:54:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:31.052+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 158 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:54:31.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:31.052+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 158 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:54:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:31 smithi081 ceph-mon[31453]: pgmap v11027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:31 smithi153 ceph-mon[36569]: pgmap v11027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:31 smithi104 ceph-mon[36624]: pgmap v11027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:33 smithi081 ceph-mon[31453]: pgmap v11028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:33 smithi153 ceph-mon[36569]: pgmap v11028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:33 smithi104 ceph-mon[36624]: pgmap v11028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:35 smithi081 ceph-mon[31453]: pgmap v11029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:35 smithi153 ceph-mon[36569]: pgmap v11029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:35 smithi104 ceph-mon[36624]: pgmap v11029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:37 smithi081 ceph-mon[31453]: pgmap v11030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:37 smithi153 ceph-mon[36569]: pgmap v11030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:37 smithi104 ceph-mon[36624]: pgmap v11030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:38.305+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:54:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:38.305+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:54:38.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:38.305+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:54:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:39 smithi081 ceph-mon[31453]: pgmap v11031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:39 smithi153 ceph-mon[36569]: pgmap v11031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:39 smithi104 ceph-mon[36624]: pgmap v11031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:41 smithi153 ceph-mon[36569]: pgmap v11032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:41 smithi104 ceph-mon[36624]: pgmap v11032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:41 smithi081 ceph-mon[31453]: pgmap v11032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:43 smithi153 ceph-mon[36569]: pgmap v11033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:43 smithi104 ceph-mon[36624]: pgmap v11033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:43 smithi081 ceph-mon[31453]: pgmap v11033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:45 smithi153 ceph-mon[36569]: pgmap v11034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:45 smithi104 ceph-mon[36624]: pgmap v11034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:45 smithi081 ceph-mon[31453]: pgmap v11034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:47 smithi153 ceph-mon[36569]: pgmap v11035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:47 smithi104 ceph-mon[36624]: pgmap v11035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:47 smithi081 ceph-mon[31453]: pgmap v11035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:49 smithi153 ceph-mon[36569]: pgmap v11036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:49 smithi104 ceph-mon[36624]: pgmap v11036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:49 smithi081 ceph-mon[31453]: pgmap v11036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:51 smithi153 ceph-mon[36569]: pgmap v11037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:51 smithi104 ceph-mon[36624]: pgmap v11037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:51 smithi081 ceph-mon[31453]: pgmap v11037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:53.308+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:54:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:53.308+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:54:53.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:54:53.308+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:54:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:53 smithi153 ceph-mon[36569]: pgmap v11038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:53 smithi104 ceph-mon[36624]: pgmap v11038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:53 smithi081 ceph-mon[31453]: pgmap v11038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:55 smithi153 ceph-mon[36569]: pgmap v11039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:55 smithi104 ceph-mon[36624]: pgmap v11039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:55 smithi081 ceph-mon[31453]: pgmap v11039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:57 smithi153 ceph-mon[36569]: pgmap v11040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:57 smithi104 ceph-mon[36624]: pgmap v11040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:57 smithi081 ceph-mon[31453]: pgmap v11040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:54:59 smithi153 ceph-mon[36569]: pgmap v11041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:54:59 smithi104 ceph-mon[36624]: pgmap v11041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:54:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:54:59 smithi081 ceph-mon[31453]: pgmap v11041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:01 smithi153 ceph-mon[36569]: pgmap v11042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:01 smithi104 ceph-mon[36624]: pgmap v11042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:01 smithi081 ceph-mon[31453]: pgmap v11042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:03 smithi153 ceph-mon[36569]: pgmap v11043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:03 smithi104 ceph-mon[36624]: pgmap v11043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:03 smithi081 ceph-mon[31453]: pgmap v11043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:05 smithi153 ceph-mon[36569]: pgmap v11044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:05 smithi104 ceph-mon[36624]: pgmap v11044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:05 smithi081 ceph-mon[31453]: pgmap v11044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:07 smithi153 ceph-mon[36569]: pgmap v11045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:07 smithi104 ceph-mon[36624]: pgmap v11045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:07 smithi081 ceph-mon[31453]: pgmap v11045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:08.315+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:55:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:08.315+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:55:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:08.315+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:55:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:09 smithi153 ceph-mon[36569]: pgmap v11046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:09 smithi104 ceph-mon[36624]: pgmap v11046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:09 smithi081 ceph-mon[31453]: pgmap v11046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:11 smithi153 ceph-mon[36569]: pgmap v11047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:11 smithi104 ceph-mon[36624]: pgmap v11047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:11 smithi081 ceph-mon[31453]: pgmap v11047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:13.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:13 smithi153 ceph-mon[36569]: pgmap v11048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:13 smithi104 ceph-mon[36624]: pgmap v11048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:13 smithi081 ceph-mon[31453]: pgmap v11048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:15 smithi153 ceph-mon[36569]: pgmap v11049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:15 smithi104 ceph-mon[36624]: pgmap v11049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:15 smithi081 ceph-mon[31453]: pgmap v11049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:16.059+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:55:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:16.059+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:55:16.060 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:16.060+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 159 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:55:16.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:16.060+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 159 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:55:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:17 smithi153 ceph-mon[36569]: pgmap v11050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:17 smithi104 ceph-mon[36624]: pgmap v11050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:17.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:17 smithi081 ceph-mon[31453]: pgmap v11050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:19 smithi153 ceph-mon[36569]: pgmap v11051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:19 smithi104 ceph-mon[36624]: pgmap v11051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:19 smithi081 ceph-mon[31453]: pgmap v11051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:21 smithi153 ceph-mon[36569]: pgmap v11052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:21 smithi104 ceph-mon[36624]: pgmap v11052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:21 smithi081 ceph-mon[31453]: pgmap v11052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:23.331+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:55:23.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:23.331+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:55:23.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:23.331+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:55:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:23 smithi153 ceph-mon[36569]: pgmap v11053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:23 smithi104 ceph-mon[36624]: pgmap v11053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:23 smithi081 ceph-mon[31453]: pgmap v11053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:25 smithi153 ceph-mon[36569]: pgmap v11054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:25 smithi104 ceph-mon[36624]: pgmap v11054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:25 smithi081 ceph-mon[31453]: pgmap v11054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:55:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:55:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: pgmap v11055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:55:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:55:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:55:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: pgmap v11055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:55:27.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:55:27.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: pgmap v11055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:55:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:55:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:28 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:55:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:28 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:55:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:28 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:55:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:28 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:55:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:28 smithi104 ceph-mon[36624]: pgmap v11056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:28 smithi104 ceph-mon[36624]: pgmap v11057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:28 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:55:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:28 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:55:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:28 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:55:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:28 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:55:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:28 smithi081 ceph-mon[31453]: pgmap v11056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:28 smithi081 ceph-mon[31453]: pgmap v11057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:28 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T02:55:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:28 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T02:55:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:28 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T02:55:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:28 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T02:55:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:28 smithi153 ceph-mon[36569]: pgmap v11056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:28 smithi153 ceph-mon[36569]: pgmap v11057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:30 smithi104 ceph-mon[36624]: pgmap v11058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:30 smithi081 ceph-mon[31453]: pgmap v11058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:30 smithi153 ceph-mon[36569]: pgmap v11058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:32 smithi104 ceph-mon[36624]: pgmap v11059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:32 smithi081 ceph-mon[31453]: pgmap v11059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:32 smithi153 ceph-mon[36569]: pgmap v11059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:34 smithi104 ceph-mon[36624]: pgmap v11060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:34 smithi081 ceph-mon[31453]: pgmap v11060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:34 smithi153 ceph-mon[36569]: pgmap v11060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:36 smithi104 ceph-mon[36624]: pgmap v11061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:36 smithi081 ceph-mon[31453]: pgmap v11061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:36 smithi153 ceph-mon[36569]: pgmap v11061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:38.347+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:55:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:38.347+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:55:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:38.347+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:55:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:38 smithi081 ceph-mon[31453]: pgmap v11062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:38 smithi153 ceph-mon[36569]: pgmap v11062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:38 smithi104 ceph-mon[36624]: pgmap v11062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:40 smithi081 ceph-mon[31453]: pgmap v11063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:40 smithi153 ceph-mon[36569]: pgmap v11063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:40 smithi104 ceph-mon[36624]: pgmap v11063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:42 smithi081 ceph-mon[31453]: pgmap v11064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:42 smithi153 ceph-mon[36569]: pgmap v11064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:42 smithi104 ceph-mon[36624]: pgmap v11064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:44 smithi081 ceph-mon[31453]: pgmap v11065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:44 smithi153 ceph-mon[36569]: pgmap v11065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:44 smithi104 ceph-mon[36624]: pgmap v11065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:46 smithi081 ceph-mon[31453]: pgmap v11066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:46 smithi153 ceph-mon[36569]: pgmap v11066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:46 smithi104 ceph-mon[36624]: pgmap v11066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:48.862 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:48 smithi153 ceph-mon[36569]: pgmap v11067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:48 smithi081 ceph-mon[31453]: pgmap v11067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:48 smithi104 ceph-mon[36624]: pgmap v11067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:50 smithi081 ceph-mon[31453]: pgmap v11068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:50 smithi153 ceph-mon[36569]: pgmap v11068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:50 smithi104 ceph-mon[36624]: pgmap v11068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:52 smithi081 ceph-mon[31453]: pgmap v11069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:52 smithi153 ceph-mon[36569]: pgmap v11069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:52 smithi104 ceph-mon[36624]: pgmap v11069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:53.360+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:55:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:53.360+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:55:53.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:55:53.360+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:55:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:54 smithi081 ceph-mon[31453]: pgmap v11070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:54 smithi153 ceph-mon[36569]: pgmap v11070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:54 smithi104 ceph-mon[36624]: pgmap v11070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:56 smithi081 ceph-mon[31453]: pgmap v11071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:56 smithi153 ceph-mon[36569]: pgmap v11071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:56 smithi104 ceph-mon[36624]: pgmap v11071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:55:58 smithi081 ceph-mon[31453]: pgmap v11072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:55:58 smithi153 ceph-mon[36569]: pgmap v11072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:55:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:55:58 smithi104 ceph-mon[36624]: pgmap v11072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:00 smithi081 ceph-mon[31453]: pgmap v11073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:00 smithi153 ceph-mon[36569]: pgmap v11073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:00 smithi104 ceph-mon[36624]: pgmap v11073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:02 smithi081 ceph-mon[31453]: pgmap v11074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:02 smithi153 ceph-mon[36569]: pgmap v11074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:02 smithi104 ceph-mon[36624]: pgmap v11074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:04.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:04 smithi081 ceph-mon[31453]: pgmap v11075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:04 smithi153 ceph-mon[36569]: pgmap v11075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:04 smithi104 ceph-mon[36624]: pgmap v11075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:06 smithi153 ceph-mon[36569]: pgmap v11076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:06 smithi104 ceph-mon[36624]: pgmap v11076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:06 smithi081 ceph-mon[31453]: pgmap v11076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:08.370+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:56:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:08.370+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:56:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:08.370+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:56:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:08 smithi153 ceph-mon[36569]: pgmap v11077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:08 smithi104 ceph-mon[36624]: pgmap v11077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:08 smithi081 ceph-mon[31453]: pgmap v11077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:10 smithi153 ceph-mon[36569]: pgmap v11078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:10 smithi104 ceph-mon[36624]: pgmap v11078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:10 smithi081 ceph-mon[31453]: pgmap v11078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:12.936 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:12 smithi081 ceph-mon[31453]: pgmap v11079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:12 smithi153 ceph-mon[36569]: pgmap v11079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:12 smithi104 ceph-mon[36624]: pgmap v11079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:14 smithi153 ceph-mon[36569]: pgmap v11080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:14 smithi104 ceph-mon[36624]: pgmap v11080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:14 smithi081 ceph-mon[31453]: pgmap v11080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:16 smithi153 ceph-mon[36569]: pgmap v11081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:16 smithi104 ceph-mon[36624]: pgmap v11081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:16 smithi081 ceph-mon[31453]: pgmap v11081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:18 smithi153 ceph-mon[36569]: pgmap v11082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:18 smithi104 ceph-mon[36624]: pgmap v11082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:18 smithi081 ceph-mon[31453]: pgmap v11082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:20 smithi153 ceph-mon[36569]: pgmap v11083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:20 smithi104 ceph-mon[36624]: pgmap v11083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:20 smithi081 ceph-mon[31453]: pgmap v11083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:22 smithi153 ceph-mon[36569]: pgmap v11084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:22 smithi104 ceph-mon[36624]: pgmap v11084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:22 smithi081 ceph-mon[31453]: pgmap v11084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:23.385+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:56:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:23.385+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:56:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:23.385+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:56:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:24 smithi153 ceph-mon[36569]: pgmap v11085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:24 smithi104 ceph-mon[36624]: pgmap v11085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:24 smithi081 ceph-mon[31453]: pgmap v11085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:25 smithi081 ceph-mon[31453]: pgmap v11086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:25 smithi153 ceph-mon[36569]: pgmap v11086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:25 smithi104 ceph-mon[36624]: pgmap v11086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:28 smithi081 ceph-mon[31453]: pgmap v11087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:56:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:28 smithi153 ceph-mon[36569]: pgmap v11087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:56:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:28 smithi104 ceph-mon[36624]: pgmap v11087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:56:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:56:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:56:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:56:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:56:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:56:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:56:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:56:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:56:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:56:30.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:30 smithi104 ceph-mon[36624]: pgmap v11088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:30.735 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:30 smithi104 ceph-mon[36624]: pgmap v11089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:30 smithi153 ceph-mon[36569]: pgmap v11088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:30 smithi153 ceph-mon[36569]: pgmap v11089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:30 smithi081 ceph-mon[31453]: pgmap v11088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:30 smithi081 ceph-mon[31453]: pgmap v11089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:32 smithi153 ceph-mon[36569]: pgmap v11090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:32 smithi104 ceph-mon[36624]: pgmap v11090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:32 smithi081 ceph-mon[31453]: pgmap v11090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:34 smithi153 ceph-mon[36569]: pgmap v11091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:34 smithi104 ceph-mon[36624]: pgmap v11091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:34 smithi081 ceph-mon[31453]: pgmap v11091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:36 smithi153 ceph-mon[36569]: pgmap v11092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:36 smithi104 ceph-mon[36624]: pgmap v11092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:36 smithi081 ceph-mon[31453]: pgmap v11092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:38.393 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:38.392+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:56:38.393 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:38.392+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:56:38.393 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:38.392+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:56:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:38 smithi153 ceph-mon[36569]: pgmap v11093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:38 smithi104 ceph-mon[36624]: pgmap v11093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:38 smithi081 ceph-mon[31453]: pgmap v11093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:40 smithi153 ceph-mon[36569]: pgmap v11094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:40 smithi104 ceph-mon[36624]: pgmap v11094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:40 smithi081 ceph-mon[31453]: pgmap v11094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:42 smithi153 ceph-mon[36569]: pgmap v11095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:42 smithi104 ceph-mon[36624]: pgmap v11095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:42 smithi081 ceph-mon[31453]: pgmap v11095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:44 smithi153 ceph-mon[36569]: pgmap v11096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:44 smithi104 ceph-mon[36624]: pgmap v11096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:44 smithi081 ceph-mon[31453]: pgmap v11096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:46 smithi153 ceph-mon[36569]: pgmap v11097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:46 smithi104 ceph-mon[36624]: pgmap v11097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:46 smithi081 ceph-mon[31453]: pgmap v11097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:48 smithi153 ceph-mon[36569]: pgmap v11098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:48 smithi104 ceph-mon[36624]: pgmap v11098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:48 smithi081 ceph-mon[31453]: pgmap v11098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:50 smithi153 ceph-mon[36569]: pgmap v11099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:50 smithi104 ceph-mon[36624]: pgmap v11099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:50 smithi081 ceph-mon[31453]: pgmap v11099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:52 smithi153 ceph-mon[36569]: pgmap v11100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:52 smithi104 ceph-mon[36624]: pgmap v11100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:52 smithi081 ceph-mon[31453]: pgmap v11100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:53.406+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:56:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:53.406+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:56:53.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:56:53.406+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:56:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:54 smithi153 ceph-mon[36569]: pgmap v11101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:54 smithi104 ceph-mon[36624]: pgmap v11101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:54 smithi081 ceph-mon[31453]: pgmap v11101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:56 smithi153 ceph-mon[36569]: pgmap v11102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:56 smithi104 ceph-mon[36624]: pgmap v11102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:56 smithi081 ceph-mon[31453]: pgmap v11102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:56:58 smithi153 ceph-mon[36569]: pgmap v11103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:56:58 smithi104 ceph-mon[36624]: pgmap v11103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:56:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:56:58 smithi081 ceph-mon[31453]: pgmap v11103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:00 smithi153 ceph-mon[36569]: pgmap v11104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:00 smithi104 ceph-mon[36624]: pgmap v11104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:00 smithi081 ceph-mon[31453]: pgmap v11104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:02.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:02 smithi153 ceph-mon[36569]: pgmap v11105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:02 smithi104 ceph-mon[36624]: pgmap v11105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:02 smithi081 ceph-mon[31453]: pgmap v11105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:04.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:04 smithi153 ceph-mon[36569]: pgmap v11106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:04 smithi104 ceph-mon[36624]: pgmap v11106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:04.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:04 smithi081 ceph-mon[31453]: pgmap v11106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:06 smithi153 ceph-mon[36569]: pgmap v11107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:06 smithi104 ceph-mon[36624]: pgmap v11107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:06 smithi081 ceph-mon[31453]: pgmap v11107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:08.422+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:57:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:08.422+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:57:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:08.422+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:57:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:08 smithi153 ceph-mon[36569]: pgmap v11108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:08 smithi104 ceph-mon[36624]: pgmap v11108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:08 smithi081 ceph-mon[31453]: pgmap v11108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:10 smithi153 ceph-mon[36569]: pgmap v11109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:10.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:10 smithi104 ceph-mon[36624]: pgmap v11109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:10 smithi081 ceph-mon[31453]: pgmap v11109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:12 smithi153 ceph-mon[36569]: pgmap v11110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:12 smithi104 ceph-mon[36624]: pgmap v11110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:12 smithi081 ceph-mon[31453]: pgmap v11110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:14 smithi153 ceph-mon[36569]: pgmap v11111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:14 smithi104 ceph-mon[36624]: pgmap v11111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:14.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:14 smithi081 ceph-mon[31453]: pgmap v11111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:16 smithi153 ceph-mon[36569]: pgmap v11112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:16 smithi104 ceph-mon[36624]: pgmap v11112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:16 smithi081 ceph-mon[31453]: pgmap v11112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:18 smithi104 ceph-mon[36624]: pgmap v11113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:18 smithi081 ceph-mon[31453]: pgmap v11113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:18 smithi153 ceph-mon[36569]: pgmap v11113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:20 smithi104 ceph-mon[36624]: pgmap v11114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:20 smithi081 ceph-mon[31453]: pgmap v11114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:20 smithi153 ceph-mon[36569]: pgmap v11114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:22 smithi104 ceph-mon[36624]: pgmap v11115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:22 smithi081 ceph-mon[31453]: pgmap v11115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:22 smithi153 ceph-mon[36569]: pgmap v11115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:23.423+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:57:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:23.423+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:57:23.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:23.423+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:57:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:24 smithi104 ceph-mon[36624]: pgmap v11116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:24.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:24 smithi081 ceph-mon[31453]: pgmap v11116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:24 smithi153 ceph-mon[36569]: pgmap v11116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:26 smithi104 ceph-mon[36624]: pgmap v11117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:26 smithi081 ceph-mon[31453]: pgmap v11117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:26 smithi153 ceph-mon[36569]: pgmap v11117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:28 smithi081 ceph-mon[31453]: pgmap v11118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:28 smithi153 ceph-mon[36569]: pgmap v11118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:28 smithi104 ceph-mon[36624]: pgmap v11118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:57:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:57:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:57:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:30 smithi081 ceph-mon[31453]: pgmap v11119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:57:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:57:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:57:30.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:30 smithi104 ceph-mon[36624]: pgmap v11119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:30.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:57:30.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:57:30.984 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:57:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:30 smithi153 ceph-mon[36569]: pgmap v11119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:57:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:57:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:57:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:31 smithi081 ceph-mon[31453]: pgmap v11120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:31 smithi081 ceph-mon[31453]: pgmap v11121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:31 smithi153 ceph-mon[36569]: pgmap v11120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:31 smithi153 ceph-mon[36569]: pgmap v11121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:31 smithi104 ceph-mon[36624]: pgmap v11120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:31 smithi104 ceph-mon[36624]: pgmap v11121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:33 smithi081 ceph-mon[31453]: pgmap v11122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:33 smithi153 ceph-mon[36569]: pgmap v11122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:33 smithi104 ceph-mon[36624]: pgmap v11122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:35 smithi081 ceph-mon[31453]: pgmap v11123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:35 smithi153 ceph-mon[36569]: pgmap v11123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:35 smithi104 ceph-mon[36624]: pgmap v11123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:37 smithi081 ceph-mon[31453]: pgmap v11124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:37 smithi153 ceph-mon[36569]: pgmap v11124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:37 smithi104 ceph-mon[36624]: pgmap v11124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:38.429+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:57:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:38.429+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:57:38.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:38.429+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:57:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:39 smithi081 ceph-mon[31453]: pgmap v11125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:39 smithi153 ceph-mon[36569]: pgmap v11125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:39 smithi104 ceph-mon[36624]: pgmap v11125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:41 smithi081 ceph-mon[31453]: pgmap v11126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:41 smithi153 ceph-mon[36569]: pgmap v11126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:41 smithi104 ceph-mon[36624]: pgmap v11126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:43 smithi081 ceph-mon[31453]: pgmap v11127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:43 smithi153 ceph-mon[36569]: pgmap v11127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:43 smithi104 ceph-mon[36624]: pgmap v11127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:45 smithi081 ceph-mon[31453]: pgmap v11128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:45 smithi153 ceph-mon[36569]: pgmap v11128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:45 smithi104 ceph-mon[36624]: pgmap v11128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:47 smithi153 ceph-mon[36569]: pgmap v11129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:47 smithi104 ceph-mon[36624]: pgmap v11129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:47 smithi081 ceph-mon[31453]: pgmap v11129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:49 smithi153 ceph-mon[36569]: pgmap v11130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:49 smithi104 ceph-mon[36624]: pgmap v11130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:49 smithi081 ceph-mon[31453]: pgmap v11130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:51 smithi153 ceph-mon[36569]: pgmap v11131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:51 smithi104 ceph-mon[36624]: pgmap v11131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:51 smithi081 ceph-mon[31453]: pgmap v11131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:53.446+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:57:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:53.446+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:57:53.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:57:53.446+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:57:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:53 smithi153 ceph-mon[36569]: pgmap v11132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:53 smithi104 ceph-mon[36624]: pgmap v11132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:53 smithi081 ceph-mon[31453]: pgmap v11132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:55 smithi153 ceph-mon[36569]: pgmap v11133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:55 smithi104 ceph-mon[36624]: pgmap v11133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:55 smithi081 ceph-mon[31453]: pgmap v11133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:57.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:57 smithi081 ceph-mon[31453]: pgmap v11134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:57 smithi153 ceph-mon[36569]: pgmap v11134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:57:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:57 smithi104 ceph-mon[36624]: pgmap v11134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:57:59 smithi153 ceph-mon[36569]: pgmap v11135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:57:59 smithi104 ceph-mon[36624]: pgmap v11135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:57:59 smithi081 ceph-mon[31453]: pgmap v11135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:01.992 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:01 smithi081 ceph-mon[31453]: pgmap v11136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:01 smithi153 ceph-mon[36569]: pgmap v11136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:01 smithi104 ceph-mon[36624]: pgmap v11136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:03 smithi153 ceph-mon[36569]: pgmap v11137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:03 smithi104 ceph-mon[36624]: pgmap v11137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:03 smithi081 ceph-mon[31453]: pgmap v11137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:05 smithi153 ceph-mon[36569]: pgmap v11138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:05 smithi104 ceph-mon[36624]: pgmap v11138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:05 smithi081 ceph-mon[31453]: pgmap v11138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:07 smithi153 ceph-mon[36569]: pgmap v11139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:07 smithi104 ceph-mon[36624]: pgmap v11139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:07 smithi081 ceph-mon[31453]: pgmap v11139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:08.459 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:08.458+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:58:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:08.458+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:58:08.460 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:08.458+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:58:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:09 smithi153 ceph-mon[36569]: pgmap v11140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:09 smithi104 ceph-mon[36624]: pgmap v11140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:09 smithi081 ceph-mon[31453]: pgmap v11140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:11 smithi153 ceph-mon[36569]: pgmap v11141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:11 smithi104 ceph-mon[36624]: pgmap v11141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:11 smithi081 ceph-mon[31453]: pgmap v11141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:14.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:13 smithi153 ceph-mon[36569]: pgmap v11142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:13 smithi104 ceph-mon[36624]: pgmap v11142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:13 smithi081 ceph-mon[31453]: pgmap v11142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:15 smithi153 ceph-mon[36569]: pgmap v11143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:15 smithi104 ceph-mon[36624]: pgmap v11143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:16.089+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:58:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:16.089+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:58:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:16.089+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 160 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T02:58:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:16.089+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 160 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T02:58:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:15 smithi081 ceph-mon[31453]: pgmap v11143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:17 smithi153 ceph-mon[36569]: pgmap v11144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:17 smithi104 ceph-mon[36624]: pgmap v11144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:17 smithi081 ceph-mon[31453]: pgmap v11144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:19 smithi153 ceph-mon[36569]: pgmap v11145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:19 smithi104 ceph-mon[36624]: pgmap v11145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:19 smithi081 ceph-mon[31453]: pgmap v11145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:21 smithi153 ceph-mon[36569]: pgmap v11146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:21 smithi104 ceph-mon[36624]: pgmap v11146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:21 smithi081 ceph-mon[31453]: pgmap v11146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:23.464+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:58:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:23.464+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:58:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:23.464+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:58:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:23 smithi153 ceph-mon[36569]: pgmap v11147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:23 smithi104 ceph-mon[36624]: pgmap v11147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:23 smithi081 ceph-mon[31453]: pgmap v11147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:25 smithi153 ceph-mon[36569]: pgmap v11148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:25 smithi104 ceph-mon[36624]: pgmap v11148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:25 smithi081 ceph-mon[31453]: pgmap v11148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:27 smithi153 ceph-mon[36569]: pgmap v11149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:27 smithi104 ceph-mon[36624]: pgmap v11149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:27 smithi081 ceph-mon[31453]: pgmap v11149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:29 smithi153 ceph-mon[36569]: pgmap v11150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:29 smithi104 ceph-mon[36624]: pgmap v11150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:29 smithi081 ceph-mon[31453]: pgmap v11150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:58:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:58:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:58:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:31 smithi153 ceph-mon[36569]: pgmap v11151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:31 smithi104 ceph-mon[36624]: pgmap v11151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:31 smithi081 ceph-mon[31453]: pgmap v11151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:58:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:58:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:58:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:58:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:58:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:58:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:58:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:58:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:58:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:33 smithi153 ceph-mon[36569]: pgmap v11152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:33 smithi153 ceph-mon[36569]: pgmap v11153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:33 smithi104 ceph-mon[36624]: pgmap v11152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:33 smithi104 ceph-mon[36624]: pgmap v11153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:33 smithi081 ceph-mon[31453]: pgmap v11152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:33 smithi081 ceph-mon[31453]: pgmap v11153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:34 smithi081 ceph-mon[31453]: pgmap v11154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:34 smithi153 ceph-mon[36569]: pgmap v11154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:34 smithi104 ceph-mon[36624]: pgmap v11154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:36 smithi153 ceph-mon[36569]: pgmap v11155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:36 smithi104 ceph-mon[36624]: pgmap v11155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:36 smithi081 ceph-mon[31453]: pgmap v11155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:38.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:38.466+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:58:38.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:38.466+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:58:38.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:38.466+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:58:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:38 smithi153 ceph-mon[36569]: pgmap v11156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:38 smithi104 ceph-mon[36624]: pgmap v11156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:38 smithi081 ceph-mon[31453]: pgmap v11156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:40 smithi153 ceph-mon[36569]: pgmap v11157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:40 smithi104 ceph-mon[36624]: pgmap v11157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:40 smithi081 ceph-mon[31453]: pgmap v11157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:42 smithi153 ceph-mon[36569]: pgmap v11158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:42 smithi104 ceph-mon[36624]: pgmap v11158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:42 smithi081 ceph-mon[31453]: pgmap v11158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:44 smithi153 ceph-mon[36569]: pgmap v11159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:44 smithi104 ceph-mon[36624]: pgmap v11159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:44 smithi081 ceph-mon[31453]: pgmap v11159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:46 smithi153 ceph-mon[36569]: pgmap v11160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:46 smithi104 ceph-mon[36624]: pgmap v11160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:46 smithi081 ceph-mon[31453]: pgmap v11160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:48 smithi153 ceph-mon[36569]: pgmap v11161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:48 smithi104 ceph-mon[36624]: pgmap v11161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:48 smithi081 ceph-mon[31453]: pgmap v11161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:50 smithi153 ceph-mon[36569]: pgmap v11162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:50 smithi104 ceph-mon[36624]: pgmap v11162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:50 smithi081 ceph-mon[31453]: pgmap v11162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:52 smithi153 ceph-mon[36569]: pgmap v11163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:52 smithi104 ceph-mon[36624]: pgmap v11163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:52 smithi081 ceph-mon[31453]: pgmap v11163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:53.478+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:58:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:53.478+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:58:53.480 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:58:53.478+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:58:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:55 smithi153 ceph-mon[36569]: pgmap v11164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:55 smithi104 ceph-mon[36624]: pgmap v11164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:55 smithi081 ceph-mon[31453]: pgmap v11164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:57 smithi153 ceph-mon[36569]: pgmap v11165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:57 smithi104 ceph-mon[36624]: pgmap v11165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:57 smithi081 ceph-mon[31453]: pgmap v11165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:58:59 smithi153 ceph-mon[36569]: pgmap v11166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:58:59 smithi104 ceph-mon[36624]: pgmap v11166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:58:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:58:59 smithi081 ceph-mon[31453]: pgmap v11166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:01 smithi153 ceph-mon[36569]: pgmap v11167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:01 smithi104 ceph-mon[36624]: pgmap v11167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:01 smithi081 ceph-mon[31453]: pgmap v11167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:03 smithi153 ceph-mon[36569]: pgmap v11168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:03 smithi104 ceph-mon[36624]: pgmap v11168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:03 smithi081 ceph-mon[31453]: pgmap v11168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:05 smithi153 ceph-mon[36569]: pgmap v11169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:05 smithi104 ceph-mon[36624]: pgmap v11169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:05 smithi081 ceph-mon[31453]: pgmap v11169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:07 smithi153 ceph-mon[36569]: pgmap v11170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:07 smithi104 ceph-mon[36624]: pgmap v11170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:07 smithi081 ceph-mon[31453]: pgmap v11170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:08.481 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:08.480+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:59:08.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:08.480+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:59:08.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:08.480+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:59:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:09 smithi153 ceph-mon[36569]: pgmap v11171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:09 smithi104 ceph-mon[36624]: pgmap v11171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:09 smithi081 ceph-mon[31453]: pgmap v11171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:11 smithi153 ceph-mon[36569]: pgmap v11172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:11 smithi104 ceph-mon[36624]: pgmap v11172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:11 smithi081 ceph-mon[31453]: pgmap v11172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:13.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:13 smithi153 ceph-mon[36569]: pgmap v11173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:13 smithi104 ceph-mon[36624]: pgmap v11173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:13 smithi081 ceph-mon[31453]: pgmap v11173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:15 smithi153 ceph-mon[36569]: pgmap v11174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:15 smithi104 ceph-mon[36624]: pgmap v11174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:15 smithi081 ceph-mon[31453]: pgmap v11174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:17 smithi104 ceph-mon[36624]: pgmap v11175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:17 smithi081 ceph-mon[31453]: pgmap v11175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:17 smithi153 ceph-mon[36569]: pgmap v11175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:19 smithi104 ceph-mon[36624]: pgmap v11176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:19.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:19 smithi153 ceph-mon[36569]: pgmap v11176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:19 smithi081 ceph-mon[31453]: pgmap v11176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:21 smithi104 ceph-mon[36624]: pgmap v11177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:21 smithi081 ceph-mon[31453]: pgmap v11177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:21 smithi153 ceph-mon[36569]: pgmap v11177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:23 smithi104 ceph-mon[36624]: pgmap v11178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:23 smithi081 ceph-mon[31453]: pgmap v11178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:23.492+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:59:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:23.492+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:59:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:23.492+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:59:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:23 smithi153 ceph-mon[36569]: pgmap v11178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:25 smithi104 ceph-mon[36624]: pgmap v11179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:25 smithi081 ceph-mon[31453]: pgmap v11179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:25 smithi153 ceph-mon[36569]: pgmap v11179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:27 smithi081 ceph-mon[31453]: pgmap v11180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:27 smithi153 ceph-mon[36569]: pgmap v11180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:27 smithi104 ceph-mon[36624]: pgmap v11180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:29 smithi081 ceph-mon[31453]: pgmap v11181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:29 smithi153 ceph-mon[36569]: pgmap v11181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:29 smithi104 ceph-mon[36624]: pgmap v11181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:31.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:31 smithi104 ceph-mon[36624]: pgmap v11182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:31 smithi081 ceph-mon[31453]: pgmap v11182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:31 smithi153 ceph-mon[36569]: pgmap v11182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:33.341 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:33 smithi153 ceph-mon[36569]: pgmap v11183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:33.341 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:59:33.341 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.342 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.342 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.342 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:33 smithi081 ceph-mon[31453]: pgmap v11183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:59:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:33 smithi104 ceph-mon[36624]: pgmap v11183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:33.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T02:59:33.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:33.569 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:35 smithi153 ceph-mon[36569]: pgmap v11184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:35 smithi104 ceph-mon[36624]: pgmap v11184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:35 smithi081 ceph-mon[31453]: pgmap v11184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:37 smithi153 ceph-mon[36569]: pgmap v11185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:37 smithi104 ceph-mon[36624]: pgmap v11185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:37 smithi081 ceph-mon[31453]: pgmap v11185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:38.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:38.508+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:59:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:38.508+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:59:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:38.508+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:59:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:59:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:59:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:59:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:59:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T02:59:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T02:59:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T02:59:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:39 smithi153 ceph-mon[36569]: pgmap v11186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:39 smithi153 ceph-mon[36569]: pgmap v11187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:39 smithi153 ceph-mon[36569]: pgmap v11188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:39 smithi104 ceph-mon[36624]: pgmap v11186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:39 smithi104 ceph-mon[36624]: pgmap v11187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:39 smithi104 ceph-mon[36624]: pgmap v11188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:39 smithi081 ceph-mon[31453]: pgmap v11186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:39 smithi081 ceph-mon[31453]: pgmap v11187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:39 smithi081 ceph-mon[31453]: pgmap v11188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:41 smithi153 ceph-mon[36569]: pgmap v11189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:41 smithi104 ceph-mon[36624]: pgmap v11189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:41 smithi081 ceph-mon[31453]: pgmap v11189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:43 smithi153 ceph-mon[36569]: pgmap v11190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:43 smithi104 ceph-mon[36624]: pgmap v11190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:43 smithi081 ceph-mon[31453]: pgmap v11190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:45 smithi153 ceph-mon[36569]: pgmap v11191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:45 smithi104 ceph-mon[36624]: pgmap v11191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:45 smithi081 ceph-mon[31453]: pgmap v11191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:47 smithi153 ceph-mon[36569]: pgmap v11192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:47 smithi104 ceph-mon[36624]: pgmap v11192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:47 smithi081 ceph-mon[31453]: pgmap v11192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:49 smithi153 ceph-mon[36569]: pgmap v11193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:49 smithi104 ceph-mon[36624]: pgmap v11193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:49 smithi081 ceph-mon[31453]: pgmap v11193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:51 smithi153 ceph-mon[36569]: pgmap v11194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:52.083 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:51 smithi081 ceph-mon[31453]: pgmap v11194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:51 smithi104 ceph-mon[36624]: pgmap v11194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:53.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:53.524+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T02:59:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:53.524+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T02:59:53.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T02:59:53.524+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T02:59:54.029 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:53 smithi081 ceph-mon[31453]: pgmap v11195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:53 smithi153 ceph-mon[36569]: pgmap v11195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:53 smithi104 ceph-mon[36624]: pgmap v11195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:55 smithi153 ceph-mon[36569]: pgmap v11196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:55 smithi104 ceph-mon[36624]: pgmap v11196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:55 smithi081 ceph-mon[31453]: pgmap v11196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:57 smithi153 ceph-mon[36569]: pgmap v11197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:57 smithi104 ceph-mon[36624]: pgmap v11197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T02:59:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:57 smithi081 ceph-mon[31453]: pgmap v11197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 02:59:59 smithi153 ceph-mon[36569]: pgmap v11198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 02:59:59 smithi104 ceph-mon[36624]: pgmap v11198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 02:59:59 smithi081 ceph-mon[31453]: pgmap v11198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:00:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:00:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:00:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:01 smithi153 ceph-mon[36569]: pgmap v11199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:01 smithi104 ceph-mon[36624]: pgmap v11199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:01 smithi081 ceph-mon[31453]: pgmap v11199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:03 smithi153 ceph-mon[36569]: pgmap v11200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:03 smithi104 ceph-mon[36624]: pgmap v11200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:03 smithi081 ceph-mon[31453]: pgmap v11200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:05 smithi153 ceph-mon[36569]: pgmap v11201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:05 smithi104 ceph-mon[36624]: pgmap v11201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:05 smithi081 ceph-mon[31453]: pgmap v11201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:07 smithi153 ceph-mon[36569]: pgmap v11202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:07 smithi104 ceph-mon[36624]: pgmap v11202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:07 smithi081 ceph-mon[31453]: pgmap v11202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:08.534+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:00:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:08.534+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:00:08.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:08.534+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:00:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:09 smithi104 ceph-mon[36624]: pgmap v11203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:09 smithi081 ceph-mon[31453]: pgmap v11203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:09 smithi153 ceph-mon[36569]: pgmap v11203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:11 smithi104 ceph-mon[36624]: pgmap v11204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:11 smithi081 ceph-mon[31453]: pgmap v11204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:11 smithi153 ceph-mon[36569]: pgmap v11204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:13 smithi104 ceph-mon[36624]: pgmap v11205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:13 smithi081 ceph-mon[31453]: pgmap v11205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:13 smithi153 ceph-mon[36569]: pgmap v11205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:15 smithi104 ceph-mon[36624]: pgmap v11206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:15 smithi081 ceph-mon[31453]: pgmap v11206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:15 smithi153 ceph-mon[36569]: pgmap v11206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:16 smithi081 ceph-mon[31453]: pgmap v11207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:16 smithi153 ceph-mon[36569]: pgmap v11207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:16 smithi104 ceph-mon[36624]: pgmap v11207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:19 smithi081 ceph-mon[31453]: pgmap v11208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:19 smithi153 ceph-mon[36569]: pgmap v11208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:19 smithi104 ceph-mon[36624]: pgmap v11208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:21 smithi153 ceph-mon[36569]: pgmap v11209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:21 smithi104 ceph-mon[36624]: pgmap v11209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:21 smithi081 ceph-mon[31453]: pgmap v11209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:23.541+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:00:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:23.542+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:00:23.543 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:23.542+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:00:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:23 smithi153 ceph-mon[36569]: pgmap v11210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:23 smithi104 ceph-mon[36624]: pgmap v11210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:23 smithi081 ceph-mon[31453]: pgmap v11210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:25 smithi153 ceph-mon[36569]: pgmap v11211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:25 smithi104 ceph-mon[36624]: pgmap v11211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:25 smithi081 ceph-mon[31453]: pgmap v11211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:27.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:27 smithi081 ceph-mon[31453]: pgmap v11212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:27 smithi153 ceph-mon[36569]: pgmap v11212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:27 smithi104 ceph-mon[36624]: pgmap v11212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:29 smithi153 ceph-mon[36569]: pgmap v11213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:29 smithi104 ceph-mon[36624]: pgmap v11213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:29 smithi081 ceph-mon[31453]: pgmap v11213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:31 smithi153 ceph-mon[36569]: pgmap v11214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:31 smithi104 ceph-mon[36624]: pgmap v11214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:31 smithi081 ceph-mon[31453]: pgmap v11214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:33 smithi153 ceph-mon[36569]: pgmap v11215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:33 smithi104 ceph-mon[36624]: pgmap v11215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:33 smithi081 ceph-mon[31453]: pgmap v11215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:35 smithi153 ceph-mon[36569]: pgmap v11216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:35 smithi104 ceph-mon[36624]: pgmap v11216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:35 smithi081 ceph-mon[31453]: pgmap v11216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:37 smithi153 ceph-mon[36569]: pgmap v11217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:37 smithi104 ceph-mon[36624]: pgmap v11217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:37 smithi081 ceph-mon[31453]: pgmap v11217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:38.557+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:00:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:38.557+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:00:38.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:38.557+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:00:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:39 smithi153 ceph-mon[36569]: pgmap v11218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:40.168 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:00:40.168 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:40.168 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:40.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:39 smithi081 ceph-mon[31453]: pgmap v11218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:40.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:00:40.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:40.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:40.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:39 smithi104 ceph-mon[36624]: pgmap v11218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:40.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:00:40.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:40.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:00:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:00:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:00:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:00:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:00:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:00:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:00:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:41 smithi153 ceph-mon[36569]: pgmap v11219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:41 smithi153 ceph-mon[36569]: pgmap v11220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:41 smithi104 ceph-mon[36624]: pgmap v11219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:41 smithi104 ceph-mon[36624]: pgmap v11220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:41 smithi081 ceph-mon[31453]: pgmap v11219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:41 smithi081 ceph-mon[31453]: pgmap v11220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:43 smithi153 ceph-mon[36569]: pgmap v11221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:43 smithi104 ceph-mon[36624]: pgmap v11221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:43 smithi081 ceph-mon[31453]: pgmap v11221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:45 smithi153 ceph-mon[36569]: pgmap v11222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:45 smithi104 ceph-mon[36624]: pgmap v11222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:45 smithi081 ceph-mon[31453]: pgmap v11222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:47 smithi153 ceph-mon[36569]: pgmap v11223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:47 smithi104 ceph-mon[36624]: pgmap v11223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:47 smithi081 ceph-mon[31453]: pgmap v11223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:49 smithi153 ceph-mon[36569]: pgmap v11224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:49 smithi104 ceph-mon[36624]: pgmap v11224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:49 smithi081 ceph-mon[31453]: pgmap v11224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:51 smithi153 ceph-mon[36569]: pgmap v11225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:51 smithi104 ceph-mon[36624]: pgmap v11225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:51 smithi081 ceph-mon[31453]: pgmap v11225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:53.571+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:00:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:53.571+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:00:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:00:53.571+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:00:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:53 smithi153 ceph-mon[36569]: pgmap v11226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:53 smithi104 ceph-mon[36624]: pgmap v11226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:53 smithi081 ceph-mon[31453]: pgmap v11226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:55 smithi153 ceph-mon[36569]: pgmap v11227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:55 smithi104 ceph-mon[36624]: pgmap v11227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:55 smithi081 ceph-mon[31453]: pgmap v11227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:57 smithi153 ceph-mon[36569]: pgmap v11228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:57 smithi104 ceph-mon[36624]: pgmap v11228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:00:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:57 smithi081 ceph-mon[31453]: pgmap v11228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:00:59 smithi153 ceph-mon[36569]: pgmap v11229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:00:59 smithi104 ceph-mon[36624]: pgmap v11229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:00:59 smithi081 ceph-mon[31453]: pgmap v11229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:01 smithi153 ceph-mon[36569]: pgmap v11230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:01 smithi104 ceph-mon[36624]: pgmap v11230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:01 smithi081 ceph-mon[31453]: pgmap v11230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:03 smithi153 ceph-mon[36569]: pgmap v11231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:03 smithi104 ceph-mon[36624]: pgmap v11231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:03 smithi081 ceph-mon[31453]: pgmap v11231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:05 smithi153 ceph-mon[36569]: pgmap v11232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:05 smithi104 ceph-mon[36624]: pgmap v11232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:05 smithi081 ceph-mon[31453]: pgmap v11232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:07 smithi153 ceph-mon[36569]: pgmap v11233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:07 smithi104 ceph-mon[36624]: pgmap v11233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:07 smithi081 ceph-mon[31453]: pgmap v11233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:08.573+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:01:08.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:08.573+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:01:08.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:08.573+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:01:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:09 smithi153 ceph-mon[36569]: pgmap v11234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:09 smithi104 ceph-mon[36624]: pgmap v11234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:09 smithi081 ceph-mon[31453]: pgmap v11234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:11 smithi153 ceph-mon[36569]: pgmap v11235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:11 smithi104 ceph-mon[36624]: pgmap v11235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:12.146+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3cf0 msgr2=0x7f16e80a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000630) for more than 900000000 us, fault. 2024-04-13T03:01:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:12.146+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3cf0 0x7f16e80a4e80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-13T03:01:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:12.146+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3cf0 msgr2=0x7f16e80a4e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T03:01:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:12.146+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3cf0 0x7f16e80a4e80 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:01:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:12.146+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:01:12.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:12.147+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:01:12.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:12.148+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b40d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f4024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:01:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:11 smithi081 ceph-mon[31453]: pgmap v11235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:14.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:13 smithi153 ceph-mon[36569]: pgmap v11236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:13 smithi104 ceph-mon[36624]: pgmap v11236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:13 smithi081 ceph-mon[31453]: pgmap v11236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:15 smithi153 ceph-mon[36569]: pgmap v11237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:15 smithi104 ceph-mon[36624]: pgmap v11237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:16.118+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:01:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:16.118+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:01:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:16.119+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 161 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:01:16.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:16.119+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 161 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:01:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:15 smithi081 ceph-mon[31453]: pgmap v11237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:17 smithi104 ceph-mon[36624]: pgmap v11238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:17 smithi081 ceph-mon[31453]: pgmap v11238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:17 smithi153 ceph-mon[36569]: pgmap v11238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:19 smithi104 ceph-mon[36624]: pgmap v11239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:19 smithi081 ceph-mon[31453]: pgmap v11239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:19 smithi153 ceph-mon[36569]: pgmap v11239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:21 smithi104 ceph-mon[36624]: pgmap v11240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:22.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:21 smithi153 ceph-mon[36569]: pgmap v11240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:21 smithi081 ceph-mon[31453]: pgmap v11240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:23.590+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:01:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:23.590+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:01:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:23.590+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:01:24.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:23 smithi153 ceph-mon[36569]: pgmap v11241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:23 smithi104 ceph-mon[36624]: pgmap v11241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:23 smithi081 ceph-mon[31453]: pgmap v11241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:25 smithi104 ceph-mon[36624]: pgmap v11242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:25 smithi081 ceph-mon[31453]: pgmap v11242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:25 smithi153 ceph-mon[36569]: pgmap v11242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:27 smithi081 ceph-mon[31453]: pgmap v11243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:27 smithi153 ceph-mon[36569]: pgmap v11243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:27 smithi104 ceph-mon[36624]: pgmap v11243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:29 smithi081 ceph-mon[31453]: pgmap v11244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:29 smithi153 ceph-mon[36569]: pgmap v11244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:29 smithi104 ceph-mon[36624]: pgmap v11244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:30 smithi081 ceph-mon[31453]: pgmap v11245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:31.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:30 smithi104 ceph-mon[36624]: pgmap v11245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:30 smithi153 ceph-mon[36569]: pgmap v11245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:33 smithi153 ceph-mon[36569]: pgmap v11246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:33 smithi104 ceph-mon[36624]: pgmap v11246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:33 smithi081 ceph-mon[31453]: pgmap v11246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:35 smithi153 ceph-mon[36569]: pgmap v11247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:35 smithi104 ceph-mon[36624]: pgmap v11247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:35 smithi081 ceph-mon[31453]: pgmap v11247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:37.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:37 smithi153 ceph-mon[36569]: pgmap v11248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:37 smithi104 ceph-mon[36624]: pgmap v11248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:37 smithi081 ceph-mon[31453]: pgmap v11248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:38.600+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:01:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:38.600+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:01:38.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:38.600+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:01:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:39 smithi081 ceph-mon[31453]: pgmap v11249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:39.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:39 smithi153 ceph-mon[36569]: pgmap v11249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:39 smithi104 ceph-mon[36624]: pgmap v11249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:41 smithi153 ceph-mon[36569]: pgmap v11250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:41.603 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:01:41.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:41 smithi104 ceph-mon[36624]: pgmap v11250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:41.604 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:01:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:41 smithi081 ceph-mon[31453]: pgmap v11250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:01:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:01:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:01:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:01:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:01:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:01:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:01:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:01:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:01:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:01:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:43 smithi104 ceph-mon[36624]: pgmap v11251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:43 smithi104 ceph-mon[36624]: pgmap v11252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:43 smithi081 ceph-mon[31453]: pgmap v11251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:43 smithi081 ceph-mon[31453]: pgmap v11252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:43 smithi153 ceph-mon[36569]: pgmap v11251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:43 smithi153 ceph-mon[36569]: pgmap v11252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:45 smithi104 ceph-mon[36624]: pgmap v11253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:45 smithi081 ceph-mon[31453]: pgmap v11253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:45 smithi153 ceph-mon[36569]: pgmap v11253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:47 smithi104 ceph-mon[36624]: pgmap v11254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:47 smithi081 ceph-mon[31453]: pgmap v11254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:47 smithi153 ceph-mon[36569]: pgmap v11254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:49 smithi104 ceph-mon[36624]: pgmap v11255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:49 smithi081 ceph-mon[31453]: pgmap v11255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:49 smithi153 ceph-mon[36569]: pgmap v11255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:51 smithi104 ceph-mon[36624]: pgmap v11256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:51 smithi081 ceph-mon[31453]: pgmap v11256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:51 smithi153 ceph-mon[36569]: pgmap v11256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:53.610 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:53 smithi153 ceph-mon[36569]: pgmap v11257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:53.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:53.610+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:01:53.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:53.610+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:01:53.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:01:53.610+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:01:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:53 smithi081 ceph-mon[31453]: pgmap v11257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:53 smithi104 ceph-mon[36624]: pgmap v11257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:55 smithi081 ceph-mon[31453]: pgmap v11258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:55 smithi153 ceph-mon[36569]: pgmap v11258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:55 smithi104 ceph-mon[36624]: pgmap v11258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:57 smithi081 ceph-mon[31453]: pgmap v11259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:57 smithi153 ceph-mon[36569]: pgmap v11259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:57 smithi104 ceph-mon[36624]: pgmap v11259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:01:59 smithi081 ceph-mon[31453]: pgmap v11260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:01:59 smithi153 ceph-mon[36569]: pgmap v11260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:01:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:01:59 smithi104 ceph-mon[36624]: pgmap v11260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:01.638 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:01 smithi104 ceph-mon[36624]: pgmap v11261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:01 smithi081 ceph-mon[31453]: pgmap v11261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:01 smithi153 ceph-mon[36569]: pgmap v11261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:03 smithi081 ceph-mon[31453]: pgmap v11262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:03 smithi153 ceph-mon[36569]: pgmap v11262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:03 smithi104 ceph-mon[36624]: pgmap v11262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:05 smithi081 ceph-mon[31453]: pgmap v11263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:05 smithi153 ceph-mon[36569]: pgmap v11263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:05 smithi104 ceph-mon[36624]: pgmap v11263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:07 smithi081 ceph-mon[31453]: pgmap v11264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:07 smithi153 ceph-mon[36569]: pgmap v11264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:07 smithi104 ceph-mon[36624]: pgmap v11264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:08.614+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:02:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:08.614+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:02:08.615 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:08.614+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:02:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:09 smithi081 ceph-mon[31453]: pgmap v11265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:09 smithi153 ceph-mon[36569]: pgmap v11265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:09 smithi104 ceph-mon[36624]: pgmap v11265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:11 smithi081 ceph-mon[31453]: pgmap v11266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:11 smithi153 ceph-mon[36569]: pgmap v11266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:11 smithi104 ceph-mon[36624]: pgmap v11266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:13 smithi081 ceph-mon[31453]: pgmap v11267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:13 smithi153 ceph-mon[36569]: pgmap v11267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:13 smithi104 ceph-mon[36624]: pgmap v11267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:15 smithi081 ceph-mon[31453]: pgmap v11268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:15 smithi153 ceph-mon[36569]: pgmap v11268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:15 smithi104 ceph-mon[36624]: pgmap v11268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:17.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:17 smithi081 ceph-mon[31453]: pgmap v11269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:17.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:17 smithi153 ceph-mon[36569]: pgmap v11269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:17.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:17 smithi104 ceph-mon[36624]: pgmap v11269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:19 smithi153 ceph-mon[36569]: pgmap v11270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:19 smithi104 ceph-mon[36624]: pgmap v11270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:19 smithi081 ceph-mon[31453]: pgmap v11270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:21 smithi153 ceph-mon[36569]: pgmap v11271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:21 smithi104 ceph-mon[36624]: pgmap v11271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:21 smithi081 ceph-mon[31453]: pgmap v11271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:23.626+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:02:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:23.626+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:02:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:23.626+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:02:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:23 smithi153 ceph-mon[36569]: pgmap v11272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:23 smithi104 ceph-mon[36624]: pgmap v11272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:23 smithi081 ceph-mon[31453]: pgmap v11272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:25 smithi153 ceph-mon[36569]: pgmap v11273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:25 smithi104 ceph-mon[36624]: pgmap v11273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:25 smithi081 ceph-mon[31453]: pgmap v11273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:27.713 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:27 smithi081 ceph-mon[31453]: pgmap v11274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:27 smithi153 ceph-mon[36569]: pgmap v11274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:27 smithi104 ceph-mon[36624]: pgmap v11274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:29 smithi153 ceph-mon[36569]: pgmap v11275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:29 smithi104 ceph-mon[36624]: pgmap v11275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:29 smithi081 ceph-mon[31453]: pgmap v11275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:31 smithi153 ceph-mon[36569]: pgmap v11276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:31 smithi104 ceph-mon[36624]: pgmap v11276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:31 smithi081 ceph-mon[31453]: pgmap v11276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:33.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:33 smithi153 ceph-mon[36569]: pgmap v11277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:33 smithi104 ceph-mon[36624]: pgmap v11277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:33 smithi081 ceph-mon[31453]: pgmap v11277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:35 smithi153 ceph-mon[36569]: pgmap v11278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:35 smithi104 ceph-mon[36624]: pgmap v11278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:35 smithi081 ceph-mon[31453]: pgmap v11278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:37 smithi153 ceph-mon[36569]: pgmap v11279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:37 smithi104 ceph-mon[36624]: pgmap v11279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:37 smithi081 ceph-mon[31453]: pgmap v11279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:38.643+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:02:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:38.643+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:02:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:38.643+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:02:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:39 smithi153 ceph-mon[36569]: pgmap v11280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:39 smithi104 ceph-mon[36624]: pgmap v11280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:39 smithi081 ceph-mon[31453]: pgmap v11280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:41 smithi153 ceph-mon[36569]: pgmap v11281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:41 smithi104 ceph-mon[36624]: pgmap v11281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:41 smithi081 ceph-mon[31453]: pgmap v11281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:02:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:02:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:02:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:43 smithi153 ceph-mon[36569]: pgmap v11282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:02:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:02:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:43 smithi104 ceph-mon[36624]: pgmap v11282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:02:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:02:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:43 smithi081 ceph-mon[31453]: pgmap v11282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:02:43.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:02:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:44 smithi153 ceph-mon[36569]: pgmap v11283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:44 smithi153 ceph-mon[36569]: pgmap v11284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:02:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:44 smithi104 ceph-mon[36624]: pgmap v11283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:44 smithi104 ceph-mon[36624]: pgmap v11284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:02:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:44 smithi081 ceph-mon[31453]: pgmap v11283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:44 smithi081 ceph-mon[31453]: pgmap v11284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:02:46.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:46.133+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:02:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:46.133+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:02:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:46.133+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 162 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:02:46.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:46.133+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 162 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:02:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:46 smithi153 ceph-mon[36569]: pgmap v11285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:46 smithi104 ceph-mon[36624]: pgmap v11285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:46 smithi081 ceph-mon[31453]: pgmap v11285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:48 smithi153 ceph-mon[36569]: pgmap v11286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:48 smithi104 ceph-mon[36624]: pgmap v11286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:48 smithi081 ceph-mon[31453]: pgmap v11286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:50 smithi153 ceph-mon[36569]: pgmap v11287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:50 smithi104 ceph-mon[36624]: pgmap v11287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:50 smithi081 ceph-mon[31453]: pgmap v11287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:52 smithi153 ceph-mon[36569]: pgmap v11288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:52 smithi104 ceph-mon[36624]: pgmap v11288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:52 smithi081 ceph-mon[31453]: pgmap v11288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:53.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:53.659+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:02:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:53.659+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:02:53.661 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:02:53.659+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:02:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:54 smithi153 ceph-mon[36569]: pgmap v11289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:54 smithi104 ceph-mon[36624]: pgmap v11289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:54 smithi081 ceph-mon[31453]: pgmap v11289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:56 smithi153 ceph-mon[36569]: pgmap v11290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:56 smithi104 ceph-mon[36624]: pgmap v11290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:56 smithi081 ceph-mon[31453]: pgmap v11290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:58.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:02:58 smithi153 ceph-mon[36569]: pgmap v11291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:02:58 smithi104 ceph-mon[36624]: pgmap v11291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:02:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:02:58 smithi081 ceph-mon[31453]: pgmap v11291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:00.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:00 smithi153 ceph-mon[36569]: pgmap v11292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:00 smithi104 ceph-mon[36624]: pgmap v11292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:00 smithi081 ceph-mon[31453]: pgmap v11292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:02.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:02 smithi153 ceph-mon[36569]: pgmap v11293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:02 smithi104 ceph-mon[36624]: pgmap v11293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:02 smithi081 ceph-mon[31453]: pgmap v11293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:04.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:04 smithi153 ceph-mon[36569]: pgmap v11294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:04 smithi104 ceph-mon[36624]: pgmap v11294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:04.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:04 smithi081 ceph-mon[31453]: pgmap v11294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:06 smithi153 ceph-mon[36569]: pgmap v11295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:06 smithi104 ceph-mon[36624]: pgmap v11295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:06 smithi081 ceph-mon[31453]: pgmap v11295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:08.675+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:03:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:08.675+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:03:08.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:08.675+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:03:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:08 smithi153 ceph-mon[36569]: pgmap v11296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:08 smithi104 ceph-mon[36624]: pgmap v11296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:08 smithi081 ceph-mon[31453]: pgmap v11296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:10.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:10 smithi104 ceph-mon[36624]: pgmap v11297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:10 smithi081 ceph-mon[31453]: pgmap v11297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:10 smithi153 ceph-mon[36569]: pgmap v11297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:12 smithi104 ceph-mon[36624]: pgmap v11298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:12 smithi081 ceph-mon[31453]: pgmap v11298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:12 smithi153 ceph-mon[36569]: pgmap v11298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:14 smithi104 ceph-mon[36624]: pgmap v11299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:14 smithi081 ceph-mon[31453]: pgmap v11299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:14 smithi153 ceph-mon[36569]: pgmap v11299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:16 smithi153 ceph-mon[36569]: pgmap v11300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:16 smithi104 ceph-mon[36624]: pgmap v11300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:16 smithi081 ceph-mon[31453]: pgmap v11300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:18 smithi153 ceph-mon[36569]: pgmap v11301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:18 smithi104 ceph-mon[36624]: pgmap v11301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:18 smithi081 ceph-mon[31453]: pgmap v11301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:20 smithi104 ceph-mon[36624]: pgmap v11302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:20 smithi081 ceph-mon[31453]: pgmap v11302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:20 smithi153 ceph-mon[36569]: pgmap v11302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:22 smithi104 ceph-mon[36624]: pgmap v11303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:22 smithi081 ceph-mon[31453]: pgmap v11303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:22 smithi153 ceph-mon[36569]: pgmap v11303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:23.690+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:03:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:23.690+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:03:23.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:23.690+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:03:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:24 smithi104 ceph-mon[36624]: pgmap v11304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:24.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:24 smithi153 ceph-mon[36569]: pgmap v11304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:24.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:24 smithi081 ceph-mon[31453]: pgmap v11304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:26 smithi104 ceph-mon[36624]: pgmap v11305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:26 smithi081 ceph-mon[31453]: pgmap v11305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:26 smithi153 ceph-mon[36569]: pgmap v11305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:28 smithi104 ceph-mon[36624]: pgmap v11306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:28.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:28 smithi081 ceph-mon[31453]: pgmap v11306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:28 smithi153 ceph-mon[36569]: pgmap v11306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:30 smithi081 ceph-mon[31453]: pgmap v11307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:30 smithi153 ceph-mon[36569]: pgmap v11307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:30 smithi104 ceph-mon[36624]: pgmap v11307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:32.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:32 smithi104 ceph-mon[36624]: pgmap v11308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:32 smithi081 ceph-mon[31453]: pgmap v11308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:32 smithi153 ceph-mon[36569]: pgmap v11308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:34 smithi081 ceph-mon[31453]: pgmap v11309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:34 smithi153 ceph-mon[36569]: pgmap v11309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:34 smithi104 ceph-mon[36624]: pgmap v11309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:36 smithi081 ceph-mon[31453]: pgmap v11310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:36 smithi153 ceph-mon[36569]: pgmap v11310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:36 smithi104 ceph-mon[36624]: pgmap v11310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:38.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:38.702+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:03:38.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:38.702+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:03:38.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:38.702+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:03:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:38 smithi081 ceph-mon[31453]: pgmap v11311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:38 smithi153 ceph-mon[36569]: pgmap v11311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:38 smithi104 ceph-mon[36624]: pgmap v11311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:40 smithi081 ceph-mon[31453]: pgmap v11312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:40 smithi153 ceph-mon[36569]: pgmap v11312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:40 smithi104 ceph-mon[36624]: pgmap v11312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:42 smithi081 ceph-mon[31453]: pgmap v11313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:42 smithi153 ceph-mon[36569]: pgmap v11313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:42 smithi104 ceph-mon[36624]: pgmap v11313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:44 smithi081 ceph-mon[31453]: pgmap v11314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:03:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:44 smithi153 ceph-mon[36569]: pgmap v11314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:03:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:44 smithi104 ceph-mon[36624]: pgmap v11314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:03:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:03:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:03:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:03:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:03:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:03:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:03:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:03:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:03:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:03:46.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:46 smithi081 ceph-mon[31453]: pgmap v11315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:46.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:46 smithi081 ceph-mon[31453]: pgmap v11316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:46 smithi153 ceph-mon[36569]: pgmap v11315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:46 smithi153 ceph-mon[36569]: pgmap v11316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:46 smithi104 ceph-mon[36624]: pgmap v11315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:46 smithi104 ceph-mon[36624]: pgmap v11316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:48 smithi081 ceph-mon[31453]: pgmap v11317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:48 smithi153 ceph-mon[36569]: pgmap v11317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:48 smithi104 ceph-mon[36624]: pgmap v11317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:50 smithi081 ceph-mon[31453]: pgmap v11318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:50 smithi153 ceph-mon[36569]: pgmap v11318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:50 smithi104 ceph-mon[36624]: pgmap v11318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:52 smithi081 ceph-mon[31453]: pgmap v11319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:52 smithi153 ceph-mon[36569]: pgmap v11319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:52 smithi104 ceph-mon[36624]: pgmap v11319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:53.704+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:03:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:53.705+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:03:53.706 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:03:53.705+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:03:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:54 smithi081 ceph-mon[31453]: pgmap v11320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:54 smithi153 ceph-mon[36569]: pgmap v11320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:54 smithi104 ceph-mon[36624]: pgmap v11320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:56 smithi153 ceph-mon[36569]: pgmap v11321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:56 smithi104 ceph-mon[36624]: pgmap v11321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:56 smithi081 ceph-mon[31453]: pgmap v11321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:03:58 smithi153 ceph-mon[36569]: pgmap v11322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:03:58 smithi104 ceph-mon[36624]: pgmap v11322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:03:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:03:58 smithi081 ceph-mon[31453]: pgmap v11322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:00 smithi153 ceph-mon[36569]: pgmap v11323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:00 smithi104 ceph-mon[36624]: pgmap v11323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:00 smithi081 ceph-mon[31453]: pgmap v11323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:02 smithi153 ceph-mon[36569]: pgmap v11324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:02 smithi104 ceph-mon[36624]: pgmap v11324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:02 smithi081 ceph-mon[31453]: pgmap v11324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:04 smithi153 ceph-mon[36569]: pgmap v11325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:04 smithi104 ceph-mon[36624]: pgmap v11325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:04 smithi081 ceph-mon[31453]: pgmap v11325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:06 smithi153 ceph-mon[36569]: pgmap v11326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:06 smithi104 ceph-mon[36624]: pgmap v11326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:06 smithi081 ceph-mon[31453]: pgmap v11326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:08.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:08.706+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:04:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:08.706+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:04:08.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:08.706+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:04:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:08 smithi153 ceph-mon[36569]: pgmap v11327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:08 smithi104 ceph-mon[36624]: pgmap v11327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:08 smithi081 ceph-mon[31453]: pgmap v11327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:10 smithi153 ceph-mon[36569]: pgmap v11328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:10 smithi104 ceph-mon[36624]: pgmap v11328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:10 smithi081 ceph-mon[31453]: pgmap v11328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:12 smithi153 ceph-mon[36569]: pgmap v11329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:12 smithi104 ceph-mon[36624]: pgmap v11329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:12 smithi081 ceph-mon[31453]: pgmap v11329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:14 smithi153 ceph-mon[36569]: pgmap v11330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:14 smithi104 ceph-mon[36624]: pgmap v11330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:14 smithi081 ceph-mon[31453]: pgmap v11330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:16.147+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:04:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:16.147+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:04:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:16.148+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 163 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:04:16.149 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:16.148+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 163 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:04:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:16 smithi153 ceph-mon[36569]: pgmap v11331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:16 smithi104 ceph-mon[36624]: pgmap v11331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:16 smithi081 ceph-mon[31453]: pgmap v11331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:18 smithi153 ceph-mon[36569]: pgmap v11332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:18 smithi104 ceph-mon[36624]: pgmap v11332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:18 smithi081 ceph-mon[31453]: pgmap v11332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:20 smithi153 ceph-mon[36569]: pgmap v11333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:20 smithi104 ceph-mon[36624]: pgmap v11333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:20 smithi081 ceph-mon[31453]: pgmap v11333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:22 smithi153 ceph-mon[36569]: pgmap v11334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:22 smithi104 ceph-mon[36624]: pgmap v11334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:22 smithi081 ceph-mon[31453]: pgmap v11334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:23.722+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:04:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:23.722+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:04:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:23.722+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:04:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:24 smithi153 ceph-mon[36569]: pgmap v11335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:24 smithi104 ceph-mon[36624]: pgmap v11335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:24 smithi081 ceph-mon[31453]: pgmap v11335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:26 smithi153 ceph-mon[36569]: pgmap v11336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:26 smithi104 ceph-mon[36624]: pgmap v11336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:26 smithi081 ceph-mon[31453]: pgmap v11336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:28 smithi153 ceph-mon[36569]: pgmap v11337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:28 smithi104 ceph-mon[36624]: pgmap v11337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:28.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:28 smithi081 ceph-mon[31453]: pgmap v11337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:30 smithi081 ceph-mon[31453]: pgmap v11338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:30 smithi153 ceph-mon[36569]: pgmap v11338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:30 smithi104 ceph-mon[36624]: pgmap v11338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:32 smithi153 ceph-mon[36569]: pgmap v11339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:32 smithi104 ceph-mon[36624]: pgmap v11339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:32 smithi081 ceph-mon[31453]: pgmap v11339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:34 smithi153 ceph-mon[36569]: pgmap v11340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:34 smithi104 ceph-mon[36624]: pgmap v11340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:34 smithi081 ceph-mon[31453]: pgmap v11340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:36 smithi153 ceph-mon[36569]: pgmap v11341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:36 smithi104 ceph-mon[36624]: pgmap v11341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:36 smithi081 ceph-mon[31453]: pgmap v11341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:38 smithi153 ceph-mon[36569]: pgmap v11342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:38 smithi104 ceph-mon[36624]: pgmap v11342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:38.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:38 smithi081 ceph-mon[31453]: pgmap v11342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:38.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:38.737+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:04:38.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:38.737+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:04:38.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:38.737+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:04:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:40 smithi153 ceph-mon[36569]: pgmap v11343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:40 smithi104 ceph-mon[36624]: pgmap v11343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:40.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:40 smithi081 ceph-mon[31453]: pgmap v11343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:42 smithi153 ceph-mon[36569]: pgmap v11344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:42 smithi104 ceph-mon[36624]: pgmap v11344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:42 smithi081 ceph-mon[31453]: pgmap v11344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:44 smithi153 ceph-mon[36569]: pgmap v11345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:44 smithi104 ceph-mon[36624]: pgmap v11345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:44 smithi081 ceph-mon[31453]: pgmap v11345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:46 smithi153 ceph-mon[36569]: pgmap v11346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:04:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:46 smithi104 ceph-mon[36624]: pgmap v11346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:04:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:46 smithi081 ceph-mon[31453]: pgmap v11346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:04:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:04:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:04:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:47 smithi153 ceph-mon[36569]: pgmap v11347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:47 smithi153 ceph-mon[36569]: pgmap v11348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:04:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:04:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:04:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:47 smithi104 ceph-mon[36624]: pgmap v11347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:47 smithi104 ceph-mon[36624]: pgmap v11348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:04:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:04:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:04:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:47 smithi081 ceph-mon[31453]: pgmap v11347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:47 smithi081 ceph-mon[31453]: pgmap v11348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:04:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:49 smithi153 ceph-mon[36569]: pgmap v11349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:49 smithi104 ceph-mon[36624]: pgmap v11349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:49 smithi081 ceph-mon[31453]: pgmap v11349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:51 smithi153 ceph-mon[36569]: pgmap v11350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:51 smithi104 ceph-mon[36624]: pgmap v11350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:51 smithi081 ceph-mon[31453]: pgmap v11350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:53.753+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:04:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:53.753+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:04:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:04:53.753+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:04:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:53 smithi153 ceph-mon[36569]: pgmap v11351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:53 smithi104 ceph-mon[36624]: pgmap v11351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:53 smithi081 ceph-mon[31453]: pgmap v11351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:55 smithi153 ceph-mon[36569]: pgmap v11352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:55 smithi104 ceph-mon[36624]: pgmap v11352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:55 smithi081 ceph-mon[31453]: pgmap v11352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:57 smithi153 ceph-mon[36569]: pgmap v11353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:57 smithi104 ceph-mon[36624]: pgmap v11353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:04:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:57 smithi081 ceph-mon[31453]: pgmap v11353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:04:59 smithi153 ceph-mon[36569]: pgmap v11354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:04:59 smithi104 ceph-mon[36624]: pgmap v11354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:04:59 smithi081 ceph-mon[31453]: pgmap v11354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:01 smithi153 ceph-mon[36569]: pgmap v11355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:01 smithi104 ceph-mon[36624]: pgmap v11355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:01 smithi081 ceph-mon[31453]: pgmap v11355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:03 smithi153 ceph-mon[36569]: pgmap v11356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:03 smithi104 ceph-mon[36624]: pgmap v11356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:03 smithi081 ceph-mon[31453]: pgmap v11356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:05 smithi153 ceph-mon[36569]: pgmap v11357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:05 smithi104 ceph-mon[36624]: pgmap v11357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:05 smithi081 ceph-mon[31453]: pgmap v11357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:07 smithi153 ceph-mon[36569]: pgmap v11358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:07 smithi104 ceph-mon[36624]: pgmap v11358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:07 smithi081 ceph-mon[31453]: pgmap v11358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:08.764 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:08.763+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:05:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:08.763+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:05:08.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:08.763+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:05:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:09 smithi153 ceph-mon[36569]: pgmap v11359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:09 smithi104 ceph-mon[36624]: pgmap v11359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:09 smithi081 ceph-mon[31453]: pgmap v11359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:11 smithi153 ceph-mon[36569]: pgmap v11360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:11 smithi104 ceph-mon[36624]: pgmap v11360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:11 smithi081 ceph-mon[31453]: pgmap v11360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:13 smithi153 ceph-mon[36569]: pgmap v11361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:13 smithi104 ceph-mon[36624]: pgmap v11361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:13 smithi081 ceph-mon[31453]: pgmap v11361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:15 smithi153 ceph-mon[36569]: pgmap v11362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:15 smithi104 ceph-mon[36624]: pgmap v11362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:15 smithi081 ceph-mon[31453]: pgmap v11362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:17 smithi153 ceph-mon[36569]: pgmap v11363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:17 smithi104 ceph-mon[36624]: pgmap v11363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:17 smithi081 ceph-mon[31453]: pgmap v11363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:19 smithi153 ceph-mon[36569]: pgmap v11364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:19 smithi104 ceph-mon[36624]: pgmap v11364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:19 smithi081 ceph-mon[31453]: pgmap v11364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:21 smithi153 ceph-mon[36569]: pgmap v11365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:21 smithi104 ceph-mon[36624]: pgmap v11365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:21 smithi081 ceph-mon[31453]: pgmap v11365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:23.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:23.767+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:05:23.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:23.767+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:05:23.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:23.767+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:05:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:23 smithi153 ceph-mon[36569]: pgmap v11366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:23 smithi104 ceph-mon[36624]: pgmap v11366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:23 smithi081 ceph-mon[31453]: pgmap v11366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:25 smithi104 ceph-mon[36624]: pgmap v11367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:25 smithi081 ceph-mon[31453]: pgmap v11367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:25 smithi153 ceph-mon[36569]: pgmap v11367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:27 smithi104 ceph-mon[36624]: pgmap v11368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:27 smithi081 ceph-mon[31453]: pgmap v11368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:27 smithi153 ceph-mon[36569]: pgmap v11368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:29 smithi104 ceph-mon[36624]: pgmap v11369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:29 smithi081 ceph-mon[31453]: pgmap v11369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:29 smithi153 ceph-mon[36569]: pgmap v11369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:31 smithi104 ceph-mon[36624]: pgmap v11370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:31 smithi081 ceph-mon[31453]: pgmap v11370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:31 smithi153 ceph-mon[36569]: pgmap v11370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:33 smithi104 ceph-mon[36624]: pgmap v11371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:33 smithi081 ceph-mon[31453]: pgmap v11371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:33 smithi153 ceph-mon[36569]: pgmap v11371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:35 smithi081 ceph-mon[31453]: pgmap v11372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:35 smithi153 ceph-mon[36569]: pgmap v11372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:35 smithi104 ceph-mon[36624]: pgmap v11372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:37 smithi081 ceph-mon[31453]: pgmap v11373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:37 smithi153 ceph-mon[36569]: pgmap v11373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:37 smithi104 ceph-mon[36624]: pgmap v11373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:38.774+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:05:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:38.774+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:05:38.776 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:38.775+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:05:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:39 smithi081 ceph-mon[31453]: pgmap v11374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:39 smithi153 ceph-mon[36569]: pgmap v11374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:39 smithi104 ceph-mon[36624]: pgmap v11374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:41 smithi081 ceph-mon[31453]: pgmap v11375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:41 smithi153 ceph-mon[36569]: pgmap v11375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:41 smithi104 ceph-mon[36624]: pgmap v11375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:43 smithi081 ceph-mon[31453]: pgmap v11376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:43 smithi153 ceph-mon[36569]: pgmap v11376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:43 smithi104 ceph-mon[36624]: pgmap v11376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:45 smithi081 ceph-mon[31453]: pgmap v11377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:45 smithi153 ceph-mon[36569]: pgmap v11377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:45 smithi104 ceph-mon[36624]: pgmap v11377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: pgmap v11378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:05:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: pgmap v11378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: pgmap v11378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:05:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:05:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:05:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:05:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:05:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:05:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:05:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:05:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:05:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:05:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:05:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:05:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:49 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:05:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:49 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:05:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:49 smithi081 ceph-mon[31453]: pgmap v11379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:49 smithi081 ceph-mon[31453]: pgmap v11380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:49 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:05:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:49 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:05:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:49 smithi153 ceph-mon[36569]: pgmap v11379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:49 smithi153 ceph-mon[36569]: pgmap v11380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:49 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:05:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:49 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:05:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:49 smithi104 ceph-mon[36624]: pgmap v11379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:49 smithi104 ceph-mon[36624]: pgmap v11380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:51 smithi081 ceph-mon[31453]: pgmap v11381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:51 smithi153 ceph-mon[36569]: pgmap v11381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:51 smithi104 ceph-mon[36624]: pgmap v11381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:53.790+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:53.790+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:05:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:05:53.790+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:05:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:53 smithi081 ceph-mon[31453]: pgmap v11382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:53 smithi153 ceph-mon[36569]: pgmap v11382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:53 smithi104 ceph-mon[36624]: pgmap v11382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:55.209 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:54 smithi153 ceph-mon[36569]: pgmap v11383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:54 smithi104 ceph-mon[36624]: pgmap v11383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:54 smithi081 ceph-mon[31453]: pgmap v11383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:57 smithi081 ceph-mon[31453]: pgmap v11384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:57 smithi153 ceph-mon[36569]: pgmap v11384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:57 smithi104 ceph-mon[36624]: pgmap v11384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:05:59 smithi081 ceph-mon[31453]: pgmap v11385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:05:59 smithi153 ceph-mon[36569]: pgmap v11385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:05:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:05:59 smithi104 ceph-mon[36624]: pgmap v11385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:01 smithi081 ceph-mon[31453]: pgmap v11386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:01 smithi153 ceph-mon[36569]: pgmap v11386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:01 smithi104 ceph-mon[36624]: pgmap v11386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:03.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:03 smithi104 ceph-mon[36624]: pgmap v11387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:03 smithi081 ceph-mon[31453]: pgmap v11387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:03 smithi153 ceph-mon[36569]: pgmap v11387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:05 smithi081 ceph-mon[31453]: pgmap v11388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:05 smithi153 ceph-mon[36569]: pgmap v11388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:05 smithi104 ceph-mon[36624]: pgmap v11388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:07 smithi081 ceph-mon[31453]: pgmap v11389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:07 smithi153 ceph-mon[36569]: pgmap v11389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:07 smithi104 ceph-mon[36624]: pgmap v11389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:08.804+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:06:08.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:08.804+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:06:08.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:08.804+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:06:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:09 smithi081 ceph-mon[31453]: pgmap v11390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:09 smithi153 ceph-mon[36569]: pgmap v11390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:09 smithi104 ceph-mon[36624]: pgmap v11390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:11 smithi081 ceph-mon[31453]: pgmap v11391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:11 smithi153 ceph-mon[36569]: pgmap v11391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:11 smithi104 ceph-mon[36624]: pgmap v11391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:13 smithi081 ceph-mon[31453]: pgmap v11392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:13.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:13 smithi153 ceph-mon[36569]: pgmap v11392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:13 smithi104 ceph-mon[36624]: pgmap v11392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:15 smithi153 ceph-mon[36569]: pgmap v11393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:15 smithi104 ceph-mon[36624]: pgmap v11393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:15 smithi081 ceph-mon[31453]: pgmap v11393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:17 smithi153 ceph-mon[36569]: pgmap v11394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:17 smithi104 ceph-mon[36624]: pgmap v11394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:17 smithi081 ceph-mon[31453]: pgmap v11394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:19 smithi153 ceph-mon[36569]: pgmap v11395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:19 smithi104 ceph-mon[36624]: pgmap v11395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:19 smithi081 ceph-mon[31453]: pgmap v11395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:21 smithi153 ceph-mon[36569]: pgmap v11396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:21 smithi104 ceph-mon[36624]: pgmap v11396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:21 smithi081 ceph-mon[31453]: pgmap v11396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:23 smithi153 ceph-mon[36569]: pgmap v11397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:23.814+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:06:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:23.814+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:06:23.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:23.814+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:06:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:23 smithi104 ceph-mon[36624]: pgmap v11397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:23 smithi081 ceph-mon[31453]: pgmap v11397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:25 smithi153 ceph-mon[36569]: pgmap v11398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:25 smithi104 ceph-mon[36624]: pgmap v11398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:25 smithi081 ceph-mon[31453]: pgmap v11398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:27 smithi153 ceph-mon[36569]: pgmap v11399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:27 smithi104 ceph-mon[36624]: pgmap v11399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:27 smithi081 ceph-mon[31453]: pgmap v11399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:29 smithi153 ceph-mon[36569]: pgmap v11400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:29 smithi104 ceph-mon[36624]: pgmap v11400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:29 smithi081 ceph-mon[31453]: pgmap v11400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:31.710 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:31 smithi081 ceph-mon[31453]: pgmap v11401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:31 smithi153 ceph-mon[36569]: pgmap v11401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:31 smithi104 ceph-mon[36624]: pgmap v11401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:33 smithi153 ceph-mon[36569]: pgmap v11402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:33 smithi104 ceph-mon[36624]: pgmap v11402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:33 smithi081 ceph-mon[31453]: pgmap v11402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:35 smithi153 ceph-mon[36569]: pgmap v11403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:35 smithi104 ceph-mon[36624]: pgmap v11403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:35 smithi081 ceph-mon[31453]: pgmap v11403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:37 smithi153 ceph-mon[36569]: pgmap v11404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:37 smithi104 ceph-mon[36624]: pgmap v11404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:37 smithi081 ceph-mon[31453]: pgmap v11404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:38.824+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:06:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:38.824+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:06:38.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:38.824+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:06:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:39 smithi153 ceph-mon[36569]: pgmap v11405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:39 smithi104 ceph-mon[36624]: pgmap v11405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:39 smithi081 ceph-mon[31453]: pgmap v11405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:41 smithi081 ceph-mon[31453]: pgmap v11406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:41 smithi153 ceph-mon[36569]: pgmap v11406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:41 smithi104 ceph-mon[36624]: pgmap v11406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:43 smithi081 ceph-mon[31453]: pgmap v11407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:43 smithi153 ceph-mon[36569]: pgmap v11407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:43 smithi104 ceph-mon[36624]: pgmap v11407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:45 smithi081 ceph-mon[31453]: pgmap v11408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:45 smithi153 ceph-mon[36569]: pgmap v11408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:45 smithi104 ceph-mon[36624]: pgmap v11408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:47 smithi081 ceph-mon[31453]: pgmap v11409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:47 smithi153 ceph-mon[36569]: pgmap v11409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:47 smithi104 ceph-mon[36624]: pgmap v11409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:49 smithi081 ceph-mon[31453]: pgmap v11410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:06:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:49 smithi153 ceph-mon[36569]: pgmap v11410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:06:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:49 smithi104 ceph-mon[36624]: pgmap v11410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:06:50.428 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:06:50.428 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:06:50.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:50 smithi081 ceph-mon[31453]: pgmap v11411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:50.429 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:50 smithi081 ceph-mon[31453]: pgmap v11412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:06:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:06:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:50 smithi153 ceph-mon[36569]: pgmap v11411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:50 smithi153 ceph-mon[36569]: pgmap v11412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:06:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:06:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:06:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:50 smithi104 ceph-mon[36624]: pgmap v11411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:50 smithi104 ceph-mon[36624]: pgmap v11412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:06:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:06:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:52 smithi081 ceph-mon[31453]: pgmap v11413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:52 smithi153 ceph-mon[36569]: pgmap v11413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:52 smithi104 ceph-mon[36624]: pgmap v11413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:53.837+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:06:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:53.837+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:06:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:06:53.837+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:06:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:54 smithi081 ceph-mon[31453]: pgmap v11414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:54 smithi153 ceph-mon[36569]: pgmap v11414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:54 smithi104 ceph-mon[36624]: pgmap v11414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:56 smithi081 ceph-mon[31453]: pgmap v11415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:56 smithi153 ceph-mon[36569]: pgmap v11415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:56 smithi104 ceph-mon[36624]: pgmap v11415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:06:58 smithi081 ceph-mon[31453]: pgmap v11416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:06:58 smithi153 ceph-mon[36569]: pgmap v11416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:06:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:06:58 smithi104 ceph-mon[36624]: pgmap v11416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:00 smithi081 ceph-mon[31453]: pgmap v11417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:00 smithi153 ceph-mon[36569]: pgmap v11417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:00 smithi104 ceph-mon[36624]: pgmap v11417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:02 smithi153 ceph-mon[36569]: pgmap v11418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:02 smithi104 ceph-mon[36624]: pgmap v11418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:02 smithi081 ceph-mon[31453]: pgmap v11418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:03 smithi153 ceph-mon[36569]: pgmap v11419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:03 smithi104 ceph-mon[36624]: pgmap v11419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:03 smithi081 ceph-mon[31453]: pgmap v11419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:06 smithi081 ceph-mon[31453]: pgmap v11420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:06 smithi153 ceph-mon[36569]: pgmap v11420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:06 smithi104 ceph-mon[36624]: pgmap v11420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:08.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:08.846+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:07:08.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:08.846+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:07:08.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:08.846+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:07:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:08 smithi081 ceph-mon[31453]: pgmap v11421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:08 smithi153 ceph-mon[36569]: pgmap v11421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:08 smithi104 ceph-mon[36624]: pgmap v11421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:10 smithi081 ceph-mon[31453]: pgmap v11422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:10 smithi153 ceph-mon[36569]: pgmap v11422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:10 smithi104 ceph-mon[36624]: pgmap v11422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:12 smithi081 ceph-mon[31453]: pgmap v11423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:12 smithi153 ceph-mon[36569]: pgmap v11423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:12 smithi104 ceph-mon[36624]: pgmap v11423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:14 smithi153 ceph-mon[36569]: pgmap v11424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:14 smithi104 ceph-mon[36624]: pgmap v11424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:14 smithi081 ceph-mon[31453]: pgmap v11424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:16.177+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:07:16.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:16.177+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:07:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:16.178+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 164 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:07:16.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:16.178+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 164 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:07:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:16 smithi153 ceph-mon[36569]: pgmap v11425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:16 smithi104 ceph-mon[36624]: pgmap v11425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:16 smithi081 ceph-mon[31453]: pgmap v11425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:18.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:17 smithi153 ceph-mon[36569]: pgmap v11426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:17 smithi104 ceph-mon[36624]: pgmap v11426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:17 smithi081 ceph-mon[31453]: pgmap v11426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:20 smithi081 ceph-mon[31453]: pgmap v11427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:20 smithi153 ceph-mon[36569]: pgmap v11427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:20 smithi104 ceph-mon[36624]: pgmap v11427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:22 smithi081 ceph-mon[31453]: pgmap v11428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:22 smithi153 ceph-mon[36569]: pgmap v11428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:22 smithi104 ceph-mon[36624]: pgmap v11428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:23.862+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:07:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:23.862+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:07:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:23.862+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:07:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:24 smithi153 ceph-mon[36569]: pgmap v11429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:24 smithi104 ceph-mon[36624]: pgmap v11429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:24 smithi081 ceph-mon[31453]: pgmap v11429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:26 smithi081 ceph-mon[31453]: pgmap v11430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:26 smithi153 ceph-mon[36569]: pgmap v11430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:26 smithi104 ceph-mon[36624]: pgmap v11430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:28 smithi081 ceph-mon[31453]: pgmap v11431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:28 smithi153 ceph-mon[36569]: pgmap v11431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:28 smithi104 ceph-mon[36624]: pgmap v11431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:30 smithi081 ceph-mon[31453]: pgmap v11432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:30 smithi153 ceph-mon[36569]: pgmap v11432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:30 smithi104 ceph-mon[36624]: pgmap v11432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:32 smithi153 ceph-mon[36569]: pgmap v11433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:32 smithi104 ceph-mon[36624]: pgmap v11433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:32 smithi081 ceph-mon[31453]: pgmap v11433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:34 smithi153 ceph-mon[36569]: pgmap v11434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:34 smithi104 ceph-mon[36624]: pgmap v11434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:34 smithi081 ceph-mon[31453]: pgmap v11434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:35 smithi153 ceph-mon[36569]: pgmap v11435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:35 smithi104 ceph-mon[36624]: pgmap v11435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:35 smithi081 ceph-mon[31453]: pgmap v11435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:38.877+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:07:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:38.878+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:07:38.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:38.878+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:07:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:38 smithi153 ceph-mon[36569]: pgmap v11436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:38 smithi104 ceph-mon[36624]: pgmap v11436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:38 smithi081 ceph-mon[31453]: pgmap v11436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:40 smithi153 ceph-mon[36569]: pgmap v11437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:40 smithi104 ceph-mon[36624]: pgmap v11437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:40 smithi081 ceph-mon[31453]: pgmap v11437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:42 smithi153 ceph-mon[36569]: pgmap v11438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:42 smithi104 ceph-mon[36624]: pgmap v11438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:42 smithi081 ceph-mon[31453]: pgmap v11438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:43 smithi153 ceph-mon[36569]: pgmap v11439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:43 smithi104 ceph-mon[36624]: pgmap v11439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:43 smithi081 ceph-mon[31453]: pgmap v11439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:46 smithi153 ceph-mon[36569]: pgmap v11440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:46 smithi104 ceph-mon[36624]: pgmap v11440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:46 smithi081 ceph-mon[31453]: pgmap v11440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:48 smithi153 ceph-mon[36569]: pgmap v11441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:48 smithi104 ceph-mon[36624]: pgmap v11441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:48 smithi081 ceph-mon[31453]: pgmap v11441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:49 smithi153 ceph-mon[36569]: pgmap v11442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:49 smithi104 ceph-mon[36624]: pgmap v11442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:49 smithi081 ceph-mon[31453]: pgmap v11442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:07:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:07:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:07:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:51 smithi153 ceph-mon[36569]: pgmap v11443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:52.329 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:51 smithi081 ceph-mon[31453]: pgmap v11443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:51 smithi104 ceph-mon[36624]: pgmap v11443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:07:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:07:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:53 smithi081 ceph-mon[31453]: pgmap v11444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:53 smithi081 ceph-mon[31453]: pgmap v11445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:07:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:07:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:53 smithi153 ceph-mon[36569]: pgmap v11444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:53 smithi153 ceph-mon[36569]: pgmap v11445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:07:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:07:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:53 smithi104 ceph-mon[36624]: pgmap v11444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:53 smithi104 ceph-mon[36624]: pgmap v11445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:07:53.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:53.892+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:07:53.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:53.892+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:07:53.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:07:53.892+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:07:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:55 smithi153 ceph-mon[36569]: pgmap v11446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:55 smithi104 ceph-mon[36624]: pgmap v11446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:55 smithi081 ceph-mon[31453]: pgmap v11446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:57 smithi153 ceph-mon[36569]: pgmap v11447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:57 smithi104 ceph-mon[36624]: pgmap v11447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:57 smithi081 ceph-mon[31453]: pgmap v11447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:07:59 smithi153 ceph-mon[36569]: pgmap v11448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:07:59 smithi104 ceph-mon[36624]: pgmap v11448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:07:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:07:59 smithi081 ceph-mon[31453]: pgmap v11448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:01 smithi153 ceph-mon[36569]: pgmap v11449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:01 smithi104 ceph-mon[36624]: pgmap v11449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:01 smithi081 ceph-mon[31453]: pgmap v11449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:03.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:03 smithi081 ceph-mon[31453]: pgmap v11450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:03 smithi153 ceph-mon[36569]: pgmap v11450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:03 smithi104 ceph-mon[36624]: pgmap v11450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:05 smithi153 ceph-mon[36569]: pgmap v11451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:05 smithi104 ceph-mon[36624]: pgmap v11451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:05 smithi081 ceph-mon[31453]: pgmap v11451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:07 smithi153 ceph-mon[36569]: pgmap v11452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:07 smithi104 ceph-mon[36624]: pgmap v11452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:07 smithi081 ceph-mon[31453]: pgmap v11452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:08.901+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:08:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:08.901+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:08:08.902 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:08.901+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:08:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:09 smithi153 ceph-mon[36569]: pgmap v11453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:09 smithi104 ceph-mon[36624]: pgmap v11453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:09 smithi081 ceph-mon[31453]: pgmap v11453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:11.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:11 smithi153 ceph-mon[36569]: pgmap v11454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:11 smithi104 ceph-mon[36624]: pgmap v11454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:11 smithi081 ceph-mon[31453]: pgmap v11454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:13 smithi153 ceph-mon[36569]: pgmap v11455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:13 smithi104 ceph-mon[36624]: pgmap v11455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:13 smithi081 ceph-mon[31453]: pgmap v11455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:15 smithi153 ceph-mon[36569]: pgmap v11456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:15 smithi104 ceph-mon[36624]: pgmap v11456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:15 smithi081 ceph-mon[31453]: pgmap v11456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:17 smithi153 ceph-mon[36569]: pgmap v11457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:17 smithi104 ceph-mon[36624]: pgmap v11457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:17 smithi081 ceph-mon[31453]: pgmap v11457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:19 smithi153 ceph-mon[36569]: pgmap v11458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:19 smithi104 ceph-mon[36624]: pgmap v11458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:19 smithi081 ceph-mon[31453]: pgmap v11458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:21.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:21 smithi153 ceph-mon[36569]: pgmap v11459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:21 smithi104 ceph-mon[36624]: pgmap v11459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:21 smithi081 ceph-mon[31453]: pgmap v11459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:23 smithi153 ceph-mon[36569]: pgmap v11460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:23 smithi104 ceph-mon[36624]: pgmap v11460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:23 smithi081 ceph-mon[31453]: pgmap v11460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:23.908+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:08:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:23.908+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:08:23.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:23.908+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:08:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:25 smithi153 ceph-mon[36569]: pgmap v11461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:25 smithi104 ceph-mon[36624]: pgmap v11461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:25.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:25 smithi081 ceph-mon[31453]: pgmap v11461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:27.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:27 smithi153 ceph-mon[36569]: pgmap v11462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:27 smithi104 ceph-mon[36624]: pgmap v11462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:27 smithi081 ceph-mon[31453]: pgmap v11462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:29 smithi153 ceph-mon[36569]: pgmap v11463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:29 smithi104 ceph-mon[36624]: pgmap v11463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:29 smithi081 ceph-mon[31453]: pgmap v11463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:31 smithi153 ceph-mon[36569]: pgmap v11464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:31 smithi104 ceph-mon[36624]: pgmap v11464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:31 smithi081 ceph-mon[31453]: pgmap v11464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:33 smithi153 ceph-mon[36569]: pgmap v11465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:33 smithi104 ceph-mon[36624]: pgmap v11465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:33 smithi081 ceph-mon[31453]: pgmap v11465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:35 smithi153 ceph-mon[36569]: pgmap v11466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:35 smithi104 ceph-mon[36624]: pgmap v11466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:35 smithi081 ceph-mon[31453]: pgmap v11466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:37 smithi153 ceph-mon[36569]: pgmap v11467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:37 smithi104 ceph-mon[36624]: pgmap v11467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:37 smithi081 ceph-mon[31453]: pgmap v11467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:38.923+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:08:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:38.923+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:08:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:38.923+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:08:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:39 smithi153 ceph-mon[36569]: pgmap v11468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:39 smithi104 ceph-mon[36624]: pgmap v11468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:39 smithi081 ceph-mon[31453]: pgmap v11468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:41 smithi153 ceph-mon[36569]: pgmap v11469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:41 smithi104 ceph-mon[36624]: pgmap v11469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:41 smithi081 ceph-mon[31453]: pgmap v11469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:43 smithi153 ceph-mon[36569]: pgmap v11470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:43 smithi104 ceph-mon[36624]: pgmap v11470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:43 smithi081 ceph-mon[31453]: pgmap v11470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:45 smithi153 ceph-mon[36569]: pgmap v11471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:45 smithi104 ceph-mon[36624]: pgmap v11471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:45 smithi081 ceph-mon[31453]: pgmap v11471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:47 smithi104 ceph-mon[36624]: pgmap v11472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:47 smithi081 ceph-mon[31453]: pgmap v11472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:47 smithi153 ceph-mon[36569]: pgmap v11472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:49.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:49 smithi104 ceph-mon[36624]: pgmap v11473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:49 smithi081 ceph-mon[31453]: pgmap v11473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:49 smithi153 ceph-mon[36569]: pgmap v11473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:51 smithi104 ceph-mon[36624]: pgmap v11474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:51 smithi081 ceph-mon[31453]: pgmap v11474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:51 smithi153 ceph-mon[36569]: pgmap v11474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:53 smithi104 ceph-mon[36624]: pgmap v11475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:08:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:53 smithi081 ceph-mon[31453]: pgmap v11475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:08:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:53 smithi153 ceph-mon[36569]: pgmap v11475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:08:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:53.937+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:08:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:53.937+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:08:53.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:08:53.937+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:08:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:08:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:08:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: pgmap v11476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: pgmap v11477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:08:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:08:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: pgmap v11476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: pgmap v11477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:08:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:08:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: pgmap v11476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: pgmap v11477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:08:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:56 smithi153 ceph-mon[36569]: pgmap v11478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:56 smithi104 ceph-mon[36624]: pgmap v11478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:56 smithi081 ceph-mon[31453]: pgmap v11478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:08:58 smithi153 ceph-mon[36569]: pgmap v11479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:08:58 smithi104 ceph-mon[36624]: pgmap v11479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:08:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:08:58 smithi081 ceph-mon[31453]: pgmap v11479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:00 smithi153 ceph-mon[36569]: pgmap v11480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:00 smithi104 ceph-mon[36624]: pgmap v11480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:00 smithi081 ceph-mon[31453]: pgmap v11480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:02 smithi153 ceph-mon[36569]: pgmap v11481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:02 smithi104 ceph-mon[36624]: pgmap v11481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:02 smithi081 ceph-mon[31453]: pgmap v11481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:04 smithi153 ceph-mon[36569]: pgmap v11482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:04 smithi104 ceph-mon[36624]: pgmap v11482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:04 smithi081 ceph-mon[31453]: pgmap v11482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:06 smithi153 ceph-mon[36569]: pgmap v11483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:06 smithi104 ceph-mon[36624]: pgmap v11483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:06 smithi081 ceph-mon[31453]: pgmap v11483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:08.954+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:09:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:08.954+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:09:08.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:08.954+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:09:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:08 smithi153 ceph-mon[36569]: pgmap v11484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:08 smithi104 ceph-mon[36624]: pgmap v11484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:08 smithi081 ceph-mon[31453]: pgmap v11484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:10 smithi104 ceph-mon[36624]: pgmap v11485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:10 smithi081 ceph-mon[31453]: pgmap v11485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:10 smithi153 ceph-mon[36569]: pgmap v11485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:12 smithi104 ceph-mon[36624]: pgmap v11486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:12 smithi081 ceph-mon[31453]: pgmap v11486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:12 smithi153 ceph-mon[36569]: pgmap v11486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:14 smithi104 ceph-mon[36624]: pgmap v11487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:14 smithi081 ceph-mon[31453]: pgmap v11487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:14 smithi153 ceph-mon[36569]: pgmap v11487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:16 smithi104 ceph-mon[36624]: pgmap v11488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:16 smithi081 ceph-mon[31453]: pgmap v11488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:16 smithi153 ceph-mon[36569]: pgmap v11488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:18 smithi104 ceph-mon[36624]: pgmap v11489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:18 smithi081 ceph-mon[31453]: pgmap v11489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:18 smithi153 ceph-mon[36569]: pgmap v11489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:20 smithi081 ceph-mon[31453]: pgmap v11490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:20 smithi153 ceph-mon[36569]: pgmap v11490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:20 smithi104 ceph-mon[36624]: pgmap v11490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:22 smithi081 ceph-mon[31453]: pgmap v11491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:22 smithi153 ceph-mon[36569]: pgmap v11491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:22 smithi104 ceph-mon[36624]: pgmap v11491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:23.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:23.963+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:09:23.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:23.963+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:09:23.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:23.963+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:09:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:24 smithi081 ceph-mon[31453]: pgmap v11492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:24 smithi153 ceph-mon[36569]: pgmap v11492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:24 smithi104 ceph-mon[36624]: pgmap v11492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:27.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:26 smithi153 ceph-mon[36569]: pgmap v11493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:26 smithi081 ceph-mon[31453]: pgmap v11493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:26 smithi104 ceph-mon[36624]: pgmap v11493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:28 smithi081 ceph-mon[31453]: pgmap v11494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:28 smithi153 ceph-mon[36569]: pgmap v11494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:28 smithi104 ceph-mon[36624]: pgmap v11494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:30 smithi081 ceph-mon[31453]: pgmap v11495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:30 smithi153 ceph-mon[36569]: pgmap v11495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:30 smithi104 ceph-mon[36624]: pgmap v11495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:32 smithi081 ceph-mon[31453]: pgmap v11496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:32 smithi153 ceph-mon[36569]: pgmap v11496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:32 smithi104 ceph-mon[36624]: pgmap v11496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:35.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:34 smithi104 ceph-mon[36624]: pgmap v11497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:34 smithi081 ceph-mon[31453]: pgmap v11497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:34 smithi153 ceph-mon[36569]: pgmap v11497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:36 smithi081 ceph-mon[31453]: pgmap v11498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:36 smithi153 ceph-mon[36569]: pgmap v11498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:36 smithi104 ceph-mon[36624]: pgmap v11498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:38.976 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:38.975+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:09:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:38.975+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:09:38.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:38.975+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:09:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:38 smithi081 ceph-mon[31453]: pgmap v11499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:38 smithi153 ceph-mon[36569]: pgmap v11499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:38 smithi104 ceph-mon[36624]: pgmap v11499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:40 smithi081 ceph-mon[31453]: pgmap v11500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:40 smithi153 ceph-mon[36569]: pgmap v11500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:40 smithi104 ceph-mon[36624]: pgmap v11500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:42 smithi081 ceph-mon[31453]: pgmap v11501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:42 smithi153 ceph-mon[36569]: pgmap v11501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:42 smithi104 ceph-mon[36624]: pgmap v11501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:44 smithi081 ceph-mon[31453]: pgmap v11502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:44 smithi153 ceph-mon[36569]: pgmap v11502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:44 smithi104 ceph-mon[36624]: pgmap v11502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:46 smithi081 ceph-mon[31453]: pgmap v11503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:46 smithi153 ceph-mon[36569]: pgmap v11503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:46 smithi104 ceph-mon[36624]: pgmap v11503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:48 smithi153 ceph-mon[36569]: pgmap v11504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:48 smithi104 ceph-mon[36624]: pgmap v11504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:48 smithi081 ceph-mon[31453]: pgmap v11504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:50 smithi153 ceph-mon[36569]: pgmap v11505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:50 smithi104 ceph-mon[36624]: pgmap v11505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:50 smithi081 ceph-mon[31453]: pgmap v11505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:52 smithi153 ceph-mon[36569]: pgmap v11506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:52 smithi104 ceph-mon[36624]: pgmap v11506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:52 smithi081 ceph-mon[31453]: pgmap v11506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:53.981+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:09:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:53.981+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:09:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:09:53.981+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:09:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:54 smithi153 ceph-mon[36569]: pgmap v11507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:09:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:54 smithi104 ceph-mon[36624]: pgmap v11507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:09:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:54 smithi081 ceph-mon[31453]: pgmap v11507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:09:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:56 smithi153 ceph-mon[36569]: pgmap v11508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:56 smithi104 ceph-mon[36624]: pgmap v11508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:56 smithi081 ceph-mon[31453]: pgmap v11508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:57.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:57.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:09:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:09:58 smithi153 ceph-mon[36569]: pgmap v11509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:09:58 smithi104 ceph-mon[36624]: pgmap v11509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:09:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:09:58 smithi081 ceph-mon[31453]: pgmap v11509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:00 smithi153 ceph-mon[36569]: pgmap v11510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:10:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:00 smithi104 ceph-mon[36624]: pgmap v11510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:00 smithi081 ceph-mon[31453]: pgmap v11510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:10:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:10:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:10:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:01 smithi153 ceph-mon[36569]: pgmap v11511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:01 smithi153 ceph-mon[36569]: pgmap v11512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:10:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:10:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:01 smithi104 ceph-mon[36624]: pgmap v11511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:01 smithi104 ceph-mon[36624]: pgmap v11512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:10:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:10:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:01 smithi081 ceph-mon[31453]: pgmap v11511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:01 smithi081 ceph-mon[31453]: pgmap v11512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:10:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:03 smithi081 ceph-mon[31453]: pgmap v11513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:03 smithi153 ceph-mon[36569]: pgmap v11513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:03 smithi104 ceph-mon[36624]: pgmap v11513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:05 smithi081 ceph-mon[31453]: pgmap v11514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:05.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:05 smithi104 ceph-mon[36624]: pgmap v11514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:05 smithi153 ceph-mon[36569]: pgmap v11514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:07 smithi081 ceph-mon[31453]: pgmap v11515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:07 smithi153 ceph-mon[36569]: pgmap v11515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:07 smithi104 ceph-mon[36624]: pgmap v11515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:08.990+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:10:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:08.990+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:10:08.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:08.990+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:10:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:09 smithi081 ceph-mon[31453]: pgmap v11516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:09 smithi153 ceph-mon[36569]: pgmap v11516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:09 smithi104 ceph-mon[36624]: pgmap v11516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:11 smithi081 ceph-mon[31453]: pgmap v11517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:11 smithi153 ceph-mon[36569]: pgmap v11517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:11 smithi104 ceph-mon[36624]: pgmap v11517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:13 smithi153 ceph-mon[36569]: pgmap v11518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:13 smithi104 ceph-mon[36624]: pgmap v11518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:13 smithi081 ceph-mon[31453]: pgmap v11518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:15 smithi153 ceph-mon[36569]: pgmap v11519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:15 smithi104 ceph-mon[36624]: pgmap v11519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:15 smithi081 ceph-mon[31453]: pgmap v11519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:16.206+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:10:16.207 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:16.206+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:10:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:16.207+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 165 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:10:16.208 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:16.207+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 165 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:10:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:17 smithi153 ceph-mon[36569]: pgmap v11520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:17 smithi104 ceph-mon[36624]: pgmap v11520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:17 smithi081 ceph-mon[31453]: pgmap v11520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:19 smithi153 ceph-mon[36569]: pgmap v11521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:19 smithi104 ceph-mon[36624]: pgmap v11521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:19 smithi081 ceph-mon[31453]: pgmap v11521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:21 smithi153 ceph-mon[36569]: pgmap v11522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:21 smithi104 ceph-mon[36624]: pgmap v11522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:21 smithi081 ceph-mon[31453]: pgmap v11522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:24.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:24.005+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:10:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:24.005+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:10:24.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:24.005+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:10:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:23 smithi153 ceph-mon[36569]: pgmap v11523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:23 smithi104 ceph-mon[36624]: pgmap v11523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:23 smithi081 ceph-mon[31453]: pgmap v11523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:25 smithi153 ceph-mon[36569]: pgmap v11524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:25 smithi104 ceph-mon[36624]: pgmap v11524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:25 smithi081 ceph-mon[31453]: pgmap v11524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:27.976 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:27 smithi081 ceph-mon[31453]: pgmap v11525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:27 smithi153 ceph-mon[36569]: pgmap v11525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:27 smithi104 ceph-mon[36624]: pgmap v11525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:29 smithi153 ceph-mon[36569]: pgmap v11526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:29 smithi104 ceph-mon[36624]: pgmap v11526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:29 smithi081 ceph-mon[31453]: pgmap v11526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:31 smithi153 ceph-mon[36569]: pgmap v11527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:31 smithi104 ceph-mon[36624]: pgmap v11527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:31 smithi081 ceph-mon[31453]: pgmap v11527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:33 smithi153 ceph-mon[36569]: pgmap v11528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:33 smithi104 ceph-mon[36624]: pgmap v11528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:33 smithi081 ceph-mon[31453]: pgmap v11528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:35 smithi153 ceph-mon[36569]: pgmap v11529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:35 smithi104 ceph-mon[36624]: pgmap v11529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:35 smithi081 ceph-mon[31453]: pgmap v11529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:37 smithi153 ceph-mon[36569]: pgmap v11530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:37 smithi104 ceph-mon[36624]: pgmap v11530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:37 smithi081 ceph-mon[31453]: pgmap v11530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:39.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:39.019+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:10:39.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:39.019+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:10:39.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:39.019+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:10:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:39 smithi153 ceph-mon[36569]: pgmap v11531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:39 smithi104 ceph-mon[36624]: pgmap v11531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:39 smithi081 ceph-mon[31453]: pgmap v11531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:41 smithi153 ceph-mon[36569]: pgmap v11532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:41 smithi104 ceph-mon[36624]: pgmap v11532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:41 smithi081 ceph-mon[31453]: pgmap v11532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:43 smithi153 ceph-mon[36569]: pgmap v11533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:44.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:43 smithi104 ceph-mon[36624]: pgmap v11533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:43 smithi081 ceph-mon[31453]: pgmap v11533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:45 smithi153 ceph-mon[36569]: pgmap v11534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:46.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:45 smithi104 ceph-mon[36624]: pgmap v11534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:45 smithi081 ceph-mon[31453]: pgmap v11534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:47 smithi153 ceph-mon[36569]: pgmap v11535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:47 smithi104 ceph-mon[36624]: pgmap v11535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:48.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:47 smithi081 ceph-mon[31453]: pgmap v11535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:49 smithi153 ceph-mon[36569]: pgmap v11536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:49 smithi104 ceph-mon[36624]: pgmap v11536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:49 smithi081 ceph-mon[31453]: pgmap v11536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:51 smithi153 ceph-mon[36569]: pgmap v11537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:51 smithi104 ceph-mon[36624]: pgmap v11537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:51 smithi081 ceph-mon[31453]: pgmap v11537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:54.021+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:10:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:54.021+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:10:54.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:10:54.021+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:10:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:53 smithi153 ceph-mon[36569]: pgmap v11538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:53 smithi104 ceph-mon[36624]: pgmap v11538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:53 smithi081 ceph-mon[31453]: pgmap v11538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:55 smithi153 ceph-mon[36569]: pgmap v11539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:55 smithi104 ceph-mon[36624]: pgmap v11539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:55 smithi081 ceph-mon[31453]: pgmap v11539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:57 smithi153 ceph-mon[36569]: pgmap v11540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:57 smithi104 ceph-mon[36624]: pgmap v11540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:10:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:57 smithi081 ceph-mon[31453]: pgmap v11540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:10:59 smithi153 ceph-mon[36569]: pgmap v11541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:10:59 smithi104 ceph-mon[36624]: pgmap v11541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:10:59 smithi081 ceph-mon[31453]: pgmap v11541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:01.214+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:11:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:01.214+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:11:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:01.214+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 166 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:11:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:01.214+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 166 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:11:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:01 smithi153 ceph-mon[36569]: pgmap v11542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:11:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:01 smithi104 ceph-mon[36624]: pgmap v11542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:11:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:01 smithi081 ceph-mon[31453]: pgmap v11542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:11:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:11:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:11:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:11:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:11:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:11:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:11:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:11:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:03 smithi153 ceph-mon[36569]: pgmap v11543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:03 smithi153 ceph-mon[36569]: pgmap v11544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:03 smithi104 ceph-mon[36624]: pgmap v11543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:03 smithi104 ceph-mon[36624]: pgmap v11544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:03 smithi081 ceph-mon[31453]: pgmap v11543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:03 smithi081 ceph-mon[31453]: pgmap v11544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:05 smithi153 ceph-mon[36569]: pgmap v11545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:05 smithi104 ceph-mon[36624]: pgmap v11545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:05 smithi081 ceph-mon[31453]: pgmap v11545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:07 smithi153 ceph-mon[36569]: pgmap v11546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:07 smithi104 ceph-mon[36624]: pgmap v11546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:07 smithi081 ceph-mon[31453]: pgmap v11546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:09.028 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:09.027+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:11:09.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:09.027+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:11:09.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:09.027+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:11:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:09 smithi104 ceph-mon[36624]: pgmap v11547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:09 smithi081 ceph-mon[31453]: pgmap v11547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:09 smithi153 ceph-mon[36569]: pgmap v11547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:11 smithi104 ceph-mon[36624]: pgmap v11548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:11 smithi081 ceph-mon[31453]: pgmap v11548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:11 smithi153 ceph-mon[36569]: pgmap v11548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:13 smithi104 ceph-mon[36624]: pgmap v11549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:13 smithi081 ceph-mon[31453]: pgmap v11549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:13 smithi153 ceph-mon[36569]: pgmap v11549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:15 smithi104 ceph-mon[36624]: pgmap v11550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:15 smithi081 ceph-mon[31453]: pgmap v11550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:15 smithi153 ceph-mon[36569]: pgmap v11550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:17 smithi081 ceph-mon[31453]: pgmap v11551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:17 smithi153 ceph-mon[36569]: pgmap v11551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:17 smithi104 ceph-mon[36624]: pgmap v11551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:19 smithi081 ceph-mon[31453]: pgmap v11552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:19 smithi153 ceph-mon[36569]: pgmap v11552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:19 smithi104 ceph-mon[36624]: pgmap v11552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:21 smithi081 ceph-mon[31453]: pgmap v11553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:21 smithi153 ceph-mon[36569]: pgmap v11553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:21 smithi104 ceph-mon[36624]: pgmap v11553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:24.029+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:11:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:24.029+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:11:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:24.029+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:11:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:23 smithi081 ceph-mon[31453]: pgmap v11554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:23 smithi153 ceph-mon[36569]: pgmap v11554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:23 smithi104 ceph-mon[36624]: pgmap v11554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:25 smithi081 ceph-mon[31453]: pgmap v11555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:25 smithi153 ceph-mon[36569]: pgmap v11555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:25 smithi104 ceph-mon[36624]: pgmap v11555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:28.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:27 smithi153 ceph-mon[36569]: pgmap v11556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:27 smithi081 ceph-mon[31453]: pgmap v11556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:27 smithi104 ceph-mon[36624]: pgmap v11556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:29 smithi081 ceph-mon[31453]: pgmap v11557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:29 smithi153 ceph-mon[36569]: pgmap v11557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:29 smithi104 ceph-mon[36624]: pgmap v11557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:31 smithi153 ceph-mon[36569]: pgmap v11558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:31 smithi104 ceph-mon[36624]: pgmap v11558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:31 smithi081 ceph-mon[31453]: pgmap v11558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:33 smithi153 ceph-mon[36569]: pgmap v11559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:33 smithi104 ceph-mon[36624]: pgmap v11559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:33 smithi081 ceph-mon[31453]: pgmap v11559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:35 smithi153 ceph-mon[36569]: pgmap v11560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:35 smithi104 ceph-mon[36624]: pgmap v11560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:35 smithi081 ceph-mon[31453]: pgmap v11560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:37 smithi153 ceph-mon[36569]: pgmap v11561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:37 smithi104 ceph-mon[36624]: pgmap v11561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:37 smithi081 ceph-mon[31453]: pgmap v11561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:39.038+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:11:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:39.038+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:11:39.039 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:39.038+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:11:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:39 smithi153 ceph-mon[36569]: pgmap v11562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:39 smithi104 ceph-mon[36624]: pgmap v11562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:39 smithi081 ceph-mon[31453]: pgmap v11562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:41 smithi153 ceph-mon[36569]: pgmap v11563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:41 smithi104 ceph-mon[36624]: pgmap v11563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:41 smithi081 ceph-mon[31453]: pgmap v11563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:43 smithi153 ceph-mon[36569]: pgmap v11564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:43 smithi104 ceph-mon[36624]: pgmap v11564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:43 smithi081 ceph-mon[31453]: pgmap v11564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:45 smithi104 ceph-mon[36624]: pgmap v11565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:45 smithi081 ceph-mon[31453]: pgmap v11565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:45 smithi153 ceph-mon[36569]: pgmap v11565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:47 smithi104 ceph-mon[36624]: pgmap v11566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:47 smithi081 ceph-mon[31453]: pgmap v11566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:47 smithi153 ceph-mon[36569]: pgmap v11566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:49 smithi104 ceph-mon[36624]: pgmap v11567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:49 smithi081 ceph-mon[31453]: pgmap v11567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:49 smithi153 ceph-mon[36569]: pgmap v11567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:51 smithi104 ceph-mon[36624]: pgmap v11568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:51 smithi081 ceph-mon[31453]: pgmap v11568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:51 smithi153 ceph-mon[36569]: pgmap v11568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:53 smithi081 ceph-mon[31453]: pgmap v11569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:53 smithi153 ceph-mon[36569]: pgmap v11569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:53 smithi104 ceph-mon[36624]: pgmap v11569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:54.054+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:11:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:54.054+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:11:54.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:11:54.054+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:11:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:55 smithi081 ceph-mon[31453]: pgmap v11570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:55 smithi153 ceph-mon[36569]: pgmap v11570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:55 smithi104 ceph-mon[36624]: pgmap v11570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:57 smithi081 ceph-mon[31453]: pgmap v11571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:57 smithi153 ceph-mon[36569]: pgmap v11571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:57 smithi104 ceph-mon[36624]: pgmap v11571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:59.610 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:11:59 smithi153 ceph-mon[36569]: pgmap v11572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:11:59 smithi081 ceph-mon[31453]: pgmap v11572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:11:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:11:59 smithi104 ceph-mon[36624]: pgmap v11572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:01 smithi081 ceph-mon[31453]: pgmap v11573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:01 smithi153 ceph-mon[36569]: pgmap v11573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:01 smithi104 ceph-mon[36624]: pgmap v11573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:03 smithi081 ceph-mon[31453]: pgmap v11574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:12:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:03 smithi153 ceph-mon[36569]: pgmap v11574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:12:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:03 smithi104 ceph-mon[36624]: pgmap v11574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:12:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:12:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:12:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:04 smithi081 ceph-mon[31453]: pgmap v11575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:04 smithi081 ceph-mon[31453]: pgmap v11576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:12:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:12:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:12:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:04 smithi153 ceph-mon[36569]: pgmap v11575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:04 smithi153 ceph-mon[36569]: pgmap v11576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:12:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:12:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:12:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:04 smithi104 ceph-mon[36624]: pgmap v11575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:04 smithi104 ceph-mon[36624]: pgmap v11576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:12:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:06 smithi153 ceph-mon[36569]: pgmap v11577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:06 smithi104 ceph-mon[36624]: pgmap v11577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:06 smithi081 ceph-mon[31453]: pgmap v11577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:08 smithi153 ceph-mon[36569]: pgmap v11578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:09.064+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:12:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:09.064+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:12:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:09.064+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:12:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:08 smithi104 ceph-mon[36624]: pgmap v11578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:08 smithi081 ceph-mon[31453]: pgmap v11578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:10 smithi153 ceph-mon[36569]: pgmap v11579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:10 smithi104 ceph-mon[36624]: pgmap v11579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:10 smithi081 ceph-mon[31453]: pgmap v11579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:12 smithi153 ceph-mon[36569]: pgmap v11580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:12 smithi104 ceph-mon[36624]: pgmap v11580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:12 smithi081 ceph-mon[31453]: pgmap v11580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:14 smithi104 ceph-mon[36624]: pgmap v11581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:14 smithi081 ceph-mon[31453]: pgmap v11581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:14 smithi153 ceph-mon[36569]: pgmap v11581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:16 smithi104 ceph-mon[36624]: pgmap v11582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:16 smithi081 ceph-mon[31453]: pgmap v11582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:16 smithi153 ceph-mon[36569]: pgmap v11582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:18 smithi104 ceph-mon[36624]: pgmap v11583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:18 smithi081 ceph-mon[31453]: pgmap v11583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:18 smithi153 ceph-mon[36569]: pgmap v11583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:20 smithi104 ceph-mon[36624]: pgmap v11584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:20 smithi081 ceph-mon[31453]: pgmap v11584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:20 smithi153 ceph-mon[36569]: pgmap v11584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:22 smithi104 ceph-mon[36624]: pgmap v11585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:22 smithi081 ceph-mon[31453]: pgmap v11585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:22 smithi153 ceph-mon[36569]: pgmap v11585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:24.080+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:12:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:24.080+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:12:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:24.080+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:12:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:24 smithi081 ceph-mon[31453]: pgmap v11586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:24 smithi153 ceph-mon[36569]: pgmap v11586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:24 smithi104 ceph-mon[36624]: pgmap v11586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:26 smithi153 ceph-mon[36569]: pgmap v11587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:26 smithi104 ceph-mon[36624]: pgmap v11587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:26 smithi081 ceph-mon[31453]: pgmap v11587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:28 smithi153 ceph-mon[36569]: pgmap v11588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:28 smithi104 ceph-mon[36624]: pgmap v11588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:28 smithi081 ceph-mon[31453]: pgmap v11588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:30 smithi153 ceph-mon[36569]: pgmap v11589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:30 smithi104 ceph-mon[36624]: pgmap v11589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:30 smithi081 ceph-mon[31453]: pgmap v11589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:32 smithi104 ceph-mon[36624]: pgmap v11590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:32 smithi081 ceph-mon[31453]: pgmap v11590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:32 smithi153 ceph-mon[36569]: pgmap v11590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:34 smithi104 ceph-mon[36624]: pgmap v11591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:34 smithi081 ceph-mon[31453]: pgmap v11591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:34 smithi153 ceph-mon[36569]: pgmap v11591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:36 smithi104 ceph-mon[36624]: pgmap v11592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:36 smithi081 ceph-mon[31453]: pgmap v11592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:36 smithi153 ceph-mon[36569]: pgmap v11592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:38 smithi104 ceph-mon[36624]: pgmap v11593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:39.085+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:12:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:39.085+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:12:39.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:39.085+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:12:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:38 smithi081 ceph-mon[31453]: pgmap v11593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:38 smithi153 ceph-mon[36569]: pgmap v11593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:40 smithi081 ceph-mon[31453]: pgmap v11594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:40 smithi153 ceph-mon[36569]: pgmap v11594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:40 smithi104 ceph-mon[36624]: pgmap v11594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:42 smithi081 ceph-mon[31453]: pgmap v11595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:42 smithi153 ceph-mon[36569]: pgmap v11595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:42 smithi104 ceph-mon[36624]: pgmap v11595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:44 smithi081 ceph-mon[31453]: pgmap v11596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:44 smithi153 ceph-mon[36569]: pgmap v11596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:44 smithi104 ceph-mon[36624]: pgmap v11596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:46 smithi081 ceph-mon[31453]: pgmap v11597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:46 smithi153 ceph-mon[36569]: pgmap v11597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:46 smithi104 ceph-mon[36624]: pgmap v11597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:48 smithi081 ceph-mon[31453]: pgmap v11598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:48 smithi153 ceph-mon[36569]: pgmap v11598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:48 smithi104 ceph-mon[36624]: pgmap v11598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:50 smithi081 ceph-mon[31453]: pgmap v11599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:50 smithi153 ceph-mon[36569]: pgmap v11599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:50 smithi104 ceph-mon[36624]: pgmap v11599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:52 smithi081 ceph-mon[31453]: pgmap v11600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:52 smithi153 ceph-mon[36569]: pgmap v11600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:52 smithi104 ceph-mon[36624]: pgmap v11600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:54.101+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:12:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:54.101+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:12:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:12:54.101+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:12:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:54 smithi081 ceph-mon[31453]: pgmap v11601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:54 smithi153 ceph-mon[36569]: pgmap v11601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:54 smithi104 ceph-mon[36624]: pgmap v11601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:56 smithi081 ceph-mon[31453]: pgmap v11602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:56 smithi153 ceph-mon[36569]: pgmap v11602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:56 smithi104 ceph-mon[36624]: pgmap v11602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:12:58 smithi081 ceph-mon[31453]: pgmap v11603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:12:58 smithi153 ceph-mon[36569]: pgmap v11603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:12:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:12:58 smithi104 ceph-mon[36624]: pgmap v11603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:00 smithi081 ceph-mon[31453]: pgmap v11604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:00 smithi153 ceph-mon[36569]: pgmap v11604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:00 smithi104 ceph-mon[36624]: pgmap v11604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:02 smithi081 ceph-mon[31453]: pgmap v11605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:02 smithi153 ceph-mon[36569]: pgmap v11605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:02 smithi104 ceph-mon[36624]: pgmap v11605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:04 smithi081 ceph-mon[31453]: pgmap v11606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:13:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:04 smithi153 ceph-mon[36569]: pgmap v11606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:13:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:04 smithi104 ceph-mon[36624]: pgmap v11606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:13:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:13:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:13:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:13:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:13:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:13:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:13:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:13:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:13:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:13:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:06 smithi153 ceph-mon[36569]: pgmap v11607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:06 smithi153 ceph-mon[36569]: pgmap v11608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:06 smithi104 ceph-mon[36624]: pgmap v11607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:06 smithi104 ceph-mon[36624]: pgmap v11608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:06 smithi081 ceph-mon[31453]: pgmap v11607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:06 smithi081 ceph-mon[31453]: pgmap v11608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:09.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:09.109+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:13:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:09.109+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:13:09.112 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:09.109+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:13:09.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:08 smithi081 ceph-mon[31453]: pgmap v11609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:08 smithi153 ceph-mon[36569]: pgmap v11609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:08 smithi104 ceph-mon[36624]: pgmap v11609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:10 smithi153 ceph-mon[36569]: pgmap v11610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:10 smithi104 ceph-mon[36624]: pgmap v11610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:10 smithi081 ceph-mon[31453]: pgmap v11610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:12 smithi153 ceph-mon[36569]: pgmap v11611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:12 smithi104 ceph-mon[36624]: pgmap v11611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:12 smithi081 ceph-mon[31453]: pgmap v11611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:14 smithi153 ceph-mon[36569]: pgmap v11612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:14 smithi104 ceph-mon[36624]: pgmap v11612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:14 smithi081 ceph-mon[31453]: pgmap v11612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:16.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:16.237+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:13:16.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:16.237+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:13:16.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:16.238+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 167 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:13:16.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:16.238+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 167 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:13:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:16 smithi153 ceph-mon[36569]: pgmap v11613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:16 smithi104 ceph-mon[36624]: pgmap v11613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:16 smithi081 ceph-mon[31453]: pgmap v11613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:19.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:18 smithi081 ceph-mon[31453]: pgmap v11614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:18 smithi153 ceph-mon[36569]: pgmap v11614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:18 smithi104 ceph-mon[36624]: pgmap v11614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:20 smithi153 ceph-mon[36569]: pgmap v11615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:20 smithi104 ceph-mon[36624]: pgmap v11615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:20 smithi081 ceph-mon[31453]: pgmap v11615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:22 smithi153 ceph-mon[36569]: pgmap v11616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:22 smithi104 ceph-mon[36624]: pgmap v11616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:22 smithi081 ceph-mon[31453]: pgmap v11616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:24.119+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:13:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:24.119+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:13:24.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:24.119+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:13:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:24 smithi153 ceph-mon[36569]: pgmap v11617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:24 smithi104 ceph-mon[36624]: pgmap v11617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:24 smithi081 ceph-mon[31453]: pgmap v11617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:26 smithi153 ceph-mon[36569]: pgmap v11618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:26 smithi104 ceph-mon[36624]: pgmap v11618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:26 smithi081 ceph-mon[31453]: pgmap v11618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:28 smithi153 ceph-mon[36569]: pgmap v11619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:28 smithi104 ceph-mon[36624]: pgmap v11619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:28 smithi081 ceph-mon[31453]: pgmap v11619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:30 smithi153 ceph-mon[36569]: pgmap v11620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:30 smithi104 ceph-mon[36624]: pgmap v11620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:30 smithi081 ceph-mon[31453]: pgmap v11620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:32 smithi153 ceph-mon[36569]: pgmap v11621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:32 smithi104 ceph-mon[36624]: pgmap v11621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:32 smithi081 ceph-mon[31453]: pgmap v11621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:34 smithi153 ceph-mon[36569]: pgmap v11622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:34 smithi104 ceph-mon[36624]: pgmap v11622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:34 smithi081 ceph-mon[31453]: pgmap v11622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:36 smithi153 ceph-mon[36569]: pgmap v11623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:36 smithi104 ceph-mon[36624]: pgmap v11623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:36 smithi081 ceph-mon[31453]: pgmap v11623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:39.135+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:13:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:39.135+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:13:39.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:39.135+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:13:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:38 smithi153 ceph-mon[36569]: pgmap v11624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:38 smithi104 ceph-mon[36624]: pgmap v11624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:38 smithi081 ceph-mon[31453]: pgmap v11624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:40 smithi153 ceph-mon[36569]: pgmap v11625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:40 smithi104 ceph-mon[36624]: pgmap v11625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:40 smithi081 ceph-mon[31453]: pgmap v11625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:42 smithi104 ceph-mon[36624]: pgmap v11626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:42 smithi081 ceph-mon[31453]: pgmap v11626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:42 smithi153 ceph-mon[36569]: pgmap v11626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:44 smithi104 ceph-mon[36624]: pgmap v11627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:44 smithi081 ceph-mon[31453]: pgmap v11627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:44 smithi153 ceph-mon[36569]: pgmap v11627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:46 smithi081 ceph-mon[31453]: pgmap v11628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:46 smithi153 ceph-mon[36569]: pgmap v11628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:46 smithi104 ceph-mon[36624]: pgmap v11628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:48 smithi081 ceph-mon[31453]: pgmap v11629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:48 smithi153 ceph-mon[36569]: pgmap v11629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:48 smithi104 ceph-mon[36624]: pgmap v11629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:50 smithi081 ceph-mon[31453]: pgmap v11630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:50 smithi153 ceph-mon[36569]: pgmap v11630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:50 smithi104 ceph-mon[36624]: pgmap v11630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:52 smithi081 ceph-mon[31453]: pgmap v11631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:52 smithi153 ceph-mon[36569]: pgmap v11631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:52 smithi104 ceph-mon[36624]: pgmap v11631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:54.151+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:13:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:54.151+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:13:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:13:54.151+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:13:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:54 smithi081 ceph-mon[31453]: pgmap v11632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:54 smithi153 ceph-mon[36569]: pgmap v11632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:54 smithi104 ceph-mon[36624]: pgmap v11632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:56 smithi081 ceph-mon[31453]: pgmap v11633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:56 smithi153 ceph-mon[36569]: pgmap v11633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:56 smithi104 ceph-mon[36624]: pgmap v11633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:13:58 smithi081 ceph-mon[31453]: pgmap v11634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:13:58 smithi153 ceph-mon[36569]: pgmap v11634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:13:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:13:58 smithi104 ceph-mon[36624]: pgmap v11634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:00 smithi081 ceph-mon[31453]: pgmap v11635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:00 smithi153 ceph-mon[36569]: pgmap v11635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:00 smithi104 ceph-mon[36624]: pgmap v11635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:02 smithi081 ceph-mon[31453]: pgmap v11636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:02 smithi153 ceph-mon[36569]: pgmap v11636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:02 smithi104 ceph-mon[36624]: pgmap v11636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:04 smithi081 ceph-mon[31453]: pgmap v11637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:04 smithi153 ceph-mon[36569]: pgmap v11637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:04 smithi104 ceph-mon[36624]: pgmap v11637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:06 smithi081 ceph-mon[31453]: pgmap v11638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:14:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:06 smithi153 ceph-mon[36569]: pgmap v11638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:14:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:06 smithi104 ceph-mon[36624]: pgmap v11638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:14:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:08 smithi081 ceph-mon[31453]: pgmap v11639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:08.711 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:08 smithi104 ceph-mon[36624]: pgmap v11639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:08 smithi153 ceph-mon[36569]: pgmap v11639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:09.166+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:14:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:09.166+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:14:09.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:09.166+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:14:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:10 smithi081 ceph-mon[31453]: pgmap v11640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:10.667 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:10 smithi104 ceph-mon[36624]: pgmap v11640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:10 smithi153 ceph-mon[36569]: pgmap v11640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:11.997 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:11.997 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:12 smithi081 ceph-mon[31453]: pgmap v11641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:12 smithi153 ceph-mon[36569]: pgmap v11641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:12 smithi104 ceph-mon[36624]: pgmap v11641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:14:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:14:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:14 smithi153 ceph-mon[36569]: pgmap v11642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:14 smithi153 ceph-mon[36569]: pgmap v11643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:14:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:14:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:14 smithi104 ceph-mon[36624]: pgmap v11642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:14 smithi104 ceph-mon[36624]: pgmap v11643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:14:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:14:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:14 smithi081 ceph-mon[31453]: pgmap v11642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:14 smithi081 ceph-mon[31453]: pgmap v11643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:14:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:16 smithi081 ceph-mon[31453]: pgmap v11644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:16 smithi153 ceph-mon[36569]: pgmap v11644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:16 smithi104 ceph-mon[36624]: pgmap v11644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:18 smithi081 ceph-mon[31453]: pgmap v11645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:18 smithi153 ceph-mon[36569]: pgmap v11645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:18 smithi104 ceph-mon[36624]: pgmap v11645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:20 smithi081 ceph-mon[31453]: pgmap v11646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:20 smithi153 ceph-mon[36569]: pgmap v11646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:20.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:20 smithi104 ceph-mon[36624]: pgmap v11646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:22 smithi081 ceph-mon[31453]: pgmap v11647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:22 smithi153 ceph-mon[36569]: pgmap v11647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:22 smithi104 ceph-mon[36624]: pgmap v11647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:24.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:24.171+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:14:24.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:24.171+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:14:24.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:24.171+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:14:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:24 smithi081 ceph-mon[31453]: pgmap v11648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:24 smithi153 ceph-mon[36569]: pgmap v11648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:24 smithi104 ceph-mon[36624]: pgmap v11648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:26 smithi081 ceph-mon[31453]: pgmap v11649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:26 smithi153 ceph-mon[36569]: pgmap v11649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:26 smithi104 ceph-mon[36624]: pgmap v11649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:28 smithi153 ceph-mon[36569]: pgmap v11650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:28 smithi104 ceph-mon[36624]: pgmap v11650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:28 smithi081 ceph-mon[31453]: pgmap v11650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:30 smithi153 ceph-mon[36569]: pgmap v11651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:30 smithi104 ceph-mon[36624]: pgmap v11651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:30 smithi081 ceph-mon[31453]: pgmap v11651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:32.489 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:32 smithi081 ceph-mon[31453]: pgmap v11652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:32 smithi153 ceph-mon[36569]: pgmap v11652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:32 smithi104 ceph-mon[36624]: pgmap v11652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:34 smithi153 ceph-mon[36569]: pgmap v11653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:34 smithi104 ceph-mon[36624]: pgmap v11653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:34 smithi081 ceph-mon[31453]: pgmap v11653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:36 smithi153 ceph-mon[36569]: pgmap v11654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:36 smithi104 ceph-mon[36624]: pgmap v11654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:36 smithi081 ceph-mon[31453]: pgmap v11654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:38 smithi153 ceph-mon[36569]: pgmap v11655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:38 smithi104 ceph-mon[36624]: pgmap v11655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:38.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:38 smithi081 ceph-mon[31453]: pgmap v11655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:39.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:39.186+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:14:39.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:39.186+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:14:39.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:39.186+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:14:40.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:40 smithi081 ceph-mon[31453]: pgmap v11656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:40 smithi153 ceph-mon[36569]: pgmap v11656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:40 smithi104 ceph-mon[36624]: pgmap v11656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:42 smithi153 ceph-mon[36569]: pgmap v11657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:42 smithi104 ceph-mon[36624]: pgmap v11657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:42 smithi081 ceph-mon[31453]: pgmap v11657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:44 smithi153 ceph-mon[36569]: pgmap v11658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:44 smithi104 ceph-mon[36624]: pgmap v11658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:44 smithi081 ceph-mon[31453]: pgmap v11658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:46 smithi153 ceph-mon[36569]: pgmap v11659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:46 smithi104 ceph-mon[36624]: pgmap v11659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:46 smithi081 ceph-mon[31453]: pgmap v11659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:48 smithi153 ceph-mon[36569]: pgmap v11660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:48 smithi104 ceph-mon[36624]: pgmap v11660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:48 smithi081 ceph-mon[31453]: pgmap v11660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:50 smithi153 ceph-mon[36569]: pgmap v11661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:50 smithi104 ceph-mon[36624]: pgmap v11661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:50 smithi081 ceph-mon[31453]: pgmap v11661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:52.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:52 smithi081 ceph-mon[31453]: pgmap v11662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:52 smithi153 ceph-mon[36569]: pgmap v11662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:52 smithi104 ceph-mon[36624]: pgmap v11662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:54.201+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:14:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:54.201+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:14:54.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:14:54.201+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:14:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:54 smithi153 ceph-mon[36569]: pgmap v11663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:54 smithi104 ceph-mon[36624]: pgmap v11663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:54 smithi081 ceph-mon[31453]: pgmap v11663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:56 smithi153 ceph-mon[36569]: pgmap v11664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:56 smithi104 ceph-mon[36624]: pgmap v11664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:56 smithi081 ceph-mon[31453]: pgmap v11664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:14:58 smithi153 ceph-mon[36569]: pgmap v11665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:14:58 smithi104 ceph-mon[36624]: pgmap v11665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:14:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:14:58 smithi081 ceph-mon[31453]: pgmap v11665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:00 smithi153 ceph-mon[36569]: pgmap v11666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:00 smithi104 ceph-mon[36624]: pgmap v11666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:00 smithi081 ceph-mon[31453]: pgmap v11666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:02 smithi153 ceph-mon[36569]: pgmap v11667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:02 smithi104 ceph-mon[36624]: pgmap v11667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:02 smithi081 ceph-mon[31453]: pgmap v11667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:04 smithi153 ceph-mon[36569]: pgmap v11668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:04 smithi104 ceph-mon[36624]: pgmap v11668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:04 smithi081 ceph-mon[31453]: pgmap v11668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:06 smithi153 ceph-mon[36569]: pgmap v11669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:06 smithi104 ceph-mon[36624]: pgmap v11669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:06 smithi081 ceph-mon[31453]: pgmap v11669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:08 smithi104 ceph-mon[36624]: pgmap v11670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:08 smithi081 ceph-mon[31453]: pgmap v11670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:08 smithi153 ceph-mon[36569]: pgmap v11670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:09.211 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:09.210+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:15:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:09.210+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:15:09.212 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:09.210+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:15:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:10 smithi104 ceph-mon[36624]: pgmap v11671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:10 smithi081 ceph-mon[31453]: pgmap v11671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:10 smithi153 ceph-mon[36569]: pgmap v11671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:12 smithi104 ceph-mon[36624]: pgmap v11672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:12 smithi081 ceph-mon[31453]: pgmap v11672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:12 smithi153 ceph-mon[36569]: pgmap v11672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:14 smithi104 ceph-mon[36624]: pgmap v11673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:15:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:14 smithi081 ceph-mon[31453]: pgmap v11673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:15:14.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:14 smithi153 ceph-mon[36569]: pgmap v11673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:14.694 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:15:16.399 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:16 smithi153 ceph-mon[36569]: pgmap v11674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:16 smithi104 ceph-mon[36624]: pgmap v11674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:16 smithi081 ceph-mon[31453]: pgmap v11674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:18 smithi153 ceph-mon[36569]: pgmap v11675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:18 smithi081 ceph-mon[31453]: pgmap v11675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:18 smithi104 ceph-mon[36624]: pgmap v11675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:15:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:15:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:19 smithi081 ceph-mon[31453]: pgmap v11676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:19 smithi081 ceph-mon[31453]: pgmap v11677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:15:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:15:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:19 smithi153 ceph-mon[36569]: pgmap v11676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:19 smithi153 ceph-mon[36569]: pgmap v11677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:15:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:15:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:19 smithi104 ceph-mon[36624]: pgmap v11676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:19 smithi104 ceph-mon[36624]: pgmap v11677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:15:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:21 smithi153 ceph-mon[36569]: pgmap v11678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:21 smithi104 ceph-mon[36624]: pgmap v11678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:21 smithi081 ceph-mon[31453]: pgmap v11678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:23 smithi153 ceph-mon[36569]: pgmap v11679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:23 smithi104 ceph-mon[36624]: pgmap v11679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:23 smithi081 ceph-mon[31453]: pgmap v11679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:24.223 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:24.222+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:15:24.223 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:24.222+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:15:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:24.222+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:15:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:25 smithi153 ceph-mon[36569]: pgmap v11680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:25 smithi104 ceph-mon[36624]: pgmap v11680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:25 smithi081 ceph-mon[31453]: pgmap v11680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:27 smithi153 ceph-mon[36569]: pgmap v11681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:27 smithi104 ceph-mon[36624]: pgmap v11681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:27 smithi081 ceph-mon[31453]: pgmap v11681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:29 smithi153 ceph-mon[36569]: pgmap v11682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:29 smithi104 ceph-mon[36624]: pgmap v11682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:29 smithi081 ceph-mon[31453]: pgmap v11682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:31 smithi153 ceph-mon[36569]: pgmap v11683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:31 smithi104 ceph-mon[36624]: pgmap v11683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:31 smithi081 ceph-mon[31453]: pgmap v11683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:33 smithi153 ceph-mon[36569]: pgmap v11684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:33 smithi104 ceph-mon[36624]: pgmap v11684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:33 smithi081 ceph-mon[31453]: pgmap v11684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:35 smithi153 ceph-mon[36569]: pgmap v11685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:35 smithi104 ceph-mon[36624]: pgmap v11685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:35 smithi081 ceph-mon[31453]: pgmap v11685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:37 smithi153 ceph-mon[36569]: pgmap v11686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:37 smithi104 ceph-mon[36624]: pgmap v11686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:37 smithi081 ceph-mon[31453]: pgmap v11686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:39.237+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:15:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:39.237+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:15:39.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:39.237+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:15:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:39 smithi153 ceph-mon[36569]: pgmap v11687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:39 smithi104 ceph-mon[36624]: pgmap v11687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:39 smithi081 ceph-mon[31453]: pgmap v11687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:41.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:41 smithi081 ceph-mon[31453]: pgmap v11688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:41 smithi153 ceph-mon[36569]: pgmap v11688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:41 smithi104 ceph-mon[36624]: pgmap v11688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:43 smithi153 ceph-mon[36569]: pgmap v11689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:43 smithi104 ceph-mon[36624]: pgmap v11689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:43 smithi081 ceph-mon[31453]: pgmap v11689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:45 smithi153 ceph-mon[36569]: pgmap v11690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:45 smithi104 ceph-mon[36624]: pgmap v11690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:45 smithi081 ceph-mon[31453]: pgmap v11690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:47 smithi153 ceph-mon[36569]: pgmap v11691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:47 smithi104 ceph-mon[36624]: pgmap v11691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:47 smithi081 ceph-mon[31453]: pgmap v11691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:49 smithi153 ceph-mon[36569]: pgmap v11692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:49 smithi104 ceph-mon[36624]: pgmap v11692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:49 smithi081 ceph-mon[31453]: pgmap v11692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:51 smithi153 ceph-mon[36569]: pgmap v11693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:51 smithi104 ceph-mon[36624]: pgmap v11693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:51 smithi081 ceph-mon[31453]: pgmap v11693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:53 smithi153 ceph-mon[36569]: pgmap v11694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:53 smithi104 ceph-mon[36624]: pgmap v11694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:53 smithi081 ceph-mon[31453]: pgmap v11694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:54.251+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:15:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:54.251+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:15:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:15:54.251+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:15:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:55 smithi153 ceph-mon[36569]: pgmap v11695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:55 smithi104 ceph-mon[36624]: pgmap v11695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:55 smithi081 ceph-mon[31453]: pgmap v11695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:57 smithi153 ceph-mon[36569]: pgmap v11696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:57 smithi104 ceph-mon[36624]: pgmap v11696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:57 smithi081 ceph-mon[31453]: pgmap v11696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:15:59 smithi153 ceph-mon[36569]: pgmap v11697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:15:59 smithi104 ceph-mon[36624]: pgmap v11697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:15:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:15:59 smithi081 ceph-mon[31453]: pgmap v11697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:01 smithi153 ceph-mon[36569]: pgmap v11698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:01 smithi104 ceph-mon[36624]: pgmap v11698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:01 smithi081 ceph-mon[31453]: pgmap v11698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:03 smithi153 ceph-mon[36569]: pgmap v11699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:03 smithi104 ceph-mon[36624]: pgmap v11699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:03 smithi081 ceph-mon[31453]: pgmap v11699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:05 smithi153 ceph-mon[36569]: pgmap v11700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:05 smithi104 ceph-mon[36624]: pgmap v11700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:05 smithi081 ceph-mon[31453]: pgmap v11700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:07 smithi153 ceph-mon[36569]: pgmap v11701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:07 smithi104 ceph-mon[36624]: pgmap v11701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:07 smithi081 ceph-mon[31453]: pgmap v11701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:09.261+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:16:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:09.261+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:16:09.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:09.261+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:16:09.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:09 smithi153 ceph-mon[36569]: pgmap v11702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:09 smithi104 ceph-mon[36624]: pgmap v11702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:09 smithi081 ceph-mon[31453]: pgmap v11702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:11 smithi153 ceph-mon[36569]: pgmap v11703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:11 smithi104 ceph-mon[36624]: pgmap v11703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:11 smithi081 ceph-mon[31453]: pgmap v11703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:12.149+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000653) for more than 900000000 us, fault. 2024-04-13T03:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:12.149+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b40d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f4024000 comp rx=0 tx=0).stop 2024-04-13T03:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:12.149+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b40d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T03:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:12.149+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b40d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:16:12.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:12.149+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:16:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:12.150+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:16:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:12.151+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f16d40065b0 tx=0x7f16f40d2000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:16:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:13 smithi153 ceph-mon[36569]: pgmap v11704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:13 smithi104 ceph-mon[36624]: pgmap v11704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:13 smithi081 ceph-mon[31453]: pgmap v11704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:15 smithi153 ceph-mon[36569]: pgmap v11705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:15 smithi104 ceph-mon[36624]: pgmap v11705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:15 smithi081 ceph-mon[31453]: pgmap v11705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:16.267+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:16:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:16.267+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:16:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:16.267+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 168 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:16:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:16.268+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 168 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:16:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:17 smithi153 ceph-mon[36569]: pgmap v11706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:17 smithi104 ceph-mon[36624]: pgmap v11706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:17 smithi081 ceph-mon[31453]: pgmap v11706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: pgmap v11707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: pgmap v11707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: pgmap v11707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:16:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:16:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:16:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:16:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:16:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:16:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:16:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:16:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: pgmap v11708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: pgmap v11709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:16:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:16:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:16:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:16:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: pgmap v11708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: pgmap v11709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:16:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:16:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:16:21.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: pgmap v11708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:21.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: pgmap v11709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:21.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:16:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:22 smithi153 ceph-mon[36569]: pgmap v11710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:22 smithi104 ceph-mon[36624]: pgmap v11710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:22 smithi081 ceph-mon[31453]: pgmap v11710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:24.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:24.273+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:16:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:24.273+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:16:24.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:24.273+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:16:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:24 smithi153 ceph-mon[36569]: pgmap v11711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:24 smithi104 ceph-mon[36624]: pgmap v11711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:25.352 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:24 smithi081 ceph-mon[31453]: pgmap v11711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:26 smithi104 ceph-mon[36624]: pgmap v11712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:26 smithi081 ceph-mon[31453]: pgmap v11712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:26 smithi153 ceph-mon[36569]: pgmap v11712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:28 smithi153 ceph-mon[36569]: pgmap v11713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:28 smithi104 ceph-mon[36624]: pgmap v11713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:28 smithi081 ceph-mon[31453]: pgmap v11713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:30 smithi153 ceph-mon[36569]: pgmap v11714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:30 smithi104 ceph-mon[36624]: pgmap v11714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:30 smithi081 ceph-mon[31453]: pgmap v11714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:32 smithi153 ceph-mon[36569]: pgmap v11715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:32 smithi104 ceph-mon[36624]: pgmap v11715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:32 smithi081 ceph-mon[31453]: pgmap v11715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:35 smithi153 ceph-mon[36569]: pgmap v11716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:35 smithi104 ceph-mon[36624]: pgmap v11716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:35 smithi081 ceph-mon[31453]: pgmap v11716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:37 smithi153 ceph-mon[36569]: pgmap v11717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:37 smithi104 ceph-mon[36624]: pgmap v11717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:37 smithi081 ceph-mon[31453]: pgmap v11717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:39.273+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:16:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:39.274+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:16:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:39.274+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:16:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:39 smithi153 ceph-mon[36569]: pgmap v11718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:39 smithi104 ceph-mon[36624]: pgmap v11718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:39 smithi081 ceph-mon[31453]: pgmap v11718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:41 smithi153 ceph-mon[36569]: pgmap v11719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:41 smithi104 ceph-mon[36624]: pgmap v11719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:41 smithi081 ceph-mon[31453]: pgmap v11719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:43 smithi153 ceph-mon[36569]: pgmap v11720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:43 smithi104 ceph-mon[36624]: pgmap v11720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:43 smithi081 ceph-mon[31453]: pgmap v11720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:45 smithi153 ceph-mon[36569]: pgmap v11721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:45 smithi104 ceph-mon[36624]: pgmap v11721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:45 smithi081 ceph-mon[31453]: pgmap v11721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:46 smithi104 ceph-mon[36624]: pgmap v11722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:46 smithi081 ceph-mon[31453]: pgmap v11722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:46 smithi153 ceph-mon[36569]: pgmap v11722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:48 smithi153 ceph-mon[36569]: pgmap v11723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:48 smithi104 ceph-mon[36624]: pgmap v11723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:48 smithi081 ceph-mon[31453]: pgmap v11723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:51 smithi153 ceph-mon[36569]: pgmap v11724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:51 smithi104 ceph-mon[36624]: pgmap v11724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:51 smithi081 ceph-mon[31453]: pgmap v11724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:53 smithi153 ceph-mon[36569]: pgmap v11725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:53 smithi104 ceph-mon[36624]: pgmap v11725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:53 smithi081 ceph-mon[31453]: pgmap v11725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:54.281+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:16:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:54.282+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:16:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:16:54.282+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:16:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:55 smithi153 ceph-mon[36569]: pgmap v11726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:55 smithi104 ceph-mon[36624]: pgmap v11726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:55 smithi081 ceph-mon[31453]: pgmap v11726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:57 smithi153 ceph-mon[36569]: pgmap v11727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:57 smithi104 ceph-mon[36624]: pgmap v11727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:57 smithi081 ceph-mon[31453]: pgmap v11727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:16:59 smithi153 ceph-mon[36569]: pgmap v11728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:16:59 smithi104 ceph-mon[36624]: pgmap v11728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:16:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:16:59 smithi081 ceph-mon[31453]: pgmap v11728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:01 smithi153 ceph-mon[36569]: pgmap v11729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:01 smithi104 ceph-mon[36624]: pgmap v11729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:01 smithi081 ceph-mon[31453]: pgmap v11729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:03 smithi153 ceph-mon[36569]: pgmap v11730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:03 smithi104 ceph-mon[36624]: pgmap v11730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:03 smithi081 ceph-mon[31453]: pgmap v11730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:04 smithi081 ceph-mon[31453]: pgmap v11731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:04 smithi153 ceph-mon[36569]: pgmap v11731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:04 smithi104 ceph-mon[36624]: pgmap v11731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:07 smithi153 ceph-mon[36569]: pgmap v11732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:07 smithi104 ceph-mon[36624]: pgmap v11732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:07 smithi081 ceph-mon[31453]: pgmap v11732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:09.289+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:17:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:09.289+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:17:09.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:09.289+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:17:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:09 smithi153 ceph-mon[36569]: pgmap v11733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:09 smithi104 ceph-mon[36624]: pgmap v11733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:09 smithi081 ceph-mon[31453]: pgmap v11733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:11 smithi153 ceph-mon[36569]: pgmap v11734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:11 smithi104 ceph-mon[36624]: pgmap v11734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:11 smithi081 ceph-mon[31453]: pgmap v11734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:13 smithi153 ceph-mon[36569]: pgmap v11735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:13 smithi104 ceph-mon[36624]: pgmap v11735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:13 smithi081 ceph-mon[31453]: pgmap v11735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:15 smithi153 ceph-mon[36569]: pgmap v11736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:15 smithi104 ceph-mon[36624]: pgmap v11736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:15 smithi081 ceph-mon[31453]: pgmap v11736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:17.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:17 smithi153 ceph-mon[36569]: pgmap v11737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:17 smithi104 ceph-mon[36624]: pgmap v11737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:17 smithi081 ceph-mon[31453]: pgmap v11737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:18 smithi081 ceph-mon[31453]: pgmap v11738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:18 smithi153 ceph-mon[36569]: pgmap v11738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:18 smithi104 ceph-mon[36624]: pgmap v11738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:21 smithi153 ceph-mon[36569]: pgmap v11739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:17:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:21 smithi104 ceph-mon[36624]: pgmap v11739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:21.363 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:17:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:21 smithi081 ceph-mon[31453]: pgmap v11739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:17:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:17:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:17:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:17:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:17:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:17:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:17:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:17:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:17:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:17:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:23 smithi153 ceph-mon[36569]: pgmap v11740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:23 smithi153 ceph-mon[36569]: pgmap v11741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:23 smithi104 ceph-mon[36624]: pgmap v11740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:23 smithi104 ceph-mon[36624]: pgmap v11741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:23 smithi081 ceph-mon[31453]: pgmap v11740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:23 smithi081 ceph-mon[31453]: pgmap v11741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:24.297+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:17:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:24.297+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:17:24.298 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:24.297+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:17:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:25 smithi153 ceph-mon[36569]: pgmap v11742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:25 smithi104 ceph-mon[36624]: pgmap v11742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:25 smithi081 ceph-mon[31453]: pgmap v11742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:26 smithi081 ceph-mon[31453]: pgmap v11743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:26 smithi153 ceph-mon[36569]: pgmap v11743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:26 smithi104 ceph-mon[36624]: pgmap v11743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:28 smithi081 ceph-mon[31453]: pgmap v11744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:28 smithi153 ceph-mon[36569]: pgmap v11744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:28 smithi104 ceph-mon[36624]: pgmap v11744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:30 smithi081 ceph-mon[31453]: pgmap v11745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:30 smithi153 ceph-mon[36569]: pgmap v11745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:30 smithi104 ceph-mon[36624]: pgmap v11745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:32.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:32 smithi153 ceph-mon[36569]: pgmap v11746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:32 smithi081 ceph-mon[31453]: pgmap v11746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:32 smithi104 ceph-mon[36624]: pgmap v11746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:34 smithi081 ceph-mon[31453]: pgmap v11747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:34 smithi153 ceph-mon[36569]: pgmap v11747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:34 smithi104 ceph-mon[36624]: pgmap v11747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:36 smithi081 ceph-mon[31453]: pgmap v11748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:36 smithi153 ceph-mon[36569]: pgmap v11748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:36 smithi104 ceph-mon[36624]: pgmap v11748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:38 smithi081 ceph-mon[31453]: pgmap v11749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:38 smithi153 ceph-mon[36569]: pgmap v11749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:38 smithi104 ceph-mon[36624]: pgmap v11749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:39.312+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:17:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:39.312+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:17:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:39.312+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:17:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:40 smithi081 ceph-mon[31453]: pgmap v11750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:40 smithi153 ceph-mon[36569]: pgmap v11750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:40 smithi104 ceph-mon[36624]: pgmap v11750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:42 smithi081 ceph-mon[31453]: pgmap v11751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:42 smithi153 ceph-mon[36569]: pgmap v11751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:42 smithi104 ceph-mon[36624]: pgmap v11751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:44 smithi081 ceph-mon[31453]: pgmap v11752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:44 smithi153 ceph-mon[36569]: pgmap v11752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:44 smithi104 ceph-mon[36624]: pgmap v11752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:46 smithi081 ceph-mon[31453]: pgmap v11753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:46 smithi153 ceph-mon[36569]: pgmap v11753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:46 smithi104 ceph-mon[36624]: pgmap v11753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:48 smithi153 ceph-mon[36569]: pgmap v11754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:48 smithi104 ceph-mon[36624]: pgmap v11754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:48 smithi081 ceph-mon[31453]: pgmap v11754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:50 smithi153 ceph-mon[36569]: pgmap v11755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:50 smithi104 ceph-mon[36624]: pgmap v11755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:50 smithi081 ceph-mon[31453]: pgmap v11755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:52 smithi153 ceph-mon[36569]: pgmap v11756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:52 smithi104 ceph-mon[36624]: pgmap v11756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:52 smithi081 ceph-mon[31453]: pgmap v11756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:54.315+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:17:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:54.315+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:17:54.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:17:54.315+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:17:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:54 smithi153 ceph-mon[36569]: pgmap v11757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:54 smithi104 ceph-mon[36624]: pgmap v11757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:54 smithi081 ceph-mon[31453]: pgmap v11757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:56 smithi153 ceph-mon[36569]: pgmap v11758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:56 smithi104 ceph-mon[36624]: pgmap v11758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:56 smithi081 ceph-mon[31453]: pgmap v11758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:17:58 smithi153 ceph-mon[36569]: pgmap v11759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:17:58 smithi104 ceph-mon[36624]: pgmap v11759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:17:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:17:58 smithi081 ceph-mon[31453]: pgmap v11759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:00 smithi153 ceph-mon[36569]: pgmap v11760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:00 smithi104 ceph-mon[36624]: pgmap v11760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:00 smithi081 ceph-mon[31453]: pgmap v11760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:02 smithi153 ceph-mon[36569]: pgmap v11761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:02 smithi104 ceph-mon[36624]: pgmap v11761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:02 smithi081 ceph-mon[31453]: pgmap v11761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:04 smithi153 ceph-mon[36569]: pgmap v11762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:04 smithi104 ceph-mon[36624]: pgmap v11762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:04 smithi081 ceph-mon[31453]: pgmap v11762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:06 smithi153 ceph-mon[36569]: pgmap v11763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:06 smithi104 ceph-mon[36624]: pgmap v11763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:06 smithi081 ceph-mon[31453]: pgmap v11763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:08 smithi153 ceph-mon[36569]: pgmap v11764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:08 smithi104 ceph-mon[36624]: pgmap v11764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:08 smithi081 ceph-mon[31453]: pgmap v11764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:09.324+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:18:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:09.324+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:18:09.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:09.324+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:10 smithi153 ceph-mon[36569]: pgmap v11765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:10 smithi104 ceph-mon[36624]: pgmap v11765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:10 smithi081 ceph-mon[31453]: pgmap v11765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:12 smithi153 ceph-mon[36569]: pgmap v11766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:12 smithi104 ceph-mon[36624]: pgmap v11766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:12 smithi081 ceph-mon[31453]: pgmap v11766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:14 smithi153 ceph-mon[36569]: pgmap v11767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:14 smithi104 ceph-mon[36624]: pgmap v11767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:14 smithi081 ceph-mon[31453]: pgmap v11767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:16 smithi153 ceph-mon[36569]: pgmap v11768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:16 smithi104 ceph-mon[36624]: pgmap v11768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:16 smithi081 ceph-mon[31453]: pgmap v11768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:18 smithi153 ceph-mon[36569]: pgmap v11769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:18 smithi104 ceph-mon[36624]: pgmap v11769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:18 smithi081 ceph-mon[31453]: pgmap v11769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:20 smithi153 ceph-mon[36569]: pgmap v11770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:20 smithi104 ceph-mon[36624]: pgmap v11770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:20 smithi081 ceph-mon[31453]: pgmap v11770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:22 smithi153 ceph-mon[36569]: pgmap v11771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:18:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:22 smithi104 ceph-mon[36624]: pgmap v11771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:18:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:22 smithi081 ceph-mon[31453]: pgmap v11771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:18:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:18:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:18:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:23 smithi153 ceph-mon[36569]: pgmap v11772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:23 smithi153 ceph-mon[36569]: pgmap v11773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:18:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:18:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:18:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:23 smithi104 ceph-mon[36624]: pgmap v11772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:23 smithi104 ceph-mon[36624]: pgmap v11773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:18:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:18:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:18:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:23 smithi081 ceph-mon[31453]: pgmap v11772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:23 smithi081 ceph-mon[31453]: pgmap v11773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:18:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:24.328+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:18:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:24.328+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:18:24.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:24.328+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:18:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:25 smithi081 ceph-mon[31453]: pgmap v11774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:25 smithi153 ceph-mon[36569]: pgmap v11774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:25 smithi104 ceph-mon[36624]: pgmap v11774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:27 smithi081 ceph-mon[31453]: pgmap v11775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:27 smithi153 ceph-mon[36569]: pgmap v11775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:27 smithi104 ceph-mon[36624]: pgmap v11775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:29 smithi081 ceph-mon[31453]: pgmap v11776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:29 smithi153 ceph-mon[36569]: pgmap v11776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:29 smithi104 ceph-mon[36624]: pgmap v11776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:31 smithi081 ceph-mon[31453]: pgmap v11777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:31 smithi153 ceph-mon[36569]: pgmap v11777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:31 smithi104 ceph-mon[36624]: pgmap v11777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:33.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:33 smithi153 ceph-mon[36569]: pgmap v11778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:33 smithi081 ceph-mon[31453]: pgmap v11778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:33 smithi104 ceph-mon[36624]: pgmap v11778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:35 smithi081 ceph-mon[31453]: pgmap v11779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:35 smithi153 ceph-mon[36569]: pgmap v11779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:35 smithi104 ceph-mon[36624]: pgmap v11779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:37 smithi081 ceph-mon[31453]: pgmap v11780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:37 smithi153 ceph-mon[36569]: pgmap v11780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:37 smithi104 ceph-mon[36624]: pgmap v11780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:39.344+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:18:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:39.344+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:18:39.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:39.344+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:18:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:39 smithi081 ceph-mon[31453]: pgmap v11781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:39 smithi153 ceph-mon[36569]: pgmap v11781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:39 smithi104 ceph-mon[36624]: pgmap v11781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:41 smithi081 ceph-mon[31453]: pgmap v11782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:41 smithi153 ceph-mon[36569]: pgmap v11782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:41 smithi104 ceph-mon[36624]: pgmap v11782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:43.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:43 smithi104 ceph-mon[36624]: pgmap v11783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:43 smithi081 ceph-mon[31453]: pgmap v11783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:43 smithi153 ceph-mon[36569]: pgmap v11783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:45 smithi081 ceph-mon[31453]: pgmap v11784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:45 smithi153 ceph-mon[36569]: pgmap v11784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:45 smithi104 ceph-mon[36624]: pgmap v11784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:47 smithi081 ceph-mon[31453]: pgmap v11785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:47 smithi153 ceph-mon[36569]: pgmap v11785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:47 smithi104 ceph-mon[36624]: pgmap v11785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:49 smithi081 ceph-mon[31453]: pgmap v11786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:49 smithi153 ceph-mon[36569]: pgmap v11786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:49 smithi104 ceph-mon[36624]: pgmap v11786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:51 smithi081 ceph-mon[31453]: pgmap v11787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:51 smithi153 ceph-mon[36569]: pgmap v11787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:51 smithi104 ceph-mon[36624]: pgmap v11787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:53 smithi081 ceph-mon[31453]: pgmap v11788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:53 smithi153 ceph-mon[36569]: pgmap v11788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:53 smithi104 ceph-mon[36624]: pgmap v11788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:54.358+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:18:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:54.358+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:18:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:18:54.358+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:18:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:55 smithi081 ceph-mon[31453]: pgmap v11789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:55 smithi153 ceph-mon[36569]: pgmap v11789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:55 smithi104 ceph-mon[36624]: pgmap v11789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:57 smithi081 ceph-mon[31453]: pgmap v11790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:57 smithi153 ceph-mon[36569]: pgmap v11790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:57 smithi104 ceph-mon[36624]: pgmap v11790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:18:59 smithi081 ceph-mon[31453]: pgmap v11791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:18:59 smithi153 ceph-mon[36569]: pgmap v11791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:18:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:18:59 smithi104 ceph-mon[36624]: pgmap v11791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:01 smithi081 ceph-mon[31453]: pgmap v11792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:01 smithi153 ceph-mon[36569]: pgmap v11792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:01 smithi104 ceph-mon[36624]: pgmap v11792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:03 smithi081 ceph-mon[31453]: pgmap v11793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:03 smithi153 ceph-mon[36569]: pgmap v11793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:03 smithi104 ceph-mon[36624]: pgmap v11793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:05 smithi081 ceph-mon[31453]: pgmap v11794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:05 smithi153 ceph-mon[36569]: pgmap v11794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:05 smithi104 ceph-mon[36624]: pgmap v11794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:07 smithi081 ceph-mon[31453]: pgmap v11795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:07 smithi153 ceph-mon[36569]: pgmap v11795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:07 smithi104 ceph-mon[36624]: pgmap v11795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:09.369+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:19:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:09.369+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:19:09.371 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:09.369+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:19:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:09 smithi081 ceph-mon[31453]: pgmap v11796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:09 smithi153 ceph-mon[36569]: pgmap v11796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:09.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:09 smithi104 ceph-mon[36624]: pgmap v11796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:11 smithi081 ceph-mon[31453]: pgmap v11797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:11 smithi153 ceph-mon[36569]: pgmap v11797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:11 smithi104 ceph-mon[36624]: pgmap v11797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:13.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:13 smithi104 ceph-mon[36624]: pgmap v11798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:13 smithi081 ceph-mon[31453]: pgmap v11798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:13 smithi153 ceph-mon[36569]: pgmap v11798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:15 smithi081 ceph-mon[31453]: pgmap v11799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:15 smithi153 ceph-mon[36569]: pgmap v11799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:15 smithi104 ceph-mon[36624]: pgmap v11799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:16.298 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:16.297+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:19:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:16.297+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:19:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:16.297+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 169 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:19:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:16.297+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 169 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:19:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:16.297+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:19:16.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:16.297+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:19:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:16.298+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 170 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:19:16.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:16.298+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 170 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:19:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:17 smithi081 ceph-mon[31453]: pgmap v11800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:17 smithi153 ceph-mon[36569]: pgmap v11800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:17 smithi104 ceph-mon[36624]: pgmap v11800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:19 smithi081 ceph-mon[31453]: pgmap v11801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:19 smithi153 ceph-mon[36569]: pgmap v11801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:19 smithi104 ceph-mon[36624]: pgmap v11801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:21 smithi081 ceph-mon[31453]: pgmap v11802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:21 smithi153 ceph-mon[36569]: pgmap v11802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:21 smithi104 ceph-mon[36624]: pgmap v11802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:23 smithi081 ceph-mon[31453]: pgmap v11803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:23 smithi153 ceph-mon[36569]: pgmap v11803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:23 smithi104 ceph-mon[36624]: pgmap v11803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:24.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:24.382+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:19:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:24.382+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:19:24.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:24.382+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:19:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:19:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:19:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:19:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:19:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:19:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:25 smithi081 ceph-mon[31453]: pgmap v11804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:25 smithi081 ceph-mon[31453]: pgmap v11805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:19:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:19:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:19:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:25 smithi153 ceph-mon[36569]: pgmap v11804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:25 smithi153 ceph-mon[36569]: pgmap v11805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:19:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:19:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:19:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:25 smithi104 ceph-mon[36624]: pgmap v11804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:25 smithi104 ceph-mon[36624]: pgmap v11805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:19:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:27 smithi104 ceph-mon[36624]: pgmap v11806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:27 smithi081 ceph-mon[31453]: pgmap v11806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:27 smithi153 ceph-mon[36569]: pgmap v11806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:29 smithi081 ceph-mon[31453]: pgmap v11807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:29 smithi153 ceph-mon[36569]: pgmap v11807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:29 smithi104 ceph-mon[36624]: pgmap v11807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:31 smithi081 ceph-mon[31453]: pgmap v11808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:31 smithi153 ceph-mon[36569]: pgmap v11808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:31 smithi104 ceph-mon[36624]: pgmap v11808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:33.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:33 smithi153 ceph-mon[36569]: pgmap v11809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:33 smithi081 ceph-mon[31453]: pgmap v11809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:33 smithi104 ceph-mon[36624]: pgmap v11809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:35 smithi081 ceph-mon[31453]: pgmap v11810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:35 smithi153 ceph-mon[36569]: pgmap v11810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:35 smithi104 ceph-mon[36624]: pgmap v11810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:37 smithi081 ceph-mon[31453]: pgmap v11811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:37 smithi153 ceph-mon[36569]: pgmap v11811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:38.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:37 smithi104 ceph-mon[36624]: pgmap v11811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:39.397+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:19:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:39.397+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:19:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:39.397+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:19:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:39 smithi081 ceph-mon[31453]: pgmap v11812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:39 smithi153 ceph-mon[36569]: pgmap v11812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:39 smithi104 ceph-mon[36624]: pgmap v11812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:41 smithi081 ceph-mon[31453]: pgmap v11813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:41 smithi153 ceph-mon[36569]: pgmap v11813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:41 smithi104 ceph-mon[36624]: pgmap v11813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:43.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:43 smithi104 ceph-mon[36624]: pgmap v11814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:43 smithi081 ceph-mon[31453]: pgmap v11814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:43 smithi153 ceph-mon[36569]: pgmap v11814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:45 smithi081 ceph-mon[31453]: pgmap v11815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:45 smithi153 ceph-mon[36569]: pgmap v11815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:46.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:45 smithi104 ceph-mon[36624]: pgmap v11815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:47 smithi081 ceph-mon[31453]: pgmap v11816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:47 smithi153 ceph-mon[36569]: pgmap v11816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:47 smithi104 ceph-mon[36624]: pgmap v11816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:49 smithi081 ceph-mon[31453]: pgmap v11817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:49 smithi153 ceph-mon[36569]: pgmap v11817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:49 smithi104 ceph-mon[36624]: pgmap v11817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:51 smithi081 ceph-mon[31453]: pgmap v11818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:51 smithi153 ceph-mon[36569]: pgmap v11818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:51 smithi104 ceph-mon[36624]: pgmap v11818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:53 smithi081 ceph-mon[31453]: pgmap v11819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:53 smithi153 ceph-mon[36569]: pgmap v11819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:53 smithi104 ceph-mon[36624]: pgmap v11819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:54.411+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:19:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:54.411+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:19:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:19:54.411+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:19:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:55 smithi081 ceph-mon[31453]: pgmap v11820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:55 smithi153 ceph-mon[36569]: pgmap v11820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:56.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:55 smithi104 ceph-mon[36624]: pgmap v11820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:57 smithi153 ceph-mon[36569]: pgmap v11821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:57 smithi104 ceph-mon[36624]: pgmap v11821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:19:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:57 smithi081 ceph-mon[31453]: pgmap v11821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:19:59 smithi153 ceph-mon[36569]: pgmap v11822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:19:59 smithi104 ceph-mon[36624]: pgmap v11822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:00.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:19:59 smithi081 ceph-mon[31453]: pgmap v11822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:20:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:20:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:20:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:01 smithi153 ceph-mon[36569]: pgmap v11823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:01 smithi104 ceph-mon[36624]: pgmap v11823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:01 smithi081 ceph-mon[31453]: pgmap v11823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:03.976 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:03 smithi081 ceph-mon[31453]: pgmap v11824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:03 smithi153 ceph-mon[36569]: pgmap v11824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:03 smithi104 ceph-mon[36624]: pgmap v11824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:05 smithi153 ceph-mon[36569]: pgmap v11825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:05 smithi104 ceph-mon[36624]: pgmap v11825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:05 smithi081 ceph-mon[31453]: pgmap v11825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:07 smithi153 ceph-mon[36569]: pgmap v11826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:07 smithi104 ceph-mon[36624]: pgmap v11826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:07 smithi081 ceph-mon[31453]: pgmap v11826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:09.424+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:20:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:09.424+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:20:09.425 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:09.424+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:20:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:09 smithi153 ceph-mon[36569]: pgmap v11827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:09 smithi104 ceph-mon[36624]: pgmap v11827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:09 smithi081 ceph-mon[31453]: pgmap v11827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:11 smithi153 ceph-mon[36569]: pgmap v11828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:11 smithi104 ceph-mon[36624]: pgmap v11828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:11 smithi081 ceph-mon[31453]: pgmap v11828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:13.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:13 smithi081 ceph-mon[31453]: pgmap v11829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:13 smithi153 ceph-mon[36569]: pgmap v11829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:13 smithi104 ceph-mon[36624]: pgmap v11829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:15 smithi153 ceph-mon[36569]: pgmap v11830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:15 smithi104 ceph-mon[36624]: pgmap v11830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:15 smithi081 ceph-mon[31453]: pgmap v11830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:17 smithi153 ceph-mon[36569]: pgmap v11831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:17 smithi104 ceph-mon[36624]: pgmap v11831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:17 smithi081 ceph-mon[31453]: pgmap v11831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:19 smithi153 ceph-mon[36569]: pgmap v11832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:19 smithi104 ceph-mon[36624]: pgmap v11832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:19 smithi081 ceph-mon[31453]: pgmap v11832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:21 smithi153 ceph-mon[36569]: pgmap v11833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:21 smithi104 ceph-mon[36624]: pgmap v11833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:21 smithi081 ceph-mon[31453]: pgmap v11833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:23 smithi153 ceph-mon[36569]: pgmap v11834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:23 smithi104 ceph-mon[36624]: pgmap v11834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:23 smithi081 ceph-mon[31453]: pgmap v11834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:24.439+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:20:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:24.439+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:20:24.440 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:24.439+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:20:25.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:25 smithi081 ceph-mon[31453]: pgmap v11835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:25.820 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:20:25.821 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:25.821 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:25.821 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:25.821 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:25 smithi153 ceph-mon[36569]: pgmap v11835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:20:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:25 smithi104 ceph-mon[36624]: pgmap v11835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:20:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:26.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:27 smithi153 ceph-mon[36569]: pgmap v11836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:27 smithi104 ceph-mon[36624]: pgmap v11836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:27 smithi081 ceph-mon[31453]: pgmap v11836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:29 smithi153 ceph-mon[36569]: pgmap v11837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:30.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:29 smithi104 ceph-mon[36624]: pgmap v11837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:29 smithi081 ceph-mon[31453]: pgmap v11837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:31 smithi081 ceph-mon[31453]: pgmap v11838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:31 smithi153 ceph-mon[36569]: pgmap v11838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:31 smithi104 ceph-mon[36624]: pgmap v11838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:20:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:20:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:32 smithi081 ceph-mon[31453]: pgmap v11839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:32 smithi081 ceph-mon[31453]: pgmap v11840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:20:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:20:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:32 smithi153 ceph-mon[36569]: pgmap v11839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:32 smithi153 ceph-mon[36569]: pgmap v11840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:20:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:20:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:32 smithi104 ceph-mon[36624]: pgmap v11839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:32 smithi104 ceph-mon[36624]: pgmap v11840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:20:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:34 smithi153 ceph-mon[36569]: pgmap v11841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:34 smithi104 ceph-mon[36624]: pgmap v11841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:34 smithi081 ceph-mon[31453]: pgmap v11841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:36 smithi153 ceph-mon[36569]: pgmap v11842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:36 smithi104 ceph-mon[36624]: pgmap v11842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:36 smithi081 ceph-mon[31453]: pgmap v11842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:38 smithi153 ceph-mon[36569]: pgmap v11843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:38 smithi104 ceph-mon[36624]: pgmap v11843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:38 smithi081 ceph-mon[31453]: pgmap v11843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:39.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:39.447+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:20:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:39.447+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:20:39.448 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:39.447+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:20:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:40 smithi153 ceph-mon[36569]: pgmap v11844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:40 smithi104 ceph-mon[36624]: pgmap v11844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:40 smithi081 ceph-mon[31453]: pgmap v11844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:42 smithi153 ceph-mon[36569]: pgmap v11845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:42 smithi104 ceph-mon[36624]: pgmap v11845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:42 smithi081 ceph-mon[31453]: pgmap v11845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:44 smithi153 ceph-mon[36569]: pgmap v11846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:44 smithi104 ceph-mon[36624]: pgmap v11846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:44 smithi081 ceph-mon[31453]: pgmap v11846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:46 smithi153 ceph-mon[36569]: pgmap v11847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:46 smithi104 ceph-mon[36624]: pgmap v11847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:46 smithi081 ceph-mon[31453]: pgmap v11847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:48 smithi153 ceph-mon[36569]: pgmap v11848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:48 smithi104 ceph-mon[36624]: pgmap v11848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:48 smithi081 ceph-mon[31453]: pgmap v11848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:50 smithi153 ceph-mon[36569]: pgmap v11849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:50 smithi104 ceph-mon[36624]: pgmap v11849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:50 smithi081 ceph-mon[31453]: pgmap v11849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:52 smithi153 ceph-mon[36569]: pgmap v11850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:52 smithi104 ceph-mon[36624]: pgmap v11850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:52 smithi081 ceph-mon[31453]: pgmap v11850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:54.460+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:20:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:54.460+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:20:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:20:54.460+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:20:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:54 smithi153 ceph-mon[36569]: pgmap v11851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:54 smithi104 ceph-mon[36624]: pgmap v11851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:54 smithi081 ceph-mon[31453]: pgmap v11851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:56 smithi153 ceph-mon[36569]: pgmap v11852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:56 smithi104 ceph-mon[36624]: pgmap v11852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:56.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:56 smithi081 ceph-mon[31453]: pgmap v11852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:20:58 smithi153 ceph-mon[36569]: pgmap v11853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:20:58 smithi104 ceph-mon[36624]: pgmap v11853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:20:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:20:58 smithi081 ceph-mon[31453]: pgmap v11853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:00 smithi104 ceph-mon[36624]: pgmap v11854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:00 smithi081 ceph-mon[31453]: pgmap v11854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:00 smithi153 ceph-mon[36569]: pgmap v11854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:02 smithi104 ceph-mon[36624]: pgmap v11855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:02 smithi081 ceph-mon[31453]: pgmap v11855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:02 smithi153 ceph-mon[36569]: pgmap v11855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:04 smithi104 ceph-mon[36624]: pgmap v11856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:04.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:04 smithi153 ceph-mon[36569]: pgmap v11856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:04 smithi081 ceph-mon[31453]: pgmap v11856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:06 smithi104 ceph-mon[36624]: pgmap v11857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:06 smithi081 ceph-mon[31453]: pgmap v11857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:06 smithi153 ceph-mon[36569]: pgmap v11857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:08 smithi104 ceph-mon[36624]: pgmap v11858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:08 smithi081 ceph-mon[31453]: pgmap v11858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:08 smithi153 ceph-mon[36569]: pgmap v11858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:09.475+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:21:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:09.475+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:21:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:09.475+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:21:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:10 smithi081 ceph-mon[31453]: pgmap v11859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:10 smithi153 ceph-mon[36569]: pgmap v11859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:10 smithi104 ceph-mon[36624]: pgmap v11859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:12 smithi081 ceph-mon[31453]: pgmap v11860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:12 smithi153 ceph-mon[36569]: pgmap v11860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:12 smithi104 ceph-mon[36624]: pgmap v11860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:14.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:14 smithi104 ceph-mon[36624]: pgmap v11861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:14 smithi081 ceph-mon[31453]: pgmap v11861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:14 smithi153 ceph-mon[36569]: pgmap v11861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:16 smithi081 ceph-mon[31453]: pgmap v11862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:16 smithi153 ceph-mon[36569]: pgmap v11862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:16 smithi104 ceph-mon[36624]: pgmap v11862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:18 smithi081 ceph-mon[31453]: pgmap v11863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:18 smithi153 ceph-mon[36569]: pgmap v11863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:18 smithi104 ceph-mon[36624]: pgmap v11863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:20 smithi081 ceph-mon[31453]: pgmap v11864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:20 smithi153 ceph-mon[36569]: pgmap v11864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:20 smithi104 ceph-mon[36624]: pgmap v11864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:22 smithi081 ceph-mon[31453]: pgmap v11865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:22 smithi153 ceph-mon[36569]: pgmap v11865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:22 smithi104 ceph-mon[36624]: pgmap v11865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:24.490+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:21:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:24.490+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:21:24.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:24.490+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:21:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:24 smithi081 ceph-mon[31453]: pgmap v11866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:24 smithi153 ceph-mon[36569]: pgmap v11866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:24 smithi104 ceph-mon[36624]: pgmap v11866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:26 smithi081 ceph-mon[31453]: pgmap v11867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:26 smithi153 ceph-mon[36569]: pgmap v11867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:26 smithi104 ceph-mon[36624]: pgmap v11867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:28.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:28 smithi081 ceph-mon[31453]: pgmap v11868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:28 smithi153 ceph-mon[36569]: pgmap v11868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:28 smithi104 ceph-mon[36624]: pgmap v11868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:30 smithi081 ceph-mon[31453]: pgmap v11869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:30 smithi153 ceph-mon[36569]: pgmap v11869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:30 smithi104 ceph-mon[36624]: pgmap v11869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:32 smithi081 ceph-mon[31453]: pgmap v11870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:21:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:32 smithi153 ceph-mon[36569]: pgmap v11870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:21:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:32 smithi104 ceph-mon[36624]: pgmap v11870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:21:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:21:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:21:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:33 smithi081 ceph-mon[31453]: pgmap v11871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:33 smithi081 ceph-mon[31453]: pgmap v11872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:33.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:21:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:21:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:33 smithi153 ceph-mon[36569]: pgmap v11871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:33 smithi153 ceph-mon[36569]: pgmap v11872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:21:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:21:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:33 smithi104 ceph-mon[36624]: pgmap v11871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:33 smithi104 ceph-mon[36624]: pgmap v11872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:21:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:36 smithi153 ceph-mon[36569]: pgmap v11873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:36 smithi104 ceph-mon[36624]: pgmap v11873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:36 smithi081 ceph-mon[31453]: pgmap v11873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:38 smithi104 ceph-mon[36624]: pgmap v11874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:38 smithi081 ceph-mon[31453]: pgmap v11874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:38 smithi153 ceph-mon[36569]: pgmap v11874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:39.500+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:21:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:39.500+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:21:39.501 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:39.500+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:21:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:40 smithi104 ceph-mon[36624]: pgmap v11875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:40 smithi081 ceph-mon[31453]: pgmap v11875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:40 smithi153 ceph-mon[36569]: pgmap v11875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:42 smithi104 ceph-mon[36624]: pgmap v11876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:42 smithi081 ceph-mon[31453]: pgmap v11876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:42 smithi153 ceph-mon[36569]: pgmap v11876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:44 smithi104 ceph-mon[36624]: pgmap v11877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:44 smithi081 ceph-mon[31453]: pgmap v11877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:44 smithi153 ceph-mon[36569]: pgmap v11877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:46 smithi104 ceph-mon[36624]: pgmap v11878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:46 smithi081 ceph-mon[31453]: pgmap v11878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:46 smithi153 ceph-mon[36569]: pgmap v11878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:48 smithi081 ceph-mon[31453]: pgmap v11879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:48 smithi153 ceph-mon[36569]: pgmap v11879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:48 smithi104 ceph-mon[36624]: pgmap v11879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:50 smithi081 ceph-mon[31453]: pgmap v11880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:50 smithi153 ceph-mon[36569]: pgmap v11880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:50 smithi104 ceph-mon[36624]: pgmap v11880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:52 smithi081 ceph-mon[31453]: pgmap v11881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:52 smithi153 ceph-mon[36569]: pgmap v11881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:52 smithi104 ceph-mon[36624]: pgmap v11881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:53 smithi081 ceph-mon[31453]: pgmap v11882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:53 smithi153 ceph-mon[36569]: pgmap v11882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:53 smithi104 ceph-mon[36624]: pgmap v11882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:54.515+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:21:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:54.515+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:21:54.516 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:21:54.515+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:21:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:56 smithi104 ceph-mon[36624]: pgmap v11883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:56 smithi081 ceph-mon[31453]: pgmap v11883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:56 smithi153 ceph-mon[36569]: pgmap v11883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:21:58 smithi104 ceph-mon[36624]: pgmap v11884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:21:58 smithi081 ceph-mon[31453]: pgmap v11884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:21:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:21:58 smithi153 ceph-mon[36569]: pgmap v11884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:00 smithi104 ceph-mon[36624]: pgmap v11885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:00 smithi081 ceph-mon[31453]: pgmap v11885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:00 smithi153 ceph-mon[36569]: pgmap v11885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:02 smithi104 ceph-mon[36624]: pgmap v11886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:02 smithi081 ceph-mon[31453]: pgmap v11886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:02 smithi153 ceph-mon[36569]: pgmap v11886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:04 smithi081 ceph-mon[31453]: pgmap v11887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:04 smithi153 ceph-mon[36569]: pgmap v11887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:04 smithi104 ceph-mon[36624]: pgmap v11887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:06.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:06 smithi153 ceph-mon[36569]: pgmap v11888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:06 smithi081 ceph-mon[31453]: pgmap v11888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:06 smithi104 ceph-mon[36624]: pgmap v11888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:08 smithi081 ceph-mon[31453]: pgmap v11889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:08 smithi153 ceph-mon[36569]: pgmap v11889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:08 smithi104 ceph-mon[36624]: pgmap v11889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:09 smithi081 ceph-mon[31453]: pgmap v11890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:09.531 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:09.530+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:22:09.531 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:09.530+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:22:09.531 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:09.530+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:22:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:09 smithi153 ceph-mon[36569]: pgmap v11890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:09 smithi104 ceph-mon[36624]: pgmap v11890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:12 smithi104 ceph-mon[36624]: pgmap v11891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:12 smithi081 ceph-mon[31453]: pgmap v11891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:12 smithi153 ceph-mon[36569]: pgmap v11891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:14 smithi104 ceph-mon[36624]: pgmap v11892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:14 smithi081 ceph-mon[31453]: pgmap v11892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:14 smithi153 ceph-mon[36569]: pgmap v11892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:16.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:16.328+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:22:16.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:16.328+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:22:16.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:16.329+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 171 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:22:16.330 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:16.329+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 171 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:22:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:16 smithi081 ceph-mon[31453]: pgmap v11893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:16 smithi153 ceph-mon[36569]: pgmap v11893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:16 smithi104 ceph-mon[36624]: pgmap v11893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:18 smithi081 ceph-mon[31453]: pgmap v11894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:18 smithi153 ceph-mon[36569]: pgmap v11894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:18 smithi104 ceph-mon[36624]: pgmap v11894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:20 smithi081 ceph-mon[31453]: pgmap v11895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:20 smithi153 ceph-mon[36569]: pgmap v11895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:20.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:20 smithi104 ceph-mon[36624]: pgmap v11895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:22 smithi081 ceph-mon[31453]: pgmap v11896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:22 smithi153 ceph-mon[36569]: pgmap v11896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:22 smithi104 ceph-mon[36624]: pgmap v11896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:23 smithi081 ceph-mon[31453]: pgmap v11897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:23 smithi153 ceph-mon[36569]: pgmap v11897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:23 smithi104 ceph-mon[36624]: pgmap v11897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:24.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:24.546+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:22:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:24.546+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:22:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:24.546+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:22:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:26 smithi104 ceph-mon[36624]: pgmap v11898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:26 smithi081 ceph-mon[31453]: pgmap v11898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:26 smithi153 ceph-mon[36569]: pgmap v11898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:28 smithi081 ceph-mon[31453]: pgmap v11899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:28 smithi153 ceph-mon[36569]: pgmap v11899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:28 smithi104 ceph-mon[36624]: pgmap v11899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:30 smithi081 ceph-mon[31453]: pgmap v11900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:30 smithi153 ceph-mon[36569]: pgmap v11900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:30 smithi104 ceph-mon[36624]: pgmap v11900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:32 smithi081 ceph-mon[31453]: pgmap v11901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:32 smithi153 ceph-mon[36569]: pgmap v11901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:32 smithi104 ceph-mon[36624]: pgmap v11901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:34 smithi081 ceph-mon[31453]: pgmap v11902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:22:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:34 smithi153 ceph-mon[36569]: pgmap v11902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:22:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:34 smithi104 ceph-mon[36624]: pgmap v11902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:22:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:22:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:22:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:35 smithi081 ceph-mon[31453]: pgmap v11903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:35 smithi081 ceph-mon[31453]: pgmap v11904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:22:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:22:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:22:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:35 smithi153 ceph-mon[36569]: pgmap v11903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:35 smithi153 ceph-mon[36569]: pgmap v11904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:22:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:22:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:22:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:35 smithi104 ceph-mon[36624]: pgmap v11903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:35 smithi104 ceph-mon[36624]: pgmap v11904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:22:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:37 smithi104 ceph-mon[36624]: pgmap v11905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:37 smithi081 ceph-mon[31453]: pgmap v11905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:37 smithi153 ceph-mon[36569]: pgmap v11905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:39.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:22:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:39.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:22:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:39.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:22:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:39 smithi104 ceph-mon[36624]: pgmap v11906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:39 smithi081 ceph-mon[31453]: pgmap v11906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:39 smithi153 ceph-mon[36569]: pgmap v11906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:41 smithi081 ceph-mon[31453]: pgmap v11907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:41 smithi153 ceph-mon[36569]: pgmap v11907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:41 smithi104 ceph-mon[36624]: pgmap v11907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:43 smithi081 ceph-mon[31453]: pgmap v11908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:43 smithi153 ceph-mon[36569]: pgmap v11908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:43 smithi104 ceph-mon[36624]: pgmap v11908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:45.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:45 smithi104 ceph-mon[36624]: pgmap v11909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:45 smithi081 ceph-mon[31453]: pgmap v11909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:45 smithi153 ceph-mon[36569]: pgmap v11909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:47 smithi081 ceph-mon[31453]: pgmap v11910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:47 smithi153 ceph-mon[36569]: pgmap v11910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:47 smithi104 ceph-mon[36624]: pgmap v11910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:49 smithi081 ceph-mon[31453]: pgmap v11911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:49 smithi153 ceph-mon[36569]: pgmap v11911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:49 smithi104 ceph-mon[36624]: pgmap v11911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:51 smithi081 ceph-mon[31453]: pgmap v11912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:51 smithi153 ceph-mon[36569]: pgmap v11912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:51 smithi104 ceph-mon[36624]: pgmap v11912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:53 smithi081 ceph-mon[31453]: pgmap v11913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:53 smithi153 ceph-mon[36569]: pgmap v11913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:53 smithi104 ceph-mon[36624]: pgmap v11913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:54.577+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:22:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:54.577+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:22:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:22:54.577+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:22:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:55 smithi081 ceph-mon[31453]: pgmap v11914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:56.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:55 smithi153 ceph-mon[36569]: pgmap v11914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:55 smithi104 ceph-mon[36624]: pgmap v11914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:57.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:57 smithi081 ceph-mon[31453]: pgmap v11915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:57 smithi153 ceph-mon[36569]: pgmap v11915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:57 smithi104 ceph-mon[36624]: pgmap v11915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:22:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:22:59 smithi081 ceph-mon[31453]: pgmap v11916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:22:59 smithi153 ceph-mon[36569]: pgmap v11916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:22:59 smithi104 ceph-mon[36624]: pgmap v11916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:01 smithi153 ceph-mon[36569]: pgmap v11917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:01 smithi104 ceph-mon[36624]: pgmap v11917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:01 smithi081 ceph-mon[31453]: pgmap v11917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:03 smithi081 ceph-mon[31453]: pgmap v11918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:03 smithi153 ceph-mon[36569]: pgmap v11918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:03 smithi104 ceph-mon[36624]: pgmap v11918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:05 smithi081 ceph-mon[31453]: pgmap v11919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:05 smithi153 ceph-mon[36569]: pgmap v11919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:05 smithi104 ceph-mon[36624]: pgmap v11919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:07 smithi081 ceph-mon[31453]: pgmap v11920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:07 smithi153 ceph-mon[36569]: pgmap v11920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:07 smithi104 ceph-mon[36624]: pgmap v11920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:09.592+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:23:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:09.592+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:23:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:09.592+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:23:09.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:09 smithi153 ceph-mon[36569]: pgmap v11921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:09 smithi081 ceph-mon[31453]: pgmap v11921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:09 smithi104 ceph-mon[36624]: pgmap v11921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:11 smithi081 ceph-mon[31453]: pgmap v11922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:11 smithi153 ceph-mon[36569]: pgmap v11922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:11 smithi104 ceph-mon[36624]: pgmap v11922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:13 smithi081 ceph-mon[31453]: pgmap v11923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:13 smithi153 ceph-mon[36569]: pgmap v11923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:13 smithi104 ceph-mon[36624]: pgmap v11923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:15.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:15 smithi104 ceph-mon[36624]: pgmap v11924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:15 smithi081 ceph-mon[31453]: pgmap v11924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:15 smithi153 ceph-mon[36569]: pgmap v11924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:17 smithi081 ceph-mon[31453]: pgmap v11925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:17 smithi153 ceph-mon[36569]: pgmap v11925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:17 smithi104 ceph-mon[36624]: pgmap v11925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:19 smithi081 ceph-mon[31453]: pgmap v11926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:19 smithi153 ceph-mon[36569]: pgmap v11926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:19 smithi104 ceph-mon[36624]: pgmap v11926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:21 smithi081 ceph-mon[31453]: pgmap v11927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:21 smithi153 ceph-mon[36569]: pgmap v11927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:21 smithi104 ceph-mon[36624]: pgmap v11927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:23 smithi081 ceph-mon[31453]: pgmap v11928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:23 smithi153 ceph-mon[36569]: pgmap v11928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:23 smithi104 ceph-mon[36624]: pgmap v11928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:24.602+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:23:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:24.602+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:23:24.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:24.602+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:23:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:25 smithi081 ceph-mon[31453]: pgmap v11929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:25 smithi153 ceph-mon[36569]: pgmap v11929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:25 smithi104 ceph-mon[36624]: pgmap v11929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:27 smithi081 ceph-mon[31453]: pgmap v11930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:27 smithi153 ceph-mon[36569]: pgmap v11930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:27 smithi104 ceph-mon[36624]: pgmap v11930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:29 smithi153 ceph-mon[36569]: pgmap v11931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:30.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:29 smithi104 ceph-mon[36624]: pgmap v11931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:29 smithi081 ceph-mon[31453]: pgmap v11931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:31 smithi153 ceph-mon[36569]: pgmap v11932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:32.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:31 smithi104 ceph-mon[36624]: pgmap v11932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:31 smithi081 ceph-mon[31453]: pgmap v11932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:33 smithi153 ceph-mon[36569]: pgmap v11933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:34.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:33 smithi104 ceph-mon[36624]: pgmap v11933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:33 smithi081 ceph-mon[31453]: pgmap v11933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:35 smithi153 ceph-mon[36569]: pgmap v11934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:35 smithi104 ceph-mon[36624]: pgmap v11934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:36.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:35 smithi081 ceph-mon[31453]: pgmap v11934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:23:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:23:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:23:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:36 smithi153 ceph-mon[36569]: pgmap v11935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:36 smithi153 ceph-mon[36569]: pgmap v11936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:23:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:23:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:23:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:36 smithi104 ceph-mon[36624]: pgmap v11935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:36 smithi104 ceph-mon[36624]: pgmap v11936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:23:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:23:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:23:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:36 smithi081 ceph-mon[31453]: pgmap v11935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:36 smithi081 ceph-mon[31453]: pgmap v11936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:23:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:39 smithi081 ceph-mon[31453]: pgmap v11937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:39 smithi153 ceph-mon[36569]: pgmap v11937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:39 smithi104 ceph-mon[36624]: pgmap v11937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:39.618+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:23:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:39.618+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:23:39.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:39.618+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:23:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:41 smithi081 ceph-mon[31453]: pgmap v11938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:41 smithi153 ceph-mon[36569]: pgmap v11938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:41 smithi104 ceph-mon[36624]: pgmap v11938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:43 smithi081 ceph-mon[31453]: pgmap v11939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:43 smithi153 ceph-mon[36569]: pgmap v11939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:43 smithi104 ceph-mon[36624]: pgmap v11939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:45.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:45 smithi104 ceph-mon[36624]: pgmap v11940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:45 smithi081 ceph-mon[31453]: pgmap v11940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:45 smithi153 ceph-mon[36569]: pgmap v11940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:47 smithi081 ceph-mon[31453]: pgmap v11941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:47 smithi153 ceph-mon[36569]: pgmap v11941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:47 smithi104 ceph-mon[36624]: pgmap v11941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:49 smithi081 ceph-mon[31453]: pgmap v11942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:49 smithi153 ceph-mon[36569]: pgmap v11942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:49 smithi104 ceph-mon[36624]: pgmap v11942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:50 smithi153 ceph-mon[36569]: pgmap v11943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:50 smithi104 ceph-mon[36624]: pgmap v11943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:50 smithi081 ceph-mon[31453]: pgmap v11943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:53 smithi081 ceph-mon[31453]: pgmap v11944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:53 smithi153 ceph-mon[36569]: pgmap v11944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:53 smithi104 ceph-mon[36624]: pgmap v11944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:54.621+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:23:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:54.621+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:23:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:23:54.621+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:23:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:55 smithi081 ceph-mon[31453]: pgmap v11945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:55 smithi153 ceph-mon[36569]: pgmap v11945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:55 smithi104 ceph-mon[36624]: pgmap v11945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:57 smithi081 ceph-mon[31453]: pgmap v11946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:57 smithi153 ceph-mon[36569]: pgmap v11946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:57 smithi104 ceph-mon[36624]: pgmap v11946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:23:59 smithi081 ceph-mon[31453]: pgmap v11947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:23:59 smithi153 ceph-mon[36569]: pgmap v11947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:23:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:23:59 smithi104 ceph-mon[36624]: pgmap v11947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:01 smithi081 ceph-mon[31453]: pgmap v11948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:01 smithi153 ceph-mon[36569]: pgmap v11948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:01 smithi104 ceph-mon[36624]: pgmap v11948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:02 smithi153 ceph-mon[36569]: pgmap v11949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:02 smithi104 ceph-mon[36624]: pgmap v11949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:02 smithi081 ceph-mon[31453]: pgmap v11949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:05 smithi081 ceph-mon[31453]: pgmap v11950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:05 smithi153 ceph-mon[36569]: pgmap v11950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:05 smithi104 ceph-mon[36624]: pgmap v11950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:07 smithi081 ceph-mon[31453]: pgmap v11951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:07 smithi153 ceph-mon[36569]: pgmap v11951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:07 smithi104 ceph-mon[36624]: pgmap v11951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:09 smithi081 ceph-mon[31453]: pgmap v11952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:09 smithi153 ceph-mon[36569]: pgmap v11952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:09 smithi104 ceph-mon[36624]: pgmap v11952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:09.634+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:24:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:09.634+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:24:09.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:09.634+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:24:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:11 smithi081 ceph-mon[31453]: pgmap v11953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:11 smithi153 ceph-mon[36569]: pgmap v11953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:11 smithi104 ceph-mon[36624]: pgmap v11953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:12 smithi153 ceph-mon[36569]: pgmap v11954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:12 smithi104 ceph-mon[36624]: pgmap v11954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:12 smithi081 ceph-mon[31453]: pgmap v11954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:15 smithi081 ceph-mon[31453]: pgmap v11955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:15 smithi153 ceph-mon[36569]: pgmap v11955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:15 smithi104 ceph-mon[36624]: pgmap v11955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:17 smithi081 ceph-mon[31453]: pgmap v11956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:17 smithi153 ceph-mon[36569]: pgmap v11956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:17 smithi104 ceph-mon[36624]: pgmap v11956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:19 smithi081 ceph-mon[31453]: pgmap v11957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:19 smithi153 ceph-mon[36569]: pgmap v11957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:19 smithi104 ceph-mon[36624]: pgmap v11957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:21 smithi081 ceph-mon[31453]: pgmap v11958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:21 smithi153 ceph-mon[36569]: pgmap v11958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:21 smithi104 ceph-mon[36624]: pgmap v11958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:22 smithi081 ceph-mon[31453]: pgmap v11959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:22 smithi153 ceph-mon[36569]: pgmap v11959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:22 smithi104 ceph-mon[36624]: pgmap v11959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:24.649+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:24:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:24.649+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:24:24.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:24.649+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:24:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:25 smithi081 ceph-mon[31453]: pgmap v11960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:25 smithi153 ceph-mon[36569]: pgmap v11960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:25 smithi104 ceph-mon[36624]: pgmap v11960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:26 smithi153 ceph-mon[36569]: pgmap v11961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:26 smithi104 ceph-mon[36624]: pgmap v11961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:26 smithi081 ceph-mon[31453]: pgmap v11961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:29 smithi081 ceph-mon[31453]: pgmap v11962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:29 smithi153 ceph-mon[36569]: pgmap v11962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:29 smithi104 ceph-mon[36624]: pgmap v11962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:31 smithi081 ceph-mon[31453]: pgmap v11963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:31 smithi153 ceph-mon[36569]: pgmap v11963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:31 smithi104 ceph-mon[36624]: pgmap v11963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:33 smithi081 ceph-mon[31453]: pgmap v11964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:33 smithi153 ceph-mon[36569]: pgmap v11964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:33 smithi104 ceph-mon[36624]: pgmap v11964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:34 smithi081 ceph-mon[31453]: pgmap v11965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:34 smithi153 ceph-mon[36569]: pgmap v11965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:34 smithi104 ceph-mon[36624]: pgmap v11965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:37 smithi081 ceph-mon[31453]: pgmap v11966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:37.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:37 smithi153 ceph-mon[36569]: pgmap v11966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:37 smithi104 ceph-mon[36624]: pgmap v11966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:24:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:24:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:24:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:24:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:24:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:24:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:39 smithi081 ceph-mon[31453]: pgmap v11967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:39 smithi081 ceph-mon[31453]: pgmap v11968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:39.458 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:39 smithi153 ceph-mon[36569]: pgmap v11967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:39.459 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:39 smithi153 ceph-mon[36569]: pgmap v11968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:39 smithi104 ceph-mon[36624]: pgmap v11967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:39 smithi104 ceph-mon[36624]: pgmap v11968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:39.661 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:39.660+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:24:39.661 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:39.660+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:24:39.661 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:39.660+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:24:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:40 smithi081 ceph-mon[31453]: pgmap v11969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:40 smithi153 ceph-mon[36569]: pgmap v11969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:40 smithi104 ceph-mon[36624]: pgmap v11969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:42 smithi153 ceph-mon[36569]: pgmap v11970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:42 smithi104 ceph-mon[36624]: pgmap v11970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:42 smithi081 ceph-mon[31453]: pgmap v11970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:44 smithi153 ceph-mon[36569]: pgmap v11971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:45.064 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:44 smithi081 ceph-mon[31453]: pgmap v11971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:44 smithi104 ceph-mon[36624]: pgmap v11971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:46 smithi153 ceph-mon[36569]: pgmap v11972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:46 smithi104 ceph-mon[36624]: pgmap v11972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:46 smithi081 ceph-mon[31453]: pgmap v11972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:48 smithi153 ceph-mon[36569]: pgmap v11973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:48 smithi104 ceph-mon[36624]: pgmap v11973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:48 smithi081 ceph-mon[31453]: pgmap v11973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:50 smithi153 ceph-mon[36569]: pgmap v11974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:50 smithi104 ceph-mon[36624]: pgmap v11974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:50 smithi081 ceph-mon[31453]: pgmap v11974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:52 smithi153 ceph-mon[36569]: pgmap v11975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:52 smithi104 ceph-mon[36624]: pgmap v11975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:52 smithi081 ceph-mon[31453]: pgmap v11975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:54.675+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:24:54.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:54.675+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:24:54.677 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:24:54.675+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:24:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:54 smithi153 ceph-mon[36569]: pgmap v11976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:54 smithi104 ceph-mon[36624]: pgmap v11976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:54 smithi081 ceph-mon[31453]: pgmap v11976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:56 smithi153 ceph-mon[36569]: pgmap v11977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:56 smithi104 ceph-mon[36624]: pgmap v11977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:56 smithi081 ceph-mon[31453]: pgmap v11977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:59.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:24:58 smithi153 ceph-mon[36569]: pgmap v11978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:24:58 smithi104 ceph-mon[36624]: pgmap v11978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:24:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:24:58 smithi081 ceph-mon[31453]: pgmap v11978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:00 smithi153 ceph-mon[36569]: pgmap v11979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:00 smithi104 ceph-mon[36624]: pgmap v11979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:00 smithi081 ceph-mon[31453]: pgmap v11979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:02 smithi153 ceph-mon[36569]: pgmap v11980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:02 smithi104 ceph-mon[36624]: pgmap v11980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:02 smithi081 ceph-mon[31453]: pgmap v11980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:04 smithi153 ceph-mon[36569]: pgmap v11981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:04 smithi104 ceph-mon[36624]: pgmap v11981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:04 smithi081 ceph-mon[31453]: pgmap v11981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:06 smithi153 ceph-mon[36569]: pgmap v11982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:06 smithi104 ceph-mon[36624]: pgmap v11982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:06 smithi081 ceph-mon[31453]: pgmap v11982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:08 smithi153 ceph-mon[36569]: pgmap v11983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:08 smithi104 ceph-mon[36624]: pgmap v11983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:08 smithi081 ceph-mon[31453]: pgmap v11983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:09.681+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:25:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:09.681+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:25:09.682 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:09.681+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:25:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:10 smithi153 ceph-mon[36569]: pgmap v11984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:10 smithi104 ceph-mon[36624]: pgmap v11984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:10 smithi081 ceph-mon[31453]: pgmap v11984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:12 smithi153 ceph-mon[36569]: pgmap v11985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:12 smithi104 ceph-mon[36624]: pgmap v11985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:12 smithi081 ceph-mon[31453]: pgmap v11985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:14 smithi153 ceph-mon[36569]: pgmap v11986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:14 smithi104 ceph-mon[36624]: pgmap v11986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:14 smithi081 ceph-mon[31453]: pgmap v11986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:16.357+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:25:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:16.357+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:25:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:16.357+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 172 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:25:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:16.357+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 172 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:25:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:16 smithi153 ceph-mon[36569]: pgmap v11987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:16 smithi104 ceph-mon[36624]: pgmap v11987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:16 smithi081 ceph-mon[31453]: pgmap v11987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:18 smithi153 ceph-mon[36569]: pgmap v11988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:18 smithi104 ceph-mon[36624]: pgmap v11988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:18 smithi081 ceph-mon[31453]: pgmap v11988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:20 smithi104 ceph-mon[36624]: pgmap v11989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:20 smithi081 ceph-mon[31453]: pgmap v11989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:20 smithi153 ceph-mon[36569]: pgmap v11989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:22 smithi104 ceph-mon[36624]: pgmap v11990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:22 smithi081 ceph-mon[31453]: pgmap v11990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:22 smithi153 ceph-mon[36569]: pgmap v11990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:24.696+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:25:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:24.696+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:25:24.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:24.696+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:25:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:24 smithi104 ceph-mon[36624]: pgmap v11991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:24 smithi081 ceph-mon[31453]: pgmap v11991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:24 smithi153 ceph-mon[36569]: pgmap v11991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:26 smithi104 ceph-mon[36624]: pgmap v11992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:26 smithi081 ceph-mon[31453]: pgmap v11992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:26 smithi153 ceph-mon[36569]: pgmap v11992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:29.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:28 smithi104 ceph-mon[36624]: pgmap v11993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:28 smithi081 ceph-mon[31453]: pgmap v11993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:28 smithi153 ceph-mon[36569]: pgmap v11993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:31.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:30 smithi081 ceph-mon[31453]: pgmap v11994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:30 smithi153 ceph-mon[36569]: pgmap v11994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:30 smithi104 ceph-mon[36624]: pgmap v11994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:32 smithi081 ceph-mon[31453]: pgmap v11995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:32 smithi153 ceph-mon[36569]: pgmap v11995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:32 smithi104 ceph-mon[36624]: pgmap v11995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:34 smithi081 ceph-mon[31453]: pgmap v11996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:34 smithi153 ceph-mon[36569]: pgmap v11996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:34 smithi104 ceph-mon[36624]: pgmap v11996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:36 smithi081 ceph-mon[31453]: pgmap v11997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:36 smithi153 ceph-mon[36569]: pgmap v11997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:36 smithi104 ceph-mon[36624]: pgmap v11997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:38 smithi081 ceph-mon[31453]: pgmap v11998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:38 smithi153 ceph-mon[36569]: pgmap v11998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:38 smithi104 ceph-mon[36624]: pgmap v11998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:25:39.712 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:39.712+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:25:39.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:39.712+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:25:39.713 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:39.712+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:25:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:25:40.208 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:40.209 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:40.209 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:25:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:25:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:25:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:25:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:40 smithi081 ceph-mon[31453]: pgmap v11999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:40 smithi081 ceph-mon[31453]: pgmap v12000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:40 smithi153 ceph-mon[36569]: pgmap v11999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:40 smithi153 ceph-mon[36569]: pgmap v12000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:40 smithi104 ceph-mon[36624]: pgmap v11999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:40 smithi104 ceph-mon[36624]: pgmap v12000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:42 smithi081 ceph-mon[31453]: pgmap v12001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:42 smithi153 ceph-mon[36569]: pgmap v12001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:42 smithi104 ceph-mon[36624]: pgmap v12001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:44 smithi081 ceph-mon[31453]: pgmap v12002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:44 smithi153 ceph-mon[36569]: pgmap v12002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:44 smithi104 ceph-mon[36624]: pgmap v12002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:46 smithi081 ceph-mon[31453]: pgmap v12003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:46 smithi153 ceph-mon[36569]: pgmap v12003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:46 smithi104 ceph-mon[36624]: pgmap v12003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:48 smithi081 ceph-mon[31453]: pgmap v12004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:48 smithi153 ceph-mon[36569]: pgmap v12004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:48 smithi104 ceph-mon[36624]: pgmap v12004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:50 smithi153 ceph-mon[36569]: pgmap v12005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:50 smithi104 ceph-mon[36624]: pgmap v12005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:50 smithi081 ceph-mon[31453]: pgmap v12005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:52 smithi153 ceph-mon[36569]: pgmap v12006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:52 smithi104 ceph-mon[36624]: pgmap v12006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:52 smithi081 ceph-mon[31453]: pgmap v12006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:54.727+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:25:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:54.727+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:25:54.728 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:25:54.727+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:25:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:54 smithi153 ceph-mon[36569]: pgmap v12007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:54 smithi104 ceph-mon[36624]: pgmap v12007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:54 smithi081 ceph-mon[31453]: pgmap v12007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:56 smithi153 ceph-mon[36569]: pgmap v12008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:56 smithi104 ceph-mon[36624]: pgmap v12008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:56 smithi081 ceph-mon[31453]: pgmap v12008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:25:58 smithi153 ceph-mon[36569]: pgmap v12009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:25:58 smithi104 ceph-mon[36624]: pgmap v12009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:25:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:25:58 smithi081 ceph-mon[31453]: pgmap v12009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:00 smithi153 ceph-mon[36569]: pgmap v12010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:00 smithi104 ceph-mon[36624]: pgmap v12010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:00 smithi081 ceph-mon[31453]: pgmap v12010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:02 smithi153 ceph-mon[36569]: pgmap v12011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:02 smithi104 ceph-mon[36624]: pgmap v12011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:02 smithi081 ceph-mon[31453]: pgmap v12011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:04 smithi153 ceph-mon[36569]: pgmap v12012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:04 smithi104 ceph-mon[36624]: pgmap v12012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:04 smithi081 ceph-mon[31453]: pgmap v12012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:06 smithi153 ceph-mon[36569]: pgmap v12013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:06 smithi104 ceph-mon[36624]: pgmap v12013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:06 smithi081 ceph-mon[31453]: pgmap v12013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:08 smithi153 ceph-mon[36569]: pgmap v12014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:08 smithi104 ceph-mon[36624]: pgmap v12014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:08 smithi081 ceph-mon[31453]: pgmap v12014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:09.740+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:26:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:09.740+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:26:09.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:09.740+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:26:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:10 smithi153 ceph-mon[36569]: pgmap v12015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:10 smithi104 ceph-mon[36624]: pgmap v12015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:10 smithi081 ceph-mon[31453]: pgmap v12015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:12 smithi153 ceph-mon[36569]: pgmap v12016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:12 smithi104 ceph-mon[36624]: pgmap v12016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:12 smithi081 ceph-mon[31453]: pgmap v12016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:14 smithi153 ceph-mon[36569]: pgmap v12017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:14 smithi104 ceph-mon[36624]: pgmap v12017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:14 smithi081 ceph-mon[31453]: pgmap v12017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:16 smithi153 ceph-mon[36569]: pgmap v12018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:16 smithi104 ceph-mon[36624]: pgmap v12018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:16 smithi081 ceph-mon[31453]: pgmap v12018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:18 smithi153 ceph-mon[36569]: pgmap v12019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:18 smithi104 ceph-mon[36624]: pgmap v12019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:18 smithi081 ceph-mon[31453]: pgmap v12019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:20 smithi153 ceph-mon[36569]: pgmap v12020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:20 smithi104 ceph-mon[36624]: pgmap v12020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:20 smithi081 ceph-mon[31453]: pgmap v12020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:22 smithi153 ceph-mon[36569]: pgmap v12021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:22 smithi104 ceph-mon[36624]: pgmap v12021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:22 smithi081 ceph-mon[31453]: pgmap v12021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:24.756+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:26:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:24.756+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:26:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:24.756+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:26:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:24 smithi153 ceph-mon[36569]: pgmap v12022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:24 smithi104 ceph-mon[36624]: pgmap v12022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:24 smithi081 ceph-mon[31453]: pgmap v12022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:27 smithi153 ceph-mon[36569]: pgmap v12023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:27 smithi104 ceph-mon[36624]: pgmap v12023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:27 smithi081 ceph-mon[31453]: pgmap v12023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:29 smithi153 ceph-mon[36569]: pgmap v12024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:29 smithi104 ceph-mon[36624]: pgmap v12024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:29 smithi081 ceph-mon[31453]: pgmap v12024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:31 smithi153 ceph-mon[36569]: pgmap v12025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:31 smithi104 ceph-mon[36624]: pgmap v12025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:31 smithi081 ceph-mon[31453]: pgmap v12025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:33 smithi153 ceph-mon[36569]: pgmap v12026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:33 smithi104 ceph-mon[36624]: pgmap v12026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:33 smithi081 ceph-mon[31453]: pgmap v12026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:35 smithi153 ceph-mon[36569]: pgmap v12027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:35 smithi104 ceph-mon[36624]: pgmap v12027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:35 smithi081 ceph-mon[31453]: pgmap v12027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:37 smithi153 ceph-mon[36569]: pgmap v12028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:37 smithi104 ceph-mon[36624]: pgmap v12028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:37 smithi081 ceph-mon[31453]: pgmap v12028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:39 smithi153 ceph-mon[36569]: pgmap v12029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:39 smithi104 ceph-mon[36624]: pgmap v12029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:39 smithi081 ceph-mon[31453]: pgmap v12029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:39.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:39.769+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:26:39.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:39.769+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:26:39.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:39.769+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:26:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:26:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: pgmap v12030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: pgmap v12030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:41.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:41.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:26:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: pgmap v12030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:26:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:26:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:42 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:26:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:42 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:26:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:42 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:26:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:42 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:26:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:42 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:26:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:42 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:26:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:42 smithi104 ceph-mon[36624]: pgmap v12031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:42 smithi104 ceph-mon[36624]: pgmap v12032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:42 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:26:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:42 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:26:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:42 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:26:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:42 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:26:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:42 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:26:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:42 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:26:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:42 smithi081 ceph-mon[31453]: pgmap v12031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:42 smithi081 ceph-mon[31453]: pgmap v12032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:42 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:26:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:42 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:26:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:42 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:26:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:42 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:26:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:42 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:26:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:42 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:26:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:42 smithi153 ceph-mon[36569]: pgmap v12031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:42 smithi153 ceph-mon[36569]: pgmap v12032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:44 smithi104 ceph-mon[36624]: pgmap v12033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:44 smithi081 ceph-mon[31453]: pgmap v12033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:44 smithi153 ceph-mon[36569]: pgmap v12033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:46 smithi104 ceph-mon[36624]: pgmap v12034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:46 smithi081 ceph-mon[31453]: pgmap v12034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:46 smithi153 ceph-mon[36569]: pgmap v12034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:48 smithi081 ceph-mon[31453]: pgmap v12035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:48 smithi153 ceph-mon[36569]: pgmap v12035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:48 smithi104 ceph-mon[36624]: pgmap v12035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:50 smithi081 ceph-mon[31453]: pgmap v12036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:50 smithi153 ceph-mon[36569]: pgmap v12036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:50 smithi104 ceph-mon[36624]: pgmap v12036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:52 smithi081 ceph-mon[31453]: pgmap v12037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:52 smithi153 ceph-mon[36569]: pgmap v12037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:52 smithi104 ceph-mon[36624]: pgmap v12037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:54 smithi081 ceph-mon[31453]: pgmap v12038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:54 smithi153 ceph-mon[36569]: pgmap v12038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:54 smithi104 ceph-mon[36624]: pgmap v12038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:54.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:54.777+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:26:54.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:54.777+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:26:54.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:26:54.777+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:26:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:56 smithi081 ceph-mon[31453]: pgmap v12039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:56 smithi153 ceph-mon[36569]: pgmap v12039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:56 smithi104 ceph-mon[36624]: pgmap v12039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:26:58 smithi081 ceph-mon[31453]: pgmap v12040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:26:58 smithi153 ceph-mon[36569]: pgmap v12040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:26:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:26:58 smithi104 ceph-mon[36624]: pgmap v12040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:00 smithi081 ceph-mon[31453]: pgmap v12041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:00 smithi153 ceph-mon[36569]: pgmap v12041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:00.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:00 smithi104 ceph-mon[36624]: pgmap v12041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:02 smithi081 ceph-mon[31453]: pgmap v12042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:02 smithi153 ceph-mon[36569]: pgmap v12042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:02 smithi104 ceph-mon[36624]: pgmap v12042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:04 smithi081 ceph-mon[31453]: pgmap v12043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:04 smithi153 ceph-mon[36569]: pgmap v12043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:04 smithi104 ceph-mon[36624]: pgmap v12043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:06 smithi081 ceph-mon[31453]: pgmap v12044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:06 smithi153 ceph-mon[36569]: pgmap v12044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:06 smithi104 ceph-mon[36624]: pgmap v12044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:08 smithi081 ceph-mon[31453]: pgmap v12045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:08 smithi153 ceph-mon[36569]: pgmap v12045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:08 smithi104 ceph-mon[36624]: pgmap v12045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:09.780+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:27:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:09.780+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:27:09.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:09.780+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:27:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:10 smithi081 ceph-mon[31453]: pgmap v12046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:10 smithi153 ceph-mon[36569]: pgmap v12046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:10 smithi104 ceph-mon[36624]: pgmap v12046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:12 smithi081 ceph-mon[31453]: pgmap v12047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:12 smithi153 ceph-mon[36569]: pgmap v12047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:12 smithi104 ceph-mon[36624]: pgmap v12047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:14 smithi081 ceph-mon[31453]: pgmap v12048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:14 smithi153 ceph-mon[36569]: pgmap v12048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:14 smithi104 ceph-mon[36624]: pgmap v12048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:16 smithi153 ceph-mon[36569]: pgmap v12049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:16 smithi104 ceph-mon[36624]: pgmap v12049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:16 smithi081 ceph-mon[31453]: pgmap v12049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:18.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:18 smithi081 ceph-mon[31453]: pgmap v12050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:18 smithi153 ceph-mon[36569]: pgmap v12050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:18 smithi104 ceph-mon[36624]: pgmap v12050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:19 smithi153 ceph-mon[36569]: pgmap v12051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:19 smithi104 ceph-mon[36624]: pgmap v12051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:19 smithi081 ceph-mon[31453]: pgmap v12051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:21 smithi081 ceph-mon[31453]: pgmap v12052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:21 smithi153 ceph-mon[36569]: pgmap v12052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:21 smithi104 ceph-mon[36624]: pgmap v12052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:23 smithi081 ceph-mon[31453]: pgmap v12053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:23 smithi153 ceph-mon[36569]: pgmap v12053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:23 smithi104 ceph-mon[36624]: pgmap v12053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:24.796 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:24.795+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:27:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:24.795+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:27:24.797 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:24.795+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:27:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:25 smithi081 ceph-mon[31453]: pgmap v12054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:25 smithi153 ceph-mon[36569]: pgmap v12054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:25 smithi104 ceph-mon[36624]: pgmap v12054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:27 smithi081 ceph-mon[31453]: pgmap v12055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:27 smithi153 ceph-mon[36569]: pgmap v12055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:27 smithi104 ceph-mon[36624]: pgmap v12055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:29 smithi081 ceph-mon[31453]: pgmap v12056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:29 smithi153 ceph-mon[36569]: pgmap v12056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:29 smithi104 ceph-mon[36624]: pgmap v12056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:31.381+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:27:31.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:31.381+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:27:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:31.381+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 173 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:27:31.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:31.381+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 173 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:27:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:31 smithi081 ceph-mon[31453]: pgmap v12057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:31 smithi153 ceph-mon[36569]: pgmap v12057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:31 smithi104 ceph-mon[36624]: pgmap v12057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:33 smithi081 ceph-mon[31453]: pgmap v12058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:33 smithi153 ceph-mon[36569]: pgmap v12058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:33 smithi104 ceph-mon[36624]: pgmap v12058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:35 smithi153 ceph-mon[36569]: pgmap v12059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:35 smithi104 ceph-mon[36624]: pgmap v12059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:35 smithi081 ceph-mon[31453]: pgmap v12059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:37 smithi153 ceph-mon[36569]: pgmap v12060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:37 smithi104 ceph-mon[36624]: pgmap v12060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:37 smithi081 ceph-mon[31453]: pgmap v12060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:39.798+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:27:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:39.798+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:27:39.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:39.798+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:27:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:39 smithi153 ceph-mon[36569]: pgmap v12061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:39 smithi104 ceph-mon[36624]: pgmap v12061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:39 smithi081 ceph-mon[31453]: pgmap v12061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:41 smithi153 ceph-mon[36569]: pgmap v12062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:41 smithi104 ceph-mon[36624]: pgmap v12062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:41 smithi081 ceph-mon[31453]: pgmap v12062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:27:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:27:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:27:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:27:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:27:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:27:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:43 smithi153 ceph-mon[36569]: pgmap v12063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:43 smithi153 ceph-mon[36569]: pgmap v12064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:43 smithi104 ceph-mon[36624]: pgmap v12063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:43 smithi104 ceph-mon[36624]: pgmap v12064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:44.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:43 smithi081 ceph-mon[31453]: pgmap v12063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:44.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:43 smithi081 ceph-mon[31453]: pgmap v12064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:45 smithi153 ceph-mon[36569]: pgmap v12065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:45 smithi104 ceph-mon[36624]: pgmap v12065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:45 smithi081 ceph-mon[31453]: pgmap v12065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:48.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:47 smithi081 ceph-mon[31453]: pgmap v12066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:47 smithi153 ceph-mon[36569]: pgmap v12066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:47 smithi104 ceph-mon[36624]: pgmap v12066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:49 smithi153 ceph-mon[36569]: pgmap v12067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:49 smithi104 ceph-mon[36624]: pgmap v12067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:49 smithi081 ceph-mon[31453]: pgmap v12067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:51 smithi153 ceph-mon[36569]: pgmap v12068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:51 smithi104 ceph-mon[36624]: pgmap v12068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:51 smithi081 ceph-mon[31453]: pgmap v12068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:53 smithi153 ceph-mon[36569]: pgmap v12069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:53 smithi104 ceph-mon[36624]: pgmap v12069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:53 smithi081 ceph-mon[31453]: pgmap v12069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:54.813+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:27:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:54.813+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:27:54.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:27:54.813+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:27:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:55 smithi153 ceph-mon[36569]: pgmap v12070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:55 smithi104 ceph-mon[36624]: pgmap v12070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:55 smithi081 ceph-mon[31453]: pgmap v12070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:57 smithi153 ceph-mon[36569]: pgmap v12071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:57 smithi104 ceph-mon[36624]: pgmap v12071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:27:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:57 smithi081 ceph-mon[31453]: pgmap v12071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:27:59 smithi153 ceph-mon[36569]: pgmap v12072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:27:59 smithi104 ceph-mon[36624]: pgmap v12072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:27:59 smithi081 ceph-mon[31453]: pgmap v12072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:01 smithi153 ceph-mon[36569]: pgmap v12073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:01 smithi104 ceph-mon[36624]: pgmap v12073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:01 smithi081 ceph-mon[31453]: pgmap v12073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:03 smithi153 ceph-mon[36569]: pgmap v12074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:03 smithi104 ceph-mon[36624]: pgmap v12074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:03 smithi081 ceph-mon[31453]: pgmap v12074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:06.280 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:05 smithi081 ceph-mon[31453]: pgmap v12075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:05 smithi153 ceph-mon[36569]: pgmap v12075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:05 smithi104 ceph-mon[36624]: pgmap v12075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:08 smithi153 ceph-mon[36569]: pgmap v12076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:08 smithi104 ceph-mon[36624]: pgmap v12076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:08 smithi081 ceph-mon[31453]: pgmap v12076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:09.829+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:28:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:09.829+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:28:09.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:09.829+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:28:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:10 smithi153 ceph-mon[36569]: pgmap v12077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:10 smithi104 ceph-mon[36624]: pgmap v12077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:10 smithi081 ceph-mon[31453]: pgmap v12077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:12 smithi153 ceph-mon[36569]: pgmap v12078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:12 smithi104 ceph-mon[36624]: pgmap v12078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:12 smithi081 ceph-mon[31453]: pgmap v12078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:14 smithi153 ceph-mon[36569]: pgmap v12079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:14 smithi104 ceph-mon[36624]: pgmap v12079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:14 smithi081 ceph-mon[31453]: pgmap v12079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:16 smithi153 ceph-mon[36569]: pgmap v12080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:16 smithi104 ceph-mon[36624]: pgmap v12080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:16.388+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:28:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:16.388+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:28:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:16.389+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 174 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:28:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:16.389+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 174 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:28:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:16 smithi081 ceph-mon[31453]: pgmap v12080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:18 smithi153 ceph-mon[36569]: pgmap v12081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:18 smithi104 ceph-mon[36624]: pgmap v12081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:18 smithi081 ceph-mon[31453]: pgmap v12081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:20 smithi153 ceph-mon[36569]: pgmap v12082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:20 smithi104 ceph-mon[36624]: pgmap v12082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:20 smithi081 ceph-mon[31453]: pgmap v12082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:22 smithi153 ceph-mon[36569]: pgmap v12083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:22 smithi104 ceph-mon[36624]: pgmap v12083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:22 smithi081 ceph-mon[31453]: pgmap v12083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:24 smithi153 ceph-mon[36569]: pgmap v12084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:24 smithi104 ceph-mon[36624]: pgmap v12084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:24 smithi081 ceph-mon[31453]: pgmap v12084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:24.845+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:28:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:24.845+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:28:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:24.845+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:28:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:26 smithi153 ceph-mon[36569]: pgmap v12085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:26 smithi104 ceph-mon[36624]: pgmap v12085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:26 smithi081 ceph-mon[31453]: pgmap v12085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:28 smithi153 ceph-mon[36569]: pgmap v12086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:28 smithi104 ceph-mon[36624]: pgmap v12086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:28 smithi081 ceph-mon[31453]: pgmap v12086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:30 smithi104 ceph-mon[36624]: pgmap v12087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:30 smithi081 ceph-mon[31453]: pgmap v12087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:30 smithi153 ceph-mon[36569]: pgmap v12087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:32 smithi104 ceph-mon[36624]: pgmap v12088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:32 smithi081 ceph-mon[31453]: pgmap v12088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:32 smithi153 ceph-mon[36569]: pgmap v12088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:34 smithi104 ceph-mon[36624]: pgmap v12089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:34 smithi081 ceph-mon[31453]: pgmap v12089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:34 smithi153 ceph-mon[36569]: pgmap v12089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:36 smithi104 ceph-mon[36624]: pgmap v12090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:36 smithi081 ceph-mon[31453]: pgmap v12090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:36 smithi153 ceph-mon[36569]: pgmap v12090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:38 smithi104 ceph-mon[36624]: pgmap v12091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:38 smithi081 ceph-mon[31453]: pgmap v12091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:38 smithi153 ceph-mon[36569]: pgmap v12091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:39.860+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:28:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:39.860+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:28:39.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:39.860+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:28:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:40 smithi081 ceph-mon[31453]: pgmap v12092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:40 smithi153 ceph-mon[36569]: pgmap v12092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:40 smithi104 ceph-mon[36624]: pgmap v12092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:42.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:42 smithi153 ceph-mon[36569]: pgmap v12093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:42 smithi081 ceph-mon[31453]: pgmap v12093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:42 smithi104 ceph-mon[36624]: pgmap v12093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:28:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:44 smithi081 ceph-mon[31453]: pgmap v12094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:28:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:44 smithi153 ceph-mon[36569]: pgmap v12094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:28:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:44 smithi104 ceph-mon[36624]: pgmap v12094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:28:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:28:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:28:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:45 smithi081 ceph-mon[31453]: pgmap v12095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:45 smithi081 ceph-mon[31453]: pgmap v12096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:45 smithi153 ceph-mon[36569]: pgmap v12095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:45 smithi153 ceph-mon[36569]: pgmap v12096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:45 smithi104 ceph-mon[36624]: pgmap v12095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:45 smithi104 ceph-mon[36624]: pgmap v12096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:47 smithi081 ceph-mon[31453]: pgmap v12097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:47 smithi153 ceph-mon[36569]: pgmap v12097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:47 smithi104 ceph-mon[36624]: pgmap v12097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:49.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:49 smithi104 ceph-mon[36624]: pgmap v12098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:49 smithi081 ceph-mon[31453]: pgmap v12098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:49 smithi153 ceph-mon[36569]: pgmap v12098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:51 smithi081 ceph-mon[31453]: pgmap v12099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:51 smithi153 ceph-mon[36569]: pgmap v12099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:51 smithi104 ceph-mon[36624]: pgmap v12099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:53 smithi081 ceph-mon[31453]: pgmap v12100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:53 smithi153 ceph-mon[36569]: pgmap v12100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:53.651 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:53 smithi104 ceph-mon[36624]: pgmap v12100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:54.873+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:28:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:54.873+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:28:54.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:28:54.873+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:28:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:55 smithi081 ceph-mon[31453]: pgmap v12101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:55 smithi153 ceph-mon[36569]: pgmap v12101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:55 smithi104 ceph-mon[36624]: pgmap v12101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:57 smithi081 ceph-mon[31453]: pgmap v12102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:57 smithi153 ceph-mon[36569]: pgmap v12102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:57 smithi104 ceph-mon[36624]: pgmap v12102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:28:59 smithi081 ceph-mon[31453]: pgmap v12103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:28:59 smithi153 ceph-mon[36569]: pgmap v12103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:28:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:28:59 smithi104 ceph-mon[36624]: pgmap v12103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:01 smithi081 ceph-mon[31453]: pgmap v12104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:01 smithi153 ceph-mon[36569]: pgmap v12104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:01 smithi104 ceph-mon[36624]: pgmap v12104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:03 smithi153 ceph-mon[36569]: pgmap v12105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:03 smithi104 ceph-mon[36624]: pgmap v12105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:03 smithi081 ceph-mon[31453]: pgmap v12105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:05 smithi153 ceph-mon[36569]: pgmap v12106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:05 smithi104 ceph-mon[36624]: pgmap v12106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:05 smithi081 ceph-mon[31453]: pgmap v12106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:07 smithi153 ceph-mon[36569]: pgmap v12107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:07 smithi104 ceph-mon[36624]: pgmap v12107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:07 smithi081 ceph-mon[31453]: pgmap v12107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:09 smithi153 ceph-mon[36569]: pgmap v12108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:09 smithi104 ceph-mon[36624]: pgmap v12108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:09 smithi081 ceph-mon[31453]: pgmap v12108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:09.889+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:29:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:09.889+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:29:09.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:09.889+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:29:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:11 smithi153 ceph-mon[36569]: pgmap v12109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:11 smithi104 ceph-mon[36624]: pgmap v12109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:11 smithi081 ceph-mon[31453]: pgmap v12109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:13 smithi153 ceph-mon[36569]: pgmap v12110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:13 smithi104 ceph-mon[36624]: pgmap v12110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:13 smithi081 ceph-mon[31453]: pgmap v12110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:14 smithi153 ceph-mon[36569]: pgmap v12111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:14 smithi104 ceph-mon[36624]: pgmap v12111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:14.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:14 smithi081 ceph-mon[31453]: pgmap v12111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:16 smithi153 ceph-mon[36569]: pgmap v12112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:16 smithi104 ceph-mon[36624]: pgmap v12112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:16 smithi081 ceph-mon[31453]: pgmap v12112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:18 smithi153 ceph-mon[36569]: pgmap v12113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:18 smithi104 ceph-mon[36624]: pgmap v12113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:18 smithi081 ceph-mon[31453]: pgmap v12113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:20 smithi153 ceph-mon[36569]: pgmap v12114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:20 smithi104 ceph-mon[36624]: pgmap v12114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:20 smithi081 ceph-mon[31453]: pgmap v12114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:22 smithi153 ceph-mon[36569]: pgmap v12115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:22 smithi104 ceph-mon[36624]: pgmap v12115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:22 smithi081 ceph-mon[31453]: pgmap v12115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:24.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:24.892+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:29:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:24.892+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:29:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:24.892+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:29:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:24 smithi153 ceph-mon[36569]: pgmap v12116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:24 smithi104 ceph-mon[36624]: pgmap v12116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:24 smithi081 ceph-mon[31453]: pgmap v12116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:26 smithi081 ceph-mon[31453]: pgmap v12117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:26 smithi153 ceph-mon[36569]: pgmap v12117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:26 smithi104 ceph-mon[36624]: pgmap v12117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:28 smithi153 ceph-mon[36569]: pgmap v12118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:28 smithi104 ceph-mon[36624]: pgmap v12118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:28 smithi081 ceph-mon[31453]: pgmap v12118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:30 smithi153 ceph-mon[36569]: pgmap v12119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:30 smithi104 ceph-mon[36624]: pgmap v12119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:30 smithi081 ceph-mon[31453]: pgmap v12119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:32 smithi153 ceph-mon[36569]: pgmap v12120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:32 smithi104 ceph-mon[36624]: pgmap v12120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:32 smithi081 ceph-mon[31453]: pgmap v12120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:34 smithi153 ceph-mon[36569]: pgmap v12121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:34 smithi104 ceph-mon[36624]: pgmap v12121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:34 smithi081 ceph-mon[31453]: pgmap v12121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:36 smithi153 ceph-mon[36569]: pgmap v12122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:36 smithi104 ceph-mon[36624]: pgmap v12122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:36 smithi081 ceph-mon[31453]: pgmap v12122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:39 smithi153 ceph-mon[36569]: pgmap v12123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:39 smithi104 ceph-mon[36624]: pgmap v12123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:39 smithi081 ceph-mon[31453]: pgmap v12123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:39.904+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:29:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:39.904+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:29:39.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:39.904+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:29:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:41 smithi153 ceph-mon[36569]: pgmap v12124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:41 smithi104 ceph-mon[36624]: pgmap v12124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:41 smithi081 ceph-mon[31453]: pgmap v12124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:43 smithi153 ceph-mon[36569]: pgmap v12125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:43 smithi104 ceph-mon[36624]: pgmap v12125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:43 smithi081 ceph-mon[31453]: pgmap v12125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:45 smithi153 ceph-mon[36569]: pgmap v12126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:45 smithi104 ceph-mon[36624]: pgmap v12126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:45 smithi081 ceph-mon[31453]: pgmap v12126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:29:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:29:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:29:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:29:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:29:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:29:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:47 smithi153 ceph-mon[36569]: pgmap v12127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:47 smithi153 ceph-mon[36569]: pgmap v12128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:47 smithi104 ceph-mon[36624]: pgmap v12127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:47 smithi104 ceph-mon[36624]: pgmap v12128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:47 smithi081 ceph-mon[31453]: pgmap v12127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:47 smithi081 ceph-mon[31453]: pgmap v12128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:49 smithi153 ceph-mon[36569]: pgmap v12129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:49 smithi104 ceph-mon[36624]: pgmap v12129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:49 smithi081 ceph-mon[31453]: pgmap v12129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:51 smithi153 ceph-mon[36569]: pgmap v12130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:51 smithi104 ceph-mon[36624]: pgmap v12130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:51 smithi081 ceph-mon[31453]: pgmap v12130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:53 smithi153 ceph-mon[36569]: pgmap v12131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:53 smithi104 ceph-mon[36624]: pgmap v12131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:53 smithi081 ceph-mon[31453]: pgmap v12131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:54.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:54.920+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:29:54.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:54.920+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:29:54.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:29:54.920+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:29:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:55 smithi153 ceph-mon[36569]: pgmap v12132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:55 smithi104 ceph-mon[36624]: pgmap v12132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:55.336 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:55 smithi081 ceph-mon[31453]: pgmap v12132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:57 smithi104 ceph-mon[36624]: pgmap v12133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:57 smithi081 ceph-mon[31453]: pgmap v12133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:57 smithi153 ceph-mon[36569]: pgmap v12133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:29:59 smithi104 ceph-mon[36624]: pgmap v12134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:29:59 smithi081 ceph-mon[31453]: pgmap v12134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:29:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:29:59 smithi153 ceph-mon[36569]: pgmap v12134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:30:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:30:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:30:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:01 smithi104 ceph-mon[36624]: pgmap v12135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:01 smithi081 ceph-mon[31453]: pgmap v12135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:01 smithi153 ceph-mon[36569]: pgmap v12135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:03 smithi104 ceph-mon[36624]: pgmap v12136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:03 smithi081 ceph-mon[31453]: pgmap v12136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:03 smithi153 ceph-mon[36569]: pgmap v12136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:05 smithi081 ceph-mon[31453]: pgmap v12137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:05 smithi153 ceph-mon[36569]: pgmap v12137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:05 smithi104 ceph-mon[36624]: pgmap v12137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:07 smithi081 ceph-mon[31453]: pgmap v12138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:07 smithi153 ceph-mon[36569]: pgmap v12138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:07 smithi104 ceph-mon[36624]: pgmap v12138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:09 smithi081 ceph-mon[31453]: pgmap v12139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:09 smithi153 ceph-mon[36569]: pgmap v12139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:09 smithi104 ceph-mon[36624]: pgmap v12139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:09.936+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:30:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:09.936+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:30:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:09.936+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:30:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:11 smithi081 ceph-mon[31453]: pgmap v12140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:11 smithi153 ceph-mon[36569]: pgmap v12140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:11 smithi104 ceph-mon[36624]: pgmap v12140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:13.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:13 smithi153 ceph-mon[36569]: pgmap v12141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:13 smithi081 ceph-mon[31453]: pgmap v12141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:13 smithi104 ceph-mon[36624]: pgmap v12141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:15 smithi081 ceph-mon[31453]: pgmap v12142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:15 smithi153 ceph-mon[36569]: pgmap v12142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:15 smithi104 ceph-mon[36624]: pgmap v12142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:17 smithi081 ceph-mon[31453]: pgmap v12143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:17 smithi153 ceph-mon[36569]: pgmap v12143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:17 smithi104 ceph-mon[36624]: pgmap v12143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:19.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:19 smithi104 ceph-mon[36624]: pgmap v12144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:19 smithi081 ceph-mon[31453]: pgmap v12144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:19 smithi153 ceph-mon[36569]: pgmap v12144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:21 smithi081 ceph-mon[31453]: pgmap v12145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:21 smithi153 ceph-mon[36569]: pgmap v12145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:21 smithi104 ceph-mon[36624]: pgmap v12145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:23 smithi081 ceph-mon[31453]: pgmap v12146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:23 smithi153 ceph-mon[36569]: pgmap v12146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:23 smithi104 ceph-mon[36624]: pgmap v12146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:24.951+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:30:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:24.951+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:30:24.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:24.951+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:30:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:25 smithi081 ceph-mon[31453]: pgmap v12147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:25 smithi153 ceph-mon[36569]: pgmap v12147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:25 smithi104 ceph-mon[36624]: pgmap v12147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:27 smithi081 ceph-mon[31453]: pgmap v12148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:27 smithi153 ceph-mon[36569]: pgmap v12148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:27 smithi104 ceph-mon[36624]: pgmap v12148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:29 smithi081 ceph-mon[31453]: pgmap v12149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:29 smithi153 ceph-mon[36569]: pgmap v12149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:29 smithi104 ceph-mon[36624]: pgmap v12149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:31 smithi081 ceph-mon[31453]: pgmap v12150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:31 smithi153 ceph-mon[36569]: pgmap v12150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:31 smithi104 ceph-mon[36624]: pgmap v12150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:33 smithi153 ceph-mon[36569]: pgmap v12151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:33 smithi104 ceph-mon[36624]: pgmap v12151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:33 smithi081 ceph-mon[31453]: pgmap v12151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:35 smithi153 ceph-mon[36569]: pgmap v12152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:35.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:35 smithi104 ceph-mon[36624]: pgmap v12152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:35 smithi081 ceph-mon[31453]: pgmap v12152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:37 smithi153 ceph-mon[36569]: pgmap v12153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:37 smithi104 ceph-mon[36624]: pgmap v12153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:37 smithi081 ceph-mon[31453]: pgmap v12153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:39 smithi153 ceph-mon[36569]: pgmap v12154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:39 smithi104 ceph-mon[36624]: pgmap v12154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:39 smithi081 ceph-mon[31453]: pgmap v12154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:39.955+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:30:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:39.955+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:30:39.956 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:39.955+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:30:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:41 smithi153 ceph-mon[36569]: pgmap v12155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:41 smithi104 ceph-mon[36624]: pgmap v12155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:41 smithi081 ceph-mon[31453]: pgmap v12155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:43 smithi081 ceph-mon[31453]: pgmap v12156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:43 smithi153 ceph-mon[36569]: pgmap v12156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:43 smithi104 ceph-mon[36624]: pgmap v12156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:45 smithi153 ceph-mon[36569]: pgmap v12157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:45 smithi104 ceph-mon[36624]: pgmap v12157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:45 smithi081 ceph-mon[31453]: pgmap v12157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:30:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:47 smithi153 ceph-mon[36569]: pgmap v12158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:47 smithi104 ceph-mon[36624]: pgmap v12158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:47 smithi081 ceph-mon[31453]: pgmap v12158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:48 smithi153 ceph-mon[36569]: pgmap v12159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:48 smithi104 ceph-mon[36624]: pgmap v12159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:48 smithi081 ceph-mon[31453]: pgmap v12159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:50 smithi153 ceph-mon[36569]: pgmap v12160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:50 smithi104 ceph-mon[36624]: pgmap v12160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:50 smithi081 ceph-mon[31453]: pgmap v12160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:53 smithi153 ceph-mon[36569]: pgmap v12161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:53 smithi104 ceph-mon[36624]: pgmap v12161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:53 smithi081 ceph-mon[31453]: pgmap v12161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:30:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:30:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:30:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:54 smithi153 ceph-mon[36569]: pgmap v12162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:54 smithi153 ceph-mon[36569]: pgmap v12163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:54 smithi104 ceph-mon[36624]: pgmap v12162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:54 smithi104 ceph-mon[36624]: pgmap v12163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:54 smithi081 ceph-mon[31453]: pgmap v12162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:54 smithi081 ceph-mon[31453]: pgmap v12163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:54.965+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:30:54.967 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:54.965+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:30:54.967 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:30:54.965+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:30:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:56 smithi153 ceph-mon[36569]: pgmap v12164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:56 smithi104 ceph-mon[36624]: pgmap v12164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:56 smithi081 ceph-mon[31453]: pgmap v12164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:58.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:30:58 smithi081 ceph-mon[31453]: pgmap v12165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:30:58 smithi153 ceph-mon[36569]: pgmap v12165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:30:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:30:58 smithi104 ceph-mon[36624]: pgmap v12165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:00 smithi153 ceph-mon[36569]: pgmap v12166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:00 smithi104 ceph-mon[36624]: pgmap v12166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:00 smithi081 ceph-mon[31453]: pgmap v12166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:02 smithi153 ceph-mon[36569]: pgmap v12167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:02 smithi104 ceph-mon[36624]: pgmap v12167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:02 smithi081 ceph-mon[31453]: pgmap v12167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:04 smithi153 ceph-mon[36569]: pgmap v12168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:04 smithi104 ceph-mon[36624]: pgmap v12168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:04 smithi081 ceph-mon[31453]: pgmap v12168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:06 smithi153 ceph-mon[36569]: pgmap v12169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:06 smithi104 ceph-mon[36624]: pgmap v12169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:06 smithi081 ceph-mon[31453]: pgmap v12169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:08 smithi153 ceph-mon[36569]: pgmap v12170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:08 smithi104 ceph-mon[36624]: pgmap v12170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:08 smithi081 ceph-mon[31453]: pgmap v12170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:09.977+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:31:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:09.977+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:31:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:09.977+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:31:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:10 smithi153 ceph-mon[36569]: pgmap v12171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:10 smithi104 ceph-mon[36624]: pgmap v12171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:10 smithi081 ceph-mon[31453]: pgmap v12171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:12.152+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 msgr2=0x7f16e80b39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000656) for more than 900000000 us, fault. 2024-04-13T03:31:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:12.152+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f16d40065b0 tx=0x7f16f40d2000 comp rx=0 tx=0).stop 2024-04-13T03:31:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:12.152+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 msgr2=0x7f16e80b39a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T03:31:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:12.152+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:31:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:12.152+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:31:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:12.152+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:31:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:12.154+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c760 tx=0x7f16f40bf000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:31:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:12 smithi153 ceph-mon[36569]: pgmap v12172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:12 smithi104 ceph-mon[36624]: pgmap v12172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:12 smithi081 ceph-mon[31453]: pgmap v12172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:13 smithi153 ceph-mon[36569]: pgmap v12173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:13 smithi104 ceph-mon[36624]: pgmap v12173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:13 smithi081 ceph-mon[31453]: pgmap v12173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:15 smithi153 ceph-mon[36569]: pgmap v12174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:15 smithi104 ceph-mon[36624]: pgmap v12174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:15 smithi081 ceph-mon[31453]: pgmap v12174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:16.417+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:31:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:16.417+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:31:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:16.418+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 175 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:31:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:16.418+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 175 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:31:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:17 smithi153 ceph-mon[36569]: pgmap v12175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:17 smithi104 ceph-mon[36624]: pgmap v12175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:17 smithi081 ceph-mon[31453]: pgmap v12175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:19 smithi153 ceph-mon[36569]: pgmap v12176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:19 smithi104 ceph-mon[36624]: pgmap v12176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:19 smithi081 ceph-mon[31453]: pgmap v12176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:21 smithi153 ceph-mon[36569]: pgmap v12177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:21 smithi104 ceph-mon[36624]: pgmap v12177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:21 smithi081 ceph-mon[31453]: pgmap v12177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:23 smithi153 ceph-mon[36569]: pgmap v12178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:23 smithi104 ceph-mon[36624]: pgmap v12178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:23.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:23 smithi081 ceph-mon[31453]: pgmap v12178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:24.992+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:31:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:24.992+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:31:24.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:24.992+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:31:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:25 smithi153 ceph-mon[36569]: pgmap v12179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:25 smithi104 ceph-mon[36624]: pgmap v12179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:25 smithi081 ceph-mon[31453]: pgmap v12179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:27 smithi153 ceph-mon[36569]: pgmap v12180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:27 smithi104 ceph-mon[36624]: pgmap v12180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:27 smithi081 ceph-mon[31453]: pgmap v12180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:29 smithi153 ceph-mon[36569]: pgmap v12181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:29 smithi104 ceph-mon[36624]: pgmap v12181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:29 smithi081 ceph-mon[31453]: pgmap v12181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:31 smithi153 ceph-mon[36569]: pgmap v12182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:31 smithi104 ceph-mon[36624]: pgmap v12182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:31 smithi081 ceph-mon[31453]: pgmap v12182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:33 smithi153 ceph-mon[36569]: pgmap v12183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:33 smithi104 ceph-mon[36624]: pgmap v12183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:33 smithi081 ceph-mon[31453]: pgmap v12183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:35 smithi153 ceph-mon[36569]: pgmap v12184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:35 smithi104 ceph-mon[36624]: pgmap v12184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:35.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:35 smithi081 ceph-mon[31453]: pgmap v12184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:37 smithi104 ceph-mon[36624]: pgmap v12185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:37.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:37 smithi081 ceph-mon[31453]: pgmap v12185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:37 smithi153 ceph-mon[36569]: pgmap v12185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:39 smithi104 ceph-mon[36624]: pgmap v12186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:39 smithi081 ceph-mon[31453]: pgmap v12186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:40.009+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:31:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:40.009+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:31:40.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:40.009+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:31:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:39 smithi153 ceph-mon[36569]: pgmap v12186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:41 smithi104 ceph-mon[36624]: pgmap v12187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:41 smithi081 ceph-mon[31453]: pgmap v12187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:41 smithi153 ceph-mon[36569]: pgmap v12187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:43 smithi104 ceph-mon[36624]: pgmap v12188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:43.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:43 smithi153 ceph-mon[36569]: pgmap v12188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:43 smithi081 ceph-mon[31453]: pgmap v12188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:45 smithi104 ceph-mon[36624]: pgmap v12189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:45.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:45 smithi081 ceph-mon[31453]: pgmap v12189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:46.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:45 smithi153 ceph-mon[36569]: pgmap v12189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:47 smithi081 ceph-mon[31453]: pgmap v12190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:47 smithi153 ceph-mon[36569]: pgmap v12190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:48.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:47 smithi104 ceph-mon[36624]: pgmap v12190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:49.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:49 smithi081 ceph-mon[31453]: pgmap v12191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:49 smithi153 ceph-mon[36569]: pgmap v12191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:50.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:49 smithi104 ceph-mon[36624]: pgmap v12191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:51.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:51 smithi104 ceph-mon[36624]: pgmap v12192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:51 smithi081 ceph-mon[31453]: pgmap v12192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:51 smithi153 ceph-mon[36569]: pgmap v12192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:53 smithi081 ceph-mon[31453]: pgmap v12193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:53 smithi153 ceph-mon[36569]: pgmap v12193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:53 smithi104 ceph-mon[36624]: pgmap v12193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:31:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:54 smithi081 ceph-mon[31453]: pgmap v12194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:54 smithi081 ceph-mon[31453]: pgmap v12195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:55.016+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:31:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:55.016+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:31:55.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:31:55.016+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:31:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:54 smithi153 ceph-mon[36569]: pgmap v12194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:54 smithi153 ceph-mon[36569]: pgmap v12195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:31:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:31:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:54 smithi104 ceph-mon[36624]: pgmap v12194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:54 smithi104 ceph-mon[36624]: pgmap v12195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:31:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:57 smithi153 ceph-mon[36569]: pgmap v12196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:57 smithi104 ceph-mon[36624]: pgmap v12196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:57 smithi081 ceph-mon[31453]: pgmap v12196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:31:59 smithi153 ceph-mon[36569]: pgmap v12197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:31:59 smithi104 ceph-mon[36624]: pgmap v12197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:31:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:31:59 smithi081 ceph-mon[31453]: pgmap v12197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:01 smithi153 ceph-mon[36569]: pgmap v12198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:01 smithi104 ceph-mon[36624]: pgmap v12198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:01 smithi081 ceph-mon[31453]: pgmap v12198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:03 smithi153 ceph-mon[36569]: pgmap v12199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:03 smithi104 ceph-mon[36624]: pgmap v12199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:03.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:03 smithi081 ceph-mon[31453]: pgmap v12199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:05 smithi153 ceph-mon[36569]: pgmap v12200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:05 smithi104 ceph-mon[36624]: pgmap v12200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:05 smithi081 ceph-mon[31453]: pgmap v12200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:07 smithi153 ceph-mon[36569]: pgmap v12201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:07 smithi104 ceph-mon[36624]: pgmap v12201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:07 smithi081 ceph-mon[31453]: pgmap v12201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:09 smithi153 ceph-mon[36569]: pgmap v12202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:09.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:09 smithi104 ceph-mon[36624]: pgmap v12202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:09 smithi081 ceph-mon[31453]: pgmap v12202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:10.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:10.028+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:32:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:10.028+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:32:10.030 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:10.028+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:32:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:11 smithi153 ceph-mon[36569]: pgmap v12203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:11 smithi104 ceph-mon[36624]: pgmap v12203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:11 smithi081 ceph-mon[31453]: pgmap v12203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:13 smithi153 ceph-mon[36569]: pgmap v12204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:13 smithi104 ceph-mon[36624]: pgmap v12204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:13 smithi081 ceph-mon[31453]: pgmap v12204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:15 smithi153 ceph-mon[36569]: pgmap v12205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:15 smithi104 ceph-mon[36624]: pgmap v12205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:15 smithi081 ceph-mon[31453]: pgmap v12205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:16 smithi081 ceph-mon[31453]: pgmap v12206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:16 smithi153 ceph-mon[36569]: pgmap v12206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:16 smithi104 ceph-mon[36624]: pgmap v12206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:19 smithi153 ceph-mon[36569]: pgmap v12207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:19 smithi104 ceph-mon[36624]: pgmap v12207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:19 smithi081 ceph-mon[31453]: pgmap v12207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:21.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:21 smithi081 ceph-mon[31453]: pgmap v12208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:21 smithi153 ceph-mon[36569]: pgmap v12208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:21 smithi104 ceph-mon[36624]: pgmap v12208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:23 smithi153 ceph-mon[36569]: pgmap v12209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:23 smithi104 ceph-mon[36624]: pgmap v12209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:23 smithi081 ceph-mon[31453]: pgmap v12209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:25.044+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:32:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:25.044+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:32:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:25.044+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:32:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:25 smithi153 ceph-mon[36569]: pgmap v12210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:25 smithi104 ceph-mon[36624]: pgmap v12210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:25 smithi081 ceph-mon[31453]: pgmap v12210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:26 smithi153 ceph-mon[36569]: pgmap v12211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:26 smithi104 ceph-mon[36624]: pgmap v12211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:26 smithi081 ceph-mon[31453]: pgmap v12211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:29 smithi153 ceph-mon[36569]: pgmap v12212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:29 smithi104 ceph-mon[36624]: pgmap v12212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:29 smithi081 ceph-mon[31453]: pgmap v12212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:31 smithi153 ceph-mon[36569]: pgmap v12213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:31 smithi104 ceph-mon[36624]: pgmap v12213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:31 smithi081 ceph-mon[31453]: pgmap v12213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:33 smithi153 ceph-mon[36569]: pgmap v12214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:33 smithi104 ceph-mon[36624]: pgmap v12214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:33 smithi081 ceph-mon[31453]: pgmap v12214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:35 smithi153 ceph-mon[36569]: pgmap v12215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:35 smithi104 ceph-mon[36624]: pgmap v12215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:35 smithi081 ceph-mon[31453]: pgmap v12215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:37 smithi153 ceph-mon[36569]: pgmap v12216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:37 smithi104 ceph-mon[36624]: pgmap v12216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:37 smithi081 ceph-mon[31453]: pgmap v12216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:39 smithi153 ceph-mon[36569]: pgmap v12217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:39 smithi104 ceph-mon[36624]: pgmap v12217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:39 smithi081 ceph-mon[31453]: pgmap v12217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:40.060+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:32:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:40.060+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:32:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:40.060+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:32:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:41 smithi153 ceph-mon[36569]: pgmap v12218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:41 smithi104 ceph-mon[36624]: pgmap v12218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:41 smithi081 ceph-mon[31453]: pgmap v12218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:42 smithi153 ceph-mon[36569]: pgmap v12219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:42 smithi104 ceph-mon[36624]: pgmap v12219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:42 smithi081 ceph-mon[31453]: pgmap v12219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:45 smithi153 ceph-mon[36569]: pgmap v12220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:45 smithi104 ceph-mon[36624]: pgmap v12220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:45 smithi081 ceph-mon[31453]: pgmap v12220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:47 smithi153 ceph-mon[36569]: pgmap v12221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:47 smithi104 ceph-mon[36624]: pgmap v12221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:47 smithi081 ceph-mon[31453]: pgmap v12221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:49 smithi153 ceph-mon[36569]: pgmap v12222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:49 smithi104 ceph-mon[36624]: pgmap v12222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:49 smithi081 ceph-mon[31453]: pgmap v12222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:51 smithi153 ceph-mon[36569]: pgmap v12223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:51 smithi104 ceph-mon[36624]: pgmap v12223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:51 smithi081 ceph-mon[31453]: pgmap v12223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:53 smithi153 ceph-mon[36569]: pgmap v12224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:53 smithi104 ceph-mon[36624]: pgmap v12224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:53 smithi081 ceph-mon[31453]: pgmap v12224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:55.075+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:32:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:55.075+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:32:55.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:32:55.075+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:32:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:55 smithi153 ceph-mon[36569]: pgmap v12225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:32:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:55 smithi104 ceph-mon[36624]: pgmap v12225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:32:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:55 smithi081 ceph-mon[31453]: pgmap v12225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:32:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:32:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:32:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:32:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:32:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:32:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:32:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:32:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:32:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:32:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:57 smithi153 ceph-mon[36569]: pgmap v12226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:57 smithi153 ceph-mon[36569]: pgmap v12227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:57.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:57 smithi104 ceph-mon[36624]: pgmap v12226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:57.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:57 smithi104 ceph-mon[36624]: pgmap v12227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:57 smithi081 ceph-mon[31453]: pgmap v12226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:57 smithi081 ceph-mon[31453]: pgmap v12227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:32:58 smithi153 ceph-mon[36569]: pgmap v12228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:32:58 smithi104 ceph-mon[36624]: pgmap v12228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:32:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:32:58 smithi081 ceph-mon[31453]: pgmap v12228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:00.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:00 smithi081 ceph-mon[31453]: pgmap v12229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:00 smithi153 ceph-mon[36569]: pgmap v12229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:00 smithi104 ceph-mon[36624]: pgmap v12229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:02 smithi081 ceph-mon[31453]: pgmap v12230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:02 smithi153 ceph-mon[36569]: pgmap v12230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:02 smithi104 ceph-mon[36624]: pgmap v12230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:04 smithi081 ceph-mon[31453]: pgmap v12231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:04 smithi153 ceph-mon[36569]: pgmap v12231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:04 smithi104 ceph-mon[36624]: pgmap v12231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:06 smithi153 ceph-mon[36569]: pgmap v12232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:06 smithi104 ceph-mon[36624]: pgmap v12232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:06 smithi081 ceph-mon[31453]: pgmap v12232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:08 smithi153 ceph-mon[36569]: pgmap v12233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:08 smithi104 ceph-mon[36624]: pgmap v12233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:09.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:08 smithi081 ceph-mon[31453]: pgmap v12233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:10.088 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:10.087+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:33:10.088 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:10.087+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:33:10.088 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:10.087+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:33:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:10 smithi153 ceph-mon[36569]: pgmap v12234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:10 smithi104 ceph-mon[36624]: pgmap v12234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:11.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:10 smithi081 ceph-mon[31453]: pgmap v12234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:12 smithi153 ceph-mon[36569]: pgmap v12235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:12 smithi104 ceph-mon[36624]: pgmap v12235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:12 smithi081 ceph-mon[31453]: pgmap v12235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:14 smithi153 ceph-mon[36569]: pgmap v12236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:14 smithi104 ceph-mon[36624]: pgmap v12236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:14 smithi081 ceph-mon[31453]: pgmap v12236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:16 smithi153 ceph-mon[36569]: pgmap v12237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:16 smithi104 ceph-mon[36624]: pgmap v12237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:16 smithi081 ceph-mon[31453]: pgmap v12237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:18 smithi153 ceph-mon[36569]: pgmap v12238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:19.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:18 smithi104 ceph-mon[36624]: pgmap v12238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:18 smithi081 ceph-mon[31453]: pgmap v12238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:20 smithi153 ceph-mon[36569]: pgmap v12239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:21.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:20 smithi104 ceph-mon[36624]: pgmap v12239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:20 smithi081 ceph-mon[31453]: pgmap v12239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:22.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:22 smithi081 ceph-mon[31453]: pgmap v12240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:22 smithi153 ceph-mon[36569]: pgmap v12240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:22 smithi104 ceph-mon[36624]: pgmap v12240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:24 smithi153 ceph-mon[36569]: pgmap v12241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:24 smithi104 ceph-mon[36624]: pgmap v12241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:25.093+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:33:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:25.093+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:33:25.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:25.093+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:33:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:24 smithi081 ceph-mon[31453]: pgmap v12241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:26 smithi153 ceph-mon[36569]: pgmap v12242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:26 smithi104 ceph-mon[36624]: pgmap v12242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:26 smithi081 ceph-mon[31453]: pgmap v12242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:28 smithi153 ceph-mon[36569]: pgmap v12243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:28 smithi104 ceph-mon[36624]: pgmap v12243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:29.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:28 smithi081 ceph-mon[31453]: pgmap v12243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:30 smithi153 ceph-mon[36569]: pgmap v12244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:30 smithi104 ceph-mon[36624]: pgmap v12244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:30 smithi081 ceph-mon[31453]: pgmap v12244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:32 smithi153 ceph-mon[36569]: pgmap v12245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:33.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:32 smithi104 ceph-mon[36624]: pgmap v12245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:32 smithi081 ceph-mon[31453]: pgmap v12245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:34 smithi153 ceph-mon[36569]: pgmap v12246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:34 smithi104 ceph-mon[36624]: pgmap v12246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:35.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:34 smithi081 ceph-mon[31453]: pgmap v12246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:36 smithi153 ceph-mon[36569]: pgmap v12247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:37.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:36 smithi104 ceph-mon[36624]: pgmap v12247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:37.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:36 smithi081 ceph-mon[31453]: pgmap v12247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:38 smithi153 ceph-mon[36569]: pgmap v12248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:38 smithi104 ceph-mon[36624]: pgmap v12248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:39.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:38 smithi081 ceph-mon[31453]: pgmap v12248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:40.109+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:33:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:40.109+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:33:40.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:40.109+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:33:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:40 smithi153 ceph-mon[36569]: pgmap v12249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:40 smithi104 ceph-mon[36624]: pgmap v12249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:40 smithi081 ceph-mon[31453]: pgmap v12249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:43.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:42 smithi153 ceph-mon[36569]: pgmap v12250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:42 smithi104 ceph-mon[36624]: pgmap v12250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:43.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:42 smithi081 ceph-mon[31453]: pgmap v12250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:44 smithi153 ceph-mon[36569]: pgmap v12251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:44 smithi104 ceph-mon[36624]: pgmap v12251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:44 smithi081 ceph-mon[31453]: pgmap v12251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:46 smithi153 ceph-mon[36569]: pgmap v12252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:46 smithi104 ceph-mon[36624]: pgmap v12252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:46 smithi081 ceph-mon[31453]: pgmap v12252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:48 smithi153 ceph-mon[36569]: pgmap v12253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:48 smithi104 ceph-mon[36624]: pgmap v12253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:49.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:48 smithi081 ceph-mon[31453]: pgmap v12253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:51.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:50 smithi153 ceph-mon[36569]: pgmap v12254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:50 smithi104 ceph-mon[36624]: pgmap v12254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:50 smithi081 ceph-mon[31453]: pgmap v12254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:52 smithi153 ceph-mon[36569]: pgmap v12255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:52 smithi104 ceph-mon[36624]: pgmap v12255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:52 smithi081 ceph-mon[31453]: pgmap v12255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:54 smithi153 ceph-mon[36569]: pgmap v12256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:54 smithi104 ceph-mon[36624]: pgmap v12256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:55.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:55.124+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:33:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:55.124+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:33:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:33:55.124+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:33:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:54 smithi081 ceph-mon[31453]: pgmap v12256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:56 smithi153 ceph-mon[36569]: pgmap v12257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:33:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:56 smithi104 ceph-mon[36624]: pgmap v12257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:57.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:33:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:56 smithi081 ceph-mon[31453]: pgmap v12257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:57.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:33:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:33:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:33:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:57 smithi104 ceph-mon[36624]: pgmap v12258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:57 smithi104 ceph-mon[36624]: pgmap v12259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:33:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:33:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:33:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:57 smithi081 ceph-mon[31453]: pgmap v12258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:57 smithi081 ceph-mon[31453]: pgmap v12259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:58.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:33:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:33:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:33:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:57 smithi153 ceph-mon[36569]: pgmap v12258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:57 smithi153 ceph-mon[36569]: pgmap v12259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:33:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:33:59 smithi153 ceph-mon[36569]: pgmap v12260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:33:59 smithi104 ceph-mon[36624]: pgmap v12260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:33:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:33:59 smithi081 ceph-mon[31453]: pgmap v12260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:01 smithi153 ceph-mon[36569]: pgmap v12261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:01 smithi104 ceph-mon[36624]: pgmap v12261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:01 smithi081 ceph-mon[31453]: pgmap v12261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:03 smithi153 ceph-mon[36569]: pgmap v12262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:03 smithi104 ceph-mon[36624]: pgmap v12262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:03 smithi081 ceph-mon[31453]: pgmap v12262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:05.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:05 smithi081 ceph-mon[31453]: pgmap v12263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:05 smithi153 ceph-mon[36569]: pgmap v12263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:05 smithi104 ceph-mon[36624]: pgmap v12263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:07 smithi153 ceph-mon[36569]: pgmap v12264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:07 smithi104 ceph-mon[36624]: pgmap v12264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:07 smithi081 ceph-mon[31453]: pgmap v12264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:09 smithi153 ceph-mon[36569]: pgmap v12265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:09 smithi104 ceph-mon[36624]: pgmap v12265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:09 smithi081 ceph-mon[31453]: pgmap v12265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:10.134+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:34:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:10.134+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:34:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:10.134+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:34:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:11 smithi153 ceph-mon[36569]: pgmap v12266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:11 smithi104 ceph-mon[36624]: pgmap v12266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:11 smithi081 ceph-mon[31453]: pgmap v12266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:13 smithi153 ceph-mon[36569]: pgmap v12267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:13 smithi104 ceph-mon[36624]: pgmap v12267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:13 smithi081 ceph-mon[31453]: pgmap v12267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:15 smithi153 ceph-mon[36569]: pgmap v12268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:15 smithi104 ceph-mon[36624]: pgmap v12268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:15 smithi081 ceph-mon[31453]: pgmap v12268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:16.448 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:16.447+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:34:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:16.447+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:34:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:16.448+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 176 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:34:16.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:16.448+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 176 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:34:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:17 smithi153 ceph-mon[36569]: pgmap v12269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:17 smithi104 ceph-mon[36624]: pgmap v12269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:17 smithi081 ceph-mon[31453]: pgmap v12269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:19 smithi153 ceph-mon[36569]: pgmap v12270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:19 smithi104 ceph-mon[36624]: pgmap v12270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:19 smithi081 ceph-mon[31453]: pgmap v12270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:21 smithi081 ceph-mon[31453]: pgmap v12271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:21 smithi153 ceph-mon[36569]: pgmap v12271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:21 smithi104 ceph-mon[36624]: pgmap v12271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:23 smithi153 ceph-mon[36569]: pgmap v12272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:23 smithi104 ceph-mon[36624]: pgmap v12272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:23 smithi081 ceph-mon[31453]: pgmap v12272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:25.145+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:34:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:25.145+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:34:25.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:25.145+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:34:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:25 smithi153 ceph-mon[36569]: pgmap v12273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:25 smithi104 ceph-mon[36624]: pgmap v12273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:25 smithi081 ceph-mon[31453]: pgmap v12273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:27 smithi153 ceph-mon[36569]: pgmap v12274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:27 smithi104 ceph-mon[36624]: pgmap v12274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:27 smithi081 ceph-mon[31453]: pgmap v12274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:29 smithi153 ceph-mon[36569]: pgmap v12275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:29 smithi104 ceph-mon[36624]: pgmap v12275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:29 smithi081 ceph-mon[31453]: pgmap v12275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:31 smithi153 ceph-mon[36569]: pgmap v12276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:31 smithi104 ceph-mon[36624]: pgmap v12276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:31 smithi081 ceph-mon[31453]: pgmap v12276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:33 smithi153 ceph-mon[36569]: pgmap v12277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:33 smithi104 ceph-mon[36624]: pgmap v12277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:33 smithi081 ceph-mon[31453]: pgmap v12277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:35 smithi153 ceph-mon[36569]: pgmap v12278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:35 smithi104 ceph-mon[36624]: pgmap v12278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:35 smithi081 ceph-mon[31453]: pgmap v12278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:37 smithi153 ceph-mon[36569]: pgmap v12279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:37 smithi104 ceph-mon[36624]: pgmap v12279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:37 smithi081 ceph-mon[31453]: pgmap v12279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:39 smithi153 ceph-mon[36569]: pgmap v12280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:39 smithi104 ceph-mon[36624]: pgmap v12280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:39 smithi081 ceph-mon[31453]: pgmap v12280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:40.155+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:34:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:40.155+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:34:40.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:40.155+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:34:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:41 smithi153 ceph-mon[36569]: pgmap v12281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:41 smithi104 ceph-mon[36624]: pgmap v12281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:41 smithi081 ceph-mon[31453]: pgmap v12281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:43 smithi153 ceph-mon[36569]: pgmap v12282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:43 smithi104 ceph-mon[36624]: pgmap v12282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:43 smithi081 ceph-mon[31453]: pgmap v12282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:45 smithi153 ceph-mon[36569]: pgmap v12283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:45 smithi104 ceph-mon[36624]: pgmap v12283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:45 smithi081 ceph-mon[31453]: pgmap v12283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:47 smithi153 ceph-mon[36569]: pgmap v12284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:47 smithi104 ceph-mon[36624]: pgmap v12284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:47 smithi081 ceph-mon[31453]: pgmap v12284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:49 smithi104 ceph-mon[36624]: pgmap v12285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:49 smithi081 ceph-mon[31453]: pgmap v12285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:49 smithi153 ceph-mon[36569]: pgmap v12285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:51 smithi104 ceph-mon[36624]: pgmap v12286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:51 smithi081 ceph-mon[31453]: pgmap v12286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:51 smithi153 ceph-mon[36569]: pgmap v12286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:53 smithi104 ceph-mon[36624]: pgmap v12287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:53 smithi081 ceph-mon[31453]: pgmap v12287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:53 smithi153 ceph-mon[36569]: pgmap v12287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:55.171+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:34:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:55.171+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:34:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:34:55.171+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:34:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:55 smithi104 ceph-mon[36624]: pgmap v12288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:55 smithi081 ceph-mon[31453]: pgmap v12288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:55 smithi153 ceph-mon[36569]: pgmap v12288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:57 smithi104 ceph-mon[36624]: pgmap v12289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:57 smithi081 ceph-mon[31453]: pgmap v12289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:57 smithi153 ceph-mon[36569]: pgmap v12289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:34:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:59 smithi104 ceph-mon[36624]: pgmap v12290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:59 smithi104 ceph-mon[36624]: pgmap v12291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:34:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:34:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:59 smithi081 ceph-mon[31453]: pgmap v12290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:59 smithi081 ceph-mon[31453]: pgmap v12291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:34:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:34:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:34:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:34:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:59 smithi153 ceph-mon[36569]: pgmap v12290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:59 smithi153 ceph-mon[36569]: pgmap v12291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:34:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:34:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:35:02.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:01 smithi153 ceph-mon[36569]: pgmap v12292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:01 smithi104 ceph-mon[36624]: pgmap v12292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:01 smithi081 ceph-mon[31453]: pgmap v12292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:04.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:03 smithi153 ceph-mon[36569]: pgmap v12293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:03 smithi104 ceph-mon[36624]: pgmap v12293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:03 smithi081 ceph-mon[31453]: pgmap v12293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:05 smithi153 ceph-mon[36569]: pgmap v12294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:05 smithi104 ceph-mon[36624]: pgmap v12294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:05 smithi081 ceph-mon[31453]: pgmap v12294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:08.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:07 smithi153 ceph-mon[36569]: pgmap v12295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:07 smithi104 ceph-mon[36624]: pgmap v12295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:07 smithi081 ceph-mon[31453]: pgmap v12295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:09 smithi104 ceph-mon[36624]: pgmap v12296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:09 smithi081 ceph-mon[31453]: pgmap v12296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:10.180+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:35:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:10.180+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:35:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:10.180+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:35:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:09 smithi153 ceph-mon[36569]: pgmap v12296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:11 smithi104 ceph-mon[36624]: pgmap v12297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:11 smithi081 ceph-mon[31453]: pgmap v12297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:11 smithi153 ceph-mon[36569]: pgmap v12297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:14.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:13 smithi104 ceph-mon[36624]: pgmap v12298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:13 smithi081 ceph-mon[31453]: pgmap v12298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:13 smithi153 ceph-mon[36569]: pgmap v12298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:15 smithi104 ceph-mon[36624]: pgmap v12299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:15 smithi081 ceph-mon[31453]: pgmap v12299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:15 smithi153 ceph-mon[36569]: pgmap v12299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:17 smithi104 ceph-mon[36624]: pgmap v12300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:18.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:17 smithi153 ceph-mon[36569]: pgmap v12300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:17 smithi081 ceph-mon[31453]: pgmap v12300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:19 smithi081 ceph-mon[31453]: pgmap v12301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:19 smithi153 ceph-mon[36569]: pgmap v12301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:19 smithi104 ceph-mon[36624]: pgmap v12301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:21 smithi081 ceph-mon[31453]: pgmap v12302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:21 smithi153 ceph-mon[36569]: pgmap v12302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:21 smithi104 ceph-mon[36624]: pgmap v12302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:24.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:23 smithi104 ceph-mon[36624]: pgmap v12303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:23 smithi081 ceph-mon[31453]: pgmap v12303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:23 smithi153 ceph-mon[36569]: pgmap v12303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:25.185+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:35:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:25.185+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:35:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:25.185+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:35:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:25 smithi081 ceph-mon[31453]: pgmap v12304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:25 smithi153 ceph-mon[36569]: pgmap v12304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:25 smithi104 ceph-mon[36624]: pgmap v12304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:27 smithi081 ceph-mon[31453]: pgmap v12305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:27 smithi153 ceph-mon[36569]: pgmap v12305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:27 smithi104 ceph-mon[36624]: pgmap v12305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:29 smithi081 ceph-mon[31453]: pgmap v12306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:29 smithi153 ceph-mon[36569]: pgmap v12306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:29 smithi104 ceph-mon[36624]: pgmap v12306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:31 smithi081 ceph-mon[31453]: pgmap v12307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:31 smithi153 ceph-mon[36569]: pgmap v12307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:31 smithi104 ceph-mon[36624]: pgmap v12307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:33 smithi081 ceph-mon[31453]: pgmap v12308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:33 smithi153 ceph-mon[36569]: pgmap v12308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:33 smithi104 ceph-mon[36624]: pgmap v12308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:35 smithi081 ceph-mon[31453]: pgmap v12309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:35 smithi153 ceph-mon[36569]: pgmap v12309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:35 smithi104 ceph-mon[36624]: pgmap v12309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:37 smithi081 ceph-mon[31453]: pgmap v12310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:37 smithi153 ceph-mon[36569]: pgmap v12310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:37 smithi104 ceph-mon[36624]: pgmap v12310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:39 smithi081 ceph-mon[31453]: pgmap v12311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:40.192+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:35:40.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:40.192+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:35:40.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:40.192+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:35:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:39 smithi153 ceph-mon[36569]: pgmap v12311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:39 smithi104 ceph-mon[36624]: pgmap v12311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:42.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:41 smithi081 ceph-mon[31453]: pgmap v12312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:41 smithi153 ceph-mon[36569]: pgmap v12312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:41 smithi104 ceph-mon[36624]: pgmap v12312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:44.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:43 smithi081 ceph-mon[31453]: pgmap v12313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:43 smithi153 ceph-mon[36569]: pgmap v12313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:43 smithi104 ceph-mon[36624]: pgmap v12313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:46.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:45 smithi081 ceph-mon[31453]: pgmap v12314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:45 smithi153 ceph-mon[36569]: pgmap v12314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:45 smithi104 ceph-mon[36624]: pgmap v12314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:46.463 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:46.462+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:35:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:46.462+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:35:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:46.463+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 177 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:35:46.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:46.463+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 177 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:35:48.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:47 smithi153 ceph-mon[36569]: pgmap v12315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:47 smithi104 ceph-mon[36624]: pgmap v12315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:47 smithi081 ceph-mon[31453]: pgmap v12315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:49 smithi153 ceph-mon[36569]: pgmap v12316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:49 smithi104 ceph-mon[36624]: pgmap v12316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:49 smithi081 ceph-mon[31453]: pgmap v12316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:51 smithi153 ceph-mon[36569]: pgmap v12317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:51 smithi104 ceph-mon[36624]: pgmap v12317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:51 smithi081 ceph-mon[31453]: pgmap v12317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:53 smithi153 ceph-mon[36569]: pgmap v12318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:53 smithi104 ceph-mon[36624]: pgmap v12318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:53 smithi081 ceph-mon[31453]: pgmap v12318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:55.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:55.208+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:35:55.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:55.208+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:35:55.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:35:55.208+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:35:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:55 smithi153 ceph-mon[36569]: pgmap v12319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:55 smithi104 ceph-mon[36624]: pgmap v12319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:55 smithi081 ceph-mon[31453]: pgmap v12319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:57 smithi153 ceph-mon[36569]: pgmap v12320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:57 smithi104 ceph-mon[36624]: pgmap v12320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:35:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:57 smithi081 ceph-mon[31453]: pgmap v12320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:59 smithi153 ceph-mon[36569]: pgmap v12321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:35:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:36:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:59 smithi104 ceph-mon[36624]: pgmap v12321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:35:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:36:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:59 smithi081 ceph-mon[31453]: pgmap v12321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:35:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:36:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:36:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:36:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:36:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:36:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:36:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:36:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:36:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:36:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:36:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:01 smithi153 ceph-mon[36569]: pgmap v12322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:01 smithi153 ceph-mon[36569]: pgmap v12323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:01 smithi104 ceph-mon[36624]: pgmap v12322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:01 smithi104 ceph-mon[36624]: pgmap v12323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:01 smithi081 ceph-mon[31453]: pgmap v12322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:01 smithi081 ceph-mon[31453]: pgmap v12323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:03 smithi153 ceph-mon[36569]: pgmap v12324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:03 smithi104 ceph-mon[36624]: pgmap v12324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:03 smithi081 ceph-mon[31453]: pgmap v12324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:05 smithi153 ceph-mon[36569]: pgmap v12325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:05 smithi104 ceph-mon[36624]: pgmap v12325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:05 smithi081 ceph-mon[31453]: pgmap v12325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:07 smithi153 ceph-mon[36569]: pgmap v12326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:07 smithi104 ceph-mon[36624]: pgmap v12326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:07 smithi081 ceph-mon[31453]: pgmap v12326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:10.224+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:36:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:10.224+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:36:10.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:10.224+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:36:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:09 smithi153 ceph-mon[36569]: pgmap v12327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:09 smithi104 ceph-mon[36624]: pgmap v12327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:09 smithi081 ceph-mon[31453]: pgmap v12327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:11 smithi153 ceph-mon[36569]: pgmap v12328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:11 smithi104 ceph-mon[36624]: pgmap v12328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:11 smithi081 ceph-mon[31453]: pgmap v12328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:13 smithi153 ceph-mon[36569]: pgmap v12329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:13 smithi104 ceph-mon[36624]: pgmap v12329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:13 smithi081 ceph-mon[31453]: pgmap v12329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:15 smithi153 ceph-mon[36569]: pgmap v12330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:15 smithi104 ceph-mon[36624]: pgmap v12330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:15 smithi081 ceph-mon[31453]: pgmap v12330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:17 smithi153 ceph-mon[36569]: pgmap v12331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:17 smithi104 ceph-mon[36624]: pgmap v12331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:17 smithi081 ceph-mon[31453]: pgmap v12331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:20 smithi153 ceph-mon[36569]: pgmap v12332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:20 smithi104 ceph-mon[36624]: pgmap v12332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:20 smithi081 ceph-mon[31453]: pgmap v12332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:22 smithi153 ceph-mon[36569]: pgmap v12333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:22 smithi104 ceph-mon[36624]: pgmap v12333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:22.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:22 smithi081 ceph-mon[31453]: pgmap v12333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:24.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:24 smithi081 ceph-mon[31453]: pgmap v12334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:24 smithi153 ceph-mon[36569]: pgmap v12334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:24 smithi104 ceph-mon[36624]: pgmap v12334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:25.234+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:36:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:25.234+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:36:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:25.234+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:36:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:26 smithi153 ceph-mon[36569]: pgmap v12335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:26 smithi104 ceph-mon[36624]: pgmap v12335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:26 smithi081 ceph-mon[31453]: pgmap v12335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:28 smithi153 ceph-mon[36569]: pgmap v12336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:28 smithi104 ceph-mon[36624]: pgmap v12336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:28 smithi081 ceph-mon[31453]: pgmap v12336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:30 smithi153 ceph-mon[36569]: pgmap v12337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:30 smithi104 ceph-mon[36624]: pgmap v12337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:30 smithi081 ceph-mon[31453]: pgmap v12337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:32 smithi153 ceph-mon[36569]: pgmap v12338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:32 smithi104 ceph-mon[36624]: pgmap v12338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:32 smithi081 ceph-mon[31453]: pgmap v12338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:34 smithi153 ceph-mon[36569]: pgmap v12339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:34 smithi104 ceph-mon[36624]: pgmap v12339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:34 smithi081 ceph-mon[31453]: pgmap v12339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:36 smithi153 ceph-mon[36569]: pgmap v12340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:36 smithi104 ceph-mon[36624]: pgmap v12340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:36 smithi081 ceph-mon[31453]: pgmap v12340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:38 smithi153 ceph-mon[36569]: pgmap v12341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:38 smithi104 ceph-mon[36624]: pgmap v12341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:38 smithi081 ceph-mon[31453]: pgmap v12341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:40.244+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:36:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:40.244+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:36:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:40.244+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:36:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:40 smithi153 ceph-mon[36569]: pgmap v12342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:40 smithi104 ceph-mon[36624]: pgmap v12342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:40 smithi081 ceph-mon[31453]: pgmap v12342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:42 smithi104 ceph-mon[36624]: pgmap v12343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:42 smithi081 ceph-mon[31453]: pgmap v12343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:42 smithi153 ceph-mon[36569]: pgmap v12343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:44 smithi104 ceph-mon[36624]: pgmap v12344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:44 smithi081 ceph-mon[31453]: pgmap v12344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:44 smithi153 ceph-mon[36569]: pgmap v12344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:46 smithi104 ceph-mon[36624]: pgmap v12345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:46 smithi081 ceph-mon[31453]: pgmap v12345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:46 smithi153 ceph-mon[36569]: pgmap v12345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:48 smithi104 ceph-mon[36624]: pgmap v12346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:48.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:48 smithi153 ceph-mon[36569]: pgmap v12346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:48 smithi081 ceph-mon[31453]: pgmap v12346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:50 smithi104 ceph-mon[36624]: pgmap v12347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:50 smithi081 ceph-mon[31453]: pgmap v12347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:50 smithi153 ceph-mon[36569]: pgmap v12347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:52 smithi081 ceph-mon[31453]: pgmap v12348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:52 smithi153 ceph-mon[36569]: pgmap v12348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:52 smithi104 ceph-mon[36624]: pgmap v12348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:54.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:54 smithi104 ceph-mon[36624]: pgmap v12349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:54 smithi081 ceph-mon[31453]: pgmap v12349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:54 smithi153 ceph-mon[36569]: pgmap v12349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:55.251+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:36:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:55.251+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:36:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:36:55.251+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:36:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:56 smithi081 ceph-mon[31453]: pgmap v12350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:56 smithi153 ceph-mon[36569]: pgmap v12350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:56 smithi104 ceph-mon[36624]: pgmap v12350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:36:58 smithi081 ceph-mon[31453]: pgmap v12351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:36:58 smithi153 ceph-mon[36569]: pgmap v12351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:36:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:36:58 smithi104 ceph-mon[36624]: pgmap v12351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:00 smithi081 ceph-mon[31453]: pgmap v12352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:00 smithi153 ceph-mon[36569]: pgmap v12352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:00 smithi104 ceph-mon[36624]: pgmap v12352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:37:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:37:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:37:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: pgmap v12353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:37:02.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:37:02.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:37:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: pgmap v12353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:37:02.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:37:02.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:37:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: pgmap v12353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:37:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:37:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:03 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:37:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:03 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:37:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:03 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:37:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:03 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:37:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:03 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:37:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:03 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:37:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:03 smithi081 ceph-mon[31453]: pgmap v12354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:03 smithi081 ceph-mon[31453]: pgmap v12355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:03 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:37:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:03 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:37:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:03 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:37:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:03 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:37:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:03 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:37:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:03 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:37:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:03 smithi153 ceph-mon[36569]: pgmap v12354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:03 smithi153 ceph-mon[36569]: pgmap v12355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:03 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:37:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:03 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:37:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:03 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:37:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:03 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:37:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:03 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:37:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:03 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:37:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:03 smithi104 ceph-mon[36624]: pgmap v12354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:03 smithi104 ceph-mon[36624]: pgmap v12355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:05 smithi081 ceph-mon[31453]: pgmap v12356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:05 smithi153 ceph-mon[36569]: pgmap v12356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:05 smithi104 ceph-mon[36624]: pgmap v12356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:07 smithi081 ceph-mon[31453]: pgmap v12357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:07 smithi153 ceph-mon[36569]: pgmap v12357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:07 smithi104 ceph-mon[36624]: pgmap v12357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:09 smithi081 ceph-mon[31453]: pgmap v12358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:09 smithi153 ceph-mon[36569]: pgmap v12358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:09 smithi104 ceph-mon[36624]: pgmap v12358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:10.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:10.254+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:37:10.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:10.254+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:37:10.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:10.254+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:37:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:11 smithi081 ceph-mon[31453]: pgmap v12359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:11 smithi153 ceph-mon[36569]: pgmap v12359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:11 smithi104 ceph-mon[36624]: pgmap v12359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:13 smithi081 ceph-mon[31453]: pgmap v12360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:13 smithi153 ceph-mon[36569]: pgmap v12360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:13 smithi104 ceph-mon[36624]: pgmap v12360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:15 smithi153 ceph-mon[36569]: pgmap v12361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:15 smithi104 ceph-mon[36624]: pgmap v12361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:15 smithi081 ceph-mon[31453]: pgmap v12361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:16.478+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:37:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:16.478+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:37:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:16.478+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 178 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065e0 con 0x7f1704076980 2024-04-13T03:37:16.479 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:16.479+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 178 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:37:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:17 smithi153 ceph-mon[36569]: pgmap v12362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:17 smithi104 ceph-mon[36624]: pgmap v12362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:17 smithi081 ceph-mon[31453]: pgmap v12362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:19 smithi153 ceph-mon[36569]: pgmap v12363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:19 smithi104 ceph-mon[36624]: pgmap v12363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:19 smithi081 ceph-mon[31453]: pgmap v12363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:21 smithi153 ceph-mon[36569]: pgmap v12364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:21 smithi104 ceph-mon[36624]: pgmap v12364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:21 smithi081 ceph-mon[31453]: pgmap v12364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:23 smithi153 ceph-mon[36569]: pgmap v12365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:23 smithi104 ceph-mon[36624]: pgmap v12365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:23 smithi081 ceph-mon[31453]: pgmap v12365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:25.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:25.267+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:37:25.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:25.267+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:37:25.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:25.267+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:37:25.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:25 smithi081 ceph-mon[31453]: pgmap v12366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:25 smithi153 ceph-mon[36569]: pgmap v12366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:25 smithi104 ceph-mon[36624]: pgmap v12366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:27 smithi153 ceph-mon[36569]: pgmap v12367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:27 smithi104 ceph-mon[36624]: pgmap v12367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:27 smithi081 ceph-mon[31453]: pgmap v12367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:29 smithi153 ceph-mon[36569]: pgmap v12368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:29 smithi104 ceph-mon[36624]: pgmap v12368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:29 smithi081 ceph-mon[31453]: pgmap v12368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:31 smithi153 ceph-mon[36569]: pgmap v12369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:31 smithi104 ceph-mon[36624]: pgmap v12369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:31 smithi081 ceph-mon[31453]: pgmap v12369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:33 smithi153 ceph-mon[36569]: pgmap v12370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:33 smithi104 ceph-mon[36624]: pgmap v12370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:33 smithi081 ceph-mon[31453]: pgmap v12370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:35 smithi153 ceph-mon[36569]: pgmap v12371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:35 smithi104 ceph-mon[36624]: pgmap v12371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:35 smithi081 ceph-mon[31453]: pgmap v12371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:37 smithi153 ceph-mon[36569]: pgmap v12372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:37 smithi104 ceph-mon[36624]: pgmap v12372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:37 smithi081 ceph-mon[31453]: pgmap v12372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:39 smithi153 ceph-mon[36569]: pgmap v12373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:39 smithi104 ceph-mon[36624]: pgmap v12373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:39 smithi081 ceph-mon[31453]: pgmap v12373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:40.284+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:37:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:40.284+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:37:40.285 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:40.284+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:37:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:41 smithi153 ceph-mon[36569]: pgmap v12374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:41 smithi104 ceph-mon[36624]: pgmap v12374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:41 smithi081 ceph-mon[31453]: pgmap v12374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:43 smithi153 ceph-mon[36569]: pgmap v12375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:43 smithi104 ceph-mon[36624]: pgmap v12375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:43 smithi081 ceph-mon[31453]: pgmap v12375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:45 smithi153 ceph-mon[36569]: pgmap v12376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:45 smithi104 ceph-mon[36624]: pgmap v12376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:45 smithi081 ceph-mon[31453]: pgmap v12376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:47 smithi153 ceph-mon[36569]: pgmap v12377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:47 smithi104 ceph-mon[36624]: pgmap v12377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:47 smithi081 ceph-mon[31453]: pgmap v12377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:49 smithi153 ceph-mon[36569]: pgmap v12378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:49 smithi104 ceph-mon[36624]: pgmap v12378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:49 smithi081 ceph-mon[31453]: pgmap v12378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:51 smithi153 ceph-mon[36569]: pgmap v12379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:51 smithi104 ceph-mon[36624]: pgmap v12379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:51 smithi081 ceph-mon[31453]: pgmap v12379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:53 smithi153 ceph-mon[36569]: pgmap v12380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:53 smithi104 ceph-mon[36624]: pgmap v12380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:53 smithi081 ceph-mon[31453]: pgmap v12380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:55.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:55.288+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:37:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:55.288+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:37:55.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:37:55.288+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:37:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:55 smithi153 ceph-mon[36569]: pgmap v12381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:55 smithi104 ceph-mon[36624]: pgmap v12381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:55 smithi081 ceph-mon[31453]: pgmap v12381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:57 smithi081 ceph-mon[31453]: pgmap v12382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:57 smithi153 ceph-mon[36569]: pgmap v12382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:57 smithi104 ceph-mon[36624]: pgmap v12382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:37:59 smithi153 ceph-mon[36569]: pgmap v12383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:37:59 smithi104 ceph-mon[36624]: pgmap v12383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:37:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:37:59 smithi081 ceph-mon[31453]: pgmap v12383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:01 smithi153 ceph-mon[36569]: pgmap v12384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:01 smithi104 ceph-mon[36624]: pgmap v12384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:01 smithi081 ceph-mon[31453]: pgmap v12384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:38:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:38:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:38:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:03 smithi153 ceph-mon[36569]: pgmap v12385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:03 smithi104 ceph-mon[36624]: pgmap v12385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:03.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:03 smithi081 ceph-mon[31453]: pgmap v12385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:05 smithi153 ceph-mon[36569]: pgmap v12386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:38:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:38:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:05 smithi153 ceph-mon[36569]: pgmap v12387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:05 smithi153 ceph-mon[36569]: pgmap v12388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:05 smithi104 ceph-mon[36624]: pgmap v12386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:38:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:38:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:05 smithi104 ceph-mon[36624]: pgmap v12387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:05 smithi104 ceph-mon[36624]: pgmap v12388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:05 smithi081 ceph-mon[31453]: pgmap v12386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:38:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:38:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:05 smithi081 ceph-mon[31453]: pgmap v12387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:05 smithi081 ceph-mon[31453]: pgmap v12388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:38:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:07 smithi153 ceph-mon[36569]: pgmap v12389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:07 smithi104 ceph-mon[36624]: pgmap v12389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:07 smithi081 ceph-mon[31453]: pgmap v12389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:08 smithi081 ceph-mon[31453]: pgmap v12390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:08 smithi153 ceph-mon[36569]: pgmap v12390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:08 smithi104 ceph-mon[36624]: pgmap v12390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:10.301+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:38:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:10.301+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:38:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:10.301+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:38:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:11 smithi153 ceph-mon[36569]: pgmap v12391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:11 smithi104 ceph-mon[36624]: pgmap v12391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:11 smithi081 ceph-mon[31453]: pgmap v12391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:12 smithi081 ceph-mon[31453]: pgmap v12392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:12 smithi153 ceph-mon[36569]: pgmap v12392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:12 smithi104 ceph-mon[36624]: pgmap v12392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:15 smithi153 ceph-mon[36569]: pgmap v12393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:15 smithi104 ceph-mon[36624]: pgmap v12393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:15 smithi081 ceph-mon[31453]: pgmap v12393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:16 smithi104 ceph-mon[36624]: pgmap v12394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:16 smithi081 ceph-mon[31453]: pgmap v12394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:16 smithi153 ceph-mon[36569]: pgmap v12394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:19 smithi153 ceph-mon[36569]: pgmap v12395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:19 smithi104 ceph-mon[36624]: pgmap v12395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:19 smithi081 ceph-mon[31453]: pgmap v12395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:20.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:20 smithi104 ceph-mon[36624]: pgmap v12396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:20 smithi081 ceph-mon[31453]: pgmap v12396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:20 smithi153 ceph-mon[36569]: pgmap v12396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:22.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:22 smithi081 ceph-mon[31453]: pgmap v12397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:22 smithi153 ceph-mon[36569]: pgmap v12397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:22 smithi104 ceph-mon[36624]: pgmap v12397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:24.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:24 smithi081 ceph-mon[31453]: pgmap v12398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:24 smithi153 ceph-mon[36569]: pgmap v12398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:24 smithi104 ceph-mon[36624]: pgmap v12398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:25.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:25.305+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:38:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:25.305+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:38:25.307 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:25.305+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:38:26.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:26 smithi104 ceph-mon[36624]: pgmap v12399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:26 smithi081 ceph-mon[31453]: pgmap v12399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:26 smithi153 ceph-mon[36569]: pgmap v12399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:28 smithi081 ceph-mon[31453]: pgmap v12400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:28 smithi153 ceph-mon[36569]: pgmap v12400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:28 smithi104 ceph-mon[36624]: pgmap v12400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:30 smithi081 ceph-mon[31453]: pgmap v12401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:30 smithi153 ceph-mon[36569]: pgmap v12401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:30 smithi104 ceph-mon[36624]: pgmap v12401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:32 smithi081 ceph-mon[31453]: pgmap v12402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:32 smithi153 ceph-mon[36569]: pgmap v12402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:32 smithi104 ceph-mon[36624]: pgmap v12402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:34 smithi081 ceph-mon[31453]: pgmap v12403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:34 smithi153 ceph-mon[36569]: pgmap v12403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:34 smithi104 ceph-mon[36624]: pgmap v12403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:36 smithi081 ceph-mon[31453]: pgmap v12404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:36 smithi153 ceph-mon[36569]: pgmap v12404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:36 smithi104 ceph-mon[36624]: pgmap v12404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:38.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:38 smithi081 ceph-mon[31453]: pgmap v12405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:38 smithi153 ceph-mon[36569]: pgmap v12405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:38 smithi104 ceph-mon[36624]: pgmap v12405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:40.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:40.320+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:38:40.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:40.320+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:38:40.321 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:40.320+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:38:40.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:40 smithi081 ceph-mon[31453]: pgmap v12406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:40 smithi153 ceph-mon[36569]: pgmap v12406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:40 smithi104 ceph-mon[36624]: pgmap v12406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:42 smithi081 ceph-mon[31453]: pgmap v12407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:42 smithi153 ceph-mon[36569]: pgmap v12407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:42 smithi104 ceph-mon[36624]: pgmap v12407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:44 smithi081 ceph-mon[31453]: pgmap v12408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:44 smithi153 ceph-mon[36569]: pgmap v12408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:44 smithi104 ceph-mon[36624]: pgmap v12408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:46 smithi081 ceph-mon[31453]: pgmap v12409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:46 smithi153 ceph-mon[36569]: pgmap v12409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:46 smithi104 ceph-mon[36624]: pgmap v12409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:48 smithi081 ceph-mon[31453]: pgmap v12410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:48 smithi153 ceph-mon[36569]: pgmap v12410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:48 smithi104 ceph-mon[36624]: pgmap v12410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:50.610 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:50 smithi153 ceph-mon[36569]: pgmap v12411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:50 smithi081 ceph-mon[31453]: pgmap v12411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:50 smithi104 ceph-mon[36624]: pgmap v12411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:52 smithi081 ceph-mon[31453]: pgmap v12412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:52 smithi153 ceph-mon[36569]: pgmap v12412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:52 smithi104 ceph-mon[36624]: pgmap v12412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:54 smithi081 ceph-mon[31453]: pgmap v12413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:54 smithi153 ceph-mon[36569]: pgmap v12413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:54 smithi104 ceph-mon[36624]: pgmap v12413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:55.331+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:38:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:55.331+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:38:55.332 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:38:55.331+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:38:56.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:56 smithi104 ceph-mon[36624]: pgmap v12414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:56 smithi081 ceph-mon[31453]: pgmap v12414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:56 smithi153 ceph-mon[36569]: pgmap v12414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:38:58 smithi081 ceph-mon[31453]: pgmap v12415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:38:58 smithi153 ceph-mon[36569]: pgmap v12415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:38:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:38:58 smithi104 ceph-mon[36624]: pgmap v12415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:00 smithi081 ceph-mon[31453]: pgmap v12416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:00 smithi153 ceph-mon[36569]: pgmap v12416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:00 smithi104 ceph-mon[36624]: pgmap v12416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:02 smithi081 ceph-mon[31453]: pgmap v12417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:02 smithi153 ceph-mon[36569]: pgmap v12417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:02 smithi104 ceph-mon[36624]: pgmap v12417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:04 smithi081 ceph-mon[31453]: pgmap v12418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:04 smithi153 ceph-mon[36569]: pgmap v12418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:04 smithi104 ceph-mon[36624]: pgmap v12418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:39:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:39:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:39:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:39:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:39:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: pgmap v12419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: pgmap v12420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:39:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:39:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: pgmap v12419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: pgmap v12420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:39:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:39:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: pgmap v12419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: pgmap v12420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:39:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:08 smithi153 ceph-mon[36569]: pgmap v12421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:08 smithi104 ceph-mon[36624]: pgmap v12421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:08 smithi081 ceph-mon[31453]: pgmap v12421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:10.335+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:39:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:10.335+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:39:10.337 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:10.335+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:39:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:10 smithi153 ceph-mon[36569]: pgmap v12422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:10 smithi104 ceph-mon[36624]: pgmap v12422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:10 smithi081 ceph-mon[31453]: pgmap v12422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:12 smithi153 ceph-mon[36569]: pgmap v12423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:12 smithi104 ceph-mon[36624]: pgmap v12423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:12 smithi081 ceph-mon[31453]: pgmap v12423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:14 smithi153 ceph-mon[36569]: pgmap v12424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:14 smithi104 ceph-mon[36624]: pgmap v12424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:14 smithi081 ceph-mon[31453]: pgmap v12424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:16 smithi153 ceph-mon[36569]: pgmap v12425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:16 smithi104 ceph-mon[36624]: pgmap v12425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:16 smithi081 ceph-mon[31453]: pgmap v12425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:18 smithi153 ceph-mon[36569]: pgmap v12426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:18 smithi104 ceph-mon[36624]: pgmap v12426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:18 smithi081 ceph-mon[31453]: pgmap v12426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:20 smithi153 ceph-mon[36569]: pgmap v12427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:20 smithi104 ceph-mon[36624]: pgmap v12427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:20 smithi081 ceph-mon[31453]: pgmap v12427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:22 smithi153 ceph-mon[36569]: pgmap v12428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:22 smithi104 ceph-mon[36624]: pgmap v12428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:22 smithi081 ceph-mon[31453]: pgmap v12428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:24 smithi153 ceph-mon[36569]: pgmap v12429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:24 smithi104 ceph-mon[36624]: pgmap v12429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:25.345+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:39:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:25.345+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:39:25.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:25.345+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:39:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:24 smithi081 ceph-mon[31453]: pgmap v12429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:26 smithi153 ceph-mon[36569]: pgmap v12430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:26 smithi104 ceph-mon[36624]: pgmap v12430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:26 smithi081 ceph-mon[31453]: pgmap v12430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:29.299 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:28 smithi081 ceph-mon[31453]: pgmap v12431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:28 smithi153 ceph-mon[36569]: pgmap v12431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:28 smithi104 ceph-mon[36624]: pgmap v12431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:30 smithi153 ceph-mon[36569]: pgmap v12432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:30 smithi104 ceph-mon[36624]: pgmap v12432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:30 smithi081 ceph-mon[31453]: pgmap v12432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:32 smithi153 ceph-mon[36569]: pgmap v12433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:32 smithi104 ceph-mon[36624]: pgmap v12433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:32 smithi081 ceph-mon[31453]: pgmap v12433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:34 smithi153 ceph-mon[36569]: pgmap v12434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:34 smithi104 ceph-mon[36624]: pgmap v12434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:34 smithi081 ceph-mon[31453]: pgmap v12434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:37 smithi153 ceph-mon[36569]: pgmap v12435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:37 smithi104 ceph-mon[36624]: pgmap v12435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:37 smithi081 ceph-mon[31453]: pgmap v12435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:39 smithi153 ceph-mon[36569]: pgmap v12436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:39 smithi104 ceph-mon[36624]: pgmap v12436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:39 smithi081 ceph-mon[31453]: pgmap v12436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:40.361+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:39:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:40.361+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:39:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:40.361+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:39:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:41 smithi153 ceph-mon[36569]: pgmap v12437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:41 smithi104 ceph-mon[36624]: pgmap v12437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:41 smithi081 ceph-mon[31453]: pgmap v12437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:43 smithi153 ceph-mon[36569]: pgmap v12438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:43 smithi104 ceph-mon[36624]: pgmap v12438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:43 smithi081 ceph-mon[31453]: pgmap v12438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:45 smithi153 ceph-mon[36569]: pgmap v12439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:45 smithi104 ceph-mon[36624]: pgmap v12439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:45 smithi081 ceph-mon[31453]: pgmap v12439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:47 smithi153 ceph-mon[36569]: pgmap v12440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:47 smithi104 ceph-mon[36624]: pgmap v12440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:47 smithi081 ceph-mon[31453]: pgmap v12440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:49 smithi153 ceph-mon[36569]: pgmap v12441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:49 smithi104 ceph-mon[36624]: pgmap v12441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:49 smithi081 ceph-mon[31453]: pgmap v12441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:51 smithi153 ceph-mon[36569]: pgmap v12442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:51.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:51 smithi104 ceph-mon[36624]: pgmap v12442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:51 smithi081 ceph-mon[31453]: pgmap v12442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:53 smithi153 ceph-mon[36569]: pgmap v12443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:53 smithi104 ceph-mon[36624]: pgmap v12443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:53 smithi081 ceph-mon[31453]: pgmap v12443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:55 smithi153 ceph-mon[36569]: pgmap v12444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:55 smithi104 ceph-mon[36624]: pgmap v12444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:55.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:55.366+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:39:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:55.366+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:39:55.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:39:55.366+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:39:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:55 smithi081 ceph-mon[31453]: pgmap v12444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:57 smithi104 ceph-mon[36624]: pgmap v12445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:57 smithi081 ceph-mon[31453]: pgmap v12445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:57 smithi153 ceph-mon[36569]: pgmap v12445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:39:59 smithi104 ceph-mon[36624]: pgmap v12446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:39:59 smithi081 ceph-mon[31453]: pgmap v12446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:39:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:39:59 smithi153 ceph-mon[36569]: pgmap v12446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:40:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:40:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:40:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:01 smithi104 ceph-mon[36624]: pgmap v12447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:01 smithi081 ceph-mon[31453]: pgmap v12447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:01 smithi153 ceph-mon[36569]: pgmap v12447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:03 smithi104 ceph-mon[36624]: pgmap v12448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:03 smithi081 ceph-mon[31453]: pgmap v12448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:03 smithi153 ceph-mon[36569]: pgmap v12448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:05 smithi081 ceph-mon[31453]: pgmap v12449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:05 smithi153 ceph-mon[36569]: pgmap v12449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:05 smithi104 ceph-mon[36624]: pgmap v12449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:07 smithi081 ceph-mon[31453]: pgmap v12450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:40:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:07 smithi153 ceph-mon[36569]: pgmap v12450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:40:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:07 smithi104 ceph-mon[36624]: pgmap v12450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:40:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:40:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:40:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:40:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:40:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:40:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:40:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:40:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:40:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:40:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:09 smithi081 ceph-mon[31453]: pgmap v12451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:09 smithi081 ceph-mon[31453]: pgmap v12452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:09 smithi153 ceph-mon[36569]: pgmap v12451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:09 smithi153 ceph-mon[36569]: pgmap v12452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:09 smithi104 ceph-mon[36624]: pgmap v12451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:09 smithi104 ceph-mon[36624]: pgmap v12452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:10.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:10.372+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:40:10.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:10.372+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:40:10.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:10.372+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:40:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:11 smithi081 ceph-mon[31453]: pgmap v12453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:11 smithi153 ceph-mon[36569]: pgmap v12453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:11 smithi104 ceph-mon[36624]: pgmap v12453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:13 smithi081 ceph-mon[31453]: pgmap v12454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:13 smithi153 ceph-mon[36569]: pgmap v12454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:13 smithi104 ceph-mon[36624]: pgmap v12454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:15 smithi081 ceph-mon[31453]: pgmap v12455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:15 smithi153 ceph-mon[36569]: pgmap v12455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:15 smithi104 ceph-mon[36624]: pgmap v12455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:16.508+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:40:16.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:16.508+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:40:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:16.509+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 179 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:40:16.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:16.509+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 179 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:40:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:17 smithi081 ceph-mon[31453]: pgmap v12456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:17 smithi153 ceph-mon[36569]: pgmap v12456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:17 smithi104 ceph-mon[36624]: pgmap v12456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:19 smithi081 ceph-mon[31453]: pgmap v12457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:19 smithi153 ceph-mon[36569]: pgmap v12457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:19 smithi104 ceph-mon[36624]: pgmap v12457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:21 smithi081 ceph-mon[31453]: pgmap v12458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:21 smithi153 ceph-mon[36569]: pgmap v12458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:21 smithi104 ceph-mon[36624]: pgmap v12458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:23 smithi081 ceph-mon[31453]: pgmap v12459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:23 smithi153 ceph-mon[36569]: pgmap v12459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:23 smithi104 ceph-mon[36624]: pgmap v12459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:25.383+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:40:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:25.383+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:40:25.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:25.383+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:40:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:25 smithi081 ceph-mon[31453]: pgmap v12460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:25 smithi153 ceph-mon[36569]: pgmap v12460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:25 smithi104 ceph-mon[36624]: pgmap v12460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:27 smithi081 ceph-mon[31453]: pgmap v12461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:27 smithi153 ceph-mon[36569]: pgmap v12461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:27 smithi104 ceph-mon[36624]: pgmap v12461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:29 smithi081 ceph-mon[31453]: pgmap v12462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:29 smithi153 ceph-mon[36569]: pgmap v12462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:29 smithi104 ceph-mon[36624]: pgmap v12462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:31 smithi153 ceph-mon[36569]: pgmap v12463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:31 smithi104 ceph-mon[36624]: pgmap v12463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:31 smithi081 ceph-mon[31453]: pgmap v12463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:33 smithi153 ceph-mon[36569]: pgmap v12464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:33 smithi104 ceph-mon[36624]: pgmap v12464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:33 smithi081 ceph-mon[31453]: pgmap v12464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:35 smithi153 ceph-mon[36569]: pgmap v12465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:35 smithi104 ceph-mon[36624]: pgmap v12465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:35 smithi081 ceph-mon[31453]: pgmap v12465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:37 smithi153 ceph-mon[36569]: pgmap v12466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:37 smithi104 ceph-mon[36624]: pgmap v12466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:37 smithi081 ceph-mon[31453]: pgmap v12466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:39.526 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:39 smithi081 ceph-mon[31453]: pgmap v12467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:39 smithi153 ceph-mon[36569]: pgmap v12467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:39 smithi104 ceph-mon[36624]: pgmap v12467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:40.390+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:40:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:40.390+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:40:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:40.390+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:40:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:41 smithi153 ceph-mon[36569]: pgmap v12468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:41 smithi104 ceph-mon[36624]: pgmap v12468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:41 smithi081 ceph-mon[31453]: pgmap v12468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:43 smithi153 ceph-mon[36569]: pgmap v12469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:43 smithi104 ceph-mon[36624]: pgmap v12469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:43 smithi081 ceph-mon[31453]: pgmap v12469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:45 smithi153 ceph-mon[36569]: pgmap v12470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:45 smithi104 ceph-mon[36624]: pgmap v12470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:45 smithi081 ceph-mon[31453]: pgmap v12470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:47 smithi081 ceph-mon[31453]: pgmap v12471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:47 smithi153 ceph-mon[36569]: pgmap v12471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:47 smithi104 ceph-mon[36624]: pgmap v12471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:49 smithi153 ceph-mon[36569]: pgmap v12472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:49 smithi104 ceph-mon[36624]: pgmap v12472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:49 smithi081 ceph-mon[31453]: pgmap v12472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:51 smithi153 ceph-mon[36569]: pgmap v12473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:51 smithi104 ceph-mon[36624]: pgmap v12473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:51 smithi081 ceph-mon[31453]: pgmap v12473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:53 smithi153 ceph-mon[36569]: pgmap v12474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:53 smithi104 ceph-mon[36624]: pgmap v12474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:53 smithi081 ceph-mon[31453]: pgmap v12474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:55.405+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:40:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:55.405+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:40:55.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:40:55.405+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:40:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:55 smithi153 ceph-mon[36569]: pgmap v12475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:55 smithi104 ceph-mon[36624]: pgmap v12475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:55 smithi081 ceph-mon[31453]: pgmap v12475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:57 smithi153 ceph-mon[36569]: pgmap v12476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:57 smithi104 ceph-mon[36624]: pgmap v12476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:57 smithi081 ceph-mon[31453]: pgmap v12476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:40:59 smithi153 ceph-mon[36569]: pgmap v12477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:40:59 smithi104 ceph-mon[36624]: pgmap v12477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:40:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:40:59 smithi081 ceph-mon[31453]: pgmap v12477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:01 smithi153 ceph-mon[36569]: pgmap v12478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:01 smithi104 ceph-mon[36624]: pgmap v12478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:01 smithi081 ceph-mon[31453]: pgmap v12478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:03 smithi153 ceph-mon[36569]: pgmap v12479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:03 smithi104 ceph-mon[36624]: pgmap v12479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:03 smithi081 ceph-mon[31453]: pgmap v12479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:05 smithi153 ceph-mon[36569]: pgmap v12480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:05 smithi104 ceph-mon[36624]: pgmap v12480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:05 smithi081 ceph-mon[31453]: pgmap v12480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:07 smithi153 ceph-mon[36569]: pgmap v12481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:07 smithi104 ceph-mon[36624]: pgmap v12481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:07 smithi081 ceph-mon[31453]: pgmap v12481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:41:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:41:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:41:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:09 smithi104 ceph-mon[36624]: pgmap v12482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:09 smithi081 ceph-mon[31453]: pgmap v12482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:09 smithi153 ceph-mon[36569]: pgmap v12482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:10.421+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:41:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:10.421+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:41:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:10.421+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:41:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:11 smithi153 ceph-mon[36569]: pgmap v12483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:11 smithi104 ceph-mon[36624]: pgmap v12483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:11 smithi081 ceph-mon[31453]: pgmap v12483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:13 smithi153 ceph-mon[36569]: pgmap v12484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:13 smithi104 ceph-mon[36624]: pgmap v12484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:13 smithi081 ceph-mon[31453]: pgmap v12484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:15 smithi153 ceph-mon[36569]: pgmap v12485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:41:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:41:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:15 smithi104 ceph-mon[36624]: pgmap v12485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:41:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:41:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:15 smithi081 ceph-mon[31453]: pgmap v12485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:41:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:41:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:41:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:16 smithi153 ceph-mon[36569]: pgmap v12486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:16 smithi153 ceph-mon[36569]: pgmap v12487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:16 smithi104 ceph-mon[36624]: pgmap v12486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:16 smithi104 ceph-mon[36624]: pgmap v12487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:16 smithi081 ceph-mon[31453]: pgmap v12486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:16 smithi081 ceph-mon[31453]: pgmap v12487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:18 smithi153 ceph-mon[36569]: pgmap v12488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:18 smithi104 ceph-mon[36624]: pgmap v12488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:18 smithi081 ceph-mon[31453]: pgmap v12488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:20 smithi153 ceph-mon[36569]: pgmap v12489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:20 smithi104 ceph-mon[36624]: pgmap v12489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:20 smithi081 ceph-mon[31453]: pgmap v12489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:22 smithi153 ceph-mon[36569]: pgmap v12490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:22 smithi104 ceph-mon[36624]: pgmap v12490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:22.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:22 smithi081 ceph-mon[31453]: pgmap v12490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:24 smithi153 ceph-mon[36569]: pgmap v12491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:24 smithi104 ceph-mon[36624]: pgmap v12491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:24 smithi081 ceph-mon[31453]: pgmap v12491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:25.436+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:41:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:25.436+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:41:25.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:25.437+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:41:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:26 smithi153 ceph-mon[36569]: pgmap v12492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:26 smithi104 ceph-mon[36624]: pgmap v12492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:26 smithi081 ceph-mon[31453]: pgmap v12492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:28 smithi153 ceph-mon[36569]: pgmap v12493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:28 smithi104 ceph-mon[36624]: pgmap v12493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:28 smithi081 ceph-mon[31453]: pgmap v12493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:30 smithi153 ceph-mon[36569]: pgmap v12494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:30 smithi104 ceph-mon[36624]: pgmap v12494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:30 smithi081 ceph-mon[31453]: pgmap v12494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:32 smithi153 ceph-mon[36569]: pgmap v12495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:32 smithi104 ceph-mon[36624]: pgmap v12495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:32 smithi081 ceph-mon[31453]: pgmap v12495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:34 smithi153 ceph-mon[36569]: pgmap v12496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:34 smithi104 ceph-mon[36624]: pgmap v12496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:34 smithi081 ceph-mon[31453]: pgmap v12496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:36 smithi153 ceph-mon[36569]: pgmap v12497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:36 smithi104 ceph-mon[36624]: pgmap v12497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:36 smithi081 ceph-mon[31453]: pgmap v12497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:38 smithi153 ceph-mon[36569]: pgmap v12498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:38 smithi104 ceph-mon[36624]: pgmap v12498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:38 smithi081 ceph-mon[31453]: pgmap v12498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:40.449+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:41:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:40.449+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:41:40.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:40.449+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:41:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:40 smithi153 ceph-mon[36569]: pgmap v12499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:40 smithi104 ceph-mon[36624]: pgmap v12499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:40 smithi081 ceph-mon[31453]: pgmap v12499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:42 smithi153 ceph-mon[36569]: pgmap v12500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:42 smithi104 ceph-mon[36624]: pgmap v12500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:42 smithi081 ceph-mon[31453]: pgmap v12500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:44 smithi153 ceph-mon[36569]: pgmap v12501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:44 smithi104 ceph-mon[36624]: pgmap v12501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:44 smithi081 ceph-mon[31453]: pgmap v12501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:46 smithi153 ceph-mon[36569]: pgmap v12502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:46 smithi104 ceph-mon[36624]: pgmap v12502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:46 smithi081 ceph-mon[31453]: pgmap v12502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:48 smithi153 ceph-mon[36569]: pgmap v12503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:48 smithi104 ceph-mon[36624]: pgmap v12503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:48 smithi081 ceph-mon[31453]: pgmap v12503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:50 smithi081 ceph-mon[31453]: pgmap v12504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:50 smithi153 ceph-mon[36569]: pgmap v12504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:50 smithi104 ceph-mon[36624]: pgmap v12504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:52 smithi153 ceph-mon[36569]: pgmap v12505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:52 smithi104 ceph-mon[36624]: pgmap v12505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:52 smithi081 ceph-mon[31453]: pgmap v12505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:54 smithi104 ceph-mon[36624]: pgmap v12506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:54 smithi081 ceph-mon[31453]: pgmap v12506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:54 smithi153 ceph-mon[36569]: pgmap v12506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:55.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:55.466+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:41:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:55.466+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:41:55.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:41:55.466+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:41:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:56 smithi104 ceph-mon[36624]: pgmap v12507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:56 smithi081 ceph-mon[31453]: pgmap v12507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:56 smithi153 ceph-mon[36569]: pgmap v12507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:41:58 smithi104 ceph-mon[36624]: pgmap v12508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:41:58 smithi081 ceph-mon[31453]: pgmap v12508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:41:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:41:58 smithi153 ceph-mon[36569]: pgmap v12508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:00 smithi104 ceph-mon[36624]: pgmap v12509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:00 smithi081 ceph-mon[31453]: pgmap v12509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:00 smithi153 ceph-mon[36569]: pgmap v12509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:02 smithi104 ceph-mon[36624]: pgmap v12510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:02.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:02 smithi081 ceph-mon[31453]: pgmap v12510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:02 smithi153 ceph-mon[36569]: pgmap v12510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:04 smithi081 ceph-mon[31453]: pgmap v12511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:04 smithi153 ceph-mon[36569]: pgmap v12511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:04 smithi104 ceph-mon[36624]: pgmap v12511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:06 smithi081 ceph-mon[31453]: pgmap v12512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:06 smithi153 ceph-mon[36569]: pgmap v12512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:06 smithi104 ceph-mon[36624]: pgmap v12512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:08 smithi081 ceph-mon[31453]: pgmap v12513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:08 smithi153 ceph-mon[36569]: pgmap v12513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:08 smithi104 ceph-mon[36624]: pgmap v12513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:09 smithi081 ceph-mon[31453]: pgmap v12514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:09 smithi153 ceph-mon[36569]: pgmap v12514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:09 smithi104 ceph-mon[36624]: pgmap v12514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:10.477+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:42:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:10.477+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:42:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:10.477+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:42:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:11 smithi153 ceph-mon[36569]: pgmap v12515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:11 smithi104 ceph-mon[36624]: pgmap v12515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:11 smithi081 ceph-mon[31453]: pgmap v12515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:13 smithi153 ceph-mon[36569]: pgmap v12516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:13 smithi104 ceph-mon[36624]: pgmap v12516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:13 smithi081 ceph-mon[31453]: pgmap v12516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:15 smithi153 ceph-mon[36569]: pgmap v12517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:42:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:15 smithi104 ceph-mon[36624]: pgmap v12517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:42:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:15 smithi081 ceph-mon[31453]: pgmap v12517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:42:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:42:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:42:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:16 smithi153 ceph-mon[36569]: pgmap v12518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:16 smithi153 ceph-mon[36569]: pgmap v12519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:42:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:42:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:16 smithi104 ceph-mon[36624]: pgmap v12518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:16 smithi104 ceph-mon[36624]: pgmap v12519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:42:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:42:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:16 smithi081 ceph-mon[31453]: pgmap v12518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:16 smithi081 ceph-mon[31453]: pgmap v12519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:42:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:19 smithi104 ceph-mon[36624]: pgmap v12520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:19 smithi081 ceph-mon[31453]: pgmap v12520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:19 smithi153 ceph-mon[36569]: pgmap v12520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:21 smithi104 ceph-mon[36624]: pgmap v12521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:21 smithi081 ceph-mon[31453]: pgmap v12521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:21 smithi153 ceph-mon[36569]: pgmap v12521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:23 smithi104 ceph-mon[36624]: pgmap v12522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:23.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:23 smithi153 ceph-mon[36569]: pgmap v12522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:23 smithi081 ceph-mon[31453]: pgmap v12522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:25 smithi153 ceph-mon[36569]: pgmap v12523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:25 smithi104 ceph-mon[36624]: pgmap v12523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:25 smithi081 ceph-mon[31453]: pgmap v12523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:25.483+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:42:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:25.484+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:42:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:25.484+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:42:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:27 smithi104 ceph-mon[36624]: pgmap v12524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:27 smithi081 ceph-mon[31453]: pgmap v12524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:27 smithi153 ceph-mon[36569]: pgmap v12524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:29 smithi081 ceph-mon[31453]: pgmap v12525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:29 smithi153 ceph-mon[36569]: pgmap v12525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:29 smithi104 ceph-mon[36624]: pgmap v12525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:31 smithi081 ceph-mon[31453]: pgmap v12526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:31 smithi153 ceph-mon[36569]: pgmap v12526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:31 smithi104 ceph-mon[36624]: pgmap v12526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:33 smithi081 ceph-mon[31453]: pgmap v12527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:33 smithi153 ceph-mon[36569]: pgmap v12527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:33 smithi104 ceph-mon[36624]: pgmap v12527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:35 smithi081 ceph-mon[31453]: pgmap v12528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:35 smithi153 ceph-mon[36569]: pgmap v12528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:35 smithi104 ceph-mon[36624]: pgmap v12528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:37 smithi081 ceph-mon[31453]: pgmap v12529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:37 smithi153 ceph-mon[36569]: pgmap v12529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:37 smithi104 ceph-mon[36624]: pgmap v12529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:39 smithi081 ceph-mon[31453]: pgmap v12530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:39 smithi153 ceph-mon[36569]: pgmap v12530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:39 smithi104 ceph-mon[36624]: pgmap v12530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:40.494+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:42:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:40.494+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:42:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:40.494+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:42:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:41 smithi081 ceph-mon[31453]: pgmap v12531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:41 smithi153 ceph-mon[36569]: pgmap v12531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:41 smithi104 ceph-mon[36624]: pgmap v12531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:43 smithi081 ceph-mon[31453]: pgmap v12532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:43 smithi153 ceph-mon[36569]: pgmap v12532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:43 smithi104 ceph-mon[36624]: pgmap v12532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:45 smithi081 ceph-mon[31453]: pgmap v12533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:45 smithi153 ceph-mon[36569]: pgmap v12533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:45 smithi104 ceph-mon[36624]: pgmap v12533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:47 smithi081 ceph-mon[31453]: pgmap v12534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:47 smithi153 ceph-mon[36569]: pgmap v12534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:47 smithi104 ceph-mon[36624]: pgmap v12534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:49 smithi081 ceph-mon[31453]: pgmap v12535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:49 smithi153 ceph-mon[36569]: pgmap v12535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:49 smithi104 ceph-mon[36624]: pgmap v12535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:51 smithi081 ceph-mon[31453]: pgmap v12536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:51 smithi153 ceph-mon[36569]: pgmap v12536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:51.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:51 smithi104 ceph-mon[36624]: pgmap v12536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:53 smithi081 ceph-mon[31453]: pgmap v12537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:53 smithi153 ceph-mon[36569]: pgmap v12537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:53 smithi104 ceph-mon[36624]: pgmap v12537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:55 smithi081 ceph-mon[31453]: pgmap v12538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:55.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:55.504+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:42:55.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:55.504+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:42:55.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:42:55.504+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:42:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:55 smithi153 ceph-mon[36569]: pgmap v12538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:55 smithi104 ceph-mon[36624]: pgmap v12538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:57 smithi153 ceph-mon[36569]: pgmap v12539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:57 smithi104 ceph-mon[36624]: pgmap v12539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:57 smithi081 ceph-mon[31453]: pgmap v12539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:59.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:42:59 smithi081 ceph-mon[31453]: pgmap v12540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:42:59 smithi153 ceph-mon[36569]: pgmap v12540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:42:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:42:59 smithi104 ceph-mon[36624]: pgmap v12540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:01 smithi153 ceph-mon[36569]: pgmap v12541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:01 smithi104 ceph-mon[36624]: pgmap v12541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:01 smithi081 ceph-mon[31453]: pgmap v12541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:03 smithi153 ceph-mon[36569]: pgmap v12542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:03 smithi104 ceph-mon[36624]: pgmap v12542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:03 smithi081 ceph-mon[31453]: pgmap v12542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:05 smithi153 ceph-mon[36569]: pgmap v12543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:05 smithi104 ceph-mon[36624]: pgmap v12543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:05 smithi081 ceph-mon[31453]: pgmap v12543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:07 smithi153 ceph-mon[36569]: pgmap v12544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:07 smithi104 ceph-mon[36624]: pgmap v12544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:07 smithi081 ceph-mon[31453]: pgmap v12544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:09 smithi153 ceph-mon[36569]: pgmap v12545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:09 smithi104 ceph-mon[36624]: pgmap v12545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:09 smithi081 ceph-mon[31453]: pgmap v12545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:10.509 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:10.509+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:43:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:10.509+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:43:10.510 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:10.509+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:43:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:11 smithi153 ceph-mon[36569]: pgmap v12546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:11 smithi104 ceph-mon[36624]: pgmap v12546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:11 smithi081 ceph-mon[31453]: pgmap v12546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:13 smithi153 ceph-mon[36569]: pgmap v12547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:13 smithi104 ceph-mon[36624]: pgmap v12547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:13 smithi081 ceph-mon[31453]: pgmap v12547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:15 smithi153 ceph-mon[36569]: pgmap v12548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:15 smithi104 ceph-mon[36624]: pgmap v12548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:15 smithi081 ceph-mon[31453]: pgmap v12548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:16.538+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:43:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:16.538+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:43:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:16.539+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 180 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:43:16.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:16.539+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 180 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:43:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:17 smithi153 ceph-mon[36569]: pgmap v12549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:43:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:17 smithi104 ceph-mon[36624]: pgmap v12549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:43:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:17 smithi081 ceph-mon[31453]: pgmap v12549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:43:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:43:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:43:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:43:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:43:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:43:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:43:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:43:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:43:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:43:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:19 smithi153 ceph-mon[36569]: pgmap v12550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:19 smithi153 ceph-mon[36569]: pgmap v12551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:19 smithi104 ceph-mon[36624]: pgmap v12550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:19 smithi104 ceph-mon[36624]: pgmap v12551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:19 smithi081 ceph-mon[31453]: pgmap v12550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:19 smithi081 ceph-mon[31453]: pgmap v12551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:21 smithi153 ceph-mon[36569]: pgmap v12552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:21 smithi104 ceph-mon[36624]: pgmap v12552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:21 smithi081 ceph-mon[31453]: pgmap v12552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:23 smithi153 ceph-mon[36569]: pgmap v12553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:23 smithi104 ceph-mon[36624]: pgmap v12553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:23 smithi081 ceph-mon[31453]: pgmap v12553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:25.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:25.513+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:43:25.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:25.513+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:43:25.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:25.513+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:43:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:25 smithi153 ceph-mon[36569]: pgmap v12554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:25 smithi104 ceph-mon[36624]: pgmap v12554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:25 smithi081 ceph-mon[31453]: pgmap v12554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:27 smithi153 ceph-mon[36569]: pgmap v12555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:27 smithi104 ceph-mon[36624]: pgmap v12555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:27 smithi081 ceph-mon[31453]: pgmap v12555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:29 smithi153 ceph-mon[36569]: pgmap v12556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:29 smithi104 ceph-mon[36624]: pgmap v12556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:29.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:29 smithi081 ceph-mon[31453]: pgmap v12556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:31 smithi153 ceph-mon[36569]: pgmap v12557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:31 smithi104 ceph-mon[36624]: pgmap v12557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:31 smithi081 ceph-mon[31453]: pgmap v12557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:33 smithi153 ceph-mon[36569]: pgmap v12558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:33 smithi104 ceph-mon[36624]: pgmap v12558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:33 smithi081 ceph-mon[31453]: pgmap v12558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:35 smithi153 ceph-mon[36569]: pgmap v12559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:35 smithi104 ceph-mon[36624]: pgmap v12559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:35 smithi081 ceph-mon[31453]: pgmap v12559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:37 smithi153 ceph-mon[36569]: pgmap v12560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:37 smithi104 ceph-mon[36624]: pgmap v12560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:37 smithi081 ceph-mon[31453]: pgmap v12560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:39 smithi153 ceph-mon[36569]: pgmap v12561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:39 smithi104 ceph-mon[36624]: pgmap v12561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:39 smithi081 ceph-mon[31453]: pgmap v12561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:40.520+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:43:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:40.520+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:43:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:40.520+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:43:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:41 smithi153 ceph-mon[36569]: pgmap v12562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:41 smithi104 ceph-mon[36624]: pgmap v12562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:41 smithi081 ceph-mon[31453]: pgmap v12562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:43 smithi153 ceph-mon[36569]: pgmap v12563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:43 smithi104 ceph-mon[36624]: pgmap v12563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:43 smithi081 ceph-mon[31453]: pgmap v12563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:45 smithi153 ceph-mon[36569]: pgmap v12564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:45 smithi104 ceph-mon[36624]: pgmap v12564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:45 smithi081 ceph-mon[31453]: pgmap v12564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:47 smithi153 ceph-mon[36569]: pgmap v12565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:47 smithi104 ceph-mon[36624]: pgmap v12565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:47 smithi081 ceph-mon[31453]: pgmap v12565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:49 smithi104 ceph-mon[36624]: pgmap v12566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:49 smithi081 ceph-mon[31453]: pgmap v12566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:49 smithi153 ceph-mon[36569]: pgmap v12566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:51 smithi104 ceph-mon[36624]: pgmap v12567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:51 smithi081 ceph-mon[31453]: pgmap v12567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:51 smithi153 ceph-mon[36569]: pgmap v12567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:53 smithi104 ceph-mon[36624]: pgmap v12568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:53.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:53 smithi153 ceph-mon[36569]: pgmap v12568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:53 smithi081 ceph-mon[31453]: pgmap v12568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:55.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:55.536+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:43:55.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:55.536+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:43:55.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:43:55.536+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:43:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:55 smithi104 ceph-mon[36624]: pgmap v12569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:55 smithi081 ceph-mon[31453]: pgmap v12569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:55 smithi153 ceph-mon[36569]: pgmap v12569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:57 smithi104 ceph-mon[36624]: pgmap v12570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:57 smithi081 ceph-mon[31453]: pgmap v12570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:57 smithi153 ceph-mon[36569]: pgmap v12570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:43:59 smithi081 ceph-mon[31453]: pgmap v12571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:43:59 smithi153 ceph-mon[36569]: pgmap v12571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:43:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:43:59 smithi104 ceph-mon[36624]: pgmap v12571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:01.546+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:44:01.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:01.546+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:44:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:01.546+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 181 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:44:01.547 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:01.546+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 181 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:44:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:01 smithi081 ceph-mon[31453]: pgmap v12572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:01 smithi153 ceph-mon[36569]: pgmap v12572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:01 smithi104 ceph-mon[36624]: pgmap v12572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:03 smithi081 ceph-mon[31453]: pgmap v12573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:03 smithi153 ceph-mon[36569]: pgmap v12573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:03 smithi104 ceph-mon[36624]: pgmap v12573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:05 smithi081 ceph-mon[31453]: pgmap v12574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:05 smithi153 ceph-mon[36569]: pgmap v12574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:05 smithi104 ceph-mon[36624]: pgmap v12574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:07 smithi081 ceph-mon[31453]: pgmap v12575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:07 smithi153 ceph-mon[36569]: pgmap v12575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:07 smithi104 ceph-mon[36624]: pgmap v12575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:09 smithi081 ceph-mon[31453]: pgmap v12576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:09 smithi153 ceph-mon[36569]: pgmap v12576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:09 smithi104 ceph-mon[36624]: pgmap v12576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:10.551+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:44:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:10.551+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:44:10.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:10.551+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:44:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:11 smithi081 ceph-mon[31453]: pgmap v12577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:11 smithi153 ceph-mon[36569]: pgmap v12577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:11 smithi104 ceph-mon[36624]: pgmap v12577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:13 smithi081 ceph-mon[31453]: pgmap v12578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:13 smithi153 ceph-mon[36569]: pgmap v12578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:13 smithi104 ceph-mon[36624]: pgmap v12578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:14 smithi153 ceph-mon[36569]: pgmap v12579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:14 smithi104 ceph-mon[36624]: pgmap v12579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:14 smithi081 ceph-mon[31453]: pgmap v12579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:16 smithi081 ceph-mon[31453]: pgmap v12580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:16 smithi153 ceph-mon[36569]: pgmap v12580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:16 smithi104 ceph-mon[36624]: pgmap v12580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:18 smithi153 ceph-mon[36569]: pgmap v12581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:44:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:18 smithi104 ceph-mon[36624]: pgmap v12581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:44:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:18 smithi081 ceph-mon[31453]: pgmap v12581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:44:20.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:20 smithi104 ceph-mon[36624]: pgmap v12582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:20 smithi153 ceph-mon[36569]: pgmap v12582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:20 smithi081 ceph-mon[31453]: pgmap v12582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:22 smithi104 ceph-mon[36624]: pgmap v12583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:22 smithi153 ceph-mon[36569]: pgmap v12583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:22 smithi081 ceph-mon[31453]: pgmap v12583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:23.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:24.873 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:24 smithi081 ceph-mon[31453]: pgmap v12584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:24 smithi153 ceph-mon[36569]: pgmap v12584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:24 smithi104 ceph-mon[36624]: pgmap v12584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:25.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:44:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:25.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:44:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:25.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:44:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:44:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:44:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:25 smithi153 ceph-mon[36569]: pgmap v12585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:25 smithi153 ceph-mon[36569]: pgmap v12586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:44:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:44:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:25 smithi104 ceph-mon[36624]: pgmap v12585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:25 smithi104 ceph-mon[36624]: pgmap v12586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:44:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:44:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:25 smithi081 ceph-mon[31453]: pgmap v12585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:25 smithi081 ceph-mon[31453]: pgmap v12586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:44:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:28 smithi153 ceph-mon[36569]: pgmap v12587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:28 smithi104 ceph-mon[36624]: pgmap v12587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:28 smithi081 ceph-mon[31453]: pgmap v12587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:30 smithi153 ceph-mon[36569]: pgmap v12588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:30 smithi104 ceph-mon[36624]: pgmap v12588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:30 smithi081 ceph-mon[31453]: pgmap v12588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:32 smithi153 ceph-mon[36569]: pgmap v12589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:32 smithi104 ceph-mon[36624]: pgmap v12589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:32 smithi081 ceph-mon[31453]: pgmap v12589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:34 smithi104 ceph-mon[36624]: pgmap v12590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:34 smithi081 ceph-mon[31453]: pgmap v12590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:34 smithi153 ceph-mon[36569]: pgmap v12590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:36 smithi104 ceph-mon[36624]: pgmap v12591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:36 smithi081 ceph-mon[31453]: pgmap v12591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:36 smithi153 ceph-mon[36569]: pgmap v12591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:38 smithi104 ceph-mon[36624]: pgmap v12592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:38 smithi081 ceph-mon[31453]: pgmap v12592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:38 smithi153 ceph-mon[36569]: pgmap v12592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:40 smithi104 ceph-mon[36624]: pgmap v12593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:40.372 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:40 smithi081 ceph-mon[31453]: pgmap v12593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:40 smithi153 ceph-mon[36569]: pgmap v12593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:40.571+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:44:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:40.571+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:44:40.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:40.571+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:44:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:42 smithi104 ceph-mon[36624]: pgmap v12594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:42 smithi081 ceph-mon[31453]: pgmap v12594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:42 smithi153 ceph-mon[36569]: pgmap v12594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:44 smithi081 ceph-mon[31453]: pgmap v12595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:44 smithi153 ceph-mon[36569]: pgmap v12595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:44 smithi104 ceph-mon[36624]: pgmap v12595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:46 smithi081 ceph-mon[31453]: pgmap v12596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:46 smithi153 ceph-mon[36569]: pgmap v12596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:46 smithi104 ceph-mon[36624]: pgmap v12596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:48 smithi081 ceph-mon[31453]: pgmap v12597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:48 smithi153 ceph-mon[36569]: pgmap v12597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:48 smithi104 ceph-mon[36624]: pgmap v12597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:50 smithi081 ceph-mon[31453]: pgmap v12598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:50 smithi153 ceph-mon[36569]: pgmap v12598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:50 smithi104 ceph-mon[36624]: pgmap v12598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:52 smithi081 ceph-mon[31453]: pgmap v12599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:52 smithi153 ceph-mon[36569]: pgmap v12599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:52 smithi104 ceph-mon[36624]: pgmap v12599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:54 smithi081 ceph-mon[31453]: pgmap v12600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:54 smithi153 ceph-mon[36569]: pgmap v12600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:54 smithi104 ceph-mon[36624]: pgmap v12600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:55.578+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:44:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:55.578+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:44:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:44:55.578+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:44:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:56 smithi081 ceph-mon[31453]: pgmap v12601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:56 smithi153 ceph-mon[36569]: pgmap v12601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:56 smithi104 ceph-mon[36624]: pgmap v12601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:44:58 smithi081 ceph-mon[31453]: pgmap v12602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:44:58 smithi153 ceph-mon[36569]: pgmap v12602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:44:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:44:58 smithi104 ceph-mon[36624]: pgmap v12602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:00 smithi081 ceph-mon[31453]: pgmap v12603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:00 smithi153 ceph-mon[36569]: pgmap v12603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:00 smithi104 ceph-mon[36624]: pgmap v12603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:02 smithi081 ceph-mon[31453]: pgmap v12604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:02 smithi153 ceph-mon[36569]: pgmap v12604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:02 smithi104 ceph-mon[36624]: pgmap v12604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:04 smithi081 ceph-mon[31453]: pgmap v12605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:04 smithi153 ceph-mon[36569]: pgmap v12605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:04 smithi104 ceph-mon[36624]: pgmap v12605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:06 smithi081 ceph-mon[31453]: pgmap v12606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:06 smithi153 ceph-mon[36569]: pgmap v12606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:06 smithi104 ceph-mon[36624]: pgmap v12606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:08 smithi153 ceph-mon[36569]: pgmap v12607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:08 smithi104 ceph-mon[36624]: pgmap v12607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:08 smithi081 ceph-mon[31453]: pgmap v12607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:10 smithi153 ceph-mon[36569]: pgmap v12608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:10.579+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:45:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:10.579+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:45:10.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:10.579+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:45:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:10 smithi104 ceph-mon[36624]: pgmap v12608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:10 smithi081 ceph-mon[31453]: pgmap v12608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:12 smithi153 ceph-mon[36569]: pgmap v12609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:12 smithi104 ceph-mon[36624]: pgmap v12609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:12 smithi081 ceph-mon[31453]: pgmap v12609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:14 smithi153 ceph-mon[36569]: pgmap v12610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:14 smithi104 ceph-mon[36624]: pgmap v12610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:14 smithi081 ceph-mon[31453]: pgmap v12610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:16 smithi153 ceph-mon[36569]: pgmap v12611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:16 smithi104 ceph-mon[36624]: pgmap v12611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:16 smithi081 ceph-mon[31453]: pgmap v12611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:18 smithi153 ceph-mon[36569]: pgmap v12612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:18 smithi104 ceph-mon[36624]: pgmap v12612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:18 smithi081 ceph-mon[31453]: pgmap v12612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:20 smithi153 ceph-mon[36569]: pgmap v12613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:20 smithi104 ceph-mon[36624]: pgmap v12613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:20 smithi081 ceph-mon[31453]: pgmap v12613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:22 smithi153 ceph-mon[36569]: pgmap v12614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:22 smithi104 ceph-mon[36624]: pgmap v12614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:22 smithi081 ceph-mon[31453]: pgmap v12614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:24 smithi153 ceph-mon[36569]: pgmap v12615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:24 smithi104 ceph-mon[36624]: pgmap v12615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:24 smithi081 ceph-mon[31453]: pgmap v12615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:25.595+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:45:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:25.595+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:45:25.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:25.595+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:45:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:26 smithi081 ceph-mon[31453]: pgmap v12616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:45:26.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:26 smithi153 ceph-mon[36569]: pgmap v12616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:26.544 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:45:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:26 smithi104 ceph-mon[36624]: pgmap v12616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:45:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:28 smithi153 ceph-mon[36569]: pgmap v12617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:28 smithi104 ceph-mon[36624]: pgmap v12617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:28 smithi081 ceph-mon[31453]: pgmap v12617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:30 smithi153 ceph-mon[36569]: pgmap v12618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:45:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:45:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:30 smithi104 ceph-mon[36624]: pgmap v12618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:45:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:45:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:30 smithi081 ceph-mon[31453]: pgmap v12618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:45:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:45:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:45:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:31 smithi153 ceph-mon[36569]: pgmap v12619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:31 smithi153 ceph-mon[36569]: pgmap v12620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:31 smithi104 ceph-mon[36624]: pgmap v12619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:31 smithi104 ceph-mon[36624]: pgmap v12620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:31 smithi081 ceph-mon[31453]: pgmap v12619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:31 smithi081 ceph-mon[31453]: pgmap v12620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:33 smithi153 ceph-mon[36569]: pgmap v12621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:33 smithi104 ceph-mon[36624]: pgmap v12621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:33 smithi081 ceph-mon[31453]: pgmap v12621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:35 smithi153 ceph-mon[36569]: pgmap v12622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:35 smithi104 ceph-mon[36624]: pgmap v12622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:35 smithi081 ceph-mon[31453]: pgmap v12622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:37 smithi153 ceph-mon[36569]: pgmap v12623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:37 smithi104 ceph-mon[36624]: pgmap v12623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:37 smithi081 ceph-mon[31453]: pgmap v12623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:39 smithi153 ceph-mon[36569]: pgmap v12624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:39 smithi104 ceph-mon[36624]: pgmap v12624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:39 smithi081 ceph-mon[31453]: pgmap v12624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:40.607+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:40.608+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:45:40.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:40.608+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:45:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:41 smithi153 ceph-mon[36569]: pgmap v12625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:41 smithi104 ceph-mon[36624]: pgmap v12625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:41 smithi081 ceph-mon[31453]: pgmap v12625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:43 smithi153 ceph-mon[36569]: pgmap v12626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:43 smithi104 ceph-mon[36624]: pgmap v12626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:43 smithi081 ceph-mon[31453]: pgmap v12626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:45 smithi153 ceph-mon[36569]: pgmap v12627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:45 smithi104 ceph-mon[36624]: pgmap v12627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:45 smithi081 ceph-mon[31453]: pgmap v12627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:47 smithi153 ceph-mon[36569]: pgmap v12628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:47 smithi104 ceph-mon[36624]: pgmap v12628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:47 smithi081 ceph-mon[31453]: pgmap v12628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:49 smithi153 ceph-mon[36569]: pgmap v12629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:49 smithi104 ceph-mon[36624]: pgmap v12629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:49 smithi081 ceph-mon[31453]: pgmap v12629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:51 smithi153 ceph-mon[36569]: pgmap v12630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:51 smithi104 ceph-mon[36624]: pgmap v12630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:51 smithi081 ceph-mon[31453]: pgmap v12630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:53 smithi153 ceph-mon[36569]: pgmap v12631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:53 smithi104 ceph-mon[36624]: pgmap v12631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:53 smithi081 ceph-mon[31453]: pgmap v12631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:55 smithi153 ceph-mon[36569]: pgmap v12632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:55 smithi104 ceph-mon[36624]: pgmap v12632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:55.619+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:45:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:55.619+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:45:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:45:55.619+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:45:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:55 smithi081 ceph-mon[31453]: pgmap v12632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:57 smithi153 ceph-mon[36569]: pgmap v12633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:57 smithi104 ceph-mon[36624]: pgmap v12633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:57 smithi081 ceph-mon[31453]: pgmap v12633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:45:59 smithi153 ceph-mon[36569]: pgmap v12634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:45:59 smithi104 ceph-mon[36624]: pgmap v12634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:45:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:45:59 smithi081 ceph-mon[31453]: pgmap v12634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:01 smithi153 ceph-mon[36569]: pgmap v12635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:01 smithi104 ceph-mon[36624]: pgmap v12635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:01 smithi081 ceph-mon[31453]: pgmap v12635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:03 smithi153 ceph-mon[36569]: pgmap v12636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:03 smithi104 ceph-mon[36624]: pgmap v12636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:03 smithi081 ceph-mon[31453]: pgmap v12636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:05 smithi153 ceph-mon[36569]: pgmap v12637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:05 smithi104 ceph-mon[36624]: pgmap v12637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:05 smithi081 ceph-mon[31453]: pgmap v12637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:07 smithi104 ceph-mon[36624]: pgmap v12638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:07 smithi081 ceph-mon[31453]: pgmap v12638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:07 smithi153 ceph-mon[36569]: pgmap v12638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:09 smithi104 ceph-mon[36624]: pgmap v12639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:09 smithi081 ceph-mon[31453]: pgmap v12639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:09 smithi153 ceph-mon[36569]: pgmap v12639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:10.627+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:46:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:10.628+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:46:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:10.628+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:46:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:11 smithi104 ceph-mon[36624]: pgmap v12640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:11 smithi081 ceph-mon[31453]: pgmap v12640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:11 smithi153 ceph-mon[36569]: pgmap v12640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:12.154+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000664) for more than 900000000 us, fault. 2024-04-13T03:46:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:12.154+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c760 tx=0x7f16f40bf000 comp rx=0 tx=0).stop 2024-04-13T03:46:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:12.154+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b3ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T03:46:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:12.154+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T03:46:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:12.154+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T03:46:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:12.155+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T03:46:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:12.156+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ce2a0 tx=0x7f16f4032000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T03:46:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:13 smithi104 ceph-mon[36624]: pgmap v12641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:13 smithi081 ceph-mon[31453]: pgmap v12641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:13.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:13 smithi153 ceph-mon[36569]: pgmap v12641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:15 smithi104 ceph-mon[36624]: pgmap v12642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:15 smithi081 ceph-mon[31453]: pgmap v12642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:15 smithi153 ceph-mon[36569]: pgmap v12642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:16.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:16.567+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:46:16.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:16.567+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:46:16.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:16.568+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 182 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:46:16.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:16.568+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 182 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:46:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:17 smithi081 ceph-mon[31453]: pgmap v12643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:17 smithi153 ceph-mon[36569]: pgmap v12643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:17 smithi104 ceph-mon[36624]: pgmap v12643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:19 smithi081 ceph-mon[31453]: pgmap v12644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:19 smithi153 ceph-mon[36569]: pgmap v12644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:19 smithi104 ceph-mon[36624]: pgmap v12644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:21 smithi081 ceph-mon[31453]: pgmap v12645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:21 smithi153 ceph-mon[36569]: pgmap v12645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:21 smithi104 ceph-mon[36624]: pgmap v12645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:23 smithi081 ceph-mon[31453]: pgmap v12646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:23 smithi153 ceph-mon[36569]: pgmap v12646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:23 smithi104 ceph-mon[36624]: pgmap v12646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:25.630+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:46:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:25.630+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:46:25.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:25.630+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:46:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:25 smithi081 ceph-mon[31453]: pgmap v12647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:25 smithi153 ceph-mon[36569]: pgmap v12647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:25 smithi104 ceph-mon[36624]: pgmap v12647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:27 smithi081 ceph-mon[31453]: pgmap v12648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:27 smithi153 ceph-mon[36569]: pgmap v12648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:27 smithi104 ceph-mon[36624]: pgmap v12648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:29 smithi081 ceph-mon[31453]: pgmap v12649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:29 smithi153 ceph-mon[36569]: pgmap v12649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:29 smithi104 ceph-mon[36624]: pgmap v12649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:46:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:46:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:46:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:31 smithi081 ceph-mon[31453]: pgmap v12650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:31.733 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:31 smithi104 ceph-mon[36624]: pgmap v12650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:31 smithi153 ceph-mon[36569]: pgmap v12650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:46:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:46:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:46:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:46:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:46:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:46:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:46:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:46:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:46:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:33 smithi081 ceph-mon[31453]: pgmap v12651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:33 smithi081 ceph-mon[31453]: pgmap v12652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:33 smithi153 ceph-mon[36569]: pgmap v12651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:33 smithi153 ceph-mon[36569]: pgmap v12652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:33 smithi104 ceph-mon[36624]: pgmap v12651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:33 smithi104 ceph-mon[36624]: pgmap v12652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:35 smithi081 ceph-mon[31453]: pgmap v12653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:35 smithi153 ceph-mon[36569]: pgmap v12653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:35 smithi104 ceph-mon[36624]: pgmap v12653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:37 smithi081 ceph-mon[31453]: pgmap v12654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:37 smithi153 ceph-mon[36569]: pgmap v12654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:37 smithi104 ceph-mon[36624]: pgmap v12654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:38 smithi153 ceph-mon[36569]: pgmap v12655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:38 smithi104 ceph-mon[36624]: pgmap v12655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:38 smithi081 ceph-mon[31453]: pgmap v12655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:40.636+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:46:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:40.636+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:46:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:40.636+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:46:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:40 smithi153 ceph-mon[36569]: pgmap v12656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:40 smithi104 ceph-mon[36624]: pgmap v12656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:40 smithi081 ceph-mon[31453]: pgmap v12656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:42 smithi153 ceph-mon[36569]: pgmap v12657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:42 smithi104 ceph-mon[36624]: pgmap v12657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:42 smithi081 ceph-mon[31453]: pgmap v12657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:44 smithi153 ceph-mon[36569]: pgmap v12658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:44 smithi104 ceph-mon[36624]: pgmap v12658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:44 smithi081 ceph-mon[31453]: pgmap v12658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:46 smithi153 ceph-mon[36569]: pgmap v12659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:46 smithi104 ceph-mon[36624]: pgmap v12659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:46 smithi081 ceph-mon[31453]: pgmap v12659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:48 smithi153 ceph-mon[36569]: pgmap v12660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:48 smithi104 ceph-mon[36624]: pgmap v12660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:48 smithi081 ceph-mon[31453]: pgmap v12660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:50 smithi153 ceph-mon[36569]: pgmap v12661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:50 smithi104 ceph-mon[36624]: pgmap v12661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:50 smithi081 ceph-mon[31453]: pgmap v12661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:52 smithi153 ceph-mon[36569]: pgmap v12662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:52 smithi104 ceph-mon[36624]: pgmap v12662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:52 smithi081 ceph-mon[31453]: pgmap v12662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:54 smithi153 ceph-mon[36569]: pgmap v12663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:54 smithi104 ceph-mon[36624]: pgmap v12663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:54 smithi081 ceph-mon[31453]: pgmap v12663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:55.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:55.649+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:46:55.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:55.649+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:46:55.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:46:55.649+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:46:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:56 smithi153 ceph-mon[36569]: pgmap v12664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:56 smithi104 ceph-mon[36624]: pgmap v12664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:56 smithi081 ceph-mon[31453]: pgmap v12664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:46:58 smithi153 ceph-mon[36569]: pgmap v12665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:46:58 smithi104 ceph-mon[36624]: pgmap v12665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:46:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:46:58 smithi081 ceph-mon[31453]: pgmap v12665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:00 smithi153 ceph-mon[36569]: pgmap v12666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:00 smithi104 ceph-mon[36624]: pgmap v12666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:00 smithi081 ceph-mon[31453]: pgmap v12666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:02 smithi104 ceph-mon[36624]: pgmap v12667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:02 smithi081 ceph-mon[31453]: pgmap v12667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:02 smithi153 ceph-mon[36569]: pgmap v12667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:04 smithi104 ceph-mon[36624]: pgmap v12668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:04 smithi081 ceph-mon[31453]: pgmap v12668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:04 smithi153 ceph-mon[36569]: pgmap v12668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:06 smithi104 ceph-mon[36624]: pgmap v12669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:06 smithi081 ceph-mon[31453]: pgmap v12669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:07.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:06 smithi153 ceph-mon[36569]: pgmap v12669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:08 smithi153 ceph-mon[36569]: pgmap v12670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:08 smithi104 ceph-mon[36624]: pgmap v12670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:08 smithi081 ceph-mon[31453]: pgmap v12670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:10.665+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:47:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:10.665+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:47:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:10.665+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:47:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:10 smithi104 ceph-mon[36624]: pgmap v12671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:10 smithi081 ceph-mon[31453]: pgmap v12671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:10 smithi153 ceph-mon[36569]: pgmap v12671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:12 smithi153 ceph-mon[36569]: pgmap v12672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:12 smithi104 ceph-mon[36624]: pgmap v12672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:12 smithi081 ceph-mon[31453]: pgmap v12672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:14 smithi153 ceph-mon[36569]: pgmap v12673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:14 smithi104 ceph-mon[36624]: pgmap v12673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:14 smithi081 ceph-mon[31453]: pgmap v12673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:17 smithi153 ceph-mon[36569]: pgmap v12674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:17 smithi104 ceph-mon[36624]: pgmap v12674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:17 smithi081 ceph-mon[31453]: pgmap v12674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:19 smithi153 ceph-mon[36569]: pgmap v12675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:19 smithi104 ceph-mon[36624]: pgmap v12675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:19 smithi081 ceph-mon[31453]: pgmap v12675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:21 smithi153 ceph-mon[36569]: pgmap v12676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:21 smithi104 ceph-mon[36624]: pgmap v12676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:21 smithi081 ceph-mon[31453]: pgmap v12676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:23 smithi153 ceph-mon[36569]: pgmap v12677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:23 smithi104 ceph-mon[36624]: pgmap v12677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:23 smithi081 ceph-mon[31453]: pgmap v12677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:25 smithi153 ceph-mon[36569]: pgmap v12678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:25 smithi104 ceph-mon[36624]: pgmap v12678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:25 smithi081 ceph-mon[31453]: pgmap v12678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:25.680+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:25.680+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:47:25.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:25.680+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:47:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:27 smithi153 ceph-mon[36569]: pgmap v12679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:27 smithi104 ceph-mon[36624]: pgmap v12679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:27 smithi081 ceph-mon[31453]: pgmap v12679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:29 smithi153 ceph-mon[36569]: pgmap v12680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:29 smithi104 ceph-mon[36624]: pgmap v12680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:29 smithi081 ceph-mon[31453]: pgmap v12680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:31 smithi153 ceph-mon[36569]: pgmap v12681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:31 smithi104 ceph-mon[36624]: pgmap v12681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:31 smithi081 ceph-mon[31453]: pgmap v12681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:47:32.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:47:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: pgmap v12682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:47:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:47:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:47:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: pgmap v12682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:47:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:47:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: pgmap v12682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:47:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:47:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:47:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:47:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:47:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:47:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:47:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:47:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:34 smithi104 ceph-mon[36624]: pgmap v12683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:34 smithi104 ceph-mon[36624]: pgmap v12684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:34 smithi081 ceph-mon[31453]: pgmap v12683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:34 smithi081 ceph-mon[31453]: pgmap v12684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:47:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:47:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:47:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:47:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:47:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:47:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:34 smithi153 ceph-mon[36569]: pgmap v12683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:34 smithi153 ceph-mon[36569]: pgmap v12684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:36 smithi104 ceph-mon[36624]: pgmap v12685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:36 smithi081 ceph-mon[31453]: pgmap v12685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:36 smithi153 ceph-mon[36569]: pgmap v12685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:38 smithi104 ceph-mon[36624]: pgmap v12686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:38 smithi081 ceph-mon[31453]: pgmap v12686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:38 smithi153 ceph-mon[36569]: pgmap v12686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:40 smithi104 ceph-mon[36624]: pgmap v12687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:40 smithi081 ceph-mon[31453]: pgmap v12687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:40 smithi153 ceph-mon[36569]: pgmap v12687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:40.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:40.692+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:47:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:40.692+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:47:40.693 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:40.692+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:47:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:42 smithi104 ceph-mon[36624]: pgmap v12688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:42 smithi081 ceph-mon[31453]: pgmap v12688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:42 smithi153 ceph-mon[36569]: pgmap v12688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:44 smithi081 ceph-mon[31453]: pgmap v12689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:44 smithi153 ceph-mon[36569]: pgmap v12689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:44 smithi104 ceph-mon[36624]: pgmap v12689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:46 smithi081 ceph-mon[31453]: pgmap v12690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:46 smithi153 ceph-mon[36569]: pgmap v12690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:46 smithi104 ceph-mon[36624]: pgmap v12690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:48 smithi081 ceph-mon[31453]: pgmap v12691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:48 smithi153 ceph-mon[36569]: pgmap v12691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:48 smithi104 ceph-mon[36624]: pgmap v12691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:50 smithi081 ceph-mon[31453]: pgmap v12692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:50 smithi153 ceph-mon[36569]: pgmap v12692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:50 smithi104 ceph-mon[36624]: pgmap v12692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:52 smithi081 ceph-mon[31453]: pgmap v12693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:52 smithi153 ceph-mon[36569]: pgmap v12693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:52 smithi104 ceph-mon[36624]: pgmap v12693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:54 smithi081 ceph-mon[31453]: pgmap v12694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:54 smithi153 ceph-mon[36569]: pgmap v12694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:54 smithi104 ceph-mon[36624]: pgmap v12694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:55.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:55.695+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:47:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:55.695+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:47:55.696 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:47:55.695+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:47:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:56 smithi081 ceph-mon[31453]: pgmap v12695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:56 smithi153 ceph-mon[36569]: pgmap v12695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:56 smithi104 ceph-mon[36624]: pgmap v12695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:47:58 smithi081 ceph-mon[31453]: pgmap v12696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:47:58 smithi153 ceph-mon[36569]: pgmap v12696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:47:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:47:58 smithi104 ceph-mon[36624]: pgmap v12696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:00 smithi081 ceph-mon[31453]: pgmap v12697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:00 smithi153 ceph-mon[36569]: pgmap v12697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:00 smithi104 ceph-mon[36624]: pgmap v12697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:02.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:02 smithi104 ceph-mon[36624]: pgmap v12698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:02 smithi081 ceph-mon[31453]: pgmap v12698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:02 smithi153 ceph-mon[36569]: pgmap v12698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:04 smithi081 ceph-mon[31453]: pgmap v12699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:04 smithi153 ceph-mon[36569]: pgmap v12699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:04 smithi104 ceph-mon[36624]: pgmap v12699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:06 smithi081 ceph-mon[31453]: pgmap v12700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:06 smithi153 ceph-mon[36569]: pgmap v12700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:06 smithi104 ceph-mon[36624]: pgmap v12700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:08 smithi081 ceph-mon[31453]: pgmap v12701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:08 smithi153 ceph-mon[36569]: pgmap v12701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:08 smithi104 ceph-mon[36624]: pgmap v12701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:10 smithi081 ceph-mon[31453]: pgmap v12702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:10 smithi153 ceph-mon[36569]: pgmap v12702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:10 smithi104 ceph-mon[36624]: pgmap v12702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:10.711+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:48:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:10.711+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:48:10.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:10.711+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:48:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:12 smithi081 ceph-mon[31453]: pgmap v12703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:12 smithi153 ceph-mon[36569]: pgmap v12703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:12 smithi104 ceph-mon[36624]: pgmap v12703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:14 smithi153 ceph-mon[36569]: pgmap v12704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:14 smithi104 ceph-mon[36624]: pgmap v12704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:14 smithi081 ceph-mon[31453]: pgmap v12704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:16 smithi153 ceph-mon[36569]: pgmap v12705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:16 smithi104 ceph-mon[36624]: pgmap v12705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:16 smithi081 ceph-mon[31453]: pgmap v12705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:18 smithi153 ceph-mon[36569]: pgmap v12706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:18 smithi104 ceph-mon[36624]: pgmap v12706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:18 smithi081 ceph-mon[31453]: pgmap v12706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:20 smithi153 ceph-mon[36569]: pgmap v12707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:20 smithi104 ceph-mon[36624]: pgmap v12707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:20 smithi081 ceph-mon[31453]: pgmap v12707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:22 smithi153 ceph-mon[36569]: pgmap v12708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:22 smithi104 ceph-mon[36624]: pgmap v12708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:22 smithi081 ceph-mon[31453]: pgmap v12708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:24 smithi153 ceph-mon[36569]: pgmap v12709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:24 smithi104 ceph-mon[36624]: pgmap v12709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:24 smithi081 ceph-mon[31453]: pgmap v12709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:25.715+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:48:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:25.715+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:48:25.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:25.715+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:48:26.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:26 smithi153 ceph-mon[36569]: pgmap v12710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:26 smithi104 ceph-mon[36624]: pgmap v12710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:26 smithi081 ceph-mon[31453]: pgmap v12710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:28 smithi153 ceph-mon[36569]: pgmap v12711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:28 smithi104 ceph-mon[36624]: pgmap v12711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:28 smithi081 ceph-mon[31453]: pgmap v12711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:30 smithi153 ceph-mon[36569]: pgmap v12712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:30 smithi104 ceph-mon[36624]: pgmap v12712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:30 smithi081 ceph-mon[31453]: pgmap v12712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:32.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:32 smithi081 ceph-mon[31453]: pgmap v12713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:32 smithi153 ceph-mon[36569]: pgmap v12713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:32 smithi104 ceph-mon[36624]: pgmap v12713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:34 smithi153 ceph-mon[36569]: pgmap v12714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:48:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:34 smithi104 ceph-mon[36624]: pgmap v12714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:48:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:34 smithi081 ceph-mon[31453]: pgmap v12714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:48:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:48:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:48:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:48:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:48:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:48:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:48:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:48:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:48:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:48:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:36 smithi153 ceph-mon[36569]: pgmap v12715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:36 smithi153 ceph-mon[36569]: pgmap v12716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:36 smithi104 ceph-mon[36624]: pgmap v12715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:36 smithi104 ceph-mon[36624]: pgmap v12716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:36 smithi081 ceph-mon[31453]: pgmap v12715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:36 smithi081 ceph-mon[31453]: pgmap v12716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:38 smithi153 ceph-mon[36569]: pgmap v12717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:38 smithi104 ceph-mon[36624]: pgmap v12717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:38 smithi081 ceph-mon[31453]: pgmap v12717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:40 smithi153 ceph-mon[36569]: pgmap v12718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:40 smithi104 ceph-mon[36624]: pgmap v12718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:40 smithi081 ceph-mon[31453]: pgmap v12718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:40.723+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:48:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:40.723+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:48:40.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:40.723+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:48:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:42 smithi153 ceph-mon[36569]: pgmap v12719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:42 smithi104 ceph-mon[36624]: pgmap v12719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:42 smithi081 ceph-mon[31453]: pgmap v12719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:44 smithi153 ceph-mon[36569]: pgmap v12720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:44 smithi104 ceph-mon[36624]: pgmap v12720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:44 smithi081 ceph-mon[31453]: pgmap v12720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:46 smithi153 ceph-mon[36569]: pgmap v12721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:46 smithi104 ceph-mon[36624]: pgmap v12721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:46 smithi081 ceph-mon[31453]: pgmap v12721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:48 smithi153 ceph-mon[36569]: pgmap v12722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:48 smithi104 ceph-mon[36624]: pgmap v12722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:48 smithi081 ceph-mon[31453]: pgmap v12722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:50 smithi153 ceph-mon[36569]: pgmap v12723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:50 smithi104 ceph-mon[36624]: pgmap v12723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:50 smithi081 ceph-mon[31453]: pgmap v12723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:52 smithi153 ceph-mon[36569]: pgmap v12724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:52 smithi104 ceph-mon[36624]: pgmap v12724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:52 smithi081 ceph-mon[31453]: pgmap v12724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:54 smithi153 ceph-mon[36569]: pgmap v12725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:54 smithi104 ceph-mon[36624]: pgmap v12725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:54 smithi081 ceph-mon[31453]: pgmap v12725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:55.726+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:48:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:55.726+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:48:55.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:48:55.726+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:48:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:56 smithi153 ceph-mon[36569]: pgmap v12726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:56 smithi104 ceph-mon[36624]: pgmap v12726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:56 smithi081 ceph-mon[31453]: pgmap v12726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:48:58 smithi153 ceph-mon[36569]: pgmap v12727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:48:58 smithi104 ceph-mon[36624]: pgmap v12727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:48:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:48:58 smithi081 ceph-mon[31453]: pgmap v12727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:00 smithi153 ceph-mon[36569]: pgmap v12728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:00 smithi104 ceph-mon[36624]: pgmap v12728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:00 smithi081 ceph-mon[31453]: pgmap v12728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:02 smithi153 ceph-mon[36569]: pgmap v12729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:02 smithi104 ceph-mon[36624]: pgmap v12729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:02 smithi081 ceph-mon[31453]: pgmap v12729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:04 smithi153 ceph-mon[36569]: pgmap v12730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:04 smithi104 ceph-mon[36624]: pgmap v12730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:04 smithi081 ceph-mon[31453]: pgmap v12730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:06 smithi104 ceph-mon[36624]: pgmap v12731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:06 smithi081 ceph-mon[31453]: pgmap v12731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:06 smithi153 ceph-mon[36569]: pgmap v12731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:08 smithi104 ceph-mon[36624]: pgmap v12732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:08 smithi081 ceph-mon[31453]: pgmap v12732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:08 smithi153 ceph-mon[36569]: pgmap v12732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:10 smithi104 ceph-mon[36624]: pgmap v12733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:10 smithi081 ceph-mon[31453]: pgmap v12733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:10.741+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:49:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:10.742+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:49:10.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:10.741+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:49:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:10 smithi153 ceph-mon[36569]: pgmap v12733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:12 smithi104 ceph-mon[36624]: pgmap v12734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:12 smithi081 ceph-mon[31453]: pgmap v12734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:12 smithi153 ceph-mon[36569]: pgmap v12734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:14 smithi104 ceph-mon[36624]: pgmap v12735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:14 smithi081 ceph-mon[31453]: pgmap v12735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:14 smithi153 ceph-mon[36569]: pgmap v12735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:16.598+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:49:16.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:16.598+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:49:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:16.598+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 183 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:49:16.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:16.598+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 183 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:49:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:16 smithi081 ceph-mon[31453]: pgmap v12736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:16 smithi153 ceph-mon[36569]: pgmap v12736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:16 smithi104 ceph-mon[36624]: pgmap v12736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:18 smithi081 ceph-mon[31453]: pgmap v12737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:18 smithi153 ceph-mon[36569]: pgmap v12737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:18 smithi104 ceph-mon[36624]: pgmap v12737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:20 smithi081 ceph-mon[31453]: pgmap v12738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:20 smithi153 ceph-mon[36569]: pgmap v12738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:20 smithi104 ceph-mon[36624]: pgmap v12738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:22 smithi081 ceph-mon[31453]: pgmap v12739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:22 smithi153 ceph-mon[36569]: pgmap v12739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:22 smithi104 ceph-mon[36624]: pgmap v12739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:24 smithi081 ceph-mon[31453]: pgmap v12740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:24 smithi153 ceph-mon[36569]: pgmap v12740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:24 smithi104 ceph-mon[36624]: pgmap v12740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:25.743+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:49:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:25.743+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:49:25.743 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:25.743+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:49:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:26 smithi081 ceph-mon[31453]: pgmap v12741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:26 smithi153 ceph-mon[36569]: pgmap v12741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:26 smithi104 ceph-mon[36624]: pgmap v12741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:28 smithi081 ceph-mon[31453]: pgmap v12742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:28 smithi153 ceph-mon[36569]: pgmap v12742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:28 smithi104 ceph-mon[36624]: pgmap v12742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:30 smithi081 ceph-mon[31453]: pgmap v12743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:30 smithi153 ceph-mon[36569]: pgmap v12743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:30 smithi104 ceph-mon[36624]: pgmap v12743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:32.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:32 smithi104 ceph-mon[36624]: pgmap v12744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:32 smithi081 ceph-mon[31453]: pgmap v12744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:32 smithi153 ceph-mon[36569]: pgmap v12744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:34 smithi081 ceph-mon[31453]: pgmap v12745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:34 smithi153 ceph-mon[36569]: pgmap v12745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:34 smithi104 ceph-mon[36624]: pgmap v12745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:49:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:49:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:49:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:36 smithi153 ceph-mon[36569]: pgmap v12746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:49:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:49:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:49:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:36 smithi104 ceph-mon[36624]: pgmap v12746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:49:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:49:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:49:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:36 smithi081 ceph-mon[31453]: pgmap v12746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:49:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:49:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:49:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:37 smithi153 ceph-mon[36569]: pgmap v12747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:37 smithi153 ceph-mon[36569]: pgmap v12748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:37 smithi104 ceph-mon[36624]: pgmap v12747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:37 smithi104 ceph-mon[36624]: pgmap v12748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:37 smithi081 ceph-mon[31453]: pgmap v12747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:37 smithi081 ceph-mon[31453]: pgmap v12748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:39 smithi153 ceph-mon[36569]: pgmap v12749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:39 smithi104 ceph-mon[36624]: pgmap v12749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:39 smithi081 ceph-mon[31453]: pgmap v12749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:40.744+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:49:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:40.745+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:49:40.745 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:40.745+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:49:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:41 smithi153 ceph-mon[36569]: pgmap v12750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:41 smithi104 ceph-mon[36624]: pgmap v12750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:41 smithi081 ceph-mon[31453]: pgmap v12750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:43 smithi153 ceph-mon[36569]: pgmap v12751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:43 smithi104 ceph-mon[36624]: pgmap v12751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:43 smithi081 ceph-mon[31453]: pgmap v12751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:45 smithi153 ceph-mon[36569]: pgmap v12752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:45 smithi104 ceph-mon[36624]: pgmap v12752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:45 smithi081 ceph-mon[31453]: pgmap v12752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:47 smithi153 ceph-mon[36569]: pgmap v12753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:47 smithi104 ceph-mon[36624]: pgmap v12753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:47 smithi081 ceph-mon[31453]: pgmap v12753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:49.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:49 smithi153 ceph-mon[36569]: pgmap v12754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:49 smithi104 ceph-mon[36624]: pgmap v12754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:49 smithi081 ceph-mon[31453]: pgmap v12754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:51 smithi153 ceph-mon[36569]: pgmap v12755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:51 smithi104 ceph-mon[36624]: pgmap v12755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:51 smithi081 ceph-mon[31453]: pgmap v12755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:52 smithi153 ceph-mon[36569]: pgmap v12756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:52 smithi104 ceph-mon[36624]: pgmap v12756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:52 smithi081 ceph-mon[31453]: pgmap v12756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:55 smithi081 ceph-mon[31453]: pgmap v12757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:55.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:55 smithi153 ceph-mon[36569]: pgmap v12757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:55 smithi104 ceph-mon[36624]: pgmap v12757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:55.760+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:49:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:55.760+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:49:55.761 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:49:55.760+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:49:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:57 smithi153 ceph-mon[36569]: pgmap v12758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:57 smithi104 ceph-mon[36624]: pgmap v12758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:57 smithi081 ceph-mon[31453]: pgmap v12758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:49:59 smithi153 ceph-mon[36569]: pgmap v12759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:49:59 smithi104 ceph-mon[36624]: pgmap v12759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:49:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:49:59 smithi081 ceph-mon[31453]: pgmap v12759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:50:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:50:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T03:50:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:01 smithi153 ceph-mon[36569]: pgmap v12760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:01 smithi104 ceph-mon[36624]: pgmap v12760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:01 smithi081 ceph-mon[31453]: pgmap v12760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:03.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:03 smithi081 ceph-mon[31453]: pgmap v12761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:03 smithi153 ceph-mon[36569]: pgmap v12761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:03 smithi104 ceph-mon[36624]: pgmap v12761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:05 smithi153 ceph-mon[36569]: pgmap v12762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:05 smithi104 ceph-mon[36624]: pgmap v12762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:05 smithi081 ceph-mon[31453]: pgmap v12762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:07 smithi153 ceph-mon[36569]: pgmap v12763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:07 smithi104 ceph-mon[36624]: pgmap v12763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:07 smithi081 ceph-mon[31453]: pgmap v12763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:09 smithi153 ceph-mon[36569]: pgmap v12764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:09 smithi104 ceph-mon[36624]: pgmap v12764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:09 smithi081 ceph-mon[31453]: pgmap v12764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:10.773 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:10.773+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:50:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:10.773+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:50:10.774 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:10.773+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:50:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:11 smithi153 ceph-mon[36569]: pgmap v12765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:11 smithi104 ceph-mon[36624]: pgmap v12765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:11 smithi081 ceph-mon[31453]: pgmap v12765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:13 smithi153 ceph-mon[36569]: pgmap v12766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:13 smithi104 ceph-mon[36624]: pgmap v12766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:13 smithi081 ceph-mon[31453]: pgmap v12766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:15 smithi153 ceph-mon[36569]: pgmap v12767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:15 smithi104 ceph-mon[36624]: pgmap v12767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:15 smithi081 ceph-mon[31453]: pgmap v12767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:17 smithi153 ceph-mon[36569]: pgmap v12768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:17 smithi104 ceph-mon[36624]: pgmap v12768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:17 smithi081 ceph-mon[31453]: pgmap v12768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:19 smithi153 ceph-mon[36569]: pgmap v12769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:19 smithi104 ceph-mon[36624]: pgmap v12769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:19 smithi081 ceph-mon[31453]: pgmap v12769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:21 smithi153 ceph-mon[36569]: pgmap v12770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:21 smithi104 ceph-mon[36624]: pgmap v12770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:21 smithi081 ceph-mon[31453]: pgmap v12770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:23 smithi153 ceph-mon[36569]: pgmap v12771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:23 smithi104 ceph-mon[36624]: pgmap v12771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:23 smithi081 ceph-mon[31453]: pgmap v12771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:25 smithi153 ceph-mon[36569]: pgmap v12772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:25 smithi104 ceph-mon[36624]: pgmap v12772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:25 smithi081 ceph-mon[31453]: pgmap v12772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:25.783+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:50:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:25.783+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:50:25.784 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:25.783+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:50:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:27 smithi153 ceph-mon[36569]: pgmap v12773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:27 smithi104 ceph-mon[36624]: pgmap v12773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:27 smithi081 ceph-mon[31453]: pgmap v12773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:29 smithi153 ceph-mon[36569]: pgmap v12774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:29 smithi104 ceph-mon[36624]: pgmap v12774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:29 smithi081 ceph-mon[31453]: pgmap v12774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:31 smithi153 ceph-mon[36569]: pgmap v12775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:31 smithi104 ceph-mon[36624]: pgmap v12775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:31 smithi081 ceph-mon[31453]: pgmap v12775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:33 smithi153 ceph-mon[36569]: pgmap v12776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:33 smithi104 ceph-mon[36624]: pgmap v12776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:33 smithi081 ceph-mon[31453]: pgmap v12776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:35 smithi153 ceph-mon[36569]: pgmap v12777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:35 smithi104 ceph-mon[36624]: pgmap v12777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:35 smithi081 ceph-mon[31453]: pgmap v12777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:37 smithi153 ceph-mon[36569]: pgmap v12778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:50:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:37 smithi104 ceph-mon[36624]: pgmap v12778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:50:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:37 smithi081 ceph-mon[31453]: pgmap v12778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:50:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:50:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:50:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:50:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:50:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:50:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:50:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:50:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:50:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:50:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:39 smithi153 ceph-mon[36569]: pgmap v12779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:39 smithi153 ceph-mon[36569]: pgmap v12780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:39 smithi104 ceph-mon[36624]: pgmap v12779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:39 smithi104 ceph-mon[36624]: pgmap v12780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:39.590 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:39 smithi081 ceph-mon[31453]: pgmap v12779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:39.590 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:39 smithi081 ceph-mon[31453]: pgmap v12780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:40.789+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:50:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:40.789+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:50:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:40.789+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:50:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:41 smithi153 ceph-mon[36569]: pgmap v12781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:41 smithi104 ceph-mon[36624]: pgmap v12781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:41 smithi081 ceph-mon[31453]: pgmap v12781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:43 smithi153 ceph-mon[36569]: pgmap v12782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:43 smithi104 ceph-mon[36624]: pgmap v12782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:43 smithi081 ceph-mon[31453]: pgmap v12782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:45 smithi153 ceph-mon[36569]: pgmap v12783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:45 smithi104 ceph-mon[36624]: pgmap v12783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:45 smithi081 ceph-mon[31453]: pgmap v12783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:47 smithi153 ceph-mon[36569]: pgmap v12784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:47 smithi104 ceph-mon[36624]: pgmap v12784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:47 smithi081 ceph-mon[31453]: pgmap v12784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:49 smithi104 ceph-mon[36624]: pgmap v12785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:49 smithi081 ceph-mon[31453]: pgmap v12785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:49 smithi153 ceph-mon[36569]: pgmap v12785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:51 smithi104 ceph-mon[36624]: pgmap v12786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:51 smithi081 ceph-mon[31453]: pgmap v12786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:51 smithi153 ceph-mon[36569]: pgmap v12786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:53 smithi104 ceph-mon[36624]: pgmap v12787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:53 smithi081 ceph-mon[31453]: pgmap v12787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:53 smithi153 ceph-mon[36569]: pgmap v12787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:55 smithi104 ceph-mon[36624]: pgmap v12788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:55 smithi081 ceph-mon[31453]: pgmap v12788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:55.801+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:50:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:55.801+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:50:55.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:50:55.801+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:50:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:55 smithi153 ceph-mon[36569]: pgmap v12788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:57 smithi104 ceph-mon[36624]: pgmap v12789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:57 smithi081 ceph-mon[31453]: pgmap v12789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:57 smithi153 ceph-mon[36569]: pgmap v12789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:59.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:50:59 smithi153 ceph-mon[36569]: pgmap v12790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:50:59 smithi081 ceph-mon[31453]: pgmap v12790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:50:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:50:59 smithi104 ceph-mon[36624]: pgmap v12790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:01 smithi081 ceph-mon[31453]: pgmap v12791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:01 smithi153 ceph-mon[36569]: pgmap v12791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:01 smithi104 ceph-mon[36624]: pgmap v12791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:03.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:03 smithi104 ceph-mon[36624]: pgmap v12792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:03 smithi081 ceph-mon[31453]: pgmap v12792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:03 smithi153 ceph-mon[36569]: pgmap v12792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:05 smithi081 ceph-mon[31453]: pgmap v12793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:05 smithi153 ceph-mon[36569]: pgmap v12793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:05 smithi104 ceph-mon[36624]: pgmap v12793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:07 smithi081 ceph-mon[31453]: pgmap v12794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:07 smithi153 ceph-mon[36569]: pgmap v12794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:07 smithi104 ceph-mon[36624]: pgmap v12794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:09 smithi081 ceph-mon[31453]: pgmap v12795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:09 smithi153 ceph-mon[36569]: pgmap v12795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:09 smithi104 ceph-mon[36624]: pgmap v12795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:10.811 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:10.810+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:51:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:10.810+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:51:10.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:10.810+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:51:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:11 smithi081 ceph-mon[31453]: pgmap v12796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:11 smithi153 ceph-mon[36569]: pgmap v12796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:11 smithi104 ceph-mon[36624]: pgmap v12796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:13 smithi081 ceph-mon[31453]: pgmap v12797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:13 smithi153 ceph-mon[36569]: pgmap v12797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:13 smithi104 ceph-mon[36624]: pgmap v12797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:15 smithi081 ceph-mon[31453]: pgmap v12798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:15.864 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:15 smithi104 ceph-mon[36624]: pgmap v12798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:15.864 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:15 smithi153 ceph-mon[36569]: pgmap v12798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:17 smithi081 ceph-mon[31453]: pgmap v12799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:17 smithi153 ceph-mon[36569]: pgmap v12799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:17 smithi104 ceph-mon[36624]: pgmap v12799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:19 smithi081 ceph-mon[31453]: pgmap v12800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:19 smithi153 ceph-mon[36569]: pgmap v12800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:19 smithi104 ceph-mon[36624]: pgmap v12800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:21 smithi081 ceph-mon[31453]: pgmap v12801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:21 smithi153 ceph-mon[36569]: pgmap v12801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:21 smithi104 ceph-mon[36624]: pgmap v12801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:23 smithi081 ceph-mon[31453]: pgmap v12802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:23 smithi153 ceph-mon[36569]: pgmap v12802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:23 smithi104 ceph-mon[36624]: pgmap v12802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:25 smithi081 ceph-mon[31453]: pgmap v12803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:25.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:25 smithi153 ceph-mon[36569]: pgmap v12803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:25.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:25.826+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:51:25.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:25.826+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:51:25.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:25.827+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:51:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:25 smithi104 ceph-mon[36624]: pgmap v12803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:27 smithi153 ceph-mon[36569]: pgmap v12804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:27 smithi104 ceph-mon[36624]: pgmap v12804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:27 smithi081 ceph-mon[31453]: pgmap v12804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:29 smithi153 ceph-mon[36569]: pgmap v12805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:29 smithi104 ceph-mon[36624]: pgmap v12805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:29 smithi081 ceph-mon[31453]: pgmap v12805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:31 smithi153 ceph-mon[36569]: pgmap v12806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:31 smithi104 ceph-mon[36624]: pgmap v12806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:31 smithi081 ceph-mon[31453]: pgmap v12806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:33.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:33 smithi081 ceph-mon[31453]: pgmap v12807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:33 smithi153 ceph-mon[36569]: pgmap v12807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:33 smithi104 ceph-mon[36624]: pgmap v12807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:35 smithi153 ceph-mon[36569]: pgmap v12808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:35 smithi104 ceph-mon[36624]: pgmap v12808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:35 smithi081 ceph-mon[31453]: pgmap v12808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:37 smithi153 ceph-mon[36569]: pgmap v12809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:37 smithi104 ceph-mon[36624]: pgmap v12809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:37 smithi081 ceph-mon[31453]: pgmap v12809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:51:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:51:38.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:51:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:39 smithi153 ceph-mon[36569]: pgmap v12810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:39 smithi104 ceph-mon[36624]: pgmap v12810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:39 smithi081 ceph-mon[31453]: pgmap v12810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:40.842+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:51:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:40.842+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:51:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:40.842+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:51:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:41 smithi153 ceph-mon[36569]: pgmap v12811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:41 smithi104 ceph-mon[36624]: pgmap v12811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:41 smithi081 ceph-mon[31453]: pgmap v12811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:43 smithi153 ceph-mon[36569]: pgmap v12812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:43 smithi104 ceph-mon[36624]: pgmap v12812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:43 smithi081 ceph-mon[31453]: pgmap v12812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:45 smithi153 ceph-mon[36569]: pgmap v12813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:51:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:51:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:45 smithi104 ceph-mon[36624]: pgmap v12813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:51:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:51:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:45 smithi081 ceph-mon[31453]: pgmap v12813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:51:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:51:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:51:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:46 smithi153 ceph-mon[36569]: pgmap v12814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:46 smithi153 ceph-mon[36569]: pgmap v12815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:46 smithi104 ceph-mon[36624]: pgmap v12814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:46 smithi104 ceph-mon[36624]: pgmap v12815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:46 smithi081 ceph-mon[31453]: pgmap v12814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:46 smithi081 ceph-mon[31453]: pgmap v12815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:47 smithi153 ceph-mon[36569]: pgmap v12816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:47 smithi104 ceph-mon[36624]: pgmap v12816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:47 smithi081 ceph-mon[31453]: pgmap v12816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:49 smithi153 ceph-mon[36569]: pgmap v12817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:49 smithi104 ceph-mon[36624]: pgmap v12817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:49 smithi081 ceph-mon[31453]: pgmap v12817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:51 smithi153 ceph-mon[36569]: pgmap v12818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:51 smithi104 ceph-mon[36624]: pgmap v12818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:51 smithi081 ceph-mon[31453]: pgmap v12818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:53 smithi153 ceph-mon[36569]: pgmap v12819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:53 smithi104 ceph-mon[36624]: pgmap v12819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:53 smithi081 ceph-mon[31453]: pgmap v12819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:55 smithi153 ceph-mon[36569]: pgmap v12820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:55 smithi104 ceph-mon[36624]: pgmap v12820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:55.844+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:51:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:55.844+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:51:55.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:51:55.844+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:51:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:55 smithi081 ceph-mon[31453]: pgmap v12820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:57 smithi153 ceph-mon[36569]: pgmap v12821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:57 smithi104 ceph-mon[36624]: pgmap v12821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:57 smithi081 ceph-mon[31453]: pgmap v12821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:51:59 smithi153 ceph-mon[36569]: pgmap v12822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:51:59 smithi104 ceph-mon[36624]: pgmap v12822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:51:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:51:59 smithi081 ceph-mon[31453]: pgmap v12822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:01 smithi153 ceph-mon[36569]: pgmap v12823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:01 smithi104 ceph-mon[36624]: pgmap v12823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:01 smithi081 ceph-mon[31453]: pgmap v12823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:03 smithi153 ceph-mon[36569]: pgmap v12824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:03 smithi104 ceph-mon[36624]: pgmap v12824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:03 smithi081 ceph-mon[31453]: pgmap v12824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:05 smithi153 ceph-mon[36569]: pgmap v12825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:05 smithi104 ceph-mon[36624]: pgmap v12825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:05 smithi081 ceph-mon[31453]: pgmap v12825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:07 smithi153 ceph-mon[36569]: pgmap v12826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:07 smithi104 ceph-mon[36624]: pgmap v12826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:07 smithi081 ceph-mon[31453]: pgmap v12826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:09 smithi153 ceph-mon[36569]: pgmap v12827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:09 smithi104 ceph-mon[36624]: pgmap v12827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:09 smithi081 ceph-mon[31453]: pgmap v12827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:10.851+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:52:10.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:10.851+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:52:10.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:10.851+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:52:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:11 smithi153 ceph-mon[36569]: pgmap v12828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:11 smithi104 ceph-mon[36624]: pgmap v12828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:11 smithi081 ceph-mon[31453]: pgmap v12828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:13 smithi153 ceph-mon[36569]: pgmap v12829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:13 smithi104 ceph-mon[36624]: pgmap v12829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:13 smithi081 ceph-mon[31453]: pgmap v12829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:15 smithi104 ceph-mon[36624]: pgmap v12830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:15 smithi081 ceph-mon[31453]: pgmap v12830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:15 smithi153 ceph-mon[36569]: pgmap v12830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:16.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:16.627+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:52:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:16.627+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:52:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:16.628+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 184 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T03:52:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:16.628+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 184 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T03:52:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:16.628+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:52:16.628 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:16.628+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:52:16.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:16.628+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 185 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:52:16.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:16.629+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 185 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:52:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:17 smithi104 ceph-mon[36624]: pgmap v12831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:17 smithi081 ceph-mon[31453]: pgmap v12831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:17 smithi153 ceph-mon[36569]: pgmap v12831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:19 smithi104 ceph-mon[36624]: pgmap v12832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:19 smithi081 ceph-mon[31453]: pgmap v12832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:19 smithi153 ceph-mon[36569]: pgmap v12832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:21 smithi104 ceph-mon[36624]: pgmap v12833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:21 smithi081 ceph-mon[31453]: pgmap v12833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:21 smithi153 ceph-mon[36569]: pgmap v12833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:23.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:23 smithi081 ceph-mon[31453]: pgmap v12834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:23 smithi153 ceph-mon[36569]: pgmap v12834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:24.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:23 smithi104 ceph-mon[36624]: pgmap v12834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:25.868+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:52:25.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:25.868+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:52:25.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:25.868+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:52:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:25 smithi081 ceph-mon[31453]: pgmap v12835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:25 smithi153 ceph-mon[36569]: pgmap v12835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:25 smithi104 ceph-mon[36624]: pgmap v12835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:28 smithi081 ceph-mon[31453]: pgmap v12836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:28 smithi153 ceph-mon[36569]: pgmap v12836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:28 smithi104 ceph-mon[36624]: pgmap v12836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:30 smithi081 ceph-mon[31453]: pgmap v12837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:30 smithi153 ceph-mon[36569]: pgmap v12837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:30 smithi104 ceph-mon[36624]: pgmap v12837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:32 smithi081 ceph-mon[31453]: pgmap v12838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:32 smithi153 ceph-mon[36569]: pgmap v12838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:32 smithi104 ceph-mon[36624]: pgmap v12838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:34.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:34 smithi104 ceph-mon[36624]: pgmap v12839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:34 smithi081 ceph-mon[31453]: pgmap v12839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:34 smithi153 ceph-mon[36569]: pgmap v12839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:36 smithi081 ceph-mon[31453]: pgmap v12840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:36 smithi153 ceph-mon[36569]: pgmap v12840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:36 smithi104 ceph-mon[36624]: pgmap v12840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:38 smithi081 ceph-mon[31453]: pgmap v12841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:38 smithi153 ceph-mon[36569]: pgmap v12841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:38 smithi104 ceph-mon[36624]: pgmap v12841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:40 smithi081 ceph-mon[31453]: pgmap v12842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:40 smithi153 ceph-mon[36569]: pgmap v12842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:40 smithi104 ceph-mon[36624]: pgmap v12842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:40.876+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:52:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:40.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:52:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:40.876+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:52:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:42 smithi081 ceph-mon[31453]: pgmap v12843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:42 smithi153 ceph-mon[36569]: pgmap v12843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:42 smithi104 ceph-mon[36624]: pgmap v12843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:44 smithi153 ceph-mon[36569]: pgmap v12844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:44 smithi104 ceph-mon[36624]: pgmap v12844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:44 smithi081 ceph-mon[31453]: pgmap v12844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:52:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:52:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:52:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:46 smithi104 ceph-mon[36624]: pgmap v12845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:52:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:52:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:46 smithi081 ceph-mon[31453]: pgmap v12845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:52:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:52:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:46 smithi153 ceph-mon[36569]: pgmap v12845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:52:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:52:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:52:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:47 smithi081 ceph-mon[31453]: pgmap v12846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:47 smithi081 ceph-mon[31453]: pgmap v12847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:47.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:47 smithi153 ceph-mon[36569]: pgmap v12846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:47.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:47 smithi153 ceph-mon[36569]: pgmap v12847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:47 smithi104 ceph-mon[36624]: pgmap v12846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:47 smithi104 ceph-mon[36624]: pgmap v12847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:49 smithi081 ceph-mon[31453]: pgmap v12848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:49 smithi153 ceph-mon[36569]: pgmap v12848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:49 smithi104 ceph-mon[36624]: pgmap v12848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:51 smithi081 ceph-mon[31453]: pgmap v12849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:51 smithi153 ceph-mon[36569]: pgmap v12849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:51 smithi104 ceph-mon[36624]: pgmap v12849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:53 smithi081 ceph-mon[31453]: pgmap v12850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:53 smithi153 ceph-mon[36569]: pgmap v12850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:53 smithi104 ceph-mon[36624]: pgmap v12850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:55 smithi081 ceph-mon[31453]: pgmap v12851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:55 smithi153 ceph-mon[36569]: pgmap v12851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:55 smithi104 ceph-mon[36624]: pgmap v12851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:55.889+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:52:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:55.889+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:52:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:52:55.889+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:52:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:57 smithi081 ceph-mon[31453]: pgmap v12852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:57 smithi153 ceph-mon[36569]: pgmap v12852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:57 smithi104 ceph-mon[36624]: pgmap v12852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:52:59 smithi081 ceph-mon[31453]: pgmap v12853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:52:59 smithi153 ceph-mon[36569]: pgmap v12853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:52:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:52:59 smithi104 ceph-mon[36624]: pgmap v12853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:01 smithi081 ceph-mon[31453]: pgmap v12854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:01 smithi153 ceph-mon[36569]: pgmap v12854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:01 smithi104 ceph-mon[36624]: pgmap v12854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:03 smithi081 ceph-mon[31453]: pgmap v12855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:03 smithi153 ceph-mon[36569]: pgmap v12855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:03 smithi104 ceph-mon[36624]: pgmap v12855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:05.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:05 smithi104 ceph-mon[36624]: pgmap v12856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:05 smithi081 ceph-mon[31453]: pgmap v12856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:05 smithi153 ceph-mon[36569]: pgmap v12856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:07 smithi081 ceph-mon[31453]: pgmap v12857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:07.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:07 smithi104 ceph-mon[36624]: pgmap v12857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:07 smithi153 ceph-mon[36569]: pgmap v12857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:09 smithi081 ceph-mon[31453]: pgmap v12858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:09 smithi153 ceph-mon[36569]: pgmap v12858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:09 smithi104 ceph-mon[36624]: pgmap v12858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:10.891+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:53:10.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:10.891+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:53:10.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:10.891+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:53:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:11 smithi081 ceph-mon[31453]: pgmap v12859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:11 smithi153 ceph-mon[36569]: pgmap v12859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:11 smithi104 ceph-mon[36624]: pgmap v12859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:13 smithi081 ceph-mon[31453]: pgmap v12860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:13 smithi153 ceph-mon[36569]: pgmap v12860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:13 smithi104 ceph-mon[36624]: pgmap v12860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:15 smithi153 ceph-mon[36569]: pgmap v12861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:15 smithi104 ceph-mon[36624]: pgmap v12861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:15 smithi081 ceph-mon[31453]: pgmap v12861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:17 smithi153 ceph-mon[36569]: pgmap v12862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:17 smithi104 ceph-mon[36624]: pgmap v12862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:17 smithi081 ceph-mon[31453]: pgmap v12862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:19 smithi153 ceph-mon[36569]: pgmap v12863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:19 smithi104 ceph-mon[36624]: pgmap v12863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:19 smithi081 ceph-mon[31453]: pgmap v12863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:21 smithi153 ceph-mon[36569]: pgmap v12864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:21 smithi104 ceph-mon[36624]: pgmap v12864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:21 smithi081 ceph-mon[31453]: pgmap v12864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:23 smithi153 ceph-mon[36569]: pgmap v12865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:23 smithi104 ceph-mon[36624]: pgmap v12865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:23 smithi081 ceph-mon[31453]: pgmap v12865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:25 smithi153 ceph-mon[36569]: pgmap v12866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:25 smithi104 ceph-mon[36624]: pgmap v12866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:25.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:25.905+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:53:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:25 smithi081 ceph-mon[31453]: pgmap v12866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:25.906+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:53:25.907 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:25.906+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:53:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:27 smithi153 ceph-mon[36569]: pgmap v12867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:27 smithi104 ceph-mon[36624]: pgmap v12867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:27 smithi081 ceph-mon[31453]: pgmap v12867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:29 smithi153 ceph-mon[36569]: pgmap v12868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:29 smithi104 ceph-mon[36624]: pgmap v12868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:29 smithi081 ceph-mon[31453]: pgmap v12868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:31 smithi153 ceph-mon[36569]: pgmap v12869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:31 smithi104 ceph-mon[36624]: pgmap v12869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:31 smithi081 ceph-mon[31453]: pgmap v12869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:33 smithi153 ceph-mon[36569]: pgmap v12870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:33 smithi104 ceph-mon[36624]: pgmap v12870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:33 smithi081 ceph-mon[31453]: pgmap v12870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:35.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:35 smithi081 ceph-mon[31453]: pgmap v12871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:35 smithi153 ceph-mon[36569]: pgmap v12871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:35 smithi104 ceph-mon[36624]: pgmap v12871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:37 smithi153 ceph-mon[36569]: pgmap v12872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:37 smithi104 ceph-mon[36624]: pgmap v12872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:37 smithi081 ceph-mon[31453]: pgmap v12872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:39 smithi153 ceph-mon[36569]: pgmap v12873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:39 smithi104 ceph-mon[36624]: pgmap v12873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:39 smithi081 ceph-mon[31453]: pgmap v12873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:40.922+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:53:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:40.922+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:53:40.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:40.922+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:53:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:41 smithi153 ceph-mon[36569]: pgmap v12874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:41 smithi104 ceph-mon[36624]: pgmap v12874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:41 smithi081 ceph-mon[31453]: pgmap v12874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:43 smithi153 ceph-mon[36569]: pgmap v12875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:43 smithi104 ceph-mon[36624]: pgmap v12875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:43 smithi081 ceph-mon[31453]: pgmap v12875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:45 smithi153 ceph-mon[36569]: pgmap v12876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:45 smithi104 ceph-mon[36624]: pgmap v12876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:45 smithi081 ceph-mon[31453]: pgmap v12876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:53:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:53:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:53:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:47 smithi153 ceph-mon[36569]: pgmap v12877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:47 smithi104 ceph-mon[36624]: pgmap v12877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:47 smithi081 ceph-mon[31453]: pgmap v12877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:53:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:53:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:48 smithi153 ceph-mon[36569]: pgmap v12878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:48 smithi153 ceph-mon[36569]: pgmap v12879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:53:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:53:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:53:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:48 smithi104 ceph-mon[36624]: pgmap v12878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:48 smithi104 ceph-mon[36624]: pgmap v12879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:53:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:53:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:53:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:48 smithi081 ceph-mon[31453]: pgmap v12878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:48 smithi081 ceph-mon[31453]: pgmap v12879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:53:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:50 smithi104 ceph-mon[36624]: pgmap v12880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:50 smithi081 ceph-mon[31453]: pgmap v12880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:50 smithi153 ceph-mon[36569]: pgmap v12880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:52 smithi104 ceph-mon[36624]: pgmap v12881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:52 smithi081 ceph-mon[31453]: pgmap v12881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:52 smithi153 ceph-mon[36569]: pgmap v12881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:54 smithi081 ceph-mon[31453]: pgmap v12882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:54 smithi153 ceph-mon[36569]: pgmap v12882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:54 smithi104 ceph-mon[36624]: pgmap v12882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:55.937+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:53:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:55.937+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:53:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:53:55.937+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:53:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:56 smithi081 ceph-mon[31453]: pgmap v12883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:56 smithi153 ceph-mon[36569]: pgmap v12883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:56 smithi104 ceph-mon[36624]: pgmap v12883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:53:58 smithi081 ceph-mon[31453]: pgmap v12884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:53:58 smithi153 ceph-mon[36569]: pgmap v12884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:53:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:53:58 smithi104 ceph-mon[36624]: pgmap v12884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:01.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:00 smithi153 ceph-mon[36569]: pgmap v12885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:00 smithi081 ceph-mon[31453]: pgmap v12885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:00 smithi104 ceph-mon[36624]: pgmap v12885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:02 smithi081 ceph-mon[31453]: pgmap v12886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:02 smithi153 ceph-mon[36569]: pgmap v12886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:02 smithi104 ceph-mon[36624]: pgmap v12886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:05.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:04 smithi104 ceph-mon[36624]: pgmap v12887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:04 smithi081 ceph-mon[31453]: pgmap v12887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:04 smithi153 ceph-mon[36569]: pgmap v12887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:06 smithi081 ceph-mon[31453]: pgmap v12888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:06 smithi153 ceph-mon[36569]: pgmap v12888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:06 smithi104 ceph-mon[36624]: pgmap v12888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:08 smithi081 ceph-mon[31453]: pgmap v12889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:08 smithi153 ceph-mon[36569]: pgmap v12889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:08 smithi104 ceph-mon[36624]: pgmap v12889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:10.945+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:54:10.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:10.945+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:54:10.946 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:10.945+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:54:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:10 smithi081 ceph-mon[31453]: pgmap v12890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:10 smithi153 ceph-mon[36569]: pgmap v12890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:10 smithi104 ceph-mon[36624]: pgmap v12890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:12 smithi081 ceph-mon[31453]: pgmap v12891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:12 smithi153 ceph-mon[36569]: pgmap v12891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:12 smithi104 ceph-mon[36624]: pgmap v12891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:14 smithi081 ceph-mon[31453]: pgmap v12892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:14 smithi153 ceph-mon[36569]: pgmap v12892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:14 smithi104 ceph-mon[36624]: pgmap v12892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:16 smithi081 ceph-mon[31453]: pgmap v12893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:16 smithi153 ceph-mon[36569]: pgmap v12893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:16 smithi104 ceph-mon[36624]: pgmap v12893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:18 smithi081 ceph-mon[31453]: pgmap v12894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:18 smithi153 ceph-mon[36569]: pgmap v12894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:18 smithi104 ceph-mon[36624]: pgmap v12894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:20 smithi153 ceph-mon[36569]: pgmap v12895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:20 smithi104 ceph-mon[36624]: pgmap v12895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:20 smithi081 ceph-mon[31453]: pgmap v12895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:22 smithi153 ceph-mon[36569]: pgmap v12896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:22 smithi104 ceph-mon[36624]: pgmap v12896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:22 smithi081 ceph-mon[31453]: pgmap v12896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:24 smithi153 ceph-mon[36569]: pgmap v12897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:24 smithi104 ceph-mon[36624]: pgmap v12897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:24 smithi081 ceph-mon[31453]: pgmap v12897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:25.950+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:54:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:25.951+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:54:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:25.951+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:54:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:26 smithi153 ceph-mon[36569]: pgmap v12898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:26 smithi104 ceph-mon[36624]: pgmap v12898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:26 smithi081 ceph-mon[31453]: pgmap v12898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:28 smithi153 ceph-mon[36569]: pgmap v12899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:28 smithi104 ceph-mon[36624]: pgmap v12899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:28 smithi081 ceph-mon[31453]: pgmap v12899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:30 smithi153 ceph-mon[36569]: pgmap v12900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:30 smithi104 ceph-mon[36624]: pgmap v12900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:30 smithi081 ceph-mon[31453]: pgmap v12900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:32 smithi153 ceph-mon[36569]: pgmap v12901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:32 smithi104 ceph-mon[36624]: pgmap v12901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:32 smithi081 ceph-mon[31453]: pgmap v12901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:34 smithi153 ceph-mon[36569]: pgmap v12902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:34 smithi104 ceph-mon[36624]: pgmap v12902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:34 smithi081 ceph-mon[31453]: pgmap v12902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:37.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:36 smithi081 ceph-mon[31453]: pgmap v12903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:36 smithi153 ceph-mon[36569]: pgmap v12903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:36 smithi104 ceph-mon[36624]: pgmap v12903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:38 smithi153 ceph-mon[36569]: pgmap v12904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:38 smithi104 ceph-mon[36624]: pgmap v12904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:38 smithi081 ceph-mon[31453]: pgmap v12904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:40.951+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:54:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:40.951+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:54:40.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:40.951+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:54:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:40 smithi153 ceph-mon[36569]: pgmap v12905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:40 smithi104 ceph-mon[36624]: pgmap v12905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:40 smithi081 ceph-mon[31453]: pgmap v12905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:42 smithi153 ceph-mon[36569]: pgmap v12906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:42 smithi104 ceph-mon[36624]: pgmap v12906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:42 smithi081 ceph-mon[31453]: pgmap v12906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:44 smithi153 ceph-mon[36569]: pgmap v12907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:44 smithi104 ceph-mon[36624]: pgmap v12907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:44 smithi081 ceph-mon[31453]: pgmap v12907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:46 smithi153 ceph-mon[36569]: pgmap v12908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:46 smithi104 ceph-mon[36624]: pgmap v12908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:46 smithi081 ceph-mon[31453]: pgmap v12908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:48 smithi153 ceph-mon[36569]: pgmap v12909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:54:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:48 smithi104 ceph-mon[36624]: pgmap v12909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:54:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:48 smithi081 ceph-mon[31453]: pgmap v12909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:54:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:54:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:54:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:49 smithi153 ceph-mon[36569]: pgmap v12910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:49 smithi153 ceph-mon[36569]: pgmap v12911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:50.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:54:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:54:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:54:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:49 smithi104 ceph-mon[36624]: pgmap v12910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:49 smithi104 ceph-mon[36624]: pgmap v12911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:54:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:54:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:54:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:49 smithi081 ceph-mon[31453]: pgmap v12910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:49 smithi081 ceph-mon[31453]: pgmap v12911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:54:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:52 smithi153 ceph-mon[36569]: pgmap v12912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:52 smithi104 ceph-mon[36624]: pgmap v12912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:52 smithi081 ceph-mon[31453]: pgmap v12912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:54 smithi153 ceph-mon[36569]: pgmap v12913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:54 smithi104 ceph-mon[36624]: pgmap v12913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:54 smithi081 ceph-mon[31453]: pgmap v12913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:55.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:54:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:55.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:54:55.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:54:55.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:54:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:56 smithi153 ceph-mon[36569]: pgmap v12914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:56 smithi104 ceph-mon[36624]: pgmap v12914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:56 smithi081 ceph-mon[31453]: pgmap v12914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:54:58 smithi153 ceph-mon[36569]: pgmap v12915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:54:58 smithi104 ceph-mon[36624]: pgmap v12915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:54:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:54:58 smithi081 ceph-mon[31453]: pgmap v12915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:00 smithi153 ceph-mon[36569]: pgmap v12916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:00 smithi104 ceph-mon[36624]: pgmap v12916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:00 smithi081 ceph-mon[31453]: pgmap v12916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:02 smithi153 ceph-mon[36569]: pgmap v12917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:02 smithi104 ceph-mon[36624]: pgmap v12917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:02 smithi081 ceph-mon[31453]: pgmap v12917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:04 smithi153 ceph-mon[36569]: pgmap v12918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:04 smithi104 ceph-mon[36624]: pgmap v12918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:04 smithi081 ceph-mon[31453]: pgmap v12918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:06 smithi153 ceph-mon[36569]: pgmap v12919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:06 smithi104 ceph-mon[36624]: pgmap v12919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:06 smithi081 ceph-mon[31453]: pgmap v12919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:08 smithi153 ceph-mon[36569]: pgmap v12920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:08 smithi104 ceph-mon[36624]: pgmap v12920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:08 smithi081 ceph-mon[31453]: pgmap v12920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:10 smithi153 ceph-mon[36569]: pgmap v12921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:10 smithi104 ceph-mon[36624]: pgmap v12921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:10 smithi081 ceph-mon[31453]: pgmap v12921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:10.972+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:55:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:10.972+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:55:10.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:10.972+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:55:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:12 smithi153 ceph-mon[36569]: pgmap v12922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:12 smithi104 ceph-mon[36624]: pgmap v12922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:12 smithi081 ceph-mon[31453]: pgmap v12922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:14 smithi153 ceph-mon[36569]: pgmap v12923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:14 smithi104 ceph-mon[36624]: pgmap v12923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:14 smithi081 ceph-mon[31453]: pgmap v12923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:16.656+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:55:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:16.656+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:55:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:16.657+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 186 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:55:16.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:16.657+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 186 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:55:16.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:16 smithi153 ceph-mon[36569]: pgmap v12924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:16 smithi104 ceph-mon[36624]: pgmap v12924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:16 smithi081 ceph-mon[31453]: pgmap v12924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:18 smithi153 ceph-mon[36569]: pgmap v12925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:18 smithi104 ceph-mon[36624]: pgmap v12925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:18 smithi081 ceph-mon[31453]: pgmap v12925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:20 smithi153 ceph-mon[36569]: pgmap v12926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:20 smithi104 ceph-mon[36624]: pgmap v12926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:20 smithi081 ceph-mon[31453]: pgmap v12926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:22 smithi153 ceph-mon[36569]: pgmap v12927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:22 smithi104 ceph-mon[36624]: pgmap v12927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:22 smithi081 ceph-mon[31453]: pgmap v12927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:24 smithi153 ceph-mon[36569]: pgmap v12928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:24 smithi104 ceph-mon[36624]: pgmap v12928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:24 smithi081 ceph-mon[31453]: pgmap v12928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:25.987+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:55:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:25.987+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:55:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:25.987+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:55:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:26 smithi153 ceph-mon[36569]: pgmap v12929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:26.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:26 smithi104 ceph-mon[36624]: pgmap v12929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:26 smithi081 ceph-mon[31453]: pgmap v12929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:28 smithi153 ceph-mon[36569]: pgmap v12930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:28 smithi104 ceph-mon[36624]: pgmap v12930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:28 smithi081 ceph-mon[31453]: pgmap v12930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:30 smithi153 ceph-mon[36569]: pgmap v12931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:30 smithi104 ceph-mon[36624]: pgmap v12931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:30 smithi081 ceph-mon[31453]: pgmap v12931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:32 smithi153 ceph-mon[36569]: pgmap v12932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:32 smithi104 ceph-mon[36624]: pgmap v12932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:32.872 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:32 smithi081 ceph-mon[31453]: pgmap v12932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:34 smithi153 ceph-mon[36569]: pgmap v12933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:34 smithi104 ceph-mon[36624]: pgmap v12933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:34 smithi081 ceph-mon[31453]: pgmap v12933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:36 smithi153 ceph-mon[36569]: pgmap v12934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:36 smithi104 ceph-mon[36624]: pgmap v12934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:36 smithi081 ceph-mon[31453]: pgmap v12934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:38 smithi153 ceph-mon[36569]: pgmap v12935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:38 smithi104 ceph-mon[36624]: pgmap v12935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:38 smithi081 ceph-mon[31453]: pgmap v12935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:39 smithi153 ceph-mon[36569]: pgmap v12936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:39 smithi104 ceph-mon[36624]: pgmap v12936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:39 smithi081 ceph-mon[31453]: pgmap v12936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:41.002+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:55:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:41.002+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:55:41.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:41.002+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:55:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:42 smithi153 ceph-mon[36569]: pgmap v12937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:42 smithi104 ceph-mon[36624]: pgmap v12937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:42 smithi081 ceph-mon[31453]: pgmap v12937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:44 smithi153 ceph-mon[36569]: pgmap v12938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:44 smithi104 ceph-mon[36624]: pgmap v12938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:44 smithi081 ceph-mon[31453]: pgmap v12938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:46 smithi153 ceph-mon[36569]: pgmap v12939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:46 smithi104 ceph-mon[36624]: pgmap v12939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:46 smithi081 ceph-mon[31453]: pgmap v12939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:48 smithi081 ceph-mon[31453]: pgmap v12940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:48 smithi153 ceph-mon[36569]: pgmap v12940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:48 smithi104 ceph-mon[36624]: pgmap v12940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:50 smithi153 ceph-mon[36569]: pgmap v12941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:55:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:50 smithi104 ceph-mon[36624]: pgmap v12941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:55:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:50 smithi081 ceph-mon[31453]: pgmap v12941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:55:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:55:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:55:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:55:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:55:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:55:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:55:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:55:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:55:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:55:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:52 smithi153 ceph-mon[36569]: pgmap v12942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:52 smithi153 ceph-mon[36569]: pgmap v12943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:52 smithi104 ceph-mon[36624]: pgmap v12942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:52 smithi104 ceph-mon[36624]: pgmap v12943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:52 smithi081 ceph-mon[31453]: pgmap v12942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:52 smithi081 ceph-mon[31453]: pgmap v12943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:54 smithi153 ceph-mon[36569]: pgmap v12944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:54 smithi104 ceph-mon[36624]: pgmap v12944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:54 smithi081 ceph-mon[31453]: pgmap v12944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:56.017+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:55:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:56.017+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:55:56.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:55:56.017+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:55:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:56 smithi153 ceph-mon[36569]: pgmap v12945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:56 smithi104 ceph-mon[36624]: pgmap v12945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:56 smithi081 ceph-mon[31453]: pgmap v12945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:58 smithi153 ceph-mon[36569]: pgmap v12946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:58 smithi104 ceph-mon[36624]: pgmap v12946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:58 smithi081 ceph-mon[31453]: pgmap v12946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:55:59 smithi153 ceph-mon[36569]: pgmap v12947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:55:59 smithi104 ceph-mon[36624]: pgmap v12947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:55:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:55:59 smithi081 ceph-mon[31453]: pgmap v12947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:02 smithi153 ceph-mon[36569]: pgmap v12948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:02 smithi104 ceph-mon[36624]: pgmap v12948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:02 smithi081 ceph-mon[31453]: pgmap v12948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:04 smithi104 ceph-mon[36624]: pgmap v12949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:04 smithi081 ceph-mon[31453]: pgmap v12949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:04 smithi153 ceph-mon[36569]: pgmap v12949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:06 smithi104 ceph-mon[36624]: pgmap v12950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:06 smithi081 ceph-mon[31453]: pgmap v12950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:06 smithi153 ceph-mon[36569]: pgmap v12950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:08 smithi104 ceph-mon[36624]: pgmap v12951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:08 smithi081 ceph-mon[31453]: pgmap v12951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:08 smithi153 ceph-mon[36569]: pgmap v12951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:10 smithi104 ceph-mon[36624]: pgmap v12952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:10 smithi081 ceph-mon[31453]: pgmap v12952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:10 smithi153 ceph-mon[36569]: pgmap v12952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:11.025+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:56:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:11.025+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:56:11.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:11.025+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:56:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:12 smithi104 ceph-mon[36624]: pgmap v12953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:12 smithi081 ceph-mon[31453]: pgmap v12953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:12 smithi153 ceph-mon[36569]: pgmap v12953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:14 smithi081 ceph-mon[31453]: pgmap v12954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:14 smithi153 ceph-mon[36569]: pgmap v12954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:14 smithi104 ceph-mon[36624]: pgmap v12954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:16 smithi081 ceph-mon[31453]: pgmap v12955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:16 smithi153 ceph-mon[36569]: pgmap v12955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:16.692 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:16 smithi104 ceph-mon[36624]: pgmap v12955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:18 smithi081 ceph-mon[31453]: pgmap v12956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:18 smithi153 ceph-mon[36569]: pgmap v12956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:18 smithi104 ceph-mon[36624]: pgmap v12956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:20 smithi081 ceph-mon[31453]: pgmap v12957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:20 smithi153 ceph-mon[36569]: pgmap v12957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:20 smithi104 ceph-mon[36624]: pgmap v12957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:22 smithi081 ceph-mon[31453]: pgmap v12958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:22 smithi153 ceph-mon[36569]: pgmap v12958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:22 smithi104 ceph-mon[36624]: pgmap v12958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:24 smithi081 ceph-mon[31453]: pgmap v12959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:24 smithi153 ceph-mon[36569]: pgmap v12959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:24 smithi104 ceph-mon[36624]: pgmap v12959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:26.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:26.033+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:56:26.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:26.033+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:56:26.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:26.033+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:56:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:26 smithi081 ceph-mon[31453]: pgmap v12960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:26 smithi153 ceph-mon[36569]: pgmap v12960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:26 smithi104 ceph-mon[36624]: pgmap v12960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:28 smithi153 ceph-mon[36569]: pgmap v12961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:28 smithi104 ceph-mon[36624]: pgmap v12961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:28 smithi081 ceph-mon[31453]: pgmap v12961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:30 smithi153 ceph-mon[36569]: pgmap v12962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:30 smithi104 ceph-mon[36624]: pgmap v12962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:30 smithi081 ceph-mon[31453]: pgmap v12962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:32 smithi153 ceph-mon[36569]: pgmap v12963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:32 smithi104 ceph-mon[36624]: pgmap v12963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:32 smithi081 ceph-mon[31453]: pgmap v12963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:34 smithi153 ceph-mon[36569]: pgmap v12964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:34 smithi104 ceph-mon[36624]: pgmap v12964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:34 smithi081 ceph-mon[31453]: pgmap v12964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:36 smithi153 ceph-mon[36569]: pgmap v12965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:36 smithi104 ceph-mon[36624]: pgmap v12965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:36 smithi081 ceph-mon[31453]: pgmap v12965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:38 smithi153 ceph-mon[36569]: pgmap v12966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:38 smithi104 ceph-mon[36624]: pgmap v12966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:38 smithi081 ceph-mon[31453]: pgmap v12966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:40 smithi153 ceph-mon[36569]: pgmap v12967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:40 smithi104 ceph-mon[36624]: pgmap v12967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:40 smithi081 ceph-mon[31453]: pgmap v12967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:41.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:41.048+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:56:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:41.048+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:56:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:41.048+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:56:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:42 smithi153 ceph-mon[36569]: pgmap v12968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:42 smithi104 ceph-mon[36624]: pgmap v12968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:42 smithi081 ceph-mon[31453]: pgmap v12968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:44 smithi153 ceph-mon[36569]: pgmap v12969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:44 smithi104 ceph-mon[36624]: pgmap v12969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:44 smithi081 ceph-mon[31453]: pgmap v12969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:46 smithi153 ceph-mon[36569]: pgmap v12970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:46 smithi104 ceph-mon[36624]: pgmap v12970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:46 smithi081 ceph-mon[31453]: pgmap v12970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:48 smithi153 ceph-mon[36569]: pgmap v12971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:48 smithi104 ceph-mon[36624]: pgmap v12971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:48 smithi081 ceph-mon[31453]: pgmap v12971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:50 smithi153 ceph-mon[36569]: pgmap v12972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:50 smithi104 ceph-mon[36624]: pgmap v12972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:50 smithi081 ceph-mon[31453]: pgmap v12972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:56:51.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:56:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:56:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:52 smithi153 ceph-mon[36569]: pgmap v12973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:52 smithi104 ceph-mon[36624]: pgmap v12973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:52 smithi081 ceph-mon[31453]: pgmap v12973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:56:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:56:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:56:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:56:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:56:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:56:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:56:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:56:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:56:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:54 smithi153 ceph-mon[36569]: pgmap v12974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:54 smithi153 ceph-mon[36569]: pgmap v12975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:54 smithi104 ceph-mon[36624]: pgmap v12974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:54 smithi104 ceph-mon[36624]: pgmap v12975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:54 smithi081 ceph-mon[31453]: pgmap v12974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:54 smithi081 ceph-mon[31453]: pgmap v12975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:56.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:56.053+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:56:56.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:56.054+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:56:56.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:56:56.054+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:56:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:56 smithi104 ceph-mon[36624]: pgmap v12976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:56 smithi081 ceph-mon[31453]: pgmap v12976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:56 smithi153 ceph-mon[36569]: pgmap v12976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:56:58 smithi104 ceph-mon[36624]: pgmap v12977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:56:58 smithi081 ceph-mon[31453]: pgmap v12977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:56:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:56:58 smithi153 ceph-mon[36569]: pgmap v12977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:00 smithi104 ceph-mon[36624]: pgmap v12978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:00 smithi081 ceph-mon[31453]: pgmap v12978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:00 smithi153 ceph-mon[36569]: pgmap v12978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:02 smithi104 ceph-mon[36624]: pgmap v12979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:02.608 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:02 smithi153 ceph-mon[36569]: pgmap v12979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:02 smithi081 ceph-mon[31453]: pgmap v12979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:04 smithi104 ceph-mon[36624]: pgmap v12980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:04 smithi081 ceph-mon[31453]: pgmap v12980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:04 smithi153 ceph-mon[36569]: pgmap v12980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:06 smithi081 ceph-mon[31453]: pgmap v12981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:06 smithi153 ceph-mon[36569]: pgmap v12981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:06 smithi104 ceph-mon[36624]: pgmap v12981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:08 smithi081 ceph-mon[31453]: pgmap v12982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:08 smithi153 ceph-mon[36569]: pgmap v12982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:08 smithi104 ceph-mon[36624]: pgmap v12982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:10 smithi081 ceph-mon[31453]: pgmap v12983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:10 smithi153 ceph-mon[36569]: pgmap v12983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:10 smithi104 ceph-mon[36624]: pgmap v12983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:11.069+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:57:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:11.069+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:57:11.069 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:11.069+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:57:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:12 smithi081 ceph-mon[31453]: pgmap v12984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:12 smithi153 ceph-mon[36569]: pgmap v12984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:12 smithi104 ceph-mon[36624]: pgmap v12984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:14 smithi081 ceph-mon[31453]: pgmap v12985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:14 smithi153 ceph-mon[36569]: pgmap v12985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:14 smithi104 ceph-mon[36624]: pgmap v12985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:16 smithi081 ceph-mon[31453]: pgmap v12986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:16 smithi153 ceph-mon[36569]: pgmap v12986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:16 smithi104 ceph-mon[36624]: pgmap v12986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:18 smithi081 ceph-mon[31453]: pgmap v12987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:18 smithi153 ceph-mon[36569]: pgmap v12987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:18 smithi104 ceph-mon[36624]: pgmap v12987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:20 smithi081 ceph-mon[31453]: pgmap v12988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:20 smithi153 ceph-mon[36569]: pgmap v12988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:20 smithi104 ceph-mon[36624]: pgmap v12988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:22 smithi081 ceph-mon[31453]: pgmap v12989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:22 smithi153 ceph-mon[36569]: pgmap v12989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:22 smithi104 ceph-mon[36624]: pgmap v12989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:24 smithi081 ceph-mon[31453]: pgmap v12990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:24 smithi153 ceph-mon[36569]: pgmap v12990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:24 smithi104 ceph-mon[36624]: pgmap v12990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:26.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:26.081+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:57:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:26.081+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:57:26.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:26.081+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:57:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:26 smithi081 ceph-mon[31453]: pgmap v12991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:26 smithi153 ceph-mon[36569]: pgmap v12991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:26 smithi104 ceph-mon[36624]: pgmap v12991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:28 smithi081 ceph-mon[31453]: pgmap v12992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:28 smithi153 ceph-mon[36569]: pgmap v12992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:28 smithi104 ceph-mon[36624]: pgmap v12992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:30 smithi081 ceph-mon[31453]: pgmap v12993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:30 smithi153 ceph-mon[36569]: pgmap v12993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:30 smithi104 ceph-mon[36624]: pgmap v12993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:32 smithi081 ceph-mon[31453]: pgmap v12994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:32 smithi153 ceph-mon[36569]: pgmap v12994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:32 smithi104 ceph-mon[36624]: pgmap v12994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:34 smithi153 ceph-mon[36569]: pgmap v12995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:34 smithi104 ceph-mon[36624]: pgmap v12995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:34 smithi081 ceph-mon[31453]: pgmap v12995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:36 smithi153 ceph-mon[36569]: pgmap v12996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:36 smithi104 ceph-mon[36624]: pgmap v12996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:36 smithi081 ceph-mon[31453]: pgmap v12996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:38 smithi153 ceph-mon[36569]: pgmap v12997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:38 smithi104 ceph-mon[36624]: pgmap v12997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:38 smithi081 ceph-mon[31453]: pgmap v12997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:40 smithi153 ceph-mon[36569]: pgmap v12998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:40 smithi104 ceph-mon[36624]: pgmap v12998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:40 smithi081 ceph-mon[31453]: pgmap v12998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:41.095+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:57:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:41.095+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:57:41.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:41.095+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:57:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:42 smithi153 ceph-mon[36569]: pgmap v12999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:42 smithi104 ceph-mon[36624]: pgmap v12999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:42 smithi081 ceph-mon[31453]: pgmap v12999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:44 smithi153 ceph-mon[36569]: pgmap v13000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:44 smithi104 ceph-mon[36624]: pgmap v13000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:44 smithi081 ceph-mon[31453]: pgmap v13000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:46 smithi153 ceph-mon[36569]: pgmap v13001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:46 smithi104 ceph-mon[36624]: pgmap v13001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:46 smithi081 ceph-mon[31453]: pgmap v13001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:48 smithi153 ceph-mon[36569]: pgmap v13002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:48 smithi104 ceph-mon[36624]: pgmap v13002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:48 smithi081 ceph-mon[31453]: pgmap v13002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:50 smithi153 ceph-mon[36569]: pgmap v13003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:50 smithi104 ceph-mon[36624]: pgmap v13003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:50 smithi081 ceph-mon[31453]: pgmap v13003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:52 smithi153 ceph-mon[36569]: pgmap v13004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:52 smithi104 ceph-mon[36624]: pgmap v13004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:52 smithi081 ceph-mon[31453]: pgmap v13004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:57:53.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:57:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: pgmap v13005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:57:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:57:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:57:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: pgmap v13005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:57:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:57:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: pgmap v13005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T03:57:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:57:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:57:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:57:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:55 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:57:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:55 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:57:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:55 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:57:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:55 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:57:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:55 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:57:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:55 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:57:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:55 smithi153 ceph-mon[36569]: pgmap v13006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:55 smithi153 ceph-mon[36569]: pgmap v13007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:55 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:57:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:55 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:57:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:55 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:57:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:55 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:57:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:55 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:57:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:55 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:57:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:55 smithi104 ceph-mon[36624]: pgmap v13006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:55 smithi104 ceph-mon[36624]: pgmap v13007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:55 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T03:57:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:55 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T03:57:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:55 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T03:57:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:55 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T03:57:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:55 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T03:57:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:55 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T03:57:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:55 smithi081 ceph-mon[31453]: pgmap v13006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:55 smithi081 ceph-mon[31453]: pgmap v13007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:56.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:56.107+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:57:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:56.107+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:57:56.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:57:56.107+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:57:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:57 smithi153 ceph-mon[36569]: pgmap v13008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:57 smithi104 ceph-mon[36624]: pgmap v13008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:57 smithi081 ceph-mon[31453]: pgmap v13008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:57:59 smithi153 ceph-mon[36569]: pgmap v13009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:57:59 smithi104 ceph-mon[36624]: pgmap v13009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:57:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:57:59 smithi081 ceph-mon[31453]: pgmap v13009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:01 smithi153 ceph-mon[36569]: pgmap v13010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:01 smithi104 ceph-mon[36624]: pgmap v13010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:01 smithi081 ceph-mon[31453]: pgmap v13010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:03 smithi153 ceph-mon[36569]: pgmap v13011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:03 smithi104 ceph-mon[36624]: pgmap v13011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:03 smithi081 ceph-mon[31453]: pgmap v13011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:05 smithi153 ceph-mon[36569]: pgmap v13012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:05 smithi104 ceph-mon[36624]: pgmap v13012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:05 smithi081 ceph-mon[31453]: pgmap v13012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:07 smithi153 ceph-mon[36569]: pgmap v13013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:07 smithi104 ceph-mon[36624]: pgmap v13013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:07 smithi081 ceph-mon[31453]: pgmap v13013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:09 smithi153 ceph-mon[36569]: pgmap v13014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:09 smithi104 ceph-mon[36624]: pgmap v13014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:09 smithi081 ceph-mon[31453]: pgmap v13014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:11.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:11.122+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:58:11.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:11.122+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:58:11.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:11.122+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:58:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:11 smithi153 ceph-mon[36569]: pgmap v13015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:11 smithi104 ceph-mon[36624]: pgmap v13015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:11 smithi081 ceph-mon[31453]: pgmap v13015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:13 smithi153 ceph-mon[36569]: pgmap v13016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:13 smithi104 ceph-mon[36624]: pgmap v13016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:13 smithi081 ceph-mon[31453]: pgmap v13016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:15 smithi153 ceph-mon[36569]: pgmap v13017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:15 smithi104 ceph-mon[36624]: pgmap v13017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:15 smithi081 ceph-mon[31453]: pgmap v13017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:16.686+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:58:16.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:16.686+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:58:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:16.687+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 187 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T03:58:16.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:16.687+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 187 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T03:58:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:17 smithi153 ceph-mon[36569]: pgmap v13018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:17 smithi104 ceph-mon[36624]: pgmap v13018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:17 smithi081 ceph-mon[31453]: pgmap v13018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:19 smithi153 ceph-mon[36569]: pgmap v13019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:19 smithi104 ceph-mon[36624]: pgmap v13019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:19 smithi081 ceph-mon[31453]: pgmap v13019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:20 smithi081 ceph-mon[31453]: pgmap v13020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:20 smithi153 ceph-mon[36569]: pgmap v13020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:20 smithi104 ceph-mon[36624]: pgmap v13020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:23 smithi153 ceph-mon[36569]: pgmap v13021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:23 smithi104 ceph-mon[36624]: pgmap v13021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:23 smithi081 ceph-mon[31453]: pgmap v13021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:25 smithi153 ceph-mon[36569]: pgmap v13022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:25 smithi104 ceph-mon[36624]: pgmap v13022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:25 smithi081 ceph-mon[31453]: pgmap v13022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:26.139+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:58:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:26.139+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:58:26.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:26.139+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:58:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:27 smithi153 ceph-mon[36569]: pgmap v13023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:27 smithi104 ceph-mon[36624]: pgmap v13023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:27 smithi081 ceph-mon[31453]: pgmap v13023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:29 smithi153 ceph-mon[36569]: pgmap v13024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:29 smithi104 ceph-mon[36624]: pgmap v13024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:29 smithi081 ceph-mon[31453]: pgmap v13024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:31 smithi153 ceph-mon[36569]: pgmap v13025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:31 smithi104 ceph-mon[36624]: pgmap v13025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:31 smithi081 ceph-mon[31453]: pgmap v13025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:33 smithi153 ceph-mon[36569]: pgmap v13026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:33 smithi104 ceph-mon[36624]: pgmap v13026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:33 smithi081 ceph-mon[31453]: pgmap v13026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:35 smithi153 ceph-mon[36569]: pgmap v13027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:35 smithi104 ceph-mon[36624]: pgmap v13027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:35 smithi081 ceph-mon[31453]: pgmap v13027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:37 smithi153 ceph-mon[36569]: pgmap v13028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:37 smithi104 ceph-mon[36624]: pgmap v13028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:37 smithi081 ceph-mon[31453]: pgmap v13028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:39 smithi153 ceph-mon[36569]: pgmap v13029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:39 smithi104 ceph-mon[36624]: pgmap v13029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:39 smithi081 ceph-mon[31453]: pgmap v13029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:41.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:41.145+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:58:41.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:41.145+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:58:41.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:41.145+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:58:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:41 smithi153 ceph-mon[36569]: pgmap v13030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:41 smithi104 ceph-mon[36624]: pgmap v13030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:41 smithi081 ceph-mon[31453]: pgmap v13030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:43 smithi153 ceph-mon[36569]: pgmap v13031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:43 smithi104 ceph-mon[36624]: pgmap v13031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:43 smithi081 ceph-mon[31453]: pgmap v13031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:45 smithi153 ceph-mon[36569]: pgmap v13032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:45 smithi104 ceph-mon[36624]: pgmap v13032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:45 smithi081 ceph-mon[31453]: pgmap v13032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:47 smithi153 ceph-mon[36569]: pgmap v13033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:47 smithi104 ceph-mon[36624]: pgmap v13033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:47 smithi081 ceph-mon[31453]: pgmap v13033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:49 smithi153 ceph-mon[36569]: pgmap v13034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:49 smithi104 ceph-mon[36624]: pgmap v13034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:49 smithi081 ceph-mon[31453]: pgmap v13034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:51 smithi153 ceph-mon[36569]: pgmap v13035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:51 smithi104 ceph-mon[36624]: pgmap v13035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:51 smithi081 ceph-mon[31453]: pgmap v13035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:53 smithi153 ceph-mon[36569]: pgmap v13036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:53 smithi104 ceph-mon[36624]: pgmap v13036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:53 smithi081 ceph-mon[31453]: pgmap v13036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:55 smithi081 ceph-mon[31453]: pgmap v13037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:58:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:55 smithi153 ceph-mon[36569]: pgmap v13037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:58:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:55 smithi104 ceph-mon[36624]: pgmap v13037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:58:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:56.157+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:58:56.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:56.157+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:58:56.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:58:56.157+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:58:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:58:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:58:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:58:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:58:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:58:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:58:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:58:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:58:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:58:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:57 smithi104 ceph-mon[36624]: pgmap v13038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:57 smithi104 ceph-mon[36624]: pgmap v13039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:57 smithi081 ceph-mon[31453]: pgmap v13038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:57 smithi081 ceph-mon[31453]: pgmap v13039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:57 smithi153 ceph-mon[36569]: pgmap v13038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:57 smithi153 ceph-mon[36569]: pgmap v13039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:58:59 smithi104 ceph-mon[36624]: pgmap v13040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:58:59 smithi081 ceph-mon[31453]: pgmap v13040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:58:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:58:59 smithi153 ceph-mon[36569]: pgmap v13040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:01 smithi104 ceph-mon[36624]: pgmap v13041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:01 smithi081 ceph-mon[31453]: pgmap v13041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:01 smithi153 ceph-mon[36569]: pgmap v13041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:03 smithi104 ceph-mon[36624]: pgmap v13042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:03.608 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:03 smithi153 ceph-mon[36569]: pgmap v13042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:03 smithi081 ceph-mon[31453]: pgmap v13042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:05 smithi081 ceph-mon[31453]: pgmap v13043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:05 smithi153 ceph-mon[36569]: pgmap v13043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:05 smithi104 ceph-mon[36624]: pgmap v13043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:07.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:07 smithi104 ceph-mon[36624]: pgmap v13044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:07 smithi081 ceph-mon[31453]: pgmap v13044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:07 smithi153 ceph-mon[36569]: pgmap v13044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:09 smithi081 ceph-mon[31453]: pgmap v13045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:09 smithi153 ceph-mon[36569]: pgmap v13045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:09 smithi104 ceph-mon[36624]: pgmap v13045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:11.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:11.170+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:59:11.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:11.170+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:59:11.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:11.170+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:59:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:11 smithi081 ceph-mon[31453]: pgmap v13046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:11 smithi153 ceph-mon[36569]: pgmap v13046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:11 smithi104 ceph-mon[36624]: pgmap v13046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:13 smithi081 ceph-mon[31453]: pgmap v13047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:13 smithi153 ceph-mon[36569]: pgmap v13047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:13 smithi104 ceph-mon[36624]: pgmap v13047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:15 smithi081 ceph-mon[31453]: pgmap v13048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:15 smithi153 ceph-mon[36569]: pgmap v13048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:15 smithi104 ceph-mon[36624]: pgmap v13048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:17 smithi081 ceph-mon[31453]: pgmap v13049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:17 smithi153 ceph-mon[36569]: pgmap v13049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:17 smithi104 ceph-mon[36624]: pgmap v13049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:19 smithi081 ceph-mon[31453]: pgmap v13050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:19 smithi153 ceph-mon[36569]: pgmap v13050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:19 smithi104 ceph-mon[36624]: pgmap v13050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:21 smithi081 ceph-mon[31453]: pgmap v13051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:21 smithi153 ceph-mon[36569]: pgmap v13051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:21 smithi104 ceph-mon[36624]: pgmap v13051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:23 smithi081 ceph-mon[31453]: pgmap v13052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:23 smithi153 ceph-mon[36569]: pgmap v13052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:23 smithi104 ceph-mon[36624]: pgmap v13052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:25 smithi081 ceph-mon[31453]: pgmap v13053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:25 smithi153 ceph-mon[36569]: pgmap v13053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:25 smithi104 ceph-mon[36624]: pgmap v13053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:26.171+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:59:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:26.171+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:59:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:26.171+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:59:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:27 smithi081 ceph-mon[31453]: pgmap v13054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:27 smithi153 ceph-mon[36569]: pgmap v13054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:27 smithi104 ceph-mon[36624]: pgmap v13054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:29 smithi081 ceph-mon[31453]: pgmap v13055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:29 smithi153 ceph-mon[36569]: pgmap v13055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:29 smithi104 ceph-mon[36624]: pgmap v13055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:31 smithi081 ceph-mon[31453]: pgmap v13056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:31 smithi153 ceph-mon[36569]: pgmap v13056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:31 smithi104 ceph-mon[36624]: pgmap v13056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:33 smithi153 ceph-mon[36569]: pgmap v13057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:33 smithi104 ceph-mon[36624]: pgmap v13057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:33 smithi081 ceph-mon[31453]: pgmap v13057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:35 smithi153 ceph-mon[36569]: pgmap v13058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:35 smithi104 ceph-mon[36624]: pgmap v13058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:35 smithi081 ceph-mon[31453]: pgmap v13058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:37 smithi153 ceph-mon[36569]: pgmap v13059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:37 smithi104 ceph-mon[36624]: pgmap v13059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:37 smithi081 ceph-mon[31453]: pgmap v13059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:39.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:39 smithi081 ceph-mon[31453]: pgmap v13060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:39 smithi153 ceph-mon[36569]: pgmap v13060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:39 smithi104 ceph-mon[36624]: pgmap v13060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:41.184 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:41.184+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:59:41.184 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:41.184+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:59:41.184 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:41.184+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:59:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:41 smithi153 ceph-mon[36569]: pgmap v13061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:41 smithi104 ceph-mon[36624]: pgmap v13061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:41 smithi081 ceph-mon[31453]: pgmap v13061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:43 smithi153 ceph-mon[36569]: pgmap v13062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:43 smithi104 ceph-mon[36624]: pgmap v13062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:43 smithi081 ceph-mon[31453]: pgmap v13062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:45 smithi153 ceph-mon[36569]: pgmap v13063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:45 smithi104 ceph-mon[36624]: pgmap v13063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:45 smithi081 ceph-mon[31453]: pgmap v13063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:47 smithi153 ceph-mon[36569]: pgmap v13064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:47 smithi104 ceph-mon[36624]: pgmap v13064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:47 smithi081 ceph-mon[31453]: pgmap v13064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:49 smithi153 ceph-mon[36569]: pgmap v13065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:49 smithi104 ceph-mon[36624]: pgmap v13065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:49 smithi081 ceph-mon[31453]: pgmap v13065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:51 smithi153 ceph-mon[36569]: pgmap v13066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:51 smithi104 ceph-mon[36624]: pgmap v13066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:51 smithi081 ceph-mon[31453]: pgmap v13066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:53 smithi153 ceph-mon[36569]: pgmap v13067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:53 smithi104 ceph-mon[36624]: pgmap v13067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:53 smithi081 ceph-mon[31453]: pgmap v13067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:55 smithi153 ceph-mon[36569]: pgmap v13068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:55 smithi104 ceph-mon[36624]: pgmap v13068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:55 smithi081 ceph-mon[31453]: pgmap v13068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:56.198+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T03:59:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:56.198+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T03:59:56.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T03:59:56.198+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T03:59:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:59:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:59:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T03:59:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:57 smithi153 ceph-mon[36569]: pgmap v13069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:59:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:59:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:59:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:57 smithi104 ceph-mon[36624]: pgmap v13069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:59:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:59:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:59:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:57 smithi081 ceph-mon[31453]: pgmap v13069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T03:59:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T03:59:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T03:59:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:58 smithi153 ceph-mon[36569]: pgmap v13070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 03:59:58 smithi153 ceph-mon[36569]: pgmap v13071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:58 smithi104 ceph-mon[36624]: pgmap v13070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 03:59:58 smithi104 ceph-mon[36624]: pgmap v13071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:58 smithi081 ceph-mon[31453]: pgmap v13070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T03:59:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 03:59:58 smithi081 ceph-mon[31453]: pgmap v13071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:00 smithi153 ceph-mon[36569]: pgmap v13072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:00:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:00 smithi104 ceph-mon[36624]: pgmap v13072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:00:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:00 smithi081 ceph-mon[31453]: pgmap v13072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:00:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:02 smithi153 ceph-mon[36569]: pgmap v13073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:02 smithi104 ceph-mon[36624]: pgmap v13073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:02 smithi081 ceph-mon[31453]: pgmap v13073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:04 smithi153 ceph-mon[36569]: pgmap v13074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:04 smithi104 ceph-mon[36624]: pgmap v13074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:04 smithi081 ceph-mon[31453]: pgmap v13074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:06 smithi153 ceph-mon[36569]: pgmap v13075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:06 smithi104 ceph-mon[36624]: pgmap v13075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:06 smithi081 ceph-mon[31453]: pgmap v13075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:08 smithi153 ceph-mon[36569]: pgmap v13076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:08 smithi104 ceph-mon[36624]: pgmap v13076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:08 smithi081 ceph-mon[31453]: pgmap v13076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:10 smithi153 ceph-mon[36569]: pgmap v13077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:10 smithi104 ceph-mon[36624]: pgmap v13077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:10 smithi081 ceph-mon[31453]: pgmap v13077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:11.214+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:00:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:11.214+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:00:11.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:11.214+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:00:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:12 smithi153 ceph-mon[36569]: pgmap v13078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:12 smithi104 ceph-mon[36624]: pgmap v13078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:12 smithi081 ceph-mon[31453]: pgmap v13078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:14 smithi153 ceph-mon[36569]: pgmap v13079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:14 smithi104 ceph-mon[36624]: pgmap v13079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:14 smithi081 ceph-mon[31453]: pgmap v13079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:16 smithi153 ceph-mon[36569]: pgmap v13080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:16 smithi104 ceph-mon[36624]: pgmap v13080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:16 smithi081 ceph-mon[31453]: pgmap v13080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:18 smithi153 ceph-mon[36569]: pgmap v13081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:18 smithi104 ceph-mon[36624]: pgmap v13081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:18 smithi081 ceph-mon[31453]: pgmap v13081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:20 smithi153 ceph-mon[36569]: pgmap v13082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:20 smithi104 ceph-mon[36624]: pgmap v13082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:20 smithi081 ceph-mon[31453]: pgmap v13082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:22 smithi153 ceph-mon[36569]: pgmap v13083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:22 smithi104 ceph-mon[36624]: pgmap v13083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:22 smithi081 ceph-mon[31453]: pgmap v13083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:24 smithi153 ceph-mon[36569]: pgmap v13084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:24 smithi104 ceph-mon[36624]: pgmap v13084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:24 smithi081 ceph-mon[31453]: pgmap v13084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:26.229+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:00:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:26.229+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:00:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:26.229+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:00:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:26 smithi104 ceph-mon[36624]: pgmap v13085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:26 smithi081 ceph-mon[31453]: pgmap v13085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:26 smithi153 ceph-mon[36569]: pgmap v13085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:28 smithi104 ceph-mon[36624]: pgmap v13086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:28 smithi081 ceph-mon[31453]: pgmap v13086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:29.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:28 smithi153 ceph-mon[36569]: pgmap v13086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:30 smithi104 ceph-mon[36624]: pgmap v13087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:30 smithi081 ceph-mon[31453]: pgmap v13087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:30 smithi153 ceph-mon[36569]: pgmap v13087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:31.709+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:00:31.709 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:31.709+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:00:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:31.710+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 188 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:00:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:31.710+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 188 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:00:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:32 smithi104 ceph-mon[36624]: pgmap v13088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:32 smithi081 ceph-mon[31453]: pgmap v13088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:32 smithi153 ceph-mon[36569]: pgmap v13088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:33 smithi081 ceph-mon[31453]: pgmap v13089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:33 smithi153 ceph-mon[36569]: pgmap v13089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:33 smithi104 ceph-mon[36624]: pgmap v13089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:36.760 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:36 smithi153 ceph-mon[36569]: pgmap v13090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:36 smithi104 ceph-mon[36624]: pgmap v13090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:36 smithi081 ceph-mon[31453]: pgmap v13090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:38 smithi153 ceph-mon[36569]: pgmap v13091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:38 smithi104 ceph-mon[36624]: pgmap v13091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:38 smithi081 ceph-mon[31453]: pgmap v13091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:40 smithi153 ceph-mon[36569]: pgmap v13092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:40 smithi104 ceph-mon[36624]: pgmap v13092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:40 smithi081 ceph-mon[31453]: pgmap v13092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:41.237 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:41.238+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:00:41.237 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:41.238+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:00:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:41.238+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:00:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:42 smithi153 ceph-mon[36569]: pgmap v13093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:42.831 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:42 smithi081 ceph-mon[31453]: pgmap v13093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:42 smithi104 ceph-mon[36624]: pgmap v13093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:44 smithi153 ceph-mon[36569]: pgmap v13094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:44 smithi104 ceph-mon[36624]: pgmap v13094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:44 smithi081 ceph-mon[31453]: pgmap v13094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:46 smithi153 ceph-mon[36569]: pgmap v13095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:46 smithi104 ceph-mon[36624]: pgmap v13095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:46 smithi081 ceph-mon[31453]: pgmap v13095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:48 smithi153 ceph-mon[36569]: pgmap v13096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:48 smithi104 ceph-mon[36624]: pgmap v13096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:48 smithi081 ceph-mon[31453]: pgmap v13096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:50 smithi153 ceph-mon[36569]: pgmap v13097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:50 smithi104 ceph-mon[36624]: pgmap v13097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:50 smithi081 ceph-mon[31453]: pgmap v13097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:52 smithi153 ceph-mon[36569]: pgmap v13098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:52 smithi104 ceph-mon[36624]: pgmap v13098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:52 smithi081 ceph-mon[31453]: pgmap v13098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:54 smithi153 ceph-mon[36569]: pgmap v13099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:54 smithi104 ceph-mon[36624]: pgmap v13099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:54 smithi081 ceph-mon[31453]: pgmap v13099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:56.253+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:00:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:56.253+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:00:56.253 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:00:56.253+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:00:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:56 smithi153 ceph-mon[36569]: pgmap v13100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:56 smithi104 ceph-mon[36624]: pgmap v13100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:56 smithi081 ceph-mon[31453]: pgmap v13100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:58 smithi153 ceph-mon[36569]: pgmap v13101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:00:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:58 smithi104 ceph-mon[36624]: pgmap v13101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:00:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:58 smithi081 ceph-mon[31453]: pgmap v13101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:00:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:00:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:00:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:00:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:00:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:00:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:00:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:00:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:00:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:01:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:01:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:01:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:00:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:01:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:00 smithi104 ceph-mon[36624]: pgmap v13102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:00 smithi104 ceph-mon[36624]: pgmap v13103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:00 smithi081 ceph-mon[31453]: pgmap v13102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:00 smithi081 ceph-mon[31453]: pgmap v13103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:00 smithi153 ceph-mon[36569]: pgmap v13102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:00 smithi153 ceph-mon[36569]: pgmap v13103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:02 smithi104 ceph-mon[36624]: pgmap v13104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:02 smithi081 ceph-mon[31453]: pgmap v13104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:02 smithi153 ceph-mon[36569]: pgmap v13104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:04 smithi104 ceph-mon[36624]: pgmap v13105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:04 smithi081 ceph-mon[31453]: pgmap v13105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:04 smithi153 ceph-mon[36569]: pgmap v13105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:06 smithi104 ceph-mon[36624]: pgmap v13106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:06.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:06 smithi153 ceph-mon[36569]: pgmap v13106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:06 smithi081 ceph-mon[31453]: pgmap v13106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:07 smithi081 ceph-mon[31453]: pgmap v13107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:07 smithi153 ceph-mon[36569]: pgmap v13107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:07 smithi104 ceph-mon[36624]: pgmap v13107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:09 smithi153 ceph-mon[36569]: pgmap v13108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:09 smithi104 ceph-mon[36624]: pgmap v13108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:09 smithi081 ceph-mon[31453]: pgmap v13108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:11.269+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:01:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:11.269+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:01:11.269 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:11.269+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:01:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:12.157+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 msgr2=0x7f16e80b39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000675) for more than 900000000 us, fault. 2024-04-13T04:01:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:12.157+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ce2a0 tx=0x7f16f4032000 comp rx=0 tx=0).stop 2024-04-13T04:01:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:12.157+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 msgr2=0x7f16e80b39a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T04:01:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:12.157+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T04:01:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:12.157+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T04:01:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:12.158+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T04:01:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:12.159+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c9f0 tx=0x7f16f4044000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T04:01:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:11 smithi153 ceph-mon[36569]: pgmap v13109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:11 smithi104 ceph-mon[36624]: pgmap v13109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:11 smithi081 ceph-mon[31453]: pgmap v13109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:14 smithi153 ceph-mon[36569]: pgmap v13110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:14 smithi104 ceph-mon[36624]: pgmap v13110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:14 smithi081 ceph-mon[31453]: pgmap v13110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:16 smithi153 ceph-mon[36569]: pgmap v13111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:16 smithi104 ceph-mon[36624]: pgmap v13111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:16 smithi081 ceph-mon[31453]: pgmap v13111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:16.716+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:01:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:16.716+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:01:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:16.717+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 189 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:01:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:16.717+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 189 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:01:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:18 smithi153 ceph-mon[36569]: pgmap v13112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:18 smithi104 ceph-mon[36624]: pgmap v13112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:18 smithi081 ceph-mon[31453]: pgmap v13112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:20 smithi153 ceph-mon[36569]: pgmap v13113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:20 smithi104 ceph-mon[36624]: pgmap v13113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:20 smithi081 ceph-mon[31453]: pgmap v13113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:22 smithi104 ceph-mon[36624]: pgmap v13114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:22 smithi081 ceph-mon[31453]: pgmap v13114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:22 smithi153 ceph-mon[36569]: pgmap v13114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:24 smithi104 ceph-mon[36624]: pgmap v13115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:24 smithi081 ceph-mon[31453]: pgmap v13115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:24 smithi153 ceph-mon[36569]: pgmap v13115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:26.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:26.277+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:01:26.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:26.277+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:01:26.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:26.277+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:01:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:26 smithi104 ceph-mon[36624]: pgmap v13116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:26 smithi081 ceph-mon[31453]: pgmap v13116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:26 smithi153 ceph-mon[36569]: pgmap v13116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:28 smithi104 ceph-mon[36624]: pgmap v13117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:28 smithi081 ceph-mon[31453]: pgmap v13117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:28 smithi153 ceph-mon[36569]: pgmap v13117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:30 smithi081 ceph-mon[31453]: pgmap v13118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:30 smithi153 ceph-mon[36569]: pgmap v13118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:30 smithi104 ceph-mon[36624]: pgmap v13118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:32 smithi081 ceph-mon[31453]: pgmap v13119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:32 smithi153 ceph-mon[36569]: pgmap v13119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:32 smithi104 ceph-mon[36624]: pgmap v13119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:34 smithi081 ceph-mon[31453]: pgmap v13120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:34 smithi153 ceph-mon[36569]: pgmap v13120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:34 smithi104 ceph-mon[36624]: pgmap v13120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:36.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:36 smithi153 ceph-mon[36569]: pgmap v13121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:36 smithi081 ceph-mon[31453]: pgmap v13121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:36 smithi104 ceph-mon[36624]: pgmap v13121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:38 smithi081 ceph-mon[31453]: pgmap v13122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:38 smithi153 ceph-mon[36569]: pgmap v13122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:38 smithi104 ceph-mon[36624]: pgmap v13122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:40.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:40 smithi104 ceph-mon[36624]: pgmap v13123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:40.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:40 smithi081 ceph-mon[31453]: pgmap v13123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:40 smithi153 ceph-mon[36569]: pgmap v13123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:41.292+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:01:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:41.292+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:01:41.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:41.292+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:01:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:42 smithi081 ceph-mon[31453]: pgmap v13124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:42 smithi153 ceph-mon[36569]: pgmap v13124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:42 smithi104 ceph-mon[36624]: pgmap v13124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:44 smithi081 ceph-mon[31453]: pgmap v13125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:44 smithi153 ceph-mon[36569]: pgmap v13125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:44 smithi104 ceph-mon[36624]: pgmap v13125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:46 smithi081 ceph-mon[31453]: pgmap v13126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:46 smithi153 ceph-mon[36569]: pgmap v13126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:46 smithi104 ceph-mon[36624]: pgmap v13126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:48 smithi081 ceph-mon[31453]: pgmap v13127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:48 smithi153 ceph-mon[36569]: pgmap v13127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:48 smithi104 ceph-mon[36624]: pgmap v13127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:50 smithi081 ceph-mon[31453]: pgmap v13128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:50 smithi153 ceph-mon[36569]: pgmap v13128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:50 smithi104 ceph-mon[36624]: pgmap v13128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:52 smithi081 ceph-mon[31453]: pgmap v13129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:52 smithi153 ceph-mon[36569]: pgmap v13129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:52 smithi104 ceph-mon[36624]: pgmap v13129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:54 smithi081 ceph-mon[31453]: pgmap v13130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:54 smithi153 ceph-mon[36569]: pgmap v13130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:54 smithi104 ceph-mon[36624]: pgmap v13130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:56.307+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:01:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:56.307+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:01:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:01:56.307+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:01:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:56 smithi081 ceph-mon[31453]: pgmap v13131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:56 smithi153 ceph-mon[36569]: pgmap v13131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:56 smithi104 ceph-mon[36624]: pgmap v13131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:58 smithi153 ceph-mon[36569]: pgmap v13132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:58 smithi104 ceph-mon[36624]: pgmap v13132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:58 smithi081 ceph-mon[31453]: pgmap v13132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:01:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:01:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:01:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:01:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:01:59.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:01:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:02:00.735 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:00 smithi104 ceph-mon[36624]: pgmap v13133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:00.736 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.736 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.736 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.736 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:00 smithi153 ceph-mon[36569]: pgmap v13133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:00 smithi081 ceph-mon[31453]: pgmap v13133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:02 smithi153 ceph-mon[36569]: pgmap v13134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:02 smithi104 ceph-mon[36624]: pgmap v13134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:02 smithi081 ceph-mon[31453]: pgmap v13134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:04 smithi153 ceph-mon[36569]: pgmap v13135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:04 smithi104 ceph-mon[36624]: pgmap v13135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:04 smithi081 ceph-mon[31453]: pgmap v13135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:06 smithi153 ceph-mon[36569]: pgmap v13136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:02:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:02:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:06 smithi104 ceph-mon[36624]: pgmap v13136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:02:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:02:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:06 smithi081 ceph-mon[31453]: pgmap v13136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:02:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:02:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:02:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:07 smithi153 ceph-mon[36569]: pgmap v13137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:07 smithi153 ceph-mon[36569]: pgmap v13138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:07 smithi104 ceph-mon[36624]: pgmap v13137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:07 smithi104 ceph-mon[36624]: pgmap v13138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:07 smithi081 ceph-mon[31453]: pgmap v13137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:07 smithi081 ceph-mon[31453]: pgmap v13138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:09 smithi153 ceph-mon[36569]: pgmap v13139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:09 smithi104 ceph-mon[36624]: pgmap v13139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:09 smithi081 ceph-mon[31453]: pgmap v13139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:11.317+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:02:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:11.317+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:02:11.317 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:11.317+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:02:11.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:11 smithi081 ceph-mon[31453]: pgmap v13140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:11 smithi153 ceph-mon[36569]: pgmap v13140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:11 smithi104 ceph-mon[36624]: pgmap v13140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:13 smithi153 ceph-mon[36569]: pgmap v13141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:13 smithi104 ceph-mon[36624]: pgmap v13141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:13 smithi081 ceph-mon[31453]: pgmap v13141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:15 smithi153 ceph-mon[36569]: pgmap v13142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:15 smithi104 ceph-mon[36624]: pgmap v13142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:15 smithi081 ceph-mon[31453]: pgmap v13142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:17 smithi153 ceph-mon[36569]: pgmap v13143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:17 smithi104 ceph-mon[36624]: pgmap v13143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:17.863 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:17 smithi081 ceph-mon[31453]: pgmap v13143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:19 smithi153 ceph-mon[36569]: pgmap v13144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:19 smithi104 ceph-mon[36624]: pgmap v13144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:19 smithi081 ceph-mon[31453]: pgmap v13144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:21 smithi153 ceph-mon[36569]: pgmap v13145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:21 smithi104 ceph-mon[36624]: pgmap v13145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:21 smithi081 ceph-mon[31453]: pgmap v13145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:23 smithi153 ceph-mon[36569]: pgmap v13146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:23 smithi104 ceph-mon[36624]: pgmap v13146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:23 smithi081 ceph-mon[31453]: pgmap v13146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:25 smithi153 ceph-mon[36569]: pgmap v13147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:25 smithi104 ceph-mon[36624]: pgmap v13147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:25 smithi081 ceph-mon[31453]: pgmap v13147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:26.330+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:02:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:26.330+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:02:26.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:26.330+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:02:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:27 smithi153 ceph-mon[36569]: pgmap v13148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:27 smithi104 ceph-mon[36624]: pgmap v13148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:27 smithi081 ceph-mon[31453]: pgmap v13148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:29 smithi153 ceph-mon[36569]: pgmap v13149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:29 smithi104 ceph-mon[36624]: pgmap v13149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:29 smithi081 ceph-mon[31453]: pgmap v13149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:31 smithi153 ceph-mon[36569]: pgmap v13150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:31 smithi104 ceph-mon[36624]: pgmap v13150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:31 smithi081 ceph-mon[31453]: pgmap v13150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:33 smithi153 ceph-mon[36569]: pgmap v13151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:33 smithi104 ceph-mon[36624]: pgmap v13151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:33 smithi081 ceph-mon[31453]: pgmap v13151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:35 smithi153 ceph-mon[36569]: pgmap v13152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:35 smithi104 ceph-mon[36624]: pgmap v13152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:35 smithi081 ceph-mon[31453]: pgmap v13152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:37 smithi153 ceph-mon[36569]: pgmap v13153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:37 smithi104 ceph-mon[36624]: pgmap v13153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:37 smithi081 ceph-mon[31453]: pgmap v13153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:39 smithi153 ceph-mon[36569]: pgmap v13154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:39 smithi104 ceph-mon[36624]: pgmap v13154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:39 smithi081 ceph-mon[31453]: pgmap v13154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:41.342+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:02:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:41.342+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:02:41.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:41.342+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:02:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:41 smithi153 ceph-mon[36569]: pgmap v13155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:41 smithi104 ceph-mon[36624]: pgmap v13155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:41 smithi081 ceph-mon[31453]: pgmap v13155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:43 smithi153 ceph-mon[36569]: pgmap v13156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:43 smithi104 ceph-mon[36624]: pgmap v13156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:43 smithi081 ceph-mon[31453]: pgmap v13156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:45 smithi153 ceph-mon[36569]: pgmap v13157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:45 smithi104 ceph-mon[36624]: pgmap v13157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:45 smithi081 ceph-mon[31453]: pgmap v13157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:47 smithi153 ceph-mon[36569]: pgmap v13158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:47 smithi104 ceph-mon[36624]: pgmap v13158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:47 smithi081 ceph-mon[31453]: pgmap v13158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:49 smithi153 ceph-mon[36569]: pgmap v13159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:49 smithi104 ceph-mon[36624]: pgmap v13159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:49 smithi081 ceph-mon[31453]: pgmap v13159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:51 smithi104 ceph-mon[36624]: pgmap v13160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:51 smithi081 ceph-mon[31453]: pgmap v13160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:51 smithi153 ceph-mon[36569]: pgmap v13160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:53 smithi104 ceph-mon[36624]: pgmap v13161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:53 smithi081 ceph-mon[31453]: pgmap v13161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:53 smithi153 ceph-mon[36569]: pgmap v13161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:55 smithi104 ceph-mon[36624]: pgmap v13162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:55 smithi081 ceph-mon[31453]: pgmap v13162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:55 smithi153 ceph-mon[36569]: pgmap v13162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:56.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:56.353+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:02:56.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:56.354+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:02:56.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:02:56.354+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:02:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:57 smithi104 ceph-mon[36624]: pgmap v13163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:57 smithi081 ceph-mon[31453]: pgmap v13163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:57 smithi153 ceph-mon[36569]: pgmap v13163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:02:59 smithi104 ceph-mon[36624]: pgmap v13164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:02:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:02:59 smithi081 ceph-mon[31453]: pgmap v13164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:02:59 smithi153 ceph-mon[36569]: pgmap v13164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:01 smithi081 ceph-mon[31453]: pgmap v13165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:01 smithi153 ceph-mon[36569]: pgmap v13165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:01 smithi104 ceph-mon[36624]: pgmap v13165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:03 smithi081 ceph-mon[31453]: pgmap v13166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:03 smithi153 ceph-mon[36569]: pgmap v13166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:03 smithi104 ceph-mon[36624]: pgmap v13166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:05 smithi081 ceph-mon[31453]: pgmap v13167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:05 smithi153 ceph-mon[36569]: pgmap v13167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:05 smithi104 ceph-mon[36624]: pgmap v13167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:03:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:03:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:03:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:07 smithi081 ceph-mon[31453]: pgmap v13168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:03:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:03:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:07.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:07 smithi153 ceph-mon[36569]: pgmap v13168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:07.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:03:07.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:03:07.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:07 smithi104 ceph-mon[36624]: pgmap v13168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:03:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:03:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:03:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:08 smithi081 ceph-mon[31453]: pgmap v13169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:08 smithi081 ceph-mon[31453]: pgmap v13170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:08 smithi153 ceph-mon[36569]: pgmap v13169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:08 smithi153 ceph-mon[36569]: pgmap v13170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:08 smithi104 ceph-mon[36624]: pgmap v13169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:08 smithi104 ceph-mon[36624]: pgmap v13170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:09 smithi081 ceph-mon[31453]: pgmap v13171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:09 smithi153 ceph-mon[36569]: pgmap v13171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:09 smithi104 ceph-mon[36624]: pgmap v13171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:11.369+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:03:11.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:11.369+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:03:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:11.369+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:03:12.610 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:12 smithi081 ceph-mon[31453]: pgmap v13172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:12.785 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:12 smithi104 ceph-mon[36624]: pgmap v13172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:12 smithi153 ceph-mon[36569]: pgmap v13172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:14.478 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:14 smithi081 ceph-mon[31453]: pgmap v13173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:14 smithi153 ceph-mon[36569]: pgmap v13173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:14 smithi104 ceph-mon[36624]: pgmap v13173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:16 smithi081 ceph-mon[31453]: pgmap v13174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:16 smithi153 ceph-mon[36569]: pgmap v13174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:16 smithi104 ceph-mon[36624]: pgmap v13174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:18 smithi081 ceph-mon[31453]: pgmap v13175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:18 smithi153 ceph-mon[36569]: pgmap v13175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:18 smithi104 ceph-mon[36624]: pgmap v13175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:20 smithi081 ceph-mon[31453]: pgmap v13176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:20 smithi153 ceph-mon[36569]: pgmap v13176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:20 smithi104 ceph-mon[36624]: pgmap v13176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:22 smithi081 ceph-mon[31453]: pgmap v13177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:22 smithi153 ceph-mon[36569]: pgmap v13177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:22 smithi104 ceph-mon[36624]: pgmap v13177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:24 smithi081 ceph-mon[31453]: pgmap v13178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:24 smithi153 ceph-mon[36569]: pgmap v13178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:24 smithi104 ceph-mon[36624]: pgmap v13178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:26.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:26.384+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:03:26.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:26.384+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:03:26.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:26.384+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:03:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:26 smithi081 ceph-mon[31453]: pgmap v13179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:26 smithi153 ceph-mon[36569]: pgmap v13179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:26 smithi104 ceph-mon[36624]: pgmap v13179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:28 smithi081 ceph-mon[31453]: pgmap v13180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:28 smithi153 ceph-mon[36569]: pgmap v13180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:28 smithi104 ceph-mon[36624]: pgmap v13180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:30 smithi081 ceph-mon[31453]: pgmap v13181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:30 smithi153 ceph-mon[36569]: pgmap v13181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:30 smithi104 ceph-mon[36624]: pgmap v13181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:32 smithi153 ceph-mon[36569]: pgmap v13182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:32 smithi104 ceph-mon[36624]: pgmap v13182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:32 smithi081 ceph-mon[31453]: pgmap v13182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:34 smithi153 ceph-mon[36569]: pgmap v13183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:34 smithi104 ceph-mon[36624]: pgmap v13183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:34 smithi081 ceph-mon[31453]: pgmap v13183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:36 smithi081 ceph-mon[31453]: pgmap v13184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:36 smithi153 ceph-mon[36569]: pgmap v13184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:36 smithi104 ceph-mon[36624]: pgmap v13184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:38 smithi153 ceph-mon[36569]: pgmap v13185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:38 smithi104 ceph-mon[36624]: pgmap v13185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:38 smithi081 ceph-mon[31453]: pgmap v13185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:40 smithi153 ceph-mon[36569]: pgmap v13186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:40 smithi104 ceph-mon[36624]: pgmap v13186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:40 smithi081 ceph-mon[31453]: pgmap v13186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:41.395+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:03:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:41.395+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:03:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:41.395+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:03:42.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:42 smithi081 ceph-mon[31453]: pgmap v13187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:42 smithi153 ceph-mon[36569]: pgmap v13187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:42 smithi104 ceph-mon[36624]: pgmap v13187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:44 smithi153 ceph-mon[36569]: pgmap v13188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:44 smithi104 ceph-mon[36624]: pgmap v13188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:44 smithi081 ceph-mon[31453]: pgmap v13188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:46 smithi153 ceph-mon[36569]: pgmap v13189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:46 smithi104 ceph-mon[36624]: pgmap v13189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:46 smithi081 ceph-mon[31453]: pgmap v13189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:48 smithi153 ceph-mon[36569]: pgmap v13190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:48 smithi104 ceph-mon[36624]: pgmap v13190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:48 smithi081 ceph-mon[31453]: pgmap v13190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:50 smithi153 ceph-mon[36569]: pgmap v13191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:50 smithi104 ceph-mon[36624]: pgmap v13191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:50 smithi081 ceph-mon[31453]: pgmap v13191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:52 smithi153 ceph-mon[36569]: pgmap v13192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:52 smithi104 ceph-mon[36624]: pgmap v13192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:52 smithi081 ceph-mon[31453]: pgmap v13192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:54 smithi153 ceph-mon[36569]: pgmap v13193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:54 smithi104 ceph-mon[36624]: pgmap v13193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:54 smithi081 ceph-mon[31453]: pgmap v13193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:56.405+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:03:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:56.406+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:03:56.406 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:03:56.406+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:03:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:56 smithi153 ceph-mon[36569]: pgmap v13194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:56 smithi104 ceph-mon[36624]: pgmap v13194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:56 smithi081 ceph-mon[31453]: pgmap v13194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:03:58 smithi153 ceph-mon[36569]: pgmap v13195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:03:58 smithi104 ceph-mon[36624]: pgmap v13195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:03:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:03:58 smithi081 ceph-mon[31453]: pgmap v13195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:00 smithi153 ceph-mon[36569]: pgmap v13196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:00 smithi104 ceph-mon[36624]: pgmap v13196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:00 smithi081 ceph-mon[31453]: pgmap v13196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:02 smithi153 ceph-mon[36569]: pgmap v13197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:02 smithi104 ceph-mon[36624]: pgmap v13197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:02 smithi081 ceph-mon[31453]: pgmap v13197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:04 smithi153 ceph-mon[36569]: pgmap v13198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:04 smithi104 ceph-mon[36624]: pgmap v13198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:04 smithi081 ceph-mon[31453]: pgmap v13198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:06 smithi153 ceph-mon[36569]: pgmap v13199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:06 smithi104 ceph-mon[36624]: pgmap v13199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:06 smithi081 ceph-mon[31453]: pgmap v13199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:08 smithi153 ceph-mon[36569]: pgmap v13200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:04:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:08 smithi104 ceph-mon[36624]: pgmap v13200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:04:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:08 smithi081 ceph-mon[31453]: pgmap v13200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:04:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:04:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:04:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:04:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:04:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:04:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:04:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:04:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:04:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:04:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:10 smithi153 ceph-mon[36569]: pgmap v13201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:10 smithi153 ceph-mon[36569]: pgmap v13202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:10 smithi104 ceph-mon[36624]: pgmap v13201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:10 smithi104 ceph-mon[36624]: pgmap v13202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:10 smithi081 ceph-mon[31453]: pgmap v13201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:10 smithi081 ceph-mon[31453]: pgmap v13202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:11.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:11.421+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:04:11.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:11.421+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:04:11.422 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:11.421+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:04:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:12 smithi153 ceph-mon[36569]: pgmap v13203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:12 smithi104 ceph-mon[36624]: pgmap v13203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:12 smithi081 ceph-mon[31453]: pgmap v13203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:14 smithi153 ceph-mon[36569]: pgmap v13204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:14 smithi104 ceph-mon[36624]: pgmap v13204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:14 smithi081 ceph-mon[31453]: pgmap v13204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:16.745+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:04:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:16.745+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:04:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:16.746+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 190 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:04:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:16.746+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 190 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:04:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:16 smithi153 ceph-mon[36569]: pgmap v13205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:16 smithi104 ceph-mon[36624]: pgmap v13205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:16 smithi081 ceph-mon[31453]: pgmap v13205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:18 smithi153 ceph-mon[36569]: pgmap v13206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:18 smithi104 ceph-mon[36624]: pgmap v13206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:18 smithi081 ceph-mon[31453]: pgmap v13206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:20 smithi153 ceph-mon[36569]: pgmap v13207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:20 smithi104 ceph-mon[36624]: pgmap v13207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:20 smithi081 ceph-mon[31453]: pgmap v13207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:22 smithi153 ceph-mon[36569]: pgmap v13208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:22 smithi104 ceph-mon[36624]: pgmap v13208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:22 smithi081 ceph-mon[31453]: pgmap v13208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:24 smithi104 ceph-mon[36624]: pgmap v13209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:24 smithi081 ceph-mon[31453]: pgmap v13209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:24 smithi153 ceph-mon[36569]: pgmap v13209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:26.437+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:04:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:26.437+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:04:26.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:26.437+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:04:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:26 smithi104 ceph-mon[36624]: pgmap v13210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:26 smithi081 ceph-mon[31453]: pgmap v13210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:26 smithi153 ceph-mon[36569]: pgmap v13210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:28 smithi104 ceph-mon[36624]: pgmap v13211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:28 smithi081 ceph-mon[31453]: pgmap v13211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:28 smithi153 ceph-mon[36569]: pgmap v13211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:30 smithi104 ceph-mon[36624]: pgmap v13212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:30 smithi081 ceph-mon[31453]: pgmap v13212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:30 smithi153 ceph-mon[36569]: pgmap v13212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:32 smithi104 ceph-mon[36624]: pgmap v13213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:32 smithi081 ceph-mon[31453]: pgmap v13213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:32 smithi153 ceph-mon[36569]: pgmap v13213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:34 smithi081 ceph-mon[31453]: pgmap v13214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:34 smithi153 ceph-mon[36569]: pgmap v13214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:34 smithi104 ceph-mon[36624]: pgmap v13214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:36 smithi081 ceph-mon[31453]: pgmap v13215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:36 smithi153 ceph-mon[36569]: pgmap v13215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:36 smithi104 ceph-mon[36624]: pgmap v13215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:38.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:38 smithi153 ceph-mon[36569]: pgmap v13216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:38 smithi081 ceph-mon[31453]: pgmap v13216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:38 smithi104 ceph-mon[36624]: pgmap v13216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:40 smithi081 ceph-mon[31453]: pgmap v13217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:40 smithi153 ceph-mon[36569]: pgmap v13217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:40 smithi104 ceph-mon[36624]: pgmap v13217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:41.444+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:04:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:41.444+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:04:41.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:41.444+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:04:42.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:42 smithi104 ceph-mon[36624]: pgmap v13218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:42 smithi081 ceph-mon[31453]: pgmap v13218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:42 smithi153 ceph-mon[36569]: pgmap v13218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:44 smithi081 ceph-mon[31453]: pgmap v13219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:44 smithi153 ceph-mon[36569]: pgmap v13219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:44 smithi104 ceph-mon[36624]: pgmap v13219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:46 smithi081 ceph-mon[31453]: pgmap v13220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:46 smithi153 ceph-mon[36569]: pgmap v13220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:46 smithi104 ceph-mon[36624]: pgmap v13220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:48 smithi081 ceph-mon[31453]: pgmap v13221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:48 smithi153 ceph-mon[36569]: pgmap v13221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:48 smithi104 ceph-mon[36624]: pgmap v13221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:50 smithi081 ceph-mon[31453]: pgmap v13222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:50 smithi153 ceph-mon[36569]: pgmap v13222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:50 smithi104 ceph-mon[36624]: pgmap v13222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:51 smithi081 ceph-mon[31453]: pgmap v13223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:51 smithi153 ceph-mon[36569]: pgmap v13223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:51 smithi104 ceph-mon[36624]: pgmap v13223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:53 smithi081 ceph-mon[31453]: pgmap v13224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:53 smithi153 ceph-mon[36569]: pgmap v13224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:53 smithi104 ceph-mon[36624]: pgmap v13224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:55 smithi081 ceph-mon[31453]: pgmap v13225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:55 smithi153 ceph-mon[36569]: pgmap v13225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:55 smithi104 ceph-mon[36624]: pgmap v13225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:56.450+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:04:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:56.450+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:04:56.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:04:56.450+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:04:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:57 smithi081 ceph-mon[31453]: pgmap v13226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:57 smithi153 ceph-mon[36569]: pgmap v13226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:04:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:57 smithi104 ceph-mon[36624]: pgmap v13226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:04:59 smithi081 ceph-mon[31453]: pgmap v13227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:04:59 smithi153 ceph-mon[36569]: pgmap v13227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:04:59 smithi104 ceph-mon[36624]: pgmap v13227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:01 smithi081 ceph-mon[31453]: pgmap v13228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:01 smithi153 ceph-mon[36569]: pgmap v13228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:01 smithi104 ceph-mon[36624]: pgmap v13228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:03 smithi081 ceph-mon[31453]: pgmap v13229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:03 smithi153 ceph-mon[36569]: pgmap v13229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:03 smithi104 ceph-mon[36624]: pgmap v13229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:05 smithi081 ceph-mon[31453]: pgmap v13230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:05 smithi153 ceph-mon[36569]: pgmap v13230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:05 smithi104 ceph-mon[36624]: pgmap v13230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:07 smithi081 ceph-mon[31453]: pgmap v13231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:07 smithi153 ceph-mon[36569]: pgmap v13231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:07 smithi104 ceph-mon[36624]: pgmap v13231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:09 smithi153 ceph-mon[36569]: pgmap v13232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:05:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:09 smithi104 ceph-mon[36624]: pgmap v13232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:05:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:09 smithi081 ceph-mon[31453]: pgmap v13232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:05:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:05:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:05:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:10 smithi153 ceph-mon[36569]: pgmap v13233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:10 smithi153 ceph-mon[36569]: pgmap v13234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:05:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:05:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:05:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:10 smithi104 ceph-mon[36624]: pgmap v13233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:10 smithi104 ceph-mon[36624]: pgmap v13234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:05:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:05:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:05:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:10 smithi081 ceph-mon[31453]: pgmap v13233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:10 smithi081 ceph-mon[31453]: pgmap v13234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:05:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:11.465+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:05:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:11.465+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:05:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:11.465+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:05:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:13 smithi081 ceph-mon[31453]: pgmap v13235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:13 smithi153 ceph-mon[36569]: pgmap v13235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:13 smithi104 ceph-mon[36624]: pgmap v13235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:15 smithi081 ceph-mon[31453]: pgmap v13236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:15 smithi153 ceph-mon[36569]: pgmap v13236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:15 smithi104 ceph-mon[36624]: pgmap v13236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:17 smithi081 ceph-mon[31453]: pgmap v13237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:17 smithi153 ceph-mon[36569]: pgmap v13237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:17 smithi104 ceph-mon[36624]: pgmap v13237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:19 smithi081 ceph-mon[31453]: pgmap v13238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:19 smithi153 ceph-mon[36569]: pgmap v13238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:19 smithi104 ceph-mon[36624]: pgmap v13238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:21.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:21 smithi081 ceph-mon[31453]: pgmap v13239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:21 smithi153 ceph-mon[36569]: pgmap v13239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:21 smithi104 ceph-mon[36624]: pgmap v13239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:23 smithi081 ceph-mon[31453]: pgmap v13240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:23 smithi153 ceph-mon[36569]: pgmap v13240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:23 smithi104 ceph-mon[36624]: pgmap v13240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:25 smithi081 ceph-mon[31453]: pgmap v13241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:25 smithi153 ceph-mon[36569]: pgmap v13241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:25 smithi104 ceph-mon[36624]: pgmap v13241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:26.468+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:05:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:26.469+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:05:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:26.469+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:05:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:27 smithi081 ceph-mon[31453]: pgmap v13242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:27 smithi153 ceph-mon[36569]: pgmap v13242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:27 smithi104 ceph-mon[36624]: pgmap v13242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:29 smithi081 ceph-mon[31453]: pgmap v13243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:29 smithi153 ceph-mon[36569]: pgmap v13243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:29 smithi104 ceph-mon[36624]: pgmap v13243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:31 smithi081 ceph-mon[31453]: pgmap v13244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:31 smithi153 ceph-mon[36569]: pgmap v13244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:31 smithi104 ceph-mon[36624]: pgmap v13244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:33 smithi153 ceph-mon[36569]: pgmap v13245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:33 smithi104 ceph-mon[36624]: pgmap v13245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:33 smithi081 ceph-mon[31453]: pgmap v13245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:35 smithi153 ceph-mon[36569]: pgmap v13246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:35 smithi104 ceph-mon[36624]: pgmap v13246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:35 smithi081 ceph-mon[31453]: pgmap v13246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:37 smithi153 ceph-mon[36569]: pgmap v13247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:37 smithi104 ceph-mon[36624]: pgmap v13247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:37 smithi081 ceph-mon[31453]: pgmap v13247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:39 smithi153 ceph-mon[36569]: pgmap v13248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:39 smithi104 ceph-mon[36624]: pgmap v13248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:39 smithi081 ceph-mon[31453]: pgmap v13248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:41.480+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:05:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:41.481+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:05:41.481 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:41.481+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:05:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:41 smithi153 ceph-mon[36569]: pgmap v13249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:41 smithi104 ceph-mon[36624]: pgmap v13249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:41 smithi081 ceph-mon[31453]: pgmap v13249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:43.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:43 smithi081 ceph-mon[31453]: pgmap v13250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:43 smithi153 ceph-mon[36569]: pgmap v13250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:43 smithi104 ceph-mon[36624]: pgmap v13250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:45 smithi153 ceph-mon[36569]: pgmap v13251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:45 smithi104 ceph-mon[36624]: pgmap v13251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:45 smithi081 ceph-mon[31453]: pgmap v13251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:47 smithi153 ceph-mon[36569]: pgmap v13252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:47 smithi104 ceph-mon[36624]: pgmap v13252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:47 smithi081 ceph-mon[31453]: pgmap v13252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:49 smithi153 ceph-mon[36569]: pgmap v13253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:49 smithi104 ceph-mon[36624]: pgmap v13253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:49 smithi081 ceph-mon[31453]: pgmap v13253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:51 smithi153 ceph-mon[36569]: pgmap v13254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:51 smithi104 ceph-mon[36624]: pgmap v13254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:51 smithi081 ceph-mon[31453]: pgmap v13254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:53 smithi153 ceph-mon[36569]: pgmap v13255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:53 smithi104 ceph-mon[36624]: pgmap v13255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:53 smithi081 ceph-mon[31453]: pgmap v13255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:55 smithi153 ceph-mon[36569]: pgmap v13256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:55 smithi104 ceph-mon[36624]: pgmap v13256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:55 smithi081 ceph-mon[31453]: pgmap v13256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:56.496+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:05:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:56.496+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:05:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:05:56.496+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:05:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:57 smithi153 ceph-mon[36569]: pgmap v13257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:57 smithi104 ceph-mon[36624]: pgmap v13257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:57 smithi081 ceph-mon[31453]: pgmap v13257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:05:59 smithi153 ceph-mon[36569]: pgmap v13258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:05:59 smithi104 ceph-mon[36624]: pgmap v13258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:05:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:05:59 smithi081 ceph-mon[31453]: pgmap v13258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:01 smithi153 ceph-mon[36569]: pgmap v13259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:01 smithi104 ceph-mon[36624]: pgmap v13259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:01 smithi081 ceph-mon[31453]: pgmap v13259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:03 smithi153 ceph-mon[36569]: pgmap v13260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:03 smithi104 ceph-mon[36624]: pgmap v13260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:03 smithi081 ceph-mon[31453]: pgmap v13260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:05 smithi153 ceph-mon[36569]: pgmap v13261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:05 smithi104 ceph-mon[36624]: pgmap v13261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:05 smithi081 ceph-mon[31453]: pgmap v13261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:07 smithi153 ceph-mon[36569]: pgmap v13262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:07 smithi104 ceph-mon[36624]: pgmap v13262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:07 smithi081 ceph-mon[31453]: pgmap v13262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:09 smithi153 ceph-mon[36569]: pgmap v13263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:09 smithi104 ceph-mon[36624]: pgmap v13263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:09 smithi081 ceph-mon[31453]: pgmap v13263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:11.511+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:06:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:11.511+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:06:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:11.511+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:06:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:11 smithi081 ceph-mon[31453]: pgmap v13264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:06:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:11 smithi153 ceph-mon[36569]: pgmap v13264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:06:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:11 smithi104 ceph-mon[36624]: pgmap v13264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:06:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:06:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:06:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:06:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:06:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:06:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:06:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:06:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:06:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:06:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:13 smithi153 ceph-mon[36569]: pgmap v13265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:13 smithi153 ceph-mon[36569]: pgmap v13266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:13.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:13 smithi081 ceph-mon[31453]: pgmap v13265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:13.808 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:13 smithi081 ceph-mon[31453]: pgmap v13266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:13 smithi104 ceph-mon[36624]: pgmap v13265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:13 smithi104 ceph-mon[36624]: pgmap v13266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:15 smithi153 ceph-mon[36569]: pgmap v13267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:15 smithi104 ceph-mon[36624]: pgmap v13267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:15 smithi081 ceph-mon[31453]: pgmap v13267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:17 smithi153 ceph-mon[36569]: pgmap v13268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:17 smithi104 ceph-mon[36624]: pgmap v13268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:17 smithi081 ceph-mon[31453]: pgmap v13268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:19 smithi153 ceph-mon[36569]: pgmap v13269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:19 smithi104 ceph-mon[36624]: pgmap v13269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:19 smithi081 ceph-mon[31453]: pgmap v13269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:21 smithi153 ceph-mon[36569]: pgmap v13270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:21 smithi104 ceph-mon[36624]: pgmap v13270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:21 smithi081 ceph-mon[31453]: pgmap v13270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:23 smithi153 ceph-mon[36569]: pgmap v13271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:23.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:23 smithi104 ceph-mon[36624]: pgmap v13271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:23 smithi081 ceph-mon[31453]: pgmap v13271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:25.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:25 smithi153 ceph-mon[36569]: pgmap v13272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:25 smithi104 ceph-mon[36624]: pgmap v13272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:25 smithi081 ceph-mon[31453]: pgmap v13272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:26.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:26.519+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:06:26.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:26.519+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:06:26.519 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:26.519+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:06:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:27 smithi153 ceph-mon[36569]: pgmap v13273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:27 smithi104 ceph-mon[36624]: pgmap v13273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:27 smithi081 ceph-mon[31453]: pgmap v13273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:29 smithi104 ceph-mon[36624]: pgmap v13274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:29 smithi081 ceph-mon[31453]: pgmap v13274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:29 smithi153 ceph-mon[36569]: pgmap v13274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:31 smithi104 ceph-mon[36624]: pgmap v13275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:31 smithi081 ceph-mon[31453]: pgmap v13275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:31 smithi153 ceph-mon[36569]: pgmap v13275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:33 smithi104 ceph-mon[36624]: pgmap v13276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:33 smithi081 ceph-mon[31453]: pgmap v13276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:33 smithi153 ceph-mon[36569]: pgmap v13276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:35 smithi104 ceph-mon[36624]: pgmap v13277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:35 smithi081 ceph-mon[31453]: pgmap v13277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:35 smithi153 ceph-mon[36569]: pgmap v13277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:37 smithi104 ceph-mon[36624]: pgmap v13278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:37 smithi081 ceph-mon[31453]: pgmap v13278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:37 smithi153 ceph-mon[36569]: pgmap v13278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:39.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:39 smithi153 ceph-mon[36569]: pgmap v13279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:39 smithi081 ceph-mon[31453]: pgmap v13279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:39 smithi104 ceph-mon[36624]: pgmap v13279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:41.535+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:41.535+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:06:41.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:41.535+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:06:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:41 smithi081 ceph-mon[31453]: pgmap v13280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:41 smithi153 ceph-mon[36569]: pgmap v13280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:41 smithi104 ceph-mon[36624]: pgmap v13280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:43.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:43 smithi104 ceph-mon[36624]: pgmap v13281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:43 smithi081 ceph-mon[31453]: pgmap v13281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:43 smithi153 ceph-mon[36569]: pgmap v13281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:45 smithi081 ceph-mon[31453]: pgmap v13282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:45 smithi153 ceph-mon[36569]: pgmap v13282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:45 smithi104 ceph-mon[36624]: pgmap v13282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:47 smithi081 ceph-mon[31453]: pgmap v13283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:47 smithi153 ceph-mon[36569]: pgmap v13283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:47 smithi104 ceph-mon[36624]: pgmap v13283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:49 smithi081 ceph-mon[31453]: pgmap v13284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:49 smithi153 ceph-mon[36569]: pgmap v13284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:49 smithi104 ceph-mon[36624]: pgmap v13284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:51 smithi081 ceph-mon[31453]: pgmap v13285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:51 smithi153 ceph-mon[36569]: pgmap v13285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:51 smithi104 ceph-mon[36624]: pgmap v13285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:53 smithi081 ceph-mon[31453]: pgmap v13286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:53 smithi153 ceph-mon[36569]: pgmap v13286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:53 smithi104 ceph-mon[36624]: pgmap v13286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:55 smithi081 ceph-mon[31453]: pgmap v13287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:55 smithi153 ceph-mon[36569]: pgmap v13287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:55 smithi104 ceph-mon[36624]: pgmap v13287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:56.546+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:06:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:56.546+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:06:56.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:06:56.546+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:06:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:57 smithi081 ceph-mon[31453]: pgmap v13288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:57 smithi153 ceph-mon[36569]: pgmap v13288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:57 smithi104 ceph-mon[36624]: pgmap v13288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:06:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:06:59 smithi081 ceph-mon[31453]: pgmap v13289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:06:59 smithi153 ceph-mon[36569]: pgmap v13289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:06:59 smithi104 ceph-mon[36624]: pgmap v13289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:01 smithi081 ceph-mon[31453]: pgmap v13290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:01 smithi153 ceph-mon[36569]: pgmap v13290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:01 smithi104 ceph-mon[36624]: pgmap v13290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:03 smithi081 ceph-mon[31453]: pgmap v13291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:03 smithi153 ceph-mon[36569]: pgmap v13291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:03 smithi104 ceph-mon[36624]: pgmap v13291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:05 smithi081 ceph-mon[31453]: pgmap v13292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:05 smithi153 ceph-mon[36569]: pgmap v13292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:05 smithi104 ceph-mon[36624]: pgmap v13292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:07 smithi153 ceph-mon[36569]: pgmap v13293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:07 smithi104 ceph-mon[36624]: pgmap v13293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:07 smithi081 ceph-mon[31453]: pgmap v13293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:08 smithi153 ceph-mon[36569]: pgmap v13294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:08 smithi104 ceph-mon[36624]: pgmap v13294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:08 smithi081 ceph-mon[31453]: pgmap v13294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:10 smithi081 ceph-mon[31453]: pgmap v13295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:10 smithi153 ceph-mon[36569]: pgmap v13295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:10 smithi104 ceph-mon[36624]: pgmap v13295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:11.551+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:07:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:11.551+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:07:11.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:11.551+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:07:13.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:12 smithi081 ceph-mon[31453]: pgmap v13296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:13.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:07:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:12 smithi153 ceph-mon[36569]: pgmap v13296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:07:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:12 smithi104 ceph-mon[36624]: pgmap v13296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:07:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:07:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:07:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:13 smithi153 ceph-mon[36569]: pgmap v13297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:13 smithi153 ceph-mon[36569]: pgmap v13298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:07:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:07:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:07:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:13 smithi104 ceph-mon[36624]: pgmap v13297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:13 smithi104 ceph-mon[36624]: pgmap v13298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:07:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:07:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:07:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:13 smithi081 ceph-mon[31453]: pgmap v13297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:13 smithi081 ceph-mon[31453]: pgmap v13298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:07:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:16.777+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:07:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:16.777+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:07:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:16.777+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 191 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:07:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:16.777+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 191 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:07:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:16 smithi153 ceph-mon[36569]: pgmap v13299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:16 smithi104 ceph-mon[36624]: pgmap v13299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:16 smithi081 ceph-mon[31453]: pgmap v13299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:18 smithi153 ceph-mon[36569]: pgmap v13300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:18 smithi104 ceph-mon[36624]: pgmap v13300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:18 smithi081 ceph-mon[31453]: pgmap v13300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:20 smithi153 ceph-mon[36569]: pgmap v13301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:20 smithi104 ceph-mon[36624]: pgmap v13301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:20 smithi081 ceph-mon[31453]: pgmap v13301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:22 smithi153 ceph-mon[36569]: pgmap v13302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:22 smithi104 ceph-mon[36624]: pgmap v13302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:22 smithi081 ceph-mon[31453]: pgmap v13302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:24 smithi153 ceph-mon[36569]: pgmap v13303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:24 smithi104 ceph-mon[36624]: pgmap v13303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:24 smithi081 ceph-mon[31453]: pgmap v13303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:26.563 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:26.563+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:07:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:26.563+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:07:26.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:26.563+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:07:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:26 smithi153 ceph-mon[36569]: pgmap v13304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:26 smithi104 ceph-mon[36624]: pgmap v13304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:26 smithi081 ceph-mon[31453]: pgmap v13304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:28 smithi153 ceph-mon[36569]: pgmap v13305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:28 smithi104 ceph-mon[36624]: pgmap v13305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:28 smithi081 ceph-mon[31453]: pgmap v13305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:30 smithi153 ceph-mon[36569]: pgmap v13306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:30 smithi104 ceph-mon[36624]: pgmap v13306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:30 smithi081 ceph-mon[31453]: pgmap v13306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:32 smithi153 ceph-mon[36569]: pgmap v13307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:32 smithi104 ceph-mon[36624]: pgmap v13307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:32 smithi081 ceph-mon[31453]: pgmap v13307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:34 smithi153 ceph-mon[36569]: pgmap v13308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:34 smithi104 ceph-mon[36624]: pgmap v13308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:34 smithi081 ceph-mon[31453]: pgmap v13308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:36 smithi153 ceph-mon[36569]: pgmap v13309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:36 smithi104 ceph-mon[36624]: pgmap v13309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:36 smithi081 ceph-mon[31453]: pgmap v13309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:38 smithi153 ceph-mon[36569]: pgmap v13310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:38 smithi104 ceph-mon[36624]: pgmap v13310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:38 smithi081 ceph-mon[31453]: pgmap v13310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:40 smithi153 ceph-mon[36569]: pgmap v13311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:40 smithi104 ceph-mon[36624]: pgmap v13311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:40 smithi081 ceph-mon[31453]: pgmap v13311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:41.563+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:41.564+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:07:41.565 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:41.564+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:07:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:42 smithi153 ceph-mon[36569]: pgmap v13312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:42 smithi104 ceph-mon[36624]: pgmap v13312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:42 smithi081 ceph-mon[31453]: pgmap v13312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:44 smithi153 ceph-mon[36569]: pgmap v13313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:44 smithi104 ceph-mon[36624]: pgmap v13313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:44 smithi081 ceph-mon[31453]: pgmap v13313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:46 smithi153 ceph-mon[36569]: pgmap v13314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:46 smithi104 ceph-mon[36624]: pgmap v13314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:46 smithi081 ceph-mon[31453]: pgmap v13314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:48 smithi153 ceph-mon[36569]: pgmap v13315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:48 smithi104 ceph-mon[36624]: pgmap v13315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:48 smithi081 ceph-mon[31453]: pgmap v13315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:50 smithi153 ceph-mon[36569]: pgmap v13316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:50 smithi104 ceph-mon[36624]: pgmap v13316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:50 smithi081 ceph-mon[31453]: pgmap v13316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:52 smithi153 ceph-mon[36569]: pgmap v13317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:52 smithi104 ceph-mon[36624]: pgmap v13317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:52 smithi081 ceph-mon[31453]: pgmap v13317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:54 smithi153 ceph-mon[36569]: pgmap v13318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:54 smithi104 ceph-mon[36624]: pgmap v13318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:54 smithi081 ceph-mon[31453]: pgmap v13318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:56.573+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:07:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:56.573+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:07:56.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:07:56.573+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:07:56.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:56 smithi153 ceph-mon[36569]: pgmap v13319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:56 smithi104 ceph-mon[36624]: pgmap v13319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:56 smithi081 ceph-mon[31453]: pgmap v13319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:07:58 smithi153 ceph-mon[36569]: pgmap v13320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:07:58 smithi104 ceph-mon[36624]: pgmap v13320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:07:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:07:58 smithi081 ceph-mon[31453]: pgmap v13320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:00 smithi153 ceph-mon[36569]: pgmap v13321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:00 smithi104 ceph-mon[36624]: pgmap v13321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:00 smithi081 ceph-mon[31453]: pgmap v13321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:02 smithi104 ceph-mon[36624]: pgmap v13322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:02 smithi081 ceph-mon[31453]: pgmap v13322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:02 smithi153 ceph-mon[36569]: pgmap v13322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:04 smithi104 ceph-mon[36624]: pgmap v13323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:04 smithi081 ceph-mon[31453]: pgmap v13323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:04 smithi153 ceph-mon[36569]: pgmap v13323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:06 smithi104 ceph-mon[36624]: pgmap v13324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:06 smithi081 ceph-mon[31453]: pgmap v13324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:06 smithi153 ceph-mon[36569]: pgmap v13324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:08 smithi104 ceph-mon[36624]: pgmap v13325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:08 smithi081 ceph-mon[31453]: pgmap v13325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:08 smithi153 ceph-mon[36569]: pgmap v13325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:10 smithi104 ceph-mon[36624]: pgmap v13326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:10 smithi081 ceph-mon[31453]: pgmap v13326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:10 smithi153 ceph-mon[36569]: pgmap v13326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:11.579+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:08:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:11.579+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:08:11.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:11.579+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:08:12.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:12 smithi153 ceph-mon[36569]: pgmap v13327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:12 smithi081 ceph-mon[31453]: pgmap v13327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:12 smithi104 ceph-mon[36624]: pgmap v13327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:14.883 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: pgmap v13328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:14.883 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:08:14.883 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.883 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.883 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: pgmap v13328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: pgmap v13328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:15 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:08:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:15 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:08:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:15 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:08:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:15 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:08:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:15 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:08:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:15 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:08:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:15 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:08:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:15 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:08:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:15 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:08:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:15 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:08:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:15 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:08:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:15 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:08:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: pgmap v13329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: pgmap v13330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: pgmap v13331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:16.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: pgmap v13329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: pgmap v13330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:17.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: pgmap v13331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: pgmap v13329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:08:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:08:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:08:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:08:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:08:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: pgmap v13330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: pgmap v13331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:08:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:17 smithi153 ceph-mon[36569]: pgmap v13332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:17 smithi104 ceph-mon[36624]: pgmap v13332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:17 smithi081 ceph-mon[31453]: pgmap v13332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:19 smithi153 ceph-mon[36569]: pgmap v13333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:19 smithi104 ceph-mon[36624]: pgmap v13333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:19 smithi081 ceph-mon[31453]: pgmap v13333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:21 smithi153 ceph-mon[36569]: pgmap v13334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:21 smithi104 ceph-mon[36624]: pgmap v13334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:21 smithi081 ceph-mon[31453]: pgmap v13334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:23 smithi153 ceph-mon[36569]: pgmap v13335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:23 smithi104 ceph-mon[36624]: pgmap v13335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:23 smithi081 ceph-mon[31453]: pgmap v13335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:25 smithi153 ceph-mon[36569]: pgmap v13336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:25 smithi104 ceph-mon[36624]: pgmap v13336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:25 smithi081 ceph-mon[31453]: pgmap v13336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:26.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:26.594+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:08:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:26.594+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:08:26.595 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:26.594+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:08:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:27 smithi153 ceph-mon[36569]: pgmap v13337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:27 smithi104 ceph-mon[36624]: pgmap v13337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:27 smithi081 ceph-mon[31453]: pgmap v13337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:29 smithi153 ceph-mon[36569]: pgmap v13338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:29 smithi104 ceph-mon[36624]: pgmap v13338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:29 smithi081 ceph-mon[31453]: pgmap v13338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:31 smithi153 ceph-mon[36569]: pgmap v13339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:31 smithi104 ceph-mon[36624]: pgmap v13339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:31 smithi081 ceph-mon[31453]: pgmap v13339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:33 smithi153 ceph-mon[36569]: pgmap v13340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:33 smithi104 ceph-mon[36624]: pgmap v13340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:33 smithi081 ceph-mon[31453]: pgmap v13340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:35 smithi153 ceph-mon[36569]: pgmap v13341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:35 smithi104 ceph-mon[36624]: pgmap v13341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:35 smithi081 ceph-mon[31453]: pgmap v13341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:37 smithi153 ceph-mon[36569]: pgmap v13342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:37 smithi104 ceph-mon[36624]: pgmap v13342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:37 smithi081 ceph-mon[31453]: pgmap v13342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:39 smithi153 ceph-mon[36569]: pgmap v13343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:39 smithi104 ceph-mon[36624]: pgmap v13343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:39 smithi081 ceph-mon[31453]: pgmap v13343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:41.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:41.609+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:08:41.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:41.609+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:08:41.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:41.609+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:08:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:41 smithi153 ceph-mon[36569]: pgmap v13344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:41 smithi104 ceph-mon[36624]: pgmap v13344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:41 smithi081 ceph-mon[31453]: pgmap v13344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:43 smithi153 ceph-mon[36569]: pgmap v13345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:43 smithi104 ceph-mon[36624]: pgmap v13345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:43 smithi081 ceph-mon[31453]: pgmap v13345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:45 smithi153 ceph-mon[36569]: pgmap v13346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:45 smithi104 ceph-mon[36624]: pgmap v13346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:45 smithi081 ceph-mon[31453]: pgmap v13346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:46.792+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:08:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:46.792+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:08:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:46.792+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 192 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:08:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:46.792+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 192 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:08:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:47 smithi153 ceph-mon[36569]: pgmap v13347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:47 smithi104 ceph-mon[36624]: pgmap v13347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:47 smithi081 ceph-mon[31453]: pgmap v13347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:49 smithi153 ceph-mon[36569]: pgmap v13348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:49 smithi104 ceph-mon[36624]: pgmap v13348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:49 smithi081 ceph-mon[31453]: pgmap v13348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:51 smithi153 ceph-mon[36569]: pgmap v13349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:51 smithi104 ceph-mon[36624]: pgmap v13349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:51 smithi081 ceph-mon[31453]: pgmap v13349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:53 smithi153 ceph-mon[36569]: pgmap v13350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:53 smithi104 ceph-mon[36624]: pgmap v13350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:53 smithi081 ceph-mon[31453]: pgmap v13350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:55 smithi153 ceph-mon[36569]: pgmap v13351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:55 smithi104 ceph-mon[36624]: pgmap v13351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:55 smithi081 ceph-mon[31453]: pgmap v13351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:56.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:56.622+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:08:56.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:56.623+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:08:56.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:08:56.623+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:08:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:57 smithi153 ceph-mon[36569]: pgmap v13352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:57 smithi104 ceph-mon[36624]: pgmap v13352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:08:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:57 smithi081 ceph-mon[31453]: pgmap v13352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:08:59 smithi153 ceph-mon[36569]: pgmap v13353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:08:59 smithi104 ceph-mon[36624]: pgmap v13353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:08:59 smithi081 ceph-mon[31453]: pgmap v13353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:02 smithi153 ceph-mon[36569]: pgmap v13354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:02 smithi104 ceph-mon[36624]: pgmap v13354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:02 smithi081 ceph-mon[31453]: pgmap v13354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:04 smithi153 ceph-mon[36569]: pgmap v13355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:04 smithi104 ceph-mon[36624]: pgmap v13355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:04 smithi081 ceph-mon[31453]: pgmap v13355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:06 smithi153 ceph-mon[36569]: pgmap v13356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:06 smithi104 ceph-mon[36624]: pgmap v13356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:06 smithi081 ceph-mon[31453]: pgmap v13356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:08 smithi153 ceph-mon[36569]: pgmap v13357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:08 smithi104 ceph-mon[36624]: pgmap v13357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:08 smithi081 ceph-mon[31453]: pgmap v13357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:10 smithi153 ceph-mon[36569]: pgmap v13358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:10 smithi104 ceph-mon[36624]: pgmap v13358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:10 smithi081 ceph-mon[31453]: pgmap v13358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:11.625+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:09:11.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:11.625+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:09:11.626 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:11.625+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:09:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:12 smithi153 ceph-mon[36569]: pgmap v13359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:12 smithi104 ceph-mon[36624]: pgmap v13359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:12 smithi081 ceph-mon[31453]: pgmap v13359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:14 smithi153 ceph-mon[36569]: pgmap v13360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:14 smithi104 ceph-mon[36624]: pgmap v13360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:14 smithi081 ceph-mon[31453]: pgmap v13360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:16 smithi153 ceph-mon[36569]: pgmap v13361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:09:17.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:16 smithi104 ceph-mon[36624]: pgmap v13361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:09:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:16 smithi081 ceph-mon[31453]: pgmap v13361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: pgmap v13362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: pgmap v13363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: pgmap v13362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: pgmap v13363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:09:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:09:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: pgmap v13362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: pgmap v13363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:09:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:20 smithi081 ceph-mon[31453]: pgmap v13364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:20 smithi153 ceph-mon[36569]: pgmap v13364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:20 smithi104 ceph-mon[36624]: pgmap v13364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:22 smithi081 ceph-mon[31453]: pgmap v13365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:22 smithi153 ceph-mon[36569]: pgmap v13365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:22 smithi104 ceph-mon[36624]: pgmap v13365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:24 smithi081 ceph-mon[31453]: pgmap v13366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:24 smithi153 ceph-mon[36569]: pgmap v13366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:24 smithi104 ceph-mon[36624]: pgmap v13366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:26.640+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:09:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:26.640+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:09:26.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:26.640+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:09:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:26 smithi081 ceph-mon[31453]: pgmap v13367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:26 smithi153 ceph-mon[36569]: pgmap v13367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:26 smithi104 ceph-mon[36624]: pgmap v13367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:28 smithi081 ceph-mon[31453]: pgmap v13368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:28 smithi153 ceph-mon[36569]: pgmap v13368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:28 smithi104 ceph-mon[36624]: pgmap v13368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:30 smithi081 ceph-mon[31453]: pgmap v13369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:30 smithi153 ceph-mon[36569]: pgmap v13369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:30 smithi104 ceph-mon[36624]: pgmap v13369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:32 smithi081 ceph-mon[31453]: pgmap v13370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:32 smithi153 ceph-mon[36569]: pgmap v13370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:32 smithi104 ceph-mon[36624]: pgmap v13370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:34 smithi081 ceph-mon[31453]: pgmap v13371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:34 smithi153 ceph-mon[36569]: pgmap v13371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:34 smithi104 ceph-mon[36624]: pgmap v13371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:36 smithi081 ceph-mon[31453]: pgmap v13372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:36 smithi153 ceph-mon[36569]: pgmap v13372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:36 smithi104 ceph-mon[36624]: pgmap v13372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:38 smithi153 ceph-mon[36569]: pgmap v13373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:38 smithi104 ceph-mon[36624]: pgmap v13373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:38 smithi081 ceph-mon[31453]: pgmap v13373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:40 smithi153 ceph-mon[36569]: pgmap v13374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:40 smithi104 ceph-mon[36624]: pgmap v13374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:40 smithi081 ceph-mon[31453]: pgmap v13374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:41.648+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:09:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:41.648+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:09:41.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:41.649+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:09:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:42 smithi153 ceph-mon[36569]: pgmap v13375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:42 smithi104 ceph-mon[36624]: pgmap v13375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:42 smithi081 ceph-mon[31453]: pgmap v13375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:44 smithi153 ceph-mon[36569]: pgmap v13376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:44 smithi104 ceph-mon[36624]: pgmap v13376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:44 smithi081 ceph-mon[31453]: pgmap v13376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:46 smithi153 ceph-mon[36569]: pgmap v13377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:46 smithi104 ceph-mon[36624]: pgmap v13377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:46 smithi081 ceph-mon[31453]: pgmap v13377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:48 smithi153 ceph-mon[36569]: pgmap v13378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:48 smithi104 ceph-mon[36624]: pgmap v13378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:48 smithi081 ceph-mon[31453]: pgmap v13378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:50 smithi153 ceph-mon[36569]: pgmap v13379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:50 smithi104 ceph-mon[36624]: pgmap v13379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:50 smithi081 ceph-mon[31453]: pgmap v13379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:52 smithi153 ceph-mon[36569]: pgmap v13380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:52 smithi104 ceph-mon[36624]: pgmap v13380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:52 smithi081 ceph-mon[31453]: pgmap v13380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:54 smithi153 ceph-mon[36569]: pgmap v13381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:54 smithi104 ceph-mon[36624]: pgmap v13381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:54 smithi081 ceph-mon[31453]: pgmap v13381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:56.663 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:56.663+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:09:56.663 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:56.663+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:09:56.663 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:09:56.663+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:09:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:56 smithi153 ceph-mon[36569]: pgmap v13382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:56 smithi104 ceph-mon[36624]: pgmap v13382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:56 smithi081 ceph-mon[31453]: pgmap v13382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:09:58 smithi104 ceph-mon[36624]: pgmap v13383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:09:58 smithi081 ceph-mon[31453]: pgmap v13383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:09:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:09:58 smithi153 ceph-mon[36569]: pgmap v13383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:00 smithi104 ceph-mon[36624]: pgmap v13384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:10:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:00 smithi081 ceph-mon[31453]: pgmap v13384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:10:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:00 smithi153 ceph-mon[36569]: pgmap v13384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:10:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:02 smithi104 ceph-mon[36624]: pgmap v13385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:02 smithi081 ceph-mon[31453]: pgmap v13385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:02 smithi153 ceph-mon[36569]: pgmap v13385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:04 smithi104 ceph-mon[36624]: pgmap v13386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:04 smithi081 ceph-mon[31453]: pgmap v13386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:04 smithi153 ceph-mon[36569]: pgmap v13386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:06 smithi104 ceph-mon[36624]: pgmap v13387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:06 smithi081 ceph-mon[31453]: pgmap v13387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:06 smithi153 ceph-mon[36569]: pgmap v13387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:08 smithi081 ceph-mon[31453]: pgmap v13388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:08 smithi153 ceph-mon[36569]: pgmap v13388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:08 smithi104 ceph-mon[36624]: pgmap v13388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:10 smithi081 ceph-mon[31453]: pgmap v13389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:10 smithi153 ceph-mon[36569]: pgmap v13389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:10 smithi104 ceph-mon[36624]: pgmap v13389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:11.679+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:10:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:11.679+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:10:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:11.679+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:10:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:12 smithi081 ceph-mon[31453]: pgmap v13390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:12 smithi153 ceph-mon[36569]: pgmap v13390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:12 smithi104 ceph-mon[36624]: pgmap v13390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:14.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:14 smithi153 ceph-mon[36569]: pgmap v13391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:14 smithi081 ceph-mon[31453]: pgmap v13391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:14 smithi104 ceph-mon[36624]: pgmap v13391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:16.806+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:10:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:16.806+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:10:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:16.807+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 193 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:10:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:16.807+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 193 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:10:16.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:16 smithi104 ceph-mon[36624]: pgmap v13392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:16 smithi081 ceph-mon[31453]: pgmap v13392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:16 smithi153 ceph-mon[36569]: pgmap v13392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:10:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:10:18.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:10:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:18 smithi081 ceph-mon[31453]: pgmap v13393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:18 smithi153 ceph-mon[36569]: pgmap v13393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:18 smithi104 ceph-mon[36624]: pgmap v13393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:10:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:10:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:10:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:10:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:10:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:10:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:10:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:10:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:10:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:20 smithi081 ceph-mon[31453]: pgmap v13394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:20 smithi081 ceph-mon[31453]: pgmap v13395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:20 smithi153 ceph-mon[36569]: pgmap v13394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:20 smithi153 ceph-mon[36569]: pgmap v13395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:20 smithi104 ceph-mon[36624]: pgmap v13394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:20 smithi104 ceph-mon[36624]: pgmap v13395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:22 smithi081 ceph-mon[31453]: pgmap v13396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:22 smithi153 ceph-mon[36569]: pgmap v13396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:22 smithi104 ceph-mon[36624]: pgmap v13396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:24 smithi081 ceph-mon[31453]: pgmap v13397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:24 smithi153 ceph-mon[36569]: pgmap v13397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:24 smithi104 ceph-mon[36624]: pgmap v13397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:26.686+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:10:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:26.686+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:10:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:26.686+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:10:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:26 smithi081 ceph-mon[31453]: pgmap v13398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:26 smithi153 ceph-mon[36569]: pgmap v13398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:26 smithi104 ceph-mon[36624]: pgmap v13398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:28 smithi081 ceph-mon[31453]: pgmap v13399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:28 smithi153 ceph-mon[36569]: pgmap v13399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:28 smithi104 ceph-mon[36624]: pgmap v13399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:30 smithi081 ceph-mon[31453]: pgmap v13400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:30 smithi153 ceph-mon[36569]: pgmap v13400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:30 smithi104 ceph-mon[36624]: pgmap v13400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:32 smithi153 ceph-mon[36569]: pgmap v13401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:32 smithi104 ceph-mon[36624]: pgmap v13401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:32 smithi081 ceph-mon[31453]: pgmap v13401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:34 smithi153 ceph-mon[36569]: pgmap v13402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:34 smithi104 ceph-mon[36624]: pgmap v13402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:34 smithi081 ceph-mon[31453]: pgmap v13402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:36 smithi153 ceph-mon[36569]: pgmap v13403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:36 smithi104 ceph-mon[36624]: pgmap v13403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:36 smithi081 ceph-mon[31453]: pgmap v13403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:38 smithi153 ceph-mon[36569]: pgmap v13404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:38 smithi104 ceph-mon[36624]: pgmap v13404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:38 smithi081 ceph-mon[31453]: pgmap v13404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:40 smithi153 ceph-mon[36569]: pgmap v13405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:40 smithi104 ceph-mon[36624]: pgmap v13405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:40 smithi081 ceph-mon[31453]: pgmap v13405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:41.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:41.689+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:10:41.690 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:41.690+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:10:41.690 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:41.690+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:10:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:42 smithi153 ceph-mon[36569]: pgmap v13406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:42 smithi104 ceph-mon[36624]: pgmap v13406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:42 smithi081 ceph-mon[31453]: pgmap v13406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:44 smithi153 ceph-mon[36569]: pgmap v13407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:44 smithi104 ceph-mon[36624]: pgmap v13407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:44 smithi081 ceph-mon[31453]: pgmap v13407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:46.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:46 smithi081 ceph-mon[31453]: pgmap v13408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:46 smithi153 ceph-mon[36569]: pgmap v13408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:46 smithi104 ceph-mon[36624]: pgmap v13408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:48 smithi153 ceph-mon[36569]: pgmap v13409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:48 smithi104 ceph-mon[36624]: pgmap v13409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:48 smithi081 ceph-mon[31453]: pgmap v13409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:49 smithi153 ceph-mon[36569]: pgmap v13410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:49 smithi104 ceph-mon[36624]: pgmap v13410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:49 smithi081 ceph-mon[31453]: pgmap v13410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:51 smithi153 ceph-mon[36569]: pgmap v13411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:51 smithi104 ceph-mon[36624]: pgmap v13411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:51 smithi081 ceph-mon[31453]: pgmap v13411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:53 smithi153 ceph-mon[36569]: pgmap v13412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:53 smithi104 ceph-mon[36624]: pgmap v13412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:53 smithi081 ceph-mon[31453]: pgmap v13412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:55 smithi153 ceph-mon[36569]: pgmap v13413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:55 smithi104 ceph-mon[36624]: pgmap v13413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:55 smithi081 ceph-mon[31453]: pgmap v13413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:56.702+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:10:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:56.702+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:10:56.702 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:10:56.702+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:10:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:57 smithi153 ceph-mon[36569]: pgmap v13414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:57 smithi104 ceph-mon[36624]: pgmap v13414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:10:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:57 smithi081 ceph-mon[31453]: pgmap v13414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:10:59 smithi153 ceph-mon[36569]: pgmap v13415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:10:59 smithi104 ceph-mon[36624]: pgmap v13415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:10:59 smithi081 ceph-mon[31453]: pgmap v13415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:01 smithi153 ceph-mon[36569]: pgmap v13416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:01 smithi104 ceph-mon[36624]: pgmap v13416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:01 smithi081 ceph-mon[31453]: pgmap v13416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:03 smithi153 ceph-mon[36569]: pgmap v13417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:03 smithi104 ceph-mon[36624]: pgmap v13417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:03 smithi081 ceph-mon[31453]: pgmap v13417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:06.249 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:05 smithi081 ceph-mon[31453]: pgmap v13418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:05 smithi153 ceph-mon[36569]: pgmap v13418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:05 smithi104 ceph-mon[36624]: pgmap v13418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:07 smithi153 ceph-mon[36569]: pgmap v13419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:07 smithi104 ceph-mon[36624]: pgmap v13419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:07 smithi081 ceph-mon[31453]: pgmap v13419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:09 smithi153 ceph-mon[36569]: pgmap v13420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:09 smithi104 ceph-mon[36624]: pgmap v13420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:09 smithi081 ceph-mon[31453]: pgmap v13420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:11.717+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:11:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:11.717+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:11:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:11.717+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:11:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:11 smithi153 ceph-mon[36569]: pgmap v13421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:11 smithi104 ceph-mon[36624]: pgmap v13421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:11 smithi081 ceph-mon[31453]: pgmap v13421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:13 smithi153 ceph-mon[36569]: pgmap v13422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:13 smithi104 ceph-mon[36624]: pgmap v13422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:13 smithi081 ceph-mon[31453]: pgmap v13422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:15 smithi153 ceph-mon[36569]: pgmap v13423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:15 smithi104 ceph-mon[36624]: pgmap v13423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:15 smithi081 ceph-mon[31453]: pgmap v13423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:17 smithi153 ceph-mon[36569]: pgmap v13424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:17 smithi104 ceph-mon[36624]: pgmap v13424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:17 smithi081 ceph-mon[31453]: pgmap v13424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:19 smithi153 ceph-mon[36569]: pgmap v13425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:11:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:19 smithi104 ceph-mon[36624]: pgmap v13425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:11:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:19 smithi081 ceph-mon[31453]: pgmap v13425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:11:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:11:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:11:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:21 smithi153 ceph-mon[36569]: pgmap v13426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:21 smithi153 ceph-mon[36569]: pgmap v13427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:11:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:11:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:11:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:21 smithi104 ceph-mon[36624]: pgmap v13426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:21 smithi104 ceph-mon[36624]: pgmap v13427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:11:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:11:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:11:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:21 smithi081 ceph-mon[31453]: pgmap v13426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:21 smithi081 ceph-mon[31453]: pgmap v13427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:11:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:23 smithi153 ceph-mon[36569]: pgmap v13428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:23 smithi104 ceph-mon[36624]: pgmap v13428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:23 smithi081 ceph-mon[31453]: pgmap v13428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:25 smithi153 ceph-mon[36569]: pgmap v13429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:25 smithi104 ceph-mon[36624]: pgmap v13429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:25 smithi081 ceph-mon[31453]: pgmap v13429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:26.719+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:11:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:26.719+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:11:26.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:26.719+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:11:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:27 smithi104 ceph-mon[36624]: pgmap v13430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:27 smithi081 ceph-mon[31453]: pgmap v13430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:27 smithi153 ceph-mon[36569]: pgmap v13430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:29 smithi104 ceph-mon[36624]: pgmap v13431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:29 smithi081 ceph-mon[31453]: pgmap v13431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:29 smithi153 ceph-mon[36569]: pgmap v13431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:31 smithi104 ceph-mon[36624]: pgmap v13432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:31 smithi081 ceph-mon[31453]: pgmap v13432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:31 smithi153 ceph-mon[36569]: pgmap v13432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:33 smithi104 ceph-mon[36624]: pgmap v13433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:33 smithi081 ceph-mon[31453]: pgmap v13433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:33 smithi153 ceph-mon[36569]: pgmap v13433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:35 smithi104 ceph-mon[36624]: pgmap v13434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:35 smithi081 ceph-mon[31453]: pgmap v13434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:35 smithi153 ceph-mon[36569]: pgmap v13434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:37 smithi081 ceph-mon[31453]: pgmap v13435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:37 smithi153 ceph-mon[36569]: pgmap v13435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:37 smithi104 ceph-mon[36624]: pgmap v13435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:39 smithi081 ceph-mon[31453]: pgmap v13436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:39 smithi153 ceph-mon[36569]: pgmap v13436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:39 smithi104 ceph-mon[36624]: pgmap v13436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:41.721+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:11:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:41.721+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:11:41.722 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:41.721+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:11:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:41 smithi081 ceph-mon[31453]: pgmap v13437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:41 smithi153 ceph-mon[36569]: pgmap v13437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:41 smithi104 ceph-mon[36624]: pgmap v13437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:43 smithi081 ceph-mon[31453]: pgmap v13438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:43 smithi153 ceph-mon[36569]: pgmap v13438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:43 smithi104 ceph-mon[36624]: pgmap v13438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:45.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:45 smithi153 ceph-mon[36569]: pgmap v13439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:45 smithi081 ceph-mon[31453]: pgmap v13439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:45 smithi104 ceph-mon[36624]: pgmap v13439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:47.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:47 smithi104 ceph-mon[36624]: pgmap v13440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:47 smithi081 ceph-mon[31453]: pgmap v13440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:47 smithi153 ceph-mon[36569]: pgmap v13440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:49 smithi081 ceph-mon[31453]: pgmap v13441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:49 smithi153 ceph-mon[36569]: pgmap v13441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:49 smithi104 ceph-mon[36624]: pgmap v13441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:51 smithi081 ceph-mon[31453]: pgmap v13442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:51 smithi153 ceph-mon[36569]: pgmap v13442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:51 smithi104 ceph-mon[36624]: pgmap v13442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:53 smithi081 ceph-mon[31453]: pgmap v13443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:53 smithi153 ceph-mon[36569]: pgmap v13443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:53 smithi104 ceph-mon[36624]: pgmap v13443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:55 smithi081 ceph-mon[31453]: pgmap v13444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:55 smithi153 ceph-mon[36569]: pgmap v13444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:55 smithi104 ceph-mon[36624]: pgmap v13444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:56.723+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:11:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:56.723+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:11:56.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:11:56.723+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:11:57.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:57 smithi081 ceph-mon[31453]: pgmap v13445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:57 smithi153 ceph-mon[36569]: pgmap v13445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:57 smithi104 ceph-mon[36624]: pgmap v13445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:11:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:11:59 smithi081 ceph-mon[31453]: pgmap v13446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:11:59 smithi153 ceph-mon[36569]: pgmap v13446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:11:59 smithi104 ceph-mon[36624]: pgmap v13446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:01 smithi081 ceph-mon[31453]: pgmap v13447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:01 smithi153 ceph-mon[36569]: pgmap v13447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:01 smithi104 ceph-mon[36624]: pgmap v13447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:03 smithi081 ceph-mon[31453]: pgmap v13448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:03 smithi153 ceph-mon[36569]: pgmap v13448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:03 smithi104 ceph-mon[36624]: pgmap v13448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:05 smithi153 ceph-mon[36569]: pgmap v13449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:05 smithi104 ceph-mon[36624]: pgmap v13449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:05 smithi081 ceph-mon[31453]: pgmap v13449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:07 smithi153 ceph-mon[36569]: pgmap v13450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:07 smithi104 ceph-mon[36624]: pgmap v13450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:07 smithi081 ceph-mon[31453]: pgmap v13450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:09 smithi153 ceph-mon[36569]: pgmap v13451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:09 smithi104 ceph-mon[36624]: pgmap v13451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:09 smithi081 ceph-mon[31453]: pgmap v13451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:11.735+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:12:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:11.735+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:12:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:11.735+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:12:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:11 smithi153 ceph-mon[36569]: pgmap v13452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:11 smithi104 ceph-mon[36624]: pgmap v13452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:11 smithi081 ceph-mon[31453]: pgmap v13452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:14.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:13 smithi153 ceph-mon[36569]: pgmap v13453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:14.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:13 smithi104 ceph-mon[36624]: pgmap v13453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:13 smithi081 ceph-mon[31453]: pgmap v13453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:15 smithi153 ceph-mon[36569]: pgmap v13454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:15 smithi104 ceph-mon[36624]: pgmap v13454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:16.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:15 smithi081 ceph-mon[31453]: pgmap v13454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:17.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:17 smithi081 ceph-mon[31453]: pgmap v13455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:17 smithi153 ceph-mon[36569]: pgmap v13455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:17 smithi104 ceph-mon[36624]: pgmap v13455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:19 smithi153 ceph-mon[36569]: pgmap v13456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:19 smithi104 ceph-mon[36624]: pgmap v13456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:19 smithi081 ceph-mon[31453]: pgmap v13456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:21 smithi153 ceph-mon[36569]: pgmap v13457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:12:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:21 smithi104 ceph-mon[36624]: pgmap v13457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:12:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:21 smithi081 ceph-mon[31453]: pgmap v13457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:12:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:23 smithi153 ceph-mon[36569]: pgmap v13458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:23 smithi104 ceph-mon[36624]: pgmap v13458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:23 smithi081 ceph-mon[31453]: pgmap v13458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:24 smithi081 ceph-mon[31453]: pgmap v13459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:24 smithi153 ceph-mon[36569]: pgmap v13459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:24 smithi104 ceph-mon[36624]: pgmap v13459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:26.751+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:12:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:26.751+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:12:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:26.751+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:12:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:27 smithi153 ceph-mon[36569]: pgmap v13460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:27 smithi104 ceph-mon[36624]: pgmap v13460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:27 smithi081 ceph-mon[31453]: pgmap v13460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:12:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:12:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:28 smithi153 ceph-mon[36569]: pgmap v13461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:28 smithi153 ceph-mon[36569]: pgmap v13462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:12:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:12:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:28 smithi104 ceph-mon[36624]: pgmap v13461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:28 smithi104 ceph-mon[36624]: pgmap v13462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:12:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:12:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:28 smithi081 ceph-mon[31453]: pgmap v13461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:28 smithi081 ceph-mon[31453]: pgmap v13462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:12:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:30 smithi081 ceph-mon[31453]: pgmap v13463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:30 smithi153 ceph-mon[36569]: pgmap v13463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:30 smithi104 ceph-mon[36624]: pgmap v13463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:32 smithi081 ceph-mon[31453]: pgmap v13464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:32 smithi153 ceph-mon[36569]: pgmap v13464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:32 smithi104 ceph-mon[36624]: pgmap v13464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:34 smithi081 ceph-mon[31453]: pgmap v13465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:34 smithi153 ceph-mon[36569]: pgmap v13465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:34 smithi104 ceph-mon[36624]: pgmap v13465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:36 smithi081 ceph-mon[31453]: pgmap v13466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:36 smithi153 ceph-mon[36569]: pgmap v13466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:36 smithi104 ceph-mon[36624]: pgmap v13466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:38 smithi081 ceph-mon[31453]: pgmap v13467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:38 smithi153 ceph-mon[36569]: pgmap v13467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:38 smithi104 ceph-mon[36624]: pgmap v13467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:40 smithi153 ceph-mon[36569]: pgmap v13468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:40 smithi104 ceph-mon[36624]: pgmap v13468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:40 smithi081 ceph-mon[31453]: pgmap v13468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:41.767+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:12:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:41.767+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:12:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:41.767+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:12:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:42 smithi153 ceph-mon[36569]: pgmap v13469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:42 smithi104 ceph-mon[36624]: pgmap v13469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:42 smithi081 ceph-mon[31453]: pgmap v13469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:44 smithi153 ceph-mon[36569]: pgmap v13470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:44 smithi104 ceph-mon[36624]: pgmap v13470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:44 smithi081 ceph-mon[31453]: pgmap v13470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:46 smithi153 ceph-mon[36569]: pgmap v13471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:46 smithi104 ceph-mon[36624]: pgmap v13471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:46 smithi081 ceph-mon[31453]: pgmap v13471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:48.628 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:48 smithi153 ceph-mon[36569]: pgmap v13472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:48.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:48 smithi081 ceph-mon[31453]: pgmap v13472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:48 smithi104 ceph-mon[36624]: pgmap v13472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:50 smithi153 ceph-mon[36569]: pgmap v13473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:50 smithi104 ceph-mon[36624]: pgmap v13473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:50 smithi081 ceph-mon[31453]: pgmap v13473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:52 smithi153 ceph-mon[36569]: pgmap v13474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:52 smithi104 ceph-mon[36624]: pgmap v13474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:52 smithi081 ceph-mon[31453]: pgmap v13474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:54 smithi153 ceph-mon[36569]: pgmap v13475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:54 smithi104 ceph-mon[36624]: pgmap v13475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:54 smithi081 ceph-mon[31453]: pgmap v13475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:56 smithi081 ceph-mon[31453]: pgmap v13476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:56.780+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:12:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:56.780+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:12:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:12:56.780+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:12:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:56 smithi153 ceph-mon[36569]: pgmap v13476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:56 smithi104 ceph-mon[36624]: pgmap v13476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:12:58 smithi153 ceph-mon[36569]: pgmap v13477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:12:58 smithi104 ceph-mon[36624]: pgmap v13477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:12:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:12:58 smithi081 ceph-mon[31453]: pgmap v13477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:00 smithi153 ceph-mon[36569]: pgmap v13478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:00.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:00 smithi104 ceph-mon[36624]: pgmap v13478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:00 smithi081 ceph-mon[31453]: pgmap v13478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:02 smithi153 ceph-mon[36569]: pgmap v13479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:02 smithi104 ceph-mon[36624]: pgmap v13479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:02 smithi081 ceph-mon[31453]: pgmap v13479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:04 smithi153 ceph-mon[36569]: pgmap v13480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:04 smithi104 ceph-mon[36624]: pgmap v13480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:04 smithi081 ceph-mon[31453]: pgmap v13480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:06 smithi153 ceph-mon[36569]: pgmap v13481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:06 smithi104 ceph-mon[36624]: pgmap v13481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:06 smithi081 ceph-mon[31453]: pgmap v13481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:08 smithi153 ceph-mon[36569]: pgmap v13482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:08 smithi104 ceph-mon[36624]: pgmap v13482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:08 smithi081 ceph-mon[31453]: pgmap v13482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:10 smithi153 ceph-mon[36569]: pgmap v13483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:10 smithi104 ceph-mon[36624]: pgmap v13483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:10 smithi081 ceph-mon[31453]: pgmap v13483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:11.788+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:13:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:11.788+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:13:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:11.788+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:13:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:12 smithi153 ceph-mon[36569]: pgmap v13484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:12 smithi104 ceph-mon[36624]: pgmap v13484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:12 smithi081 ceph-mon[31453]: pgmap v13484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:14 smithi153 ceph-mon[36569]: pgmap v13485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:14.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:14 smithi104 ceph-mon[36624]: pgmap v13485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:14 smithi081 ceph-mon[31453]: pgmap v13485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:16 smithi153 ceph-mon[36569]: pgmap v13486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:16 smithi104 ceph-mon[36624]: pgmap v13486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:16.837+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:13:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:16.837+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:13:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:16.837+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 194 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:13:16.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:16.837+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 194 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:13:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:16 smithi081 ceph-mon[31453]: pgmap v13486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:18 smithi153 ceph-mon[36569]: pgmap v13487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:18 smithi104 ceph-mon[36624]: pgmap v13487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:18 smithi081 ceph-mon[31453]: pgmap v13487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:20 smithi153 ceph-mon[36569]: pgmap v13488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:20 smithi104 ceph-mon[36624]: pgmap v13488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:20 smithi081 ceph-mon[31453]: pgmap v13488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:22 smithi153 ceph-mon[36569]: pgmap v13489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:22 smithi104 ceph-mon[36624]: pgmap v13489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:22 smithi081 ceph-mon[31453]: pgmap v13489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:24 smithi153 ceph-mon[36569]: pgmap v13490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:24 smithi104 ceph-mon[36624]: pgmap v13490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:24 smithi081 ceph-mon[31453]: pgmap v13490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:26.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:26.792+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:13:26.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:26.792+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:13:26.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:26.792+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:13:26.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:26 smithi153 ceph-mon[36569]: pgmap v13491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:26 smithi104 ceph-mon[36624]: pgmap v13491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:26.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:26 smithi081 ceph-mon[31453]: pgmap v13491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:28.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:28 smithi081 ceph-mon[31453]: pgmap v13492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:28.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:13:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:29.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:28 smithi153 ceph-mon[36569]: pgmap v13492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:13:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:28 smithi104 ceph-mon[36624]: pgmap v13492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:13:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:29.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:13:29.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:13:29.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:13:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:13:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:13:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:13:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:13:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:30 smithi081 ceph-mon[31453]: pgmap v13493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:30 smithi081 ceph-mon[31453]: pgmap v13494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:30 smithi153 ceph-mon[36569]: pgmap v13493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:30 smithi153 ceph-mon[36569]: pgmap v13494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:30 smithi104 ceph-mon[36624]: pgmap v13493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:30 smithi104 ceph-mon[36624]: pgmap v13494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:32.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:32 smithi081 ceph-mon[31453]: pgmap v13495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:33.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:32 smithi153 ceph-mon[36569]: pgmap v13495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:32 smithi104 ceph-mon[36624]: pgmap v13495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:34 smithi081 ceph-mon[31453]: pgmap v13496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:35.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:34 smithi153 ceph-mon[36569]: pgmap v13496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:34 smithi104 ceph-mon[36624]: pgmap v13496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:36.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:36 smithi081 ceph-mon[31453]: pgmap v13497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:37.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:36 smithi153 ceph-mon[36569]: pgmap v13497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:36 smithi104 ceph-mon[36624]: pgmap v13497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:38 smithi081 ceph-mon[31453]: pgmap v13498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:39.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:38 smithi153 ceph-mon[36569]: pgmap v13498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:38 smithi104 ceph-mon[36624]: pgmap v13498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:40 smithi081 ceph-mon[31453]: pgmap v13499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:41.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:40 smithi153 ceph-mon[36569]: pgmap v13499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:40 smithi104 ceph-mon[36624]: pgmap v13499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:41.801 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:41.802+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:13:41.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:41.802+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:13:41.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:41.802+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:13:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:42 smithi081 ceph-mon[31453]: pgmap v13500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:43.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:42 smithi153 ceph-mon[36569]: pgmap v13500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:42 smithi104 ceph-mon[36624]: pgmap v13500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:44 smithi081 ceph-mon[31453]: pgmap v13501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:45.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:44 smithi153 ceph-mon[36569]: pgmap v13501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:44 smithi104 ceph-mon[36624]: pgmap v13501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:46.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:46 smithi081 ceph-mon[31453]: pgmap v13502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:47.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:46 smithi153 ceph-mon[36569]: pgmap v13502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:47.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:46 smithi104 ceph-mon[36624]: pgmap v13502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:48.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:48 smithi081 ceph-mon[31453]: pgmap v13503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:49.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:48 smithi153 ceph-mon[36569]: pgmap v13503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:49.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:48 smithi104 ceph-mon[36624]: pgmap v13503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:51.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:50 smithi153 ceph-mon[36569]: pgmap v13504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:51.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:50 smithi104 ceph-mon[36624]: pgmap v13504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:51.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:50 smithi081 ceph-mon[31453]: pgmap v13504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:53.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:52 smithi153 ceph-mon[36569]: pgmap v13505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:53.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:52 smithi104 ceph-mon[36624]: pgmap v13505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:53.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:52 smithi081 ceph-mon[31453]: pgmap v13505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:55.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:54 smithi153 ceph-mon[36569]: pgmap v13506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:55.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:54 smithi104 ceph-mon[36624]: pgmap v13506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:55.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:54 smithi081 ceph-mon[31453]: pgmap v13506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:56.811+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:13:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:56.811+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:13:56.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:13:56.811+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:13:57.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:56 smithi153 ceph-mon[36569]: pgmap v13507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:57.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:56 smithi104 ceph-mon[36624]: pgmap v13507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:57.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:56 smithi081 ceph-mon[31453]: pgmap v13507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:59.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:13:58 smithi153 ceph-mon[36569]: pgmap v13508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:59.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:13:58 smithi104 ceph-mon[36624]: pgmap v13508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:13:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:13:58 smithi081 ceph-mon[31453]: pgmap v13508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:01.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:00 smithi153 ceph-mon[36569]: pgmap v13509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:01.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:00 smithi104 ceph-mon[36624]: pgmap v13509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:01.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:00 smithi081 ceph-mon[31453]: pgmap v13509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:03.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:02 smithi153 ceph-mon[36569]: pgmap v13510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:03.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:02 smithi104 ceph-mon[36624]: pgmap v13510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:03.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:02 smithi081 ceph-mon[31453]: pgmap v13510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:04 smithi153 ceph-mon[36569]: pgmap v13511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:05.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:04 smithi104 ceph-mon[36624]: pgmap v13511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:05.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:04 smithi081 ceph-mon[31453]: pgmap v13511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:07.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:06 smithi153 ceph-mon[36569]: pgmap v13512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:07.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:06 smithi104 ceph-mon[36624]: pgmap v13512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:07.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:06 smithi081 ceph-mon[31453]: pgmap v13512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:09.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:08 smithi153 ceph-mon[36569]: pgmap v13513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:09.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:08 smithi104 ceph-mon[36624]: pgmap v13513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:09.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:08 smithi081 ceph-mon[31453]: pgmap v13513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:11.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:10 smithi153 ceph-mon[36569]: pgmap v13514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:11.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:10 smithi104 ceph-mon[36624]: pgmap v13514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:11.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:10 smithi081 ceph-mon[31453]: pgmap v13514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:11.824+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:14:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:11.824+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:14:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:11.824+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:14:13.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:12 smithi153 ceph-mon[36569]: pgmap v13515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:13.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:12 smithi104 ceph-mon[36624]: pgmap v13515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:13.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:12 smithi081 ceph-mon[31453]: pgmap v13515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:15.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:14 smithi153 ceph-mon[36569]: pgmap v13516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:15.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:14 smithi104 ceph-mon[36624]: pgmap v13516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:15.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:14 smithi081 ceph-mon[31453]: pgmap v13516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:17.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:16 smithi153 ceph-mon[36569]: pgmap v13517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:17.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:16 smithi104 ceph-mon[36624]: pgmap v13517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:17.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:16 smithi081 ceph-mon[31453]: pgmap v13517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:19.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:18 smithi153 ceph-mon[36569]: pgmap v13518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:19.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:18 smithi104 ceph-mon[36624]: pgmap v13518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:19.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:18 smithi081 ceph-mon[31453]: pgmap v13518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:21.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:20 smithi153 ceph-mon[36569]: pgmap v13519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:21.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:20 smithi104 ceph-mon[36624]: pgmap v13519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:21.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:20 smithi081 ceph-mon[31453]: pgmap v13519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:23.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:22 smithi153 ceph-mon[36569]: pgmap v13520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:23.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:22 smithi104 ceph-mon[36624]: pgmap v13520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:23.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:22 smithi081 ceph-mon[31453]: pgmap v13520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:24.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:23 smithi153 ceph-mon[36569]: pgmap v13521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:24.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:23 smithi104 ceph-mon[36624]: pgmap v13521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:24.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:23 smithi081 ceph-mon[31453]: pgmap v13521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:26.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:26 smithi081 ceph-mon[31453]: pgmap v13522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:26.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:26 smithi153 ceph-mon[36569]: pgmap v13522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:26.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:26 smithi104 ceph-mon[36624]: pgmap v13522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:26.825+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:14:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:26.825+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:14:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:26.825+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:14:28.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:28 smithi081 ceph-mon[31453]: pgmap v13523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:28.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:28 smithi153 ceph-mon[36569]: pgmap v13523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:28.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:28 smithi104 ceph-mon[36624]: pgmap v13523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:30 smithi081 ceph-mon[31453]: pgmap v13524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:14:30.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:30 smithi153 ceph-mon[36569]: pgmap v13524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:30.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:14:30.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:30 smithi104 ceph-mon[36624]: pgmap v13524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:30.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:14:32.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:32 smithi081 ceph-mon[31453]: pgmap v13525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:32.241 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:32 smithi104 ceph-mon[36624]: pgmap v13525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:32.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:32 smithi153 ceph-mon[36569]: pgmap v13525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:34.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:34 smithi081 ceph-mon[31453]: pgmap v13526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:34.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:34 smithi153 ceph-mon[36569]: pgmap v13526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:34.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:34 smithi104 ceph-mon[36624]: pgmap v13526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:35.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:35.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:35.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:35.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:35.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:35.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:36.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:36 smithi153 ceph-mon[36569]: pgmap v13527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:36.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:36 smithi104 ceph-mon[36624]: pgmap v13527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:36.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:36 smithi081 ceph-mon[31453]: pgmap v13527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:37.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:37.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:37.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:14:37.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:14:37.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:37 smithi081 ceph-mon[31453]: pgmap v13528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:37.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:37 smithi081 ceph-mon[31453]: pgmap v13529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:37.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:37.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:37.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:37.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:14:37.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:14:37.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:37 smithi153 ceph-mon[36569]: pgmap v13528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:37.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:37 smithi153 ceph-mon[36569]: pgmap v13529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:37.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:37.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:37.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:37.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:14:37.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:14:37.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:37 smithi104 ceph-mon[36624]: pgmap v13528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:37.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:37 smithi104 ceph-mon[36624]: pgmap v13529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:37.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:14:39.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:39 smithi153 ceph-mon[36569]: pgmap v13530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:39.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:39 smithi104 ceph-mon[36624]: pgmap v13530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:39.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:39 smithi081 ceph-mon[31453]: pgmap v13530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:41.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:41 smithi153 ceph-mon[36569]: pgmap v13531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:41.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:41.830+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:14:41.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:41.830+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:14:41.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:41.830+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:14:41.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:41 smithi104 ceph-mon[36624]: pgmap v13531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:41.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:41 smithi081 ceph-mon[31453]: pgmap v13531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:43.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:43 smithi104 ceph-mon[36624]: pgmap v13532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:43.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:43 smithi153 ceph-mon[36569]: pgmap v13532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:43.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:43 smithi081 ceph-mon[31453]: pgmap v13532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:45.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:45 smithi153 ceph-mon[36569]: pgmap v13533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:45.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:45 smithi104 ceph-mon[36624]: pgmap v13533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:45.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:45 smithi081 ceph-mon[31453]: pgmap v13533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:47.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:47 smithi153 ceph-mon[36569]: pgmap v13534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:47.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:47 smithi104 ceph-mon[36624]: pgmap v13534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:47.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:47 smithi081 ceph-mon[31453]: pgmap v13534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:49.706 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:49 smithi081 ceph-mon[31453]: pgmap v13535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:49.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:49 smithi153 ceph-mon[36569]: pgmap v13535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:49.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:49 smithi104 ceph-mon[36624]: pgmap v13535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:51.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:51 smithi153 ceph-mon[36569]: pgmap v13536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:51.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:51 smithi104 ceph-mon[36624]: pgmap v13536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:51.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:51 smithi081 ceph-mon[31453]: pgmap v13536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:53.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:53 smithi153 ceph-mon[36569]: pgmap v13537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:53.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:53 smithi104 ceph-mon[36624]: pgmap v13537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:53.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:53 smithi081 ceph-mon[31453]: pgmap v13537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:55.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:55 smithi153 ceph-mon[36569]: pgmap v13538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:55.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:55 smithi104 ceph-mon[36624]: pgmap v13538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:55.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:55 smithi081 ceph-mon[31453]: pgmap v13538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:56.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:56.846+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:14:56.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:56.846+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:14:56.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:14:56.846+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:14:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:57 smithi153 ceph-mon[36569]: pgmap v13539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:57.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:57 smithi104 ceph-mon[36624]: pgmap v13539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:57.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:57 smithi081 ceph-mon[31453]: pgmap v13539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:59.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:14:59 smithi153 ceph-mon[36569]: pgmap v13540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:59.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:14:59 smithi104 ceph-mon[36624]: pgmap v13540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:14:59.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:14:59 smithi081 ceph-mon[31453]: pgmap v13540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:01.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:01 smithi153 ceph-mon[36569]: pgmap v13541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:01.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:01 smithi104 ceph-mon[36624]: pgmap v13541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:01.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:01 smithi081 ceph-mon[31453]: pgmap v13541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:03 smithi153 ceph-mon[36569]: pgmap v13542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:03.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:03 smithi104 ceph-mon[36624]: pgmap v13542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:03.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:03 smithi081 ceph-mon[31453]: pgmap v13542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:05 smithi153 ceph-mon[36569]: pgmap v13543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:05.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:05 smithi104 ceph-mon[36624]: pgmap v13543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:05.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:05 smithi081 ceph-mon[31453]: pgmap v13543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:07 smithi153 ceph-mon[36569]: pgmap v13544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:07.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:07 smithi104 ceph-mon[36624]: pgmap v13544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:07.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:07 smithi081 ceph-mon[31453]: pgmap v13544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:09.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:09 smithi153 ceph-mon[36569]: pgmap v13545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:09.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:09 smithi104 ceph-mon[36624]: pgmap v13545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:09.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:09 smithi081 ceph-mon[31453]: pgmap v13545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:11.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:11 smithi153 ceph-mon[36569]: pgmap v13546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:11.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:11 smithi104 ceph-mon[36624]: pgmap v13546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:11.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:11.857+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:15:11.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:11.857+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:15:11.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:11.857+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:15:11.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:11 smithi081 ceph-mon[31453]: pgmap v13546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:13.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:13 smithi153 ceph-mon[36569]: pgmap v13547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:13.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:13 smithi104 ceph-mon[36624]: pgmap v13547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:13.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:13 smithi081 ceph-mon[31453]: pgmap v13547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:15.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:15 smithi153 ceph-mon[36569]: pgmap v13548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:15 smithi104 ceph-mon[36624]: pgmap v13548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:15.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:15 smithi081 ceph-mon[31453]: pgmap v13548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:17 smithi153 ceph-mon[36569]: pgmap v13549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:17 smithi104 ceph-mon[36624]: pgmap v13549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:17.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:17 smithi081 ceph-mon[31453]: pgmap v13549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:19.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:19 smithi153 ceph-mon[36569]: pgmap v13550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:19.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:19 smithi104 ceph-mon[36624]: pgmap v13550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:19.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:19 smithi081 ceph-mon[31453]: pgmap v13550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:21.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:21 smithi153 ceph-mon[36569]: pgmap v13551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:21.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:21 smithi104 ceph-mon[36624]: pgmap v13551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:21.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:21 smithi081 ceph-mon[31453]: pgmap v13551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:23.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:23 smithi153 ceph-mon[36569]: pgmap v13552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:23.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:23 smithi104 ceph-mon[36624]: pgmap v13552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:23.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:23 smithi081 ceph-mon[31453]: pgmap v13552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:25.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:25 smithi153 ceph-mon[36569]: pgmap v13553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:25.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:25 smithi104 ceph-mon[36624]: pgmap v13553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:25.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:25 smithi081 ceph-mon[31453]: pgmap v13553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:26.863+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:15:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:26.863+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:15:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:26.863+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:15:27.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:27 smithi153 ceph-mon[36569]: pgmap v13554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:27.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:27 smithi104 ceph-mon[36624]: pgmap v13554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:27.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:27 smithi081 ceph-mon[31453]: pgmap v13554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:29.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:29 smithi153 ceph-mon[36569]: pgmap v13555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:29.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:29 smithi104 ceph-mon[36624]: pgmap v13555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:29.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:29 smithi081 ceph-mon[31453]: pgmap v13555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:31.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:31 smithi104 ceph-mon[36624]: pgmap v13556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:31.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:31 smithi081 ceph-mon[31453]: pgmap v13556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:32.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:31 smithi153 ceph-mon[36569]: pgmap v13556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:33.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:33 smithi104 ceph-mon[36624]: pgmap v13557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:33.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:33 smithi081 ceph-mon[31453]: pgmap v13557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:34.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:33 smithi153 ceph-mon[36569]: pgmap v13557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:35.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:35 smithi104 ceph-mon[36624]: pgmap v13558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:35.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:35 smithi081 ceph-mon[31453]: pgmap v13558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:36.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:35 smithi153 ceph-mon[36569]: pgmap v13558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:37 smithi104 ceph-mon[36624]: pgmap v13559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:37.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:15:37.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:37 smithi081 ceph-mon[31453]: pgmap v13559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:37.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:15:38.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:37 smithi153 ceph-mon[36569]: pgmap v13559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:38.015 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:15:39.764 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:39 smithi153 ceph-mon[36569]: pgmap v13560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:39.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:39 smithi104 ceph-mon[36624]: pgmap v13560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:39.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:39 smithi081 ceph-mon[31453]: pgmap v13560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:41.802 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:41 smithi153 ceph-mon[36569]: pgmap v13561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:41.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:41.879+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:15:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:41.879+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:15:41.875 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:41.879+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:15:41.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:41 smithi081 ceph-mon[31453]: pgmap v13561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:42.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:41 smithi104 ceph-mon[36624]: pgmap v13561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:15:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:15:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:42 smithi153 ceph-mon[36569]: pgmap v13562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:42 smithi153 ceph-mon[36569]: pgmap v13563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:43.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:43.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:43.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:43.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:15:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:15:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:42 smithi104 ceph-mon[36624]: pgmap v13562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:42 smithi104 ceph-mon[36624]: pgmap v13563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:43.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:43.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:15:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:15:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:42 smithi081 ceph-mon[31453]: pgmap v13562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:42 smithi081 ceph-mon[31453]: pgmap v13563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:43.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:15:45.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:44 smithi153 ceph-mon[36569]: pgmap v13564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:45.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:44 smithi104 ceph-mon[36624]: pgmap v13564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:45.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:44 smithi081 ceph-mon[31453]: pgmap v13564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:47.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:46 smithi153 ceph-mon[36569]: pgmap v13565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:47.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:46 smithi104 ceph-mon[36624]: pgmap v13565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:47.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:46 smithi081 ceph-mon[31453]: pgmap v13565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:49.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:48 smithi153 ceph-mon[36569]: pgmap v13566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:49.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:48 smithi104 ceph-mon[36624]: pgmap v13566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:49.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:48 smithi081 ceph-mon[31453]: pgmap v13566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:51.052 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:50 smithi153 ceph-mon[36569]: pgmap v13567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:51.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:50 smithi104 ceph-mon[36624]: pgmap v13567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:51.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:50 smithi081 ceph-mon[31453]: pgmap v13567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:53.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:52 smithi153 ceph-mon[36569]: pgmap v13568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:53.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:52 smithi104 ceph-mon[36624]: pgmap v13568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:53.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:52 smithi081 ceph-mon[31453]: pgmap v13568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:54.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:53 smithi104 ceph-mon[36624]: pgmap v13569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:54.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:53 smithi081 ceph-mon[31453]: pgmap v13569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:54.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:53 smithi153 ceph-mon[36569]: pgmap v13569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:56.894+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:15:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:56.894+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:15:56.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:15:56.894+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:15:57.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:56 smithi153 ceph-mon[36569]: pgmap v13570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:57.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:56 smithi104 ceph-mon[36624]: pgmap v13570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:57.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:56 smithi081 ceph-mon[31453]: pgmap v13570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:59.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:58 smithi153 ceph-mon[36569]: pgmap v13571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:59.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:58 smithi104 ceph-mon[36624]: pgmap v13571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:15:59.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:58 smithi081 ceph-mon[31453]: pgmap v13571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:00.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:15:59 smithi104 ceph-mon[36624]: pgmap v13572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:00.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:15:59 smithi081 ceph-mon[31453]: pgmap v13572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:00.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:15:59 smithi153 ceph-mon[36569]: pgmap v13572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:03.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:02 smithi153 ceph-mon[36569]: pgmap v13573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:03.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:02 smithi104 ceph-mon[36624]: pgmap v13573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:03.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:02 smithi081 ceph-mon[31453]: pgmap v13573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:05.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:04 smithi153 ceph-mon[36569]: pgmap v13574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:05.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:04 smithi104 ceph-mon[36624]: pgmap v13574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:05.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:04 smithi081 ceph-mon[31453]: pgmap v13574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:07.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:06 smithi153 ceph-mon[36569]: pgmap v13575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:07.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:06 smithi104 ceph-mon[36624]: pgmap v13575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:07.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:06 smithi081 ceph-mon[31453]: pgmap v13575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:09.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:08 smithi153 ceph-mon[36569]: pgmap v13576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:09.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:08 smithi104 ceph-mon[36624]: pgmap v13576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:09.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:08 smithi081 ceph-mon[31453]: pgmap v13576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:10.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:09 smithi104 ceph-mon[36624]: pgmap v13577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:10.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:09 smithi081 ceph-mon[31453]: pgmap v13577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:10.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:09 smithi153 ceph-mon[36569]: pgmap v13577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:11.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:11.903+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:16:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:11.904+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:16:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:11.904+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:16:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:12.160+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000686) for more than 900000000 us, fault. 2024-04-13T04:16:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:12.160+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c9f0 tx=0x7f16f4044000 comp rx=0 tx=0).stop 2024-04-13T04:16:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:12.160+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b3ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T04:16:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:12.160+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T04:16:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:12.160+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T04:16:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:12.161+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T04:16:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:12.162+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f16d40036c0 tx=0x7f16f40ac000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T04:16:13.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:12 smithi153 ceph-mon[36569]: pgmap v13578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:13.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:12 smithi104 ceph-mon[36624]: pgmap v13578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:13.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:12 smithi081 ceph-mon[31453]: pgmap v13578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:15.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:14 smithi153 ceph-mon[36569]: pgmap v13579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:15.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:14 smithi104 ceph-mon[36624]: pgmap v13579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:15.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:14 smithi081 ceph-mon[31453]: pgmap v13579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:16.867+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:16:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:16.867+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:16:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:16.867+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 195 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:16:16.862 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:16.867+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 195 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:16:17.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:16 smithi153 ceph-mon[36569]: pgmap v13580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:17.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:16 smithi104 ceph-mon[36624]: pgmap v13580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:17.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:16 smithi081 ceph-mon[31453]: pgmap v13580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:18.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:17 smithi104 ceph-mon[36624]: pgmap v13581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:18.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:17 smithi081 ceph-mon[31453]: pgmap v13581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:18.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:17 smithi153 ceph-mon[36569]: pgmap v13581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:21.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:20 smithi153 ceph-mon[36569]: pgmap v13582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:21.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:20 smithi104 ceph-mon[36624]: pgmap v13582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:21.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:20 smithi081 ceph-mon[31453]: pgmap v13582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:23.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:22 smithi153 ceph-mon[36569]: pgmap v13583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:23.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:22 smithi104 ceph-mon[36624]: pgmap v13583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:23.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:22 smithi081 ceph-mon[31453]: pgmap v13583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:25.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:24 smithi153 ceph-mon[36569]: pgmap v13584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:25.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:24 smithi104 ceph-mon[36624]: pgmap v13584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:25.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:24 smithi081 ceph-mon[31453]: pgmap v13584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:26.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:25 smithi153 ceph-mon[36569]: pgmap v13585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:26.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:25 smithi104 ceph-mon[36624]: pgmap v13585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:26.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:25 smithi081 ceph-mon[31453]: pgmap v13585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:26.906+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:16:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:26.906+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:16:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:26.906+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:16:29.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:28 smithi153 ceph-mon[36569]: pgmap v13586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:29.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:28 smithi104 ceph-mon[36624]: pgmap v13586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:29.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:28 smithi081 ceph-mon[31453]: pgmap v13586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:31.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:30 smithi153 ceph-mon[36569]: pgmap v13587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:31.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:30 smithi104 ceph-mon[36624]: pgmap v13587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:31.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:30 smithi081 ceph-mon[31453]: pgmap v13587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:32.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:31 smithi104 ceph-mon[36624]: pgmap v13588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:32.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:31 smithi081 ceph-mon[31453]: pgmap v13588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:32.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:31 smithi153 ceph-mon[36569]: pgmap v13588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:35.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:34 smithi153 ceph-mon[36569]: pgmap v13589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:35.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:34 smithi104 ceph-mon[36624]: pgmap v13589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:35.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:34 smithi081 ceph-mon[31453]: pgmap v13589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:37.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:36 smithi153 ceph-mon[36569]: pgmap v13590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:37.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:36 smithi104 ceph-mon[36624]: pgmap v13590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:37.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:36 smithi081 ceph-mon[31453]: pgmap v13590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:38.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:37 smithi153 ceph-mon[36569]: pgmap v13591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:38.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:37 smithi104 ceph-mon[36624]: pgmap v13591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:38.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:37 smithi081 ceph-mon[31453]: pgmap v13591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:41.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:40 smithi153 ceph-mon[36569]: pgmap v13592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:41.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:40 smithi104 ceph-mon[36624]: pgmap v13592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:41.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:40 smithi081 ceph-mon[31453]: pgmap v13592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:41.916 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:41.922+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:16:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:41.922+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:16:41.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:41.922+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:16:42.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:41 smithi104 ceph-mon[36624]: pgmap v13593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:42.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:41 smithi081 ceph-mon[31453]: pgmap v13593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:42.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:41 smithi153 ceph-mon[36569]: pgmap v13593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:43.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:16:43.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:16:43.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:16:44.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:16:44.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:16:44.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:43 smithi081 ceph-mon[31453]: pgmap v13594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:44.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:43 smithi081 ceph-mon[31453]: pgmap v13595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:44.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:16:44.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:16:44.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:16:44.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:43 smithi153 ceph-mon[36569]: pgmap v13594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:44.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:43 smithi153 ceph-mon[36569]: pgmap v13595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:44.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:16:44.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:16:44.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:16:44.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:43 smithi104 ceph-mon[36624]: pgmap v13594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:44.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:43 smithi104 ceph-mon[36624]: pgmap v13595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:44.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:16:46.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:46 smithi081 ceph-mon[31453]: pgmap v13596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:46.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:46 smithi153 ceph-mon[36569]: pgmap v13596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:46.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:46 smithi104 ceph-mon[36624]: pgmap v13596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:48.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:48 smithi081 ceph-mon[31453]: pgmap v13597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:48.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:48 smithi153 ceph-mon[36569]: pgmap v13597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:48.826 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:48 smithi104 ceph-mon[36624]: pgmap v13597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:50.628 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:50 smithi104 ceph-mon[36624]: pgmap v13598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:50.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:50 smithi081 ceph-mon[31453]: pgmap v13598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:50.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:50 smithi153 ceph-mon[36569]: pgmap v13598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:52.614 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:52 smithi153 ceph-mon[36569]: pgmap v13599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:52.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:52 smithi081 ceph-mon[31453]: pgmap v13599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:52.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:52 smithi104 ceph-mon[36624]: pgmap v13599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:54.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:54 smithi081 ceph-mon[31453]: pgmap v13600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:54.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:54 smithi153 ceph-mon[36569]: pgmap v13600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:54.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:54 smithi104 ceph-mon[36624]: pgmap v13600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:56.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:56 smithi081 ceph-mon[31453]: pgmap v13601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:56.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:56 smithi153 ceph-mon[36569]: pgmap v13601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:56.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:56 smithi104 ceph-mon[36624]: pgmap v13601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:56.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:56.938+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:16:56.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:56.938+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:16:56.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:16:56.938+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:16:58.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:16:58 smithi081 ceph-mon[31453]: pgmap v13602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:58.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:16:58 smithi153 ceph-mon[36569]: pgmap v13602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:16:58.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:16:58 smithi104 ceph-mon[36624]: pgmap v13602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:00.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:00 smithi081 ceph-mon[31453]: pgmap v13603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:00.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:00 smithi153 ceph-mon[36569]: pgmap v13603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:00.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:00 smithi104 ceph-mon[36624]: pgmap v13603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:01.875+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:17:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:01.875+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:17:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:01.875+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 196 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:17:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:01.875+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 196 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:17:02.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:02 smithi081 ceph-mon[31453]: pgmap v13604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:02.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:02 smithi153 ceph-mon[36569]: pgmap v13604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:02.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:02 smithi104 ceph-mon[36624]: pgmap v13604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:04.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:04 smithi081 ceph-mon[31453]: pgmap v13605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:04.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:04 smithi153 ceph-mon[36569]: pgmap v13605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:04.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:04 smithi104 ceph-mon[36624]: pgmap v13605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:06.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:06 smithi081 ceph-mon[31453]: pgmap v13606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:06.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:06 smithi153 ceph-mon[36569]: pgmap v13606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:06.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:06 smithi104 ceph-mon[36624]: pgmap v13606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:08.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:08 smithi153 ceph-mon[36569]: pgmap v13607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:08.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:08 smithi104 ceph-mon[36624]: pgmap v13607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:08.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:08 smithi081 ceph-mon[31453]: pgmap v13607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:10.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:10 smithi153 ceph-mon[36569]: pgmap v13608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:10.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:10 smithi104 ceph-mon[36624]: pgmap v13608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:10.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:10 smithi081 ceph-mon[31453]: pgmap v13608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:11.949+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:17:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:11.949+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:17:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:11.949+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:17:12.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:12 smithi153 ceph-mon[36569]: pgmap v13609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:12.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:12 smithi104 ceph-mon[36624]: pgmap v13609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:12.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:12 smithi081 ceph-mon[31453]: pgmap v13609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:14.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:14 smithi153 ceph-mon[36569]: pgmap v13610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:14.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:14 smithi104 ceph-mon[36624]: pgmap v13610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:14.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:14 smithi081 ceph-mon[31453]: pgmap v13610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:16.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:16 smithi153 ceph-mon[36569]: pgmap v13611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:16.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:16 smithi104 ceph-mon[36624]: pgmap v13611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:16.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:16 smithi081 ceph-mon[31453]: pgmap v13611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:18.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:18 smithi153 ceph-mon[36569]: pgmap v13612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:18.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:18 smithi104 ceph-mon[36624]: pgmap v13612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:18.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:18 smithi081 ceph-mon[31453]: pgmap v13612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:20.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:20 smithi153 ceph-mon[36569]: pgmap v13613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:20.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:20 smithi104 ceph-mon[36624]: pgmap v13613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:20.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:20 smithi081 ceph-mon[31453]: pgmap v13613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:22.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:22 smithi153 ceph-mon[36569]: pgmap v13614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:22.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:22 smithi104 ceph-mon[36624]: pgmap v13614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:22.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:22 smithi081 ceph-mon[31453]: pgmap v13614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:24.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:24 smithi153 ceph-mon[36569]: pgmap v13615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:24.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:24 smithi104 ceph-mon[36624]: pgmap v13615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:24.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:24 smithi081 ceph-mon[31453]: pgmap v13615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:26.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:26 smithi153 ceph-mon[36569]: pgmap v13616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:26.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:26 smithi104 ceph-mon[36624]: pgmap v13616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:26.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:26 smithi081 ceph-mon[31453]: pgmap v13616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:26.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:17:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:26.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:17:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:26.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:17:28.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:28 smithi153 ceph-mon[36569]: pgmap v13617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:28.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:28 smithi104 ceph-mon[36624]: pgmap v13617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:28.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:28 smithi081 ceph-mon[31453]: pgmap v13617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:30.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:30 smithi153 ceph-mon[36569]: pgmap v13618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:30.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:30 smithi104 ceph-mon[36624]: pgmap v13618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:30.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:30 smithi081 ceph-mon[31453]: pgmap v13618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:32.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:32 smithi153 ceph-mon[36569]: pgmap v13619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:32.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:32 smithi104 ceph-mon[36624]: pgmap v13619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:32.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:32 smithi081 ceph-mon[31453]: pgmap v13619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:34.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:34 smithi153 ceph-mon[36569]: pgmap v13620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:34.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:34 smithi104 ceph-mon[36624]: pgmap v13620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:34.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:34 smithi081 ceph-mon[31453]: pgmap v13620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:36.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:36 smithi153 ceph-mon[36569]: pgmap v13621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:36.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:36 smithi104 ceph-mon[36624]: pgmap v13621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:36.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:36 smithi081 ceph-mon[31453]: pgmap v13621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:38.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:38 smithi153 ceph-mon[36569]: pgmap v13622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:38.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:38 smithi104 ceph-mon[36624]: pgmap v13622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:38.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:38 smithi081 ceph-mon[31453]: pgmap v13622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:40.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:40 smithi153 ceph-mon[36569]: pgmap v13623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:40.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:40 smithi104 ceph-mon[36624]: pgmap v13623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:40.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:40 smithi081 ceph-mon[31453]: pgmap v13623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:41.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:41.970+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:17:41.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:41.970+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:17:41.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:41.970+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:17:42.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:42 smithi153 ceph-mon[36569]: pgmap v13624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:42.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:42 smithi104 ceph-mon[36624]: pgmap v13624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:42.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:42 smithi081 ceph-mon[31453]: pgmap v13624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:44.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:44 smithi153 ceph-mon[36569]: pgmap v13625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:44.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:17:44.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:44 smithi104 ceph-mon[36624]: pgmap v13625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:44.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:17:44.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:44 smithi081 ceph-mon[31453]: pgmap v13625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:44.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:17:45.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:17:45.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:17:45.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:17:45.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:17:45.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:17:45.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:17:45.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:17:45.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:17:45.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:17:46.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:46 smithi153 ceph-mon[36569]: pgmap v13626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:46.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:46 smithi153 ceph-mon[36569]: pgmap v13627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:46.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:46 smithi104 ceph-mon[36624]: pgmap v13626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:46.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:46 smithi104 ceph-mon[36624]: pgmap v13627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:46.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:46 smithi081 ceph-mon[31453]: pgmap v13626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:46.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:46 smithi081 ceph-mon[31453]: pgmap v13627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:48.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:48 smithi153 ceph-mon[36569]: pgmap v13628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:48.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:48 smithi104 ceph-mon[36624]: pgmap v13628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:48.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:48 smithi081 ceph-mon[31453]: pgmap v13628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:50.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:50 smithi153 ceph-mon[36569]: pgmap v13629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:50.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:50 smithi104 ceph-mon[36624]: pgmap v13629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:50.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:50 smithi081 ceph-mon[31453]: pgmap v13629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:53.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:52 smithi153 ceph-mon[36569]: pgmap v13630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:53.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:52 smithi104 ceph-mon[36624]: pgmap v13630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:53.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:52 smithi081 ceph-mon[31453]: pgmap v13630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:55.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:54 smithi153 ceph-mon[36569]: pgmap v13631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:55.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:54 smithi104 ceph-mon[36624]: pgmap v13631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:55.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:54 smithi081 ceph-mon[31453]: pgmap v13631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:56.972+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:17:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:56.972+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:17:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:17:56.972+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:17:57.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:56 smithi153 ceph-mon[36569]: pgmap v13632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:57.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:56 smithi104 ceph-mon[36624]: pgmap v13632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:57.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:56 smithi081 ceph-mon[31453]: pgmap v13632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:59.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:17:58 smithi153 ceph-mon[36569]: pgmap v13633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:59.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:17:58 smithi104 ceph-mon[36624]: pgmap v13633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:17:59.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:17:58 smithi081 ceph-mon[31453]: pgmap v13633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:01.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:00 smithi153 ceph-mon[36569]: pgmap v13634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:01.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:00 smithi104 ceph-mon[36624]: pgmap v13634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:01.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:00 smithi081 ceph-mon[31453]: pgmap v13634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:03.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:02 smithi153 ceph-mon[36569]: pgmap v13635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:03.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:02 smithi104 ceph-mon[36624]: pgmap v13635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:03.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:02 smithi081 ceph-mon[31453]: pgmap v13635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:05.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:04 smithi153 ceph-mon[36569]: pgmap v13636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:05.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:04 smithi104 ceph-mon[36624]: pgmap v13636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:05.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:04 smithi081 ceph-mon[31453]: pgmap v13636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:07.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:06 smithi153 ceph-mon[36569]: pgmap v13637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:07.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:06 smithi104 ceph-mon[36624]: pgmap v13637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:07.103 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:06 smithi081 ceph-mon[31453]: pgmap v13637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:09.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:08 smithi153 ceph-mon[36569]: pgmap v13638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:09.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:08 smithi104 ceph-mon[36624]: pgmap v13638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:09.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:08 smithi081 ceph-mon[31453]: pgmap v13638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:11.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:10 smithi153 ceph-mon[36569]: pgmap v13639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:11.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:10 smithi104 ceph-mon[36624]: pgmap v13639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:11.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:10 smithi081 ceph-mon[31453]: pgmap v13639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:11.987+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:18:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:11.987+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:18:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:11.987+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:18:13.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:12 smithi153 ceph-mon[36569]: pgmap v13640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:13.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:12 smithi104 ceph-mon[36624]: pgmap v13640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:13.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:12 smithi081 ceph-mon[31453]: pgmap v13640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:15.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:14 smithi153 ceph-mon[36569]: pgmap v13641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:15.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:14 smithi104 ceph-mon[36624]: pgmap v13641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:15.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:14 smithi081 ceph-mon[31453]: pgmap v13641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:17.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:16 smithi153 ceph-mon[36569]: pgmap v13642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:17.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:16 smithi104 ceph-mon[36624]: pgmap v13642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:17.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:16 smithi081 ceph-mon[31453]: pgmap v13642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:19.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:18 smithi153 ceph-mon[36569]: pgmap v13643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:19.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:18 smithi104 ceph-mon[36624]: pgmap v13643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:19.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:18 smithi081 ceph-mon[31453]: pgmap v13643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:21.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:20 smithi153 ceph-mon[36569]: pgmap v13644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:21.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:20 smithi104 ceph-mon[36624]: pgmap v13644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:21.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:20 smithi081 ceph-mon[31453]: pgmap v13644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:23.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:22 smithi153 ceph-mon[36569]: pgmap v13645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:23.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:22 smithi104 ceph-mon[36624]: pgmap v13645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:23.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:22 smithi081 ceph-mon[31453]: pgmap v13645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:25.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:24 smithi153 ceph-mon[36569]: pgmap v13646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:25.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:24 smithi104 ceph-mon[36624]: pgmap v13646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:25.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:24 smithi081 ceph-mon[31453]: pgmap v13646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:27.002+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:18:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:27.002+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:18:26.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:27.002+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:18:27.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:26 smithi153 ceph-mon[36569]: pgmap v13647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:27.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:26 smithi104 ceph-mon[36624]: pgmap v13647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:27.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:26 smithi081 ceph-mon[31453]: pgmap v13647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:29.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:28 smithi153 ceph-mon[36569]: pgmap v13648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:29.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:28 smithi104 ceph-mon[36624]: pgmap v13648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:29.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:28 smithi081 ceph-mon[31453]: pgmap v13648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:31.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:30 smithi153 ceph-mon[36569]: pgmap v13649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:31.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:30 smithi104 ceph-mon[36624]: pgmap v13649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:31.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:30 smithi081 ceph-mon[31453]: pgmap v13649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:33.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:32 smithi153 ceph-mon[36569]: pgmap v13650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:33.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:32 smithi104 ceph-mon[36624]: pgmap v13650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:33.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:32 smithi081 ceph-mon[31453]: pgmap v13650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:35.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:34 smithi153 ceph-mon[36569]: pgmap v13651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:35.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:34 smithi104 ceph-mon[36624]: pgmap v13651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:35.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:34 smithi081 ceph-mon[31453]: pgmap v13651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:36.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:35 smithi081 ceph-mon[31453]: pgmap v13652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:36.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:35 smithi153 ceph-mon[36569]: pgmap v13652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:36.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:35 smithi104 ceph-mon[36624]: pgmap v13652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:38.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:37 smithi153 ceph-mon[36569]: pgmap v13653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:38.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:37 smithi104 ceph-mon[36624]: pgmap v13653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:38.334 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:37 smithi081 ceph-mon[31453]: pgmap v13653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:40.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:39 smithi153 ceph-mon[36569]: pgmap v13654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:40.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:39 smithi104 ceph-mon[36624]: pgmap v13654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:40.397 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:39 smithi081 ceph-mon[31453]: pgmap v13654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:42.010+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:18:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:42.010+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:18:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:42.010+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:18:42.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:41 smithi153 ceph-mon[36569]: pgmap v13655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:42.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:41 smithi104 ceph-mon[36624]: pgmap v13655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:42.397 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:41 smithi081 ceph-mon[31453]: pgmap v13655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:44.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:43 smithi153 ceph-mon[36569]: pgmap v13656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:44.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:43 smithi104 ceph-mon[36624]: pgmap v13656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:44.397 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:43 smithi081 ceph-mon[31453]: pgmap v13656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:46.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: pgmap v13657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:46.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:18:46.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: pgmap v13657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:46.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:18:46.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.397 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: pgmap v13657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:46.397 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:18:46.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:46.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:18:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:18:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:18:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:18:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:18:47.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:18:47.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:18:47.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:18:47.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: pgmap v13658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:47.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: pgmap v13659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:47.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:18:47.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:18:47.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:18:47.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:18:47.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: pgmap v13658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: pgmap v13659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:47.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:18:47.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:18:47.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:18:47.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:18:47.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:18:47.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:18:47.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:18:47.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:18:47.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:18:47.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:18:47.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: pgmap v13658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:47.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: pgmap v13659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:47.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:18:49.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:49 smithi153 ceph-mon[36569]: pgmap v13660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:49.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:49 smithi104 ceph-mon[36624]: pgmap v13660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:49.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:49 smithi081 ceph-mon[31453]: pgmap v13660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:51.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:51 smithi153 ceph-mon[36569]: pgmap v13661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:51.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:51 smithi104 ceph-mon[36624]: pgmap v13661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:51.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:51 smithi081 ceph-mon[31453]: pgmap v13661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:53.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:53 smithi153 ceph-mon[36569]: pgmap v13662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:53.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:53 smithi104 ceph-mon[36624]: pgmap v13662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:53.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:53 smithi081 ceph-mon[31453]: pgmap v13662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:55.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:55 smithi153 ceph-mon[36569]: pgmap v13663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:55.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:55 smithi104 ceph-mon[36624]: pgmap v13663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:55.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:55 smithi081 ceph-mon[31453]: pgmap v13663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:57.024+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:18:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:57.024+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:18:57.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:18:57.024+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:18:57.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:57 smithi153 ceph-mon[36569]: pgmap v13664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:57.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:57 smithi104 ceph-mon[36624]: pgmap v13664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:57.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:57 smithi081 ceph-mon[31453]: pgmap v13664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:59.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:18:59 smithi153 ceph-mon[36569]: pgmap v13665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:59.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:18:59 smithi104 ceph-mon[36624]: pgmap v13665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:18:59.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:18:59 smithi081 ceph-mon[31453]: pgmap v13665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:01.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:01 smithi153 ceph-mon[36569]: pgmap v13666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:01.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:01 smithi104 ceph-mon[36624]: pgmap v13666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:01.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:01 smithi081 ceph-mon[31453]: pgmap v13666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:03.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:03 smithi153 ceph-mon[36569]: pgmap v13667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:03.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:03 smithi104 ceph-mon[36624]: pgmap v13667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:03.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:03 smithi081 ceph-mon[31453]: pgmap v13667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:05.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:05 smithi153 ceph-mon[36569]: pgmap v13668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:05.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:05 smithi104 ceph-mon[36624]: pgmap v13668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:05.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:05 smithi081 ceph-mon[31453]: pgmap v13668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:07.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:07 smithi153 ceph-mon[36569]: pgmap v13669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:07.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:07 smithi104 ceph-mon[36624]: pgmap v13669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:07.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:07 smithi081 ceph-mon[31453]: pgmap v13669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:09.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:09 smithi153 ceph-mon[36569]: pgmap v13670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:09.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:09 smithi104 ceph-mon[36624]: pgmap v13670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:09.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:09 smithi081 ceph-mon[31453]: pgmap v13670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:11.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:11 smithi153 ceph-mon[36569]: pgmap v13671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:11.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:11 smithi104 ceph-mon[36624]: pgmap v13671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:11.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:11 smithi081 ceph-mon[31453]: pgmap v13671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:12.032+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:19:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:12.032+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:19:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:12.032+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:19:13.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:13 smithi153 ceph-mon[36569]: pgmap v13672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:13.823 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:13 smithi104 ceph-mon[36624]: pgmap v13672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:13.897 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:13 smithi081 ceph-mon[31453]: pgmap v13672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:15.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:15 smithi153 ceph-mon[36569]: pgmap v13673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:15.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:15 smithi104 ceph-mon[36624]: pgmap v13673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:15.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:15 smithi081 ceph-mon[31453]: pgmap v13673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:16.897+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:19:16.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:16.897+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:19:16.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:16.898+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 197 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:19:16.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:16.898+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 197 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:19:17.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:17 smithi153 ceph-mon[36569]: pgmap v13674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:17.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:17 smithi104 ceph-mon[36624]: pgmap v13674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:17.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:17 smithi081 ceph-mon[31453]: pgmap v13674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:19.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:19 smithi153 ceph-mon[36569]: pgmap v13675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:19.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:19 smithi104 ceph-mon[36624]: pgmap v13675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:19.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:19 smithi081 ceph-mon[31453]: pgmap v13675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:21.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:21 smithi153 ceph-mon[36569]: pgmap v13676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:21.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:21 smithi104 ceph-mon[36624]: pgmap v13676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:21.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:21 smithi081 ceph-mon[31453]: pgmap v13676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:23.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:23 smithi153 ceph-mon[36569]: pgmap v13677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:23.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:23 smithi104 ceph-mon[36624]: pgmap v13677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:23.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:23 smithi081 ceph-mon[31453]: pgmap v13677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:25.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:25 smithi153 ceph-mon[36569]: pgmap v13678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:25.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:25 smithi104 ceph-mon[36624]: pgmap v13678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:25.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:25 smithi081 ceph-mon[31453]: pgmap v13678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:27.037 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:27.047+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:19:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:27.047+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:19:27.038 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:27.047+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:19:27.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:27 smithi104 ceph-mon[36624]: pgmap v13679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:27.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:27 smithi081 ceph-mon[31453]: pgmap v13679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:28.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:27 smithi153 ceph-mon[36569]: pgmap v13679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:29.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:29 smithi104 ceph-mon[36624]: pgmap v13680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:29.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:29 smithi081 ceph-mon[31453]: pgmap v13680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:30.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:29 smithi153 ceph-mon[36569]: pgmap v13680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:31.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:31 smithi104 ceph-mon[36624]: pgmap v13681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:31.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:31 smithi081 ceph-mon[31453]: pgmap v13681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:32.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:31 smithi153 ceph-mon[36569]: pgmap v13681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:34.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:33 smithi153 ceph-mon[36569]: pgmap v13682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:34.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:33 smithi104 ceph-mon[36624]: pgmap v13682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:34.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:33 smithi081 ceph-mon[31453]: pgmap v13682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:36.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:35 smithi153 ceph-mon[36569]: pgmap v13683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:36.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:35 smithi104 ceph-mon[36624]: pgmap v13683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:36.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:35 smithi081 ceph-mon[31453]: pgmap v13683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:38.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:37 smithi153 ceph-mon[36569]: pgmap v13684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:38.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:37 smithi104 ceph-mon[36624]: pgmap v13684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:38.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:37 smithi081 ceph-mon[31453]: pgmap v13684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:40.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:39 smithi153 ceph-mon[36569]: pgmap v13685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:40.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:39 smithi104 ceph-mon[36624]: pgmap v13685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:40.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:39 smithi081 ceph-mon[31453]: pgmap v13685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:42.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:41 smithi153 ceph-mon[36569]: pgmap v13686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:42.062+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:19:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:42.062+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:19:42.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:42.062+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:19:42.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:41 smithi104 ceph-mon[36624]: pgmap v13686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:42.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:41 smithi081 ceph-mon[31453]: pgmap v13686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:44.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:43 smithi153 ceph-mon[36569]: pgmap v13687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:44.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:43 smithi104 ceph-mon[36624]: pgmap v13687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:44.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:43 smithi081 ceph-mon[31453]: pgmap v13687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:46.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:45 smithi153 ceph-mon[36569]: pgmap v13688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:46.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:45 smithi104 ceph-mon[36624]: pgmap v13688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:46.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:45 smithi081 ceph-mon[31453]: pgmap v13688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:47.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:19:47.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:19:47.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:19:48.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:47 smithi153 ceph-mon[36569]: pgmap v13689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:48.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:47 smithi104 ceph-mon[36624]: pgmap v13689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:48.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:47 smithi081 ceph-mon[31453]: pgmap v13689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:49.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:19:49.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:19:49.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:19:49.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:19:49.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:19:49.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:19:49.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:19:49.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:19:49.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:19:50.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:49 smithi153 ceph-mon[36569]: pgmap v13690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:50.050 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:49 smithi153 ceph-mon[36569]: pgmap v13691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:50.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:49 smithi104 ceph-mon[36624]: pgmap v13690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:50.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:49 smithi104 ceph-mon[36624]: pgmap v13691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:50.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:49 smithi081 ceph-mon[31453]: pgmap v13690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:50.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:49 smithi081 ceph-mon[31453]: pgmap v13691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:52.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:51 smithi153 ceph-mon[36569]: pgmap v13692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:52.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:51 smithi104 ceph-mon[36624]: pgmap v13692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:52.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:51 smithi081 ceph-mon[31453]: pgmap v13692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:54.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:53 smithi153 ceph-mon[36569]: pgmap v13693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:54.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:53 smithi104 ceph-mon[36624]: pgmap v13693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:54.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:53 smithi081 ceph-mon[31453]: pgmap v13693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:56.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:55 smithi104 ceph-mon[36624]: pgmap v13694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:56.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:55 smithi081 ceph-mon[31453]: pgmap v13694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:56.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:55 smithi153 ceph-mon[36569]: pgmap v13694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:57.073+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:19:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:57.074+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:19:57.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:19:57.074+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:19:58.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:57 smithi104 ceph-mon[36624]: pgmap v13695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:58.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:57 smithi081 ceph-mon[31453]: pgmap v13695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:58.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:57 smithi153 ceph-mon[36569]: pgmap v13695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:59.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:19:58 smithi104 ceph-mon[36624]: pgmap v13696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:59.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:19:58 smithi081 ceph-mon[31453]: pgmap v13696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:19:59.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:19:58 smithi153 ceph-mon[36569]: pgmap v13696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:00.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:20:00.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:20:00.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:20:01.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:01 smithi153 ceph-mon[36569]: pgmap v13697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:01.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:01 smithi104 ceph-mon[36624]: pgmap v13697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:01.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:01 smithi081 ceph-mon[31453]: pgmap v13697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:03.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:03 smithi104 ceph-mon[36624]: pgmap v13698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:03.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:03 smithi081 ceph-mon[31453]: pgmap v13698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:03.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:03 smithi153 ceph-mon[36569]: pgmap v13698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:05.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:05 smithi104 ceph-mon[36624]: pgmap v13699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:05.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:05 smithi081 ceph-mon[31453]: pgmap v13699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:05.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:05 smithi153 ceph-mon[36569]: pgmap v13699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:07.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:07 smithi104 ceph-mon[36624]: pgmap v13700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:07.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:07 smithi081 ceph-mon[31453]: pgmap v13700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:07.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:07 smithi153 ceph-mon[36569]: pgmap v13700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:09.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:09 smithi104 ceph-mon[36624]: pgmap v13701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:09.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:09 smithi081 ceph-mon[31453]: pgmap v13701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:09.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:09 smithi153 ceph-mon[36569]: pgmap v13701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:11.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:11 smithi104 ceph-mon[36624]: pgmap v13702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:11.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:11 smithi081 ceph-mon[31453]: pgmap v13702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:11.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:11 smithi153 ceph-mon[36569]: pgmap v13702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:12.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:12.075+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:20:12.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:12.075+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:20:12.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:12.075+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:20:13.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:13 smithi081 ceph-mon[31453]: pgmap v13703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:13.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:13 smithi153 ceph-mon[36569]: pgmap v13703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:13.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:13 smithi104 ceph-mon[36624]: pgmap v13703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:15.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:15 smithi081 ceph-mon[31453]: pgmap v13704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:15.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:15 smithi153 ceph-mon[36569]: pgmap v13704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:15.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:15 smithi104 ceph-mon[36624]: pgmap v13704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:17.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:17 smithi081 ceph-mon[31453]: pgmap v13705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:17.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:17 smithi153 ceph-mon[36569]: pgmap v13705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:17.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:17 smithi104 ceph-mon[36624]: pgmap v13705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:19.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:19 smithi081 ceph-mon[31453]: pgmap v13706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:19.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:19 smithi153 ceph-mon[36569]: pgmap v13706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:19.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:19 smithi104 ceph-mon[36624]: pgmap v13706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:21.374 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:21 smithi104 ceph-mon[36624]: pgmap v13707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:21.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:21 smithi081 ceph-mon[31453]: pgmap v13707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:21.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:21 smithi153 ceph-mon[36569]: pgmap v13707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:23.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:23 smithi081 ceph-mon[31453]: pgmap v13708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:23.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:23 smithi153 ceph-mon[36569]: pgmap v13708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:23.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:23 smithi104 ceph-mon[36624]: pgmap v13708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:25.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:25 smithi081 ceph-mon[31453]: pgmap v13709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:25.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:25 smithi153 ceph-mon[36569]: pgmap v13709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:25.571 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:25 smithi104 ceph-mon[36624]: pgmap v13709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:27.091+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:20:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:27.091+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:20:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:27.091+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:20:28.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:27 smithi153 ceph-mon[36569]: pgmap v13710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:28.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:27 smithi104 ceph-mon[36624]: pgmap v13710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:28.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:27 smithi081 ceph-mon[31453]: pgmap v13710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:30.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:29 smithi153 ceph-mon[36569]: pgmap v13711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:30.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:29 smithi104 ceph-mon[36624]: pgmap v13711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:30.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:29 smithi081 ceph-mon[31453]: pgmap v13711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:32.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:31 smithi153 ceph-mon[36569]: pgmap v13712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:32.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:31 smithi104 ceph-mon[36624]: pgmap v13712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:32.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:31 smithi081 ceph-mon[31453]: pgmap v13712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:34.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:33 smithi153 ceph-mon[36569]: pgmap v13713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:34.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:33 smithi104 ceph-mon[36624]: pgmap v13713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:34.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:33 smithi081 ceph-mon[31453]: pgmap v13713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:36.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:35 smithi153 ceph-mon[36569]: pgmap v13714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:36.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:35 smithi104 ceph-mon[36624]: pgmap v13714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:36.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:35 smithi081 ceph-mon[31453]: pgmap v13714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:38.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:37 smithi153 ceph-mon[36569]: pgmap v13715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:38.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:37 smithi104 ceph-mon[36624]: pgmap v13715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:38.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:37 smithi081 ceph-mon[31453]: pgmap v13715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:39.896 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:39 smithi081 ceph-mon[31453]: pgmap v13716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:40.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:39 smithi153 ceph-mon[36569]: pgmap v13716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:40.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:39 smithi104 ceph-mon[36624]: pgmap v13716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:42.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:41 smithi153 ceph-mon[36569]: pgmap v13717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:42.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:41 smithi104 ceph-mon[36624]: pgmap v13717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:42.106+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:20:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:42.106+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:20:42.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:42.106+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:20:42.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:41 smithi081 ceph-mon[31453]: pgmap v13717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:44.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:43 smithi153 ceph-mon[36569]: pgmap v13718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:44.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:43 smithi104 ceph-mon[36624]: pgmap v13718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:44.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:43 smithi081 ceph-mon[31453]: pgmap v13718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:46.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:45 smithi153 ceph-mon[36569]: pgmap v13719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:46.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:45 smithi104 ceph-mon[36624]: pgmap v13719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:46.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:45 smithi081 ceph-mon[31453]: pgmap v13719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:48.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:47 smithi153 ceph-mon[36569]: pgmap v13720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:48.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:47 smithi104 ceph-mon[36624]: pgmap v13720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:48.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:47 smithi081 ceph-mon[31453]: pgmap v13720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:49.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:20:49.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:20:49.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:20:50.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:49 smithi104 ceph-mon[36624]: pgmap v13721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:50.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:20:50.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:20:50.072 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:20:50.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:49 smithi081 ceph-mon[31453]: pgmap v13721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:50.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:20:50.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:20:50.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:20:50.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:49 smithi153 ceph-mon[36569]: pgmap v13721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:50.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:20:50.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:20:50.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:20:51.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:50 smithi104 ceph-mon[36624]: pgmap v13722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:51.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:50 smithi104 ceph-mon[36624]: pgmap v13723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:51.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:50 smithi081 ceph-mon[31453]: pgmap v13722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:51.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:50 smithi081 ceph-mon[31453]: pgmap v13723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:51.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:50 smithi153 ceph-mon[36569]: pgmap v13722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:51.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:50 smithi153 ceph-mon[36569]: pgmap v13723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:53.071 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:52 smithi104 ceph-mon[36624]: pgmap v13724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:53.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:52 smithi081 ceph-mon[31453]: pgmap v13724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:53.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:52 smithi153 ceph-mon[36569]: pgmap v13724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:54.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:53 smithi081 ceph-mon[31453]: pgmap v13725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:53 smithi153 ceph-mon[36569]: pgmap v13725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:54.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:53 smithi104 ceph-mon[36624]: pgmap v13725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:56.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:56 smithi153 ceph-mon[36569]: pgmap v13726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:56.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:56 smithi104 ceph-mon[36624]: pgmap v13726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:56.849 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:56 smithi081 ceph-mon[31453]: pgmap v13726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:57.120+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:20:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:57.120+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:20:57.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:20:57.120+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:20:58.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:20:58 smithi153 ceph-mon[36569]: pgmap v13727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:58.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:20:58 smithi104 ceph-mon[36624]: pgmap v13727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:20:58.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:20:58 smithi081 ceph-mon[31453]: pgmap v13727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:00.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:00 smithi153 ceph-mon[36569]: pgmap v13728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:00.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:00 smithi104 ceph-mon[36624]: pgmap v13728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:00.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:00 smithi081 ceph-mon[31453]: pgmap v13728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:02.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:02 smithi153 ceph-mon[36569]: pgmap v13729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:02.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:02 smithi104 ceph-mon[36624]: pgmap v13729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:02.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:02 smithi081 ceph-mon[31453]: pgmap v13729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:04.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:04 smithi104 ceph-mon[36624]: pgmap v13730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:04.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:04 smithi081 ceph-mon[31453]: pgmap v13730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:05.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:04 smithi153 ceph-mon[36569]: pgmap v13730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:06.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:06 smithi104 ceph-mon[36624]: pgmap v13731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:06.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:06 smithi081 ceph-mon[31453]: pgmap v13731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:07.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:06 smithi153 ceph-mon[36569]: pgmap v13731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:08.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:08 smithi104 ceph-mon[36624]: pgmap v13732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:08.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:08 smithi081 ceph-mon[31453]: pgmap v13732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:09.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:08 smithi153 ceph-mon[36569]: pgmap v13732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:10.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:10 smithi104 ceph-mon[36624]: pgmap v13733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:10.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:10 smithi081 ceph-mon[31453]: pgmap v13733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:11.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:10 smithi153 ceph-mon[36569]: pgmap v13733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:12.134+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:21:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:12.134+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:21:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:12.134+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:21:12.820 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:12 smithi104 ceph-mon[36624]: pgmap v13734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:12.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:12 smithi081 ceph-mon[31453]: pgmap v13734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:13.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:12 smithi153 ceph-mon[36569]: pgmap v13734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:14.895 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:14 smithi081 ceph-mon[31453]: pgmap v13735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:15.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:14 smithi153 ceph-mon[36569]: pgmap v13735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:15.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:14 smithi104 ceph-mon[36624]: pgmap v13735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:16.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:16 smithi081 ceph-mon[31453]: pgmap v13736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:17.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:16 smithi153 ceph-mon[36569]: pgmap v13736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:17.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:16 smithi104 ceph-mon[36624]: pgmap v13736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:18.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:18 smithi081 ceph-mon[31453]: pgmap v13737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:19.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:18 smithi153 ceph-mon[36569]: pgmap v13737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:19.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:18 smithi104 ceph-mon[36624]: pgmap v13737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:20.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:20 smithi081 ceph-mon[31453]: pgmap v13738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:21.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:20 smithi153 ceph-mon[36569]: pgmap v13738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:21.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:20 smithi104 ceph-mon[36624]: pgmap v13738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:22.873 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:22 smithi104 ceph-mon[36624]: pgmap v13739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:22.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:22 smithi081 ceph-mon[31453]: pgmap v13739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:23.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:22 smithi153 ceph-mon[36569]: pgmap v13739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:24.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:24 smithi081 ceph-mon[31453]: pgmap v13740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:25.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:24 smithi153 ceph-mon[36569]: pgmap v13740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:25.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:24 smithi104 ceph-mon[36624]: pgmap v13740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:26.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:26 smithi081 ceph-mon[31453]: pgmap v13741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:27.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:26 smithi153 ceph-mon[36569]: pgmap v13741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:27.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:26 smithi104 ceph-mon[36624]: pgmap v13741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:27.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:27.150+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:21:27.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:27.150+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:21:27.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:27.150+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:21:28.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:28 smithi081 ceph-mon[31453]: pgmap v13742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:29.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:28 smithi153 ceph-mon[36569]: pgmap v13742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:29.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:28 smithi104 ceph-mon[36624]: pgmap v13742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:30.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:30 smithi081 ceph-mon[31453]: pgmap v13743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:31.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:30 smithi153 ceph-mon[36569]: pgmap v13743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:31.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:30 smithi104 ceph-mon[36624]: pgmap v13743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:32.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:32 smithi081 ceph-mon[31453]: pgmap v13744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:33.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:32 smithi153 ceph-mon[36569]: pgmap v13744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:33.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:32 smithi104 ceph-mon[36624]: pgmap v13744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:34.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:34 smithi081 ceph-mon[31453]: pgmap v13745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:35.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:34 smithi153 ceph-mon[36569]: pgmap v13745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:35.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:34 smithi104 ceph-mon[36624]: pgmap v13745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:36.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:36 smithi081 ceph-mon[31453]: pgmap v13746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:37.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:36 smithi153 ceph-mon[36569]: pgmap v13746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:37.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:36 smithi104 ceph-mon[36624]: pgmap v13746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:38.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:38 smithi081 ceph-mon[31453]: pgmap v13747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:39.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:38 smithi153 ceph-mon[36569]: pgmap v13747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:39.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:38 smithi104 ceph-mon[36624]: pgmap v13747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:40.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:40 smithi081 ceph-mon[31453]: pgmap v13748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:41.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:40 smithi153 ceph-mon[36569]: pgmap v13748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:41.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:40 smithi104 ceph-mon[36624]: pgmap v13748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:42.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:42.164+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:21:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:42.164+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:21:42.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:42.164+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:21:42.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:42 smithi081 ceph-mon[31453]: pgmap v13749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:43.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:42 smithi153 ceph-mon[36569]: pgmap v13749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:43.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:42 smithi104 ceph-mon[36624]: pgmap v13749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:45.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:44 smithi153 ceph-mon[36569]: pgmap v13750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:45.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:44 smithi104 ceph-mon[36624]: pgmap v13750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:45.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:44 smithi081 ceph-mon[31453]: pgmap v13750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:47.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:46 smithi153 ceph-mon[36569]: pgmap v13751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:47.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:46 smithi104 ceph-mon[36624]: pgmap v13751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:47.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:46 smithi081 ceph-mon[31453]: pgmap v13751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:49.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:48 smithi153 ceph-mon[36569]: pgmap v13752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:49.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:48 smithi104 ceph-mon[36624]: pgmap v13752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:49.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:48 smithi081 ceph-mon[31453]: pgmap v13752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:51.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:50 smithi153 ceph-mon[36569]: pgmap v13753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:51.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:21:51.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:50 smithi104 ceph-mon[36624]: pgmap v13753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:51.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:21:51.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:50 smithi081 ceph-mon[31453]: pgmap v13753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:51.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:21:52.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:21:52.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:21:52.045 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:21:52.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:21:52.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:21:52.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:21:52.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:21:52.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:21:52.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:21:52.971 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:52 smithi104 ceph-mon[36624]: pgmap v13754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:52.971 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:52 smithi104 ceph-mon[36624]: pgmap v13755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:52 smithi153 ceph-mon[36569]: pgmap v13754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:53.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:52 smithi153 ceph-mon[36569]: pgmap v13755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:53.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:52 smithi081 ceph-mon[31453]: pgmap v13754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:53.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:52 smithi081 ceph-mon[31453]: pgmap v13755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:55.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:54 smithi153 ceph-mon[36569]: pgmap v13756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:55.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:54 smithi104 ceph-mon[36624]: pgmap v13756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:55.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:54 smithi081 ceph-mon[31453]: pgmap v13756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:56.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:56 smithi081 ceph-mon[31453]: pgmap v13757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:57.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:56 smithi153 ceph-mon[36569]: pgmap v13757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:57.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:56 smithi104 ceph-mon[36624]: pgmap v13757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:57.180+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:21:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:57.180+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:21:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:21:57.180+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:21:59.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:21:58 smithi153 ceph-mon[36569]: pgmap v13758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:59.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:21:58 smithi104 ceph-mon[36624]: pgmap v13758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:21:59.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:21:58 smithi081 ceph-mon[31453]: pgmap v13758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:01.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:00 smithi153 ceph-mon[36569]: pgmap v13759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:01.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:00 smithi104 ceph-mon[36624]: pgmap v13759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:01.083 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:00 smithi081 ceph-mon[31453]: pgmap v13759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:03.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:02 smithi153 ceph-mon[36569]: pgmap v13760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:03.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:02 smithi104 ceph-mon[36624]: pgmap v13760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:03.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:02 smithi081 ceph-mon[31453]: pgmap v13760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:05.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:04 smithi153 ceph-mon[36569]: pgmap v13761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:05.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:04 smithi104 ceph-mon[36624]: pgmap v13761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:05.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:04 smithi081 ceph-mon[31453]: pgmap v13761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:07.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:06 smithi153 ceph-mon[36569]: pgmap v13762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:07.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:06 smithi104 ceph-mon[36624]: pgmap v13762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:07.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:06 smithi081 ceph-mon[31453]: pgmap v13762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:09.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:08 smithi153 ceph-mon[36569]: pgmap v13763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:09.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:08 smithi104 ceph-mon[36624]: pgmap v13763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:09.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:08 smithi081 ceph-mon[31453]: pgmap v13763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:11.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:10 smithi153 ceph-mon[36569]: pgmap v13764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:11.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:10 smithi104 ceph-mon[36624]: pgmap v13764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:11.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:10 smithi081 ceph-mon[31453]: pgmap v13764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:12.186+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:22:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:12.186+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:22:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:12.186+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:22:13.046 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:12 smithi153 ceph-mon[36569]: pgmap v13765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:13.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:12 smithi104 ceph-mon[36624]: pgmap v13765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:13.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:12 smithi081 ceph-mon[31453]: pgmap v13765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:15.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:14 smithi153 ceph-mon[36569]: pgmap v13766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:15.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:14 smithi104 ceph-mon[36624]: pgmap v13766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:15.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:14 smithi081 ceph-mon[31453]: pgmap v13766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:16.929+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:22:16.916 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:16.929+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:22:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:16.929+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 198 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:22:16.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:16.929+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 198 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:22:17.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:16 smithi153 ceph-mon[36569]: pgmap v13767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:17.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:16 smithi104 ceph-mon[36624]: pgmap v13767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:17.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:16 smithi081 ceph-mon[31453]: pgmap v13767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:19.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:18 smithi153 ceph-mon[36569]: pgmap v13768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:19.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:18 smithi104 ceph-mon[36624]: pgmap v13768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:19.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:18 smithi081 ceph-mon[31453]: pgmap v13768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:21.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:20 smithi153 ceph-mon[36569]: pgmap v13769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:21.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:20 smithi104 ceph-mon[36624]: pgmap v13769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:21.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:20 smithi081 ceph-mon[31453]: pgmap v13769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:23.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:22 smithi153 ceph-mon[36569]: pgmap v13770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:23.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:22 smithi104 ceph-mon[36624]: pgmap v13770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:23.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:22 smithi081 ceph-mon[31453]: pgmap v13770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:25.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:24 smithi153 ceph-mon[36569]: pgmap v13771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:25.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:24 smithi104 ceph-mon[36624]: pgmap v13771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:25.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:24 smithi081 ceph-mon[31453]: pgmap v13771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:27.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:26 smithi153 ceph-mon[36569]: pgmap v13772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:27.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:26 smithi104 ceph-mon[36624]: pgmap v13772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:27.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:26 smithi081 ceph-mon[31453]: pgmap v13772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:27.190+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:22:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:27.190+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:22:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:27.190+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:22:29.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:28 smithi153 ceph-mon[36569]: pgmap v13773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:29.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:28 smithi104 ceph-mon[36624]: pgmap v13773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:29.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:28 smithi081 ceph-mon[31453]: pgmap v13773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:31.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:30 smithi153 ceph-mon[36569]: pgmap v13774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:31.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:30 smithi104 ceph-mon[36624]: pgmap v13774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:31.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:30 smithi081 ceph-mon[31453]: pgmap v13774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:33.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:32 smithi153 ceph-mon[36569]: pgmap v13775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:33.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:32 smithi104 ceph-mon[36624]: pgmap v13775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:33.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:32 smithi081 ceph-mon[31453]: pgmap v13775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:35.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:34 smithi153 ceph-mon[36569]: pgmap v13776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:35.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:34 smithi104 ceph-mon[36624]: pgmap v13776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:35.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:34 smithi081 ceph-mon[31453]: pgmap v13776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:37.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:36 smithi153 ceph-mon[36569]: pgmap v13777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:37.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:36 smithi104 ceph-mon[36624]: pgmap v13777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:37.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:36 smithi081 ceph-mon[31453]: pgmap v13777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:39.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:38 smithi104 ceph-mon[36624]: pgmap v13778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:39.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:38 smithi081 ceph-mon[31453]: pgmap v13778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:38 smithi153 ceph-mon[36569]: pgmap v13778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:41.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:40 smithi104 ceph-mon[36624]: pgmap v13779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:41.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:40 smithi081 ceph-mon[31453]: pgmap v13779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:41.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:40 smithi153 ceph-mon[36569]: pgmap v13779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:42.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:42.196+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:22:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:42.196+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:22:42.184 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:42.196+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:22:43.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:42 smithi104 ceph-mon[36624]: pgmap v13780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:43.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:42 smithi081 ceph-mon[31453]: pgmap v13780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:43.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:42 smithi153 ceph-mon[36569]: pgmap v13780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:45.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:44 smithi104 ceph-mon[36624]: pgmap v13781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:45.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:44 smithi081 ceph-mon[31453]: pgmap v13781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:45.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:44 smithi153 ceph-mon[36569]: pgmap v13781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:47.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:46 smithi104 ceph-mon[36624]: pgmap v13782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:47.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:46 smithi081 ceph-mon[31453]: pgmap v13782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:46 smithi153 ceph-mon[36569]: pgmap v13782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:49.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:48 smithi081 ceph-mon[31453]: pgmap v13783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:49.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:48 smithi153 ceph-mon[36569]: pgmap v13783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:49.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:48 smithi104 ceph-mon[36624]: pgmap v13783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:51.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:50 smithi081 ceph-mon[31453]: pgmap v13784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:51.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:50 smithi153 ceph-mon[36569]: pgmap v13784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:51.319 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:50 smithi104 ceph-mon[36624]: pgmap v13784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:52.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:22:52.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:52.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:52.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:22:52.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:52.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:52.220 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:22:52.220 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:52.220 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:53.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:52 smithi104 ceph-mon[36624]: pgmap v13785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:53.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:53.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:53.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:52 smithi081 ceph-mon[31453]: pgmap v13785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:53.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:53.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:53.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:52 smithi153 ceph-mon[36569]: pgmap v13785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:53.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:53.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:54 smithi153 ceph-mon[36569]: pgmap v13786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:55.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:54 smithi104 ceph-mon[36624]: pgmap v13786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:55.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:54 smithi081 ceph-mon[31453]: pgmap v13786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:55.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:55.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:56.946 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:56 smithi081 ceph-mon[31453]: pgmap v13787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:56.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:56.947 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:57.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:56 smithi153 ceph-mon[36569]: pgmap v13787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:57.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:57.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:57.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:56 smithi104 ceph-mon[36624]: pgmap v13787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:57.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:57.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:57.211+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:22:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:57.211+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:22:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:22:57.211+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:22:58.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:22:58.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:22:58.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:22:58.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:22:58.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:22:58.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:22:59.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:58 smithi153 ceph-mon[36569]: pgmap v13788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:58 smithi153 ceph-mon[36569]: pgmap v13789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:58 smithi153 ceph-mon[36569]: pgmap v13790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:22:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:59.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:58 smithi104 ceph-mon[36624]: pgmap v13788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:58 smithi104 ceph-mon[36624]: pgmap v13789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:58 smithi104 ceph-mon[36624]: pgmap v13790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:22:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:22:59.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:58 smithi081 ceph-mon[31453]: pgmap v13788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:58 smithi081 ceph-mon[31453]: pgmap v13789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:58 smithi081 ceph-mon[31453]: pgmap v13790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:22:59.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:22:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:23:01.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:00 smithi153 ceph-mon[36569]: pgmap v13791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:01.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:00 smithi104 ceph-mon[36624]: pgmap v13791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:01.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:00 smithi081 ceph-mon[31453]: pgmap v13791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:03.034 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:02 smithi081 ceph-mon[31453]: pgmap v13792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:03.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:02 smithi153 ceph-mon[36569]: pgmap v13792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:03.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:02 smithi104 ceph-mon[36624]: pgmap v13792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:05.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:04 smithi153 ceph-mon[36569]: pgmap v13793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:05.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:04 smithi104 ceph-mon[36624]: pgmap v13793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:05.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:04 smithi081 ceph-mon[31453]: pgmap v13793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:07.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:06 smithi153 ceph-mon[36569]: pgmap v13794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:07.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:06 smithi104 ceph-mon[36624]: pgmap v13794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:07.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:06 smithi081 ceph-mon[31453]: pgmap v13794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:09.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:08 smithi153 ceph-mon[36569]: pgmap v13795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:09.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:08 smithi104 ceph-mon[36624]: pgmap v13795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:09.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:08 smithi081 ceph-mon[31453]: pgmap v13795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:11.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:10 smithi153 ceph-mon[36569]: pgmap v13796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:11.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:10 smithi104 ceph-mon[36624]: pgmap v13796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:11.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:10 smithi081 ceph-mon[31453]: pgmap v13796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:12.227+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:23:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:12.227+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:23:12.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:12.227+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:23:13.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:12 smithi153 ceph-mon[36569]: pgmap v13797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:13.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:12 smithi104 ceph-mon[36624]: pgmap v13797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:13.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:12 smithi081 ceph-mon[31453]: pgmap v13797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:15.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:14 smithi153 ceph-mon[36569]: pgmap v13798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:15.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:14 smithi104 ceph-mon[36624]: pgmap v13798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:15.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:14 smithi081 ceph-mon[31453]: pgmap v13798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:17.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:16 smithi153 ceph-mon[36569]: pgmap v13799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:17.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:16 smithi104 ceph-mon[36624]: pgmap v13799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:17.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:16 smithi081 ceph-mon[31453]: pgmap v13799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:19.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:18 smithi153 ceph-mon[36569]: pgmap v13800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:19.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:18 smithi104 ceph-mon[36624]: pgmap v13800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:19.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:18 smithi081 ceph-mon[31453]: pgmap v13800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:21.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:20 smithi153 ceph-mon[36569]: pgmap v13801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:21.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:20 smithi104 ceph-mon[36624]: pgmap v13801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:21.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:20 smithi081 ceph-mon[31453]: pgmap v13801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:23.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:22 smithi153 ceph-mon[36569]: pgmap v13802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:23.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:22 smithi104 ceph-mon[36624]: pgmap v13802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:23.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:22 smithi081 ceph-mon[31453]: pgmap v13802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:25.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:24 smithi153 ceph-mon[36569]: pgmap v13803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:25.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:24 smithi104 ceph-mon[36624]: pgmap v13803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:25.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:24 smithi081 ceph-mon[31453]: pgmap v13803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:27.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:26 smithi153 ceph-mon[36569]: pgmap v13804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:27.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:26 smithi104 ceph-mon[36624]: pgmap v13804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:27.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:26 smithi081 ceph-mon[31453]: pgmap v13804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:27.240+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:23:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:27.240+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:23:27.227 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:27.240+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:23:29.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:28 smithi153 ceph-mon[36569]: pgmap v13805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:29.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:28 smithi104 ceph-mon[36624]: pgmap v13805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:29.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:28 smithi081 ceph-mon[31453]: pgmap v13805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:31.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:30 smithi153 ceph-mon[36569]: pgmap v13806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:31.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:30 smithi104 ceph-mon[36624]: pgmap v13806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:31.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:30 smithi081 ceph-mon[31453]: pgmap v13806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:33.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:32 smithi153 ceph-mon[36569]: pgmap v13807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:33.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:32 smithi104 ceph-mon[36624]: pgmap v13807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:33.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:32 smithi081 ceph-mon[31453]: pgmap v13807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:35.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:34 smithi153 ceph-mon[36569]: pgmap v13808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:35.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:34 smithi104 ceph-mon[36624]: pgmap v13808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:35.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:34 smithi081 ceph-mon[31453]: pgmap v13808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:37.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:36 smithi104 ceph-mon[36624]: pgmap v13809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:37.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:36 smithi081 ceph-mon[31453]: pgmap v13809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:37.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:36 smithi153 ceph-mon[36569]: pgmap v13809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:39.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:38 smithi104 ceph-mon[36624]: pgmap v13810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:39.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:38 smithi081 ceph-mon[31453]: pgmap v13810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:39.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:38 smithi153 ceph-mon[36569]: pgmap v13810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:41.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:40 smithi104 ceph-mon[36624]: pgmap v13811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:41.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:40 smithi081 ceph-mon[31453]: pgmap v13811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:41.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:40 smithi153 ceph-mon[36569]: pgmap v13811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:42.249+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:23:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:42.250+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:23:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:42.250+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:23:43.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:42 smithi104 ceph-mon[36624]: pgmap v13812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:43.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:42 smithi081 ceph-mon[31453]: pgmap v13812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:43.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:42 smithi153 ceph-mon[36569]: pgmap v13812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:45.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:44 smithi081 ceph-mon[31453]: pgmap v13813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:45.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:44 smithi153 ceph-mon[36569]: pgmap v13813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:45.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:44 smithi104 ceph-mon[36624]: pgmap v13813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:47.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:46 smithi081 ceph-mon[31453]: pgmap v13814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:47.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:46 smithi153 ceph-mon[36569]: pgmap v13814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:47.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:46 smithi104 ceph-mon[36624]: pgmap v13814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:49.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:48 smithi081 ceph-mon[31453]: pgmap v13815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:49.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:48 smithi153 ceph-mon[36569]: pgmap v13815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:49.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:48 smithi104 ceph-mon[36624]: pgmap v13815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:51.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:50 smithi081 ceph-mon[31453]: pgmap v13816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:51.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:50 smithi153 ceph-mon[36569]: pgmap v13816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:51.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:50 smithi104 ceph-mon[36624]: pgmap v13816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:52.219 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:51 smithi104 ceph-mon[36624]: pgmap v13817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:52.293 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:51 smithi153 ceph-mon[36569]: pgmap v13817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:52.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:51 smithi081 ceph-mon[31453]: pgmap v13817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:55.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:54 smithi153 ceph-mon[36569]: pgmap v13818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:55.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:54 smithi104 ceph-mon[36624]: pgmap v13818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:55.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:54 smithi081 ceph-mon[31453]: pgmap v13818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:57.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:56 smithi153 ceph-mon[36569]: pgmap v13819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:57.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:56 smithi104 ceph-mon[36624]: pgmap v13819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:57.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:56 smithi081 ceph-mon[31453]: pgmap v13819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:57.259+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:23:57.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:57.259+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:23:57.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:23:57.259+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:23:59.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:58 smithi153 ceph-mon[36569]: pgmap v13820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:59.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:23:59.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:23:59.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:23:59.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:58 smithi104 ceph-mon[36624]: pgmap v13820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:59.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:23:59.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:23:59.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:23:59.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:58 smithi081 ceph-mon[31453]: pgmap v13820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:23:59.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:23:59.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:23:59.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:24:00.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:24:00.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:24:00.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:23:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:24:00.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:24:00.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:24:00.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:23:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:24:00.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:24:00.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:24:00.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:23:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:24:01.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:00 smithi153 ceph-mon[36569]: pgmap v13821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:01.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:00 smithi153 ceph-mon[36569]: pgmap v13822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:01.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:00 smithi104 ceph-mon[36624]: pgmap v13821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:01.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:00 smithi104 ceph-mon[36624]: pgmap v13822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:01.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:00 smithi081 ceph-mon[31453]: pgmap v13821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:01.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:00 smithi081 ceph-mon[31453]: pgmap v13822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:03.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:02 smithi104 ceph-mon[36624]: pgmap v13823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:03.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:02 smithi081 ceph-mon[31453]: pgmap v13823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:03.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:02 smithi153 ceph-mon[36569]: pgmap v13823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:05.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:04 smithi104 ceph-mon[36624]: pgmap v13824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:05.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:04 smithi081 ceph-mon[31453]: pgmap v13824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:05.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:04 smithi153 ceph-mon[36569]: pgmap v13824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:07.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:06 smithi104 ceph-mon[36624]: pgmap v13825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:07.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:06 smithi081 ceph-mon[31453]: pgmap v13825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:07.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:06 smithi153 ceph-mon[36569]: pgmap v13825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:09.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:08 smithi104 ceph-mon[36624]: pgmap v13826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:09.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:08 smithi081 ceph-mon[31453]: pgmap v13826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:09.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:08 smithi153 ceph-mon[36569]: pgmap v13826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:11.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:10 smithi081 ceph-mon[31453]: pgmap v13827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:11.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:10 smithi153 ceph-mon[36569]: pgmap v13827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:11.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:10 smithi104 ceph-mon[36624]: pgmap v13827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:12.260+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:24:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:12.260+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:24:12.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:12.260+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:24:13.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:12 smithi081 ceph-mon[31453]: pgmap v13828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:13.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:12 smithi153 ceph-mon[36569]: pgmap v13828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:13.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:12 smithi104 ceph-mon[36624]: pgmap v13828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:15.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:14 smithi081 ceph-mon[31453]: pgmap v13829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:15.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:14 smithi153 ceph-mon[36569]: pgmap v13829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:15.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:14 smithi104 ceph-mon[36624]: pgmap v13829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:17.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:16 smithi081 ceph-mon[31453]: pgmap v13830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:17.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:16 smithi153 ceph-mon[36569]: pgmap v13830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:17.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:16 smithi104 ceph-mon[36624]: pgmap v13830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:19.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:18 smithi081 ceph-mon[31453]: pgmap v13831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:19.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:18 smithi153 ceph-mon[36569]: pgmap v13831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:19.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:18 smithi104 ceph-mon[36624]: pgmap v13831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:21.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:20 smithi081 ceph-mon[31453]: pgmap v13832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:21.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:20 smithi153 ceph-mon[36569]: pgmap v13832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:21.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:20 smithi104 ceph-mon[36624]: pgmap v13832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:23.120 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:22 smithi104 ceph-mon[36624]: pgmap v13833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:23.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:22 smithi081 ceph-mon[31453]: pgmap v13833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:23.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:22 smithi153 ceph-mon[36569]: pgmap v13833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:25.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:24 smithi081 ceph-mon[31453]: pgmap v13834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:25.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:24 smithi153 ceph-mon[36569]: pgmap v13834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:25.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:24 smithi104 ceph-mon[36624]: pgmap v13834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:26.193 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:25 smithi153 ceph-mon[36569]: pgmap v13835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:26.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:25 smithi104 ceph-mon[36624]: pgmap v13835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:26.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:25 smithi081 ceph-mon[31453]: pgmap v13835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:27.264+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:24:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:27.265+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:24:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:27.265+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:24:29.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:28 smithi153 ceph-mon[36569]: pgmap v13836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:29.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:28 smithi081 ceph-mon[31453]: pgmap v13836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:29.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:28 smithi104 ceph-mon[36624]: pgmap v13836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:31.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:30 smithi081 ceph-mon[31453]: pgmap v13837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:31.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:30 smithi153 ceph-mon[36569]: pgmap v13837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:31.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:30 smithi104 ceph-mon[36624]: pgmap v13837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:33.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:32 smithi081 ceph-mon[31453]: pgmap v13838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:33.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:32 smithi153 ceph-mon[36569]: pgmap v13838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:33.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:32 smithi104 ceph-mon[36624]: pgmap v13838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:35.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:34 smithi081 ceph-mon[31453]: pgmap v13839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:35.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:34 smithi153 ceph-mon[36569]: pgmap v13839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:35.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:34 smithi104 ceph-mon[36624]: pgmap v13839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:37.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:36 smithi081 ceph-mon[31453]: pgmap v13840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:37.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:36 smithi153 ceph-mon[36569]: pgmap v13840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:37.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:36 smithi104 ceph-mon[36624]: pgmap v13840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:39.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:38 smithi153 ceph-mon[36569]: pgmap v13841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:39.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:38 smithi081 ceph-mon[31453]: pgmap v13841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:39.318 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:38 smithi104 ceph-mon[36624]: pgmap v13841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:41.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:40 smithi081 ceph-mon[31453]: pgmap v13842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:41.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:40 smithi153 ceph-mon[36569]: pgmap v13842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:41.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:40 smithi104 ceph-mon[36624]: pgmap v13842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:42.280+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:24:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:42.280+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:24:42.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:42.280+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:24:43.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:42 smithi081 ceph-mon[31453]: pgmap v13843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:43.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:42 smithi153 ceph-mon[36569]: pgmap v13843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:43.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:42 smithi104 ceph-mon[36624]: pgmap v13843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:44.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:43 smithi153 ceph-mon[36569]: pgmap v13844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:44.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:43 smithi104 ceph-mon[36624]: pgmap v13844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:44.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:43 smithi081 ceph-mon[31453]: pgmap v13844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:46.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:46 smithi081 ceph-mon[31453]: pgmap v13845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:46.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:46 smithi153 ceph-mon[36569]: pgmap v13845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:46.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:46 smithi104 ceph-mon[36624]: pgmap v13845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:48.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:48 smithi153 ceph-mon[36569]: pgmap v13846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:48.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:48 smithi104 ceph-mon[36624]: pgmap v13846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:48.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:48 smithi081 ceph-mon[31453]: pgmap v13846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:50.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:50 smithi153 ceph-mon[36569]: pgmap v13847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:50.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:50 smithi104 ceph-mon[36624]: pgmap v13847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:50.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:50 smithi081 ceph-mon[31453]: pgmap v13847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:52.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:52 smithi153 ceph-mon[36569]: pgmap v13848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:52.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:52 smithi104 ceph-mon[36624]: pgmap v13848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:52.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:52 smithi081 ceph-mon[31453]: pgmap v13848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:54.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:54 smithi153 ceph-mon[36569]: pgmap v13849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:54.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:54 smithi104 ceph-mon[36624]: pgmap v13849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:54.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:54 smithi081 ceph-mon[31453]: pgmap v13849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:56.694 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:56 smithi081 ceph-mon[31453]: pgmap v13850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:56.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:56 smithi153 ceph-mon[36569]: pgmap v13850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:56.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:56 smithi104 ceph-mon[36624]: pgmap v13850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:57.295+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:24:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:57.295+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:24:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:24:57.295+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:24:58.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:24:58 smithi153 ceph-mon[36569]: pgmap v13851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:58.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:24:58 smithi104 ceph-mon[36624]: pgmap v13851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:24:58.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:24:58 smithi081 ceph-mon[31453]: pgmap v13851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:00.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:00 smithi153 ceph-mon[36569]: pgmap v13852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:00.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:25:00.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:00 smithi104 ceph-mon[36624]: pgmap v13852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:00.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:25:00.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:00 smithi081 ceph-mon[31453]: pgmap v13852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:00.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:25:01.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:25:01.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:25:01.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:01 smithi153 ceph-mon[36569]: pgmap v13853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:01.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:01 smithi153 ceph-mon[36569]: pgmap v13854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:01.792 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:25:01.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:25:01.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:25:01.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:01 smithi104 ceph-mon[36624]: pgmap v13853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:01.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:01 smithi104 ceph-mon[36624]: pgmap v13854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:01.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:25:01.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:25:01.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:25:01.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:01 smithi081 ceph-mon[31453]: pgmap v13853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:01.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:01 smithi081 ceph-mon[31453]: pgmap v13854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:01.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:25:04.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:03 smithi081 ceph-mon[31453]: pgmap v13855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:04.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:03 smithi153 ceph-mon[36569]: pgmap v13855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:04.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:03 smithi104 ceph-mon[36624]: pgmap v13855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:06.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:05 smithi081 ceph-mon[31453]: pgmap v13856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:06.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:05 smithi153 ceph-mon[36569]: pgmap v13856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:06.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:05 smithi104 ceph-mon[36624]: pgmap v13856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:08.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:07 smithi081 ceph-mon[31453]: pgmap v13857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:08.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:07 smithi153 ceph-mon[36569]: pgmap v13857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:08.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:07 smithi104 ceph-mon[36624]: pgmap v13857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:10.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:09 smithi081 ceph-mon[31453]: pgmap v13858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:10.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:09 smithi153 ceph-mon[36569]: pgmap v13858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:10.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:09 smithi104 ceph-mon[36624]: pgmap v13858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:12.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:11 smithi081 ceph-mon[31453]: pgmap v13859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:12.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:11 smithi153 ceph-mon[36569]: pgmap v13859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:12.312+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:25:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:12.312+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:25:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:12.312+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:25:12.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:11 smithi104 ceph-mon[36624]: pgmap v13859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:14.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:13 smithi081 ceph-mon[31453]: pgmap v13860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:14.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:13 smithi153 ceph-mon[36569]: pgmap v13860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:14.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:13 smithi104 ceph-mon[36624]: pgmap v13860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:16.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:15 smithi081 ceph-mon[31453]: pgmap v13861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:16.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:15 smithi153 ceph-mon[36569]: pgmap v13861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:16.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:15 smithi104 ceph-mon[36624]: pgmap v13861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:16.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:16.958+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:25:16.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:16.958+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:25:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:16.959+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 199 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:25:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:16.959+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 199 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:25:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:16.959+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:25:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:16.959+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:25:16.944 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:16.959+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 200 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:25:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:16.959+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 200 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:25:18.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:17 smithi081 ceph-mon[31453]: pgmap v13862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:18.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:17 smithi153 ceph-mon[36569]: pgmap v13862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:18.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:17 smithi104 ceph-mon[36624]: pgmap v13862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:20.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:19 smithi081 ceph-mon[31453]: pgmap v13863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:20.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:19 smithi153 ceph-mon[36569]: pgmap v13863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:20.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:19 smithi104 ceph-mon[36624]: pgmap v13863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:21.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:20 smithi081 ceph-mon[31453]: pgmap v13864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:21.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:20 smithi153 ceph-mon[36569]: pgmap v13864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:21.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:20 smithi104 ceph-mon[36624]: pgmap v13864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:24.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:23 smithi081 ceph-mon[31453]: pgmap v13865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:24.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:23 smithi153 ceph-mon[36569]: pgmap v13865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:24.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:23 smithi104 ceph-mon[36624]: pgmap v13865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:26.094 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:25 smithi153 ceph-mon[36569]: pgmap v13866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:26.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:25 smithi081 ceph-mon[31453]: pgmap v13866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:26.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:25 smithi104 ceph-mon[36624]: pgmap v13866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:27.318+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:25:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:27.318+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:25:27.303 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:27.318+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:25:28.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:27 smithi081 ceph-mon[31453]: pgmap v13867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:28.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:27 smithi153 ceph-mon[36569]: pgmap v13867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:28.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:27 smithi104 ceph-mon[36624]: pgmap v13867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:30.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:29 smithi081 ceph-mon[31453]: pgmap v13868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:30.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:29 smithi153 ceph-mon[36569]: pgmap v13868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:30.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:29 smithi104 ceph-mon[36624]: pgmap v13868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:32.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:31 smithi081 ceph-mon[31453]: pgmap v13869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:32.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:31 smithi153 ceph-mon[36569]: pgmap v13869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:32.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:31 smithi104 ceph-mon[36624]: pgmap v13869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:34.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:33 smithi081 ceph-mon[31453]: pgmap v13870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:34.292 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:33 smithi153 ceph-mon[36569]: pgmap v13870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:34.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:33 smithi104 ceph-mon[36624]: pgmap v13870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:36.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:35 smithi081 ceph-mon[31453]: pgmap v13871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:36.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:35 smithi153 ceph-mon[36569]: pgmap v13871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:36.374 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:35 smithi104 ceph-mon[36624]: pgmap v13871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:38.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:37 smithi081 ceph-mon[31453]: pgmap v13872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:38.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:37 smithi153 ceph-mon[36569]: pgmap v13872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:38.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:37 smithi104 ceph-mon[36624]: pgmap v13872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:39.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:38 smithi081 ceph-mon[31453]: pgmap v13873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:39.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:38 smithi153 ceph-mon[36569]: pgmap v13873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:39.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:38 smithi104 ceph-mon[36624]: pgmap v13873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:42.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:41 smithi081 ceph-mon[31453]: pgmap v13874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:42.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:41 smithi153 ceph-mon[36569]: pgmap v13874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:42.323+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:25:42.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:42.323+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:25:42.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:42.323+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:25:42.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:41 smithi104 ceph-mon[36624]: pgmap v13874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:44.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:43 smithi081 ceph-mon[31453]: pgmap v13875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:44.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:43 smithi153 ceph-mon[36569]: pgmap v13875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:44.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:43 smithi104 ceph-mon[36624]: pgmap v13875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:46.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:45 smithi081 ceph-mon[31453]: pgmap v13876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:46.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:45 smithi153 ceph-mon[36569]: pgmap v13876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:46.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:45 smithi104 ceph-mon[36624]: pgmap v13876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:48.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:47 smithi081 ceph-mon[31453]: pgmap v13877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:48.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:47 smithi153 ceph-mon[36569]: pgmap v13877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:48.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:47 smithi104 ceph-mon[36624]: pgmap v13877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:50.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:49 smithi081 ceph-mon[31453]: pgmap v13878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:50.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:49 smithi153 ceph-mon[36569]: pgmap v13878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:50.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:49 smithi104 ceph-mon[36624]: pgmap v13878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:52.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:51 smithi081 ceph-mon[31453]: pgmap v13879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:52.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:51 smithi153 ceph-mon[36569]: pgmap v13879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:52.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:51 smithi104 ceph-mon[36624]: pgmap v13879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:53.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:52 smithi153 ceph-mon[36569]: pgmap v13880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:53.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:52 smithi104 ceph-mon[36624]: pgmap v13880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:53.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:52 smithi081 ceph-mon[31453]: pgmap v13880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:56.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:55 smithi081 ceph-mon[31453]: pgmap v13881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:56.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:55 smithi153 ceph-mon[36569]: pgmap v13881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:56.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:55 smithi104 ceph-mon[36624]: pgmap v13881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:57.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:57.333+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:25:57.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:57.333+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:25:57.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:25:57.333+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:25:58.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:57 smithi081 ceph-mon[31453]: pgmap v13882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:58.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:57 smithi153 ceph-mon[36569]: pgmap v13882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:25:58.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:57 smithi104 ceph-mon[36624]: pgmap v13882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:00.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:25:59 smithi081 ceph-mon[31453]: pgmap v13883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:00.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:25:59 smithi153 ceph-mon[36569]: pgmap v13883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:00.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:25:59 smithi104 ceph-mon[36624]: pgmap v13883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:02.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:01 smithi081 ceph-mon[31453]: pgmap v13884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:02.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:26:02.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:01 smithi153 ceph-mon[36569]: pgmap v13884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:02.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:26:02.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:01 smithi104 ceph-mon[36624]: pgmap v13884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:02.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:26:03.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:26:03.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:26:03.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:26:03.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:26:03.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:26:03.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:26:03.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:26:03.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:26:03.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:26:04.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:03 smithi081 ceph-mon[31453]: pgmap v13885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:04.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:03 smithi081 ceph-mon[31453]: pgmap v13886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:04.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:03 smithi153 ceph-mon[36569]: pgmap v13885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:04.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:03 smithi153 ceph-mon[36569]: pgmap v13886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:04.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:03 smithi104 ceph-mon[36624]: pgmap v13885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:04.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:03 smithi104 ceph-mon[36624]: pgmap v13886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:05.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:04 smithi153 ceph-mon[36569]: pgmap v13887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:05.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:04 smithi104 ceph-mon[36624]: pgmap v13887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:05.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:04 smithi081 ceph-mon[31453]: pgmap v13887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:08.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:07 smithi081 ceph-mon[31453]: pgmap v13888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:08.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:07 smithi153 ceph-mon[36569]: pgmap v13888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:08.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:07 smithi104 ceph-mon[36624]: pgmap v13888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:10.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:09 smithi081 ceph-mon[31453]: pgmap v13889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:10.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:09 smithi153 ceph-mon[36569]: pgmap v13889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:10.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:09 smithi104 ceph-mon[36624]: pgmap v13889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:12.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:11 smithi081 ceph-mon[31453]: pgmap v13890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:12.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:11 smithi153 ceph-mon[36569]: pgmap v13890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:12.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:11 smithi104 ceph-mon[36624]: pgmap v13890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:12.341+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:26:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:12.341+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:26:12.326 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:12.341+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:26:14.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:13 smithi081 ceph-mon[31453]: pgmap v13891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:14.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:13 smithi153 ceph-mon[36569]: pgmap v13891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:14.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:13 smithi104 ceph-mon[36624]: pgmap v13891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:16.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:15 smithi081 ceph-mon[31453]: pgmap v13892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:16.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:15 smithi153 ceph-mon[36569]: pgmap v13892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:16.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:15 smithi104 ceph-mon[36624]: pgmap v13892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:17.291 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:16 smithi153 ceph-mon[36569]: pgmap v13893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:17.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:16 smithi104 ceph-mon[36624]: pgmap v13893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:17.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:16 smithi081 ceph-mon[31453]: pgmap v13893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:19.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:19 smithi153 ceph-mon[36569]: pgmap v13894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:19.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:19 smithi104 ceph-mon[36624]: pgmap v13894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:19.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:19 smithi081 ceph-mon[31453]: pgmap v13894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:21.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:21 smithi153 ceph-mon[36569]: pgmap v13895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:21.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:21 smithi104 ceph-mon[36624]: pgmap v13895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:21.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:21 smithi081 ceph-mon[31453]: pgmap v13895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:23.640 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:23 smithi081 ceph-mon[31453]: pgmap v13896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:23.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:23 smithi153 ceph-mon[36569]: pgmap v13896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:23.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:23 smithi104 ceph-mon[36624]: pgmap v13896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:25.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:25 smithi153 ceph-mon[36569]: pgmap v13897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:25.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:25 smithi104 ceph-mon[36624]: pgmap v13897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:25.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:25 smithi081 ceph-mon[31453]: pgmap v13897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:27.357+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:26:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:27.357+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:26:27.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:27.357+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:26:27.693 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:27 smithi081 ceph-mon[31453]: pgmap v13898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:27.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:27 smithi153 ceph-mon[36569]: pgmap v13898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:27.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:27 smithi104 ceph-mon[36624]: pgmap v13898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:29.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:29 smithi153 ceph-mon[36569]: pgmap v13899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:29.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:29 smithi104 ceph-mon[36624]: pgmap v13899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:29.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:29 smithi081 ceph-mon[31453]: pgmap v13899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:31.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:31 smithi153 ceph-mon[36569]: pgmap v13900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:31.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:31 smithi104 ceph-mon[36624]: pgmap v13900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:31.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:31 smithi081 ceph-mon[31453]: pgmap v13900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:33.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:33 smithi153 ceph-mon[36569]: pgmap v13901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:33.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:33 smithi104 ceph-mon[36624]: pgmap v13901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:33.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:33 smithi081 ceph-mon[31453]: pgmap v13901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:35.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:35 smithi153 ceph-mon[36569]: pgmap v13902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:35.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:35 smithi104 ceph-mon[36624]: pgmap v13902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:35.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:35 smithi081 ceph-mon[31453]: pgmap v13902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:37.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:37 smithi153 ceph-mon[36569]: pgmap v13903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:37.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:37 smithi104 ceph-mon[36624]: pgmap v13903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:37.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:37 smithi081 ceph-mon[31453]: pgmap v13903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:39.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:39 smithi153 ceph-mon[36569]: pgmap v13904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:39.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:39 smithi104 ceph-mon[36624]: pgmap v13904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:39.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:39 smithi081 ceph-mon[31453]: pgmap v13904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:41.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:41 smithi153 ceph-mon[36569]: pgmap v13905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:41.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:41 smithi104 ceph-mon[36624]: pgmap v13905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:41.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:41 smithi081 ceph-mon[31453]: pgmap v13905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:42.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:42.357+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:26:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:42.357+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:26:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:42.357+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:26:43.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:43 smithi153 ceph-mon[36569]: pgmap v13906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:43.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:43 smithi104 ceph-mon[36624]: pgmap v13906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:43.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:43 smithi081 ceph-mon[31453]: pgmap v13906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:45.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:45 smithi153 ceph-mon[36569]: pgmap v13907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:45.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:45 smithi104 ceph-mon[36624]: pgmap v13907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:45.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:45 smithi081 ceph-mon[31453]: pgmap v13907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:47.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:47 smithi153 ceph-mon[36569]: pgmap v13908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:47.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:47 smithi104 ceph-mon[36624]: pgmap v13908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:47.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:47 smithi081 ceph-mon[31453]: pgmap v13908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:49.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:49 smithi153 ceph-mon[36569]: pgmap v13909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:49.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:49 smithi104 ceph-mon[36624]: pgmap v13909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:49.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:49 smithi081 ceph-mon[31453]: pgmap v13909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:51.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:51 smithi153 ceph-mon[36569]: pgmap v13910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:51.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:51 smithi104 ceph-mon[36624]: pgmap v13910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:51.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:51 smithi081 ceph-mon[31453]: pgmap v13910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:53.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:53 smithi153 ceph-mon[36569]: pgmap v13911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:53.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:53 smithi104 ceph-mon[36624]: pgmap v13911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:53.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:53 smithi081 ceph-mon[31453]: pgmap v13911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:55.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:55 smithi153 ceph-mon[36569]: pgmap v13912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:55.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:55 smithi104 ceph-mon[36624]: pgmap v13912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:55.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:55 smithi081 ceph-mon[31453]: pgmap v13912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:57.374+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:26:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:57.374+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:26:57.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:26:57.374+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:26:57.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:57 smithi153 ceph-mon[36569]: pgmap v13913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:57.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:57 smithi104 ceph-mon[36624]: pgmap v13913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:57.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:57 smithi081 ceph-mon[31453]: pgmap v13913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:59.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:26:59 smithi153 ceph-mon[36569]: pgmap v13914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:59.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:26:59 smithi104 ceph-mon[36624]: pgmap v13914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:26:59.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:26:59 smithi081 ceph-mon[31453]: pgmap v13914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:01.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:01 smithi153 ceph-mon[36569]: pgmap v13915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:01.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:01 smithi104 ceph-mon[36624]: pgmap v13915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:01.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:01 smithi081 ceph-mon[31453]: pgmap v13915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:03.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:03 smithi153 ceph-mon[36569]: pgmap v13916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:03.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:27:03.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:03 smithi104 ceph-mon[36624]: pgmap v13916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:03.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:27:03.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:03 smithi081 ceph-mon[31453]: pgmap v13916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:03.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:27:04.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:27:04.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:27:04.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:27:04.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:27:04.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:27:04.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:27:04.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:27:04.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:27:04.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:27:05.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:05 smithi153 ceph-mon[36569]: pgmap v13917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:05.791 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:05 smithi153 ceph-mon[36569]: pgmap v13918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:05.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:05 smithi104 ceph-mon[36624]: pgmap v13917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:05.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:05 smithi104 ceph-mon[36624]: pgmap v13918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:05.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:05 smithi081 ceph-mon[31453]: pgmap v13917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:05.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:05 smithi081 ceph-mon[31453]: pgmap v13918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:07.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:07 smithi104 ceph-mon[36624]: pgmap v13919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:07.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:07 smithi081 ceph-mon[31453]: pgmap v13919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:07 smithi153 ceph-mon[36569]: pgmap v13919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:09.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:09 smithi104 ceph-mon[36624]: pgmap v13920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:09.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:09 smithi081 ceph-mon[31453]: pgmap v13920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:10.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:09 smithi153 ceph-mon[36569]: pgmap v13920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:11.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:11 smithi104 ceph-mon[36624]: pgmap v13921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:11.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:11 smithi081 ceph-mon[31453]: pgmap v13921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:12.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:11 smithi153 ceph-mon[36569]: pgmap v13921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:12.389+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:27:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:12.389+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:27:12.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:12.389+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:27:13.818 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:13 smithi104 ceph-mon[36624]: pgmap v13922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:13.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:13 smithi081 ceph-mon[31453]: pgmap v13922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:14.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:13 smithi153 ceph-mon[36569]: pgmap v13922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:15.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:15 smithi104 ceph-mon[36624]: pgmap v13923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:15.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:15 smithi081 ceph-mon[31453]: pgmap v13923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:16.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:15 smithi153 ceph-mon[36569]: pgmap v13923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:17.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:17 smithi081 ceph-mon[31453]: pgmap v13924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:18.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:17 smithi153 ceph-mon[36569]: pgmap v13924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:18.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:17 smithi104 ceph-mon[36624]: pgmap v13924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:19.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:19 smithi081 ceph-mon[31453]: pgmap v13925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:20.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:19 smithi153 ceph-mon[36569]: pgmap v13925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:20.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:19 smithi104 ceph-mon[36624]: pgmap v13925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:21.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:21 smithi081 ceph-mon[31453]: pgmap v13926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:22.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:21 smithi153 ceph-mon[36569]: pgmap v13926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:22.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:21 smithi104 ceph-mon[36624]: pgmap v13926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:23.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:23 smithi081 ceph-mon[31453]: pgmap v13927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:24.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:23 smithi153 ceph-mon[36569]: pgmap v13927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:24.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:23 smithi104 ceph-mon[36624]: pgmap v13927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:25.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:25 smithi081 ceph-mon[31453]: pgmap v13928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:26.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:25 smithi153 ceph-mon[36569]: pgmap v13928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:26.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:25 smithi104 ceph-mon[36624]: pgmap v13928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:27.404+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:27:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:27.404+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:27:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:27.404+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:27:27.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:27 smithi104 ceph-mon[36624]: pgmap v13929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:27.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:27 smithi081 ceph-mon[31453]: pgmap v13929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:28.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:27 smithi153 ceph-mon[36569]: pgmap v13929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:29.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:29 smithi081 ceph-mon[31453]: pgmap v13930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:30.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:29 smithi153 ceph-mon[36569]: pgmap v13930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:30.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:29 smithi104 ceph-mon[36624]: pgmap v13930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:31.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:31 smithi081 ceph-mon[31453]: pgmap v13931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:32.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:31 smithi153 ceph-mon[36569]: pgmap v13931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:32.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:31 smithi104 ceph-mon[36624]: pgmap v13931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:34.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:33 smithi153 ceph-mon[36569]: pgmap v13932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:34.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:33 smithi104 ceph-mon[36624]: pgmap v13932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:34.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:33 smithi081 ceph-mon[31453]: pgmap v13932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:36.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:35 smithi153 ceph-mon[36569]: pgmap v13933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:36.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:35 smithi104 ceph-mon[36624]: pgmap v13933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:36.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:35 smithi081 ceph-mon[31453]: pgmap v13933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:37.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:36 smithi153 ceph-mon[36569]: pgmap v13934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:37.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:36 smithi104 ceph-mon[36624]: pgmap v13934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:37.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:36 smithi081 ceph-mon[31453]: pgmap v13934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:39.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:38 smithi153 ceph-mon[36569]: pgmap v13935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:39.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:38 smithi104 ceph-mon[36624]: pgmap v13935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:39.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:38 smithi081 ceph-mon[31453]: pgmap v13935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:41.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:41 smithi153 ceph-mon[36569]: pgmap v13936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:41.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:41 smithi104 ceph-mon[36624]: pgmap v13936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:41.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:41 smithi081 ceph-mon[31453]: pgmap v13936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:42.418+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:27:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:42.418+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:27:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:42.418+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:27:43.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:43 smithi153 ceph-mon[36569]: pgmap v13937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:43.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:43 smithi104 ceph-mon[36624]: pgmap v13937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:43.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:43 smithi081 ceph-mon[31453]: pgmap v13937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:45.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:45 smithi153 ceph-mon[36569]: pgmap v13938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:45.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:45 smithi104 ceph-mon[36624]: pgmap v13938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:45.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:45 smithi081 ceph-mon[31453]: pgmap v13938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:47.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:47 smithi153 ceph-mon[36569]: pgmap v13939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:47.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:47 smithi104 ceph-mon[36624]: pgmap v13939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:47.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:47 smithi081 ceph-mon[31453]: pgmap v13939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:49.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:49 smithi153 ceph-mon[36569]: pgmap v13940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:49.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:49 smithi104 ceph-mon[36624]: pgmap v13940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:49.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:49 smithi081 ceph-mon[31453]: pgmap v13940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:51.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:51 smithi153 ceph-mon[36569]: pgmap v13941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:51.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:51 smithi104 ceph-mon[36624]: pgmap v13941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:51.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:51 smithi081 ceph-mon[31453]: pgmap v13941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:53.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:53 smithi153 ceph-mon[36569]: pgmap v13942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:53.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:53 smithi104 ceph-mon[36624]: pgmap v13942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:53.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:53 smithi081 ceph-mon[31453]: pgmap v13942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:55.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:55 smithi153 ceph-mon[36569]: pgmap v13943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:55.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:55 smithi104 ceph-mon[36624]: pgmap v13943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:55.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:55 smithi081 ceph-mon[31453]: pgmap v13943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:57.192 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:57 smithi081 ceph-mon[31453]: pgmap v13944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:57.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:57 smithi153 ceph-mon[36569]: pgmap v13944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:57.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:57 smithi104 ceph-mon[36624]: pgmap v13944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:57.433+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:27:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:57.433+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:27:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:27:57.433+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:27:59.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:27:59 smithi104 ceph-mon[36624]: pgmap v13945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:59.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:27:59 smithi081 ceph-mon[31453]: pgmap v13945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:27:59.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:27:59 smithi153 ceph-mon[36569]: pgmap v13945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:01.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:01 smithi104 ceph-mon[36624]: pgmap v13946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:01.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:01 smithi081 ceph-mon[31453]: pgmap v13946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:01.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:01 smithi153 ceph-mon[36569]: pgmap v13946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:03.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:03 smithi104 ceph-mon[36624]: pgmap v13947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:03.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:03 smithi081 ceph-mon[31453]: pgmap v13947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:03.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:03 smithi153 ceph-mon[36569]: pgmap v13947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:05.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:05 smithi104 ceph-mon[36624]: pgmap v13948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:05.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:28:05.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:05 smithi081 ceph-mon[31453]: pgmap v13948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:05.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:28:05.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:05 smithi153 ceph-mon[36569]: pgmap v13948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:05.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:28:06.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:28:06.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:28:06.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:06 smithi081 ceph-mon[31453]: pgmap v13949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:06.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:06 smithi081 ceph-mon[31453]: pgmap v13950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:06.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:28:06.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:28:06.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:28:06.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:06 smithi153 ceph-mon[36569]: pgmap v13949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:06.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:06 smithi153 ceph-mon[36569]: pgmap v13950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:06.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:28:06.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:28:06.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:28:06.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:06 smithi104 ceph-mon[36624]: pgmap v13949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:06.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:06 smithi104 ceph-mon[36624]: pgmap v13950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:06.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:28:08.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:08 smithi153 ceph-mon[36569]: pgmap v13951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:08.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:08 smithi104 ceph-mon[36624]: pgmap v13951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:08.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:08 smithi081 ceph-mon[31453]: pgmap v13951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:10.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:10 smithi153 ceph-mon[36569]: pgmap v13952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:10.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:10 smithi104 ceph-mon[36624]: pgmap v13952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:10.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:10 smithi081 ceph-mon[31453]: pgmap v13952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:12.449+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:28:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:12.449+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:28:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:12.449+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:28:12.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:12 smithi153 ceph-mon[36569]: pgmap v13953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:12.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:12 smithi104 ceph-mon[36624]: pgmap v13953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:12.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:12 smithi081 ceph-mon[31453]: pgmap v13953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:14.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:14 smithi104 ceph-mon[36624]: pgmap v13954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:14.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:14 smithi081 ceph-mon[31453]: pgmap v13954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:15.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:14 smithi153 ceph-mon[36569]: pgmap v13954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:16.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:16 smithi104 ceph-mon[36624]: pgmap v13955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:16.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:16 smithi081 ceph-mon[31453]: pgmap v13955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:16.987+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:28:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:16.987+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:28:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:16.988+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 201 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:28:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:16.988+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 201 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:28:17.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:16 smithi153 ceph-mon[36569]: pgmap v13955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:18.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:18 smithi104 ceph-mon[36624]: pgmap v13956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:18.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:18 smithi081 ceph-mon[31453]: pgmap v13956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:19.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:18 smithi153 ceph-mon[36569]: pgmap v13956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:20.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:20 smithi104 ceph-mon[36624]: pgmap v13957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:20.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:20 smithi081 ceph-mon[31453]: pgmap v13957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:21.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:20 smithi153 ceph-mon[36569]: pgmap v13957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:22.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:22 smithi104 ceph-mon[36624]: pgmap v13958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:22.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:22 smithi081 ceph-mon[31453]: pgmap v13958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:23.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:22 smithi153 ceph-mon[36569]: pgmap v13958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:24.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:24 smithi081 ceph-mon[31453]: pgmap v13959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:25.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:24 smithi153 ceph-mon[36569]: pgmap v13959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:25.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:24 smithi104 ceph-mon[36624]: pgmap v13959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:26.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:26 smithi081 ceph-mon[31453]: pgmap v13960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:27.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:26 smithi153 ceph-mon[36569]: pgmap v13960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:27.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:26 smithi104 ceph-mon[36624]: pgmap v13960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:27.461+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:28:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:27.461+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:28:27.445 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:27.461+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:28:28.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:27 smithi153 ceph-mon[36569]: pgmap v13961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:28.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:27 smithi104 ceph-mon[36624]: pgmap v13961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:28.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:27 smithi081 ceph-mon[31453]: pgmap v13961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:30.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:30 smithi153 ceph-mon[36569]: pgmap v13962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:30.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:30 smithi104 ceph-mon[36624]: pgmap v13962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:30.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:30 smithi081 ceph-mon[31453]: pgmap v13962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:32.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:32 smithi104 ceph-mon[36624]: pgmap v13963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:32.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:32 smithi081 ceph-mon[31453]: pgmap v13963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:33.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:32 smithi153 ceph-mon[36569]: pgmap v13963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:34.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:34 smithi104 ceph-mon[36624]: pgmap v13964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:34.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:34 smithi081 ceph-mon[31453]: pgmap v13964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:35.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:34 smithi153 ceph-mon[36569]: pgmap v13964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:36.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:36 smithi104 ceph-mon[36624]: pgmap v13965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:36.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:36 smithi081 ceph-mon[31453]: pgmap v13965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:37.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:36 smithi153 ceph-mon[36569]: pgmap v13965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:38.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:38 smithi104 ceph-mon[36624]: pgmap v13966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:38.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:38 smithi081 ceph-mon[31453]: pgmap v13966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:39.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:38 smithi153 ceph-mon[36569]: pgmap v13966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:40.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:40 smithi104 ceph-mon[36624]: pgmap v13967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:40.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:40 smithi081 ceph-mon[31453]: pgmap v13967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:41.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:40 smithi153 ceph-mon[36569]: pgmap v13967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:42.477+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:28:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:42.477+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:28:42.461 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:42.477+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:28:42.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:42 smithi081 ceph-mon[31453]: pgmap v13968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:43.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:42 smithi153 ceph-mon[36569]: pgmap v13968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:43.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:42 smithi104 ceph-mon[36624]: pgmap v13968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:44.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:44 smithi081 ceph-mon[31453]: pgmap v13969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:45.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:44 smithi153 ceph-mon[36569]: pgmap v13969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:45.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:44 smithi104 ceph-mon[36624]: pgmap v13969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:46.891 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:46 smithi081 ceph-mon[31453]: pgmap v13970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:47.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:46 smithi153 ceph-mon[36569]: pgmap v13970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:47.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:46 smithi104 ceph-mon[36624]: pgmap v13970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:48.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:48 smithi081 ceph-mon[31453]: pgmap v13971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:49.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:48 smithi153 ceph-mon[36569]: pgmap v13971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:49.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:48 smithi104 ceph-mon[36624]: pgmap v13971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:50.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:50 smithi081 ceph-mon[31453]: pgmap v13972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:51.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:50 smithi153 ceph-mon[36569]: pgmap v13972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:51.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:50 smithi104 ceph-mon[36624]: pgmap v13972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:52.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:52 smithi081 ceph-mon[31453]: pgmap v13973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:53.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:52 smithi153 ceph-mon[36569]: pgmap v13973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:53.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:52 smithi104 ceph-mon[36624]: pgmap v13973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:54.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:54 smithi081 ceph-mon[31453]: pgmap v13974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:55.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:54 smithi153 ceph-mon[36569]: pgmap v13974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:55.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:54 smithi104 ceph-mon[36624]: pgmap v13974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:56.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:56 smithi081 ceph-mon[31453]: pgmap v13975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:57.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:56 smithi153 ceph-mon[36569]: pgmap v13975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:57.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:56 smithi104 ceph-mon[36624]: pgmap v13975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:57.493+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:28:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:57.493+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:28:57.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:28:57.493+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:28:58.867 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:28:58 smithi104 ceph-mon[36624]: pgmap v13976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:58.870 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:28:58 smithi081 ceph-mon[31453]: pgmap v13976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:28:59.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:28:58 smithi153 ceph-mon[36569]: pgmap v13976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:00.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:00 smithi081 ceph-mon[31453]: pgmap v13977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:01.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:00 smithi153 ceph-mon[36569]: pgmap v13977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:01.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:00 smithi104 ceph-mon[36624]: pgmap v13977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:02.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:02 smithi081 ceph-mon[31453]: pgmap v13978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:03.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:02 smithi153 ceph-mon[36569]: pgmap v13978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:03.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:02 smithi104 ceph-mon[36624]: pgmap v13978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:04.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:04 smithi081 ceph-mon[31453]: pgmap v13979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:05.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:04 smithi153 ceph-mon[36569]: pgmap v13979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:05.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:04 smithi104 ceph-mon[36624]: pgmap v13979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:06.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: pgmap v13980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:06.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:29:06.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:06.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:06.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:06.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:06.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:06.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:06.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:06.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: pgmap v13980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:07.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:29:07.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: pgmap v13980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:07.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:29:07.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:29:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:29:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:29:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:29:07.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:07.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:29:07.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:29:07.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:29:08.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:29:08.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:29:08.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:29:08.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:29:08.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:29:09.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:08 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:29:09.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:08 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:29:09.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:08 smithi153 ceph-mon[36569]: pgmap v13981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:09.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:08 smithi153 ceph-mon[36569]: pgmap v13982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:09.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:08 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:29:09.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:08 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:29:09.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:08 smithi104 ceph-mon[36624]: pgmap v13981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:09.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:08 smithi104 ceph-mon[36624]: pgmap v13982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:09.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:08 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:29:09.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:08 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:29:09.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:08 smithi081 ceph-mon[31453]: pgmap v13981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:09.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:08 smithi081 ceph-mon[31453]: pgmap v13982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:11.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:10 smithi153 ceph-mon[36569]: pgmap v13983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:11.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:10 smithi104 ceph-mon[36624]: pgmap v13983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:11.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:10 smithi081 ceph-mon[31453]: pgmap v13983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:12.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:12.509+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:29:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:12.509+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:29:12.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:12.509+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:29:13.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:12 smithi153 ceph-mon[36569]: pgmap v13984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:13.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:12 smithi104 ceph-mon[36624]: pgmap v13984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:13.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:12 smithi081 ceph-mon[31453]: pgmap v13984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:15.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:14 smithi153 ceph-mon[36569]: pgmap v13985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:15.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:14 smithi104 ceph-mon[36624]: pgmap v13985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:15.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:14 smithi081 ceph-mon[31453]: pgmap v13985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:17.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:16 smithi153 ceph-mon[36569]: pgmap v13986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:17.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:16 smithi104 ceph-mon[36624]: pgmap v13986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:17.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:16 smithi081 ceph-mon[31453]: pgmap v13986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:19.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:18 smithi153 ceph-mon[36569]: pgmap v13987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:19.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:18 smithi104 ceph-mon[36624]: pgmap v13987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:19.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:18 smithi081 ceph-mon[31453]: pgmap v13987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:21.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:20 smithi153 ceph-mon[36569]: pgmap v13988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:21.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:20 smithi104 ceph-mon[36624]: pgmap v13988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:21.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:20 smithi081 ceph-mon[31453]: pgmap v13988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:23.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:22 smithi153 ceph-mon[36569]: pgmap v13989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:23.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:22 smithi104 ceph-mon[36624]: pgmap v13989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:23.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:22 smithi081 ceph-mon[31453]: pgmap v13989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:25.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:24 smithi153 ceph-mon[36569]: pgmap v13990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:25.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:24 smithi104 ceph-mon[36624]: pgmap v13990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:25.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:24 smithi081 ceph-mon[31453]: pgmap v13990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:27.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:26 smithi153 ceph-mon[36569]: pgmap v13991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:27.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:26 smithi104 ceph-mon[36624]: pgmap v13991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:27.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:26 smithi081 ceph-mon[31453]: pgmap v13991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:27.524+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:29:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:27.524+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:29:27.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:27.524+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:29:29.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:28 smithi153 ceph-mon[36569]: pgmap v13992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:29.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:28 smithi104 ceph-mon[36624]: pgmap v13992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:29.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:28 smithi081 ceph-mon[31453]: pgmap v13992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:31.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:30 smithi153 ceph-mon[36569]: pgmap v13993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:31.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:30 smithi104 ceph-mon[36624]: pgmap v13993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:31.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:30 smithi081 ceph-mon[31453]: pgmap v13993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:33.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:32 smithi153 ceph-mon[36569]: pgmap v13994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:33.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:32 smithi104 ceph-mon[36624]: pgmap v13994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:33.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:32 smithi081 ceph-mon[31453]: pgmap v13994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:35.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:34 smithi153 ceph-mon[36569]: pgmap v13995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:35.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:34 smithi104 ceph-mon[36624]: pgmap v13995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:35.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:34 smithi081 ceph-mon[31453]: pgmap v13995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:37.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:36 smithi153 ceph-mon[36569]: pgmap v13996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:37.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:36 smithi104 ceph-mon[36624]: pgmap v13996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:37.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:36 smithi081 ceph-mon[31453]: pgmap v13996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:39.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:38 smithi153 ceph-mon[36569]: pgmap v13997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:39.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:38 smithi104 ceph-mon[36624]: pgmap v13997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:39.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:38 smithi081 ceph-mon[31453]: pgmap v13997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:41.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:40 smithi153 ceph-mon[36569]: pgmap v13998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:41.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:40 smithi104 ceph-mon[36624]: pgmap v13998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:41.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:40 smithi081 ceph-mon[31453]: pgmap v13998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:42.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:42.540+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:29:42.523 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:42.540+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:29:42.524 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:42.540+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:29:43.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:42 smithi153 ceph-mon[36569]: pgmap v13999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:43.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:42 smithi104 ceph-mon[36624]: pgmap v13999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:43.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:42 smithi081 ceph-mon[31453]: pgmap v13999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:45.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:44 smithi153 ceph-mon[36569]: pgmap v14000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:45.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:44 smithi104 ceph-mon[36624]: pgmap v14000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:45.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:44 smithi081 ceph-mon[31453]: pgmap v14000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:47.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:46 smithi153 ceph-mon[36569]: pgmap v14001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:47.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:46 smithi104 ceph-mon[36624]: pgmap v14001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:47.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:46 smithi081 ceph-mon[31453]: pgmap v14001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:49.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:48 smithi153 ceph-mon[36569]: pgmap v14002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:49.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:48 smithi104 ceph-mon[36624]: pgmap v14002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:49.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:48 smithi081 ceph-mon[31453]: pgmap v14002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:51.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:50 smithi153 ceph-mon[36569]: pgmap v14003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:51.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:50 smithi104 ceph-mon[36624]: pgmap v14003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:51.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:50 smithi081 ceph-mon[31453]: pgmap v14003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:53.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:52 smithi153 ceph-mon[36569]: pgmap v14004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:53.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:52 smithi104 ceph-mon[36624]: pgmap v14004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:53.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:52 smithi081 ceph-mon[31453]: pgmap v14004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:55.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:54 smithi153 ceph-mon[36569]: pgmap v14005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:55.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:54 smithi104 ceph-mon[36624]: pgmap v14005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:55.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:54 smithi081 ceph-mon[31453]: pgmap v14005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:57.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:56 smithi153 ceph-mon[36569]: pgmap v14006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:57.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:56 smithi104 ceph-mon[36624]: pgmap v14006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:57.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:56 smithi081 ceph-mon[31453]: pgmap v14006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:57.555+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:29:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:57.556+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:29:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:29:57.556+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:29:59.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:29:58 smithi153 ceph-mon[36569]: pgmap v14007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:59.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:29:58 smithi104 ceph-mon[36624]: pgmap v14007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:29:59.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:29:58 smithi081 ceph-mon[31453]: pgmap v14007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:01.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:00 smithi153 ceph-mon[36569]: pgmap v14008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:01.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:30:01.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:00 smithi104 ceph-mon[36624]: pgmap v14008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:01.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:30:01.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:00 smithi081 ceph-mon[31453]: pgmap v14008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:01.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:30:03.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:02 smithi153 ceph-mon[36569]: pgmap v14009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:03.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:02 smithi104 ceph-mon[36624]: pgmap v14009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:03.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:02 smithi081 ceph-mon[31453]: pgmap v14009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:05.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:04 smithi104 ceph-mon[36624]: pgmap v14010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:05.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:04 smithi081 ceph-mon[31453]: pgmap v14010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:05.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:04 smithi153 ceph-mon[36569]: pgmap v14010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:07.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:06 smithi104 ceph-mon[36624]: pgmap v14011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:07.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:06 smithi081 ceph-mon[31453]: pgmap v14011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:07.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:06 smithi153 ceph-mon[36569]: pgmap v14011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:08.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:30:08.139 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:30:08.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:30:09.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:08 smithi081 ceph-mon[31453]: pgmap v14012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:09.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:30:09.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:30:09.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:30:09.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:08 smithi153 ceph-mon[36569]: pgmap v14012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:09.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:30:09.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:30:09.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:30:09.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:08 smithi104 ceph-mon[36624]: pgmap v14012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:09.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:30:09.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:30:09.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:30:10.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:09 smithi081 ceph-mon[31453]: pgmap v14013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:10.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:09 smithi081 ceph-mon[31453]: pgmap v14014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:10.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:09 smithi153 ceph-mon[36569]: pgmap v14013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:10.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:09 smithi153 ceph-mon[36569]: pgmap v14014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:10.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:09 smithi104 ceph-mon[36624]: pgmap v14013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:10.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:09 smithi104 ceph-mon[36624]: pgmap v14014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:12.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:11 smithi081 ceph-mon[31453]: pgmap v14015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:12.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:11 smithi153 ceph-mon[36569]: pgmap v14015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:12.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:11 smithi104 ceph-mon[36624]: pgmap v14015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:12.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:12.571+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:30:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:12.571+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:30:12.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:12.571+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:30:14.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:13 smithi081 ceph-mon[31453]: pgmap v14016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:14.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:13 smithi153 ceph-mon[36569]: pgmap v14016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:14.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:13 smithi104 ceph-mon[36624]: pgmap v14016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:16.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:15 smithi081 ceph-mon[31453]: pgmap v14017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:16.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:15 smithi153 ceph-mon[36569]: pgmap v14017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:16.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:15 smithi104 ceph-mon[36624]: pgmap v14017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:18.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:17 smithi081 ceph-mon[31453]: pgmap v14018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:18.289 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:17 smithi153 ceph-mon[36569]: pgmap v14018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:18.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:17 smithi104 ceph-mon[36624]: pgmap v14018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:20.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:19 smithi081 ceph-mon[31453]: pgmap v14019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:20.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:19 smithi153 ceph-mon[36569]: pgmap v14019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:20.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:19 smithi104 ceph-mon[36624]: pgmap v14019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:22.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:21 smithi081 ceph-mon[31453]: pgmap v14020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:22.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:21 smithi153 ceph-mon[36569]: pgmap v14020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:22.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:21 smithi104 ceph-mon[36624]: pgmap v14020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:24.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:23 smithi081 ceph-mon[31453]: pgmap v14021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:24.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:23 smithi153 ceph-mon[36569]: pgmap v14021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:24.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:23 smithi104 ceph-mon[36624]: pgmap v14021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:26.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:25 smithi081 ceph-mon[31453]: pgmap v14022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:26.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:25 smithi153 ceph-mon[36569]: pgmap v14022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:26.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:25 smithi104 ceph-mon[36624]: pgmap v14022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:27.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:27.576+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:30:27.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:27.576+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:30:27.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:27.576+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:30:28.117 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:27 smithi104 ceph-mon[36624]: pgmap v14023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:28.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:27 smithi081 ceph-mon[31453]: pgmap v14023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:28.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:27 smithi153 ceph-mon[36569]: pgmap v14023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:30.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:29 smithi104 ceph-mon[36624]: pgmap v14024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:30.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:29 smithi081 ceph-mon[31453]: pgmap v14024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:30.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:29 smithi153 ceph-mon[36569]: pgmap v14024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:32.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:31 smithi081 ceph-mon[31453]: pgmap v14025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:32.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:31 smithi153 ceph-mon[36569]: pgmap v14025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:32.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:31 smithi104 ceph-mon[36624]: pgmap v14025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:34.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:33 smithi081 ceph-mon[31453]: pgmap v14026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:34.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:33 smithi153 ceph-mon[36569]: pgmap v14026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:34.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:33 smithi104 ceph-mon[36624]: pgmap v14026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:36.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:35 smithi081 ceph-mon[31453]: pgmap v14027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:36.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:35 smithi153 ceph-mon[36569]: pgmap v14027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:36.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:35 smithi104 ceph-mon[36624]: pgmap v14027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:38.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:37 smithi153 ceph-mon[36569]: pgmap v14028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:38.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:37 smithi104 ceph-mon[36624]: pgmap v14028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:38.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:37 smithi081 ceph-mon[31453]: pgmap v14028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:40.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:39 smithi153 ceph-mon[36569]: pgmap v14029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:40.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:39 smithi104 ceph-mon[36624]: pgmap v14029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:40.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:39 smithi081 ceph-mon[31453]: pgmap v14029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:42.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:41 smithi081 ceph-mon[31453]: pgmap v14030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:42.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:41 smithi153 ceph-mon[36569]: pgmap v14030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:42.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:41 smithi104 ceph-mon[36624]: pgmap v14030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:42.586+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:30:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:42.586+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:30:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:42.586+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:30:44.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:43 smithi153 ceph-mon[36569]: pgmap v14031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:44.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:43 smithi104 ceph-mon[36624]: pgmap v14031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:44.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:43 smithi081 ceph-mon[31453]: pgmap v14031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:46.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:45 smithi153 ceph-mon[36569]: pgmap v14032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:46.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:45 smithi104 ceph-mon[36624]: pgmap v14032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:46.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:45 smithi081 ceph-mon[31453]: pgmap v14032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:48.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:47 smithi153 ceph-mon[36569]: pgmap v14033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:48.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:47 smithi104 ceph-mon[36624]: pgmap v14033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:48.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:47 smithi081 ceph-mon[31453]: pgmap v14033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:50.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:49 smithi153 ceph-mon[36569]: pgmap v14034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:50.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:49 smithi104 ceph-mon[36624]: pgmap v14034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:50.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:49 smithi081 ceph-mon[31453]: pgmap v14034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:52.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:51 smithi153 ceph-mon[36569]: pgmap v14035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:52.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:51 smithi104 ceph-mon[36624]: pgmap v14035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:52.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:51 smithi081 ceph-mon[31453]: pgmap v14035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:54.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:53 smithi153 ceph-mon[36569]: pgmap v14036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:54.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:53 smithi104 ceph-mon[36624]: pgmap v14036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:54.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:53 smithi081 ceph-mon[31453]: pgmap v14036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:55.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:54 smithi153 ceph-mon[36569]: pgmap v14037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:55.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:54 smithi104 ceph-mon[36624]: pgmap v14037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:55.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:54 smithi081 ceph-mon[31453]: pgmap v14037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:57.602+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:30:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:57.602+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:30:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:30:57.602+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:30:57.991 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:57 smithi153 ceph-mon[36569]: pgmap v14038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:58.017 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:57 smithi104 ceph-mon[36624]: pgmap v14038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:30:58.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:57 smithi081 ceph-mon[31453]: pgmap v14038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:00.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:30:59 smithi153 ceph-mon[36569]: pgmap v14039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:00.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:30:59 smithi104 ceph-mon[36624]: pgmap v14039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:00.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:30:59 smithi081 ceph-mon[31453]: pgmap v14039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:02.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:01 smithi153 ceph-mon[36569]: pgmap v14040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:02.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:01 smithi104 ceph-mon[36624]: pgmap v14040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:02.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:01 smithi081 ceph-mon[31453]: pgmap v14040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:04.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:03 smithi153 ceph-mon[36569]: pgmap v14041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:04.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:03 smithi104 ceph-mon[36624]: pgmap v14041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:04.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:03 smithi081 ceph-mon[31453]: pgmap v14041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:06.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:05 smithi153 ceph-mon[36569]: pgmap v14042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:06.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:05 smithi104 ceph-mon[36624]: pgmap v14042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:06.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:05 smithi081 ceph-mon[31453]: pgmap v14042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:08.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:07 smithi153 ceph-mon[36569]: pgmap v14043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:08.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:07 smithi104 ceph-mon[36624]: pgmap v14043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:08.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:07 smithi081 ceph-mon[31453]: pgmap v14043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:10.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:09 smithi153 ceph-mon[36569]: pgmap v14044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:10.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:31:10.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:09 smithi104 ceph-mon[36624]: pgmap v14044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:10.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:31:10.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:09 smithi081 ceph-mon[31453]: pgmap v14044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:10.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:31:11.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:31:11.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:31:11.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:31:11.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:31:11.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:31:11.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:31:11.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:31:11.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:31:11.138 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:31:12.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:11 smithi153 ceph-mon[36569]: pgmap v14045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:12.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:11 smithi153 ceph-mon[36569]: pgmap v14046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:12.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:11 smithi104 ceph-mon[36624]: pgmap v14045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:12.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:11 smithi104 ceph-mon[36624]: pgmap v14046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:12.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:11 smithi081 ceph-mon[31453]: pgmap v14045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:12.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:11 smithi081 ceph-mon[31453]: pgmap v14046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.162+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 msgr2=0x7f16e80b39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000628) for more than 900000000 us, fault. 2024-04-13T04:31:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.162+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f16d40036c0 tx=0x7f16f40ac000 comp rx=0 tx=0).stop 2024-04-13T04:31:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.162+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 msgr2=0x7f16e80b39a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T04:31:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.162+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4e80 0x7f16e80b39a0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T04:31:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.162+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T04:31:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.163+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T04:31:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.164+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f16f4004a80 tx=0x7f16f4039000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T04:31:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.609+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:31:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.609+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:31:12.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:12.609+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:31:14.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:13 smithi153 ceph-mon[36569]: pgmap v14047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:14.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:13 smithi104 ceph-mon[36624]: pgmap v14047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:14.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:13 smithi081 ceph-mon[31453]: pgmap v14047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:16.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:15 smithi153 ceph-mon[36569]: pgmap v14048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:16.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:15 smithi104 ceph-mon[36624]: pgmap v14048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:16.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:15 smithi081 ceph-mon[31453]: pgmap v14048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:17.017+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:31:16.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:17.017+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:31:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:17.018+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 202 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:31:17.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:17.018+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 202 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:31:18.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:17 smithi153 ceph-mon[36569]: pgmap v14049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:18.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:17 smithi104 ceph-mon[36624]: pgmap v14049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:18.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:17 smithi081 ceph-mon[31453]: pgmap v14049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:20.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:19 smithi153 ceph-mon[36569]: pgmap v14050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:20.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:19 smithi104 ceph-mon[36624]: pgmap v14050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:20.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:19 smithi081 ceph-mon[31453]: pgmap v14050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:22.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:21 smithi153 ceph-mon[36569]: pgmap v14051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:22.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:21 smithi104 ceph-mon[36624]: pgmap v14051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:22.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:21 smithi081 ceph-mon[31453]: pgmap v14051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:24.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:23 smithi153 ceph-mon[36569]: pgmap v14052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:24.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:23 smithi104 ceph-mon[36624]: pgmap v14052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:24.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:23 smithi081 ceph-mon[31453]: pgmap v14052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:26.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:25 smithi153 ceph-mon[36569]: pgmap v14053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:26.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:25 smithi104 ceph-mon[36624]: pgmap v14053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:26.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:25 smithi081 ceph-mon[31453]: pgmap v14053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:27.616+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:31:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:27.616+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:31:27.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:27.616+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:31:28.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:27 smithi153 ceph-mon[36569]: pgmap v14054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:28.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:27 smithi104 ceph-mon[36624]: pgmap v14054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:28.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:27 smithi081 ceph-mon[31453]: pgmap v14054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:30.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:29 smithi153 ceph-mon[36569]: pgmap v14055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:30.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:29 smithi104 ceph-mon[36624]: pgmap v14055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:30.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:29 smithi081 ceph-mon[31453]: pgmap v14055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:32.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:31 smithi153 ceph-mon[36569]: pgmap v14056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:32.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:31 smithi104 ceph-mon[36624]: pgmap v14056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:32.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:31 smithi081 ceph-mon[31453]: pgmap v14056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:34.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:33 smithi104 ceph-mon[36624]: pgmap v14057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:34.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:33 smithi081 ceph-mon[31453]: pgmap v14057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:34.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:33 smithi153 ceph-mon[36569]: pgmap v14057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:36.064 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:35 smithi104 ceph-mon[36624]: pgmap v14058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:36.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:35 smithi081 ceph-mon[31453]: pgmap v14058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:36.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:35 smithi153 ceph-mon[36569]: pgmap v14058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:38.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:37 smithi104 ceph-mon[36624]: pgmap v14059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:38.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:37 smithi081 ceph-mon[31453]: pgmap v14059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:38.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:37 smithi153 ceph-mon[36569]: pgmap v14059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:40.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:39 smithi104 ceph-mon[36624]: pgmap v14060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:40.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:39 smithi081 ceph-mon[31453]: pgmap v14060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:40.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:39 smithi153 ceph-mon[36569]: pgmap v14060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:42.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:41 smithi081 ceph-mon[31453]: pgmap v14061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:42.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:41 smithi153 ceph-mon[36569]: pgmap v14061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:42.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:41 smithi104 ceph-mon[36624]: pgmap v14061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:42.618+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:31:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:42.618+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:31:42.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:42.618+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:31:44.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:43 smithi081 ceph-mon[31453]: pgmap v14062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:44.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:43 smithi153 ceph-mon[36569]: pgmap v14062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:44.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:43 smithi104 ceph-mon[36624]: pgmap v14062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:46.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:45 smithi081 ceph-mon[31453]: pgmap v14063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:46.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:45 smithi153 ceph-mon[36569]: pgmap v14063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:46.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:45 smithi104 ceph-mon[36624]: pgmap v14063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:48.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:47 smithi081 ceph-mon[31453]: pgmap v14064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:48.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:47 smithi153 ceph-mon[36569]: pgmap v14064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:48.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:47 smithi104 ceph-mon[36624]: pgmap v14064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:50.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:49 smithi081 ceph-mon[31453]: pgmap v14065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:50.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:49 smithi153 ceph-mon[36569]: pgmap v14065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:50.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:49 smithi104 ceph-mon[36624]: pgmap v14065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:52.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:51 smithi081 ceph-mon[31453]: pgmap v14066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:52.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:51 smithi153 ceph-mon[36569]: pgmap v14066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:52.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:51 smithi104 ceph-mon[36624]: pgmap v14066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:54.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:53 smithi081 ceph-mon[31453]: pgmap v14067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:53 smithi153 ceph-mon[36569]: pgmap v14067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:54.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:53 smithi104 ceph-mon[36624]: pgmap v14067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:56.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:55 smithi081 ceph-mon[31453]: pgmap v14068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:56.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:55 smithi153 ceph-mon[36569]: pgmap v14068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:56.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:55 smithi104 ceph-mon[36624]: pgmap v14068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:57.613 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:57.632+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:31:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:57.632+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:31:57.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:31:57.632+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:31:58.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:57 smithi081 ceph-mon[31453]: pgmap v14069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:58.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:57 smithi153 ceph-mon[36569]: pgmap v14069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:31:58.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:57 smithi104 ceph-mon[36624]: pgmap v14069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:00.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:31:59 smithi081 ceph-mon[31453]: pgmap v14070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:00.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:31:59 smithi153 ceph-mon[36569]: pgmap v14070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:00.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:31:59 smithi104 ceph-mon[36624]: pgmap v14070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:02.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:01 smithi081 ceph-mon[31453]: pgmap v14071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:02.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:01 smithi153 ceph-mon[36569]: pgmap v14071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:02.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:01 smithi104 ceph-mon[36624]: pgmap v14071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:04.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:03 smithi081 ceph-mon[31453]: pgmap v14072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:04.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:03 smithi153 ceph-mon[36569]: pgmap v14072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:04.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:03 smithi104 ceph-mon[36624]: pgmap v14072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:06.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:05 smithi081 ceph-mon[31453]: pgmap v14073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:06.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:05 smithi153 ceph-mon[36569]: pgmap v14073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:06.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:05 smithi104 ceph-mon[36624]: pgmap v14073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:08.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:07 smithi153 ceph-mon[36569]: pgmap v14074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:08.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:07 smithi104 ceph-mon[36624]: pgmap v14074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:08.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:07 smithi081 ceph-mon[31453]: pgmap v14074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:10.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:09 smithi153 ceph-mon[36569]: pgmap v14075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:10.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:09 smithi104 ceph-mon[36624]: pgmap v14075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:10.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:09 smithi081 ceph-mon[31453]: pgmap v14075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:11.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:32:11.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:32:11.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:32:12.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:11 smithi153 ceph-mon[36569]: pgmap v14076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:12.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:32:12.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:32:12.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:32:12.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:11 smithi104 ceph-mon[36624]: pgmap v14076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:12.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:32:12.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:32:12.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:32:12.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:11 smithi081 ceph-mon[31453]: pgmap v14076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:12.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:32:12.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:32:12.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:32:12.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:12.648+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:32:12.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:12.648+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:32:12.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:12.648+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:32:13.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:12 smithi153 ceph-mon[36569]: pgmap v14077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:13.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:12 smithi153 ceph-mon[36569]: pgmap v14078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:13.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:12 smithi104 ceph-mon[36624]: pgmap v14077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:13.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:12 smithi104 ceph-mon[36624]: pgmap v14078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:13.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:12 smithi081 ceph-mon[31453]: pgmap v14077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:13.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:12 smithi081 ceph-mon[31453]: pgmap v14078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:15.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:14 smithi153 ceph-mon[36569]: pgmap v14079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:15.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:14 smithi104 ceph-mon[36624]: pgmap v14079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:15.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:14 smithi081 ceph-mon[31453]: pgmap v14079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:17.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:16 smithi153 ceph-mon[36569]: pgmap v14080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:17.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:16 smithi104 ceph-mon[36624]: pgmap v14080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:17.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:16 smithi081 ceph-mon[31453]: pgmap v14080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:19.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:18 smithi153 ceph-mon[36569]: pgmap v14081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:19.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:18 smithi104 ceph-mon[36624]: pgmap v14081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:19.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:18 smithi081 ceph-mon[31453]: pgmap v14081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:21.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:20 smithi153 ceph-mon[36569]: pgmap v14082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:21.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:20 smithi104 ceph-mon[36624]: pgmap v14082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:21.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:20 smithi081 ceph-mon[31453]: pgmap v14082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:23.288 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:22 smithi153 ceph-mon[36569]: pgmap v14083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:23.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:22 smithi104 ceph-mon[36624]: pgmap v14083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:23.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:22 smithi081 ceph-mon[31453]: pgmap v14083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:25.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:24 smithi153 ceph-mon[36569]: pgmap v14084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:25.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:24 smithi104 ceph-mon[36624]: pgmap v14084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:25.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:24 smithi081 ceph-mon[31453]: pgmap v14084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:27.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:26 smithi153 ceph-mon[36569]: pgmap v14085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:27.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:26 smithi104 ceph-mon[36624]: pgmap v14085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:27.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:26 smithi081 ceph-mon[31453]: pgmap v14085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:27.663+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:32:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:27.663+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:32:27.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:27.663+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:32:28.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:28 smithi153 ceph-mon[36569]: pgmap v14086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:28.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:28 smithi104 ceph-mon[36624]: pgmap v14086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:28.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:28 smithi081 ceph-mon[31453]: pgmap v14086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:31.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:30 smithi081 ceph-mon[31453]: pgmap v14087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:31.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:30 smithi153 ceph-mon[36569]: pgmap v14087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:31.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:30 smithi104 ceph-mon[36624]: pgmap v14087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:33.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:32 smithi081 ceph-mon[31453]: pgmap v14088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:33.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:32 smithi153 ceph-mon[36569]: pgmap v14088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:33.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:32 smithi104 ceph-mon[36624]: pgmap v14088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:35.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:34 smithi081 ceph-mon[31453]: pgmap v14089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:35.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:34 smithi153 ceph-mon[36569]: pgmap v14089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:35.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:34 smithi104 ceph-mon[36624]: pgmap v14089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:37.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:36 smithi081 ceph-mon[31453]: pgmap v14090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:37.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:36 smithi153 ceph-mon[36569]: pgmap v14090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:37.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:36 smithi104 ceph-mon[36624]: pgmap v14090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:39.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:38 smithi081 ceph-mon[31453]: pgmap v14091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:39.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:38 smithi153 ceph-mon[36569]: pgmap v14091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:39.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:38 smithi104 ceph-mon[36624]: pgmap v14091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:41.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:40 smithi081 ceph-mon[31453]: pgmap v14092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:41.290 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:40 smithi153 ceph-mon[36569]: pgmap v14092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:41.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:40 smithi104 ceph-mon[36624]: pgmap v14092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:42.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:42.675+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:32:42.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:42.676+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:32:42.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:42.676+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:32:43.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:42 smithi081 ceph-mon[31453]: pgmap v14093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:43.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:42 smithi153 ceph-mon[36569]: pgmap v14093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:43.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:42 smithi104 ceph-mon[36624]: pgmap v14093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:45.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:44 smithi153 ceph-mon[36569]: pgmap v14094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:45.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:44 smithi104 ceph-mon[36624]: pgmap v14094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:45.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:44 smithi081 ceph-mon[31453]: pgmap v14094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:47.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:46 smithi153 ceph-mon[36569]: pgmap v14095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:47.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:46 smithi104 ceph-mon[36624]: pgmap v14095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:47.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:46 smithi081 ceph-mon[31453]: pgmap v14095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:49.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:48 smithi153 ceph-mon[36569]: pgmap v14096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:49.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:48 smithi104 ceph-mon[36624]: pgmap v14096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:49.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:48 smithi081 ceph-mon[31453]: pgmap v14096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:51.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:50 smithi153 ceph-mon[36569]: pgmap v14097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:51.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:50 smithi104 ceph-mon[36624]: pgmap v14097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:51.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:50 smithi081 ceph-mon[31453]: pgmap v14097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:53.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:52 smithi153 ceph-mon[36569]: pgmap v14098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:53.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:52 smithi104 ceph-mon[36624]: pgmap v14098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:53.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:52 smithi081 ceph-mon[31453]: pgmap v14098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:55.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:54 smithi153 ceph-mon[36569]: pgmap v14099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:55.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:54 smithi104 ceph-mon[36624]: pgmap v14099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:55.389 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:54 smithi081 ceph-mon[31453]: pgmap v14099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:57.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:56 smithi153 ceph-mon[36569]: pgmap v14100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:57.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:56 smithi104 ceph-mon[36624]: pgmap v14100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:57.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:56 smithi081 ceph-mon[31453]: pgmap v14100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:57.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:57.690+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:32:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:57.691+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:32:57.672 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:32:57.691+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:32:59.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:32:58 smithi153 ceph-mon[36569]: pgmap v14101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:59.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:32:58 smithi104 ceph-mon[36624]: pgmap v14101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:32:59.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:32:58 smithi081 ceph-mon[31453]: pgmap v14101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:01.189 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:00 smithi081 ceph-mon[31453]: pgmap v14102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:01.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:00 smithi153 ceph-mon[36569]: pgmap v14102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:01.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:00 smithi104 ceph-mon[36624]: pgmap v14102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:03.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:02 smithi153 ceph-mon[36569]: pgmap v14103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:03.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:02 smithi104 ceph-mon[36624]: pgmap v14103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:03.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:02 smithi081 ceph-mon[31453]: pgmap v14103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:05.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:04 smithi153 ceph-mon[36569]: pgmap v14104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:05.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:04 smithi104 ceph-mon[36624]: pgmap v14104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:05.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:04 smithi081 ceph-mon[31453]: pgmap v14104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:07.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:06 smithi153 ceph-mon[36569]: pgmap v14105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:07.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:06 smithi104 ceph-mon[36624]: pgmap v14105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:07.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:06 smithi081 ceph-mon[31453]: pgmap v14105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:09.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:08 smithi153 ceph-mon[36569]: pgmap v14106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:09.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:08 smithi104 ceph-mon[36624]: pgmap v14106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:09.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:08 smithi081 ceph-mon[31453]: pgmap v14106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:11.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:10 smithi153 ceph-mon[36569]: pgmap v14107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:11.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:10 smithi104 ceph-mon[36624]: pgmap v14107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:11.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:10 smithi081 ceph-mon[31453]: pgmap v14107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:12.706+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:33:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:12.706+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:33:12.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:12.706+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:33:13.247 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:12 smithi153 ceph-mon[36569]: pgmap v14108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:13.247 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:33:13.248 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:13.248 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:13.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:12 smithi104 ceph-mon[36624]: pgmap v14108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:13.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:33:13.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:13.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:13.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:12 smithi081 ceph-mon[31453]: pgmap v14108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:13.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:33:13.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:13.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:14.024 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:14.024 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:14.024 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:13 smithi104 ceph-mon[36624]: pgmap v14109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:14.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:14.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:14.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:13 smithi153 ceph-mon[36569]: pgmap v14109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:14.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:14.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:14.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:13 smithi081 ceph-mon[31453]: pgmap v14109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:15.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:15.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:15.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:15.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.063 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:16.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:16 smithi081 ceph-mon[31453]: pgmap v14110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:17.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:16 smithi153 ceph-mon[36569]: pgmap v14110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:17.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:16 smithi104 ceph-mon[36624]: pgmap v14110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:18.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:18.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:18.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:18 smithi153 ceph-mon[36569]: pgmap v14111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:18.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:18 smithi104 ceph-mon[36624]: pgmap v14111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:18.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:18.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:18.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:18 smithi081 ceph-mon[31453]: pgmap v14111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:19.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:33:19.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:33:19.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:19 smithi153 ceph-mon[36569]: pgmap v14112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:19.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:19 smithi153 ceph-mon[36569]: pgmap v14113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:19.787 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:19.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:33:19.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:33:19.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:19 smithi104 ceph-mon[36624]: pgmap v14112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:19.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:19 smithi104 ceph-mon[36624]: pgmap v14113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:19.813 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:19.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:33:19.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:33:19.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:19 smithi081 ceph-mon[31453]: pgmap v14112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:19.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:19 smithi081 ceph-mon[31453]: pgmap v14113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:19.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:33:22.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:21 smithi081 ceph-mon[31453]: pgmap v14114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:22.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:21 smithi153 ceph-mon[36569]: pgmap v14114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:22.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:21 smithi104 ceph-mon[36624]: pgmap v14114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:24.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:23 smithi081 ceph-mon[31453]: pgmap v14115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:24.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:23 smithi153 ceph-mon[36569]: pgmap v14115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:24.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:23 smithi104 ceph-mon[36624]: pgmap v14115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:26.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:25 smithi081 ceph-mon[31453]: pgmap v14116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:26.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:25 smithi153 ceph-mon[36569]: pgmap v14116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:26.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:25 smithi104 ceph-mon[36624]: pgmap v14116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:27.720+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:33:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:27.720+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:33:27.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:27.720+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:33:28.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:27 smithi081 ceph-mon[31453]: pgmap v14117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:28.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:27 smithi153 ceph-mon[36569]: pgmap v14117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:28.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:27 smithi104 ceph-mon[36624]: pgmap v14117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:29 smithi081 ceph-mon[31453]: pgmap v14118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:30.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:29 smithi153 ceph-mon[36569]: pgmap v14118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:29 smithi104 ceph-mon[36624]: pgmap v14118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:32.041+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:33:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:32.041+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:33:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:32.041+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 203 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:33:32.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:32.041+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 203 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:33:32.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:31 smithi081 ceph-mon[31453]: pgmap v14119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:32.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:31 smithi153 ceph-mon[36569]: pgmap v14119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:32.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:31 smithi104 ceph-mon[36624]: pgmap v14119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:34.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:33 smithi081 ceph-mon[31453]: pgmap v14120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:34.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:33 smithi153 ceph-mon[36569]: pgmap v14120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:34.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:33 smithi104 ceph-mon[36624]: pgmap v14120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:36.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:35 smithi081 ceph-mon[31453]: pgmap v14121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:36.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:35 smithi153 ceph-mon[36569]: pgmap v14121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:36.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:35 smithi104 ceph-mon[36624]: pgmap v14121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:38.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:37 smithi153 ceph-mon[36569]: pgmap v14122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:38.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:37 smithi104 ceph-mon[36624]: pgmap v14122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:38.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:37 smithi081 ceph-mon[31453]: pgmap v14122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:40.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:39 smithi153 ceph-mon[36569]: pgmap v14123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:40.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:39 smithi104 ceph-mon[36624]: pgmap v14123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:40.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:39 smithi081 ceph-mon[31453]: pgmap v14123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:41.904 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:41 smithi153 ceph-mon[36569]: pgmap v14124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:42.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:41 smithi104 ceph-mon[36624]: pgmap v14124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:42.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:41 smithi081 ceph-mon[31453]: pgmap v14124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:42.727+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:33:42.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:42.727+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:33:42.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:42.727+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:33:44.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:43 smithi153 ceph-mon[36569]: pgmap v14125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:44.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:43 smithi104 ceph-mon[36624]: pgmap v14125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:44.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:43 smithi081 ceph-mon[31453]: pgmap v14125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:46.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:45 smithi153 ceph-mon[36569]: pgmap v14126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:46.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:45 smithi104 ceph-mon[36624]: pgmap v14126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:46.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:45 smithi081 ceph-mon[31453]: pgmap v14126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:48.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:47 smithi153 ceph-mon[36569]: pgmap v14127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:48.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:47 smithi104 ceph-mon[36624]: pgmap v14127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:48.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:47 smithi081 ceph-mon[31453]: pgmap v14127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:50.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:49 smithi153 ceph-mon[36569]: pgmap v14128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:50.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:49 smithi104 ceph-mon[36624]: pgmap v14128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:50.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:49 smithi081 ceph-mon[31453]: pgmap v14128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:52.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:51 smithi153 ceph-mon[36569]: pgmap v14129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:52.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:51 smithi104 ceph-mon[36624]: pgmap v14129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:52.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:51 smithi081 ceph-mon[31453]: pgmap v14129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:54.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:53 smithi153 ceph-mon[36569]: pgmap v14130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:54.291 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:53 smithi081 ceph-mon[31453]: pgmap v14130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:54.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:53 smithi104 ceph-mon[36624]: pgmap v14130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:56.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:55 smithi153 ceph-mon[36569]: pgmap v14131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:56.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:55 smithi104 ceph-mon[36624]: pgmap v14131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:56.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:55 smithi081 ceph-mon[31453]: pgmap v14131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:57.735+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:33:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:57.735+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:33:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:33:57.735+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:33:58.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:57 smithi153 ceph-mon[36569]: pgmap v14132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:57 smithi104 ceph-mon[36624]: pgmap v14132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:33:58.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:57 smithi081 ceph-mon[31453]: pgmap v14132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:00.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:33:59 smithi153 ceph-mon[36569]: pgmap v14133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:00.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:33:59 smithi104 ceph-mon[36624]: pgmap v14133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:00.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:33:59 smithi081 ceph-mon[31453]: pgmap v14133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:02.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:01 smithi153 ceph-mon[36569]: pgmap v14134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:02.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:01 smithi104 ceph-mon[36624]: pgmap v14134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:02.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:01 smithi081 ceph-mon[31453]: pgmap v14134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:04.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:03 smithi153 ceph-mon[36569]: pgmap v14135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:04.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:03 smithi104 ceph-mon[36624]: pgmap v14135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:04.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:03 smithi081 ceph-mon[31453]: pgmap v14135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:06.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:05 smithi153 ceph-mon[36569]: pgmap v14136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:06.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:05 smithi104 ceph-mon[36624]: pgmap v14136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:06.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:05 smithi081 ceph-mon[31453]: pgmap v14136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:08.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:07 smithi153 ceph-mon[36569]: pgmap v14137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:08.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:07 smithi104 ceph-mon[36624]: pgmap v14137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:08.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:07 smithi081 ceph-mon[31453]: pgmap v14137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:09.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:09 smithi153 ceph-mon[36569]: pgmap v14138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:09.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:09 smithi104 ceph-mon[36624]: pgmap v14138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:09.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:09 smithi081 ceph-mon[31453]: pgmap v14138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:12.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:11 smithi081 ceph-mon[31453]: pgmap v14139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:12.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:11 smithi153 ceph-mon[36569]: pgmap v14139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:12.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:11 smithi104 ceph-mon[36624]: pgmap v14139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:12.751+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:34:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:12.751+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:34:12.731 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:12.751+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:34:14.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:13 smithi153 ceph-mon[36569]: pgmap v14140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:14.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:13 smithi081 ceph-mon[31453]: pgmap v14140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:14.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:13 smithi104 ceph-mon[36624]: pgmap v14140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:16.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:15 smithi081 ceph-mon[31453]: pgmap v14141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:16.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:15 smithi153 ceph-mon[36569]: pgmap v14141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:16.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:15 smithi104 ceph-mon[36624]: pgmap v14141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:17.048+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:34:17.028 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:17.048+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:34:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:17.048+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 204 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:34:17.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:17.049+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 204 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:34:18.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:17 smithi153 ceph-mon[36569]: pgmap v14142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:18.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:17 smithi104 ceph-mon[36624]: pgmap v14142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:18.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:17 smithi081 ceph-mon[31453]: pgmap v14142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:20.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:19 smithi153 ceph-mon[36569]: pgmap v14143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:20.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:34:20.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:20.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:20.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:19 smithi104 ceph-mon[36624]: pgmap v14143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:20.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:34:20.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:20.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:20.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:19 smithi081 ceph-mon[31453]: pgmap v14143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:20.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:34:20.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:20.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:21.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:34:21.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:34:21.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:21.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:34:21.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:34:21.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:21.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:34:21.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:34:21.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:34:22.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:21 smithi153 ceph-mon[36569]: pgmap v14144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:22.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:21 smithi153 ceph-mon[36569]: pgmap v14145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:22.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:21 smithi104 ceph-mon[36624]: pgmap v14144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:22.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:21 smithi104 ceph-mon[36624]: pgmap v14145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:21 smithi081 ceph-mon[31453]: pgmap v14144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:22.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:21 smithi081 ceph-mon[31453]: pgmap v14145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:24.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:23 smithi153 ceph-mon[36569]: pgmap v14146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:24.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:23 smithi104 ceph-mon[36624]: pgmap v14146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:24.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:23 smithi081 ceph-mon[31453]: pgmap v14146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:26.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:25 smithi153 ceph-mon[36569]: pgmap v14147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:26.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:25 smithi104 ceph-mon[36624]: pgmap v14147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:26.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:25 smithi081 ceph-mon[31453]: pgmap v14147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:27.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:27.758+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:34:27.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:27.758+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:34:27.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:27.758+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:34:28.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:27 smithi153 ceph-mon[36569]: pgmap v14148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:28.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:27 smithi104 ceph-mon[36624]: pgmap v14148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:28.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:27 smithi081 ceph-mon[31453]: pgmap v14148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:29 smithi153 ceph-mon[36569]: pgmap v14149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:29 smithi104 ceph-mon[36624]: pgmap v14149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:30.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:29 smithi081 ceph-mon[31453]: pgmap v14149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:32.189 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:31 smithi081 ceph-mon[31453]: pgmap v14150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:32.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:31 smithi153 ceph-mon[36569]: pgmap v14150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:32.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:31 smithi104 ceph-mon[36624]: pgmap v14150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:34.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:33 smithi153 ceph-mon[36569]: pgmap v14151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:34.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:33 smithi104 ceph-mon[36624]: pgmap v14151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:34.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:33 smithi081 ceph-mon[31453]: pgmap v14151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:36.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:35 smithi153 ceph-mon[36569]: pgmap v14152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:36.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:35 smithi104 ceph-mon[36624]: pgmap v14152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:36.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:35 smithi081 ceph-mon[31453]: pgmap v14152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:38.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:37 smithi153 ceph-mon[36569]: pgmap v14153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:38.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:37 smithi104 ceph-mon[36624]: pgmap v14153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:38.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:37 smithi081 ceph-mon[31453]: pgmap v14153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:40.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:39 smithi153 ceph-mon[36569]: pgmap v14154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:40.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:39 smithi104 ceph-mon[36624]: pgmap v14154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:40.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:39 smithi081 ceph-mon[31453]: pgmap v14154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:42.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:41 smithi153 ceph-mon[36569]: pgmap v14155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:42.314 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:41 smithi104 ceph-mon[36624]: pgmap v14155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:42.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:41 smithi081 ceph-mon[31453]: pgmap v14155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:42.774+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:34:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:42.774+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:34:42.754 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:42.774+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:34:44.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:43 smithi153 ceph-mon[36569]: pgmap v14156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:44.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:43 smithi104 ceph-mon[36624]: pgmap v14156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:44.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:43 smithi081 ceph-mon[31453]: pgmap v14156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:46.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:45 smithi153 ceph-mon[36569]: pgmap v14157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:46.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:45 smithi104 ceph-mon[36624]: pgmap v14157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:46.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:45 smithi081 ceph-mon[31453]: pgmap v14157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:48.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:47 smithi153 ceph-mon[36569]: pgmap v14158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:48.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:47 smithi104 ceph-mon[36624]: pgmap v14158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:48.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:47 smithi081 ceph-mon[31453]: pgmap v14158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:49.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:49 smithi153 ceph-mon[36569]: pgmap v14159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:49.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:49 smithi104 ceph-mon[36624]: pgmap v14159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:49.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:49 smithi081 ceph-mon[31453]: pgmap v14159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:51.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:51 smithi153 ceph-mon[36569]: pgmap v14160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:51.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:51 smithi104 ceph-mon[36624]: pgmap v14160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:51.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:51 smithi081 ceph-mon[31453]: pgmap v14160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:53.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:53 smithi153 ceph-mon[36569]: pgmap v14161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:53.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:53 smithi104 ceph-mon[36624]: pgmap v14161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:53.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:53 smithi081 ceph-mon[31453]: pgmap v14161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:55.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:55 smithi153 ceph-mon[36569]: pgmap v14162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:55.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:55 smithi104 ceph-mon[36624]: pgmap v14162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:55.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:55 smithi081 ceph-mon[31453]: pgmap v14162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:57.790+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:34:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:57.790+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:34:57.770 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:34:57.790+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:34:57.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:57 smithi153 ceph-mon[36569]: pgmap v14163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:57.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:57 smithi104 ceph-mon[36624]: pgmap v14163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:57.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:57 smithi081 ceph-mon[31453]: pgmap v14163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:59.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:34:59 smithi153 ceph-mon[36569]: pgmap v14164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:59.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:34:59 smithi104 ceph-mon[36624]: pgmap v14164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:34:59.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:34:59 smithi081 ceph-mon[31453]: pgmap v14164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:01.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:01 smithi153 ceph-mon[36569]: pgmap v14165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:01.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:01 smithi104 ceph-mon[36624]: pgmap v14165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:01.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:01 smithi081 ceph-mon[31453]: pgmap v14165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:03.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:03 smithi153 ceph-mon[36569]: pgmap v14166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:03.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:03 smithi104 ceph-mon[36624]: pgmap v14166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:03.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:03 smithi081 ceph-mon[31453]: pgmap v14166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:05.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:05 smithi153 ceph-mon[36569]: pgmap v14167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:05.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:05 smithi104 ceph-mon[36624]: pgmap v14167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:05.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:05 smithi081 ceph-mon[31453]: pgmap v14167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:07.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:07 smithi153 ceph-mon[36569]: pgmap v14168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:07.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:07 smithi104 ceph-mon[36624]: pgmap v14168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:07.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:07 smithi081 ceph-mon[31453]: pgmap v14168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:09.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:09 smithi153 ceph-mon[36569]: pgmap v14169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:09.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:09 smithi104 ceph-mon[36624]: pgmap v14169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:09.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:09 smithi081 ceph-mon[31453]: pgmap v14169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:11.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:11 smithi104 ceph-mon[36624]: pgmap v14170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:11.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:11 smithi081 ceph-mon[31453]: pgmap v14170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:12.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:11 smithi153 ceph-mon[36569]: pgmap v14170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:12.792+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:35:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:12.792+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:35:12.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:12.792+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:35:13.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:13 smithi104 ceph-mon[36624]: pgmap v14171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:13.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:13 smithi081 ceph-mon[31453]: pgmap v14171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:14.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:13 smithi153 ceph-mon[36569]: pgmap v14171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:15.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:15 smithi104 ceph-mon[36624]: pgmap v14172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:15.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:15 smithi081 ceph-mon[31453]: pgmap v14172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:16.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:15 smithi153 ceph-mon[36569]: pgmap v14172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:17.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:17 smithi104 ceph-mon[36624]: pgmap v14173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:17.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:17 smithi081 ceph-mon[31453]: pgmap v14173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:18.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:17 smithi153 ceph-mon[36569]: pgmap v14173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:19.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:19 smithi104 ceph-mon[36624]: pgmap v14174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:19.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:19 smithi081 ceph-mon[31453]: pgmap v14174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:20.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:19 smithi153 ceph-mon[36569]: pgmap v14174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:21.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:21 smithi081 ceph-mon[31453]: pgmap v14175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:21.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:35:22.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:21 smithi153 ceph-mon[36569]: pgmap v14175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:22.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:35:22.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:21 smithi104 ceph-mon[36624]: pgmap v14175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:22.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:35:22.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:35:22.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:35:22.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:22 smithi081 ceph-mon[31453]: pgmap v14176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:22.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:22 smithi081 ceph-mon[31453]: pgmap v14177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:22.887 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:35:23.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:35:23.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:35:23.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:22 smithi153 ceph-mon[36569]: pgmap v14176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:23.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:22 smithi153 ceph-mon[36569]: pgmap v14177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:23.037 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:35:23.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:35:23.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:35:23.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:22 smithi104 ceph-mon[36624]: pgmap v14176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:23.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:22 smithi104 ceph-mon[36624]: pgmap v14177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:23.062 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:35:25.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:25 smithi153 ceph-mon[36569]: pgmap v14178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:25.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:25 smithi104 ceph-mon[36624]: pgmap v14178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:25.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:25 smithi081 ceph-mon[31453]: pgmap v14178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:26.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:26 smithi153 ceph-mon[36569]: pgmap v14179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:26.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:26 smithi104 ceph-mon[36624]: pgmap v14179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:26.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:26 smithi081 ceph-mon[31453]: pgmap v14179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:27.800+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:35:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:27.800+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:35:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:27.800+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:35:28.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:28 smithi081 ceph-mon[31453]: pgmap v14180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:28.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:28 smithi153 ceph-mon[36569]: pgmap v14180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:28.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:28 smithi104 ceph-mon[36624]: pgmap v14180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:30 smithi104 ceph-mon[36624]: pgmap v14181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:30.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:30 smithi081 ceph-mon[31453]: pgmap v14181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:30.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:30 smithi153 ceph-mon[36569]: pgmap v14181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:32.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:32 smithi104 ceph-mon[36624]: pgmap v14182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:32.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:32 smithi081 ceph-mon[31453]: pgmap v14182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:32.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:32 smithi153 ceph-mon[36569]: pgmap v14182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:34.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:34 smithi104 ceph-mon[36624]: pgmap v14183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:34.338 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:34 smithi153 ceph-mon[36569]: pgmap v14183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:34.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:34 smithi081 ceph-mon[31453]: pgmap v14183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:36.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:36 smithi104 ceph-mon[36624]: pgmap v14184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:36.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:36 smithi081 ceph-mon[31453]: pgmap v14184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:36.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:36 smithi153 ceph-mon[36569]: pgmap v14184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:38.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:38 smithi081 ceph-mon[31453]: pgmap v14185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:38.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:38 smithi153 ceph-mon[36569]: pgmap v14185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:38.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:38 smithi104 ceph-mon[36624]: pgmap v14185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:40.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:40 smithi104 ceph-mon[36624]: pgmap v14186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:40.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:40 smithi081 ceph-mon[31453]: pgmap v14186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:40.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:40 smithi153 ceph-mon[36569]: pgmap v14186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:42.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:42 smithi104 ceph-mon[36624]: pgmap v14187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:42.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:42 smithi081 ceph-mon[31453]: pgmap v14187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:42.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:42 smithi153 ceph-mon[36569]: pgmap v14187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:42.814+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:35:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:42.814+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:35:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:42.814+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:35:44.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:44 smithi104 ceph-mon[36624]: pgmap v14188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:44.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:44 smithi081 ceph-mon[31453]: pgmap v14188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:44.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:44 smithi153 ceph-mon[36569]: pgmap v14188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:46.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:46 smithi104 ceph-mon[36624]: pgmap v14189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:46.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:46 smithi081 ceph-mon[31453]: pgmap v14189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:46.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:46 smithi153 ceph-mon[36569]: pgmap v14189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:48.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:48 smithi104 ceph-mon[36624]: pgmap v14190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:48.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:48 smithi081 ceph-mon[31453]: pgmap v14190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:48.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:48 smithi153 ceph-mon[36569]: pgmap v14190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:50.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:50 smithi104 ceph-mon[36624]: pgmap v14191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:50.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:50 smithi081 ceph-mon[31453]: pgmap v14191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:50.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:50 smithi153 ceph-mon[36569]: pgmap v14191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:52.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:52 smithi104 ceph-mon[36624]: pgmap v14192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:52.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:52 smithi081 ceph-mon[31453]: pgmap v14192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:52.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:52 smithi153 ceph-mon[36569]: pgmap v14192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:54.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:54 smithi104 ceph-mon[36624]: pgmap v14193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:54.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:54 smithi081 ceph-mon[31453]: pgmap v14193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:54.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:54 smithi153 ceph-mon[36569]: pgmap v14193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:56.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:56 smithi104 ceph-mon[36624]: pgmap v14194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:56.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:56 smithi081 ceph-mon[31453]: pgmap v14194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:56.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:56 smithi153 ceph-mon[36569]: pgmap v14194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:57.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:57.830+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:35:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:57.830+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:35:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:35:57.830+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:35:58.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:35:58 smithi104 ceph-mon[36624]: pgmap v14195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:58.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:35:58 smithi081 ceph-mon[31453]: pgmap v14195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:35:58.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:35:58 smithi153 ceph-mon[36569]: pgmap v14195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:00.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:00 smithi104 ceph-mon[36624]: pgmap v14196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:00.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:00 smithi081 ceph-mon[31453]: pgmap v14196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:00.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:00 smithi153 ceph-mon[36569]: pgmap v14196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:02.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:02 smithi104 ceph-mon[36624]: pgmap v14197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:02.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:02 smithi081 ceph-mon[31453]: pgmap v14197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:02.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:02 smithi153 ceph-mon[36569]: pgmap v14197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:04.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:04 smithi104 ceph-mon[36624]: pgmap v14198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:04.338 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:04 smithi153 ceph-mon[36569]: pgmap v14198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:04.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:04 smithi081 ceph-mon[31453]: pgmap v14198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:06.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:06 smithi104 ceph-mon[36624]: pgmap v14199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:06.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:06 smithi081 ceph-mon[31453]: pgmap v14199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:06.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:06 smithi153 ceph-mon[36569]: pgmap v14199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:08.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:08 smithi104 ceph-mon[36624]: pgmap v14200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:08.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:08 smithi081 ceph-mon[31453]: pgmap v14200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:08.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:08 smithi153 ceph-mon[36569]: pgmap v14200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:10.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:10 smithi104 ceph-mon[36624]: pgmap v14201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:10.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:10 smithi081 ceph-mon[31453]: pgmap v14201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:10.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:10 smithi153 ceph-mon[36569]: pgmap v14201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:12.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:12 smithi104 ceph-mon[36624]: pgmap v14202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:12.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:12 smithi081 ceph-mon[31453]: pgmap v14202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:12.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:12 smithi153 ceph-mon[36569]: pgmap v14202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:12.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:12.838+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:36:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:12.838+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:36:12.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:12.838+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:36:14.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:14 smithi104 ceph-mon[36624]: pgmap v14203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:14.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:14 smithi081 ceph-mon[31453]: pgmap v14203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:14.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:14 smithi153 ceph-mon[36569]: pgmap v14203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:16.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:16 smithi104 ceph-mon[36624]: pgmap v14204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:16.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:16 smithi081 ceph-mon[31453]: pgmap v14204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:16.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:16 smithi153 ceph-mon[36569]: pgmap v14204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:18.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:18 smithi104 ceph-mon[36624]: pgmap v14205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:18.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:18 smithi081 ceph-mon[31453]: pgmap v14205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:18.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:18 smithi153 ceph-mon[36569]: pgmap v14205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:20.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:20 smithi104 ceph-mon[36624]: pgmap v14206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:20.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:20 smithi081 ceph-mon[31453]: pgmap v14206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:20.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:20 smithi153 ceph-mon[36569]: pgmap v14206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:22.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:22 smithi104 ceph-mon[36624]: pgmap v14207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:22.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:22 smithi081 ceph-mon[31453]: pgmap v14207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:22.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:22 smithi153 ceph-mon[36569]: pgmap v14207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:23.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:36:23.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:36:23.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:36:24.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:36:24.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:36:24.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:24 smithi081 ceph-mon[31453]: pgmap v14208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:24.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:24 smithi081 ceph-mon[31453]: pgmap v14209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:24.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:36:24.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:36:24.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:36:24.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:24 smithi153 ceph-mon[36569]: pgmap v14208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:24.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:24 smithi153 ceph-mon[36569]: pgmap v14209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:24.536 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:36:24.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:36:24.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:36:24.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:24 smithi104 ceph-mon[36624]: pgmap v14208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:24.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:24 smithi104 ceph-mon[36624]: pgmap v14209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:24.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:36:26.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:26 smithi153 ceph-mon[36569]: pgmap v14210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:26.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:26 smithi104 ceph-mon[36624]: pgmap v14210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:26.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:26 smithi081 ceph-mon[31453]: pgmap v14210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:27.843+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:36:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:27.843+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:36:27.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:27.843+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:36:28.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:28 smithi081 ceph-mon[31453]: pgmap v14211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:28.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:28 smithi153 ceph-mon[36569]: pgmap v14211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:28.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:28 smithi104 ceph-mon[36624]: pgmap v14211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:30.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:30 smithi153 ceph-mon[36569]: pgmap v14212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:30.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:30 smithi104 ceph-mon[36624]: pgmap v14212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:30.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:30 smithi081 ceph-mon[31453]: pgmap v14212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:32.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:32 smithi153 ceph-mon[36569]: pgmap v14213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:32.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:32 smithi104 ceph-mon[36624]: pgmap v14213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:32.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:32 smithi081 ceph-mon[31453]: pgmap v14213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:34.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:34 smithi153 ceph-mon[36569]: pgmap v14214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:34.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:34 smithi104 ceph-mon[36624]: pgmap v14214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:34.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:34 smithi081 ceph-mon[31453]: pgmap v14214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:36.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:36 smithi153 ceph-mon[36569]: pgmap v14215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:36.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:36 smithi104 ceph-mon[36624]: pgmap v14215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:36.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:36 smithi081 ceph-mon[31453]: pgmap v14215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:38.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:38 smithi153 ceph-mon[36569]: pgmap v14216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:38.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:38 smithi104 ceph-mon[36624]: pgmap v14216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:38.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:38 smithi081 ceph-mon[31453]: pgmap v14216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:40.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:40 smithi153 ceph-mon[36569]: pgmap v14217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:40.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:40 smithi104 ceph-mon[36624]: pgmap v14217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:40.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:40 smithi081 ceph-mon[31453]: pgmap v14217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:42.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:42 smithi153 ceph-mon[36569]: pgmap v14218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:42.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:42 smithi104 ceph-mon[36624]: pgmap v14218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:42.857+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:36:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:42.858+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:36:42.837 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:42.858+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:36:42.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:42 smithi081 ceph-mon[31453]: pgmap v14218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:44.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:44 smithi104 ceph-mon[36624]: pgmap v14219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:44.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:44 smithi081 ceph-mon[31453]: pgmap v14219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:45.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:44 smithi153 ceph-mon[36569]: pgmap v14219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:46.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:46 smithi104 ceph-mon[36624]: pgmap v14220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:46.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:46 smithi081 ceph-mon[31453]: pgmap v14220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:47.036 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:46 smithi153 ceph-mon[36569]: pgmap v14220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:48.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:48 smithi104 ceph-mon[36624]: pgmap v14221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:48.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:48 smithi081 ceph-mon[31453]: pgmap v14221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:49.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:48 smithi153 ceph-mon[36569]: pgmap v14221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:50.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:50 smithi104 ceph-mon[36624]: pgmap v14222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:50.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:50 smithi081 ceph-mon[31453]: pgmap v14222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:51.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:50 smithi153 ceph-mon[36569]: pgmap v14222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:52.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:52 smithi104 ceph-mon[36624]: pgmap v14223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:52.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:52 smithi081 ceph-mon[31453]: pgmap v14223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:53.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:52 smithi153 ceph-mon[36569]: pgmap v14223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:54.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:54 smithi081 ceph-mon[31453]: pgmap v14224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:55.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:54 smithi153 ceph-mon[36569]: pgmap v14224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:55.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:54 smithi104 ceph-mon[36624]: pgmap v14224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:56.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:56 smithi081 ceph-mon[31453]: pgmap v14225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:57.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:56 smithi153 ceph-mon[36569]: pgmap v14225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:57.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:56 smithi104 ceph-mon[36624]: pgmap v14225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:57.852 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:57.873+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:36:57.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:57.873+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:36:57.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:36:57.873+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:36:58.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:36:58 smithi081 ceph-mon[31453]: pgmap v14226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:59.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:36:58 smithi153 ceph-mon[36569]: pgmap v14226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:36:59.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:36:58 smithi104 ceph-mon[36624]: pgmap v14226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:00.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:00 smithi081 ceph-mon[31453]: pgmap v14227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:01.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:00 smithi153 ceph-mon[36569]: pgmap v14227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:01.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:00 smithi104 ceph-mon[36624]: pgmap v14227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:02.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:02 smithi081 ceph-mon[31453]: pgmap v14228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:03.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:02 smithi153 ceph-mon[36569]: pgmap v14228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:03.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:02 smithi104 ceph-mon[36624]: pgmap v14228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:04.863 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:04 smithi104 ceph-mon[36624]: pgmap v14229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:04.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:04 smithi081 ceph-mon[31453]: pgmap v14229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:05.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:04 smithi153 ceph-mon[36569]: pgmap v14229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:06.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:06 smithi081 ceph-mon[31453]: pgmap v14230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:07.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:06 smithi153 ceph-mon[36569]: pgmap v14230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:07.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:06 smithi104 ceph-mon[36624]: pgmap v14230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:08.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:08 smithi081 ceph-mon[31453]: pgmap v14231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:09.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:08 smithi153 ceph-mon[36569]: pgmap v14231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:09.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:08 smithi104 ceph-mon[36624]: pgmap v14231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:10.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:10 smithi081 ceph-mon[31453]: pgmap v14232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:11.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:10 smithi153 ceph-mon[36569]: pgmap v14232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:11.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:10 smithi104 ceph-mon[36624]: pgmap v14232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:12.875+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:37:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:12.875+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:37:12.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:12.875+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:37:12.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:12 smithi081 ceph-mon[31453]: pgmap v14233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:13.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:12 smithi153 ceph-mon[36569]: pgmap v14233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:13.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:12 smithi104 ceph-mon[36624]: pgmap v14233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:14.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:14 smithi081 ceph-mon[31453]: pgmap v14234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:15.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:14 smithi153 ceph-mon[36569]: pgmap v14234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:15.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:14 smithi104 ceph-mon[36624]: pgmap v14234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:16.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:16 smithi081 ceph-mon[31453]: pgmap v14235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:17.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:16 smithi153 ceph-mon[36569]: pgmap v14235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:17.077+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:37:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:17.077+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:37:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:17.077+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 205 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:37:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:17.078+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 205 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:37:17.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:16 smithi104 ceph-mon[36624]: pgmap v14235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:18.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:18 smithi081 ceph-mon[31453]: pgmap v14236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:19.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:18 smithi153 ceph-mon[36569]: pgmap v14236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:19.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:18 smithi104 ceph-mon[36624]: pgmap v14236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:20.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:20 smithi081 ceph-mon[31453]: pgmap v14237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:21.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:20 smithi153 ceph-mon[36569]: pgmap v14237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:21.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:20 smithi104 ceph-mon[36624]: pgmap v14237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:23.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:22 smithi153 ceph-mon[36569]: pgmap v14238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:23.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:22 smithi104 ceph-mon[36624]: pgmap v14238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:23.134 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:22 smithi081 ceph-mon[31453]: pgmap v14238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:25.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:24 smithi153 ceph-mon[36569]: pgmap v14239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:25.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:37:25.060 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:24 smithi104 ceph-mon[36624]: pgmap v14239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:25.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:37:25.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:24 smithi081 ceph-mon[31453]: pgmap v14239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:25.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:37:26.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:37:26.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:37:26.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:25 smithi153 ceph-mon[36569]: pgmap v14240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:26.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:25 smithi153 ceph-mon[36569]: pgmap v14241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:26.035 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:37:26.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:37:26.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:37:26.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:25 smithi104 ceph-mon[36624]: pgmap v14240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:26.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:25 smithi104 ceph-mon[36624]: pgmap v14241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:26.061 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:37:26.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:37:26.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:37:26.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:25 smithi081 ceph-mon[31453]: pgmap v14240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:26.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:25 smithi081 ceph-mon[31453]: pgmap v14241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:26.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:37:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:27.887+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:37:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:27.887+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:37:27.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:27.887+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:37:28.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:28 smithi153 ceph-mon[36569]: pgmap v14242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:28.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:28 smithi104 ceph-mon[36624]: pgmap v14242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:28.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:28 smithi081 ceph-mon[31453]: pgmap v14242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:30 smithi153 ceph-mon[36569]: pgmap v14243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:30.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:30 smithi104 ceph-mon[36624]: pgmap v14243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:30.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:30 smithi081 ceph-mon[31453]: pgmap v14243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:32.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:32 smithi153 ceph-mon[36569]: pgmap v14244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:32.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:32 smithi104 ceph-mon[36624]: pgmap v14244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:32.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:32 smithi081 ceph-mon[31453]: pgmap v14244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:33.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:33 smithi081 ceph-mon[31453]: pgmap v14245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:33.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:33 smithi153 ceph-mon[36569]: pgmap v14245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:33.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:33 smithi104 ceph-mon[36624]: pgmap v14245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:36.264 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:36 smithi104 ceph-mon[36624]: pgmap v14246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:36.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:36 smithi153 ceph-mon[36569]: pgmap v14246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:36.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:36 smithi081 ceph-mon[31453]: pgmap v14246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:38.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:38 smithi153 ceph-mon[36569]: pgmap v14247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:38.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:38 smithi104 ceph-mon[36624]: pgmap v14247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:38.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:38 smithi081 ceph-mon[31453]: pgmap v14247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:40.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:40 smithi153 ceph-mon[36569]: pgmap v14248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:40.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:40 smithi104 ceph-mon[36624]: pgmap v14248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:40.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:40 smithi081 ceph-mon[31453]: pgmap v14248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:41.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:41 smithi153 ceph-mon[36569]: pgmap v14249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:41.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:41 smithi104 ceph-mon[36624]: pgmap v14249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:41.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:41 smithi081 ceph-mon[31453]: pgmap v14249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:42.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:42.903+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:37:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:42.903+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:37:42.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:42.903+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:37:44.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:44 smithi153 ceph-mon[36569]: pgmap v14250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:44.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:44 smithi104 ceph-mon[36624]: pgmap v14250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:44.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:44 smithi081 ceph-mon[31453]: pgmap v14250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:46.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:46 smithi153 ceph-mon[36569]: pgmap v14251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:46.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:46 smithi104 ceph-mon[36624]: pgmap v14251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:46.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:46 smithi081 ceph-mon[31453]: pgmap v14251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:47.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:47 smithi081 ceph-mon[31453]: pgmap v14252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:47.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:47 smithi153 ceph-mon[36569]: pgmap v14252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:47.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:47 smithi104 ceph-mon[36624]: pgmap v14252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:50.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:50 smithi153 ceph-mon[36569]: pgmap v14253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:50.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:50 smithi104 ceph-mon[36624]: pgmap v14253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:50.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:50 smithi081 ceph-mon[31453]: pgmap v14253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:52.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:52 smithi153 ceph-mon[36569]: pgmap v14254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:52.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:52 smithi104 ceph-mon[36624]: pgmap v14254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:52.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:52 smithi081 ceph-mon[31453]: pgmap v14254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:53.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:53 smithi153 ceph-mon[36569]: pgmap v14255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:53.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:53 smithi104 ceph-mon[36624]: pgmap v14255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:53 smithi081 ceph-mon[31453]: pgmap v14255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:56.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:56 smithi153 ceph-mon[36569]: pgmap v14256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:56.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:56 smithi104 ceph-mon[36624]: pgmap v14256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:56.385 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:56 smithi081 ceph-mon[31453]: pgmap v14256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:57.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:37:57 smithi081 ceph-mon[31453]: pgmap v14257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:57.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:37:57 smithi153 ceph-mon[36569]: pgmap v14257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:57.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:37:57 smithi104 ceph-mon[36624]: pgmap v14257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:37:57.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:57.917+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:37:57.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:57.917+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:37:57.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:37:57.917+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:38:00.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:00 smithi153 ceph-mon[36569]: pgmap v14258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:00.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:00 smithi104 ceph-mon[36624]: pgmap v14258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:00.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:00 smithi081 ceph-mon[31453]: pgmap v14258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:01.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:01 smithi081 ceph-mon[31453]: pgmap v14259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:01.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:01 smithi153 ceph-mon[36569]: pgmap v14259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:01.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:01 smithi104 ceph-mon[36624]: pgmap v14259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:04.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:04 smithi153 ceph-mon[36569]: pgmap v14260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:04.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:04 smithi104 ceph-mon[36624]: pgmap v14260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:04.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:04 smithi081 ceph-mon[31453]: pgmap v14260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:05.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:05 smithi153 ceph-mon[36569]: pgmap v14261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:05.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:05 smithi104 ceph-mon[36624]: pgmap v14261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:05.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:05 smithi081 ceph-mon[31453]: pgmap v14261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:07.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:07 smithi081 ceph-mon[31453]: pgmap v14262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:07.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:07 smithi153 ceph-mon[36569]: pgmap v14262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:07.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:07 smithi104 ceph-mon[36624]: pgmap v14262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:09.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:09 smithi081 ceph-mon[31453]: pgmap v14263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:09.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:09 smithi153 ceph-mon[36569]: pgmap v14263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:09.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:09 smithi104 ceph-mon[36624]: pgmap v14263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:11.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:11 smithi081 ceph-mon[31453]: pgmap v14264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:11.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:11 smithi153 ceph-mon[36569]: pgmap v14264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:11.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:11 smithi104 ceph-mon[36624]: pgmap v14264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:12.934+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:38:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:12.934+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:38:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:12.934+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:38:13.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:13 smithi081 ceph-mon[31453]: pgmap v14265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:13.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:13 smithi153 ceph-mon[36569]: pgmap v14265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:13.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:13 smithi104 ceph-mon[36624]: pgmap v14265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:15.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:15 smithi081 ceph-mon[31453]: pgmap v14266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:15.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:15 smithi153 ceph-mon[36569]: pgmap v14266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:15.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:15 smithi104 ceph-mon[36624]: pgmap v14266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:17.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:17 smithi081 ceph-mon[31453]: pgmap v14267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:17.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:17 smithi153 ceph-mon[36569]: pgmap v14267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:17.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:17 smithi104 ceph-mon[36624]: pgmap v14267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:19.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:19 smithi081 ceph-mon[31453]: pgmap v14268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:19.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:19 smithi153 ceph-mon[36569]: pgmap v14268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:19.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:19 smithi104 ceph-mon[36624]: pgmap v14268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:21.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:21 smithi081 ceph-mon[31453]: pgmap v14269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:21.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:21 smithi153 ceph-mon[36569]: pgmap v14269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:21.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:21 smithi104 ceph-mon[36624]: pgmap v14269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:23.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:23 smithi081 ceph-mon[31453]: pgmap v14270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:23.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:23 smithi153 ceph-mon[36569]: pgmap v14270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:23.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:23 smithi104 ceph-mon[36624]: pgmap v14270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:25.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:25 smithi081 ceph-mon[31453]: pgmap v14271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:25.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:25 smithi153 ceph-mon[36569]: pgmap v14271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:25.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:25 smithi104 ceph-mon[36624]: pgmap v14271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:26.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:38:26.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:38:26.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:38:27.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:27 smithi081 ceph-mon[31453]: pgmap v14272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:27.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:27 smithi153 ceph-mon[36569]: pgmap v14272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:27.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:27 smithi104 ceph-mon[36624]: pgmap v14272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:27.936+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:38:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:27.936+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:38:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:27.936+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:38:28.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:28.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:28.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:38:28.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:38:28.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:28 smithi153 ceph-mon[36569]: pgmap v14273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:28.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:28 smithi153 ceph-mon[36569]: pgmap v14274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:28.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:28.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:28.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:28.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:38:28.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:38:28.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:28 smithi104 ceph-mon[36624]: pgmap v14273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:28.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:28 smithi104 ceph-mon[36624]: pgmap v14274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:28.561 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:28.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:28.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:28.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:38:28.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:38:28.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:28 smithi081 ceph-mon[31453]: pgmap v14273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:28.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:28 smithi081 ceph-mon[31453]: pgmap v14274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:28.635 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:38:30.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:30 smithi081 ceph-mon[31453]: pgmap v14275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:30.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:30 smithi153 ceph-mon[36569]: pgmap v14275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:30.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:30 smithi104 ceph-mon[36624]: pgmap v14275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:32.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:32 smithi081 ceph-mon[31453]: pgmap v14276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:32.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:32 smithi153 ceph-mon[36569]: pgmap v14276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:32.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:32 smithi104 ceph-mon[36624]: pgmap v14276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:34.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:34 smithi081 ceph-mon[31453]: pgmap v14277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:34.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:34 smithi153 ceph-mon[36569]: pgmap v14277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:34.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:34 smithi104 ceph-mon[36624]: pgmap v14277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:36.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:36 smithi081 ceph-mon[31453]: pgmap v14278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:36.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:36 smithi153 ceph-mon[36569]: pgmap v14278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:36.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:36 smithi104 ceph-mon[36624]: pgmap v14278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:38.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:38 smithi081 ceph-mon[31453]: pgmap v14279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:38.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:38 smithi153 ceph-mon[36569]: pgmap v14279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:38.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:38 smithi104 ceph-mon[36624]: pgmap v14279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:40.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:40 smithi081 ceph-mon[31453]: pgmap v14280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:40.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:40 smithi153 ceph-mon[36569]: pgmap v14280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:40.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:40 smithi104 ceph-mon[36624]: pgmap v14280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:42.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:42 smithi081 ceph-mon[31453]: pgmap v14281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:42.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:42 smithi153 ceph-mon[36569]: pgmap v14281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:42.821 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:42 smithi104 ceph-mon[36624]: pgmap v14281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:42.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:42.945+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:38:42.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:42.945+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:38:42.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:42.945+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:38:44.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:44 smithi081 ceph-mon[31453]: pgmap v14282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:44.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:44 smithi153 ceph-mon[36569]: pgmap v14282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:44.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:44 smithi104 ceph-mon[36624]: pgmap v14282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:46.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:46 smithi153 ceph-mon[36569]: pgmap v14283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:46.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:46 smithi104 ceph-mon[36624]: pgmap v14283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:46.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:46 smithi081 ceph-mon[31453]: pgmap v14283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:48.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:48 smithi153 ceph-mon[36569]: pgmap v14284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:48.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:48 smithi104 ceph-mon[36624]: pgmap v14284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:48.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:48 smithi081 ceph-mon[31453]: pgmap v14284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:50.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:50 smithi153 ceph-mon[36569]: pgmap v14285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:50.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:50 smithi104 ceph-mon[36624]: pgmap v14285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:50.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:50 smithi081 ceph-mon[31453]: pgmap v14285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:52.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:52 smithi153 ceph-mon[36569]: pgmap v14286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:52.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:52 smithi104 ceph-mon[36624]: pgmap v14286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:52.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:52 smithi081 ceph-mon[31453]: pgmap v14286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:54.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:54 smithi153 ceph-mon[36569]: pgmap v14287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:54.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:54 smithi104 ceph-mon[36624]: pgmap v14287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:54.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:54 smithi081 ceph-mon[31453]: pgmap v14287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:56.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:56 smithi153 ceph-mon[36569]: pgmap v14288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:56.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:56 smithi104 ceph-mon[36624]: pgmap v14288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:56.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:56 smithi081 ceph-mon[31453]: pgmap v14288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:57.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:38:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:57.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:38:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:38:57.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:38:58.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:38:58 smithi153 ceph-mon[36569]: pgmap v14289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:58.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:38:58 smithi104 ceph-mon[36624]: pgmap v14289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:38:58.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:38:58 smithi081 ceph-mon[31453]: pgmap v14289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:00.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:00 smithi153 ceph-mon[36569]: pgmap v14290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:00.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:00 smithi104 ceph-mon[36624]: pgmap v14290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:00.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:00 smithi081 ceph-mon[31453]: pgmap v14290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:02.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:02 smithi153 ceph-mon[36569]: pgmap v14291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:02.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:02 smithi104 ceph-mon[36624]: pgmap v14291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:02.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:02 smithi081 ceph-mon[31453]: pgmap v14291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:04.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:04 smithi153 ceph-mon[36569]: pgmap v14292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:04.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:04 smithi104 ceph-mon[36624]: pgmap v14292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:04.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:04 smithi081 ceph-mon[31453]: pgmap v14292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:06.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:06 smithi153 ceph-mon[36569]: pgmap v14293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:06.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:06 smithi104 ceph-mon[36624]: pgmap v14293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:06.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:06 smithi081 ceph-mon[31453]: pgmap v14293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:08.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:08 smithi153 ceph-mon[36569]: pgmap v14294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:08.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:08 smithi104 ceph-mon[36624]: pgmap v14294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:08.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:08 smithi081 ceph-mon[31453]: pgmap v14294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:10.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:10 smithi153 ceph-mon[36569]: pgmap v14295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:10.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:10 smithi104 ceph-mon[36624]: pgmap v14295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:10.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:10 smithi081 ceph-mon[31453]: pgmap v14295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:12.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:12 smithi153 ceph-mon[36569]: pgmap v14296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:12.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:12 smithi104 ceph-mon[36624]: pgmap v14296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:12.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:12 smithi081 ceph-mon[31453]: pgmap v14296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:12.970+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:39:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:12.970+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:39:12.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:12.970+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:39:14.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:14 smithi153 ceph-mon[36569]: pgmap v14297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:14.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:14 smithi104 ceph-mon[36624]: pgmap v14297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:14.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:14 smithi081 ceph-mon[31453]: pgmap v14297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:16.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:16 smithi153 ceph-mon[36569]: pgmap v14298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:16.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:16 smithi104 ceph-mon[36624]: pgmap v14298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:16.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:16 smithi081 ceph-mon[31453]: pgmap v14298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:18.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:18 smithi153 ceph-mon[36569]: pgmap v14299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:18.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:18 smithi104 ceph-mon[36624]: pgmap v14299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:18.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:18 smithi081 ceph-mon[31453]: pgmap v14299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:20.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:20 smithi153 ceph-mon[36569]: pgmap v14300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:20.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:20 smithi104 ceph-mon[36624]: pgmap v14300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:20.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:20 smithi081 ceph-mon[31453]: pgmap v14300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:22.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:22 smithi153 ceph-mon[36569]: pgmap v14301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:22.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:22 smithi104 ceph-mon[36624]: pgmap v14301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:22.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:22 smithi081 ceph-mon[31453]: pgmap v14301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:24.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:24 smithi153 ceph-mon[36569]: pgmap v14302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:24.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:24 smithi104 ceph-mon[36624]: pgmap v14302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:24.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:24 smithi081 ceph-mon[31453]: pgmap v14302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:26.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:26 smithi153 ceph-mon[36569]: pgmap v14303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:26.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:26 smithi104 ceph-mon[36624]: pgmap v14303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:26.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:26 smithi081 ceph-mon[31453]: pgmap v14303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:27.986+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:39:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:27.986+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:39:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:27.986+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:39:28.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:28 smithi153 ceph-mon[36569]: pgmap v14304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:28.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:39:28.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:28 smithi104 ceph-mon[36624]: pgmap v14304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:28.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:39:28.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:28 smithi081 ceph-mon[31453]: pgmap v14304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:28.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:39:30.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.135 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:39:30.136 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:39:30.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:39:30.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:39:30.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:39:30.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: pgmap v14305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:30.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: pgmap v14306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:30.137 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:39:30.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: pgmap v14305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:30.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: pgmap v14306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:30.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: pgmap v14305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: pgmap v14306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:30.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:39:32.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:32 smithi153 ceph-mon[36569]: pgmap v14307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:32.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:32 smithi104 ceph-mon[36624]: pgmap v14307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:32.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:32 smithi081 ceph-mon[31453]: pgmap v14307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:34.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:34 smithi153 ceph-mon[36569]: pgmap v14308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:34.315 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:34 smithi104 ceph-mon[36624]: pgmap v14308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:34.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:34 smithi081 ceph-mon[31453]: pgmap v14308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:36.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:36 smithi153 ceph-mon[36569]: pgmap v14309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:36.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:36 smithi104 ceph-mon[36624]: pgmap v14309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:36.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:36 smithi081 ceph-mon[31453]: pgmap v14309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:38.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:38 smithi153 ceph-mon[36569]: pgmap v14310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:38.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:38 smithi104 ceph-mon[36624]: pgmap v14310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:38.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:38 smithi081 ceph-mon[31453]: pgmap v14310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:40.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:40 smithi153 ceph-mon[36569]: pgmap v14311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:40.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:40 smithi104 ceph-mon[36624]: pgmap v14311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:40.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:40 smithi081 ceph-mon[31453]: pgmap v14311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:42.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:42 smithi153 ceph-mon[36569]: pgmap v14312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:42.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:42 smithi104 ceph-mon[36624]: pgmap v14312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:42.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:42 smithi081 ceph-mon[31453]: pgmap v14312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:42.976 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:42.997+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:39:42.976 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:42.997+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:39:42.976 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:42.997+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:39:44.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:44 smithi153 ceph-mon[36569]: pgmap v14313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:44.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:44 smithi104 ceph-mon[36624]: pgmap v14313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:44.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:44 smithi081 ceph-mon[31453]: pgmap v14313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:46.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:46 smithi153 ceph-mon[36569]: pgmap v14314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:46.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:46 smithi104 ceph-mon[36624]: pgmap v14314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:46.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:46 smithi081 ceph-mon[31453]: pgmap v14314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:48.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:48 smithi153 ceph-mon[36569]: pgmap v14315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:48.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:48 smithi104 ceph-mon[36624]: pgmap v14315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:48.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:48 smithi081 ceph-mon[31453]: pgmap v14315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:50.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:50 smithi153 ceph-mon[36569]: pgmap v14316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:50.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:50 smithi104 ceph-mon[36624]: pgmap v14316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:50.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:50 smithi081 ceph-mon[31453]: pgmap v14316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:52.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:52 smithi104 ceph-mon[36624]: pgmap v14317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:52.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:52 smithi081 ceph-mon[31453]: pgmap v14317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:52.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:52 smithi153 ceph-mon[36569]: pgmap v14317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:53.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:53 smithi081 ceph-mon[31453]: pgmap v14318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:53.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:53 smithi153 ceph-mon[36569]: pgmap v14318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:53.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:53 smithi104 ceph-mon[36624]: pgmap v14318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:56.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:56 smithi153 ceph-mon[36569]: pgmap v14319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:56.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:56 smithi104 ceph-mon[36624]: pgmap v14319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:56.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:56 smithi081 ceph-mon[31453]: pgmap v14319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:58.000+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:39:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:58.000+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:39:57.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:39:58.000+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:39:58.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:39:58 smithi153 ceph-mon[36569]: pgmap v14320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:58.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:39:58 smithi104 ceph-mon[36624]: pgmap v14320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:39:58.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:39:58 smithi081 ceph-mon[31453]: pgmap v14320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:00.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:00 smithi153 ceph-mon[36569]: pgmap v14321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:00.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:40:00.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:00 smithi104 ceph-mon[36624]: pgmap v14321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:00.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:40:00.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:00 smithi081 ceph-mon[31453]: pgmap v14321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:00.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:40:02.285 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:02 smithi153 ceph-mon[36569]: pgmap v14322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:02.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:02 smithi104 ceph-mon[36624]: pgmap v14322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:02.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:02 smithi081 ceph-mon[31453]: pgmap v14322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:04.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:04 smithi153 ceph-mon[36569]: pgmap v14323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:04.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:04 smithi104 ceph-mon[36624]: pgmap v14323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:04.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:04 smithi081 ceph-mon[31453]: pgmap v14323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:06.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:06 smithi153 ceph-mon[36569]: pgmap v14324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:06.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:06 smithi104 ceph-mon[36624]: pgmap v14324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:06.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:06 smithi081 ceph-mon[31453]: pgmap v14324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:08.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:08 smithi153 ceph-mon[36569]: pgmap v14325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:08.285 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:08 smithi081 ceph-mon[31453]: pgmap v14325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:08.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:08 smithi104 ceph-mon[36624]: pgmap v14325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:10.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:10 smithi104 ceph-mon[36624]: pgmap v14326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:10.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:10 smithi081 ceph-mon[31453]: pgmap v14326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:10.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:10 smithi153 ceph-mon[36569]: pgmap v14326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:12.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:12 smithi104 ceph-mon[36624]: pgmap v14327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:12.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:12 smithi081 ceph-mon[31453]: pgmap v14327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:12.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:12 smithi153 ceph-mon[36569]: pgmap v14327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:13.010+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:40:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:13.010+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:40:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:13.010+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:40:13.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:13 smithi081 ceph-mon[31453]: pgmap v14328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:13.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:13 smithi153 ceph-mon[36569]: pgmap v14328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:13.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:13 smithi104 ceph-mon[36624]: pgmap v14328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:16.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:16 smithi153 ceph-mon[36569]: pgmap v14329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:16.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:16 smithi104 ceph-mon[36624]: pgmap v14329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:16.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:16 smithi081 ceph-mon[31453]: pgmap v14329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:17.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:17.106+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:40:17.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:17.106+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:40:17.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:17.107+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 206 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:40:17.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:17.107+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 206 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:40:18.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:18 smithi153 ceph-mon[36569]: pgmap v14330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:18.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:18 smithi104 ceph-mon[36624]: pgmap v14330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:18.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:18 smithi081 ceph-mon[31453]: pgmap v14330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:20.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:20 smithi153 ceph-mon[36569]: pgmap v14331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:20.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:20 smithi104 ceph-mon[36624]: pgmap v14331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:20.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:20 smithi081 ceph-mon[31453]: pgmap v14331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:22.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:22 smithi081 ceph-mon[31453]: pgmap v14332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:22.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:22 smithi153 ceph-mon[36569]: pgmap v14332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:22.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:22 smithi104 ceph-mon[36624]: pgmap v14332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:24.284 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:24 smithi153 ceph-mon[36569]: pgmap v14333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:24.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:24 smithi104 ceph-mon[36624]: pgmap v14333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:24.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:24 smithi081 ceph-mon[31453]: pgmap v14333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:26.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:26 smithi104 ceph-mon[36624]: pgmap v14334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:26.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:26 smithi081 ceph-mon[31453]: pgmap v14334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:26.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:26 smithi153 ceph-mon[36569]: pgmap v14334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:28.019+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:40:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:28.019+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:40:27.998 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:28.019+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:40:28.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:28 smithi104 ceph-mon[36624]: pgmap v14335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:28.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:28 smithi081 ceph-mon[31453]: pgmap v14335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:28.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:28 smithi153 ceph-mon[36569]: pgmap v14335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:30.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:30 smithi104 ceph-mon[36624]: pgmap v14336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:30.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:40:30.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:30 smithi081 ceph-mon[31453]: pgmap v14336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:30.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:40:30.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:30 smithi153 ceph-mon[36569]: pgmap v14336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:30.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:40:31.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:40:31.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:40:31.310 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:40:31.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:40:31.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:40:31.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:40:31.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:40:31.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:40:31.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:40:32.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:32 smithi081 ceph-mon[31453]: pgmap v14337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:32.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:32 smithi081 ceph-mon[31453]: pgmap v14338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:32.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:32 smithi153 ceph-mon[36569]: pgmap v14337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:32.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:32 smithi153 ceph-mon[36569]: pgmap v14338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:32.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:32 smithi104 ceph-mon[36624]: pgmap v14337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:32.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:32 smithi104 ceph-mon[36624]: pgmap v14338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:33.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:33 smithi081 ceph-mon[31453]: pgmap v14339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:33.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:33 smithi153 ceph-mon[36569]: pgmap v14339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:33.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:33 smithi104 ceph-mon[36624]: pgmap v14339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:35.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:35 smithi153 ceph-mon[36569]: pgmap v14340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:35.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:35 smithi104 ceph-mon[36624]: pgmap v14340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:35.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:35 smithi081 ceph-mon[31453]: pgmap v14340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:37.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:37 smithi153 ceph-mon[36569]: pgmap v14341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:37.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:37 smithi104 ceph-mon[36624]: pgmap v14341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:37.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:37 smithi081 ceph-mon[31453]: pgmap v14341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:39.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:39 smithi153 ceph-mon[36569]: pgmap v14342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:39.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:39 smithi104 ceph-mon[36624]: pgmap v14342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:39.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:39 smithi081 ceph-mon[31453]: pgmap v14342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:41.784 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:41 smithi153 ceph-mon[36569]: pgmap v14343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:41.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:41 smithi104 ceph-mon[36624]: pgmap v14343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:41.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:41 smithi081 ceph-mon[31453]: pgmap v14343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:43.021+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:40:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:43.021+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:40:42.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:43.021+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:40:43.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:43 smithi104 ceph-mon[36624]: pgmap v14344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:43.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:43 smithi081 ceph-mon[31453]: pgmap v14344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:44.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:43 smithi153 ceph-mon[36569]: pgmap v14344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:45.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:45 smithi104 ceph-mon[36624]: pgmap v14345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:45.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:45 smithi081 ceph-mon[31453]: pgmap v14345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:46.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:45 smithi153 ceph-mon[36569]: pgmap v14345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:47.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:47 smithi104 ceph-mon[36624]: pgmap v14346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:47.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:47 smithi081 ceph-mon[31453]: pgmap v14346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:48.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:47 smithi153 ceph-mon[36569]: pgmap v14346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:49.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:49 smithi104 ceph-mon[36624]: pgmap v14347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:49.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:49 smithi081 ceph-mon[31453]: pgmap v14347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:50.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:49 smithi153 ceph-mon[36569]: pgmap v14347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:51.809 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:51 smithi104 ceph-mon[36624]: pgmap v14348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:51.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:51 smithi081 ceph-mon[31453]: pgmap v14348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:52.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:51 smithi153 ceph-mon[36569]: pgmap v14348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:53.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:53 smithi081 ceph-mon[31453]: pgmap v14349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:54.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:53 smithi153 ceph-mon[36569]: pgmap v14349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:54.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:53 smithi104 ceph-mon[36624]: pgmap v14349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:55.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:55 smithi081 ceph-mon[31453]: pgmap v14350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:56.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:55 smithi153 ceph-mon[36569]: pgmap v14350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:56.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:55 smithi104 ceph-mon[36624]: pgmap v14350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:57.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:57 smithi081 ceph-mon[31453]: pgmap v14351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:58.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:58.023+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:40:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:58.023+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:40:58.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:40:58.023+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:40:58.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:57 smithi153 ceph-mon[36569]: pgmap v14351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:58.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:57 smithi104 ceph-mon[36624]: pgmap v14351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:40:59.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:40:59 smithi081 ceph-mon[31453]: pgmap v14352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:00.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:40:59 smithi153 ceph-mon[36569]: pgmap v14352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:00.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:40:59 smithi104 ceph-mon[36624]: pgmap v14352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:01.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:01 smithi081 ceph-mon[31453]: pgmap v14353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:02.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:01 smithi153 ceph-mon[36569]: pgmap v14353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:02.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:01 smithi104 ceph-mon[36624]: pgmap v14353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:03.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:03 smithi081 ceph-mon[31453]: pgmap v14354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:04.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:03 smithi153 ceph-mon[36569]: pgmap v14354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:04.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:03 smithi104 ceph-mon[36624]: pgmap v14354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:05.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:05 smithi081 ceph-mon[31453]: pgmap v14355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:06.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:05 smithi153 ceph-mon[36569]: pgmap v14355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:06.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:05 smithi104 ceph-mon[36624]: pgmap v14355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:07.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:07 smithi081 ceph-mon[31453]: pgmap v14356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:08.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:07 smithi153 ceph-mon[36569]: pgmap v14356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:08.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:07 smithi104 ceph-mon[36624]: pgmap v14356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:09.861 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:09 smithi104 ceph-mon[36624]: pgmap v14357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:09.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:09 smithi081 ceph-mon[31453]: pgmap v14357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:10.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:09 smithi153 ceph-mon[36569]: pgmap v14357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:11.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:11 smithi081 ceph-mon[31453]: pgmap v14358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:12.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:11 smithi153 ceph-mon[36569]: pgmap v14358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:12.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:11 smithi104 ceph-mon[36624]: pgmap v14358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:13.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:13.024+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:41:13.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:13.024+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:41:13.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:13.024+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:41:13.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:13 smithi081 ceph-mon[31453]: pgmap v14359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:14.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:13 smithi153 ceph-mon[36569]: pgmap v14359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:14.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:13 smithi104 ceph-mon[36624]: pgmap v14359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:15.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:15 smithi081 ceph-mon[31453]: pgmap v14360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:16.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:15 smithi153 ceph-mon[36569]: pgmap v14360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:16.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:15 smithi104 ceph-mon[36624]: pgmap v14360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:17.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:17 smithi081 ceph-mon[31453]: pgmap v14361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:18.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:17 smithi153 ceph-mon[36569]: pgmap v14361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:18.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:17 smithi104 ceph-mon[36624]: pgmap v14361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:19.883 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:19 smithi081 ceph-mon[31453]: pgmap v14362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:20.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:19 smithi153 ceph-mon[36569]: pgmap v14362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:20.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:19 smithi104 ceph-mon[36624]: pgmap v14362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:22.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:21 smithi153 ceph-mon[36569]: pgmap v14363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:22.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:21 smithi104 ceph-mon[36624]: pgmap v14363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:22.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:21 smithi081 ceph-mon[31453]: pgmap v14363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:24.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:23 smithi153 ceph-mon[36569]: pgmap v14364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:24.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:23 smithi104 ceph-mon[36624]: pgmap v14364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:24.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:23 smithi081 ceph-mon[31453]: pgmap v14364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:26.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:25 smithi153 ceph-mon[36569]: pgmap v14365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:26.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:25 smithi104 ceph-mon[36624]: pgmap v14365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:26.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:25 smithi081 ceph-mon[31453]: pgmap v14365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:28.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:28.035+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:41:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:28.035+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:41:28.013 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:28.035+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:41:28.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:27 smithi153 ceph-mon[36569]: pgmap v14366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:28.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:27 smithi104 ceph-mon[36624]: pgmap v14366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:28.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:27 smithi081 ceph-mon[31453]: pgmap v14366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:30.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:29 smithi153 ceph-mon[36569]: pgmap v14367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:30.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:29 smithi104 ceph-mon[36624]: pgmap v14367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:30.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:29 smithi081 ceph-mon[31453]: pgmap v14367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:32.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:31 smithi153 ceph-mon[36569]: pgmap v14368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:32.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:41:32.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:31 smithi104 ceph-mon[36624]: pgmap v14368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:32.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:41:32.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:31 smithi081 ceph-mon[31453]: pgmap v14368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:32.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:41:33.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:41:33.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:41:33.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:32 smithi153 ceph-mon[36569]: pgmap v14369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:33.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:32 smithi153 ceph-mon[36569]: pgmap v14370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:33.034 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:41:33.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:41:33.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:41:33.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:32 smithi104 ceph-mon[36624]: pgmap v14369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:33.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:32 smithi104 ceph-mon[36624]: pgmap v14370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:33.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:41:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:41:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:41:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:32 smithi081 ceph-mon[31453]: pgmap v14369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:32 smithi081 ceph-mon[31453]: pgmap v14370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:33.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:41:35.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:35 smithi153 ceph-mon[36569]: pgmap v14371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:35.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:35 smithi104 ceph-mon[36624]: pgmap v14371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:35.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:35 smithi081 ceph-mon[31453]: pgmap v14371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:37.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:37 smithi104 ceph-mon[36624]: pgmap v14372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:37.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:37 smithi081 ceph-mon[31453]: pgmap v14372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:37.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:37 smithi153 ceph-mon[36569]: pgmap v14372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:39.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:39 smithi104 ceph-mon[36624]: pgmap v14373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:39.335 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:39 smithi153 ceph-mon[36569]: pgmap v14373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:39.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:39 smithi081 ceph-mon[31453]: pgmap v14373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:41.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:41 smithi104 ceph-mon[36624]: pgmap v14374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:41.368 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:41 smithi081 ceph-mon[31453]: pgmap v14374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:41.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:41 smithi153 ceph-mon[36569]: pgmap v14374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:43.051+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:41:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:43.051+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:41:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:43.051+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:41:43.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:43 smithi104 ceph-mon[36624]: pgmap v14375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:43.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:43 smithi081 ceph-mon[31453]: pgmap v14375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:43.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:43 smithi153 ceph-mon[36569]: pgmap v14375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:45.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:45 smithi104 ceph-mon[36624]: pgmap v14376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:45.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:45 smithi081 ceph-mon[31453]: pgmap v14376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:45.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:45 smithi153 ceph-mon[36569]: pgmap v14376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:47.121+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:41:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:47.121+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:41:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:47.122+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 207 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:41:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:47.122+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 207 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:41:47.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:47 smithi081 ceph-mon[31453]: pgmap v14377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:47.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:47 smithi153 ceph-mon[36569]: pgmap v14377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:47.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:47 smithi104 ceph-mon[36624]: pgmap v14377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:48.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:48 smithi081 ceph-mon[31453]: pgmap v14378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:48.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:48 smithi153 ceph-mon[36569]: pgmap v14378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:48.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:48 smithi104 ceph-mon[36624]: pgmap v14378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:51.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:51 smithi081 ceph-mon[31453]: pgmap v14379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:51.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:51 smithi104 ceph-mon[36624]: pgmap v14379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:51.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:51 smithi153 ceph-mon[36569]: pgmap v14379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:53.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:53 smithi104 ceph-mon[36624]: pgmap v14380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:53.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:53 smithi081 ceph-mon[31453]: pgmap v14380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:53.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:53 smithi153 ceph-mon[36569]: pgmap v14380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:55.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:55 smithi104 ceph-mon[36624]: pgmap v14381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:55.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:55 smithi081 ceph-mon[31453]: pgmap v14381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:55.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:55 smithi153 ceph-mon[36569]: pgmap v14381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:57.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:57 smithi104 ceph-mon[36624]: pgmap v14382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:57.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:57 smithi081 ceph-mon[31453]: pgmap v14382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:57.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:57 smithi153 ceph-mon[36569]: pgmap v14382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:58.033 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:58.053+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:41:58.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:58.054+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:41:58.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:41:58.054+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:41:59.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:41:59 smithi081 ceph-mon[31453]: pgmap v14383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:59.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:41:59 smithi153 ceph-mon[36569]: pgmap v14383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:41:59.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:41:59 smithi104 ceph-mon[36624]: pgmap v14383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:01.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:01 smithi081 ceph-mon[31453]: pgmap v14384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:01.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:01 smithi153 ceph-mon[36569]: pgmap v14384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:01.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:01 smithi104 ceph-mon[36624]: pgmap v14384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:02.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:02 smithi081 ceph-mon[31453]: pgmap v14385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:02.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:02 smithi153 ceph-mon[36569]: pgmap v14385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:02.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:02 smithi104 ceph-mon[36624]: pgmap v14385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:05.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:05 smithi104 ceph-mon[36624]: pgmap v14386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:05.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:05 smithi081 ceph-mon[31453]: pgmap v14386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:05.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:05 smithi153 ceph-mon[36569]: pgmap v14386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:07.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:07 smithi104 ceph-mon[36624]: pgmap v14387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:07.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:07 smithi081 ceph-mon[31453]: pgmap v14387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:07.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:07 smithi153 ceph-mon[36569]: pgmap v14387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:09.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:09 smithi104 ceph-mon[36624]: pgmap v14388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:09.336 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:09 smithi153 ceph-mon[36569]: pgmap v14388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:09.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:09 smithi081 ceph-mon[31453]: pgmap v14388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:11.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:11 smithi081 ceph-mon[31453]: pgmap v14389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:11.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:11 smithi153 ceph-mon[36569]: pgmap v14389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:11.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:11 smithi104 ceph-mon[36624]: pgmap v14389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:13.055+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:42:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:13.055+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:42:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:13.055+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:42:13.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:13 smithi081 ceph-mon[31453]: pgmap v14390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:13.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:13 smithi153 ceph-mon[36569]: pgmap v14390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:13.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:13 smithi104 ceph-mon[36624]: pgmap v14390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:14.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:14 smithi081 ceph-mon[31453]: pgmap v14391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:14.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:14 smithi153 ceph-mon[36569]: pgmap v14391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:14.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:14 smithi104 ceph-mon[36624]: pgmap v14391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:17.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:17 smithi104 ceph-mon[36624]: pgmap v14392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:17.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:17 smithi081 ceph-mon[31453]: pgmap v14392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:17.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:17 smithi153 ceph-mon[36569]: pgmap v14392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:19.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:19 smithi081 ceph-mon[31453]: pgmap v14393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:19.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:19 smithi153 ceph-mon[36569]: pgmap v14393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:19.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:19 smithi104 ceph-mon[36624]: pgmap v14393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:21.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:21 smithi081 ceph-mon[31453]: pgmap v14394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:21.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:21 smithi153 ceph-mon[36569]: pgmap v14394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:21.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:21 smithi104 ceph-mon[36624]: pgmap v14394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:23.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:23 smithi104 ceph-mon[36624]: pgmap v14395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:23.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:23 smithi081 ceph-mon[31453]: pgmap v14395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:23.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:23 smithi153 ceph-mon[36569]: pgmap v14395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:24.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:24 smithi081 ceph-mon[31453]: pgmap v14396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:24.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:24 smithi153 ceph-mon[36569]: pgmap v14396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:24.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:24 smithi104 ceph-mon[36624]: pgmap v14396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:27.312 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:27 smithi081 ceph-mon[31453]: pgmap v14397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:27.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:27 smithi153 ceph-mon[36569]: pgmap v14397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:27.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:27 smithi104 ceph-mon[36624]: pgmap v14397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:28.071+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:42:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:28.071+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:42:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:28.071+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:42:29.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:29 smithi081 ceph-mon[31453]: pgmap v14398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:29.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:29 smithi153 ceph-mon[36569]: pgmap v14398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:29.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:29 smithi104 ceph-mon[36624]: pgmap v14398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:31.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:31 smithi081 ceph-mon[31453]: pgmap v14399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:31.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:31 smithi153 ceph-mon[36569]: pgmap v14399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:31.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:31 smithi104 ceph-mon[36624]: pgmap v14399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:32.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:32 smithi081 ceph-mon[31453]: pgmap v14400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:32.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:32 smithi153 ceph-mon[36569]: pgmap v14400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:32.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:32 smithi104 ceph-mon[36624]: pgmap v14400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:33.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:42:33.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:42:33.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:42:34.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:42:34.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:42:34.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:34 smithi153 ceph-mon[36569]: pgmap v14401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:34.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:34 smithi153 ceph-mon[36569]: pgmap v14402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:34.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:42:34.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:42:34.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:42:34.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:34 smithi104 ceph-mon[36624]: pgmap v14401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:34.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:34 smithi104 ceph-mon[36624]: pgmap v14402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:34.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:42:34.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:42:34.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:42:34.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:34 smithi081 ceph-mon[31453]: pgmap v14401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:34.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:34 smithi081 ceph-mon[31453]: pgmap v14402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:34.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:42:37.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:36 smithi153 ceph-mon[36569]: pgmap v14403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:37.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:36 smithi104 ceph-mon[36624]: pgmap v14403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:37.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:36 smithi081 ceph-mon[31453]: pgmap v14403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:39.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:38 smithi153 ceph-mon[36569]: pgmap v14404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:39.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:38 smithi104 ceph-mon[36624]: pgmap v14404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:39.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:38 smithi081 ceph-mon[31453]: pgmap v14404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:40.935 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:40 smithi081 ceph-mon[31453]: pgmap v14405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:41.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:40 smithi153 ceph-mon[36569]: pgmap v14405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:41.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:40 smithi104 ceph-mon[36624]: pgmap v14405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:43.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:42 smithi153 ceph-mon[36569]: pgmap v14406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:43.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:42 smithi104 ceph-mon[36624]: pgmap v14406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:43.086+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:42:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:43.087+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:42:43.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:43.087+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:42:43.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:42 smithi081 ceph-mon[31453]: pgmap v14406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:45.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:44 smithi153 ceph-mon[36569]: pgmap v14407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:45.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:44 smithi104 ceph-mon[36624]: pgmap v14407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:45.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:44 smithi081 ceph-mon[31453]: pgmap v14407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:47.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:46 smithi153 ceph-mon[36569]: pgmap v14408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:47.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:46 smithi104 ceph-mon[36624]: pgmap v14408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:47.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:46 smithi081 ceph-mon[31453]: pgmap v14408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:49.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:48 smithi153 ceph-mon[36569]: pgmap v14409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:49.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:48 smithi104 ceph-mon[36624]: pgmap v14409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:49.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:48 smithi081 ceph-mon[31453]: pgmap v14409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:51.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:50 smithi153 ceph-mon[36569]: pgmap v14410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:51.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:50 smithi104 ceph-mon[36624]: pgmap v14410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:51.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:50 smithi081 ceph-mon[31453]: pgmap v14410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:53.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:52 smithi153 ceph-mon[36569]: pgmap v14411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:53.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:52 smithi104 ceph-mon[36624]: pgmap v14411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:53.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:52 smithi081 ceph-mon[31453]: pgmap v14411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:55.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:54 smithi153 ceph-mon[36569]: pgmap v14412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:55.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:54 smithi104 ceph-mon[36624]: pgmap v14412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:55.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:54 smithi081 ceph-mon[31453]: pgmap v14412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:57.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:56 smithi153 ceph-mon[36569]: pgmap v14413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:57.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:56 smithi104 ceph-mon[36624]: pgmap v14413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:57.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:56 smithi081 ceph-mon[31453]: pgmap v14413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:58.102+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:42:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:58.102+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:42:58.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:42:58.102+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:42:59.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:42:58 smithi153 ceph-mon[36569]: pgmap v14414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:59.056 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:42:58 smithi081 ceph-mon[31453]: pgmap v14414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:42:59.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:42:58 smithi104 ceph-mon[36624]: pgmap v14414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:01.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:00 smithi153 ceph-mon[36569]: pgmap v14415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:01.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:00 smithi104 ceph-mon[36624]: pgmap v14415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:01.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:00 smithi081 ceph-mon[31453]: pgmap v14415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:03.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:02 smithi153 ceph-mon[36569]: pgmap v14416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:03.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:02 smithi104 ceph-mon[36624]: pgmap v14416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:03.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:02 smithi081 ceph-mon[31453]: pgmap v14416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:05.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:04 smithi153 ceph-mon[36569]: pgmap v14417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:05.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:04 smithi104 ceph-mon[36624]: pgmap v14417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:05.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:04 smithi081 ceph-mon[31453]: pgmap v14417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:07.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:06 smithi153 ceph-mon[36569]: pgmap v14418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:07.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:06 smithi104 ceph-mon[36624]: pgmap v14418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:07.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:06 smithi081 ceph-mon[31453]: pgmap v14418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:09.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:08 smithi153 ceph-mon[36569]: pgmap v14419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:09.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:08 smithi104 ceph-mon[36624]: pgmap v14419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:09.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:08 smithi081 ceph-mon[31453]: pgmap v14419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:11.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:10 smithi153 ceph-mon[36569]: pgmap v14420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:11.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:10 smithi104 ceph-mon[36624]: pgmap v14420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:11.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:10 smithi081 ceph-mon[31453]: pgmap v14420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:13.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:12 smithi153 ceph-mon[36569]: pgmap v14421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:13.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:12 smithi104 ceph-mon[36624]: pgmap v14421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:13.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:13.118+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:43:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:13.118+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:43:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:13.118+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:43:13.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:12 smithi081 ceph-mon[31453]: pgmap v14421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:15.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:14 smithi153 ceph-mon[36569]: pgmap v14422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:15.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:14 smithi104 ceph-mon[36624]: pgmap v14422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:15.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:14 smithi081 ceph-mon[31453]: pgmap v14422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:17.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:16 smithi153 ceph-mon[36569]: pgmap v14423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:17.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:16 smithi104 ceph-mon[36624]: pgmap v14423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:17.137+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:43:17.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:17.137+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:43:17.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:17.137+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 208 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:43:17.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:17.137+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 208 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:43:17.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:16 smithi081 ceph-mon[31453]: pgmap v14423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:19.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:18 smithi153 ceph-mon[36569]: pgmap v14424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:19.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:18 smithi104 ceph-mon[36624]: pgmap v14424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:19.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:18 smithi081 ceph-mon[31453]: pgmap v14424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:21.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:20 smithi153 ceph-mon[36569]: pgmap v14425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:21.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:20 smithi104 ceph-mon[36624]: pgmap v14425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:21.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:20 smithi081 ceph-mon[31453]: pgmap v14425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:23.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:22 smithi153 ceph-mon[36569]: pgmap v14426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:23.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:22 smithi104 ceph-mon[36624]: pgmap v14426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:23.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:22 smithi081 ceph-mon[31453]: pgmap v14426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:25.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:24 smithi153 ceph-mon[36569]: pgmap v14427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:25.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:24 smithi104 ceph-mon[36624]: pgmap v14427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:25.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:24 smithi081 ceph-mon[31453]: pgmap v14427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:27.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:26 smithi104 ceph-mon[36624]: pgmap v14428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:27.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:26 smithi081 ceph-mon[31453]: pgmap v14428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:27.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:26 smithi153 ceph-mon[36569]: pgmap v14428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:28.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:28.133+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:43:28.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:28.133+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:43:28.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:28.133+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:43:29.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:28 smithi104 ceph-mon[36624]: pgmap v14429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:29.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:28 smithi081 ceph-mon[31453]: pgmap v14429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:29.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:28 smithi153 ceph-mon[36569]: pgmap v14429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:31.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:30 smithi104 ceph-mon[36624]: pgmap v14430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:31.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:30 smithi081 ceph-mon[31453]: pgmap v14430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:31.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:30 smithi153 ceph-mon[36569]: pgmap v14430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:33.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:32 smithi104 ceph-mon[36624]: pgmap v14431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:33.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:32 smithi081 ceph-mon[31453]: pgmap v14431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:33.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:32 smithi153 ceph-mon[36569]: pgmap v14431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:35.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:34 smithi153 ceph-mon[36569]: pgmap v14432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:35.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:43:35.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.033 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:34 smithi104 ceph-mon[36624]: pgmap v14432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:35.058 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:43:35.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.059 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:34 smithi081 ceph-mon[31453]: pgmap v14432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:35.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:43:35.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:35.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:36 smithi081 ceph-mon[31453]: pgmap v14433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:37.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:36 smithi153 ceph-mon[36569]: pgmap v14433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:37.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:36 smithi104 ceph-mon[36624]: pgmap v14433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:37.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:37.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:39.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:38 smithi081 ceph-mon[31453]: pgmap v14434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:39.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:38 smithi153 ceph-mon[36569]: pgmap v14434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:39.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:38 smithi104 ceph-mon[36624]: pgmap v14434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:40.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:40.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:40.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:40.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:40.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:40.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:41.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:41 smithi081 ceph-mon[31453]: pgmap v14435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:41.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:43:41.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:43:41.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:41.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:41 smithi153 ceph-mon[36569]: pgmap v14435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:41.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:43:41.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:43:41.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:41.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:41 smithi104 ceph-mon[36624]: pgmap v14435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:41.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:43:41.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:43:41.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:43:42.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:42 smithi081 ceph-mon[31453]: pgmap v14436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:42.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:42 smithi081 ceph-mon[31453]: pgmap v14437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:42.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:42 smithi153 ceph-mon[36569]: pgmap v14436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:42.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:42 smithi153 ceph-mon[36569]: pgmap v14437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:42.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:42 smithi104 ceph-mon[36624]: pgmap v14436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:42.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:42 smithi104 ceph-mon[36624]: pgmap v14437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:43.146+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:43:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:43.146+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:43:43.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:43.146+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:43:43.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:43 smithi153 ceph-mon[36569]: pgmap v14438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:43.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:43 smithi104 ceph-mon[36624]: pgmap v14438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:43.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:43 smithi081 ceph-mon[31453]: pgmap v14438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:45.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:45 smithi081 ceph-mon[31453]: pgmap v14439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:45.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:45 smithi153 ceph-mon[36569]: pgmap v14439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:45.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:45 smithi104 ceph-mon[36624]: pgmap v14439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:47.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:47 smithi081 ceph-mon[31453]: pgmap v14440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:47.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:47 smithi153 ceph-mon[36569]: pgmap v14440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:47.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:47 smithi104 ceph-mon[36624]: pgmap v14440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:49.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:49 smithi081 ceph-mon[31453]: pgmap v14441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:49.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:49 smithi153 ceph-mon[36569]: pgmap v14441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:49.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:49 smithi104 ceph-mon[36624]: pgmap v14441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:51.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:51 smithi081 ceph-mon[31453]: pgmap v14442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:51.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:51 smithi153 ceph-mon[36569]: pgmap v14442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:51.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:51 smithi104 ceph-mon[36624]: pgmap v14442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:53.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:53 smithi153 ceph-mon[36569]: pgmap v14443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:53.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:53 smithi104 ceph-mon[36624]: pgmap v14443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:53.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:53 smithi081 ceph-mon[31453]: pgmap v14443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:55.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:55 smithi153 ceph-mon[36569]: pgmap v14444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:55.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:55 smithi104 ceph-mon[36624]: pgmap v14444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:55.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:55 smithi081 ceph-mon[31453]: pgmap v14444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:57.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:57 smithi153 ceph-mon[36569]: pgmap v14445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:57.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:57 smithi104 ceph-mon[36624]: pgmap v14445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:57.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:57 smithi081 ceph-mon[31453]: pgmap v14445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:58.148+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:43:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:58.149+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:43:58.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:43:58.149+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:43:59.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:43:59 smithi153 ceph-mon[36569]: pgmap v14446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:59.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:43:59 smithi104 ceph-mon[36624]: pgmap v14446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:43:59.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:43:59 smithi081 ceph-mon[31453]: pgmap v14446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:01.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:01 smithi153 ceph-mon[36569]: pgmap v14447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:01.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:01 smithi104 ceph-mon[36624]: pgmap v14447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:01.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:01 smithi081 ceph-mon[31453]: pgmap v14447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:03.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:03 smithi153 ceph-mon[36569]: pgmap v14448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:03.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:03 smithi104 ceph-mon[36624]: pgmap v14448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:03.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:03 smithi081 ceph-mon[31453]: pgmap v14448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:05.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:05 smithi153 ceph-mon[36569]: pgmap v14449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:05.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:05 smithi104 ceph-mon[36624]: pgmap v14449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:05.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:05 smithi081 ceph-mon[31453]: pgmap v14449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:07.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:07 smithi153 ceph-mon[36569]: pgmap v14450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:07.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:07 smithi104 ceph-mon[36624]: pgmap v14450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:07.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:07 smithi081 ceph-mon[31453]: pgmap v14450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:09.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:09 smithi153 ceph-mon[36569]: pgmap v14451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:09.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:09 smithi104 ceph-mon[36624]: pgmap v14451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:09.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:09 smithi081 ceph-mon[31453]: pgmap v14451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:11.685 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:11 smithi081 ceph-mon[31453]: pgmap v14452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:11.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:11 smithi153 ceph-mon[36569]: pgmap v14452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:11.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:11 smithi104 ceph-mon[36624]: pgmap v14452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:13.159+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:44:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:13.160+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:44:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:13.160+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:44:13.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:13 smithi153 ceph-mon[36569]: pgmap v14453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:13.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:13 smithi104 ceph-mon[36624]: pgmap v14453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:13.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:13 smithi081 ceph-mon[31453]: pgmap v14453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:15.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:15 smithi153 ceph-mon[36569]: pgmap v14454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:15.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:15 smithi104 ceph-mon[36624]: pgmap v14454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:15.855 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:15 smithi081 ceph-mon[31453]: pgmap v14454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:17.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:17 smithi153 ceph-mon[36569]: pgmap v14455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:17.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:17 smithi104 ceph-mon[36624]: pgmap v14455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:17.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:17 smithi081 ceph-mon[31453]: pgmap v14455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:19.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:19 smithi153 ceph-mon[36569]: pgmap v14456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:19.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:19 smithi104 ceph-mon[36624]: pgmap v14456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:19.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:19 smithi081 ceph-mon[31453]: pgmap v14456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:21.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:21 smithi153 ceph-mon[36569]: pgmap v14457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:21.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:21 smithi104 ceph-mon[36624]: pgmap v14457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:21.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:21 smithi081 ceph-mon[31453]: pgmap v14457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:23.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:23 smithi153 ceph-mon[36569]: pgmap v14458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:23.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:23 smithi104 ceph-mon[36624]: pgmap v14458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:23.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:23 smithi081 ceph-mon[31453]: pgmap v14458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:25.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:25 smithi153 ceph-mon[36569]: pgmap v14459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:25 smithi104 ceph-mon[36624]: pgmap v14459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:25.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:25 smithi081 ceph-mon[31453]: pgmap v14459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:27.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:27 smithi153 ceph-mon[36569]: pgmap v14460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:27.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:27 smithi104 ceph-mon[36624]: pgmap v14460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:27.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:27 smithi081 ceph-mon[31453]: pgmap v14460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:28.171+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:44:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:28.171+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:44:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:28.171+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:44:29.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:29 smithi153 ceph-mon[36569]: pgmap v14461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:29.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:29 smithi104 ceph-mon[36624]: pgmap v14461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:29.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:29 smithi081 ceph-mon[31453]: pgmap v14461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:31.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:31 smithi153 ceph-mon[36569]: pgmap v14462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:31.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:31 smithi104 ceph-mon[36624]: pgmap v14462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:31.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:31 smithi081 ceph-mon[31453]: pgmap v14462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:33.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:33 smithi153 ceph-mon[36569]: pgmap v14463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:33.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:33 smithi104 ceph-mon[36624]: pgmap v14463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:33.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:33 smithi081 ceph-mon[31453]: pgmap v14463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:35.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:35 smithi153 ceph-mon[36569]: pgmap v14464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:35.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:35 smithi104 ceph-mon[36624]: pgmap v14464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:35.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:35 smithi081 ceph-mon[31453]: pgmap v14464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:37.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:37 smithi153 ceph-mon[36569]: pgmap v14465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:37.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:37 smithi104 ceph-mon[36624]: pgmap v14465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:37.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:37 smithi081 ceph-mon[31453]: pgmap v14465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:39.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:39 smithi153 ceph-mon[36569]: pgmap v14466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:39.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:39 smithi104 ceph-mon[36624]: pgmap v14466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:39.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:39 smithi081 ceph-mon[31453]: pgmap v14466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:41.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:41 smithi153 ceph-mon[36569]: pgmap v14467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:41.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:44:41.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:41.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:41.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:41 smithi104 ceph-mon[36624]: pgmap v14467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:41.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:44:41.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:41.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:41.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:41 smithi081 ceph-mon[31453]: pgmap v14467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:41.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:44:41.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:41.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:43.187+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:44:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:43.187+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:44:43.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:43.187+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:44:43.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:43 smithi153 ceph-mon[36569]: pgmap v14468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:43.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:43 smithi104 ceph-mon[36624]: pgmap v14468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:43.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:43 smithi081 ceph-mon[31453]: pgmap v14468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:45.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:45 smithi153 ceph-mon[36569]: pgmap v14469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:45.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:45.783 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:45.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:45 smithi104 ceph-mon[36624]: pgmap v14469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:45.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:45.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:45.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:45 smithi081 ceph-mon[31453]: pgmap v14469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:45.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:45.882 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:47.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:47 smithi081 ceph-mon[31453]: pgmap v14470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:47.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:47 smithi153 ceph-mon[36569]: pgmap v14470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:47.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:47 smithi104 ceph-mon[36624]: pgmap v14470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:49.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:49.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:49.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:44:49.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:44:49.132 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:48 smithi081 ceph-mon[31453]: pgmap v14471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:49.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:48 smithi081 ceph-mon[31453]: pgmap v14472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:49.133 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:49.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:44:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:44:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:48 smithi153 ceph-mon[36569]: pgmap v14471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:48 smithi153 ceph-mon[36569]: pgmap v14472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:49.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:44:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:44:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:48 smithi104 ceph-mon[36624]: pgmap v14471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:48 smithi104 ceph-mon[36624]: pgmap v14472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:44:51.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:50 smithi153 ceph-mon[36569]: pgmap v14473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:51.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:50 smithi104 ceph-mon[36624]: pgmap v14473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:51.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:50 smithi081 ceph-mon[31453]: pgmap v14473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:53.032 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:52 smithi153 ceph-mon[36569]: pgmap v14474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:53.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:52 smithi104 ceph-mon[36624]: pgmap v14474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:53.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:52 smithi081 ceph-mon[31453]: pgmap v14474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:55.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:54 smithi153 ceph-mon[36569]: pgmap v14475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:55.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:54 smithi104 ceph-mon[36624]: pgmap v14475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:55.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:54 smithi081 ceph-mon[31453]: pgmap v14475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:57.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:56 smithi153 ceph-mon[36569]: pgmap v14476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:57.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:56 smithi104 ceph-mon[36624]: pgmap v14476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:57.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:56 smithi081 ceph-mon[31453]: pgmap v14476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:58.188+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:44:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:58.188+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:44:58.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:44:58.188+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:44:59.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:44:58 smithi153 ceph-mon[36569]: pgmap v14477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:59.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:44:58 smithi104 ceph-mon[36624]: pgmap v14477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:44:59.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:44:58 smithi081 ceph-mon[31453]: pgmap v14477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:01.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:00 smithi153 ceph-mon[36569]: pgmap v14478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:01.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:00 smithi104 ceph-mon[36624]: pgmap v14478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:01.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:00 smithi081 ceph-mon[31453]: pgmap v14478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:03.185 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:02 smithi081 ceph-mon[31453]: pgmap v14479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:03.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:02 smithi153 ceph-mon[36569]: pgmap v14479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:03.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:02 smithi104 ceph-mon[36624]: pgmap v14479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:05.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:04 smithi153 ceph-mon[36569]: pgmap v14480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:05.292 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:04 smithi081 ceph-mon[31453]: pgmap v14480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:05.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:04 smithi104 ceph-mon[36624]: pgmap v14480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:07.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:06 smithi153 ceph-mon[36569]: pgmap v14481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:07.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:06 smithi104 ceph-mon[36624]: pgmap v14481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:07.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:06 smithi081 ceph-mon[31453]: pgmap v14481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:09.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:08 smithi153 ceph-mon[36569]: pgmap v14482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:09.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:08 smithi104 ceph-mon[36624]: pgmap v14482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:09.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:08 smithi081 ceph-mon[31453]: pgmap v14482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:11.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:10 smithi081 ceph-mon[31453]: pgmap v14483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:11.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:10 smithi153 ceph-mon[36569]: pgmap v14483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:11.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:10 smithi104 ceph-mon[36624]: pgmap v14483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:13.203+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:45:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:13.203+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:45:13.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:13.203+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:45:13.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:12 smithi153 ceph-mon[36569]: pgmap v14484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:13.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:12 smithi104 ceph-mon[36624]: pgmap v14484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:13.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:12 smithi081 ceph-mon[31453]: pgmap v14484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:15.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:14 smithi153 ceph-mon[36569]: pgmap v14485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:15.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:14 smithi104 ceph-mon[36624]: pgmap v14485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:15.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:14 smithi081 ceph-mon[31453]: pgmap v14485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:17.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:16 smithi153 ceph-mon[36569]: pgmap v14486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:17.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:16 smithi104 ceph-mon[36624]: pgmap v14486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:17.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:16 smithi081 ceph-mon[31453]: pgmap v14486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:19.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:18 smithi153 ceph-mon[36569]: pgmap v14487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:19.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:18 smithi104 ceph-mon[36624]: pgmap v14487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:19.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:18 smithi081 ceph-mon[31453]: pgmap v14487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:21.251 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:21 smithi081 ceph-mon[31453]: pgmap v14488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:21.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:21 smithi153 ceph-mon[36569]: pgmap v14488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:21.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:21 smithi104 ceph-mon[36624]: pgmap v14488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:23.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:23 smithi153 ceph-mon[36569]: pgmap v14489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:23.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:23 smithi104 ceph-mon[36624]: pgmap v14489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:23.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:23 smithi081 ceph-mon[31453]: pgmap v14489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:25.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:25 smithi153 ceph-mon[36569]: pgmap v14490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:25.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:25 smithi104 ceph-mon[36624]: pgmap v14490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:25.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:25 smithi081 ceph-mon[31453]: pgmap v14490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:27.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:27 smithi153 ceph-mon[36569]: pgmap v14491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:27.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:27 smithi104 ceph-mon[36624]: pgmap v14491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:27.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:27 smithi081 ceph-mon[31453]: pgmap v14491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:28.217+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:45:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:28.217+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:45:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:28.217+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:45:29.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:29 smithi153 ceph-mon[36569]: pgmap v14492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:29.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:29 smithi104 ceph-mon[36624]: pgmap v14492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:29.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:29 smithi081 ceph-mon[31453]: pgmap v14492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:31.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:31 smithi153 ceph-mon[36569]: pgmap v14493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:31.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:31 smithi104 ceph-mon[36624]: pgmap v14493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:31.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:31 smithi081 ceph-mon[31453]: pgmap v14493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:33.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:33 smithi153 ceph-mon[36569]: pgmap v14494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:33.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:33 smithi104 ceph-mon[36624]: pgmap v14494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:33.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:33 smithi081 ceph-mon[31453]: pgmap v14494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:35.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:35 smithi153 ceph-mon[36569]: pgmap v14495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:35.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:35 smithi104 ceph-mon[36624]: pgmap v14495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:35.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:35 smithi081 ceph-mon[31453]: pgmap v14495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:37.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:37 smithi153 ceph-mon[36569]: pgmap v14496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:37.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:37 smithi104 ceph-mon[36624]: pgmap v14496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:37.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:37 smithi081 ceph-mon[31453]: pgmap v14496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:39.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:39 smithi153 ceph-mon[36569]: pgmap v14497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:39.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:39 smithi104 ceph-mon[36624]: pgmap v14497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:39.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:39 smithi081 ceph-mon[31453]: pgmap v14497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:41.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:41 smithi153 ceph-mon[36569]: pgmap v14498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:41.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:41 smithi104 ceph-mon[36624]: pgmap v14498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:41.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:41 smithi081 ceph-mon[31453]: pgmap v14498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:43.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:43.232+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:45:43.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:43.232+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:45:43.209 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:43.232+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:45:43.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:43 smithi104 ceph-mon[36624]: pgmap v14499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:43.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:43 smithi081 ceph-mon[31453]: pgmap v14499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:43.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:43 smithi153 ceph-mon[36569]: pgmap v14499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:45.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:45 smithi104 ceph-mon[36624]: pgmap v14500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:45.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:45 smithi081 ceph-mon[31453]: pgmap v14500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:45.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:45 smithi153 ceph-mon[36569]: pgmap v14500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:47.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:47 smithi104 ceph-mon[36624]: pgmap v14501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:47.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:47 smithi081 ceph-mon[31453]: pgmap v14501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:47.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:47 smithi153 ceph-mon[36569]: pgmap v14501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:49 smithi104 ceph-mon[36624]: pgmap v14502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:49.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:45:49.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:49 smithi081 ceph-mon[31453]: pgmap v14502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:49.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:45:49.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:49 smithi153 ceph-mon[36569]: pgmap v14502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:49.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:45:51.255 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:51 smithi153 ceph-mon[36569]: pgmap v14503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:51.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:51 smithi104 ceph-mon[36624]: pgmap v14503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:51.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:51 smithi081 ceph-mon[31453]: pgmap v14503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:53.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:53 smithi081 ceph-mon[31453]: pgmap v14504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:53.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:53 smithi153 ceph-mon[36569]: pgmap v14504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:53.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:53 smithi104 ceph-mon[36624]: pgmap v14504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:54.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:54.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:54.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:45:54.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:45:54.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:54 smithi153 ceph-mon[36569]: pgmap v14505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:54.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:54 smithi153 ceph-mon[36569]: pgmap v14506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:54.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:54.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:54.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:54.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:45:54.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:45:54.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:54 smithi104 ceph-mon[36624]: pgmap v14505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:54.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:54 smithi104 ceph-mon[36624]: pgmap v14506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:54.558 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:54.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:54.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:54.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:45:54.632 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:45:54.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:54 smithi081 ceph-mon[31453]: pgmap v14505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:54.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:54 smithi081 ceph-mon[31453]: pgmap v14506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:54.633 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:45:56.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:56 smithi081 ceph-mon[31453]: pgmap v14507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:56.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:56 smithi153 ceph-mon[36569]: pgmap v14507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:56.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:56 smithi104 ceph-mon[36624]: pgmap v14507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:58.243+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:45:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:58.243+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:45:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:45:58.243+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:45:58.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:45:58 smithi081 ceph-mon[31453]: pgmap v14508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:58.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:45:58 smithi153 ceph-mon[36569]: pgmap v14508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:45:58.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:45:58 smithi104 ceph-mon[36624]: pgmap v14508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:00.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:00 smithi081 ceph-mon[31453]: pgmap v14509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:00.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:00 smithi153 ceph-mon[36569]: pgmap v14509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:00.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:00 smithi104 ceph-mon[36624]: pgmap v14509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:02.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:02 smithi081 ceph-mon[31453]: pgmap v14510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:02.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:02 smithi153 ceph-mon[36569]: pgmap v14510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:02.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:02 smithi104 ceph-mon[36624]: pgmap v14510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:04.532 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:04 smithi153 ceph-mon[36569]: pgmap v14511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:04.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:04 smithi081 ceph-mon[31453]: pgmap v14511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:04.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:04 smithi104 ceph-mon[36624]: pgmap v14511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:06.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:06 smithi081 ceph-mon[31453]: pgmap v14512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:06.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:06 smithi153 ceph-mon[36569]: pgmap v14512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:06.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:06 smithi104 ceph-mon[36624]: pgmap v14512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:08.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:08 smithi081 ceph-mon[31453]: pgmap v14513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:08.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:08 smithi153 ceph-mon[36569]: pgmap v14513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:08.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:08 smithi104 ceph-mon[36624]: pgmap v14513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:10.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:10 smithi081 ceph-mon[31453]: pgmap v14514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:10.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:10 smithi153 ceph-mon[36569]: pgmap v14514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:10.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:10 smithi104 ceph-mon[36624]: pgmap v14514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:12.165+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000548) for more than 900000000 us, fault. 2024-04-13T04:46:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:12.165+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f16f4004a80 tx=0x7f16f4039000 comp rx=0 tx=0).stop 2024-04-13T04:46:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:12.165+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b3ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T04:46:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:12.165+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3ee0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T04:46:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:12.165+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T04:46:12.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:12.165+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T04:46:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:12.166+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c6a0 tx=0x7f16f4044000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T04:46:12.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:12 smithi081 ceph-mon[31453]: pgmap v14515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:12.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:12 smithi153 ceph-mon[36569]: pgmap v14515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:12.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:12 smithi104 ceph-mon[36624]: pgmap v14515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:13.259+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:46:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:13.259+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:46:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:13.259+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:46:14.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:14 smithi081 ceph-mon[31453]: pgmap v14516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:14.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:14 smithi153 ceph-mon[36569]: pgmap v14516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:14.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:14 smithi104 ceph-mon[36624]: pgmap v14516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:16.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:16 smithi153 ceph-mon[36569]: pgmap v14517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:16.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:16 smithi104 ceph-mon[36624]: pgmap v14517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:16.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:16 smithi081 ceph-mon[31453]: pgmap v14517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:17.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:17.167+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:46:17.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:17.167+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:46:17.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:17.168+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 209 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:46:17.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:17.168+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 209 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40065b0 con 0x7f1704076980 2024-04-13T04:46:18.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:18 smithi153 ceph-mon[36569]: pgmap v14518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:18.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:18 smithi104 ceph-mon[36624]: pgmap v14518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:18.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:18 smithi081 ceph-mon[31453]: pgmap v14518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:20.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:20 smithi081 ceph-mon[31453]: pgmap v14519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:20.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:20 smithi153 ceph-mon[36569]: pgmap v14519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:20.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:20 smithi104 ceph-mon[36624]: pgmap v14519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:22.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:22 smithi153 ceph-mon[36569]: pgmap v14520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:22.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:22 smithi104 ceph-mon[36624]: pgmap v14520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:22.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:22 smithi081 ceph-mon[31453]: pgmap v14520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:24.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:24 smithi153 ceph-mon[36569]: pgmap v14521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:24.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:24 smithi104 ceph-mon[36624]: pgmap v14521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:24.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:24 smithi081 ceph-mon[31453]: pgmap v14521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:26.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:26 smithi153 ceph-mon[36569]: pgmap v14522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:26.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:26 smithi104 ceph-mon[36624]: pgmap v14522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:26.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:26 smithi081 ceph-mon[31453]: pgmap v14522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:28.272+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:46:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:28.272+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:46:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:28.272+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:46:28.557 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:28 smithi104 ceph-mon[36624]: pgmap v14523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:28.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:28 smithi081 ceph-mon[31453]: pgmap v14523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:28.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:28 smithi153 ceph-mon[36569]: pgmap v14523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:30.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:30 smithi081 ceph-mon[31453]: pgmap v14524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:30.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:30 smithi153 ceph-mon[36569]: pgmap v14524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:30.808 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:30 smithi104 ceph-mon[36624]: pgmap v14524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:32.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:32 smithi081 ceph-mon[31453]: pgmap v14525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:32.781 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:32 smithi153 ceph-mon[36569]: pgmap v14525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:32.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:32 smithi104 ceph-mon[36624]: pgmap v14525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:34.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:34 smithi081 ceph-mon[31453]: pgmap v14526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:34.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:34 smithi153 ceph-mon[36569]: pgmap v14526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:34.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:34 smithi104 ceph-mon[36624]: pgmap v14526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:36.434 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:36 smithi081 ceph-mon[31453]: pgmap v14527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:36.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:36 smithi153 ceph-mon[36569]: pgmap v14527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:36.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:36 smithi104 ceph-mon[36624]: pgmap v14527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:38.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:38 smithi081 ceph-mon[31453]: pgmap v14528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:38.781 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:38 smithi153 ceph-mon[36569]: pgmap v14528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:38.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:38 smithi104 ceph-mon[36624]: pgmap v14528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:40.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:40 smithi081 ceph-mon[31453]: pgmap v14529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:40.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:40 smithi153 ceph-mon[36569]: pgmap v14529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:40.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:40 smithi104 ceph-mon[36624]: pgmap v14529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:42.609 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:42 smithi104 ceph-mon[36624]: pgmap v14530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:42.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:42 smithi081 ceph-mon[31453]: pgmap v14530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:42.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:42 smithi153 ceph-mon[36569]: pgmap v14530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:43.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:43.284+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:46:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:43.284+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:46:43.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:43.284+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:46:44.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:44 smithi081 ceph-mon[31453]: pgmap v14531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:44.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:44 smithi153 ceph-mon[36569]: pgmap v14531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:44.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:44 smithi104 ceph-mon[36624]: pgmap v14531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:46.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:46 smithi081 ceph-mon[31453]: pgmap v14532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:46.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:46 smithi153 ceph-mon[36569]: pgmap v14532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:46.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:46 smithi104 ceph-mon[36624]: pgmap v14532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:48.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:48 smithi081 ceph-mon[31453]: pgmap v14533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:48.781 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:48 smithi153 ceph-mon[36569]: pgmap v14533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:48.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:48 smithi104 ceph-mon[36624]: pgmap v14533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:50.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:50 smithi081 ceph-mon[31453]: pgmap v14534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:50.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:50 smithi153 ceph-mon[36569]: pgmap v14534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:50.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:50 smithi104 ceph-mon[36624]: pgmap v14534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:52.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:52 smithi081 ceph-mon[31453]: pgmap v14535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:52.781 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:52 smithi153 ceph-mon[36569]: pgmap v14535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:52.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:52 smithi104 ceph-mon[36624]: pgmap v14535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:54.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:54 smithi081 ceph-mon[31453]: pgmap v14536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:54.631 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:46:54.781 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:54 smithi153 ceph-mon[36569]: pgmap v14536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:54.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:46:54.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:54 smithi104 ceph-mon[36624]: pgmap v14536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:54.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:46:55.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:46:55.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:46:55.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:55 smithi153 ceph-mon[36569]: pgmap v14537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:55.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:55 smithi153 ceph-mon[36569]: pgmap v14538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:55.782 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:46:55.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:46:55.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:46:55.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:55 smithi104 ceph-mon[36624]: pgmap v14537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:55.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:55 smithi104 ceph-mon[36624]: pgmap v14538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:55.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:46:55.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:46:55.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:46:55.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:55 smithi081 ceph-mon[31453]: pgmap v14537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:55.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:55 smithi081 ceph-mon[31453]: pgmap v14538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:55.881 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:46:58.264 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:58.288+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:46:58.264 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:58.288+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:46:58.264 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:46:58.288+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:46:58.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:57 smithi153 ceph-mon[36569]: pgmap v14539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:58.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:57 smithi104 ceph-mon[36624]: pgmap v14539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:46:58.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:57 smithi081 ceph-mon[31453]: pgmap v14539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:00.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:46:59 smithi153 ceph-mon[36569]: pgmap v14540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:00.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:46:59 smithi104 ceph-mon[36624]: pgmap v14540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:00.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:46:59 smithi081 ceph-mon[31453]: pgmap v14540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:02.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:01 smithi153 ceph-mon[36569]: pgmap v14541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:02.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:01 smithi104 ceph-mon[36624]: pgmap v14541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:02.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:01 smithi081 ceph-mon[31453]: pgmap v14541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:04.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:03 smithi153 ceph-mon[36569]: pgmap v14542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:04.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:03 smithi104 ceph-mon[36624]: pgmap v14542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:04.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:03 smithi081 ceph-mon[31453]: pgmap v14542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:06.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:05 smithi153 ceph-mon[36569]: pgmap v14543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:06.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:05 smithi104 ceph-mon[36624]: pgmap v14543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:06.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:05 smithi081 ceph-mon[31453]: pgmap v14543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:08.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:07 smithi153 ceph-mon[36569]: pgmap v14544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:08.311 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:07 smithi104 ceph-mon[36624]: pgmap v14544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:08.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:07 smithi081 ceph-mon[31453]: pgmap v14544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:10.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:09 smithi153 ceph-mon[36569]: pgmap v14545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:10.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:09 smithi104 ceph-mon[36624]: pgmap v14545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:10.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:09 smithi081 ceph-mon[31453]: pgmap v14545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:12.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:11 smithi153 ceph-mon[36569]: pgmap v14546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:12.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:11 smithi104 ceph-mon[36624]: pgmap v14546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:12.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:11 smithi081 ceph-mon[31453]: pgmap v14546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:13.276 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:13.301+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:47:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:13.301+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:47:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:13.301+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:47:14.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:13 smithi153 ceph-mon[36569]: pgmap v14547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:14.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:13 smithi104 ceph-mon[36624]: pgmap v14547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:14.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:13 smithi081 ceph-mon[31453]: pgmap v14547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:16.281 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:15 smithi153 ceph-mon[36569]: pgmap v14548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:16.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:15 smithi104 ceph-mon[36624]: pgmap v14548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:16.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:15 smithi081 ceph-mon[31453]: pgmap v14548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:18.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:17 smithi153 ceph-mon[36569]: pgmap v14549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:18.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:17 smithi104 ceph-mon[36624]: pgmap v14549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:18.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:17 smithi081 ceph-mon[31453]: pgmap v14549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:20.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:19 smithi153 ceph-mon[36569]: pgmap v14550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:20.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:19 smithi104 ceph-mon[36624]: pgmap v14550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:20.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:19 smithi081 ceph-mon[31453]: pgmap v14550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:22.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:21 smithi153 ceph-mon[36569]: pgmap v14551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:22.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:21 smithi104 ceph-mon[36624]: pgmap v14551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:22.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:21 smithi081 ceph-mon[31453]: pgmap v14551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:24.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:24 smithi153 ceph-mon[36569]: pgmap v14552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:24.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:24 smithi104 ceph-mon[36624]: pgmap v14552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:24.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:24 smithi081 ceph-mon[31453]: pgmap v14552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:26.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:26 smithi153 ceph-mon[36569]: pgmap v14553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:26.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:26 smithi104 ceph-mon[36624]: pgmap v14553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:26.381 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:26 smithi081 ceph-mon[31453]: pgmap v14553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:28.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:28 smithi153 ceph-mon[36569]: pgmap v14554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:28.316+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:47:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:28.316+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:47:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:28.316+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:47:28.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:28 smithi104 ceph-mon[36624]: pgmap v14554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:28.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:28 smithi081 ceph-mon[31453]: pgmap v14554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:30.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:30 smithi153 ceph-mon[36569]: pgmap v14555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:30.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:30 smithi104 ceph-mon[36624]: pgmap v14555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:30.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:30 smithi081 ceph-mon[31453]: pgmap v14555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:32.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:32 smithi153 ceph-mon[36569]: pgmap v14556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:32.307 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:32 smithi104 ceph-mon[36624]: pgmap v14556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:32.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:32 smithi081 ceph-mon[31453]: pgmap v14556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:34.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:34 smithi153 ceph-mon[36569]: pgmap v14557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:34.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:34 smithi104 ceph-mon[36624]: pgmap v14557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:34.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:34 smithi081 ceph-mon[31453]: pgmap v14557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:36.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:36 smithi153 ceph-mon[36569]: pgmap v14558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:36.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:36 smithi104 ceph-mon[36624]: pgmap v14558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:36.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:36 smithi081 ceph-mon[31453]: pgmap v14558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:38.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:38 smithi153 ceph-mon[36569]: pgmap v14559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:38.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:38 smithi104 ceph-mon[36624]: pgmap v14559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:38.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:38 smithi081 ceph-mon[31453]: pgmap v14559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:40.282 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:40 smithi153 ceph-mon[36569]: pgmap v14560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:40.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:40 smithi104 ceph-mon[36624]: pgmap v14560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:40.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:40 smithi081 ceph-mon[31453]: pgmap v14560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:42.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:42 smithi153 ceph-mon[36569]: pgmap v14561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:42.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:42 smithi104 ceph-mon[36624]: pgmap v14561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:42.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:42 smithi081 ceph-mon[31453]: pgmap v14561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:43.328+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:47:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:43.328+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:47:43.305 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:43.328+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:47:44.283 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:44 smithi153 ceph-mon[36569]: pgmap v14562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:44.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:44 smithi104 ceph-mon[36624]: pgmap v14562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:44.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:44 smithi081 ceph-mon[31453]: pgmap v14562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:46.308 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:46 smithi104 ceph-mon[36624]: pgmap v14563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:46.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:46 smithi081 ceph-mon[31453]: pgmap v14563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:46.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:46 smithi153 ceph-mon[36569]: pgmap v14563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:48.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:48 smithi104 ceph-mon[36624]: pgmap v14564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:48.382 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:48 smithi081 ceph-mon[31453]: pgmap v14564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:48.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:48 smithi153 ceph-mon[36569]: pgmap v14564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:50.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:50 smithi104 ceph-mon[36624]: pgmap v14565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:50.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:50 smithi081 ceph-mon[31453]: pgmap v14565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:50.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:50 smithi153 ceph-mon[36569]: pgmap v14565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:52.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:52 smithi104 ceph-mon[36624]: pgmap v14566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:52.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:52 smithi081 ceph-mon[31453]: pgmap v14566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:52.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:52 smithi153 ceph-mon[36569]: pgmap v14566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:54.309 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:54 smithi104 ceph-mon[36624]: pgmap v14567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:54.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:54 smithi081 ceph-mon[31453]: pgmap v14567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:54.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:54 smithi153 ceph-mon[36569]: pgmap v14567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:56.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:56 smithi081 ceph-mon[31453]: pgmap v14568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:56.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:47:56.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:56 smithi153 ceph-mon[36569]: pgmap v14568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:56.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:47:56.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:56 smithi104 ceph-mon[36624]: pgmap v14568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:56.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:47:57.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:47:57.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:47:57.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:47:57.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:47:57.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:47:57.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:47:57.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:47:57.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:47:57.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:47:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:58.335+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:47:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:58.335+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:47:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:47:58.335+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:47:58.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:58 smithi081 ceph-mon[31453]: pgmap v14569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:58.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:47:58 smithi081 ceph-mon[31453]: pgmap v14570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:58.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:58 smithi153 ceph-mon[36569]: pgmap v14569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:58.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:47:58 smithi153 ceph-mon[36569]: pgmap v14570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:58.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:58 smithi104 ceph-mon[36624]: pgmap v14569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:47:58.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:47:58 smithi104 ceph-mon[36624]: pgmap v14570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:00.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:00 smithi081 ceph-mon[31453]: pgmap v14571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:00.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:00 smithi153 ceph-mon[36569]: pgmap v14571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:00.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:00 smithi104 ceph-mon[36624]: pgmap v14571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:02.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:02 smithi081 ceph-mon[31453]: pgmap v14572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:02.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:02 smithi153 ceph-mon[36569]: pgmap v14572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:02.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:02 smithi104 ceph-mon[36624]: pgmap v14572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:04.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:04 smithi081 ceph-mon[31453]: pgmap v14573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:04.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:04 smithi153 ceph-mon[36569]: pgmap v14573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:04.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:04 smithi104 ceph-mon[36624]: pgmap v14573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:06.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:06 smithi081 ceph-mon[31453]: pgmap v14574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:06.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:06 smithi153 ceph-mon[36569]: pgmap v14574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:06.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:06 smithi104 ceph-mon[36624]: pgmap v14574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:08.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:08 smithi081 ceph-mon[31453]: pgmap v14575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:08.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:08 smithi153 ceph-mon[36569]: pgmap v14575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:08.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:08 smithi104 ceph-mon[36624]: pgmap v14575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:10.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:10 smithi081 ceph-mon[31453]: pgmap v14576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:10.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:10 smithi153 ceph-mon[36569]: pgmap v14576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:10.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:10 smithi104 ceph-mon[36624]: pgmap v14576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:12.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:12 smithi081 ceph-mon[31453]: pgmap v14577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:12.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:12 smithi153 ceph-mon[36569]: pgmap v14577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:12.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:12 smithi104 ceph-mon[36624]: pgmap v14577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:13.348+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:48:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:13.348+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:48:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:13.348+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:48:14.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:14 smithi081 ceph-mon[31453]: pgmap v14578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:14.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:14 smithi153 ceph-mon[36569]: pgmap v14578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:14.559 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:14 smithi104 ceph-mon[36624]: pgmap v14578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:16.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:16 smithi081 ceph-mon[31453]: pgmap v14579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:16.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:16 smithi153 ceph-mon[36569]: pgmap v14579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:16.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:16 smithi104 ceph-mon[36624]: pgmap v14579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:18.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:18 smithi081 ceph-mon[31453]: pgmap v14580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:18.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:18 smithi153 ceph-mon[36569]: pgmap v14580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:18.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:18 smithi104 ceph-mon[36624]: pgmap v14580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:20.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:20 smithi153 ceph-mon[36569]: pgmap v14581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:20.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:20 smithi104 ceph-mon[36624]: pgmap v14581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:20.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:20 smithi081 ceph-mon[31453]: pgmap v14581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:22.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:22 smithi153 ceph-mon[36569]: pgmap v14582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:22.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:22 smithi104 ceph-mon[36624]: pgmap v14582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:22.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:22 smithi081 ceph-mon[31453]: pgmap v14582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:24.534 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:24 smithi153 ceph-mon[36569]: pgmap v14583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:24.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:24 smithi104 ceph-mon[36624]: pgmap v14583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:24.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:24 smithi081 ceph-mon[31453]: pgmap v14583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:25.535 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:25 smithi153 ceph-mon[36569]: pgmap v14584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:25.560 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:25 smithi104 ceph-mon[36624]: pgmap v14584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:25.634 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:25 smithi081 ceph-mon[31453]: pgmap v14584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:27.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:27 smithi153 ceph-mon[36569]: pgmap v14585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:27.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:27 smithi104 ceph-mon[36624]: pgmap v14585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:27.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:27 smithi081 ceph-mon[31453]: pgmap v14585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:28.355+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:48:28.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:28.355+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:48:28.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:28.355+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:48:29.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:29 smithi153 ceph-mon[36569]: pgmap v14586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:29.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:29 smithi104 ceph-mon[36624]: pgmap v14586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:29.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:29 smithi081 ceph-mon[31453]: pgmap v14586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:31.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:31 smithi153 ceph-mon[36569]: pgmap v14587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:31.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:31 smithi104 ceph-mon[36624]: pgmap v14587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:31.884 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:31 smithi081 ceph-mon[31453]: pgmap v14587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:33.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:33 smithi153 ceph-mon[36569]: pgmap v14588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:33.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:33 smithi104 ceph-mon[36624]: pgmap v14588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:33.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:33 smithi081 ceph-mon[31453]: pgmap v14588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:35.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:35 smithi153 ceph-mon[36569]: pgmap v14589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:35.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:35 smithi104 ceph-mon[36624]: pgmap v14589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:35.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:35 smithi081 ceph-mon[31453]: pgmap v14589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:37.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:37 smithi153 ceph-mon[36569]: pgmap v14590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:37.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:37 smithi104 ceph-mon[36624]: pgmap v14590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:37.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:37 smithi081 ceph-mon[31453]: pgmap v14590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:39.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:39 smithi153 ceph-mon[36569]: pgmap v14591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:39.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:39 smithi104 ceph-mon[36624]: pgmap v14591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:39.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:39 smithi081 ceph-mon[31453]: pgmap v14591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:41.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:41 smithi153 ceph-mon[36569]: pgmap v14592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:41.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:41 smithi104 ceph-mon[36624]: pgmap v14592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:41.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:41 smithi081 ceph-mon[31453]: pgmap v14592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:43.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:43.370+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:48:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:43.370+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:48:43.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:43.370+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:48:43.785 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:43 smithi153 ceph-mon[36569]: pgmap v14593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:43.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:43 smithi104 ceph-mon[36624]: pgmap v14593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:43.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:43 smithi081 ceph-mon[31453]: pgmap v14593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:45.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:45 smithi153 ceph-mon[36569]: pgmap v14594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:45.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:45 smithi104 ceph-mon[36624]: pgmap v14594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:45.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:45 smithi081 ceph-mon[31453]: pgmap v14594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:47.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:47 smithi153 ceph-mon[36569]: pgmap v14595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:47.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:47 smithi104 ceph-mon[36624]: pgmap v14595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:47.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:47 smithi081 ceph-mon[31453]: pgmap v14595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:49.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:49 smithi153 ceph-mon[36569]: pgmap v14596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:49.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:49 smithi104 ceph-mon[36624]: pgmap v14596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:49.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:49 smithi081 ceph-mon[31453]: pgmap v14596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:51.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:51 smithi153 ceph-mon[36569]: pgmap v14597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:51.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:51 smithi104 ceph-mon[36624]: pgmap v14597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:51.885 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:51 smithi081 ceph-mon[31453]: pgmap v14597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:53.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:53 smithi153 ceph-mon[36569]: pgmap v14598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:53.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:53 smithi104 ceph-mon[36624]: pgmap v14598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:53.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:53 smithi081 ceph-mon[31453]: pgmap v14598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:55.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:55 smithi153 ceph-mon[36569]: pgmap v14599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:55.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:55 smithi104 ceph-mon[36624]: pgmap v14599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:55.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:55 smithi081 ceph-mon[31453]: pgmap v14599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:57.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:57 smithi153 ceph-mon[36569]: pgmap v14600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:57.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:48:57.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:57 smithi104 ceph-mon[36624]: pgmap v14600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:57.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:48:57.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:57 smithi081 ceph-mon[31453]: pgmap v14600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:57.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:48:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:58.384+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:48:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:58.384+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:48:58.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:48:58.384+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:48:58.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:48:58.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:48:58.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:58 smithi153 ceph-mon[36569]: pgmap v14601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:58.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:58 smithi153 ceph-mon[36569]: pgmap v14602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:58.786 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:48:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:48:58.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:48:58.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:48:58.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:58 smithi104 ceph-mon[36624]: pgmap v14601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:58.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:58 smithi104 ceph-mon[36624]: pgmap v14602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:58.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:48:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:48:58.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:48:58.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:48:58.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:58 smithi081 ceph-mon[31453]: pgmap v14601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:58.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:58 smithi081 ceph-mon[31453]: pgmap v14602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:48:58.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:48:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:49:01.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:01 smithi153 ceph-mon[36569]: pgmap v14603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:01.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:01 smithi104 ceph-mon[36624]: pgmap v14603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:01.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:01 smithi081 ceph-mon[31453]: pgmap v14603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:03.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:03 smithi153 ceph-mon[36569]: pgmap v14604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:03.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:03 smithi104 ceph-mon[36624]: pgmap v14604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:03.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:03 smithi081 ceph-mon[31453]: pgmap v14604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:05.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:05 smithi153 ceph-mon[36569]: pgmap v14605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:05.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:05 smithi104 ceph-mon[36624]: pgmap v14605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:05.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:05 smithi081 ceph-mon[31453]: pgmap v14605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:07.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:07 smithi153 ceph-mon[36569]: pgmap v14606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:07.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:07 smithi104 ceph-mon[36624]: pgmap v14606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:07.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:07 smithi081 ceph-mon[31453]: pgmap v14606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:09.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:09 smithi153 ceph-mon[36569]: pgmap v14607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:09.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:09 smithi104 ceph-mon[36624]: pgmap v14607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:09.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:09 smithi081 ceph-mon[31453]: pgmap v14607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:11.286 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:11 smithi153 ceph-mon[36569]: pgmap v14608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:11.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:11 smithi104 ceph-mon[36624]: pgmap v14608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:11.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:11 smithi081 ceph-mon[31453]: pgmap v14608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:13.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:13 smithi153 ceph-mon[36569]: pgmap v14609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:13.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:13 smithi104 ceph-mon[36624]: pgmap v14609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:13.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:13.399+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:49:13.380 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:13.399+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:49:13.380 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:13.399+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:49:13.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:13 smithi081 ceph-mon[31453]: pgmap v14609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:15.287 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:15 smithi153 ceph-mon[36569]: pgmap v14610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:15.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:15 smithi104 ceph-mon[36624]: pgmap v14610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:15.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:15 smithi081 ceph-mon[31453]: pgmap v14610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:17.196+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:49:17.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:17.196+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:49:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:17.197+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 210 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:49:17.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:17.197+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 210 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:49:17.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:17 smithi104 ceph-mon[36624]: pgmap v14611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:17.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:17 smithi081 ceph-mon[31453]: pgmap v14611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:17.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:17 smithi153 ceph-mon[36569]: pgmap v14611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:19.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:19 smithi104 ceph-mon[36624]: pgmap v14612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:19.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:19 smithi081 ceph-mon[31453]: pgmap v14612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:19.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:19 smithi153 ceph-mon[36569]: pgmap v14612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:21.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:21 smithi104 ceph-mon[36624]: pgmap v14613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:21.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:21 smithi081 ceph-mon[31453]: pgmap v14613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:21.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:21 smithi153 ceph-mon[36569]: pgmap v14613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:23.313 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:23 smithi104 ceph-mon[36624]: pgmap v14614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:23.386 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:23 smithi081 ceph-mon[31453]: pgmap v14614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:23.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:23 smithi153 ceph-mon[36569]: pgmap v14614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:25.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:25 smithi104 ceph-mon[36624]: pgmap v14615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:25.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:25 smithi081 ceph-mon[31453]: pgmap v14615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:25.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:25 smithi153 ceph-mon[36569]: pgmap v14615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:27.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:27 smithi081 ceph-mon[31453]: pgmap v14616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:27.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:27 smithi153 ceph-mon[36569]: pgmap v14616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:27.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:27 smithi104 ceph-mon[36624]: pgmap v14616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:28.414+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:49:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:28.415+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:49:28.396 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:28.415+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:49:29.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:29 smithi081 ceph-mon[31453]: pgmap v14617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:29.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:29 smithi153 ceph-mon[36569]: pgmap v14617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:29.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:29 smithi104 ceph-mon[36624]: pgmap v14617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:31.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:31 smithi081 ceph-mon[31453]: pgmap v14618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:31.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:31 smithi153 ceph-mon[36569]: pgmap v14618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:31.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:31 smithi104 ceph-mon[36624]: pgmap v14618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:33.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:33 smithi081 ceph-mon[31453]: pgmap v14619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:33.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:33 smithi153 ceph-mon[36569]: pgmap v14619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:33.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:33 smithi104 ceph-mon[36624]: pgmap v14619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:35.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:35 smithi081 ceph-mon[31453]: pgmap v14620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:35.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:35 smithi153 ceph-mon[36569]: pgmap v14620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:35.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:35 smithi104 ceph-mon[36624]: pgmap v14620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:37.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:37 smithi081 ceph-mon[31453]: pgmap v14621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:37.537 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:37 smithi153 ceph-mon[36569]: pgmap v14621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:37.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:37 smithi104 ceph-mon[36624]: pgmap v14621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:39.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:39 smithi081 ceph-mon[31453]: pgmap v14622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:39.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:39 smithi153 ceph-mon[36569]: pgmap v14622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:39.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:39 smithi104 ceph-mon[36624]: pgmap v14622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:41.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:41 smithi081 ceph-mon[31453]: pgmap v14623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:41.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:41 smithi153 ceph-mon[36569]: pgmap v14623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:41.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:41 smithi104 ceph-mon[36624]: pgmap v14623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:43.366 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:43 smithi104 ceph-mon[36624]: pgmap v14624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:43.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:43 smithi081 ceph-mon[31453]: pgmap v14624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:43.430+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:49:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:43.430+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:49:43.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:43.430+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:49:43.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:43 smithi153 ceph-mon[36569]: pgmap v14624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:45.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:45 smithi081 ceph-mon[31453]: pgmap v14625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:45.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:45 smithi153 ceph-mon[36569]: pgmap v14625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:45.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:45 smithi104 ceph-mon[36624]: pgmap v14625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:47.387 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:47 smithi081 ceph-mon[31453]: pgmap v14626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:47.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:47 smithi153 ceph-mon[36569]: pgmap v14626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:47.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:47 smithi104 ceph-mon[36624]: pgmap v14626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:49.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:49 smithi081 ceph-mon[31453]: pgmap v14627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:49.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:49 smithi153 ceph-mon[36569]: pgmap v14627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:49.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:49 smithi104 ceph-mon[36624]: pgmap v14627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:51.388 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:51 smithi081 ceph-mon[31453]: pgmap v14628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:51.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:51 smithi153 ceph-mon[36569]: pgmap v14628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:51.563 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:51 smithi104 ceph-mon[36624]: pgmap v14628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:53.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:53 smithi153 ceph-mon[36569]: pgmap v14629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:53.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:53 smithi104 ceph-mon[36624]: pgmap v14629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:53.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:53 smithi081 ceph-mon[31453]: pgmap v14629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:55.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:55 smithi153 ceph-mon[36569]: pgmap v14630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:55.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:55 smithi104 ceph-mon[36624]: pgmap v14630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:55.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:55 smithi081 ceph-mon[31453]: pgmap v14630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:57.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:57 smithi153 ceph-mon[36569]: pgmap v14631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:57.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:57 smithi104 ceph-mon[36624]: pgmap v14631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:57.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:57 smithi081 ceph-mon[31453]: pgmap v14631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:58.444+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:49:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:58.444+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:49:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:49:58.444+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:49:59.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: pgmap v14632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:49:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: pgmap v14632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:59.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:49:59.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:49:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: pgmap v14632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:49:59.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:49:59.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:49:59.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:49:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:50:00.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:50:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:50:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:50:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:50:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:50:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:50:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:50:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:50:00.565 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:50:00.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T04:50:01.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:01 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:50:01.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:01 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:50:01.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:01 smithi153 ceph-mon[36569]: pgmap v14633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:01.539 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:01 smithi153 ceph-mon[36569]: pgmap v14634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:01.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:01 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:50:01.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:01 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:50:01.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:01 smithi104 ceph-mon[36624]: pgmap v14633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:01.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:01 smithi104 ceph-mon[36624]: pgmap v14634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:01.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:01 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T04:50:01.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:01 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T04:50:01.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:01 smithi081 ceph-mon[31453]: pgmap v14633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:01.639 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:01 smithi081 ceph-mon[31453]: pgmap v14634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:02.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:02.204+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:50:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:02.204+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:50:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:02.205+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 211 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:50:02.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:02.205+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 211 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:50:03.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:03 smithi153 ceph-mon[36569]: pgmap v14635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:03.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:03 smithi104 ceph-mon[36624]: pgmap v14635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:03.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:03 smithi081 ceph-mon[31453]: pgmap v14635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:04.538 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:04 smithi153 ceph-mon[36569]: pgmap v14636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:04.564 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:04 smithi104 ceph-mon[36624]: pgmap v14636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:04.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:04 smithi081 ceph-mon[31453]: pgmap v14636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:06.789 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:06 smithi153 ceph-mon[36569]: pgmap v14637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:06.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:06 smithi104 ceph-mon[36624]: pgmap v14637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:06.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:06 smithi081 ceph-mon[31453]: pgmap v14637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:08.788 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:08 smithi153 ceph-mon[36569]: pgmap v14638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:08.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:08 smithi104 ceph-mon[36624]: pgmap v14638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:08.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:08 smithi081 ceph-mon[31453]: pgmap v14638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:10.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:10 smithi104 ceph-mon[36624]: pgmap v14639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:10.888 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:10 smithi081 ceph-mon[31453]: pgmap v14639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:11.038 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:10 smithi153 ceph-mon[36569]: pgmap v14639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:12.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:12 smithi104 ceph-mon[36624]: pgmap v14640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:12.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:12 smithi081 ceph-mon[31453]: pgmap v14640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:13.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:12 smithi153 ceph-mon[36569]: pgmap v14640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:13.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:13.446+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:50:13.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:13.446+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:50:13.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:13.446+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:50:14.814 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:14 smithi104 ceph-mon[36624]: pgmap v14641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:14.841 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:14 smithi153 ceph-mon[36569]: pgmap v14641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:14.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:14 smithi081 ceph-mon[31453]: pgmap v14641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:16.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:16 smithi104 ceph-mon[36624]: pgmap v14642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:16.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:16 smithi081 ceph-mon[31453]: pgmap v14642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:17.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:16 smithi153 ceph-mon[36569]: pgmap v14642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:18.815 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:18 smithi104 ceph-mon[36624]: pgmap v14643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:18.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:18 smithi081 ceph-mon[31453]: pgmap v14643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:19.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:18 smithi153 ceph-mon[36569]: pgmap v14643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:20.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:20 smithi081 ceph-mon[31453]: pgmap v14644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:21.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:20 smithi153 ceph-mon[36569]: pgmap v14644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:21.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:20 smithi104 ceph-mon[36624]: pgmap v14644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:22.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:22 smithi081 ceph-mon[31453]: pgmap v14645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:23.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:22 smithi153 ceph-mon[36569]: pgmap v14645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:23.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:22 smithi104 ceph-mon[36624]: pgmap v14645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:24.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:24 smithi081 ceph-mon[31453]: pgmap v14646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:25.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:24 smithi153 ceph-mon[36569]: pgmap v14646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:25.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:24 smithi104 ceph-mon[36624]: pgmap v14646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:26.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:26 smithi081 ceph-mon[31453]: pgmap v14647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:27.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:26 smithi153 ceph-mon[36569]: pgmap v14647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:27.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:26 smithi104 ceph-mon[36624]: pgmap v14647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:28.452+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:50:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:28.452+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:50:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:28.452+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:50:28.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:28 smithi081 ceph-mon[31453]: pgmap v14648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:29.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:28 smithi153 ceph-mon[36569]: pgmap v14648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:29.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:28 smithi104 ceph-mon[36624]: pgmap v14648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:30.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:30 smithi081 ceph-mon[31453]: pgmap v14649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:31.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:30 smithi153 ceph-mon[36569]: pgmap v14649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:31.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:30 smithi104 ceph-mon[36624]: pgmap v14649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:32.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:32 smithi081 ceph-mon[31453]: pgmap v14650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:33.039 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:32 smithi153 ceph-mon[36569]: pgmap v14650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:33.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:32 smithi104 ceph-mon[36624]: pgmap v14650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:34.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:34 smithi081 ceph-mon[31453]: pgmap v14651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:35.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:34 smithi153 ceph-mon[36569]: pgmap v14651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:35.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:34 smithi104 ceph-mon[36624]: pgmap v14651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:36.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:36 smithi081 ceph-mon[31453]: pgmap v14652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:37.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:36 smithi153 ceph-mon[36569]: pgmap v14652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:37.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:36 smithi104 ceph-mon[36624]: pgmap v14652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:38.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:38 smithi081 ceph-mon[31453]: pgmap v14653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:39.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:38 smithi153 ceph-mon[36569]: pgmap v14653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:39.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:38 smithi104 ceph-mon[36624]: pgmap v14653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:40.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:40 smithi081 ceph-mon[31453]: pgmap v14654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:41.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:40 smithi153 ceph-mon[36569]: pgmap v14654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:41.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:40 smithi104 ceph-mon[36624]: pgmap v14654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:42.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:42 smithi081 ceph-mon[31453]: pgmap v14655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:43.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:42 smithi153 ceph-mon[36569]: pgmap v14655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:43.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:42 smithi104 ceph-mon[36624]: pgmap v14655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:43.459+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:50:43.443 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:43.459+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:50:43.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:43.459+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:50:44.889 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:44 smithi081 ceph-mon[31453]: pgmap v14656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:45.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:44 smithi153 ceph-mon[36569]: pgmap v14656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:45.065 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:44 smithi104 ceph-mon[36624]: pgmap v14656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:46.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:46 smithi081 ceph-mon[31453]: pgmap v14657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:47.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:46 smithi153 ceph-mon[36569]: pgmap v14657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:47.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:46 smithi104 ceph-mon[36624]: pgmap v14657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:49.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:48 smithi153 ceph-mon[36569]: pgmap v14658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:49.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:48 smithi104 ceph-mon[36624]: pgmap v14658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:49.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:48 smithi081 ceph-mon[31453]: pgmap v14658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:51.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:50 smithi153 ceph-mon[36569]: pgmap v14659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:51.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:50 smithi104 ceph-mon[36624]: pgmap v14659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:51.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:50 smithi081 ceph-mon[31453]: pgmap v14659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:53.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:52 smithi153 ceph-mon[36569]: pgmap v14660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:53.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:52 smithi104 ceph-mon[36624]: pgmap v14660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:53.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:52 smithi081 ceph-mon[31453]: pgmap v14660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:55.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:54 smithi153 ceph-mon[36569]: pgmap v14661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:55.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:54 smithi104 ceph-mon[36624]: pgmap v14661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:55.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:54 smithi081 ceph-mon[31453]: pgmap v14661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:57.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:56 smithi153 ceph-mon[36569]: pgmap v14662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:57.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:56 smithi104 ceph-mon[36624]: pgmap v14662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:57.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:56 smithi081 ceph-mon[31453]: pgmap v14662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:58.475+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:50:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:58.475+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:50:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:50:58.475+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:50:59.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:50:58 smithi153 ceph-mon[36569]: pgmap v14663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:59.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:50:58 smithi104 ceph-mon[36624]: pgmap v14663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:50:59.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:50:58 smithi081 ceph-mon[31453]: pgmap v14663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:01.040 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:00 smithi153 ceph-mon[36569]: pgmap v14664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:01.041 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:51:01.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:00 smithi104 ceph-mon[36624]: pgmap v14664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:01.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:51:01.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:00 smithi081 ceph-mon[31453]: pgmap v14664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:01.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:51:02.041 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:51:02.041 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:51:02.041 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:01 smithi153 ceph-mon[36569]: pgmap v14665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:02.041 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:01 smithi153 ceph-mon[36569]: pgmap v14666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:02.041 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:51:02.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:51:02.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:51:02.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:01 smithi104 ceph-mon[36624]: pgmap v14665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:02.066 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:01 smithi104 ceph-mon[36624]: pgmap v14666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:02.067 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:51:02.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:51:02.140 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:51:02.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:01 smithi081 ceph-mon[31453]: pgmap v14665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:02.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:01 smithi081 ceph-mon[31453]: pgmap v14666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:02.141 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:51:03.790 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:03 smithi153 ceph-mon[36569]: pgmap v14667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:03.816 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:03 smithi104 ceph-mon[36624]: pgmap v14667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:03.890 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:03 smithi081 ceph-mon[31453]: pgmap v14667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:06.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:06 smithi104 ceph-mon[36624]: pgmap v14668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:06.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:06 smithi081 ceph-mon[31453]: pgmap v14668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:06.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:06 smithi153 ceph-mon[36569]: pgmap v14668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:08.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:08 smithi081 ceph-mon[31453]: pgmap v14669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:08.540 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:08 smithi153 ceph-mon[36569]: pgmap v14669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:08.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:08 smithi104 ceph-mon[36624]: pgmap v14669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:10.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:10 smithi081 ceph-mon[31453]: pgmap v14670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:10.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:10 smithi153 ceph-mon[36569]: pgmap v14670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:10.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:10 smithi104 ceph-mon[36624]: pgmap v14670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:12.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:12 smithi081 ceph-mon[31453]: pgmap v14671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:12.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:12 smithi153 ceph-mon[36569]: pgmap v14671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:12.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:12 smithi104 ceph-mon[36624]: pgmap v14671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:13.489+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:51:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:13.489+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:51:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:13.489+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:51:14.343 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:14 smithi153 ceph-mon[36569]: pgmap v14672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:14.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:14 smithi104 ceph-mon[36624]: pgmap v14672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:14.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:14 smithi081 ceph-mon[31453]: pgmap v14672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:16.390 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:16 smithi081 ceph-mon[31453]: pgmap v14673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:16.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:16 smithi153 ceph-mon[36569]: pgmap v14673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:16.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:16 smithi104 ceph-mon[36624]: pgmap v14673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:18.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:18 smithi081 ceph-mon[31453]: pgmap v14674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:18.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:18 smithi153 ceph-mon[36569]: pgmap v14674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:18.566 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:18 smithi104 ceph-mon[36624]: pgmap v14674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:20.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:20 smithi081 ceph-mon[31453]: pgmap v14675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:20.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:20 smithi153 ceph-mon[36569]: pgmap v14675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:20.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:20 smithi104 ceph-mon[36624]: pgmap v14675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:22.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:22 smithi081 ceph-mon[31453]: pgmap v14676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:22.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:22 smithi153 ceph-mon[36569]: pgmap v14676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:22.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:22 smithi104 ceph-mon[36624]: pgmap v14676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:24.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:24 smithi081 ceph-mon[31453]: pgmap v14677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:24.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:24 smithi153 ceph-mon[36569]: pgmap v14677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:24.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:24 smithi104 ceph-mon[36624]: pgmap v14677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:26.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:26 smithi081 ceph-mon[31453]: pgmap v14678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:26.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:26 smithi153 ceph-mon[36569]: pgmap v14678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:26.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:26 smithi104 ceph-mon[36624]: pgmap v14678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:28.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:28 smithi081 ceph-mon[31453]: pgmap v14679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:28.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:28.491+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:51:28.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:28.491+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:51:28.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:28.491+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:51:28.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:28 smithi153 ceph-mon[36569]: pgmap v14679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:28.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:28 smithi104 ceph-mon[36624]: pgmap v14679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:30.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:30 smithi081 ceph-mon[31453]: pgmap v14680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:30.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:30 smithi153 ceph-mon[36569]: pgmap v14680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:30.570 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:30 smithi104 ceph-mon[36624]: pgmap v14680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:32.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:32 smithi081 ceph-mon[31453]: pgmap v14681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:32.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:32 smithi153 ceph-mon[36569]: pgmap v14681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:32.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:32 smithi104 ceph-mon[36624]: pgmap v14681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:34.391 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:34 smithi081 ceph-mon[31453]: pgmap v14682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:34.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:34 smithi153 ceph-mon[36569]: pgmap v14682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:34.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:34 smithi104 ceph-mon[36624]: pgmap v14682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:36.541 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:36 smithi153 ceph-mon[36569]: pgmap v14683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:36.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:36 smithi104 ceph-mon[36624]: pgmap v14683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:36.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:36 smithi081 ceph-mon[31453]: pgmap v14683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:38.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:38 smithi153 ceph-mon[36569]: pgmap v14684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:38.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:38 smithi104 ceph-mon[36624]: pgmap v14684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:38.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:38 smithi081 ceph-mon[31453]: pgmap v14684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:40.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:40 smithi153 ceph-mon[36569]: pgmap v14685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:40.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:40 smithi104 ceph-mon[36624]: pgmap v14685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:40.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:40 smithi081 ceph-mon[31453]: pgmap v14685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:42.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:42 smithi153 ceph-mon[36569]: pgmap v14686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:42.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:42 smithi104 ceph-mon[36624]: pgmap v14686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:42.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:42 smithi081 ceph-mon[31453]: pgmap v14686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:43.505+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:51:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:43.505+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:51:43.491 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:43.505+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:51:44.444 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:44 smithi081 ceph-mon[31453]: pgmap v14687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:44.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:44 smithi153 ceph-mon[36569]: pgmap v14687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:44.567 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:44 smithi104 ceph-mon[36624]: pgmap v14687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:46.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:46 smithi153 ceph-mon[36569]: pgmap v14688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:46.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:46 smithi104 ceph-mon[36624]: pgmap v14688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:46.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:46 smithi081 ceph-mon[31453]: pgmap v14688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:48.392 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:48 smithi081 ceph-mon[31453]: pgmap v14689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:48.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:48 smithi153 ceph-mon[36569]: pgmap v14689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:48.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:48 smithi104 ceph-mon[36624]: pgmap v14689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:50.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:50 smithi153 ceph-mon[36569]: pgmap v14690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:50.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:50 smithi104 ceph-mon[36624]: pgmap v14690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:50.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:50 smithi081 ceph-mon[31453]: pgmap v14690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:52.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:52 smithi153 ceph-mon[36569]: pgmap v14691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:52.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:52 smithi104 ceph-mon[36624]: pgmap v14691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:52.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:52 smithi081 ceph-mon[31453]: pgmap v14691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:54.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:54 smithi153 ceph-mon[36569]: pgmap v14692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:54.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:54 smithi104 ceph-mon[36624]: pgmap v14692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:54.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:54 smithi081 ceph-mon[31453]: pgmap v14692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:56.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:56 smithi153 ceph-mon[36569]: pgmap v14693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:56.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:56 smithi104 ceph-mon[36624]: pgmap v14693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:56.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:56 smithi081 ceph-mon[31453]: pgmap v14693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:58.513+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:51:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:58.513+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:51:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:51:58.513+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:51:58.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:51:58 smithi153 ceph-mon[36569]: pgmap v14694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:58.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:51:58 smithi104 ceph-mon[36624]: pgmap v14694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:51:58.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:51:58 smithi081 ceph-mon[31453]: pgmap v14694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:00.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:00 smithi153 ceph-mon[36569]: pgmap v14695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:00.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:00 smithi104 ceph-mon[36624]: pgmap v14695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:00.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:00 smithi081 ceph-mon[31453]: pgmap v14695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:01.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:01 smithi153 ceph-mon[36569]: pgmap v14696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:01.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:01 smithi104 ceph-mon[36624]: pgmap v14696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:01 smithi081 ceph-mon[31453]: pgmap v14696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:02.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:52:02.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:52:02.641 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:52:03.542 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:52:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:52:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:03 smithi153 ceph-mon[36569]: pgmap v14697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:03 smithi153 ceph-mon[36569]: pgmap v14698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:03.543 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:52:03.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:52:03.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:52:03.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:03 smithi104 ceph-mon[36624]: pgmap v14697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:03.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:03 smithi104 ceph-mon[36624]: pgmap v14698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:03.568 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:52:03.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:52:03.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:52:03.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:03 smithi081 ceph-mon[31453]: pgmap v14697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:03.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:03 smithi081 ceph-mon[31453]: pgmap v14698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:03.642 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:52:05.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:05 smithi081 ceph-mon[31453]: pgmap v14699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:06.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:05 smithi153 ceph-mon[36569]: pgmap v14699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:06.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:05 smithi104 ceph-mon[36624]: pgmap v14699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:07.892 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:07 smithi081 ceph-mon[31453]: pgmap v14700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:08.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:07 smithi153 ceph-mon[36569]: pgmap v14700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:08.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:07 smithi104 ceph-mon[36624]: pgmap v14700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:10.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:09 smithi153 ceph-mon[36569]: pgmap v14701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:10.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:09 smithi104 ceph-mon[36624]: pgmap v14701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:10.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:09 smithi081 ceph-mon[31453]: pgmap v14701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:12.042 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:11 smithi153 ceph-mon[36569]: pgmap v14702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:12.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:11 smithi104 ceph-mon[36624]: pgmap v14702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:12.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:11 smithi081 ceph-mon[31453]: pgmap v14702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:13.516+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:52:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:13.516+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:52:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:13.516+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:52:14.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:13 smithi153 ceph-mon[36569]: pgmap v14703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:14.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:13 smithi104 ceph-mon[36624]: pgmap v14703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:14.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:13 smithi081 ceph-mon[31453]: pgmap v14703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:15.945 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:15 smithi081 ceph-mon[31453]: pgmap v14704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:16.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:15 smithi153 ceph-mon[36569]: pgmap v14704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:16.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:15 smithi104 ceph-mon[36624]: pgmap v14704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:17.213 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:17.226+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:52:17.213 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:17.226+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:52:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:17.227+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 212 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:52:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:17.227+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 212 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:52:18.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:17 smithi153 ceph-mon[36569]: pgmap v14705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:18.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:17 smithi104 ceph-mon[36624]: pgmap v14705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:18.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:17 smithi081 ceph-mon[31453]: pgmap v14705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:20.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:19 smithi153 ceph-mon[36569]: pgmap v14706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:20.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:19 smithi104 ceph-mon[36624]: pgmap v14706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:20.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:19 smithi081 ceph-mon[31453]: pgmap v14706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:22.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:21 smithi153 ceph-mon[36569]: pgmap v14707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:22.068 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:21 smithi104 ceph-mon[36624]: pgmap v14707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:22.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:21 smithi081 ceph-mon[31453]: pgmap v14707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:24.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:23 smithi153 ceph-mon[36569]: pgmap v14708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:24.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:23 smithi104 ceph-mon[36624]: pgmap v14708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:24.142 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:23 smithi081 ceph-mon[31453]: pgmap v14708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:26.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:25 smithi153 ceph-mon[36569]: pgmap v14709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:26.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:25 smithi104 ceph-mon[36624]: pgmap v14709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:26.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:25 smithi081 ceph-mon[31453]: pgmap v14709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:28.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:27 smithi153 ceph-mon[36569]: pgmap v14710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:28.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:27 smithi104 ceph-mon[36624]: pgmap v14710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:28.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:27 smithi081 ceph-mon[31453]: pgmap v14710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:28.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:28.519+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:52:28.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:28.519+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:52:28.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:28.519+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:52:30.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:29 smithi153 ceph-mon[36569]: pgmap v14711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:30.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:29 smithi104 ceph-mon[36624]: pgmap v14711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:30.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:29 smithi081 ceph-mon[31453]: pgmap v14711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:32.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:31 smithi153 ceph-mon[36569]: pgmap v14712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:32.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:31 smithi104 ceph-mon[36624]: pgmap v14712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:32.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:31 smithi081 ceph-mon[31453]: pgmap v14712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:34.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:33 smithi153 ceph-mon[36569]: pgmap v14713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:34.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:33 smithi104 ceph-mon[36624]: pgmap v14713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:34.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:33 smithi081 ceph-mon[31453]: pgmap v14713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:36.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:35 smithi153 ceph-mon[36569]: pgmap v14714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:36.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:35 smithi104 ceph-mon[36624]: pgmap v14714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:36.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:35 smithi081 ceph-mon[31453]: pgmap v14714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:38.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:37 smithi153 ceph-mon[36569]: pgmap v14715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:38.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:37 smithi104 ceph-mon[36624]: pgmap v14715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:38.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:37 smithi081 ceph-mon[31453]: pgmap v14715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:40.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:39 smithi153 ceph-mon[36569]: pgmap v14716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:40.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:39 smithi104 ceph-mon[36624]: pgmap v14716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:40.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:39 smithi081 ceph-mon[31453]: pgmap v14716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:42.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:41 smithi153 ceph-mon[36569]: pgmap v14717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:42.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:41 smithi104 ceph-mon[36624]: pgmap v14717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:42.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:41 smithi081 ceph-mon[31453]: pgmap v14717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:43.525+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:52:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:43.525+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:52:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:43.525+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:52:44.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:43 smithi153 ceph-mon[36569]: pgmap v14718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:44.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:43 smithi104 ceph-mon[36624]: pgmap v14718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:44.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:43 smithi081 ceph-mon[31453]: pgmap v14718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:46.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:45 smithi153 ceph-mon[36569]: pgmap v14719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:46.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:45 smithi104 ceph-mon[36624]: pgmap v14719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:46.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:45 smithi081 ceph-mon[31453]: pgmap v14719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:48.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:47 smithi153 ceph-mon[36569]: pgmap v14720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:48.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:47 smithi104 ceph-mon[36624]: pgmap v14720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:48.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:47 smithi081 ceph-mon[31453]: pgmap v14720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:50.043 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:49 smithi153 ceph-mon[36569]: pgmap v14721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:50.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:49 smithi104 ceph-mon[36624]: pgmap v14721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:50.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:49 smithi081 ceph-mon[31453]: pgmap v14721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:52.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:51 smithi153 ceph-mon[36569]: pgmap v14722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:52.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:51 smithi104 ceph-mon[36624]: pgmap v14722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:52.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:51 smithi081 ceph-mon[31453]: pgmap v14722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:54.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:53 smithi153 ceph-mon[36569]: pgmap v14723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:54.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:53 smithi104 ceph-mon[36624]: pgmap v14723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:54.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:53 smithi081 ceph-mon[31453]: pgmap v14723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:56.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:55 smithi153 ceph-mon[36569]: pgmap v14724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:56.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:55 smithi104 ceph-mon[36624]: pgmap v14724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:56.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:55 smithi081 ceph-mon[31453]: pgmap v14724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:58.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:57 smithi153 ceph-mon[36569]: pgmap v14725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:58.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:57 smithi104 ceph-mon[36624]: pgmap v14725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:58.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:57 smithi081 ceph-mon[31453]: pgmap v14725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:52:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:58.540+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:52:58.528 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:58.540+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:52:58.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:52:58.540+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:00.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:52:59 smithi153 ceph-mon[36569]: pgmap v14726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:00.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:52:59 smithi104 ceph-mon[36624]: pgmap v14726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:00.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:52:59 smithi081 ceph-mon[31453]: pgmap v14726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:02.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:01 smithi153 ceph-mon[36569]: pgmap v14727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:02.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:01 smithi104 ceph-mon[36624]: pgmap v14727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:02.143 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:01 smithi081 ceph-mon[31453]: pgmap v14727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:03 smithi153 ceph-mon[36569]: pgmap v14728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:04.044 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:53:04.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:03 smithi104 ceph-mon[36624]: pgmap v14728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:04.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:53:04.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:03 smithi081 ceph-mon[31453]: pgmap v14728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:04.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:53:05.069 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:53:05.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:53:05.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:53:05.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:53:05.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:53:05.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:53:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:53:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:53:05.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:53:06.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:05 smithi104 ceph-mon[36624]: pgmap v14729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:06.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:05 smithi104 ceph-mon[36624]: pgmap v14730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:06.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:05 smithi081 ceph-mon[31453]: pgmap v14729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:06.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:05 smithi081 ceph-mon[31453]: pgmap v14730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:06.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:05 smithi153 ceph-mon[36569]: pgmap v14729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:06.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:05 smithi153 ceph-mon[36569]: pgmap v14730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:08.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:07 smithi104 ceph-mon[36624]: pgmap v14731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:08.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:07 smithi081 ceph-mon[31453]: pgmap v14731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:08.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:07 smithi153 ceph-mon[36569]: pgmap v14731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:10.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:09 smithi104 ceph-mon[36624]: pgmap v14732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:10.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:09 smithi081 ceph-mon[31453]: pgmap v14732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:10.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:09 smithi153 ceph-mon[36569]: pgmap v14732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:12.070 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:11 smithi104 ceph-mon[36624]: pgmap v14733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:12.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:11 smithi081 ceph-mon[31453]: pgmap v14733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:12.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:11 smithi153 ceph-mon[36569]: pgmap v14733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:13.556+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:13.556+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:13.544 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:13.556+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:53:14.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:13 smithi081 ceph-mon[31453]: pgmap v14734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:14.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:13 smithi153 ceph-mon[36569]: pgmap v14734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:14.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:13 smithi104 ceph-mon[36624]: pgmap v14734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:16.102 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:15 smithi153 ceph-mon[36569]: pgmap v14735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:16.123 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:15 smithi104 ceph-mon[36624]: pgmap v14735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:16.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:15 smithi081 ceph-mon[31453]: pgmap v14735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:18.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:17 smithi081 ceph-mon[31453]: pgmap v14736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:18.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:17 smithi153 ceph-mon[36569]: pgmap v14736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:18.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:17 smithi104 ceph-mon[36624]: pgmap v14736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:20.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:19 smithi081 ceph-mon[31453]: pgmap v14737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:20.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:19 smithi153 ceph-mon[36569]: pgmap v14737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:20.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:19 smithi104 ceph-mon[36624]: pgmap v14737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:22.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:21 smithi081 ceph-mon[31453]: pgmap v14738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:22.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:21 smithi153 ceph-mon[36569]: pgmap v14738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:22.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:21 smithi104 ceph-mon[36624]: pgmap v14738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:24.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:23 smithi081 ceph-mon[31453]: pgmap v14739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:24.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:23 smithi153 ceph-mon[36569]: pgmap v14739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:24.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:23 smithi104 ceph-mon[36624]: pgmap v14739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:26.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:25 smithi081 ceph-mon[31453]: pgmap v14740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:26.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:25 smithi153 ceph-mon[36569]: pgmap v14740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:26.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:25 smithi104 ceph-mon[36624]: pgmap v14740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:28.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:27 smithi081 ceph-mon[31453]: pgmap v14741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:28.294 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:27 smithi153 ceph-mon[36569]: pgmap v14741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:28.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:27 smithi104 ceph-mon[36624]: pgmap v14741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:28.552 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:28.564+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:28.564+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:28.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:28.564+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:53:30.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:29 smithi081 ceph-mon[31453]: pgmap v14742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:30.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:29 smithi153 ceph-mon[36569]: pgmap v14742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:30.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:29 smithi104 ceph-mon[36624]: pgmap v14742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:32.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:31 smithi081 ceph-mon[31453]: pgmap v14743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:32.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:31 smithi153 ceph-mon[36569]: pgmap v14743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:32.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:31 smithi104 ceph-mon[36624]: pgmap v14743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:34.144 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:33 smithi081 ceph-mon[31453]: pgmap v14744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:34.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:33 smithi153 ceph-mon[36569]: pgmap v14744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:34.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:33 smithi104 ceph-mon[36624]: pgmap v14744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:36.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:35 smithi081 ceph-mon[31453]: pgmap v14745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:36.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:35 smithi153 ceph-mon[36569]: pgmap v14745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:36.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:35 smithi104 ceph-mon[36624]: pgmap v14745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:38.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:37 smithi081 ceph-mon[31453]: pgmap v14746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:38.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:37 smithi153 ceph-mon[36569]: pgmap v14746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:38.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:37 smithi104 ceph-mon[36624]: pgmap v14746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:40.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:39 smithi081 ceph-mon[31453]: pgmap v14747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:40.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:39 smithi153 ceph-mon[36569]: pgmap v14747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:40.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:39 smithi104 ceph-mon[36624]: pgmap v14747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:42.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:41 smithi153 ceph-mon[36569]: pgmap v14748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:42.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:41 smithi104 ceph-mon[36624]: pgmap v14748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:42.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:41 smithi081 ceph-mon[31453]: pgmap v14748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:43.569+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:43.569+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:43.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:43.569+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:53:44.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:43 smithi153 ceph-mon[36569]: pgmap v14749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:44.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:43 smithi104 ceph-mon[36624]: pgmap v14749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:44.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:43 smithi081 ceph-mon[31453]: pgmap v14749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:46.197 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:45 smithi081 ceph-mon[31453]: pgmap v14750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:46.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:45 smithi153 ceph-mon[36569]: pgmap v14750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:46.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:45 smithi104 ceph-mon[36624]: pgmap v14750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:48.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:47 smithi153 ceph-mon[36569]: pgmap v14751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:48.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:47 smithi104 ceph-mon[36624]: pgmap v14751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:48.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:47 smithi081 ceph-mon[31453]: pgmap v14751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:50.297 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:49 smithi153 ceph-mon[36569]: pgmap v14752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:50.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:49 smithi104 ceph-mon[36624]: pgmap v14752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:50.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:49 smithi081 ceph-mon[31453]: pgmap v14752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:52.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:51 smithi081 ceph-mon[31453]: pgmap v14753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:52.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:51 smithi153 ceph-mon[36569]: pgmap v14753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:52.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:51 smithi104 ceph-mon[36624]: pgmap v14753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:54.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:53 smithi153 ceph-mon[36569]: pgmap v14754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:54.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:53 smithi104 ceph-mon[36624]: pgmap v14754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:54.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:53 smithi081 ceph-mon[31453]: pgmap v14754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:56.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:55 smithi153 ceph-mon[36569]: pgmap v14755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:56.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:55 smithi104 ceph-mon[36624]: pgmap v14755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:56.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:55 smithi081 ceph-mon[31453]: pgmap v14755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:58.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:57 smithi153 ceph-mon[36569]: pgmap v14756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:58.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:57 smithi104 ceph-mon[36624]: pgmap v14756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:58.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:57 smithi081 ceph-mon[31453]: pgmap v14756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:53:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:58.584+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:58.584+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:53:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:53:58.584+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:54:00.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:53:59 smithi153 ceph-mon[36569]: pgmap v14757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:00.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:53:59 smithi104 ceph-mon[36624]: pgmap v14757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:00.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:53:59 smithi081 ceph-mon[31453]: pgmap v14757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:02.295 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:01 smithi153 ceph-mon[36569]: pgmap v14758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:02.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:01 smithi104 ceph-mon[36624]: pgmap v14758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:02.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:01 smithi081 ceph-mon[31453]: pgmap v14758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:04.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:03 smithi153 ceph-mon[36569]: pgmap v14759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:04.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:03 smithi104 ceph-mon[36624]: pgmap v14759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:04.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:03 smithi081 ceph-mon[31453]: pgmap v14759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:05.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:54:05.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:54:05.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:54:06.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:06 smithi104 ceph-mon[36624]: pgmap v14760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:06.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:06 smithi081 ceph-mon[31453]: pgmap v14760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:06.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:06 smithi153 ceph-mon[36569]: pgmap v14760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:06.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:06.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:08 smithi153 ceph-mon[36569]: pgmap v14761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:08.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:08 smithi104 ceph-mon[36624]: pgmap v14761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:08.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:08 smithi081 ceph-mon[31453]: pgmap v14761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:08.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:08.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:10.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:10 smithi153 ceph-mon[36569]: pgmap v14762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:10.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:10.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:10.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:10 smithi104 ceph-mon[36624]: pgmap v14762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:10.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:10.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:10.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:10 smithi081 ceph-mon[31453]: pgmap v14762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:10.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:10.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:11.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:54:11.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:54:11.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:11.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:54:11.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:54:11.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:11.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:54:11.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:54:11.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:54:12.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:12 smithi153 ceph-mon[36569]: pgmap v14763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:12.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:12 smithi153 ceph-mon[36569]: pgmap v14764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:12.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:12 smithi153 ceph-mon[36569]: pgmap v14765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:12.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:12 smithi104 ceph-mon[36624]: pgmap v14763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:12.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:12 smithi104 ceph-mon[36624]: pgmap v14764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:12.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:12 smithi104 ceph-mon[36624]: pgmap v14765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:12.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:12 smithi081 ceph-mon[31453]: pgmap v14763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:12.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:12 smithi081 ceph-mon[31453]: pgmap v14764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:12.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:12 smithi081 ceph-mon[31453]: pgmap v14765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:13.599+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:54:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:13.599+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:54:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:13.599+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:54:14.296 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:14 smithi153 ceph-mon[36569]: pgmap v14766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:14.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:14 smithi104 ceph-mon[36624]: pgmap v14766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:14.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:14 smithi081 ceph-mon[31453]: pgmap v14766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:16.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:16 smithi104 ceph-mon[36624]: pgmap v14767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:16.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:16 smithi081 ceph-mon[31453]: pgmap v14767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:16.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:16 smithi153 ceph-mon[36569]: pgmap v14767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:18.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:18 smithi104 ceph-mon[36624]: pgmap v14768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:18.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:18 smithi081 ceph-mon[31453]: pgmap v14768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:18.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:18 smithi153 ceph-mon[36569]: pgmap v14768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:20.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:20 smithi104 ceph-mon[36624]: pgmap v14769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:20.395 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:20 smithi081 ceph-mon[31453]: pgmap v14769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:20.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:20 smithi153 ceph-mon[36569]: pgmap v14769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:22.322 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:22 smithi104 ceph-mon[36624]: pgmap v14770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:22.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:22 smithi081 ceph-mon[31453]: pgmap v14770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:22.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:22 smithi153 ceph-mon[36569]: pgmap v14770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:24.321 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:24 smithi104 ceph-mon[36624]: pgmap v14771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:24.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:24 smithi081 ceph-mon[31453]: pgmap v14771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:24.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:24 smithi153 ceph-mon[36569]: pgmap v14771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:26.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:26 smithi081 ceph-mon[31453]: pgmap v14772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:26.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:26 smithi153 ceph-mon[36569]: pgmap v14772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:26.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:26 smithi104 ceph-mon[36624]: pgmap v14772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:28.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:28 smithi081 ceph-mon[31453]: pgmap v14773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:28.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:28 smithi153 ceph-mon[36569]: pgmap v14773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:28.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:28 smithi104 ceph-mon[36624]: pgmap v14773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:28.611+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:54:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:28.611+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:54:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:28.611+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:54:30.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:30 smithi081 ceph-mon[31453]: pgmap v14774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:30.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:30 smithi153 ceph-mon[36569]: pgmap v14774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:30.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:30 smithi104 ceph-mon[36624]: pgmap v14774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:32.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:32 smithi081 ceph-mon[31453]: pgmap v14775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:32.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:32 smithi153 ceph-mon[36569]: pgmap v14775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:32.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:32 smithi104 ceph-mon[36624]: pgmap v14775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:34.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:34 smithi081 ceph-mon[31453]: pgmap v14776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:34.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:34 smithi153 ceph-mon[36569]: pgmap v14776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:34.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:34 smithi104 ceph-mon[36624]: pgmap v14776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:36.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:36 smithi081 ceph-mon[31453]: pgmap v14777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:36.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:36 smithi153 ceph-mon[36569]: pgmap v14777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:36.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:36 smithi104 ceph-mon[36624]: pgmap v14777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:38.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:38 smithi081 ceph-mon[31453]: pgmap v14778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:38.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:38 smithi153 ceph-mon[36569]: pgmap v14778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:38.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:38 smithi104 ceph-mon[36624]: pgmap v14778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:40.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:40 smithi081 ceph-mon[31453]: pgmap v14779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:40.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:40 smithi153 ceph-mon[36569]: pgmap v14779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:40.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:40 smithi104 ceph-mon[36624]: pgmap v14779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:42.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:42 smithi081 ceph-mon[31453]: pgmap v14780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:42.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:42 smithi153 ceph-mon[36569]: pgmap v14780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:42.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:42 smithi104 ceph-mon[36624]: pgmap v14780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:43.618+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:54:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:43.618+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:54:43.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:43.618+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:54:44.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:44 smithi081 ceph-mon[31453]: pgmap v14781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:44.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:44 smithi153 ceph-mon[36569]: pgmap v14781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:44.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:44 smithi104 ceph-mon[36624]: pgmap v14781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:46.374 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:46 smithi104 ceph-mon[36624]: pgmap v14782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:46.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:46 smithi081 ceph-mon[31453]: pgmap v14782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:46.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:46 smithi153 ceph-mon[36569]: pgmap v14782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:48.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:48 smithi081 ceph-mon[31453]: pgmap v14783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:48.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:48 smithi153 ceph-mon[36569]: pgmap v14783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:48.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:48 smithi104 ceph-mon[36624]: pgmap v14783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:50.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:50 smithi081 ceph-mon[31453]: pgmap v14784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:50.546 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:50 smithi153 ceph-mon[36569]: pgmap v14784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:50.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:50 smithi104 ceph-mon[36624]: pgmap v14784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:52.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:52 smithi081 ceph-mon[31453]: pgmap v14785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:52.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:52 smithi153 ceph-mon[36569]: pgmap v14785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:52.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:52 smithi104 ceph-mon[36624]: pgmap v14785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:54.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:54 smithi153 ceph-mon[36569]: pgmap v14786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:54.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:54 smithi104 ceph-mon[36624]: pgmap v14786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:54.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:54 smithi081 ceph-mon[31453]: pgmap v14786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:56.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:56 smithi153 ceph-mon[36569]: pgmap v14787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:56.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:56 smithi104 ceph-mon[36624]: pgmap v14787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:56.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:56 smithi081 ceph-mon[31453]: pgmap v14787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:58.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:54:58 smithi153 ceph-mon[36569]: pgmap v14788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:58.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:54:58 smithi104 ceph-mon[36624]: pgmap v14788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:54:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:58.634+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:54:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:58.634+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:54:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:54:58.634+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:54:58.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:54:58 smithi081 ceph-mon[31453]: pgmap v14788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:00.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:00 smithi153 ceph-mon[36569]: pgmap v14789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:00.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:00 smithi104 ceph-mon[36624]: pgmap v14789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:00.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:00 smithi081 ceph-mon[31453]: pgmap v14789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:02.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:02 smithi153 ceph-mon[36569]: pgmap v14790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:02.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:02 smithi104 ceph-mon[36624]: pgmap v14790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:02.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:02 smithi081 ceph-mon[31453]: pgmap v14790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:04.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:04 smithi153 ceph-mon[36569]: pgmap v14791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:04.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:04 smithi104 ceph-mon[36624]: pgmap v14791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:04.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:04 smithi081 ceph-mon[31453]: pgmap v14791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:06.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:06 smithi153 ceph-mon[36569]: pgmap v14792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:06.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:06 smithi104 ceph-mon[36624]: pgmap v14792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:06.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:06 smithi081 ceph-mon[31453]: pgmap v14792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:08.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:08 smithi153 ceph-mon[36569]: pgmap v14793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:08.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:08 smithi104 ceph-mon[36624]: pgmap v14793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:08.646 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:08 smithi081 ceph-mon[31453]: pgmap v14793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:10.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:10 smithi153 ceph-mon[36569]: pgmap v14794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:10.572 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:10 smithi104 ceph-mon[36624]: pgmap v14794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:10.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:10 smithi081 ceph-mon[31453]: pgmap v14794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:12.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:12 smithi153 ceph-mon[36569]: pgmap v14795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:12.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:55:12.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:12.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:12.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:12 smithi104 ceph-mon[36624]: pgmap v14795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:12.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:55:12.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:12.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:12.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:12 smithi081 ceph-mon[31453]: pgmap v14795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:12.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:55:12.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:12.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:13.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:55:13.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:55:13.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:13.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:55:13.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:55:13.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:13.644+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:55:13.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:13.644+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:55:13.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:13.644+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:55:13.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:55:13.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:55:13.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:55:14.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:14 smithi153 ceph-mon[36569]: pgmap v14796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:14.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:14 smithi153 ceph-mon[36569]: pgmap v14797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:14.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:14 smithi104 ceph-mon[36624]: pgmap v14796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:14.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:14 smithi104 ceph-mon[36624]: pgmap v14797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:14.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:14 smithi081 ceph-mon[31453]: pgmap v14796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:14.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:14 smithi081 ceph-mon[31453]: pgmap v14797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:16.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:16 smithi153 ceph-mon[36569]: pgmap v14798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:16.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:16 smithi104 ceph-mon[36624]: pgmap v14798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:16.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:16 smithi081 ceph-mon[31453]: pgmap v14798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:17.255+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:55:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:17.255+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:55:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:17.256+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 213 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:55:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:17.256+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 213 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:55:18.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:18 smithi153 ceph-mon[36569]: pgmap v14799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:18.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:18 smithi104 ceph-mon[36624]: pgmap v14799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:18.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:18 smithi081 ceph-mon[31453]: pgmap v14799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:20.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:20 smithi153 ceph-mon[36569]: pgmap v14800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:20.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:20 smithi104 ceph-mon[36624]: pgmap v14800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:20.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:20 smithi081 ceph-mon[31453]: pgmap v14800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:22.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:22 smithi153 ceph-mon[36569]: pgmap v14801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:22.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:22 smithi104 ceph-mon[36624]: pgmap v14801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:22.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:22 smithi081 ceph-mon[31453]: pgmap v14801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:24.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:24 smithi153 ceph-mon[36569]: pgmap v14802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:24.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:24 smithi104 ceph-mon[36624]: pgmap v14802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:24.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:24 smithi081 ceph-mon[31453]: pgmap v14802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:26.397 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:26 smithi081 ceph-mon[31453]: pgmap v14803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:26.547 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:26 smithi153 ceph-mon[36569]: pgmap v14803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:26.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:26 smithi104 ceph-mon[36624]: pgmap v14803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:27.573 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:27 smithi104 ceph-mon[36624]: pgmap v14804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:27.647 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:27 smithi081 ceph-mon[31453]: pgmap v14804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:27.797 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:27 smithi153 ceph-mon[36569]: pgmap v14804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:28.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:28.660+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:55:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:28.660+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:55:28.652 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:28.660+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:55:30.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:29 smithi153 ceph-mon[36569]: pgmap v14805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:30.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:29 smithi104 ceph-mon[36624]: pgmap v14805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:30.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:29 smithi081 ceph-mon[31453]: pgmap v14805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:32.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:31 smithi153 ceph-mon[36569]: pgmap v14806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:32.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:31 smithi104 ceph-mon[36624]: pgmap v14806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:32.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:31 smithi081 ceph-mon[31453]: pgmap v14806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:34.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:33 smithi153 ceph-mon[36569]: pgmap v14807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:34.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:33 smithi104 ceph-mon[36624]: pgmap v14807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:34.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:33 smithi081 ceph-mon[31453]: pgmap v14807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:36.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:35 smithi153 ceph-mon[36569]: pgmap v14808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:36.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:35 smithi104 ceph-mon[36624]: pgmap v14808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:36.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:35 smithi081 ceph-mon[31453]: pgmap v14808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:38.047 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:37 smithi153 ceph-mon[36569]: pgmap v14809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:38.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:37 smithi104 ceph-mon[36624]: pgmap v14809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:38.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:37 smithi081 ceph-mon[31453]: pgmap v14809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:40.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:39 smithi153 ceph-mon[36569]: pgmap v14810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:40.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:39 smithi104 ceph-mon[36624]: pgmap v14810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:40.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:39 smithi081 ceph-mon[31453]: pgmap v14810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:42.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:41 smithi153 ceph-mon[36569]: pgmap v14811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:42.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:41 smithi104 ceph-mon[36624]: pgmap v14811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:42.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:41 smithi081 ceph-mon[31453]: pgmap v14811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:43.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:43.672+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:55:43.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:43.672+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:55:43.664 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:43.672+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:55:44.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:43 smithi153 ceph-mon[36569]: pgmap v14812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:44.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:43 smithi104 ceph-mon[36624]: pgmap v14812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:44.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:43 smithi081 ceph-mon[31453]: pgmap v14812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:46.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:45 smithi153 ceph-mon[36569]: pgmap v14813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:46.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:45 smithi104 ceph-mon[36624]: pgmap v14813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:46.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:45 smithi081 ceph-mon[31453]: pgmap v14813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:48.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:47 smithi153 ceph-mon[36569]: pgmap v14814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:48.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:47 smithi104 ceph-mon[36624]: pgmap v14814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:48.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:47 smithi081 ceph-mon[31453]: pgmap v14814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:50.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:49 smithi153 ceph-mon[36569]: pgmap v14815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:50.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:49 smithi104 ceph-mon[36624]: pgmap v14815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:50.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:49 smithi081 ceph-mon[31453]: pgmap v14815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:52.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:51 smithi153 ceph-mon[36569]: pgmap v14816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:52.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:51 smithi104 ceph-mon[36624]: pgmap v14816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:52.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:51 smithi081 ceph-mon[31453]: pgmap v14816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:54.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:53 smithi153 ceph-mon[36569]: pgmap v14817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:54.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:53 smithi104 ceph-mon[36624]: pgmap v14817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:54.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:53 smithi081 ceph-mon[31453]: pgmap v14817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:56.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:55 smithi153 ceph-mon[36569]: pgmap v14818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:56.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:55 smithi104 ceph-mon[36624]: pgmap v14818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:56.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:55 smithi081 ceph-mon[31453]: pgmap v14818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:58.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:57 smithi153 ceph-mon[36569]: pgmap v14819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:58.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:57 smithi104 ceph-mon[36624]: pgmap v14819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:58.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:57 smithi081 ceph-mon[31453]: pgmap v14819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:55:58.676 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:58.683+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:55:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:58.684+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:55:58.677 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:55:58.684+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:56:00.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:55:59 smithi153 ceph-mon[36569]: pgmap v14820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:00.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:55:59 smithi104 ceph-mon[36624]: pgmap v14820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:00.147 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:55:59 smithi081 ceph-mon[31453]: pgmap v14820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:02.048 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:01 smithi153 ceph-mon[36569]: pgmap v14821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:02.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:01 smithi104 ceph-mon[36624]: pgmap v14821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:02.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:01 smithi081 ceph-mon[31453]: pgmap v14821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:04.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:03 smithi104 ceph-mon[36624]: pgmap v14822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:04.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:03 smithi081 ceph-mon[31453]: pgmap v14822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:04.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:03 smithi153 ceph-mon[36569]: pgmap v14822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:06.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:05 smithi104 ceph-mon[36624]: pgmap v14823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:06.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:05 smithi081 ceph-mon[31453]: pgmap v14823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:06.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:05 smithi153 ceph-mon[36569]: pgmap v14823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:08.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:07 smithi104 ceph-mon[36624]: pgmap v14824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:08.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:07 smithi081 ceph-mon[31453]: pgmap v14824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:08.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:07 smithi153 ceph-mon[36569]: pgmap v14824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:10.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:09 smithi104 ceph-mon[36624]: pgmap v14825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:10.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:09 smithi081 ceph-mon[31453]: pgmap v14825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:10.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:09 smithi153 ceph-mon[36569]: pgmap v14825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:12.074 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:11 smithi104 ceph-mon[36624]: pgmap v14826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:12.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:11 smithi081 ceph-mon[31453]: pgmap v14826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:12.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:11 smithi153 ceph-mon[36569]: pgmap v14826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:13.700+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:56:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:13.700+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:56:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:13.700+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:56:14.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:13 smithi081 ceph-mon[31453]: pgmap v14827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:14.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:56:14.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:13 smithi153 ceph-mon[36569]: pgmap v14827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:14.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:56:14.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:13 smithi104 ceph-mon[36624]: pgmap v14827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:14.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:56:15.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:56:15.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:56:15.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:14 smithi081 ceph-mon[31453]: pgmap v14828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:15.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:14 smithi081 ceph-mon[31453]: pgmap v14829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:15.148 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:56:15.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:56:15.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:56:15.298 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:14 smithi153 ceph-mon[36569]: pgmap v14828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:15.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:14 smithi153 ceph-mon[36569]: pgmap v14829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:15.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:56:15.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:56:15.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:56:15.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:14 smithi104 ceph-mon[36624]: pgmap v14828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:15.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:14 smithi104 ceph-mon[36624]: pgmap v14829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:15.324 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:56:17.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:17 smithi153 ceph-mon[36569]: pgmap v14830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:17.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:17 smithi104 ceph-mon[36624]: pgmap v14830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:17.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:17 smithi081 ceph-mon[31453]: pgmap v14830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:19.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:19 smithi153 ceph-mon[36569]: pgmap v14831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:19.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:19 smithi104 ceph-mon[36624]: pgmap v14831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:19.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:19 smithi081 ceph-mon[31453]: pgmap v14831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:20.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:20 smithi104 ceph-mon[36624]: pgmap v14832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:20.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:20 smithi081 ceph-mon[31453]: pgmap v14832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:20.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:20 smithi153 ceph-mon[36569]: pgmap v14832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:23.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:23 smithi153 ceph-mon[36569]: pgmap v14833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:23.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:23 smithi104 ceph-mon[36624]: pgmap v14833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:23.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:23 smithi081 ceph-mon[31453]: pgmap v14833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:24.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:24 smithi104 ceph-mon[36624]: pgmap v14834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:24.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:24 smithi081 ceph-mon[31453]: pgmap v14834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:24.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:24 smithi153 ceph-mon[36569]: pgmap v14834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:27.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:27 smithi153 ceph-mon[36569]: pgmap v14835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:27.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:27 smithi104 ceph-mon[36624]: pgmap v14835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:27.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:27 smithi081 ceph-mon[31453]: pgmap v14835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:28.716+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:56:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:28.716+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:56:28.708 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:28.716+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:56:28.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:28 smithi153 ceph-mon[36569]: pgmap v14836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:28.824 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:28 smithi104 ceph-mon[36624]: pgmap v14836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:28.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:28 smithi081 ceph-mon[31453]: pgmap v14836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:31.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:31 smithi153 ceph-mon[36569]: pgmap v14837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:31.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:31 smithi104 ceph-mon[36624]: pgmap v14837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:31.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:31 smithi081 ceph-mon[31453]: pgmap v14837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:32.548 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:32 smithi153 ceph-mon[36569]: pgmap v14838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:32.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:32 smithi104 ceph-mon[36624]: pgmap v14838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:32.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:32 smithi081 ceph-mon[31453]: pgmap v14838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:35.549 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:35 smithi153 ceph-mon[36569]: pgmap v14839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:35.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:35 smithi104 ceph-mon[36624]: pgmap v14839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:35.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:35 smithi081 ceph-mon[31453]: pgmap v14839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:36.550 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:36 smithi153 ceph-mon[36569]: pgmap v14840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:36.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:36 smithi104 ceph-mon[36624]: pgmap v14840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:36.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:36 smithi081 ceph-mon[31453]: pgmap v14840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:38.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:38 smithi104 ceph-mon[36624]: pgmap v14841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:38.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:38 smithi081 ceph-mon[31453]: pgmap v14841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:38.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:38 smithi153 ceph-mon[36569]: pgmap v14841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:40.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:40 smithi104 ceph-mon[36624]: pgmap v14842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:40.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:40 smithi081 ceph-mon[31453]: pgmap v14842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:40.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:40 smithi153 ceph-mon[36569]: pgmap v14842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:42.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:42 smithi104 ceph-mon[36624]: pgmap v14843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:42.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:42 smithi081 ceph-mon[31453]: pgmap v14843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:42.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:42 smithi153 ceph-mon[36569]: pgmap v14843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:43.726+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:56:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:43.726+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:56:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:43.726+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:56:44.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:44 smithi104 ceph-mon[36624]: pgmap v14844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:44.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:44 smithi081 ceph-mon[31453]: pgmap v14844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:44.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:44 smithi153 ceph-mon[36569]: pgmap v14844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:46.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:46 smithi104 ceph-mon[36624]: pgmap v14845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:46.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:46 smithi081 ceph-mon[31453]: pgmap v14845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:46.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:46 smithi153 ceph-mon[36569]: pgmap v14845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:48.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:48 smithi104 ceph-mon[36624]: pgmap v14846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:48.601 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:48 smithi153 ceph-mon[36569]: pgmap v14846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:48.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:48 smithi081 ceph-mon[31453]: pgmap v14846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:50.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:50 smithi104 ceph-mon[36624]: pgmap v14847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:50.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:50 smithi081 ceph-mon[31453]: pgmap v14847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:50.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:50 smithi153 ceph-mon[36569]: pgmap v14847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:52.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:52 smithi104 ceph-mon[36624]: pgmap v14848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:52.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:52 smithi081 ceph-mon[31453]: pgmap v14848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:52.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:52 smithi153 ceph-mon[36569]: pgmap v14848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:54.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:54 smithi104 ceph-mon[36624]: pgmap v14849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:54.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:54 smithi081 ceph-mon[31453]: pgmap v14849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:54.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:54 smithi153 ceph-mon[36569]: pgmap v14849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:56.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:56 smithi104 ceph-mon[36624]: pgmap v14850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:56.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:56 smithi081 ceph-mon[31453]: pgmap v14850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:56.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:56 smithi153 ceph-mon[36569]: pgmap v14850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:58.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:56:58 smithi104 ceph-mon[36624]: pgmap v14851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:58.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:56:58 smithi081 ceph-mon[31453]: pgmap v14851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:56:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:58.736+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:56:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:58.736+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:56:58.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:56:58.736+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:56:58.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:56:58 smithi153 ceph-mon[36569]: pgmap v14851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:00.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:00 smithi104 ceph-mon[36624]: pgmap v14852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:00.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:00 smithi081 ceph-mon[31453]: pgmap v14852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:00.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:00 smithi153 ceph-mon[36569]: pgmap v14852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:02.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:02 smithi104 ceph-mon[36624]: pgmap v14853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:02.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:02 smithi081 ceph-mon[31453]: pgmap v14853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:02.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:02 smithi153 ceph-mon[36569]: pgmap v14853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:04.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:04 smithi104 ceph-mon[36624]: pgmap v14854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:04.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:04 smithi081 ceph-mon[31453]: pgmap v14854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:04.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:04 smithi153 ceph-mon[36569]: pgmap v14854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:06.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:06 smithi104 ceph-mon[36624]: pgmap v14855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:06.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:06 smithi081 ceph-mon[31453]: pgmap v14855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:06.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:06 smithi153 ceph-mon[36569]: pgmap v14855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:08.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:08 smithi104 ceph-mon[36624]: pgmap v14856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:08.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:08 smithi081 ceph-mon[31453]: pgmap v14856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:08.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:08 smithi153 ceph-mon[36569]: pgmap v14856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:10.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:10 smithi104 ceph-mon[36624]: pgmap v14857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:10.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:10 smithi081 ceph-mon[31453]: pgmap v14857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:10.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:10 smithi153 ceph-mon[36569]: pgmap v14857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:12.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:12 smithi104 ceph-mon[36624]: pgmap v14858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:12.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:12 smithi081 ceph-mon[31453]: pgmap v14858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:12.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:12 smithi153 ceph-mon[36569]: pgmap v14858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:13.748+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:57:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:13.748+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:57:13.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:13.748+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:57:14.399 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:14 smithi081 ceph-mon[31453]: pgmap v14859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:14.575 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:14 smithi104 ceph-mon[36624]: pgmap v14859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:14.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:14 smithi153 ceph-mon[36569]: pgmap v14859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:15.648 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:57:15.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:57:15.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:57:16.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:57:16.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:57:16.649 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:16 smithi081 ceph-mon[31453]: pgmap v14860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:16.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:16 smithi081 ceph-mon[31453]: pgmap v14861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:16.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:57:16.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:57:16.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:57:16.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:16 smithi153 ceph-mon[36569]: pgmap v14860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:16.799 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:16 smithi153 ceph-mon[36569]: pgmap v14861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:16.800 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:57:16.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:57:16.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:57:16.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:16 smithi104 ceph-mon[36624]: pgmap v14860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:16.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:16 smithi104 ceph-mon[36624]: pgmap v14861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:16.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:57:19.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:18 smithi104 ceph-mon[36624]: pgmap v14862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:19.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:18 smithi081 ceph-mon[31453]: pgmap v14862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:19.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:18 smithi153 ceph-mon[36569]: pgmap v14862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:21.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:20 smithi104 ceph-mon[36624]: pgmap v14863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:21.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:20 smithi081 ceph-mon[31453]: pgmap v14863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:21.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:20 smithi153 ceph-mon[36569]: pgmap v14863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:23.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:22 smithi081 ceph-mon[31453]: pgmap v14864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:23.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:22 smithi153 ceph-mon[36569]: pgmap v14864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:23.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:22 smithi104 ceph-mon[36624]: pgmap v14864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:25.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:24 smithi081 ceph-mon[31453]: pgmap v14865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:25.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:24 smithi153 ceph-mon[36569]: pgmap v14865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:25.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:24 smithi104 ceph-mon[36624]: pgmap v14865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:27.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:26 smithi081 ceph-mon[31453]: pgmap v14866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:27.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:26 smithi153 ceph-mon[36569]: pgmap v14866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:27.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:26 smithi104 ceph-mon[36624]: pgmap v14866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:28.762+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:57:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:28.762+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:57:28.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:28.762+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:57:28.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:28 smithi104 ceph-mon[36624]: pgmap v14867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:28.899 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:28 smithi081 ceph-mon[31453]: pgmap v14867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:29.049 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:28 smithi153 ceph-mon[36569]: pgmap v14867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:31.075 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:30 smithi104 ceph-mon[36624]: pgmap v14868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:31.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:30 smithi081 ceph-mon[31453]: pgmap v14868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:31.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:30 smithi153 ceph-mon[36569]: pgmap v14868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:33.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:32 smithi081 ceph-mon[31453]: pgmap v14869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:33.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:32 smithi153 ceph-mon[36569]: pgmap v14869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:33.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:32 smithi104 ceph-mon[36624]: pgmap v14869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:35.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:34 smithi081 ceph-mon[31453]: pgmap v14870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:35.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:34 smithi153 ceph-mon[36569]: pgmap v14870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:35.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:34 smithi104 ceph-mon[36624]: pgmap v14870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:37.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:36 smithi081 ceph-mon[31453]: pgmap v14871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:37.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:36 smithi153 ceph-mon[36569]: pgmap v14871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:37.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:36 smithi104 ceph-mon[36624]: pgmap v14871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:39.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:38 smithi081 ceph-mon[31453]: pgmap v14872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:39.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:38 smithi153 ceph-mon[36569]: pgmap v14872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:39.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:38 smithi104 ceph-mon[36624]: pgmap v14872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:41.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:40 smithi081 ceph-mon[31453]: pgmap v14873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:41.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:40 smithi153 ceph-mon[36569]: pgmap v14873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:41.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:40 smithi104 ceph-mon[36624]: pgmap v14873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:43.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:42 smithi081 ceph-mon[31453]: pgmap v14874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:43.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:42 smithi153 ceph-mon[36569]: pgmap v14874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:43.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:42 smithi104 ceph-mon[36624]: pgmap v14874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:43.771+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:57:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:43.771+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:57:43.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:43.771+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:57:45.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:44 smithi081 ceph-mon[31453]: pgmap v14875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:45.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:44 smithi153 ceph-mon[36569]: pgmap v14875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:45.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:44 smithi104 ceph-mon[36624]: pgmap v14875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:47.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:46 smithi081 ceph-mon[31453]: pgmap v14876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:47.299 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:46 smithi153 ceph-mon[36569]: pgmap v14876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:47.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:46 smithi104 ceph-mon[36624]: pgmap v14876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:49.128 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:48 smithi104 ceph-mon[36624]: pgmap v14877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:49.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:48 smithi081 ceph-mon[31453]: pgmap v14877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:49.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:48 smithi153 ceph-mon[36569]: pgmap v14877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:51.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:50 smithi081 ceph-mon[31453]: pgmap v14878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:51.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:50 smithi153 ceph-mon[36569]: pgmap v14878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:51.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:50 smithi104 ceph-mon[36624]: pgmap v14878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:53.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:52 smithi081 ceph-mon[31453]: pgmap v14879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:53.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:52 smithi153 ceph-mon[36569]: pgmap v14879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:53.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:52 smithi104 ceph-mon[36624]: pgmap v14879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:55.149 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:54 smithi081 ceph-mon[31453]: pgmap v14880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:55.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:54 smithi153 ceph-mon[36569]: pgmap v14880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:55.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:54 smithi104 ceph-mon[36624]: pgmap v14880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:57.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:56 smithi081 ceph-mon[31453]: pgmap v14881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:57.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:56 smithi153 ceph-mon[36569]: pgmap v14881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:57.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:56 smithi104 ceph-mon[36624]: pgmap v14881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:58.784+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:57:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:58.784+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:57:58.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:57:58.784+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:57:59.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:57:58 smithi153 ceph-mon[36569]: pgmap v14882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:59.325 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:57:58 smithi104 ceph-mon[36624]: pgmap v14882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:57:59.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:57:58 smithi081 ceph-mon[31453]: pgmap v14882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:01.217 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:00 smithi081 ceph-mon[31453]: pgmap v14883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:01.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:00 smithi153 ceph-mon[36569]: pgmap v14883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:01.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:00 smithi104 ceph-mon[36624]: pgmap v14883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:03.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:02 smithi153 ceph-mon[36569]: pgmap v14884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:03.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:02 smithi104 ceph-mon[36624]: pgmap v14884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:03.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:02 smithi081 ceph-mon[31453]: pgmap v14884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:05.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:04 smithi153 ceph-mon[36569]: pgmap v14885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:05.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:04 smithi104 ceph-mon[36624]: pgmap v14885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:05.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:04 smithi081 ceph-mon[31453]: pgmap v14885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:07.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:06 smithi153 ceph-mon[36569]: pgmap v14886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:07.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:06 smithi104 ceph-mon[36624]: pgmap v14886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:07.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:06 smithi081 ceph-mon[31453]: pgmap v14886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:09.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:08 smithi153 ceph-mon[36569]: pgmap v14887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:09.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:08 smithi104 ceph-mon[36624]: pgmap v14887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:09.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:08 smithi081 ceph-mon[31453]: pgmap v14887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:11.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:10 smithi153 ceph-mon[36569]: pgmap v14888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:11.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:10 smithi104 ceph-mon[36624]: pgmap v14888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:11.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:10 smithi081 ceph-mon[31453]: pgmap v14888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:13.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:12 smithi153 ceph-mon[36569]: pgmap v14889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:13.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:12 smithi104 ceph-mon[36624]: pgmap v14889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:13.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:12 smithi081 ceph-mon[31453]: pgmap v14889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:13.791 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:13.797+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:58:13.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:13.797+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:58:13.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:13.797+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:58:15.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:14 smithi153 ceph-mon[36569]: pgmap v14890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:15.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:14 smithi104 ceph-mon[36624]: pgmap v14890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:15.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:14 smithi081 ceph-mon[31453]: pgmap v14890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:17.285+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:58:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:17.285+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:58:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:17.285+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 214 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T04:58:17.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:17.285+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 214 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T04:58:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:17.285+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:58:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:17.285+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:58:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:17.286+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 215 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T04:58:17.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:17.286+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 215 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T04:58:17.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:16 smithi153 ceph-mon[36569]: pgmap v14891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:17.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:58:17.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:16 smithi104 ceph-mon[36624]: pgmap v14891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:17.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:58:17.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:16 smithi081 ceph-mon[31453]: pgmap v14891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:17.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:58:18.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:58:18.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:58:18.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:58:18.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:58:18.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:58:18.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:58:18.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:58:18.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:58:18.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:58:19.227 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:18 smithi104 ceph-mon[36624]: pgmap v14892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:19.227 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:18 smithi104 ceph-mon[36624]: pgmap v14893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:19.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:18 smithi153 ceph-mon[36569]: pgmap v14892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:19.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:18 smithi153 ceph-mon[36569]: pgmap v14893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:19.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:18 smithi081 ceph-mon[31453]: pgmap v14892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:19.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:18 smithi081 ceph-mon[31453]: pgmap v14893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:21.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:20 smithi153 ceph-mon[36569]: pgmap v14894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:21.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:20 smithi104 ceph-mon[36624]: pgmap v14894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:21.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:20 smithi081 ceph-mon[31453]: pgmap v14894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:23.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:22 smithi153 ceph-mon[36569]: pgmap v14895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:23.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:22 smithi104 ceph-mon[36624]: pgmap v14895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:23.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:22 smithi081 ceph-mon[31453]: pgmap v14895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:25.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:24 smithi153 ceph-mon[36569]: pgmap v14896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:25.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:24 smithi104 ceph-mon[36624]: pgmap v14896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:25.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:24 smithi081 ceph-mon[31453]: pgmap v14896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:27.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:26 smithi153 ceph-mon[36569]: pgmap v14897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:27.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:26 smithi104 ceph-mon[36624]: pgmap v14897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:27.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:26 smithi081 ceph-mon[31453]: pgmap v14897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:28.808+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:58:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:28.808+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:58:28.803 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:28.808+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:58:29.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:28 smithi153 ceph-mon[36569]: pgmap v14898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:29.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:28 smithi104 ceph-mon[36624]: pgmap v14898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:29.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:28 smithi081 ceph-mon[31453]: pgmap v14898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:31.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:31 smithi153 ceph-mon[36569]: pgmap v14899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:31.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:31 smithi104 ceph-mon[36624]: pgmap v14899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:31.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:31 smithi081 ceph-mon[31453]: pgmap v14899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:33.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:33 smithi153 ceph-mon[36569]: pgmap v14900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:33.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:33 smithi104 ceph-mon[36624]: pgmap v14900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:33.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:33 smithi081 ceph-mon[31453]: pgmap v14900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:35.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:35 smithi153 ceph-mon[36569]: pgmap v14901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:35.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:35 smithi104 ceph-mon[36624]: pgmap v14901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:35.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:35 smithi081 ceph-mon[31453]: pgmap v14901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:37.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:37 smithi153 ceph-mon[36569]: pgmap v14902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:37.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:37 smithi104 ceph-mon[36624]: pgmap v14902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:37.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:37 smithi081 ceph-mon[31453]: pgmap v14902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:39.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:39 smithi153 ceph-mon[36569]: pgmap v14903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:39.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:39 smithi104 ceph-mon[36624]: pgmap v14903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:39.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:39 smithi081 ceph-mon[31453]: pgmap v14903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:41.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:41 smithi153 ceph-mon[36569]: pgmap v14904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:41.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:41 smithi104 ceph-mon[36624]: pgmap v14904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:41.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:41 smithi081 ceph-mon[31453]: pgmap v14904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:43.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:43 smithi153 ceph-mon[36569]: pgmap v14905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:43.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:43 smithi104 ceph-mon[36624]: pgmap v14905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:43.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:43 smithi081 ceph-mon[31453]: pgmap v14905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:43.818 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:43.824+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:58:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:43.824+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:58:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:43.824+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:58:45.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:45 smithi153 ceph-mon[36569]: pgmap v14906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:45.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:45 smithi104 ceph-mon[36624]: pgmap v14906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:45.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:45 smithi081 ceph-mon[31453]: pgmap v14906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:47.298 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:47 smithi081 ceph-mon[31453]: pgmap v14907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:47.300 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:47 smithi153 ceph-mon[36569]: pgmap v14907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:47.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:47 smithi104 ceph-mon[36624]: pgmap v14907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:49.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:49 smithi153 ceph-mon[36569]: pgmap v14908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:49.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:49 smithi104 ceph-mon[36624]: pgmap v14908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:49.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:49 smithi081 ceph-mon[31453]: pgmap v14908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:51.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:51 smithi153 ceph-mon[36569]: pgmap v14909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:51.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:51 smithi104 ceph-mon[36624]: pgmap v14909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:51.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:51 smithi081 ceph-mon[31453]: pgmap v14909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:53.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:53 smithi104 ceph-mon[36624]: pgmap v14910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:53.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:53 smithi081 ceph-mon[31453]: pgmap v14910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:53.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:53 smithi153 ceph-mon[36569]: pgmap v14910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:55.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:55 smithi104 ceph-mon[36624]: pgmap v14911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:55.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:55 smithi081 ceph-mon[31453]: pgmap v14911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:55.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:55 smithi153 ceph-mon[36569]: pgmap v14911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:57.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:57 smithi081 ceph-mon[31453]: pgmap v14912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:57.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:57 smithi153 ceph-mon[36569]: pgmap v14912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:57.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:57 smithi104 ceph-mon[36624]: pgmap v14912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:58.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:58.840+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:58:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:58.840+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:58:58.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:58:58.840+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:58:59.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:58:59 smithi081 ceph-mon[31453]: pgmap v14913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:59.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:58:59 smithi153 ceph-mon[36569]: pgmap v14913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:58:59.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:58:59 smithi104 ceph-mon[36624]: pgmap v14913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:01.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:01 smithi081 ceph-mon[31453]: pgmap v14914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:01.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:01 smithi153 ceph-mon[36569]: pgmap v14914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:01.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:01 smithi104 ceph-mon[36624]: pgmap v14914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:03.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:03 smithi081 ceph-mon[31453]: pgmap v14915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:03.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:03 smithi153 ceph-mon[36569]: pgmap v14915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:03.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:03 smithi104 ceph-mon[36624]: pgmap v14915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:05.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:05 smithi081 ceph-mon[31453]: pgmap v14916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:05.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:05 smithi153 ceph-mon[36569]: pgmap v14916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:05.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:05 smithi104 ceph-mon[36624]: pgmap v14916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:07.400 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:07 smithi081 ceph-mon[31453]: pgmap v14917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:07.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:07 smithi153 ceph-mon[36569]: pgmap v14917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:07.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:07 smithi104 ceph-mon[36624]: pgmap v14917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:09.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:09 smithi081 ceph-mon[31453]: pgmap v14918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:09.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:09 smithi153 ceph-mon[36569]: pgmap v14918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:09.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:09 smithi104 ceph-mon[36624]: pgmap v14918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:11.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:11 smithi081 ceph-mon[31453]: pgmap v14919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:11.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:11 smithi153 ceph-mon[36569]: pgmap v14919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:11.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:11 smithi104 ceph-mon[36624]: pgmap v14919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:13.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:13 smithi081 ceph-mon[31453]: pgmap v14920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:13.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:13 smithi153 ceph-mon[36569]: pgmap v14920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:13.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:13 smithi104 ceph-mon[36624]: pgmap v14920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:13.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:13.849+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:59:13.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:13.849+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:59:13.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:13.849+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:59:15.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:15 smithi081 ceph-mon[31453]: pgmap v14921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:15.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:15 smithi153 ceph-mon[36569]: pgmap v14921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:15.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:15 smithi104 ceph-mon[36624]: pgmap v14921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:17.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:17 smithi081 ceph-mon[31453]: pgmap v14922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:17.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:17 smithi153 ceph-mon[36569]: pgmap v14922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:17.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:17 smithi104 ceph-mon[36624]: pgmap v14922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:18.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:59:18.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:59:18.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T04:59:19.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:19 smithi081 ceph-mon[31453]: pgmap v14923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:19.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:59:19.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:59:19.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:59:19.453 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:19 smithi153 ceph-mon[36569]: pgmap v14923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:19.453 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:59:19.453 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:59:19.453 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:59:19.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:19 smithi104 ceph-mon[36624]: pgmap v14923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:19.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T04:59:19.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T04:59:19.478 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T04:59:20.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:20 smithi153 ceph-mon[36569]: pgmap v14924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:20.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:20 smithi153 ceph-mon[36569]: pgmap v14925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:20.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:20 smithi104 ceph-mon[36624]: pgmap v14924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:20.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:20 smithi104 ceph-mon[36624]: pgmap v14925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:20.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:20 smithi081 ceph-mon[31453]: pgmap v14924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:20.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:20 smithi081 ceph-mon[31453]: pgmap v14925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:22.454 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:22 smithi081 ceph-mon[31453]: pgmap v14926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:22.551 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:22 smithi153 ceph-mon[36569]: pgmap v14926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:22.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:22 smithi104 ceph-mon[36624]: pgmap v14926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:23.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:23 smithi081 ceph-mon[31453]: pgmap v14927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:24.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:23 smithi153 ceph-mon[36569]: pgmap v14927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:24.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:23 smithi104 ceph-mon[36624]: pgmap v14927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:26.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:25 smithi153 ceph-mon[36569]: pgmap v14928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:26.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:25 smithi104 ceph-mon[36624]: pgmap v14928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:26.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:25 smithi081 ceph-mon[31453]: pgmap v14928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:28.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:27 smithi153 ceph-mon[36569]: pgmap v14929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:28.326 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:27 smithi104 ceph-mon[36624]: pgmap v14929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:28.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:27 smithi081 ceph-mon[31453]: pgmap v14929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:28.854+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:59:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:28.854+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:59:28.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:28.854+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:59:30.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:29 smithi153 ceph-mon[36569]: pgmap v14930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:30.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:29 smithi104 ceph-mon[36624]: pgmap v14930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:30.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:29 smithi081 ceph-mon[31453]: pgmap v14930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:32.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:31 smithi153 ceph-mon[36569]: pgmap v14931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:32.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:31 smithi104 ceph-mon[36624]: pgmap v14931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:32.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:31 smithi081 ceph-mon[31453]: pgmap v14931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:34.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:33 smithi081 ceph-mon[31453]: pgmap v14932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:34.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:33 smithi153 ceph-mon[36569]: pgmap v14932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:34.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:33 smithi104 ceph-mon[36624]: pgmap v14932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:36.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:35 smithi153 ceph-mon[36569]: pgmap v14933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:36.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:35 smithi104 ceph-mon[36624]: pgmap v14933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:36.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:35 smithi081 ceph-mon[31453]: pgmap v14933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:38.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:37 smithi153 ceph-mon[36569]: pgmap v14934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:38.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:37 smithi104 ceph-mon[36624]: pgmap v14934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:38.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:37 smithi081 ceph-mon[31453]: pgmap v14934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:40.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:39 smithi153 ceph-mon[36569]: pgmap v14935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:40.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:39 smithi104 ceph-mon[36624]: pgmap v14935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:40.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:39 smithi081 ceph-mon[31453]: pgmap v14935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:42.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:41 smithi153 ceph-mon[36569]: pgmap v14936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:42.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:41 smithi104 ceph-mon[36624]: pgmap v14936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:42.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:41 smithi081 ceph-mon[31453]: pgmap v14936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:43.868+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:59:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:43.868+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:59:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:43.868+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T04:59:44.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:43 smithi153 ceph-mon[36569]: pgmap v14937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:44.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:43 smithi104 ceph-mon[36624]: pgmap v14937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:44.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:43 smithi081 ceph-mon[31453]: pgmap v14937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:46.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:46 smithi153 ceph-mon[36569]: pgmap v14938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:46.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:46 smithi104 ceph-mon[36624]: pgmap v14938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:46.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:46 smithi081 ceph-mon[31453]: pgmap v14938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:48.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:48 smithi153 ceph-mon[36569]: pgmap v14939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:48.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:48 smithi104 ceph-mon[36624]: pgmap v14939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:48.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:48 smithi081 ceph-mon[31453]: pgmap v14939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:50.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:50 smithi153 ceph-mon[36569]: pgmap v14940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:50.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:50 smithi104 ceph-mon[36624]: pgmap v14940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:50.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:50 smithi081 ceph-mon[31453]: pgmap v14940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:52.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:52 smithi153 ceph-mon[36569]: pgmap v14941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:52.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:52 smithi104 ceph-mon[36624]: pgmap v14941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:52.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:52 smithi081 ceph-mon[31453]: pgmap v14941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:54.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:54 smithi153 ceph-mon[36569]: pgmap v14942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:54.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:54 smithi104 ceph-mon[36624]: pgmap v14942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:54.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:54 smithi081 ceph-mon[31453]: pgmap v14942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:56.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:56 smithi153 ceph-mon[36569]: pgmap v14943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:56.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:56 smithi104 ceph-mon[36624]: pgmap v14943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:56.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:56 smithi081 ceph-mon[31453]: pgmap v14943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:58.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 04:59:58 smithi153 ceph-mon[36569]: pgmap v14944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:58.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 04:59:58 smithi104 ceph-mon[36624]: pgmap v14944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:58.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 04:59:58 smithi081 ceph-mon[31453]: pgmap v14944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T04:59:58.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:58.871+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T04:59:58.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:58.871+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T04:59:58.867 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T04:59:58.871+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:00:00.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:00 smithi153 ceph-mon[36569]: pgmap v14945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:00.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:00:00.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:00 smithi104 ceph-mon[36624]: pgmap v14945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:00.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:00:00.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:00 smithi081 ceph-mon[31453]: pgmap v14945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:00.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:00:02.301 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:02 smithi153 ceph-mon[36569]: pgmap v14946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:02.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:02 smithi104 ceph-mon[36624]: pgmap v14946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:02.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:02 smithi081 ceph-mon[31453]: pgmap v14946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:03.901 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:03 smithi081 ceph-mon[31453]: pgmap v14947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:04.051 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:03 smithi153 ceph-mon[36569]: pgmap v14947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:04.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:03 smithi104 ceph-mon[36624]: pgmap v14947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:06.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:05 smithi153 ceph-mon[36569]: pgmap v14948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:06.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:05 smithi104 ceph-mon[36624]: pgmap v14948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:06.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:05 smithi081 ceph-mon[31453]: pgmap v14948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:08.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:07 smithi153 ceph-mon[36569]: pgmap v14949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:08.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:07 smithi104 ceph-mon[36624]: pgmap v14949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:08.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:07 smithi081 ceph-mon[31453]: pgmap v14949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:10.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:09 smithi153 ceph-mon[36569]: pgmap v14950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:10.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:09 smithi104 ceph-mon[36624]: pgmap v14950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:10.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:09 smithi081 ceph-mon[31453]: pgmap v14950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:12.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:11 smithi153 ceph-mon[36569]: pgmap v14951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:12.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:11 smithi104 ceph-mon[36624]: pgmap v14951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:12.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:11 smithi081 ceph-mon[31453]: pgmap v14951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:13.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:13.876+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:00:13.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:13.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:00:13.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:13.876+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:00:14.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:13 smithi153 ceph-mon[36569]: pgmap v14952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:14.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:13 smithi104 ceph-mon[36624]: pgmap v14952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:14.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:13 smithi081 ceph-mon[31453]: pgmap v14952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:16.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:15 smithi153 ceph-mon[36569]: pgmap v14953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:16.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:15 smithi104 ceph-mon[36624]: pgmap v14953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:16.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:15 smithi081 ceph-mon[31453]: pgmap v14953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:18.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:18 smithi153 ceph-mon[36569]: pgmap v14954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:18.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:18 smithi104 ceph-mon[36624]: pgmap v14954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:18.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:18 smithi081 ceph-mon[31453]: pgmap v14954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:20.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: pgmap v14955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: pgmap v14955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: pgmap v14955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:20.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:00:20.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:20.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:00:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:00:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:00:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:00:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:00:21.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:00:21.303 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:00:21.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:00:21.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:00:21.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:00:21.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:00:21.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:00:21.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:00:21.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:00:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:00:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:00:21.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:00:22.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:22 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:00:22.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:22 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:00:22.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:22 smithi153 ceph-mon[36569]: pgmap v14956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:22.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:22 smithi153 ceph-mon[36569]: pgmap v14957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:22.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:22 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:00:22.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:22 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:00:22.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:22 smithi081 ceph-mon[31453]: pgmap v14956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:22.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:22 smithi081 ceph-mon[31453]: pgmap v14957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:22.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:22 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:00:22.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:22 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:00:22.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:22 smithi104 ceph-mon[36624]: pgmap v14956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:22.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:22 smithi104 ceph-mon[36624]: pgmap v14957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:24.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:24 smithi153 ceph-mon[36569]: pgmap v14958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:24.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:24 smithi104 ceph-mon[36624]: pgmap v14958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:24.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:24 smithi081 ceph-mon[31453]: pgmap v14958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:26.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:26 smithi153 ceph-mon[36569]: pgmap v14959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:26.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:26 smithi104 ceph-mon[36624]: pgmap v14959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:26.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:26 smithi081 ceph-mon[31453]: pgmap v14959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:28.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:28 smithi153 ceph-mon[36569]: pgmap v14960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:28.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:28 smithi104 ceph-mon[36624]: pgmap v14960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:28.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:28 smithi081 ceph-mon[31453]: pgmap v14960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:28.893+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:00:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:28.893+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:00:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:28.893+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:00:30.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:30 smithi153 ceph-mon[36569]: pgmap v14961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:30.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:30 smithi104 ceph-mon[36624]: pgmap v14961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:30.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:30 smithi081 ceph-mon[31453]: pgmap v14961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:32.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:32 smithi153 ceph-mon[36569]: pgmap v14962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:32.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:32 smithi104 ceph-mon[36624]: pgmap v14962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:32.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:32 smithi081 ceph-mon[31453]: pgmap v14962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:34.302 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:34 smithi153 ceph-mon[36569]: pgmap v14963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:34.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:34 smithi104 ceph-mon[36624]: pgmap v14963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:34.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:34 smithi081 ceph-mon[31453]: pgmap v14963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:36.152 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:36 smithi081 ceph-mon[31453]: pgmap v14964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:36.327 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:36 smithi104 ceph-mon[36624]: pgmap v14964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:36.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:36 smithi153 ceph-mon[36569]: pgmap v14964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:38.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:38 smithi104 ceph-mon[36624]: pgmap v14965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:38.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:38 smithi081 ceph-mon[31453]: pgmap v14965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:38.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:38 smithi153 ceph-mon[36569]: pgmap v14965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:40.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:40 smithi104 ceph-mon[36624]: pgmap v14966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:40.401 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:40 smithi081 ceph-mon[31453]: pgmap v14966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:40.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:40 smithi153 ceph-mon[36569]: pgmap v14966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:42.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:42 smithi104 ceph-mon[36624]: pgmap v14967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:42.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:42 smithi081 ceph-mon[31453]: pgmap v14967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:42.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:42 smithi153 ceph-mon[36569]: pgmap v14967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:43.904+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:00:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:43.904+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:00:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:43.904+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:00:44.328 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:44 smithi104 ceph-mon[36624]: pgmap v14968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:44.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:44 smithi081 ceph-mon[31453]: pgmap v14968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:44.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:44 smithi153 ceph-mon[36569]: pgmap v14968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:46.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:46 smithi081 ceph-mon[31453]: pgmap v14969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:46.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:46 smithi153 ceph-mon[36569]: pgmap v14969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:46.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:46 smithi104 ceph-mon[36624]: pgmap v14969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:48.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:48 smithi081 ceph-mon[31453]: pgmap v14970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:48.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:48 smithi153 ceph-mon[36569]: pgmap v14970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:48.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:48 smithi104 ceph-mon[36624]: pgmap v14970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:50.354 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:50 smithi153 ceph-mon[36569]: pgmap v14971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:50.380 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:50 smithi104 ceph-mon[36624]: pgmap v14971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:50.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:50 smithi081 ceph-mon[31453]: pgmap v14971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:52.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:52 smithi081 ceph-mon[31453]: pgmap v14972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:52.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:52 smithi153 ceph-mon[36569]: pgmap v14972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:52.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:52 smithi104 ceph-mon[36624]: pgmap v14972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:54.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:54 smithi081 ceph-mon[31453]: pgmap v14973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:54.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:54 smithi153 ceph-mon[36569]: pgmap v14973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:54.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:54 smithi104 ceph-mon[36624]: pgmap v14973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:56.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:56 smithi081 ceph-mon[31453]: pgmap v14974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:56.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:56 smithi153 ceph-mon[36569]: pgmap v14974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:56.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:56 smithi104 ceph-mon[36624]: pgmap v14974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:58.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:00:58 smithi081 ceph-mon[31453]: pgmap v14975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:58.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:00:58 smithi153 ceph-mon[36569]: pgmap v14975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:58.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:00:58 smithi104 ceph-mon[36624]: pgmap v14975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:00:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:58.918+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:00:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:58.918+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:00:58.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:00:58.918+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:00.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:00 smithi081 ceph-mon[31453]: pgmap v14976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:00.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:00 smithi153 ceph-mon[36569]: pgmap v14976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:00.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:00 smithi104 ceph-mon[36624]: pgmap v14976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:02.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:02 smithi081 ceph-mon[31453]: pgmap v14977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:02.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:02 smithi153 ceph-mon[36569]: pgmap v14977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:02.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:02 smithi104 ceph-mon[36624]: pgmap v14977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:04.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:04 smithi081 ceph-mon[31453]: pgmap v14978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:04.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:04 smithi153 ceph-mon[36569]: pgmap v14978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:04.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:04 smithi104 ceph-mon[36624]: pgmap v14978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:06.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:06 smithi081 ceph-mon[31453]: pgmap v14979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:06.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:06 smithi153 ceph-mon[36569]: pgmap v14979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:06.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:06 smithi104 ceph-mon[36624]: pgmap v14979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:08.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:08 smithi081 ceph-mon[31453]: pgmap v14980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:08.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:08 smithi153 ceph-mon[36569]: pgmap v14980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:08.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:08 smithi104 ceph-mon[36624]: pgmap v14980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:10.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:10 smithi081 ceph-mon[31453]: pgmap v14981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:10.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:10 smithi153 ceph-mon[36569]: pgmap v14981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:10.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:10 smithi104 ceph-mon[36624]: pgmap v14981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:12.162 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:12.166+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 msgr2=0x7f16e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000463) for more than 900000000 us, fault. 2024-04-13T05:01:12.162 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:12.166+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c6a0 tx=0x7f16f4044000 comp rx=0 tx=0).stop 2024-04-13T05:01:12.162 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:12.166+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 msgr2=0x7f16e80a4d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T05:01:12.162 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:12.166+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T05:01:12.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:12.166+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T05:01:12.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:12.167+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T05:01:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:12.168+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3d80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f16d40038e0 tx=0x7f16f40bf000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T05:01:12.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:12 smithi081 ceph-mon[31453]: pgmap v14982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:12.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:12 smithi153 ceph-mon[36569]: pgmap v14982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:12.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:12 smithi104 ceph-mon[36624]: pgmap v14982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:13.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:13.928+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:13.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:13.928+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:13.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:13.928+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:01:14.402 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:14 smithi081 ceph-mon[31453]: pgmap v14983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:14.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:14 smithi153 ceph-mon[36569]: pgmap v14983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:14.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:14 smithi104 ceph-mon[36624]: pgmap v14983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:16.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:16 smithi153 ceph-mon[36569]: pgmap v14984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:16.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:16 smithi104 ceph-mon[36624]: pgmap v14984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:16.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:16 smithi081 ceph-mon[31453]: pgmap v14984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:17.315+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:01:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:17.315+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:01:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:17.316+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 216 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:01:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:17.316+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 216 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:01:18.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:18 smithi153 ceph-mon[36569]: pgmap v14985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:18.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:18 smithi104 ceph-mon[36624]: pgmap v14985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:18.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:18 smithi081 ceph-mon[31453]: pgmap v14985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:20.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:20 smithi153 ceph-mon[36569]: pgmap v14986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:20.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:20 smithi104 ceph-mon[36624]: pgmap v14986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:20.638 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:20 smithi081 ceph-mon[31453]: pgmap v14986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:21.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:01:21.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:01:21.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:01:22.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:22 smithi153 ceph-mon[36569]: pgmap v14987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:01:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:01:22.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:01:22.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:22 smithi104 ceph-mon[36624]: pgmap v14987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:22.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:01:22.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:01:22.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:01:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:22 smithi081 ceph-mon[31453]: pgmap v14987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:22.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:01:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:01:22.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:01:23.552 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:23 smithi153 ceph-mon[36569]: pgmap v14988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:23.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:23 smithi153 ceph-mon[36569]: pgmap v14989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:23.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:23 smithi104 ceph-mon[36624]: pgmap v14988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:23.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:23 smithi104 ceph-mon[36624]: pgmap v14989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:23.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:23 smithi081 ceph-mon[31453]: pgmap v14988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:23.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:23 smithi081 ceph-mon[31453]: pgmap v14989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:25.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:25 smithi153 ceph-mon[36569]: pgmap v14990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:25.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:25 smithi104 ceph-mon[36624]: pgmap v14990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:25.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:25 smithi081 ceph-mon[31453]: pgmap v14990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:27.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:27 smithi153 ceph-mon[36569]: pgmap v14991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:27.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:27 smithi104 ceph-mon[36624]: pgmap v14991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:27.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:27 smithi081 ceph-mon[31453]: pgmap v14991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:28.935+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:28.936+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:28.936+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:01:29.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:29 smithi153 ceph-mon[36569]: pgmap v14992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:29.605 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:29 smithi104 ceph-mon[36624]: pgmap v14992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:29.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:29 smithi081 ceph-mon[31453]: pgmap v14992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:31.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:31 smithi153 ceph-mon[36569]: pgmap v14993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:31.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:31 smithi104 ceph-mon[36624]: pgmap v14993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:31.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:31 smithi081 ceph-mon[31453]: pgmap v14993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:33.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:33 smithi153 ceph-mon[36569]: pgmap v14994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:33 smithi104 ceph-mon[36624]: pgmap v14994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:33.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:33 smithi081 ceph-mon[31453]: pgmap v14994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:35.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:35 smithi153 ceph-mon[36569]: pgmap v14995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:35.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:35 smithi104 ceph-mon[36624]: pgmap v14995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:35 smithi081 ceph-mon[31453]: pgmap v14995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:37.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:37 smithi153 ceph-mon[36569]: pgmap v14996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:37.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:37 smithi104 ceph-mon[36624]: pgmap v14996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:37.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:37 smithi081 ceph-mon[31453]: pgmap v14996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:39.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:39 smithi153 ceph-mon[36569]: pgmap v14997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:39.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:39 smithi104 ceph-mon[36624]: pgmap v14997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:39.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:39 smithi081 ceph-mon[31453]: pgmap v14997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:41.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:41 smithi153 ceph-mon[36569]: pgmap v14998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:41.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:41 smithi104 ceph-mon[36624]: pgmap v14998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:41.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:41 smithi081 ceph-mon[31453]: pgmap v14998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:43.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:43 smithi153 ceph-mon[36569]: pgmap v14999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:43.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:43 smithi104 ceph-mon[36624]: pgmap v14999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:43.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:43 smithi081 ceph-mon[31453]: pgmap v14999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:43.947 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:43.951+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:43.951+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:43.951+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:01:45.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:45 smithi153 ceph-mon[36569]: pgmap v15000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:45.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:45 smithi104 ceph-mon[36624]: pgmap v15000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:45.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:45 smithi081 ceph-mon[31453]: pgmap v15000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:47 smithi153 ceph-mon[36569]: pgmap v15001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:47.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:47 smithi104 ceph-mon[36624]: pgmap v15001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:47.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:47 smithi081 ceph-mon[31453]: pgmap v15001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:49.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:49 smithi153 ceph-mon[36569]: pgmap v15002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:49.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:49 smithi104 ceph-mon[36624]: pgmap v15002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:49.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:49 smithi081 ceph-mon[31453]: pgmap v15002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:51.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:51 smithi153 ceph-mon[36569]: pgmap v15003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:51.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:51 smithi104 ceph-mon[36624]: pgmap v15003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:51.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:51 smithi081 ceph-mon[31453]: pgmap v15003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:53.456 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:53 smithi081 ceph-mon[31453]: pgmap v15004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:53.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:53 smithi153 ceph-mon[36569]: pgmap v15004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:53.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:53 smithi104 ceph-mon[36624]: pgmap v15004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:55.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:55 smithi153 ceph-mon[36569]: pgmap v15005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:55.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:55 smithi104 ceph-mon[36624]: pgmap v15005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:55.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:55 smithi081 ceph-mon[31453]: pgmap v15005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:57.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:57 smithi153 ceph-mon[36569]: pgmap v15006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:57.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:57 smithi104 ceph-mon[36624]: pgmap v15006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:57.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:57 smithi081 ceph-mon[31453]: pgmap v15006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:58.954+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:58.954+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:01:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:01:58.954+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:01:59.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:01:59 smithi153 ceph-mon[36569]: pgmap v15007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:59.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:01:59 smithi104 ceph-mon[36624]: pgmap v15007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:01:59.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:01:59 smithi081 ceph-mon[31453]: pgmap v15007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:01.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:01 smithi153 ceph-mon[36569]: pgmap v15008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:01.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:01 smithi104 ceph-mon[36624]: pgmap v15008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:01.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:01 smithi081 ceph-mon[31453]: pgmap v15008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:03.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:03 smithi153 ceph-mon[36569]: pgmap v15009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:03.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:03 smithi104 ceph-mon[36624]: pgmap v15009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:03 smithi081 ceph-mon[31453]: pgmap v15009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:05.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:05 smithi104 ceph-mon[36624]: pgmap v15010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:05.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:05 smithi081 ceph-mon[31453]: pgmap v15010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:05 smithi153 ceph-mon[36569]: pgmap v15010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:07.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:07 smithi104 ceph-mon[36624]: pgmap v15011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:07.652 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:07 smithi081 ceph-mon[31453]: pgmap v15011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:07 smithi153 ceph-mon[36569]: pgmap v15011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:09.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:09 smithi104 ceph-mon[36624]: pgmap v15012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:09.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:09 smithi081 ceph-mon[31453]: pgmap v15012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:09.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:09 smithi153 ceph-mon[36569]: pgmap v15012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:11.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:11 smithi104 ceph-mon[36624]: pgmap v15013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:11.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:11 smithi081 ceph-mon[31453]: pgmap v15013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:11.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:11 smithi153 ceph-mon[36569]: pgmap v15013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:13.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:13 smithi104 ceph-mon[36624]: pgmap v15014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:13.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:13 smithi081 ceph-mon[31453]: pgmap v15014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:13.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:13 smithi153 ceph-mon[36569]: pgmap v15014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:13.969+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:02:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:13.969+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:02:13.966 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:13.969+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:02:15.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:15 smithi081 ceph-mon[31453]: pgmap v15015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:15.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:15 smithi153 ceph-mon[36569]: pgmap v15015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:15.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:15 smithi104 ceph-mon[36624]: pgmap v15015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:17.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:17 smithi081 ceph-mon[31453]: pgmap v15016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:17.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:17 smithi153 ceph-mon[36569]: pgmap v15016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:17.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:17 smithi104 ceph-mon[36624]: pgmap v15016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:19.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:19 smithi081 ceph-mon[31453]: pgmap v15017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:19.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:19 smithi153 ceph-mon[36569]: pgmap v15017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:19.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:19 smithi104 ceph-mon[36624]: pgmap v15017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:20.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:20 smithi081 ceph-mon[31453]: pgmap v15018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:20.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:20 smithi153 ceph-mon[36569]: pgmap v15018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:20.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:20 smithi104 ceph-mon[36624]: pgmap v15018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:23.506 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:23 smithi153 ceph-mon[36569]: pgmap v15019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:23.506 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:02:23.532 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:23 smithi104 ceph-mon[36624]: pgmap v15019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:23.532 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:02:23.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:23 smithi081 ceph-mon[31453]: pgmap v15019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:23.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:02:24.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:02:24.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:02:24.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:02:24.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:02:24.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:02:24.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:02:24.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:02:24.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:02:24.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:02:25.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:25 smithi153 ceph-mon[36569]: pgmap v15020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:25.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:25 smithi153 ceph-mon[36569]: pgmap v15021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:25.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:25 smithi104 ceph-mon[36624]: pgmap v15020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:25.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:25 smithi104 ceph-mon[36624]: pgmap v15021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:25.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:25 smithi081 ceph-mon[31453]: pgmap v15020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:25.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:25 smithi081 ceph-mon[31453]: pgmap v15021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:27.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:27 smithi153 ceph-mon[36569]: pgmap v15022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:27.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:27 smithi104 ceph-mon[36624]: pgmap v15022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:27.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:27 smithi081 ceph-mon[31453]: pgmap v15022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:28.976+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:02:28.973 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:28.976+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:02:28.974 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:28.976+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:02:29.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:29 smithi153 ceph-mon[36569]: pgmap v15023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:29.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:29 smithi104 ceph-mon[36624]: pgmap v15023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:29.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:29 smithi081 ceph-mon[31453]: pgmap v15023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:31.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:31 smithi153 ceph-mon[36569]: pgmap v15024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:31.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:31 smithi104 ceph-mon[36624]: pgmap v15024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:31.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:31 smithi081 ceph-mon[31453]: pgmap v15024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:33.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:33 smithi153 ceph-mon[36569]: pgmap v15025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:33.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:33 smithi104 ceph-mon[36624]: pgmap v15025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:33.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:33 smithi081 ceph-mon[31453]: pgmap v15025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:35.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:35 smithi153 ceph-mon[36569]: pgmap v15026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:35.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:35 smithi104 ceph-mon[36624]: pgmap v15026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:35.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:35 smithi081 ceph-mon[31453]: pgmap v15026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:37.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:37 smithi153 ceph-mon[36569]: pgmap v15027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:37.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:37 smithi104 ceph-mon[36624]: pgmap v15027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:37.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:37 smithi081 ceph-mon[31453]: pgmap v15027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:39.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:39 smithi153 ceph-mon[36569]: pgmap v15028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:39.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:39 smithi104 ceph-mon[36624]: pgmap v15028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:39.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:39 smithi081 ceph-mon[31453]: pgmap v15028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:41.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:41 smithi153 ceph-mon[36569]: pgmap v15029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:41.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:41 smithi104 ceph-mon[36624]: pgmap v15029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:41.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:41 smithi081 ceph-mon[31453]: pgmap v15029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:43.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:43 smithi153 ceph-mon[36569]: pgmap v15030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:43.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:43 smithi104 ceph-mon[36624]: pgmap v15030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:43.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:43 smithi081 ceph-mon[31453]: pgmap v15030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:43.991+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:02:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:43.991+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:02:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:43.991+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:02:45.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:45 smithi153 ceph-mon[36569]: pgmap v15031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:45.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:45 smithi104 ceph-mon[36624]: pgmap v15031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:45.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:45 smithi081 ceph-mon[31453]: pgmap v15031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:47.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:47 smithi153 ceph-mon[36569]: pgmap v15032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:47.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:47 smithi104 ceph-mon[36624]: pgmap v15032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:47.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:47 smithi081 ceph-mon[31453]: pgmap v15032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:49.553 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:49 smithi153 ceph-mon[36569]: pgmap v15033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:49.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:49 smithi104 ceph-mon[36624]: pgmap v15033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:49.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:49 smithi081 ceph-mon[31453]: pgmap v15033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:51.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:51 smithi104 ceph-mon[36624]: pgmap v15034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:51.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:51 smithi081 ceph-mon[31453]: pgmap v15034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:51.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:51 smithi153 ceph-mon[36569]: pgmap v15034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:53.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:53 smithi104 ceph-mon[36624]: pgmap v15035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:53.606 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:53 smithi153 ceph-mon[36569]: pgmap v15035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:53.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:53 smithi081 ceph-mon[31453]: pgmap v15035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:55.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:55 smithi081 ceph-mon[31453]: pgmap v15036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:55.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:55 smithi104 ceph-mon[36624]: pgmap v15036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:55.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:55 smithi153 ceph-mon[36569]: pgmap v15036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:57.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:57 smithi104 ceph-mon[36624]: pgmap v15037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:57.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:57 smithi081 ceph-mon[31453]: pgmap v15037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:57.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:57 smithi153 ceph-mon[36569]: pgmap v15037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:59.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:59.004+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:02:59.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:59.004+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:02:59.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:02:59.004+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:02:59.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:02:59 smithi104 ceph-mon[36624]: pgmap v15038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:59.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:02:59 smithi081 ceph-mon[31453]: pgmap v15038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:02:59.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:02:59 smithi153 ceph-mon[36569]: pgmap v15038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:01.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:01 smithi081 ceph-mon[31453]: pgmap v15039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:01.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:01 smithi153 ceph-mon[36569]: pgmap v15039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:01.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:01 smithi104 ceph-mon[36624]: pgmap v15039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:03.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:03 smithi081 ceph-mon[31453]: pgmap v15040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:03.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:03 smithi153 ceph-mon[36569]: pgmap v15040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:03.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:03 smithi104 ceph-mon[36624]: pgmap v15040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:05.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:05 smithi081 ceph-mon[31453]: pgmap v15041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:05.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:05 smithi153 ceph-mon[36569]: pgmap v15041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:05.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:05 smithi104 ceph-mon[36624]: pgmap v15041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:07.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:07 smithi081 ceph-mon[31453]: pgmap v15042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:07.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:07 smithi153 ceph-mon[36569]: pgmap v15042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:07.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:07 smithi104 ceph-mon[36624]: pgmap v15042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:08.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:08 smithi081 ceph-mon[31453]: pgmap v15043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:08.803 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:08 smithi153 ceph-mon[36569]: pgmap v15043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:08.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:08 smithi104 ceph-mon[36624]: pgmap v15043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:11.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:10 smithi153 ceph-mon[36569]: pgmap v15044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:11.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:10 smithi104 ceph-mon[36624]: pgmap v15044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:11.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:10 smithi081 ceph-mon[31453]: pgmap v15044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:13.053 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:12 smithi153 ceph-mon[36569]: pgmap v15045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:13.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:12 smithi104 ceph-mon[36624]: pgmap v15045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:13.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:12 smithi081 ceph-mon[31453]: pgmap v15045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:14.017+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:03:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:14.017+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:03:14.015 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:14.017+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:03:15.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:14 smithi153 ceph-mon[36569]: pgmap v15046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:15.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:14 smithi104 ceph-mon[36624]: pgmap v15046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:15.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:14 smithi081 ceph-mon[31453]: pgmap v15046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:17.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:16 smithi104 ceph-mon[36624]: pgmap v15047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:17.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:16 smithi081 ceph-mon[31453]: pgmap v15047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:17.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:16 smithi153 ceph-mon[36569]: pgmap v15047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:19.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:18 smithi104 ceph-mon[36624]: pgmap v15048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:19.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:18 smithi081 ceph-mon[31453]: pgmap v15048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:19.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:18 smithi153 ceph-mon[36569]: pgmap v15048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:21.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:20 smithi104 ceph-mon[36624]: pgmap v15049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:21.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:20 smithi081 ceph-mon[31453]: pgmap v15049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:21.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:20 smithi153 ceph-mon[36569]: pgmap v15049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:23.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:22 smithi104 ceph-mon[36624]: pgmap v15050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:23.106 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:22 smithi153 ceph-mon[36569]: pgmap v15050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:23.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:22 smithi081 ceph-mon[31453]: pgmap v15050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:25.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:24 smithi104 ceph-mon[36624]: pgmap v15051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:25.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:03:25.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:24 smithi081 ceph-mon[31453]: pgmap v15051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:25.153 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:03:25.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:24 smithi153 ceph-mon[36569]: pgmap v15051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:25.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:03:26.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:03:26.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:03:26.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:25 smithi081 ceph-mon[31453]: pgmap v15052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:26.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:25 smithi081 ceph-mon[31453]: pgmap v15053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:26.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:03:26.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:03:26.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:03:26.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:25 smithi153 ceph-mon[36569]: pgmap v15052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:26.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:25 smithi153 ceph-mon[36569]: pgmap v15053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:26.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:03:26.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:03:26.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:03:26.329 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:25 smithi104 ceph-mon[36624]: pgmap v15052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:26.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:25 smithi104 ceph-mon[36624]: pgmap v15053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:26.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:03:28.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:28 smithi081 ceph-mon[31453]: pgmap v15054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:28.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:28 smithi153 ceph-mon[36569]: pgmap v15054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:28.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:28 smithi104 ceph-mon[36624]: pgmap v15054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:29.021+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:03:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:29.021+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:03:29.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:29.021+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:03:30.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:30 smithi081 ceph-mon[31453]: pgmap v15055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:30.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:30 smithi153 ceph-mon[36569]: pgmap v15055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:30.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:30 smithi104 ceph-mon[36624]: pgmap v15055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:32.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:32 smithi081 ceph-mon[31453]: pgmap v15056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:32.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:32 smithi153 ceph-mon[36569]: pgmap v15056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:32.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:32 smithi104 ceph-mon[36624]: pgmap v15056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:34.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:34 smithi081 ceph-mon[31453]: pgmap v15057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:34.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:34 smithi153 ceph-mon[36569]: pgmap v15057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:34.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:34 smithi104 ceph-mon[36624]: pgmap v15057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:35.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:35 smithi153 ceph-mon[36569]: pgmap v15058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:35.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:35 smithi104 ceph-mon[36624]: pgmap v15058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:35.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:35 smithi081 ceph-mon[31453]: pgmap v15058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:38.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:38 smithi081 ceph-mon[31453]: pgmap v15059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:38.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:38 smithi153 ceph-mon[36569]: pgmap v15059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:38.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:38 smithi104 ceph-mon[36624]: pgmap v15059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:40.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:40 smithi081 ceph-mon[31453]: pgmap v15060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:40.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:40 smithi153 ceph-mon[36569]: pgmap v15060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:40.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:40 smithi104 ceph-mon[36624]: pgmap v15060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:42.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:42 smithi081 ceph-mon[31453]: pgmap v15061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:42.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:42 smithi153 ceph-mon[36569]: pgmap v15061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:42.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:42 smithi104 ceph-mon[36624]: pgmap v15061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:44.033+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:44.033+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:44.033+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:03:44.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:44 smithi081 ceph-mon[31453]: pgmap v15062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:44.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:44 smithi153 ceph-mon[36569]: pgmap v15062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:44.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:44 smithi104 ceph-mon[36624]: pgmap v15062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:45.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:45 smithi081 ceph-mon[31453]: pgmap v15063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:45.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:45 smithi153 ceph-mon[36569]: pgmap v15063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:45.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:45 smithi104 ceph-mon[36624]: pgmap v15063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:48.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:48 smithi081 ceph-mon[31453]: pgmap v15064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:48.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:48 smithi153 ceph-mon[36569]: pgmap v15064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:48.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:48 smithi104 ceph-mon[36624]: pgmap v15064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:50.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:50 smithi081 ceph-mon[31453]: pgmap v15065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:50.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:50 smithi153 ceph-mon[36569]: pgmap v15065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:50.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:50 smithi104 ceph-mon[36624]: pgmap v15065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:52.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:52 smithi081 ceph-mon[31453]: pgmap v15066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:52.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:52 smithi153 ceph-mon[36569]: pgmap v15066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:52.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:52 smithi104 ceph-mon[36624]: pgmap v15066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:53.705 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:53 smithi153 ceph-mon[36569]: pgmap v15067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:53.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:53 smithi104 ceph-mon[36624]: pgmap v15067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:53.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:53 smithi081 ceph-mon[31453]: pgmap v15067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:56.607 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:56 smithi081 ceph-mon[31453]: pgmap v15068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:56.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:56 smithi153 ceph-mon[36569]: pgmap v15068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:56.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:56 smithi104 ceph-mon[36624]: pgmap v15068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:58 smithi081 ceph-mon[31453]: pgmap v15069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:58.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:58 smithi153 ceph-mon[36569]: pgmap v15069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:58.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:58 smithi104 ceph-mon[36624]: pgmap v15069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:59.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:59.042+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:03:59.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:59.042+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:03:59.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:03:59.042+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:03:59.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:03:59 smithi153 ceph-mon[36569]: pgmap v15070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:59.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:03:59 smithi104 ceph-mon[36624]: pgmap v15070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:03:59.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:03:59 smithi081 ceph-mon[31453]: pgmap v15070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:02.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:02 smithi081 ceph-mon[31453]: pgmap v15071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:02.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:02 smithi153 ceph-mon[36569]: pgmap v15071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:02.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:02 smithi104 ceph-mon[36624]: pgmap v15071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:04.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:04 smithi081 ceph-mon[31453]: pgmap v15072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:04.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:04 smithi153 ceph-mon[36569]: pgmap v15072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:04.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:04 smithi104 ceph-mon[36624]: pgmap v15072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:05.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:05 smithi153 ceph-mon[36569]: pgmap v15073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:05.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:05 smithi104 ceph-mon[36624]: pgmap v15073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:05.903 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:05 smithi081 ceph-mon[31453]: pgmap v15073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:08.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:08 smithi081 ceph-mon[31453]: pgmap v15074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:08.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:08 smithi153 ceph-mon[36569]: pgmap v15074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:08.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:08 smithi104 ceph-mon[36624]: pgmap v15074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:09.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:09 smithi153 ceph-mon[36569]: pgmap v15075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:09.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:09 smithi104 ceph-mon[36624]: pgmap v15075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:09.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:09 smithi081 ceph-mon[31453]: pgmap v15075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:12.653 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:12 smithi081 ceph-mon[31453]: pgmap v15076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:12.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:12 smithi153 ceph-mon[36569]: pgmap v15076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:12.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:12 smithi104 ceph-mon[36624]: pgmap v15076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:13.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:13 smithi153 ceph-mon[36569]: pgmap v15077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:13.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:13 smithi104 ceph-mon[36624]: pgmap v15077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:13.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:13 smithi081 ceph-mon[31453]: pgmap v15077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:14.054+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:14.054+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:04:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:14.054+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:04:16.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:16 smithi081 ceph-mon[31453]: pgmap v15078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:16.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:16 smithi153 ceph-mon[36569]: pgmap v15078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:16.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:16 smithi104 ceph-mon[36624]: pgmap v15078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:17.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:17.345+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:04:17.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:17.345+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:04:17.343 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:17.346+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 217 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:04:17.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:17.346+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 217 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:04:17.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:17 smithi081 ceph-mon[31453]: pgmap v15079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:17.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:17 smithi153 ceph-mon[36569]: pgmap v15079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:17.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:17 smithi104 ceph-mon[36624]: pgmap v15079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:19.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:19 smithi153 ceph-mon[36569]: pgmap v15080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:19.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:19 smithi104 ceph-mon[36624]: pgmap v15080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:19.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:19 smithi081 ceph-mon[31453]: pgmap v15080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:21.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:21 smithi153 ceph-mon[36569]: pgmap v15081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:21.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:21 smithi104 ceph-mon[36624]: pgmap v15081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:21.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:21 smithi081 ceph-mon[31453]: pgmap v15081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:23.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:23 smithi153 ceph-mon[36569]: pgmap v15082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:23.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:23 smithi104 ceph-mon[36624]: pgmap v15082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:23.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:23 smithi081 ceph-mon[31453]: pgmap v15082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:25.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:25 smithi153 ceph-mon[36569]: pgmap v15083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:25.829 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:25 smithi104 ceph-mon[36624]: pgmap v15083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:25.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:25 smithi081 ceph-mon[31453]: pgmap v15083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:26.731 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:04:26.731 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.731 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.731 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.731 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:04:26.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:04:26.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:26.806 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:27.790 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:27 smithi081 ceph-mon[31453]: pgmap v15084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:27.804 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:27 smithi153 ceph-mon[36569]: pgmap v15084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:27.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:27 smithi104 ceph-mon[36624]: pgmap v15084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:29.068+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:04:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:29.068+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:04:29.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:29.068+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:04:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.321 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.322 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:30.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:30 smithi153 ceph-mon[36569]: pgmap v15085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:30.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:30 smithi104 ceph-mon[36624]: pgmap v15085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:30.403 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:30 smithi081 ceph-mon[31453]: pgmap v15085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:31.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:31.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:31.904 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:31 smithi081 ceph-mon[31453]: pgmap v15086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:32.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:32.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:32.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:31 smithi153 ceph-mon[36569]: pgmap v15086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:32.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:32.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:32.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:31 smithi104 ceph-mon[36624]: pgmap v15086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:33.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:04:33.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:04:33.054 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:32 smithi153 ceph-mon[36569]: pgmap v15087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:33.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:32 smithi153 ceph-mon[36569]: pgmap v15088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:33.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:04:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:04:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:32 smithi104 ceph-mon[36624]: pgmap v15087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:32 smithi104 ceph-mon[36624]: pgmap v15088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:33.154 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:04:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:04:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:32 smithi081 ceph-mon[31453]: pgmap v15087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:32 smithi081 ceph-mon[31453]: pgmap v15088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:04:35.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:35 smithi153 ceph-mon[36569]: pgmap v15089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:35.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:35 smithi104 ceph-mon[36624]: pgmap v15089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:35.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:35 smithi081 ceph-mon[31453]: pgmap v15089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:37.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:37 smithi153 ceph-mon[36569]: pgmap v15090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:37.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:37 smithi104 ceph-mon[36624]: pgmap v15090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:37.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:37 smithi081 ceph-mon[31453]: pgmap v15090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:39.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:39 smithi153 ceph-mon[36569]: pgmap v15091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:39.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:39 smithi104 ceph-mon[36624]: pgmap v15091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:39.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:39 smithi081 ceph-mon[31453]: pgmap v15091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:41.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:41 smithi153 ceph-mon[36569]: pgmap v15092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:41.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:41 smithi104 ceph-mon[36624]: pgmap v15092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:41.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:41 smithi081 ceph-mon[31453]: pgmap v15092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:43.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:43 smithi153 ceph-mon[36569]: pgmap v15093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:43.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:43 smithi104 ceph-mon[36624]: pgmap v15093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:43.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:43 smithi081 ceph-mon[31453]: pgmap v15093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:44.083+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:04:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:44.083+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:04:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:44.083+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:04:45.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:45 smithi153 ceph-mon[36569]: pgmap v15094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:45.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:45 smithi104 ceph-mon[36624]: pgmap v15094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:45.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:45 smithi081 ceph-mon[31453]: pgmap v15094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:47.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:47 smithi153 ceph-mon[36569]: pgmap v15095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:47.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:47 smithi104 ceph-mon[36624]: pgmap v15095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:47.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:47 smithi081 ceph-mon[31453]: pgmap v15095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:49.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:49 smithi153 ceph-mon[36569]: pgmap v15096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:49.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:49 smithi104 ceph-mon[36624]: pgmap v15096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:49.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:49 smithi081 ceph-mon[31453]: pgmap v15096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:51.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:51 smithi153 ceph-mon[36569]: pgmap v15097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:51.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:51 smithi104 ceph-mon[36624]: pgmap v15097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:51.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:51 smithi081 ceph-mon[31453]: pgmap v15097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:53.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:53 smithi153 ceph-mon[36569]: pgmap v15098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:53.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:53 smithi104 ceph-mon[36624]: pgmap v15098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:53.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:53 smithi081 ceph-mon[31453]: pgmap v15098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:55.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:55 smithi104 ceph-mon[36624]: pgmap v15099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:55.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:55 smithi081 ceph-mon[31453]: pgmap v15099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:55.528 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:55 smithi153 ceph-mon[36569]: pgmap v15099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:57.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:57 smithi104 ceph-mon[36624]: pgmap v15100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:57.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:57 smithi081 ceph-mon[31453]: pgmap v15100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:57.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:57 smithi153 ceph-mon[36569]: pgmap v15100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:59.098+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:04:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:59.099+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:04:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:04:59.099+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:04:59.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:04:59 smithi104 ceph-mon[36624]: pgmap v15101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:59.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:04:59 smithi081 ceph-mon[31453]: pgmap v15101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:04:59.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:04:59 smithi153 ceph-mon[36569]: pgmap v15101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:01.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:01 smithi104 ceph-mon[36624]: pgmap v15102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:01.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:01 smithi081 ceph-mon[31453]: pgmap v15102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:01.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:01 smithi153 ceph-mon[36569]: pgmap v15102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:03.330 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:03 smithi104 ceph-mon[36624]: pgmap v15103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:03.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:03 smithi081 ceph-mon[31453]: pgmap v15103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:03.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:03 smithi153 ceph-mon[36569]: pgmap v15103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:05.304 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:05 smithi153 ceph-mon[36569]: pgmap v15104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:05.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:05 smithi081 ceph-mon[31453]: pgmap v15104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:05.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:05 smithi104 ceph-mon[36624]: pgmap v15104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:07.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:07 smithi081 ceph-mon[31453]: pgmap v15105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:07.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:07 smithi153 ceph-mon[36569]: pgmap v15105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:07.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:07 smithi104 ceph-mon[36624]: pgmap v15105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:09.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:09 smithi081 ceph-mon[31453]: pgmap v15106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:09.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:09 smithi153 ceph-mon[36569]: pgmap v15106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:09.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:09 smithi104 ceph-mon[36624]: pgmap v15106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:11.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:11 smithi081 ceph-mon[31453]: pgmap v15107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:11.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:11 smithi153 ceph-mon[36569]: pgmap v15107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:11.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:11 smithi104 ceph-mon[36624]: pgmap v15107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:13.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:13 smithi081 ceph-mon[31453]: pgmap v15108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:13.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:13 smithi153 ceph-mon[36569]: pgmap v15108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:13.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:13 smithi104 ceph-mon[36624]: pgmap v15108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:14.112+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:05:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:14.112+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:05:14.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:14.112+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:05:15.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:15 smithi081 ceph-mon[31453]: pgmap v15109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:15.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:15 smithi153 ceph-mon[36569]: pgmap v15109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:15.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:15 smithi104 ceph-mon[36624]: pgmap v15109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:17.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:17 smithi081 ceph-mon[31453]: pgmap v15110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:17.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:17 smithi153 ceph-mon[36569]: pgmap v15110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:17 smithi104 ceph-mon[36624]: pgmap v15110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:19.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:19 smithi081 ceph-mon[31453]: pgmap v15111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:19.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:19 smithi153 ceph-mon[36569]: pgmap v15111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:19.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:19 smithi104 ceph-mon[36624]: pgmap v15111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:21.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:21 smithi081 ceph-mon[31453]: pgmap v15112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:21.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:21 smithi153 ceph-mon[36569]: pgmap v15112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:21.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:21 smithi104 ceph-mon[36624]: pgmap v15112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:23.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:23 smithi081 ceph-mon[31453]: pgmap v15113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:23.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:23 smithi153 ceph-mon[36569]: pgmap v15113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:23.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:23 smithi104 ceph-mon[36624]: pgmap v15113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:25.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:25 smithi081 ceph-mon[31453]: pgmap v15114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:25.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:25 smithi153 ceph-mon[36569]: pgmap v15114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:25.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:25 smithi104 ceph-mon[36624]: pgmap v15114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:27.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:27 smithi081 ceph-mon[31453]: pgmap v15115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:27.554 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:27 smithi153 ceph-mon[36569]: pgmap v15115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:27.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:27 smithi104 ceph-mon[36624]: pgmap v15115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:29.116+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:05:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:29.116+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:05:29.115 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:29.116+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:05:29.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:29 smithi081 ceph-mon[31453]: pgmap v15116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:29.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:29 smithi153 ceph-mon[36569]: pgmap v15116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:29.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:29 smithi104 ceph-mon[36624]: pgmap v15116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:31.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:31 smithi081 ceph-mon[31453]: pgmap v15117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:31 smithi153 ceph-mon[36569]: pgmap v15117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:31.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:31 smithi104 ceph-mon[36624]: pgmap v15117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:33.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:33 smithi081 ceph-mon[31453]: pgmap v15118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:05:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:33.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:33 smithi153 ceph-mon[36569]: pgmap v15118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:33.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:05:33.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:33.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:33.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:33 smithi104 ceph-mon[36624]: pgmap v15118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:33.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:05:33.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:05:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:05:34.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:34.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:05:34.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:05:34.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:34.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:05:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:05:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:05:35.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:35 smithi153 ceph-mon[36569]: pgmap v15119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:35.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:35 smithi153 ceph-mon[36569]: pgmap v15120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:35.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:35 smithi104 ceph-mon[36624]: pgmap v15119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:35 smithi104 ceph-mon[36624]: pgmap v15120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:35.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:35 smithi081 ceph-mon[31453]: pgmap v15119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:35.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:35 smithi081 ceph-mon[31453]: pgmap v15120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:37.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:37 smithi153 ceph-mon[36569]: pgmap v15121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:37.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:37 smithi104 ceph-mon[36624]: pgmap v15121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:37.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:37 smithi081 ceph-mon[31453]: pgmap v15121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:39.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:39 smithi153 ceph-mon[36569]: pgmap v15122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:39.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:39 smithi104 ceph-mon[36624]: pgmap v15122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:39.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:39 smithi081 ceph-mon[31453]: pgmap v15122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:41.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:41 smithi153 ceph-mon[36569]: pgmap v15123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:41.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:41 smithi104 ceph-mon[36624]: pgmap v15123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:41.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:41 smithi081 ceph-mon[31453]: pgmap v15123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:43.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:43 smithi153 ceph-mon[36569]: pgmap v15124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:43.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:43 smithi104 ceph-mon[36624]: pgmap v15124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:43.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:43 smithi081 ceph-mon[31453]: pgmap v15124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:44.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:44.127+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:05:44.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:44.127+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:05:44.126 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:44.127+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:05:45.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:45 smithi153 ceph-mon[36569]: pgmap v15125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:45.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:45 smithi104 ceph-mon[36624]: pgmap v15125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:45.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:45 smithi081 ceph-mon[31453]: pgmap v15125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:47.404 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:47 smithi081 ceph-mon[31453]: pgmap v15126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:47.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:47 smithi153 ceph-mon[36569]: pgmap v15126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:47.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:47 smithi104 ceph-mon[36624]: pgmap v15126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:49.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:49 smithi153 ceph-mon[36569]: pgmap v15127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:49.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:49 smithi104 ceph-mon[36624]: pgmap v15127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:49.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:49 smithi081 ceph-mon[31453]: pgmap v15127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:51.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:51 smithi153 ceph-mon[36569]: pgmap v15128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:51.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:51 smithi104 ceph-mon[36624]: pgmap v15128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:51.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:51 smithi081 ceph-mon[31453]: pgmap v15128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:53.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:53 smithi153 ceph-mon[36569]: pgmap v15129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:53.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:53 smithi104 ceph-mon[36624]: pgmap v15129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:53.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:53 smithi081 ceph-mon[31453]: pgmap v15129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:55.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:55 smithi153 ceph-mon[36569]: pgmap v15130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:55.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:55 smithi104 ceph-mon[36624]: pgmap v15130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:55.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:55 smithi081 ceph-mon[31453]: pgmap v15130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:57.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:57 smithi153 ceph-mon[36569]: pgmap v15131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:57.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:57 smithi104 ceph-mon[36624]: pgmap v15131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:57.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:57 smithi081 ceph-mon[31453]: pgmap v15131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:59.135+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:05:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:59.135+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:05:59.134 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:05:59.135+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:05:59.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:05:59 smithi153 ceph-mon[36569]: pgmap v15132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:59.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:05:59 smithi104 ceph-mon[36624]: pgmap v15132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:05:59.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:05:59 smithi081 ceph-mon[31453]: pgmap v15132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:01.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:01 smithi153 ceph-mon[36569]: pgmap v15133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:01.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:01 smithi104 ceph-mon[36624]: pgmap v15133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:01.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:01 smithi081 ceph-mon[31453]: pgmap v15133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:03 smithi081 ceph-mon[31453]: pgmap v15134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:03.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:03 smithi153 ceph-mon[36569]: pgmap v15134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:03.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:03 smithi104 ceph-mon[36624]: pgmap v15134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:05.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:05 smithi153 ceph-mon[36569]: pgmap v15135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:05.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:05 smithi104 ceph-mon[36624]: pgmap v15135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:05.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:05 smithi081 ceph-mon[31453]: pgmap v15135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:07.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:07 smithi153 ceph-mon[36569]: pgmap v15136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:07.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:07 smithi104 ceph-mon[36624]: pgmap v15136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:07.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:07 smithi081 ceph-mon[31453]: pgmap v15136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:09.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:09 smithi153 ceph-mon[36569]: pgmap v15137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:09.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:09 smithi104 ceph-mon[36624]: pgmap v15137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:09.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:09 smithi081 ceph-mon[31453]: pgmap v15137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:11.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:11 smithi153 ceph-mon[36569]: pgmap v15138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:11.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:11 smithi104 ceph-mon[36624]: pgmap v15138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:11 smithi081 ceph-mon[31453]: pgmap v15138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:13.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:13 smithi153 ceph-mon[36569]: pgmap v15139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:13.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:13 smithi104 ceph-mon[36624]: pgmap v15139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:13.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:13 smithi081 ceph-mon[31453]: pgmap v15139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:14.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:14.150+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:06:14.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:14.150+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:06:14.150 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:14.150+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:06:15.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:15 smithi153 ceph-mon[36569]: pgmap v15140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:15 smithi104 ceph-mon[36624]: pgmap v15140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:15 smithi081 ceph-mon[31453]: pgmap v15140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:17.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:17 smithi153 ceph-mon[36569]: pgmap v15141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:17.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:17 smithi104 ceph-mon[36624]: pgmap v15141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:17.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:17 smithi081 ceph-mon[31453]: pgmap v15141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:19.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:19 smithi153 ceph-mon[36569]: pgmap v15142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:19 smithi104 ceph-mon[36624]: pgmap v15142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:19.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:19 smithi081 ceph-mon[31453]: pgmap v15142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:21.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:21 smithi153 ceph-mon[36569]: pgmap v15143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:21.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:21 smithi104 ceph-mon[36624]: pgmap v15143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:21.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:21 smithi081 ceph-mon[31453]: pgmap v15143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:23.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:23 smithi153 ceph-mon[36569]: pgmap v15144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:23 smithi104 ceph-mon[36624]: pgmap v15144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:23.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:23 smithi081 ceph-mon[31453]: pgmap v15144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:25.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:25 smithi153 ceph-mon[36569]: pgmap v15145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:25 smithi104 ceph-mon[36624]: pgmap v15145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:25 smithi081 ceph-mon[31453]: pgmap v15145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:27.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:27 smithi153 ceph-mon[36569]: pgmap v15146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:27.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:27 smithi104 ceph-mon[36624]: pgmap v15146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:27 smithi081 ceph-mon[31453]: pgmap v15146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:29.159+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:06:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:29.159+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:06:29.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:29.159+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:06:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:29 smithi104 ceph-mon[36624]: pgmap v15147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:29.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:29 smithi081 ceph-mon[31453]: pgmap v15147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:29.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:29 smithi153 ceph-mon[36569]: pgmap v15147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:31.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:31 smithi104 ceph-mon[36624]: pgmap v15148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:31.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:31 smithi081 ceph-mon[31453]: pgmap v15148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:31.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:31 smithi153 ceph-mon[36569]: pgmap v15148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:32.367+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:06:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:32.367+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:06:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:32.368+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 218 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:06:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:32.368+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 218 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:06:33.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:33 smithi104 ceph-mon[36624]: pgmap v15149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:33 smithi081 ceph-mon[31453]: pgmap v15149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:33.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:33 smithi153 ceph-mon[36569]: pgmap v15149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:34.580 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:06:34.654 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:06:34.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:06:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:35 smithi081 ceph-mon[31453]: pgmap v15150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:06:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:06:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:06:35.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:35 smithi153 ceph-mon[36569]: pgmap v15150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:35.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:06:35.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:06:35.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:06:35.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:35 smithi104 ceph-mon[36624]: pgmap v15150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:35.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:06:35.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:06:35.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:06:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:36 smithi081 ceph-mon[31453]: pgmap v15151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:36 smithi081 ceph-mon[31453]: pgmap v15152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:36.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:36 smithi153 ceph-mon[36569]: pgmap v15151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:36.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:36 smithi153 ceph-mon[36569]: pgmap v15152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:36.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:36 smithi104 ceph-mon[36624]: pgmap v15151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:36.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:36 smithi104 ceph-mon[36624]: pgmap v15152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:38 smithi081 ceph-mon[31453]: pgmap v15153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:38.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:38 smithi153 ceph-mon[36569]: pgmap v15153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:38.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:38 smithi104 ceph-mon[36624]: pgmap v15153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:40.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:40 smithi081 ceph-mon[31453]: pgmap v15154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:40.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:40 smithi153 ceph-mon[36569]: pgmap v15154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:40 smithi104 ceph-mon[36624]: pgmap v15154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:42 smithi081 ceph-mon[31453]: pgmap v15155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:42 smithi153 ceph-mon[36569]: pgmap v15155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:42.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:42 smithi104 ceph-mon[36624]: pgmap v15155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:44.167+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:06:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:44.167+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:06:44.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:44.167+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:06:44.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:44 smithi081 ceph-mon[31453]: pgmap v15156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:44.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:44 smithi153 ceph-mon[36569]: pgmap v15156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:44.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:44 smithi104 ceph-mon[36624]: pgmap v15156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:46 smithi081 ceph-mon[31453]: pgmap v15157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:46.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:46 smithi153 ceph-mon[36569]: pgmap v15157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:46.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:46 smithi104 ceph-mon[36624]: pgmap v15157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:48 smithi081 ceph-mon[31453]: pgmap v15158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:48.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:48 smithi153 ceph-mon[36569]: pgmap v15158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:48.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:48 smithi104 ceph-mon[36624]: pgmap v15158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:50 smithi081 ceph-mon[31453]: pgmap v15159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:50.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:50 smithi153 ceph-mon[36569]: pgmap v15159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:50.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:50 smithi104 ceph-mon[36624]: pgmap v15159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:52 smithi081 ceph-mon[31453]: pgmap v15160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:52.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:52 smithi153 ceph-mon[36569]: pgmap v15160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:52.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:52 smithi104 ceph-mon[36624]: pgmap v15160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:54.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:54 smithi081 ceph-mon[31453]: pgmap v15161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:54.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:54 smithi153 ceph-mon[36569]: pgmap v15161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:54 smithi104 ceph-mon[36624]: pgmap v15161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:56.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:56 smithi081 ceph-mon[31453]: pgmap v15162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:56.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:56 smithi153 ceph-mon[36569]: pgmap v15162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:56 smithi104 ceph-mon[36624]: pgmap v15162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:57.732 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:06:57 smithi104 ceph-mon[36624]: pgmap v15163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:57.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:06:57 smithi153 ceph-mon[36569]: pgmap v15163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:57.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:06:57 smithi081 ceph-mon[31453]: pgmap v15163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:06:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:59.183+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:06:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:59.183+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:06:59.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:06:59.183+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:07:00.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:00 smithi153 ceph-mon[36569]: pgmap v15164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:00 smithi104 ceph-mon[36624]: pgmap v15164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:00.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:00 smithi081 ceph-mon[31453]: pgmap v15164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:02.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:02 smithi153 ceph-mon[36569]: pgmap v15165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:02.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:02 smithi104 ceph-mon[36624]: pgmap v15165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:02.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:02 smithi081 ceph-mon[31453]: pgmap v15165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:04.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:04 smithi153 ceph-mon[36569]: pgmap v15166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:04.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:04 smithi104 ceph-mon[36624]: pgmap v15166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:04.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:04 smithi081 ceph-mon[31453]: pgmap v15166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:06.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:06 smithi153 ceph-mon[36569]: pgmap v15167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:06.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:06 smithi104 ceph-mon[36624]: pgmap v15167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:06 smithi081 ceph-mon[31453]: pgmap v15167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:08.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:08 smithi153 ceph-mon[36569]: pgmap v15168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:08 smithi104 ceph-mon[36624]: pgmap v15168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:08.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:08 smithi081 ceph-mon[31453]: pgmap v15168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:10.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:10 smithi153 ceph-mon[36569]: pgmap v15169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:10.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:10 smithi104 ceph-mon[36624]: pgmap v15169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:10.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:10 smithi081 ceph-mon[31453]: pgmap v15169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:12.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:12 smithi153 ceph-mon[36569]: pgmap v15170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:12.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:12 smithi104 ceph-mon[36624]: pgmap v15170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:12 smithi081 ceph-mon[31453]: pgmap v15170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:14.195+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:07:14.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:14.195+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:07:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:14.195+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:07:14.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:14 smithi153 ceph-mon[36569]: pgmap v15171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:14 smithi104 ceph-mon[36624]: pgmap v15171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:14.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:14 smithi081 ceph-mon[31453]: pgmap v15171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:16.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:16 smithi153 ceph-mon[36569]: pgmap v15172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:16 smithi104 ceph-mon[36624]: pgmap v15172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:16.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:16 smithi081 ceph-mon[31453]: pgmap v15172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:17.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:17.374+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:07:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:17.374+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:07:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:17.375+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 219 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:07:17.374 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:17.375+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 219 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:07:18.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:18 smithi153 ceph-mon[36569]: pgmap v15173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:18 smithi104 ceph-mon[36624]: pgmap v15173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:18 smithi081 ceph-mon[31453]: pgmap v15173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:20.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:20 smithi153 ceph-mon[36569]: pgmap v15174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:20 smithi104 ceph-mon[36624]: pgmap v15174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:20 smithi081 ceph-mon[31453]: pgmap v15174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:22.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:22 smithi153 ceph-mon[36569]: pgmap v15175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:22.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:22 smithi104 ceph-mon[36624]: pgmap v15175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:22.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:22 smithi081 ceph-mon[31453]: pgmap v15175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:24.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:24 smithi153 ceph-mon[36569]: pgmap v15176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:24 smithi104 ceph-mon[36624]: pgmap v15176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:24 smithi081 ceph-mon[31453]: pgmap v15176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:26.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:26 smithi153 ceph-mon[36569]: pgmap v15177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:26 smithi104 ceph-mon[36624]: pgmap v15177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:26 smithi081 ceph-mon[31453]: pgmap v15177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:28 smithi104 ceph-mon[36624]: pgmap v15178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:28 smithi081 ceph-mon[31453]: pgmap v15178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:28.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:28 smithi153 ceph-mon[36569]: pgmap v15178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:29.211+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:07:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:29.211+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:07:29.210 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:29.211+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:07:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:30 smithi104 ceph-mon[36624]: pgmap v15179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:30 smithi081 ceph-mon[31453]: pgmap v15179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:30.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:30 smithi153 ceph-mon[36569]: pgmap v15179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:32 smithi104 ceph-mon[36624]: pgmap v15180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:32 smithi081 ceph-mon[31453]: pgmap v15180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:32.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:32 smithi153 ceph-mon[36569]: pgmap v15180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:34 smithi104 ceph-mon[36624]: pgmap v15181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:34 smithi081 ceph-mon[31453]: pgmap v15181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:34.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:34 smithi153 ceph-mon[36569]: pgmap v15181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:36 smithi104 ceph-mon[36624]: pgmap v15182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:07:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:36 smithi081 ceph-mon[31453]: pgmap v15182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:07:36.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:36 smithi153 ceph-mon[36569]: pgmap v15182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:07:37.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:07:37.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:07:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:07:37.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:07:37.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:07:37.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:07:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:07:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:07:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:07:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:38 smithi081 ceph-mon[31453]: pgmap v15183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:38 smithi081 ceph-mon[31453]: pgmap v15184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:38.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:38 smithi153 ceph-mon[36569]: pgmap v15183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:38 smithi153 ceph-mon[36569]: pgmap v15184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:38 smithi104 ceph-mon[36624]: pgmap v15183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:38 smithi104 ceph-mon[36624]: pgmap v15184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:40.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:40 smithi081 ceph-mon[31453]: pgmap v15185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:40.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:40 smithi153 ceph-mon[36569]: pgmap v15185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:40.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:40 smithi104 ceph-mon[36624]: pgmap v15185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:42 smithi081 ceph-mon[31453]: pgmap v15186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:42.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:42 smithi153 ceph-mon[36569]: pgmap v15186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:42 smithi104 ceph-mon[36624]: pgmap v15186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:44.226+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:07:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:44.226+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:07:44.226 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:44.226+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:07:44.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:44 smithi081 ceph-mon[31453]: pgmap v15187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:44.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:44 smithi153 ceph-mon[36569]: pgmap v15187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:44 smithi104 ceph-mon[36624]: pgmap v15187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:46 smithi081 ceph-mon[31453]: pgmap v15188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:46.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:46 smithi153 ceph-mon[36569]: pgmap v15188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:46.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:46 smithi104 ceph-mon[36624]: pgmap v15188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:48 smithi081 ceph-mon[31453]: pgmap v15189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:48.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:48 smithi153 ceph-mon[36569]: pgmap v15189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:48.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:48 smithi104 ceph-mon[36624]: pgmap v15189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:50 smithi081 ceph-mon[31453]: pgmap v15190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:50.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:50 smithi153 ceph-mon[36569]: pgmap v15190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:50 smithi104 ceph-mon[36624]: pgmap v15190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:52 smithi081 ceph-mon[31453]: pgmap v15191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:52.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:52 smithi153 ceph-mon[36569]: pgmap v15191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:52 smithi104 ceph-mon[36624]: pgmap v15191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:54 smithi081 ceph-mon[31453]: pgmap v15192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:54.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:54 smithi153 ceph-mon[36569]: pgmap v15192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:54.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:54 smithi104 ceph-mon[36624]: pgmap v15192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:56.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:56 smithi081 ceph-mon[31453]: pgmap v15193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:56.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:56 smithi153 ceph-mon[36569]: pgmap v15193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:56 smithi104 ceph-mon[36624]: pgmap v15193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:07:58 smithi081 ceph-mon[31453]: pgmap v15194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:58.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:07:58 smithi153 ceph-mon[36569]: pgmap v15194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:07:58 smithi104 ceph-mon[36624]: pgmap v15194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:07:59.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:59.235+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:07:59.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:59.235+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:07:59.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:07:59.235+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:08:00.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:00 smithi081 ceph-mon[31453]: pgmap v15195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:00 smithi153 ceph-mon[36569]: pgmap v15195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:00 smithi104 ceph-mon[36624]: pgmap v15195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:02.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:02 smithi081 ceph-mon[31453]: pgmap v15196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:02.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:02 smithi153 ceph-mon[36569]: pgmap v15196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:02 smithi104 ceph-mon[36624]: pgmap v15196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:04.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:04 smithi153 ceph-mon[36569]: pgmap v15197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:04 smithi104 ceph-mon[36624]: pgmap v15197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:04 smithi081 ceph-mon[31453]: pgmap v15197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:06.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:06 smithi153 ceph-mon[36569]: pgmap v15198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:06 smithi104 ceph-mon[36624]: pgmap v15198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:06 smithi081 ceph-mon[31453]: pgmap v15198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:07.805 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:07 smithi153 ceph-mon[36569]: pgmap v15199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:07.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:07 smithi104 ceph-mon[36624]: pgmap v15199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:07.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:07 smithi081 ceph-mon[31453]: pgmap v15199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:10.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:09 smithi153 ceph-mon[36569]: pgmap v15200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:09 smithi104 ceph-mon[36624]: pgmap v15200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:09 smithi081 ceph-mon[31453]: pgmap v15200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:11 smithi153 ceph-mon[36569]: pgmap v15201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:12.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:11 smithi104 ceph-mon[36624]: pgmap v15201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:11 smithi081 ceph-mon[31453]: pgmap v15201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:13 smithi153 ceph-mon[36569]: pgmap v15202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:14.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:13 smithi104 ceph-mon[36624]: pgmap v15202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:14.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:13 smithi081 ceph-mon[31453]: pgmap v15202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:14.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:14.247+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:08:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:14.247+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:08:14.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:14.247+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:08:16.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:15 smithi153 ceph-mon[36569]: pgmap v15203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:16.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:15 smithi104 ceph-mon[36624]: pgmap v15203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:16.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:15 smithi081 ceph-mon[31453]: pgmap v15203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:17 smithi153 ceph-mon[36569]: pgmap v15204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:18.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:17 smithi104 ceph-mon[36624]: pgmap v15204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:18.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:17 smithi081 ceph-mon[31453]: pgmap v15204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:19 smithi153 ceph-mon[36569]: pgmap v15205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:19 smithi104 ceph-mon[36624]: pgmap v15205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:20.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:19 smithi081 ceph-mon[31453]: pgmap v15205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:22.055 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:21 smithi153 ceph-mon[36569]: pgmap v15206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:22.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:21 smithi104 ceph-mon[36624]: pgmap v15206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:22.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:21 smithi081 ceph-mon[31453]: pgmap v15206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:23 smithi153 ceph-mon[36569]: pgmap v15207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:23 smithi104 ceph-mon[36624]: pgmap v15207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:24.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:23 smithi081 ceph-mon[31453]: pgmap v15207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:25 smithi153 ceph-mon[36569]: pgmap v15208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:25 smithi104 ceph-mon[36624]: pgmap v15208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:25 smithi081 ceph-mon[31453]: pgmap v15208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:28.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:27 smithi104 ceph-mon[36624]: pgmap v15209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:27 smithi081 ceph-mon[31453]: pgmap v15209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:27 smithi153 ceph-mon[36569]: pgmap v15209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:29.250+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:08:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:29.250+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:08:29.250 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:29.251+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:08:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:29 smithi104 ceph-mon[36624]: pgmap v15210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:29 smithi081 ceph-mon[31453]: pgmap v15210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:29 smithi153 ceph-mon[36569]: pgmap v15210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:32.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:31 smithi104 ceph-mon[36624]: pgmap v15211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:32.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:31 smithi081 ceph-mon[31453]: pgmap v15211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:31 smithi153 ceph-mon[36569]: pgmap v15211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:33 smithi104 ceph-mon[36624]: pgmap v15212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:34.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:33 smithi081 ceph-mon[31453]: pgmap v15212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:33 smithi153 ceph-mon[36569]: pgmap v15212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:36.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:35 smithi081 ceph-mon[31453]: pgmap v15213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:35 smithi153 ceph-mon[36569]: pgmap v15213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:36.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:35 smithi104 ceph-mon[36624]: pgmap v15213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:38.150 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:37 smithi081 ceph-mon[31453]: pgmap v15214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:38.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:08:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:37 smithi153 ceph-mon[36569]: pgmap v15214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:08:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:37 smithi104 ceph-mon[36624]: pgmap v15214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:08:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:39 smithi153 ceph-mon[36569]: pgmap v15215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:08:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:08:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:39 smithi153 ceph-mon[36569]: pgmap v15216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:39 smithi153 ceph-mon[36569]: pgmap v15217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:40.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:39 smithi104 ceph-mon[36624]: pgmap v15215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:40.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:08:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:08:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:39 smithi104 ceph-mon[36624]: pgmap v15216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:39 smithi104 ceph-mon[36624]: pgmap v15217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:39 smithi081 ceph-mon[31453]: pgmap v15215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:39 smithi081 ceph-mon[31453]: pgmap v15216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:39 smithi081 ceph-mon[31453]: pgmap v15217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:08:42.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:42 smithi104 ceph-mon[36624]: pgmap v15218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:42.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:42 smithi081 ceph-mon[31453]: pgmap v15218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:42 smithi153 ceph-mon[36569]: pgmap v15218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:44.265+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:08:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:44.265+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:08:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:44.265+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:08:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:44 smithi081 ceph-mon[31453]: pgmap v15219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:44 smithi153 ceph-mon[36569]: pgmap v15219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:44 smithi104 ceph-mon[36624]: pgmap v15219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:46 smithi081 ceph-mon[31453]: pgmap v15220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:46 smithi153 ceph-mon[36569]: pgmap v15220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:46 smithi104 ceph-mon[36624]: pgmap v15220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:48.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:48 smithi081 ceph-mon[31453]: pgmap v15221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:48 smithi153 ceph-mon[36569]: pgmap v15221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:48 smithi104 ceph-mon[36624]: pgmap v15221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:50 smithi081 ceph-mon[31453]: pgmap v15222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:50 smithi153 ceph-mon[36569]: pgmap v15222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:50.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:50 smithi104 ceph-mon[36624]: pgmap v15222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:52 smithi081 ceph-mon[31453]: pgmap v15223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:52 smithi153 ceph-mon[36569]: pgmap v15223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:52 smithi104 ceph-mon[36624]: pgmap v15223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:54 smithi081 ceph-mon[31453]: pgmap v15224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:54 smithi153 ceph-mon[36569]: pgmap v15224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:54.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:54 smithi104 ceph-mon[36624]: pgmap v15224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:56 smithi081 ceph-mon[31453]: pgmap v15225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:56 smithi153 ceph-mon[36569]: pgmap v15225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:56.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:56 smithi104 ceph-mon[36624]: pgmap v15225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:58.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:08:58 smithi081 ceph-mon[31453]: pgmap v15226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:08:58 smithi153 ceph-mon[36569]: pgmap v15226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:58.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:08:58 smithi104 ceph-mon[36624]: pgmap v15226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:08:59.281 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:59.282+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:08:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:59.282+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:08:59.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:08:59.282+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:09:00.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:00 smithi104 ceph-mon[36624]: pgmap v15227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:00.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:00 smithi081 ceph-mon[31453]: pgmap v15227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:00 smithi153 ceph-mon[36569]: pgmap v15227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:02 smithi081 ceph-mon[31453]: pgmap v15228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:02 smithi153 ceph-mon[36569]: pgmap v15228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:02.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:02 smithi104 ceph-mon[36624]: pgmap v15228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:04.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:04 smithi081 ceph-mon[31453]: pgmap v15229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:04 smithi153 ceph-mon[36569]: pgmap v15229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:04 smithi104 ceph-mon[36624]: pgmap v15229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:06.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:06 smithi081 ceph-mon[31453]: pgmap v15230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:06 smithi153 ceph-mon[36569]: pgmap v15230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:06.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:06 smithi104 ceph-mon[36624]: pgmap v15230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:08.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:08 smithi081 ceph-mon[31453]: pgmap v15231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:08 smithi153 ceph-mon[36569]: pgmap v15231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:08 smithi104 ceph-mon[36624]: pgmap v15231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:10 smithi081 ceph-mon[31453]: pgmap v15232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:10 smithi153 ceph-mon[36569]: pgmap v15232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:10.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:10 smithi104 ceph-mon[36624]: pgmap v15232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:12 smithi153 ceph-mon[36569]: pgmap v15233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:12.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:12 smithi104 ceph-mon[36624]: pgmap v15233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:12 smithi081 ceph-mon[31453]: pgmap v15233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:14.297+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:09:14.297 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:14.297+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:09:14.298 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:14.297+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:09:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:14 smithi153 ceph-mon[36569]: pgmap v15234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:14 smithi104 ceph-mon[36624]: pgmap v15234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:14.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:14 smithi081 ceph-mon[31453]: pgmap v15234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:16 smithi153 ceph-mon[36569]: pgmap v15235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:16 smithi104 ceph-mon[36624]: pgmap v15235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:16.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:16 smithi081 ceph-mon[31453]: pgmap v15235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:18 smithi153 ceph-mon[36569]: pgmap v15236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:18 smithi104 ceph-mon[36624]: pgmap v15236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:18 smithi081 ceph-mon[31453]: pgmap v15236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:20 smithi153 ceph-mon[36569]: pgmap v15237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:20 smithi104 ceph-mon[36624]: pgmap v15237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:20 smithi081 ceph-mon[31453]: pgmap v15237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:22 smithi153 ceph-mon[36569]: pgmap v15238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:22.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:22 smithi104 ceph-mon[36624]: pgmap v15238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:22 smithi081 ceph-mon[31453]: pgmap v15238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:24.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:24 smithi081 ceph-mon[31453]: pgmap v15239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:24 smithi153 ceph-mon[36569]: pgmap v15239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:24 smithi104 ceph-mon[36624]: pgmap v15239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:26 smithi153 ceph-mon[36569]: pgmap v15240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:26 smithi104 ceph-mon[36624]: pgmap v15240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:26 smithi081 ceph-mon[31453]: pgmap v15240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:28 smithi153 ceph-mon[36569]: pgmap v15241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:28 smithi104 ceph-mon[36624]: pgmap v15241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:28 smithi081 ceph-mon[31453]: pgmap v15241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:29.313+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:09:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:29.313+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:09:29.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:29.313+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:09:30.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:30 smithi081 ceph-mon[31453]: pgmap v15242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:30 smithi153 ceph-mon[36569]: pgmap v15242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:30 smithi104 ceph-mon[36624]: pgmap v15242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:32 smithi153 ceph-mon[36569]: pgmap v15243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:32 smithi104 ceph-mon[36624]: pgmap v15243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:32 smithi081 ceph-mon[31453]: pgmap v15243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:34 smithi153 ceph-mon[36569]: pgmap v15244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:34 smithi104 ceph-mon[36624]: pgmap v15244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:34 smithi081 ceph-mon[31453]: pgmap v15244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:36 smithi153 ceph-mon[36569]: pgmap v15245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:36 smithi104 ceph-mon[36624]: pgmap v15245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:36 smithi081 ceph-mon[31453]: pgmap v15245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:38 smithi153 ceph-mon[36569]: pgmap v15246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:38 smithi104 ceph-mon[36624]: pgmap v15246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:38 smithi081 ceph-mon[31453]: pgmap v15246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:40 smithi081 ceph-mon[31453]: pgmap v15247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:09:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:40 smithi153 ceph-mon[36569]: pgmap v15247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:09:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:40 smithi104 ceph-mon[36624]: pgmap v15247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:09:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:09:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:09:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: pgmap v15248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: pgmap v15249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:41.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: pgmap v15248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: pgmap v15249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: pgmap v15248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: pgmap v15249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:42.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:09:44.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:43 smithi081 ceph-mon[31453]: pgmap v15250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:43 smithi153 ceph-mon[36569]: pgmap v15250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:44.329+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:09:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:44.329+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:09:44.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:44.329+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:09:44.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:43 smithi104 ceph-mon[36624]: pgmap v15250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:45 smithi081 ceph-mon[31453]: pgmap v15251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:45 smithi153 ceph-mon[36569]: pgmap v15251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:45 smithi104 ceph-mon[36624]: pgmap v15251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:47 smithi081 ceph-mon[31453]: pgmap v15252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:47 smithi153 ceph-mon[36569]: pgmap v15252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:48.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:47 smithi104 ceph-mon[36624]: pgmap v15252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:49 smithi081 ceph-mon[31453]: pgmap v15253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:49 smithi153 ceph-mon[36569]: pgmap v15253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:49 smithi104 ceph-mon[36624]: pgmap v15253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:51 smithi081 ceph-mon[31453]: pgmap v15254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:51 smithi153 ceph-mon[36569]: pgmap v15254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:51 smithi104 ceph-mon[36624]: pgmap v15254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:53 smithi153 ceph-mon[36569]: pgmap v15255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:54.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:53 smithi104 ceph-mon[36624]: pgmap v15255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:53 smithi081 ceph-mon[31453]: pgmap v15255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:56.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:55 smithi081 ceph-mon[31453]: pgmap v15256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:55 smithi153 ceph-mon[36569]: pgmap v15256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:55 smithi104 ceph-mon[36624]: pgmap v15256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:57 smithi153 ceph-mon[36569]: pgmap v15257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:58.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:57 smithi104 ceph-mon[36624]: pgmap v15257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:57 smithi081 ceph-mon[31453]: pgmap v15257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:09:59.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:59.333+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:09:59.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:59.333+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:09:59.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:09:59.333+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:10:00.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:09:59 smithi081 ceph-mon[31453]: pgmap v15258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:09:59 smithi153 ceph-mon[36569]: pgmap v15258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:00.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:09:59 smithi104 ceph-mon[36624]: pgmap v15258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:10:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:10:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:10:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:01 smithi153 ceph-mon[36569]: pgmap v15259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:01 smithi104 ceph-mon[36624]: pgmap v15259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:01 smithi081 ceph-mon[31453]: pgmap v15259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:03 smithi153 ceph-mon[36569]: pgmap v15260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:03 smithi104 ceph-mon[36624]: pgmap v15260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:03 smithi081 ceph-mon[31453]: pgmap v15260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:05 smithi153 ceph-mon[36569]: pgmap v15261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:05 smithi104 ceph-mon[36624]: pgmap v15261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:06.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:05 smithi081 ceph-mon[31453]: pgmap v15261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:07 smithi153 ceph-mon[36569]: pgmap v15262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:07 smithi104 ceph-mon[36624]: pgmap v15262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:07 smithi081 ceph-mon[31453]: pgmap v15262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:09 smithi153 ceph-mon[36569]: pgmap v15263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:09 smithi104 ceph-mon[36624]: pgmap v15263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:09 smithi081 ceph-mon[31453]: pgmap v15263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:11 smithi153 ceph-mon[36569]: pgmap v15264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:11 smithi104 ceph-mon[36624]: pgmap v15264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:11 smithi081 ceph-mon[31453]: pgmap v15264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:13 smithi153 ceph-mon[36569]: pgmap v15265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:13 smithi104 ceph-mon[36624]: pgmap v15265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:14.349+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:10:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:14.349+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:10:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:14.349+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:10:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:13 smithi081 ceph-mon[31453]: pgmap v15265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:15 smithi153 ceph-mon[36569]: pgmap v15266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:15 smithi104 ceph-mon[36624]: pgmap v15266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:15 smithi081 ceph-mon[31453]: pgmap v15266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:17.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:17.404+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:10:17.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:17.404+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:10:17.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:17.405+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 220 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:10:17.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:17.405+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 220 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:10:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:17 smithi153 ceph-mon[36569]: pgmap v15267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:17 smithi104 ceph-mon[36624]: pgmap v15267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:17 smithi081 ceph-mon[31453]: pgmap v15267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:19 smithi153 ceph-mon[36569]: pgmap v15268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:20.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:19 smithi104 ceph-mon[36624]: pgmap v15268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:19 smithi081 ceph-mon[31453]: pgmap v15268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:21 smithi153 ceph-mon[36569]: pgmap v15269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:21 smithi104 ceph-mon[36624]: pgmap v15269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:21 smithi081 ceph-mon[31453]: pgmap v15269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:23 smithi153 ceph-mon[36569]: pgmap v15270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:23 smithi104 ceph-mon[36624]: pgmap v15270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:23 smithi081 ceph-mon[31453]: pgmap v15270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:26 smithi153 ceph-mon[36569]: pgmap v15271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:26 smithi104 ceph-mon[36624]: pgmap v15271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:26 smithi081 ceph-mon[31453]: pgmap v15271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:28 smithi153 ceph-mon[36569]: pgmap v15272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:28 smithi104 ceph-mon[36624]: pgmap v15272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:28 smithi081 ceph-mon[31453]: pgmap v15272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:29.359+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:10:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:29.359+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:10:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:29.359+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:10:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:30 smithi153 ceph-mon[36569]: pgmap v15273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:30 smithi104 ceph-mon[36624]: pgmap v15273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:30 smithi081 ceph-mon[31453]: pgmap v15273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:32 smithi153 ceph-mon[36569]: pgmap v15274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:32 smithi104 ceph-mon[36624]: pgmap v15274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:32 smithi081 ceph-mon[31453]: pgmap v15274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:34 smithi153 ceph-mon[36569]: pgmap v15275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:34 smithi104 ceph-mon[36624]: pgmap v15275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:34 smithi081 ceph-mon[31453]: pgmap v15275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:36 smithi153 ceph-mon[36569]: pgmap v15276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:36 smithi104 ceph-mon[36624]: pgmap v15276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:36 smithi081 ceph-mon[31453]: pgmap v15276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:38 smithi153 ceph-mon[36569]: pgmap v15277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:38 smithi104 ceph-mon[36624]: pgmap v15277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:38 smithi081 ceph-mon[31453]: pgmap v15277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:40 smithi153 ceph-mon[36569]: pgmap v15278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:40 smithi104 ceph-mon[36624]: pgmap v15278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:40 smithi081 ceph-mon[31453]: pgmap v15278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: pgmap v15279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: pgmap v15279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: pgmap v15279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:10:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:10:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:10:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:10:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:10:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: pgmap v15280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: pgmap v15281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:10:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: pgmap v15280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: pgmap v15281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:43.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:10:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: pgmap v15280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: pgmap v15281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:10:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:44.376+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:10:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:44.376+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:10:44.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:44.376+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:10:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:44 smithi153 ceph-mon[36569]: pgmap v15282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:44 smithi104 ceph-mon[36624]: pgmap v15282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:44 smithi081 ceph-mon[31453]: pgmap v15282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:46 smithi153 ceph-mon[36569]: pgmap v15283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:46 smithi104 ceph-mon[36624]: pgmap v15283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:46 smithi081 ceph-mon[31453]: pgmap v15283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:48 smithi153 ceph-mon[36569]: pgmap v15284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:48 smithi104 ceph-mon[36624]: pgmap v15284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:48 smithi081 ceph-mon[31453]: pgmap v15284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:50 smithi153 ceph-mon[36569]: pgmap v15285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:50 smithi104 ceph-mon[36624]: pgmap v15285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:50 smithi081 ceph-mon[31453]: pgmap v15285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:52 smithi153 ceph-mon[36569]: pgmap v15286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:52 smithi104 ceph-mon[36624]: pgmap v15286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:52 smithi081 ceph-mon[31453]: pgmap v15286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:54 smithi153 ceph-mon[36569]: pgmap v15287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:54 smithi104 ceph-mon[36624]: pgmap v15287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:54 smithi081 ceph-mon[31453]: pgmap v15287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:56 smithi153 ceph-mon[36569]: pgmap v15288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:56 smithi104 ceph-mon[36624]: pgmap v15288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:56 smithi081 ceph-mon[31453]: pgmap v15288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:10:58 smithi153 ceph-mon[36569]: pgmap v15289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:10:58 smithi104 ceph-mon[36624]: pgmap v15289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:10:58 smithi081 ceph-mon[31453]: pgmap v15289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:10:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:59.391+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:10:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:59.391+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:10:59.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:10:59.391+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:11:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:00 smithi153 ceph-mon[36569]: pgmap v15290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:00.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:00 smithi081 ceph-mon[31453]: pgmap v15290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:00 smithi104 ceph-mon[36624]: pgmap v15290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:02 smithi153 ceph-mon[36569]: pgmap v15291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:02 smithi104 ceph-mon[36624]: pgmap v15291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:02.857 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:02 smithi081 ceph-mon[31453]: pgmap v15291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:04 smithi153 ceph-mon[36569]: pgmap v15292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:04 smithi104 ceph-mon[36624]: pgmap v15292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:04 smithi081 ceph-mon[31453]: pgmap v15292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:06 smithi153 ceph-mon[36569]: pgmap v15293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:06 smithi104 ceph-mon[36624]: pgmap v15293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:06 smithi081 ceph-mon[31453]: pgmap v15293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:08 smithi153 ceph-mon[36569]: pgmap v15294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:08 smithi104 ceph-mon[36624]: pgmap v15294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:08 smithi081 ceph-mon[31453]: pgmap v15294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:10 smithi153 ceph-mon[36569]: pgmap v15295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:10 smithi104 ceph-mon[36624]: pgmap v15295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:10 smithi081 ceph-mon[31453]: pgmap v15295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:12 smithi081 ceph-mon[31453]: pgmap v15296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:12 smithi153 ceph-mon[36569]: pgmap v15296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:12 smithi104 ceph-mon[36624]: pgmap v15296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:14.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:14.407+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:11:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:14.407+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:11:14.408 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:14.407+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:11:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:14 smithi081 ceph-mon[31453]: pgmap v15297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:14 smithi153 ceph-mon[36569]: pgmap v15297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:14 smithi104 ceph-mon[36624]: pgmap v15297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:16 smithi153 ceph-mon[36569]: pgmap v15298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:16 smithi104 ceph-mon[36624]: pgmap v15298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:16 smithi081 ceph-mon[31453]: pgmap v15298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:18 smithi153 ceph-mon[36569]: pgmap v15299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:18 smithi104 ceph-mon[36624]: pgmap v15299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:18 smithi081 ceph-mon[31453]: pgmap v15299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:20 smithi153 ceph-mon[36569]: pgmap v15300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:20 smithi104 ceph-mon[36624]: pgmap v15300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:20 smithi081 ceph-mon[31453]: pgmap v15300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:22 smithi153 ceph-mon[36569]: pgmap v15301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:22 smithi104 ceph-mon[36624]: pgmap v15301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:22 smithi081 ceph-mon[31453]: pgmap v15301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:24 smithi153 ceph-mon[36569]: pgmap v15302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:24 smithi104 ceph-mon[36624]: pgmap v15302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:24 smithi081 ceph-mon[31453]: pgmap v15302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:26 smithi153 ceph-mon[36569]: pgmap v15303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:26 smithi104 ceph-mon[36624]: pgmap v15303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:26 smithi081 ceph-mon[31453]: pgmap v15303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:28 smithi153 ceph-mon[36569]: pgmap v15304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:28 smithi104 ceph-mon[36624]: pgmap v15304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:28 smithi081 ceph-mon[31453]: pgmap v15304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:29.416+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:11:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:29.416+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:11:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:29.416+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:11:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:30 smithi153 ceph-mon[36569]: pgmap v15305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:30 smithi104 ceph-mon[36624]: pgmap v15305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:30 smithi081 ceph-mon[31453]: pgmap v15305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:32 smithi153 ceph-mon[36569]: pgmap v15306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:32 smithi104 ceph-mon[36624]: pgmap v15306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:32 smithi081 ceph-mon[31453]: pgmap v15306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:34 smithi153 ceph-mon[36569]: pgmap v15307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:34 smithi104 ceph-mon[36624]: pgmap v15307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:34 smithi081 ceph-mon[31453]: pgmap v15307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:36 smithi153 ceph-mon[36569]: pgmap v15308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:36 smithi104 ceph-mon[36624]: pgmap v15308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:36 smithi081 ceph-mon[31453]: pgmap v15308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:38.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:38 smithi153 ceph-mon[36569]: pgmap v15309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:38 smithi104 ceph-mon[36624]: pgmap v15309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:38 smithi081 ceph-mon[31453]: pgmap v15309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:40 smithi153 ceph-mon[36569]: pgmap v15310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:40 smithi104 ceph-mon[36624]: pgmap v15310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:40 smithi081 ceph-mon[31453]: pgmap v15310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:42 smithi153 ceph-mon[36569]: pgmap v15311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:42 smithi104 ceph-mon[36624]: pgmap v15311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:42 smithi081 ceph-mon[31453]: pgmap v15311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:11:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:11:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:11:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:44.430+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:11:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:44.430+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:11:44.430 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:44.430+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:11:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:11:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:11:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:44 smithi153 ceph-mon[36569]: pgmap v15312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:44 smithi153 ceph-mon[36569]: pgmap v15313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:11:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:11:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:11:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:44 smithi104 ceph-mon[36624]: pgmap v15312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:44 smithi104 ceph-mon[36624]: pgmap v15313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:11:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:11:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:11:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:44 smithi081 ceph-mon[31453]: pgmap v15312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:44 smithi081 ceph-mon[31453]: pgmap v15313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:11:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:47 smithi153 ceph-mon[36569]: pgmap v15314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:47 smithi104 ceph-mon[36624]: pgmap v15314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:47 smithi081 ceph-mon[31453]: pgmap v15314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:49 smithi153 ceph-mon[36569]: pgmap v15315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:49 smithi104 ceph-mon[36624]: pgmap v15315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:49 smithi081 ceph-mon[31453]: pgmap v15315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:51 smithi153 ceph-mon[36569]: pgmap v15316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:51 smithi104 ceph-mon[36624]: pgmap v15316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:51 smithi081 ceph-mon[31453]: pgmap v15316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:53 smithi153 ceph-mon[36569]: pgmap v15317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:53 smithi104 ceph-mon[36624]: pgmap v15317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:53 smithi081 ceph-mon[31453]: pgmap v15317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:55 smithi153 ceph-mon[36569]: pgmap v15318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:55 smithi104 ceph-mon[36624]: pgmap v15318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:55 smithi081 ceph-mon[31453]: pgmap v15318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:57 smithi153 ceph-mon[36569]: pgmap v15319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:57 smithi104 ceph-mon[36624]: pgmap v15319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:57 smithi081 ceph-mon[31453]: pgmap v15319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:11:59 smithi153 ceph-mon[36569]: pgmap v15320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:11:59 smithi104 ceph-mon[36624]: pgmap v15320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:11:59 smithi081 ceph-mon[31453]: pgmap v15320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:11:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:59.437+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:11:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:59.437+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:11:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:11:59.437+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:12:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:01 smithi081 ceph-mon[31453]: pgmap v15321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:01 smithi104 ceph-mon[36624]: pgmap v15321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:01 smithi153 ceph-mon[36569]: pgmap v15321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:03 smithi104 ceph-mon[36624]: pgmap v15322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:03 smithi081 ceph-mon[31453]: pgmap v15322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:03 smithi153 ceph-mon[36569]: pgmap v15322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:05 smithi104 ceph-mon[36624]: pgmap v15323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:05 smithi081 ceph-mon[31453]: pgmap v15323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:05 smithi153 ceph-mon[36569]: pgmap v15323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:07 smithi104 ceph-mon[36624]: pgmap v15324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:07 smithi081 ceph-mon[31453]: pgmap v15324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:07 smithi153 ceph-mon[36569]: pgmap v15324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:09 smithi104 ceph-mon[36624]: pgmap v15325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:09 smithi081 ceph-mon[31453]: pgmap v15325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:09 smithi153 ceph-mon[36569]: pgmap v15325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:11 smithi081 ceph-mon[31453]: pgmap v15326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:11 smithi153 ceph-mon[36569]: pgmap v15326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:11 smithi104 ceph-mon[36624]: pgmap v15326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:13 smithi081 ceph-mon[31453]: pgmap v15327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:13 smithi153 ceph-mon[36569]: pgmap v15327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:13 smithi104 ceph-mon[36624]: pgmap v15327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:14.445+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:12:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:14.445+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:12:14.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:14.445+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:12:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:15 smithi081 ceph-mon[31453]: pgmap v15328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:15 smithi153 ceph-mon[36569]: pgmap v15328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:15 smithi104 ceph-mon[36624]: pgmap v15328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:17 smithi081 ceph-mon[31453]: pgmap v15329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:17 smithi153 ceph-mon[36569]: pgmap v15329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:17 smithi104 ceph-mon[36624]: pgmap v15329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:19 smithi081 ceph-mon[31453]: pgmap v15330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:19 smithi153 ceph-mon[36569]: pgmap v15330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:19 smithi104 ceph-mon[36624]: pgmap v15330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:21 smithi081 ceph-mon[31453]: pgmap v15331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:21 smithi153 ceph-mon[36569]: pgmap v15331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:21.641 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:21 smithi104 ceph-mon[36624]: pgmap v15331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:23 smithi081 ceph-mon[31453]: pgmap v15332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:23 smithi153 ceph-mon[36569]: pgmap v15332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:23 smithi104 ceph-mon[36624]: pgmap v15332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:25 smithi081 ceph-mon[31453]: pgmap v15333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:25 smithi153 ceph-mon[36569]: pgmap v15333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:25 smithi104 ceph-mon[36624]: pgmap v15333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:27 smithi081 ceph-mon[31453]: pgmap v15334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:27 smithi153 ceph-mon[36569]: pgmap v15334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:27 smithi104 ceph-mon[36624]: pgmap v15334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:28 smithi081 ceph-mon[31453]: pgmap v15335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:28 smithi153 ceph-mon[36569]: pgmap v15335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:28 smithi104 ceph-mon[36624]: pgmap v15335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:29.450+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:12:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:29.450+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:12:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:29.450+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:12:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:31 smithi153 ceph-mon[36569]: pgmap v15336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:31 smithi104 ceph-mon[36624]: pgmap v15336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:31 smithi081 ceph-mon[31453]: pgmap v15336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:33 smithi153 ceph-mon[36569]: pgmap v15337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:33 smithi104 ceph-mon[36624]: pgmap v15337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:33 smithi081 ceph-mon[31453]: pgmap v15337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:35 smithi104 ceph-mon[36624]: pgmap v15338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:35 smithi081 ceph-mon[31453]: pgmap v15338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:35 smithi153 ceph-mon[36569]: pgmap v15338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:37 smithi104 ceph-mon[36624]: pgmap v15339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:37 smithi081 ceph-mon[31453]: pgmap v15339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:37 smithi153 ceph-mon[36569]: pgmap v15339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:39 smithi104 ceph-mon[36624]: pgmap v15340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:39 smithi081 ceph-mon[31453]: pgmap v15340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:39 smithi153 ceph-mon[36569]: pgmap v15340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:41 smithi104 ceph-mon[36624]: pgmap v15341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:41 smithi081 ceph-mon[31453]: pgmap v15341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:41 smithi153 ceph-mon[36569]: pgmap v15341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:43 smithi104 ceph-mon[36624]: pgmap v15342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:43 smithi081 ceph-mon[31453]: pgmap v15342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:43 smithi153 ceph-mon[36569]: pgmap v15342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:44.465+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:12:44.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:44.465+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:12:44.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:44.465+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:12:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:45 smithi081 ceph-mon[31453]: pgmap v15343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:12:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:45 smithi153 ceph-mon[36569]: pgmap v15343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:12:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:45 smithi104 ceph-mon[36624]: pgmap v15343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:12:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:12:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:12:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:12:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:12:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:12:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:12:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:12:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:12:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:12:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:47 smithi081 ceph-mon[31453]: pgmap v15344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:47 smithi081 ceph-mon[31453]: pgmap v15345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:47 smithi153 ceph-mon[36569]: pgmap v15344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:47 smithi153 ceph-mon[36569]: pgmap v15345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:47 smithi104 ceph-mon[36624]: pgmap v15344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:47 smithi104 ceph-mon[36624]: pgmap v15345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:49 smithi081 ceph-mon[31453]: pgmap v15346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:49 smithi153 ceph-mon[36569]: pgmap v15346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:49 smithi104 ceph-mon[36624]: pgmap v15346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:51 smithi081 ceph-mon[31453]: pgmap v15347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:51 smithi153 ceph-mon[36569]: pgmap v15347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:51 smithi104 ceph-mon[36624]: pgmap v15347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:53 smithi081 ceph-mon[31453]: pgmap v15348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:53 smithi153 ceph-mon[36569]: pgmap v15348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:53 smithi104 ceph-mon[36624]: pgmap v15348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:55 smithi081 ceph-mon[31453]: pgmap v15349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:55 smithi153 ceph-mon[36569]: pgmap v15349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:55 smithi104 ceph-mon[36624]: pgmap v15349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:57 smithi081 ceph-mon[31453]: pgmap v15350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:57 smithi153 ceph-mon[36569]: pgmap v15350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:57 smithi104 ceph-mon[36624]: pgmap v15350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:12:59 smithi081 ceph-mon[31453]: pgmap v15351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:59.472+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:12:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:59.472+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:12:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:12:59.472+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:12:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:12:59 smithi153 ceph-mon[36569]: pgmap v15351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:12:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:12:59 smithi104 ceph-mon[36624]: pgmap v15351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:01 smithi081 ceph-mon[31453]: pgmap v15352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:01 smithi153 ceph-mon[36569]: pgmap v15352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:01 smithi104 ceph-mon[36624]: pgmap v15352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:03 smithi081 ceph-mon[31453]: pgmap v15353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:03 smithi153 ceph-mon[36569]: pgmap v15353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:03 smithi104 ceph-mon[36624]: pgmap v15353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:05 smithi081 ceph-mon[31453]: pgmap v15354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:05 smithi153 ceph-mon[36569]: pgmap v15354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:05 smithi104 ceph-mon[36624]: pgmap v15354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:07.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:07 smithi081 ceph-mon[31453]: pgmap v15355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:07 smithi153 ceph-mon[36569]: pgmap v15355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:07 smithi104 ceph-mon[36624]: pgmap v15355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:09 smithi153 ceph-mon[36569]: pgmap v15356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:09 smithi104 ceph-mon[36624]: pgmap v15356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:09 smithi081 ceph-mon[31453]: pgmap v15356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:11 smithi153 ceph-mon[36569]: pgmap v15357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:11 smithi104 ceph-mon[36624]: pgmap v15357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:11 smithi081 ceph-mon[31453]: pgmap v15357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:13 smithi153 ceph-mon[36569]: pgmap v15358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:13 smithi104 ceph-mon[36624]: pgmap v15358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:13 smithi081 ceph-mon[31453]: pgmap v15358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:14.487 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:14.487+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:13:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:14.487+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:13:14.488 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:14.487+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:13:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:15 smithi153 ceph-mon[36569]: pgmap v15359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:15 smithi104 ceph-mon[36624]: pgmap v15359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:15 smithi081 ceph-mon[31453]: pgmap v15359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:17.433+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:13:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:17.433+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:13:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:17.434+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 221 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:13:17.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:17.434+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 221 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:13:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:17 smithi153 ceph-mon[36569]: pgmap v15360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:17 smithi104 ceph-mon[36624]: pgmap v15360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:17 smithi081 ceph-mon[31453]: pgmap v15360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:19 smithi153 ceph-mon[36569]: pgmap v15361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:19 smithi104 ceph-mon[36624]: pgmap v15361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:19 smithi081 ceph-mon[31453]: pgmap v15361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:21 smithi153 ceph-mon[36569]: pgmap v15362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:21 smithi104 ceph-mon[36624]: pgmap v15362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:21 smithi081 ceph-mon[31453]: pgmap v15362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:23 smithi153 ceph-mon[36569]: pgmap v15363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:23 smithi104 ceph-mon[36624]: pgmap v15363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:23 smithi081 ceph-mon[31453]: pgmap v15363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:25 smithi153 ceph-mon[36569]: pgmap v15364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:25 smithi104 ceph-mon[36624]: pgmap v15364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:25 smithi081 ceph-mon[31453]: pgmap v15364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:27 smithi153 ceph-mon[36569]: pgmap v15365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:27 smithi104 ceph-mon[36624]: pgmap v15365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:27 smithi081 ceph-mon[31453]: pgmap v15365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:29.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:29.502+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:13:29.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:29.502+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:13:29.503 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:29.502+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:13:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:29 smithi153 ceph-mon[36569]: pgmap v15366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:29 smithi104 ceph-mon[36624]: pgmap v15366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:29 smithi081 ceph-mon[31453]: pgmap v15366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:31 smithi153 ceph-mon[36569]: pgmap v15367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:31 smithi104 ceph-mon[36624]: pgmap v15367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:31 smithi081 ceph-mon[31453]: pgmap v15367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:33 smithi153 ceph-mon[36569]: pgmap v15368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:33 smithi104 ceph-mon[36624]: pgmap v15368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:33 smithi081 ceph-mon[31453]: pgmap v15368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:35 smithi153 ceph-mon[36569]: pgmap v15369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:35.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:35 smithi104 ceph-mon[36624]: pgmap v15369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:35 smithi081 ceph-mon[31453]: pgmap v15369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:37 smithi153 ceph-mon[36569]: pgmap v15370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:37 smithi104 ceph-mon[36624]: pgmap v15370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:37 smithi081 ceph-mon[31453]: pgmap v15370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:39 smithi153 ceph-mon[36569]: pgmap v15371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:39 smithi104 ceph-mon[36624]: pgmap v15371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:39 smithi081 ceph-mon[31453]: pgmap v15371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:41 smithi153 ceph-mon[36569]: pgmap v15372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:41 smithi104 ceph-mon[36624]: pgmap v15372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:41 smithi081 ceph-mon[31453]: pgmap v15372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:43 smithi153 ceph-mon[36569]: pgmap v15373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:43 smithi104 ceph-mon[36624]: pgmap v15373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:43 smithi081 ceph-mon[31453]: pgmap v15373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:44.505+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:13:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:44.505+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:13:44.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:44.505+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:13:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:45 smithi153 ceph-mon[36569]: pgmap v15374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:45 smithi104 ceph-mon[36624]: pgmap v15374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:45 smithi081 ceph-mon[31453]: pgmap v15374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:13:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:13:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:13:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:47 smithi153 ceph-mon[36569]: pgmap v15375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:13:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:13:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:13:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:47 smithi104 ceph-mon[36624]: pgmap v15375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:13:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:13:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:13:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:47 smithi081 ceph-mon[31453]: pgmap v15375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:13:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:13:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:13:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:48 smithi153 ceph-mon[36569]: pgmap v15376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:48 smithi153 ceph-mon[36569]: pgmap v15377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:48 smithi104 ceph-mon[36624]: pgmap v15376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:48 smithi104 ceph-mon[36624]: pgmap v15377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:48 smithi081 ceph-mon[31453]: pgmap v15376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:48 smithi081 ceph-mon[31453]: pgmap v15377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:50 smithi153 ceph-mon[36569]: pgmap v15378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:50 smithi104 ceph-mon[36624]: pgmap v15378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:50 smithi081 ceph-mon[31453]: pgmap v15378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:52 smithi153 ceph-mon[36569]: pgmap v15379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:52 smithi104 ceph-mon[36624]: pgmap v15379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:52 smithi081 ceph-mon[31453]: pgmap v15379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:54.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:54 smithi153 ceph-mon[36569]: pgmap v15380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:54 smithi104 ceph-mon[36624]: pgmap v15380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:54 smithi081 ceph-mon[31453]: pgmap v15380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:56 smithi153 ceph-mon[36569]: pgmap v15381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:56 smithi104 ceph-mon[36624]: pgmap v15381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:56 smithi081 ceph-mon[31453]: pgmap v15381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:13:58 smithi153 ceph-mon[36569]: pgmap v15382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:13:58 smithi104 ceph-mon[36624]: pgmap v15382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:13:58 smithi081 ceph-mon[31453]: pgmap v15382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:13:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:59.520+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:13:59.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:59.520+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:13:59.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:13:59.520+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:14:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:00 smithi153 ceph-mon[36569]: pgmap v15383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:00 smithi104 ceph-mon[36624]: pgmap v15383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:00 smithi081 ceph-mon[31453]: pgmap v15383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:02 smithi104 ceph-mon[36624]: pgmap v15384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:02 smithi081 ceph-mon[31453]: pgmap v15384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:02 smithi153 ceph-mon[36569]: pgmap v15384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:04 smithi104 ceph-mon[36624]: pgmap v15385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:04 smithi081 ceph-mon[31453]: pgmap v15385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:04.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:04 smithi153 ceph-mon[36569]: pgmap v15385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:06 smithi104 ceph-mon[36624]: pgmap v15386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:06 smithi081 ceph-mon[31453]: pgmap v15386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:06 smithi153 ceph-mon[36569]: pgmap v15386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:08 smithi104 ceph-mon[36624]: pgmap v15387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:08 smithi081 ceph-mon[31453]: pgmap v15387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:08 smithi153 ceph-mon[36569]: pgmap v15387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:10 smithi081 ceph-mon[31453]: pgmap v15388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:10 smithi153 ceph-mon[36569]: pgmap v15388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:10.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:10 smithi104 ceph-mon[36624]: pgmap v15388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:12 smithi081 ceph-mon[31453]: pgmap v15389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:12 smithi153 ceph-mon[36569]: pgmap v15389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:12.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:12 smithi104 ceph-mon[36624]: pgmap v15389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:14.535+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:14:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:14.536+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:14:14.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:14.536+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:14:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:14 smithi081 ceph-mon[31453]: pgmap v15390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:14 smithi153 ceph-mon[36569]: pgmap v15390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:14.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:14 smithi104 ceph-mon[36624]: pgmap v15390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:16 smithi081 ceph-mon[31453]: pgmap v15391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:16 smithi153 ceph-mon[36569]: pgmap v15391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:16 smithi104 ceph-mon[36624]: pgmap v15391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:18 smithi081 ceph-mon[31453]: pgmap v15392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:18 smithi153 ceph-mon[36569]: pgmap v15392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:18 smithi104 ceph-mon[36624]: pgmap v15392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:20 smithi081 ceph-mon[31453]: pgmap v15393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:20 smithi153 ceph-mon[36569]: pgmap v15393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:20 smithi104 ceph-mon[36624]: pgmap v15393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:22.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:22 smithi081 ceph-mon[31453]: pgmap v15394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:22 smithi153 ceph-mon[36569]: pgmap v15394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:22 smithi104 ceph-mon[36624]: pgmap v15394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:24.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:24 smithi081 ceph-mon[31453]: pgmap v15395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:24 smithi153 ceph-mon[36569]: pgmap v15395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:24 smithi104 ceph-mon[36624]: pgmap v15395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:26 smithi081 ceph-mon[31453]: pgmap v15396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:26 smithi153 ceph-mon[36569]: pgmap v15396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:26 smithi104 ceph-mon[36624]: pgmap v15396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:28 smithi081 ceph-mon[31453]: pgmap v15397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:28 smithi153 ceph-mon[36569]: pgmap v15397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:28.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:28 smithi104 ceph-mon[36624]: pgmap v15397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:29.548+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:14:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:29.548+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:14:29.549 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:29.548+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:14:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:30 smithi081 ceph-mon[31453]: pgmap v15398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:30 smithi153 ceph-mon[36569]: pgmap v15398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:30 smithi104 ceph-mon[36624]: pgmap v15398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:32 smithi081 ceph-mon[31453]: pgmap v15399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:32 smithi153 ceph-mon[36569]: pgmap v15399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:32 smithi104 ceph-mon[36624]: pgmap v15399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:34 smithi153 ceph-mon[36569]: pgmap v15400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:34 smithi104 ceph-mon[36624]: pgmap v15400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:34 smithi081 ceph-mon[31453]: pgmap v15400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:36 smithi153 ceph-mon[36569]: pgmap v15401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:36 smithi104 ceph-mon[36624]: pgmap v15401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:36 smithi081 ceph-mon[31453]: pgmap v15401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:38.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:38 smithi153 ceph-mon[36569]: pgmap v15402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:38 smithi104 ceph-mon[36624]: pgmap v15402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:38 smithi081 ceph-mon[31453]: pgmap v15402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:40.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:40 smithi153 ceph-mon[36569]: pgmap v15403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:40.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:40 smithi104 ceph-mon[36624]: pgmap v15403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:40 smithi081 ceph-mon[31453]: pgmap v15403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:42 smithi153 ceph-mon[36569]: pgmap v15404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:42 smithi104 ceph-mon[36624]: pgmap v15404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:42.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:42 smithi081 ceph-mon[31453]: pgmap v15404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:44.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:44.548+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:14:44.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:44.548+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:14:44.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:44.548+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:14:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:44 smithi153 ceph-mon[36569]: pgmap v15405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:44 smithi104 ceph-mon[36624]: pgmap v15405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:44 smithi081 ceph-mon[31453]: pgmap v15405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:46 smithi153 ceph-mon[36569]: pgmap v15406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:46 smithi104 ceph-mon[36624]: pgmap v15406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:46 smithi081 ceph-mon[31453]: pgmap v15406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:47.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:47.448+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:14:47.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:47.448+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:14:47.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:47.450+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 222 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:14:47.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:47.450+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 222 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:14:48.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:48 smithi153 ceph-mon[36569]: pgmap v15407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:48.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:14:48.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.545 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:48 smithi104 ceph-mon[36624]: pgmap v15407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:14:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:48 smithi081 ceph-mon[31453]: pgmap v15407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:14:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:50 smithi153 ceph-mon[36569]: pgmap v15408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:50 smithi104 ceph-mon[36624]: pgmap v15408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:50 smithi081 ceph-mon[31453]: pgmap v15408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:51 smithi153 ceph-mon[36569]: pgmap v15409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:52.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:51 smithi104 ceph-mon[36624]: pgmap v15409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:52.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:51 smithi081 ceph-mon[31453]: pgmap v15409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:53 smithi104 ceph-mon[36624]: pgmap v15410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:53 smithi153 ceph-mon[36569]: pgmap v15410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:53 smithi081 ceph-mon[31453]: pgmap v15410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:56 smithi081 ceph-mon[31453]: pgmap v15411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:56 smithi153 ceph-mon[36569]: pgmap v15411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:56 smithi104 ceph-mon[36624]: pgmap v15411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:14:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:14:58 smithi153 ceph-mon[36569]: pgmap v15412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:14:58 smithi104 ceph-mon[36624]: pgmap v15412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:14:58 smithi081 ceph-mon[31453]: pgmap v15412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:14:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:59.565+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:14:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:59.565+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:14:59.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:14:59.565+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:15:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:00 smithi153 ceph-mon[36569]: pgmap v15413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:15:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:15:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:00 smithi104 ceph-mon[36624]: pgmap v15413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:15:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:15:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:00 smithi081 ceph-mon[31453]: pgmap v15413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:15:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:15:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:15:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:01 smithi153 ceph-mon[36569]: pgmap v15414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:01 smithi153 ceph-mon[36569]: pgmap v15415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:01 smithi104 ceph-mon[36624]: pgmap v15414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:01 smithi104 ceph-mon[36624]: pgmap v15415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:01.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:01 smithi081 ceph-mon[31453]: pgmap v15414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:01.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:01 smithi081 ceph-mon[31453]: pgmap v15415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:03.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:03 smithi081 ceph-mon[31453]: pgmap v15416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:03 smithi153 ceph-mon[36569]: pgmap v15416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:03 smithi104 ceph-mon[36624]: pgmap v15416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:05.373 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:05 smithi081 ceph-mon[31453]: pgmap v15417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:05 smithi153 ceph-mon[36569]: pgmap v15417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:05 smithi104 ceph-mon[36624]: pgmap v15417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:07 smithi153 ceph-mon[36569]: pgmap v15418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:07 smithi104 ceph-mon[36624]: pgmap v15418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:07 smithi081 ceph-mon[31453]: pgmap v15418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:09 smithi153 ceph-mon[36569]: pgmap v15419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:09 smithi104 ceph-mon[36624]: pgmap v15419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:09 smithi081 ceph-mon[31453]: pgmap v15419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:11 smithi153 ceph-mon[36569]: pgmap v15420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:11 smithi104 ceph-mon[36624]: pgmap v15420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:11 smithi081 ceph-mon[31453]: pgmap v15420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:13 smithi153 ceph-mon[36569]: pgmap v15421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:13 smithi104 ceph-mon[36624]: pgmap v15421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:13 smithi081 ceph-mon[31453]: pgmap v15421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:14.581+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:15:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:14.581+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:15:14.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:14.581+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:15:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:15 smithi153 ceph-mon[36569]: pgmap v15422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:15 smithi104 ceph-mon[36624]: pgmap v15422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:15 smithi081 ceph-mon[31453]: pgmap v15422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:17 smithi153 ceph-mon[36569]: pgmap v15423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:17 smithi104 ceph-mon[36624]: pgmap v15423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:17 smithi081 ceph-mon[31453]: pgmap v15423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:19 smithi153 ceph-mon[36569]: pgmap v15424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:19 smithi104 ceph-mon[36624]: pgmap v15424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:19 smithi081 ceph-mon[31453]: pgmap v15424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:21 smithi153 ceph-mon[36569]: pgmap v15425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:21 smithi104 ceph-mon[36624]: pgmap v15425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:21 smithi081 ceph-mon[31453]: pgmap v15425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:23 smithi081 ceph-mon[31453]: pgmap v15426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:23 smithi153 ceph-mon[36569]: pgmap v15426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:23 smithi104 ceph-mon[36624]: pgmap v15426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:25 smithi153 ceph-mon[36569]: pgmap v15427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:25 smithi104 ceph-mon[36624]: pgmap v15427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:25.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:25 smithi081 ceph-mon[31453]: pgmap v15427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:27 smithi153 ceph-mon[36569]: pgmap v15428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:27 smithi104 ceph-mon[36624]: pgmap v15428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:27 smithi081 ceph-mon[31453]: pgmap v15428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:29 smithi153 ceph-mon[36569]: pgmap v15429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:29 smithi104 ceph-mon[36624]: pgmap v15429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:29.590+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:15:29.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:29.590+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:15:29.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:29.590+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:15:29.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:29 smithi081 ceph-mon[31453]: pgmap v15429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:31 smithi153 ceph-mon[36569]: pgmap v15430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:31 smithi104 ceph-mon[36624]: pgmap v15430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:31 smithi081 ceph-mon[31453]: pgmap v15430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:33 smithi153 ceph-mon[36569]: pgmap v15431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:33 smithi104 ceph-mon[36624]: pgmap v15431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:33 smithi081 ceph-mon[31453]: pgmap v15431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:35 smithi153 ceph-mon[36569]: pgmap v15432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:35 smithi104 ceph-mon[36624]: pgmap v15432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:35 smithi081 ceph-mon[31453]: pgmap v15432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:37 smithi153 ceph-mon[36569]: pgmap v15433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:37.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:37 smithi104 ceph-mon[36624]: pgmap v15433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:37 smithi081 ceph-mon[31453]: pgmap v15433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:39 smithi153 ceph-mon[36569]: pgmap v15434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:39 smithi104 ceph-mon[36624]: pgmap v15434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:39 smithi081 ceph-mon[31453]: pgmap v15434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:41 smithi153 ceph-mon[36569]: pgmap v15435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:41 smithi104 ceph-mon[36624]: pgmap v15435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:41 smithi081 ceph-mon[31453]: pgmap v15435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:43 smithi153 ceph-mon[36569]: pgmap v15436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:43 smithi104 ceph-mon[36624]: pgmap v15436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:43 smithi081 ceph-mon[31453]: pgmap v15436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:44.607 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:44.606+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:15:44.607 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:44.606+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:15:44.607 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:44.606+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:15:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:45 smithi153 ceph-mon[36569]: pgmap v15437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:45 smithi104 ceph-mon[36624]: pgmap v15437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:45 smithi081 ceph-mon[31453]: pgmap v15437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:47 smithi104 ceph-mon[36624]: pgmap v15438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:47 smithi081 ceph-mon[31453]: pgmap v15438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:47 smithi153 ceph-mon[36569]: pgmap v15438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:49 smithi104 ceph-mon[36624]: pgmap v15439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:49 smithi081 ceph-mon[31453]: pgmap v15439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:49 smithi153 ceph-mon[36569]: pgmap v15439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:51 smithi104 ceph-mon[36624]: pgmap v15440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:51.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:51 smithi081 ceph-mon[31453]: pgmap v15440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:51 smithi153 ceph-mon[36569]: pgmap v15440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:53 smithi104 ceph-mon[36624]: pgmap v15441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:53 smithi081 ceph-mon[31453]: pgmap v15441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:53.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:53 smithi153 ceph-mon[36569]: pgmap v15441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:55 smithi104 ceph-mon[36624]: pgmap v15442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:55 smithi081 ceph-mon[31453]: pgmap v15442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:55 smithi153 ceph-mon[36569]: pgmap v15442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:57.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:57 smithi081 ceph-mon[31453]: pgmap v15443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:57 smithi153 ceph-mon[36569]: pgmap v15443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:57 smithi104 ceph-mon[36624]: pgmap v15443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:59.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:15:59 smithi153 ceph-mon[36569]: pgmap v15444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:59.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:59.617+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:15:59.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:59.617+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:15:59.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:15:59.617+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:15:59.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:15:59 smithi081 ceph-mon[31453]: pgmap v15444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:15:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:15:59 smithi104 ceph-mon[36624]: pgmap v15444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:16:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:16:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:16:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:01 smithi081 ceph-mon[31453]: pgmap v15445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:01.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:01 smithi153 ceph-mon[36569]: pgmap v15445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:01 smithi104 ceph-mon[36624]: pgmap v15445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:03 smithi153 ceph-mon[36569]: pgmap v15446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:03.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:03 smithi104 ceph-mon[36624]: pgmap v15446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:03 smithi081 ceph-mon[31453]: pgmap v15446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:05.509 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:05 smithi153 ceph-mon[36569]: pgmap v15447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:05 smithi081 ceph-mon[31453]: pgmap v15447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:05 smithi104 ceph-mon[36624]: pgmap v15447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:16:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:16:06.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:06 smithi153 ceph-mon[36569]: pgmap v15448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:06.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:06 smithi153 ceph-mon[36569]: pgmap v15449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:06.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:16:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:16:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:06 smithi104 ceph-mon[36624]: pgmap v15448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:06 smithi104 ceph-mon[36624]: pgmap v15449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:16:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:16:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:06 smithi081 ceph-mon[31453]: pgmap v15448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:06 smithi081 ceph-mon[31453]: pgmap v15449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:16:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:08 smithi081 ceph-mon[31453]: pgmap v15450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:09.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:08 smithi153 ceph-mon[36569]: pgmap v15450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:08 smithi104 ceph-mon[36624]: pgmap v15450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:10 smithi081 ceph-mon[31453]: pgmap v15451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:10 smithi153 ceph-mon[36569]: pgmap v15451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:10 smithi104 ceph-mon[36624]: pgmap v15451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:12.169+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b3d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000550) for more than 900000000 us, fault. 2024-04-13T05:16:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:12.169+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3d80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f16d40038e0 tx=0x7f16f40bf000 comp rx=0 tx=0).stop 2024-04-13T05:16:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:12.169+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80b3d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T05:16:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:12.169+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80b3d80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T05:16:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:12.169+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T05:16:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:12.170+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T05:16:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:12.171+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c9f0 tx=0x7f16f4032000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T05:16:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:12 smithi153 ceph-mon[36569]: pgmap v15452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:12 smithi104 ceph-mon[36624]: pgmap v15452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:13.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:12 smithi081 ceph-mon[31453]: pgmap v15452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:14.633+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:16:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:14.633+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:16:14.635 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:14.633+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:16:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:14 smithi153 ceph-mon[36569]: pgmap v15453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:14 smithi104 ceph-mon[36624]: pgmap v15453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:14 smithi081 ceph-mon[31453]: pgmap v15453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:16 smithi153 ceph-mon[36569]: pgmap v15454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:16 smithi104 ceph-mon[36624]: pgmap v15454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:16 smithi081 ceph-mon[31453]: pgmap v15454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:17.464+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:16:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:17.464+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:16:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:17.464+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 223 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:16:17.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:17.464+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 223 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:16:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:18 smithi153 ceph-mon[36569]: pgmap v15455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:19 smithi104 ceph-mon[36624]: pgmap v15455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:18 smithi081 ceph-mon[31453]: pgmap v15455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:21 smithi153 ceph-mon[36569]: pgmap v15456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:21 smithi104 ceph-mon[36624]: pgmap v15456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:21 smithi081 ceph-mon[31453]: pgmap v15456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:23 smithi153 ceph-mon[36569]: pgmap v15457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:23 smithi104 ceph-mon[36624]: pgmap v15457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:23 smithi081 ceph-mon[31453]: pgmap v15457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:25 smithi153 ceph-mon[36569]: pgmap v15458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:25 smithi104 ceph-mon[36624]: pgmap v15458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:25 smithi081 ceph-mon[31453]: pgmap v15458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:27 smithi153 ceph-mon[36569]: pgmap v15459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:27 smithi104 ceph-mon[36624]: pgmap v15459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:27 smithi081 ceph-mon[31453]: pgmap v15459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:29 smithi153 ceph-mon[36569]: pgmap v15460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:29 smithi104 ceph-mon[36624]: pgmap v15460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:29 smithi081 ceph-mon[31453]: pgmap v15460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:29.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:29.642+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:16:29.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:29.643+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:16:29.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:29.643+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:16:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:31 smithi153 ceph-mon[36569]: pgmap v15461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:31 smithi104 ceph-mon[36624]: pgmap v15461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:31 smithi081 ceph-mon[31453]: pgmap v15461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:33 smithi153 ceph-mon[36569]: pgmap v15462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:33 smithi104 ceph-mon[36624]: pgmap v15462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:33 smithi081 ceph-mon[31453]: pgmap v15462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:35 smithi153 ceph-mon[36569]: pgmap v15463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:35 smithi104 ceph-mon[36624]: pgmap v15463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:35 smithi081 ceph-mon[31453]: pgmap v15463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:37.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:37 smithi153 ceph-mon[36569]: pgmap v15464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:37 smithi104 ceph-mon[36624]: pgmap v15464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:37 smithi081 ceph-mon[31453]: pgmap v15464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:39.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:39 smithi153 ceph-mon[36569]: pgmap v15465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:39 smithi104 ceph-mon[36624]: pgmap v15465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:39 smithi081 ceph-mon[31453]: pgmap v15465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:41 smithi153 ceph-mon[36569]: pgmap v15466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:41 smithi104 ceph-mon[36624]: pgmap v15466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:41 smithi081 ceph-mon[31453]: pgmap v15466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:43 smithi104 ceph-mon[36624]: pgmap v15467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:43 smithi081 ceph-mon[31453]: pgmap v15467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:43 smithi153 ceph-mon[36569]: pgmap v15467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:44.655 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:44.654+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:16:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:44.654+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:16:44.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:44.654+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:16:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:45 smithi104 ceph-mon[36624]: pgmap v15468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:45 smithi081 ceph-mon[31453]: pgmap v15468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:45.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:45 smithi153 ceph-mon[36569]: pgmap v15468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:47 smithi104 ceph-mon[36624]: pgmap v15469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:47 smithi081 ceph-mon[31453]: pgmap v15469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:47 smithi153 ceph-mon[36569]: pgmap v15469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:49 smithi104 ceph-mon[36624]: pgmap v15470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:49 smithi081 ceph-mon[31453]: pgmap v15470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:49 smithi153 ceph-mon[36569]: pgmap v15470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:51 smithi104 ceph-mon[36624]: pgmap v15471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:51 smithi081 ceph-mon[31453]: pgmap v15471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:51 smithi153 ceph-mon[36569]: pgmap v15471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:53 smithi081 ceph-mon[31453]: pgmap v15472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:53 smithi153 ceph-mon[36569]: pgmap v15472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:53 smithi104 ceph-mon[36624]: pgmap v15472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:55 smithi081 ceph-mon[31453]: pgmap v15473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:55 smithi153 ceph-mon[36569]: pgmap v15473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:55 smithi104 ceph-mon[36624]: pgmap v15473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:57 smithi081 ceph-mon[31453]: pgmap v15474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:57 smithi153 ceph-mon[36569]: pgmap v15474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:57 smithi104 ceph-mon[36624]: pgmap v15474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:59.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:16:59 smithi153 ceph-mon[36569]: pgmap v15475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:16:59 smithi081 ceph-mon[31453]: pgmap v15475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:16:59 smithi104 ceph-mon[36624]: pgmap v15475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:16:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:59.668+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:16:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:59.668+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:16:59.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:16:59.668+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:01 smithi081 ceph-mon[31453]: pgmap v15476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:01 smithi153 ceph-mon[36569]: pgmap v15476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:01 smithi104 ceph-mon[36624]: pgmap v15476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:03.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:03 smithi104 ceph-mon[36624]: pgmap v15477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:03 smithi081 ceph-mon[31453]: pgmap v15477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:03 smithi153 ceph-mon[36569]: pgmap v15477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:05 smithi081 ceph-mon[31453]: pgmap v15478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:05 smithi153 ceph-mon[36569]: pgmap v15478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:05 smithi104 ceph-mon[36624]: pgmap v15478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:17:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:17:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:17:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:07 smithi081 ceph-mon[31453]: pgmap v15479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:17:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:17:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:17:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:07 smithi153 ceph-mon[36569]: pgmap v15479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:17:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:17:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:17:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:07 smithi104 ceph-mon[36624]: pgmap v15479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:17:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:17:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:17:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:08 smithi081 ceph-mon[31453]: pgmap v15480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:08 smithi081 ceph-mon[31453]: pgmap v15481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:08 smithi153 ceph-mon[36569]: pgmap v15480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:08 smithi153 ceph-mon[36569]: pgmap v15481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:08 smithi104 ceph-mon[36624]: pgmap v15480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:08 smithi104 ceph-mon[36624]: pgmap v15481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:10 smithi081 ceph-mon[31453]: pgmap v15482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:10 smithi153 ceph-mon[36569]: pgmap v15482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:10 smithi104 ceph-mon[36624]: pgmap v15482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:12 smithi081 ceph-mon[31453]: pgmap v15483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:12 smithi153 ceph-mon[36569]: pgmap v15483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:12 smithi104 ceph-mon[36624]: pgmap v15483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:14 smithi081 ceph-mon[31453]: pgmap v15484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:14 smithi153 ceph-mon[36569]: pgmap v15484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:14 smithi104 ceph-mon[36624]: pgmap v15484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:14.685 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:14.684+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:14.685 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:14.684+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:14.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:14.684+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:17:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:16 smithi081 ceph-mon[31453]: pgmap v15485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:16 smithi153 ceph-mon[36569]: pgmap v15485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:16 smithi104 ceph-mon[36624]: pgmap v15485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:18 smithi153 ceph-mon[36569]: pgmap v15486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:18 smithi104 ceph-mon[36624]: pgmap v15486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:18 smithi081 ceph-mon[31453]: pgmap v15486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:20 smithi153 ceph-mon[36569]: pgmap v15487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:20.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:20 smithi104 ceph-mon[36624]: pgmap v15487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:20 smithi081 ceph-mon[31453]: pgmap v15487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:22 smithi153 ceph-mon[36569]: pgmap v15488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:22 smithi104 ceph-mon[36624]: pgmap v15488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:22.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:22 smithi081 ceph-mon[31453]: pgmap v15488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:24 smithi153 ceph-mon[36569]: pgmap v15489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:24 smithi104 ceph-mon[36624]: pgmap v15489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:24.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:24 smithi081 ceph-mon[31453]: pgmap v15489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:26 smithi153 ceph-mon[36569]: pgmap v15490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:26 smithi104 ceph-mon[36624]: pgmap v15490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:26 smithi081 ceph-mon[31453]: pgmap v15490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:28 smithi081 ceph-mon[31453]: pgmap v15491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:28 smithi153 ceph-mon[36569]: pgmap v15491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:28 smithi104 ceph-mon[36624]: pgmap v15491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:29.699 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:29.698+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:29.698+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:17:29.700 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:29.698+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:30 smithi153 ceph-mon[36569]: pgmap v15492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:30 smithi104 ceph-mon[36624]: pgmap v15492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:30 smithi081 ceph-mon[31453]: pgmap v15492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:32 smithi153 ceph-mon[36569]: pgmap v15493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:32 smithi104 ceph-mon[36624]: pgmap v15493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:32 smithi081 ceph-mon[31453]: pgmap v15493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:34.460 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:34 smithi081 ceph-mon[31453]: pgmap v15494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:34 smithi153 ceph-mon[36569]: pgmap v15494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:34 smithi104 ceph-mon[36624]: pgmap v15494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:36 smithi153 ceph-mon[36569]: pgmap v15495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:36 smithi104 ceph-mon[36624]: pgmap v15495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:36 smithi081 ceph-mon[31453]: pgmap v15495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:38 smithi153 ceph-mon[36569]: pgmap v15496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:38 smithi104 ceph-mon[36624]: pgmap v15496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:38.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:38 smithi081 ceph-mon[31453]: pgmap v15496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:40 smithi153 ceph-mon[36569]: pgmap v15497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:40 smithi104 ceph-mon[36624]: pgmap v15497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:40.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:40 smithi081 ceph-mon[31453]: pgmap v15497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:42 smithi153 ceph-mon[36569]: pgmap v15498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:42 smithi104 ceph-mon[36624]: pgmap v15498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:42 smithi081 ceph-mon[31453]: pgmap v15498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:44 smithi153 ceph-mon[36569]: pgmap v15499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:44 smithi104 ceph-mon[36624]: pgmap v15499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:44 smithi081 ceph-mon[31453]: pgmap v15499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:44.712+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:44.712+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:17:44.714 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:44.712+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:46 smithi153 ceph-mon[36569]: pgmap v15500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:46 smithi104 ceph-mon[36624]: pgmap v15500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:46 smithi081 ceph-mon[31453]: pgmap v15500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:48 smithi153 ceph-mon[36569]: pgmap v15501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:48 smithi104 ceph-mon[36624]: pgmap v15501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:48 smithi081 ceph-mon[31453]: pgmap v15501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:50 smithi153 ceph-mon[36569]: pgmap v15502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:50 smithi104 ceph-mon[36624]: pgmap v15502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:50 smithi081 ceph-mon[31453]: pgmap v15502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:52 smithi153 ceph-mon[36569]: pgmap v15503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:52 smithi104 ceph-mon[36624]: pgmap v15503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:52 smithi081 ceph-mon[31453]: pgmap v15503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:54.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:54 smithi153 ceph-mon[36569]: pgmap v15504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:54 smithi104 ceph-mon[36624]: pgmap v15504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:54 smithi081 ceph-mon[31453]: pgmap v15504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:56 smithi153 ceph-mon[36569]: pgmap v15505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:56 smithi104 ceph-mon[36624]: pgmap v15505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:56 smithi081 ceph-mon[31453]: pgmap v15505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:17:58 smithi153 ceph-mon[36569]: pgmap v15506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:17:58 smithi104 ceph-mon[36624]: pgmap v15506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:17:58 smithi081 ceph-mon[31453]: pgmap v15506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:17:59.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:59.728+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:59.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:59.728+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:17:59.729 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:17:59.728+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:18:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:00 smithi153 ceph-mon[36569]: pgmap v15507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:00 smithi104 ceph-mon[36624]: pgmap v15507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:00 smithi081 ceph-mon[31453]: pgmap v15507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:02 smithi153 ceph-mon[36569]: pgmap v15508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:02 smithi104 ceph-mon[36624]: pgmap v15508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:02 smithi081 ceph-mon[31453]: pgmap v15508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:04 smithi153 ceph-mon[36569]: pgmap v15509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:04 smithi104 ceph-mon[36624]: pgmap v15509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:04 smithi081 ceph-mon[31453]: pgmap v15509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:06 smithi153 ceph-mon[36569]: pgmap v15510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:06.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:06 smithi104 ceph-mon[36624]: pgmap v15510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:06 smithi081 ceph-mon[31453]: pgmap v15510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:08 smithi153 ceph-mon[36569]: pgmap v15511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:18:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:08 smithi104 ceph-mon[36624]: pgmap v15511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:08.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:18:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:08 smithi081 ceph-mon[31453]: pgmap v15511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:18:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:18:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:18:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:18:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:18:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:18:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:18:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:18:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:18:09.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:18:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:10 smithi153 ceph-mon[36569]: pgmap v15512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:10 smithi153 ceph-mon[36569]: pgmap v15513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:10 smithi104 ceph-mon[36624]: pgmap v15512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:10 smithi104 ceph-mon[36624]: pgmap v15513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:10 smithi081 ceph-mon[31453]: pgmap v15512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:10.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:10 smithi081 ceph-mon[31453]: pgmap v15513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:12 smithi104 ceph-mon[36624]: pgmap v15514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:12 smithi081 ceph-mon[31453]: pgmap v15514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:12 smithi153 ceph-mon[36569]: pgmap v15514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:14 smithi081 ceph-mon[31453]: pgmap v15515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:14 smithi104 ceph-mon[36624]: pgmap v15515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:14.740 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:14.738+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:18:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:14.738+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:18:14.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:14.738+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:18:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:14 smithi153 ceph-mon[36569]: pgmap v15515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:16 smithi081 ceph-mon[31453]: pgmap v15516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:16 smithi153 ceph-mon[36569]: pgmap v15516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:16.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:16 smithi104 ceph-mon[36624]: pgmap v15516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:18 smithi081 ceph-mon[31453]: pgmap v15517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:18 smithi153 ceph-mon[36569]: pgmap v15517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:18 smithi104 ceph-mon[36624]: pgmap v15517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:20 smithi081 ceph-mon[31453]: pgmap v15518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:20.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:20 smithi153 ceph-mon[36569]: pgmap v15518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:20.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:20 smithi104 ceph-mon[36624]: pgmap v15518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:22.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:22 smithi081 ceph-mon[31453]: pgmap v15519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:22.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:22 smithi153 ceph-mon[36569]: pgmap v15519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:22 smithi104 ceph-mon[36624]: pgmap v15519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:24.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:24 smithi081 ceph-mon[31453]: pgmap v15520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:24.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:24 smithi153 ceph-mon[36569]: pgmap v15520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:24.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:24 smithi104 ceph-mon[36624]: pgmap v15520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:26 smithi081 ceph-mon[31453]: pgmap v15521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:26.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:26 smithi153 ceph-mon[36569]: pgmap v15521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:26.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:26 smithi104 ceph-mon[36624]: pgmap v15521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:28 smithi081 ceph-mon[31453]: pgmap v15522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:28 smithi153 ceph-mon[36569]: pgmap v15522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:28.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:28 smithi104 ceph-mon[36624]: pgmap v15522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:29.751+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:18:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:29.751+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:18:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:29.751+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:18:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:30 smithi081 ceph-mon[31453]: pgmap v15523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:30.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:30 smithi153 ceph-mon[36569]: pgmap v15523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:30 smithi104 ceph-mon[36624]: pgmap v15523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:32 smithi081 ceph-mon[31453]: pgmap v15524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:32.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:32 smithi153 ceph-mon[36569]: pgmap v15524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:32.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:32 smithi104 ceph-mon[36624]: pgmap v15524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:34.636 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:34 smithi104 ceph-mon[36624]: pgmap v15525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:34.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:34 smithi081 ceph-mon[31453]: pgmap v15525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:34.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:34 smithi153 ceph-mon[36569]: pgmap v15525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:36 smithi081 ceph-mon[31453]: pgmap v15526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:36.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:36 smithi153 ceph-mon[36569]: pgmap v15526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:36 smithi104 ceph-mon[36624]: pgmap v15526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:38.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:38 smithi081 ceph-mon[31453]: pgmap v15527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:38 smithi153 ceph-mon[36569]: pgmap v15527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:38 smithi104 ceph-mon[36624]: pgmap v15527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:40.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:40 smithi081 ceph-mon[31453]: pgmap v15528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:40.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:40 smithi153 ceph-mon[36569]: pgmap v15528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:40.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:40 smithi104 ceph-mon[36624]: pgmap v15528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:42 smithi081 ceph-mon[31453]: pgmap v15529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:42 smithi153 ceph-mon[36569]: pgmap v15529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:42.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:42 smithi104 ceph-mon[36624]: pgmap v15529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:44 smithi081 ceph-mon[31453]: pgmap v15530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:44.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:44.765+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:18:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:44.765+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:18:44.767 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:44.765+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:18:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:44 smithi153 ceph-mon[36569]: pgmap v15530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:44 smithi104 ceph-mon[36624]: pgmap v15530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:46 smithi153 ceph-mon[36569]: pgmap v15531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:46.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:46 smithi104 ceph-mon[36624]: pgmap v15531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:46 smithi081 ceph-mon[31453]: pgmap v15531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:48 smithi153 ceph-mon[36569]: pgmap v15532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:48 smithi104 ceph-mon[36624]: pgmap v15532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:48.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:48 smithi081 ceph-mon[31453]: pgmap v15532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:50 smithi153 ceph-mon[36569]: pgmap v15533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:50.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:50 smithi104 ceph-mon[36624]: pgmap v15533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:50.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:50 smithi081 ceph-mon[31453]: pgmap v15533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:52 smithi153 ceph-mon[36569]: pgmap v15534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:52 smithi104 ceph-mon[36624]: pgmap v15534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:52.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:52 smithi081 ceph-mon[31453]: pgmap v15534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:54 smithi153 ceph-mon[36569]: pgmap v15535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:54 smithi104 ceph-mon[36624]: pgmap v15535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:54 smithi081 ceph-mon[31453]: pgmap v15535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:56 smithi153 ceph-mon[36569]: pgmap v15536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:56.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:56 smithi104 ceph-mon[36624]: pgmap v15536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:56.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:56 smithi081 ceph-mon[31453]: pgmap v15536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:58.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:18:58 smithi153 ceph-mon[36569]: pgmap v15537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:58.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:18:58 smithi104 ceph-mon[36624]: pgmap v15537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:18:58 smithi081 ceph-mon[31453]: pgmap v15537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:18:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:59.777+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:18:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:59.777+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:18:59.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:18:59.777+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:19:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:00 smithi081 ceph-mon[31453]: pgmap v15538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:00.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:00 smithi153 ceph-mon[36569]: pgmap v15538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:00.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:00 smithi104 ceph-mon[36624]: pgmap v15538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:02.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:02 smithi153 ceph-mon[36569]: pgmap v15539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:02 smithi104 ceph-mon[36624]: pgmap v15539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:02 smithi081 ceph-mon[31453]: pgmap v15539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:04.711 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:04 smithi081 ceph-mon[31453]: pgmap v15540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:04.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:04 smithi153 ceph-mon[36569]: pgmap v15540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:04 smithi104 ceph-mon[36624]: pgmap v15540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:06 smithi153 ceph-mon[36569]: pgmap v15541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:06.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:06 smithi104 ceph-mon[36624]: pgmap v15541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:06.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:06 smithi081 ceph-mon[31453]: pgmap v15541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:08 smithi153 ceph-mon[36569]: pgmap v15542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:08 smithi104 ceph-mon[36624]: pgmap v15542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:08.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:08 smithi081 ceph-mon[31453]: pgmap v15542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:19:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:19:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:19:10.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:10 smithi153 ceph-mon[36569]: pgmap v15543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:10.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:19:10.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:19:10.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:19:10.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:10 smithi104 ceph-mon[36624]: pgmap v15543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:19:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:19:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:19:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:10 smithi081 ceph-mon[31453]: pgmap v15543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:19:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:19:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:19:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:11 smithi153 ceph-mon[36569]: pgmap v15544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:11 smithi153 ceph-mon[36569]: pgmap v15545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:11 smithi104 ceph-mon[36624]: pgmap v15544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:11 smithi104 ceph-mon[36624]: pgmap v15545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:11 smithi081 ceph-mon[31453]: pgmap v15544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:11 smithi081 ceph-mon[31453]: pgmap v15545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:13 smithi153 ceph-mon[36569]: pgmap v15546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:13 smithi104 ceph-mon[36624]: pgmap v15546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:13 smithi081 ceph-mon[31453]: pgmap v15546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:14.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:14.787+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:19:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:14.787+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:19:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:14.787+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:19:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:15 smithi153 ceph-mon[36569]: pgmap v15547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:15.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:15 smithi104 ceph-mon[36624]: pgmap v15547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:15 smithi081 ceph-mon[31453]: pgmap v15547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:17.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:17.494+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:19:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:17.494+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:19:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:17.494+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 224 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:19:17.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:17.494+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 224 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:19:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:17 smithi153 ceph-mon[36569]: pgmap v15548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:17 smithi104 ceph-mon[36624]: pgmap v15548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:17 smithi081 ceph-mon[31453]: pgmap v15548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:19 smithi153 ceph-mon[36569]: pgmap v15549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:19 smithi104 ceph-mon[36624]: pgmap v15549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:19 smithi081 ceph-mon[31453]: pgmap v15549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:21 smithi153 ceph-mon[36569]: pgmap v15550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:21 smithi104 ceph-mon[36624]: pgmap v15550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:21 smithi081 ceph-mon[31453]: pgmap v15550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:23 smithi153 ceph-mon[36569]: pgmap v15551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:23 smithi104 ceph-mon[36624]: pgmap v15551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:23 smithi081 ceph-mon[31453]: pgmap v15551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:25 smithi153 ceph-mon[36569]: pgmap v15552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:25 smithi104 ceph-mon[36624]: pgmap v15552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:25 smithi081 ceph-mon[31453]: pgmap v15552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:27 smithi153 ceph-mon[36569]: pgmap v15553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:27 smithi104 ceph-mon[36624]: pgmap v15553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:27.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:27 smithi081 ceph-mon[31453]: pgmap v15553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:29 smithi104 ceph-mon[36624]: pgmap v15554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:29 smithi081 ceph-mon[31453]: pgmap v15554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:29 smithi153 ceph-mon[36569]: pgmap v15554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:29.803+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:19:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:29.803+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:19:29.805 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:29.803+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:19:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:31 smithi081 ceph-mon[31453]: pgmap v15555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:31 smithi153 ceph-mon[36569]: pgmap v15555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:31 smithi104 ceph-mon[36624]: pgmap v15555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:33 smithi081 ceph-mon[31453]: pgmap v15556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:33 smithi153 ceph-mon[36569]: pgmap v15556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:33 smithi104 ceph-mon[36624]: pgmap v15556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:35.636 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:35 smithi104 ceph-mon[36624]: pgmap v15557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:35 smithi081 ceph-mon[31453]: pgmap v15557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:35 smithi153 ceph-mon[36569]: pgmap v15557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:37.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:37 smithi081 ceph-mon[31453]: pgmap v15558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:37 smithi153 ceph-mon[36569]: pgmap v15558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:37 smithi104 ceph-mon[36624]: pgmap v15558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:39 smithi081 ceph-mon[31453]: pgmap v15559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:39 smithi153 ceph-mon[36569]: pgmap v15559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:39 smithi104 ceph-mon[36624]: pgmap v15559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:41.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:41 smithi081 ceph-mon[31453]: pgmap v15560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:41.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:41 smithi153 ceph-mon[36569]: pgmap v15560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:41 smithi104 ceph-mon[36624]: pgmap v15560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:43 smithi081 ceph-mon[31453]: pgmap v15561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:43 smithi153 ceph-mon[36569]: pgmap v15561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:43 smithi104 ceph-mon[36624]: pgmap v15561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:44.817+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:19:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:44.817+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:19:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:44.817+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:19:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:45 smithi081 ceph-mon[31453]: pgmap v15562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:45 smithi153 ceph-mon[36569]: pgmap v15562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:45 smithi104 ceph-mon[36624]: pgmap v15562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:47 smithi081 ceph-mon[31453]: pgmap v15563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:47 smithi153 ceph-mon[36569]: pgmap v15563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:47 smithi104 ceph-mon[36624]: pgmap v15563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:49 smithi081 ceph-mon[31453]: pgmap v15564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:49 smithi153 ceph-mon[36569]: pgmap v15564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:49 smithi104 ceph-mon[36624]: pgmap v15564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:51 smithi081 ceph-mon[31453]: pgmap v15565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:51 smithi153 ceph-mon[36569]: pgmap v15565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:51 smithi104 ceph-mon[36624]: pgmap v15565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:53 smithi081 ceph-mon[31453]: pgmap v15566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:53 smithi153 ceph-mon[36569]: pgmap v15566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:53 smithi104 ceph-mon[36624]: pgmap v15566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:55 smithi081 ceph-mon[31453]: pgmap v15567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:55 smithi153 ceph-mon[36569]: pgmap v15567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:55 smithi104 ceph-mon[36624]: pgmap v15567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:57 smithi153 ceph-mon[36569]: pgmap v15568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:57 smithi104 ceph-mon[36624]: pgmap v15568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:57 smithi081 ceph-mon[31453]: pgmap v15568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:19:59 smithi153 ceph-mon[36569]: pgmap v15569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:59.831+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:19:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:59.831+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:19:59.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:19:59.831+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:19:59.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:19:59 smithi104 ceph-mon[36624]: pgmap v15569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:19:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:19:59 smithi081 ceph-mon[31453]: pgmap v15569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:00.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:20:00.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:20:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:20:01.709 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:01 smithi153 ceph-mon[36569]: pgmap v15570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:01 smithi104 ceph-mon[36624]: pgmap v15570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:01 smithi081 ceph-mon[31453]: pgmap v15570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:03.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:03 smithi081 ceph-mon[31453]: pgmap v15571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:03 smithi153 ceph-mon[36569]: pgmap v15571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:03.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:03 smithi104 ceph-mon[36624]: pgmap v15571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:05 smithi153 ceph-mon[36569]: pgmap v15572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:05 smithi104 ceph-mon[36624]: pgmap v15572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:05 smithi081 ceph-mon[31453]: pgmap v15572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:07 smithi153 ceph-mon[36569]: pgmap v15573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:07 smithi104 ceph-mon[36624]: pgmap v15573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:07 smithi081 ceph-mon[31453]: pgmap v15573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:09 smithi153 ceph-mon[36569]: pgmap v15574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:09 smithi104 ceph-mon[36624]: pgmap v15574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:09 smithi081 ceph-mon[31453]: pgmap v15574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:11 smithi153 ceph-mon[36569]: pgmap v15575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:20:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:11 smithi104 ceph-mon[36624]: pgmap v15575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:20:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:11 smithi081 ceph-mon[31453]: pgmap v15575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:20:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:20:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:20:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:20:12.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:20:12.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:20:12.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:20:12.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:20:12.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:20:12.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:20:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:13 smithi153 ceph-mon[36569]: pgmap v15576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:13 smithi153 ceph-mon[36569]: pgmap v15577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:13 smithi104 ceph-mon[36624]: pgmap v15576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:13 smithi104 ceph-mon[36624]: pgmap v15577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:13.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:13 smithi081 ceph-mon[31453]: pgmap v15576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:13 smithi081 ceph-mon[31453]: pgmap v15577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:14.847+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:20:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:14.847+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:20:14.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:14.847+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:20:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:15 smithi153 ceph-mon[36569]: pgmap v15578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:15 smithi104 ceph-mon[36624]: pgmap v15578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:15 smithi081 ceph-mon[31453]: pgmap v15578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:17 smithi153 ceph-mon[36569]: pgmap v15579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:17.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:17 smithi104 ceph-mon[36624]: pgmap v15579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:17.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:17 smithi081 ceph-mon[31453]: pgmap v15579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:19 smithi153 ceph-mon[36569]: pgmap v15580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:19 smithi104 ceph-mon[36624]: pgmap v15580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:19 smithi081 ceph-mon[31453]: pgmap v15580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:21 smithi153 ceph-mon[36569]: pgmap v15581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:21 smithi104 ceph-mon[36624]: pgmap v15581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:21 smithi081 ceph-mon[31453]: pgmap v15581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:23 smithi153 ceph-mon[36569]: pgmap v15582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:23 smithi104 ceph-mon[36624]: pgmap v15582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:23 smithi081 ceph-mon[31453]: pgmap v15582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:25 smithi153 ceph-mon[36569]: pgmap v15583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:25 smithi104 ceph-mon[36624]: pgmap v15583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:25 smithi081 ceph-mon[31453]: pgmap v15583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:27 smithi153 ceph-mon[36569]: pgmap v15584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:27 smithi104 ceph-mon[36624]: pgmap v15584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:27 smithi081 ceph-mon[31453]: pgmap v15584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:29 smithi153 ceph-mon[36569]: pgmap v15585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:29 smithi104 ceph-mon[36624]: pgmap v15585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:29.862+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:20:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:29.862+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:20:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:29.862+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:20:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:29 smithi081 ceph-mon[31453]: pgmap v15585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:31 smithi153 ceph-mon[36569]: pgmap v15586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:31 smithi104 ceph-mon[36624]: pgmap v15586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:31 smithi081 ceph-mon[31453]: pgmap v15586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:33 smithi153 ceph-mon[36569]: pgmap v15587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:33 smithi104 ceph-mon[36624]: pgmap v15587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:33 smithi081 ceph-mon[31453]: pgmap v15587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:35 smithi153 ceph-mon[36569]: pgmap v15588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:35 smithi104 ceph-mon[36624]: pgmap v15588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:35.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:35 smithi081 ceph-mon[31453]: pgmap v15588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:37 smithi153 ceph-mon[36569]: pgmap v15589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:37 smithi104 ceph-mon[36624]: pgmap v15589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:37 smithi081 ceph-mon[31453]: pgmap v15589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:39 smithi153 ceph-mon[36569]: pgmap v15590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:39 smithi104 ceph-mon[36624]: pgmap v15590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:39 smithi081 ceph-mon[31453]: pgmap v15590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:41.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:41 smithi153 ceph-mon[36569]: pgmap v15591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:41.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:41 smithi104 ceph-mon[36624]: pgmap v15591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:41 smithi081 ceph-mon[31453]: pgmap v15591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:43.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:43 smithi153 ceph-mon[36569]: pgmap v15592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:43 smithi104 ceph-mon[36624]: pgmap v15592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:43 smithi081 ceph-mon[31453]: pgmap v15592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:44.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:44.876+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:20:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:44.876+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:20:44.879 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:44.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:20:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:45 smithi153 ceph-mon[36569]: pgmap v15593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:45 smithi104 ceph-mon[36624]: pgmap v15593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:45 smithi081 ceph-mon[31453]: pgmap v15593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:47 smithi104 ceph-mon[36624]: pgmap v15594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:47 smithi081 ceph-mon[31453]: pgmap v15594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:47 smithi153 ceph-mon[36569]: pgmap v15594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:49 smithi104 ceph-mon[36624]: pgmap v15595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:49 smithi081 ceph-mon[31453]: pgmap v15595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:49 smithi153 ceph-mon[36569]: pgmap v15595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:51 smithi104 ceph-mon[36624]: pgmap v15596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:51 smithi081 ceph-mon[31453]: pgmap v15596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:51 smithi153 ceph-mon[36569]: pgmap v15596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:53.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:53 smithi104 ceph-mon[36624]: pgmap v15597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:53 smithi081 ceph-mon[31453]: pgmap v15597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:53 smithi153 ceph-mon[36569]: pgmap v15597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:54 smithi081 ceph-mon[31453]: pgmap v15598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:54 smithi153 ceph-mon[36569]: pgmap v15598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:54 smithi104 ceph-mon[36624]: pgmap v15598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:56 smithi081 ceph-mon[31453]: pgmap v15599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:56 smithi153 ceph-mon[36569]: pgmap v15599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:56 smithi104 ceph-mon[36624]: pgmap v15599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:59.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:20:58 smithi081 ceph-mon[31453]: pgmap v15600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:20:58 smithi153 ceph-mon[36569]: pgmap v15600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:20:58 smithi104 ceph-mon[36624]: pgmap v15600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:20:59.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:59.892+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:20:59.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:59.892+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:20:59.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:20:59.892+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:21:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:00 smithi081 ceph-mon[31453]: pgmap v15601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:00 smithi153 ceph-mon[36569]: pgmap v15601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:00 smithi104 ceph-mon[36624]: pgmap v15601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:02 smithi081 ceph-mon[31453]: pgmap v15602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:02 smithi153 ceph-mon[36569]: pgmap v15602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:02 smithi104 ceph-mon[36624]: pgmap v15602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:04 smithi081 ceph-mon[31453]: pgmap v15603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:04 smithi153 ceph-mon[36569]: pgmap v15603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:04 smithi104 ceph-mon[36624]: pgmap v15603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:06 smithi081 ceph-mon[31453]: pgmap v15604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:06 smithi153 ceph-mon[36569]: pgmap v15604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:06 smithi104 ceph-mon[36624]: pgmap v15604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:08 smithi081 ceph-mon[31453]: pgmap v15605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:08 smithi153 ceph-mon[36569]: pgmap v15605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:08 smithi104 ceph-mon[36624]: pgmap v15605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:10 smithi081 ceph-mon[31453]: pgmap v15606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:10 smithi153 ceph-mon[36569]: pgmap v15606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:10 smithi104 ceph-mon[36624]: pgmap v15606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: pgmap v15607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:21:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: pgmap v15607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: pgmap v15607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:21:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:21:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: pgmap v15608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: pgmap v15609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:21:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:21:14.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:21:14.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: pgmap v15608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:14.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: pgmap v15609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:14.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:21:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: pgmap v15608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: pgmap v15609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:21:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:14.898+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:21:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:14.898+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:21:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:14.898+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:21:16.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:16 smithi081 ceph-mon[31453]: pgmap v15610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:16 smithi153 ceph-mon[36569]: pgmap v15610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:16 smithi104 ceph-mon[36624]: pgmap v15610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:18 smithi081 ceph-mon[31453]: pgmap v15611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:18 smithi153 ceph-mon[36569]: pgmap v15611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:18.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:18 smithi104 ceph-mon[36624]: pgmap v15611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:20.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:20 smithi081 ceph-mon[31453]: pgmap v15612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:20.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:20 smithi153 ceph-mon[36569]: pgmap v15612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:20 smithi104 ceph-mon[36624]: pgmap v15612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:22.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:22 smithi081 ceph-mon[31453]: pgmap v15613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:22.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:22 smithi153 ceph-mon[36569]: pgmap v15613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:22.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:22 smithi104 ceph-mon[36624]: pgmap v15613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:24.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:24 smithi081 ceph-mon[31453]: pgmap v15614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:24.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:24 smithi153 ceph-mon[36569]: pgmap v15614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:24.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:24 smithi104 ceph-mon[36624]: pgmap v15614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:26.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:26 smithi081 ceph-mon[31453]: pgmap v15615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:26.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:26 smithi153 ceph-mon[36569]: pgmap v15615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:26.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:26 smithi104 ceph-mon[36624]: pgmap v15615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:28.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:28 smithi081 ceph-mon[31453]: pgmap v15616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:28 smithi153 ceph-mon[36569]: pgmap v15616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:28 smithi104 ceph-mon[36624]: pgmap v15616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:29.914+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:21:29.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:29.914+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:21:29.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:29.914+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:21:30.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:30 smithi081 ceph-mon[31453]: pgmap v15617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:30.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:30 smithi153 ceph-mon[36569]: pgmap v15617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:30.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:30 smithi104 ceph-mon[36624]: pgmap v15617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:32.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:32 smithi153 ceph-mon[36569]: pgmap v15618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:32.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:32 smithi104 ceph-mon[36624]: pgmap v15618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:32.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:32 smithi081 ceph-mon[31453]: pgmap v15618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:34.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:34 smithi153 ceph-mon[36569]: pgmap v15619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:34 smithi104 ceph-mon[36624]: pgmap v15619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:34 smithi081 ceph-mon[31453]: pgmap v15619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:36.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:36 smithi153 ceph-mon[36569]: pgmap v15620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:36 smithi104 ceph-mon[36624]: pgmap v15620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:36.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:36 smithi081 ceph-mon[31453]: pgmap v15620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:38.710 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:38 smithi081 ceph-mon[31453]: pgmap v15621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:38 smithi153 ceph-mon[36569]: pgmap v15621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:38 smithi104 ceph-mon[36624]: pgmap v15621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:40.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:40 smithi153 ceph-mon[36569]: pgmap v15622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:40.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:40 smithi104 ceph-mon[36624]: pgmap v15622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:40.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:40 smithi081 ceph-mon[31453]: pgmap v15622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:42 smithi153 ceph-mon[36569]: pgmap v15623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:42.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:42 smithi104 ceph-mon[36624]: pgmap v15623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:42 smithi081 ceph-mon[31453]: pgmap v15623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:44 smithi153 ceph-mon[36569]: pgmap v15624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:44.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:44 smithi104 ceph-mon[36624]: pgmap v15624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:44.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:44 smithi081 ceph-mon[31453]: pgmap v15624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:44.917+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:21:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:44.917+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:21:44.920 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:44.917+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:21:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:46 smithi153 ceph-mon[36569]: pgmap v15625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:46 smithi104 ceph-mon[36624]: pgmap v15625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:46.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:46 smithi081 ceph-mon[31453]: pgmap v15625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:48 smithi153 ceph-mon[36569]: pgmap v15626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:48 smithi104 ceph-mon[36624]: pgmap v15626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:48 smithi081 ceph-mon[31453]: pgmap v15626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:50 smithi153 ceph-mon[36569]: pgmap v15627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:50.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:50 smithi104 ceph-mon[36624]: pgmap v15627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:50 smithi081 ceph-mon[31453]: pgmap v15627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:52 smithi153 ceph-mon[36569]: pgmap v15628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:52 smithi104 ceph-mon[36624]: pgmap v15628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:52 smithi081 ceph-mon[31453]: pgmap v15628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:54 smithi153 ceph-mon[36569]: pgmap v15629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:54.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:54 smithi104 ceph-mon[36624]: pgmap v15629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:54 smithi081 ceph-mon[31453]: pgmap v15629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:56 smithi153 ceph-mon[36569]: pgmap v15630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:56 smithi104 ceph-mon[36624]: pgmap v15630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:56.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:56 smithi081 ceph-mon[31453]: pgmap v15630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:58.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:21:58 smithi153 ceph-mon[36569]: pgmap v15631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:21:58 smithi104 ceph-mon[36624]: pgmap v15631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:21:58 smithi081 ceph-mon[31453]: pgmap v15631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:21:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:59.928+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:21:59.930 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:59.928+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:21:59.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:21:59.928+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:00.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:00 smithi153 ceph-mon[36569]: pgmap v15632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:00.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:00 smithi104 ceph-mon[36624]: pgmap v15632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:00 smithi081 ceph-mon[31453]: pgmap v15632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:02.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:02 smithi153 ceph-mon[36569]: pgmap v15633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:02 smithi104 ceph-mon[36624]: pgmap v15633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:02 smithi081 ceph-mon[31453]: pgmap v15633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:04.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:04 smithi153 ceph-mon[36569]: pgmap v15634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:04.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:04 smithi104 ceph-mon[36624]: pgmap v15634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:04.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:04 smithi081 ceph-mon[31453]: pgmap v15634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:06 smithi153 ceph-mon[36569]: pgmap v15635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:06.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:06 smithi104 ceph-mon[36624]: pgmap v15635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:06 smithi081 ceph-mon[31453]: pgmap v15635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:08 smithi153 ceph-mon[36569]: pgmap v15636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:08 smithi104 ceph-mon[36624]: pgmap v15636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:08.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:08 smithi081 ceph-mon[31453]: pgmap v15636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:10.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:10 smithi153 ceph-mon[36569]: pgmap v15637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:10 smithi104 ceph-mon[36624]: pgmap v15637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:10 smithi081 ceph-mon[31453]: pgmap v15637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:12 smithi153 ceph-mon[36569]: pgmap v15638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:12.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:12 smithi104 ceph-mon[36624]: pgmap v15638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:12.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:12 smithi081 ceph-mon[31453]: pgmap v15638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:14 smithi153 ceph-mon[36569]: pgmap v15639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:14.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:22:14.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:14 smithi104 ceph-mon[36624]: pgmap v15639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:14.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:22:14.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:14 smithi081 ceph-mon[31453]: pgmap v15639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:14.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:22:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:14.943+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:14.943+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:14.943+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:22:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:22:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:22:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:22:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:22:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:22:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:22:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:22:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:22:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:22:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:16 smithi153 ceph-mon[36569]: pgmap v15640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:16.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:16 smithi153 ceph-mon[36569]: pgmap v15641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:16 smithi104 ceph-mon[36624]: pgmap v15640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:16 smithi104 ceph-mon[36624]: pgmap v15641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:16.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:16 smithi081 ceph-mon[31453]: pgmap v15640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:16.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:16 smithi081 ceph-mon[31453]: pgmap v15641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:17.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:17.524+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:22:17.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:17.524+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:22:17.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:17.524+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 225 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40036f0 con 0x7f1704076980 2024-04-13T05:22:17.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:17.524+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 225 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:22:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:18 smithi153 ceph-mon[36569]: pgmap v15642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:18.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:18 smithi104 ceph-mon[36624]: pgmap v15642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:18.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:18 smithi081 ceph-mon[31453]: pgmap v15642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:20.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:20 smithi153 ceph-mon[36569]: pgmap v15643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:20 smithi104 ceph-mon[36624]: pgmap v15643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:20 smithi081 ceph-mon[31453]: pgmap v15643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:22.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:22 smithi104 ceph-mon[36624]: pgmap v15644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:22.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:22 smithi081 ceph-mon[31453]: pgmap v15644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:23.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:22 smithi153 ceph-mon[36569]: pgmap v15644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:24.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:24 smithi104 ceph-mon[36624]: pgmap v15645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:24.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:24 smithi081 ceph-mon[31453]: pgmap v15645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:24 smithi153 ceph-mon[36569]: pgmap v15645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:26.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:26 smithi104 ceph-mon[36624]: pgmap v15646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:26.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:26 smithi081 ceph-mon[31453]: pgmap v15646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:26 smithi153 ceph-mon[36569]: pgmap v15646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:28 smithi104 ceph-mon[36624]: pgmap v15647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:28 smithi081 ceph-mon[31453]: pgmap v15647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:28 smithi153 ceph-mon[36569]: pgmap v15647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:29.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:29.958+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:29.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:29.958+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:29.961 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:29.958+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:22:30.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:30 smithi104 ceph-mon[36624]: pgmap v15648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:30.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:30 smithi081 ceph-mon[31453]: pgmap v15648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:30 smithi153 ceph-mon[36569]: pgmap v15648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:32.861 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:32 smithi153 ceph-mon[36569]: pgmap v15649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:32.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:32 smithi081 ceph-mon[31453]: pgmap v15649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:32 smithi104 ceph-mon[36624]: pgmap v15649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:34 smithi081 ceph-mon[31453]: pgmap v15650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:34 smithi153 ceph-mon[36569]: pgmap v15650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:34 smithi104 ceph-mon[36624]: pgmap v15650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:36.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:36 smithi104 ceph-mon[36624]: pgmap v15651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:36 smithi081 ceph-mon[31453]: pgmap v15651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:36 smithi153 ceph-mon[36569]: pgmap v15651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:37 smithi081 ceph-mon[31453]: pgmap v15652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:37 smithi153 ceph-mon[36569]: pgmap v15652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:38.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:37 smithi104 ceph-mon[36624]: pgmap v15652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:39 smithi153 ceph-mon[36569]: pgmap v15653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:39 smithi104 ceph-mon[36624]: pgmap v15653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:39 smithi081 ceph-mon[31453]: pgmap v15653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:41 smithi153 ceph-mon[36569]: pgmap v15654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:41 smithi104 ceph-mon[36624]: pgmap v15654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:41 smithi081 ceph-mon[31453]: pgmap v15654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:43 smithi153 ceph-mon[36569]: pgmap v15655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:43 smithi104 ceph-mon[36624]: pgmap v15655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:43 smithi081 ceph-mon[31453]: pgmap v15655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:44.973+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:44.973+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:44.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:44.973+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:22:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:45 smithi153 ceph-mon[36569]: pgmap v15656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:45 smithi104 ceph-mon[36624]: pgmap v15656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:45 smithi081 ceph-mon[31453]: pgmap v15656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:47 smithi153 ceph-mon[36569]: pgmap v15657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:47 smithi104 ceph-mon[36624]: pgmap v15657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:47 smithi081 ceph-mon[31453]: pgmap v15657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:49 smithi153 ceph-mon[36569]: pgmap v15658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:49 smithi104 ceph-mon[36624]: pgmap v15658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:49 smithi081 ceph-mon[31453]: pgmap v15658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:51 smithi153 ceph-mon[36569]: pgmap v15659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:51 smithi104 ceph-mon[36624]: pgmap v15659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:51 smithi081 ceph-mon[31453]: pgmap v15659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:53 smithi153 ceph-mon[36569]: pgmap v15660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:53 smithi104 ceph-mon[36624]: pgmap v15660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:53 smithi081 ceph-mon[31453]: pgmap v15660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:55 smithi153 ceph-mon[36569]: pgmap v15661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:55 smithi104 ceph-mon[36624]: pgmap v15661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:55 smithi081 ceph-mon[31453]: pgmap v15661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:57 smithi153 ceph-mon[36569]: pgmap v15662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:57 smithi104 ceph-mon[36624]: pgmap v15662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:57 smithi081 ceph-mon[31453]: pgmap v15662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:22:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:59.988+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:59.988+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:22:59.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:22:59.988+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:23:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:22:59 smithi153 ceph-mon[36569]: pgmap v15663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:22:59 smithi104 ceph-mon[36624]: pgmap v15663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:22:59 smithi081 ceph-mon[31453]: pgmap v15663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:01 smithi153 ceph-mon[36569]: pgmap v15664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:01 smithi104 ceph-mon[36624]: pgmap v15664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:01 smithi081 ceph-mon[31453]: pgmap v15664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:02.531+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:23:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:02.531+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T05:23:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:02.532+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 226 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T05:23:02.533 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:02.532+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 226 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:23:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:03 smithi153 ceph-mon[36569]: pgmap v15665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:03 smithi104 ceph-mon[36624]: pgmap v15665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:03 smithi081 ceph-mon[31453]: pgmap v15665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:06 smithi153 ceph-mon[36569]: pgmap v15666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:06 smithi104 ceph-mon[36624]: pgmap v15666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:06 smithi081 ceph-mon[31453]: pgmap v15666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:08 smithi153 ceph-mon[36569]: pgmap v15667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:08 smithi104 ceph-mon[36624]: pgmap v15667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:08 smithi081 ceph-mon[31453]: pgmap v15667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:10 smithi153 ceph-mon[36569]: pgmap v15668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:10 smithi104 ceph-mon[36624]: pgmap v15668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:10 smithi081 ceph-mon[31453]: pgmap v15668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:12 smithi153 ceph-mon[36569]: pgmap v15669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:12 smithi104 ceph-mon[36624]: pgmap v15669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:12 smithi081 ceph-mon[31453]: pgmap v15669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:14 smithi153 ceph-mon[36569]: pgmap v15670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:14 smithi104 ceph-mon[36624]: pgmap v15670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:14 smithi081 ceph-mon[31453]: pgmap v15670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:15.002+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:23:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:15.002+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:23:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:15.002+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:23:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:16 smithi153 ceph-mon[36569]: pgmap v15671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:23:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:16 smithi104 ceph-mon[36624]: pgmap v15671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:23:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:16 smithi081 ceph-mon[31453]: pgmap v15671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:23:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:23:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:23:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:17 smithi153 ceph-mon[36569]: pgmap v15672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:17 smithi153 ceph-mon[36569]: pgmap v15673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:23:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:23:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:23:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:17 smithi104 ceph-mon[36624]: pgmap v15672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:17 smithi104 ceph-mon[36624]: pgmap v15673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:23:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:23:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:23:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:17 smithi081 ceph-mon[31453]: pgmap v15672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:17 smithi081 ceph-mon[31453]: pgmap v15673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:23:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:19 smithi153 ceph-mon[36569]: pgmap v15674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:19 smithi104 ceph-mon[36624]: pgmap v15674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:19 smithi081 ceph-mon[31453]: pgmap v15674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:21 smithi153 ceph-mon[36569]: pgmap v15675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:21.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:21 smithi104 ceph-mon[36624]: pgmap v15675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:21 smithi081 ceph-mon[31453]: pgmap v15675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:23.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:23 smithi153 ceph-mon[36569]: pgmap v15676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:23.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:23 smithi104 ceph-mon[36624]: pgmap v15676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:23.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:23 smithi081 ceph-mon[31453]: pgmap v15676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:25 smithi153 ceph-mon[36569]: pgmap v15677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:25 smithi104 ceph-mon[36624]: pgmap v15677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:25 smithi081 ceph-mon[31453]: pgmap v15677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:27 smithi153 ceph-mon[36569]: pgmap v15678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:27 smithi104 ceph-mon[36624]: pgmap v15678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:27 smithi081 ceph-mon[31453]: pgmap v15678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:29 smithi153 ceph-mon[36569]: pgmap v15679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:29 smithi104 ceph-mon[36624]: pgmap v15679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:29.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:29 smithi081 ceph-mon[31453]: pgmap v15679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:30.007 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:30.006+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:23:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:30.006+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:23:30.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:30.006+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:23:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:31 smithi153 ceph-mon[36569]: pgmap v15680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:31 smithi104 ceph-mon[36624]: pgmap v15680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:31 smithi081 ceph-mon[31453]: pgmap v15680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:33 smithi153 ceph-mon[36569]: pgmap v15681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:33 smithi104 ceph-mon[36624]: pgmap v15681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:33 smithi081 ceph-mon[31453]: pgmap v15681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:35 smithi153 ceph-mon[36569]: pgmap v15682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:35 smithi104 ceph-mon[36624]: pgmap v15682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:35.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:35 smithi081 ceph-mon[31453]: pgmap v15682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:37 smithi153 ceph-mon[36569]: pgmap v15683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:37 smithi104 ceph-mon[36624]: pgmap v15683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:37 smithi081 ceph-mon[31453]: pgmap v15683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:39 smithi153 ceph-mon[36569]: pgmap v15684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:39 smithi104 ceph-mon[36624]: pgmap v15684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:39.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:39 smithi081 ceph-mon[31453]: pgmap v15684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:41.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:41 smithi153 ceph-mon[36569]: pgmap v15685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:41.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:41 smithi104 ceph-mon[36624]: pgmap v15685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:41 smithi081 ceph-mon[31453]: pgmap v15685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:43 smithi153 ceph-mon[36569]: pgmap v15686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:43 smithi104 ceph-mon[36624]: pgmap v15686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:43.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:43 smithi081 ceph-mon[31453]: pgmap v15686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:45.019+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:23:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:45.019+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:23:45.021 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:23:45.019+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:23:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:45 smithi153 ceph-mon[36569]: pgmap v15687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:45 smithi104 ceph-mon[36624]: pgmap v15687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:45 smithi081 ceph-mon[31453]: pgmap v15687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:47 smithi153 ceph-mon[36569]: pgmap v15688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:47 smithi104 ceph-mon[36624]: pgmap v15688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:47 smithi081 ceph-mon[31453]: pgmap v15688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:49 smithi153 ceph-mon[36569]: pgmap v15689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:49 smithi104 ceph-mon[36624]: pgmap v15689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:49 smithi081 ceph-mon[31453]: pgmap v15689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:51 smithi153 ceph-mon[36569]: pgmap v15690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:51 smithi104 ceph-mon[36624]: pgmap v15690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:51.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:51 smithi081 ceph-mon[31453]: pgmap v15690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:53 smithi153 ceph-mon[36569]: pgmap v15691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:53.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:53 smithi104 ceph-mon[36624]: pgmap v15691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:53 smithi081 ceph-mon[31453]: pgmap v15691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:55 smithi153 ceph-mon[36569]: pgmap v15692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:55 smithi104 ceph-mon[36624]: pgmap v15692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:55 smithi081 ceph-mon[31453]: pgmap v15692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:57 smithi153 ceph-mon[36569]: pgmap v15693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:57 smithi104 ceph-mon[36624]: pgmap v15693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:57 smithi081 ceph-mon[31453]: pgmap v15693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:23:59 smithi153 ceph-mon[36569]: pgmap v15694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:23:59 smithi104 ceph-mon[36624]: pgmap v15694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:23:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:23:59 smithi081 ceph-mon[31453]: pgmap v15694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:00.035+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:24:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:00.035+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:24:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:00.035+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:24:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:01 smithi153 ceph-mon[36569]: pgmap v15695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:01 smithi104 ceph-mon[36624]: pgmap v15695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:01 smithi081 ceph-mon[31453]: pgmap v15695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:03 smithi153 ceph-mon[36569]: pgmap v15696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:03.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:03 smithi104 ceph-mon[36624]: pgmap v15696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:03 smithi081 ceph-mon[31453]: pgmap v15696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:05 smithi153 ceph-mon[36569]: pgmap v15697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:05 smithi104 ceph-mon[36624]: pgmap v15697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:05.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:05 smithi081 ceph-mon[31453]: pgmap v15697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:07.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:07 smithi104 ceph-mon[36624]: pgmap v15698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:07 smithi081 ceph-mon[31453]: pgmap v15698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:07 smithi153 ceph-mon[36569]: pgmap v15698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:09.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:09 smithi104 ceph-mon[36624]: pgmap v15699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:09.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:09 smithi081 ceph-mon[31453]: pgmap v15699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:09 smithi153 ceph-mon[36569]: pgmap v15699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:11 smithi104 ceph-mon[36624]: pgmap v15700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:11 smithi081 ceph-mon[31453]: pgmap v15700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:11 smithi153 ceph-mon[36569]: pgmap v15700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:13 smithi104 ceph-mon[36624]: pgmap v15701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:13 smithi081 ceph-mon[31453]: pgmap v15701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:14.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:13 smithi153 ceph-mon[36569]: pgmap v15701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:15.050+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:24:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:15.050+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:24:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:15.050+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:24:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:15 smithi104 ceph-mon[36624]: pgmap v15702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:15 smithi081 ceph-mon[31453]: pgmap v15702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:15 smithi153 ceph-mon[36569]: pgmap v15702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:17.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:17 smithi081 ceph-mon[31453]: pgmap v15703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:17.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:24:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:17 smithi153 ceph-mon[36569]: pgmap v15703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:24:18.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:17 smithi104 ceph-mon[36624]: pgmap v15703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:24:18.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:24:18.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:24:18.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:24:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:24:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:24:19.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:24:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:24:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:24:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:24:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:19 smithi081 ceph-mon[31453]: pgmap v15704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:19 smithi081 ceph-mon[31453]: pgmap v15705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:19 smithi153 ceph-mon[36569]: pgmap v15704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:19 smithi153 ceph-mon[36569]: pgmap v15705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:20.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:19 smithi104 ceph-mon[36624]: pgmap v15704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:20.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:19 smithi104 ceph-mon[36624]: pgmap v15705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:21 smithi081 ceph-mon[31453]: pgmap v15706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:21 smithi153 ceph-mon[36569]: pgmap v15706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:21 smithi104 ceph-mon[36624]: pgmap v15706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:23 smithi081 ceph-mon[31453]: pgmap v15707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:23 smithi153 ceph-mon[36569]: pgmap v15707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:24.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:23 smithi104 ceph-mon[36624]: pgmap v15707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:25 smithi081 ceph-mon[31453]: pgmap v15708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:25 smithi153 ceph-mon[36569]: pgmap v15708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:25 smithi104 ceph-mon[36624]: pgmap v15708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:27 smithi081 ceph-mon[31453]: pgmap v15709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:28.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:27 smithi153 ceph-mon[36569]: pgmap v15709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:27 smithi104 ceph-mon[36624]: pgmap v15709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:29 smithi153 ceph-mon[36569]: pgmap v15710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:29.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:29 smithi104 ceph-mon[36624]: pgmap v15710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:29.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:29 smithi081 ceph-mon[31453]: pgmap v15710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:30.066+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:24:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:30.066+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:24:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:30.066+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:24:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:31 smithi153 ceph-mon[36569]: pgmap v15711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:31.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:31 smithi104 ceph-mon[36624]: pgmap v15711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:31.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:31 smithi081 ceph-mon[31453]: pgmap v15711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:33 smithi153 ceph-mon[36569]: pgmap v15712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:33 smithi104 ceph-mon[36624]: pgmap v15712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:33 smithi081 ceph-mon[31453]: pgmap v15712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:35 smithi153 ceph-mon[36569]: pgmap v15713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:35.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:35 smithi104 ceph-mon[36624]: pgmap v15713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:35 smithi081 ceph-mon[31453]: pgmap v15713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:37.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:37 smithi153 ceph-mon[36569]: pgmap v15714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:37.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:37 smithi104 ceph-mon[36624]: pgmap v15714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:37 smithi081 ceph-mon[31453]: pgmap v15714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:39.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:39 smithi153 ceph-mon[36569]: pgmap v15715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:39.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:39 smithi104 ceph-mon[36624]: pgmap v15715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:39 smithi081 ceph-mon[31453]: pgmap v15715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:41.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:41 smithi081 ceph-mon[31453]: pgmap v15716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:41 smithi153 ceph-mon[36569]: pgmap v15716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:41.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:41 smithi104 ceph-mon[36624]: pgmap v15716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:43 smithi153 ceph-mon[36569]: pgmap v15717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:43.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:43 smithi104 ceph-mon[36624]: pgmap v15717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:43 smithi081 ceph-mon[31453]: pgmap v15717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:45.070+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:24:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:45.070+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:24:45.072 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:24:45.070+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:24:45.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:45 smithi153 ceph-mon[36569]: pgmap v15718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:45 smithi104 ceph-mon[36624]: pgmap v15718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:45.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:45 smithi081 ceph-mon[31453]: pgmap v15718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:47 smithi153 ceph-mon[36569]: pgmap v15719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:47 smithi104 ceph-mon[36624]: pgmap v15719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:47 smithi081 ceph-mon[31453]: pgmap v15719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:49 smithi153 ceph-mon[36569]: pgmap v15720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:49 smithi104 ceph-mon[36624]: pgmap v15720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:49 smithi081 ceph-mon[31453]: pgmap v15720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:51.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:51 smithi153 ceph-mon[36569]: pgmap v15721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:51.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:51 smithi104 ceph-mon[36624]: pgmap v15721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:51 smithi081 ceph-mon[31453]: pgmap v15721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:53 smithi153 ceph-mon[36569]: pgmap v15722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:53 smithi104 ceph-mon[36624]: pgmap v15722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:53.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:53 smithi081 ceph-mon[31453]: pgmap v15722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:55 smithi153 ceph-mon[36569]: pgmap v15723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:55.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:55 smithi104 ceph-mon[36624]: pgmap v15723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:55 smithi081 ceph-mon[31453]: pgmap v15723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:57 smithi153 ceph-mon[36569]: pgmap v15724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:57 smithi104 ceph-mon[36624]: pgmap v15724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:57 smithi081 ceph-mon[31453]: pgmap v15724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:59.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:24:59 smithi153 ceph-mon[36569]: pgmap v15725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:59.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:24:59 smithi104 ceph-mon[36624]: pgmap v15725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:24:59.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:24:59 smithi081 ceph-mon[31453]: pgmap v15725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:00.084+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:25:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:00.084+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:25:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:00.084+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:25:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:01 smithi153 ceph-mon[36569]: pgmap v15726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:01.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:01 smithi104 ceph-mon[36624]: pgmap v15726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:01.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:01 smithi081 ceph-mon[31453]: pgmap v15726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:03 smithi153 ceph-mon[36569]: pgmap v15727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:03.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:03 smithi104 ceph-mon[36624]: pgmap v15727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:03.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:03 smithi081 ceph-mon[31453]: pgmap v15727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:05 smithi153 ceph-mon[36569]: pgmap v15728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:05.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:05 smithi104 ceph-mon[36624]: pgmap v15728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:05 smithi081 ceph-mon[31453]: pgmap v15728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:07 smithi153 ceph-mon[36569]: pgmap v15729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:07 smithi104 ceph-mon[36624]: pgmap v15729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:07 smithi081 ceph-mon[31453]: pgmap v15729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:09 smithi153 ceph-mon[36569]: pgmap v15730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:09.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:09 smithi104 ceph-mon[36624]: pgmap v15730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:09.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:09 smithi081 ceph-mon[31453]: pgmap v15730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:11.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:11 smithi153 ceph-mon[36569]: pgmap v15731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:11.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:11 smithi104 ceph-mon[36624]: pgmap v15731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:11.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:11 smithi081 ceph-mon[31453]: pgmap v15731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:13.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:13 smithi153 ceph-mon[36569]: pgmap v15732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:13.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:13 smithi104 ceph-mon[36624]: pgmap v15732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:13.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:13 smithi081 ceph-mon[31453]: pgmap v15732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:15.092+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:25:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:15.093+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:25:15.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:15.093+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:25:15.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:15 smithi153 ceph-mon[36569]: pgmap v15733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:15.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:15 smithi104 ceph-mon[36624]: pgmap v15733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:15.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:15 smithi081 ceph-mon[31453]: pgmap v15733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:17.553+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:25:17.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:17.553+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T05:25:17.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:17.554+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 227 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:25:17.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:17.554+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 227 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T05:25:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:17 smithi153 ceph-mon[36569]: pgmap v15734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:17.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:17 smithi104 ceph-mon[36624]: pgmap v15734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:17.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:17 smithi081 ceph-mon[31453]: pgmap v15734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:25:18.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:25:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:25:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:19 smithi153 ceph-mon[36569]: pgmap v15735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:19 smithi104 ceph-mon[36624]: pgmap v15735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:19.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:19 smithi081 ceph-mon[31453]: pgmap v15735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:21.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:21 smithi081 ceph-mon[31453]: pgmap v15736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:21 smithi153 ceph-mon[36569]: pgmap v15736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:22.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:22.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:22.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:22.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:21 smithi104 ceph-mon[36624]: pgmap v15736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:23 smithi081 ceph-mon[31453]: pgmap v15737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:23 smithi153 ceph-mon[36569]: pgmap v15737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:24.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:24.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:23 smithi104 ceph-mon[36624]: pgmap v15737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:24.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:24.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:25 smithi081 ceph-mon[31453]: pgmap v15738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:25:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:25:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:25 smithi153 ceph-mon[36569]: pgmap v15738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:25:26.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:25:26.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:25 smithi104 ceph-mon[36624]: pgmap v15738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:25:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:25:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:25:26.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:26 smithi081 ceph-mon[31453]: pgmap v15739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:26.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:26 smithi081 ceph-mon[31453]: pgmap v15740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:26 smithi153 ceph-mon[36569]: pgmap v15739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:26 smithi153 ceph-mon[36569]: pgmap v15740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:26 smithi104 ceph-mon[36624]: pgmap v15739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:26 smithi104 ceph-mon[36624]: pgmap v15740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:28 smithi081 ceph-mon[31453]: pgmap v15741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:28 smithi153 ceph-mon[36569]: pgmap v15741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:28 smithi104 ceph-mon[36624]: pgmap v15741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:30.094+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:25:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:30.094+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:25:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:30.094+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:25:30.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:30 smithi081 ceph-mon[31453]: pgmap v15742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:30 smithi153 ceph-mon[36569]: pgmap v15742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:30 smithi104 ceph-mon[36624]: pgmap v15742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:31 smithi081 ceph-mon[31453]: pgmap v15743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:31 smithi153 ceph-mon[36569]: pgmap v15743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:32.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:31 smithi104 ceph-mon[36624]: pgmap v15743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:33 smithi153 ceph-mon[36569]: pgmap v15744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:33 smithi104 ceph-mon[36624]: pgmap v15744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:33 smithi081 ceph-mon[31453]: pgmap v15744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:35 smithi153 ceph-mon[36569]: pgmap v15745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:36.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:35 smithi104 ceph-mon[36624]: pgmap v15745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:35 smithi081 ceph-mon[31453]: pgmap v15745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:37 smithi153 ceph-mon[36569]: pgmap v15746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:38.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:37 smithi104 ceph-mon[36624]: pgmap v15746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:37 smithi081 ceph-mon[31453]: pgmap v15746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:40.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:39 smithi153 ceph-mon[36569]: pgmap v15747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:39 smithi104 ceph-mon[36624]: pgmap v15747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:39 smithi081 ceph-mon[31453]: pgmap v15747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:41 smithi153 ceph-mon[36569]: pgmap v15748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:42.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:41 smithi104 ceph-mon[36624]: pgmap v15748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:42.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:41 smithi081 ceph-mon[31453]: pgmap v15748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:44.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:43 smithi153 ceph-mon[36569]: pgmap v15749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:44.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:43 smithi104 ceph-mon[36624]: pgmap v15749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:43 smithi081 ceph-mon[31453]: pgmap v15749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:45.103+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:25:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:45.103+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:25:45.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:25:45.103+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:25:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:45 smithi153 ceph-mon[36569]: pgmap v15750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:46.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:45 smithi104 ceph-mon[36624]: pgmap v15750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:46.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:45 smithi081 ceph-mon[31453]: pgmap v15750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:47 smithi153 ceph-mon[36569]: pgmap v15751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:47 smithi104 ceph-mon[36624]: pgmap v15751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:47 smithi081 ceph-mon[31453]: pgmap v15751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:49 smithi153 ceph-mon[36569]: pgmap v15752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:50.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:49 smithi104 ceph-mon[36624]: pgmap v15752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:49 smithi081 ceph-mon[31453]: pgmap v15752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:51 smithi153 ceph-mon[36569]: pgmap v15753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:51 smithi104 ceph-mon[36624]: pgmap v15753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:51 smithi081 ceph-mon[31453]: pgmap v15753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:53 smithi153 ceph-mon[36569]: pgmap v15754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:53 smithi104 ceph-mon[36624]: pgmap v15754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:53 smithi081 ceph-mon[31453]: pgmap v15754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:55 smithi153 ceph-mon[36569]: pgmap v15755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:55 smithi104 ceph-mon[36624]: pgmap v15755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:55 smithi081 ceph-mon[31453]: pgmap v15755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:57 smithi153 ceph-mon[36569]: pgmap v15756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:57 smithi104 ceph-mon[36624]: pgmap v15756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:25:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:57 smithi081 ceph-mon[31453]: pgmap v15756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:25:59 smithi104 ceph-mon[36624]: pgmap v15757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:00.107+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:26:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:00.107+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:26:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:00.107+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:26:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:25:59 smithi081 ceph-mon[31453]: pgmap v15757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:25:59 smithi153 ceph-mon[36569]: pgmap v15757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:01 smithi104 ceph-mon[36624]: pgmap v15758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:01 smithi081 ceph-mon[31453]: pgmap v15758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:01 smithi153 ceph-mon[36569]: pgmap v15758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:04.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:03 smithi104 ceph-mon[36624]: pgmap v15759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:04.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:03 smithi153 ceph-mon[36569]: pgmap v15759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:03 smithi081 ceph-mon[31453]: pgmap v15759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:05 smithi081 ceph-mon[31453]: pgmap v15760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:05 smithi153 ceph-mon[36569]: pgmap v15760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:05 smithi104 ceph-mon[36624]: pgmap v15760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:08.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:07 smithi104 ceph-mon[36624]: pgmap v15761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:08.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:07 smithi081 ceph-mon[31453]: pgmap v15761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:07 smithi153 ceph-mon[36569]: pgmap v15761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:09 smithi081 ceph-mon[31453]: pgmap v15762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:09 smithi153 ceph-mon[36569]: pgmap v15762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:09 smithi104 ceph-mon[36624]: pgmap v15762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:11 smithi081 ceph-mon[31453]: pgmap v15763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:11 smithi153 ceph-mon[36569]: pgmap v15763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:11 smithi104 ceph-mon[36624]: pgmap v15763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:13 smithi081 ceph-mon[31453]: pgmap v15764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:13 smithi153 ceph-mon[36569]: pgmap v15764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:13 smithi104 ceph-mon[36624]: pgmap v15764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:15.114+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:26:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:15.114+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:26:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:15.114+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:26:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:15 smithi081 ceph-mon[31453]: pgmap v15765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:15 smithi153 ceph-mon[36569]: pgmap v15765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:15 smithi104 ceph-mon[36624]: pgmap v15765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:17 smithi081 ceph-mon[31453]: pgmap v15766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:17 smithi153 ceph-mon[36569]: pgmap v15766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:17 smithi104 ceph-mon[36624]: pgmap v15766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:20.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:19 smithi081 ceph-mon[31453]: pgmap v15767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:19 smithi153 ceph-mon[36569]: pgmap v15767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:19 smithi104 ceph-mon[36624]: pgmap v15767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:21 smithi081 ceph-mon[31453]: pgmap v15768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:21 smithi153 ceph-mon[36569]: pgmap v15768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:21 smithi104 ceph-mon[36624]: pgmap v15768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:23 smithi081 ceph-mon[31453]: pgmap v15769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:23 smithi153 ceph-mon[36569]: pgmap v15769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:23 smithi104 ceph-mon[36624]: pgmap v15769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:25 smithi081 ceph-mon[31453]: pgmap v15770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:26:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:25 smithi153 ceph-mon[36569]: pgmap v15770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:26:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:25 smithi104 ceph-mon[36624]: pgmap v15770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:26:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:26:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:26:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:26 smithi081 ceph-mon[31453]: pgmap v15771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:26 smithi081 ceph-mon[31453]: pgmap v15772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:26:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:26:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:26 smithi153 ceph-mon[36569]: pgmap v15771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:26 smithi153 ceph-mon[36569]: pgmap v15772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:26:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:26:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:26 smithi104 ceph-mon[36624]: pgmap v15771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:26 smithi104 ceph-mon[36624]: pgmap v15772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:26:29.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:29 smithi081 ceph-mon[31453]: pgmap v15773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:29 smithi153 ceph-mon[36569]: pgmap v15773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:29 smithi104 ceph-mon[36624]: pgmap v15773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:30.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:30.127+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:26:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:30.127+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:26:30.130 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:30.127+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:26:31.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:31 smithi081 ceph-mon[31453]: pgmap v15774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:31 smithi153 ceph-mon[36569]: pgmap v15774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:31 smithi104 ceph-mon[36624]: pgmap v15774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:33 smithi081 ceph-mon[31453]: pgmap v15775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:33 smithi153 ceph-mon[36569]: pgmap v15775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:33 smithi104 ceph-mon[36624]: pgmap v15775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:35 smithi081 ceph-mon[31453]: pgmap v15776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:35 smithi153 ceph-mon[36569]: pgmap v15776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:35 smithi104 ceph-mon[36624]: pgmap v15776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:37 smithi081 ceph-mon[31453]: pgmap v15777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:37 smithi153 ceph-mon[36569]: pgmap v15777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:37 smithi104 ceph-mon[36624]: pgmap v15777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:39.636 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:39 smithi104 ceph-mon[36624]: pgmap v15778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:39 smithi081 ceph-mon[31453]: pgmap v15778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:39 smithi153 ceph-mon[36569]: pgmap v15778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:41.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:41 smithi081 ceph-mon[31453]: pgmap v15779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:41.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:41 smithi153 ceph-mon[36569]: pgmap v15779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:41.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:41 smithi104 ceph-mon[36624]: pgmap v15779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:43 smithi081 ceph-mon[31453]: pgmap v15780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:43 smithi153 ceph-mon[36569]: pgmap v15780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:43 smithi104 ceph-mon[36624]: pgmap v15780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:45.142+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:26:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:45.142+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:26:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:26:45.142+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:26:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:45 smithi081 ceph-mon[31453]: pgmap v15781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:45 smithi153 ceph-mon[36569]: pgmap v15781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:45 smithi104 ceph-mon[36624]: pgmap v15781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:47 smithi153 ceph-mon[36569]: pgmap v15782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:47 smithi104 ceph-mon[36624]: pgmap v15782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:47 smithi081 ceph-mon[31453]: pgmap v15782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:49 smithi153 ceph-mon[36569]: pgmap v15783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:49 smithi104 ceph-mon[36624]: pgmap v15783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:49 smithi081 ceph-mon[31453]: pgmap v15783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:51 smithi153 ceph-mon[36569]: pgmap v15784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:51 smithi104 ceph-mon[36624]: pgmap v15784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:51.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:51 smithi081 ceph-mon[31453]: pgmap v15784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:53 smithi153 ceph-mon[36569]: pgmap v15785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:53.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:53 smithi104 ceph-mon[36624]: pgmap v15785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:53 smithi081 ceph-mon[31453]: pgmap v15785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:55 smithi153 ceph-mon[36569]: pgmap v15786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:55 smithi104 ceph-mon[36624]: pgmap v15786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:55 smithi081 ceph-mon[31453]: pgmap v15786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:57 smithi153 ceph-mon[36569]: pgmap v15787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:57 smithi104 ceph-mon[36624]: pgmap v15787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:57 smithi081 ceph-mon[31453]: pgmap v15787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:26:59 smithi153 ceph-mon[36569]: pgmap v15788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:26:59 smithi104 ceph-mon[36624]: pgmap v15788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:26:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:26:59 smithi081 ceph-mon[31453]: pgmap v15788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:00.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:00.149+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:27:00.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:00.149+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:27:00.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:00.149+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:27:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:01 smithi153 ceph-mon[36569]: pgmap v15789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:01 smithi104 ceph-mon[36624]: pgmap v15789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:01 smithi081 ceph-mon[31453]: pgmap v15789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:03 smithi153 ceph-mon[36569]: pgmap v15790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:03.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:03 smithi104 ceph-mon[36624]: pgmap v15790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:03 smithi081 ceph-mon[31453]: pgmap v15790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:05 smithi153 ceph-mon[36569]: pgmap v15791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:05 smithi104 ceph-mon[36624]: pgmap v15791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:05.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:05 smithi081 ceph-mon[31453]: pgmap v15791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:07 smithi153 ceph-mon[36569]: pgmap v15792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:07.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:07 smithi104 ceph-mon[36624]: pgmap v15792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:07.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:07 smithi081 ceph-mon[31453]: pgmap v15792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:09 smithi153 ceph-mon[36569]: pgmap v15793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:09.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:09 smithi104 ceph-mon[36624]: pgmap v15793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:09.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:09 smithi081 ceph-mon[31453]: pgmap v15793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:11 smithi153 ceph-mon[36569]: pgmap v15794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:11 smithi104 ceph-mon[36624]: pgmap v15794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:11 smithi081 ceph-mon[31453]: pgmap v15794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:13 smithi153 ceph-mon[36569]: pgmap v15795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:13 smithi104 ceph-mon[36624]: pgmap v15795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:13 smithi081 ceph-mon[31453]: pgmap v15795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:15.156+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:27:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:15.156+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:27:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:15.156+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:27:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:15 smithi153 ceph-mon[36569]: pgmap v15796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:15 smithi104 ceph-mon[36624]: pgmap v15796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:15 smithi081 ceph-mon[31453]: pgmap v15796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:17 smithi153 ceph-mon[36569]: pgmap v15797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:17.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:17 smithi104 ceph-mon[36624]: pgmap v15797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:17.968 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:17 smithi081 ceph-mon[31453]: pgmap v15797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:19 smithi153 ceph-mon[36569]: pgmap v15798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:19 smithi104 ceph-mon[36624]: pgmap v15798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:19.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:19 smithi081 ceph-mon[31453]: pgmap v15798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:21 smithi153 ceph-mon[36569]: pgmap v15799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:21.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:21 smithi104 ceph-mon[36624]: pgmap v15799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:21.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:21 smithi081 ceph-mon[31453]: pgmap v15799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:23 smithi153 ceph-mon[36569]: pgmap v15800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:23.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:23 smithi104 ceph-mon[36624]: pgmap v15800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:23 smithi081 ceph-mon[31453]: pgmap v15800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:25 smithi153 ceph-mon[36569]: pgmap v15801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:25 smithi104 ceph-mon[36624]: pgmap v15801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:25 smithi081 ceph-mon[31453]: pgmap v15801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:27 smithi153 ceph-mon[36569]: pgmap v15802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:27:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:27 smithi104 ceph-mon[36624]: pgmap v15802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:27:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:27 smithi081 ceph-mon[31453]: pgmap v15802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:27:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:27:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:27:28.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:27:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:27:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:27:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:27:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:27:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:27:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:27:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:29 smithi153 ceph-mon[36569]: pgmap v15803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:29 smithi153 ceph-mon[36569]: pgmap v15804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:29 smithi104 ceph-mon[36624]: pgmap v15803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:29.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:29 smithi104 ceph-mon[36624]: pgmap v15804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:29.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:29 smithi081 ceph-mon[31453]: pgmap v15803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:29.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:29 smithi081 ceph-mon[31453]: pgmap v15804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:30.166+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:27:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:30.166+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:27:30.168 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:30.166+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:27:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:31 smithi153 ceph-mon[36569]: pgmap v15805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:31 smithi104 ceph-mon[36624]: pgmap v15805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:31 smithi081 ceph-mon[31453]: pgmap v15805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:33 smithi153 ceph-mon[36569]: pgmap v15806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:33 smithi104 ceph-mon[36624]: pgmap v15806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:33 smithi081 ceph-mon[31453]: pgmap v15806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:35 smithi153 ceph-mon[36569]: pgmap v15807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:35 smithi104 ceph-mon[36624]: pgmap v15807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:35.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:35 smithi081 ceph-mon[31453]: pgmap v15807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:37 smithi153 ceph-mon[36569]: pgmap v15808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:37 smithi104 ceph-mon[36624]: pgmap v15808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:37 smithi081 ceph-mon[31453]: pgmap v15808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:39 smithi153 ceph-mon[36569]: pgmap v15809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:39 smithi104 ceph-mon[36624]: pgmap v15809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:39.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:39 smithi081 ceph-mon[31453]: pgmap v15809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:41.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:41 smithi104 ceph-mon[36624]: pgmap v15810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:41 smithi081 ceph-mon[31453]: pgmap v15810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:42.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:41 smithi153 ceph-mon[36569]: pgmap v15810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:43 smithi104 ceph-mon[36624]: pgmap v15811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:43.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:43 smithi081 ceph-mon[31453]: pgmap v15811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:44.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:43 smithi153 ceph-mon[36569]: pgmap v15811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:45.166+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:45.166+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:27:45.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:27:45.166+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:27:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:45 smithi104 ceph-mon[36624]: pgmap v15812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:45 smithi081 ceph-mon[31453]: pgmap v15812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:45 smithi153 ceph-mon[36569]: pgmap v15812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:47 smithi104 ceph-mon[36624]: pgmap v15813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:47 smithi081 ceph-mon[31453]: pgmap v15813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:47 smithi153 ceph-mon[36569]: pgmap v15813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:49 smithi104 ceph-mon[36624]: pgmap v15814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:49 smithi081 ceph-mon[31453]: pgmap v15814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:50.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:49 smithi153 ceph-mon[36569]: pgmap v15814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:51.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:51 smithi081 ceph-mon[31453]: pgmap v15815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:51 smithi153 ceph-mon[36569]: pgmap v15815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:51 smithi104 ceph-mon[36624]: pgmap v15815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:53 smithi081 ceph-mon[31453]: pgmap v15816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:53 smithi153 ceph-mon[36569]: pgmap v15816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:53 smithi104 ceph-mon[36624]: pgmap v15816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:55 smithi081 ceph-mon[31453]: pgmap v15817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:55 smithi153 ceph-mon[36569]: pgmap v15817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:55 smithi104 ceph-mon[36624]: pgmap v15817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:57 smithi081 ceph-mon[31453]: pgmap v15818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:57 smithi153 ceph-mon[36569]: pgmap v15818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:57 smithi104 ceph-mon[36624]: pgmap v15818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:27:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:27:59 smithi081 ceph-mon[31453]: pgmap v15819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:27:59 smithi153 ceph-mon[36569]: pgmap v15819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:27:59 smithi104 ceph-mon[36624]: pgmap v15819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:00.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:00.176+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:28:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:00.176+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:28:00.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:00.176+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:28:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:01 smithi081 ceph-mon[31453]: pgmap v15820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:01 smithi153 ceph-mon[36569]: pgmap v15820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:01 smithi104 ceph-mon[36624]: pgmap v15820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:03 smithi081 ceph-mon[31453]: pgmap v15821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:03 smithi153 ceph-mon[36569]: pgmap v15821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:04.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:03 smithi104 ceph-mon[36624]: pgmap v15821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:05.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:05 smithi081 ceph-mon[31453]: pgmap v15822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:05 smithi153 ceph-mon[36569]: pgmap v15822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:05 smithi104 ceph-mon[36624]: pgmap v15822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:07.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:07 smithi081 ceph-mon[31453]: pgmap v15823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:07 smithi153 ceph-mon[36569]: pgmap v15823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:07 smithi104 ceph-mon[36624]: pgmap v15823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:09.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:09 smithi104 ceph-mon[36624]: pgmap v15824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:09.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:09 smithi081 ceph-mon[31453]: pgmap v15824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:09 smithi153 ceph-mon[36569]: pgmap v15824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:11 smithi081 ceph-mon[31453]: pgmap v15825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:11 smithi153 ceph-mon[36569]: pgmap v15825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:12.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:11 smithi104 ceph-mon[36624]: pgmap v15825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:13 smithi081 ceph-mon[31453]: pgmap v15826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:14.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:13 smithi153 ceph-mon[36569]: pgmap v15826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:13 smithi104 ceph-mon[36624]: pgmap v15826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:15.190+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:28:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:15.190+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:28:15.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:15.190+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:28:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:15 smithi081 ceph-mon[31453]: pgmap v15827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:15 smithi153 ceph-mon[36569]: pgmap v15827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:15 smithi104 ceph-mon[36624]: pgmap v15827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:17.584+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:28:17.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:17.584+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T05:28:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:17.584+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 228 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T05:28:17.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:17.584+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 228 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:28:17.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:17 smithi081 ceph-mon[31453]: pgmap v15828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:17 smithi153 ceph-mon[36569]: pgmap v15828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:17 smithi104 ceph-mon[36624]: pgmap v15828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:19 smithi153 ceph-mon[36569]: pgmap v15829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:20.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:19 smithi104 ceph-mon[36624]: pgmap v15829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:20.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:19 smithi081 ceph-mon[31453]: pgmap v15829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:21 smithi153 ceph-mon[36569]: pgmap v15830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:21 smithi104 ceph-mon[36624]: pgmap v15830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:21 smithi081 ceph-mon[31453]: pgmap v15830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:22 smithi153 ceph-mon[36569]: pgmap v15831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:22 smithi104 ceph-mon[36624]: pgmap v15831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:22 smithi081 ceph-mon[31453]: pgmap v15831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:24 smithi153 ceph-mon[36569]: pgmap v15832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:24 smithi104 ceph-mon[36624]: pgmap v15832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:24 smithi081 ceph-mon[31453]: pgmap v15832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:26 smithi153 ceph-mon[36569]: pgmap v15833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:26 smithi104 ceph-mon[36624]: pgmap v15833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:26 smithi081 ceph-mon[31453]: pgmap v15833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:28 smithi153 ceph-mon[36569]: pgmap v15834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:28:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:28 smithi104 ceph-mon[36624]: pgmap v15834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:28:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:28 smithi081 ceph-mon[31453]: pgmap v15834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:28:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:30.201+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:28:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:30.201+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:28:30.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:30.201+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:28:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:28:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:28:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:29 smithi153 ceph-mon[36569]: pgmap v15835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:29 smithi153 ceph-mon[36569]: pgmap v15836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:28:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:28:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:28:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:29 smithi104 ceph-mon[36624]: pgmap v15835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:29 smithi104 ceph-mon[36624]: pgmap v15836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:28:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:28:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:28:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:29 smithi081 ceph-mon[31453]: pgmap v15835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:29 smithi081 ceph-mon[31453]: pgmap v15836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:28:32.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:32 smithi104 ceph-mon[36624]: pgmap v15837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:32.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:32 smithi081 ceph-mon[31453]: pgmap v15837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:32 smithi153 ceph-mon[36569]: pgmap v15837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:34 smithi081 ceph-mon[31453]: pgmap v15838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:34 smithi153 ceph-mon[36569]: pgmap v15838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:34 smithi104 ceph-mon[36624]: pgmap v15838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:36.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:36 smithi153 ceph-mon[36569]: pgmap v15839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:36.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:36 smithi081 ceph-mon[31453]: pgmap v15839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:36 smithi104 ceph-mon[36624]: pgmap v15839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:38 smithi081 ceph-mon[31453]: pgmap v15840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:38 smithi153 ceph-mon[36569]: pgmap v15840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:38 smithi104 ceph-mon[36624]: pgmap v15840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:40.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:40 smithi104 ceph-mon[36624]: pgmap v15841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:40.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:40 smithi081 ceph-mon[31453]: pgmap v15841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:40 smithi153 ceph-mon[36569]: pgmap v15841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:42 smithi081 ceph-mon[31453]: pgmap v15842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:42 smithi153 ceph-mon[36569]: pgmap v15842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:43.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:42 smithi104 ceph-mon[36624]: pgmap v15842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:44.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:44 smithi081 ceph-mon[31453]: pgmap v15843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:44 smithi153 ceph-mon[36569]: pgmap v15843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:44 smithi104 ceph-mon[36624]: pgmap v15843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:45.218+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:28:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:45.218+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:28:45.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:28:45.218+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:28:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:46 smithi081 ceph-mon[31453]: pgmap v15844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:46 smithi153 ceph-mon[36569]: pgmap v15844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:47.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:46 smithi104 ceph-mon[36624]: pgmap v15844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:48 smithi081 ceph-mon[31453]: pgmap v15845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:48 smithi153 ceph-mon[36569]: pgmap v15845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:48 smithi104 ceph-mon[36624]: pgmap v15845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:50 smithi081 ceph-mon[31453]: pgmap v15846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:50 smithi153 ceph-mon[36569]: pgmap v15846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:50 smithi104 ceph-mon[36624]: pgmap v15846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:52 smithi081 ceph-mon[31453]: pgmap v15847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:52 smithi153 ceph-mon[36569]: pgmap v15847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:52 smithi104 ceph-mon[36624]: pgmap v15847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:54 smithi081 ceph-mon[31453]: pgmap v15848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:54 smithi153 ceph-mon[36569]: pgmap v15848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:54 smithi104 ceph-mon[36624]: pgmap v15848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:56.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:56 smithi081 ceph-mon[31453]: pgmap v15849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:56 smithi153 ceph-mon[36569]: pgmap v15849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:56 smithi104 ceph-mon[36624]: pgmap v15849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:28:58 smithi081 ceph-mon[31453]: pgmap v15850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:28:58 smithi153 ceph-mon[36569]: pgmap v15850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:28:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:28:58 smithi104 ceph-mon[36624]: pgmap v15850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:00.231+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:29:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:00.231+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:29:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:00.231+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:29:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:00 smithi081 ceph-mon[31453]: pgmap v15851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:00 smithi153 ceph-mon[36569]: pgmap v15851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:00 smithi104 ceph-mon[36624]: pgmap v15851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:02 smithi153 ceph-mon[36569]: pgmap v15852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:02 smithi104 ceph-mon[36624]: pgmap v15852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:02 smithi081 ceph-mon[31453]: pgmap v15852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:04 smithi153 ceph-mon[36569]: pgmap v15853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:04 smithi104 ceph-mon[36624]: pgmap v15853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:04 smithi081 ceph-mon[31453]: pgmap v15853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:06 smithi153 ceph-mon[36569]: pgmap v15854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:06 smithi104 ceph-mon[36624]: pgmap v15854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:06 smithi081 ceph-mon[31453]: pgmap v15854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:08 smithi153 ceph-mon[36569]: pgmap v15855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:08 smithi104 ceph-mon[36624]: pgmap v15855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:08 smithi081 ceph-mon[31453]: pgmap v15855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:09 smithi153 ceph-mon[36569]: pgmap v15856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:10.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:09 smithi104 ceph-mon[36624]: pgmap v15856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:09 smithi081 ceph-mon[31453]: pgmap v15856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:12.862 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:12 smithi081 ceph-mon[31453]: pgmap v15857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:12 smithi153 ceph-mon[36569]: pgmap v15857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:12 smithi104 ceph-mon[36624]: pgmap v15857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:14.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:14 smithi081 ceph-mon[31453]: pgmap v15858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:14 smithi153 ceph-mon[36569]: pgmap v15858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:14 smithi104 ceph-mon[36624]: pgmap v15858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:15.246+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:29:15.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:15.246+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:29:15.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:15.246+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:29:16.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:16 smithi081 ceph-mon[31453]: pgmap v15859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:16 smithi153 ceph-mon[36569]: pgmap v15859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:17.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:16 smithi104 ceph-mon[36624]: pgmap v15859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:18.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:18 smithi081 ceph-mon[31453]: pgmap v15860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:18 smithi153 ceph-mon[36569]: pgmap v15860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:18 smithi104 ceph-mon[36624]: pgmap v15860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:20 smithi081 ceph-mon[31453]: pgmap v15861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:20 smithi153 ceph-mon[36569]: pgmap v15861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:20 smithi104 ceph-mon[36624]: pgmap v15861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:22.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:22 smithi081 ceph-mon[31453]: pgmap v15862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:22 smithi153 ceph-mon[36569]: pgmap v15862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:22 smithi104 ceph-mon[36624]: pgmap v15862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:24.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:24 smithi081 ceph-mon[31453]: pgmap v15863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:24 smithi153 ceph-mon[36569]: pgmap v15863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:25.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:24 smithi104 ceph-mon[36624]: pgmap v15863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:26.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:26 smithi081 ceph-mon[31453]: pgmap v15864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:26 smithi153 ceph-mon[36569]: pgmap v15864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:26 smithi104 ceph-mon[36624]: pgmap v15864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:28 smithi081 ceph-mon[31453]: pgmap v15865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:28 smithi153 ceph-mon[36569]: pgmap v15865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:28 smithi104 ceph-mon[36624]: pgmap v15865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:30.260+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:29:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:30.260+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:29:30.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:30.260+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:29:30.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:30 smithi081 ceph-mon[31453]: pgmap v15866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:30.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:29:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:30 smithi153 ceph-mon[36569]: pgmap v15866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:29:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:30 smithi104 ceph-mon[36624]: pgmap v15866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:29:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:29:32.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:29:32.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:29:32.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:29:32.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:29:32.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:29:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:29:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:29:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:29:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:32 smithi153 ceph-mon[36569]: pgmap v15867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:33.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:32 smithi153 ceph-mon[36569]: pgmap v15868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:32 smithi104 ceph-mon[36624]: pgmap v15867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:32 smithi104 ceph-mon[36624]: pgmap v15868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:32 smithi081 ceph-mon[31453]: pgmap v15867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:32 smithi081 ceph-mon[31453]: pgmap v15868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:34 smithi153 ceph-mon[36569]: pgmap v15869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:34 smithi104 ceph-mon[36624]: pgmap v15869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:34 smithi081 ceph-mon[31453]: pgmap v15869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:36 smithi153 ceph-mon[36569]: pgmap v15870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:36 smithi104 ceph-mon[36624]: pgmap v15870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:36 smithi081 ceph-mon[31453]: pgmap v15870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:38 smithi153 ceph-mon[36569]: pgmap v15871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:38 smithi104 ceph-mon[36624]: pgmap v15871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:38 smithi081 ceph-mon[31453]: pgmap v15871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:39 smithi153 ceph-mon[36569]: pgmap v15872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:39 smithi104 ceph-mon[36624]: pgmap v15872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:39 smithi081 ceph-mon[31453]: pgmap v15872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:42 smithi153 ceph-mon[36569]: pgmap v15873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:42 smithi104 ceph-mon[36624]: pgmap v15873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:42 smithi081 ceph-mon[31453]: pgmap v15873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:44 smithi153 ceph-mon[36569]: pgmap v15874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:44 smithi104 ceph-mon[36624]: pgmap v15874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:44.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:44 smithi081 ceph-mon[31453]: pgmap v15874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:45.273+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:29:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:45.273+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:29:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:29:45.273+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:29:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:46 smithi153 ceph-mon[36569]: pgmap v15875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:46.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:46 smithi104 ceph-mon[36624]: pgmap v15875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:46 smithi081 ceph-mon[31453]: pgmap v15875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:48 smithi153 ceph-mon[36569]: pgmap v15876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:48.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:48 smithi104 ceph-mon[36624]: pgmap v15876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:48 smithi081 ceph-mon[31453]: pgmap v15876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:50 smithi153 ceph-mon[36569]: pgmap v15877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:50 smithi104 ceph-mon[36624]: pgmap v15877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:50 smithi081 ceph-mon[31453]: pgmap v15877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:52 smithi153 ceph-mon[36569]: pgmap v15878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:52 smithi104 ceph-mon[36624]: pgmap v15878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:52 smithi081 ceph-mon[31453]: pgmap v15878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:54.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:54 smithi153 ceph-mon[36569]: pgmap v15879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:54.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:54 smithi104 ceph-mon[36624]: pgmap v15879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:54.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:54 smithi081 ceph-mon[31453]: pgmap v15879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:56 smithi153 ceph-mon[36569]: pgmap v15880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:56.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:56 smithi104 ceph-mon[36624]: pgmap v15880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:56 smithi081 ceph-mon[31453]: pgmap v15880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:29:58 smithi153 ceph-mon[36569]: pgmap v15881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:29:58 smithi104 ceph-mon[36624]: pgmap v15881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:29:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:29:58 smithi081 ceph-mon[31453]: pgmap v15881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:00.274+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:30:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:00.274+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:30:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:00.274+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:30:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:00 smithi153 ceph-mon[36569]: pgmap v15882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:00.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:30:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:00 smithi104 ceph-mon[36624]: pgmap v15882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:30:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:00 smithi081 ceph-mon[31453]: pgmap v15882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:30:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:02 smithi153 ceph-mon[36569]: pgmap v15883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:02 smithi104 ceph-mon[36624]: pgmap v15883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:02.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:02 smithi081 ceph-mon[31453]: pgmap v15883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:04 smithi153 ceph-mon[36569]: pgmap v15884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:04.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:04 smithi104 ceph-mon[36624]: pgmap v15884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:04.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:04 smithi081 ceph-mon[31453]: pgmap v15884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:06 smithi153 ceph-mon[36569]: pgmap v15885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:06 smithi104 ceph-mon[36624]: pgmap v15885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:06 smithi081 ceph-mon[31453]: pgmap v15885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:08 smithi153 ceph-mon[36569]: pgmap v15886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:08 smithi104 ceph-mon[36624]: pgmap v15886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:08 smithi081 ceph-mon[31453]: pgmap v15886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:10 smithi153 ceph-mon[36569]: pgmap v15887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:10.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:10 smithi104 ceph-mon[36624]: pgmap v15887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:10.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:10 smithi081 ceph-mon[31453]: pgmap v15887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:12 smithi153 ceph-mon[36569]: pgmap v15888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:12.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:12 smithi104 ceph-mon[36624]: pgmap v15888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:12.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:12 smithi081 ceph-mon[31453]: pgmap v15888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:14 smithi153 ceph-mon[36569]: pgmap v15889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:14.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:14 smithi104 ceph-mon[36624]: pgmap v15889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:14 smithi081 ceph-mon[31453]: pgmap v15889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:15.291 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:15.289+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:30:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:15.289+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:30:15.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:15.289+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:30:16.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:16 smithi153 ceph-mon[36569]: pgmap v15890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:16.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:16 smithi104 ceph-mon[36624]: pgmap v15890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:16.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:16 smithi081 ceph-mon[31453]: pgmap v15890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:18 smithi153 ceph-mon[36569]: pgmap v15891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:18.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:18 smithi104 ceph-mon[36624]: pgmap v15891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:18 smithi081 ceph-mon[31453]: pgmap v15891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:20.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:20 smithi153 ceph-mon[36569]: pgmap v15892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:20.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:20 smithi104 ceph-mon[36624]: pgmap v15892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:20.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:20 smithi081 ceph-mon[31453]: pgmap v15892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:22 smithi153 ceph-mon[36569]: pgmap v15893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:22.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:22 smithi104 ceph-mon[36624]: pgmap v15893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:22.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:22 smithi081 ceph-mon[31453]: pgmap v15893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:24 smithi153 ceph-mon[36569]: pgmap v15894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:24.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:24 smithi104 ceph-mon[36624]: pgmap v15894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:24.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:24 smithi081 ceph-mon[31453]: pgmap v15894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:26 smithi153 ceph-mon[36569]: pgmap v15895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:26.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:26 smithi104 ceph-mon[36624]: pgmap v15895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:26.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:26 smithi081 ceph-mon[31453]: pgmap v15895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:28 smithi153 ceph-mon[36569]: pgmap v15896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:28.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:28 smithi104 ceph-mon[36624]: pgmap v15896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:28.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:28 smithi081 ceph-mon[31453]: pgmap v15896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:30.305+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:30:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:30.305+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:30:30.307 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:30.305+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:30:30.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:30 smithi104 ceph-mon[36624]: pgmap v15897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:30.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:30 smithi081 ceph-mon[31453]: pgmap v15897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:30.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:30 smithi153 ceph-mon[36569]: pgmap v15897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:32.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:32 smithi104 ceph-mon[36624]: pgmap v15898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:32.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:30:32.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:32 smithi081 ceph-mon[31453]: pgmap v15898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:32.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:30:32.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:32 smithi153 ceph-mon[36569]: pgmap v15898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:32.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:30:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:30:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:30:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:33 smithi104 ceph-mon[36624]: pgmap v15899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:33 smithi104 ceph-mon[36624]: pgmap v15900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:30:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:30:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:30:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:33 smithi081 ceph-mon[31453]: pgmap v15899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:33 smithi081 ceph-mon[31453]: pgmap v15900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:30:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:30:33.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:30:33.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:33 smithi153 ceph-mon[36569]: pgmap v15899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:33.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:33 smithi153 ceph-mon[36569]: pgmap v15900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:33.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:30:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:34 smithi153 ceph-mon[36569]: pgmap v15901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:34 smithi104 ceph-mon[36624]: pgmap v15901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:34 smithi081 ceph-mon[31453]: pgmap v15901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:36 smithi153 ceph-mon[36569]: pgmap v15902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:36 smithi104 ceph-mon[36624]: pgmap v15902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:36 smithi081 ceph-mon[31453]: pgmap v15902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:38 smithi153 ceph-mon[36569]: pgmap v15903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:38 smithi104 ceph-mon[36624]: pgmap v15903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:38 smithi081 ceph-mon[31453]: pgmap v15903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:40 smithi153 ceph-mon[36569]: pgmap v15904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:40 smithi104 ceph-mon[36624]: pgmap v15904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:40 smithi081 ceph-mon[31453]: pgmap v15904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:42 smithi153 ceph-mon[36569]: pgmap v15905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:43.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:42 smithi104 ceph-mon[36624]: pgmap v15905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:42 smithi081 ceph-mon[31453]: pgmap v15905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:45.025 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:44 smithi081 ceph-mon[31453]: pgmap v15906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:44 smithi153 ceph-mon[36569]: pgmap v15906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:44 smithi104 ceph-mon[36624]: pgmap v15906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:45.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:45.317+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:30:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:45.318+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:30:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:30:45.318+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:30:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:46 smithi153 ceph-mon[36569]: pgmap v15907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:47.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:46 smithi104 ceph-mon[36624]: pgmap v15907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:46 smithi081 ceph-mon[31453]: pgmap v15907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:48 smithi153 ceph-mon[36569]: pgmap v15908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:48 smithi104 ceph-mon[36624]: pgmap v15908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:48 smithi081 ceph-mon[31453]: pgmap v15908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:50 smithi153 ceph-mon[36569]: pgmap v15909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:50 smithi104 ceph-mon[36624]: pgmap v15909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:50 smithi081 ceph-mon[31453]: pgmap v15909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:52 smithi153 ceph-mon[36569]: pgmap v15910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:52 smithi104 ceph-mon[36624]: pgmap v15910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:52 smithi081 ceph-mon[31453]: pgmap v15910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:54 smithi153 ceph-mon[36569]: pgmap v15911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:54 smithi104 ceph-mon[36624]: pgmap v15911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:54 smithi081 ceph-mon[31453]: pgmap v15911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:57.033 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:56 smithi081 ceph-mon[31453]: pgmap v15912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:56 smithi153 ceph-mon[36569]: pgmap v15912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:56 smithi104 ceph-mon[36624]: pgmap v15912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:30:58 smithi153 ceph-mon[36569]: pgmap v15913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:30:58 smithi104 ceph-mon[36624]: pgmap v15913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:30:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:30:58 smithi081 ceph-mon[31453]: pgmap v15913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:00.333+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:31:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:00.333+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:31:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:00.333+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:31:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:00 smithi153 ceph-mon[36569]: pgmap v15914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:00 smithi104 ceph-mon[36624]: pgmap v15914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:00 smithi081 ceph-mon[31453]: pgmap v15914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:02 smithi153 ceph-mon[36569]: pgmap v15915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:02 smithi104 ceph-mon[36624]: pgmap v15915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:02 smithi081 ceph-mon[31453]: pgmap v15915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:04 smithi153 ceph-mon[36569]: pgmap v15916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:04 smithi104 ceph-mon[36624]: pgmap v15916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:04 smithi081 ceph-mon[31453]: pgmap v15916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:06 smithi153 ceph-mon[36569]: pgmap v15917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:06 smithi104 ceph-mon[36624]: pgmap v15917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:06 smithi081 ceph-mon[31453]: pgmap v15917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:08 smithi153 ceph-mon[36569]: pgmap v15918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:08 smithi104 ceph-mon[36624]: pgmap v15918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:08 smithi081 ceph-mon[31453]: pgmap v15918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:10 smithi153 ceph-mon[36569]: pgmap v15919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:10 smithi104 ceph-mon[36624]: pgmap v15919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:10 smithi081 ceph-mon[31453]: pgmap v15919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:12.171+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 msgr2=0x7f16e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000540) for more than 900000000 us, fault. 2024-04-13T05:31:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:12.171+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c9f0 tx=0x7f16f4032000 comp rx=0 tx=0).stop 2024-04-13T05:31:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:12.171+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 msgr2=0x7f16e80a4d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T05:31:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:12.171+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b39a0 0x7f16e80a4d40 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T05:31:12.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:12.171+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T05:31:12.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:12.171+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T05:31:12.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:12.172+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80a5280 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f16d40065b0 tx=0x7f16f404c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T05:31:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:12 smithi153 ceph-mon[36569]: pgmap v15920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:12 smithi104 ceph-mon[36624]: pgmap v15920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:12 smithi081 ceph-mon[31453]: pgmap v15920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:14 smithi153 ceph-mon[36569]: pgmap v15921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:14 smithi104 ceph-mon[36624]: pgmap v15921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:14 smithi081 ceph-mon[31453]: pgmap v15921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:15.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:15.339+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:31:15.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:15.339+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:31:15.342 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:15.340+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:31:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:16 smithi153 ceph-mon[36569]: pgmap v15922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:16 smithi104 ceph-mon[36624]: pgmap v15922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:16 smithi081 ceph-mon[31453]: pgmap v15922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:17.613+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:31:17.615 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:17.613+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T05:31:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:17.614+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 229 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T05:31:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:17.614+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:31:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:17.614+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 229 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T05:31:17.616 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:17.614+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T05:31:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:17.614+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 230 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:31:17.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:17.615+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 230 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003bc0 con 0x7f1704076980 2024-04-13T05:31:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:18 smithi153 ceph-mon[36569]: pgmap v15923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:18 smithi104 ceph-mon[36624]: pgmap v15923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:18 smithi081 ceph-mon[31453]: pgmap v15923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:20 smithi153 ceph-mon[36569]: pgmap v15924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:20 smithi104 ceph-mon[36624]: pgmap v15924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:20 smithi081 ceph-mon[31453]: pgmap v15924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:22 smithi153 ceph-mon[36569]: pgmap v15925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:22 smithi104 ceph-mon[36624]: pgmap v15925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:22 smithi081 ceph-mon[31453]: pgmap v15925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:24 smithi153 ceph-mon[36569]: pgmap v15926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:25.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:24 smithi104 ceph-mon[36624]: pgmap v15926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:24 smithi081 ceph-mon[31453]: pgmap v15926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:26 smithi153 ceph-mon[36569]: pgmap v15927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:26 smithi104 ceph-mon[36624]: pgmap v15927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:26 smithi081 ceph-mon[31453]: pgmap v15927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:28 smithi153 ceph-mon[36569]: pgmap v15928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:28 smithi104 ceph-mon[36624]: pgmap v15928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:28 smithi081 ceph-mon[31453]: pgmap v15928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:30.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:30.342+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:31:30.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:30.342+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:31:30.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:30.342+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:31:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:30 smithi153 ceph-mon[36569]: pgmap v15929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:30 smithi104 ceph-mon[36624]: pgmap v15929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:30 smithi081 ceph-mon[31453]: pgmap v15929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:32 smithi153 ceph-mon[36569]: pgmap v15930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:32 smithi104 ceph-mon[36624]: pgmap v15930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:32 smithi081 ceph-mon[31453]: pgmap v15930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:34.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:31:34.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:31:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:31:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:34.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: pgmap v15931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: pgmap v15932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:35.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:31:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:31:35.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:31:35.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:31:35.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: pgmap v15931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:35.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: pgmap v15932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:35.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:31:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: pgmap v15931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: pgmap v15932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:31:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:37 smithi081 ceph-mon[31453]: pgmap v15933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:37.761 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:37 smithi153 ceph-mon[36569]: pgmap v15933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:37 smithi104 ceph-mon[36624]: pgmap v15933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:39 smithi081 ceph-mon[31453]: pgmap v15934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:39 smithi153 ceph-mon[36569]: pgmap v15934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:39 smithi104 ceph-mon[36624]: pgmap v15934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:41.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:41 smithi153 ceph-mon[36569]: pgmap v15935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:41.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:41 smithi104 ceph-mon[36624]: pgmap v15935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:41 smithi081 ceph-mon[31453]: pgmap v15935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:43 smithi153 ceph-mon[36569]: pgmap v15936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:43 smithi104 ceph-mon[36624]: pgmap v15936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:43.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:43 smithi081 ceph-mon[31453]: pgmap v15936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:45.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:45.344+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:31:45.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:45.344+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:31:45.346 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:31:45.344+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:31:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:45 smithi153 ceph-mon[36569]: pgmap v15937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:45 smithi104 ceph-mon[36624]: pgmap v15937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:45 smithi081 ceph-mon[31453]: pgmap v15937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:47 smithi153 ceph-mon[36569]: pgmap v15938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:47 smithi104 ceph-mon[36624]: pgmap v15938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:47 smithi081 ceph-mon[31453]: pgmap v15938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:49 smithi153 ceph-mon[36569]: pgmap v15939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:49 smithi104 ceph-mon[36624]: pgmap v15939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:49 smithi081 ceph-mon[31453]: pgmap v15939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:51 smithi153 ceph-mon[36569]: pgmap v15940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:51 smithi104 ceph-mon[36624]: pgmap v15940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:51.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:51 smithi081 ceph-mon[31453]: pgmap v15940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:53 smithi153 ceph-mon[36569]: pgmap v15941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:53.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:53 smithi104 ceph-mon[36624]: pgmap v15941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:53 smithi081 ceph-mon[31453]: pgmap v15941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:55 smithi153 ceph-mon[36569]: pgmap v15942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:55 smithi104 ceph-mon[36624]: pgmap v15942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:55 smithi081 ceph-mon[31453]: pgmap v15942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:57 smithi153 ceph-mon[36569]: pgmap v15943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:57 smithi104 ceph-mon[36624]: pgmap v15943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:57 smithi081 ceph-mon[31453]: pgmap v15943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:31:59 smithi153 ceph-mon[36569]: pgmap v15944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:31:59 smithi104 ceph-mon[36624]: pgmap v15944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:31:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:31:59 smithi081 ceph-mon[31453]: pgmap v15944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:00.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:00.359+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:32:00.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:00.359+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:32:00.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:00.359+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:32:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:01 smithi153 ceph-mon[36569]: pgmap v15945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:01.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:01 smithi104 ceph-mon[36624]: pgmap v15945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:01 smithi081 ceph-mon[31453]: pgmap v15945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:03 smithi153 ceph-mon[36569]: pgmap v15946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:03.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:03 smithi104 ceph-mon[36624]: pgmap v15946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:03 smithi081 ceph-mon[31453]: pgmap v15946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:05 smithi153 ceph-mon[36569]: pgmap v15947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:05 smithi104 ceph-mon[36624]: pgmap v15947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:05.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:05 smithi081 ceph-mon[31453]: pgmap v15947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:07 smithi153 ceph-mon[36569]: pgmap v15948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:07.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:07 smithi104 ceph-mon[36624]: pgmap v15948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:07.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:07 smithi081 ceph-mon[31453]: pgmap v15948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:09 smithi153 ceph-mon[36569]: pgmap v15949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:09.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:09 smithi104 ceph-mon[36624]: pgmap v15949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:09.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:09 smithi081 ceph-mon[31453]: pgmap v15949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:11.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:11 smithi153 ceph-mon[36569]: pgmap v15950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:11 smithi104 ceph-mon[36624]: pgmap v15950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:11 smithi081 ceph-mon[31453]: pgmap v15950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:13 smithi153 ceph-mon[36569]: pgmap v15951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:13 smithi104 ceph-mon[36624]: pgmap v15951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:13 smithi081 ceph-mon[31453]: pgmap v15951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:15.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:15.374+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:32:15.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:15.374+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:32:15.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:15.374+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:32:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:15 smithi153 ceph-mon[36569]: pgmap v15952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:15 smithi104 ceph-mon[36624]: pgmap v15952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:15 smithi081 ceph-mon[31453]: pgmap v15952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:17 smithi153 ceph-mon[36569]: pgmap v15953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:17.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:17 smithi104 ceph-mon[36624]: pgmap v15953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:17.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:17 smithi081 ceph-mon[31453]: pgmap v15953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:19 smithi153 ceph-mon[36569]: pgmap v15954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:19 smithi104 ceph-mon[36624]: pgmap v15954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:19 smithi081 ceph-mon[31453]: pgmap v15954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:21 smithi153 ceph-mon[36569]: pgmap v15955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:21.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:21 smithi104 ceph-mon[36624]: pgmap v15955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:21 smithi081 ceph-mon[31453]: pgmap v15955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:23 smithi153 ceph-mon[36569]: pgmap v15956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:23.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:23 smithi104 ceph-mon[36624]: pgmap v15956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:23 smithi081 ceph-mon[31453]: pgmap v15956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:25 smithi153 ceph-mon[36569]: pgmap v15957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:25 smithi104 ceph-mon[36624]: pgmap v15957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:25 smithi081 ceph-mon[31453]: pgmap v15957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:27 smithi153 ceph-mon[36569]: pgmap v15958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:27 smithi104 ceph-mon[36624]: pgmap v15958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:27 smithi081 ceph-mon[31453]: pgmap v15958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:29.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:29 smithi081 ceph-mon[31453]: pgmap v15959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:29 smithi153 ceph-mon[36569]: pgmap v15959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:29 smithi104 ceph-mon[36624]: pgmap v15959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:30.391 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:30.389+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:32:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:30.389+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:32:30.392 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:30.389+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:32:31.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:31 smithi081 ceph-mon[31453]: pgmap v15960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:31 smithi153 ceph-mon[36569]: pgmap v15960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:31 smithi104 ceph-mon[36624]: pgmap v15960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:33 smithi081 ceph-mon[31453]: pgmap v15961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:33 smithi153 ceph-mon[36569]: pgmap v15961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:33 smithi104 ceph-mon[36624]: pgmap v15961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:35 smithi081 ceph-mon[31453]: pgmap v15962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:32:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:35 smithi153 ceph-mon[36569]: pgmap v15962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:35.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:32:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:35 smithi104 ceph-mon[36624]: pgmap v15962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:32:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:32:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:32:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:32:36.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:32:36.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:32:36.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:32:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:32:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:32:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:32:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:37 smithi153 ceph-mon[36569]: pgmap v15963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:37 smithi153 ceph-mon[36569]: pgmap v15964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:37 smithi104 ceph-mon[36624]: pgmap v15963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:37 smithi104 ceph-mon[36624]: pgmap v15964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:37 smithi081 ceph-mon[31453]: pgmap v15963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:37 smithi081 ceph-mon[31453]: pgmap v15964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:39 smithi153 ceph-mon[36569]: pgmap v15965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:39 smithi104 ceph-mon[36624]: pgmap v15965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:39.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:39 smithi081 ceph-mon[31453]: pgmap v15965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:41.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:41 smithi153 ceph-mon[36569]: pgmap v15966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:41.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:41 smithi104 ceph-mon[36624]: pgmap v15966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:41 smithi081 ceph-mon[31453]: pgmap v15966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:43.711 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:43 smithi081 ceph-mon[31453]: pgmap v15967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:43 smithi153 ceph-mon[36569]: pgmap v15967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:43 smithi104 ceph-mon[36624]: pgmap v15967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:45.402+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:32:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:45.402+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:32:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:32:45.402+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:32:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:45 smithi153 ceph-mon[36569]: pgmap v15968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:45 smithi104 ceph-mon[36624]: pgmap v15968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:45 smithi081 ceph-mon[31453]: pgmap v15968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:47 smithi153 ceph-mon[36569]: pgmap v15969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:47 smithi104 ceph-mon[36624]: pgmap v15969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:47 smithi081 ceph-mon[31453]: pgmap v15969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:49 smithi153 ceph-mon[36569]: pgmap v15970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:49 smithi104 ceph-mon[36624]: pgmap v15970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:49 smithi081 ceph-mon[31453]: pgmap v15970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:51 smithi153 ceph-mon[36569]: pgmap v15971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:51 smithi104 ceph-mon[36624]: pgmap v15971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:51.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:51 smithi081 ceph-mon[31453]: pgmap v15971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:53 smithi153 ceph-mon[36569]: pgmap v15972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:53.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:53 smithi104 ceph-mon[36624]: pgmap v15972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:53 smithi081 ceph-mon[31453]: pgmap v15972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:55 smithi153 ceph-mon[36569]: pgmap v15973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:55 smithi104 ceph-mon[36624]: pgmap v15973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:55 smithi081 ceph-mon[31453]: pgmap v15973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:57.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:57 smithi153 ceph-mon[36569]: pgmap v15974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:57 smithi104 ceph-mon[36624]: pgmap v15974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:57 smithi081 ceph-mon[31453]: pgmap v15974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:32:59 smithi153 ceph-mon[36569]: pgmap v15975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:32:59 smithi104 ceph-mon[36624]: pgmap v15975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:32:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:32:59 smithi081 ceph-mon[31453]: pgmap v15975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:00.416+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:33:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:00.416+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:33:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:00.416+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:33:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:01 smithi153 ceph-mon[36569]: pgmap v15976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:01.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:01 smithi104 ceph-mon[36624]: pgmap v15976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:01 smithi081 ceph-mon[31453]: pgmap v15976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:03 smithi153 ceph-mon[36569]: pgmap v15977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:03.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:03 smithi104 ceph-mon[36624]: pgmap v15977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:03 smithi081 ceph-mon[31453]: pgmap v15977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:05.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:05 smithi153 ceph-mon[36569]: pgmap v15978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:05 smithi104 ceph-mon[36624]: pgmap v15978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:05.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:05 smithi081 ceph-mon[31453]: pgmap v15978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:07 smithi153 ceph-mon[36569]: pgmap v15979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:07.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:07 smithi104 ceph-mon[36624]: pgmap v15979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:07.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:07 smithi081 ceph-mon[31453]: pgmap v15979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:09 smithi153 ceph-mon[36569]: pgmap v15980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:09.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:09 smithi104 ceph-mon[36624]: pgmap v15980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:09.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:09 smithi081 ceph-mon[31453]: pgmap v15980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:11 smithi153 ceph-mon[36569]: pgmap v15981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:11 smithi104 ceph-mon[36624]: pgmap v15981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:11 smithi081 ceph-mon[31453]: pgmap v15981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:13 smithi153 ceph-mon[36569]: pgmap v15982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:13 smithi104 ceph-mon[36624]: pgmap v15982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:13 smithi081 ceph-mon[31453]: pgmap v15982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:15.432+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:33:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:15.432+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:33:15.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:15.432+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:33:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:15 smithi153 ceph-mon[36569]: pgmap v15983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:15 smithi104 ceph-mon[36624]: pgmap v15983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:15 smithi081 ceph-mon[31453]: pgmap v15983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:17.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:17 smithi081 ceph-mon[31453]: pgmap v15984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:17 smithi153 ceph-mon[36569]: pgmap v15984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:17.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:17 smithi104 ceph-mon[36624]: pgmap v15984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:19 smithi153 ceph-mon[36569]: pgmap v15985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:19 smithi104 ceph-mon[36624]: pgmap v15985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:19 smithi081 ceph-mon[31453]: pgmap v15985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:21 smithi153 ceph-mon[36569]: pgmap v15986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:21.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:21 smithi104 ceph-mon[36624]: pgmap v15986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:21 smithi081 ceph-mon[31453]: pgmap v15986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:23 smithi153 ceph-mon[36569]: pgmap v15987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:23.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:23 smithi104 ceph-mon[36624]: pgmap v15987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:23 smithi081 ceph-mon[31453]: pgmap v15987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:25 smithi153 ceph-mon[36569]: pgmap v15988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:25 smithi104 ceph-mon[36624]: pgmap v15988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:25 smithi081 ceph-mon[31453]: pgmap v15988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:27 smithi153 ceph-mon[36569]: pgmap v15989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:27 smithi104 ceph-mon[36624]: pgmap v15989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:27 smithi081 ceph-mon[31453]: pgmap v15989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:29 smithi153 ceph-mon[36569]: pgmap v15990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:29 smithi104 ceph-mon[36624]: pgmap v15990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:29.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:29 smithi081 ceph-mon[31453]: pgmap v15990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:30.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:30.447+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:33:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:30.447+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:33:30.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:30.447+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:33:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:31 smithi153 ceph-mon[36569]: pgmap v15991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:31 smithi104 ceph-mon[36624]: pgmap v15991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:31 smithi081 ceph-mon[31453]: pgmap v15991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:33 smithi153 ceph-mon[36569]: pgmap v15992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:33 smithi104 ceph-mon[36624]: pgmap v15992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:33.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:33 smithi081 ceph-mon[31453]: pgmap v15992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:35 smithi104 ceph-mon[36624]: pgmap v15993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:35.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:35 smithi081 ceph-mon[31453]: pgmap v15993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:36.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:35 smithi153 ceph-mon[36569]: pgmap v15993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:33:36.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:33:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:33:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:37 smithi104 ceph-mon[36624]: pgmap v15994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:33:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:33:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:33:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:37 smithi081 ceph-mon[31453]: pgmap v15994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:33:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:33:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:33:37.960 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:37 smithi153 ceph-mon[36569]: pgmap v15994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:37.960 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:33:37.960 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:33:37.960 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:33:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:38 smithi104 ceph-mon[36624]: pgmap v15995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:38 smithi104 ceph-mon[36624]: pgmap v15996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:38 smithi081 ceph-mon[31453]: pgmap v15995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:38 smithi081 ceph-mon[31453]: pgmap v15996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:38 smithi153 ceph-mon[36569]: pgmap v15995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:38 smithi153 ceph-mon[36569]: pgmap v15996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:40.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:40 smithi081 ceph-mon[31453]: pgmap v15997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:40 smithi153 ceph-mon[36569]: pgmap v15997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:40 smithi104 ceph-mon[36624]: pgmap v15997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:42 smithi081 ceph-mon[31453]: pgmap v15998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:43.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:42 smithi153 ceph-mon[36569]: pgmap v15998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:43.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:42 smithi104 ceph-mon[36624]: pgmap v15998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:44.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:44 smithi104 ceph-mon[36624]: pgmap v15999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:44.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:44 smithi081 ceph-mon[31453]: pgmap v15999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:44 smithi153 ceph-mon[36569]: pgmap v15999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:45.455 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:45.452+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:33:45.455 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:45.452+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:33:45.455 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:33:45.452+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:33:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:46 smithi081 ceph-mon[31453]: pgmap v16000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:46 smithi153 ceph-mon[36569]: pgmap v16000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:47.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:46 smithi104 ceph-mon[36624]: pgmap v16000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:48 smithi081 ceph-mon[31453]: pgmap v16001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:48 smithi153 ceph-mon[36569]: pgmap v16001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:48 smithi104 ceph-mon[36624]: pgmap v16001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:50 smithi081 ceph-mon[31453]: pgmap v16002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:50 smithi153 ceph-mon[36569]: pgmap v16002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:50 smithi104 ceph-mon[36624]: pgmap v16002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:52.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:52 smithi081 ceph-mon[31453]: pgmap v16003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:52 smithi153 ceph-mon[36569]: pgmap v16003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:52 smithi104 ceph-mon[36624]: pgmap v16003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:54 smithi081 ceph-mon[31453]: pgmap v16004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:54 smithi153 ceph-mon[36569]: pgmap v16004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:54 smithi104 ceph-mon[36624]: pgmap v16004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:56.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:56 smithi081 ceph-mon[31453]: pgmap v16005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:56 smithi153 ceph-mon[36569]: pgmap v16005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:56 smithi104 ceph-mon[36624]: pgmap v16005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:33:58 smithi081 ceph-mon[31453]: pgmap v16006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:33:58 smithi153 ceph-mon[36569]: pgmap v16006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:33:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:33:58 smithi104 ceph-mon[36624]: pgmap v16006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:00.461+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:34:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:00.461+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:34:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:00.461+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:34:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:00 smithi081 ceph-mon[31453]: pgmap v16007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:00 smithi153 ceph-mon[36569]: pgmap v16007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:00 smithi104 ceph-mon[36624]: pgmap v16007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:02 smithi081 ceph-mon[31453]: pgmap v16008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:02 smithi153 ceph-mon[36569]: pgmap v16008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:02 smithi104 ceph-mon[36624]: pgmap v16008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:04 smithi153 ceph-mon[36569]: pgmap v16009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:04 smithi104 ceph-mon[36624]: pgmap v16009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:04 smithi081 ceph-mon[31453]: pgmap v16009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:06 smithi153 ceph-mon[36569]: pgmap v16010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:06 smithi104 ceph-mon[36624]: pgmap v16010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:06 smithi081 ceph-mon[31453]: pgmap v16010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:08 smithi153 ceph-mon[36569]: pgmap v16011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:09.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:08 smithi104 ceph-mon[36624]: pgmap v16011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:08 smithi081 ceph-mon[31453]: pgmap v16011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:10 smithi153 ceph-mon[36569]: pgmap v16012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:10 smithi104 ceph-mon[36624]: pgmap v16012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:10 smithi081 ceph-mon[31453]: pgmap v16012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:12 smithi153 ceph-mon[36569]: pgmap v16013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:12 smithi104 ceph-mon[36624]: pgmap v16013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:12 smithi081 ceph-mon[31453]: pgmap v16013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:14.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:14 smithi081 ceph-mon[31453]: pgmap v16014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:15.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:14 smithi153 ceph-mon[36569]: pgmap v16014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:14 smithi104 ceph-mon[36624]: pgmap v16014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:15.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:15.474+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:34:15.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:15.474+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:34:15.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:15.474+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:34:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:16 smithi153 ceph-mon[36569]: pgmap v16015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:16 smithi104 ceph-mon[36624]: pgmap v16015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:16 smithi081 ceph-mon[31453]: pgmap v16015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:17.642+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:34:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:17.642+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:34:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:17.643+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 231 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:34:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:17.643+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 231 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:34:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:18 smithi153 ceph-mon[36569]: pgmap v16016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:18 smithi104 ceph-mon[36624]: pgmap v16016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:18 smithi081 ceph-mon[31453]: pgmap v16016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:20 smithi153 ceph-mon[36569]: pgmap v16017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:20 smithi104 ceph-mon[36624]: pgmap v16017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:20 smithi081 ceph-mon[31453]: pgmap v16017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:23.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:22 smithi153 ceph-mon[36569]: pgmap v16018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:22 smithi104 ceph-mon[36624]: pgmap v16018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:22 smithi081 ceph-mon[31453]: pgmap v16018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:24 smithi153 ceph-mon[36569]: pgmap v16019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:25.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:24 smithi104 ceph-mon[36624]: pgmap v16019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:24 smithi081 ceph-mon[31453]: pgmap v16019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:26 smithi153 ceph-mon[36569]: pgmap v16020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:26 smithi104 ceph-mon[36624]: pgmap v16020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:26 smithi081 ceph-mon[31453]: pgmap v16020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:29.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:28 smithi153 ceph-mon[36569]: pgmap v16021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:28 smithi104 ceph-mon[36624]: pgmap v16021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:28 smithi081 ceph-mon[31453]: pgmap v16021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:30.482+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:30.482+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:34:30.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:30.482+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:34:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:30 smithi153 ceph-mon[36569]: pgmap v16022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:30 smithi104 ceph-mon[36624]: pgmap v16022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:30 smithi081 ceph-mon[31453]: pgmap v16022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:32 smithi153 ceph-mon[36569]: pgmap v16023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:32 smithi104 ceph-mon[36624]: pgmap v16023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:32 smithi081 ceph-mon[31453]: pgmap v16023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:35.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:34 smithi153 ceph-mon[36569]: pgmap v16024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:34 smithi104 ceph-mon[36624]: pgmap v16024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:35.145 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:34 smithi081 ceph-mon[31453]: pgmap v16024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:36 smithi153 ceph-mon[36569]: pgmap v16025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:36 smithi104 ceph-mon[36624]: pgmap v16025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:36 smithi081 ceph-mon[31453]: pgmap v16025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:38.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:34:38.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:34:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:34:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:38 smithi153 ceph-mon[36569]: pgmap v16026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:38 smithi104 ceph-mon[36624]: pgmap v16026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:38 smithi081 ceph-mon[31453]: pgmap v16026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:40.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:34:40.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:34:40.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:39 smithi153 ceph-mon[36569]: pgmap v16027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:40.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:39 smithi153 ceph-mon[36569]: pgmap v16028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:40.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:34:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:34:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:34:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:39 smithi104 ceph-mon[36624]: pgmap v16027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:39 smithi104 ceph-mon[36624]: pgmap v16028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:34:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:34:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:34:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:39 smithi081 ceph-mon[31453]: pgmap v16027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:39 smithi081 ceph-mon[31453]: pgmap v16028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:34:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:42 smithi153 ceph-mon[36569]: pgmap v16029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:42 smithi104 ceph-mon[36624]: pgmap v16029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:42 smithi081 ceph-mon[31453]: pgmap v16029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:44 smithi153 ceph-mon[36569]: pgmap v16030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:44 smithi104 ceph-mon[36624]: pgmap v16030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:44 smithi081 ceph-mon[31453]: pgmap v16030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:45.494+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:34:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:45.494+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:34:45.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:34:45.494+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:34:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:46 smithi153 ceph-mon[36569]: pgmap v16031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:46 smithi104 ceph-mon[36624]: pgmap v16031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:46 smithi081 ceph-mon[31453]: pgmap v16031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:48 smithi153 ceph-mon[36569]: pgmap v16032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:48 smithi104 ceph-mon[36624]: pgmap v16032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:48 smithi081 ceph-mon[31453]: pgmap v16032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:50 smithi104 ceph-mon[36624]: pgmap v16033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:50 smithi081 ceph-mon[31453]: pgmap v16033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:50 smithi153 ceph-mon[36569]: pgmap v16033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:52 smithi104 ceph-mon[36624]: pgmap v16034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:52 smithi081 ceph-mon[31453]: pgmap v16034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:52 smithi153 ceph-mon[36569]: pgmap v16034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:54 smithi104 ceph-mon[36624]: pgmap v16035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:54 smithi081 ceph-mon[31453]: pgmap v16035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:54.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:54 smithi153 ceph-mon[36569]: pgmap v16035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:56 smithi104 ceph-mon[36624]: pgmap v16036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:56 smithi081 ceph-mon[31453]: pgmap v16036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:56 smithi153 ceph-mon[36569]: pgmap v16036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:34:58 smithi104 ceph-mon[36624]: pgmap v16037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:34:58 smithi081 ceph-mon[31453]: pgmap v16037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:34:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:34:58 smithi153 ceph-mon[36569]: pgmap v16037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:00 smithi104 ceph-mon[36624]: pgmap v16038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:00 smithi081 ceph-mon[31453]: pgmap v16038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:00.503+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:35:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:00.503+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:35:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:00.503+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:35:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:00 smithi153 ceph-mon[36569]: pgmap v16038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:02 smithi081 ceph-mon[31453]: pgmap v16039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:02 smithi153 ceph-mon[36569]: pgmap v16039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:02 smithi104 ceph-mon[36624]: pgmap v16039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:04 smithi081 ceph-mon[31453]: pgmap v16040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:04 smithi153 ceph-mon[36569]: pgmap v16040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:04.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:04 smithi104 ceph-mon[36624]: pgmap v16040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:06.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:06 smithi081 ceph-mon[31453]: pgmap v16041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:06 smithi153 ceph-mon[36569]: pgmap v16041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:06 smithi104 ceph-mon[36624]: pgmap v16041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:08.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:08 smithi153 ceph-mon[36569]: pgmap v16042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:08 smithi081 ceph-mon[31453]: pgmap v16042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:08.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:08 smithi104 ceph-mon[36624]: pgmap v16042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:10 smithi081 ceph-mon[31453]: pgmap v16043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:10 smithi153 ceph-mon[36569]: pgmap v16043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:10.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:10 smithi104 ceph-mon[36624]: pgmap v16043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:12 smithi081 ceph-mon[31453]: pgmap v16044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:12 smithi153 ceph-mon[36569]: pgmap v16044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:12.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:12 smithi104 ceph-mon[36624]: pgmap v16044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:14.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:14 smithi104 ceph-mon[36624]: pgmap v16045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:14 smithi081 ceph-mon[31453]: pgmap v16045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:14 smithi153 ceph-mon[36569]: pgmap v16045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:15.504+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:35:15.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:15.504+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:35:15.507 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:15.504+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:35:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:16 smithi081 ceph-mon[31453]: pgmap v16046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:16 smithi153 ceph-mon[36569]: pgmap v16046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:16.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:16 smithi104 ceph-mon[36624]: pgmap v16046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:18.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:18 smithi081 ceph-mon[31453]: pgmap v16047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:18 smithi153 ceph-mon[36569]: pgmap v16047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:18.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:18 smithi104 ceph-mon[36624]: pgmap v16047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:20.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:20 smithi081 ceph-mon[31453]: pgmap v16048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:20.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:20 smithi153 ceph-mon[36569]: pgmap v16048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:20.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:20 smithi104 ceph-mon[36624]: pgmap v16048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:22 smithi081 ceph-mon[31453]: pgmap v16049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:22.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:22 smithi153 ceph-mon[36569]: pgmap v16049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:22.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:22 smithi104 ceph-mon[36624]: pgmap v16049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:24 smithi081 ceph-mon[31453]: pgmap v16050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:24 smithi153 ceph-mon[36569]: pgmap v16050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:24.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:24 smithi104 ceph-mon[36624]: pgmap v16050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:26 smithi081 ceph-mon[31453]: pgmap v16051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:26 smithi153 ceph-mon[36569]: pgmap v16051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:26.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:26 smithi104 ceph-mon[36624]: pgmap v16051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:28.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:28 smithi081 ceph-mon[31453]: pgmap v16052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:28 smithi153 ceph-mon[36569]: pgmap v16052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:28.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:28 smithi104 ceph-mon[36624]: pgmap v16052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:30.519+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:35:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:30.519+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:35:30.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:30.519+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:35:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:30 smithi153 ceph-mon[36569]: pgmap v16053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:30.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:30 smithi104 ceph-mon[36624]: pgmap v16053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:30.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:30 smithi081 ceph-mon[31453]: pgmap v16053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:32 smithi153 ceph-mon[36569]: pgmap v16054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:32.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:32 smithi104 ceph-mon[36624]: pgmap v16054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:32.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:32 smithi081 ceph-mon[31453]: pgmap v16054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:34 smithi153 ceph-mon[36569]: pgmap v16055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:34.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:34 smithi104 ceph-mon[36624]: pgmap v16055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:34.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:34 smithi081 ceph-mon[31453]: pgmap v16055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:36 smithi153 ceph-mon[36569]: pgmap v16056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:36.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:36 smithi104 ceph-mon[36624]: pgmap v16056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:36 smithi081 ceph-mon[31453]: pgmap v16056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:38 smithi153 ceph-mon[36569]: pgmap v16057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:38.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:38 smithi104 ceph-mon[36624]: pgmap v16057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:38 smithi081 ceph-mon[31453]: pgmap v16057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:40 smithi153 ceph-mon[36569]: pgmap v16058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:40.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:35:40.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:40.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:40.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:40 smithi104 ceph-mon[36624]: pgmap v16058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:35:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:40.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:40 smithi081 ceph-mon[31453]: pgmap v16058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:40.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:35:40.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:40.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:40.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:41.534 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:41.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:42 smithi153 ceph-mon[36569]: pgmap v16059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:42.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:42.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:42 smithi104 ceph-mon[36624]: pgmap v16059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:42 smithi081 ceph-mon[31453]: pgmap v16059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:43.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:43.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:43.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:44 smithi153 ceph-mon[36569]: pgmap v16060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:44.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:44 smithi104 ceph-mon[36624]: pgmap v16060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:44.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:44 smithi081 ceph-mon[31453]: pgmap v16060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:45.537 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:45.535+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:35:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:45.535+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:35:45.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:35:45.535+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:35:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:45.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:46 smithi153 ceph-mon[36569]: pgmap v16061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:35:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:35:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:46 smithi153 ceph-mon[36569]: pgmap v16062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:46 smithi153 ceph-mon[36569]: pgmap v16063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:46 smithi104 ceph-mon[36624]: pgmap v16061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:35:46.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:35:46.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:46 smithi104 ceph-mon[36624]: pgmap v16062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:46 smithi104 ceph-mon[36624]: pgmap v16063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:46 smithi081 ceph-mon[31453]: pgmap v16061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:35:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:35:46.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:46 smithi081 ceph-mon[31453]: pgmap v16062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:46 smithi081 ceph-mon[31453]: pgmap v16063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:46.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:35:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:47 smithi104 ceph-mon[36624]: pgmap v16064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:47 smithi081 ceph-mon[31453]: pgmap v16064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:47 smithi153 ceph-mon[36569]: pgmap v16064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:50.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:49 smithi104 ceph-mon[36624]: pgmap v16065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:49 smithi081 ceph-mon[31453]: pgmap v16065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:49 smithi153 ceph-mon[36569]: pgmap v16065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:51 smithi104 ceph-mon[36624]: pgmap v16066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:51 smithi081 ceph-mon[31453]: pgmap v16066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:51 smithi153 ceph-mon[36569]: pgmap v16066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:53 smithi104 ceph-mon[36624]: pgmap v16067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:53 smithi081 ceph-mon[31453]: pgmap v16067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:53 smithi153 ceph-mon[36569]: pgmap v16067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:55 smithi104 ceph-mon[36624]: pgmap v16068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:55 smithi081 ceph-mon[31453]: pgmap v16068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:55 smithi153 ceph-mon[36569]: pgmap v16068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:57 smithi104 ceph-mon[36624]: pgmap v16069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:57 smithi081 ceph-mon[31453]: pgmap v16069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:35:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:57 smithi153 ceph-mon[36569]: pgmap v16069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:35:59 smithi104 ceph-mon[36624]: pgmap v16070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:35:59 smithi081 ceph-mon[31453]: pgmap v16070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:35:59 smithi153 ceph-mon[36569]: pgmap v16070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:00.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:00.551+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:36:00.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:00.551+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:36:00.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:00.551+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:36:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:01 smithi104 ceph-mon[36624]: pgmap v16071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:01 smithi081 ceph-mon[31453]: pgmap v16071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:01 smithi153 ceph-mon[36569]: pgmap v16071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:04.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:03 smithi104 ceph-mon[36624]: pgmap v16072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:03 smithi081 ceph-mon[31453]: pgmap v16072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:03 smithi153 ceph-mon[36569]: pgmap v16072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:05 smithi104 ceph-mon[36624]: pgmap v16073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:05 smithi081 ceph-mon[31453]: pgmap v16073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:05 smithi153 ceph-mon[36569]: pgmap v16073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:07 smithi104 ceph-mon[36624]: pgmap v16074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:08.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:07 smithi153 ceph-mon[36569]: pgmap v16074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:08.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:07 smithi081 ceph-mon[31453]: pgmap v16074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:10.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:09 smithi104 ceph-mon[36624]: pgmap v16075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:09 smithi081 ceph-mon[31453]: pgmap v16075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:10.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:09 smithi153 ceph-mon[36569]: pgmap v16075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:12.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:11 smithi104 ceph-mon[36624]: pgmap v16076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:12.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:11 smithi081 ceph-mon[31453]: pgmap v16076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:12.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:11 smithi153 ceph-mon[36569]: pgmap v16076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:13 smithi104 ceph-mon[36624]: pgmap v16077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:13 smithi081 ceph-mon[31453]: pgmap v16077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:13 smithi153 ceph-mon[36569]: pgmap v16077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:15.566+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:36:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:15.566+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:36:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:15.566+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:36:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:15 smithi104 ceph-mon[36624]: pgmap v16078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:15 smithi081 ceph-mon[31453]: pgmap v16078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:15 smithi153 ceph-mon[36569]: pgmap v16078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:17 smithi104 ceph-mon[36624]: pgmap v16079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:17 smithi081 ceph-mon[31453]: pgmap v16079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:18.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:17 smithi153 ceph-mon[36569]: pgmap v16079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:20 smithi104 ceph-mon[36624]: pgmap v16080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:20 smithi081 ceph-mon[31453]: pgmap v16080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:20 smithi153 ceph-mon[36569]: pgmap v16080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:22 smithi104 ceph-mon[36624]: pgmap v16081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:22 smithi081 ceph-mon[31453]: pgmap v16081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:22 smithi153 ceph-mon[36569]: pgmap v16081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:25.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:24 smithi104 ceph-mon[36624]: pgmap v16082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:24 smithi081 ceph-mon[31453]: pgmap v16082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:24 smithi153 ceph-mon[36569]: pgmap v16082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:26 smithi104 ceph-mon[36624]: pgmap v16083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:26 smithi081 ceph-mon[31453]: pgmap v16083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:26 smithi153 ceph-mon[36569]: pgmap v16083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:29 smithi153 ceph-mon[36569]: pgmap v16084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:29 smithi104 ceph-mon[36624]: pgmap v16084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:29.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:29 smithi081 ceph-mon[31453]: pgmap v16084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:30.582+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:36:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:30.582+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:36:30.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:30.582+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:36:31.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:31 smithi153 ceph-mon[36569]: pgmap v16085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:31 smithi104 ceph-mon[36624]: pgmap v16085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:31 smithi081 ceph-mon[31453]: pgmap v16085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:33 smithi153 ceph-mon[36569]: pgmap v16086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:33 smithi104 ceph-mon[36624]: pgmap v16086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:33 smithi081 ceph-mon[31453]: pgmap v16086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:35.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:35 smithi153 ceph-mon[36569]: pgmap v16087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:35 smithi104 ceph-mon[36624]: pgmap v16087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:35.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:35 smithi081 ceph-mon[31453]: pgmap v16087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:37.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:37 smithi153 ceph-mon[36569]: pgmap v16088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:37 smithi104 ceph-mon[36624]: pgmap v16088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:37 smithi081 ceph-mon[31453]: pgmap v16088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:39 smithi153 ceph-mon[36569]: pgmap v16089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:39 smithi104 ceph-mon[36624]: pgmap v16089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:39.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:39 smithi081 ceph-mon[31453]: pgmap v16089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:41.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:41 smithi153 ceph-mon[36569]: pgmap v16090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:41.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:41 smithi104 ceph-mon[36624]: pgmap v16090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:41 smithi081 ceph-mon[31453]: pgmap v16090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:43 smithi153 ceph-mon[36569]: pgmap v16091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:43 smithi104 ceph-mon[36624]: pgmap v16091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:43.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:43 smithi081 ceph-mon[31453]: pgmap v16091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:45.585+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:36:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:45.585+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:36:45.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:36:45.585+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:36:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:45 smithi153 ceph-mon[36569]: pgmap v16092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:45 smithi104 ceph-mon[36624]: pgmap v16092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:45 smithi081 ceph-mon[31453]: pgmap v16092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:36:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:36:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:36:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:47 smithi081 ceph-mon[31453]: pgmap v16093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:36:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:36:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:47 smithi153 ceph-mon[36569]: pgmap v16093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:36:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:36:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:47 smithi104 ceph-mon[36624]: pgmap v16093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:36:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:36:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:36:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:48 smithi081 ceph-mon[31453]: pgmap v16094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:48 smithi081 ceph-mon[31453]: pgmap v16095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:49.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:48 smithi153 ceph-mon[36569]: pgmap v16094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:49.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:48 smithi153 ceph-mon[36569]: pgmap v16095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:48 smithi104 ceph-mon[36624]: pgmap v16094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:48 smithi104 ceph-mon[36624]: pgmap v16095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:50 smithi081 ceph-mon[31453]: pgmap v16096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:51.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:50 smithi153 ceph-mon[36569]: pgmap v16096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:50 smithi104 ceph-mon[36624]: pgmap v16096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:52 smithi081 ceph-mon[31453]: pgmap v16097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:53.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:52 smithi153 ceph-mon[36569]: pgmap v16097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:52 smithi104 ceph-mon[36624]: pgmap v16097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:54 smithi081 ceph-mon[31453]: pgmap v16098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:55.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:54 smithi153 ceph-mon[36569]: pgmap v16098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:54 smithi104 ceph-mon[36624]: pgmap v16098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:56.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:56 smithi081 ceph-mon[31453]: pgmap v16099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:56 smithi153 ceph-mon[36569]: pgmap v16099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:56 smithi104 ceph-mon[36624]: pgmap v16099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:36:58 smithi081 ceph-mon[31453]: pgmap v16100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:36:58 smithi153 ceph-mon[36569]: pgmap v16100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:36:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:36:58 smithi104 ceph-mon[36624]: pgmap v16100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:00.602 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:00.600+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:37:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:00.600+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:37:00.603 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:00.600+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:37:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:00 smithi081 ceph-mon[31453]: pgmap v16101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:00 smithi153 ceph-mon[36569]: pgmap v16101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:00 smithi104 ceph-mon[36624]: pgmap v16101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:02 smithi081 ceph-mon[31453]: pgmap v16102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:02 smithi153 ceph-mon[36569]: pgmap v16102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:02 smithi104 ceph-mon[36624]: pgmap v16102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:04.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:04 smithi081 ceph-mon[31453]: pgmap v16103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:04 smithi153 ceph-mon[36569]: pgmap v16103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:05.102 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:04 smithi104 ceph-mon[36624]: pgmap v16103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:06 smithi081 ceph-mon[31453]: pgmap v16104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:06 smithi153 ceph-mon[36569]: pgmap v16104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:06 smithi104 ceph-mon[36624]: pgmap v16104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:08 smithi153 ceph-mon[36569]: pgmap v16105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:08 smithi104 ceph-mon[36624]: pgmap v16105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:08 smithi081 ceph-mon[31453]: pgmap v16105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:10 smithi153 ceph-mon[36569]: pgmap v16106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:10 smithi104 ceph-mon[36624]: pgmap v16106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:10 smithi081 ceph-mon[31453]: pgmap v16106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:12 smithi153 ceph-mon[36569]: pgmap v16107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:12 smithi104 ceph-mon[36624]: pgmap v16107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:12 smithi081 ceph-mon[31453]: pgmap v16107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:14 smithi153 ceph-mon[36569]: pgmap v16108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:14 smithi104 ceph-mon[36624]: pgmap v16108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:14 smithi081 ceph-mon[31453]: pgmap v16108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:15.616+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:37:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:15.616+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:37:15.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:15.616+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:37:16.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:16 smithi081 ceph-mon[31453]: pgmap v16109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:16 smithi153 ceph-mon[36569]: pgmap v16109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:16 smithi104 ceph-mon[36624]: pgmap v16109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:17.672+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:37:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:17.672+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:37:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:17.672+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 232 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:37:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:17.673+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 232 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:37:19.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:18 smithi153 ceph-mon[36569]: pgmap v16110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:18 smithi104 ceph-mon[36624]: pgmap v16110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:18 smithi081 ceph-mon[31453]: pgmap v16110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:20 smithi153 ceph-mon[36569]: pgmap v16111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:20 smithi104 ceph-mon[36624]: pgmap v16111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:20 smithi081 ceph-mon[31453]: pgmap v16111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:23.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:22 smithi153 ceph-mon[36569]: pgmap v16112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:22 smithi104 ceph-mon[36624]: pgmap v16112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:22 smithi081 ceph-mon[31453]: pgmap v16112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:25.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:24 smithi153 ceph-mon[36569]: pgmap v16113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:25.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:24 smithi104 ceph-mon[36624]: pgmap v16113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:24 smithi081 ceph-mon[31453]: pgmap v16113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:27.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:26 smithi153 ceph-mon[36569]: pgmap v16114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:26 smithi104 ceph-mon[36624]: pgmap v16114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:26 smithi081 ceph-mon[31453]: pgmap v16114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:29.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:28 smithi153 ceph-mon[36569]: pgmap v16115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:28 smithi104 ceph-mon[36624]: pgmap v16115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:28 smithi081 ceph-mon[31453]: pgmap v16115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:30.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:30.628+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:37:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:30.628+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:37:30.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:30.628+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:37:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:30 smithi153 ceph-mon[36569]: pgmap v16116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:30 smithi104 ceph-mon[36624]: pgmap v16116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:30 smithi081 ceph-mon[31453]: pgmap v16116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:32 smithi153 ceph-mon[36569]: pgmap v16117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:32 smithi104 ceph-mon[36624]: pgmap v16117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:32 smithi081 ceph-mon[31453]: pgmap v16117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:34 smithi153 ceph-mon[36569]: pgmap v16118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:34 smithi104 ceph-mon[36624]: pgmap v16118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:34 smithi081 ceph-mon[31453]: pgmap v16118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:36 smithi153 ceph-mon[36569]: pgmap v16119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:36 smithi104 ceph-mon[36624]: pgmap v16119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:36 smithi081 ceph-mon[31453]: pgmap v16119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:38 smithi153 ceph-mon[36569]: pgmap v16120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:38 smithi104 ceph-mon[36624]: pgmap v16120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:38 smithi081 ceph-mon[31453]: pgmap v16120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:41.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:40 smithi153 ceph-mon[36569]: pgmap v16121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:40 smithi104 ceph-mon[36624]: pgmap v16121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:40 smithi081 ceph-mon[31453]: pgmap v16121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:42 smithi153 ceph-mon[36569]: pgmap v16122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:43.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:42 smithi104 ceph-mon[36624]: pgmap v16122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:42 smithi081 ceph-mon[31453]: pgmap v16122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:44 smithi153 ceph-mon[36569]: pgmap v16123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:44 smithi104 ceph-mon[36624]: pgmap v16123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:45.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:44 smithi081 ceph-mon[31453]: pgmap v16123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:45.638+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:37:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:45.638+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:37:45.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:37:45.638+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:37:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:46 smithi153 ceph-mon[36569]: pgmap v16124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:47.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:46 smithi104 ceph-mon[36624]: pgmap v16124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:46 smithi081 ceph-mon[31453]: pgmap v16124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:48.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:37:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:37:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:37:49.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:48 smithi153 ceph-mon[36569]: pgmap v16125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:48 smithi104 ceph-mon[36624]: pgmap v16125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:48 smithi081 ceph-mon[31453]: pgmap v16125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:50.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:37:50.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:37:50.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:37:50.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:37:50.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:37:50.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:37:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:37:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:37:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:37:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:50 smithi153 ceph-mon[36569]: pgmap v16126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:51.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:50 smithi153 ceph-mon[36569]: pgmap v16127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:50 smithi104 ceph-mon[36624]: pgmap v16126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:50 smithi104 ceph-mon[36624]: pgmap v16127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:50 smithi081 ceph-mon[31453]: pgmap v16126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:50 smithi081 ceph-mon[31453]: pgmap v16127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:51 smithi153 ceph-mon[36569]: pgmap v16128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:51 smithi104 ceph-mon[36624]: pgmap v16128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:51 smithi081 ceph-mon[31453]: pgmap v16128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:53 smithi153 ceph-mon[36569]: pgmap v16129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:53 smithi104 ceph-mon[36624]: pgmap v16129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:53 smithi081 ceph-mon[31453]: pgmap v16129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:56 smithi153 ceph-mon[36569]: pgmap v16130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:56 smithi104 ceph-mon[36624]: pgmap v16130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:56 smithi081 ceph-mon[31453]: pgmap v16130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:58.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:37:58 smithi153 ceph-mon[36569]: pgmap v16131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:37:58 smithi104 ceph-mon[36624]: pgmap v16131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:37:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:37:58 smithi081 ceph-mon[31453]: pgmap v16131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:00 smithi153 ceph-mon[36569]: pgmap v16132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:00 smithi104 ceph-mon[36624]: pgmap v16132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:00 smithi081 ceph-mon[31453]: pgmap v16132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:00.646+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:38:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:00.646+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:38:00.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:00.646+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:38:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:02 smithi153 ceph-mon[36569]: pgmap v16133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:02 smithi104 ceph-mon[36624]: pgmap v16133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:02 smithi081 ceph-mon[31453]: pgmap v16133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:04 smithi153 ceph-mon[36569]: pgmap v16134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:04 smithi104 ceph-mon[36624]: pgmap v16134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:04 smithi081 ceph-mon[31453]: pgmap v16134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:06 smithi153 ceph-mon[36569]: pgmap v16135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:06 smithi104 ceph-mon[36624]: pgmap v16135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:06 smithi081 ceph-mon[31453]: pgmap v16135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:08 smithi153 ceph-mon[36569]: pgmap v16136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:08 smithi104 ceph-mon[36624]: pgmap v16136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:08 smithi081 ceph-mon[31453]: pgmap v16136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:10 smithi153 ceph-mon[36569]: pgmap v16137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:10 smithi104 ceph-mon[36624]: pgmap v16137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:10 smithi081 ceph-mon[31453]: pgmap v16137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:12.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:12 smithi153 ceph-mon[36569]: pgmap v16138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:12 smithi104 ceph-mon[36624]: pgmap v16138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:12 smithi081 ceph-mon[31453]: pgmap v16138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:14 smithi153 ceph-mon[36569]: pgmap v16139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:14 smithi104 ceph-mon[36624]: pgmap v16139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:14 smithi081 ceph-mon[31453]: pgmap v16139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:15.651+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:38:15.653 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:15.651+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:38:15.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:15.651+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:38:16.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:16 smithi153 ceph-mon[36569]: pgmap v16140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:16 smithi104 ceph-mon[36624]: pgmap v16140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:16 smithi081 ceph-mon[31453]: pgmap v16140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:18.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:18 smithi153 ceph-mon[36569]: pgmap v16141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:18 smithi104 ceph-mon[36624]: pgmap v16141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:18.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:18 smithi081 ceph-mon[31453]: pgmap v16141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:20 smithi104 ceph-mon[36624]: pgmap v16142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:20.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:20 smithi081 ceph-mon[31453]: pgmap v16142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:20.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:20 smithi153 ceph-mon[36569]: pgmap v16142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:22 smithi104 ceph-mon[36624]: pgmap v16143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:22 smithi081 ceph-mon[31453]: pgmap v16143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:22 smithi153 ceph-mon[36569]: pgmap v16143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:24 smithi104 ceph-mon[36624]: pgmap v16144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:24 smithi081 ceph-mon[31453]: pgmap v16144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:24.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:24 smithi153 ceph-mon[36569]: pgmap v16144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:26 smithi104 ceph-mon[36624]: pgmap v16145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:26 smithi081 ceph-mon[31453]: pgmap v16145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:26 smithi153 ceph-mon[36569]: pgmap v16145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:28.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:28 smithi081 ceph-mon[31453]: pgmap v16146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:28 smithi153 ceph-mon[36569]: pgmap v16146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:28.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:28 smithi104 ceph-mon[36624]: pgmap v16146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:30.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:30 smithi081 ceph-mon[31453]: pgmap v16147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:30 smithi153 ceph-mon[36569]: pgmap v16147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:30.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:30 smithi104 ceph-mon[36624]: pgmap v16147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:30.667 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:30.665+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:38:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:30.665+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:38:30.668 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:30.665+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:38:32.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:32 smithi081 ceph-mon[31453]: pgmap v16148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:32 smithi153 ceph-mon[36569]: pgmap v16148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:32.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:32 smithi104 ceph-mon[36624]: pgmap v16148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:34 smithi081 ceph-mon[31453]: pgmap v16149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:34 smithi153 ceph-mon[36569]: pgmap v16149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:34.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:34 smithi104 ceph-mon[36624]: pgmap v16149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:36.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:36 smithi081 ceph-mon[31453]: pgmap v16150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:36.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:36 smithi153 ceph-mon[36569]: pgmap v16150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:36.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:36 smithi104 ceph-mon[36624]: pgmap v16150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:38 smithi081 ceph-mon[31453]: pgmap v16151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:38 smithi153 ceph-mon[36569]: pgmap v16151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:38.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:38 smithi104 ceph-mon[36624]: pgmap v16151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:40 smithi081 ceph-mon[31453]: pgmap v16152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:40 smithi153 ceph-mon[36569]: pgmap v16152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:40 smithi104 ceph-mon[36624]: pgmap v16152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:42 smithi081 ceph-mon[31453]: pgmap v16153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:42 smithi153 ceph-mon[36569]: pgmap v16153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:42 smithi104 ceph-mon[36624]: pgmap v16153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:44 smithi081 ceph-mon[31453]: pgmap v16154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:44 smithi153 ceph-mon[36569]: pgmap v16154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:44 smithi104 ceph-mon[36624]: pgmap v16154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:45.681+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:38:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:45.681+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:38:45.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:38:45.681+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:38:46.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:46 smithi104 ceph-mon[36624]: pgmap v16155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:46 smithi081 ceph-mon[31453]: pgmap v16155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:46.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:46 smithi153 ceph-mon[36569]: pgmap v16155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:48 smithi081 ceph-mon[31453]: pgmap v16156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:48.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:48 smithi153 ceph-mon[36569]: pgmap v16156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:48.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:48 smithi104 ceph-mon[36624]: pgmap v16156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:50 smithi081 ceph-mon[31453]: pgmap v16157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:38:50.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:50 smithi153 ceph-mon[36569]: pgmap v16157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:50.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:38:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:50 smithi104 ceph-mon[36624]: pgmap v16157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:50.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:38:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:52 smithi153 ceph-mon[36569]: pgmap v16158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:38:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:38:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:52 smithi153 ceph-mon[36569]: pgmap v16159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:52 smithi153 ceph-mon[36569]: pgmap v16160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:52 smithi104 ceph-mon[36624]: pgmap v16158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:52.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:38:52.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:38:52.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:52 smithi104 ceph-mon[36624]: pgmap v16159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:52 smithi104 ceph-mon[36624]: pgmap v16160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:52 smithi081 ceph-mon[31453]: pgmap v16158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:52.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:52.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:38:52.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:38:52.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:52 smithi081 ceph-mon[31453]: pgmap v16159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:52 smithi081 ceph-mon[31453]: pgmap v16160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:52.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:38:54.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:54 smithi153 ceph-mon[36569]: pgmap v16161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:54.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:54 smithi104 ceph-mon[36624]: pgmap v16161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:54.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:54 smithi081 ceph-mon[31453]: pgmap v16161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:56.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:56 smithi153 ceph-mon[36569]: pgmap v16162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:56.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:56 smithi104 ceph-mon[36624]: pgmap v16162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:56 smithi081 ceph-mon[31453]: pgmap v16162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:38:58 smithi104 ceph-mon[36624]: pgmap v16163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:38:58 smithi081 ceph-mon[31453]: pgmap v16163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:38:58.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:38:58 smithi153 ceph-mon[36569]: pgmap v16163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:00 smithi104 ceph-mon[36624]: pgmap v16164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:00 smithi081 ceph-mon[31453]: pgmap v16164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:00.691+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:39:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:00.691+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:39:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:00.691+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:39:00.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:00 smithi153 ceph-mon[36569]: pgmap v16164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:02 smithi104 ceph-mon[36624]: pgmap v16165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:02.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:02 smithi081 ceph-mon[31453]: pgmap v16165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:02.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:02 smithi153 ceph-mon[36569]: pgmap v16165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:04.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:04 smithi104 ceph-mon[36624]: pgmap v16166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:04.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:04 smithi081 ceph-mon[31453]: pgmap v16166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:04.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:04 smithi153 ceph-mon[36569]: pgmap v16166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:06 smithi104 ceph-mon[36624]: pgmap v16167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:06 smithi081 ceph-mon[31453]: pgmap v16167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:06.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:06 smithi153 ceph-mon[36569]: pgmap v16167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:08 smithi081 ceph-mon[31453]: pgmap v16168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:08 smithi153 ceph-mon[36569]: pgmap v16168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:08.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:08 smithi104 ceph-mon[36624]: pgmap v16168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:10.611 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:10 smithi153 ceph-mon[36569]: pgmap v16169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:10.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:10 smithi081 ceph-mon[31453]: pgmap v16169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:10 smithi104 ceph-mon[36624]: pgmap v16169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:12.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:12 smithi081 ceph-mon[31453]: pgmap v16170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:12.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:12 smithi153 ceph-mon[36569]: pgmap v16170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:12.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:12 smithi104 ceph-mon[36624]: pgmap v16170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:14.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:14 smithi081 ceph-mon[31453]: pgmap v16171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:14.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:14 smithi153 ceph-mon[36569]: pgmap v16171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:14.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:14 smithi104 ceph-mon[36624]: pgmap v16171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:15.704+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:39:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:15.705+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:39:15.707 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:15.705+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:39:16.636 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:16 smithi104 ceph-mon[36624]: pgmap v16172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:16.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:16 smithi081 ceph-mon[31453]: pgmap v16172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:16.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:16 smithi153 ceph-mon[36569]: pgmap v16172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:18 smithi081 ceph-mon[31453]: pgmap v16173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:18.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:18 smithi153 ceph-mon[36569]: pgmap v16173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:18.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:18 smithi104 ceph-mon[36624]: pgmap v16173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:20.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:20 smithi081 ceph-mon[31453]: pgmap v16174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:20.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:20 smithi153 ceph-mon[36569]: pgmap v16174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:20 smithi104 ceph-mon[36624]: pgmap v16174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:22.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:22 smithi081 ceph-mon[31453]: pgmap v16175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:22.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:22 smithi153 ceph-mon[36569]: pgmap v16175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:22.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:22 smithi104 ceph-mon[36624]: pgmap v16175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:24.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:24 smithi081 ceph-mon[31453]: pgmap v16176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:24.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:24 smithi153 ceph-mon[36569]: pgmap v16176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:24.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:24 smithi104 ceph-mon[36624]: pgmap v16176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:26.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:26 smithi081 ceph-mon[31453]: pgmap v16177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:26.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:26 smithi153 ceph-mon[36569]: pgmap v16177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:26.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:26 smithi104 ceph-mon[36624]: pgmap v16177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:28.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:28 smithi081 ceph-mon[31453]: pgmap v16178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:28.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:28 smithi153 ceph-mon[36569]: pgmap v16178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:28 smithi104 ceph-mon[36624]: pgmap v16178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:30.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:30 smithi081 ceph-mon[31453]: pgmap v16179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:30.708+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:39:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:30.708+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:39:30.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:30.708+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:39:30.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:30 smithi153 ceph-mon[36569]: pgmap v16179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:30.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:30 smithi104 ceph-mon[36624]: pgmap v16179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:32.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:32 smithi081 ceph-mon[31453]: pgmap v16180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:32.694+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:39:32.696 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:32.694+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:39:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:32.694+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 233 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:39:32.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:32.695+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 233 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:39:32.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:32 smithi153 ceph-mon[36569]: pgmap v16180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:32.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:32 smithi104 ceph-mon[36624]: pgmap v16180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:34.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:34 smithi081 ceph-mon[31453]: pgmap v16181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:34.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:34 smithi153 ceph-mon[36569]: pgmap v16181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:34 smithi104 ceph-mon[36624]: pgmap v16181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:36 smithi081 ceph-mon[31453]: pgmap v16182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:36.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:36 smithi153 ceph-mon[36569]: pgmap v16182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:36 smithi104 ceph-mon[36624]: pgmap v16182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:38.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:38 smithi153 ceph-mon[36569]: pgmap v16183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:38 smithi104 ceph-mon[36624]: pgmap v16183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:38 smithi081 ceph-mon[31453]: pgmap v16183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:40.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:40 smithi153 ceph-mon[36569]: pgmap v16184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:40.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:40 smithi104 ceph-mon[36624]: pgmap v16184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:40.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:40 smithi081 ceph-mon[31453]: pgmap v16184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:42.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:42 smithi153 ceph-mon[36569]: pgmap v16185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:42.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:42 smithi104 ceph-mon[36624]: pgmap v16185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:42 smithi081 ceph-mon[31453]: pgmap v16185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:44.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:44 smithi153 ceph-mon[36569]: pgmap v16186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:44.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:44 smithi104 ceph-mon[36624]: pgmap v16186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:44.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:44 smithi081 ceph-mon[31453]: pgmap v16186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:45.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:45.722+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:39:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:45.722+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:39:45.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:39:45.722+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:39:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:46 smithi153 ceph-mon[36569]: pgmap v16187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:46 smithi104 ceph-mon[36624]: pgmap v16187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:46 smithi081 ceph-mon[31453]: pgmap v16187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:48 smithi153 ceph-mon[36569]: pgmap v16188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:48 smithi104 ceph-mon[36624]: pgmap v16188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:48 smithi081 ceph-mon[31453]: pgmap v16188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:50.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:50 smithi153 ceph-mon[36569]: pgmap v16189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:50.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:50 smithi104 ceph-mon[36624]: pgmap v16189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:50 smithi081 ceph-mon[31453]: pgmap v16189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:52.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:52 smithi153 ceph-mon[36569]: pgmap v16190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:52.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:39:52.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:52 smithi104 ceph-mon[36624]: pgmap v16190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:52.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:39:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:52 smithi081 ceph-mon[31453]: pgmap v16190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: pgmap v16191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: pgmap v16192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: pgmap v16191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: pgmap v16192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:39:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:39:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: pgmap v16191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: pgmap v16192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:39:56.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:55 smithi153 ceph-mon[36569]: pgmap v16193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:55 smithi104 ceph-mon[36624]: pgmap v16193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:55 smithi081 ceph-mon[31453]: pgmap v16193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:39:57 smithi153 ceph-mon[36569]: pgmap v16194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:39:57 smithi104 ceph-mon[36624]: pgmap v16194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:39:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:39:57 smithi081 ceph-mon[31453]: pgmap v16194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:00.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:00 smithi153 ceph-mon[36569]: pgmap v16195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:00 smithi104 ceph-mon[36624]: pgmap v16195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:00 smithi081 ceph-mon[31453]: pgmap v16195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:00.740 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:00.738+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:40:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:00.738+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:40:00.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:00.738+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:40:01.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:01 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:40:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:01 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:40:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:01 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:40:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:02 smithi153 ceph-mon[36569]: pgmap v16196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:02 smithi104 ceph-mon[36624]: pgmap v16196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:02 smithi081 ceph-mon[31453]: pgmap v16196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:04 smithi153 ceph-mon[36569]: pgmap v16197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:04 smithi104 ceph-mon[36624]: pgmap v16197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:04 smithi081 ceph-mon[31453]: pgmap v16197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:06 smithi153 ceph-mon[36569]: pgmap v16198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:06 smithi104 ceph-mon[36624]: pgmap v16198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:06 smithi081 ceph-mon[31453]: pgmap v16198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:08 smithi153 ceph-mon[36569]: pgmap v16199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:08 smithi104 ceph-mon[36624]: pgmap v16199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:08 smithi081 ceph-mon[31453]: pgmap v16199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:10.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:10 smithi153 ceph-mon[36569]: pgmap v16200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:10 smithi104 ceph-mon[36624]: pgmap v16200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:10 smithi081 ceph-mon[31453]: pgmap v16200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:12 smithi153 ceph-mon[36569]: pgmap v16201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:12 smithi104 ceph-mon[36624]: pgmap v16201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:12 smithi081 ceph-mon[31453]: pgmap v16201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:14 smithi153 ceph-mon[36569]: pgmap v16202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:14 smithi104 ceph-mon[36624]: pgmap v16202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:14 smithi081 ceph-mon[31453]: pgmap v16202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:15.747+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:40:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:15.747+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:40:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:15.747+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:40:16.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:16 smithi153 ceph-mon[36569]: pgmap v16203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:16 smithi104 ceph-mon[36624]: pgmap v16203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:16 smithi081 ceph-mon[31453]: pgmap v16203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:17.701+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:40:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:17.701+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:40:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:17.702+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 234 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:40:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:17.702+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 234 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:40:18.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:18 smithi104 ceph-mon[36624]: pgmap v16204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:18.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:18 smithi081 ceph-mon[31453]: pgmap v16204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:18.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:18 smithi153 ceph-mon[36569]: pgmap v16204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:20 smithi104 ceph-mon[36624]: pgmap v16205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:20.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:20 smithi081 ceph-mon[31453]: pgmap v16205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:20.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:20 smithi153 ceph-mon[36569]: pgmap v16205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:22 smithi104 ceph-mon[36624]: pgmap v16206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:22 smithi081 ceph-mon[31453]: pgmap v16206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:22 smithi153 ceph-mon[36569]: pgmap v16206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:24 smithi104 ceph-mon[36624]: pgmap v16207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:24 smithi081 ceph-mon[31453]: pgmap v16207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:24 smithi153 ceph-mon[36569]: pgmap v16207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:26 smithi104 ceph-mon[36624]: pgmap v16208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:26 smithi081 ceph-mon[31453]: pgmap v16208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:26.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:26 smithi153 ceph-mon[36569]: pgmap v16208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:28.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:28 smithi081 ceph-mon[31453]: pgmap v16209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:28.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:28 smithi153 ceph-mon[36569]: pgmap v16209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:28.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:28 smithi104 ceph-mon[36624]: pgmap v16209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:30.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:30.763+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:40:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:30.763+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:40:30.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:30.763+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:40:30.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:30 smithi153 ceph-mon[36569]: pgmap v16210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:30.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:30 smithi104 ceph-mon[36624]: pgmap v16210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:30.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:30 smithi081 ceph-mon[31453]: pgmap v16210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:32.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:32 smithi153 ceph-mon[36569]: pgmap v16211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:32.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:32 smithi104 ceph-mon[36624]: pgmap v16211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:32.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:32 smithi081 ceph-mon[31453]: pgmap v16211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:34.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:34 smithi153 ceph-mon[36569]: pgmap v16212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:34 smithi104 ceph-mon[36624]: pgmap v16212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:34 smithi081 ceph-mon[31453]: pgmap v16212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:36 smithi104 ceph-mon[36624]: pgmap v16213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:36.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:36 smithi081 ceph-mon[31453]: pgmap v16213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:37.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:36 smithi153 ceph-mon[36569]: pgmap v16213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:38 smithi104 ceph-mon[36624]: pgmap v16214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:38 smithi081 ceph-mon[31453]: pgmap v16214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:39.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:38 smithi153 ceph-mon[36569]: pgmap v16214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:40.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:40 smithi104 ceph-mon[36624]: pgmap v16215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:40.861 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:40 smithi153 ceph-mon[36569]: pgmap v16215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:40.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:40 smithi081 ceph-mon[31453]: pgmap v16215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:42.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:42 smithi104 ceph-mon[36624]: pgmap v16216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:42 smithi081 ceph-mon[31453]: pgmap v16216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:43.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:42 smithi153 ceph-mon[36569]: pgmap v16216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:44.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:44 smithi104 ceph-mon[36624]: pgmap v16217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:44.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:44 smithi081 ceph-mon[31453]: pgmap v16217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:45.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:44 smithi153 ceph-mon[36569]: pgmap v16217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:45.764+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:40:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:45.764+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:40:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:40:45.764+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:40:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:46 smithi081 ceph-mon[31453]: pgmap v16218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:47.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:46 smithi153 ceph-mon[36569]: pgmap v16218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:47.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:46 smithi104 ceph-mon[36624]: pgmap v16218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:48 smithi081 ceph-mon[31453]: pgmap v16219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:49.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:48 smithi153 ceph-mon[36569]: pgmap v16219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:48 smithi104 ceph-mon[36624]: pgmap v16219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:50 smithi081 ceph-mon[31453]: pgmap v16220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:51.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:50 smithi153 ceph-mon[36569]: pgmap v16220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:50 smithi104 ceph-mon[36624]: pgmap v16220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:52 smithi081 ceph-mon[31453]: pgmap v16221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:53.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:52 smithi153 ceph-mon[36569]: pgmap v16221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:52 smithi104 ceph-mon[36624]: pgmap v16221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:40:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:40:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:40:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:54 smithi081 ceph-mon[31453]: pgmap v16222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:40:54.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:40:54.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:40:55.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:54 smithi153 ceph-mon[36569]: pgmap v16222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:55.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:40:55.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:40:55.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:40:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:54 smithi104 ceph-mon[36624]: pgmap v16222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:40:55.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:40:55.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:40:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:55 smithi081 ceph-mon[31453]: pgmap v16223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:55 smithi081 ceph-mon[31453]: pgmap v16224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:55 smithi153 ceph-mon[36569]: pgmap v16223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:55 smithi153 ceph-mon[36569]: pgmap v16224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:55 smithi104 ceph-mon[36624]: pgmap v16223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:55 smithi104 ceph-mon[36624]: pgmap v16224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:57 smithi081 ceph-mon[31453]: pgmap v16225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:58.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:57 smithi153 ceph-mon[36569]: pgmap v16225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:57 smithi104 ceph-mon[36624]: pgmap v16225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:40:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:40:59 smithi081 ceph-mon[31453]: pgmap v16226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:40:59 smithi153 ceph-mon[36569]: pgmap v16226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:40:59 smithi104 ceph-mon[36624]: pgmap v16226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:00.779+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:41:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:00.779+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:41:00.782 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:00.779+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:41:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:01 smithi081 ceph-mon[31453]: pgmap v16227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:01 smithi153 ceph-mon[36569]: pgmap v16227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:01 smithi104 ceph-mon[36624]: pgmap v16227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:03 smithi081 ceph-mon[31453]: pgmap v16228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:03 smithi153 ceph-mon[36569]: pgmap v16228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:04.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:03 smithi104 ceph-mon[36624]: pgmap v16228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:05.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:05 smithi081 ceph-mon[31453]: pgmap v16229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:06.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:05 smithi153 ceph-mon[36569]: pgmap v16229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:05 smithi104 ceph-mon[36624]: pgmap v16229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:07.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:07 smithi081 ceph-mon[31453]: pgmap v16230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:07 smithi153 ceph-mon[36569]: pgmap v16230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:07 smithi104 ceph-mon[36624]: pgmap v16230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:10.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:09 smithi153 ceph-mon[36569]: pgmap v16231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:10.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:09 smithi104 ceph-mon[36624]: pgmap v16231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:09 smithi081 ceph-mon[31453]: pgmap v16231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:12.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:11 smithi153 ceph-mon[36569]: pgmap v16232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:12.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:11 smithi104 ceph-mon[36624]: pgmap v16232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:12.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:11 smithi081 ceph-mon[31453]: pgmap v16232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:14.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:13 smithi153 ceph-mon[36569]: pgmap v16233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:13 smithi104 ceph-mon[36624]: pgmap v16233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:13 smithi081 ceph-mon[31453]: pgmap v16233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:15.784+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:41:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:15.784+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:41:15.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:15.784+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:41:15.936 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:15 smithi081 ceph-mon[31453]: pgmap v16234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:16.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:15 smithi153 ceph-mon[36569]: pgmap v16234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:15 smithi104 ceph-mon[36624]: pgmap v16234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:18.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:17 smithi153 ceph-mon[36569]: pgmap v16235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:17 smithi104 ceph-mon[36624]: pgmap v16235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:17 smithi081 ceph-mon[31453]: pgmap v16235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:20.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:19 smithi153 ceph-mon[36569]: pgmap v16236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:20.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:19 smithi104 ceph-mon[36624]: pgmap v16236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:20.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:19 smithi081 ceph-mon[31453]: pgmap v16236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:22.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:21 smithi153 ceph-mon[36569]: pgmap v16237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:21 smithi104 ceph-mon[36624]: pgmap v16237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:21 smithi081 ceph-mon[31453]: pgmap v16237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:24.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:23 smithi153 ceph-mon[36569]: pgmap v16238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:24.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:23 smithi104 ceph-mon[36624]: pgmap v16238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:23 smithi081 ceph-mon[31453]: pgmap v16238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:26.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:25 smithi153 ceph-mon[36569]: pgmap v16239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:25 smithi104 ceph-mon[36624]: pgmap v16239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:25 smithi081 ceph-mon[31453]: pgmap v16239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:28.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:27 smithi153 ceph-mon[36569]: pgmap v16240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:28.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:27 smithi104 ceph-mon[36624]: pgmap v16240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:27 smithi081 ceph-mon[31453]: pgmap v16240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:30.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:29 smithi153 ceph-mon[36569]: pgmap v16241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:30.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:29 smithi104 ceph-mon[36624]: pgmap v16241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:29 smithi081 ceph-mon[31453]: pgmap v16241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:30.797+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:41:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:30.797+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:41:30.801 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:30.797+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:41:32.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:31 smithi153 ceph-mon[36569]: pgmap v16242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:32.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:31 smithi104 ceph-mon[36624]: pgmap v16242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:31 smithi081 ceph-mon[31453]: pgmap v16242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:34.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:33 smithi153 ceph-mon[36569]: pgmap v16243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:33 smithi104 ceph-mon[36624]: pgmap v16243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:33 smithi081 ceph-mon[31453]: pgmap v16243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:36.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:35 smithi153 ceph-mon[36569]: pgmap v16244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:36.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:35 smithi104 ceph-mon[36624]: pgmap v16244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:35 smithi081 ceph-mon[31453]: pgmap v16244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:38.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:37 smithi153 ceph-mon[36569]: pgmap v16245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:38.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:37 smithi104 ceph-mon[36624]: pgmap v16245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:37 smithi081 ceph-mon[31453]: pgmap v16245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:40.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:39 smithi153 ceph-mon[36569]: pgmap v16246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:39 smithi104 ceph-mon[36624]: pgmap v16246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:39 smithi081 ceph-mon[31453]: pgmap v16246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:42.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:41 smithi153 ceph-mon[36569]: pgmap v16247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:42.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:41 smithi104 ceph-mon[36624]: pgmap v16247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:42.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:41 smithi081 ceph-mon[31453]: pgmap v16247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:44.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:43 smithi153 ceph-mon[36569]: pgmap v16248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:44.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:43 smithi104 ceph-mon[36624]: pgmap v16248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:43 smithi081 ceph-mon[31453]: pgmap v16248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:45.811 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:45.809+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:41:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:45.809+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:41:45.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:41:45.809+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:41:46.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:45 smithi153 ceph-mon[36569]: pgmap v16249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:46.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:45 smithi104 ceph-mon[36624]: pgmap v16249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:46.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:45 smithi081 ceph-mon[31453]: pgmap v16249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:47 smithi153 ceph-mon[36569]: pgmap v16250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:47 smithi104 ceph-mon[36624]: pgmap v16250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:47 smithi081 ceph-mon[31453]: pgmap v16250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:50.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:49 smithi153 ceph-mon[36569]: pgmap v16251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:50.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:49 smithi104 ceph-mon[36624]: pgmap v16251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:49 smithi081 ceph-mon[31453]: pgmap v16251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:52.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:51 smithi153 ceph-mon[36569]: pgmap v16252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:51 smithi104 ceph-mon[36624]: pgmap v16252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:51 smithi081 ceph-mon[31453]: pgmap v16252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:54.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:53 smithi153 ceph-mon[36569]: pgmap v16253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:53 smithi104 ceph-mon[36624]: pgmap v16253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:53 smithi081 ceph-mon[31453]: pgmap v16253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:41:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:41:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:41:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: pgmap v16254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: pgmap v16254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: pgmap v16254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:56.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:41:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:41:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:41:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:41:57.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:41:57.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:41:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:57 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:41:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:57 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:41:58.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:57 smithi104 ceph-mon[36624]: pgmap v16255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:58.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:57 smithi104 ceph-mon[36624]: pgmap v16256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:57 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:41:58.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:57 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:41:58.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:57 smithi081 ceph-mon[31453]: pgmap v16255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:58.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:57 smithi081 ceph-mon[31453]: pgmap v16256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:57 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:41:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:57 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:41:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:57 smithi153 ceph-mon[36569]: pgmap v16255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:41:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:57 smithi153 ceph-mon[36569]: pgmap v16256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:41:59 smithi104 ceph-mon[36624]: pgmap v16257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:41:59 smithi081 ceph-mon[31453]: pgmap v16257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:41:59 smithi153 ceph-mon[36569]: pgmap v16257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:00.814+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:42:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:00.815+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:42:00.817 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:00.815+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:42:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:01 smithi104 ceph-mon[36624]: pgmap v16258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:01 smithi081 ceph-mon[31453]: pgmap v16258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:01 smithi153 ceph-mon[36569]: pgmap v16258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:04.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:03 smithi104 ceph-mon[36624]: pgmap v16259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:03 smithi081 ceph-mon[31453]: pgmap v16259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:03 smithi153 ceph-mon[36569]: pgmap v16259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:05 smithi104 ceph-mon[36624]: pgmap v16260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:05 smithi081 ceph-mon[31453]: pgmap v16260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:05 smithi153 ceph-mon[36569]: pgmap v16260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:06 smithi081 ceph-mon[31453]: pgmap v16261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:07.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:06 smithi153 ceph-mon[36569]: pgmap v16261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:07.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:06 smithi104 ceph-mon[36624]: pgmap v16261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:09 smithi081 ceph-mon[31453]: pgmap v16262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:09.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:09 smithi153 ceph-mon[36569]: pgmap v16262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:09.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:09 smithi104 ceph-mon[36624]: pgmap v16262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:11.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:11 smithi153 ceph-mon[36569]: pgmap v16263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:11 smithi081 ceph-mon[31453]: pgmap v16263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:11.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:11 smithi104 ceph-mon[36624]: pgmap v16263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:13 smithi081 ceph-mon[31453]: pgmap v16264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:13.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:13 smithi153 ceph-mon[36569]: pgmap v16264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:13.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:13 smithi104 ceph-mon[36624]: pgmap v16264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:15.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:15 smithi081 ceph-mon[31453]: pgmap v16265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:15.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:15 smithi153 ceph-mon[36569]: pgmap v16265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:15.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:15 smithi104 ceph-mon[36624]: pgmap v16265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:15.819+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:42:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:15.819+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:42:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:15.819+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:42:17.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:17 smithi104 ceph-mon[36624]: pgmap v16266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:17 smithi081 ceph-mon[31453]: pgmap v16266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:17.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:17 smithi153 ceph-mon[36569]: pgmap v16266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:19.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:19 smithi081 ceph-mon[31453]: pgmap v16267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:19.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:19 smithi153 ceph-mon[36569]: pgmap v16267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:19.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:19 smithi104 ceph-mon[36624]: pgmap v16267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:21 smithi081 ceph-mon[31453]: pgmap v16268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:21.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:21 smithi153 ceph-mon[36569]: pgmap v16268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:21.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:21 smithi104 ceph-mon[36624]: pgmap v16268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:23 smithi081 ceph-mon[31453]: pgmap v16269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:23 smithi153 ceph-mon[36569]: pgmap v16269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:23.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:23 smithi104 ceph-mon[36624]: pgmap v16269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:25 smithi081 ceph-mon[31453]: pgmap v16270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:25.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:25 smithi153 ceph-mon[36569]: pgmap v16270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:25.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:25 smithi104 ceph-mon[36624]: pgmap v16270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:27 smithi081 ceph-mon[31453]: pgmap v16271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:27.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:27 smithi153 ceph-mon[36569]: pgmap v16271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:27.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:27 smithi104 ceph-mon[36624]: pgmap v16271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:29 smithi081 ceph-mon[31453]: pgmap v16272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:29.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:29 smithi153 ceph-mon[36569]: pgmap v16272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:29.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:29 smithi104 ceph-mon[36624]: pgmap v16272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:30.831+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:42:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:30.831+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:42:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:30.831+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:42:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:31 smithi081 ceph-mon[31453]: pgmap v16273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:31.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:31 smithi153 ceph-mon[36569]: pgmap v16273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:31.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:31 smithi104 ceph-mon[36624]: pgmap v16273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:33 smithi081 ceph-mon[31453]: pgmap v16274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:33.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:33 smithi153 ceph-mon[36569]: pgmap v16274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:33 smithi104 ceph-mon[36624]: pgmap v16274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:35.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:35 smithi153 ceph-mon[36569]: pgmap v16275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:35.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:35 smithi104 ceph-mon[36624]: pgmap v16275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:35 smithi081 ceph-mon[31453]: pgmap v16275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:37.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:37 smithi153 ceph-mon[36569]: pgmap v16276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:37.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:37 smithi104 ceph-mon[36624]: pgmap v16276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:37 smithi081 ceph-mon[31453]: pgmap v16276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:39.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:39 smithi153 ceph-mon[36569]: pgmap v16277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:39.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:39 smithi104 ceph-mon[36624]: pgmap v16277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:39.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:39 smithi081 ceph-mon[31453]: pgmap v16277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:41.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:41 smithi153 ceph-mon[36569]: pgmap v16278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:41.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:41 smithi104 ceph-mon[36624]: pgmap v16278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:41.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:41 smithi081 ceph-mon[31453]: pgmap v16278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:43.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:43 smithi153 ceph-mon[36569]: pgmap v16279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:43.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:43 smithi104 ceph-mon[36624]: pgmap v16279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:43 smithi081 ceph-mon[31453]: pgmap v16279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:45.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:45 smithi153 ceph-mon[36569]: pgmap v16280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:45 smithi104 ceph-mon[36624]: pgmap v16280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:45.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:45 smithi081 ceph-mon[31453]: pgmap v16280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:45.848 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:45.846+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:42:45.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:45.846+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:42:45.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:42:45.846+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:42:47.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:47 smithi081 ceph-mon[31453]: pgmap v16281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:47.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:47 smithi153 ceph-mon[36569]: pgmap v16281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:47 smithi104 ceph-mon[36624]: pgmap v16281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:49.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:49 smithi153 ceph-mon[36569]: pgmap v16282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:49 smithi104 ceph-mon[36624]: pgmap v16282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:49 smithi081 ceph-mon[31453]: pgmap v16282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:51.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:51 smithi153 ceph-mon[36569]: pgmap v16283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:51.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:51 smithi104 ceph-mon[36624]: pgmap v16283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:51 smithi081 ceph-mon[31453]: pgmap v16283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:53.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:53 smithi153 ceph-mon[36569]: pgmap v16284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:53.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:53 smithi104 ceph-mon[36624]: pgmap v16284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:53.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:53 smithi081 ceph-mon[31453]: pgmap v16284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:55.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:55 smithi153 ceph-mon[36569]: pgmap v16285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:55.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:55 smithi104 ceph-mon[36624]: pgmap v16285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:55 smithi081 ceph-mon[31453]: pgmap v16285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:57.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:57 smithi153 ceph-mon[36569]: pgmap v16286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:57.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:42:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:57 smithi104 ceph-mon[36624]: pgmap v16286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:42:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:57 smithi081 ceph-mon[31453]: pgmap v16286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:57.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:42:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:42:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:42:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:58 smithi153 ceph-mon[36569]: pgmap v16287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:58 smithi153 ceph-mon[36569]: pgmap v16288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:42:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:42:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:42:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:42:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:58 smithi104 ceph-mon[36624]: pgmap v16287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:58.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:58 smithi104 ceph-mon[36624]: pgmap v16288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:58.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:42:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:42:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:42:58.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:42:58.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:58 smithi081 ceph-mon[31453]: pgmap v16287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:58.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:58 smithi081 ceph-mon[31453]: pgmap v16288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:42:58.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:42:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:43:00.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:00.848+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:43:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:00.848+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:43:00.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:00.848+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:43:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:00 smithi081 ceph-mon[31453]: pgmap v16289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:00 smithi153 ceph-mon[36569]: pgmap v16289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:00 smithi104 ceph-mon[36624]: pgmap v16289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:02 smithi081 ceph-mon[31453]: pgmap v16290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:02 smithi153 ceph-mon[36569]: pgmap v16290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:02 smithi104 ceph-mon[36624]: pgmap v16290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:04.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:04 smithi081 ceph-mon[31453]: pgmap v16291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:05.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:04 smithi153 ceph-mon[36569]: pgmap v16291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:04 smithi104 ceph-mon[36624]: pgmap v16291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:06 smithi081 ceph-mon[31453]: pgmap v16292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:07.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:06 smithi153 ceph-mon[36569]: pgmap v16292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:06 smithi104 ceph-mon[36624]: pgmap v16292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:08.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:08 smithi081 ceph-mon[31453]: pgmap v16293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:09.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:08 smithi153 ceph-mon[36569]: pgmap v16293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:09.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:08 smithi104 ceph-mon[36624]: pgmap v16293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:10 smithi081 ceph-mon[31453]: pgmap v16294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:11.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:10 smithi153 ceph-mon[36569]: pgmap v16294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:10 smithi104 ceph-mon[36624]: pgmap v16294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:12.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:12 smithi081 ceph-mon[31453]: pgmap v16295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:13.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:12 smithi153 ceph-mon[36569]: pgmap v16295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:12 smithi104 ceph-mon[36624]: pgmap v16295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:14.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:14 smithi081 ceph-mon[31453]: pgmap v16296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:15.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:14 smithi153 ceph-mon[36569]: pgmap v16296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:14 smithi104 ceph-mon[36624]: pgmap v16296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:15.860+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:43:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:15.860+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:43:15.863 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:15.860+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:43:16.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:16 smithi081 ceph-mon[31453]: pgmap v16297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:17.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:16 smithi153 ceph-mon[36569]: pgmap v16297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:16 smithi104 ceph-mon[36624]: pgmap v16297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:17.732+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:43:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:17.732+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:43:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:17.732+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 235 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:43:17.735 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:17.732+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 235 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:43:19.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:18 smithi153 ceph-mon[36569]: pgmap v16298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:18 smithi104 ceph-mon[36624]: pgmap v16298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:18 smithi081 ceph-mon[31453]: pgmap v16298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:20 smithi153 ceph-mon[36569]: pgmap v16299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:20 smithi104 ceph-mon[36624]: pgmap v16299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:20 smithi081 ceph-mon[31453]: pgmap v16299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:23.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:22 smithi153 ceph-mon[36569]: pgmap v16300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:22 smithi104 ceph-mon[36624]: pgmap v16300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:22 smithi081 ceph-mon[31453]: pgmap v16300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:25.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:24 smithi153 ceph-mon[36569]: pgmap v16301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:25.065 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:24 smithi081 ceph-mon[31453]: pgmap v16301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:25.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:24 smithi104 ceph-mon[36624]: pgmap v16301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:27.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:26 smithi153 ceph-mon[36569]: pgmap v16302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:26 smithi104 ceph-mon[36624]: pgmap v16302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:26 smithi081 ceph-mon[31453]: pgmap v16302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:29.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:28 smithi153 ceph-mon[36569]: pgmap v16303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:28 smithi104 ceph-mon[36624]: pgmap v16303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:28 smithi081 ceph-mon[31453]: pgmap v16303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:30.874+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:43:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:30.874+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:43:30.877 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:30.874+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:43:31.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:30 smithi153 ceph-mon[36569]: pgmap v16304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:30 smithi104 ceph-mon[36624]: pgmap v16304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:30 smithi081 ceph-mon[31453]: pgmap v16304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:33.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:32 smithi153 ceph-mon[36569]: pgmap v16305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:32 smithi104 ceph-mon[36624]: pgmap v16305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:33.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:32 smithi081 ceph-mon[31453]: pgmap v16305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:34 smithi153 ceph-mon[36569]: pgmap v16306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:34 smithi104 ceph-mon[36624]: pgmap v16306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:34 smithi081 ceph-mon[31453]: pgmap v16306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:37.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:36 smithi153 ceph-mon[36569]: pgmap v16307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:37.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:36 smithi104 ceph-mon[36624]: pgmap v16307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:37.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:36 smithi081 ceph-mon[31453]: pgmap v16307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:39.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:38 smithi153 ceph-mon[36569]: pgmap v16308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:38 smithi104 ceph-mon[36624]: pgmap v16308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:39.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:38 smithi081 ceph-mon[31453]: pgmap v16308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:41.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:40 smithi153 ceph-mon[36569]: pgmap v16309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:40 smithi104 ceph-mon[36624]: pgmap v16309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:40 smithi081 ceph-mon[31453]: pgmap v16309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:43.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:42 smithi153 ceph-mon[36569]: pgmap v16310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:43.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:42 smithi104 ceph-mon[36624]: pgmap v16310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:42 smithi081 ceph-mon[31453]: pgmap v16310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:45.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:44 smithi153 ceph-mon[36569]: pgmap v16311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:44 smithi104 ceph-mon[36624]: pgmap v16311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:45.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:44 smithi081 ceph-mon[31453]: pgmap v16311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:45.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:45.889+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:43:45.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:45.889+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:43:45.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:43:45.889+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:43:47.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:46 smithi153 ceph-mon[36569]: pgmap v16312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:47.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:46 smithi104 ceph-mon[36624]: pgmap v16312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:46 smithi081 ceph-mon[31453]: pgmap v16312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:49.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:48 smithi153 ceph-mon[36569]: pgmap v16313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:48 smithi104 ceph-mon[36624]: pgmap v16313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:48 smithi081 ceph-mon[31453]: pgmap v16313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:51.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:50 smithi153 ceph-mon[36569]: pgmap v16314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:50 smithi104 ceph-mon[36624]: pgmap v16314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:50 smithi081 ceph-mon[31453]: pgmap v16314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:53.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:52 smithi153 ceph-mon[36569]: pgmap v16315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:52 smithi104 ceph-mon[36624]: pgmap v16315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:52 smithi081 ceph-mon[31453]: pgmap v16315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:55.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:54 smithi153 ceph-mon[36569]: pgmap v16316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:54 smithi104 ceph-mon[36624]: pgmap v16316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:54 smithi081 ceph-mon[31453]: pgmap v16316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:57.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:56 smithi153 ceph-mon[36569]: pgmap v16317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:56 smithi104 ceph-mon[36624]: pgmap v16317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:56 smithi081 ceph-mon[31453]: pgmap v16317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:59.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:58 smithi153 ceph-mon[36569]: pgmap v16318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:59.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:43:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:58 smithi104 ceph-mon[36624]: pgmap v16318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:43:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:58 smithi081 ceph-mon[31453]: pgmap v16318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:43:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:44:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:44:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:44:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:43:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:44:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:44:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:44:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:43:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:44:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:44:00.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:44:00.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:43:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:44:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:00.890+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:44:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:00.890+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:44:00.893 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:00.890+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:44:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:00 smithi153 ceph-mon[36569]: pgmap v16319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:00 smithi153 ceph-mon[36569]: pgmap v16320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:00 smithi104 ceph-mon[36624]: pgmap v16319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:01.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:00 smithi104 ceph-mon[36624]: pgmap v16320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:00 smithi081 ceph-mon[31453]: pgmap v16319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:01.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:00 smithi081 ceph-mon[31453]: pgmap v16320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:02 smithi153 ceph-mon[36569]: pgmap v16321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:02 smithi104 ceph-mon[36624]: pgmap v16321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:02 smithi081 ceph-mon[31453]: pgmap v16321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:05.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:04 smithi153 ceph-mon[36569]: pgmap v16322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:04 smithi104 ceph-mon[36624]: pgmap v16322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:04 smithi081 ceph-mon[31453]: pgmap v16322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:07.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:06 smithi153 ceph-mon[36569]: pgmap v16323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:06 smithi104 ceph-mon[36624]: pgmap v16323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:06 smithi081 ceph-mon[31453]: pgmap v16323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:09.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:08 smithi153 ceph-mon[36569]: pgmap v16324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:08 smithi104 ceph-mon[36624]: pgmap v16324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:08 smithi081 ceph-mon[31453]: pgmap v16324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:11.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:10 smithi153 ceph-mon[36569]: pgmap v16325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:10 smithi104 ceph-mon[36624]: pgmap v16325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:11.146 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:10 smithi081 ceph-mon[31453]: pgmap v16325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:12 smithi104 ceph-mon[36624]: pgmap v16326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:12 smithi081 ceph-mon[31453]: pgmap v16326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:12 smithi153 ceph-mon[36569]: pgmap v16326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:14 smithi104 ceph-mon[36624]: pgmap v16327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:14 smithi081 ceph-mon[31453]: pgmap v16327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:15.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:14 smithi153 ceph-mon[36569]: pgmap v16327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:15.906+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:44:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:15.906+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:44:15.909 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:15.906+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:44:17.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:16 smithi104 ceph-mon[36624]: pgmap v16328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:16 smithi081 ceph-mon[31453]: pgmap v16328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:16 smithi153 ceph-mon[36569]: pgmap v16328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:18 smithi104 ceph-mon[36624]: pgmap v16329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:18 smithi081 ceph-mon[31453]: pgmap v16329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:18 smithi153 ceph-mon[36569]: pgmap v16329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:20 smithi104 ceph-mon[36624]: pgmap v16330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:20 smithi081 ceph-mon[31453]: pgmap v16330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:20 smithi153 ceph-mon[36569]: pgmap v16330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:22 smithi081 ceph-mon[31453]: pgmap v16331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:22 smithi153 ceph-mon[36569]: pgmap v16331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:22 smithi104 ceph-mon[36624]: pgmap v16331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:24 smithi081 ceph-mon[31453]: pgmap v16332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:25.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:24 smithi153 ceph-mon[36569]: pgmap v16332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:24 smithi104 ceph-mon[36624]: pgmap v16332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:26 smithi081 ceph-mon[31453]: pgmap v16333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:26 smithi153 ceph-mon[36569]: pgmap v16333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:26 smithi104 ceph-mon[36624]: pgmap v16333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:28 smithi081 ceph-mon[31453]: pgmap v16334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:28 smithi153 ceph-mon[36569]: pgmap v16334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:28 smithi104 ceph-mon[36624]: pgmap v16334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:30.924 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:30.922+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:44:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:30.922+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:44:30.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:30.922+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:44:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:30 smithi081 ceph-mon[31453]: pgmap v16335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:31.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:30 smithi153 ceph-mon[36569]: pgmap v16335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:30 smithi104 ceph-mon[36624]: pgmap v16335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:32 smithi081 ceph-mon[31453]: pgmap v16336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:32 smithi153 ceph-mon[36569]: pgmap v16336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:33.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:32 smithi104 ceph-mon[36624]: pgmap v16336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:34 smithi081 ceph-mon[31453]: pgmap v16337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:35.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:34 smithi153 ceph-mon[36569]: pgmap v16337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:34 smithi104 ceph-mon[36624]: pgmap v16337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:35 smithi081 ceph-mon[31453]: pgmap v16338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:36.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:35 smithi153 ceph-mon[36569]: pgmap v16338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:35 smithi104 ceph-mon[36624]: pgmap v16338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:38.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:38 smithi153 ceph-mon[36569]: pgmap v16339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:38.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:38 smithi104 ceph-mon[36624]: pgmap v16339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:38 smithi081 ceph-mon[31453]: pgmap v16339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:40.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:40 smithi153 ceph-mon[36569]: pgmap v16340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:40 smithi104 ceph-mon[36624]: pgmap v16340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:40.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:40 smithi081 ceph-mon[31453]: pgmap v16340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:42.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:42 smithi153 ceph-mon[36569]: pgmap v16341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:42 smithi104 ceph-mon[36624]: pgmap v16341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:42 smithi081 ceph-mon[31453]: pgmap v16341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:44.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:44 smithi153 ceph-mon[36569]: pgmap v16342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:44 smithi104 ceph-mon[36624]: pgmap v16342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:44.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:44 smithi081 ceph-mon[31453]: pgmap v16342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:45.940 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:45.938+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:44:45.940 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:45.938+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:44:45.940 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:44:45.938+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:44:46.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:46 smithi153 ceph-mon[36569]: pgmap v16343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:46.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:46 smithi104 ceph-mon[36624]: pgmap v16343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:46 smithi081 ceph-mon[31453]: pgmap v16343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:48.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:48 smithi081 ceph-mon[31453]: pgmap v16344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:48.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:48 smithi153 ceph-mon[36569]: pgmap v16344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:48.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:48 smithi104 ceph-mon[36624]: pgmap v16344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:50.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:50 smithi153 ceph-mon[36569]: pgmap v16345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:50 smithi104 ceph-mon[36624]: pgmap v16345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:50 smithi081 ceph-mon[31453]: pgmap v16345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:52.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:52 smithi153 ceph-mon[36569]: pgmap v16346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:52 smithi104 ceph-mon[36624]: pgmap v16346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:52 smithi081 ceph-mon[31453]: pgmap v16346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:54.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:54 smithi153 ceph-mon[36569]: pgmap v16347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:54.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:54 smithi104 ceph-mon[36624]: pgmap v16347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:54.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:54 smithi081 ceph-mon[31453]: pgmap v16347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:56.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:56 smithi153 ceph-mon[36569]: pgmap v16348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:56.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:56 smithi104 ceph-mon[36624]: pgmap v16348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:56 smithi081 ceph-mon[31453]: pgmap v16348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:44:58 smithi153 ceph-mon[36569]: pgmap v16349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:44:58 smithi104 ceph-mon[36624]: pgmap v16349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:44:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:44:58 smithi081 ceph-mon[31453]: pgmap v16349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:00.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:00 smithi153 ceph-mon[36569]: pgmap v16350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:00.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:45:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:00 smithi104 ceph-mon[36624]: pgmap v16350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:45:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:00 smithi081 ceph-mon[31453]: pgmap v16350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:00.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:45:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:00.952+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:45:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:00.952+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:45:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:00.952+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:45:02.490 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:02 smithi104 ceph-mon[36624]: pgmap v16351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:02.494 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:02 smithi081 ceph-mon[31453]: pgmap v16351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:02.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:02 smithi153 ceph-mon[36569]: pgmap v16351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:04.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:04 smithi153 ceph-mon[36569]: pgmap v16352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:04.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:04 smithi104 ceph-mon[36624]: pgmap v16352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:04.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:04 smithi081 ceph-mon[31453]: pgmap v16352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:05.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:05.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:05.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:05.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:06.650 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:06 smithi081 ceph-mon[31453]: pgmap v16353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:06.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:06 smithi153 ceph-mon[36569]: pgmap v16353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:06.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:06 smithi104 ceph-mon[36624]: pgmap v16353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:45:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:45:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:07 smithi153 ceph-mon[36569]: pgmap v16354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:07 smithi153 ceph-mon[36569]: pgmap v16355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:08.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:08.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:45:08.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:45:08.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:07 smithi104 ceph-mon[36624]: pgmap v16354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:08.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:07 smithi104 ceph-mon[36624]: pgmap v16355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:08.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:45:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:45:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:07 smithi081 ceph-mon[31453]: pgmap v16354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:07 smithi081 ceph-mon[31453]: pgmap v16355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:45:10.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:09 smithi104 ceph-mon[36624]: pgmap v16356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:09 smithi081 ceph-mon[31453]: pgmap v16356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:10.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:09 smithi153 ceph-mon[36569]: pgmap v16356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:12.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:11 smithi104 ceph-mon[36624]: pgmap v16357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:12.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:11 smithi153 ceph-mon[36569]: pgmap v16357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:12.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:11 smithi081 ceph-mon[31453]: pgmap v16357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:14.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:13 smithi081 ceph-mon[31453]: pgmap v16358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:13 smithi153 ceph-mon[36569]: pgmap v16358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:13 smithi104 ceph-mon[36624]: pgmap v16358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:15.956+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:45:15.959 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:15.957+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:45:15.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:15.957+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:45:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:15 smithi081 ceph-mon[31453]: pgmap v16359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:16.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:15 smithi153 ceph-mon[36569]: pgmap v16359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:15 smithi104 ceph-mon[36624]: pgmap v16359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:18.137 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:17 smithi104 ceph-mon[36624]: pgmap v16360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:17 smithi081 ceph-mon[31453]: pgmap v16360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:18.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:17 smithi153 ceph-mon[36569]: pgmap v16360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:20.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:19 smithi081 ceph-mon[31453]: pgmap v16361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:19 smithi153 ceph-mon[36569]: pgmap v16361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:19 smithi104 ceph-mon[36624]: pgmap v16361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:21 smithi081 ceph-mon[31453]: pgmap v16362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:21 smithi153 ceph-mon[36569]: pgmap v16362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:21 smithi104 ceph-mon[36624]: pgmap v16362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:23 smithi081 ceph-mon[31453]: pgmap v16363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:24.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:23 smithi153 ceph-mon[36569]: pgmap v16363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:23 smithi104 ceph-mon[36624]: pgmap v16363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:25 smithi081 ceph-mon[31453]: pgmap v16364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:25 smithi153 ceph-mon[36569]: pgmap v16364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:25 smithi104 ceph-mon[36624]: pgmap v16364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:27 smithi081 ceph-mon[31453]: pgmap v16365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:28.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:27 smithi153 ceph-mon[36569]: pgmap v16365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:27 smithi104 ceph-mon[36624]: pgmap v16365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:28 smithi153 ceph-mon[36569]: pgmap v16366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:28 smithi104 ceph-mon[36624]: pgmap v16366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:28 smithi081 ceph-mon[31453]: pgmap v16366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:30.972+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:45:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:30.972+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:45:30.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:30.972+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:45:32.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:31 smithi081 ceph-mon[31453]: pgmap v16367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:31 smithi153 ceph-mon[36569]: pgmap v16367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:31 smithi104 ceph-mon[36624]: pgmap v16367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:33 smithi081 ceph-mon[31453]: pgmap v16368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:33 smithi153 ceph-mon[36569]: pgmap v16368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:33 smithi104 ceph-mon[36624]: pgmap v16368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:35 smithi081 ceph-mon[31453]: pgmap v16369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:36.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:35 smithi153 ceph-mon[36569]: pgmap v16369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:35 smithi104 ceph-mon[36624]: pgmap v16369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:37 smithi081 ceph-mon[31453]: pgmap v16370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:37 smithi153 ceph-mon[36569]: pgmap v16370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:37 smithi104 ceph-mon[36624]: pgmap v16370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:39 smithi081 ceph-mon[31453]: pgmap v16371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:39 smithi153 ceph-mon[36569]: pgmap v16371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:39 smithi104 ceph-mon[36624]: pgmap v16371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:42.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:41 smithi081 ceph-mon[31453]: pgmap v16372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:41 smithi153 ceph-mon[36569]: pgmap v16372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:41 smithi104 ceph-mon[36624]: pgmap v16372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:43 smithi081 ceph-mon[31453]: pgmap v16373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:43 smithi153 ceph-mon[36569]: pgmap v16373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:43 smithi104 ceph-mon[36624]: pgmap v16373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:45.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:45.974+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:45:45.977 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:45.974+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:45:45.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:45:45.974+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:45:46.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:45 smithi081 ceph-mon[31453]: pgmap v16374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:45 smithi153 ceph-mon[36569]: pgmap v16374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:45 smithi104 ceph-mon[36624]: pgmap v16374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:48.137 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:47 smithi104 ceph-mon[36624]: pgmap v16375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:47 smithi081 ceph-mon[31453]: pgmap v16375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:47 smithi153 ceph-mon[36569]: pgmap v16375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:49.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:48 smithi153 ceph-mon[36569]: pgmap v16376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:48 smithi104 ceph-mon[36624]: pgmap v16376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:48 smithi081 ceph-mon[31453]: pgmap v16376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:51 smithi081 ceph-mon[31453]: pgmap v16377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:52.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:51 smithi153 ceph-mon[36569]: pgmap v16377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:51 smithi104 ceph-mon[36624]: pgmap v16377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:53 smithi081 ceph-mon[31453]: pgmap v16378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:53 smithi153 ceph-mon[36569]: pgmap v16378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:53 smithi104 ceph-mon[36624]: pgmap v16378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:55 smithi081 ceph-mon[31453]: pgmap v16379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:56.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:55 smithi153 ceph-mon[36569]: pgmap v16379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:56.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:55 smithi104 ceph-mon[36624]: pgmap v16379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:58.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:57 smithi081 ceph-mon[31453]: pgmap v16380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:57 smithi153 ceph-mon[36569]: pgmap v16380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:45:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:57 smithi104 ceph-mon[36624]: pgmap v16380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:45:59 smithi081 ceph-mon[31453]: pgmap v16381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:45:59 smithi153 ceph-mon[36569]: pgmap v16381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:45:59 smithi104 ceph-mon[36624]: pgmap v16381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:00.978+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:46:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:00.978+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:46:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:00.978+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:46:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:01 smithi081 ceph-mon[31453]: pgmap v16382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:01 smithi153 ceph-mon[36569]: pgmap v16382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:01 smithi104 ceph-mon[36624]: pgmap v16382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:03 smithi081 ceph-mon[31453]: pgmap v16383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:03 smithi153 ceph-mon[36569]: pgmap v16383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:03 smithi104 ceph-mon[36624]: pgmap v16383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:05.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:04 smithi153 ceph-mon[36569]: pgmap v16384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:04 smithi104 ceph-mon[36624]: pgmap v16384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:04 smithi081 ceph-mon[31453]: pgmap v16384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:08.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:07 smithi081 ceph-mon[31453]: pgmap v16385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:46:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:07 smithi153 ceph-mon[36569]: pgmap v16385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:46:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:07 smithi104 ceph-mon[36624]: pgmap v16385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:46:08.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:08.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:08 smithi153 ceph-mon[36569]: pgmap v16386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:08 smithi104 ceph-mon[36624]: pgmap v16386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:09.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:09.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:09.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:08 smithi081 ceph-mon[31453]: pgmap v16386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:10.794 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.795 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:10.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:11.798 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:11 smithi153 ceph-mon[36569]: pgmap v16387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:11 smithi104 ceph-mon[36624]: pgmap v16387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:11 smithi081 ceph-mon[31453]: pgmap v16387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:12.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:12.173+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80a5280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000565) for more than 900000000 us, fault. 2024-04-13T05:46:12.175 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:12.173+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80a5280 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f16d40065b0 tx=0x7f16f404c000 comp rx=0 tx=0).stop 2024-04-13T05:46:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:12.173+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 msgr2=0x7f16e80a5280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T05:46:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:12.173+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b3530 0x7f16e80a5280 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T05:46:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:12.173+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 0x7f16e80b1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T05:46:12.176 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:12.174+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 0x7f16e80b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T05:46:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:12.175+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 0x7f16e80b1900 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f40bf000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T05:46:13.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:13.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:13.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:13 smithi081 ceph-mon[31453]: pgmap v16388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:13.963 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:13.963 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:13.963 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:13 smithi153 ceph-mon[36569]: pgmap v16388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:14.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:13 smithi104 ceph-mon[36624]: pgmap v16388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:15.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:46:15.796 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:46:15.797 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:15 smithi081 ceph-mon[31453]: pgmap v16389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:15.797 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:15 smithi081 ceph-mon[31453]: pgmap v16390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:15.797 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:46:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:46:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:15 smithi153 ceph-mon[36569]: pgmap v16389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:15 smithi153 ceph-mon[36569]: pgmap v16390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:46:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:46:15.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:15 smithi104 ceph-mon[36624]: pgmap v16389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:15.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:15 smithi104 ceph-mon[36624]: pgmap v16390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:15.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:46:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:15.984+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:46:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:15.984+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:46:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:15.984+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:46:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:17.762+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:46:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:17.762+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:46:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:17.763+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 236 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:46:17.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:17.763+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 236 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:46:17.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:17 smithi081 ceph-mon[31453]: pgmap v16391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:18.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:17 smithi153 ceph-mon[36569]: pgmap v16391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:17 smithi104 ceph-mon[36624]: pgmap v16391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:19 smithi081 ceph-mon[31453]: pgmap v16392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:20.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:19 smithi153 ceph-mon[36569]: pgmap v16392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:20.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:19 smithi104 ceph-mon[36624]: pgmap v16392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:21 smithi081 ceph-mon[31453]: pgmap v16393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:22.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:21 smithi153 ceph-mon[36569]: pgmap v16393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:21 smithi104 ceph-mon[36624]: pgmap v16393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:23 smithi081 ceph-mon[31453]: pgmap v16394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:24.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:23 smithi153 ceph-mon[36569]: pgmap v16394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:24.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:23 smithi104 ceph-mon[36624]: pgmap v16394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:26.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:25 smithi153 ceph-mon[36569]: pgmap v16395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:25 smithi104 ceph-mon[36624]: pgmap v16395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:25 smithi081 ceph-mon[31453]: pgmap v16395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:28.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:27 smithi153 ceph-mon[36569]: pgmap v16396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:28.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:27 smithi104 ceph-mon[36624]: pgmap v16396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:27 smithi081 ceph-mon[31453]: pgmap v16396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:29 smithi104 ceph-mon[36624]: pgmap v16397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:30.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:29 smithi153 ceph-mon[36569]: pgmap v16397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:29 smithi081 ceph-mon[31453]: pgmap v16397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:31.000+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:46:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:31.000+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:46:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:31.000+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:46:32.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:31 smithi153 ceph-mon[36569]: pgmap v16398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:32.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:31 smithi104 ceph-mon[36624]: pgmap v16398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:32.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:31 smithi081 ceph-mon[31453]: pgmap v16398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:34.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:33 smithi153 ceph-mon[36569]: pgmap v16399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:33 smithi104 ceph-mon[36624]: pgmap v16399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:33 smithi081 ceph-mon[31453]: pgmap v16399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:36.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:35 smithi153 ceph-mon[36569]: pgmap v16400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:36.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:35 smithi104 ceph-mon[36624]: pgmap v16400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:35 smithi081 ceph-mon[31453]: pgmap v16400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:38.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:37 smithi153 ceph-mon[36569]: pgmap v16401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:38.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:37 smithi104 ceph-mon[36624]: pgmap v16401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:37 smithi081 ceph-mon[31453]: pgmap v16401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:40.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:39 smithi153 ceph-mon[36569]: pgmap v16402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:40.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:39 smithi104 ceph-mon[36624]: pgmap v16402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:40.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:39 smithi081 ceph-mon[31453]: pgmap v16402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:42.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:41 smithi153 ceph-mon[36569]: pgmap v16403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:42.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:41 smithi104 ceph-mon[36624]: pgmap v16403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:42.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:41 smithi081 ceph-mon[31453]: pgmap v16403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:44.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:43 smithi153 ceph-mon[36569]: pgmap v16404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:44.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:43 smithi104 ceph-mon[36624]: pgmap v16404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:44.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:43 smithi081 ceph-mon[31453]: pgmap v16404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:46.007+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:46:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:46.007+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:46:46.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:46:46.007+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:46:46.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:45 smithi153 ceph-mon[36569]: pgmap v16405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:46.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:45 smithi104 ceph-mon[36624]: pgmap v16405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:46.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:45 smithi081 ceph-mon[31453]: pgmap v16405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:47 smithi153 ceph-mon[36569]: pgmap v16406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:48.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:47 smithi104 ceph-mon[36624]: pgmap v16406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:47 smithi081 ceph-mon[31453]: pgmap v16406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:50.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:49 smithi153 ceph-mon[36569]: pgmap v16407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:50.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:49 smithi104 ceph-mon[36624]: pgmap v16407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:49 smithi081 ceph-mon[31453]: pgmap v16407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:52.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:51 smithi153 ceph-mon[36569]: pgmap v16408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:51 smithi104 ceph-mon[36624]: pgmap v16408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:52.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:51 smithi081 ceph-mon[31453]: pgmap v16408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:54.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:53 smithi153 ceph-mon[36569]: pgmap v16409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:53 smithi104 ceph-mon[36624]: pgmap v16409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:53 smithi081 ceph-mon[31453]: pgmap v16409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:55 smithi153 ceph-mon[36569]: pgmap v16410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:55 smithi104 ceph-mon[36624]: pgmap v16410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:55 smithi081 ceph-mon[31453]: pgmap v16410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:58.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:57 smithi153 ceph-mon[36569]: pgmap v16411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:57 smithi104 ceph-mon[36624]: pgmap v16411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:46:58.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:57 smithi081 ceph-mon[31453]: pgmap v16411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:46:59 smithi153 ceph-mon[36569]: pgmap v16412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:46:59 smithi104 ceph-mon[36624]: pgmap v16412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:46:59 smithi081 ceph-mon[31453]: pgmap v16412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:01.014+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:47:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:01.014+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:47:01.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:01.014+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:47:02.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:01 smithi153 ceph-mon[36569]: pgmap v16413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:01 smithi104 ceph-mon[36624]: pgmap v16413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:01 smithi081 ceph-mon[31453]: pgmap v16413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:04.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:03 smithi153 ceph-mon[36569]: pgmap v16414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:04.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:03 smithi104 ceph-mon[36624]: pgmap v16414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:03 smithi081 ceph-mon[31453]: pgmap v16414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:06.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:05 smithi153 ceph-mon[36569]: pgmap v16415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:05 smithi104 ceph-mon[36624]: pgmap v16415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:05 smithi081 ceph-mon[31453]: pgmap v16415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:07 smithi153 ceph-mon[36569]: pgmap v16416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:07 smithi104 ceph-mon[36624]: pgmap v16416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:08.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:07 smithi081 ceph-mon[31453]: pgmap v16416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:10.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:09 smithi153 ceph-mon[36569]: pgmap v16417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:10.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:09 smithi104 ceph-mon[36624]: pgmap v16417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:09 smithi081 ceph-mon[31453]: pgmap v16417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:12.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:11 smithi153 ceph-mon[36569]: pgmap v16418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:12.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:11 smithi104 ceph-mon[36624]: pgmap v16418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:12.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:11 smithi081 ceph-mon[31453]: pgmap v16418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:14.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:13 smithi153 ceph-mon[36569]: pgmap v16419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:13 smithi104 ceph-mon[36624]: pgmap v16419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:13 smithi081 ceph-mon[31453]: pgmap v16419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:16.024+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:47:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:16.024+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:47:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:16.024+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:47:16.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:15 smithi153 ceph-mon[36569]: pgmap v16420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:16.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:47:16.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:16.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:15 smithi104 ceph-mon[36624]: pgmap v16420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:16.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:47:16.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:16.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:15 smithi081 ceph-mon[31453]: pgmap v16420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:16.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:47:16.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:16.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:17.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:47:17.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:47:17.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:17.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:47:17.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:47:17.061 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:17.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:47:17.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:47:17.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:47:18.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:17 smithi153 ceph-mon[36569]: pgmap v16421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:18.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:17 smithi153 ceph-mon[36569]: pgmap v16422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:17 smithi104 ceph-mon[36624]: pgmap v16421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:18.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:17 smithi104 ceph-mon[36624]: pgmap v16422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:18.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:17 smithi081 ceph-mon[31453]: pgmap v16421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:18.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:17 smithi081 ceph-mon[31453]: pgmap v16422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:20.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:19 smithi104 ceph-mon[36624]: pgmap v16423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:20.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:19 smithi081 ceph-mon[31453]: pgmap v16423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:19 smithi153 ceph-mon[36569]: pgmap v16423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:21 smithi104 ceph-mon[36624]: pgmap v16424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:21 smithi081 ceph-mon[31453]: pgmap v16424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:21 smithi153 ceph-mon[36569]: pgmap v16424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:24.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:23 smithi104 ceph-mon[36624]: pgmap v16425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:23 smithi081 ceph-mon[31453]: pgmap v16425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:24.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:23 smithi153 ceph-mon[36569]: pgmap v16425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:25 smithi104 ceph-mon[36624]: pgmap v16426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:25 smithi081 ceph-mon[31453]: pgmap v16426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:25 smithi153 ceph-mon[36569]: pgmap v16426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:28.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:27 smithi104 ceph-mon[36624]: pgmap v16427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:27 smithi081 ceph-mon[31453]: pgmap v16427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:28.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:27 smithi153 ceph-mon[36569]: pgmap v16427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:29 smithi081 ceph-mon[31453]: pgmap v16428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:29 smithi153 ceph-mon[36569]: pgmap v16428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:29 smithi104 ceph-mon[36624]: pgmap v16428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:31.036 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:31.033+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:47:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:31.033+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:47:31.037 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:31.033+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:47:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:31 smithi081 ceph-mon[31453]: pgmap v16429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:31 smithi153 ceph-mon[36569]: pgmap v16429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:31 smithi104 ceph-mon[36624]: pgmap v16429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:33 smithi081 ceph-mon[31453]: pgmap v16430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:33 smithi153 ceph-mon[36569]: pgmap v16430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:33 smithi104 ceph-mon[36624]: pgmap v16430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:35 smithi081 ceph-mon[31453]: pgmap v16431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:36.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:35 smithi153 ceph-mon[36569]: pgmap v16431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:35 smithi104 ceph-mon[36624]: pgmap v16431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:38.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:37 smithi081 ceph-mon[31453]: pgmap v16432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:37 smithi153 ceph-mon[36569]: pgmap v16432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:37 smithi104 ceph-mon[36624]: pgmap v16432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:39 smithi081 ceph-mon[31453]: pgmap v16433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:40.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:39 smithi153 ceph-mon[36569]: pgmap v16433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:39 smithi104 ceph-mon[36624]: pgmap v16433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:42.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:41 smithi081 ceph-mon[31453]: pgmap v16434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:41 smithi153 ceph-mon[36569]: pgmap v16434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:41 smithi104 ceph-mon[36624]: pgmap v16434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:43 smithi081 ceph-mon[31453]: pgmap v16435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:43 smithi153 ceph-mon[36569]: pgmap v16435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:43 smithi104 ceph-mon[36624]: pgmap v16435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:46.048+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:47:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:46.048+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:47:46.051 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:46.048+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:47:46.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:45 smithi081 ceph-mon[31453]: pgmap v16436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:45 smithi153 ceph-mon[36569]: pgmap v16436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:45 smithi104 ceph-mon[36624]: pgmap v16436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:47.780 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:47.777+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:47:47.780 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:47.777+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:47:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:47.778+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 237 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:47:47.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:47:47.778+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 237 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:47:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:47 smithi081 ceph-mon[31453]: pgmap v16437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:47 smithi153 ceph-mon[36569]: pgmap v16437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:47 smithi104 ceph-mon[36624]: pgmap v16437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:50.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:49 smithi081 ceph-mon[31453]: pgmap v16438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:49 smithi153 ceph-mon[36569]: pgmap v16438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:50.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:49 smithi104 ceph-mon[36624]: pgmap v16438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:51 smithi081 ceph-mon[31453]: pgmap v16439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:52.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:51 smithi153 ceph-mon[36569]: pgmap v16439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:51 smithi104 ceph-mon[36624]: pgmap v16439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:53 smithi081 ceph-mon[31453]: pgmap v16440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:53 smithi153 ceph-mon[36569]: pgmap v16440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:53 smithi104 ceph-mon[36624]: pgmap v16440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:55 smithi081 ceph-mon[31453]: pgmap v16441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:56.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:55 smithi153 ceph-mon[36569]: pgmap v16441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:55 smithi104 ceph-mon[36624]: pgmap v16441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:57 smithi153 ceph-mon[36569]: pgmap v16442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:58.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:57 smithi104 ceph-mon[36624]: pgmap v16442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:57 smithi081 ceph-mon[31453]: pgmap v16442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:59.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:47:58 smithi153 ceph-mon[36569]: pgmap v16443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:47:58 smithi104 ceph-mon[36624]: pgmap v16443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:47:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:47:58 smithi081 ceph-mon[31453]: pgmap v16443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:01.062+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:48:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:01.062+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:48:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:01.062+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:48:01.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:01 smithi153 ceph-mon[36569]: pgmap v16444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:01.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:01 smithi104 ceph-mon[36624]: pgmap v16444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:01.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:01 smithi081 ceph-mon[31453]: pgmap v16444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:03 smithi081 ceph-mon[31453]: pgmap v16445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:03.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:03 smithi153 ceph-mon[36569]: pgmap v16445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:03.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:03 smithi104 ceph-mon[36624]: pgmap v16445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:05.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:05 smithi153 ceph-mon[36569]: pgmap v16446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:05.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:05 smithi104 ceph-mon[36624]: pgmap v16446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:05 smithi081 ceph-mon[31453]: pgmap v16446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:07 smithi153 ceph-mon[36569]: pgmap v16447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:07 smithi104 ceph-mon[36624]: pgmap v16447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:07 smithi081 ceph-mon[31453]: pgmap v16447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:09.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:09 smithi153 ceph-mon[36569]: pgmap v16448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:09.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:09 smithi104 ceph-mon[36624]: pgmap v16448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:09.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:09 smithi081 ceph-mon[31453]: pgmap v16448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:11.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:11 smithi153 ceph-mon[36569]: pgmap v16449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:11.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:11 smithi104 ceph-mon[36624]: pgmap v16449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:11.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:11 smithi081 ceph-mon[31453]: pgmap v16449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:13.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:13 smithi153 ceph-mon[36569]: pgmap v16450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:13.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:13 smithi104 ceph-mon[36624]: pgmap v16450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:13.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:13 smithi081 ceph-mon[31453]: pgmap v16450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:15.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:15 smithi104 ceph-mon[36624]: pgmap v16451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:15.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:15 smithi081 ceph-mon[31453]: pgmap v16451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:15.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:15 smithi153 ceph-mon[36569]: pgmap v16451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:16.074+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:48:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:16.074+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:48:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:16.074+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:48:17.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:17 smithi104 ceph-mon[36624]: pgmap v16452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:17.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:48:17.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:17 smithi081 ceph-mon[31453]: pgmap v16452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:17.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:48:17.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:17 smithi153 ceph-mon[36569]: pgmap v16452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:17.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:48:18.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:48:18.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:48:18.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:18 smithi104 ceph-mon[36624]: pgmap v16453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:18.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:18 smithi104 ceph-mon[36624]: pgmap v16454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:18.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:48:18.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:48:18.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:48:18.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:18 smithi081 ceph-mon[31453]: pgmap v16453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:18.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:18 smithi081 ceph-mon[31453]: pgmap v16454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:18.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:48:18.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:48:18.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:48:18.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:18 smithi153 ceph-mon[36569]: pgmap v16453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:18.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:18 smithi153 ceph-mon[36569]: pgmap v16454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:18.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:48:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:20 smithi081 ceph-mon[31453]: pgmap v16455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:21.288 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:20 smithi104 ceph-mon[36624]: pgmap v16455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:20 smithi153 ceph-mon[36569]: pgmap v16455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:23.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:22 smithi081 ceph-mon[31453]: pgmap v16456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:22 smithi153 ceph-mon[36569]: pgmap v16456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:22 smithi104 ceph-mon[36624]: pgmap v16456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:24 smithi081 ceph-mon[31453]: pgmap v16457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:25.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:24 smithi153 ceph-mon[36569]: pgmap v16457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:24 smithi104 ceph-mon[36624]: pgmap v16457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:26 smithi081 ceph-mon[31453]: pgmap v16458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:26 smithi153 ceph-mon[36569]: pgmap v16458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:26 smithi104 ceph-mon[36624]: pgmap v16458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:28 smithi081 ceph-mon[31453]: pgmap v16459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:28 smithi153 ceph-mon[36569]: pgmap v16459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:28 smithi104 ceph-mon[36624]: pgmap v16459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:31.089+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:48:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:31.089+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:48:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:31.089+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:48:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:30 smithi081 ceph-mon[31453]: pgmap v16460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:31.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:30 smithi153 ceph-mon[36569]: pgmap v16460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:30 smithi104 ceph-mon[36624]: pgmap v16460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:32 smithi081 ceph-mon[31453]: pgmap v16461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:32 smithi153 ceph-mon[36569]: pgmap v16461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:33.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:32 smithi104 ceph-mon[36624]: pgmap v16461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:34 smithi153 ceph-mon[36569]: pgmap v16462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:35.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:34 smithi104 ceph-mon[36624]: pgmap v16462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:34 smithi081 ceph-mon[31453]: pgmap v16462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:36 smithi081 ceph-mon[31453]: pgmap v16463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:37.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:36 smithi153 ceph-mon[36569]: pgmap v16463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:36 smithi104 ceph-mon[36624]: pgmap v16463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:38 smithi081 ceph-mon[31453]: pgmap v16464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:39.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:38 smithi153 ceph-mon[36569]: pgmap v16464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:39.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:38 smithi104 ceph-mon[36624]: pgmap v16464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:41.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:40 smithi081 ceph-mon[31453]: pgmap v16465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:40 smithi153 ceph-mon[36569]: pgmap v16465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:40 smithi104 ceph-mon[36624]: pgmap v16465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:42 smithi081 ceph-mon[31453]: pgmap v16466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:43.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:42 smithi153 ceph-mon[36569]: pgmap v16466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:42 smithi104 ceph-mon[36624]: pgmap v16466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:45.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:44 smithi081 ceph-mon[31453]: pgmap v16467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:44 smithi153 ceph-mon[36569]: pgmap v16467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:44 smithi104 ceph-mon[36624]: pgmap v16467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:46.103+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:48:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:46.103+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:48:46.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:48:46.103+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:48:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:46 smithi081 ceph-mon[31453]: pgmap v16468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:46 smithi153 ceph-mon[36569]: pgmap v16468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:47.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:46 smithi104 ceph-mon[36624]: pgmap v16468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:48 smithi081 ceph-mon[31453]: pgmap v16469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:49.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:48 smithi153 ceph-mon[36569]: pgmap v16469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:48 smithi104 ceph-mon[36624]: pgmap v16469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:51.137 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:50 smithi104 ceph-mon[36624]: pgmap v16470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:50 smithi081 ceph-mon[31453]: pgmap v16470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:51.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:50 smithi153 ceph-mon[36569]: pgmap v16470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:53.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:52 smithi081 ceph-mon[31453]: pgmap v16471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:52 smithi153 ceph-mon[36569]: pgmap v16471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:52 smithi104 ceph-mon[36624]: pgmap v16471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:54 smithi081 ceph-mon[31453]: pgmap v16472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:55.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:54 smithi153 ceph-mon[36569]: pgmap v16472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:54 smithi104 ceph-mon[36624]: pgmap v16472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:56 smithi081 ceph-mon[31453]: pgmap v16473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:56 smithi153 ceph-mon[36569]: pgmap v16473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:56 smithi104 ceph-mon[36624]: pgmap v16473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:48:58 smithi081 ceph-mon[31453]: pgmap v16474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:59.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:48:58 smithi153 ceph-mon[36569]: pgmap v16474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:48:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:48:58 smithi104 ceph-mon[36624]: pgmap v16474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:01.119+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:49:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:01.119+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:49:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:01.119+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:49:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:00 smithi153 ceph-mon[36569]: pgmap v16475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:00 smithi104 ceph-mon[36624]: pgmap v16475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:01.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:00 smithi081 ceph-mon[31453]: pgmap v16475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:03.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:02 smithi153 ceph-mon[36569]: pgmap v16476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:03.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:02 smithi104 ceph-mon[36624]: pgmap v16476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:03.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:02 smithi081 ceph-mon[31453]: pgmap v16476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:05.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:04 smithi153 ceph-mon[36569]: pgmap v16477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:05.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:04 smithi104 ceph-mon[36624]: pgmap v16477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:04 smithi081 ceph-mon[31453]: pgmap v16477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:05 smithi153 ceph-mon[36569]: pgmap v16478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:05 smithi104 ceph-mon[36624]: pgmap v16478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:05 smithi081 ceph-mon[31453]: pgmap v16478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:09.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:08 smithi081 ceph-mon[31453]: pgmap v16479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:08 smithi153 ceph-mon[36569]: pgmap v16479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:09.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:08 smithi104 ceph-mon[36624]: pgmap v16479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:11.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:10 smithi081 ceph-mon[31453]: pgmap v16480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:11.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:10 smithi153 ceph-mon[36569]: pgmap v16480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:11.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:10 smithi104 ceph-mon[36624]: pgmap v16480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:13.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:12 smithi081 ceph-mon[31453]: pgmap v16481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:12 smithi153 ceph-mon[36569]: pgmap v16481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:13.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:12 smithi104 ceph-mon[36624]: pgmap v16481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:14 smithi081 ceph-mon[31453]: pgmap v16482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:15.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:14 smithi153 ceph-mon[36569]: pgmap v16482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:15.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:14 smithi104 ceph-mon[36624]: pgmap v16482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:16.124+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:16.124+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:49:16.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:16.124+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:49:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:16 smithi081 ceph-mon[31453]: pgmap v16483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:16 smithi153 ceph-mon[36569]: pgmap v16483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:16 smithi104 ceph-mon[36624]: pgmap v16483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:17.792+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:49:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:17.792+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:49:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:17.793+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 238 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:49:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:17.793+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 238 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:49:19.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:18 smithi081 ceph-mon[31453]: pgmap v16484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:19.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:49:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:18 smithi153 ceph-mon[36569]: pgmap v16484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:49:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:18 smithi104 ceph-mon[36624]: pgmap v16484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:49:20.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:49:20.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:49:20.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:49:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:49:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:49:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:49:20.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:49:20.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:49:20.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:49:21.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:20 smithi104 ceph-mon[36624]: pgmap v16485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:21.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:20 smithi104 ceph-mon[36624]: pgmap v16486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:20 smithi153 ceph-mon[36569]: pgmap v16485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:20 smithi153 ceph-mon[36569]: pgmap v16486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:21.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:20 smithi081 ceph-mon[31453]: pgmap v16485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:21.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:20 smithi081 ceph-mon[31453]: pgmap v16486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:22 smithi153 ceph-mon[36569]: pgmap v16487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:22 smithi104 ceph-mon[36624]: pgmap v16487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:23.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:22 smithi081 ceph-mon[31453]: pgmap v16487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:25.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:24 smithi153 ceph-mon[36569]: pgmap v16488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:25.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:24 smithi104 ceph-mon[36624]: pgmap v16488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:24 smithi081 ceph-mon[31453]: pgmap v16488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:25 smithi153 ceph-mon[36569]: pgmap v16489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:26.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:25 smithi104 ceph-mon[36624]: pgmap v16489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:25 smithi081 ceph-mon[31453]: pgmap v16489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:28.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:28 smithi081 ceph-mon[31453]: pgmap v16490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:28.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:28 smithi153 ceph-mon[36569]: pgmap v16490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:28.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:28 smithi104 ceph-mon[36624]: pgmap v16490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:30.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:30 smithi153 ceph-mon[36569]: pgmap v16491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:30.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:30 smithi104 ceph-mon[36624]: pgmap v16491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:30.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:30 smithi081 ceph-mon[31453]: pgmap v16491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:31.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:31.140+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:49:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:31.140+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:49:31.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:31.140+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:49:32.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:32 smithi153 ceph-mon[36569]: pgmap v16492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:32.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:32 smithi104 ceph-mon[36624]: pgmap v16492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:32.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:32 smithi081 ceph-mon[31453]: pgmap v16492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:34.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:34 smithi153 ceph-mon[36569]: pgmap v16493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:34 smithi104 ceph-mon[36624]: pgmap v16493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:34 smithi081 ceph-mon[31453]: pgmap v16493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:36.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:36 smithi153 ceph-mon[36569]: pgmap v16494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:36 smithi104 ceph-mon[36624]: pgmap v16494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:36.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:36 smithi081 ceph-mon[31453]: pgmap v16494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:38.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:38 smithi153 ceph-mon[36569]: pgmap v16495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:38 smithi104 ceph-mon[36624]: pgmap v16495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:38.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:38 smithi081 ceph-mon[31453]: pgmap v16495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:40.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:40 smithi153 ceph-mon[36569]: pgmap v16496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:40.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:40 smithi104 ceph-mon[36624]: pgmap v16496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:40.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:40 smithi081 ceph-mon[31453]: pgmap v16496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:42.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:42 smithi153 ceph-mon[36569]: pgmap v16497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:42.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:42 smithi104 ceph-mon[36624]: pgmap v16497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:42 smithi081 ceph-mon[31453]: pgmap v16497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:44.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:44 smithi153 ceph-mon[36569]: pgmap v16498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:44.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:44 smithi104 ceph-mon[36624]: pgmap v16498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:44.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:44 smithi081 ceph-mon[31453]: pgmap v16498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:46.149+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:49:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:46.149+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:49:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:49:46.149+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:49:46.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:46 smithi153 ceph-mon[36569]: pgmap v16499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:46.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:46 smithi104 ceph-mon[36624]: pgmap v16499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:46.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:46 smithi081 ceph-mon[31453]: pgmap v16499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:48.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:48 smithi153 ceph-mon[36569]: pgmap v16500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:48 smithi104 ceph-mon[36624]: pgmap v16500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:48.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:48 smithi081 ceph-mon[31453]: pgmap v16500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:50.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:50 smithi153 ceph-mon[36569]: pgmap v16501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:50.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:50 smithi104 ceph-mon[36624]: pgmap v16501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:50.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:50 smithi081 ceph-mon[31453]: pgmap v16501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:52.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:52 smithi153 ceph-mon[36569]: pgmap v16502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:52.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:52 smithi104 ceph-mon[36624]: pgmap v16502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:52.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:52 smithi081 ceph-mon[31453]: pgmap v16502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:54.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:54 smithi153 ceph-mon[36569]: pgmap v16503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:54.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:54 smithi104 ceph-mon[36624]: pgmap v16503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:54.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:54 smithi081 ceph-mon[31453]: pgmap v16503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:56.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:56 smithi153 ceph-mon[36569]: pgmap v16504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:56 smithi104 ceph-mon[36624]: pgmap v16504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:56.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:56 smithi081 ceph-mon[31453]: pgmap v16504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:58.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:49:58 smithi153 ceph-mon[36569]: pgmap v16505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:49:58 smithi104 ceph-mon[36624]: pgmap v16505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:49:58.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:49:58 smithi081 ceph-mon[31453]: pgmap v16505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:00.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:00 smithi153 ceph-mon[36569]: pgmap v16506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:00.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:50:00.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:00 smithi104 ceph-mon[36624]: pgmap v16506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:00.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:50:00.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:00 smithi081 ceph-mon[31453]: pgmap v16506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:00.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T05:50:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:01.157+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:50:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:01.157+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:50:01.161 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:01.157+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:50:02.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:02 smithi153 ceph-mon[36569]: pgmap v16507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:02 smithi104 ceph-mon[36624]: pgmap v16507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:02.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:02 smithi081 ceph-mon[31453]: pgmap v16507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:04.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:04 smithi153 ceph-mon[36569]: pgmap v16508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:04.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:04 smithi104 ceph-mon[36624]: pgmap v16508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:04.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:04 smithi081 ceph-mon[31453]: pgmap v16508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:06.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:06 smithi153 ceph-mon[36569]: pgmap v16509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:06.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:06 smithi104 ceph-mon[36624]: pgmap v16509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:06.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:06 smithi081 ceph-mon[31453]: pgmap v16509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:08.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:08 smithi153 ceph-mon[36569]: pgmap v16510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:08.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:08 smithi104 ceph-mon[36624]: pgmap v16510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:08.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:08 smithi081 ceph-mon[31453]: pgmap v16510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:10.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:10 smithi153 ceph-mon[36569]: pgmap v16511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:10.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:10 smithi104 ceph-mon[36624]: pgmap v16511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:10.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:10 smithi081 ceph-mon[31453]: pgmap v16511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:12.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:12 smithi153 ceph-mon[36569]: pgmap v16512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:12.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:12 smithi104 ceph-mon[36624]: pgmap v16512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:12.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:12 smithi081 ceph-mon[31453]: pgmap v16512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:14.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:14 smithi153 ceph-mon[36569]: pgmap v16513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:14.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:14 smithi104 ceph-mon[36624]: pgmap v16513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:14.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:14 smithi081 ceph-mon[31453]: pgmap v16513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:16.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:16.171+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:50:16.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:16.171+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:50:16.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:16.171+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:50:16.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:16 smithi153 ceph-mon[36569]: pgmap v16514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:16 smithi104 ceph-mon[36624]: pgmap v16514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:16.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:16 smithi081 ceph-mon[31453]: pgmap v16514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:18.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:18 smithi153 ceph-mon[36569]: pgmap v16515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:18.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:18 smithi104 ceph-mon[36624]: pgmap v16515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:18.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:18 smithi081 ceph-mon[31453]: pgmap v16515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:20.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:20 smithi153 ceph-mon[36569]: pgmap v16516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:20.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:50:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:20 smithi104 ceph-mon[36624]: pgmap v16516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:50:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:20 smithi081 ceph-mon[31453]: pgmap v16516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:20.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:50:21.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:50:21.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:50:21.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:21 smithi153 ceph-mon[36569]: pgmap v16517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:21.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:21 smithi153 ceph-mon[36569]: pgmap v16518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:21.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:50:21.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:50:21.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:50:21.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:21 smithi081 ceph-mon[31453]: pgmap v16517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:21.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:21 smithi081 ceph-mon[31453]: pgmap v16518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:21.811 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:50:21.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:50:21.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:50:21.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:21 smithi104 ceph-mon[36624]: pgmap v16517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:21.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:21 smithi104 ceph-mon[36624]: pgmap v16518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:21.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:50:24.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:23 smithi153 ceph-mon[36569]: pgmap v16519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:23 smithi104 ceph-mon[36624]: pgmap v16519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:24.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:23 smithi081 ceph-mon[31453]: pgmap v16519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:26.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:25 smithi104 ceph-mon[36624]: pgmap v16520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:25 smithi153 ceph-mon[36569]: pgmap v16520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:26.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:25 smithi081 ceph-mon[31453]: pgmap v16520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:28.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:27 smithi153 ceph-mon[36569]: pgmap v16521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:27 smithi104 ceph-mon[36624]: pgmap v16521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:28.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:27 smithi081 ceph-mon[31453]: pgmap v16521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:29 smithi153 ceph-mon[36569]: pgmap v16522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:29 smithi104 ceph-mon[36624]: pgmap v16522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:30.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:29 smithi081 ceph-mon[31453]: pgmap v16522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:31.180+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:50:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:31.180+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:50:31.183 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:31.180+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:50:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:31 smithi153 ceph-mon[36569]: pgmap v16523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:31 smithi104 ceph-mon[36624]: pgmap v16523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:32.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:31 smithi081 ceph-mon[31453]: pgmap v16523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:33 smithi153 ceph-mon[36569]: pgmap v16524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:33 smithi104 ceph-mon[36624]: pgmap v16524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:34.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:33 smithi081 ceph-mon[31453]: pgmap v16524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:35.270 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:34 smithi104 ceph-mon[36624]: pgmap v16525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:35.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:34 smithi153 ceph-mon[36569]: pgmap v16525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:34 smithi081 ceph-mon[31453]: pgmap v16525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:38.219 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:37 smithi081 ceph-mon[31453]: pgmap v16526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:37 smithi153 ceph-mon[36569]: pgmap v16526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:38.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:37 smithi104 ceph-mon[36624]: pgmap v16526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:39 smithi153 ceph-mon[36569]: pgmap v16527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:40.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:39 smithi104 ceph-mon[36624]: pgmap v16527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:40.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:39 smithi081 ceph-mon[31453]: pgmap v16527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:41 smithi153 ceph-mon[36569]: pgmap v16528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:42.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:41 smithi104 ceph-mon[36624]: pgmap v16528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:42.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:41 smithi081 ceph-mon[31453]: pgmap v16528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:43 smithi153 ceph-mon[36569]: pgmap v16529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:43 smithi104 ceph-mon[36624]: pgmap v16529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:44.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:43 smithi081 ceph-mon[31453]: pgmap v16529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:46.196+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:50:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:46.196+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:50:46.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:50:46.196+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:50:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:45 smithi153 ceph-mon[36569]: pgmap v16530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:45 smithi104 ceph-mon[36624]: pgmap v16530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:45 smithi081 ceph-mon[31453]: pgmap v16530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:46 smithi153 ceph-mon[36569]: pgmap v16531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:46 smithi104 ceph-mon[36624]: pgmap v16531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:47.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:46 smithi081 ceph-mon[31453]: pgmap v16531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:49 smithi153 ceph-mon[36569]: pgmap v16532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:49 smithi104 ceph-mon[36624]: pgmap v16532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:50.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:49 smithi081 ceph-mon[31453]: pgmap v16532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:52.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:51 smithi081 ceph-mon[31453]: pgmap v16533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:52.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:51 smithi153 ceph-mon[36569]: pgmap v16533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:51 smithi104 ceph-mon[36624]: pgmap v16533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:53 smithi153 ceph-mon[36569]: pgmap v16534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:54.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:53 smithi104 ceph-mon[36624]: pgmap v16534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:54.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:53 smithi081 ceph-mon[31453]: pgmap v16534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:56.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:55 smithi153 ceph-mon[36569]: pgmap v16535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:55 smithi104 ceph-mon[36624]: pgmap v16535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:56.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:55 smithi081 ceph-mon[31453]: pgmap v16535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:56 smithi153 ceph-mon[36569]: pgmap v16536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:56 smithi104 ceph-mon[36624]: pgmap v16536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:50:57.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:56 smithi081 ceph-mon[31453]: pgmap v16536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:50:59 smithi153 ceph-mon[36569]: pgmap v16537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:50:59 smithi104 ceph-mon[36624]: pgmap v16537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:00.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:50:59 smithi081 ceph-mon[31453]: pgmap v16537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:01.211+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:51:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:01.212+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:51:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:01.212+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:51:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:01 smithi153 ceph-mon[36569]: pgmap v16538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:01 smithi104 ceph-mon[36624]: pgmap v16538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:02.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:01 smithi081 ceph-mon[31453]: pgmap v16538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:03 smithi153 ceph-mon[36569]: pgmap v16539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:04.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:03 smithi104 ceph-mon[36624]: pgmap v16539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:04.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:03 smithi081 ceph-mon[31453]: pgmap v16539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:05.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:04 smithi153 ceph-mon[36569]: pgmap v16540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:05.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:04 smithi104 ceph-mon[36624]: pgmap v16540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:05.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:04 smithi081 ceph-mon[31453]: pgmap v16540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:07 smithi153 ceph-mon[36569]: pgmap v16541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:07 smithi104 ceph-mon[36624]: pgmap v16541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:08.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:07 smithi081 ceph-mon[31453]: pgmap v16541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:10.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:09 smithi153 ceph-mon[36569]: pgmap v16542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:10.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:09 smithi104 ceph-mon[36624]: pgmap v16542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:10.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:09 smithi081 ceph-mon[31453]: pgmap v16542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:11.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:10 smithi153 ceph-mon[36569]: pgmap v16543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:11.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:10 smithi104 ceph-mon[36624]: pgmap v16543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:11.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:10 smithi081 ceph-mon[31453]: pgmap v16543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:13 smithi153 ceph-mon[36569]: pgmap v16544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:14.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:13 smithi104 ceph-mon[36624]: pgmap v16544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:13 smithi081 ceph-mon[31453]: pgmap v16544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:16.227+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:51:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:16.227+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:51:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:16.227+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:51:16.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:15 smithi153 ceph-mon[36569]: pgmap v16545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:15 smithi104 ceph-mon[36624]: pgmap v16545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:16.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:15 smithi081 ceph-mon[31453]: pgmap v16545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:16 smithi153 ceph-mon[36569]: pgmap v16546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:17.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:16 smithi104 ceph-mon[36624]: pgmap v16546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:17.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:16 smithi081 ceph-mon[31453]: pgmap v16546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:19 smithi153 ceph-mon[36569]: pgmap v16547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:19 smithi104 ceph-mon[36624]: pgmap v16547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:20.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:19 smithi081 ceph-mon[31453]: pgmap v16547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:22.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:21 smithi081 ceph-mon[31453]: pgmap v16548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:22.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:51:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:21 smithi153 ceph-mon[36569]: pgmap v16548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:51:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:21 smithi104 ceph-mon[36624]: pgmap v16548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:22.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:51:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:51:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:51:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:22 smithi153 ceph-mon[36569]: pgmap v16549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:22 smithi153 ceph-mon[36569]: pgmap v16550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:23.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:51:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:51:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:51:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:22 smithi104 ceph-mon[36624]: pgmap v16549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:22 smithi104 ceph-mon[36624]: pgmap v16550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:51:23.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:51:23.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:51:23.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:22 smithi081 ceph-mon[31453]: pgmap v16549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:23.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:22 smithi081 ceph-mon[31453]: pgmap v16550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:23.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:51:25.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:25 smithi153 ceph-mon[36569]: pgmap v16551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:25.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:25 smithi104 ceph-mon[36624]: pgmap v16551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:25.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:25 smithi081 ceph-mon[31453]: pgmap v16551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:27.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:27 smithi153 ceph-mon[36569]: pgmap v16552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:27 smithi104 ceph-mon[36624]: pgmap v16552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:27 smithi081 ceph-mon[31453]: pgmap v16552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:29.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:29 smithi153 ceph-mon[36569]: pgmap v16553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:29.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:29 smithi104 ceph-mon[36624]: pgmap v16553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:29.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:29 smithi081 ceph-mon[31453]: pgmap v16553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:31.238+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:51:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:31.239+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:51:31.242 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:31.239+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:51:31.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:31 smithi153 ceph-mon[36569]: pgmap v16554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:31.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:31 smithi104 ceph-mon[36624]: pgmap v16554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:31.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:31 smithi081 ceph-mon[31453]: pgmap v16554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:33.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:33 smithi153 ceph-mon[36569]: pgmap v16555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:33.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:33 smithi104 ceph-mon[36624]: pgmap v16555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:33.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:33 smithi081 ceph-mon[31453]: pgmap v16555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:35.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:35 smithi153 ceph-mon[36569]: pgmap v16556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:35 smithi104 ceph-mon[36624]: pgmap v16556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:35.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:35 smithi081 ceph-mon[31453]: pgmap v16556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:37.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:37 smithi153 ceph-mon[36569]: pgmap v16557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:37 smithi104 ceph-mon[36624]: pgmap v16557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:37.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:37 smithi081 ceph-mon[31453]: pgmap v16557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:39.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:39 smithi104 ceph-mon[36624]: pgmap v16558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:39.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:39 smithi081 ceph-mon[31453]: pgmap v16558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:40.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:39 smithi153 ceph-mon[36569]: pgmap v16558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:41.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:41 smithi104 ceph-mon[36624]: pgmap v16559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:41.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:41 smithi081 ceph-mon[31453]: pgmap v16559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:42.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:41 smithi153 ceph-mon[36569]: pgmap v16559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:43.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:43 smithi104 ceph-mon[36624]: pgmap v16560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:43.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:43 smithi081 ceph-mon[31453]: pgmap v16560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:44.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:43 smithi153 ceph-mon[36569]: pgmap v16560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:45.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:45 smithi104 ceph-mon[36624]: pgmap v16561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:45.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:45 smithi081 ceph-mon[31453]: pgmap v16561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:46.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:45 smithi153 ceph-mon[36569]: pgmap v16561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:46.245+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:51:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:46.245+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:51:46.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:51:46.245+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:51:47.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:47 smithi104 ceph-mon[36624]: pgmap v16562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:47.862 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:47 smithi153 ceph-mon[36569]: pgmap v16562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:47 smithi081 ceph-mon[31453]: pgmap v16562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:49.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:49 smithi104 ceph-mon[36624]: pgmap v16563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:49.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:49 smithi081 ceph-mon[31453]: pgmap v16563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:50.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:49 smithi153 ceph-mon[36569]: pgmap v16563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:51.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:51 smithi081 ceph-mon[31453]: pgmap v16564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:52.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:51 smithi153 ceph-mon[36569]: pgmap v16564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:52.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:51 smithi104 ceph-mon[36624]: pgmap v16564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:53.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:53 smithi081 ceph-mon[31453]: pgmap v16565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:54.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:53 smithi153 ceph-mon[36569]: pgmap v16565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:54.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:53 smithi104 ceph-mon[36624]: pgmap v16565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:55.887 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:55 smithi104 ceph-mon[36624]: pgmap v16566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:55.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:55 smithi081 ceph-mon[31453]: pgmap v16566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:55 smithi153 ceph-mon[36569]: pgmap v16566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:57.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:57 smithi081 ceph-mon[31453]: pgmap v16567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:58.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:57 smithi153 ceph-mon[36569]: pgmap v16567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:58.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:57 smithi104 ceph-mon[36624]: pgmap v16567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:51:59.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:51:59 smithi081 ceph-mon[31453]: pgmap v16568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:51:59 smithi153 ceph-mon[36569]: pgmap v16568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:00.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:51:59 smithi104 ceph-mon[36624]: pgmap v16568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:01.261+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:52:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:01.262+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:52:01.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:01.262+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:52:01.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:01 smithi081 ceph-mon[31453]: pgmap v16569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:02.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:01 smithi153 ceph-mon[36569]: pgmap v16569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:01 smithi104 ceph-mon[36624]: pgmap v16569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:03.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:03 smithi081 ceph-mon[31453]: pgmap v16570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:04.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:03 smithi153 ceph-mon[36569]: pgmap v16570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:04.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:03 smithi104 ceph-mon[36624]: pgmap v16570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:05.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:05 smithi081 ceph-mon[31453]: pgmap v16571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:06.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:05 smithi153 ceph-mon[36569]: pgmap v16571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:06.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:05 smithi104 ceph-mon[36624]: pgmap v16571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:07.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:07 smithi081 ceph-mon[31453]: pgmap v16572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:08.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:07 smithi153 ceph-mon[36569]: pgmap v16572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:08.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:07 smithi104 ceph-mon[36624]: pgmap v16572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:09.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:09 smithi081 ceph-mon[31453]: pgmap v16573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:10.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:09 smithi153 ceph-mon[36569]: pgmap v16573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:10.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:09 smithi104 ceph-mon[36624]: pgmap v16573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:11.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:11 smithi081 ceph-mon[31453]: pgmap v16574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:12.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:11 smithi153 ceph-mon[36569]: pgmap v16574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:12.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:11 smithi104 ceph-mon[36624]: pgmap v16574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:13.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:13 smithi081 ceph-mon[31453]: pgmap v16575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:14.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:13 smithi153 ceph-mon[36569]: pgmap v16575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:14.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:13 smithi104 ceph-mon[36624]: pgmap v16575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:15.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:15 smithi081 ceph-mon[31453]: pgmap v16576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:16.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:15 smithi153 ceph-mon[36569]: pgmap v16576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:16.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:15 smithi104 ceph-mon[36624]: pgmap v16576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:16.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:16.270+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:52:16.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:16.270+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:52:16.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:16.270+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:52:17.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:17.822+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:52:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:17.822+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:52:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:17.822+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 239 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:52:17.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:17.822+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 239 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:52:17.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:17 smithi081 ceph-mon[31453]: pgmap v16577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:18.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:17 smithi153 ceph-mon[36569]: pgmap v16577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:18.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:17 smithi104 ceph-mon[36624]: pgmap v16577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:20.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:19 smithi153 ceph-mon[36569]: pgmap v16578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:20.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:19 smithi104 ceph-mon[36624]: pgmap v16578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:20.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:19 smithi081 ceph-mon[31453]: pgmap v16578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:22.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:21 smithi153 ceph-mon[36569]: pgmap v16579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:22.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:21 smithi104 ceph-mon[36624]: pgmap v16579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:21 smithi081 ceph-mon[31453]: pgmap v16579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:23.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: pgmap v16580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:23.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: pgmap v16580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: pgmap v16580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:24.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:52:25.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:52:25.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:52:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:52:25.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:52:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T05:52:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T05:52:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T05:52:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T05:52:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T05:52:25.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:52:25.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:52:25.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:52:25.986 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:25 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:52:25.986 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:25 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:52:25.986 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:25 smithi104 ceph-mon[36624]: pgmap v16581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:25.987 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:25 smithi104 ceph-mon[36624]: pgmap v16582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:26.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:25 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:52:26.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:25 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:52:26.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:25 smithi153 ceph-mon[36569]: pgmap v16581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:26.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:25 smithi153 ceph-mon[36569]: pgmap v16582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:26.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:25 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T05:52:26.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:25 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T05:52:26.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:25 smithi081 ceph-mon[31453]: pgmap v16581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:26.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:25 smithi081 ceph-mon[31453]: pgmap v16582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:28.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:27 smithi153 ceph-mon[36569]: pgmap v16583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:28.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:27 smithi104 ceph-mon[36624]: pgmap v16583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:28.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:27 smithi081 ceph-mon[31453]: pgmap v16583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:30.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:29 smithi104 ceph-mon[36624]: pgmap v16584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:30.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:29 smithi081 ceph-mon[31453]: pgmap v16584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:29 smithi153 ceph-mon[36569]: pgmap v16584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:31.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:31.284+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:52:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:31.284+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:52:31.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:31.284+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:52:32.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:31 smithi104 ceph-mon[36624]: pgmap v16585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:32.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:31 smithi081 ceph-mon[31453]: pgmap v16585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:31 smithi153 ceph-mon[36569]: pgmap v16585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:34.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:33 smithi104 ceph-mon[36624]: pgmap v16586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:33 smithi081 ceph-mon[31453]: pgmap v16586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:33 smithi153 ceph-mon[36569]: pgmap v16586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:36.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:35 smithi104 ceph-mon[36624]: pgmap v16587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:36.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:35 smithi081 ceph-mon[31453]: pgmap v16587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:36.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:35 smithi153 ceph-mon[36569]: pgmap v16587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:37 smithi081 ceph-mon[31453]: pgmap v16588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:37 smithi153 ceph-mon[36569]: pgmap v16588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:38.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:37 smithi104 ceph-mon[36624]: pgmap v16588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:40.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:39 smithi081 ceph-mon[31453]: pgmap v16589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:39 smithi153 ceph-mon[36569]: pgmap v16589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:40.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:39 smithi104 ceph-mon[36624]: pgmap v16589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:42.122 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:41 smithi081 ceph-mon[31453]: pgmap v16590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:41 smithi153 ceph-mon[36569]: pgmap v16590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:42.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:41 smithi104 ceph-mon[36624]: pgmap v16590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:44.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:43 smithi081 ceph-mon[31453]: pgmap v16591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:43 smithi153 ceph-mon[36569]: pgmap v16591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:44.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:43 smithi104 ceph-mon[36624]: pgmap v16591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:46.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:45 smithi081 ceph-mon[31453]: pgmap v16592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:46.298+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:52:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:46.298+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:52:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:52:46.298+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:52:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:45 smithi153 ceph-mon[36569]: pgmap v16592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:46.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:45 smithi104 ceph-mon[36624]: pgmap v16592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:48.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:47 smithi081 ceph-mon[31453]: pgmap v16593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:47 smithi153 ceph-mon[36569]: pgmap v16593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:48.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:47 smithi104 ceph-mon[36624]: pgmap v16593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:50.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:49 smithi081 ceph-mon[31453]: pgmap v16594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:49 smithi153 ceph-mon[36569]: pgmap v16594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:50.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:49 smithi104 ceph-mon[36624]: pgmap v16594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:52.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:51 smithi081 ceph-mon[31453]: pgmap v16595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:52.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:51 smithi153 ceph-mon[36569]: pgmap v16595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:51 smithi104 ceph-mon[36624]: pgmap v16595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:54.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:53 smithi081 ceph-mon[31453]: pgmap v16596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:53 smithi153 ceph-mon[36569]: pgmap v16596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:53 smithi104 ceph-mon[36624]: pgmap v16596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:56.137 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:55 smithi104 ceph-mon[36624]: pgmap v16597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:56.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:55 smithi081 ceph-mon[31453]: pgmap v16597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:56.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:55 smithi153 ceph-mon[36569]: pgmap v16597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:58.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:57 smithi081 ceph-mon[31453]: pgmap v16598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:58.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:57 smithi153 ceph-mon[36569]: pgmap v16598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:52:58.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:57 smithi104 ceph-mon[36624]: pgmap v16598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:00.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:52:59 smithi081 ceph-mon[31453]: pgmap v16599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:52:59 smithi153 ceph-mon[36569]: pgmap v16599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:00.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:52:59 smithi104 ceph-mon[36624]: pgmap v16599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:01.307+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:53:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:01.307+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:53:01.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:01.307+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:53:02.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:01 smithi081 ceph-mon[31453]: pgmap v16600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:01 smithi153 ceph-mon[36569]: pgmap v16600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:02.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:01 smithi104 ceph-mon[36624]: pgmap v16600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:04.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:03 smithi081 ceph-mon[31453]: pgmap v16601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:03 smithi153 ceph-mon[36569]: pgmap v16601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:04.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:03 smithi104 ceph-mon[36624]: pgmap v16601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:05 smithi153 ceph-mon[36569]: pgmap v16602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:06.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:05 smithi104 ceph-mon[36624]: pgmap v16602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:06.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:05 smithi081 ceph-mon[31453]: pgmap v16602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:07 smithi153 ceph-mon[36569]: pgmap v16603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:08.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:07 smithi104 ceph-mon[36624]: pgmap v16603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:08.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:07 smithi081 ceph-mon[31453]: pgmap v16603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:10.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:09 smithi153 ceph-mon[36569]: pgmap v16604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:10.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:09 smithi104 ceph-mon[36624]: pgmap v16604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:10.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:09 smithi081 ceph-mon[31453]: pgmap v16604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:12.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:11 smithi153 ceph-mon[36569]: pgmap v16605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:12.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:11 smithi104 ceph-mon[36624]: pgmap v16605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:12.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:11 smithi081 ceph-mon[31453]: pgmap v16605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:13 smithi153 ceph-mon[36569]: pgmap v16606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:14.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:13 smithi104 ceph-mon[36624]: pgmap v16606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:14.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:13 smithi081 ceph-mon[31453]: pgmap v16606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:16.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:15 smithi153 ceph-mon[36569]: pgmap v16607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:16.321+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:53:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:16.322+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:53:16.324 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:16.322+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:53:16.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:15 smithi104 ceph-mon[36624]: pgmap v16607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:16.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:15 smithi081 ceph-mon[31453]: pgmap v16607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:18.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:17 smithi153 ceph-mon[36569]: pgmap v16608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:18.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:17 smithi104 ceph-mon[36624]: pgmap v16608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:18.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:17 smithi081 ceph-mon[31453]: pgmap v16608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:19 smithi153 ceph-mon[36569]: pgmap v16609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:20.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:19 smithi104 ceph-mon[36624]: pgmap v16609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:20.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:19 smithi081 ceph-mon[31453]: pgmap v16609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:21 smithi153 ceph-mon[36569]: pgmap v16610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:22.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:21 smithi104 ceph-mon[36624]: pgmap v16610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:22.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:21 smithi081 ceph-mon[31453]: pgmap v16610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:24.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:23 smithi153 ceph-mon[36569]: pgmap v16611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:23 smithi104 ceph-mon[36624]: pgmap v16611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:24.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:23 smithi081 ceph-mon[31453]: pgmap v16611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:25.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:53:25.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:53:25.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:53:26.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:25 smithi104 ceph-mon[36624]: pgmap v16612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:26.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:53:26.236 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:53:26.237 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:53:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:25 smithi153 ceph-mon[36569]: pgmap v16612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:53:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:53:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:53:26.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:25 smithi081 ceph-mon[31453]: pgmap v16612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:26.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:53:26.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:53:26.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:53:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:26 smithi153 ceph-mon[36569]: pgmap v16613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:26 smithi153 ceph-mon[36569]: pgmap v16614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:27.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:26 smithi104 ceph-mon[36624]: pgmap v16613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:27.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:26 smithi104 ceph-mon[36624]: pgmap v16614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:27.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:26 smithi081 ceph-mon[31453]: pgmap v16613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:27.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:26 smithi081 ceph-mon[31453]: pgmap v16614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:28.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:27 smithi153 ceph-mon[36569]: pgmap v16615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:28.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:27 smithi081 ceph-mon[31453]: pgmap v16615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:28.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:27 smithi104 ceph-mon[36624]: pgmap v16615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:30.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:30 smithi081 ceph-mon[31453]: pgmap v16616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:31.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:30 smithi153 ceph-mon[36569]: pgmap v16616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:31.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:30 smithi104 ceph-mon[36624]: pgmap v16616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:31.335+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:53:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:31.335+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:53:31.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:31.335+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:53:32.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:32 smithi081 ceph-mon[31453]: pgmap v16617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:33.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:32 smithi153 ceph-mon[36569]: pgmap v16617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:33.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:32 smithi104 ceph-mon[36624]: pgmap v16617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:34.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:34 smithi081 ceph-mon[31453]: pgmap v16618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:34 smithi153 ceph-mon[36569]: pgmap v16618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:34 smithi104 ceph-mon[36624]: pgmap v16618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:36.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:36 smithi081 ceph-mon[31453]: pgmap v16619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:37.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:36 smithi153 ceph-mon[36569]: pgmap v16619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:37.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:36 smithi104 ceph-mon[36624]: pgmap v16619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:38.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:38 smithi081 ceph-mon[31453]: pgmap v16620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:39.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:38 smithi153 ceph-mon[36569]: pgmap v16620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:39.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:38 smithi104 ceph-mon[36624]: pgmap v16620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:40.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:40 smithi081 ceph-mon[31453]: pgmap v16621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:41.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:40 smithi153 ceph-mon[36569]: pgmap v16621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:41.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:40 smithi104 ceph-mon[36624]: pgmap v16621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:42.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:42 smithi081 ceph-mon[31453]: pgmap v16622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:43.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:42 smithi153 ceph-mon[36569]: pgmap v16622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:43.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:42 smithi104 ceph-mon[36624]: pgmap v16622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:44.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:44 smithi081 ceph-mon[31453]: pgmap v16623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:45.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:44 smithi153 ceph-mon[36569]: pgmap v16623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:45.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:44 smithi104 ceph-mon[36624]: pgmap v16623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:46.350+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:53:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:46.350+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:53:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:53:46.350+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:53:46.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:46 smithi081 ceph-mon[31453]: pgmap v16624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:47.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:46 smithi153 ceph-mon[36569]: pgmap v16624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:47.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:46 smithi104 ceph-mon[36624]: pgmap v16624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:48.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:48 smithi081 ceph-mon[31453]: pgmap v16625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:49.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:48 smithi153 ceph-mon[36569]: pgmap v16625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:49.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:48 smithi104 ceph-mon[36624]: pgmap v16625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:51.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:50 smithi153 ceph-mon[36569]: pgmap v16626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:51.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:50 smithi104 ceph-mon[36624]: pgmap v16626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:51.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:50 smithi081 ceph-mon[31453]: pgmap v16626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:53.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:52 smithi153 ceph-mon[36569]: pgmap v16627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:52 smithi104 ceph-mon[36624]: pgmap v16627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:53.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:52 smithi081 ceph-mon[31453]: pgmap v16627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:54.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:54 smithi081 ceph-mon[31453]: pgmap v16628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:55.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:54 smithi153 ceph-mon[36569]: pgmap v16628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:55.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:54 smithi104 ceph-mon[36624]: pgmap v16628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:57.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:56 smithi153 ceph-mon[36569]: pgmap v16629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:56 smithi104 ceph-mon[36624]: pgmap v16629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:56 smithi081 ceph-mon[31453]: pgmap v16629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:58.962 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:53:58 smithi081 ceph-mon[31453]: pgmap v16630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:59.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:53:58 smithi153 ceph-mon[36569]: pgmap v16630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:53:59.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:53:58 smithi104 ceph-mon[36624]: pgmap v16630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:00 smithi153 ceph-mon[36569]: pgmap v16631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:01.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:00 smithi104 ceph-mon[36624]: pgmap v16631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:01.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:00 smithi081 ceph-mon[31453]: pgmap v16631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:01.366+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:54:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:01.366+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:54:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:01.366+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:54:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:02 smithi153 ceph-mon[36569]: pgmap v16632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:03.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:02 smithi104 ceph-mon[36624]: pgmap v16632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:03.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:02 smithi081 ceph-mon[31453]: pgmap v16632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:05.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:04 smithi153 ceph-mon[36569]: pgmap v16633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:05.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:04 smithi104 ceph-mon[36624]: pgmap v16633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:05.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:04 smithi081 ceph-mon[31453]: pgmap v16633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:07.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:06 smithi153 ceph-mon[36569]: pgmap v16634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:07.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:06 smithi104 ceph-mon[36624]: pgmap v16634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:07.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:06 smithi081 ceph-mon[31453]: pgmap v16634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:09.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:08 smithi153 ceph-mon[36569]: pgmap v16635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:09.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:08 smithi104 ceph-mon[36624]: pgmap v16635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:09.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:08 smithi081 ceph-mon[31453]: pgmap v16635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:11.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:10 smithi153 ceph-mon[36569]: pgmap v16636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:11.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:10 smithi104 ceph-mon[36624]: pgmap v16636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:11.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:10 smithi081 ceph-mon[31453]: pgmap v16636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:13.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:12 smithi153 ceph-mon[36569]: pgmap v16637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:13.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:12 smithi104 ceph-mon[36624]: pgmap v16637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:13.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:12 smithi081 ceph-mon[31453]: pgmap v16637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:15.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:14 smithi153 ceph-mon[36569]: pgmap v16638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:15.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:14 smithi104 ceph-mon[36624]: pgmap v16638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:15.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:14 smithi081 ceph-mon[31453]: pgmap v16638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:16.379+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:16.379+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:54:16.382 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:16.379+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:54:17.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:16 smithi153 ceph-mon[36569]: pgmap v16639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:17.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:16 smithi104 ceph-mon[36624]: pgmap v16639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:17.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:16 smithi081 ceph-mon[31453]: pgmap v16639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:19.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:18 smithi153 ceph-mon[36569]: pgmap v16640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:19.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:18 smithi104 ceph-mon[36624]: pgmap v16640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:19.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:18 smithi081 ceph-mon[31453]: pgmap v16640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:21.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:20 smithi153 ceph-mon[36569]: pgmap v16641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:21.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:20 smithi104 ceph-mon[36624]: pgmap v16641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:21.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:20 smithi081 ceph-mon[31453]: pgmap v16641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:23.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:22 smithi153 ceph-mon[36569]: pgmap v16642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:23.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:22 smithi104 ceph-mon[36624]: pgmap v16642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:23.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:22 smithi081 ceph-mon[31453]: pgmap v16642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:25.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:24 smithi153 ceph-mon[36569]: pgmap v16643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:25.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:24 smithi104 ceph-mon[36624]: pgmap v16643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:24 smithi081 ceph-mon[31453]: pgmap v16643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:27.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:26 smithi153 ceph-mon[36569]: pgmap v16644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:27.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:54:27.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:26 smithi104 ceph-mon[36624]: pgmap v16644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:27.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:54:27.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:26 smithi081 ceph-mon[31453]: pgmap v16644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:27.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:54:28.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:54:28.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:54:28.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:54:28.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:54:28.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:54:28.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:54:28.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:54:28.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:54:28.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:54:29.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:28 smithi153 ceph-mon[36569]: pgmap v16645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:29.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:28 smithi153 ceph-mon[36569]: pgmap v16646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:29.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:28 smithi104 ceph-mon[36624]: pgmap v16645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:29.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:28 smithi104 ceph-mon[36624]: pgmap v16646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:29.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:28 smithi081 ceph-mon[31453]: pgmap v16645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:29.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:28 smithi081 ceph-mon[31453]: pgmap v16646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:31.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:30 smithi153 ceph-mon[36569]: pgmap v16647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:31.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:30 smithi104 ceph-mon[36624]: pgmap v16647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:31.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:30 smithi081 ceph-mon[31453]: pgmap v16647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:31.393+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:54:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:31.393+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:54:31.396 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:31.393+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:54:33.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:32 smithi153 ceph-mon[36569]: pgmap v16648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:33.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:32 smithi104 ceph-mon[36624]: pgmap v16648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:33.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:32 smithi081 ceph-mon[31453]: pgmap v16648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:35.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:34 smithi153 ceph-mon[36569]: pgmap v16649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:35.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:34 smithi104 ceph-mon[36624]: pgmap v16649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:34 smithi081 ceph-mon[31453]: pgmap v16649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:37.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:36 smithi153 ceph-mon[36569]: pgmap v16650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:37.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:36 smithi104 ceph-mon[36624]: pgmap v16650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:37.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:36 smithi081 ceph-mon[31453]: pgmap v16650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:39.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:38 smithi153 ceph-mon[36569]: pgmap v16651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:39.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:38 smithi104 ceph-mon[36624]: pgmap v16651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:39.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:38 smithi081 ceph-mon[31453]: pgmap v16651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:41.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:40 smithi153 ceph-mon[36569]: pgmap v16652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:41.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:40 smithi104 ceph-mon[36624]: pgmap v16652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:41.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:40 smithi081 ceph-mon[31453]: pgmap v16652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:43.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:42 smithi153 ceph-mon[36569]: pgmap v16653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:43.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:42 smithi104 ceph-mon[36624]: pgmap v16653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:43.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:42 smithi081 ceph-mon[31453]: pgmap v16653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:45.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:44 smithi153 ceph-mon[36569]: pgmap v16654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:45.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:44 smithi104 ceph-mon[36624]: pgmap v16654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:45.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:44 smithi081 ceph-mon[31453]: pgmap v16654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:46.408+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:54:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:46.408+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:54:46.411 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:54:46.408+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:54:47.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:46 smithi104 ceph-mon[36624]: pgmap v16655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:47.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:46 smithi081 ceph-mon[31453]: pgmap v16655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:46 smithi153 ceph-mon[36569]: pgmap v16655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:49.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:48 smithi104 ceph-mon[36624]: pgmap v16656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:49.112 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:48 smithi153 ceph-mon[36569]: pgmap v16656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:49.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:48 smithi081 ceph-mon[31453]: pgmap v16656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:51.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:50 smithi104 ceph-mon[36624]: pgmap v16657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:51.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:50 smithi081 ceph-mon[31453]: pgmap v16657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:51.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:50 smithi153 ceph-mon[36569]: pgmap v16657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:53.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:52 smithi104 ceph-mon[36624]: pgmap v16658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:53.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:52 smithi081 ceph-mon[31453]: pgmap v16658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:52 smithi153 ceph-mon[36569]: pgmap v16658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:55.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:54 smithi104 ceph-mon[36624]: pgmap v16659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:55.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:54 smithi081 ceph-mon[31453]: pgmap v16659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:55.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:54 smithi153 ceph-mon[36569]: pgmap v16659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:56 smithi081 ceph-mon[31453]: pgmap v16660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:56 smithi153 ceph-mon[36569]: pgmap v16660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:57.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:56 smithi104 ceph-mon[36624]: pgmap v16660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:59.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:54:58 smithi081 ceph-mon[31453]: pgmap v16661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:59.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:54:58 smithi153 ceph-mon[36569]: pgmap v16661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:54:59.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:54:58 smithi104 ceph-mon[36624]: pgmap v16661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:01.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:00 smithi081 ceph-mon[31453]: pgmap v16662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:00 smithi153 ceph-mon[36569]: pgmap v16662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:00 smithi104 ceph-mon[36624]: pgmap v16662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:01.423+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:55:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:01.423+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:55:01.426 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:01.423+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:55:03.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:02 smithi081 ceph-mon[31453]: pgmap v16663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:03.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:02 smithi153 ceph-mon[36569]: pgmap v16663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:03.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:02 smithi104 ceph-mon[36624]: pgmap v16663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:05.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:04 smithi081 ceph-mon[31453]: pgmap v16664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:05.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:04 smithi153 ceph-mon[36569]: pgmap v16664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:05.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:04 smithi104 ceph-mon[36624]: pgmap v16664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:07.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:06 smithi081 ceph-mon[31453]: pgmap v16665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:07.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:06 smithi153 ceph-mon[36569]: pgmap v16665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:07.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:06 smithi104 ceph-mon[36624]: pgmap v16665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:09.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:08 smithi081 ceph-mon[31453]: pgmap v16666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:08 smithi153 ceph-mon[36569]: pgmap v16666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:09.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:08 smithi104 ceph-mon[36624]: pgmap v16666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:11.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:10 smithi081 ceph-mon[31453]: pgmap v16667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:11.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:10 smithi153 ceph-mon[36569]: pgmap v16667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:11.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:10 smithi104 ceph-mon[36624]: pgmap v16667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:13.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:12 smithi081 ceph-mon[31453]: pgmap v16668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:12 smithi153 ceph-mon[36569]: pgmap v16668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:13.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:12 smithi104 ceph-mon[36624]: pgmap v16668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:15.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:14 smithi081 ceph-mon[31453]: pgmap v16669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:15.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:14 smithi153 ceph-mon[36569]: pgmap v16669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:15.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:14 smithi104 ceph-mon[36624]: pgmap v16669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:16.434+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:55:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:16.434+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:55:16.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:16.434+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:55:17.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:16 smithi081 ceph-mon[31453]: pgmap v16670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:16 smithi153 ceph-mon[36569]: pgmap v16670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:17.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:16 smithi104 ceph-mon[36624]: pgmap v16670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:17.853 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:17.850+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:55:17.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:17.850+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:55:17.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:17.850+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 240 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:55:17.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:17.850+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 240 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:55:19.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:18 smithi081 ceph-mon[31453]: pgmap v16671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:18 smithi153 ceph-mon[36569]: pgmap v16671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:19.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:18 smithi104 ceph-mon[36624]: pgmap v16671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:21.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:20 smithi081 ceph-mon[31453]: pgmap v16672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:20 smithi153 ceph-mon[36569]: pgmap v16672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:21.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:20 smithi104 ceph-mon[36624]: pgmap v16672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:23.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:22 smithi081 ceph-mon[31453]: pgmap v16673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:22 smithi153 ceph-mon[36569]: pgmap v16673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:22 smithi104 ceph-mon[36624]: pgmap v16673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:25.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:24 smithi153 ceph-mon[36569]: pgmap v16674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:25.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:24 smithi104 ceph-mon[36624]: pgmap v16674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:25.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:24 smithi081 ceph-mon[31453]: pgmap v16674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:26 smithi153 ceph-mon[36569]: pgmap v16675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:27.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:26 smithi104 ceph-mon[36624]: pgmap v16675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:27.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:26 smithi081 ceph-mon[31453]: pgmap v16675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:28.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:55:28.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:55:28.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:55:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:28 smithi153 ceph-mon[36569]: pgmap v16676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:55:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:55:29.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:55:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:28 smithi104 ceph-mon[36624]: pgmap v16676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:55:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:55:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:55:29.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:28 smithi081 ceph-mon[31453]: pgmap v16676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:29.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:55:29.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:55:29.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:55:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:29 smithi153 ceph-mon[36569]: pgmap v16677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:30.398 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:29 smithi153 ceph-mon[36569]: pgmap v16678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:30.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:29 smithi104 ceph-mon[36624]: pgmap v16677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:30.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:29 smithi104 ceph-mon[36624]: pgmap v16678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:30.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:29 smithi081 ceph-mon[31453]: pgmap v16677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:30.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:29 smithi081 ceph-mon[31453]: pgmap v16678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:31.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:31.450+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:55:31.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:31.450+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:55:31.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:31.450+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:55:32.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:31 smithi081 ceph-mon[31453]: pgmap v16679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:31 smithi153 ceph-mon[36569]: pgmap v16679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:32.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:31 smithi104 ceph-mon[36624]: pgmap v16679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:33 smithi153 ceph-mon[36569]: pgmap v16680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:34.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:33 smithi104 ceph-mon[36624]: pgmap v16680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:34.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:33 smithi081 ceph-mon[31453]: pgmap v16680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:36.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:35 smithi153 ceph-mon[36569]: pgmap v16681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:36.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:35 smithi104 ceph-mon[36624]: pgmap v16681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:36.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:35 smithi081 ceph-mon[31453]: pgmap v16681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:37 smithi153 ceph-mon[36569]: pgmap v16682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:38.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:37 smithi104 ceph-mon[36624]: pgmap v16682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:38.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:37 smithi081 ceph-mon[31453]: pgmap v16682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:39 smithi153 ceph-mon[36569]: pgmap v16683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:40.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:39 smithi104 ceph-mon[36624]: pgmap v16683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:40.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:39 smithi081 ceph-mon[31453]: pgmap v16683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:41 smithi153 ceph-mon[36569]: pgmap v16684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:42.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:41 smithi104 ceph-mon[36624]: pgmap v16684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:42.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:41 smithi081 ceph-mon[31453]: pgmap v16684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:43 smithi153 ceph-mon[36569]: pgmap v16685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:44.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:43 smithi104 ceph-mon[36624]: pgmap v16685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:44.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:43 smithi081 ceph-mon[31453]: pgmap v16685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:45 smithi153 ceph-mon[36569]: pgmap v16686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:46.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:45 smithi104 ceph-mon[36624]: pgmap v16686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:46.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:45 smithi081 ceph-mon[31453]: pgmap v16686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:46.465+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:55:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:46.465+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:55:46.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:55:46.465+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:55:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:47 smithi153 ceph-mon[36569]: pgmap v16687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:48.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:47 smithi104 ceph-mon[36624]: pgmap v16687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:48.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:47 smithi081 ceph-mon[31453]: pgmap v16687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:49 smithi153 ceph-mon[36569]: pgmap v16688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:50.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:49 smithi104 ceph-mon[36624]: pgmap v16688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:50.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:49 smithi081 ceph-mon[31453]: pgmap v16688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:52.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:51 smithi153 ceph-mon[36569]: pgmap v16689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:52.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:51 smithi104 ceph-mon[36624]: pgmap v16689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:52.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:51 smithi081 ceph-mon[31453]: pgmap v16689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:54 smithi153 ceph-mon[36569]: pgmap v16690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:54 smithi104 ceph-mon[36624]: pgmap v16690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:54.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:54 smithi081 ceph-mon[31453]: pgmap v16690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:56.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:56 smithi153 ceph-mon[36569]: pgmap v16691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:56 smithi104 ceph-mon[36624]: pgmap v16691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:56.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:56 smithi081 ceph-mon[31453]: pgmap v16691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:57 smithi153 ceph-mon[36569]: pgmap v16692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:57.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:57 smithi104 ceph-mon[36624]: pgmap v16692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:55:57.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:57 smithi081 ceph-mon[31453]: pgmap v16692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:00.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:55:59 smithi153 ceph-mon[36569]: pgmap v16693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:00.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:55:59 smithi104 ceph-mon[36624]: pgmap v16693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:00.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:55:59 smithi081 ceph-mon[31453]: pgmap v16693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:01.480+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:56:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:01.480+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:56:01.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:01.480+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:56:02.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:01 smithi153 ceph-mon[36569]: pgmap v16694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:02.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:01 smithi104 ceph-mon[36624]: pgmap v16694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:02.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:01 smithi081 ceph-mon[31453]: pgmap v16694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:02.857+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:56:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:02.857+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:56:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:02.857+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 241 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:56:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:02.858+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 241 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:56:04.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:03 smithi104 ceph-mon[36624]: pgmap v16695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:04.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:03 smithi081 ceph-mon[31453]: pgmap v16695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:03 smithi153 ceph-mon[36569]: pgmap v16695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:06.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:05 smithi104 ceph-mon[36624]: pgmap v16696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:06.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:05 smithi081 ceph-mon[31453]: pgmap v16696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:05 smithi153 ceph-mon[36569]: pgmap v16696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:08.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:07 smithi104 ceph-mon[36624]: pgmap v16697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:08.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:07 smithi081 ceph-mon[31453]: pgmap v16697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:07 smithi153 ceph-mon[36569]: pgmap v16697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:10.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:09 smithi104 ceph-mon[36624]: pgmap v16698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:10.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:09 smithi081 ceph-mon[31453]: pgmap v16698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:10.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:09 smithi153 ceph-mon[36569]: pgmap v16698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:12.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:11 smithi104 ceph-mon[36624]: pgmap v16699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:12.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:11 smithi081 ceph-mon[31453]: pgmap v16699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:12.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:11 smithi153 ceph-mon[36569]: pgmap v16699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:14.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:13 smithi081 ceph-mon[31453]: pgmap v16700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:13 smithi153 ceph-mon[36569]: pgmap v16700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:13 smithi104 ceph-mon[36624]: pgmap v16700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:16.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:15 smithi081 ceph-mon[31453]: pgmap v16701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:16.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:15 smithi153 ceph-mon[36569]: pgmap v16701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:15 smithi104 ceph-mon[36624]: pgmap v16701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:16.495+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:56:16.498 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:16.495+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:56:16.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:16.495+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:56:18.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:17 smithi081 ceph-mon[31453]: pgmap v16702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:18.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:17 smithi153 ceph-mon[36569]: pgmap v16702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:18.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:17 smithi104 ceph-mon[36624]: pgmap v16702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:20.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:19 smithi153 ceph-mon[36569]: pgmap v16703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:20.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:19 smithi081 ceph-mon[31453]: pgmap v16703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:20.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:19 smithi104 ceph-mon[36624]: pgmap v16703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:22.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:21 smithi081 ceph-mon[31453]: pgmap v16704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:21 smithi153 ceph-mon[36569]: pgmap v16704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:22.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:21 smithi104 ceph-mon[36624]: pgmap v16704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:24.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:23 smithi081 ceph-mon[31453]: pgmap v16705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:24.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:23 smithi153 ceph-mon[36569]: pgmap v16705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:24.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:23 smithi104 ceph-mon[36624]: pgmap v16705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:26.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:25 smithi081 ceph-mon[31453]: pgmap v16706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:25 smithi153 ceph-mon[36569]: pgmap v16706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:26.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:25 smithi104 ceph-mon[36624]: pgmap v16706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:28.137 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:27 smithi104 ceph-mon[36624]: pgmap v16707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:28.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:27 smithi081 ceph-mon[31453]: pgmap v16707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:28.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:27 smithi153 ceph-mon[36569]: pgmap v16707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:30.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:29 smithi081 ceph-mon[31453]: pgmap v16708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:30.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:56:30.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:29 smithi153 ceph-mon[36569]: pgmap v16708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:56:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:29 smithi104 ceph-mon[36624]: pgmap v16708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:30.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:56:30.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:30.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:31.513 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:31.510+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:56:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:31.510+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:56:31.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:31.510+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:56:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:31 smithi081 ceph-mon[31453]: pgmap v16709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:32.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:32.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:31 smithi153 ceph-mon[36569]: pgmap v16709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:31 smithi104 ceph-mon[36624]: pgmap v16709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:32.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:32.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:33.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:33.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:33 smithi153 ceph-mon[36569]: pgmap v16710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:33 smithi104 ceph-mon[36624]: pgmap v16710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:33 smithi081 ceph-mon[31453]: pgmap v16710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:35 smithi081 ceph-mon[31453]: pgmap v16711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:35.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:35.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:35.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:35 smithi153 ceph-mon[36569]: pgmap v16711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:35.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:35.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:35.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:35 smithi104 ceph-mon[36624]: pgmap v16711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:36.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:56:36.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:56:36.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:36 smithi153 ceph-mon[36569]: pgmap v16712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:36.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:36 smithi153 ceph-mon[36569]: pgmap v16713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:36.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:36.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:56:36.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:56:36.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:36 smithi104 ceph-mon[36624]: pgmap v16712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:36.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:36 smithi104 ceph-mon[36624]: pgmap v16713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:36.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:56:36.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:56:36.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:36 smithi081 ceph-mon[31453]: pgmap v16712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:36.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:36 smithi081 ceph-mon[31453]: pgmap v16713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:36.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:56:38.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:38 smithi081 ceph-mon[31453]: pgmap v16714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:39.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:38 smithi153 ceph-mon[36569]: pgmap v16714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:38 smithi104 ceph-mon[36624]: pgmap v16714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:40 smithi153 ceph-mon[36569]: pgmap v16715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:40 smithi104 ceph-mon[36624]: pgmap v16715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:40 smithi081 ceph-mon[31453]: pgmap v16715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:43.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:42 smithi153 ceph-mon[36569]: pgmap v16716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:42 smithi104 ceph-mon[36624]: pgmap v16716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:42 smithi081 ceph-mon[31453]: pgmap v16716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:44 smithi153 ceph-mon[36569]: pgmap v16717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:44 smithi104 ceph-mon[36624]: pgmap v16717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:44 smithi081 ceph-mon[31453]: pgmap v16717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:46.519+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:56:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:46.519+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:56:46.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:56:46.519+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:56:47.180 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:46 smithi081 ceph-mon[31453]: pgmap v16718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:46 smithi153 ceph-mon[36569]: pgmap v16718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:46 smithi104 ceph-mon[36624]: pgmap v16718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:49.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:48 smithi153 ceph-mon[36569]: pgmap v16719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:49.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:48 smithi104 ceph-mon[36624]: pgmap v16719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:49.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:48 smithi081 ceph-mon[31453]: pgmap v16719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:51.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:50 smithi153 ceph-mon[36569]: pgmap v16720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:50 smithi104 ceph-mon[36624]: pgmap v16720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:51.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:50 smithi081 ceph-mon[31453]: pgmap v16720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:52 smithi153 ceph-mon[36569]: pgmap v16721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:52 smithi104 ceph-mon[36624]: pgmap v16721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:53.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:52 smithi081 ceph-mon[31453]: pgmap v16721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:55.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:54 smithi081 ceph-mon[31453]: pgmap v16722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:55.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:54 smithi153 ceph-mon[36569]: pgmap v16722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:55.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:54 smithi104 ceph-mon[36624]: pgmap v16722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:56 smithi153 ceph-mon[36569]: pgmap v16723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:57.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:56 smithi104 ceph-mon[36624]: pgmap v16723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:57.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:56 smithi081 ceph-mon[31453]: pgmap v16723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:59.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:56:58 smithi153 ceph-mon[36569]: pgmap v16724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:56:58 smithi104 ceph-mon[36624]: pgmap v16724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:56:59.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:56:58 smithi081 ceph-mon[31453]: pgmap v16724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:00 smithi153 ceph-mon[36569]: pgmap v16725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:00 smithi104 ceph-mon[36624]: pgmap v16725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:01.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:00 smithi081 ceph-mon[31453]: pgmap v16725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:01.522+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:57:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:01.522+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:57:01.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:01.522+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:57:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:02 smithi153 ceph-mon[36569]: pgmap v16726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:03.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:02 smithi104 ceph-mon[36624]: pgmap v16726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:03.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:02 smithi081 ceph-mon[31453]: pgmap v16726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:05.291 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:04 smithi081 ceph-mon[31453]: pgmap v16727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:05.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:04 smithi153 ceph-mon[36569]: pgmap v16727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:04 smithi104 ceph-mon[36624]: pgmap v16727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:07.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:06 smithi153 ceph-mon[36569]: pgmap v16728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:06 smithi104 ceph-mon[36624]: pgmap v16728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:07.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:06 smithi081 ceph-mon[31453]: pgmap v16728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:08 smithi153 ceph-mon[36569]: pgmap v16729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:08 smithi104 ceph-mon[36624]: pgmap v16729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:08 smithi081 ceph-mon[31453]: pgmap v16729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:11.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:10 smithi153 ceph-mon[36569]: pgmap v16730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:10 smithi104 ceph-mon[36624]: pgmap v16730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:11.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:10 smithi081 ceph-mon[31453]: pgmap v16730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:13.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:13 smithi153 ceph-mon[36569]: pgmap v16731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:13.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:13 smithi104 ceph-mon[36624]: pgmap v16731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:13.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:13 smithi081 ceph-mon[31453]: pgmap v16731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:15.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:15 smithi153 ceph-mon[36569]: pgmap v16732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:15 smithi104 ceph-mon[36624]: pgmap v16732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:15.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:15 smithi081 ceph-mon[31453]: pgmap v16732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:16.538+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:57:16.541 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:16.538+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:57:16.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:16.538+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:57:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:17 smithi153 ceph-mon[36569]: pgmap v16733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:17 smithi104 ceph-mon[36624]: pgmap v16733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:17.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:17 smithi081 ceph-mon[31453]: pgmap v16733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:19 smithi153 ceph-mon[36569]: pgmap v16734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:19 smithi104 ceph-mon[36624]: pgmap v16734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:19.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:19 smithi081 ceph-mon[31453]: pgmap v16734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:21 smithi153 ceph-mon[36569]: pgmap v16735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:21 smithi104 ceph-mon[36624]: pgmap v16735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:21 smithi081 ceph-mon[31453]: pgmap v16735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:22 smithi104 ceph-mon[36624]: pgmap v16736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:22.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:22 smithi081 ceph-mon[31453]: pgmap v16736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:22.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:22 smithi153 ceph-mon[36569]: pgmap v16736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:25.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:24 smithi153 ceph-mon[36569]: pgmap v16737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:24 smithi104 ceph-mon[36624]: pgmap v16737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:24 smithi081 ceph-mon[31453]: pgmap v16737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:26 smithi153 ceph-mon[36569]: pgmap v16738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:26 smithi104 ceph-mon[36624]: pgmap v16738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:26 smithi081 ceph-mon[31453]: pgmap v16738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:28 smithi153 ceph-mon[36569]: pgmap v16739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:28 smithi104 ceph-mon[36624]: pgmap v16739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:29.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:28 smithi081 ceph-mon[31453]: pgmap v16739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:31.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:30 smithi153 ceph-mon[36569]: pgmap v16740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:30 smithi104 ceph-mon[36624]: pgmap v16740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:30 smithi081 ceph-mon[31453]: pgmap v16740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:31.550 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:31.547+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:57:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:31.547+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:57:31.551 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:31.547+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:57:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:32 smithi153 ceph-mon[36569]: pgmap v16741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:33.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:32 smithi104 ceph-mon[36624]: pgmap v16741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:33.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:32 smithi081 ceph-mon[31453]: pgmap v16741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:35.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:34 smithi153 ceph-mon[36569]: pgmap v16742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:34 smithi104 ceph-mon[36624]: pgmap v16742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:34 smithi081 ceph-mon[31453]: pgmap v16742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:36.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:57:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:57:36.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:57:37.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:36 smithi153 ceph-mon[36569]: pgmap v16743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:37.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:37.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:37.310 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:36 smithi081 ceph-mon[31453]: pgmap v16743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:37.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:37.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:37.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:36 smithi104 ceph-mon[36624]: pgmap v16743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:37.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:37.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:57:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:57:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:57:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:57:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:38.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:57:38.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:57:38.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:57:39.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:38 smithi153 ceph-mon[36569]: pgmap v16744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:39.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:38 smithi153 ceph-mon[36569]: pgmap v16745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:38 smithi104 ceph-mon[36624]: pgmap v16744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:38 smithi104 ceph-mon[36624]: pgmap v16745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:38 smithi081 ceph-mon[31453]: pgmap v16744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:38 smithi081 ceph-mon[31453]: pgmap v16745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:40 smithi153 ceph-mon[36569]: pgmap v16746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:40 smithi104 ceph-mon[36624]: pgmap v16746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:41.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:40 smithi081 ceph-mon[31453]: pgmap v16746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:43.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:42 smithi153 ceph-mon[36569]: pgmap v16747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:42 smithi104 ceph-mon[36624]: pgmap v16747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:42 smithi081 ceph-mon[31453]: pgmap v16747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:44 smithi153 ceph-mon[36569]: pgmap v16748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:44 smithi104 ceph-mon[36624]: pgmap v16748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:44 smithi081 ceph-mon[31453]: pgmap v16748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:46.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:57:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:46.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:57:46.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:57:46.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:57:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:46 smithi153 ceph-mon[36569]: pgmap v16749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:46 smithi104 ceph-mon[36624]: pgmap v16749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:47.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:46 smithi081 ceph-mon[31453]: pgmap v16749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:49.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:49 smithi153 ceph-mon[36569]: pgmap v16750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:49 smithi104 ceph-mon[36624]: pgmap v16750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:49.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:49 smithi081 ceph-mon[31453]: pgmap v16750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:51.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:51 smithi153 ceph-mon[36569]: pgmap v16751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:51 smithi104 ceph-mon[36624]: pgmap v16751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:51 smithi081 ceph-mon[31453]: pgmap v16751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:53 smithi153 ceph-mon[36569]: pgmap v16752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:53 smithi104 ceph-mon[36624]: pgmap v16752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:53.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:53 smithi081 ceph-mon[31453]: pgmap v16752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:55.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:55 smithi153 ceph-mon[36569]: pgmap v16753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:55.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:55 smithi104 ceph-mon[36624]: pgmap v16753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:55.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:55 smithi081 ceph-mon[31453]: pgmap v16753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:57 smithi153 ceph-mon[36569]: pgmap v16754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:57 smithi104 ceph-mon[36624]: pgmap v16754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:57.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:57 smithi081 ceph-mon[31453]: pgmap v16754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:59.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:57:59 smithi153 ceph-mon[36569]: pgmap v16755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:57:59 smithi104 ceph-mon[36624]: pgmap v16755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:57:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:57:59 smithi081 ceph-mon[31453]: pgmap v16755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:00 smithi081 ceph-mon[31453]: pgmap v16756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:00.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:00 smithi153 ceph-mon[36569]: pgmap v16756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:00 smithi104 ceph-mon[36624]: pgmap v16756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:01.575+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:58:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:01.575+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:58:01.578 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:01.575+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:58:02.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:02 smithi153 ceph-mon[36569]: pgmap v16757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:02 smithi104 ceph-mon[36624]: pgmap v16757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:02.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:02 smithi081 ceph-mon[31453]: pgmap v16757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:04.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:04 smithi153 ceph-mon[36569]: pgmap v16758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:04.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:04 smithi104 ceph-mon[36624]: pgmap v16758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:04.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:04 smithi081 ceph-mon[31453]: pgmap v16758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:06.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:06 smithi153 ceph-mon[36569]: pgmap v16759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:06 smithi104 ceph-mon[36624]: pgmap v16759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:06 smithi081 ceph-mon[31453]: pgmap v16759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:08.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:08 smithi153 ceph-mon[36569]: pgmap v16760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:08.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:08 smithi104 ceph-mon[36624]: pgmap v16760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:08 smithi081 ceph-mon[31453]: pgmap v16760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:10.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:10 smithi153 ceph-mon[36569]: pgmap v16761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:10.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:10 smithi104 ceph-mon[36624]: pgmap v16761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:10.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:10 smithi081 ceph-mon[31453]: pgmap v16761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:12.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:12 smithi153 ceph-mon[36569]: pgmap v16762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:12.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:12 smithi104 ceph-mon[36624]: pgmap v16762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:12.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:12 smithi081 ceph-mon[31453]: pgmap v16762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:14.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:14 smithi153 ceph-mon[36569]: pgmap v16763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:14.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:14 smithi104 ceph-mon[36624]: pgmap v16763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:14 smithi081 ceph-mon[31453]: pgmap v16763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:16.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:16 smithi153 ceph-mon[36569]: pgmap v16764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:16.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:16 smithi104 ceph-mon[36624]: pgmap v16764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:16.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:16.591+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:58:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:16.591+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:58:16.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:16.591+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:58:16.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:16 smithi081 ceph-mon[31453]: pgmap v16764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:17.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:17.880+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:58:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:17.880+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:58:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:17.881+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 242 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T05:58:17.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:17.881+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 242 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T05:58:18.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:18 smithi153 ceph-mon[36569]: pgmap v16765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:18.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:18 smithi104 ceph-mon[36624]: pgmap v16765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:18 smithi081 ceph-mon[31453]: pgmap v16765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:20.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:20 smithi153 ceph-mon[36569]: pgmap v16766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:20.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:20 smithi104 ceph-mon[36624]: pgmap v16766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:20.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:20 smithi081 ceph-mon[31453]: pgmap v16766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:22.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:22 smithi104 ceph-mon[36624]: pgmap v16767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:22.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:22 smithi081 ceph-mon[31453]: pgmap v16767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:22.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:22 smithi153 ceph-mon[36569]: pgmap v16767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:24.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:24 smithi104 ceph-mon[36624]: pgmap v16768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:24.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:24 smithi081 ceph-mon[31453]: pgmap v16768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:24.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:24 smithi153 ceph-mon[36569]: pgmap v16768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:26.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:26 smithi104 ceph-mon[36624]: pgmap v16769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:26.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:26 smithi081 ceph-mon[31453]: pgmap v16769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:26.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:26 smithi153 ceph-mon[36569]: pgmap v16769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:28.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:28 smithi104 ceph-mon[36624]: pgmap v16770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:28.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:28 smithi081 ceph-mon[31453]: pgmap v16770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:28.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:28 smithi153 ceph-mon[36569]: pgmap v16770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:30.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:30 smithi104 ceph-mon[36624]: pgmap v16771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:30.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:30 smithi081 ceph-mon[31453]: pgmap v16771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:30.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:30 smithi153 ceph-mon[36569]: pgmap v16771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:31.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:31.606+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:58:31.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:31.607+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:58:31.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:31.607+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:58:32.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:32 smithi081 ceph-mon[31453]: pgmap v16772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:32.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:32 smithi153 ceph-mon[36569]: pgmap v16772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:32.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:32 smithi104 ceph-mon[36624]: pgmap v16772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:34.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:34 smithi081 ceph-mon[31453]: pgmap v16773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:34.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:34 smithi153 ceph-mon[36569]: pgmap v16773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:34 smithi104 ceph-mon[36624]: pgmap v16773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:36.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:36 smithi081 ceph-mon[31453]: pgmap v16774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:36.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:36 smithi153 ceph-mon[36569]: pgmap v16774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:36.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:36 smithi104 ceph-mon[36624]: pgmap v16774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:38 smithi081 ceph-mon[31453]: pgmap v16775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:38.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:58:38.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:38 smithi153 ceph-mon[36569]: pgmap v16775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:38.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:58:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:38 smithi104 ceph-mon[36624]: pgmap v16775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:38.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:58:39.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:58:39.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:58:39.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:39 smithi081 ceph-mon[31453]: pgmap v16776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:39.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:39 smithi081 ceph-mon[31453]: pgmap v16777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:39.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:58:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:58:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:58:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:39 smithi153 ceph-mon[36569]: pgmap v16776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:39 smithi153 ceph-mon[36569]: pgmap v16777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:39.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:58:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:58:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:58:39.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:39 smithi104 ceph-mon[36624]: pgmap v16776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:39.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:39 smithi104 ceph-mon[36624]: pgmap v16777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:39.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:58:42.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:41 smithi153 ceph-mon[36569]: pgmap v16778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:42.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:41 smithi104 ceph-mon[36624]: pgmap v16778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:42.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:41 smithi081 ceph-mon[31453]: pgmap v16778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:44.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:43 smithi153 ceph-mon[36569]: pgmap v16779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:44.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:43 smithi104 ceph-mon[36624]: pgmap v16779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:43 smithi081 ceph-mon[31453]: pgmap v16779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:46.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:45 smithi153 ceph-mon[36569]: pgmap v16780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:46.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:45 smithi104 ceph-mon[36624]: pgmap v16780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:46.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:45 smithi081 ceph-mon[31453]: pgmap v16780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:46.622+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:58:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:46.622+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:58:46.625 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:58:46.622+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:58:48.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:47 smithi153 ceph-mon[36569]: pgmap v16781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:48.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:47 smithi104 ceph-mon[36624]: pgmap v16781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:47 smithi081 ceph-mon[31453]: pgmap v16781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:50.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:49 smithi153 ceph-mon[36569]: pgmap v16782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:50.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:49 smithi104 ceph-mon[36624]: pgmap v16782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:49 smithi081 ceph-mon[31453]: pgmap v16782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:52.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:51 smithi153 ceph-mon[36569]: pgmap v16783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:51 smithi104 ceph-mon[36624]: pgmap v16783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:51 smithi081 ceph-mon[31453]: pgmap v16783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:54.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:53 smithi153 ceph-mon[36569]: pgmap v16784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:53 smithi104 ceph-mon[36624]: pgmap v16784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:53 smithi081 ceph-mon[31453]: pgmap v16784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:56.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:55 smithi153 ceph-mon[36569]: pgmap v16785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:56.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:55 smithi104 ceph-mon[36624]: pgmap v16785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:55 smithi081 ceph-mon[31453]: pgmap v16785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:58.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:57 smithi153 ceph-mon[36569]: pgmap v16786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:57 smithi104 ceph-mon[36624]: pgmap v16786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:58:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:57 smithi081 ceph-mon[31453]: pgmap v16786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:58:59 smithi104 ceph-mon[36624]: pgmap v16787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:58:59 smithi081 ceph-mon[31453]: pgmap v16787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:58:59 smithi153 ceph-mon[36569]: pgmap v16787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:01.638+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:59:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:01.638+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:59:01.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:01.638+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:59:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:01 smithi104 ceph-mon[36624]: pgmap v16788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:01 smithi081 ceph-mon[31453]: pgmap v16788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:01 smithi153 ceph-mon[36569]: pgmap v16788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:04.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:03 smithi104 ceph-mon[36624]: pgmap v16789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:03 smithi081 ceph-mon[31453]: pgmap v16789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:04.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:03 smithi153 ceph-mon[36569]: pgmap v16789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:06.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:05 smithi104 ceph-mon[36624]: pgmap v16790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:05 smithi081 ceph-mon[31453]: pgmap v16790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:05 smithi153 ceph-mon[36569]: pgmap v16790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:07 smithi104 ceph-mon[36624]: pgmap v16791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:08.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:07 smithi081 ceph-mon[31453]: pgmap v16791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:07 smithi153 ceph-mon[36569]: pgmap v16791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:09 smithi081 ceph-mon[31453]: pgmap v16792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:09 smithi153 ceph-mon[36569]: pgmap v16792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:09 smithi104 ceph-mon[36624]: pgmap v16792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:12.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:11 smithi081 ceph-mon[31453]: pgmap v16793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:12.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:11 smithi153 ceph-mon[36569]: pgmap v16793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:11 smithi104 ceph-mon[36624]: pgmap v16793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:13 smithi081 ceph-mon[31453]: pgmap v16794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:13 smithi153 ceph-mon[36569]: pgmap v16794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:13 smithi104 ceph-mon[36624]: pgmap v16794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:15 smithi081 ceph-mon[31453]: pgmap v16795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:16.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:15 smithi153 ceph-mon[36569]: pgmap v16795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:15 smithi104 ceph-mon[36624]: pgmap v16795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:16.643+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:59:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:16.643+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:59:16.646 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:16.643+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:59:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:17 smithi081 ceph-mon[31453]: pgmap v16796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:17 smithi153 ceph-mon[36569]: pgmap v16796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:17 smithi104 ceph-mon[36624]: pgmap v16796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:20.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:19 smithi081 ceph-mon[31453]: pgmap v16797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:19 smithi153 ceph-mon[36569]: pgmap v16797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:19 smithi104 ceph-mon[36624]: pgmap v16797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:21 smithi081 ceph-mon[31453]: pgmap v16798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:21 smithi153 ceph-mon[36569]: pgmap v16798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:21 smithi104 ceph-mon[36624]: pgmap v16798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:23 smithi081 ceph-mon[31453]: pgmap v16799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:24.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:23 smithi153 ceph-mon[36569]: pgmap v16799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:23 smithi104 ceph-mon[36624]: pgmap v16799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:25 smithi081 ceph-mon[31453]: pgmap v16800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:26.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:25 smithi153 ceph-mon[36569]: pgmap v16800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:25 smithi104 ceph-mon[36624]: pgmap v16800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:27 smithi081 ceph-mon[31453]: pgmap v16801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:28.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:27 smithi153 ceph-mon[36569]: pgmap v16801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:27 smithi104 ceph-mon[36624]: pgmap v16801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:29 smithi081 ceph-mon[31453]: pgmap v16802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:29 smithi153 ceph-mon[36569]: pgmap v16802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:29 smithi104 ceph-mon[36624]: pgmap v16802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:31.658+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:59:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:31.658+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:59:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:31.658+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:59:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:31 smithi081 ceph-mon[31453]: pgmap v16803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:32.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:31 smithi153 ceph-mon[36569]: pgmap v16803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:31 smithi104 ceph-mon[36624]: pgmap v16803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:33 smithi081 ceph-mon[31453]: pgmap v16804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:34.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:33 smithi153 ceph-mon[36569]: pgmap v16804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:33 smithi104 ceph-mon[36624]: pgmap v16804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:35 smithi081 ceph-mon[31453]: pgmap v16805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:36.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:35 smithi153 ceph-mon[36569]: pgmap v16805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:35 smithi104 ceph-mon[36624]: pgmap v16805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:38.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:37 smithi153 ceph-mon[36569]: pgmap v16806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:37 smithi104 ceph-mon[36624]: pgmap v16806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:37 smithi081 ceph-mon[31453]: pgmap v16806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:39 smithi153 ceph-mon[36569]: pgmap v16807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:40.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:59:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:39 smithi104 ceph-mon[36624]: pgmap v16807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:59:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:39 smithi081 ceph-mon[31453]: pgmap v16807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T05:59:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:59:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:59:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:59:41.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:59:41.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:59:41.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:59:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T05:59:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T05:59:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T05:59:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:41 smithi153 ceph-mon[36569]: pgmap v16808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:41 smithi153 ceph-mon[36569]: pgmap v16809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:41 smithi104 ceph-mon[36624]: pgmap v16808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:42.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:41 smithi104 ceph-mon[36624]: pgmap v16809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:41 smithi081 ceph-mon[31453]: pgmap v16808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:42.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:41 smithi081 ceph-mon[31453]: pgmap v16809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:43 smithi153 ceph-mon[36569]: pgmap v16810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:43 smithi104 ceph-mon[36624]: pgmap v16810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:43 smithi081 ceph-mon[31453]: pgmap v16810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:45 smithi153 ceph-mon[36569]: pgmap v16811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:45 smithi104 ceph-mon[36624]: pgmap v16811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:45 smithi081 ceph-mon[31453]: pgmap v16811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:46.674+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T05:59:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:46.674+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T05:59:46.677 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T05:59:46.674+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T05:59:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:47 smithi153 ceph-mon[36569]: pgmap v16812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:47 smithi104 ceph-mon[36624]: pgmap v16812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:47 smithi081 ceph-mon[31453]: pgmap v16812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:49 smithi153 ceph-mon[36569]: pgmap v16813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:49 smithi104 ceph-mon[36624]: pgmap v16813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:50.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:49 smithi081 ceph-mon[31453]: pgmap v16813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:52.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:51 smithi153 ceph-mon[36569]: pgmap v16814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:51 smithi104 ceph-mon[36624]: pgmap v16814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:51 smithi081 ceph-mon[31453]: pgmap v16814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:53 smithi153 ceph-mon[36569]: pgmap v16815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:53 smithi104 ceph-mon[36624]: pgmap v16815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:53 smithi081 ceph-mon[31453]: pgmap v16815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:56.298 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:55 smithi081 ceph-mon[31453]: pgmap v16816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:55 smithi153 ceph-mon[36569]: pgmap v16816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:55 smithi104 ceph-mon[36624]: pgmap v16816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:57 smithi153 ceph-mon[36569]: pgmap v16817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:57 smithi104 ceph-mon[36624]: pgmap v16817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T05:59:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:57 smithi081 ceph-mon[31453]: pgmap v16817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 05:59:59 smithi153 ceph-mon[36569]: pgmap v16818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 05:59:59 smithi104 ceph-mon[36624]: pgmap v16818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 05:59:59 smithi081 ceph-mon[31453]: pgmap v16818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:00:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:00:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:00:01.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:01.681+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:00:01.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:01.681+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:00:01.683 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:01.681+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:00:02.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:01 smithi153 ceph-mon[36569]: pgmap v16819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:01 smithi104 ceph-mon[36624]: pgmap v16819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:01 smithi081 ceph-mon[31453]: pgmap v16819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:04 smithi153 ceph-mon[36569]: pgmap v16820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:04 smithi104 ceph-mon[36624]: pgmap v16820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:03 smithi081 ceph-mon[31453]: pgmap v16820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:06.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:06 smithi153 ceph-mon[36569]: pgmap v16821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:06 smithi104 ceph-mon[36624]: pgmap v16821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:06 smithi081 ceph-mon[31453]: pgmap v16821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:08.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:08 smithi153 ceph-mon[36569]: pgmap v16822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:08 smithi104 ceph-mon[36624]: pgmap v16822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:08 smithi081 ceph-mon[31453]: pgmap v16822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:10.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:10 smithi153 ceph-mon[36569]: pgmap v16823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:10 smithi104 ceph-mon[36624]: pgmap v16823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:10 smithi081 ceph-mon[31453]: pgmap v16823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:12.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:12 smithi153 ceph-mon[36569]: pgmap v16824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:12 smithi104 ceph-mon[36624]: pgmap v16824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:12 smithi081 ceph-mon[31453]: pgmap v16824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:14.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:14 smithi153 ceph-mon[36569]: pgmap v16825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:14 smithi104 ceph-mon[36624]: pgmap v16825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:14 smithi081 ceph-mon[31453]: pgmap v16825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:16 smithi153 ceph-mon[36569]: pgmap v16826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:16 smithi104 ceph-mon[36624]: pgmap v16826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:16 smithi081 ceph-mon[31453]: pgmap v16826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:16.693+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:00:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:16.693+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:00:16.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:16.693+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:00:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:18 smithi153 ceph-mon[36569]: pgmap v16827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:18 smithi104 ceph-mon[36624]: pgmap v16827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:18.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:18 smithi081 ceph-mon[31453]: pgmap v16827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:20.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:20 smithi153 ceph-mon[36569]: pgmap v16828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:20 smithi104 ceph-mon[36624]: pgmap v16828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:20.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:20 smithi081 ceph-mon[31453]: pgmap v16828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:22.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:22 smithi153 ceph-mon[36569]: pgmap v16829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:22 smithi104 ceph-mon[36624]: pgmap v16829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:22 smithi081 ceph-mon[31453]: pgmap v16829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:24 smithi153 ceph-mon[36569]: pgmap v16830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:24 smithi104 ceph-mon[36624]: pgmap v16830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:24 smithi081 ceph-mon[31453]: pgmap v16830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:26 smithi104 ceph-mon[36624]: pgmap v16831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:26 smithi081 ceph-mon[31453]: pgmap v16831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:26 smithi153 ceph-mon[36569]: pgmap v16831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:28 smithi104 ceph-mon[36624]: pgmap v16832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:28.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:28 smithi081 ceph-mon[31453]: pgmap v16832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:28 smithi153 ceph-mon[36569]: pgmap v16832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:30.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:29 smithi153 ceph-mon[36569]: pgmap v16833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:29 smithi104 ceph-mon[36624]: pgmap v16833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:29 smithi081 ceph-mon[31453]: pgmap v16833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:31.710 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:31.708+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:00:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:31.708+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:00:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:31.708+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:00:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:31 smithi153 ceph-mon[36569]: pgmap v16834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:31 smithi104 ceph-mon[36624]: pgmap v16834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:32.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:31 smithi081 ceph-mon[31453]: pgmap v16834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:34.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:34 smithi104 ceph-mon[36624]: pgmap v16835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:34 smithi153 ceph-mon[36569]: pgmap v16835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:34 smithi081 ceph-mon[31453]: pgmap v16835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:36 smithi153 ceph-mon[36569]: pgmap v16836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:36.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:36 smithi104 ceph-mon[36624]: pgmap v16836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:36.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:36 smithi081 ceph-mon[31453]: pgmap v16836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:38 smithi153 ceph-mon[36569]: pgmap v16837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:38 smithi104 ceph-mon[36624]: pgmap v16837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:38 smithi081 ceph-mon[31453]: pgmap v16837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:40 smithi153 ceph-mon[36569]: pgmap v16838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:40 smithi104 ceph-mon[36624]: pgmap v16838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:40 smithi081 ceph-mon[31453]: pgmap v16838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:00:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:00:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:00:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:42 smithi153 ceph-mon[36569]: pgmap v16839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:00:42.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:00:42.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:00:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:42 smithi104 ceph-mon[36624]: pgmap v16839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:00:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:00:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:00:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:42 smithi081 ceph-mon[31453]: pgmap v16839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:00:42.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:00:42.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:00:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:43 smithi081 ceph-mon[31453]: pgmap v16840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:43.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:43 smithi081 ceph-mon[31453]: pgmap v16841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:43.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:43 smithi153 ceph-mon[36569]: pgmap v16840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:43.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:43 smithi153 ceph-mon[36569]: pgmap v16841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:43 smithi104 ceph-mon[36624]: pgmap v16840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:43.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:43 smithi104 ceph-mon[36624]: pgmap v16841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:45 smithi153 ceph-mon[36569]: pgmap v16842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:45 smithi104 ceph-mon[36624]: pgmap v16842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:45.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:45 smithi081 ceph-mon[31453]: pgmap v16842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:46.724+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:00:46.726 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:46.724+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:00:46.727 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:00:46.724+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:00:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:47 smithi153 ceph-mon[36569]: pgmap v16843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:47 smithi104 ceph-mon[36624]: pgmap v16843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:47.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:47 smithi081 ceph-mon[31453]: pgmap v16843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:49 smithi104 ceph-mon[36624]: pgmap v16844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:49 smithi081 ceph-mon[31453]: pgmap v16844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:49 smithi153 ceph-mon[36569]: pgmap v16844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:51 smithi104 ceph-mon[36624]: pgmap v16845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:51 smithi081 ceph-mon[31453]: pgmap v16845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:51.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:51 smithi153 ceph-mon[36569]: pgmap v16845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:53 smithi104 ceph-mon[36624]: pgmap v16846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:53 smithi081 ceph-mon[31453]: pgmap v16846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:53 smithi153 ceph-mon[36569]: pgmap v16846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:54 smithi104 ceph-mon[36624]: pgmap v16847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:54 smithi081 ceph-mon[31453]: pgmap v16847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:54.460 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:54 smithi153 ceph-mon[36569]: pgmap v16847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:56 smithi153 ceph-mon[36569]: pgmap v16848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:56 smithi104 ceph-mon[36624]: pgmap v16848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:56 smithi081 ceph-mon[31453]: pgmap v16848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:00:58 smithi153 ceph-mon[36569]: pgmap v16849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:00:58 smithi104 ceph-mon[36624]: pgmap v16849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:00:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:00:58 smithi081 ceph-mon[31453]: pgmap v16849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:00 smithi153 ceph-mon[36569]: pgmap v16850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:00 smithi104 ceph-mon[36624]: pgmap v16850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:00 smithi081 ceph-mon[31453]: pgmap v16850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:01.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:01.739+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:01:01.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:01.739+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:01:01.742 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:01.739+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:01:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:02 smithi153 ceph-mon[36569]: pgmap v16851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:02 smithi104 ceph-mon[36624]: pgmap v16851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:02 smithi081 ceph-mon[31453]: pgmap v16851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:04 smithi153 ceph-mon[36569]: pgmap v16852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:04 smithi104 ceph-mon[36624]: pgmap v16852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:04 smithi081 ceph-mon[31453]: pgmap v16852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:06 smithi153 ceph-mon[36569]: pgmap v16853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:06 smithi104 ceph-mon[36624]: pgmap v16853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:06 smithi081 ceph-mon[31453]: pgmap v16853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:09.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:08 smithi153 ceph-mon[36569]: pgmap v16854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:08 smithi104 ceph-mon[36624]: pgmap v16854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:08 smithi081 ceph-mon[31453]: pgmap v16854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:10 smithi153 ceph-mon[36569]: pgmap v16855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:10 smithi104 ceph-mon[36624]: pgmap v16855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:10 smithi081 ceph-mon[31453]: pgmap v16855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:12.176+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 msgr2=0x7f16e80b1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000602) for more than 900000000 us, fault. 2024-04-13T06:01:12.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:12.176+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 0x7f16e80b1900 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f40bf000 comp rx=0 tx=0).stop 2024-04-13T06:01:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:12.176+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 msgr2=0x7f16e80b1900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T06:01:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:12.176+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 0x7f16e80b1900 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T06:01:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:12.176+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T06:01:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:12.177+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T06:01:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:12.178+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1e40 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006750 tx=0x7f16f4044000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T06:01:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:12 smithi153 ceph-mon[36569]: pgmap v16856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:12 smithi104 ceph-mon[36624]: pgmap v16856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:12 smithi081 ceph-mon[31453]: pgmap v16856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:15.037 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:14 smithi081 ceph-mon[31453]: pgmap v16857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:14 smithi153 ceph-mon[36569]: pgmap v16857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:14 smithi104 ceph-mon[36624]: pgmap v16857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:16.754+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:01:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:16.754+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:01:16.758 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:16.754+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:01:16.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:16 smithi081 ceph-mon[31453]: pgmap v16858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:16 smithi153 ceph-mon[36569]: pgmap v16858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:16 smithi104 ceph-mon[36624]: pgmap v16858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:17.909+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:01:17.912 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:17.909+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:01:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:17.909+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 243 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:01:17.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:17.910+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 243 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:01:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:18 smithi153 ceph-mon[36569]: pgmap v16859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:18 smithi104 ceph-mon[36624]: pgmap v16859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:19.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:18 smithi081 ceph-mon[31453]: pgmap v16859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:20 smithi153 ceph-mon[36569]: pgmap v16860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:20 smithi104 ceph-mon[36624]: pgmap v16860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:20 smithi081 ceph-mon[31453]: pgmap v16860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:22 smithi153 ceph-mon[36569]: pgmap v16861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:22 smithi104 ceph-mon[36624]: pgmap v16861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:22 smithi081 ceph-mon[31453]: pgmap v16861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:24 smithi153 ceph-mon[36569]: pgmap v16862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:25.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:24 smithi104 ceph-mon[36624]: pgmap v16862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:24 smithi081 ceph-mon[31453]: pgmap v16862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:26.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:26 smithi081 ceph-mon[31453]: pgmap v16863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:26 smithi153 ceph-mon[36569]: pgmap v16863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:26 smithi104 ceph-mon[36624]: pgmap v16863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:28 smithi153 ceph-mon[36569]: pgmap v16864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:28 smithi104 ceph-mon[36624]: pgmap v16864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:28 smithi081 ceph-mon[31453]: pgmap v16864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:30 smithi104 ceph-mon[36624]: pgmap v16865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:30 smithi081 ceph-mon[31453]: pgmap v16865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:30 smithi153 ceph-mon[36569]: pgmap v16865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:31.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:31.765+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:01:31.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:31.765+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:01:31.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:31.765+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:01:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:32 smithi104 ceph-mon[36624]: pgmap v16866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:32 smithi081 ceph-mon[31453]: pgmap v16866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:32 smithi153 ceph-mon[36569]: pgmap v16866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:34 smithi104 ceph-mon[36624]: pgmap v16867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:34 smithi081 ceph-mon[31453]: pgmap v16867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:34 smithi153 ceph-mon[36569]: pgmap v16867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:36 smithi104 ceph-mon[36624]: pgmap v16868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:36 smithi081 ceph-mon[31453]: pgmap v16868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:36 smithi153 ceph-mon[36569]: pgmap v16868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:38 smithi104 ceph-mon[36624]: pgmap v16869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:38 smithi081 ceph-mon[31453]: pgmap v16869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:38 smithi153 ceph-mon[36569]: pgmap v16869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:40 smithi081 ceph-mon[31453]: pgmap v16870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:40 smithi153 ceph-mon[36569]: pgmap v16870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:40 smithi104 ceph-mon[36624]: pgmap v16870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:42 smithi081 ceph-mon[31453]: pgmap v16871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:01:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:42 smithi153 ceph-mon[36569]: pgmap v16871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:01:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:42 smithi104 ceph-mon[36624]: pgmap v16871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:01:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:01:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:01:44.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:01:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:01:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:01:44.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:01:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:01:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:01:44.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:01:45.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:44 smithi081 ceph-mon[31453]: pgmap v16872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:45.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:44 smithi081 ceph-mon[31453]: pgmap v16873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:44 smithi153 ceph-mon[36569]: pgmap v16872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:44 smithi153 ceph-mon[36569]: pgmap v16873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:44 smithi104 ceph-mon[36624]: pgmap v16872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:44 smithi104 ceph-mon[36624]: pgmap v16873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:46.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:46.781+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:01:46.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:46.781+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:01:46.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:01:46.781+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:01:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:46 smithi081 ceph-mon[31453]: pgmap v16874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:46 smithi153 ceph-mon[36569]: pgmap v16874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:46 smithi104 ceph-mon[36624]: pgmap v16874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:48 smithi081 ceph-mon[31453]: pgmap v16875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:48 smithi153 ceph-mon[36569]: pgmap v16875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:48 smithi104 ceph-mon[36624]: pgmap v16875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:50 smithi081 ceph-mon[31453]: pgmap v16876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:50 smithi153 ceph-mon[36569]: pgmap v16876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:50 smithi104 ceph-mon[36624]: pgmap v16876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:52 smithi081 ceph-mon[31453]: pgmap v16877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:52 smithi153 ceph-mon[36569]: pgmap v16877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:52 smithi104 ceph-mon[36624]: pgmap v16877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:54 smithi081 ceph-mon[31453]: pgmap v16878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:54 smithi153 ceph-mon[36569]: pgmap v16878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:54 smithi104 ceph-mon[36624]: pgmap v16878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:56 smithi081 ceph-mon[31453]: pgmap v16879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:56 smithi153 ceph-mon[36569]: pgmap v16879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:56 smithi104 ceph-mon[36624]: pgmap v16879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:01:58 smithi081 ceph-mon[31453]: pgmap v16880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:01:58 smithi153 ceph-mon[36569]: pgmap v16880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:01:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:01:58 smithi104 ceph-mon[36624]: pgmap v16880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:00 smithi081 ceph-mon[31453]: pgmap v16881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:00 smithi153 ceph-mon[36569]: pgmap v16881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:00 smithi104 ceph-mon[36624]: pgmap v16881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:01.796+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:02:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:01.796+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:02:01.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:01.796+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:02:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:02 smithi081 ceph-mon[31453]: pgmap v16882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:02 smithi153 ceph-mon[36569]: pgmap v16882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:02 smithi104 ceph-mon[36624]: pgmap v16882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:04 smithi081 ceph-mon[31453]: pgmap v16883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:04 smithi153 ceph-mon[36569]: pgmap v16883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:04 smithi104 ceph-mon[36624]: pgmap v16883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:06 smithi153 ceph-mon[36569]: pgmap v16884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:06 smithi104 ceph-mon[36624]: pgmap v16884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:06 smithi081 ceph-mon[31453]: pgmap v16884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:08 smithi153 ceph-mon[36569]: pgmap v16885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:08 smithi104 ceph-mon[36624]: pgmap v16885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:08 smithi081 ceph-mon[31453]: pgmap v16885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:10 smithi153 ceph-mon[36569]: pgmap v16886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:10 smithi104 ceph-mon[36624]: pgmap v16886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:10 smithi081 ceph-mon[31453]: pgmap v16886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:12 smithi153 ceph-mon[36569]: pgmap v16887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:12 smithi104 ceph-mon[36624]: pgmap v16887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:12 smithi081 ceph-mon[31453]: pgmap v16887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:14 smithi153 ceph-mon[36569]: pgmap v16888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:14 smithi104 ceph-mon[36624]: pgmap v16888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:15.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:14 smithi081 ceph-mon[31453]: pgmap v16888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:16.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:16.812+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:02:16.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:16.812+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:02:16.815 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:16.812+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:02:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:16 smithi153 ceph-mon[36569]: pgmap v16889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:16 smithi104 ceph-mon[36624]: pgmap v16889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:16 smithi081 ceph-mon[31453]: pgmap v16889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:18 smithi153 ceph-mon[36569]: pgmap v16890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:18 smithi104 ceph-mon[36624]: pgmap v16890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:19.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:18 smithi081 ceph-mon[31453]: pgmap v16890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:20 smithi153 ceph-mon[36569]: pgmap v16891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:20 smithi104 ceph-mon[36624]: pgmap v16891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:20 smithi081 ceph-mon[31453]: pgmap v16891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:22 smithi153 ceph-mon[36569]: pgmap v16892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:22 smithi104 ceph-mon[36624]: pgmap v16892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:22 smithi081 ceph-mon[31453]: pgmap v16892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:24 smithi153 ceph-mon[36569]: pgmap v16893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:24 smithi104 ceph-mon[36624]: pgmap v16893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:24 smithi081 ceph-mon[31453]: pgmap v16893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:26 smithi153 ceph-mon[36569]: pgmap v16894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:26 smithi104 ceph-mon[36624]: pgmap v16894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:26 smithi081 ceph-mon[31453]: pgmap v16894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:28 smithi153 ceph-mon[36569]: pgmap v16895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:28 smithi104 ceph-mon[36624]: pgmap v16895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:28 smithi081 ceph-mon[31453]: pgmap v16895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:31.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:30 smithi081 ceph-mon[31453]: pgmap v16896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:30 smithi153 ceph-mon[36569]: pgmap v16896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:30 smithi104 ceph-mon[36624]: pgmap v16896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:31.822+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:02:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:31.822+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:02:31.825 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:31.822+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:02:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:32 smithi153 ceph-mon[36569]: pgmap v16897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:33.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:32 smithi104 ceph-mon[36624]: pgmap v16897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:32 smithi081 ceph-mon[31453]: pgmap v16897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:34 smithi153 ceph-mon[36569]: pgmap v16898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:34 smithi104 ceph-mon[36624]: pgmap v16898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:34 smithi081 ceph-mon[31453]: pgmap v16898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:36 smithi153 ceph-mon[36569]: pgmap v16899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:36 smithi104 ceph-mon[36624]: pgmap v16899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:37.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:36 smithi081 ceph-mon[31453]: pgmap v16899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:38 smithi153 ceph-mon[36569]: pgmap v16900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:39.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:38 smithi104 ceph-mon[36624]: pgmap v16900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:39.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:38 smithi081 ceph-mon[31453]: pgmap v16900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:41 smithi153 ceph-mon[36569]: pgmap v16901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:41 smithi104 ceph-mon[36624]: pgmap v16901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:41 smithi081 ceph-mon[31453]: pgmap v16901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:43 smithi153 ceph-mon[36569]: pgmap v16902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:43 smithi104 ceph-mon[36624]: pgmap v16902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:43.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:43 smithi081 ceph-mon[31453]: pgmap v16902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:02:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:02:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:02:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: pgmap v16903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:02:45.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:02:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: pgmap v16903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:02:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: pgmap v16903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:02:45.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:02:45.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:02:45.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:02:46.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:46 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:02:46.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:46 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:02:46.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:46 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:02:46.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:46 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:02:46.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:46 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:02:46.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:46 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:02:46.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:46 smithi081 ceph-mon[31453]: pgmap v16904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:46.282 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:46 smithi081 ceph-mon[31453]: pgmap v16905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:46 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:02:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:46 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:02:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:46 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:02:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:46 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:02:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:46 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:02:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:46 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:02:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:46 smithi153 ceph-mon[36569]: pgmap v16904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:46.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:46 smithi153 ceph-mon[36569]: pgmap v16905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:46 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:02:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:46 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:02:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:46 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:02:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:46 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:02:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:46 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:02:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:46 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:02:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:46 smithi104 ceph-mon[36624]: pgmap v16904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:46 smithi104 ceph-mon[36624]: pgmap v16905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:46.838+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:02:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:46.838+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:02:46.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:02:46.838+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:02:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:48 smithi153 ceph-mon[36569]: pgmap v16906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:48 smithi104 ceph-mon[36624]: pgmap v16906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:48 smithi081 ceph-mon[31453]: pgmap v16906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:50 smithi153 ceph-mon[36569]: pgmap v16907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:50 smithi104 ceph-mon[36624]: pgmap v16907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:50 smithi081 ceph-mon[31453]: pgmap v16907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:52 smithi153 ceph-mon[36569]: pgmap v16908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:52 smithi104 ceph-mon[36624]: pgmap v16908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:52 smithi081 ceph-mon[31453]: pgmap v16908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:54 smithi153 ceph-mon[36569]: pgmap v16909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:54.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:54 smithi104 ceph-mon[36624]: pgmap v16909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:54 smithi081 ceph-mon[31453]: pgmap v16909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:56.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:56 smithi153 ceph-mon[36569]: pgmap v16910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:56 smithi104 ceph-mon[36624]: pgmap v16910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:56.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:56 smithi081 ceph-mon[31453]: pgmap v16910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:02:58 smithi104 ceph-mon[36624]: pgmap v16911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:02:58 smithi081 ceph-mon[31453]: pgmap v16911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:02:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:02:58 smithi153 ceph-mon[36569]: pgmap v16911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:00 smithi104 ceph-mon[36624]: pgmap v16912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:00 smithi081 ceph-mon[31453]: pgmap v16912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:00 smithi153 ceph-mon[36569]: pgmap v16912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:01.853+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:03:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:01.853+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:03:01.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:01.853+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:03:02.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:02 smithi104 ceph-mon[36624]: pgmap v16913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:02 smithi081 ceph-mon[31453]: pgmap v16913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:02 smithi153 ceph-mon[36569]: pgmap v16913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:04 smithi104 ceph-mon[36624]: pgmap v16914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:04 smithi081 ceph-mon[31453]: pgmap v16914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:04 smithi153 ceph-mon[36569]: pgmap v16914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:06 smithi104 ceph-mon[36624]: pgmap v16915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:06 smithi081 ceph-mon[31453]: pgmap v16915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:06 smithi153 ceph-mon[36569]: pgmap v16915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:08 smithi081 ceph-mon[31453]: pgmap v16916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:08 smithi153 ceph-mon[36569]: pgmap v16916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:08 smithi104 ceph-mon[36624]: pgmap v16916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:10 smithi081 ceph-mon[31453]: pgmap v16917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:10 smithi153 ceph-mon[36569]: pgmap v16917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:10.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:10 smithi104 ceph-mon[36624]: pgmap v16917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:11 smithi081 ceph-mon[31453]: pgmap v16918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:11.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:11 smithi153 ceph-mon[36569]: pgmap v16918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:11.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:11 smithi104 ceph-mon[36624]: pgmap v16918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:13 smithi081 ceph-mon[31453]: pgmap v16919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:13 smithi153 ceph-mon[36569]: pgmap v16919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:13 smithi104 ceph-mon[36624]: pgmap v16919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:15 smithi153 ceph-mon[36569]: pgmap v16920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:15 smithi104 ceph-mon[36624]: pgmap v16920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:15 smithi081 ceph-mon[31453]: pgmap v16920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:16.869+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:03:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:16.869+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:03:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:16.869+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:03:18.212 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:17 smithi081 ceph-mon[31453]: pgmap v16921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:17 smithi153 ceph-mon[36569]: pgmap v16921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:17 smithi104 ceph-mon[36624]: pgmap v16921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:19 smithi153 ceph-mon[36569]: pgmap v16922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:19 smithi104 ceph-mon[36624]: pgmap v16922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:20.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:19 smithi081 ceph-mon[31453]: pgmap v16922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:21 smithi153 ceph-mon[36569]: pgmap v16923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:21 smithi104 ceph-mon[36624]: pgmap v16923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:21 smithi081 ceph-mon[31453]: pgmap v16923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:23 smithi153 ceph-mon[36569]: pgmap v16924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:23 smithi104 ceph-mon[36624]: pgmap v16924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:23 smithi081 ceph-mon[31453]: pgmap v16924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:25 smithi153 ceph-mon[36569]: pgmap v16925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:25 smithi104 ceph-mon[36624]: pgmap v16925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:25 smithi081 ceph-mon[31453]: pgmap v16925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:27 smithi153 ceph-mon[36569]: pgmap v16926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:27 smithi104 ceph-mon[36624]: pgmap v16926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:28.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:27 smithi081 ceph-mon[31453]: pgmap v16926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:30 smithi153 ceph-mon[36569]: pgmap v16927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:30 smithi104 ceph-mon[36624]: pgmap v16927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:30 smithi081 ceph-mon[31453]: pgmap v16927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:31.881+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:03:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:31.881+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:03:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:31.881+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:03:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:32 smithi104 ceph-mon[36624]: pgmap v16928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:32.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:32 smithi081 ceph-mon[31453]: pgmap v16928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:32 smithi153 ceph-mon[36569]: pgmap v16928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:34 smithi104 ceph-mon[36624]: pgmap v16929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:34 smithi081 ceph-mon[31453]: pgmap v16929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:34 smithi153 ceph-mon[36569]: pgmap v16929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:36 smithi104 ceph-mon[36624]: pgmap v16930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:36.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:36 smithi081 ceph-mon[31453]: pgmap v16930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:36 smithi153 ceph-mon[36569]: pgmap v16930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:38 smithi104 ceph-mon[36624]: pgmap v16931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:38 smithi081 ceph-mon[31453]: pgmap v16931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:38 smithi153 ceph-mon[36569]: pgmap v16931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:40 smithi104 ceph-mon[36624]: pgmap v16932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:40 smithi081 ceph-mon[31453]: pgmap v16932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:40 smithi153 ceph-mon[36569]: pgmap v16932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:42 smithi081 ceph-mon[31453]: pgmap v16933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:42 smithi153 ceph-mon[36569]: pgmap v16933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:42 smithi104 ceph-mon[36624]: pgmap v16933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:44 smithi081 ceph-mon[31453]: pgmap v16934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:44 smithi153 ceph-mon[36569]: pgmap v16934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:44 smithi104 ceph-mon[36624]: pgmap v16934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:46 smithi081 ceph-mon[31453]: pgmap v16935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:03:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:46 smithi153 ceph-mon[36569]: pgmap v16935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:03:46.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:46 smithi104 ceph-mon[36624]: pgmap v16935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:46.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:03:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:46.897+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:03:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:46.897+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:03:46.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:03:46.897+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:03:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:03:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:03:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:03:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:03:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:03:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:03:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:03:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:03:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:03:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:48 smithi081 ceph-mon[31453]: pgmap v16936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:48 smithi081 ceph-mon[31453]: pgmap v16937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:48 smithi153 ceph-mon[36569]: pgmap v16936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:48 smithi153 ceph-mon[36569]: pgmap v16937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:48.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:48 smithi104 ceph-mon[36624]: pgmap v16936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:48.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:48 smithi104 ceph-mon[36624]: pgmap v16937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:49 smithi081 ceph-mon[31453]: pgmap v16938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:49 smithi153 ceph-mon[36569]: pgmap v16938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:49 smithi104 ceph-mon[36624]: pgmap v16938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:51 smithi153 ceph-mon[36569]: pgmap v16939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:51 smithi104 ceph-mon[36624]: pgmap v16939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:51.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:51 smithi081 ceph-mon[31453]: pgmap v16939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:53 smithi153 ceph-mon[36569]: pgmap v16940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:53.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:53 smithi104 ceph-mon[36624]: pgmap v16940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:53.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:53 smithi081 ceph-mon[31453]: pgmap v16940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:55 smithi153 ceph-mon[36569]: pgmap v16941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:55 smithi104 ceph-mon[36624]: pgmap v16941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:55 smithi081 ceph-mon[31453]: pgmap v16941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:57 smithi153 ceph-mon[36569]: pgmap v16942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:57 smithi104 ceph-mon[36624]: pgmap v16942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:57 smithi081 ceph-mon[31453]: pgmap v16942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:03:59 smithi153 ceph-mon[36569]: pgmap v16943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:03:59 smithi104 ceph-mon[36624]: pgmap v16943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:03:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:03:59 smithi081 ceph-mon[31453]: pgmap v16943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:01 smithi153 ceph-mon[36569]: pgmap v16944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:01.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:01 smithi104 ceph-mon[36624]: pgmap v16944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:01.904+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:04:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:01.904+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:04:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:01.904+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:04:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:01 smithi081 ceph-mon[31453]: pgmap v16944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:03 smithi153 ceph-mon[36569]: pgmap v16945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:03.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:03 smithi104 ceph-mon[36624]: pgmap v16945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:03.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:03 smithi081 ceph-mon[31453]: pgmap v16945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:05 smithi153 ceph-mon[36569]: pgmap v16946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:05 smithi104 ceph-mon[36624]: pgmap v16946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:05.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:05 smithi081 ceph-mon[31453]: pgmap v16946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:07 smithi153 ceph-mon[36569]: pgmap v16947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:07.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:07 smithi104 ceph-mon[36624]: pgmap v16947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:07.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:07 smithi081 ceph-mon[31453]: pgmap v16947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:09 smithi153 ceph-mon[36569]: pgmap v16948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:09.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:09 smithi104 ceph-mon[36624]: pgmap v16948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:09.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:09 smithi081 ceph-mon[31453]: pgmap v16948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:11 smithi153 ceph-mon[36569]: pgmap v16949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:11 smithi104 ceph-mon[36624]: pgmap v16949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:11.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:11 smithi081 ceph-mon[31453]: pgmap v16949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:13 smithi153 ceph-mon[36569]: pgmap v16950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:13 smithi104 ceph-mon[36624]: pgmap v16950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:13 smithi081 ceph-mon[31453]: pgmap v16950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:15 smithi153 ceph-mon[36569]: pgmap v16951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:15 smithi104 ceph-mon[36624]: pgmap v16951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:15 smithi081 ceph-mon[31453]: pgmap v16951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:16.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:16.919+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:04:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:16.919+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:04:16.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:16.919+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:04:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:17 smithi153 ceph-mon[36569]: pgmap v16952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:17.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:17 smithi104 ceph-mon[36624]: pgmap v16952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:17.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:17 smithi081 ceph-mon[31453]: pgmap v16952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:17.940+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:04:17.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:17.940+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:04:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:17.940+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 244 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:04:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:17.940+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 244 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:04:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:17.940+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:04:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:17.940+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:04:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:17.941+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 245 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:04:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:17.941+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 245 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:04:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:19 smithi153 ceph-mon[36569]: pgmap v16953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:19 smithi104 ceph-mon[36624]: pgmap v16953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:19 smithi081 ceph-mon[31453]: pgmap v16953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:21 smithi153 ceph-mon[36569]: pgmap v16954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:21.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:21 smithi104 ceph-mon[36624]: pgmap v16954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:21 smithi081 ceph-mon[31453]: pgmap v16954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:23 smithi153 ceph-mon[36569]: pgmap v16955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:23.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:23 smithi104 ceph-mon[36624]: pgmap v16955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:23 smithi081 ceph-mon[31453]: pgmap v16955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:25 smithi153 ceph-mon[36569]: pgmap v16956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:25 smithi104 ceph-mon[36624]: pgmap v16956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:25 smithi081 ceph-mon[31453]: pgmap v16956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:27 smithi153 ceph-mon[36569]: pgmap v16957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:27 smithi104 ceph-mon[36624]: pgmap v16957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:27 smithi081 ceph-mon[31453]: pgmap v16957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:30 smithi104 ceph-mon[36624]: pgmap v16958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:30 smithi081 ceph-mon[31453]: pgmap v16958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:30 smithi153 ceph-mon[36569]: pgmap v16958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:31.931 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:31.929+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:04:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:31.929+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:04:31.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:31.929+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:04:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:32 smithi104 ceph-mon[36624]: pgmap v16959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:32.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:32 smithi081 ceph-mon[31453]: pgmap v16959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:32 smithi153 ceph-mon[36569]: pgmap v16959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:34 smithi081 ceph-mon[31453]: pgmap v16960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:34 smithi153 ceph-mon[36569]: pgmap v16960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:34.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:34 smithi104 ceph-mon[36624]: pgmap v16960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:36.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:36 smithi081 ceph-mon[31453]: pgmap v16961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:36 smithi153 ceph-mon[36569]: pgmap v16961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:36.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:36 smithi104 ceph-mon[36624]: pgmap v16961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:38.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:38 smithi104 ceph-mon[36624]: pgmap v16962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:38 smithi081 ceph-mon[31453]: pgmap v16962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:38 smithi153 ceph-mon[36569]: pgmap v16962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:40 smithi081 ceph-mon[31453]: pgmap v16963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:40 smithi153 ceph-mon[36569]: pgmap v16963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:40 smithi104 ceph-mon[36624]: pgmap v16963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:42 smithi081 ceph-mon[31453]: pgmap v16964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:42 smithi153 ceph-mon[36569]: pgmap v16964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:42 smithi104 ceph-mon[36624]: pgmap v16964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:44 smithi081 ceph-mon[31453]: pgmap v16965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:44 smithi153 ceph-mon[36569]: pgmap v16965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:44 smithi104 ceph-mon[36624]: pgmap v16965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:45.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:45 smithi153 ceph-mon[36569]: pgmap v16966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:45 smithi104 ceph-mon[36624]: pgmap v16966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:45.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:45 smithi081 ceph-mon[31453]: pgmap v16966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:46.943+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:04:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:46.943+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:04:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:04:46.943+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:04:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:47 smithi153 ceph-mon[36569]: pgmap v16967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:04:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:47 smithi104 ceph-mon[36624]: pgmap v16967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:04:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:47 smithi081 ceph-mon[31453]: pgmap v16967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:04:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:04:48.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:04:48.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:48 smithi153 ceph-mon[36569]: pgmap v16968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:48.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:48 smithi153 ceph-mon[36569]: pgmap v16969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:48.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:04:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:04:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:04:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:48 smithi104 ceph-mon[36624]: pgmap v16968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:48 smithi104 ceph-mon[36624]: pgmap v16969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:04:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:04:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:04:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:48 smithi081 ceph-mon[31453]: pgmap v16968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:48 smithi081 ceph-mon[31453]: pgmap v16969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:04:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:50 smithi153 ceph-mon[36569]: pgmap v16970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:50 smithi104 ceph-mon[36624]: pgmap v16970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:50 smithi081 ceph-mon[31453]: pgmap v16970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:52 smithi153 ceph-mon[36569]: pgmap v16971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:52 smithi104 ceph-mon[36624]: pgmap v16971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:52 smithi081 ceph-mon[31453]: pgmap v16971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:54 smithi153 ceph-mon[36569]: pgmap v16972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:54 smithi104 ceph-mon[36624]: pgmap v16972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:55.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:54 smithi081 ceph-mon[31453]: pgmap v16972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:56 smithi153 ceph-mon[36569]: pgmap v16973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:56 smithi104 ceph-mon[36624]: pgmap v16973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:57.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:56 smithi081 ceph-mon[31453]: pgmap v16973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:04:58 smithi153 ceph-mon[36569]: pgmap v16974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:04:58 smithi104 ceph-mon[36624]: pgmap v16974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:04:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:04:58 smithi081 ceph-mon[31453]: pgmap v16974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:01 smithi153 ceph-mon[36569]: pgmap v16975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:01 smithi104 ceph-mon[36624]: pgmap v16975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:01 smithi081 ceph-mon[31453]: pgmap v16975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:01.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:05:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:01.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:05:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:01.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:05:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:03 smithi153 ceph-mon[36569]: pgmap v16976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:03 smithi104 ceph-mon[36624]: pgmap v16976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:03 smithi081 ceph-mon[31453]: pgmap v16976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:05 smithi153 ceph-mon[36569]: pgmap v16977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:05 smithi104 ceph-mon[36624]: pgmap v16977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:05 smithi081 ceph-mon[31453]: pgmap v16977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:07 smithi153 ceph-mon[36569]: pgmap v16978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:07 smithi104 ceph-mon[36624]: pgmap v16978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:07 smithi081 ceph-mon[31453]: pgmap v16978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:09 smithi153 ceph-mon[36569]: pgmap v16979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:09 smithi104 ceph-mon[36624]: pgmap v16979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:09 smithi081 ceph-mon[31453]: pgmap v16979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:11 smithi153 ceph-mon[36569]: pgmap v16980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:11 smithi104 ceph-mon[36624]: pgmap v16980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:11 smithi081 ceph-mon[31453]: pgmap v16980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:13 smithi153 ceph-mon[36569]: pgmap v16981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:13 smithi104 ceph-mon[36624]: pgmap v16981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:13 smithi081 ceph-mon[31453]: pgmap v16981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:15 smithi153 ceph-mon[36569]: pgmap v16982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:15 smithi104 ceph-mon[36624]: pgmap v16982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:15.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:15 smithi081 ceph-mon[31453]: pgmap v16982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:16.962+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:05:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:16.962+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:05:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:16.962+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:05:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:17 smithi153 ceph-mon[36569]: pgmap v16983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:17 smithi104 ceph-mon[36624]: pgmap v16983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:17 smithi081 ceph-mon[31453]: pgmap v16983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:19 smithi153 ceph-mon[36569]: pgmap v16984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:19 smithi104 ceph-mon[36624]: pgmap v16984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:19.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:19 smithi081 ceph-mon[31453]: pgmap v16984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:21 smithi153 ceph-mon[36569]: pgmap v16985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:21 smithi104 ceph-mon[36624]: pgmap v16985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:21 smithi081 ceph-mon[31453]: pgmap v16985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:23 smithi104 ceph-mon[36624]: pgmap v16986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:23 smithi081 ceph-mon[31453]: pgmap v16986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:23 smithi153 ceph-mon[36569]: pgmap v16986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:25 smithi104 ceph-mon[36624]: pgmap v16987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:25 smithi081 ceph-mon[31453]: pgmap v16987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:25 smithi153 ceph-mon[36569]: pgmap v16987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:27 smithi104 ceph-mon[36624]: pgmap v16988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:27.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:27 smithi153 ceph-mon[36569]: pgmap v16988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:27 smithi081 ceph-mon[31453]: pgmap v16988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:29 smithi104 ceph-mon[36624]: pgmap v16989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:29 smithi081 ceph-mon[31453]: pgmap v16989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:29 smithi153 ceph-mon[36569]: pgmap v16989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:31 smithi104 ceph-mon[36624]: pgmap v16990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:31 smithi081 ceph-mon[31453]: pgmap v16990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:31 smithi153 ceph-mon[36569]: pgmap v16990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:31.977+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:05:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:31.977+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:05:31.980 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:31.977+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:05:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:33 smithi081 ceph-mon[31453]: pgmap v16991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:33 smithi153 ceph-mon[36569]: pgmap v16991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:33 smithi104 ceph-mon[36624]: pgmap v16991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:35 smithi081 ceph-mon[31453]: pgmap v16992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:35 smithi153 ceph-mon[36569]: pgmap v16992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:35.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:35 smithi104 ceph-mon[36624]: pgmap v16992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:37.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:37 smithi081 ceph-mon[31453]: pgmap v16993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:37.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:37 smithi153 ceph-mon[36569]: pgmap v16993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:37.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:37 smithi104 ceph-mon[36624]: pgmap v16993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:39.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:39 smithi104 ceph-mon[36624]: pgmap v16994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:39.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:39 smithi081 ceph-mon[31453]: pgmap v16994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:39.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:39 smithi153 ceph-mon[36569]: pgmap v16994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:41 smithi081 ceph-mon[31453]: pgmap v16995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:41 smithi153 ceph-mon[36569]: pgmap v16995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:41.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:41 smithi104 ceph-mon[36624]: pgmap v16995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:43.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:43 smithi081 ceph-mon[31453]: pgmap v16996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:43 smithi153 ceph-mon[36569]: pgmap v16996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:43.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:43 smithi104 ceph-mon[36624]: pgmap v16996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:45.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:45 smithi081 ceph-mon[31453]: pgmap v16997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:45.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:45 smithi153 ceph-mon[36569]: pgmap v16997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:45 smithi104 ceph-mon[36624]: pgmap v16997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:46.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:46.989+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:05:46.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:46.989+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:05:46.991 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:05:46.989+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:05:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:47 smithi081 ceph-mon[31453]: pgmap v16998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:47 smithi153 ceph-mon[36569]: pgmap v16998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:47 smithi104 ceph-mon[36624]: pgmap v16998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:48 smithi153 ceph-mon[36569]: pgmap v16999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:48.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:48 smithi104 ceph-mon[36624]: pgmap v16999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:48 smithi081 ceph-mon[31453]: pgmap v16999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:05:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:05:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:05:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:05:50.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:05:50.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:50 smithi153 ceph-mon[36569]: pgmap v17000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:50.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:50 smithi153 ceph-mon[36569]: pgmap v17001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:50.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:05:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:05:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:05:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:50 smithi104 ceph-mon[36624]: pgmap v17000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:50 smithi104 ceph-mon[36624]: pgmap v17001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:05:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:05:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:05:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:50 smithi081 ceph-mon[31453]: pgmap v17000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:50 smithi081 ceph-mon[31453]: pgmap v17001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:05:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:52 smithi153 ceph-mon[36569]: pgmap v17002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:52.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:52 smithi104 ceph-mon[36624]: pgmap v17002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:52 smithi081 ceph-mon[31453]: pgmap v17002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:54 smithi153 ceph-mon[36569]: pgmap v17003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:54.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:54 smithi104 ceph-mon[36624]: pgmap v17003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:54 smithi081 ceph-mon[31453]: pgmap v17003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:56 smithi153 ceph-mon[36569]: pgmap v17004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:56 smithi104 ceph-mon[36624]: pgmap v17004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:56.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:56 smithi081 ceph-mon[31453]: pgmap v17004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:58.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:05:58 smithi153 ceph-mon[36569]: pgmap v17005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:58.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:05:58 smithi104 ceph-mon[36624]: pgmap v17005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:05:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:05:58 smithi081 ceph-mon[31453]: pgmap v17005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:00.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:00 smithi153 ceph-mon[36569]: pgmap v17006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:00.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:00 smithi104 ceph-mon[36624]: pgmap v17006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:00 smithi081 ceph-mon[31453]: pgmap v17006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:02.004+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:06:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:02.004+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:06:02.006 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:02.004+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:06:02.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:02 smithi153 ceph-mon[36569]: pgmap v17007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:02 smithi104 ceph-mon[36624]: pgmap v17007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:02 smithi081 ceph-mon[31453]: pgmap v17007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:04.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:04 smithi153 ceph-mon[36569]: pgmap v17008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:04.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:04 smithi104 ceph-mon[36624]: pgmap v17008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:04.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:04 smithi081 ceph-mon[31453]: pgmap v17008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:06.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:06 smithi153 ceph-mon[36569]: pgmap v17009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:06.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:06 smithi104 ceph-mon[36624]: pgmap v17009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:06 smithi081 ceph-mon[31453]: pgmap v17009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:08 smithi153 ceph-mon[36569]: pgmap v17010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:08.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:08 smithi104 ceph-mon[36624]: pgmap v17010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:08.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:08 smithi081 ceph-mon[31453]: pgmap v17010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:10.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:10 smithi153 ceph-mon[36569]: pgmap v17011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:10.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:10 smithi104 ceph-mon[36624]: pgmap v17011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:10.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:10 smithi081 ceph-mon[31453]: pgmap v17011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:12.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:12 smithi153 ceph-mon[36569]: pgmap v17012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:12.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:12 smithi104 ceph-mon[36624]: pgmap v17012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:12.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:12 smithi081 ceph-mon[31453]: pgmap v17012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:14.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:14 smithi153 ceph-mon[36569]: pgmap v17013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:14.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:14 smithi104 ceph-mon[36624]: pgmap v17013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:14.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:14 smithi081 ceph-mon[31453]: pgmap v17013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:16 smithi153 ceph-mon[36569]: pgmap v17014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:16 smithi104 ceph-mon[36624]: pgmap v17014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:16.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:16 smithi081 ceph-mon[31453]: pgmap v17014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:17.007+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:06:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:17.007+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:06:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:17.007+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:06:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:18 smithi153 ceph-mon[36569]: pgmap v17015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:18.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:18 smithi104 ceph-mon[36624]: pgmap v17015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:18.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:18 smithi081 ceph-mon[31453]: pgmap v17015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:20 smithi104 ceph-mon[36624]: pgmap v17016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:20.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:20 smithi081 ceph-mon[31453]: pgmap v17016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:20 smithi153 ceph-mon[36569]: pgmap v17016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:22.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:22 smithi104 ceph-mon[36624]: pgmap v17017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:22.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:22 smithi081 ceph-mon[31453]: pgmap v17017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:22 smithi153 ceph-mon[36569]: pgmap v17017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:24.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:24 smithi104 ceph-mon[36624]: pgmap v17018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:24.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:24 smithi081 ceph-mon[31453]: pgmap v17018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:24 smithi153 ceph-mon[36569]: pgmap v17018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:26.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:26 smithi104 ceph-mon[36624]: pgmap v17019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:26.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:26 smithi081 ceph-mon[31453]: pgmap v17019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:26 smithi153 ceph-mon[36569]: pgmap v17019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:28 smithi104 ceph-mon[36624]: pgmap v17020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:28.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:28 smithi153 ceph-mon[36569]: pgmap v17020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:28 smithi081 ceph-mon[31453]: pgmap v17020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:30.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:30 smithi081 ceph-mon[31453]: pgmap v17021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:30 smithi153 ceph-mon[36569]: pgmap v17021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:30 smithi104 ceph-mon[36624]: pgmap v17021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:32.017+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:06:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:32.017+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:06:32.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:32.017+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:06:32.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:32 smithi081 ceph-mon[31453]: pgmap v17022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:32 smithi153 ceph-mon[36569]: pgmap v17022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:32 smithi104 ceph-mon[36624]: pgmap v17022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:34 smithi081 ceph-mon[31453]: pgmap v17023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:34 smithi153 ceph-mon[36569]: pgmap v17023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:34 smithi104 ceph-mon[36624]: pgmap v17023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:36.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:36 smithi081 ceph-mon[31453]: pgmap v17024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:36 smithi153 ceph-mon[36569]: pgmap v17024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:36 smithi104 ceph-mon[36624]: pgmap v17024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:38 smithi081 ceph-mon[31453]: pgmap v17025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:38 smithi153 ceph-mon[36569]: pgmap v17025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:38 smithi104 ceph-mon[36624]: pgmap v17025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:40.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:40 smithi104 ceph-mon[36624]: pgmap v17026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:40.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:40 smithi081 ceph-mon[31453]: pgmap v17026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:40 smithi153 ceph-mon[36569]: pgmap v17026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:42 smithi081 ceph-mon[31453]: pgmap v17027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:42 smithi153 ceph-mon[36569]: pgmap v17027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:42 smithi104 ceph-mon[36624]: pgmap v17027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:44.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:44 smithi081 ceph-mon[31453]: pgmap v17028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:44 smithi153 ceph-mon[36569]: pgmap v17028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:44 smithi104 ceph-mon[36624]: pgmap v17028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:46.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:46 smithi081 ceph-mon[31453]: pgmap v17029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:47.022+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:06:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:47.022+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:06:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:06:47.022+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:06:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:46 smithi153 ceph-mon[36569]: pgmap v17029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:47.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:46 smithi104 ceph-mon[36624]: pgmap v17029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:48 smithi081 ceph-mon[31453]: pgmap v17030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:48 smithi153 ceph-mon[36569]: pgmap v17030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:48 smithi104 ceph-mon[36624]: pgmap v17030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:50 smithi153 ceph-mon[36569]: pgmap v17031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:06:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:50.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:50.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:50.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:50 smithi081 ceph-mon[31453]: pgmap v17031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:06:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:50 smithi104 ceph-mon[36624]: pgmap v17031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:06:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:52 smithi081 ceph-mon[31453]: pgmap v17032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:52 smithi153 ceph-mon[36569]: pgmap v17032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:52 smithi104 ceph-mon[36624]: pgmap v17032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:54.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:54 smithi153 ceph-mon[36569]: pgmap v17033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:54 smithi104 ceph-mon[36624]: pgmap v17033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:55.114 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:54 smithi081 ceph-mon[31453]: pgmap v17033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:56.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:56 smithi081 ceph-mon[31453]: pgmap v17034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:06:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:06:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:56 smithi081 ceph-mon[31453]: pgmap v17035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:56 smithi081 ceph-mon[31453]: pgmap v17036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:56 smithi153 ceph-mon[36569]: pgmap v17034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:06:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:06:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:56 smithi153 ceph-mon[36569]: pgmap v17035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:56 smithi153 ceph-mon[36569]: pgmap v17036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:56 smithi104 ceph-mon[36624]: pgmap v17034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:06:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:06:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:56 smithi104 ceph-mon[36624]: pgmap v17035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:56 smithi104 ceph-mon[36624]: pgmap v17036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:06:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:06:58 smithi153 ceph-mon[36569]: pgmap v17037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:06:58 smithi104 ceph-mon[36624]: pgmap v17037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:06:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:06:58 smithi081 ceph-mon[31453]: pgmap v17037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:00 smithi153 ceph-mon[36569]: pgmap v17038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:00 smithi104 ceph-mon[36624]: pgmap v17038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:00 smithi081 ceph-mon[31453]: pgmap v17038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:02.038+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:02.038+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:07:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:02.038+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:07:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:02 smithi153 ceph-mon[36569]: pgmap v17039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:02 smithi104 ceph-mon[36624]: pgmap v17039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:02.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:02 smithi081 ceph-mon[31453]: pgmap v17039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:04 smithi153 ceph-mon[36569]: pgmap v17040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:04.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:04 smithi104 ceph-mon[36624]: pgmap v17040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:04.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:04 smithi081 ceph-mon[31453]: pgmap v17040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:06.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:06 smithi081 ceph-mon[31453]: pgmap v17041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:06 smithi153 ceph-mon[36569]: pgmap v17041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:06 smithi104 ceph-mon[36624]: pgmap v17041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:08 smithi153 ceph-mon[36569]: pgmap v17042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:08 smithi104 ceph-mon[36624]: pgmap v17042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:08 smithi081 ceph-mon[31453]: pgmap v17042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:10 smithi153 ceph-mon[36569]: pgmap v17043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:10.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:10 smithi104 ceph-mon[36624]: pgmap v17043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:10.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:10 smithi081 ceph-mon[31453]: pgmap v17043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:12 smithi153 ceph-mon[36569]: pgmap v17044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:12.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:12 smithi104 ceph-mon[36624]: pgmap v17044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:12.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:12 smithi081 ceph-mon[31453]: pgmap v17044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:14 smithi153 ceph-mon[36569]: pgmap v17045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:14.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:14 smithi104 ceph-mon[36624]: pgmap v17045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:14 smithi081 ceph-mon[31453]: pgmap v17045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:16 smithi153 ceph-mon[36569]: pgmap v17046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:16.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:16 smithi104 ceph-mon[36624]: pgmap v17046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:16.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:16 smithi081 ceph-mon[31453]: pgmap v17046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:17.055 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:17.053+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:07:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:17.053+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:07:17.056 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:17.053+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:07:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:17.969+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:07:17.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:17.969+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:07:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:17.970+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 246 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:07:17.973 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:17.970+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 246 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:07:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:18 smithi153 ceph-mon[36569]: pgmap v17047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:18.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:18 smithi104 ceph-mon[36624]: pgmap v17047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:18 smithi081 ceph-mon[31453]: pgmap v17047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:20.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:20 smithi153 ceph-mon[36569]: pgmap v17048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:20.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:20 smithi104 ceph-mon[36624]: pgmap v17048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:20.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:20 smithi081 ceph-mon[31453]: pgmap v17048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:22 smithi153 ceph-mon[36569]: pgmap v17049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:22.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:22 smithi104 ceph-mon[36624]: pgmap v17049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:22.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:22 smithi081 ceph-mon[31453]: pgmap v17049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:24 smithi153 ceph-mon[36569]: pgmap v17050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:24.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:24 smithi104 ceph-mon[36624]: pgmap v17050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:24.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:24 smithi081 ceph-mon[31453]: pgmap v17050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:26.462 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:26 smithi081 ceph-mon[31453]: pgmap v17051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:26 smithi153 ceph-mon[36569]: pgmap v17051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:26 smithi104 ceph-mon[36624]: pgmap v17051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:28 smithi153 ceph-mon[36569]: pgmap v17052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:28.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:28 smithi104 ceph-mon[36624]: pgmap v17052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:28.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:28 smithi081 ceph-mon[31453]: pgmap v17052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:30 smithi153 ceph-mon[36569]: pgmap v17053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:30.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:30 smithi104 ceph-mon[36624]: pgmap v17053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:30.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:30 smithi081 ceph-mon[31453]: pgmap v17053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:32.055+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:07:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:32.056+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:07:32.058 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:32.056+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:07:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:32 smithi153 ceph-mon[36569]: pgmap v17054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:32.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:32 smithi104 ceph-mon[36624]: pgmap v17054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:32.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:32 smithi081 ceph-mon[31453]: pgmap v17054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:34 smithi153 ceph-mon[36569]: pgmap v17055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:34.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:34 smithi104 ceph-mon[36624]: pgmap v17055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:34.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:34 smithi081 ceph-mon[31453]: pgmap v17055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:36.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:36 smithi081 ceph-mon[31453]: pgmap v17056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:36 smithi153 ceph-mon[36569]: pgmap v17056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:36.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:36 smithi104 ceph-mon[36624]: pgmap v17056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:38 smithi153 ceph-mon[36569]: pgmap v17057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:38.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:38 smithi104 ceph-mon[36624]: pgmap v17057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:38 smithi081 ceph-mon[31453]: pgmap v17057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:40 smithi153 ceph-mon[36569]: pgmap v17058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:40 smithi104 ceph-mon[36624]: pgmap v17058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:40.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:40 smithi081 ceph-mon[31453]: pgmap v17058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:42 smithi153 ceph-mon[36569]: pgmap v17059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:42 smithi104 ceph-mon[36624]: pgmap v17059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:42 smithi081 ceph-mon[31453]: pgmap v17059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:44 smithi153 ceph-mon[36569]: pgmap v17060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:44 smithi104 ceph-mon[36624]: pgmap v17060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:44.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:44 smithi081 ceph-mon[31453]: pgmap v17060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:47.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:47.071+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:07:47.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:47.071+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:07:47.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:07:47.071+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:07:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:47 smithi153 ceph-mon[36569]: pgmap v17061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:47 smithi104 ceph-mon[36624]: pgmap v17061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:47 smithi081 ceph-mon[31453]: pgmap v17061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:49 smithi153 ceph-mon[36569]: pgmap v17062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:49 smithi104 ceph-mon[36624]: pgmap v17062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:49 smithi081 ceph-mon[31453]: pgmap v17062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:51.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:51 smithi153 ceph-mon[36569]: pgmap v17063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:51.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:51 smithi104 ceph-mon[36624]: pgmap v17063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:51 smithi081 ceph-mon[31453]: pgmap v17063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:53 smithi153 ceph-mon[36569]: pgmap v17064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:53 smithi104 ceph-mon[36624]: pgmap v17064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:53.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:53 smithi081 ceph-mon[31453]: pgmap v17064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:55 smithi153 ceph-mon[36569]: pgmap v17065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:55.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:55 smithi104 ceph-mon[36624]: pgmap v17065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:55 smithi081 ceph-mon[31453]: pgmap v17065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:57 smithi153 ceph-mon[36569]: pgmap v17066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:07:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:07:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:07:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:57 smithi104 ceph-mon[36624]: pgmap v17066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:07:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:07:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:07:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:57 smithi081 ceph-mon[31453]: pgmap v17066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:07:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:07:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:07:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:07:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:07:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:58 smithi081 ceph-mon[31453]: pgmap v17067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:58 smithi081 ceph-mon[31453]: pgmap v17068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:07:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:07:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:07:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:07:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:58 smithi153 ceph-mon[36569]: pgmap v17067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:58 smithi153 ceph-mon[36569]: pgmap v17068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:58.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:07:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:07:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:07:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:07:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:58 smithi104 ceph-mon[36624]: pgmap v17067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:58 smithi104 ceph-mon[36624]: pgmap v17068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:07:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:07:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:08:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:00 smithi153 ceph-mon[36569]: pgmap v17069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:00 smithi104 ceph-mon[36624]: pgmap v17069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:00 smithi081 ceph-mon[31453]: pgmap v17069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:02.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:02.082+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:08:02.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:02.082+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:08:02.084 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:02.082+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:08:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:02 smithi153 ceph-mon[36569]: pgmap v17070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:02 smithi104 ceph-mon[36624]: pgmap v17070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:02 smithi081 ceph-mon[31453]: pgmap v17070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:04 smithi104 ceph-mon[36624]: pgmap v17071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:04 smithi081 ceph-mon[31453]: pgmap v17071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:04 smithi153 ceph-mon[36569]: pgmap v17071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:06 smithi104 ceph-mon[36624]: pgmap v17072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:06 smithi081 ceph-mon[31453]: pgmap v17072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:06 smithi153 ceph-mon[36569]: pgmap v17072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:09.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:08 smithi104 ceph-mon[36624]: pgmap v17073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:08 smithi081 ceph-mon[31453]: pgmap v17073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:08 smithi153 ceph-mon[36569]: pgmap v17073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:10 smithi104 ceph-mon[36624]: pgmap v17074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:10 smithi081 ceph-mon[31453]: pgmap v17074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:10 smithi153 ceph-mon[36569]: pgmap v17074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:12 smithi104 ceph-mon[36624]: pgmap v17075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:12 smithi081 ceph-mon[31453]: pgmap v17075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:12 smithi153 ceph-mon[36569]: pgmap v17075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:14 smithi081 ceph-mon[31453]: pgmap v17076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:14 smithi153 ceph-mon[36569]: pgmap v17076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:14 smithi104 ceph-mon[36624]: pgmap v17076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:17.097+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:08:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:17.097+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:08:17.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:17.097+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:08:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:16 smithi081 ceph-mon[31453]: pgmap v17077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:16 smithi153 ceph-mon[36569]: pgmap v17077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:16 smithi104 ceph-mon[36624]: pgmap v17077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:18 smithi081 ceph-mon[31453]: pgmap v17078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:19.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:18 smithi153 ceph-mon[36569]: pgmap v17078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:18 smithi104 ceph-mon[36624]: pgmap v17078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:20 smithi081 ceph-mon[31453]: pgmap v17079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:20 smithi153 ceph-mon[36569]: pgmap v17079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:20 smithi104 ceph-mon[36624]: pgmap v17079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:22 smithi081 ceph-mon[31453]: pgmap v17080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:22 smithi153 ceph-mon[36569]: pgmap v17080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:22 smithi104 ceph-mon[36624]: pgmap v17080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:24 smithi081 ceph-mon[31453]: pgmap v17081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:24 smithi153 ceph-mon[36569]: pgmap v17081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:24 smithi104 ceph-mon[36624]: pgmap v17081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:26 smithi081 ceph-mon[31453]: pgmap v17082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:26 smithi153 ceph-mon[36569]: pgmap v17082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:26 smithi104 ceph-mon[36624]: pgmap v17082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:28 smithi081 ceph-mon[31453]: pgmap v17083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:28 smithi153 ceph-mon[36569]: pgmap v17083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:28 smithi104 ceph-mon[36624]: pgmap v17083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:30 smithi153 ceph-mon[36569]: pgmap v17084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:30.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:30 smithi104 ceph-mon[36624]: pgmap v17084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:30.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:30 smithi081 ceph-mon[31453]: pgmap v17084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:32.103+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:08:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:32.103+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:08:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:32.103+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:08:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:32 smithi104 ceph-mon[36624]: pgmap v17085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:32 smithi081 ceph-mon[31453]: pgmap v17085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:32 smithi153 ceph-mon[36569]: pgmap v17085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:34 smithi104 ceph-mon[36624]: pgmap v17086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:34 smithi081 ceph-mon[31453]: pgmap v17086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:34 smithi153 ceph-mon[36569]: pgmap v17086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:37.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:36 smithi104 ceph-mon[36624]: pgmap v17087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:36 smithi081 ceph-mon[31453]: pgmap v17087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:36 smithi153 ceph-mon[36569]: pgmap v17087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:38 smithi104 ceph-mon[36624]: pgmap v17088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:38 smithi081 ceph-mon[31453]: pgmap v17088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:38 smithi153 ceph-mon[36569]: pgmap v17088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:40 smithi081 ceph-mon[31453]: pgmap v17089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:40 smithi153 ceph-mon[36569]: pgmap v17089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:40 smithi104 ceph-mon[36624]: pgmap v17089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:42 smithi081 ceph-mon[31453]: pgmap v17090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:42 smithi153 ceph-mon[36569]: pgmap v17090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:42 smithi104 ceph-mon[36624]: pgmap v17090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:45.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:44 smithi081 ceph-mon[31453]: pgmap v17091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:44 smithi153 ceph-mon[36569]: pgmap v17091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:44 smithi104 ceph-mon[36624]: pgmap v17091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:47.106+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:08:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:47.106+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:08:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:08:47.106+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:08:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:46 smithi081 ceph-mon[31453]: pgmap v17092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:46 smithi153 ceph-mon[36569]: pgmap v17092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:46 smithi104 ceph-mon[36624]: pgmap v17092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:48 smithi081 ceph-mon[31453]: pgmap v17093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:48 smithi153 ceph-mon[36569]: pgmap v17093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:48 smithi104 ceph-mon[36624]: pgmap v17093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:50 smithi081 ceph-mon[31453]: pgmap v17094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:50 smithi153 ceph-mon[36569]: pgmap v17094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:51.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:50 smithi104 ceph-mon[36624]: pgmap v17094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:52 smithi081 ceph-mon[31453]: pgmap v17095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:52 smithi153 ceph-mon[36569]: pgmap v17095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:53.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:52 smithi104 ceph-mon[36624]: pgmap v17095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:54 smithi081 ceph-mon[31453]: pgmap v17096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:54 smithi153 ceph-mon[36569]: pgmap v17096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:54 smithi104 ceph-mon[36624]: pgmap v17096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:56 smithi081 ceph-mon[31453]: pgmap v17097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:56 smithi153 ceph-mon[36569]: pgmap v17097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:56 smithi104 ceph-mon[36624]: pgmap v17097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:58 smithi081 ceph-mon[31453]: pgmap v17098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:08:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:08:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:58 smithi153 ceph-mon[36569]: pgmap v17098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:08:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:08:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:58 smithi104 ceph-mon[36624]: pgmap v17098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:08:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:08:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:09:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:01 smithi104 ceph-mon[36624]: pgmap v17099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:09:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:09:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:01 smithi104 ceph-mon[36624]: pgmap v17100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:01 smithi104 ceph-mon[36624]: pgmap v17101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:01 smithi081 ceph-mon[31453]: pgmap v17099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:09:01.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:09:01.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:01 smithi081 ceph-mon[31453]: pgmap v17100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:01 smithi081 ceph-mon[31453]: pgmap v17101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:01 smithi153 ceph-mon[36569]: pgmap v17099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:09:01.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:09:01.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:01 smithi153 ceph-mon[36569]: pgmap v17100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:01 smithi153 ceph-mon[36569]: pgmap v17101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:01.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:09:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:02.117+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:09:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:02.117+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:09:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:02.117+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:09:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:03 smithi081 ceph-mon[31453]: pgmap v17102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:03 smithi153 ceph-mon[36569]: pgmap v17102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:03 smithi104 ceph-mon[36624]: pgmap v17102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:05 smithi081 ceph-mon[31453]: pgmap v17103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:05 smithi153 ceph-mon[36569]: pgmap v17103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:05.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:05 smithi104 ceph-mon[36624]: pgmap v17103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:07 smithi081 ceph-mon[31453]: pgmap v17104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:07 smithi153 ceph-mon[36569]: pgmap v17104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:07 smithi104 ceph-mon[36624]: pgmap v17104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:09 smithi153 ceph-mon[36569]: pgmap v17105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:09.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:09 smithi104 ceph-mon[36624]: pgmap v17105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:09.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:09 smithi081 ceph-mon[31453]: pgmap v17105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:11.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:11 smithi153 ceph-mon[36569]: pgmap v17106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:11.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:11 smithi104 ceph-mon[36624]: pgmap v17106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:11.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:11 smithi081 ceph-mon[31453]: pgmap v17106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:13.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:13 smithi153 ceph-mon[36569]: pgmap v17107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:13.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:13 smithi104 ceph-mon[36624]: pgmap v17107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:13.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:13 smithi081 ceph-mon[31453]: pgmap v17107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:14 smithi153 ceph-mon[36569]: pgmap v17108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:14 smithi104 ceph-mon[36624]: pgmap v17108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:14 smithi081 ceph-mon[31453]: pgmap v17108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:17.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:17.120+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:09:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:17.120+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:09:17.123 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:17.120+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:09:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:17 smithi081 ceph-mon[31453]: pgmap v17109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:17 smithi153 ceph-mon[36569]: pgmap v17109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:17 smithi104 ceph-mon[36624]: pgmap v17109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:19.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:19 smithi081 ceph-mon[31453]: pgmap v17110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:19 smithi153 ceph-mon[36569]: pgmap v17110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:19.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:19 smithi104 ceph-mon[36624]: pgmap v17110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:21.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:21 smithi153 ceph-mon[36569]: pgmap v17111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:21 smithi104 ceph-mon[36624]: pgmap v17111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:21.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:21 smithi081 ceph-mon[31453]: pgmap v17111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:23 smithi153 ceph-mon[36569]: pgmap v17112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:23 smithi104 ceph-mon[36624]: pgmap v17112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:23.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:23 smithi081 ceph-mon[31453]: pgmap v17112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:25 smithi153 ceph-mon[36569]: pgmap v17113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:25.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:25 smithi104 ceph-mon[36624]: pgmap v17113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:25.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:25 smithi081 ceph-mon[31453]: pgmap v17113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:26 smithi153 ceph-mon[36569]: pgmap v17114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:26.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:26 smithi104 ceph-mon[36624]: pgmap v17114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:26.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:26 smithi081 ceph-mon[31453]: pgmap v17114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:29 smithi153 ceph-mon[36569]: pgmap v17115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:29.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:29 smithi104 ceph-mon[36624]: pgmap v17115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:29.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:29 smithi081 ceph-mon[31453]: pgmap v17115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:31.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:31 smithi081 ceph-mon[31453]: pgmap v17116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:31 smithi153 ceph-mon[36569]: pgmap v17116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:31.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:31 smithi104 ceph-mon[36624]: pgmap v17116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:32.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:32.135+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:09:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:32.135+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:09:32.138 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:32.135+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:09:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:33 smithi153 ceph-mon[36569]: pgmap v17117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:33 smithi104 ceph-mon[36624]: pgmap v17117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:33 smithi081 ceph-mon[31453]: pgmap v17117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:35 smithi153 ceph-mon[36569]: pgmap v17118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:35.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:35 smithi104 ceph-mon[36624]: pgmap v17118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:35 smithi081 ceph-mon[31453]: pgmap v17118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:36 smithi153 ceph-mon[36569]: pgmap v17119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:36.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:36 smithi104 ceph-mon[36624]: pgmap v17119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:36 smithi081 ceph-mon[31453]: pgmap v17119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:39.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:39 smithi153 ceph-mon[36569]: pgmap v17120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:39 smithi104 ceph-mon[36624]: pgmap v17120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:39 smithi081 ceph-mon[31453]: pgmap v17120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:41 smithi153 ceph-mon[36569]: pgmap v17121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:41.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:41 smithi104 ceph-mon[36624]: pgmap v17121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:41.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:41 smithi081 ceph-mon[31453]: pgmap v17121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:43 smithi153 ceph-mon[36569]: pgmap v17122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:43 smithi104 ceph-mon[36624]: pgmap v17122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:43 smithi081 ceph-mon[31453]: pgmap v17122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:44 smithi153 ceph-mon[36569]: pgmap v17123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:44 smithi104 ceph-mon[36624]: pgmap v17123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:44.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:44 smithi081 ceph-mon[31453]: pgmap v17123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:47.149+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:09:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:47.149+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:09:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:09:47.149+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:09:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:47 smithi081 ceph-mon[31453]: pgmap v17124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:47 smithi153 ceph-mon[36569]: pgmap v17124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:47 smithi104 ceph-mon[36624]: pgmap v17124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:49 smithi153 ceph-mon[36569]: pgmap v17125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:49 smithi104 ceph-mon[36624]: pgmap v17125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:49 smithi081 ceph-mon[31453]: pgmap v17125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:51.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:51 smithi153 ceph-mon[36569]: pgmap v17126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:51.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:51 smithi104 ceph-mon[36624]: pgmap v17126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:51 smithi081 ceph-mon[31453]: pgmap v17126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:52 smithi153 ceph-mon[36569]: pgmap v17127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:52 smithi104 ceph-mon[36624]: pgmap v17127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:52 smithi081 ceph-mon[31453]: pgmap v17127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:55 smithi153 ceph-mon[36569]: pgmap v17128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:55.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:55 smithi104 ceph-mon[36624]: pgmap v17128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:55 smithi081 ceph-mon[31453]: pgmap v17128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:57 smithi153 ceph-mon[36569]: pgmap v17129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:57 smithi104 ceph-mon[36624]: pgmap v17129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:57 smithi081 ceph-mon[31453]: pgmap v17129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:09:58 smithi153 ceph-mon[36569]: pgmap v17130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:09:58 smithi104 ceph-mon[36624]: pgmap v17130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:09:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:09:58 smithi081 ceph-mon[31453]: pgmap v17130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:10:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:10:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:10:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:01 smithi104 ceph-mon[36624]: pgmap v17131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:10:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:01 smithi081 ceph-mon[31453]: pgmap v17131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:10:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:01 smithi153 ceph-mon[36569]: pgmap v17131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:10:02.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:02.157+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:10:02.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:02.158+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:10:02.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:02.158+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:10:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:10:02.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:10:02.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: pgmap v17132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:02.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: pgmap v17133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:02.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:02.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:10:02.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:10:02.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: pgmap v17132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:02.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: pgmap v17133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:02.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:10:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:10:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: pgmap v17132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: pgmap v17133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:10:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:04 smithi153 ceph-mon[36569]: pgmap v17134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:04 smithi104 ceph-mon[36624]: pgmap v17134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:04 smithi081 ceph-mon[31453]: pgmap v17134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:06 smithi153 ceph-mon[36569]: pgmap v17135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:07.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:06 smithi104 ceph-mon[36624]: pgmap v17135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:06 smithi081 ceph-mon[31453]: pgmap v17135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:08 smithi153 ceph-mon[36569]: pgmap v17136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:08 smithi104 ceph-mon[36624]: pgmap v17136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:08 smithi081 ceph-mon[31453]: pgmap v17136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:10 smithi153 ceph-mon[36569]: pgmap v17137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:10 smithi104 ceph-mon[36624]: pgmap v17137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:10 smithi081 ceph-mon[31453]: pgmap v17137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:12 smithi153 ceph-mon[36569]: pgmap v17138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:13.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:12 smithi104 ceph-mon[36624]: pgmap v17138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:12 smithi081 ceph-mon[31453]: pgmap v17138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:14 smithi153 ceph-mon[36569]: pgmap v17139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:15.156 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:14 smithi104 ceph-mon[36624]: pgmap v17139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:14 smithi081 ceph-mon[31453]: pgmap v17139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:16 smithi153 ceph-mon[36569]: pgmap v17140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:16 smithi104 ceph-mon[36624]: pgmap v17140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:16 smithi081 ceph-mon[31453]: pgmap v17140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:17.171+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:10:17.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:17.171+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:10:17.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:17.171+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:10:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:18.000+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:10:18.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:18.000+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:10:18.003 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:18.001+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 247 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:10:18.003 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:18.001+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 247 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:10:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:18 smithi153 ceph-mon[36569]: pgmap v17141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:18 smithi104 ceph-mon[36624]: pgmap v17141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:18 smithi081 ceph-mon[31453]: pgmap v17141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:20 smithi153 ceph-mon[36569]: pgmap v17142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:20 smithi104 ceph-mon[36624]: pgmap v17142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:20 smithi081 ceph-mon[31453]: pgmap v17142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:22 smithi153 ceph-mon[36569]: pgmap v17143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:23.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:22 smithi104 ceph-mon[36624]: pgmap v17143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:22 smithi081 ceph-mon[31453]: pgmap v17143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:24 smithi153 ceph-mon[36569]: pgmap v17144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:25.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:24 smithi104 ceph-mon[36624]: pgmap v17144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:24 smithi081 ceph-mon[31453]: pgmap v17144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:26 smithi153 ceph-mon[36569]: pgmap v17145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:27.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:26 smithi104 ceph-mon[36624]: pgmap v17145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:26 smithi081 ceph-mon[31453]: pgmap v17145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:28 smithi153 ceph-mon[36569]: pgmap v17146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:28 smithi104 ceph-mon[36624]: pgmap v17146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:28 smithi081 ceph-mon[31453]: pgmap v17146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:30 smithi153 ceph-mon[36569]: pgmap v17147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:30 smithi104 ceph-mon[36624]: pgmap v17147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:30 smithi081 ceph-mon[31453]: pgmap v17147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:32.187+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:10:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:32.187+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:10:32.189 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:32.187+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:10:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:32 smithi153 ceph-mon[36569]: pgmap v17148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:32 smithi104 ceph-mon[36624]: pgmap v17148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:32 smithi081 ceph-mon[31453]: pgmap v17148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:34 smithi153 ceph-mon[36569]: pgmap v17149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:34 smithi104 ceph-mon[36624]: pgmap v17149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:34 smithi081 ceph-mon[31453]: pgmap v17149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:36 smithi153 ceph-mon[36569]: pgmap v17150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:36 smithi104 ceph-mon[36624]: pgmap v17150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:36 smithi081 ceph-mon[31453]: pgmap v17150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:38 smithi153 ceph-mon[36569]: pgmap v17151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:39.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:38 smithi104 ceph-mon[36624]: pgmap v17151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:38 smithi081 ceph-mon[31453]: pgmap v17151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:40 smithi153 ceph-mon[36569]: pgmap v17152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:40 smithi104 ceph-mon[36624]: pgmap v17152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:40 smithi081 ceph-mon[31453]: pgmap v17152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:42 smithi153 ceph-mon[36569]: pgmap v17153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:43.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:42 smithi104 ceph-mon[36624]: pgmap v17153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:42 smithi081 ceph-mon[31453]: pgmap v17153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:44 smithi153 ceph-mon[36569]: pgmap v17154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:44 smithi104 ceph-mon[36624]: pgmap v17154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:45.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:44 smithi081 ceph-mon[31453]: pgmap v17154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:47.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:46 smithi104 ceph-mon[36624]: pgmap v17155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:46 smithi081 ceph-mon[31453]: pgmap v17155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:47.202+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:10:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:47.202+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:10:47.205 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:10:47.202+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:10:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:46 smithi153 ceph-mon[36569]: pgmap v17155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:48 smithi104 ceph-mon[36624]: pgmap v17156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:49.097 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:48 smithi081 ceph-mon[31453]: pgmap v17156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:48 smithi153 ceph-mon[36569]: pgmap v17156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:50 smithi104 ceph-mon[36624]: pgmap v17157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:50 smithi081 ceph-mon[31453]: pgmap v17157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:50 smithi153 ceph-mon[36569]: pgmap v17157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:53.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:52 smithi104 ceph-mon[36624]: pgmap v17158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:52 smithi081 ceph-mon[31453]: pgmap v17158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:52 smithi153 ceph-mon[36569]: pgmap v17158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:55.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:54 smithi104 ceph-mon[36624]: pgmap v17159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:54 smithi081 ceph-mon[31453]: pgmap v17159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:54 smithi153 ceph-mon[36569]: pgmap v17159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:56 smithi081 ceph-mon[31453]: pgmap v17160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:56 smithi153 ceph-mon[36569]: pgmap v17160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:56 smithi104 ceph-mon[36624]: pgmap v17160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:10:58 smithi081 ceph-mon[31453]: pgmap v17161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:10:58 smithi153 ceph-mon[36569]: pgmap v17161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:10:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:10:58 smithi104 ceph-mon[36624]: pgmap v17161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:01.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:00 smithi153 ceph-mon[36569]: pgmap v17162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:00 smithi081 ceph-mon[31453]: pgmap v17162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:00 smithi104 ceph-mon[36624]: pgmap v17162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:02.218+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:11:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:02.218+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:11:02.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:02.218+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:11:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:02 smithi081 ceph-mon[31453]: pgmap v17163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:11:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:02 smithi153 ceph-mon[36569]: pgmap v17163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:11:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:02 smithi104 ceph-mon[36624]: pgmap v17163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:11:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:11:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:11:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:11:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:11:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:11:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:11:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:11:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:11:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:11:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:04 smithi081 ceph-mon[31453]: pgmap v17164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:04 smithi081 ceph-mon[31453]: pgmap v17165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:04 smithi153 ceph-mon[36569]: pgmap v17164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:04 smithi153 ceph-mon[36569]: pgmap v17165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:04 smithi104 ceph-mon[36624]: pgmap v17164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:04 smithi104 ceph-mon[36624]: pgmap v17165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:06 smithi081 ceph-mon[31453]: pgmap v17166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:06 smithi153 ceph-mon[36569]: pgmap v17166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:06 smithi104 ceph-mon[36624]: pgmap v17166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:08 smithi081 ceph-mon[31453]: pgmap v17167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:08 smithi153 ceph-mon[36569]: pgmap v17167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:08 smithi104 ceph-mon[36624]: pgmap v17167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:10 smithi081 ceph-mon[31453]: pgmap v17168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:10 smithi153 ceph-mon[36569]: pgmap v17168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:10 smithi104 ceph-mon[36624]: pgmap v17168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:12 smithi081 ceph-mon[31453]: pgmap v17169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:12 smithi153 ceph-mon[36569]: pgmap v17169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:12 smithi104 ceph-mon[36624]: pgmap v17169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:14 smithi081 ceph-mon[31453]: pgmap v17170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:14 smithi153 ceph-mon[36569]: pgmap v17170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:14 smithi104 ceph-mon[36624]: pgmap v17170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:16 smithi081 ceph-mon[31453]: pgmap v17171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:17.233+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:11:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:17.233+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:11:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:17.233+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:11:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:16 smithi153 ceph-mon[36569]: pgmap v17171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:16 smithi104 ceph-mon[36624]: pgmap v17171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:18 smithi081 ceph-mon[31453]: pgmap v17172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:18 smithi153 ceph-mon[36569]: pgmap v17172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:18 smithi104 ceph-mon[36624]: pgmap v17172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:20 smithi153 ceph-mon[36569]: pgmap v17173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:20 smithi104 ceph-mon[36624]: pgmap v17173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:20 smithi081 ceph-mon[31453]: pgmap v17173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:22 smithi153 ceph-mon[36569]: pgmap v17174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:22 smithi104 ceph-mon[36624]: pgmap v17174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:22 smithi081 ceph-mon[31453]: pgmap v17174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:24 smithi153 ceph-mon[36569]: pgmap v17175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:24 smithi104 ceph-mon[36624]: pgmap v17175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:24 smithi081 ceph-mon[31453]: pgmap v17175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:26 smithi153 ceph-mon[36569]: pgmap v17176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:26 smithi104 ceph-mon[36624]: pgmap v17176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:26 smithi081 ceph-mon[31453]: pgmap v17176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:28 smithi153 ceph-mon[36569]: pgmap v17177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:28 smithi104 ceph-mon[36624]: pgmap v17177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:28 smithi081 ceph-mon[31453]: pgmap v17177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:30 smithi153 ceph-mon[36569]: pgmap v17178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:30 smithi104 ceph-mon[36624]: pgmap v17178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:30 smithi081 ceph-mon[31453]: pgmap v17178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:32.239+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:11:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:32.239+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:11:32.241 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:32.239+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:11:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:32 smithi153 ceph-mon[36569]: pgmap v17179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:32 smithi104 ceph-mon[36624]: pgmap v17179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:32 smithi081 ceph-mon[31453]: pgmap v17179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:34 smithi153 ceph-mon[36569]: pgmap v17180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:34 smithi104 ceph-mon[36624]: pgmap v17180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:34 smithi081 ceph-mon[31453]: pgmap v17180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:36 smithi153 ceph-mon[36569]: pgmap v17181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:36 smithi104 ceph-mon[36624]: pgmap v17181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:37.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:36 smithi081 ceph-mon[31453]: pgmap v17181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:38 smithi153 ceph-mon[36569]: pgmap v17182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:38 smithi104 ceph-mon[36624]: pgmap v17182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:39.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:38 smithi081 ceph-mon[31453]: pgmap v17182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:41.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:40 smithi081 ceph-mon[31453]: pgmap v17183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:40 smithi153 ceph-mon[36569]: pgmap v17183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:40 smithi104 ceph-mon[36624]: pgmap v17183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:42 smithi153 ceph-mon[36569]: pgmap v17184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:42 smithi104 ceph-mon[36624]: pgmap v17184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:43.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:42 smithi081 ceph-mon[31453]: pgmap v17184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:44 smithi153 ceph-mon[36569]: pgmap v17185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:44 smithi104 ceph-mon[36624]: pgmap v17185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:45.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:44 smithi081 ceph-mon[31453]: pgmap v17185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:47.254+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:11:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:47.255+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:11:47.257 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:11:47.255+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:11:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:46 smithi153 ceph-mon[36569]: pgmap v17186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:46 smithi104 ceph-mon[36624]: pgmap v17186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:46 smithi081 ceph-mon[31453]: pgmap v17186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:48 smithi153 ceph-mon[36569]: pgmap v17187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:48 smithi104 ceph-mon[36624]: pgmap v17187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:48 smithi081 ceph-mon[31453]: pgmap v17187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:50 smithi153 ceph-mon[36569]: pgmap v17188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:50 smithi104 ceph-mon[36624]: pgmap v17188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:50 smithi081 ceph-mon[31453]: pgmap v17188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:52 smithi153 ceph-mon[36569]: pgmap v17189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:52 smithi104 ceph-mon[36624]: pgmap v17189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:52 smithi081 ceph-mon[31453]: pgmap v17189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:54 smithi153 ceph-mon[36569]: pgmap v17190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:54 smithi104 ceph-mon[36624]: pgmap v17190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:54 smithi081 ceph-mon[31453]: pgmap v17190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:57 smithi153 ceph-mon[36569]: pgmap v17191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:57 smithi104 ceph-mon[36624]: pgmap v17191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:57.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:57 smithi081 ceph-mon[31453]: pgmap v17191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:11:59 smithi153 ceph-mon[36569]: pgmap v17192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:11:59 smithi104 ceph-mon[36624]: pgmap v17192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:11:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:11:59 smithi081 ceph-mon[31453]: pgmap v17192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:01 smithi153 ceph-mon[36569]: pgmap v17193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:01 smithi104 ceph-mon[36624]: pgmap v17193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:01 smithi081 ceph-mon[31453]: pgmap v17193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:02.266 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:02.264+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:12:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:02.264+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:12:02.267 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:02.265+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:12:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:03 smithi153 ceph-mon[36569]: pgmap v17194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:03 smithi104 ceph-mon[36624]: pgmap v17194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:03 smithi081 ceph-mon[31453]: pgmap v17194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:12:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:12:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:12:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:05 smithi153 ceph-mon[36569]: pgmap v17195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:12:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:12:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:12:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:05 smithi104 ceph-mon[36624]: pgmap v17195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:12:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:12:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:12:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:05 smithi081 ceph-mon[31453]: pgmap v17195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:12:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:12:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:12:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:06 smithi153 ceph-mon[36569]: pgmap v17196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:06 smithi153 ceph-mon[36569]: pgmap v17197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:06 smithi104 ceph-mon[36624]: pgmap v17196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:06 smithi104 ceph-mon[36624]: pgmap v17197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:06 smithi081 ceph-mon[31453]: pgmap v17196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:06 smithi081 ceph-mon[31453]: pgmap v17197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:08 smithi153 ceph-mon[36569]: pgmap v17198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:08 smithi104 ceph-mon[36624]: pgmap v17198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:08 smithi081 ceph-mon[31453]: pgmap v17198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:10 smithi153 ceph-mon[36569]: pgmap v17199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:10 smithi104 ceph-mon[36624]: pgmap v17199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:10 smithi081 ceph-mon[31453]: pgmap v17199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:12 smithi104 ceph-mon[36624]: pgmap v17200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:12 smithi081 ceph-mon[31453]: pgmap v17200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:12 smithi153 ceph-mon[36569]: pgmap v17200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:14 smithi104 ceph-mon[36624]: pgmap v17201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:14 smithi081 ceph-mon[31453]: pgmap v17201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:14 smithi153 ceph-mon[36569]: pgmap v17201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:16 smithi104 ceph-mon[36624]: pgmap v17202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:16 smithi081 ceph-mon[31453]: pgmap v17202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:16 smithi153 ceph-mon[36569]: pgmap v17202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:17.280+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:12:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:17.280+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:12:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:17.280+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:12:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:18 smithi104 ceph-mon[36624]: pgmap v17203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:18 smithi081 ceph-mon[31453]: pgmap v17203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:18 smithi153 ceph-mon[36569]: pgmap v17203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:20.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:20 smithi081 ceph-mon[31453]: pgmap v17204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:20.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:20 smithi153 ceph-mon[36569]: pgmap v17204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:20.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:20 smithi104 ceph-mon[36624]: pgmap v17204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:22 smithi081 ceph-mon[31453]: pgmap v17205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:22 smithi153 ceph-mon[36569]: pgmap v17205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:22.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:22 smithi104 ceph-mon[36624]: pgmap v17205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:24 smithi081 ceph-mon[31453]: pgmap v17206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:24 smithi153 ceph-mon[36569]: pgmap v17206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:24 smithi104 ceph-mon[36624]: pgmap v17206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:26 smithi081 ceph-mon[31453]: pgmap v17207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:26 smithi153 ceph-mon[36569]: pgmap v17207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:26.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:26 smithi104 ceph-mon[36624]: pgmap v17207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:28.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:28 smithi081 ceph-mon[31453]: pgmap v17208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:28 smithi153 ceph-mon[36569]: pgmap v17208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:28 smithi104 ceph-mon[36624]: pgmap v17208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:30.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:30 smithi081 ceph-mon[31453]: pgmap v17209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:30 smithi153 ceph-mon[36569]: pgmap v17209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:30.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:30 smithi104 ceph-mon[36624]: pgmap v17209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:32.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:32.283+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:12:32.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:32.283+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:12:32.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:32.283+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:12:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:32 smithi081 ceph-mon[31453]: pgmap v17210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:32 smithi153 ceph-mon[36569]: pgmap v17210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:32 smithi104 ceph-mon[36624]: pgmap v17210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:33.021+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:12:33.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:33.021+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:12:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:33.022+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 248 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:12:33.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:33.022+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 248 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:12:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:34 smithi081 ceph-mon[31453]: pgmap v17211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:34 smithi153 ceph-mon[36569]: pgmap v17211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:34 smithi104 ceph-mon[36624]: pgmap v17211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:36.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:36 smithi081 ceph-mon[31453]: pgmap v17212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:36 smithi153 ceph-mon[36569]: pgmap v17212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:36 smithi104 ceph-mon[36624]: pgmap v17212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:38 smithi081 ceph-mon[31453]: pgmap v17213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:38 smithi153 ceph-mon[36569]: pgmap v17213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:38 smithi104 ceph-mon[36624]: pgmap v17213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:40 smithi081 ceph-mon[31453]: pgmap v17214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:40 smithi153 ceph-mon[36569]: pgmap v17214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:40 smithi104 ceph-mon[36624]: pgmap v17214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:42.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:42 smithi081 ceph-mon[31453]: pgmap v17215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:42 smithi153 ceph-mon[36569]: pgmap v17215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:42 smithi104 ceph-mon[36624]: pgmap v17215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:44 smithi081 ceph-mon[31453]: pgmap v17216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:44 smithi153 ceph-mon[36569]: pgmap v17216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:44 smithi104 ceph-mon[36624]: pgmap v17216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:46 smithi153 ceph-mon[36569]: pgmap v17217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:46 smithi104 ceph-mon[36624]: pgmap v17217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:46 smithi081 ceph-mon[31453]: pgmap v17217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:47.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:47.299+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:12:47.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:47.299+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:12:47.301 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:12:47.299+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:12:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:48 smithi153 ceph-mon[36569]: pgmap v17218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:48 smithi104 ceph-mon[36624]: pgmap v17218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:48 smithi081 ceph-mon[31453]: pgmap v17218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:50 smithi153 ceph-mon[36569]: pgmap v17219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:50.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:50 smithi104 ceph-mon[36624]: pgmap v17219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:50 smithi081 ceph-mon[31453]: pgmap v17219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:52 smithi153 ceph-mon[36569]: pgmap v17220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:52 smithi104 ceph-mon[36624]: pgmap v17220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:52 smithi081 ceph-mon[31453]: pgmap v17220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:54.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:54 smithi081 ceph-mon[31453]: pgmap v17221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:54.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:54 smithi153 ceph-mon[36569]: pgmap v17221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:54 smithi104 ceph-mon[36624]: pgmap v17221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:56 smithi153 ceph-mon[36569]: pgmap v17222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:56 smithi104 ceph-mon[36624]: pgmap v17222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:56 smithi081 ceph-mon[31453]: pgmap v17222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:12:58 smithi153 ceph-mon[36569]: pgmap v17223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:12:58 smithi104 ceph-mon[36624]: pgmap v17223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:12:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:12:58 smithi081 ceph-mon[31453]: pgmap v17223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:00 smithi153 ceph-mon[36569]: pgmap v17224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:00.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:00 smithi104 ceph-mon[36624]: pgmap v17224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:00.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:00 smithi081 ceph-mon[31453]: pgmap v17224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:02.314+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:02.314+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:13:02.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:02.314+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:13:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:02 smithi153 ceph-mon[36569]: pgmap v17225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:02.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:02 smithi104 ceph-mon[36624]: pgmap v17225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:02.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:02 smithi081 ceph-mon[31453]: pgmap v17225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:04 smithi153 ceph-mon[36569]: pgmap v17226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:04 smithi104 ceph-mon[36624]: pgmap v17226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:04 smithi081 ceph-mon[31453]: pgmap v17226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:13:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:13:05.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:13:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: pgmap v17227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: pgmap v17227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: pgmap v17227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:06.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:13:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:13:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:13:07.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:13:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:13:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:13:07.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:13:07.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:13:07.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:13:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:13:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:13:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:13:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:13:07.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:13:07.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:13:07.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:13:07.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:13:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:08 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:13:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:08 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:13:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:08 smithi153 ceph-mon[36569]: pgmap v17228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:08 smithi153 ceph-mon[36569]: pgmap v17229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:08 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:13:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:08 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:13:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:08 smithi104 ceph-mon[36624]: pgmap v17228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:08 smithi104 ceph-mon[36624]: pgmap v17229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:08 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:13:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:08 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:13:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:08 smithi081 ceph-mon[31453]: pgmap v17228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:08.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:08 smithi081 ceph-mon[31453]: pgmap v17229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:09 smithi153 ceph-mon[36569]: pgmap v17230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:09.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:09 smithi104 ceph-mon[36624]: pgmap v17230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:09.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:09 smithi081 ceph-mon[31453]: pgmap v17230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:11.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:11 smithi081 ceph-mon[31453]: pgmap v17231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:11 smithi153 ceph-mon[36569]: pgmap v17231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:11.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:11 smithi104 ceph-mon[36624]: pgmap v17231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:13.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:13 smithi081 ceph-mon[31453]: pgmap v17232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:13 smithi153 ceph-mon[36569]: pgmap v17232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:13 smithi104 ceph-mon[36624]: pgmap v17232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:15.636 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:15 smithi104 ceph-mon[36624]: pgmap v17233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:15.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:15 smithi081 ceph-mon[31453]: pgmap v17233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:15 smithi153 ceph-mon[36569]: pgmap v17233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:17.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:17.321+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:13:17.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:17.321+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:13:17.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:17.321+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:13:17.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:17 smithi081 ceph-mon[31453]: pgmap v17234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:17 smithi153 ceph-mon[36569]: pgmap v17234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:17.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:17 smithi104 ceph-mon[36624]: pgmap v17234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:18.030+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:13:18.031 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:18.030+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:13:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:18.030+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 249 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:13:18.032 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:18.030+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 249 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:13:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:19 smithi081 ceph-mon[31453]: pgmap v17235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:19 smithi153 ceph-mon[36569]: pgmap v17235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:19 smithi104 ceph-mon[36624]: pgmap v17235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:21.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:21 smithi081 ceph-mon[31453]: pgmap v17236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:21 smithi153 ceph-mon[36569]: pgmap v17236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:21 smithi104 ceph-mon[36624]: pgmap v17236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:23 smithi081 ceph-mon[31453]: pgmap v17237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:23 smithi153 ceph-mon[36569]: pgmap v17237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:23.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:23 smithi104 ceph-mon[36624]: pgmap v17237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:25.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:25 smithi081 ceph-mon[31453]: pgmap v17238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:25 smithi153 ceph-mon[36569]: pgmap v17238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:25 smithi104 ceph-mon[36624]: pgmap v17238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:27.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:27 smithi081 ceph-mon[31453]: pgmap v17239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:27 smithi153 ceph-mon[36569]: pgmap v17239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:27 smithi104 ceph-mon[36624]: pgmap v17239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:29 smithi153 ceph-mon[36569]: pgmap v17240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:29 smithi104 ceph-mon[36624]: pgmap v17240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:29 smithi081 ceph-mon[31453]: pgmap v17240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:31 smithi153 ceph-mon[36569]: pgmap v17241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:31.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:31 smithi104 ceph-mon[36624]: pgmap v17241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:31.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:31 smithi081 ceph-mon[31453]: pgmap v17241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:32.337+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:13:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:32.337+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:13:32.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:32.337+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:13:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:33 smithi153 ceph-mon[36569]: pgmap v17242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:33 smithi104 ceph-mon[36624]: pgmap v17242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:33 smithi081 ceph-mon[31453]: pgmap v17242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:35 smithi153 ceph-mon[36569]: pgmap v17243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:35 smithi104 ceph-mon[36624]: pgmap v17243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:35.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:35 smithi081 ceph-mon[31453]: pgmap v17243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:37 smithi153 ceph-mon[36569]: pgmap v17244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:37 smithi104 ceph-mon[36624]: pgmap v17244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:37.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:37 smithi081 ceph-mon[31453]: pgmap v17244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:39 smithi153 ceph-mon[36569]: pgmap v17245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:39 smithi104 ceph-mon[36624]: pgmap v17245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:39.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:39 smithi081 ceph-mon[31453]: pgmap v17245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:41.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:41 smithi153 ceph-mon[36569]: pgmap v17246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:41 smithi104 ceph-mon[36624]: pgmap v17246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:41 smithi081 ceph-mon[31453]: pgmap v17246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:43.711 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:43 smithi081 ceph-mon[31453]: pgmap v17247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:43 smithi153 ceph-mon[36569]: pgmap v17247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:43 smithi104 ceph-mon[36624]: pgmap v17247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:45 smithi153 ceph-mon[36569]: pgmap v17248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:45 smithi104 ceph-mon[36624]: pgmap v17248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:45 smithi081 ceph-mon[31453]: pgmap v17248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:47.352+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:13:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:47.352+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:13:47.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:13:47.352+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:13:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:47 smithi153 ceph-mon[36569]: pgmap v17249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:47.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:47 smithi104 ceph-mon[36624]: pgmap v17249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:47.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:47 smithi081 ceph-mon[31453]: pgmap v17249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:49 smithi153 ceph-mon[36569]: pgmap v17250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:49.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:49 smithi104 ceph-mon[36624]: pgmap v17250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:49 smithi081 ceph-mon[31453]: pgmap v17250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:51 smithi153 ceph-mon[36569]: pgmap v17251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:51 smithi104 ceph-mon[36624]: pgmap v17251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:51.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:51 smithi081 ceph-mon[31453]: pgmap v17251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:53 smithi153 ceph-mon[36569]: pgmap v17252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:53 smithi104 ceph-mon[36624]: pgmap v17252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:53 smithi081 ceph-mon[31453]: pgmap v17252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:55 smithi153 ceph-mon[36569]: pgmap v17253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:55 smithi104 ceph-mon[36624]: pgmap v17253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:55 smithi081 ceph-mon[31453]: pgmap v17253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:57 smithi153 ceph-mon[36569]: pgmap v17254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:57 smithi104 ceph-mon[36624]: pgmap v17254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:57 smithi081 ceph-mon[31453]: pgmap v17254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:13:59 smithi153 ceph-mon[36569]: pgmap v17255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:13:59 smithi104 ceph-mon[36624]: pgmap v17255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:13:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:13:59 smithi081 ceph-mon[31453]: pgmap v17255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:01 smithi153 ceph-mon[36569]: pgmap v17256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:01.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:01 smithi104 ceph-mon[36624]: pgmap v17256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:01 smithi081 ceph-mon[31453]: pgmap v17256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:02.368+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:14:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:02.368+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:14:02.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:02.368+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:14:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:03 smithi153 ceph-mon[36569]: pgmap v17257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:03 smithi104 ceph-mon[36624]: pgmap v17257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:03 smithi081 ceph-mon[31453]: pgmap v17257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:05 smithi153 ceph-mon[36569]: pgmap v17258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:05 smithi104 ceph-mon[36624]: pgmap v17258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:05.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:05 smithi081 ceph-mon[31453]: pgmap v17258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:07 smithi153 ceph-mon[36569]: pgmap v17259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:14:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:07 smithi104 ceph-mon[36624]: pgmap v17259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:07.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:14:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:07 smithi081 ceph-mon[31453]: pgmap v17259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:07.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:14:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:14:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:14:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:08 smithi153 ceph-mon[36569]: pgmap v17260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:08 smithi153 ceph-mon[36569]: pgmap v17261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:08.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:14:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:14:08.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:14:08.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:08 smithi104 ceph-mon[36624]: pgmap v17260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:08.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:08 smithi104 ceph-mon[36624]: pgmap v17261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:08.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:14:08.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:14:08.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:14:08.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:08 smithi081 ceph-mon[31453]: pgmap v17260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:08.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:08 smithi081 ceph-mon[31453]: pgmap v17261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:08.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:14:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:10 smithi081 ceph-mon[31453]: pgmap v17262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:10 smithi153 ceph-mon[36569]: pgmap v17262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:10 smithi104 ceph-mon[36624]: pgmap v17262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:12 smithi081 ceph-mon[31453]: pgmap v17263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:12 smithi153 ceph-mon[36569]: pgmap v17263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:12 smithi104 ceph-mon[36624]: pgmap v17263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:15.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:14 smithi104 ceph-mon[36624]: pgmap v17264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:14 smithi081 ceph-mon[31453]: pgmap v17264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:14 smithi153 ceph-mon[36569]: pgmap v17264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:16 smithi081 ceph-mon[31453]: pgmap v17265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:16 smithi153 ceph-mon[36569]: pgmap v17265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:16 smithi104 ceph-mon[36624]: pgmap v17265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:17.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:17.383+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:14:17.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:17.383+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:14:17.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:17.383+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:14:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:18 smithi081 ceph-mon[31453]: pgmap v17266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:18 smithi153 ceph-mon[36569]: pgmap v17266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:18 smithi104 ceph-mon[36624]: pgmap v17266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:20 smithi081 ceph-mon[31453]: pgmap v17267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:20 smithi153 ceph-mon[36569]: pgmap v17267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:20 smithi104 ceph-mon[36624]: pgmap v17267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:22 smithi081 ceph-mon[31453]: pgmap v17268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:22 smithi153 ceph-mon[36569]: pgmap v17268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:22 smithi104 ceph-mon[36624]: pgmap v17268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:24 smithi081 ceph-mon[31453]: pgmap v17269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:24 smithi153 ceph-mon[36569]: pgmap v17269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:24 smithi104 ceph-mon[36624]: pgmap v17269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:27.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:26 smithi081 ceph-mon[31453]: pgmap v17270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:26 smithi153 ceph-mon[36569]: pgmap v17270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:26 smithi104 ceph-mon[36624]: pgmap v17270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:28 smithi153 ceph-mon[36569]: pgmap v17271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:28 smithi104 ceph-mon[36624]: pgmap v17271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:29.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:28 smithi081 ceph-mon[31453]: pgmap v17271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:30.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:30 smithi104 ceph-mon[36624]: pgmap v17272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:30 smithi081 ceph-mon[31453]: pgmap v17272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:30.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:30 smithi153 ceph-mon[36569]: pgmap v17272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:32.399+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:14:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:32.399+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:14:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:32.399+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:14:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:32 smithi081 ceph-mon[31453]: pgmap v17273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:32 smithi153 ceph-mon[36569]: pgmap v17273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:32 smithi104 ceph-mon[36624]: pgmap v17273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:34 smithi081 ceph-mon[31453]: pgmap v17274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:34 smithi153 ceph-mon[36569]: pgmap v17274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:34 smithi104 ceph-mon[36624]: pgmap v17274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:36 smithi081 ceph-mon[31453]: pgmap v17275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:36 smithi153 ceph-mon[36569]: pgmap v17275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:36 smithi104 ceph-mon[36624]: pgmap v17275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:38 smithi081 ceph-mon[31453]: pgmap v17276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:38 smithi153 ceph-mon[36569]: pgmap v17276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:38 smithi104 ceph-mon[36624]: pgmap v17276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:40 smithi081 ceph-mon[31453]: pgmap v17277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:40 smithi153 ceph-mon[36569]: pgmap v17277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:40 smithi104 ceph-mon[36624]: pgmap v17277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:42 smithi153 ceph-mon[36569]: pgmap v17278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:42 smithi104 ceph-mon[36624]: pgmap v17278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:43.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:42 smithi081 ceph-mon[31453]: pgmap v17278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:44 smithi153 ceph-mon[36569]: pgmap v17279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:44 smithi104 ceph-mon[36624]: pgmap v17279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:45.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:44 smithi081 ceph-mon[31453]: pgmap v17279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:46 smithi153 ceph-mon[36569]: pgmap v17280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:46 smithi104 ceph-mon[36624]: pgmap v17280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:47.400+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:14:47.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:47.400+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:14:47.403 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:14:47.400+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:14:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:46 smithi081 ceph-mon[31453]: pgmap v17280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:48 smithi153 ceph-mon[36569]: pgmap v17281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:48 smithi104 ceph-mon[36624]: pgmap v17281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:48 smithi081 ceph-mon[31453]: pgmap v17281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:50 smithi153 ceph-mon[36569]: pgmap v17282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:50 smithi104 ceph-mon[36624]: pgmap v17282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:50 smithi081 ceph-mon[31453]: pgmap v17282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:52 smithi153 ceph-mon[36569]: pgmap v17283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:52 smithi104 ceph-mon[36624]: pgmap v17283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:52 smithi081 ceph-mon[31453]: pgmap v17283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:54 smithi153 ceph-mon[36569]: pgmap v17284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:54 smithi104 ceph-mon[36624]: pgmap v17284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:55.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:54 smithi081 ceph-mon[31453]: pgmap v17284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:56 smithi153 ceph-mon[36569]: pgmap v17285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:56 smithi104 ceph-mon[36624]: pgmap v17285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:56 smithi081 ceph-mon[31453]: pgmap v17285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:14:58 smithi153 ceph-mon[36569]: pgmap v17286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:14:58 smithi104 ceph-mon[36624]: pgmap v17286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:14:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:14:58 smithi081 ceph-mon[31453]: pgmap v17286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:00 smithi153 ceph-mon[36569]: pgmap v17287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:00 smithi104 ceph-mon[36624]: pgmap v17287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:00 smithi081 ceph-mon[31453]: pgmap v17287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:02.405+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:15:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:02.405+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:15:02.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:02.405+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:15:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:02 smithi153 ceph-mon[36569]: pgmap v17288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:02 smithi104 ceph-mon[36624]: pgmap v17288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:02 smithi081 ceph-mon[31453]: pgmap v17288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:04 smithi153 ceph-mon[36569]: pgmap v17289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:04 smithi104 ceph-mon[36624]: pgmap v17289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:04 smithi081 ceph-mon[31453]: pgmap v17289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:06 smithi153 ceph-mon[36569]: pgmap v17290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:06 smithi104 ceph-mon[36624]: pgmap v17290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:06 smithi081 ceph-mon[31453]: pgmap v17290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:08 smithi153 ceph-mon[36569]: pgmap v17291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:15:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:08 smithi104 ceph-mon[36624]: pgmap v17291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:15:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:08 smithi081 ceph-mon[31453]: pgmap v17291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:15:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:10 smithi153 ceph-mon[36569]: pgmap v17292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:10 smithi104 ceph-mon[36624]: pgmap v17292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:11.398 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:10 smithi081 ceph-mon[31453]: pgmap v17292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:13.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:12 smithi081 ceph-mon[31453]: pgmap v17293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:13.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:13.289 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:12 smithi153 ceph-mon[36569]: pgmap v17293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:12 smithi104 ceph-mon[36624]: pgmap v17293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:14 smithi153 ceph-mon[36569]: pgmap v17294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:14 smithi104 ceph-mon[36624]: pgmap v17294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:15.343 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:14 smithi081 ceph-mon[31453]: pgmap v17294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:16.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:16.720 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:16.721 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:15:16.721 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:15:16.721 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:16 smithi081 ceph-mon[31453]: pgmap v17295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:16.721 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:16 smithi081 ceph-mon[31453]: pgmap v17296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:16.721 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:15:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:15:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:16 smithi153 ceph-mon[36569]: pgmap v17295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:16 smithi153 ceph-mon[36569]: pgmap v17296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:16.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:15:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:15:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:16 smithi104 ceph-mon[36624]: pgmap v17295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:16 smithi104 ceph-mon[36624]: pgmap v17296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:16.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:15:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:17.411+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:15:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:17.411+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:15:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:17.411+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:15:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:18 smithi153 ceph-mon[36569]: pgmap v17297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:18 smithi104 ceph-mon[36624]: pgmap v17297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:18.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:18 smithi081 ceph-mon[31453]: pgmap v17297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:20.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:20 smithi153 ceph-mon[36569]: pgmap v17298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:20.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:20 smithi104 ceph-mon[36624]: pgmap v17298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:20.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:20 smithi081 ceph-mon[31453]: pgmap v17298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:22.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:22 smithi153 ceph-mon[36569]: pgmap v17299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:22.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:22 smithi104 ceph-mon[36624]: pgmap v17299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:22.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:22 smithi081 ceph-mon[31453]: pgmap v17299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:24.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:24 smithi153 ceph-mon[36569]: pgmap v17300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:24.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:24 smithi104 ceph-mon[36624]: pgmap v17300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:24.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:24 smithi081 ceph-mon[31453]: pgmap v17300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:26.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:26 smithi104 ceph-mon[36624]: pgmap v17301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:26.849 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:26 smithi081 ceph-mon[31453]: pgmap v17301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:26 smithi153 ceph-mon[36569]: pgmap v17301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:28 smithi104 ceph-mon[36624]: pgmap v17302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:28 smithi081 ceph-mon[31453]: pgmap v17302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:29.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:28 smithi153 ceph-mon[36569]: pgmap v17302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:30.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:30 smithi104 ceph-mon[36624]: pgmap v17303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:30.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:30 smithi081 ceph-mon[31453]: pgmap v17303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:30 smithi153 ceph-mon[36569]: pgmap v17303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:32.426+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:15:32.428 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:32.426+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:15:32.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:32.426+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:15:32.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:32 smithi104 ceph-mon[36624]: pgmap v17304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:32.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:32 smithi081 ceph-mon[31453]: pgmap v17304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:32 smithi153 ceph-mon[36569]: pgmap v17304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:34.860 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:34 smithi153 ceph-mon[36569]: pgmap v17305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:34.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:34 smithi081 ceph-mon[31453]: pgmap v17305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:34 smithi104 ceph-mon[36624]: pgmap v17305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:36.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:36 smithi081 ceph-mon[31453]: pgmap v17306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:36 smithi153 ceph-mon[36569]: pgmap v17306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:36 smithi104 ceph-mon[36624]: pgmap v17306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:38.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:38 smithi081 ceph-mon[31453]: pgmap v17307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:38 smithi153 ceph-mon[36569]: pgmap v17307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:38 smithi104 ceph-mon[36624]: pgmap v17307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:40.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:40 smithi081 ceph-mon[31453]: pgmap v17308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:40 smithi153 ceph-mon[36569]: pgmap v17308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:41.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:40 smithi104 ceph-mon[36624]: pgmap v17308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:42.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:42 smithi081 ceph-mon[31453]: pgmap v17309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:42 smithi153 ceph-mon[36569]: pgmap v17309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:43.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:42 smithi104 ceph-mon[36624]: pgmap v17309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:44.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:44 smithi081 ceph-mon[31453]: pgmap v17310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:44 smithi153 ceph-mon[36569]: pgmap v17310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:45.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:44 smithi104 ceph-mon[36624]: pgmap v17310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:46.886 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:46 smithi104 ceph-mon[36624]: pgmap v17311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:46 smithi081 ceph-mon[31453]: pgmap v17311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:46 smithi153 ceph-mon[36569]: pgmap v17311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:47.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:47.442+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:15:47.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:47.442+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:15:47.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:15:47.442+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:15:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:48 smithi081 ceph-mon[31453]: pgmap v17312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:48 smithi153 ceph-mon[36569]: pgmap v17312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:49.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:48 smithi104 ceph-mon[36624]: pgmap v17312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:50.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:50 smithi081 ceph-mon[31453]: pgmap v17313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:50 smithi153 ceph-mon[36569]: pgmap v17313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:51.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:50 smithi104 ceph-mon[36624]: pgmap v17313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:52.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:52 smithi081 ceph-mon[31453]: pgmap v17314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:52 smithi153 ceph-mon[36569]: pgmap v17314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:52 smithi104 ceph-mon[36624]: pgmap v17314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:54.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:54 smithi081 ceph-mon[31453]: pgmap v17315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:54 smithi153 ceph-mon[36569]: pgmap v17315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:54 smithi104 ceph-mon[36624]: pgmap v17315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:56.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:56 smithi081 ceph-mon[31453]: pgmap v17316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:56 smithi153 ceph-mon[36569]: pgmap v17316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:56 smithi104 ceph-mon[36624]: pgmap v17316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:15:58 smithi081 ceph-mon[31453]: pgmap v17317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:15:58 smithi153 ceph-mon[36569]: pgmap v17317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:15:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:15:58 smithi104 ceph-mon[36624]: pgmap v17317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:00 smithi081 ceph-mon[31453]: pgmap v17318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:00 smithi153 ceph-mon[36569]: pgmap v17318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:00 smithi104 ceph-mon[36624]: pgmap v17318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:02.455+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:16:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:02.455+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:16:02.458 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:02.455+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:16:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:02 smithi081 ceph-mon[31453]: pgmap v17319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:02 smithi153 ceph-mon[36569]: pgmap v17319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:02 smithi104 ceph-mon[36624]: pgmap v17319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:04 smithi153 ceph-mon[36569]: pgmap v17320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:05.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:04 smithi104 ceph-mon[36624]: pgmap v17320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:04 smithi081 ceph-mon[31453]: pgmap v17320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:06 smithi153 ceph-mon[36569]: pgmap v17321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:07.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:06 smithi104 ceph-mon[36624]: pgmap v17321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:06 smithi081 ceph-mon[31453]: pgmap v17321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:08 smithi153 ceph-mon[36569]: pgmap v17322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:09.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:08 smithi104 ceph-mon[36624]: pgmap v17322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:08 smithi081 ceph-mon[31453]: pgmap v17322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:10 smithi153 ceph-mon[36569]: pgmap v17323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:10 smithi104 ceph-mon[36624]: pgmap v17323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:10 smithi081 ceph-mon[31453]: pgmap v17323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:12.178+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 msgr2=0x7f16e80b1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000678) for more than 900000000 us, fault. 2024-04-13T06:16:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:12.178+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1e40 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006750 tx=0x7f16f4044000 comp rx=0 tx=0).stop 2024-04-13T06:16:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:12.178+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 msgr2=0x7f16e80b1e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T06:16:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:12.178+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1e40 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T06:16:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:12.178+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T06:16:12.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:12.179+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T06:16:12.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:12.180+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006660 tx=0x7f16f402b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T06:16:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:12 smithi153 ceph-mon[36569]: pgmap v17324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:12 smithi104 ceph-mon[36624]: pgmap v17324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:12 smithi081 ceph-mon[31453]: pgmap v17324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:14.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:14 smithi081 ceph-mon[31453]: pgmap v17325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:14 smithi153 ceph-mon[36569]: pgmap v17325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:15.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:14 smithi104 ceph-mon[36624]: pgmap v17325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:16 smithi153 ceph-mon[36569]: pgmap v17326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:16:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:16 smithi104 ceph-mon[36624]: pgmap v17326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:17.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:16:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:16 smithi081 ceph-mon[31453]: pgmap v17326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:16:17.473 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:17.472+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:16:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:17.472+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:16:17.474 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:17.472+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:16:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:18.060+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:16:18.062 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:18.060+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:16:18.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:18.061+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 250 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:16:18.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:18.061+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 250 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:16:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:18 smithi153 ceph-mon[36569]: pgmap v17327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:18 smithi104 ceph-mon[36624]: pgmap v17327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:18 smithi081 ceph-mon[31453]: pgmap v17327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:20.963 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:20 smithi153 ceph-mon[36569]: pgmap v17328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:20 smithi104 ceph-mon[36624]: pgmap v17328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:21.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:20 smithi081 ceph-mon[31453]: pgmap v17328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:16:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:16:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:21 smithi104 ceph-mon[36624]: pgmap v17329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:21 smithi104 ceph-mon[36624]: pgmap v17330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:22.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:16:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:16:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:21 smithi081 ceph-mon[31453]: pgmap v17329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:21 smithi081 ceph-mon[31453]: pgmap v17330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:22.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:16:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:16:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:21 smithi153 ceph-mon[36569]: pgmap v17329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:21 smithi153 ceph-mon[36569]: pgmap v17330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:16:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:23 smithi081 ceph-mon[31453]: pgmap v17331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:23 smithi153 ceph-mon[36569]: pgmap v17331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:23 smithi104 ceph-mon[36624]: pgmap v17331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:25 smithi153 ceph-mon[36569]: pgmap v17332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:25 smithi104 ceph-mon[36624]: pgmap v17332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:26.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:25 smithi081 ceph-mon[31453]: pgmap v17332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:27 smithi153 ceph-mon[36569]: pgmap v17333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:27 smithi104 ceph-mon[36624]: pgmap v17333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:27 smithi081 ceph-mon[31453]: pgmap v17333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:29 smithi153 ceph-mon[36569]: pgmap v17334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:29 smithi104 ceph-mon[36624]: pgmap v17334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:29 smithi081 ceph-mon[31453]: pgmap v17334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:31 smithi153 ceph-mon[36569]: pgmap v17335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:31 smithi104 ceph-mon[36624]: pgmap v17335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:32.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:31 smithi081 ceph-mon[31453]: pgmap v17335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:32.489 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:32.486+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:16:32.489 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:32.486+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:16:32.489 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:32.486+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:16:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:33 smithi153 ceph-mon[36569]: pgmap v17336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:33 smithi104 ceph-mon[36624]: pgmap v17336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:33 smithi081 ceph-mon[31453]: pgmap v17336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:35 smithi153 ceph-mon[36569]: pgmap v17337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:35 smithi104 ceph-mon[36624]: pgmap v17337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:35 smithi081 ceph-mon[31453]: pgmap v17337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:37 smithi153 ceph-mon[36569]: pgmap v17338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:37 smithi104 ceph-mon[36624]: pgmap v17338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:37 smithi081 ceph-mon[31453]: pgmap v17338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:39 smithi153 ceph-mon[36569]: pgmap v17339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:39 smithi104 ceph-mon[36624]: pgmap v17339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:39 smithi081 ceph-mon[31453]: pgmap v17339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:41 smithi153 ceph-mon[36569]: pgmap v17340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:41 smithi104 ceph-mon[36624]: pgmap v17340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:41 smithi081 ceph-mon[31453]: pgmap v17340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:44.240 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:43 smithi081 ceph-mon[31453]: pgmap v17341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:43 smithi153 ceph-mon[36569]: pgmap v17341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:43 smithi104 ceph-mon[36624]: pgmap v17341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:46.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:45 smithi081 ceph-mon[31453]: pgmap v17342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:45 smithi153 ceph-mon[36569]: pgmap v17342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:45 smithi104 ceph-mon[36624]: pgmap v17342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:47.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:47.502+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:16:47.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:47.502+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:16:47.505 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:16:47.502+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:16:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:47 smithi153 ceph-mon[36569]: pgmap v17343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:47 smithi104 ceph-mon[36624]: pgmap v17343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:47 smithi081 ceph-mon[31453]: pgmap v17343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:49 smithi153 ceph-mon[36569]: pgmap v17344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:49 smithi104 ceph-mon[36624]: pgmap v17344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:49 smithi081 ceph-mon[31453]: pgmap v17344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:51 smithi153 ceph-mon[36569]: pgmap v17345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:51 smithi104 ceph-mon[36624]: pgmap v17345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:51 smithi081 ceph-mon[31453]: pgmap v17345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:54.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:53 smithi153 ceph-mon[36569]: pgmap v17346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:53 smithi104 ceph-mon[36624]: pgmap v17346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:53 smithi081 ceph-mon[31453]: pgmap v17346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:55 smithi153 ceph-mon[36569]: pgmap v17347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:55 smithi104 ceph-mon[36624]: pgmap v17347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:55 smithi081 ceph-mon[31453]: pgmap v17347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:16:58 smithi153 ceph-mon[36569]: pgmap v17348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:16:58 smithi104 ceph-mon[36624]: pgmap v17348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:16:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:16:58 smithi081 ceph-mon[31453]: pgmap v17348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:00 smithi153 ceph-mon[36569]: pgmap v17349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:00 smithi104 ceph-mon[36624]: pgmap v17349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:00 smithi081 ceph-mon[31453]: pgmap v17349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:02 smithi153 ceph-mon[36569]: pgmap v17350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:02 smithi104 ceph-mon[36624]: pgmap v17350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:02.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:02 smithi081 ceph-mon[31453]: pgmap v17350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:02.518+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:17:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:02.518+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:17:02.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:02.518+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:17:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:04 smithi153 ceph-mon[36569]: pgmap v17351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:04 smithi104 ceph-mon[36624]: pgmap v17351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:04 smithi081 ceph-mon[31453]: pgmap v17351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:06 smithi153 ceph-mon[36569]: pgmap v17352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:06 smithi104 ceph-mon[36624]: pgmap v17352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:06 smithi081 ceph-mon[31453]: pgmap v17352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:08 smithi153 ceph-mon[36569]: pgmap v17353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:08 smithi104 ceph-mon[36624]: pgmap v17353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:08.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:08 smithi081 ceph-mon[31453]: pgmap v17353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:10 smithi153 ceph-mon[36569]: pgmap v17354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:10 smithi104 ceph-mon[36624]: pgmap v17354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:10.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:10 smithi081 ceph-mon[31453]: pgmap v17354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:12.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:12 smithi153 ceph-mon[36569]: pgmap v17355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:12.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:12 smithi104 ceph-mon[36624]: pgmap v17355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:12.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:12 smithi081 ceph-mon[31453]: pgmap v17355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:14 smithi153 ceph-mon[36569]: pgmap v17356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:14 smithi104 ceph-mon[36624]: pgmap v17356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:14.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:14 smithi081 ceph-mon[31453]: pgmap v17356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:16 smithi153 ceph-mon[36569]: pgmap v17357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:16 smithi104 ceph-mon[36624]: pgmap v17357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:16.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:16 smithi081 ceph-mon[31453]: pgmap v17357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:17.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:17.533+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:17:17.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:17.533+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:17:17.535 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:17.533+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:17:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:18 smithi153 ceph-mon[36569]: pgmap v17358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:18 smithi104 ceph-mon[36624]: pgmap v17358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:18.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:18 smithi081 ceph-mon[31453]: pgmap v17358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:20 smithi104 ceph-mon[36624]: pgmap v17359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:20.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:20 smithi081 ceph-mon[31453]: pgmap v17359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:20.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:20 smithi153 ceph-mon[36569]: pgmap v17359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:22 smithi153 ceph-mon[36569]: pgmap v17360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:17:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:22 smithi104 ceph-mon[36624]: pgmap v17360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:17:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:22 smithi081 ceph-mon[31453]: pgmap v17360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:17:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:24 smithi104 ceph-mon[36624]: pgmap v17361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:24 smithi081 ceph-mon[31453]: pgmap v17361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:24.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:24 smithi153 ceph-mon[36569]: pgmap v17361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:25.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:25.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:25.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:25 smithi081 ceph-mon[31453]: pgmap v17362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:25 smithi153 ceph-mon[36569]: pgmap v17362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:25.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:25 smithi104 ceph-mon[36624]: pgmap v17362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:27 smithi153 ceph-mon[36569]: pgmap v17363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:27.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:27 smithi104 ceph-mon[36624]: pgmap v17363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:27 smithi081 ceph-mon[31453]: pgmap v17363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:17:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:17:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:28 smithi153 ceph-mon[36569]: pgmap v17364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:28.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:28 smithi153 ceph-mon[36569]: pgmap v17365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:28.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:17:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:17:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:28 smithi104 ceph-mon[36624]: pgmap v17364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:28 smithi104 ceph-mon[36624]: pgmap v17365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:28.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:17:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:17:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:28 smithi081 ceph-mon[31453]: pgmap v17364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:28 smithi081 ceph-mon[31453]: pgmap v17365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:28.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:17:31.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:30 smithi153 ceph-mon[36569]: pgmap v17366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:31.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:30 smithi104 ceph-mon[36624]: pgmap v17366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:30 smithi081 ceph-mon[31453]: pgmap v17366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:32.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:32.544+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:17:32.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:32.544+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:17:32.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:32.544+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:17:33.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:32 smithi153 ceph-mon[36569]: pgmap v17367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:32 smithi104 ceph-mon[36624]: pgmap v17367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:32 smithi081 ceph-mon[31453]: pgmap v17367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:34 smithi153 ceph-mon[36569]: pgmap v17368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:34 smithi104 ceph-mon[36624]: pgmap v17368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:34 smithi081 ceph-mon[31453]: pgmap v17368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:37.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:36 smithi153 ceph-mon[36569]: pgmap v17369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:37.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:36 smithi104 ceph-mon[36624]: pgmap v17369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:36 smithi081 ceph-mon[31453]: pgmap v17369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:39.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:38 smithi153 ceph-mon[36569]: pgmap v17370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:38 smithi104 ceph-mon[36624]: pgmap v17370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:38 smithi081 ceph-mon[31453]: pgmap v17370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:41.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:40 smithi153 ceph-mon[36569]: pgmap v17371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:41.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:40 smithi104 ceph-mon[36624]: pgmap v17371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:41.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:40 smithi081 ceph-mon[31453]: pgmap v17371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:43.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:42 smithi153 ceph-mon[36569]: pgmap v17372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:43.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:42 smithi104 ceph-mon[36624]: pgmap v17372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:42 smithi081 ceph-mon[31453]: pgmap v17372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:45.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:44 smithi153 ceph-mon[36569]: pgmap v17373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:45.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:44 smithi104 ceph-mon[36624]: pgmap v17373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:45.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:44 smithi081 ceph-mon[31453]: pgmap v17373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:47.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:46 smithi153 ceph-mon[36569]: pgmap v17374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:47.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:46 smithi104 ceph-mon[36624]: pgmap v17374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:47.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:46 smithi081 ceph-mon[31453]: pgmap v17374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:47.559+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:17:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:47.559+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:17:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:17:47.559+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:17:49.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:48 smithi153 ceph-mon[36569]: pgmap v17375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:49.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:48 smithi104 ceph-mon[36624]: pgmap v17375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:48 smithi081 ceph-mon[31453]: pgmap v17375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:51.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:50 smithi153 ceph-mon[36569]: pgmap v17376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:51.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:50 smithi104 ceph-mon[36624]: pgmap v17376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:51.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:50 smithi081 ceph-mon[31453]: pgmap v17376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:53.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:52 smithi153 ceph-mon[36569]: pgmap v17377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:53.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:52 smithi104 ceph-mon[36624]: pgmap v17377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:53.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:52 smithi081 ceph-mon[31453]: pgmap v17377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:55.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:54 smithi153 ceph-mon[36569]: pgmap v17378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:54 smithi104 ceph-mon[36624]: pgmap v17378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:54 smithi081 ceph-mon[31453]: pgmap v17378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:56 smithi153 ceph-mon[36569]: pgmap v17379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:57.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:56 smithi104 ceph-mon[36624]: pgmap v17379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:56 smithi081 ceph-mon[31453]: pgmap v17379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:17:58 smithi153 ceph-mon[36569]: pgmap v17380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:59.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:17:58 smithi104 ceph-mon[36624]: pgmap v17380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:17:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:17:58 smithi081 ceph-mon[31453]: pgmap v17380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:00 smithi153 ceph-mon[36569]: pgmap v17381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:00 smithi104 ceph-mon[36624]: pgmap v17381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:01.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:00 smithi081 ceph-mon[31453]: pgmap v17381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:02.565+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:02.565+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:18:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:02.565+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:18:03.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:02 smithi104 ceph-mon[36624]: pgmap v17382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:03.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:02 smithi081 ceph-mon[31453]: pgmap v17382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:02 smithi153 ceph-mon[36569]: pgmap v17382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:04 smithi104 ceph-mon[36624]: pgmap v17383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:05.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:04 smithi153 ceph-mon[36569]: pgmap v17383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:04 smithi081 ceph-mon[31453]: pgmap v17383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:06 smithi104 ceph-mon[36624]: pgmap v17384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:07.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:06 smithi081 ceph-mon[31453]: pgmap v17384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:06 smithi153 ceph-mon[36569]: pgmap v17384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:08 smithi104 ceph-mon[36624]: pgmap v17385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:09.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:08 smithi081 ceph-mon[31453]: pgmap v17385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:08 smithi153 ceph-mon[36569]: pgmap v17385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:10 smithi104 ceph-mon[36624]: pgmap v17386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:10 smithi081 ceph-mon[31453]: pgmap v17386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:10 smithi153 ceph-mon[36569]: pgmap v17386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:12 smithi081 ceph-mon[31453]: pgmap v17387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:12 smithi153 ceph-mon[36569]: pgmap v17387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:12 smithi104 ceph-mon[36624]: pgmap v17387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:15.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:14 smithi081 ceph-mon[31453]: pgmap v17388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:14 smithi153 ceph-mon[36569]: pgmap v17388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:14 smithi104 ceph-mon[36624]: pgmap v17388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:17.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:16 smithi104 ceph-mon[36624]: pgmap v17389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:17.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:16 smithi081 ceph-mon[31453]: pgmap v17389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:16 smithi153 ceph-mon[36569]: pgmap v17389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:17.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:17.572+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:18:17.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:17.572+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:18:17.581 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:17.572+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:18:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:18 smithi081 ceph-mon[31453]: pgmap v17390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:18 smithi153 ceph-mon[36569]: pgmap v17390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:18 smithi104 ceph-mon[36624]: pgmap v17390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:20 smithi081 ceph-mon[31453]: pgmap v17391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:20 smithi153 ceph-mon[36569]: pgmap v17391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:20 smithi104 ceph-mon[36624]: pgmap v17391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:22 smithi081 ceph-mon[31453]: pgmap v17392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:22 smithi153 ceph-mon[36569]: pgmap v17392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:22 smithi104 ceph-mon[36624]: pgmap v17392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:25.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:24 smithi081 ceph-mon[31453]: pgmap v17393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:24 smithi153 ceph-mon[36569]: pgmap v17393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:24 smithi104 ceph-mon[36624]: pgmap v17393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:26 smithi081 ceph-mon[31453]: pgmap v17394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:26 smithi153 ceph-mon[36569]: pgmap v17394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:26 smithi104 ceph-mon[36624]: pgmap v17394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:28 smithi081 ceph-mon[31453]: pgmap v17395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:18:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:28 smithi153 ceph-mon[36569]: pgmap v17395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:18:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:28 smithi104 ceph-mon[36624]: pgmap v17395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:18:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:18:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:18:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:18:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:18:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:18:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:18:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:18:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:30 smithi081 ceph-mon[31453]: pgmap v17396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:31.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:30 smithi081 ceph-mon[31453]: pgmap v17397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:30 smithi153 ceph-mon[36569]: pgmap v17396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:30 smithi153 ceph-mon[36569]: pgmap v17397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:30 smithi104 ceph-mon[36624]: pgmap v17396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:30 smithi104 ceph-mon[36624]: pgmap v17397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:32.581+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:18:32.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:32.581+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:18:32.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:32.581+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:18:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:32 smithi081 ceph-mon[31453]: pgmap v17398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:32 smithi153 ceph-mon[36569]: pgmap v17398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:33.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:32 smithi104 ceph-mon[36624]: pgmap v17398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:34 smithi081 ceph-mon[31453]: pgmap v17399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:34 smithi153 ceph-mon[36569]: pgmap v17399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:34 smithi104 ceph-mon[36624]: pgmap v17399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:36 smithi153 ceph-mon[36569]: pgmap v17400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:36 smithi104 ceph-mon[36624]: pgmap v17400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:37.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:36 smithi081 ceph-mon[31453]: pgmap v17400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:38 smithi153 ceph-mon[36569]: pgmap v17401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:38 smithi104 ceph-mon[36624]: pgmap v17401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:39.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:38 smithi081 ceph-mon[31453]: pgmap v17401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:40 smithi153 ceph-mon[36569]: pgmap v17402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:40 smithi104 ceph-mon[36624]: pgmap v17402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:41.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:40 smithi081 ceph-mon[31453]: pgmap v17402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:42 smithi153 ceph-mon[36569]: pgmap v17403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:42 smithi104 ceph-mon[36624]: pgmap v17403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:43.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:42 smithi081 ceph-mon[31453]: pgmap v17403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:45.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:44 smithi081 ceph-mon[31453]: pgmap v17404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:44 smithi153 ceph-mon[36569]: pgmap v17404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:44 smithi104 ceph-mon[36624]: pgmap v17404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:46 smithi153 ceph-mon[36569]: pgmap v17405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:46 smithi104 ceph-mon[36624]: pgmap v17405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:46 smithi081 ceph-mon[31453]: pgmap v17405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:47.597+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:18:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:47.597+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:18:47.599 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:18:47.597+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:18:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:48 smithi153 ceph-mon[36569]: pgmap v17406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:48 smithi104 ceph-mon[36624]: pgmap v17406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:48 smithi081 ceph-mon[31453]: pgmap v17406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:50 smithi153 ceph-mon[36569]: pgmap v17407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:50 smithi104 ceph-mon[36624]: pgmap v17407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:50 smithi081 ceph-mon[31453]: pgmap v17407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:52 smithi153 ceph-mon[36569]: pgmap v17408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:52 smithi104 ceph-mon[36624]: pgmap v17408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:52 smithi081 ceph-mon[31453]: pgmap v17408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:54 smithi153 ceph-mon[36569]: pgmap v17409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:54 smithi104 ceph-mon[36624]: pgmap v17409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:55.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:54 smithi081 ceph-mon[31453]: pgmap v17409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:56 smithi153 ceph-mon[36569]: pgmap v17410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:56 smithi104 ceph-mon[36624]: pgmap v17410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:56 smithi081 ceph-mon[31453]: pgmap v17410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:18:58 smithi153 ceph-mon[36569]: pgmap v17411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:18:58 smithi104 ceph-mon[36624]: pgmap v17411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:18:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:18:58 smithi081 ceph-mon[31453]: pgmap v17411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:00 smithi153 ceph-mon[36569]: pgmap v17412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:00 smithi104 ceph-mon[36624]: pgmap v17412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:00 smithi081 ceph-mon[31453]: pgmap v17412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:02.605+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:19:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:02.606+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:19:02.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:02.606+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:19:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:02 smithi153 ceph-mon[36569]: pgmap v17413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:02 smithi104 ceph-mon[36624]: pgmap v17413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:02 smithi081 ceph-mon[31453]: pgmap v17413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:04 smithi153 ceph-mon[36569]: pgmap v17414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:04 smithi104 ceph-mon[36624]: pgmap v17414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:04 smithi081 ceph-mon[31453]: pgmap v17414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:07.273 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:06 smithi081 ceph-mon[31453]: pgmap v17415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:06 smithi153 ceph-mon[36569]: pgmap v17415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:06 smithi104 ceph-mon[36624]: pgmap v17415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:08 smithi153 ceph-mon[36569]: pgmap v17416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:08 smithi104 ceph-mon[36624]: pgmap v17416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:08 smithi081 ceph-mon[31453]: pgmap v17416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:10 smithi153 ceph-mon[36569]: pgmap v17417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:10 smithi104 ceph-mon[36624]: pgmap v17417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:10 smithi081 ceph-mon[31453]: pgmap v17417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:13 smithi153 ceph-mon[36569]: pgmap v17418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:13 smithi104 ceph-mon[36624]: pgmap v17418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:13 smithi081 ceph-mon[31453]: pgmap v17418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:15 smithi153 ceph-mon[36569]: pgmap v17419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:15 smithi104 ceph-mon[36624]: pgmap v17419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:15.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:15 smithi081 ceph-mon[31453]: pgmap v17419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:17 smithi153 ceph-mon[36569]: pgmap v17420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:17 smithi104 ceph-mon[36624]: pgmap v17420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:17 smithi081 ceph-mon[31453]: pgmap v17420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:17.621+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:19:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:17.621+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:19:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:17.621+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:19:18.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:18.091+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:19:18.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:18.091+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:19:18.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:18.091+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 251 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:19:18.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:18.091+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 251 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:19:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:19 smithi153 ceph-mon[36569]: pgmap v17421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:19 smithi104 ceph-mon[36624]: pgmap v17421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:19 smithi081 ceph-mon[31453]: pgmap v17421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:21 smithi153 ceph-mon[36569]: pgmap v17422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:21 smithi104 ceph-mon[36624]: pgmap v17422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:21 smithi081 ceph-mon[31453]: pgmap v17422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:23 smithi153 ceph-mon[36569]: pgmap v17423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:23 smithi104 ceph-mon[36624]: pgmap v17423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:23 smithi081 ceph-mon[31453]: pgmap v17423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:25 smithi153 ceph-mon[36569]: pgmap v17424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:25 smithi104 ceph-mon[36624]: pgmap v17424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:25 smithi081 ceph-mon[31453]: pgmap v17424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:27 smithi153 ceph-mon[36569]: pgmap v17425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:27 smithi104 ceph-mon[36624]: pgmap v17425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:27 smithi081 ceph-mon[31453]: pgmap v17425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:29 smithi153 ceph-mon[36569]: pgmap v17426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:29 smithi104 ceph-mon[36624]: pgmap v17426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:29 smithi081 ceph-mon[31453]: pgmap v17426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:19:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:19:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:19:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:31 smithi104 ceph-mon[36624]: pgmap v17427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:19:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:19:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:19:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:31 smithi081 ceph-mon[31453]: pgmap v17427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:19:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:19:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:19:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:31 smithi153 ceph-mon[36569]: pgmap v17427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:19:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:19:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:19:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:32 smithi104 ceph-mon[36624]: pgmap v17428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:32 smithi104 ceph-mon[36624]: pgmap v17429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:32.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:32 smithi081 ceph-mon[31453]: pgmap v17428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:32.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:32 smithi081 ceph-mon[31453]: pgmap v17429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:32 smithi153 ceph-mon[36569]: pgmap v17428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:32 smithi153 ceph-mon[36569]: pgmap v17429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:32.623+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:19:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:32.623+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:19:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:32.623+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:19:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:34 smithi104 ceph-mon[36624]: pgmap v17430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:34 smithi081 ceph-mon[31453]: pgmap v17430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:34 smithi153 ceph-mon[36569]: pgmap v17430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:36 smithi104 ceph-mon[36624]: pgmap v17431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:36.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:36 smithi153 ceph-mon[36569]: pgmap v17431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:36 smithi081 ceph-mon[31453]: pgmap v17431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:38 smithi081 ceph-mon[31453]: pgmap v17432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:38 smithi153 ceph-mon[36569]: pgmap v17432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:38 smithi104 ceph-mon[36624]: pgmap v17432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:40 smithi081 ceph-mon[31453]: pgmap v17433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:40 smithi153 ceph-mon[36569]: pgmap v17433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:40 smithi104 ceph-mon[36624]: pgmap v17433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:42 smithi081 ceph-mon[31453]: pgmap v17434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:42 smithi153 ceph-mon[36569]: pgmap v17434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:42 smithi104 ceph-mon[36624]: pgmap v17434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:44 smithi081 ceph-mon[31453]: pgmap v17435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:44 smithi153 ceph-mon[36569]: pgmap v17435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:44 smithi104 ceph-mon[36624]: pgmap v17435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:46 smithi081 ceph-mon[31453]: pgmap v17436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:46 smithi153 ceph-mon[36569]: pgmap v17436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:46 smithi104 ceph-mon[36624]: pgmap v17436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:47.636+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:19:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:47.636+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:19:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:19:47.636+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:19:48.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:48 smithi104 ceph-mon[36624]: pgmap v17437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:48 smithi081 ceph-mon[31453]: pgmap v17437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:48 smithi153 ceph-mon[36569]: pgmap v17437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:50 smithi081 ceph-mon[31453]: pgmap v17438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:50 smithi153 ceph-mon[36569]: pgmap v17438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:50.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:50 smithi104 ceph-mon[36624]: pgmap v17438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:52 smithi081 ceph-mon[31453]: pgmap v17439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:52 smithi153 ceph-mon[36569]: pgmap v17439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:52 smithi104 ceph-mon[36624]: pgmap v17439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:54 smithi081 ceph-mon[31453]: pgmap v17440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:54.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:54 smithi153 ceph-mon[36569]: pgmap v17440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:54.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:54 smithi104 ceph-mon[36624]: pgmap v17440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:56 smithi081 ceph-mon[31453]: pgmap v17441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:56 smithi153 ceph-mon[36569]: pgmap v17441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:56 smithi104 ceph-mon[36624]: pgmap v17441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:19:58 smithi081 ceph-mon[31453]: pgmap v17442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:19:58 smithi153 ceph-mon[36569]: pgmap v17442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:19:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:19:58 smithi104 ceph-mon[36624]: pgmap v17442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:00 smithi081 ceph-mon[31453]: pgmap v17443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:00.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:20:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:00 smithi153 ceph-mon[36569]: pgmap v17443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:20:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:00 smithi104 ceph-mon[36624]: pgmap v17443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:20:02.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:02 smithi081 ceph-mon[31453]: pgmap v17444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:02 smithi153 ceph-mon[36569]: pgmap v17444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:02 smithi104 ceph-mon[36624]: pgmap v17444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:02.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:02.652+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:20:02.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:02.652+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:20:02.654 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:02.652+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:20:04.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:04 smithi081 ceph-mon[31453]: pgmap v17445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:04 smithi153 ceph-mon[36569]: pgmap v17445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:04.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:04 smithi104 ceph-mon[36624]: pgmap v17445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:06 smithi153 ceph-mon[36569]: pgmap v17446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:06 smithi104 ceph-mon[36624]: pgmap v17446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:06.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:06 smithi081 ceph-mon[31453]: pgmap v17446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:08 smithi153 ceph-mon[36569]: pgmap v17447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:08 smithi104 ceph-mon[36624]: pgmap v17447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:08 smithi081 ceph-mon[31453]: pgmap v17447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:10 smithi153 ceph-mon[36569]: pgmap v17448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:10.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:10 smithi104 ceph-mon[36624]: pgmap v17448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:10.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:10 smithi081 ceph-mon[31453]: pgmap v17448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:12 smithi153 ceph-mon[36569]: pgmap v17449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:12.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:12 smithi104 ceph-mon[36624]: pgmap v17449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:12.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:12 smithi081 ceph-mon[31453]: pgmap v17449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:14 smithi153 ceph-mon[36569]: pgmap v17450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:14.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:14 smithi104 ceph-mon[36624]: pgmap v17450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:14 smithi081 ceph-mon[31453]: pgmap v17450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:16.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:16 smithi081 ceph-mon[31453]: pgmap v17451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:16 smithi153 ceph-mon[36569]: pgmap v17451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:16 smithi104 ceph-mon[36624]: pgmap v17451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:17.662+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:20:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:17.662+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:20:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:17.662+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:20:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:18 smithi153 ceph-mon[36569]: pgmap v17452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:18.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:18 smithi104 ceph-mon[36624]: pgmap v17452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:18 smithi081 ceph-mon[31453]: pgmap v17452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:20.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:20 smithi153 ceph-mon[36569]: pgmap v17453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:20.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:20 smithi104 ceph-mon[36624]: pgmap v17453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:20.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:20 smithi081 ceph-mon[31453]: pgmap v17453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:22 smithi153 ceph-mon[36569]: pgmap v17454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:22 smithi104 ceph-mon[36624]: pgmap v17454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:22.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:22 smithi081 ceph-mon[31453]: pgmap v17454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:24 smithi153 ceph-mon[36569]: pgmap v17455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:24 smithi104 ceph-mon[36624]: pgmap v17455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:24.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:24 smithi081 ceph-mon[31453]: pgmap v17455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:26 smithi153 ceph-mon[36569]: pgmap v17456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:26 smithi104 ceph-mon[36624]: pgmap v17456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:26.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:26 smithi081 ceph-mon[31453]: pgmap v17456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:28.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:28 smithi153 ceph-mon[36569]: pgmap v17457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:28 smithi104 ceph-mon[36624]: pgmap v17457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:28 smithi081 ceph-mon[31453]: pgmap v17457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:30 smithi153 ceph-mon[36569]: pgmap v17458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:30 smithi104 ceph-mon[36624]: pgmap v17458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:30.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:30 smithi081 ceph-mon[31453]: pgmap v17458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:20:31.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:20:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:20:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:32 smithi153 ceph-mon[36569]: pgmap v17459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:32.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:32 smithi104 ceph-mon[36624]: pgmap v17459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:32 smithi081 ceph-mon[31453]: pgmap v17459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:32.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:32.669+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:20:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:32.669+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:20:32.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:32.669+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:20:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:20:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:20:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:20:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:20:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:20:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:20:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:20:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:20:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:20:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:34 smithi153 ceph-mon[36569]: pgmap v17460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:34 smithi153 ceph-mon[36569]: pgmap v17461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:34.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:34 smithi104 ceph-mon[36624]: pgmap v17460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:34.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:34 smithi104 ceph-mon[36624]: pgmap v17461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:34.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:34 smithi081 ceph-mon[31453]: pgmap v17460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:34.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:34 smithi081 ceph-mon[31453]: pgmap v17461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:36 smithi153 ceph-mon[36569]: pgmap v17462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:36 smithi104 ceph-mon[36624]: pgmap v17462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:36 smithi081 ceph-mon[31453]: pgmap v17462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:38.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:38 smithi153 ceph-mon[36569]: pgmap v17463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:38.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:38 smithi104 ceph-mon[36624]: pgmap v17463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:38 smithi081 ceph-mon[31453]: pgmap v17463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:40 smithi153 ceph-mon[36569]: pgmap v17464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:40.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:40 smithi104 ceph-mon[36624]: pgmap v17464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:40.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:40 smithi081 ceph-mon[31453]: pgmap v17464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:42 smithi153 ceph-mon[36569]: pgmap v17465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:42.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:42 smithi104 ceph-mon[36624]: pgmap v17465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:42 smithi081 ceph-mon[31453]: pgmap v17465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:44 smithi153 ceph-mon[36569]: pgmap v17466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:44 smithi104 ceph-mon[36624]: pgmap v17466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:44.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:44 smithi081 ceph-mon[31453]: pgmap v17466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:46 smithi153 ceph-mon[36569]: pgmap v17467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:46 smithi104 ceph-mon[36624]: pgmap v17467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:46.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:46 smithi081 ceph-mon[31453]: pgmap v17467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:47.678+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:20:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:47.678+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:20:47.680 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:47.678+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:20:48.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:48.106+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:20:48.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:48.106+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:20:48.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:48.106+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 252 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:20:48.108 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:20:48.107+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 252 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:20:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:48 smithi153 ceph-mon[36569]: pgmap v17468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:48 smithi104 ceph-mon[36624]: pgmap v17468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:48.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:48 smithi081 ceph-mon[31453]: pgmap v17468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:50 smithi153 ceph-mon[36569]: pgmap v17469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:50 smithi104 ceph-mon[36624]: pgmap v17469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:50.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:50 smithi081 ceph-mon[31453]: pgmap v17469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:52 smithi153 ceph-mon[36569]: pgmap v17470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:52.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:52 smithi104 ceph-mon[36624]: pgmap v17470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:52 smithi081 ceph-mon[31453]: pgmap v17470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:54.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:54 smithi104 ceph-mon[36624]: pgmap v17471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:54.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:54 smithi081 ceph-mon[31453]: pgmap v17471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:54 smithi153 ceph-mon[36569]: pgmap v17471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:56.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:56 smithi104 ceph-mon[36624]: pgmap v17472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:56 smithi081 ceph-mon[31453]: pgmap v17472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:56.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:56 smithi153 ceph-mon[36569]: pgmap v17472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:57 smithi081 ceph-mon[31453]: pgmap v17473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:57 smithi153 ceph-mon[36569]: pgmap v17473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:57 smithi104 ceph-mon[36624]: pgmap v17473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:20:59 smithi153 ceph-mon[36569]: pgmap v17474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:20:59 smithi104 ceph-mon[36624]: pgmap v17474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:20:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:20:59 smithi081 ceph-mon[31453]: pgmap v17474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:01 smithi153 ceph-mon[36569]: pgmap v17475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:01 smithi104 ceph-mon[36624]: pgmap v17475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:01 smithi081 ceph-mon[31453]: pgmap v17475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:02.694+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:21:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:02.694+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:21:02.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:02.694+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:21:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:03 smithi153 ceph-mon[36569]: pgmap v17476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:03 smithi104 ceph-mon[36624]: pgmap v17476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:03.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:03 smithi081 ceph-mon[31453]: pgmap v17476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:05 smithi153 ceph-mon[36569]: pgmap v17477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:05.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:05 smithi104 ceph-mon[36624]: pgmap v17477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:05.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:05 smithi081 ceph-mon[31453]: pgmap v17477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:07 smithi153 ceph-mon[36569]: pgmap v17478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:07 smithi104 ceph-mon[36624]: pgmap v17478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:07.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:07 smithi081 ceph-mon[31453]: pgmap v17478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:09 smithi153 ceph-mon[36569]: pgmap v17479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:09 smithi104 ceph-mon[36624]: pgmap v17479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:09.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:09 smithi081 ceph-mon[31453]: pgmap v17479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:11 smithi153 ceph-mon[36569]: pgmap v17480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:11 smithi104 ceph-mon[36624]: pgmap v17480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:11 smithi081 ceph-mon[31453]: pgmap v17480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:13 smithi153 ceph-mon[36569]: pgmap v17481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:13.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:13 smithi104 ceph-mon[36624]: pgmap v17481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:13.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:13 smithi081 ceph-mon[31453]: pgmap v17481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:15 smithi153 ceph-mon[36569]: pgmap v17482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:15 smithi104 ceph-mon[36624]: pgmap v17482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:15.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:15 smithi081 ceph-mon[31453]: pgmap v17482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:17.704+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:21:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:17.704+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:21:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:17.704+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:21:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:17 smithi153 ceph-mon[36569]: pgmap v17483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:17 smithi104 ceph-mon[36624]: pgmap v17483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:17.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:17 smithi081 ceph-mon[31453]: pgmap v17483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:19 smithi153 ceph-mon[36569]: pgmap v17484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:19 smithi104 ceph-mon[36624]: pgmap v17484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:19.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:19 smithi081 ceph-mon[31453]: pgmap v17484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:21 smithi153 ceph-mon[36569]: pgmap v17485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:21.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:21 smithi104 ceph-mon[36624]: pgmap v17485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:21 smithi081 ceph-mon[31453]: pgmap v17485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:23 smithi153 ceph-mon[36569]: pgmap v17486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:23 smithi104 ceph-mon[36624]: pgmap v17486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:23.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:23 smithi081 ceph-mon[31453]: pgmap v17486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:25 smithi153 ceph-mon[36569]: pgmap v17487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:25 smithi104 ceph-mon[36624]: pgmap v17487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:25 smithi081 ceph-mon[31453]: pgmap v17487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:27 smithi153 ceph-mon[36569]: pgmap v17488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:27 smithi104 ceph-mon[36624]: pgmap v17488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:27.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:27 smithi081 ceph-mon[31453]: pgmap v17488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:29 smithi153 ceph-mon[36569]: pgmap v17489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:29.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:29 smithi104 ceph-mon[36624]: pgmap v17489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:29.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:29 smithi081 ceph-mon[31453]: pgmap v17489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:31 smithi153 ceph-mon[36569]: pgmap v17490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:31 smithi104 ceph-mon[36624]: pgmap v17490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:31.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:31 smithi081 ceph-mon[31453]: pgmap v17490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:32.719+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:21:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:32.719+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:21:32.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:32.719+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:21:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:33 smithi153 ceph-mon[36569]: pgmap v17491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:21:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:33 smithi104 ceph-mon[36624]: pgmap v17491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:21:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:33 smithi081 ceph-mon[31453]: pgmap v17491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:33.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:21:34.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:21:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:21:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:34 smithi104 ceph-mon[36624]: pgmap v17492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:34 smithi104 ceph-mon[36624]: pgmap v17493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:34.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:21:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:21:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:21:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:34 smithi081 ceph-mon[31453]: pgmap v17492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:34 smithi081 ceph-mon[31453]: pgmap v17493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:34.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:21:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:21:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:21:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:34 smithi153 ceph-mon[36569]: pgmap v17492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:34 smithi153 ceph-mon[36569]: pgmap v17493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:21:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:37 smithi153 ceph-mon[36569]: pgmap v17494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:37 smithi104 ceph-mon[36624]: pgmap v17494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:37 smithi081 ceph-mon[31453]: pgmap v17494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:39 smithi153 ceph-mon[36569]: pgmap v17495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:39 smithi104 ceph-mon[36624]: pgmap v17495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:39 smithi081 ceph-mon[31453]: pgmap v17495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:41 smithi153 ceph-mon[36569]: pgmap v17496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:41 smithi104 ceph-mon[36624]: pgmap v17496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:41 smithi081 ceph-mon[31453]: pgmap v17496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:43 smithi153 ceph-mon[36569]: pgmap v17497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:43 smithi104 ceph-mon[36624]: pgmap v17497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:43 smithi081 ceph-mon[31453]: pgmap v17497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:45 smithi153 ceph-mon[36569]: pgmap v17498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:45 smithi104 ceph-mon[36624]: pgmap v17498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:45 smithi081 ceph-mon[31453]: pgmap v17498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:47 smithi153 ceph-mon[36569]: pgmap v17499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:47 smithi104 ceph-mon[36624]: pgmap v17499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:47 smithi081 ceph-mon[31453]: pgmap v17499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:47.723+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:21:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:47.723+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:21:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:21:47.723+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:21:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:49 smithi153 ceph-mon[36569]: pgmap v17500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:49 smithi104 ceph-mon[36624]: pgmap v17500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:49 smithi081 ceph-mon[31453]: pgmap v17500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:51 smithi104 ceph-mon[36624]: pgmap v17501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:51 smithi081 ceph-mon[31453]: pgmap v17501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:51.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:51 smithi153 ceph-mon[36569]: pgmap v17501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:53 smithi104 ceph-mon[36624]: pgmap v17502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:53 smithi081 ceph-mon[31453]: pgmap v17502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:53 smithi153 ceph-mon[36569]: pgmap v17502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:55 smithi104 ceph-mon[36624]: pgmap v17503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:55.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:55 smithi081 ceph-mon[31453]: pgmap v17503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:55 smithi153 ceph-mon[36569]: pgmap v17503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:57 smithi104 ceph-mon[36624]: pgmap v17504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:57 smithi081 ceph-mon[31453]: pgmap v17504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:57 smithi153 ceph-mon[36569]: pgmap v17504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:21:59 smithi104 ceph-mon[36624]: pgmap v17505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:21:59 smithi081 ceph-mon[31453]: pgmap v17505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:21:59.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:21:59 smithi153 ceph-mon[36569]: pgmap v17505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:01 smithi081 ceph-mon[31453]: pgmap v17506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:01 smithi153 ceph-mon[36569]: pgmap v17506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:01 smithi104 ceph-mon[36624]: pgmap v17506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:02.740 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:02.738+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:22:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:02.738+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:22:02.741 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:02.738+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:22:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:03 smithi081 ceph-mon[31453]: pgmap v17507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:03 smithi153 ceph-mon[36569]: pgmap v17507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:03.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:03 smithi104 ceph-mon[36624]: pgmap v17507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:05 smithi081 ceph-mon[31453]: pgmap v17508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:05 smithi153 ceph-mon[36569]: pgmap v17508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:05.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:05 smithi104 ceph-mon[36624]: pgmap v17508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:07.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:07 smithi153 ceph-mon[36569]: pgmap v17509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:07 smithi081 ceph-mon[31453]: pgmap v17509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:07 smithi104 ceph-mon[36624]: pgmap v17509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:09 smithi081 ceph-mon[31453]: pgmap v17510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:09 smithi153 ceph-mon[36569]: pgmap v17510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:09 smithi104 ceph-mon[36624]: pgmap v17510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:11 smithi081 ceph-mon[31453]: pgmap v17511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:11.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:11 smithi153 ceph-mon[36569]: pgmap v17511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:11.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:11 smithi104 ceph-mon[36624]: pgmap v17511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:13 smithi081 ceph-mon[31453]: pgmap v17512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:13.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:13 smithi153 ceph-mon[36569]: pgmap v17512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:13.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:13 smithi104 ceph-mon[36624]: pgmap v17512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:15.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:15 smithi081 ceph-mon[31453]: pgmap v17513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:15.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:15 smithi153 ceph-mon[36569]: pgmap v17513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:15 smithi104 ceph-mon[36624]: pgmap v17513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:17 smithi081 ceph-mon[31453]: pgmap v17514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:17 smithi153 ceph-mon[36569]: pgmap v17514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:17.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:17 smithi104 ceph-mon[36624]: pgmap v17514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:17.754+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:22:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:17.754+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:22:17.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:17.754+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:22:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:18.120+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:22:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:18.120+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:22:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:18.121+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 253 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:22:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:18.121+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 253 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:22:19.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:19 smithi104 ceph-mon[36624]: pgmap v17515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:19 smithi081 ceph-mon[31453]: pgmap v17515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:19 smithi153 ceph-mon[36569]: pgmap v17515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:21 smithi081 ceph-mon[31453]: pgmap v17516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:21.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:21 smithi153 ceph-mon[36569]: pgmap v17516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:21.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:21 smithi104 ceph-mon[36624]: pgmap v17516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:23 smithi081 ceph-mon[31453]: pgmap v17517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:23 smithi153 ceph-mon[36569]: pgmap v17517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:23.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:23 smithi104 ceph-mon[36624]: pgmap v17517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:25 smithi081 ceph-mon[31453]: pgmap v17518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:25.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:25 smithi153 ceph-mon[36569]: pgmap v17518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:25.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:25 smithi104 ceph-mon[36624]: pgmap v17518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:27 smithi081 ceph-mon[31453]: pgmap v17519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:27.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:27 smithi153 ceph-mon[36569]: pgmap v17519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:27 smithi104 ceph-mon[36624]: pgmap v17519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:29 smithi081 ceph-mon[31453]: pgmap v17520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:29.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:29 smithi153 ceph-mon[36569]: pgmap v17520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:29.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:29 smithi104 ceph-mon[36624]: pgmap v17520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:31.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:31 smithi153 ceph-mon[36569]: pgmap v17521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:31.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:31 smithi104 ceph-mon[36624]: pgmap v17521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:31.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:31 smithi081 ceph-mon[31453]: pgmap v17521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:32.761 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:32.759+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:22:32.761 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:32.759+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:22:32.761 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:32.759+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:22:33.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:33 smithi153 ceph-mon[36569]: pgmap v17522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:33.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:33 smithi104 ceph-mon[36624]: pgmap v17522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:33 smithi081 ceph-mon[31453]: pgmap v17522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:35 smithi153 ceph-mon[36569]: pgmap v17523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:35.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:22:35.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:35 smithi104 ceph-mon[36624]: pgmap v17523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:35.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:22:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:35 smithi081 ceph-mon[31453]: pgmap v17523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:35.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:22:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:22:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:22:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:22:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:22:36.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:22:36.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:22:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:22:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:22:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:22:37.459 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:37 smithi153 ceph-mon[36569]: pgmap v17524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:37.459 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:37 smithi153 ceph-mon[36569]: pgmap v17525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:37.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:37 smithi104 ceph-mon[36624]: pgmap v17524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:37.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:37 smithi104 ceph-mon[36624]: pgmap v17525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:37 smithi081 ceph-mon[31453]: pgmap v17524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:37 smithi081 ceph-mon[31453]: pgmap v17525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:39.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:39 smithi153 ceph-mon[36569]: pgmap v17526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:39 smithi104 ceph-mon[36624]: pgmap v17526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:39 smithi081 ceph-mon[31453]: pgmap v17526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:41.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:41 smithi153 ceph-mon[36569]: pgmap v17527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:41.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:41 smithi104 ceph-mon[36624]: pgmap v17527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:41.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:41 smithi081 ceph-mon[31453]: pgmap v17527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:43 smithi153 ceph-mon[36569]: pgmap v17528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:43 smithi104 ceph-mon[36624]: pgmap v17528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:43 smithi081 ceph-mon[31453]: pgmap v17528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:45.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:45 smithi153 ceph-mon[36569]: pgmap v17529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:45 smithi104 ceph-mon[36624]: pgmap v17529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:45.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:45 smithi081 ceph-mon[31453]: pgmap v17529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:47 smithi153 ceph-mon[36569]: pgmap v17530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:47 smithi104 ceph-mon[36624]: pgmap v17530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:47.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:47 smithi081 ceph-mon[31453]: pgmap v17530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:47.764+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:22:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:47.764+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:22:47.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:22:47.764+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:22:49.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:49 smithi153 ceph-mon[36569]: pgmap v17531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:49 smithi104 ceph-mon[36624]: pgmap v17531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:49 smithi081 ceph-mon[31453]: pgmap v17531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:51.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:51 smithi153 ceph-mon[36569]: pgmap v17532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:51.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:51 smithi104 ceph-mon[36624]: pgmap v17532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:51 smithi081 ceph-mon[31453]: pgmap v17532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:53.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:53 smithi153 ceph-mon[36569]: pgmap v17533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:53.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:53 smithi104 ceph-mon[36624]: pgmap v17533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:53.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:53 smithi081 ceph-mon[31453]: pgmap v17533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:55 smithi153 ceph-mon[36569]: pgmap v17534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:55.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:55 smithi104 ceph-mon[36624]: pgmap v17534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:55 smithi081 ceph-mon[31453]: pgmap v17534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:57.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:57 smithi153 ceph-mon[36569]: pgmap v17535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:57.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:57 smithi104 ceph-mon[36624]: pgmap v17535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:57 smithi081 ceph-mon[31453]: pgmap v17535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:59.462 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:22:59 smithi081 ceph-mon[31453]: pgmap v17536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:59.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:22:59 smithi153 ceph-mon[36569]: pgmap v17536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:22:59.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:22:59 smithi104 ceph-mon[36624]: pgmap v17536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:01 smithi153 ceph-mon[36569]: pgmap v17537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:01 smithi104 ceph-mon[36624]: pgmap v17537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:01.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:01 smithi081 ceph-mon[31453]: pgmap v17537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:02.776+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:23:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:02.776+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:23:02.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:02.776+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:23:03.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:03 smithi153 ceph-mon[36569]: pgmap v17538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:03.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:03 smithi104 ceph-mon[36624]: pgmap v17538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:03.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:03 smithi081 ceph-mon[31453]: pgmap v17538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:05.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:05 smithi153 ceph-mon[36569]: pgmap v17539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:05.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:05 smithi104 ceph-mon[36624]: pgmap v17539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:05 smithi081 ceph-mon[31453]: pgmap v17539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:07.362 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:07 smithi153 ceph-mon[36569]: pgmap v17540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:07 smithi104 ceph-mon[36624]: pgmap v17540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:07 smithi081 ceph-mon[31453]: pgmap v17540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:09.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:09 smithi153 ceph-mon[36569]: pgmap v17541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:09 smithi104 ceph-mon[36624]: pgmap v17541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:09 smithi081 ceph-mon[31453]: pgmap v17541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:11.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:11 smithi153 ceph-mon[36569]: pgmap v17542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:11 smithi104 ceph-mon[36624]: pgmap v17542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:11.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:11 smithi081 ceph-mon[31453]: pgmap v17542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:13.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:13 smithi153 ceph-mon[36569]: pgmap v17543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:13.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:13 smithi104 ceph-mon[36624]: pgmap v17543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:13.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:13 smithi081 ceph-mon[31453]: pgmap v17543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:15.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:15 smithi081 ceph-mon[31453]: pgmap v17544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:15.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:15 smithi153 ceph-mon[36569]: pgmap v17544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:15.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:15 smithi104 ceph-mon[36624]: pgmap v17544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:17.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:17 smithi153 ceph-mon[36569]: pgmap v17545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:17.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:17 smithi104 ceph-mon[36624]: pgmap v17545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:17.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:17 smithi081 ceph-mon[31453]: pgmap v17545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:17.791+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:23:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:17.791+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:23:17.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:17.791+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:23:19.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:19 smithi153 ceph-mon[36569]: pgmap v17546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:19.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:19 smithi104 ceph-mon[36624]: pgmap v17546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:19 smithi081 ceph-mon[31453]: pgmap v17546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:21.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:21 smithi153 ceph-mon[36569]: pgmap v17547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:21.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:21 smithi104 ceph-mon[36624]: pgmap v17547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:21.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:21 smithi081 ceph-mon[31453]: pgmap v17547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:23.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:23 smithi153 ceph-mon[36569]: pgmap v17548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:23.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:23 smithi104 ceph-mon[36624]: pgmap v17548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:23.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:23 smithi081 ceph-mon[31453]: pgmap v17548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:25.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:25 smithi104 ceph-mon[36624]: pgmap v17549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:25.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:25 smithi081 ceph-mon[31453]: pgmap v17549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:25 smithi153 ceph-mon[36569]: pgmap v17549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:27.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:27 smithi104 ceph-mon[36624]: pgmap v17550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:27.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:27 smithi081 ceph-mon[31453]: pgmap v17550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:27 smithi153 ceph-mon[36569]: pgmap v17550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:29.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:29 smithi104 ceph-mon[36624]: pgmap v17551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:29.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:29 smithi081 ceph-mon[31453]: pgmap v17551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:29 smithi153 ceph-mon[36569]: pgmap v17551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:31.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:31 smithi104 ceph-mon[36624]: pgmap v17552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:31.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:31 smithi081 ceph-mon[31453]: pgmap v17552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:31 smithi153 ceph-mon[36569]: pgmap v17552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:32.807+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:23:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:32.807+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:23:32.809 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:32.807+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:23:33.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:33 smithi104 ceph-mon[36624]: pgmap v17553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:33 smithi081 ceph-mon[31453]: pgmap v17553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:33 smithi153 ceph-mon[36569]: pgmap v17553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:35 smithi081 ceph-mon[31453]: pgmap v17554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:35 smithi153 ceph-mon[36569]: pgmap v17554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:35.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:35 smithi104 ceph-mon[36624]: pgmap v17554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:36.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:23:36.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:23:36.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:23:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: pgmap v17555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:23:37.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:23:37.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:23:37.709 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: pgmap v17555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:37.709 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:23:37.710 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:23:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: pgmap v17555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:23:37.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:23:37.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:23:37.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:23:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:38 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:23:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:38 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:23:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:38 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:23:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:38 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:23:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:38 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:23:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:38 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:23:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:38 smithi081 ceph-mon[31453]: pgmap v17556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:38 smithi081 ceph-mon[31453]: pgmap v17557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:38 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:23:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:38 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:23:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:38 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:23:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:38 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:23:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:38 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:23:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:38 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:23:38.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:38 smithi153 ceph-mon[36569]: pgmap v17556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:38.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:38 smithi153 ceph-mon[36569]: pgmap v17557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:38.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:38 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:23:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:38 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:23:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:38 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:23:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:38 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:23:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:38 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:23:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:38 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:23:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:38 smithi104 ceph-mon[36624]: pgmap v17556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:38.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:38 smithi104 ceph-mon[36624]: pgmap v17557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:39.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:39 smithi081 ceph-mon[31453]: pgmap v17558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:39 smithi153 ceph-mon[36569]: pgmap v17558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:39 smithi104 ceph-mon[36624]: pgmap v17558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:42.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:42 smithi153 ceph-mon[36569]: pgmap v17559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:42 smithi104 ceph-mon[36624]: pgmap v17559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:42.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:42 smithi081 ceph-mon[31453]: pgmap v17559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:44 smithi153 ceph-mon[36569]: pgmap v17560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:44 smithi104 ceph-mon[36624]: pgmap v17560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:44.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:44 smithi081 ceph-mon[31453]: pgmap v17560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:46 smithi153 ceph-mon[36569]: pgmap v17561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:46 smithi104 ceph-mon[36624]: pgmap v17561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:46 smithi081 ceph-mon[31453]: pgmap v17561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:47.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:47.821+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:23:47.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:47.821+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:23:47.824 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:23:47.821+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:23:48.461 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:48 smithi081 ceph-mon[31453]: pgmap v17562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:48.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:48 smithi153 ceph-mon[36569]: pgmap v17562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:48.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:48 smithi104 ceph-mon[36624]: pgmap v17562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:50.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:50 smithi153 ceph-mon[36569]: pgmap v17563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:50.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:50 smithi104 ceph-mon[36624]: pgmap v17563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:50 smithi081 ceph-mon[31453]: pgmap v17563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:52.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:52 smithi153 ceph-mon[36569]: pgmap v17564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:52.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:52 smithi104 ceph-mon[36624]: pgmap v17564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:52 smithi081 ceph-mon[31453]: pgmap v17564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:54.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:54 smithi153 ceph-mon[36569]: pgmap v17565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:54.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:54 smithi104 ceph-mon[36624]: pgmap v17565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:54.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:54 smithi081 ceph-mon[31453]: pgmap v17565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:56 smithi153 ceph-mon[36569]: pgmap v17566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:56.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:56 smithi104 ceph-mon[36624]: pgmap v17566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:56.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:56 smithi081 ceph-mon[31453]: pgmap v17566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:23:58 smithi153 ceph-mon[36569]: pgmap v17567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:58.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:23:58 smithi104 ceph-mon[36624]: pgmap v17567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:23:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:23:58 smithi081 ceph-mon[31453]: pgmap v17567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:00 smithi153 ceph-mon[36569]: pgmap v17568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:00.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:00 smithi104 ceph-mon[36624]: pgmap v17568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:00.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:00 smithi081 ceph-mon[31453]: pgmap v17568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:02.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:02 smithi153 ceph-mon[36569]: pgmap v17569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:02.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:02 smithi104 ceph-mon[36624]: pgmap v17569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:02.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:02 smithi081 ceph-mon[31453]: pgmap v17569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:02.838+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:24:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:02.838+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:24:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:02.838+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:24:04.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:04 smithi153 ceph-mon[36569]: pgmap v17570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:04.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:04 smithi104 ceph-mon[36624]: pgmap v17570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:04.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:04 smithi081 ceph-mon[31453]: pgmap v17570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:06.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:06 smithi153 ceph-mon[36569]: pgmap v17571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:06.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:06 smithi104 ceph-mon[36624]: pgmap v17571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:06.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:06 smithi081 ceph-mon[31453]: pgmap v17571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:08.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:08 smithi153 ceph-mon[36569]: pgmap v17572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:08.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:08 smithi104 ceph-mon[36624]: pgmap v17572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:08.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:08 smithi081 ceph-mon[31453]: pgmap v17572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:10.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:10 smithi153 ceph-mon[36569]: pgmap v17573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:10.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:10 smithi104 ceph-mon[36624]: pgmap v17573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:10.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:10 smithi081 ceph-mon[31453]: pgmap v17573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:12.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:12 smithi153 ceph-mon[36569]: pgmap v17574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:12.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:12 smithi104 ceph-mon[36624]: pgmap v17574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:12.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:12 smithi081 ceph-mon[31453]: pgmap v17574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:14.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:14 smithi153 ceph-mon[36569]: pgmap v17575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:14.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:14 smithi104 ceph-mon[36624]: pgmap v17575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:14.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:14 smithi081 ceph-mon[31453]: pgmap v17575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:16.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:16 smithi153 ceph-mon[36569]: pgmap v17576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:16.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:16 smithi104 ceph-mon[36624]: pgmap v17576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:16.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:16 smithi081 ceph-mon[31453]: pgmap v17576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:17.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:17.848+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:24:17.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:17.848+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:24:17.851 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:17.848+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:24:18.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:18 smithi153 ceph-mon[36569]: pgmap v17577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:18.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:18 smithi104 ceph-mon[36624]: pgmap v17577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:18 smithi081 ceph-mon[31453]: pgmap v17577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:20.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:20 smithi153 ceph-mon[36569]: pgmap v17578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:20.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:20 smithi104 ceph-mon[36624]: pgmap v17578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:20.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:20 smithi081 ceph-mon[31453]: pgmap v17578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:22.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:22 smithi153 ceph-mon[36569]: pgmap v17579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:22.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:22 smithi104 ceph-mon[36624]: pgmap v17579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:22.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:22 smithi081 ceph-mon[31453]: pgmap v17579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:24.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:24 smithi153 ceph-mon[36569]: pgmap v17580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:24 smithi104 ceph-mon[36624]: pgmap v17580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:24.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:24 smithi081 ceph-mon[31453]: pgmap v17580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:26.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:26 smithi153 ceph-mon[36569]: pgmap v17581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:26.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:26 smithi104 ceph-mon[36624]: pgmap v17581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:26.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:26 smithi081 ceph-mon[31453]: pgmap v17581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:28.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:28 smithi153 ceph-mon[36569]: pgmap v17582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:28.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:28 smithi104 ceph-mon[36624]: pgmap v17582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:28.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:28 smithi081 ceph-mon[31453]: pgmap v17582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:30.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:30 smithi153 ceph-mon[36569]: pgmap v17583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:30.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:30 smithi104 ceph-mon[36624]: pgmap v17583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:30.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:30 smithi081 ceph-mon[31453]: pgmap v17583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:32.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:32 smithi153 ceph-mon[36569]: pgmap v17584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:32.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:32 smithi104 ceph-mon[36624]: pgmap v17584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:32.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:32 smithi081 ceph-mon[31453]: pgmap v17584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:32.864+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:24:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:32.864+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:24:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:32.864+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:24:34.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:34 smithi153 ceph-mon[36569]: pgmap v17585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:34.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:34 smithi104 ceph-mon[36624]: pgmap v17585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:34.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:34 smithi081 ceph-mon[31453]: pgmap v17585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:36.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:36 smithi153 ceph-mon[36569]: pgmap v17586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:36.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:36 smithi104 ceph-mon[36624]: pgmap v17586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:36 smithi081 ceph-mon[31453]: pgmap v17586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:38.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:38 smithi104 ceph-mon[36624]: pgmap v17587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:38.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:24:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:38 smithi081 ceph-mon[31453]: pgmap v17587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:24:38.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:38 smithi153 ceph-mon[36569]: pgmap v17587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:38.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:24:39.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:24:39.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:24:39.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:24:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:24:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:24:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:24:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:24:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:24:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:24:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:40 smithi104 ceph-mon[36624]: pgmap v17588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:40 smithi104 ceph-mon[36624]: pgmap v17589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:40.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:40 smithi081 ceph-mon[31453]: pgmap v17588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:40.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:40 smithi081 ceph-mon[31453]: pgmap v17589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:40.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:40 smithi153 ceph-mon[36569]: pgmap v17588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:40.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:40 smithi153 ceph-mon[36569]: pgmap v17589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:42 smithi104 ceph-mon[36624]: pgmap v17590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:42 smithi081 ceph-mon[31453]: pgmap v17590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:42 smithi153 ceph-mon[36569]: pgmap v17590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:44.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:44 smithi081 ceph-mon[31453]: pgmap v17591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:44.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:44 smithi153 ceph-mon[36569]: pgmap v17591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:44.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:44 smithi104 ceph-mon[36624]: pgmap v17591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:46 smithi081 ceph-mon[31453]: pgmap v17592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:46 smithi153 ceph-mon[36569]: pgmap v17592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:46 smithi104 ceph-mon[36624]: pgmap v17592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:47.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:47.876+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:24:47.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:47.876+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:24:47.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:24:47.876+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:24:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:48 smithi081 ceph-mon[31453]: pgmap v17593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:48 smithi153 ceph-mon[36569]: pgmap v17593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:48 smithi104 ceph-mon[36624]: pgmap v17593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:50.636 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:50 smithi104 ceph-mon[36624]: pgmap v17594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:50.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:50 smithi081 ceph-mon[31453]: pgmap v17594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:50.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:50 smithi153 ceph-mon[36569]: pgmap v17594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:52 smithi081 ceph-mon[31453]: pgmap v17595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:52 smithi153 ceph-mon[36569]: pgmap v17595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:52.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:52 smithi104 ceph-mon[36624]: pgmap v17595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:54.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:54 smithi081 ceph-mon[31453]: pgmap v17596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:54.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:54 smithi153 ceph-mon[36569]: pgmap v17596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:54.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:54 smithi104 ceph-mon[36624]: pgmap v17596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:55 smithi153 ceph-mon[36569]: pgmap v17597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:55 smithi104 ceph-mon[36624]: pgmap v17597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:55 smithi081 ceph-mon[31453]: pgmap v17597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:57 smithi153 ceph-mon[36569]: pgmap v17598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:57 smithi104 ceph-mon[36624]: pgmap v17598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:24:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:57 smithi081 ceph-mon[31453]: pgmap v17598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:24:59 smithi153 ceph-mon[36569]: pgmap v17599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:24:59 smithi104 ceph-mon[36624]: pgmap v17599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:24:59 smithi081 ceph-mon[31453]: pgmap v17599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:01 smithi153 ceph-mon[36569]: pgmap v17600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:02.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:01 smithi104 ceph-mon[36624]: pgmap v17600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:02.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:01 smithi081 ceph-mon[31453]: pgmap v17600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:02.881+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:25:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:02.881+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:25:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:02.881+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:25:04.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:03 smithi153 ceph-mon[36569]: pgmap v17601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:03 smithi104 ceph-mon[36624]: pgmap v17601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:03 smithi081 ceph-mon[31453]: pgmap v17601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:06.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:05 smithi153 ceph-mon[36569]: pgmap v17602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:05 smithi104 ceph-mon[36624]: pgmap v17602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:05 smithi081 ceph-mon[31453]: pgmap v17602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:08.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:07 smithi153 ceph-mon[36569]: pgmap v17603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:07 smithi104 ceph-mon[36624]: pgmap v17603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:07 smithi081 ceph-mon[31453]: pgmap v17603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:10.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:09 smithi153 ceph-mon[36569]: pgmap v17604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:09 smithi104 ceph-mon[36624]: pgmap v17604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:09 smithi081 ceph-mon[31453]: pgmap v17604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:12.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:11 smithi153 ceph-mon[36569]: pgmap v17605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:12.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:11 smithi104 ceph-mon[36624]: pgmap v17605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:12.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:11 smithi081 ceph-mon[31453]: pgmap v17605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:14.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:13 smithi153 ceph-mon[36569]: pgmap v17606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:14.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:13 smithi104 ceph-mon[36624]: pgmap v17606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:13 smithi081 ceph-mon[31453]: pgmap v17606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:16.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:15 smithi153 ceph-mon[36569]: pgmap v17607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:16.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:15 smithi104 ceph-mon[36624]: pgmap v17607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:15 smithi081 ceph-mon[31453]: pgmap v17607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:17.897+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:25:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:17.897+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:25:17.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:17.897+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:25:18.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:17 smithi153 ceph-mon[36569]: pgmap v17608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:18.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:17 smithi104 ceph-mon[36624]: pgmap v17608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:18.150+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:25:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:18.150+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:25:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:18.151+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 254 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:25:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:18.151+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 254 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:25:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:17 smithi081 ceph-mon[31453]: pgmap v17608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:20.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:19 smithi153 ceph-mon[36569]: pgmap v17609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:20.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:19 smithi104 ceph-mon[36624]: pgmap v17609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:19 smithi081 ceph-mon[31453]: pgmap v17609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:22.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:21 smithi153 ceph-mon[36569]: pgmap v17610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:22.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:21 smithi104 ceph-mon[36624]: pgmap v17610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:21 smithi081 ceph-mon[31453]: pgmap v17610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:24.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:23 smithi104 ceph-mon[36624]: pgmap v17611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:23 smithi081 ceph-mon[31453]: pgmap v17611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:23 smithi153 ceph-mon[36569]: pgmap v17611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:26.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:25 smithi104 ceph-mon[36624]: pgmap v17612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:25 smithi081 ceph-mon[31453]: pgmap v17612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:25 smithi153 ceph-mon[36569]: pgmap v17612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:28.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:27 smithi104 ceph-mon[36624]: pgmap v17613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:27 smithi081 ceph-mon[31453]: pgmap v17613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:27 smithi153 ceph-mon[36569]: pgmap v17613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:30.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:29 smithi104 ceph-mon[36624]: pgmap v17614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:29 smithi081 ceph-mon[31453]: pgmap v17614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:29 smithi153 ceph-mon[36569]: pgmap v17614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:32.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:31 smithi104 ceph-mon[36624]: pgmap v17615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:31 smithi081 ceph-mon[31453]: pgmap v17615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:31 smithi153 ceph-mon[36569]: pgmap v17615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:32.902+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:25:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:32.902+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:25:32.905 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:32.902+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:25:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:33 smithi081 ceph-mon[31453]: pgmap v17616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:33 smithi153 ceph-mon[36569]: pgmap v17616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:33 smithi104 ceph-mon[36624]: pgmap v17616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:36.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:35 smithi081 ceph-mon[31453]: pgmap v17617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:35 smithi153 ceph-mon[36569]: pgmap v17617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:35 smithi104 ceph-mon[36624]: pgmap v17617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:38.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:37 smithi081 ceph-mon[31453]: pgmap v17618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:37 smithi153 ceph-mon[36569]: pgmap v17618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:37 smithi104 ceph-mon[36624]: pgmap v17618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:40.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:39 smithi153 ceph-mon[36569]: pgmap v17619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:40.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:25:40.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:39 smithi081 ceph-mon[31453]: pgmap v17619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:40.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:25:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:39 smithi104 ceph-mon[36624]: pgmap v17619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:25:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:25:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:25:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:40 smithi081 ceph-mon[31453]: pgmap v17620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:40 smithi081 ceph-mon[31453]: pgmap v17621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:25:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:25:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:25:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:40 smithi153 ceph-mon[36569]: pgmap v17620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:40 smithi153 ceph-mon[36569]: pgmap v17621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:25:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:25:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:25:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:40 smithi104 ceph-mon[36624]: pgmap v17620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:40 smithi104 ceph-mon[36624]: pgmap v17621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:25:43.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:43 smithi153 ceph-mon[36569]: pgmap v17622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:43.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:43 smithi104 ceph-mon[36624]: pgmap v17622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:43.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:43 smithi081 ceph-mon[31453]: pgmap v17622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:45.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:45 smithi153 ceph-mon[36569]: pgmap v17623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:45.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:45 smithi104 ceph-mon[36624]: pgmap v17623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:45 smithi081 ceph-mon[31453]: pgmap v17623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:47 smithi153 ceph-mon[36569]: pgmap v17624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:47 smithi104 ceph-mon[36624]: pgmap v17624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:47.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:47 smithi081 ceph-mon[31453]: pgmap v17624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:47.920 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:47.919+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:25:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:47.919+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:25:47.921 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:25:47.919+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:25:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:49 smithi104 ceph-mon[36624]: pgmap v17625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:49 smithi081 ceph-mon[31453]: pgmap v17625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:49 smithi153 ceph-mon[36569]: pgmap v17625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:51.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:51 smithi104 ceph-mon[36624]: pgmap v17626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:51 smithi081 ceph-mon[31453]: pgmap v17626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:51 smithi153 ceph-mon[36569]: pgmap v17626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:53.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:53 smithi104 ceph-mon[36624]: pgmap v17627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:53.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:53 smithi081 ceph-mon[31453]: pgmap v17627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:53 smithi153 ceph-mon[36569]: pgmap v17627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:55.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:55 smithi104 ceph-mon[36624]: pgmap v17628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:55 smithi081 ceph-mon[31453]: pgmap v17628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:55 smithi153 ceph-mon[36569]: pgmap v17628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:57.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:57 smithi104 ceph-mon[36624]: pgmap v17629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:57 smithi081 ceph-mon[31453]: pgmap v17629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:57 smithi153 ceph-mon[36569]: pgmap v17629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:59.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:25:59 smithi081 ceph-mon[31453]: pgmap v17630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:25:59 smithi153 ceph-mon[36569]: pgmap v17630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:25:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:25:59 smithi104 ceph-mon[36624]: pgmap v17630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:01.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:01 smithi081 ceph-mon[31453]: pgmap v17631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:01 smithi153 ceph-mon[36569]: pgmap v17631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:01.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:01 smithi104 ceph-mon[36624]: pgmap v17631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:02.931+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:26:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:02.932+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:26:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:02.932+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:26:03.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:03 smithi081 ceph-mon[31453]: pgmap v17632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:03 smithi153 ceph-mon[36569]: pgmap v17632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:03 smithi104 ceph-mon[36624]: pgmap v17632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:05 smithi081 ceph-mon[31453]: pgmap v17633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:05 smithi153 ceph-mon[36569]: pgmap v17633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:05 smithi104 ceph-mon[36624]: pgmap v17633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:07 smithi081 ceph-mon[31453]: pgmap v17634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:07 smithi153 ceph-mon[36569]: pgmap v17634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:07.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:07 smithi104 ceph-mon[36624]: pgmap v17634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:09 smithi081 ceph-mon[31453]: pgmap v17635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:09 smithi153 ceph-mon[36569]: pgmap v17635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:09.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:09 smithi104 ceph-mon[36624]: pgmap v17635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:11 smithi081 ceph-mon[31453]: pgmap v17636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:11 smithi153 ceph-mon[36569]: pgmap v17636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:11 smithi104 ceph-mon[36624]: pgmap v17636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:13.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:13 smithi081 ceph-mon[31453]: pgmap v17637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:13 smithi153 ceph-mon[36569]: pgmap v17637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:13.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:13 smithi104 ceph-mon[36624]: pgmap v17637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:15.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:15 smithi081 ceph-mon[31453]: pgmap v17638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:15 smithi153 ceph-mon[36569]: pgmap v17638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:15 smithi104 ceph-mon[36624]: pgmap v17638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:17.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:17 smithi081 ceph-mon[31453]: pgmap v17639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:17 smithi153 ceph-mon[36569]: pgmap v17639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:17.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:17 smithi104 ceph-mon[36624]: pgmap v17639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:17.947+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:26:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:17.947+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:26:17.949 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:17.947+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:26:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:18 smithi153 ceph-mon[36569]: pgmap v17640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:18.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:18 smithi104 ceph-mon[36624]: pgmap v17640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:18.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:18 smithi081 ceph-mon[31453]: pgmap v17640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:21.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:21 smithi104 ceph-mon[36624]: pgmap v17641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:21.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:21 smithi081 ceph-mon[31453]: pgmap v17641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:21 smithi153 ceph-mon[36569]: pgmap v17641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:23.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:23 smithi104 ceph-mon[36624]: pgmap v17642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:23.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:23 smithi081 ceph-mon[31453]: pgmap v17642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:23 smithi153 ceph-mon[36569]: pgmap v17642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:25.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:25 smithi104 ceph-mon[36624]: pgmap v17643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:25.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:25 smithi081 ceph-mon[31453]: pgmap v17643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:25 smithi153 ceph-mon[36569]: pgmap v17643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:27.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:27 smithi104 ceph-mon[36624]: pgmap v17644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:27.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:27 smithi081 ceph-mon[31453]: pgmap v17644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:27.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:27 smithi153 ceph-mon[36569]: pgmap v17644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:29.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:29 smithi081 ceph-mon[31453]: pgmap v17645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:29 smithi153 ceph-mon[36569]: pgmap v17645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:29 smithi104 ceph-mon[36624]: pgmap v17645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:31.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:31 smithi081 ceph-mon[31453]: pgmap v17646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:31 smithi153 ceph-mon[36569]: pgmap v17646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:31 smithi104 ceph-mon[36624]: pgmap v17646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:32.955+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:26:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:32.955+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:26:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:32.955+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:26:33.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:33 smithi081 ceph-mon[31453]: pgmap v17647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:33 smithi153 ceph-mon[36569]: pgmap v17647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:33.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:33 smithi104 ceph-mon[36624]: pgmap v17647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:35 smithi081 ceph-mon[31453]: pgmap v17648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:35 smithi153 ceph-mon[36569]: pgmap v17648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:35 smithi104 ceph-mon[36624]: pgmap v17648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:37 smithi081 ceph-mon[31453]: pgmap v17649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:37 smithi153 ceph-mon[36569]: pgmap v17649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:37.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:37 smithi104 ceph-mon[36624]: pgmap v17649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:39 smithi081 ceph-mon[31453]: pgmap v17650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:39 smithi153 ceph-mon[36569]: pgmap v17650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:39.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:39 smithi104 ceph-mon[36624]: pgmap v17650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:41.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:41 smithi081 ceph-mon[31453]: pgmap v17651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:41.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:26:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:41 smithi153 ceph-mon[36569]: pgmap v17651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:41.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:26:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:41 smithi104 ceph-mon[36624]: pgmap v17651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:41.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:26:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:26:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:26:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:26:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:26:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:26:42.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:26:42.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:26:42.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:26:42.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:26:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:43 smithi081 ceph-mon[31453]: pgmap v17652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:43 smithi081 ceph-mon[31453]: pgmap v17653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:43 smithi153 ceph-mon[36569]: pgmap v17652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:43 smithi153 ceph-mon[36569]: pgmap v17653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:43 smithi104 ceph-mon[36624]: pgmap v17652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:43 smithi104 ceph-mon[36624]: pgmap v17653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:45.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:45 smithi081 ceph-mon[31453]: pgmap v17654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:45 smithi153 ceph-mon[36569]: pgmap v17654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:45 smithi104 ceph-mon[36624]: pgmap v17654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:46 smithi153 ceph-mon[36569]: pgmap v17655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:46.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:46 smithi104 ceph-mon[36624]: pgmap v17655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:46.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:46 smithi081 ceph-mon[31453]: pgmap v17655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:47.970+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:26:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:47.970+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:26:47.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:26:47.970+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:26:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:48 smithi081 ceph-mon[31453]: pgmap v17656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:48 smithi153 ceph-mon[36569]: pgmap v17656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:48 smithi104 ceph-mon[36624]: pgmap v17656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:51.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:50 smithi104 ceph-mon[36624]: pgmap v17657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:50 smithi081 ceph-mon[31453]: pgmap v17657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:50 smithi153 ceph-mon[36569]: pgmap v17657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:52 smithi081 ceph-mon[31453]: pgmap v17658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:52 smithi153 ceph-mon[36569]: pgmap v17658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:52 smithi104 ceph-mon[36624]: pgmap v17658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:54 smithi081 ceph-mon[31453]: pgmap v17659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:54 smithi153 ceph-mon[36569]: pgmap v17659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:54 smithi104 ceph-mon[36624]: pgmap v17659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:56 smithi081 ceph-mon[31453]: pgmap v17660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:56 smithi153 ceph-mon[36569]: pgmap v17660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:56 smithi104 ceph-mon[36624]: pgmap v17660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:26:58 smithi081 ceph-mon[31453]: pgmap v17661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:26:58 smithi153 ceph-mon[36569]: pgmap v17661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:26:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:26:58 smithi104 ceph-mon[36624]: pgmap v17661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:01.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:00 smithi081 ceph-mon[31453]: pgmap v17662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:00 smithi153 ceph-mon[36569]: pgmap v17662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:00 smithi104 ceph-mon[36624]: pgmap v17662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:02.985+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:27:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:02.985+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:27:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:02.985+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:27:03.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:02 smithi081 ceph-mon[31453]: pgmap v17663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:03.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:02 smithi153 ceph-mon[36569]: pgmap v17663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:02 smithi104 ceph-mon[36624]: pgmap v17663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:05.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:04 smithi081 ceph-mon[31453]: pgmap v17664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:04 smithi153 ceph-mon[36569]: pgmap v17664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:04 smithi104 ceph-mon[36624]: pgmap v17664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:07.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:06 smithi081 ceph-mon[31453]: pgmap v17665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:06 smithi153 ceph-mon[36569]: pgmap v17665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:06 smithi104 ceph-mon[36624]: pgmap v17665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:08 smithi153 ceph-mon[36569]: pgmap v17666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:08 smithi104 ceph-mon[36624]: pgmap v17666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:08 smithi081 ceph-mon[31453]: pgmap v17666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:10 smithi153 ceph-mon[36569]: pgmap v17667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:10 smithi104 ceph-mon[36624]: pgmap v17667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:10 smithi081 ceph-mon[31453]: pgmap v17667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:12 smithi153 ceph-mon[36569]: pgmap v17668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:12 smithi104 ceph-mon[36624]: pgmap v17668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:12 smithi081 ceph-mon[31453]: pgmap v17668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:14 smithi153 ceph-mon[36569]: pgmap v17669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:14 smithi104 ceph-mon[36624]: pgmap v17669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:15.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:14 smithi081 ceph-mon[31453]: pgmap v17669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:16 smithi153 ceph-mon[36569]: pgmap v17670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:16 smithi104 ceph-mon[36624]: pgmap v17670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:16 smithi081 ceph-mon[31453]: pgmap v17670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:17.997+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:27:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:17.997+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:27:18.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:17.997+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:27:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:18 smithi153 ceph-mon[36569]: pgmap v17671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:18 smithi104 ceph-mon[36624]: pgmap v17671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:19.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:18 smithi081 ceph-mon[31453]: pgmap v17671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:20 smithi153 ceph-mon[36569]: pgmap v17672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:20 smithi104 ceph-mon[36624]: pgmap v17672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:20 smithi081 ceph-mon[31453]: pgmap v17672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:22 smithi081 ceph-mon[31453]: pgmap v17673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:23.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:22 smithi153 ceph-mon[36569]: pgmap v17673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:22 smithi104 ceph-mon[36624]: pgmap v17673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:24 smithi153 ceph-mon[36569]: pgmap v17674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:24 smithi104 ceph-mon[36624]: pgmap v17674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:24 smithi081 ceph-mon[31453]: pgmap v17674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:26 smithi153 ceph-mon[36569]: pgmap v17675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:26 smithi104 ceph-mon[36624]: pgmap v17675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:26 smithi081 ceph-mon[31453]: pgmap v17675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:28 smithi153 ceph-mon[36569]: pgmap v17676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:28 smithi104 ceph-mon[36624]: pgmap v17676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:28 smithi081 ceph-mon[31453]: pgmap v17676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:30 smithi153 ceph-mon[36569]: pgmap v17677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:30 smithi104 ceph-mon[36624]: pgmap v17677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:30 smithi081 ceph-mon[31453]: pgmap v17677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:33.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:33.006+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:27:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:33.006+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:27:33.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:33.006+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:27:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:32 smithi153 ceph-mon[36569]: pgmap v17678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:33.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:32 smithi104 ceph-mon[36624]: pgmap v17678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:32 smithi081 ceph-mon[31453]: pgmap v17678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:34 smithi153 ceph-mon[36569]: pgmap v17679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:34 smithi104 ceph-mon[36624]: pgmap v17679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:35.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:34 smithi081 ceph-mon[31453]: pgmap v17679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:36 smithi153 ceph-mon[36569]: pgmap v17680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:36 smithi104 ceph-mon[36624]: pgmap v17680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:36 smithi081 ceph-mon[31453]: pgmap v17680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:38 smithi153 ceph-mon[36569]: pgmap v17681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:38 smithi104 ceph-mon[36624]: pgmap v17681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:39.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:38 smithi081 ceph-mon[31453]: pgmap v17681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:40 smithi153 ceph-mon[36569]: pgmap v17682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:40 smithi104 ceph-mon[36624]: pgmap v17682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:41.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:40 smithi081 ceph-mon[31453]: pgmap v17682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:43.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:43 smithi153 ceph-mon[36569]: pgmap v17683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:43.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:27:43.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.194 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.195 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:43 smithi104 ceph-mon[36624]: pgmap v17683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:27:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:42 smithi081 ceph-mon[31453]: pgmap v17683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:27:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:45 smithi153 ceph-mon[36569]: pgmap v17684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:45.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:45 smithi104 ceph-mon[36624]: pgmap v17684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:45 smithi081 ceph-mon[31453]: pgmap v17684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:45.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:45.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:46.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:46.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:46.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:47.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:47 smithi081 ceph-mon[31453]: pgmap v17685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:47.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:47 smithi153 ceph-mon[36569]: pgmap v17685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:47.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:47 smithi104 ceph-mon[36624]: pgmap v17685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:48.021+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:27:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:48.021+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:27:48.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:27:48.021+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:27:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:48 smithi153 ceph-mon[36569]: pgmap v17686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:48.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:48 smithi104 ceph-mon[36624]: pgmap v17686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:48 smithi081 ceph-mon[31453]: pgmap v17686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:27:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:27:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:49 smithi153 ceph-mon[36569]: pgmap v17687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:49 smithi153 ceph-mon[36569]: pgmap v17688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:49.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:27:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:27:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:49 smithi104 ceph-mon[36624]: pgmap v17687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:49 smithi104 ceph-mon[36624]: pgmap v17688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:49.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:27:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:27:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:49 smithi081 ceph-mon[31453]: pgmap v17687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:49 smithi081 ceph-mon[31453]: pgmap v17688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:49.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:27:52.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:51 smithi153 ceph-mon[36569]: pgmap v17689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:51 smithi104 ceph-mon[36624]: pgmap v17689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:52.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:51 smithi081 ceph-mon[31453]: pgmap v17689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:54.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:53 smithi153 ceph-mon[36569]: pgmap v17690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:53 smithi104 ceph-mon[36624]: pgmap v17690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:54.121 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:53 smithi081 ceph-mon[31453]: pgmap v17690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:56.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:55 smithi153 ceph-mon[36569]: pgmap v17691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:55 smithi104 ceph-mon[36624]: pgmap v17691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:55 smithi081 ceph-mon[31453]: pgmap v17691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:58.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:57 smithi153 ceph-mon[36569]: pgmap v17692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:58.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:57 smithi104 ceph-mon[36624]: pgmap v17692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:27:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:57 smithi081 ceph-mon[31453]: pgmap v17692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:00.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:27:59 smithi153 ceph-mon[36569]: pgmap v17693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:00.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:27:59 smithi104 ceph-mon[36624]: pgmap v17693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:27:59 smithi081 ceph-mon[31453]: pgmap v17693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:01 smithi104 ceph-mon[36624]: pgmap v17694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:01 smithi081 ceph-mon[31453]: pgmap v17694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:01 smithi153 ceph-mon[36569]: pgmap v17694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:03.031+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:28:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:03.031+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:28:03.033 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:03.031+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:28:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:03 smithi104 ceph-mon[36624]: pgmap v17695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:03 smithi081 ceph-mon[31453]: pgmap v17695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:03 smithi153 ceph-mon[36569]: pgmap v17695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:06.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:05 smithi104 ceph-mon[36624]: pgmap v17696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:05 smithi081 ceph-mon[31453]: pgmap v17696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:05 smithi153 ceph-mon[36569]: pgmap v17696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:08.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:07 smithi104 ceph-mon[36624]: pgmap v17697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:07 smithi081 ceph-mon[31453]: pgmap v17697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:07 smithi153 ceph-mon[36569]: pgmap v17697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:09 smithi081 ceph-mon[31453]: pgmap v17698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:09 smithi153 ceph-mon[36569]: pgmap v17698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:09 smithi104 ceph-mon[36624]: pgmap v17698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:12.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:11 smithi153 ceph-mon[36569]: pgmap v17699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:12.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:11 smithi081 ceph-mon[31453]: pgmap v17699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:11 smithi104 ceph-mon[36624]: pgmap v17699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:13 smithi081 ceph-mon[31453]: pgmap v17700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:13 smithi153 ceph-mon[36569]: pgmap v17700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:13 smithi104 ceph-mon[36624]: pgmap v17700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:15 smithi081 ceph-mon[31453]: pgmap v17701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:15 smithi153 ceph-mon[36569]: pgmap v17701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:15 smithi104 ceph-mon[36624]: pgmap v17701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:18.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:18.047+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:28:18.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:18.047+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:28:18.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:18.047+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:28:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:17 smithi081 ceph-mon[31453]: pgmap v17702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:18.179+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:18.179+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:18.180+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 255 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:28:18.181 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:18.180+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 255 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:28:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:17 smithi153 ceph-mon[36569]: pgmap v17702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:17 smithi104 ceph-mon[36624]: pgmap v17702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:20.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:19 smithi081 ceph-mon[31453]: pgmap v17703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:19 smithi153 ceph-mon[36569]: pgmap v17703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:19 smithi104 ceph-mon[36624]: pgmap v17703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:22.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:21 smithi104 ceph-mon[36624]: pgmap v17704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:21 smithi081 ceph-mon[31453]: pgmap v17704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:21 smithi153 ceph-mon[36569]: pgmap v17704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:23 smithi081 ceph-mon[31453]: pgmap v17705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:23 smithi153 ceph-mon[36569]: pgmap v17705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:23 smithi104 ceph-mon[36624]: pgmap v17705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:25 smithi081 ceph-mon[31453]: pgmap v17706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:25 smithi153 ceph-mon[36569]: pgmap v17706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:25 smithi104 ceph-mon[36624]: pgmap v17706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:27 smithi081 ceph-mon[31453]: pgmap v17707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:27 smithi153 ceph-mon[36569]: pgmap v17707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:27 smithi104 ceph-mon[36624]: pgmap v17707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:29 smithi081 ceph-mon[31453]: pgmap v17708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:29 smithi153 ceph-mon[36569]: pgmap v17708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:29 smithi104 ceph-mon[36624]: pgmap v17708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:32.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:31 smithi081 ceph-mon[31453]: pgmap v17709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:31 smithi153 ceph-mon[36569]: pgmap v17709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:31 smithi104 ceph-mon[36624]: pgmap v17709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:33.050+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:33.050+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:28:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:33.050+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:28:34.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:33 smithi081 ceph-mon[31453]: pgmap v17710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:33 smithi153 ceph-mon[36569]: pgmap v17710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:33 smithi104 ceph-mon[36624]: pgmap v17710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:35 smithi153 ceph-mon[36569]: pgmap v17711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:35 smithi104 ceph-mon[36624]: pgmap v17711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:35 smithi081 ceph-mon[31453]: pgmap v17711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:37 smithi153 ceph-mon[36569]: pgmap v17712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:37 smithi104 ceph-mon[36624]: pgmap v17712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:37 smithi081 ceph-mon[31453]: pgmap v17712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:39 smithi153 ceph-mon[36569]: pgmap v17713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:39 smithi104 ceph-mon[36624]: pgmap v17713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:39 smithi081 ceph-mon[31453]: pgmap v17713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:41 smithi153 ceph-mon[36569]: pgmap v17714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:41 smithi104 ceph-mon[36624]: pgmap v17714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:41 smithi081 ceph-mon[31453]: pgmap v17714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:43 smithi153 ceph-mon[36569]: pgmap v17715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:43 smithi104 ceph-mon[36624]: pgmap v17715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:43 smithi081 ceph-mon[31453]: pgmap v17715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:45 smithi153 ceph-mon[36569]: pgmap v17716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:45 smithi104 ceph-mon[36624]: pgmap v17716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:46.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:45 smithi081 ceph-mon[31453]: pgmap v17716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:48.065+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:28:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:48.065+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:28:48.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:28:48.065+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:28:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:47 smithi153 ceph-mon[36569]: pgmap v17717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:47 smithi104 ceph-mon[36624]: pgmap v17717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:47 smithi081 ceph-mon[31453]: pgmap v17717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:49 smithi153 ceph-mon[36569]: pgmap v17718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:28:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:49 smithi104 ceph-mon[36624]: pgmap v17718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:28:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:49 smithi081 ceph-mon[31453]: pgmap v17718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:28:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:28:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:28:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:50 smithi153 ceph-mon[36569]: pgmap v17719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:50 smithi153 ceph-mon[36569]: pgmap v17720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:28:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:28:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:50 smithi104 ceph-mon[36624]: pgmap v17719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:50 smithi104 ceph-mon[36624]: pgmap v17720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:28:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:28:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:50 smithi081 ceph-mon[31453]: pgmap v17719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:50 smithi081 ceph-mon[31453]: pgmap v17720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:28:53.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:53 smithi081 ceph-mon[31453]: pgmap v17721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:53 smithi153 ceph-mon[36569]: pgmap v17721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:53 smithi104 ceph-mon[36624]: pgmap v17721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:55.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:55 smithi104 ceph-mon[36624]: pgmap v17722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:55.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:55 smithi081 ceph-mon[31453]: pgmap v17722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:55 smithi153 ceph-mon[36569]: pgmap v17722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:57.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:57 smithi081 ceph-mon[31453]: pgmap v17723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:57.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:57 smithi153 ceph-mon[36569]: pgmap v17723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:57 smithi104 ceph-mon[36624]: pgmap v17723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:59.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:28:59 smithi081 ceph-mon[31453]: pgmap v17724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:28:59 smithi153 ceph-mon[36569]: pgmap v17724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:28:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:28:59 smithi104 ceph-mon[36624]: pgmap v17724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:00 smithi081 ceph-mon[31453]: pgmap v17725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:00 smithi153 ceph-mon[36569]: pgmap v17725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:01.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:00 smithi104 ceph-mon[36624]: pgmap v17725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:03.075+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:29:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:03.075+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:29:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:03.075+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:29:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:03.187+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:29:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:03.187+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:29:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:03.187+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 256 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:29:03.189 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:03.187+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 256 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:29:03.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:03 smithi081 ceph-mon[31453]: pgmap v17726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:03.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:03 smithi153 ceph-mon[36569]: pgmap v17726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:03.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:03 smithi104 ceph-mon[36624]: pgmap v17726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:05.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:05 smithi104 ceph-mon[36624]: pgmap v17727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:05.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:05 smithi081 ceph-mon[31453]: pgmap v17727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:05.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:05 smithi153 ceph-mon[36569]: pgmap v17727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:07.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:07 smithi081 ceph-mon[31453]: pgmap v17728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:07.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:07 smithi153 ceph-mon[36569]: pgmap v17728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:07.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:07 smithi104 ceph-mon[36624]: pgmap v17728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:09 smithi081 ceph-mon[31453]: pgmap v17729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:09.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:09 smithi153 ceph-mon[36569]: pgmap v17729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:09 smithi104 ceph-mon[36624]: pgmap v17729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:11 smithi081 ceph-mon[31453]: pgmap v17730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:11.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:11 smithi153 ceph-mon[36569]: pgmap v17730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:11.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:11 smithi104 ceph-mon[36624]: pgmap v17730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:13.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:13 smithi081 ceph-mon[31453]: pgmap v17731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:13.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:13 smithi153 ceph-mon[36569]: pgmap v17731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:13.951 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:13 smithi104 ceph-mon[36624]: pgmap v17731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:15.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:15 smithi081 ceph-mon[31453]: pgmap v17732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:15.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:15 smithi153 ceph-mon[36569]: pgmap v17732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:15.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:15 smithi104 ceph-mon[36624]: pgmap v17732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:17.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:17 smithi081 ceph-mon[31453]: pgmap v17733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:17.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:17 smithi153 ceph-mon[36569]: pgmap v17733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:17.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:17 smithi104 ceph-mon[36624]: pgmap v17733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:18.088+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:29:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:18.088+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:29:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:18.088+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:29:19.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:19 smithi081 ceph-mon[31453]: pgmap v17734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:19.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:19 smithi153 ceph-mon[36569]: pgmap v17734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:19.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:19 smithi104 ceph-mon[36624]: pgmap v17734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:21.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:21 smithi153 ceph-mon[36569]: pgmap v17735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:21.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:21 smithi104 ceph-mon[36624]: pgmap v17735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:21.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:21 smithi081 ceph-mon[31453]: pgmap v17735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:23.710 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:23 smithi081 ceph-mon[31453]: pgmap v17736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:23.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:23 smithi153 ceph-mon[36569]: pgmap v17736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:23.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:23 smithi104 ceph-mon[36624]: pgmap v17736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:25.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:25 smithi153 ceph-mon[36569]: pgmap v17737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:25.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:25 smithi104 ceph-mon[36624]: pgmap v17737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:25.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:25 smithi081 ceph-mon[31453]: pgmap v17737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:26.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:26 smithi153 ceph-mon[36569]: pgmap v17738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:26.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:26 smithi104 ceph-mon[36624]: pgmap v17738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:26.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:26 smithi081 ceph-mon[31453]: pgmap v17738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:29.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:29 smithi081 ceph-mon[31453]: pgmap v17739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:29.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:29 smithi153 ceph-mon[36569]: pgmap v17739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:29.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:29 smithi104 ceph-mon[36624]: pgmap v17739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:31.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:31 smithi081 ceph-mon[31453]: pgmap v17740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:31.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:31 smithi153 ceph-mon[36569]: pgmap v17740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:31.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:31 smithi104 ceph-mon[36624]: pgmap v17740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:33.094+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:29:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:33.094+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:29:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:33.094+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:29:33.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:33 smithi081 ceph-mon[31453]: pgmap v17741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:33.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:33 smithi153 ceph-mon[36569]: pgmap v17741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:33.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:33 smithi104 ceph-mon[36624]: pgmap v17741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:35.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:35 smithi081 ceph-mon[31453]: pgmap v17742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:35.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:35 smithi153 ceph-mon[36569]: pgmap v17742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:35.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:35 smithi104 ceph-mon[36624]: pgmap v17742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:37.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:37 smithi081 ceph-mon[31453]: pgmap v17743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:37.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:37 smithi153 ceph-mon[36569]: pgmap v17743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:37.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:37 smithi104 ceph-mon[36624]: pgmap v17743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:39.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:39 smithi081 ceph-mon[31453]: pgmap v17744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:39.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:39 smithi153 ceph-mon[36569]: pgmap v17744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:39.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:39 smithi104 ceph-mon[36624]: pgmap v17744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:41.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:41 smithi153 ceph-mon[36569]: pgmap v17745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:41.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:41 smithi104 ceph-mon[36624]: pgmap v17745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:41.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:41 smithi081 ceph-mon[31453]: pgmap v17745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:43.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:43 smithi081 ceph-mon[31453]: pgmap v17746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:43.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:43 smithi153 ceph-mon[36569]: pgmap v17746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:43.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:43 smithi104 ceph-mon[36624]: pgmap v17746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:45.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:45 smithi153 ceph-mon[36569]: pgmap v17747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:45.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:45 smithi104 ceph-mon[36624]: pgmap v17747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:45.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:45 smithi081 ceph-mon[31453]: pgmap v17747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:47.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:47 smithi153 ceph-mon[36569]: pgmap v17748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:47.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:47 smithi104 ceph-mon[36624]: pgmap v17748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:47.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:47 smithi081 ceph-mon[31453]: pgmap v17748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:48.109+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:29:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:48.109+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:29:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:29:48.109+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:29:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:48 smithi153 ceph-mon[36569]: pgmap v17749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:48.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:48 smithi104 ceph-mon[36624]: pgmap v17749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:48.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:48 smithi081 ceph-mon[31453]: pgmap v17749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:51 smithi081 ceph-mon[31453]: pgmap v17750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:51.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:29:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:51 smithi153 ceph-mon[36569]: pgmap v17750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:51.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:29:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:51 smithi104 ceph-mon[36624]: pgmap v17750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:51.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:29:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:29:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:29:52.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:29:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:29:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:29:52.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:29:52.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:29:52.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:29:52.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:29:53.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:53 smithi081 ceph-mon[31453]: pgmap v17751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:53.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:53 smithi081 ceph-mon[31453]: pgmap v17752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:53.735 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:53 smithi104 ceph-mon[36624]: pgmap v17751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:53.735 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:53 smithi104 ceph-mon[36624]: pgmap v17752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:53 smithi153 ceph-mon[36569]: pgmap v17751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:53.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:53 smithi153 ceph-mon[36569]: pgmap v17752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:55.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:55 smithi153 ceph-mon[36569]: pgmap v17753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:55.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:55 smithi104 ceph-mon[36624]: pgmap v17753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:55.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:55 smithi081 ceph-mon[31453]: pgmap v17753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:57.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:57 smithi153 ceph-mon[36569]: pgmap v17754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:57.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:57 smithi104 ceph-mon[36624]: pgmap v17754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:57.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:57 smithi081 ceph-mon[31453]: pgmap v17754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:59.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:29:59 smithi153 ceph-mon[36569]: pgmap v17755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:59.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:29:59 smithi104 ceph-mon[36624]: pgmap v17755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:29:59.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:29:59 smithi081 ceph-mon[31453]: pgmap v17755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:00.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:30:00.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:30:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:30:01.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:01 smithi153 ceph-mon[36569]: pgmap v17756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:01.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:01 smithi104 ceph-mon[36624]: pgmap v17756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:01.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:01 smithi081 ceph-mon[31453]: pgmap v17756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:02.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:02 smithi153 ceph-mon[36569]: pgmap v17757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:02 smithi104 ceph-mon[36624]: pgmap v17757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:02 smithi081 ceph-mon[31453]: pgmap v17757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:03.124+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:30:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:03.124+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:30:03.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:03.124+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:30:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:04 smithi153 ceph-mon[36569]: pgmap v17758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:04 smithi104 ceph-mon[36624]: pgmap v17758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:04 smithi081 ceph-mon[31453]: pgmap v17758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:06 smithi153 ceph-mon[36569]: pgmap v17759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:06 smithi104 ceph-mon[36624]: pgmap v17759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:06 smithi081 ceph-mon[31453]: pgmap v17759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:08 smithi153 ceph-mon[36569]: pgmap v17760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:08 smithi104 ceph-mon[36624]: pgmap v17760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:08 smithi081 ceph-mon[31453]: pgmap v17760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:11.289 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:10 smithi104 ceph-mon[36624]: pgmap v17761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:10 smithi153 ceph-mon[36569]: pgmap v17761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:11.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:10 smithi081 ceph-mon[31453]: pgmap v17761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:12 smithi153 ceph-mon[36569]: pgmap v17762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:12 smithi104 ceph-mon[36624]: pgmap v17762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:13.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:12 smithi081 ceph-mon[31453]: pgmap v17762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:14 smithi153 ceph-mon[36569]: pgmap v17763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:14 smithi104 ceph-mon[36624]: pgmap v17763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:15.384 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:14 smithi081 ceph-mon[31453]: pgmap v17763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:17.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:16 smithi081 ceph-mon[31453]: pgmap v17764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:16 smithi153 ceph-mon[36569]: pgmap v17764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:16 smithi104 ceph-mon[36624]: pgmap v17764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:18.134+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:30:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:18.134+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:30:18.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:18.134+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:30:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:18 smithi153 ceph-mon[36569]: pgmap v17765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:18 smithi104 ceph-mon[36624]: pgmap v17765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:19.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:18 smithi081 ceph-mon[31453]: pgmap v17765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:20 smithi153 ceph-mon[36569]: pgmap v17766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:21.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:20 smithi104 ceph-mon[36624]: pgmap v17766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:20 smithi081 ceph-mon[31453]: pgmap v17766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:22 smithi153 ceph-mon[36569]: pgmap v17767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:22 smithi104 ceph-mon[36624]: pgmap v17767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:22 smithi081 ceph-mon[31453]: pgmap v17767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:25.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:24 smithi153 ceph-mon[36569]: pgmap v17768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:24 smithi104 ceph-mon[36624]: pgmap v17768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:24 smithi081 ceph-mon[31453]: pgmap v17768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:27.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:26 smithi081 ceph-mon[31453]: pgmap v17769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:26 smithi153 ceph-mon[36569]: pgmap v17769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:26 smithi104 ceph-mon[36624]: pgmap v17769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:29.258 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:28 smithi081 ceph-mon[31453]: pgmap v17770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:28 smithi153 ceph-mon[36569]: pgmap v17770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:28 smithi104 ceph-mon[36624]: pgmap v17770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:31.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:30 smithi153 ceph-mon[36569]: pgmap v17771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:30 smithi104 ceph-mon[36624]: pgmap v17771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:30 smithi081 ceph-mon[31453]: pgmap v17771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:33.150+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:30:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:33.150+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:30:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:33.150+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:30:33.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:32 smithi153 ceph-mon[36569]: pgmap v17772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:33.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:32 smithi104 ceph-mon[36624]: pgmap v17772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:33.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:32 smithi081 ceph-mon[31453]: pgmap v17772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:35.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:35 smithi153 ceph-mon[36569]: pgmap v17773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:35 smithi104 ceph-mon[36624]: pgmap v17773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:35.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:35 smithi081 ceph-mon[31453]: pgmap v17773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:37.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:37 smithi153 ceph-mon[36569]: pgmap v17774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:37 smithi104 ceph-mon[36624]: pgmap v17774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:37.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:37 smithi081 ceph-mon[31453]: pgmap v17774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:39.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:39 smithi153 ceph-mon[36569]: pgmap v17775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:39.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:39 smithi104 ceph-mon[36624]: pgmap v17775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:39 smithi081 ceph-mon[31453]: pgmap v17775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:41 smithi153 ceph-mon[36569]: pgmap v17776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:41 smithi104 ceph-mon[36624]: pgmap v17776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:41.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:41 smithi081 ceph-mon[31453]: pgmap v17776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:43.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:43 smithi153 ceph-mon[36569]: pgmap v17777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:43.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:43 smithi104 ceph-mon[36624]: pgmap v17777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:43.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:43 smithi081 ceph-mon[31453]: pgmap v17777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:45 smithi153 ceph-mon[36569]: pgmap v17778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:45 smithi104 ceph-mon[36624]: pgmap v17778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:45.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:45 smithi081 ceph-mon[31453]: pgmap v17778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:47 smithi153 ceph-mon[36569]: pgmap v17779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:47.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:47 smithi104 ceph-mon[36624]: pgmap v17779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:47.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:47 smithi081 ceph-mon[31453]: pgmap v17779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:48.156+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:30:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:48.156+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:30:48.160 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:30:48.156+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:30:49.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:49 smithi153 ceph-mon[36569]: pgmap v17780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:49.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:49 smithi104 ceph-mon[36624]: pgmap v17780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:49.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:49 smithi081 ceph-mon[31453]: pgmap v17780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:51.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:51 smithi153 ceph-mon[36569]: pgmap v17781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:51.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:51 smithi104 ceph-mon[36624]: pgmap v17781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:51.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:51 smithi081 ceph-mon[31453]: pgmap v17781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:53 smithi153 ceph-mon[36569]: pgmap v17782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:30:53.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:53 smithi104 ceph-mon[36624]: pgmap v17782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:53.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:30:53.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:53 smithi081 ceph-mon[31453]: pgmap v17782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:53.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:30:54.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:54 smithi104 ceph-mon[36624]: pgmap v17783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:54.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:54 smithi104 ceph-mon[36624]: pgmap v17784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:54.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:30:54.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:30:54.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:30:54.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:54 smithi081 ceph-mon[31453]: pgmap v17783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:54.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:54 smithi081 ceph-mon[31453]: pgmap v17784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:54.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:30:54.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:30:54.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:30:54.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:54 smithi153 ceph-mon[36569]: pgmap v17783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:54.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:54 smithi153 ceph-mon[36569]: pgmap v17784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:54.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:30:56.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:56 smithi153 ceph-mon[36569]: pgmap v17785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:56.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:56 smithi104 ceph-mon[36624]: pgmap v17785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:56.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:56 smithi081 ceph-mon[31453]: pgmap v17785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:57.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:57 smithi153 ceph-mon[36569]: pgmap v17786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:57.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:57 smithi104 ceph-mon[36624]: pgmap v17786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:57.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:57 smithi081 ceph-mon[31453]: pgmap v17786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:59.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:30:59 smithi153 ceph-mon[36569]: pgmap v17787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:59.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:30:59 smithi104 ceph-mon[36624]: pgmap v17787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:30:59.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:30:59 smithi081 ceph-mon[31453]: pgmap v17787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:01.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:01 smithi153 ceph-mon[36569]: pgmap v17788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:01.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:01 smithi104 ceph-mon[36624]: pgmap v17788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:01.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:01 smithi081 ceph-mon[31453]: pgmap v17788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:03.167+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:31:03.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:03.167+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:31:03.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:03.167+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:31:03.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:03 smithi153 ceph-mon[36569]: pgmap v17789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:03.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:03 smithi104 ceph-mon[36624]: pgmap v17789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:03.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:03 smithi081 ceph-mon[31453]: pgmap v17789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:05.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:05 smithi153 ceph-mon[36569]: pgmap v17790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:05.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:05 smithi104 ceph-mon[36624]: pgmap v17790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:05.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:05 smithi081 ceph-mon[31453]: pgmap v17790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:07.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:07 smithi153 ceph-mon[36569]: pgmap v17791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:07.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:07 smithi104 ceph-mon[36624]: pgmap v17791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:07.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:07 smithi081 ceph-mon[31453]: pgmap v17791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:09.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:09 smithi153 ceph-mon[36569]: pgmap v17792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:09.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:09 smithi104 ceph-mon[36624]: pgmap v17792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:09.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:09 smithi081 ceph-mon[31453]: pgmap v17792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:11.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:11 smithi153 ceph-mon[36569]: pgmap v17793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:11.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:11 smithi104 ceph-mon[36624]: pgmap v17793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:11.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:11 smithi081 ceph-mon[31453]: pgmap v17793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:12.184 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:12.181+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000716) for more than 900000000 us, fault. 2024-04-13T06:31:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:12.181+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006660 tx=0x7f16f402b000 comp rx=0 tx=0).stop 2024-04-13T06:31:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:12.181+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T06:31:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:12.181+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T06:31:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:12.181+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T06:31:12.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:12.182+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T06:31:12.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:12.183+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006530 tx=0x7f16f4039000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T06:31:13.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:13 smithi153 ceph-mon[36569]: pgmap v17794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:13.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:13 smithi104 ceph-mon[36624]: pgmap v17794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:13.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:13 smithi081 ceph-mon[31453]: pgmap v17794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:15.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:15 smithi153 ceph-mon[36569]: pgmap v17795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:15.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:15 smithi104 ceph-mon[36624]: pgmap v17795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:15.851 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:15 smithi081 ceph-mon[31453]: pgmap v17795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:17.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:17 smithi153 ceph-mon[36569]: pgmap v17796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:17.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:17 smithi104 ceph-mon[36624]: pgmap v17796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:17.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:17 smithi081 ceph-mon[31453]: pgmap v17796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:18.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:18.181+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:31:18.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:18.181+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:31:18.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:18.181+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:31:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:18.210+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:31:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:18.210+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:31:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:18.210+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 257 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:31:18.214 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:18.210+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 257 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:31:19.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:19 smithi153 ceph-mon[36569]: pgmap v17797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:19.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:19 smithi104 ceph-mon[36624]: pgmap v17797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:19.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:19 smithi081 ceph-mon[31453]: pgmap v17797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:21.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:21 smithi153 ceph-mon[36569]: pgmap v17798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:21.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:21 smithi104 ceph-mon[36624]: pgmap v17798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:21.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:21 smithi081 ceph-mon[31453]: pgmap v17798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:23.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:23 smithi153 ceph-mon[36569]: pgmap v17799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:23.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:23 smithi104 ceph-mon[36624]: pgmap v17799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:23.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:23 smithi081 ceph-mon[31453]: pgmap v17799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:25.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:25 smithi153 ceph-mon[36569]: pgmap v17800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:25.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:25 smithi104 ceph-mon[36624]: pgmap v17800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:25.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:25 smithi081 ceph-mon[31453]: pgmap v17800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:27.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:27 smithi153 ceph-mon[36569]: pgmap v17801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:27.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:27 smithi104 ceph-mon[36624]: pgmap v17801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:27 smithi081 ceph-mon[31453]: pgmap v17801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:29.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:29 smithi153 ceph-mon[36569]: pgmap v17802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:29.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:29 smithi104 ceph-mon[36624]: pgmap v17802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:29.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:29 smithi081 ceph-mon[31453]: pgmap v17802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:31.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:31 smithi153 ceph-mon[36569]: pgmap v17803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:31.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:31 smithi104 ceph-mon[36624]: pgmap v17803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:31.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:31 smithi081 ceph-mon[31453]: pgmap v17803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:33.189+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:31:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:33.189+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:31:33.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:33.189+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:31:33.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:33 smithi153 ceph-mon[36569]: pgmap v17804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:33.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:33 smithi104 ceph-mon[36624]: pgmap v17804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:33.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:33 smithi081 ceph-mon[31453]: pgmap v17804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:35.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:35 smithi153 ceph-mon[36569]: pgmap v17805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:35.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:35 smithi104 ceph-mon[36624]: pgmap v17805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:35.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:35 smithi081 ceph-mon[31453]: pgmap v17805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:37.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:37 smithi153 ceph-mon[36569]: pgmap v17806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:37.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:37 smithi104 ceph-mon[36624]: pgmap v17806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:37.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:37 smithi081 ceph-mon[31453]: pgmap v17806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:39.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:39 smithi153 ceph-mon[36569]: pgmap v17807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:39.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:39 smithi104 ceph-mon[36624]: pgmap v17807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:39.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:39 smithi081 ceph-mon[31453]: pgmap v17807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:41 smithi153 ceph-mon[36569]: pgmap v17808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:41.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:41 smithi104 ceph-mon[36624]: pgmap v17808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:41.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:41 smithi081 ceph-mon[31453]: pgmap v17808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:43.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:43 smithi153 ceph-mon[36569]: pgmap v17809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:43.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:43 smithi104 ceph-mon[36624]: pgmap v17809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:43.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:43 smithi081 ceph-mon[31453]: pgmap v17809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:45.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:45 smithi153 ceph-mon[36569]: pgmap v17810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:45.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:45 smithi104 ceph-mon[36624]: pgmap v17810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:45.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:45 smithi081 ceph-mon[31453]: pgmap v17810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:47.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:47 smithi153 ceph-mon[36569]: pgmap v17811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:47.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:47 smithi104 ceph-mon[36624]: pgmap v17811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:47.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:47 smithi081 ceph-mon[31453]: pgmap v17811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:48.199+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:31:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:48.199+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:31:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:31:48.199+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:31:49.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:49 smithi153 ceph-mon[36569]: pgmap v17812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:49.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:49 smithi104 ceph-mon[36624]: pgmap v17812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:49.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:49 smithi081 ceph-mon[31453]: pgmap v17812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:51.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:51 smithi153 ceph-mon[36569]: pgmap v17813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:51.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:51 smithi104 ceph-mon[36624]: pgmap v17813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:51.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:51 smithi081 ceph-mon[31453]: pgmap v17813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:53.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:53 smithi153 ceph-mon[36569]: pgmap v17814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:53.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:53 smithi104 ceph-mon[36624]: pgmap v17814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:53.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:53 smithi081 ceph-mon[31453]: pgmap v17814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:54.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:31:54.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:31:54.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:31:55.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:31:55.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:31:55.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:55 smithi153 ceph-mon[36569]: pgmap v17815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:55.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:55 smithi153 ceph-mon[36569]: pgmap v17816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:55.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:31:55.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:31:55.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:31:55.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:55 smithi104 ceph-mon[36624]: pgmap v17815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:55.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:55 smithi104 ceph-mon[36624]: pgmap v17816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:55.836 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:31:55.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:31:55.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:31:55.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:55 smithi081 ceph-mon[31453]: pgmap v17815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:55.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:55 smithi081 ceph-mon[31453]: pgmap v17816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:55.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:31:58.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:31:58 smithi153 ceph-mon[36569]: pgmap v17817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:58.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:31:58 smithi104 ceph-mon[36624]: pgmap v17817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:31:58.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:31:58 smithi081 ceph-mon[31453]: pgmap v17817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:00.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:00 smithi153 ceph-mon[36569]: pgmap v17818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:00.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:00 smithi104 ceph-mon[36624]: pgmap v17818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:00.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:00 smithi081 ceph-mon[31453]: pgmap v17818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:02.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:02 smithi153 ceph-mon[36569]: pgmap v17819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:02.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:02 smithi104 ceph-mon[36624]: pgmap v17819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:02.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:02 smithi081 ceph-mon[31453]: pgmap v17819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:03.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:03.211+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:32:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:03.211+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:32:03.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:03.211+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:32:04.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:04 smithi104 ceph-mon[36624]: pgmap v17820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:04.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:04 smithi081 ceph-mon[31453]: pgmap v17820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:04.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:04 smithi153 ceph-mon[36569]: pgmap v17820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:06.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:06 smithi104 ceph-mon[36624]: pgmap v17821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:06.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:06 smithi081 ceph-mon[31453]: pgmap v17821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:06.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:06 smithi153 ceph-mon[36569]: pgmap v17821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:08.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:08 smithi081 ceph-mon[31453]: pgmap v17822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:08.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:08 smithi153 ceph-mon[36569]: pgmap v17822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:08.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:08 smithi104 ceph-mon[36624]: pgmap v17822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:10.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:10 smithi081 ceph-mon[31453]: pgmap v17823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:10.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:10 smithi153 ceph-mon[36569]: pgmap v17823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:10.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:10 smithi104 ceph-mon[36624]: pgmap v17823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:12.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:12 smithi081 ceph-mon[31453]: pgmap v17824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:12.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:12 smithi153 ceph-mon[36569]: pgmap v17824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:12.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:12 smithi104 ceph-mon[36624]: pgmap v17824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:14.363 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:14 smithi153 ceph-mon[36569]: pgmap v17825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:14.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:14 smithi081 ceph-mon[31453]: pgmap v17825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:14.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:14 smithi104 ceph-mon[36624]: pgmap v17825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:16.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:16 smithi081 ceph-mon[31453]: pgmap v17826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:16.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:16 smithi153 ceph-mon[36569]: pgmap v17826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:16.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:16 smithi104 ceph-mon[36624]: pgmap v17826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:18.217+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:32:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:18.217+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:32:18.222 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:18.217+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:32:18.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:18 smithi081 ceph-mon[31453]: pgmap v17827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:18.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:18 smithi153 ceph-mon[36569]: pgmap v17827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:18.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:18 smithi104 ceph-mon[36624]: pgmap v17827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:20.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:20 smithi081 ceph-mon[31453]: pgmap v17828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:20.560 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:20 smithi153 ceph-mon[36569]: pgmap v17828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:20.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:20 smithi104 ceph-mon[36624]: pgmap v17828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:22.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:22 smithi081 ceph-mon[31453]: pgmap v17829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:22.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:22 smithi153 ceph-mon[36569]: pgmap v17829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:22.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:22 smithi104 ceph-mon[36624]: pgmap v17829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:24.388 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:24 smithi104 ceph-mon[36624]: pgmap v17830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:24.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:24 smithi081 ceph-mon[31453]: pgmap v17830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:24.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:24 smithi153 ceph-mon[36569]: pgmap v17830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:26.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:26 smithi081 ceph-mon[31453]: pgmap v17831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:26.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:26 smithi153 ceph-mon[36569]: pgmap v17831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:26.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:26 smithi104 ceph-mon[36624]: pgmap v17831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:28.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:28 smithi081 ceph-mon[31453]: pgmap v17832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:28.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:28 smithi153 ceph-mon[36569]: pgmap v17832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:28.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:28 smithi104 ceph-mon[36624]: pgmap v17832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:30.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:30 smithi081 ceph-mon[31453]: pgmap v17833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:30.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:30 smithi153 ceph-mon[36569]: pgmap v17833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:30.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:30 smithi104 ceph-mon[36624]: pgmap v17833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:32.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:32 smithi081 ceph-mon[31453]: pgmap v17834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:32.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:32 smithi153 ceph-mon[36569]: pgmap v17834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:32.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:32 smithi104 ceph-mon[36624]: pgmap v17834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:33.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:33.225+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:32:33.230 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:33.225+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:32:33.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:33.225+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:32:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:34 smithi081 ceph-mon[31453]: pgmap v17835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:34.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:34 smithi153 ceph-mon[36569]: pgmap v17835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:34.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:34 smithi104 ceph-mon[36624]: pgmap v17835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:36.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:36 smithi153 ceph-mon[36569]: pgmap v17836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:36.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:36 smithi104 ceph-mon[36624]: pgmap v17836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:36.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:36 smithi081 ceph-mon[31453]: pgmap v17836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:38.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:38 smithi153 ceph-mon[36569]: pgmap v17837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:38.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:38 smithi104 ceph-mon[36624]: pgmap v17837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:38.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:38 smithi081 ceph-mon[31453]: pgmap v17837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:40.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:40 smithi153 ceph-mon[36569]: pgmap v17838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:40.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:40 smithi104 ceph-mon[36624]: pgmap v17838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:40.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:40 smithi081 ceph-mon[31453]: pgmap v17838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:42.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:42 smithi153 ceph-mon[36569]: pgmap v17839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:42.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:42 smithi104 ceph-mon[36624]: pgmap v17839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:42.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:42 smithi081 ceph-mon[31453]: pgmap v17839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:44.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:44 smithi153 ceph-mon[36569]: pgmap v17840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:44.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:44 smithi104 ceph-mon[36624]: pgmap v17840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:44.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:44 smithi081 ceph-mon[31453]: pgmap v17840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:46.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:46 smithi153 ceph-mon[36569]: pgmap v17841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:46.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:46 smithi104 ceph-mon[36624]: pgmap v17841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:46.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:46 smithi081 ceph-mon[31453]: pgmap v17841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:48.241+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:32:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:48.241+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:32:48.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:32:48.241+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:32:48.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:48 smithi153 ceph-mon[36569]: pgmap v17842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:48.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:48 smithi104 ceph-mon[36624]: pgmap v17842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:48.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:48 smithi081 ceph-mon[31453]: pgmap v17842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:50.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:50 smithi153 ceph-mon[36569]: pgmap v17843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:50.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:50 smithi104 ceph-mon[36624]: pgmap v17843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:50.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:50 smithi081 ceph-mon[31453]: pgmap v17843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:52.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:52 smithi153 ceph-mon[36569]: pgmap v17844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:52.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:52 smithi104 ceph-mon[36624]: pgmap v17844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:52.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:52 smithi081 ceph-mon[31453]: pgmap v17844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:54.464 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:54 smithi081 ceph-mon[31453]: pgmap v17845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:54.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:54 smithi153 ceph-mon[36569]: pgmap v17845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:54.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:54 smithi104 ceph-mon[36624]: pgmap v17845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:56.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:56 smithi153 ceph-mon[36569]: pgmap v17846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:56.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:32:56.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:56 smithi104 ceph-mon[36624]: pgmap v17846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:56.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:32:56.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:56 smithi081 ceph-mon[31453]: pgmap v17846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:56.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:32:57.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:32:57.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:32:57.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:32:57.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:32:57.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:32:57.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:32:57.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:32:57.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:32:57.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:32:58.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:58 smithi153 ceph-mon[36569]: pgmap v17847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:58.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:32:58 smithi153 ceph-mon[36569]: pgmap v17848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:58.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:58 smithi104 ceph-mon[36624]: pgmap v17847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:58.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:32:58 smithi104 ceph-mon[36624]: pgmap v17848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:58.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:58 smithi081 ceph-mon[31453]: pgmap v17847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:32:58.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:32:58 smithi081 ceph-mon[31453]: pgmap v17848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:00.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:00 smithi153 ceph-mon[36569]: pgmap v17849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:00.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:00 smithi104 ceph-mon[36624]: pgmap v17849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:00.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:00 smithi081 ceph-mon[31453]: pgmap v17849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:02.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:02 smithi153 ceph-mon[36569]: pgmap v17850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:02.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:02 smithi104 ceph-mon[36624]: pgmap v17850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:02.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:02 smithi081 ceph-mon[31453]: pgmap v17850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:03.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:03.257+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:33:03.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:03.257+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:33:03.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:03.257+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:33:04.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:04 smithi153 ceph-mon[36569]: pgmap v17851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:04.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:04 smithi104 ceph-mon[36624]: pgmap v17851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:04.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:04 smithi081 ceph-mon[31453]: pgmap v17851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:06.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:06 smithi153 ceph-mon[36569]: pgmap v17852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:06.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:06 smithi104 ceph-mon[36624]: pgmap v17852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:06.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:06 smithi081 ceph-mon[31453]: pgmap v17852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:08.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:08 smithi153 ceph-mon[36569]: pgmap v17853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:08.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:08 smithi104 ceph-mon[36624]: pgmap v17853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:08.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:08 smithi081 ceph-mon[31453]: pgmap v17853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:10.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:10 smithi153 ceph-mon[36569]: pgmap v17854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:10.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:10 smithi104 ceph-mon[36624]: pgmap v17854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:10.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:10 smithi081 ceph-mon[31453]: pgmap v17854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:12.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:12 smithi153 ceph-mon[36569]: pgmap v17855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:12.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:12 smithi104 ceph-mon[36624]: pgmap v17855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:12.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:12 smithi081 ceph-mon[31453]: pgmap v17855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:14.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:14 smithi153 ceph-mon[36569]: pgmap v17856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:14.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:14 smithi104 ceph-mon[36624]: pgmap v17856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:14.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:14 smithi081 ceph-mon[31453]: pgmap v17856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:16.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:16 smithi153 ceph-mon[36569]: pgmap v17857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:16.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:16 smithi104 ceph-mon[36624]: pgmap v17857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:16.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:16 smithi081 ceph-mon[31453]: pgmap v17857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:18.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:18.266+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:33:18.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:18.266+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:33:18.273 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:18.266+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:33:18.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:18 smithi153 ceph-mon[36569]: pgmap v17858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:18.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:18 smithi104 ceph-mon[36624]: pgmap v17858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:18.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:18 smithi081 ceph-mon[31453]: pgmap v17858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:20.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:20 smithi153 ceph-mon[36569]: pgmap v17859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:20.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:20 smithi104 ceph-mon[36624]: pgmap v17859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:20.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:20 smithi081 ceph-mon[31453]: pgmap v17859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:22.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:22 smithi153 ceph-mon[36569]: pgmap v17860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:22.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:22 smithi104 ceph-mon[36624]: pgmap v17860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:22.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:22 smithi081 ceph-mon[31453]: pgmap v17860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:24.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:24 smithi153 ceph-mon[36569]: pgmap v17861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:24.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:24 smithi104 ceph-mon[36624]: pgmap v17861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:24.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:24 smithi081 ceph-mon[31453]: pgmap v17861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:26.561 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:26 smithi153 ceph-mon[36569]: pgmap v17862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:26.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:26 smithi104 ceph-mon[36624]: pgmap v17862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:26.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:26 smithi081 ceph-mon[31453]: pgmap v17862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:28.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:28 smithi153 ceph-mon[36569]: pgmap v17863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:28.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:28 smithi104 ceph-mon[36624]: pgmap v17863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:28.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:28 smithi081 ceph-mon[31453]: pgmap v17863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:30.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:30 smithi104 ceph-mon[36624]: pgmap v17864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:30.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:30 smithi081 ceph-mon[31453]: pgmap v17864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:30.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:30 smithi153 ceph-mon[36569]: pgmap v17864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:32.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:32 smithi104 ceph-mon[36624]: pgmap v17865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:32.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:32 smithi081 ceph-mon[31453]: pgmap v17865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:32.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:32 smithi153 ceph-mon[36569]: pgmap v17865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:33.280+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:33:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:33.280+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:33:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:33.280+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:33:34.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:34 smithi104 ceph-mon[36624]: pgmap v17866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:34.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:34 smithi081 ceph-mon[31453]: pgmap v17866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:34.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:34 smithi153 ceph-mon[36569]: pgmap v17866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:36.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:36 smithi104 ceph-mon[36624]: pgmap v17867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:36.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:36 smithi081 ceph-mon[31453]: pgmap v17867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:36.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:36 smithi153 ceph-mon[36569]: pgmap v17867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:38.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:38 smithi104 ceph-mon[36624]: pgmap v17868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:38.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:38 smithi081 ceph-mon[31453]: pgmap v17868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:38.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:38 smithi153 ceph-mon[36569]: pgmap v17868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:40.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:40 smithi081 ceph-mon[31453]: pgmap v17869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:40.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:40 smithi153 ceph-mon[36569]: pgmap v17869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:40.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:40 smithi104 ceph-mon[36624]: pgmap v17869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:42.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:42 smithi081 ceph-mon[31453]: pgmap v17870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:42.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:42 smithi153 ceph-mon[36569]: pgmap v17870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:42.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:42 smithi104 ceph-mon[36624]: pgmap v17870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:44.614 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:44 smithi153 ceph-mon[36569]: pgmap v17871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:44.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:44 smithi081 ceph-mon[31453]: pgmap v17871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:44.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:44 smithi104 ceph-mon[36624]: pgmap v17871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:46.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:46 smithi081 ceph-mon[31453]: pgmap v17872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:46.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:46 smithi153 ceph-mon[36569]: pgmap v17872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:46.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:46 smithi104 ceph-mon[36624]: pgmap v17872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:48.296+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:33:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:48.296+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:33:48.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:33:48.296+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:33:48.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:48 smithi081 ceph-mon[31453]: pgmap v17873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:48.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:48 smithi153 ceph-mon[36569]: pgmap v17873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:48.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:48 smithi104 ceph-mon[36624]: pgmap v17873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:50.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:50 smithi081 ceph-mon[31453]: pgmap v17874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:50.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:50 smithi153 ceph-mon[36569]: pgmap v17874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:50.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:50 smithi104 ceph-mon[36624]: pgmap v17874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:52.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:52 smithi081 ceph-mon[31453]: pgmap v17875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:52.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:52 smithi153 ceph-mon[36569]: pgmap v17875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:52.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:52 smithi104 ceph-mon[36624]: pgmap v17875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:54.640 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:54 smithi104 ceph-mon[36624]: pgmap v17876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:54.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:54 smithi081 ceph-mon[31453]: pgmap v17876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:54.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:54 smithi153 ceph-mon[36569]: pgmap v17876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:56.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:56 smithi081 ceph-mon[31453]: pgmap v17877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:56.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:56 smithi153 ceph-mon[36569]: pgmap v17877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:56.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:56 smithi104 ceph-mon[36624]: pgmap v17877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:57.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:33:57.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:33:57.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:33:58.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: pgmap v17878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:58.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:33:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:33:58.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:33:58.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: pgmap v17878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:58.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:33:58.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: pgmap v17878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:33:58.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:33:58.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:33:58.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:33:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:59 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:33:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:59 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:33:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:59 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:33:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:59 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:33:59.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:59 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:33:59.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:59 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:33:59.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:59 smithi081 ceph-mon[31453]: pgmap v17879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:59.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:33:59 smithi081 ceph-mon[31453]: pgmap v17880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:59.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:59 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:33:59.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:59 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:33:59.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:59 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:33:59.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:59 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:33:59.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:59 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:33:59.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:59 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:33:59.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:59 smithi153 ceph-mon[36569]: pgmap v17879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:59.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:33:59 smithi153 ceph-mon[36569]: pgmap v17880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:59.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:59 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:33:59.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:59 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:33:59.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:59 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:33:59.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:59 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:33:59.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:59 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:33:59.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:59 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:33:59.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:59 smithi104 ceph-mon[36624]: pgmap v17879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:33:59.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:33:59 smithi104 ceph-mon[36624]: pgmap v17880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:01.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:01 smithi153 ceph-mon[36569]: pgmap v17881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:01.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:01 smithi104 ceph-mon[36624]: pgmap v17881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:01.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:01 smithi081 ceph-mon[31453]: pgmap v17881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:03.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:03.309+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:34:03.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:03.309+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:34:03.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:03.309+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:34:03.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:03 smithi153 ceph-mon[36569]: pgmap v17882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:03.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:03 smithi104 ceph-mon[36624]: pgmap v17882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:03.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:03 smithi081 ceph-mon[31453]: pgmap v17882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:05.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:05 smithi153 ceph-mon[36569]: pgmap v17883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:05.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:05 smithi104 ceph-mon[36624]: pgmap v17883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:05.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:05 smithi081 ceph-mon[31453]: pgmap v17883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:07.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:07 smithi153 ceph-mon[36569]: pgmap v17884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:07.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:07 smithi104 ceph-mon[36624]: pgmap v17884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:07.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:07 smithi081 ceph-mon[31453]: pgmap v17884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:09.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:09 smithi153 ceph-mon[36569]: pgmap v17885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:10.014 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:09 smithi081 ceph-mon[31453]: pgmap v17885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:10.014 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:09 smithi104 ceph-mon[36624]: pgmap v17885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:11.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:11 smithi153 ceph-mon[36569]: pgmap v17886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:11.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:11 smithi104 ceph-mon[36624]: pgmap v17886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:11.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:11 smithi081 ceph-mon[31453]: pgmap v17886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:13.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:13 smithi153 ceph-mon[36569]: pgmap v17887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:13.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:13 smithi104 ceph-mon[36624]: pgmap v17887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:13.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:13 smithi081 ceph-mon[31453]: pgmap v17887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:15.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:15 smithi153 ceph-mon[36569]: pgmap v17888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:15.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:15 smithi104 ceph-mon[36624]: pgmap v17888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:15.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:15 smithi081 ceph-mon[31453]: pgmap v17888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:17.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:17 smithi153 ceph-mon[36569]: pgmap v17889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:17.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:17 smithi104 ceph-mon[36624]: pgmap v17889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:17.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:17 smithi081 ceph-mon[31453]: pgmap v17889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:18.240+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:34:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:18.240+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:34:18.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:18.241+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 258 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:34:18.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:18.241+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 258 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:34:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:18.325+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:34:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:18.325+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:34:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:18.325+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:34:19.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:19 smithi153 ceph-mon[36569]: pgmap v17890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:19.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:19 smithi104 ceph-mon[36624]: pgmap v17890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:19.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:19 smithi081 ceph-mon[31453]: pgmap v17890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:21.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:21 smithi153 ceph-mon[36569]: pgmap v17891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:21.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:21 smithi104 ceph-mon[36624]: pgmap v17891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:21.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:21 smithi081 ceph-mon[31453]: pgmap v17891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:23.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:23 smithi153 ceph-mon[36569]: pgmap v17892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:23.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:23 smithi104 ceph-mon[36624]: pgmap v17892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:23.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:23 smithi081 ceph-mon[31453]: pgmap v17892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:25.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:25 smithi153 ceph-mon[36569]: pgmap v17893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:25.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:25 smithi104 ceph-mon[36624]: pgmap v17893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:25.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:25 smithi081 ceph-mon[31453]: pgmap v17893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:27.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:27 smithi153 ceph-mon[36569]: pgmap v17894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:27.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:27 smithi104 ceph-mon[36624]: pgmap v17894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:27.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:27 smithi081 ceph-mon[31453]: pgmap v17894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:28.812 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:28 smithi153 ceph-mon[36569]: pgmap v17895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:28.838 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:28 smithi104 ceph-mon[36624]: pgmap v17895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:28.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:28 smithi081 ceph-mon[31453]: pgmap v17895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:31.412 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:31 smithi081 ceph-mon[31453]: pgmap v17896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:31.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:31 smithi153 ceph-mon[36569]: pgmap v17896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:31.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:31 smithi104 ceph-mon[36624]: pgmap v17896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:33.327+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:34:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:33.327+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:34:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:33.327+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:34:33.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:33 smithi153 ceph-mon[36569]: pgmap v17897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:33.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:33 smithi104 ceph-mon[36624]: pgmap v17897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:33.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:33 smithi081 ceph-mon[31453]: pgmap v17897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:35.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:35 smithi153 ceph-mon[36569]: pgmap v17898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:35.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:35 smithi104 ceph-mon[36624]: pgmap v17898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:35.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:35 smithi081 ceph-mon[31453]: pgmap v17898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:37.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:37 smithi153 ceph-mon[36569]: pgmap v17899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:37.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:37 smithi104 ceph-mon[36624]: pgmap v17899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:37.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:37 smithi081 ceph-mon[31453]: pgmap v17899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:39.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:39 smithi153 ceph-mon[36569]: pgmap v17900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:39.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:39 smithi104 ceph-mon[36624]: pgmap v17900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:39.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:39 smithi081 ceph-mon[31453]: pgmap v17900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:41.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:41 smithi153 ceph-mon[36569]: pgmap v17901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:41.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:41 smithi104 ceph-mon[36624]: pgmap v17901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:41.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:41 smithi081 ceph-mon[31453]: pgmap v17901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:43.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:43 smithi153 ceph-mon[36569]: pgmap v17902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:43.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:43 smithi104 ceph-mon[36624]: pgmap v17902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:43.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:43 smithi081 ceph-mon[31453]: pgmap v17902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:45.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:45 smithi153 ceph-mon[36569]: pgmap v17903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:45.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:45 smithi104 ceph-mon[36624]: pgmap v17903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:45.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:45 smithi081 ceph-mon[31453]: pgmap v17903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:47.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:47 smithi153 ceph-mon[36569]: pgmap v17904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:47.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:47 smithi104 ceph-mon[36624]: pgmap v17904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:47.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:47 smithi081 ceph-mon[31453]: pgmap v17904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:48.332+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:34:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:48.332+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:34:48.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:34:48.332+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:34:49.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:49 smithi153 ceph-mon[36569]: pgmap v17905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:49.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:49 smithi104 ceph-mon[36624]: pgmap v17905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:49.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:49 smithi081 ceph-mon[31453]: pgmap v17905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:51.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:51 smithi153 ceph-mon[36569]: pgmap v17906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:51.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:51 smithi104 ceph-mon[36624]: pgmap v17906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:51.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:51 smithi081 ceph-mon[31453]: pgmap v17906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:53.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:53 smithi153 ceph-mon[36569]: pgmap v17907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:53.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:53 smithi104 ceph-mon[36624]: pgmap v17907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:53.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:53 smithi081 ceph-mon[31453]: pgmap v17907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:55.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:55 smithi153 ceph-mon[36569]: pgmap v17908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:55.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:55 smithi104 ceph-mon[36624]: pgmap v17908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:55.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:55 smithi081 ceph-mon[31453]: pgmap v17908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:57.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:57 smithi153 ceph-mon[36569]: pgmap v17909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:57.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:57 smithi104 ceph-mon[36624]: pgmap v17909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:57.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:57 smithi081 ceph-mon[31453]: pgmap v17909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:59.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:59 smithi153 ceph-mon[36569]: pgmap v17910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:59.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:34:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:34:59.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:59 smithi104 ceph-mon[36624]: pgmap v17910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:59.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:34:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:34:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:59 smithi081 ceph-mon[31453]: pgmap v17910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:34:59.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:34:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:35:00.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:35:00.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:35:00.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:35:00.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:35:00.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:35:00.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:35:00.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:35:00.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:35:00.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:35:01.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:01 smithi153 ceph-mon[36569]: pgmap v17911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:01.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:01 smithi153 ceph-mon[36569]: pgmap v17912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:01.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:01 smithi104 ceph-mon[36624]: pgmap v17911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:01.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:01 smithi104 ceph-mon[36624]: pgmap v17912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:01.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:01 smithi081 ceph-mon[31453]: pgmap v17911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:01.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:01 smithi081 ceph-mon[31453]: pgmap v17912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:03.337+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:35:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:03.337+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:35:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:03.337+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:35:03.562 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:03 smithi153 ceph-mon[36569]: pgmap v17913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:03.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:03 smithi104 ceph-mon[36624]: pgmap v17913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:03.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:03 smithi081 ceph-mon[31453]: pgmap v17913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:05.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:05 smithi153 ceph-mon[36569]: pgmap v17914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:05.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:05 smithi104 ceph-mon[36624]: pgmap v17914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:05.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:05 smithi081 ceph-mon[31453]: pgmap v17914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:07.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:07 smithi153 ceph-mon[36569]: pgmap v17915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:07.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:07 smithi104 ceph-mon[36624]: pgmap v17915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:07.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:07 smithi081 ceph-mon[31453]: pgmap v17915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:09.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:09 smithi153 ceph-mon[36569]: pgmap v17916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:09.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:09 smithi104 ceph-mon[36624]: pgmap v17916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:09.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:09 smithi081 ceph-mon[31453]: pgmap v17916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:11.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:11 smithi153 ceph-mon[36569]: pgmap v17917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:11.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:11 smithi104 ceph-mon[36624]: pgmap v17917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:11.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:11 smithi081 ceph-mon[31453]: pgmap v17917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:13.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:13 smithi153 ceph-mon[36569]: pgmap v17918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:13.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:13 smithi104 ceph-mon[36624]: pgmap v17918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:13.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:13 smithi081 ceph-mon[31453]: pgmap v17918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:15.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:15 smithi153 ceph-mon[36569]: pgmap v17919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:15.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:15 smithi104 ceph-mon[36624]: pgmap v17919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:15.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:15 smithi081 ceph-mon[31453]: pgmap v17919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:17.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:17 smithi153 ceph-mon[36569]: pgmap v17920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:17.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:17 smithi104 ceph-mon[36624]: pgmap v17920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:17.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:17 smithi081 ceph-mon[31453]: pgmap v17920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:18.342+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:35:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:18.342+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:35:18.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:18.342+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:35:19.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:19 smithi153 ceph-mon[36569]: pgmap v17921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:19.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:19 smithi104 ceph-mon[36624]: pgmap v17921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:19.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:19 smithi081 ceph-mon[31453]: pgmap v17921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:21.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:21 smithi153 ceph-mon[36569]: pgmap v17922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:21.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:21 smithi104 ceph-mon[36624]: pgmap v17922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:21.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:21 smithi081 ceph-mon[31453]: pgmap v17922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:23.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:23 smithi153 ceph-mon[36569]: pgmap v17923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:23.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:23 smithi104 ceph-mon[36624]: pgmap v17923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:23.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:23 smithi081 ceph-mon[31453]: pgmap v17923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:25.465 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:25 smithi081 ceph-mon[31453]: pgmap v17924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:25.588 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:25 smithi104 ceph-mon[36624]: pgmap v17924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:25.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:25 smithi153 ceph-mon[36569]: pgmap v17924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:27.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:27 smithi104 ceph-mon[36624]: pgmap v17925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:27.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:27 smithi081 ceph-mon[31453]: pgmap v17925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:27.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:27 smithi153 ceph-mon[36569]: pgmap v17925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:29.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:29 smithi104 ceph-mon[36624]: pgmap v17926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:29.662 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:29 smithi081 ceph-mon[31453]: pgmap v17926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:29.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:29 smithi153 ceph-mon[36569]: pgmap v17926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:31.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:31 smithi104 ceph-mon[36624]: pgmap v17927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:31.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:31 smithi081 ceph-mon[31453]: pgmap v17927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:31.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:31 smithi153 ceph-mon[36569]: pgmap v17927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:33.358+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:35:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:33.358+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:35:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:33.358+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:35:33.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:33 smithi104 ceph-mon[36624]: pgmap v17928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:33.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:33 smithi081 ceph-mon[31453]: pgmap v17928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:33.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:33 smithi153 ceph-mon[36569]: pgmap v17928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:35.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:35 smithi081 ceph-mon[31453]: pgmap v17929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:35.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:35 smithi153 ceph-mon[36569]: pgmap v17929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:35.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:35 smithi104 ceph-mon[36624]: pgmap v17929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:37.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:37 smithi081 ceph-mon[31453]: pgmap v17930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:37.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:37 smithi153 ceph-mon[36569]: pgmap v17930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:37.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:37 smithi104 ceph-mon[36624]: pgmap v17930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:39.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:39 smithi081 ceph-mon[31453]: pgmap v17931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:39.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:39 smithi153 ceph-mon[36569]: pgmap v17931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:39.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:39 smithi104 ceph-mon[36624]: pgmap v17931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:41.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:41 smithi081 ceph-mon[31453]: pgmap v17932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:41.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:41 smithi153 ceph-mon[36569]: pgmap v17932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:41.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:41 smithi104 ceph-mon[36624]: pgmap v17932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:43.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:43 smithi081 ceph-mon[31453]: pgmap v17933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:43.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:43 smithi153 ceph-mon[36569]: pgmap v17933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:43.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:43 smithi104 ceph-mon[36624]: pgmap v17933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:45.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:45 smithi081 ceph-mon[31453]: pgmap v17934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:45.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:45 smithi153 ceph-mon[36569]: pgmap v17934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:45.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:45 smithi104 ceph-mon[36624]: pgmap v17934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:47.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:47 smithi081 ceph-mon[31453]: pgmap v17935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:47.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:47 smithi153 ceph-mon[36569]: pgmap v17935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:47.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:47 smithi104 ceph-mon[36624]: pgmap v17935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:48.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:48.365+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:35:48.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:48.365+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:35:48.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:35:48.365+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:35:49.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:49 smithi081 ceph-mon[31453]: pgmap v17936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:49.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:49 smithi153 ceph-mon[36569]: pgmap v17936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:49.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:49 smithi104 ceph-mon[36624]: pgmap v17936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:51.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:51 smithi081 ceph-mon[31453]: pgmap v17937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:51.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:51 smithi153 ceph-mon[36569]: pgmap v17937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:51.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:51 smithi104 ceph-mon[36624]: pgmap v17937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:53.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:53 smithi081 ceph-mon[31453]: pgmap v17938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:53.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:53 smithi153 ceph-mon[36569]: pgmap v17938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:53.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:53 smithi104 ceph-mon[36624]: pgmap v17938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:55.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:55 smithi081 ceph-mon[31453]: pgmap v17939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:55.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:55 smithi153 ceph-mon[36569]: pgmap v17939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:55.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:55 smithi104 ceph-mon[36624]: pgmap v17939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:57.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:57 smithi081 ceph-mon[31453]: pgmap v17940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:57.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:57 smithi153 ceph-mon[36569]: pgmap v17940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:57.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:57 smithi104 ceph-mon[36624]: pgmap v17940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:59.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:35:59 smithi081 ceph-mon[31453]: pgmap v17941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:59.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:35:59 smithi153 ceph-mon[36569]: pgmap v17941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:35:59.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:35:59 smithi104 ceph-mon[36624]: pgmap v17941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:00.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:36:00.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:36:00.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:36:01.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:01 smithi153 ceph-mon[36569]: pgmap v17942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:01.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:36:01.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:36:01.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:36:01.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:01 smithi104 ceph-mon[36624]: pgmap v17942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:01.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:36:01.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:36:01.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:36:01.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:01 smithi081 ceph-mon[31453]: pgmap v17942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:01.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:36:01.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:36:01.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:36:02.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:02 smithi153 ceph-mon[36569]: pgmap v17943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:02.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:02 smithi153 ceph-mon[36569]: pgmap v17944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:02.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:02 smithi104 ceph-mon[36624]: pgmap v17943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:02.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:02 smithi104 ceph-mon[36624]: pgmap v17944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:02.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:02 smithi081 ceph-mon[31453]: pgmap v17943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:02.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:02 smithi081 ceph-mon[31453]: pgmap v17944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:03.368+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:36:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:03.368+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:36:03.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:03.368+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:36:04.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:04 smithi153 ceph-mon[36569]: pgmap v17945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:04.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:04 smithi104 ceph-mon[36624]: pgmap v17945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:04.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:04 smithi081 ceph-mon[31453]: pgmap v17945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:06.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:06 smithi153 ceph-mon[36569]: pgmap v17946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:06.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:06 smithi104 ceph-mon[36624]: pgmap v17946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:06.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:06 smithi081 ceph-mon[31453]: pgmap v17946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:08.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:08 smithi153 ceph-mon[36569]: pgmap v17947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:08.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:08 smithi104 ceph-mon[36624]: pgmap v17947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:08.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:08 smithi081 ceph-mon[31453]: pgmap v17947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:10.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:10 smithi153 ceph-mon[36569]: pgmap v17948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:10.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:10 smithi104 ceph-mon[36624]: pgmap v17948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:10.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:10 smithi081 ceph-mon[31453]: pgmap v17948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:12.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:12 smithi153 ceph-mon[36569]: pgmap v17949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:12.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:12 smithi104 ceph-mon[36624]: pgmap v17949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:12.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:12 smithi081 ceph-mon[31453]: pgmap v17949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:14.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:14 smithi153 ceph-mon[36569]: pgmap v17950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:14.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:14 smithi104 ceph-mon[36624]: pgmap v17950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:14.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:14 smithi081 ceph-mon[31453]: pgmap v17950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:16.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:16 smithi153 ceph-mon[36569]: pgmap v17951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:16.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:16 smithi104 ceph-mon[36624]: pgmap v17951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:16.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:16 smithi081 ceph-mon[31453]: pgmap v17951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:18.375+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:36:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:18.375+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:36:18.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:18.375+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:36:18.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:18 smithi153 ceph-mon[36569]: pgmap v17952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:18.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:18 smithi104 ceph-mon[36624]: pgmap v17952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:18.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:18 smithi081 ceph-mon[31453]: pgmap v17952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:20.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:20 smithi153 ceph-mon[36569]: pgmap v17953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:20.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:20 smithi104 ceph-mon[36624]: pgmap v17953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:20.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:20 smithi081 ceph-mon[31453]: pgmap v17953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:22.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:22 smithi153 ceph-mon[36569]: pgmap v17954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:22.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:22 smithi104 ceph-mon[36624]: pgmap v17954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:22.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:22 smithi081 ceph-mon[31453]: pgmap v17954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:24.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:24 smithi153 ceph-mon[36569]: pgmap v17955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:24.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:24 smithi104 ceph-mon[36624]: pgmap v17955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:24.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:24 smithi081 ceph-mon[31453]: pgmap v17955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:26.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:26 smithi153 ceph-mon[36569]: pgmap v17956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:26.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:26 smithi104 ceph-mon[36624]: pgmap v17956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:26.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:26 smithi081 ceph-mon[31453]: pgmap v17956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:28.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:28 smithi153 ceph-mon[36569]: pgmap v17957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:28.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:28 smithi104 ceph-mon[36624]: pgmap v17957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:28.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:28 smithi081 ceph-mon[31453]: pgmap v17957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:30.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:30 smithi153 ceph-mon[36569]: pgmap v17958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:30.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:30 smithi104 ceph-mon[36624]: pgmap v17958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:30.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:30 smithi081 ceph-mon[31453]: pgmap v17958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:31.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:31 smithi153 ceph-mon[36569]: pgmap v17959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:31.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:31 smithi104 ceph-mon[36624]: pgmap v17959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:31.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:31 smithi081 ceph-mon[31453]: pgmap v17959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:33.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:33.377+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:36:33.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:33.377+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:36:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:33.377+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:36:34.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:34 smithi153 ceph-mon[36569]: pgmap v17960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:34.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:34 smithi104 ceph-mon[36624]: pgmap v17960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:34.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:34 smithi081 ceph-mon[31453]: pgmap v17960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:36.564 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:36 smithi153 ceph-mon[36569]: pgmap v17961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:36.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:36 smithi104 ceph-mon[36624]: pgmap v17961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:36.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:36 smithi081 ceph-mon[31453]: pgmap v17961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:38.563 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:38 smithi153 ceph-mon[36569]: pgmap v17962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:38.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:38 smithi104 ceph-mon[36624]: pgmap v17962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:38.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:38 smithi081 ceph-mon[31453]: pgmap v17962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:40.564 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:40 smithi153 ceph-mon[36569]: pgmap v17963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:40.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:40 smithi104 ceph-mon[36624]: pgmap v17963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:40.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:40 smithi081 ceph-mon[31453]: pgmap v17963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:42.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:42 smithi104 ceph-mon[36624]: pgmap v17964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:42.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:42 smithi081 ceph-mon[31453]: pgmap v17964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:42.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:42 smithi153 ceph-mon[36569]: pgmap v17964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:44.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:44 smithi104 ceph-mon[36624]: pgmap v17965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:44.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:44 smithi081 ceph-mon[31453]: pgmap v17965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:44.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:44 smithi153 ceph-mon[36569]: pgmap v17965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:46.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:46 smithi104 ceph-mon[36624]: pgmap v17966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:46.616 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:46 smithi153 ceph-mon[36569]: pgmap v17966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:46.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:46 smithi081 ceph-mon[31453]: pgmap v17966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:48.392+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:36:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:48.392+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:36:48.400 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:36:48.392+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:36:48.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:48 smithi104 ceph-mon[36624]: pgmap v17967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:48.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:48 smithi081 ceph-mon[31453]: pgmap v17967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:48.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:48 smithi153 ceph-mon[36569]: pgmap v17967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:50.589 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:50 smithi104 ceph-mon[36624]: pgmap v17968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:50.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:50 smithi081 ceph-mon[31453]: pgmap v17968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:50.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:50 smithi153 ceph-mon[36569]: pgmap v17968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:52.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:52 smithi081 ceph-mon[31453]: pgmap v17969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:52.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:52 smithi153 ceph-mon[36569]: pgmap v17969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:52.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:52 smithi104 ceph-mon[36624]: pgmap v17969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:54.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:54 smithi081 ceph-mon[31453]: pgmap v17970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:54.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:54 smithi153 ceph-mon[36569]: pgmap v17970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:54.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:54 smithi104 ceph-mon[36624]: pgmap v17970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:56.642 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:56 smithi104 ceph-mon[36624]: pgmap v17971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:56.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:56 smithi081 ceph-mon[31453]: pgmap v17971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:56.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:56 smithi153 ceph-mon[36569]: pgmap v17971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:58.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:36:58 smithi081 ceph-mon[31453]: pgmap v17972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:58.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:36:58 smithi153 ceph-mon[36569]: pgmap v17972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:36:58.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:36:58 smithi104 ceph-mon[36624]: pgmap v17972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:00.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:00 smithi081 ceph-mon[31453]: pgmap v17973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:00.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:00 smithi153 ceph-mon[36569]: pgmap v17973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:00 smithi104 ceph-mon[36624]: pgmap v17973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:02.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:02 smithi081 ceph-mon[31453]: pgmap v17974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:02.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:37:02.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:02 smithi153 ceph-mon[36569]: pgmap v17974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:02.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:37:02.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:02 smithi104 ceph-mon[36624]: pgmap v17974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:02.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:37:03.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:03.405+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:37:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:03.405+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:37:03.413 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:03.405+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:37:03.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:37:03.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:37:03.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:37:03.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:37:03.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:37:03.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:37:03.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:37:03.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:37:03.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:37:04.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:04 smithi081 ceph-mon[31453]: pgmap v17975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:04.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:04 smithi081 ceph-mon[31453]: pgmap v17976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:04.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:04 smithi153 ceph-mon[36569]: pgmap v17975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:04.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:04 smithi153 ceph-mon[36569]: pgmap v17976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:04.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:04 smithi104 ceph-mon[36624]: pgmap v17975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:04.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:04 smithi104 ceph-mon[36624]: pgmap v17976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:06.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:06 smithi081 ceph-mon[31453]: pgmap v17977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:06.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:06 smithi153 ceph-mon[36569]: pgmap v17977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:06.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:06 smithi104 ceph-mon[36624]: pgmap v17977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:08.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:08 smithi081 ceph-mon[31453]: pgmap v17978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:08.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:08 smithi153 ceph-mon[36569]: pgmap v17978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:08.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:08 smithi104 ceph-mon[36624]: pgmap v17978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:10.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:10 smithi081 ceph-mon[31453]: pgmap v17979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:10.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:10 smithi153 ceph-mon[36569]: pgmap v17979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:10.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:10 smithi104 ceph-mon[36624]: pgmap v17979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:12.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:12 smithi081 ceph-mon[31453]: pgmap v17980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:12.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:12 smithi153 ceph-mon[36569]: pgmap v17980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:12.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:12 smithi104 ceph-mon[36624]: pgmap v17980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:14.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:14 smithi081 ceph-mon[31453]: pgmap v17981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:14.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:14 smithi153 ceph-mon[36569]: pgmap v17981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:14.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:14 smithi104 ceph-mon[36624]: pgmap v17981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:16.663 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:16 smithi081 ceph-mon[31453]: pgmap v17982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:16.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:16 smithi153 ceph-mon[36569]: pgmap v17982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:16.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:16 smithi104 ceph-mon[36624]: pgmap v17982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.270+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:37:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.270+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:37:18.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.271+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 259 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T06:37:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.271+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 259 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T06:37:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.271+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:37:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.271+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:37:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.272+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 260 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:37:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.272+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 260 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:37:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.420+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:37:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.420+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:37:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:18.420+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:37:18.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:18 smithi153 ceph-mon[36569]: pgmap v17983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:18.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:18 smithi104 ceph-mon[36624]: pgmap v17983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:18.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:18 smithi081 ceph-mon[31453]: pgmap v17983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:20.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:20 smithi153 ceph-mon[36569]: pgmap v17984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:20.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:20 smithi104 ceph-mon[36624]: pgmap v17984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:20.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:20 smithi081 ceph-mon[31453]: pgmap v17984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:22.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:22 smithi153 ceph-mon[36569]: pgmap v17985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:22.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:22 smithi104 ceph-mon[36624]: pgmap v17985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:22.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:22 smithi081 ceph-mon[31453]: pgmap v17985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:24.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:24 smithi153 ceph-mon[36569]: pgmap v17986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:24.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:24 smithi104 ceph-mon[36624]: pgmap v17986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:24.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:24 smithi081 ceph-mon[31453]: pgmap v17986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:26.717 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:26 smithi081 ceph-mon[31453]: pgmap v17987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:26.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:26 smithi153 ceph-mon[36569]: pgmap v17987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:26.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:26 smithi104 ceph-mon[36624]: pgmap v17987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:28.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:28 smithi153 ceph-mon[36569]: pgmap v17988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:28.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:28 smithi104 ceph-mon[36624]: pgmap v17988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:28.990 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:28 smithi081 ceph-mon[31453]: pgmap v17988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:30.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:30 smithi153 ceph-mon[36569]: pgmap v17989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:30.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:30 smithi104 ceph-mon[36624]: pgmap v17989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:30.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:30 smithi081 ceph-mon[31453]: pgmap v17989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:32.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:32 smithi153 ceph-mon[36569]: pgmap v17990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:32.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:32 smithi104 ceph-mon[36624]: pgmap v17990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:32.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:32 smithi081 ceph-mon[31453]: pgmap v17990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:33.436+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:37:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:33.436+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:37:33.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:33.436+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:37:34.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:34 smithi153 ceph-mon[36569]: pgmap v17991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:34.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:34 smithi104 ceph-mon[36624]: pgmap v17991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:34.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:34 smithi081 ceph-mon[31453]: pgmap v17991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:36.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:36 smithi153 ceph-mon[36569]: pgmap v17992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:36.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:36 smithi104 ceph-mon[36624]: pgmap v17992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:36.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:36 smithi081 ceph-mon[31453]: pgmap v17992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:38.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:38 smithi153 ceph-mon[36569]: pgmap v17993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:38.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:38 smithi104 ceph-mon[36624]: pgmap v17993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:38.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:38 smithi081 ceph-mon[31453]: pgmap v17993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:40.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:40 smithi153 ceph-mon[36569]: pgmap v17994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:40.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:40 smithi104 ceph-mon[36624]: pgmap v17994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:40.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:40 smithi081 ceph-mon[31453]: pgmap v17994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:42.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:42 smithi153 ceph-mon[36569]: pgmap v17995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:42.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:42 smithi104 ceph-mon[36624]: pgmap v17995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:42.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:42 smithi081 ceph-mon[31453]: pgmap v17995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:44.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:44 smithi153 ceph-mon[36569]: pgmap v17996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:44.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:44 smithi104 ceph-mon[36624]: pgmap v17996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:44.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:44 smithi081 ceph-mon[31453]: pgmap v17996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:46.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:46 smithi153 ceph-mon[36569]: pgmap v17997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:46.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:46 smithi104 ceph-mon[36624]: pgmap v17997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:46.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:46 smithi081 ceph-mon[31453]: pgmap v17997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:48.439+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:37:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:48.439+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:37:48.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:37:48.439+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:37:48.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:48 smithi153 ceph-mon[36569]: pgmap v17998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:48.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:48 smithi104 ceph-mon[36624]: pgmap v17998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:48.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:48 smithi081 ceph-mon[31453]: pgmap v17998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:50.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:50 smithi153 ceph-mon[36569]: pgmap v17999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:50.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:50 smithi104 ceph-mon[36624]: pgmap v17999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:50.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:50 smithi081 ceph-mon[31453]: pgmap v17999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:52.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:52 smithi153 ceph-mon[36569]: pgmap v18000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:52.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:52 smithi104 ceph-mon[36624]: pgmap v18000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:52.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:52 smithi081 ceph-mon[31453]: pgmap v18000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:54.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:54 smithi153 ceph-mon[36569]: pgmap v18001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:54.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:54 smithi104 ceph-mon[36624]: pgmap v18001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:54.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:54 smithi081 ceph-mon[31453]: pgmap v18001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:56.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:56 smithi153 ceph-mon[36569]: pgmap v18002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:56.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:56 smithi104 ceph-mon[36624]: pgmap v18002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:56.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:56 smithi081 ceph-mon[31453]: pgmap v18002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:58.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:37:58 smithi153 ceph-mon[36569]: pgmap v18003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:58.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:37:58 smithi104 ceph-mon[36624]: pgmap v18003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:37:58.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:37:58 smithi081 ceph-mon[31453]: pgmap v18003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:00.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:00 smithi081 ceph-mon[31453]: pgmap v18004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:00.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:00 smithi153 ceph-mon[36569]: pgmap v18004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:00 smithi104 ceph-mon[36624]: pgmap v18004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:02.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:02 smithi153 ceph-mon[36569]: pgmap v18005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:02.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:02 smithi104 ceph-mon[36624]: pgmap v18005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:02.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:02 smithi081 ceph-mon[31453]: pgmap v18005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:03.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:03.454+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:38:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:03.454+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:38:03.463 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:03.454+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:38:03.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:38:03.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:38:03.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:38:04.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:04 smithi153 ceph-mon[36569]: pgmap v18006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:04.869 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.870 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.870 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.870 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.870 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:04 smithi104 ceph-mon[36624]: pgmap v18006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:04.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.871 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:04 smithi081 ceph-mon[31453]: pgmap v18006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:04.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:04.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:06 smithi153 ceph-mon[36569]: pgmap v18007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:06.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:06 smithi104 ceph-mon[36624]: pgmap v18007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:06.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:06 smithi081 ceph-mon[31453]: pgmap v18007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:06.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:06.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:08.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:08 smithi153 ceph-mon[36569]: pgmap v18008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:08.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:08 smithi104 ceph-mon[36624]: pgmap v18008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:08.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:08 smithi081 ceph-mon[31453]: pgmap v18008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:09.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:09.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:09.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:09 smithi153 ceph-mon[36569]: pgmap v18009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:09.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:09.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:09.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:09 smithi104 ceph-mon[36624]: pgmap v18009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:09.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:09.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:09.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:09 smithi081 ceph-mon[31453]: pgmap v18009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:10.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:38:10.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:38:10.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:10 smithi104 ceph-mon[36624]: pgmap v18010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:10.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:10 smithi104 ceph-mon[36624]: pgmap v18011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:10.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:10.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:38:10.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:38:10.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:10 smithi081 ceph-mon[31453]: pgmap v18010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:10.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:10 smithi081 ceph-mon[31453]: pgmap v18011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:10.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:11.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:38:11.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:38:11.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:10 smithi153 ceph-mon[36569]: pgmap v18010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:11.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:10 smithi153 ceph-mon[36569]: pgmap v18011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:11.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:38:13.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:12 smithi153 ceph-mon[36569]: pgmap v18012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:13.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:12 smithi104 ceph-mon[36624]: pgmap v18012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:13.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:12 smithi081 ceph-mon[31453]: pgmap v18012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:15.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:14 smithi153 ceph-mon[36569]: pgmap v18013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:15.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:14 smithi104 ceph-mon[36624]: pgmap v18013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:15.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:14 smithi081 ceph-mon[31453]: pgmap v18013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:17.317 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:16 smithi153 ceph-mon[36569]: pgmap v18014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:17.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:16 smithi104 ceph-mon[36624]: pgmap v18014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:17.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:16 smithi081 ceph-mon[31453]: pgmap v18014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:18.468+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:38:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:18.468+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:38:18.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:18.468+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:38:19.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:18 smithi153 ceph-mon[36569]: pgmap v18015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:19.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:18 smithi104 ceph-mon[36624]: pgmap v18015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:19.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:18 smithi081 ceph-mon[31453]: pgmap v18015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:21.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:20 smithi153 ceph-mon[36569]: pgmap v18016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:21.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:20 smithi104 ceph-mon[36624]: pgmap v18016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:21.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:20 smithi081 ceph-mon[31453]: pgmap v18016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:23.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:22 smithi153 ceph-mon[36569]: pgmap v18017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:23.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:22 smithi104 ceph-mon[36624]: pgmap v18017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:23.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:22 smithi081 ceph-mon[31453]: pgmap v18017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:25.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:24 smithi153 ceph-mon[36569]: pgmap v18018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:25.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:24 smithi104 ceph-mon[36624]: pgmap v18018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:25.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:24 smithi081 ceph-mon[31453]: pgmap v18018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:27.217 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:26 smithi081 ceph-mon[31453]: pgmap v18019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:27.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:26 smithi153 ceph-mon[36569]: pgmap v18019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:27.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:26 smithi104 ceph-mon[36624]: pgmap v18019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:29.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:28 smithi153 ceph-mon[36569]: pgmap v18020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:29.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:28 smithi104 ceph-mon[36624]: pgmap v18020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:29.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:28 smithi081 ceph-mon[31453]: pgmap v18020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:31.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:30 smithi153 ceph-mon[36569]: pgmap v18021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:31.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:30 smithi104 ceph-mon[36624]: pgmap v18021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:31.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:30 smithi081 ceph-mon[31453]: pgmap v18021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:33.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:32 smithi153 ceph-mon[36569]: pgmap v18022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:33.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:32 smithi104 ceph-mon[36624]: pgmap v18022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:33.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:32 smithi081 ceph-mon[31453]: pgmap v18022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:33.474+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:38:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:33.474+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:38:33.483 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:33.474+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:38:35.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:34 smithi153 ceph-mon[36569]: pgmap v18023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:35.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:34 smithi104 ceph-mon[36624]: pgmap v18023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:35.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:34 smithi081 ceph-mon[31453]: pgmap v18023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:37.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:36 smithi153 ceph-mon[36569]: pgmap v18024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:37.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:36 smithi104 ceph-mon[36624]: pgmap v18024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:37.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:36 smithi081 ceph-mon[31453]: pgmap v18024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:39.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:38 smithi153 ceph-mon[36569]: pgmap v18025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:39.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:38 smithi104 ceph-mon[36624]: pgmap v18025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:39.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:38 smithi081 ceph-mon[31453]: pgmap v18025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:41.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:40 smithi153 ceph-mon[36569]: pgmap v18026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:41.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:40 smithi104 ceph-mon[36624]: pgmap v18026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:41.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:40 smithi081 ceph-mon[31453]: pgmap v18026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:43.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:42 smithi153 ceph-mon[36569]: pgmap v18027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:43.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:42 smithi104 ceph-mon[36624]: pgmap v18027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:43.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:42 smithi081 ceph-mon[31453]: pgmap v18027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:45.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:45 smithi153 ceph-mon[36569]: pgmap v18028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:45.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:45 smithi104 ceph-mon[36624]: pgmap v18028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:45.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:45 smithi081 ceph-mon[31453]: pgmap v18028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:47.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:47 smithi153 ceph-mon[36569]: pgmap v18029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:47.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:47 smithi104 ceph-mon[36624]: pgmap v18029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:47.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:47 smithi081 ceph-mon[31453]: pgmap v18029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:48.480+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:38:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:48.480+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:38:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:38:48.480+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:38:49.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:49 smithi153 ceph-mon[36569]: pgmap v18030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:49.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:49 smithi104 ceph-mon[36624]: pgmap v18030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:49.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:49 smithi081 ceph-mon[31453]: pgmap v18030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:51.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:51 smithi153 ceph-mon[36569]: pgmap v18031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:51.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:51 smithi104 ceph-mon[36624]: pgmap v18031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:51.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:51 smithi081 ceph-mon[31453]: pgmap v18031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:53.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:53 smithi153 ceph-mon[36569]: pgmap v18032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:53.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:53 smithi104 ceph-mon[36624]: pgmap v18032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:53.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:53 smithi081 ceph-mon[31453]: pgmap v18032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:55.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:55 smithi153 ceph-mon[36569]: pgmap v18033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:55.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:55 smithi104 ceph-mon[36624]: pgmap v18033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:55.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:55 smithi081 ceph-mon[31453]: pgmap v18033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:57.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:57 smithi153 ceph-mon[36569]: pgmap v18034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:57.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:57 smithi104 ceph-mon[36624]: pgmap v18034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:57.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:57 smithi081 ceph-mon[31453]: pgmap v18034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:59.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:38:59 smithi153 ceph-mon[36569]: pgmap v18035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:59.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:38:59 smithi104 ceph-mon[36624]: pgmap v18035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:38:59.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:38:59 smithi081 ceph-mon[31453]: pgmap v18035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:01.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:01 smithi153 ceph-mon[36569]: pgmap v18036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:01.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:01 smithi104 ceph-mon[36624]: pgmap v18036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:01.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:01 smithi081 ceph-mon[31453]: pgmap v18036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:03.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:03 smithi153 ceph-mon[36569]: pgmap v18037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:03.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:03 smithi104 ceph-mon[36624]: pgmap v18037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:03.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:03 smithi081 ceph-mon[31453]: pgmap v18037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:03.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:03.495+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:39:03.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:03.495+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:39:03.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:03.495+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:39:05.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:05 smithi153 ceph-mon[36569]: pgmap v18038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:05.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:05 smithi104 ceph-mon[36624]: pgmap v18038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:05.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:05 smithi081 ceph-mon[31453]: pgmap v18038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:07.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:07 smithi104 ceph-mon[36624]: pgmap v18039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:07.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:07 smithi081 ceph-mon[31453]: pgmap v18039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:07.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:07 smithi153 ceph-mon[36569]: pgmap v18039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:09.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:09 smithi104 ceph-mon[36624]: pgmap v18040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:09.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:09 smithi081 ceph-mon[31453]: pgmap v18040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:09.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:09 smithi153 ceph-mon[36569]: pgmap v18040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:11.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:11 smithi104 ceph-mon[36624]: pgmap v18041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:11.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:39:11.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:11.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:11.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:11 smithi081 ceph-mon[31453]: pgmap v18041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:11.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:39:11.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:11.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:11.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:11 smithi153 ceph-mon[36569]: pgmap v18041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:11.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:39:11.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:11.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:13 smithi153 ceph-mon[36569]: pgmap v18042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:39:13.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:39:13.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:13 smithi153 ceph-mon[36569]: pgmap v18043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:13 smithi153 ceph-mon[36569]: pgmap v18044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:13 smithi104 ceph-mon[36624]: pgmap v18042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:39:13.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:39:13.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:13 smithi104 ceph-mon[36624]: pgmap v18043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:13 smithi104 ceph-mon[36624]: pgmap v18044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:13 smithi081 ceph-mon[31453]: pgmap v18042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:13.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:39:13.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:39:13.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:13 smithi081 ceph-mon[31453]: pgmap v18043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:13 smithi081 ceph-mon[31453]: pgmap v18044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:13.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:39:15.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:15 smithi104 ceph-mon[36624]: pgmap v18045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:15.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:15 smithi081 ceph-mon[31453]: pgmap v18045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:15.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:15 smithi153 ceph-mon[36569]: pgmap v18045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:17.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:17 smithi104 ceph-mon[36624]: pgmap v18046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:17.618 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:17 smithi153 ceph-mon[36569]: pgmap v18046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:17.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:17 smithi081 ceph-mon[31453]: pgmap v18046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:18.511+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:39:18.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:18.511+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:39:18.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:18.511+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:39:19.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:19 smithi081 ceph-mon[31453]: pgmap v18047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:19.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:19 smithi153 ceph-mon[36569]: pgmap v18047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:19.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:19 smithi104 ceph-mon[36624]: pgmap v18047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:21.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:21 smithi081 ceph-mon[31453]: pgmap v18048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:21.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:21 smithi153 ceph-mon[36569]: pgmap v18048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:21.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:21 smithi104 ceph-mon[36624]: pgmap v18048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:23.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:23 smithi081 ceph-mon[31453]: pgmap v18049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:23.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:23 smithi153 ceph-mon[36569]: pgmap v18049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:23.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:23 smithi104 ceph-mon[36624]: pgmap v18049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:25.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:25 smithi081 ceph-mon[31453]: pgmap v18050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:25.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:25 smithi153 ceph-mon[36569]: pgmap v18050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:25.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:25 smithi104 ceph-mon[36624]: pgmap v18050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:27.643 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:27 smithi104 ceph-mon[36624]: pgmap v18051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:27.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:27 smithi081 ceph-mon[31453]: pgmap v18051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:27.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:27 smithi153 ceph-mon[36569]: pgmap v18051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:29.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:29 smithi104 ceph-mon[36624]: pgmap v18052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:29.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:29 smithi081 ceph-mon[31453]: pgmap v18052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:29.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:29 smithi153 ceph-mon[36569]: pgmap v18052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:31.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:31 smithi081 ceph-mon[31453]: pgmap v18053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:31.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:31 smithi153 ceph-mon[36569]: pgmap v18053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:31.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:31 smithi104 ceph-mon[36624]: pgmap v18053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:33.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:33.526+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:39:33.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:33.527+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:39:33.536 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:33.527+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:39:33.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:33 smithi081 ceph-mon[31453]: pgmap v18054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:33.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:33 smithi153 ceph-mon[36569]: pgmap v18054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:33.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:33 smithi104 ceph-mon[36624]: pgmap v18054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:35.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:35 smithi081 ceph-mon[31453]: pgmap v18055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:35.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:35 smithi153 ceph-mon[36569]: pgmap v18055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:35.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:35 smithi104 ceph-mon[36624]: pgmap v18055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:37.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:37 smithi081 ceph-mon[31453]: pgmap v18056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:37.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:37 smithi153 ceph-mon[36569]: pgmap v18056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:37.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:37 smithi104 ceph-mon[36624]: pgmap v18056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:39.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:39 smithi104 ceph-mon[36624]: pgmap v18057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:39.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:39 smithi081 ceph-mon[31453]: pgmap v18057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:39.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:39 smithi153 ceph-mon[36569]: pgmap v18057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:41.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:41 smithi081 ceph-mon[31453]: pgmap v18058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:41.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:41 smithi153 ceph-mon[36569]: pgmap v18058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:41.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:41 smithi104 ceph-mon[36624]: pgmap v18058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:43.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:43 smithi081 ceph-mon[31453]: pgmap v18059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:43.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:43 smithi153 ceph-mon[36569]: pgmap v18059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:43.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:43 smithi104 ceph-mon[36624]: pgmap v18059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:45.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:45 smithi081 ceph-mon[31453]: pgmap v18060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:45.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:45 smithi153 ceph-mon[36569]: pgmap v18060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:45.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:45 smithi104 ceph-mon[36624]: pgmap v18060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:47.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:47 smithi153 ceph-mon[36569]: pgmap v18061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:47.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:47 smithi104 ceph-mon[36624]: pgmap v18061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:47.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:47 smithi081 ceph-mon[31453]: pgmap v18061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:48.536+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:39:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:48.536+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:39:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:39:48.536+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:39:49.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:49 smithi153 ceph-mon[36569]: pgmap v18062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:49.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:49 smithi104 ceph-mon[36624]: pgmap v18062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:49.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:49 smithi081 ceph-mon[31453]: pgmap v18062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:51.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:51 smithi153 ceph-mon[36569]: pgmap v18063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:51.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:51 smithi104 ceph-mon[36624]: pgmap v18063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:51.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:51 smithi081 ceph-mon[31453]: pgmap v18063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:53.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:53 smithi153 ceph-mon[36569]: pgmap v18064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:53.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:53 smithi104 ceph-mon[36624]: pgmap v18064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:53.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:53 smithi081 ceph-mon[31453]: pgmap v18064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:55.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:55 smithi153 ceph-mon[36569]: pgmap v18065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:55.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:55 smithi104 ceph-mon[36624]: pgmap v18065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:55.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:55 smithi081 ceph-mon[31453]: pgmap v18065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:57.718 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:57 smithi081 ceph-mon[31453]: pgmap v18066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:57.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:57 smithi153 ceph-mon[36569]: pgmap v18066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:57.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:57 smithi104 ceph-mon[36624]: pgmap v18066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:59.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:39:59 smithi153 ceph-mon[36569]: pgmap v18067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:59.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:39:59 smithi104 ceph-mon[36624]: pgmap v18067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:39:59.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:39:59 smithi081 ceph-mon[31453]: pgmap v18067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:00.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:40:00.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:40:00.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:40:01.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:01 smithi153 ceph-mon[36569]: pgmap v18068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:01.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:01 smithi104 ceph-mon[36624]: pgmap v18068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:01.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:01 smithi081 ceph-mon[31453]: pgmap v18068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:03.550+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:40:03.559 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:03.550+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:40:03.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:03.550+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:40:03.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:03 smithi153 ceph-mon[36569]: pgmap v18069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:03.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:03 smithi104 ceph-mon[36624]: pgmap v18069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:03.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:03 smithi081 ceph-mon[31453]: pgmap v18069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:05.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:05 smithi153 ceph-mon[36569]: pgmap v18070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:05.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:05 smithi104 ceph-mon[36624]: pgmap v18070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:05.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:05 smithi081 ceph-mon[31453]: pgmap v18070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:07.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:07 smithi153 ceph-mon[36569]: pgmap v18071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:07.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:07 smithi104 ceph-mon[36624]: pgmap v18071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:07.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:07 smithi081 ceph-mon[31453]: pgmap v18071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:09.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:09 smithi153 ceph-mon[36569]: pgmap v18072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:09.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:09 smithi104 ceph-mon[36624]: pgmap v18072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:09.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:09 smithi081 ceph-mon[31453]: pgmap v18072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:11.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:11 smithi153 ceph-mon[36569]: pgmap v18073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:11.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:11 smithi104 ceph-mon[36624]: pgmap v18073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:11.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:11 smithi081 ceph-mon[31453]: pgmap v18073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:13.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:13 smithi153 ceph-mon[36569]: pgmap v18074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:13.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:40:13.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:13 smithi104 ceph-mon[36624]: pgmap v18074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:13.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:40:13.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:13 smithi081 ceph-mon[31453]: pgmap v18074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:13.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:40:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:40:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:40:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: pgmap v18075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:15.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: pgmap v18076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:15.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: pgmap v18075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: pgmap v18076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:15.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:40:15.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:40:15.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: pgmap v18075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:15.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: pgmap v18076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:15.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:40:17.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:16 smithi153 ceph-mon[36569]: pgmap v18077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:17.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:16 smithi104 ceph-mon[36624]: pgmap v18077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:17.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:16 smithi081 ceph-mon[31453]: pgmap v18077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:18.298+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:40:18.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:18.298+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:40:18.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:18.299+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 261 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:40:18.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:18.299+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 261 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:40:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:18.565+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:40:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:18.565+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:40:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:18.565+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:40:19.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:18 smithi153 ceph-mon[36569]: pgmap v18078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:19.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:18 smithi104 ceph-mon[36624]: pgmap v18078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:19.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:18 smithi081 ceph-mon[31453]: pgmap v18078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:21.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:20 smithi153 ceph-mon[36569]: pgmap v18079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:21.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:20 smithi104 ceph-mon[36624]: pgmap v18079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:21.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:20 smithi081 ceph-mon[31453]: pgmap v18079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:23.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:22 smithi153 ceph-mon[36569]: pgmap v18080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:23.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:22 smithi104 ceph-mon[36624]: pgmap v18080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:23.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:22 smithi081 ceph-mon[31453]: pgmap v18080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:25.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:24 smithi153 ceph-mon[36569]: pgmap v18081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:25.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:24 smithi104 ceph-mon[36624]: pgmap v18081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:25.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:24 smithi081 ceph-mon[31453]: pgmap v18081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:27.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:26 smithi153 ceph-mon[36569]: pgmap v18082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:27.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:26 smithi104 ceph-mon[36624]: pgmap v18082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:27.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:26 smithi081 ceph-mon[31453]: pgmap v18082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:29.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:28 smithi153 ceph-mon[36569]: pgmap v18083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:29.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:28 smithi104 ceph-mon[36624]: pgmap v18083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:29.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:28 smithi081 ceph-mon[31453]: pgmap v18083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:31.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:30 smithi081 ceph-mon[31453]: pgmap v18084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:31.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:30 smithi153 ceph-mon[36569]: pgmap v18084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:31.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:30 smithi104 ceph-mon[36624]: pgmap v18084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:33.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:32 smithi153 ceph-mon[36569]: pgmap v18085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:33.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:32 smithi104 ceph-mon[36624]: pgmap v18085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:33.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:32 smithi081 ceph-mon[31453]: pgmap v18085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:33.578+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:40:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:33.578+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:40:33.588 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:33.578+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:40:35.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:34 smithi153 ceph-mon[36569]: pgmap v18086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:35.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:34 smithi104 ceph-mon[36624]: pgmap v18086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:35.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:34 smithi081 ceph-mon[31453]: pgmap v18086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:37.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:36 smithi153 ceph-mon[36569]: pgmap v18087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:37.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:36 smithi104 ceph-mon[36624]: pgmap v18087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:37.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:36 smithi081 ceph-mon[31453]: pgmap v18087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:39.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:38 smithi153 ceph-mon[36569]: pgmap v18088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:39.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:38 smithi104 ceph-mon[36624]: pgmap v18088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:39.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:38 smithi081 ceph-mon[31453]: pgmap v18088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:41.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:40 smithi153 ceph-mon[36569]: pgmap v18089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:41.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:40 smithi104 ceph-mon[36624]: pgmap v18089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:41.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:40 smithi081 ceph-mon[31453]: pgmap v18089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:43.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:42 smithi153 ceph-mon[36569]: pgmap v18090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:43.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:42 smithi104 ceph-mon[36624]: pgmap v18090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:43.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:42 smithi081 ceph-mon[31453]: pgmap v18090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:45.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:44 smithi153 ceph-mon[36569]: pgmap v18091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:45.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:44 smithi104 ceph-mon[36624]: pgmap v18091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:45.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:44 smithi081 ceph-mon[31453]: pgmap v18091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:47.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:46 smithi153 ceph-mon[36569]: pgmap v18092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:47.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:46 smithi104 ceph-mon[36624]: pgmap v18092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:47.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:46 smithi081 ceph-mon[31453]: pgmap v18092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:48.591+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:40:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:48.591+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:40:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:40:48.591+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:40:49.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:48 smithi153 ceph-mon[36569]: pgmap v18093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:49.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:48 smithi104 ceph-mon[36624]: pgmap v18093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:49.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:48 smithi081 ceph-mon[31453]: pgmap v18093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:51.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:50 smithi153 ceph-mon[36569]: pgmap v18094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:51.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:50 smithi104 ceph-mon[36624]: pgmap v18094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:51.344 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:50 smithi081 ceph-mon[31453]: pgmap v18094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:53.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:52 smithi153 ceph-mon[36569]: pgmap v18095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:53.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:52 smithi104 ceph-mon[36624]: pgmap v18095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:53.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:52 smithi081 ceph-mon[31453]: pgmap v18095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:55.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:54 smithi153 ceph-mon[36569]: pgmap v18096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:55.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:54 smithi104 ceph-mon[36624]: pgmap v18096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:55.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:54 smithi081 ceph-mon[31453]: pgmap v18096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:57.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:56 smithi153 ceph-mon[36569]: pgmap v18097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:57.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:56 smithi104 ceph-mon[36624]: pgmap v18097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:57.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:56 smithi081 ceph-mon[31453]: pgmap v18097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:59.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:40:58 smithi153 ceph-mon[36569]: pgmap v18098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:59.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:40:58 smithi104 ceph-mon[36624]: pgmap v18098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:40:59.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:40:58 smithi081 ceph-mon[31453]: pgmap v18098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:01.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:00 smithi153 ceph-mon[36569]: pgmap v18099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:01.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:00 smithi104 ceph-mon[36624]: pgmap v18099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:01.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:00 smithi081 ceph-mon[31453]: pgmap v18099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:03.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:02 smithi153 ceph-mon[36569]: pgmap v18100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:03.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:02 smithi104 ceph-mon[36624]: pgmap v18100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:03.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:02 smithi081 ceph-mon[31453]: pgmap v18100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:03.607+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:41:03.617 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:03.607+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:41:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:03.607+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:41:05.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:04 smithi153 ceph-mon[36569]: pgmap v18101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:05.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:04 smithi104 ceph-mon[36624]: pgmap v18101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:05.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:04 smithi081 ceph-mon[31453]: pgmap v18101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:07.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:07 smithi153 ceph-mon[36569]: pgmap v18102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:07.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:07 smithi104 ceph-mon[36624]: pgmap v18102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:07.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:07 smithi081 ceph-mon[31453]: pgmap v18102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:09.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:09 smithi153 ceph-mon[36569]: pgmap v18103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:09.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:09 smithi104 ceph-mon[36624]: pgmap v18103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:09.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:09 smithi081 ceph-mon[31453]: pgmap v18103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:11.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:11 smithi153 ceph-mon[36569]: pgmap v18104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:11.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:11 smithi104 ceph-mon[36624]: pgmap v18104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:11.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:11 smithi081 ceph-mon[31453]: pgmap v18104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:13.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:13 smithi153 ceph-mon[36569]: pgmap v18105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:13.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:13 smithi104 ceph-mon[36624]: pgmap v18105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:13.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:13 smithi081 ceph-mon[31453]: pgmap v18105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:15 smithi153 ceph-mon[36569]: pgmap v18106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:15.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:41:15.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:15 smithi104 ceph-mon[36624]: pgmap v18106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:15.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:41:15.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:15 smithi081 ceph-mon[31453]: pgmap v18106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:15.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:41:16.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:41:16.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:41:16.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:16 smithi153 ceph-mon[36569]: pgmap v18107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:16.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:16 smithi153 ceph-mon[36569]: pgmap v18108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:16.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:41:16.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:41:16.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:41:16.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:16 smithi104 ceph-mon[36624]: pgmap v18107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:16.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:16 smithi104 ceph-mon[36624]: pgmap v18108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:16.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:41:16.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:41:16.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:41:16.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:16 smithi081 ceph-mon[31453]: pgmap v18107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:16.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:16 smithi081 ceph-mon[31453]: pgmap v18108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:16.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:41:18.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:18.608+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:41:18.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:18.609+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:41:18.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:18.609+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:41:18.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:18 smithi153 ceph-mon[36569]: pgmap v18109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:18.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:18 smithi104 ceph-mon[36624]: pgmap v18109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:18.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:18 smithi081 ceph-mon[31453]: pgmap v18109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:20.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:20 smithi153 ceph-mon[36569]: pgmap v18110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:20.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:20 smithi104 ceph-mon[36624]: pgmap v18110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:20.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:20 smithi081 ceph-mon[31453]: pgmap v18110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:22.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:22 smithi153 ceph-mon[36569]: pgmap v18111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:22.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:22 smithi104 ceph-mon[36624]: pgmap v18111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:22.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:22 smithi081 ceph-mon[31453]: pgmap v18111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:24.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:24 smithi153 ceph-mon[36569]: pgmap v18112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:24.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:24 smithi104 ceph-mon[36624]: pgmap v18112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:24.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:24 smithi081 ceph-mon[31453]: pgmap v18112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:26.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:26 smithi153 ceph-mon[36569]: pgmap v18113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:26.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:26 smithi104 ceph-mon[36624]: pgmap v18113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:26.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:26 smithi081 ceph-mon[31453]: pgmap v18113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:28.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:28 smithi153 ceph-mon[36569]: pgmap v18114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:28.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:28 smithi104 ceph-mon[36624]: pgmap v18114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:28.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:28 smithi081 ceph-mon[31453]: pgmap v18114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:30.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:30 smithi153 ceph-mon[36569]: pgmap v18115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:30.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:30 smithi104 ceph-mon[36624]: pgmap v18115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:30.915 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:30 smithi081 ceph-mon[31453]: pgmap v18115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:32.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:32 smithi153 ceph-mon[36569]: pgmap v18116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:32.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:32 smithi104 ceph-mon[36624]: pgmap v18116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:32.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:32 smithi081 ceph-mon[31453]: pgmap v18116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:33.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:33.621+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:41:33.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:33.621+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:41:33.631 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:33.621+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:41:34.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:34 smithi104 ceph-mon[36624]: pgmap v18117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:34.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:34 smithi081 ceph-mon[31453]: pgmap v18117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:35.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:34 smithi153 ceph-mon[36569]: pgmap v18117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:36.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:36 smithi104 ceph-mon[36624]: pgmap v18118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:36.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:36 smithi081 ceph-mon[31453]: pgmap v18118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:37.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:36 smithi153 ceph-mon[36569]: pgmap v18118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:38.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:38 smithi104 ceph-mon[36624]: pgmap v18119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:38.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:38 smithi081 ceph-mon[31453]: pgmap v18119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:39.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:38 smithi153 ceph-mon[36569]: pgmap v18119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:39.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:39 smithi081 ceph-mon[31453]: pgmap v18120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:40.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:39 smithi153 ceph-mon[36569]: pgmap v18120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:40.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:39 smithi104 ceph-mon[36624]: pgmap v18120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:42.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:42 smithi153 ceph-mon[36569]: pgmap v18121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:42.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:42 smithi104 ceph-mon[36624]: pgmap v18121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:42.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:42 smithi081 ceph-mon[31453]: pgmap v18121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:44.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:44 smithi153 ceph-mon[36569]: pgmap v18122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:44.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:44 smithi104 ceph-mon[36624]: pgmap v18122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:44.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:44 smithi081 ceph-mon[31453]: pgmap v18122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:46.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:46 smithi153 ceph-mon[36569]: pgmap v18123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:46.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:46 smithi104 ceph-mon[36624]: pgmap v18123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:46.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:46 smithi081 ceph-mon[31453]: pgmap v18123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:48.631+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:41:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:48.631+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:41:48.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:41:48.631+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:41:48.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:48 smithi153 ceph-mon[36569]: pgmap v18124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:48.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:48 smithi104 ceph-mon[36624]: pgmap v18124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:48.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:48 smithi081 ceph-mon[31453]: pgmap v18124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:50.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:50 smithi153 ceph-mon[36569]: pgmap v18125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:50.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:50 smithi104 ceph-mon[36624]: pgmap v18125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:50.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:50 smithi081 ceph-mon[31453]: pgmap v18125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:52.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:52 smithi153 ceph-mon[36569]: pgmap v18126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:52.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:52 smithi104 ceph-mon[36624]: pgmap v18126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:52.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:52 smithi081 ceph-mon[31453]: pgmap v18126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:54.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:54 smithi104 ceph-mon[36624]: pgmap v18127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:54.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:54 smithi081 ceph-mon[31453]: pgmap v18127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:55.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:54 smithi153 ceph-mon[36569]: pgmap v18127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:56.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:56 smithi104 ceph-mon[36624]: pgmap v18128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:56.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:56 smithi081 ceph-mon[31453]: pgmap v18128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:57.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:56 smithi153 ceph-mon[36569]: pgmap v18128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:58.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:58 smithi104 ceph-mon[36624]: pgmap v18129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:58.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:58 smithi081 ceph-mon[31453]: pgmap v18129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:59.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:58 smithi153 ceph-mon[36569]: pgmap v18129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:41:59.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:41:59 smithi081 ceph-mon[31453]: pgmap v18130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:00.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:41:59 smithi153 ceph-mon[36569]: pgmap v18130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:00.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:41:59 smithi104 ceph-mon[36624]: pgmap v18130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:02.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:02 smithi153 ceph-mon[36569]: pgmap v18131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:02.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:02 smithi104 ceph-mon[36624]: pgmap v18131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:02.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:02 smithi081 ceph-mon[31453]: pgmap v18131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:03.645+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:42:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:03.645+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:42:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:03.645+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:42:04.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:04 smithi153 ceph-mon[36569]: pgmap v18132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:04.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:04 smithi104 ceph-mon[36624]: pgmap v18132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:04.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:04 smithi081 ceph-mon[31453]: pgmap v18132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:06.816 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:06 smithi153 ceph-mon[36569]: pgmap v18133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:06.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:06 smithi104 ceph-mon[36624]: pgmap v18133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:06.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:06 smithi081 ceph-mon[31453]: pgmap v18133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:08.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:08 smithi104 ceph-mon[36624]: pgmap v18134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:08.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:08 smithi081 ceph-mon[31453]: pgmap v18134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:09.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:08 smithi153 ceph-mon[36569]: pgmap v18134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:10.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:10 smithi104 ceph-mon[36624]: pgmap v18135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:10.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:10 smithi081 ceph-mon[31453]: pgmap v18135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:11.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:10 smithi153 ceph-mon[36569]: pgmap v18135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:12.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:12 smithi104 ceph-mon[36624]: pgmap v18136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:12.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:12 smithi081 ceph-mon[31453]: pgmap v18136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:13.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:12 smithi153 ceph-mon[36569]: pgmap v18136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:14.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:14 smithi104 ceph-mon[36624]: pgmap v18137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:14.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:14 smithi081 ceph-mon[31453]: pgmap v18137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:15.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:14 smithi153 ceph-mon[36569]: pgmap v18137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:16.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:16 smithi104 ceph-mon[36624]: pgmap v18138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:16.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:42:16.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:16 smithi081 ceph-mon[31453]: pgmap v18138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:16.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:42:17.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:16 smithi153 ceph-mon[36569]: pgmap v18138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:17.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:42:17.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:42:17.916 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:42:17.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:17 smithi081 ceph-mon[31453]: pgmap v18139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:17.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:17 smithi081 ceph-mon[31453]: pgmap v18140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:17.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:42:18.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:42:18.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:42:18.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:17 smithi153 ceph-mon[36569]: pgmap v18139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:18.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:17 smithi153 ceph-mon[36569]: pgmap v18140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:18.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:42:18.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:42:18.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:42:18.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:17 smithi104 ceph-mon[36624]: pgmap v18139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:18.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:17 smithi104 ceph-mon[36624]: pgmap v18140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:18.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:42:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:18.660+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:42:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:18.660+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:42:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:18.661+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:42:20.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:20 smithi081 ceph-mon[31453]: pgmap v18141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:20.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:20 smithi153 ceph-mon[36569]: pgmap v18141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:20.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:20 smithi104 ceph-mon[36624]: pgmap v18141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:22.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:22 smithi081 ceph-mon[31453]: pgmap v18142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:22.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:22 smithi153 ceph-mon[36569]: pgmap v18142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:22.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:22 smithi104 ceph-mon[36624]: pgmap v18142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:24.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:24 smithi081 ceph-mon[31453]: pgmap v18143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:24.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:24 smithi153 ceph-mon[36569]: pgmap v18143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:24.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:24 smithi104 ceph-mon[36624]: pgmap v18143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:26.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:26 smithi081 ceph-mon[31453]: pgmap v18144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:26.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:26 smithi153 ceph-mon[36569]: pgmap v18144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:26.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:26 smithi104 ceph-mon[36624]: pgmap v18144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:28.416 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:28 smithi081 ceph-mon[31453]: pgmap v18145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:28.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:28 smithi153 ceph-mon[36569]: pgmap v18145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:28.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:28 smithi104 ceph-mon[36624]: pgmap v18145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:30.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:30 smithi153 ceph-mon[36569]: pgmap v18146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:30.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:30 smithi104 ceph-mon[36624]: pgmap v18146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:30.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:30 smithi081 ceph-mon[31453]: pgmap v18146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:32.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:32 smithi153 ceph-mon[36569]: pgmap v18147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:32.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:32 smithi104 ceph-mon[36624]: pgmap v18147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:32.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:32 smithi081 ceph-mon[31453]: pgmap v18147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:33.674+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:42:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:33.674+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:42:33.685 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:33.674+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:42:34.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:34 smithi153 ceph-mon[36569]: pgmap v18148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:34.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:34 smithi104 ceph-mon[36624]: pgmap v18148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:34.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:34 smithi081 ceph-mon[31453]: pgmap v18148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:36.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:36 smithi153 ceph-mon[36569]: pgmap v18149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:36.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:36 smithi104 ceph-mon[36624]: pgmap v18149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:36.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:36 smithi081 ceph-mon[31453]: pgmap v18149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:38.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:38 smithi153 ceph-mon[36569]: pgmap v18150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:38.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:38 smithi104 ceph-mon[36624]: pgmap v18150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:38.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:38 smithi081 ceph-mon[31453]: pgmap v18150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:40.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:40 smithi153 ceph-mon[36569]: pgmap v18151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:40.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:40 smithi104 ceph-mon[36624]: pgmap v18151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:40.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:40 smithi081 ceph-mon[31453]: pgmap v18151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:42.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:42 smithi153 ceph-mon[36569]: pgmap v18152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:42.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:42 smithi104 ceph-mon[36624]: pgmap v18152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:42.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:42 smithi081 ceph-mon[31453]: pgmap v18152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:44.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:44 smithi153 ceph-mon[36569]: pgmap v18153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:44.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:44 smithi104 ceph-mon[36624]: pgmap v18153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:44.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:44 smithi081 ceph-mon[31453]: pgmap v18153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:46.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:46 smithi153 ceph-mon[36569]: pgmap v18154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:46.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:46 smithi104 ceph-mon[36624]: pgmap v18154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:46.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:46 smithi081 ceph-mon[31453]: pgmap v18154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:48.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:48 smithi153 ceph-mon[36569]: pgmap v18155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:48.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:48 smithi104 ceph-mon[36624]: pgmap v18155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:48.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:48 smithi081 ceph-mon[31453]: pgmap v18155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:48.676+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:42:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:48.676+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:42:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:42:48.676+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:42:50.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:50 smithi153 ceph-mon[36569]: pgmap v18156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:50.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:50 smithi104 ceph-mon[36624]: pgmap v18156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:50.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:50 smithi081 ceph-mon[31453]: pgmap v18156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:52.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:52 smithi153 ceph-mon[36569]: pgmap v18157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:52.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:52 smithi104 ceph-mon[36624]: pgmap v18157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:52.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:52 smithi081 ceph-mon[31453]: pgmap v18157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:54.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:54 smithi153 ceph-mon[36569]: pgmap v18158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:54.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:54 smithi104 ceph-mon[36624]: pgmap v18158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:54.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:54 smithi081 ceph-mon[31453]: pgmap v18158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:56.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:56 smithi153 ceph-mon[36569]: pgmap v18159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:56.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:56 smithi104 ceph-mon[36624]: pgmap v18159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:56.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:56 smithi081 ceph-mon[31453]: pgmap v18159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:58.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:42:58 smithi153 ceph-mon[36569]: pgmap v18160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:58.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:42:58 smithi104 ceph-mon[36624]: pgmap v18160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:42:58.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:42:58 smithi081 ceph-mon[31453]: pgmap v18160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:00.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:00 smithi153 ceph-mon[36569]: pgmap v18161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:00.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:00 smithi104 ceph-mon[36624]: pgmap v18161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:00.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:00 smithi081 ceph-mon[31453]: pgmap v18161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:02.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:02 smithi153 ceph-mon[36569]: pgmap v18162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:02.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:02 smithi104 ceph-mon[36624]: pgmap v18162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:02.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:02 smithi081 ceph-mon[31453]: pgmap v18162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:03.690+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:43:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:03.690+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:43:03.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:03.690+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:43:04.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:04 smithi153 ceph-mon[36569]: pgmap v18163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:04.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:04 smithi104 ceph-mon[36624]: pgmap v18163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:04.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:04 smithi081 ceph-mon[31453]: pgmap v18163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:06.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:06 smithi153 ceph-mon[36569]: pgmap v18164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:06.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:06 smithi104 ceph-mon[36624]: pgmap v18164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:06.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:06 smithi081 ceph-mon[31453]: pgmap v18164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:08.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:08 smithi153 ceph-mon[36569]: pgmap v18165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:08.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:08 smithi104 ceph-mon[36624]: pgmap v18165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:08.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:08 smithi081 ceph-mon[31453]: pgmap v18165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:10.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:10 smithi153 ceph-mon[36569]: pgmap v18166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:10.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:10 smithi104 ceph-mon[36624]: pgmap v18166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:10.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:10 smithi081 ceph-mon[31453]: pgmap v18166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:12.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:12 smithi153 ceph-mon[36569]: pgmap v18167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:12.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:12 smithi104 ceph-mon[36624]: pgmap v18167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:12.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:12 smithi081 ceph-mon[31453]: pgmap v18167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:14.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:14 smithi153 ceph-mon[36569]: pgmap v18168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:14.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:14 smithi104 ceph-mon[36624]: pgmap v18168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:14.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:14 smithi081 ceph-mon[31453]: pgmap v18168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:16.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:16 smithi153 ceph-mon[36569]: pgmap v18169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:16.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:16 smithi104 ceph-mon[36624]: pgmap v18169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:16.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:16 smithi081 ceph-mon[31453]: pgmap v18169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:18.330+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:43:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:18.330+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:43:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:18.330+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 262 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:43:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:18.330+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 262 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:43:18.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:18 smithi153 ceph-mon[36569]: pgmap v18170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:18.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:43:18.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:18 smithi104 ceph-mon[36624]: pgmap v18170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:18.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:43:18.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:18 smithi081 ceph-mon[31453]: pgmap v18170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:18.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:43:18.715 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:18.705+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:43:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:18.705+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:43:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:18.705+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:43:19.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:43:19.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:43:19.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:43:19.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:43:19.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:43:19.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:43:19.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:43:19.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:43:19.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:43:20.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:20 smithi153 ceph-mon[36569]: pgmap v18171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:20.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:20 smithi153 ceph-mon[36569]: pgmap v18172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:20.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:20 smithi104 ceph-mon[36624]: pgmap v18171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:20.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:20 smithi104 ceph-mon[36624]: pgmap v18172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:20.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:20 smithi081 ceph-mon[31453]: pgmap v18171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:20.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:20 smithi081 ceph-mon[31453]: pgmap v18172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:22.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:22 smithi153 ceph-mon[36569]: pgmap v18173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:22.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:22 smithi104 ceph-mon[36624]: pgmap v18173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:22.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:22 smithi081 ceph-mon[31453]: pgmap v18173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:24.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:24 smithi104 ceph-mon[36624]: pgmap v18174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:24.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:24 smithi081 ceph-mon[31453]: pgmap v18174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:24.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:24 smithi153 ceph-mon[36569]: pgmap v18174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:26.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:26 smithi104 ceph-mon[36624]: pgmap v18175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:26.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:26 smithi081 ceph-mon[31453]: pgmap v18175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:26.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:26 smithi153 ceph-mon[36569]: pgmap v18175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:28.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:28 smithi104 ceph-mon[36624]: pgmap v18176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:28.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:28 smithi081 ceph-mon[31453]: pgmap v18176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:28.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:28 smithi153 ceph-mon[36569]: pgmap v18176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:30.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:30 smithi104 ceph-mon[36624]: pgmap v18177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:30.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:30 smithi081 ceph-mon[31453]: pgmap v18177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:30.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:30 smithi153 ceph-mon[36569]: pgmap v18177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:32.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:32 smithi081 ceph-mon[31453]: pgmap v18178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:32.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:32 smithi153 ceph-mon[36569]: pgmap v18178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:32.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:32 smithi104 ceph-mon[36624]: pgmap v18178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:33.720+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:43:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:33.721+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:43:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:33.721+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:43:34.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:34 smithi081 ceph-mon[31453]: pgmap v18179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:34.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:34 smithi153 ceph-mon[36569]: pgmap v18179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:34.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:34 smithi104 ceph-mon[36624]: pgmap v18179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:36.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:36 smithi081 ceph-mon[31453]: pgmap v18180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:36.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:36 smithi153 ceph-mon[36569]: pgmap v18180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:36.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:36 smithi104 ceph-mon[36624]: pgmap v18180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:38.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:38 smithi081 ceph-mon[31453]: pgmap v18181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:38.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:38 smithi153 ceph-mon[36569]: pgmap v18181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:38.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:38 smithi104 ceph-mon[36624]: pgmap v18181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:40.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:40 smithi081 ceph-mon[31453]: pgmap v18182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:40.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:40 smithi153 ceph-mon[36569]: pgmap v18182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:40.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:40 smithi104 ceph-mon[36624]: pgmap v18182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:42.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:42 smithi081 ceph-mon[31453]: pgmap v18183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:42.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:42 smithi153 ceph-mon[36569]: pgmap v18183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:42.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:42 smithi104 ceph-mon[36624]: pgmap v18183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:44.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:44 smithi081 ceph-mon[31453]: pgmap v18184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:44.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:44 smithi153 ceph-mon[36569]: pgmap v18184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:44.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:44 smithi104 ceph-mon[36624]: pgmap v18184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:46.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:46 smithi081 ceph-mon[31453]: pgmap v18185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:46.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:46 smithi153 ceph-mon[36569]: pgmap v18185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:46.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:46 smithi104 ceph-mon[36624]: pgmap v18185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:48.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:48 smithi081 ceph-mon[31453]: pgmap v18186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:48.725+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:43:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:48.725+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:43:48.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:43:48.725+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:43:48.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:48 smithi153 ceph-mon[36569]: pgmap v18186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:48.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:48 smithi104 ceph-mon[36624]: pgmap v18186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:50.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:50 smithi081 ceph-mon[31453]: pgmap v18187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:50.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:50 smithi153 ceph-mon[36569]: pgmap v18187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:50.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:50 smithi104 ceph-mon[36624]: pgmap v18187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:52.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:52 smithi081 ceph-mon[31453]: pgmap v18188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:52.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:52 smithi153 ceph-mon[36569]: pgmap v18188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:52.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:52 smithi104 ceph-mon[36624]: pgmap v18188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:54.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:54 smithi081 ceph-mon[31453]: pgmap v18189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:54.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:54 smithi153 ceph-mon[36569]: pgmap v18189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:54.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:54 smithi104 ceph-mon[36624]: pgmap v18189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:56.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:56 smithi081 ceph-mon[31453]: pgmap v18190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:56.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:56 smithi153 ceph-mon[36569]: pgmap v18190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:56.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:56 smithi104 ceph-mon[36624]: pgmap v18190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:58.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:43:58 smithi081 ceph-mon[31453]: pgmap v18191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:58.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:43:58 smithi153 ceph-mon[36569]: pgmap v18191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:43:58.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:43:58 smithi104 ceph-mon[36624]: pgmap v18191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:00.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:00 smithi153 ceph-mon[36569]: pgmap v18192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:00.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:00 smithi104 ceph-mon[36624]: pgmap v18192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:00.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:00 smithi081 ceph-mon[31453]: pgmap v18192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:02.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:02 smithi153 ceph-mon[36569]: pgmap v18193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:02.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:02 smithi104 ceph-mon[36624]: pgmap v18193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:02.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:02 smithi081 ceph-mon[31453]: pgmap v18193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:03.755 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:03.741+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:44:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:03.741+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:44:03.756 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:03.741+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:44:04.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:04 smithi153 ceph-mon[36569]: pgmap v18194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:04.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:04 smithi104 ceph-mon[36624]: pgmap v18194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:04.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:04 smithi081 ceph-mon[31453]: pgmap v18194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:06.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:06 smithi153 ceph-mon[36569]: pgmap v18195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:06.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:06 smithi104 ceph-mon[36624]: pgmap v18195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:06.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:06 smithi081 ceph-mon[31453]: pgmap v18195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:08.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:08 smithi153 ceph-mon[36569]: pgmap v18196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:08.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:08 smithi104 ceph-mon[36624]: pgmap v18196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:08.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:08 smithi081 ceph-mon[31453]: pgmap v18196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:10.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:10 smithi153 ceph-mon[36569]: pgmap v18197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:10.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:10 smithi104 ceph-mon[36624]: pgmap v18197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:10.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:10 smithi081 ceph-mon[31453]: pgmap v18197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:12.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:12 smithi153 ceph-mon[36569]: pgmap v18198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:12.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:12 smithi104 ceph-mon[36624]: pgmap v18198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:12.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:12 smithi081 ceph-mon[31453]: pgmap v18198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:14.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:14 smithi153 ceph-mon[36569]: pgmap v18199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:14.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:14 smithi104 ceph-mon[36624]: pgmap v18199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:14.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:14 smithi081 ceph-mon[31453]: pgmap v18199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:16.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:16 smithi153 ceph-mon[36569]: pgmap v18200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:16.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:16 smithi104 ceph-mon[36624]: pgmap v18200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:16.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:16 smithi081 ceph-mon[31453]: pgmap v18200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:18.757+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:44:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:18.757+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:44:18.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:18.757+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:44:18.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:18 smithi153 ceph-mon[36569]: pgmap v18201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:18.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:18 smithi104 ceph-mon[36624]: pgmap v18201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:18.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:18 smithi081 ceph-mon[31453]: pgmap v18201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:19.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:44:19.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:44:19.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: pgmap v18202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:44:20.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:44:20.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:44:20.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: pgmap v18202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:20.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:44:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:44:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:44:20.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: pgmap v18202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:44:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:44:20.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:44:21.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:21 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:44:21.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:21 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:44:21.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:21 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:44:21.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:21 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:44:21.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:21 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:44:21.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:21 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:44:21.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:21 smithi153 ceph-mon[36569]: pgmap v18203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:21.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:21 smithi153 ceph-mon[36569]: pgmap v18204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:21.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:21 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:44:21.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:21 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:44:21.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:21 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:44:21.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:21 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:44:21.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:21 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:44:21.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:21 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:44:21.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:21 smithi104 ceph-mon[36624]: pgmap v18203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:21.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:21 smithi104 ceph-mon[36624]: pgmap v18204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:21.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:21 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:44:21.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:21 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:44:21.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:21 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:44:21.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:21 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:44:21.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:21 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:44:21.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:21 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:44:21.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:21 smithi081 ceph-mon[31453]: pgmap v18203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:21.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:21 smithi081 ceph-mon[31453]: pgmap v18204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:23.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:23 smithi153 ceph-mon[36569]: pgmap v18205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:23.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:23 smithi104 ceph-mon[36624]: pgmap v18205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:23.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:23 smithi081 ceph-mon[31453]: pgmap v18205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:25.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:25 smithi153 ceph-mon[36569]: pgmap v18206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:25.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:25 smithi104 ceph-mon[36624]: pgmap v18206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:25.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:25 smithi081 ceph-mon[31453]: pgmap v18206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:27.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:27 smithi153 ceph-mon[36569]: pgmap v18207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:27.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:27 smithi104 ceph-mon[36624]: pgmap v18207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:27.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:27 smithi081 ceph-mon[31453]: pgmap v18207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:29.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:29 smithi153 ceph-mon[36569]: pgmap v18208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:29.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:29 smithi104 ceph-mon[36624]: pgmap v18208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:29.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:29 smithi081 ceph-mon[31453]: pgmap v18208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:31.317 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:30 smithi153 ceph-mon[36569]: pgmap v18209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:31.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:30 smithi104 ceph-mon[36624]: pgmap v18209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:31.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:30 smithi081 ceph-mon[31453]: pgmap v18209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:33.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:33 smithi153 ceph-mon[36569]: pgmap v18210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:33.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:33 smithi104 ceph-mon[36624]: pgmap v18210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:33.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:33 smithi081 ceph-mon[31453]: pgmap v18210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:33.762+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:44:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:33.762+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:44:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:33.762+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:44:35.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:35 smithi153 ceph-mon[36569]: pgmap v18211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:35.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:35 smithi104 ceph-mon[36624]: pgmap v18211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:35.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:35 smithi081 ceph-mon[31453]: pgmap v18211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:37.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:37 smithi153 ceph-mon[36569]: pgmap v18212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:37.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:37 smithi104 ceph-mon[36624]: pgmap v18212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:37.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:37 smithi081 ceph-mon[31453]: pgmap v18212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:39.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:39 smithi153 ceph-mon[36569]: pgmap v18213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:39.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:39 smithi104 ceph-mon[36624]: pgmap v18213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:39.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:39 smithi081 ceph-mon[31453]: pgmap v18213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:41.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:41 smithi153 ceph-mon[36569]: pgmap v18214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:41.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:41 smithi104 ceph-mon[36624]: pgmap v18214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:41.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:41 smithi081 ceph-mon[31453]: pgmap v18214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:43.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:43 smithi153 ceph-mon[36569]: pgmap v18215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:43.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:43 smithi104 ceph-mon[36624]: pgmap v18215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:43.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:43 smithi081 ceph-mon[31453]: pgmap v18215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:45.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:45 smithi104 ceph-mon[36624]: pgmap v18216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:45.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:45 smithi081 ceph-mon[31453]: pgmap v18216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:45.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:45 smithi153 ceph-mon[36569]: pgmap v18216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:47.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:47 smithi104 ceph-mon[36624]: pgmap v18217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:47.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:47 smithi081 ceph-mon[31453]: pgmap v18217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:47.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:47 smithi153 ceph-mon[36569]: pgmap v18217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:48.777+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:44:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:48.778+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:44:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:44:48.778+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:44:49.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:49 smithi104 ceph-mon[36624]: pgmap v18218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:49.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:49 smithi081 ceph-mon[31453]: pgmap v18218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:49.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:49 smithi153 ceph-mon[36569]: pgmap v18218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:51.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:51 smithi104 ceph-mon[36624]: pgmap v18219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:51.620 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:51 smithi153 ceph-mon[36569]: pgmap v18219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:51.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:51 smithi081 ceph-mon[31453]: pgmap v18219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:53.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:53 smithi104 ceph-mon[36624]: pgmap v18220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:53.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:53 smithi081 ceph-mon[31453]: pgmap v18220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:53.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:53 smithi153 ceph-mon[36569]: pgmap v18220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:55.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:55 smithi081 ceph-mon[31453]: pgmap v18221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:55.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:55 smithi153 ceph-mon[36569]: pgmap v18221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:55.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:55 smithi104 ceph-mon[36624]: pgmap v18221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:57.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:57 smithi081 ceph-mon[31453]: pgmap v18222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:57.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:57 smithi153 ceph-mon[36569]: pgmap v18222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:57.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:57 smithi104 ceph-mon[36624]: pgmap v18222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:59.645 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:44:59 smithi104 ceph-mon[36624]: pgmap v18223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:59.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:44:59 smithi081 ceph-mon[31453]: pgmap v18223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:44:59.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:44:59 smithi153 ceph-mon[36569]: pgmap v18223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:01.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:01 smithi081 ceph-mon[31453]: pgmap v18224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:01.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:01 smithi153 ceph-mon[36569]: pgmap v18224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:01.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:01 smithi104 ceph-mon[36624]: pgmap v18224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:03.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:03 smithi081 ceph-mon[31453]: pgmap v18225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:03.785+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:45:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:03.785+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:45:03.796 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:03.785+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:45:03.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:03 smithi153 ceph-mon[36569]: pgmap v18225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:03.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:03 smithi104 ceph-mon[36624]: pgmap v18225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:05.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:05 smithi081 ceph-mon[31453]: pgmap v18226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:05.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:05 smithi153 ceph-mon[36569]: pgmap v18226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:05.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:05 smithi104 ceph-mon[36624]: pgmap v18226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:07.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:07 smithi081 ceph-mon[31453]: pgmap v18227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:07.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:07 smithi153 ceph-mon[36569]: pgmap v18227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:07.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:07 smithi104 ceph-mon[36624]: pgmap v18227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:09.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:09 smithi081 ceph-mon[31453]: pgmap v18228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:09.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:09 smithi153 ceph-mon[36569]: pgmap v18228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:09.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:09 smithi104 ceph-mon[36624]: pgmap v18228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:11.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:11 smithi081 ceph-mon[31453]: pgmap v18229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:11.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:11 smithi153 ceph-mon[36569]: pgmap v18229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:11.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:11 smithi104 ceph-mon[36624]: pgmap v18229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:13.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:13 smithi081 ceph-mon[31453]: pgmap v18230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:13.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:13 smithi153 ceph-mon[36569]: pgmap v18230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:13.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:13 smithi104 ceph-mon[36624]: pgmap v18230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:15.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:15 smithi081 ceph-mon[31453]: pgmap v18231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:15.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:15 smithi153 ceph-mon[36569]: pgmap v18231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:15.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:15 smithi104 ceph-mon[36624]: pgmap v18231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:17.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:17 smithi081 ceph-mon[31453]: pgmap v18232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:17.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:17 smithi153 ceph-mon[36569]: pgmap v18232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:17.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:17 smithi104 ceph-mon[36624]: pgmap v18232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:18.790+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:45:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:18.790+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:45:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:18.790+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:45:19.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:19 smithi081 ceph-mon[31453]: pgmap v18233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:19.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:19 smithi153 ceph-mon[36569]: pgmap v18233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:19.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:19 smithi104 ceph-mon[36624]: pgmap v18233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:21.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:21 smithi081 ceph-mon[31453]: pgmap v18234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:21.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:45:21.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:21 smithi153 ceph-mon[36569]: pgmap v18234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:21.817 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:45:21.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:21 smithi104 ceph-mon[36624]: pgmap v18234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:21.825 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:45:23.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:23 smithi153 ceph-mon[36569]: pgmap v18235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:23.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:23 smithi104 ceph-mon[36624]: pgmap v18235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:23.898 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:23 smithi081 ceph-mon[31453]: pgmap v18235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:25.542 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:25 smithi104 ceph-mon[36624]: pgmap v18236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:25.543 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:25.543 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:25.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:25 smithi153 ceph-mon[36569]: pgmap v18236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:25.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:25.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:25.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:25 smithi081 ceph-mon[31453]: pgmap v18236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:25.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:25.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:27.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:27 smithi153 ceph-mon[36569]: pgmap v18237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:27.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:27 smithi104 ceph-mon[36624]: pgmap v18237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:27.902 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:27 smithi081 ceph-mon[31453]: pgmap v18237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:29.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:29.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:29.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:45:29.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:45:29.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:29 smithi153 ceph-mon[36569]: pgmap v18238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:29.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:29 smithi153 ceph-mon[36569]: pgmap v18239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:29.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:29.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:29.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:29.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:45:29.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:45:29.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:29 smithi104 ceph-mon[36624]: pgmap v18238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:29.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:29 smithi104 ceph-mon[36624]: pgmap v18239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:29.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:29.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:29.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:29.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:45:29.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:45:29.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:29 smithi081 ceph-mon[31453]: pgmap v18238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:29.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:29 smithi081 ceph-mon[31453]: pgmap v18239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:29.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:45:31.383 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:31 smithi081 ceph-mon[31453]: pgmap v18240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:31.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:31 smithi153 ceph-mon[36569]: pgmap v18240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:31.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:31 smithi104 ceph-mon[36624]: pgmap v18240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:33.352+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:45:33.364 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:33.352+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:45:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:33.352+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 263 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:45:33.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:33.352+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 263 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:45:33.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:33 smithi081 ceph-mon[31453]: pgmap v18241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:33.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:33 smithi153 ceph-mon[36569]: pgmap v18241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:33.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:33 smithi104 ceph-mon[36624]: pgmap v18241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:33.800+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:45:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:33.800+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:45:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:33.800+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:45:35.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:35 smithi081 ceph-mon[31453]: pgmap v18242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:35.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:35 smithi153 ceph-mon[36569]: pgmap v18242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:35.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:35 smithi104 ceph-mon[36624]: pgmap v18242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:37.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:37 smithi081 ceph-mon[31453]: pgmap v18243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:37.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:37 smithi153 ceph-mon[36569]: pgmap v18243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:37.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:37 smithi104 ceph-mon[36624]: pgmap v18243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:39.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:39 smithi081 ceph-mon[31453]: pgmap v18244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:39.567 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:39 smithi153 ceph-mon[36569]: pgmap v18244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:39.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:39 smithi104 ceph-mon[36624]: pgmap v18244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:41.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:41 smithi081 ceph-mon[31453]: pgmap v18245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:41.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:41 smithi153 ceph-mon[36569]: pgmap v18245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:41.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:41 smithi104 ceph-mon[36624]: pgmap v18245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:43.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:43 smithi081 ceph-mon[31453]: pgmap v18246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:43.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:43 smithi153 ceph-mon[36569]: pgmap v18246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:43.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:43 smithi104 ceph-mon[36624]: pgmap v18246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:45.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:45 smithi081 ceph-mon[31453]: pgmap v18247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:45.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:45 smithi153 ceph-mon[36569]: pgmap v18247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:45.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:45 smithi104 ceph-mon[36624]: pgmap v18247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:47.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:47 smithi081 ceph-mon[31453]: pgmap v18248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:47.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:47 smithi153 ceph-mon[36569]: pgmap v18248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:47.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:47 smithi104 ceph-mon[36624]: pgmap v18248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:48.816+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:45:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:48.816+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:45:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:45:48.816+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:45:49.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:49 smithi081 ceph-mon[31453]: pgmap v18249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:49.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:49 smithi153 ceph-mon[36569]: pgmap v18249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:49.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:49 smithi104 ceph-mon[36624]: pgmap v18249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:51.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:51 smithi081 ceph-mon[31453]: pgmap v18250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:51.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:51 smithi153 ceph-mon[36569]: pgmap v18250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:51.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:51 smithi104 ceph-mon[36624]: pgmap v18250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:53.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:53 smithi081 ceph-mon[31453]: pgmap v18251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:53.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:53 smithi153 ceph-mon[36569]: pgmap v18251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:53.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:53 smithi104 ceph-mon[36624]: pgmap v18251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:55.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:55 smithi081 ceph-mon[31453]: pgmap v18252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:55.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:55 smithi153 ceph-mon[36569]: pgmap v18252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:55.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:55 smithi104 ceph-mon[36624]: pgmap v18252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:57.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:57 smithi153 ceph-mon[36569]: pgmap v18253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:57.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:57 smithi104 ceph-mon[36624]: pgmap v18253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:57.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:57 smithi081 ceph-mon[31453]: pgmap v18253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:59.471 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:45:59 smithi081 ceph-mon[31453]: pgmap v18254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:59.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:45:59 smithi153 ceph-mon[36569]: pgmap v18254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:45:59.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:45:59 smithi104 ceph-mon[36624]: pgmap v18254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:01.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:01 smithi153 ceph-mon[36569]: pgmap v18255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:01.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:01 smithi104 ceph-mon[36624]: pgmap v18255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:01.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:01 smithi081 ceph-mon[31453]: pgmap v18255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:03.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:03 smithi081 ceph-mon[31453]: pgmap v18256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:03.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:03 smithi153 ceph-mon[36569]: pgmap v18256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:03.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:03 smithi104 ceph-mon[36624]: pgmap v18256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:03.832+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:46:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:03.832+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:46:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:03.832+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:46:05.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:05 smithi153 ceph-mon[36569]: pgmap v18257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:05.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:05 smithi104 ceph-mon[36624]: pgmap v18257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:05.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:05 smithi081 ceph-mon[31453]: pgmap v18257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:07.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:07 smithi153 ceph-mon[36569]: pgmap v18258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:07.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:07 smithi104 ceph-mon[36624]: pgmap v18258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:07.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:07 smithi081 ceph-mon[31453]: pgmap v18258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:09.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:09 smithi153 ceph-mon[36569]: pgmap v18259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:09.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:09 smithi104 ceph-mon[36624]: pgmap v18259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:09.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:09 smithi081 ceph-mon[31453]: pgmap v18259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:11.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:11 smithi153 ceph-mon[36569]: pgmap v18260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:11.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:11 smithi104 ceph-mon[36624]: pgmap v18260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:11.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:11 smithi081 ceph-mon[31453]: pgmap v18260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:12.184+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 msgr2=0x7f16e80b1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000725) for more than 900000000 us, fault. 2024-04-13T06:46:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:12.184+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006530 tx=0x7f16f4039000 comp rx=0 tx=0).stop 2024-04-13T06:46:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:12.184+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 msgr2=0x7f16e80b1a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T06:46:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:12.184+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T06:46:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:12.184+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T06:46:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:12.185+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T06:46:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:12.186+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f40ba000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T06:46:13.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:13 smithi153 ceph-mon[36569]: pgmap v18261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:13.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:13 smithi104 ceph-mon[36624]: pgmap v18261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:13.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:13 smithi081 ceph-mon[31453]: pgmap v18261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:15.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:15 smithi153 ceph-mon[36569]: pgmap v18262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:15.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:15 smithi104 ceph-mon[36624]: pgmap v18262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:15.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:15 smithi081 ceph-mon[31453]: pgmap v18262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:17.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:17 smithi153 ceph-mon[36569]: pgmap v18263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:17.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:17 smithi104 ceph-mon[36624]: pgmap v18263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:17.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:17 smithi081 ceph-mon[31453]: pgmap v18263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:18.360+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:46:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:18.360+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:46:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:18.361+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 264 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:46:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:18.361+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 264 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:46:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:18.848+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:46:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:18.848+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:46:18.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:18.848+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:46:19.417 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:19 smithi081 ceph-mon[31453]: pgmap v18264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:19.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:19 smithi153 ceph-mon[36569]: pgmap v18264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:19.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:19 smithi104 ceph-mon[36624]: pgmap v18264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:21.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:21 smithi153 ceph-mon[36569]: pgmap v18265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:21.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:21 smithi104 ceph-mon[36624]: pgmap v18265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:21.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:21 smithi081 ceph-mon[31453]: pgmap v18265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:23.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:23 smithi153 ceph-mon[36569]: pgmap v18266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:23.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:23 smithi104 ceph-mon[36624]: pgmap v18266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:23.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:23 smithi081 ceph-mon[31453]: pgmap v18266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:25.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:25 smithi153 ceph-mon[36569]: pgmap v18267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:25.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:25 smithi104 ceph-mon[36624]: pgmap v18267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:25.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:25 smithi081 ceph-mon[31453]: pgmap v18267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:27.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:27 smithi153 ceph-mon[36569]: pgmap v18268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:27.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:27 smithi104 ceph-mon[36624]: pgmap v18268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:27.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:27 smithi081 ceph-mon[31453]: pgmap v18268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:29.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:29 smithi153 ceph-mon[36569]: pgmap v18269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:29.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:46:29.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:29 smithi104 ceph-mon[36624]: pgmap v18269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:29.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:46:29.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:29 smithi081 ceph-mon[31453]: pgmap v18269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:29.651 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:46:31.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:31 smithi153 ceph-mon[36569]: pgmap v18270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:31.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:31 smithi104 ceph-mon[36624]: pgmap v18270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:31.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:31 smithi081 ceph-mon[31453]: pgmap v18270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:33.533 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:33 smithi153 ceph-mon[36569]: pgmap v18271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:33.593 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:33 smithi104 ceph-mon[36624]: pgmap v18271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:33.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:33 smithi081 ceph-mon[31453]: pgmap v18271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:33.848+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:46:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:33.848+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:46:33.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:33.848+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:46:34.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:34.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:34.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:46:34.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:46:34.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:34 smithi081 ceph-mon[31453]: pgmap v18272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:34.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:34 smithi081 ceph-mon[31453]: pgmap v18273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:34.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:34.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:34.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:34.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:46:34.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:46:34.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:34 smithi153 ceph-mon[36569]: pgmap v18272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:34.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:34 smithi153 ceph-mon[36569]: pgmap v18273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:34.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:34.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:34.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:34.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:46:34.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:46:34.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:34 smithi104 ceph-mon[36624]: pgmap v18272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:34.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:34 smithi104 ceph-mon[36624]: pgmap v18273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:34.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:46:36.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:36 smithi153 ceph-mon[36569]: pgmap v18274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:36.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:36 smithi104 ceph-mon[36624]: pgmap v18274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:36.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:36 smithi081 ceph-mon[31453]: pgmap v18274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:38.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:38 smithi153 ceph-mon[36569]: pgmap v18275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:38.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:38 smithi104 ceph-mon[36624]: pgmap v18275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:38.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:38 smithi081 ceph-mon[31453]: pgmap v18275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:40.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:40 smithi153 ceph-mon[36569]: pgmap v18276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:40.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:40 smithi104 ceph-mon[36624]: pgmap v18276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:40.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:40 smithi081 ceph-mon[31453]: pgmap v18276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:42.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:42 smithi153 ceph-mon[36569]: pgmap v18277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:42.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:42 smithi104 ceph-mon[36624]: pgmap v18277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:42.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:42 smithi081 ceph-mon[31453]: pgmap v18277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:44.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:44 smithi153 ceph-mon[36569]: pgmap v18278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:44.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:44 smithi104 ceph-mon[36624]: pgmap v18278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:44.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:44 smithi081 ceph-mon[31453]: pgmap v18278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:46.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:46 smithi153 ceph-mon[36569]: pgmap v18279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:46.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:46 smithi104 ceph-mon[36624]: pgmap v18279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:46.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:46 smithi081 ceph-mon[31453]: pgmap v18279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:48.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:48 smithi153 ceph-mon[36569]: pgmap v18280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:48.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:48 smithi104 ceph-mon[36624]: pgmap v18280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:48.864+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:46:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:48.864+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:46:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:46:48.864+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:46:48.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:48 smithi081 ceph-mon[31453]: pgmap v18280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:50.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:50 smithi153 ceph-mon[36569]: pgmap v18281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:50.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:50 smithi104 ceph-mon[36624]: pgmap v18281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:50.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:50 smithi081 ceph-mon[31453]: pgmap v18281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:52.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:52 smithi153 ceph-mon[36569]: pgmap v18282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:52.896 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:52 smithi104 ceph-mon[36624]: pgmap v18282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:52.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:52 smithi081 ceph-mon[31453]: pgmap v18282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:54.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:54 smithi153 ceph-mon[36569]: pgmap v18283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:54.877 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:54 smithi104 ceph-mon[36624]: pgmap v18283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:54.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:54 smithi081 ceph-mon[31453]: pgmap v18283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:56.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:56 smithi153 ceph-mon[36569]: pgmap v18284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:56.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:56 smithi104 ceph-mon[36624]: pgmap v18284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:56.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:56 smithi081 ceph-mon[31453]: pgmap v18284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:58.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:46:58 smithi153 ceph-mon[36569]: pgmap v18285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:58.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:46:58 smithi104 ceph-mon[36624]: pgmap v18285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:46:58.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:46:58 smithi081 ceph-mon[31453]: pgmap v18285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:00.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:00 smithi153 ceph-mon[36569]: pgmap v18286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:00.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:00 smithi104 ceph-mon[36624]: pgmap v18286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:00.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:00 smithi081 ceph-mon[31453]: pgmap v18286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:02.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:02 smithi153 ceph-mon[36569]: pgmap v18287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:02.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:02 smithi104 ceph-mon[36624]: pgmap v18287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:02.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:02 smithi081 ceph-mon[31453]: pgmap v18287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:03.891 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:03.880+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:47:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:03.880+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:47:03.892 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:03.880+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:47:04.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:04 smithi153 ceph-mon[36569]: pgmap v18288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:04.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:04 smithi104 ceph-mon[36624]: pgmap v18288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:04.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:04 smithi081 ceph-mon[31453]: pgmap v18288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:06.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:06 smithi153 ceph-mon[36569]: pgmap v18289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:06.843 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:06 smithi104 ceph-mon[36624]: pgmap v18289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:06.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:06 smithi081 ceph-mon[31453]: pgmap v18289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:08.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:08 smithi153 ceph-mon[36569]: pgmap v18290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:08.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:08 smithi104 ceph-mon[36624]: pgmap v18290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:08.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:08 smithi081 ceph-mon[31453]: pgmap v18290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:10.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:10 smithi153 ceph-mon[36569]: pgmap v18291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:10.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:10 smithi104 ceph-mon[36624]: pgmap v18291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:10.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:10 smithi081 ceph-mon[31453]: pgmap v18291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:12.818 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:12 smithi153 ceph-mon[36569]: pgmap v18292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:12.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:12 smithi104 ceph-mon[36624]: pgmap v18292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:12.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:12 smithi081 ceph-mon[31453]: pgmap v18292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:14.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:14 smithi104 ceph-mon[36624]: pgmap v18293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:14.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:14 smithi081 ceph-mon[31453]: pgmap v18293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:15.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:14 smithi153 ceph-mon[36569]: pgmap v18293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:16.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:16 smithi104 ceph-mon[36624]: pgmap v18294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:16.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:16 smithi081 ceph-mon[31453]: pgmap v18294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:17.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:16 smithi153 ceph-mon[36569]: pgmap v18294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:18.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:18 smithi104 ceph-mon[36624]: pgmap v18295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:18.894+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:47:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:18.894+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:47:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:18.894+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:47:18.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:18 smithi081 ceph-mon[31453]: pgmap v18295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:19.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:18 smithi153 ceph-mon[36569]: pgmap v18295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:20.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:20 smithi104 ceph-mon[36624]: pgmap v18296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:20 smithi081 ceph-mon[31453]: pgmap v18296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:21.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:20 smithi153 ceph-mon[36569]: pgmap v18296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:22.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:22 smithi104 ceph-mon[36624]: pgmap v18297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:22.870 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:22 smithi153 ceph-mon[36569]: pgmap v18297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:22.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:22 smithi081 ceph-mon[31453]: pgmap v18297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:24.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:24 smithi081 ceph-mon[31453]: pgmap v18298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:25.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:24 smithi153 ceph-mon[36569]: pgmap v18298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:25.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:24 smithi104 ceph-mon[36624]: pgmap v18298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:26.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:26 smithi081 ceph-mon[31453]: pgmap v18299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:27.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:26 smithi153 ceph-mon[36569]: pgmap v18299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:27.155 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:26 smithi104 ceph-mon[36624]: pgmap v18299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:28.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:28 smithi081 ceph-mon[31453]: pgmap v18300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:29.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:28 smithi153 ceph-mon[36569]: pgmap v18300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:29.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:28 smithi104 ceph-mon[36624]: pgmap v18300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:30.896 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:30 smithi104 ceph-mon[36624]: pgmap v18301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:30.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:30 smithi081 ceph-mon[31453]: pgmap v18301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:31.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:30 smithi153 ceph-mon[36569]: pgmap v18301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:32.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:32 smithi081 ceph-mon[31453]: pgmap v18302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:33.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:32 smithi153 ceph-mon[36569]: pgmap v18302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:33.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:32 smithi104 ceph-mon[36624]: pgmap v18302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:33.910+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:47:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:33.910+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:47:33.922 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:33.910+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:47:34.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:34 smithi081 ceph-mon[31453]: pgmap v18303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:34.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:47:35.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:34 smithi153 ceph-mon[36569]: pgmap v18303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:35.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:47:35.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:34 smithi104 ceph-mon[36624]: pgmap v18303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:35.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:47:35.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:47:35.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:47:35.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:47:36.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:47:36.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:47:36.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:47:36.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:47:36.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:47:36.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:47:36.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:36 smithi081 ceph-mon[31453]: pgmap v18304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:36.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:36 smithi081 ceph-mon[31453]: pgmap v18305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:37.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:36 smithi153 ceph-mon[36569]: pgmap v18304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:37.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:36 smithi153 ceph-mon[36569]: pgmap v18305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:37.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:36 smithi104 ceph-mon[36624]: pgmap v18304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:37.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:36 smithi104 ceph-mon[36624]: pgmap v18305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:38.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:38 smithi081 ceph-mon[31453]: pgmap v18306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:39.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:38 smithi153 ceph-mon[36569]: pgmap v18306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:39.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:38 smithi104 ceph-mon[36624]: pgmap v18306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:40.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:40 smithi081 ceph-mon[31453]: pgmap v18307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:41.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:40 smithi153 ceph-mon[36569]: pgmap v18307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:41.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:40 smithi104 ceph-mon[36624]: pgmap v18307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:41.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:41 smithi081 ceph-mon[31453]: pgmap v18308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:42.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:41 smithi153 ceph-mon[36569]: pgmap v18308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:42.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:41 smithi104 ceph-mon[36624]: pgmap v18308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:44.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:43 smithi153 ceph-mon[36569]: pgmap v18309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:44.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:43 smithi104 ceph-mon[36624]: pgmap v18309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:44.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:43 smithi081 ceph-mon[31453]: pgmap v18309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:46.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:45 smithi153 ceph-mon[36569]: pgmap v18310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:46.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:45 smithi104 ceph-mon[36624]: pgmap v18310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:46.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:45 smithi081 ceph-mon[31453]: pgmap v18310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:48.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:47 smithi153 ceph-mon[36569]: pgmap v18311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:48.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:47 smithi104 ceph-mon[36624]: pgmap v18311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:48.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:47 smithi081 ceph-mon[31453]: pgmap v18311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:48.925+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:47:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:48.926+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:47:48.938 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:47:48.926+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:47:50.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:49 smithi153 ceph-mon[36569]: pgmap v18312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:50.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:49 smithi104 ceph-mon[36624]: pgmap v18312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:50.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:49 smithi081 ceph-mon[31453]: pgmap v18312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:52.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:51 smithi153 ceph-mon[36569]: pgmap v18313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:52.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:51 smithi104 ceph-mon[36624]: pgmap v18313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:52.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:51 smithi081 ceph-mon[31453]: pgmap v18313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:54.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:53 smithi153 ceph-mon[36569]: pgmap v18314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:54.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:53 smithi104 ceph-mon[36624]: pgmap v18314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:54.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:53 smithi081 ceph-mon[31453]: pgmap v18314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:56.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:55 smithi153 ceph-mon[36569]: pgmap v18315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:56.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:55 smithi104 ceph-mon[36624]: pgmap v18315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:56.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:55 smithi081 ceph-mon[31453]: pgmap v18315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:58.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:47:58 smithi153 ceph-mon[36569]: pgmap v18316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:58.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:47:58 smithi104 ceph-mon[36624]: pgmap v18316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:47:58.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:47:58 smithi081 ceph-mon[31453]: pgmap v18316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:00.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:00 smithi153 ceph-mon[36569]: pgmap v18317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:00.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:00 smithi104 ceph-mon[36624]: pgmap v18317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:00.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:00 smithi081 ceph-mon[31453]: pgmap v18317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:02.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:02 smithi153 ceph-mon[36569]: pgmap v18318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:02.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:02 smithi104 ceph-mon[36624]: pgmap v18318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:02.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:02 smithi081 ceph-mon[31453]: pgmap v18318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:03.928+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:48:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:03.928+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:48:03.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:03.928+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:48:04.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:04 smithi153 ceph-mon[36569]: pgmap v18319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:04.345 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:04 smithi104 ceph-mon[36624]: pgmap v18319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:04.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:04 smithi081 ceph-mon[31453]: pgmap v18319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:06.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:06 smithi153 ceph-mon[36569]: pgmap v18320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:06.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:06 smithi104 ceph-mon[36624]: pgmap v18320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:06.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:06 smithi081 ceph-mon[31453]: pgmap v18320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:08.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:08 smithi153 ceph-mon[36569]: pgmap v18321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:08.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:08 smithi104 ceph-mon[36624]: pgmap v18321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:08.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:08 smithi081 ceph-mon[31453]: pgmap v18321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:10.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:10 smithi153 ceph-mon[36569]: pgmap v18322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:10.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:10 smithi104 ceph-mon[36624]: pgmap v18322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:10.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:10 smithi081 ceph-mon[31453]: pgmap v18322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:12.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:12 smithi153 ceph-mon[36569]: pgmap v18323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:12.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:12 smithi104 ceph-mon[36624]: pgmap v18323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:12.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:12 smithi081 ceph-mon[31453]: pgmap v18323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:14.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:14 smithi153 ceph-mon[36569]: pgmap v18324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:14.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:14 smithi104 ceph-mon[36624]: pgmap v18324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:14.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:14 smithi081 ceph-mon[31453]: pgmap v18324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:16.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:16 smithi153 ceph-mon[36569]: pgmap v18325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:16.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:16 smithi104 ceph-mon[36624]: pgmap v18325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:16.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:16 smithi081 ceph-mon[31453]: pgmap v18325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:18.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:18 smithi153 ceph-mon[36569]: pgmap v18326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:18.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:18 smithi104 ceph-mon[36624]: pgmap v18326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:18.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:18 smithi081 ceph-mon[31453]: pgmap v18326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:18.944+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:48:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:18.944+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:48:18.957 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:18.944+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:48:20.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:20 smithi104 ceph-mon[36624]: pgmap v18327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:20.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:20 smithi081 ceph-mon[31453]: pgmap v18327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:20.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:20 smithi153 ceph-mon[36569]: pgmap v18327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:22.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:22 smithi081 ceph-mon[31453]: pgmap v18328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:22.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:22 smithi104 ceph-mon[36624]: pgmap v18328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:22.371 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:22 smithi153 ceph-mon[36569]: pgmap v18328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:24.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:24 smithi104 ceph-mon[36624]: pgmap v18329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:24.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:24 smithi081 ceph-mon[31453]: pgmap v18329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:24.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:24 smithi153 ceph-mon[36569]: pgmap v18329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:26.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:26 smithi104 ceph-mon[36624]: pgmap v18330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:26.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:26 smithi081 ceph-mon[31453]: pgmap v18330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:26.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:26 smithi153 ceph-mon[36569]: pgmap v18330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:28.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:28 smithi104 ceph-mon[36624]: pgmap v18331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:28.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:28 smithi081 ceph-mon[31453]: pgmap v18331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:28.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:28 smithi153 ceph-mon[36569]: pgmap v18331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:30.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:30 smithi081 ceph-mon[31453]: pgmap v18332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:30.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:30 smithi153 ceph-mon[36569]: pgmap v18332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:30.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:30 smithi104 ceph-mon[36624]: pgmap v18332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:32.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:32 smithi081 ceph-mon[31453]: pgmap v18333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:32.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:32 smithi153 ceph-mon[36569]: pgmap v18333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:32.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:32 smithi104 ceph-mon[36624]: pgmap v18333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:33.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:48:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:33.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:48:33.973 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:33.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:48:34.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:34 smithi081 ceph-mon[31453]: pgmap v18334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:34.568 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:34 smithi153 ceph-mon[36569]: pgmap v18334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:34.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:34 smithi104 ceph-mon[36624]: pgmap v18334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:36.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:36 smithi153 ceph-mon[36569]: pgmap v18335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:36.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:48:36.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.319 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:36 smithi081 ceph-mon[31453]: pgmap v18335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:36.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:48:36.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.419 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.541 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:36 smithi104 ceph-mon[36624]: pgmap v18335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:36.541 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:48:36.541 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.542 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.542 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:36.542 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:38.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:38 smithi081 ceph-mon[31453]: pgmap v18336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:38.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:38.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:38.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:38.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:39.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:38 smithi153 ceph-mon[36569]: pgmap v18336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:39.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:39.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:39.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:39.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:39.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:38 smithi104 ceph-mon[36624]: pgmap v18336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:39.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:39.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:39.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:39.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:40.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:40 smithi081 ceph-mon[31453]: pgmap v18337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:40.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:40.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:41.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:40 smithi153 ceph-mon[36569]: pgmap v18337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:41.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:41.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:41.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:40 smithi104 ceph-mon[36624]: pgmap v18337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:41.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:41.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:41.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:48:41.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:48:41.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:42.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:48:42.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:48:42.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:42.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:48:42.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:48:42.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:48:42.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:42 smithi081 ceph-mon[31453]: pgmap v18338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:42.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:42 smithi081 ceph-mon[31453]: pgmap v18339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:42.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:42 smithi081 ceph-mon[31453]: pgmap v18340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:43.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:42 smithi153 ceph-mon[36569]: pgmap v18338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:43.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:42 smithi153 ceph-mon[36569]: pgmap v18339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:43.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:42 smithi153 ceph-mon[36569]: pgmap v18340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:43.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:42 smithi104 ceph-mon[36624]: pgmap v18338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:43.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:42 smithi104 ceph-mon[36624]: pgmap v18339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:43.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:42 smithi104 ceph-mon[36624]: pgmap v18340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:44.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:44 smithi081 ceph-mon[31453]: pgmap v18341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:45.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:44 smithi153 ceph-mon[36569]: pgmap v18341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:45.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:44 smithi104 ceph-mon[36624]: pgmap v18341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:46.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:46 smithi081 ceph-mon[31453]: pgmap v18342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:47.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:46 smithi153 ceph-mon[36569]: pgmap v18342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:47.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:46 smithi104 ceph-mon[36624]: pgmap v18342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:48.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:48 smithi081 ceph-mon[31453]: pgmap v18343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:48.975+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:48:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:48.975+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:48:48.988 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:48:48.975+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:48:49.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:48 smithi153 ceph-mon[36569]: pgmap v18343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:49.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:48 smithi104 ceph-mon[36624]: pgmap v18343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:50.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:50 smithi081 ceph-mon[31453]: pgmap v18344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:51.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:50 smithi153 ceph-mon[36569]: pgmap v18344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:51.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:50 smithi104 ceph-mon[36624]: pgmap v18344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:52.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:52 smithi081 ceph-mon[31453]: pgmap v18345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:53.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:52 smithi153 ceph-mon[36569]: pgmap v18345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:53.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:52 smithi104 ceph-mon[36624]: pgmap v18345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:55.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:54 smithi153 ceph-mon[36569]: pgmap v18346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:55.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:54 smithi104 ceph-mon[36624]: pgmap v18346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:55.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:54 smithi081 ceph-mon[31453]: pgmap v18346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:56.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:55 smithi153 ceph-mon[36569]: pgmap v18347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:56.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:55 smithi104 ceph-mon[36624]: pgmap v18347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:56.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:55 smithi081 ceph-mon[31453]: pgmap v18347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:58.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:48:58 smithi081 ceph-mon[31453]: pgmap v18348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:59.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:48:58 smithi153 ceph-mon[36569]: pgmap v18348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:48:59.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:48:58 smithi104 ceph-mon[36624]: pgmap v18348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:00.896 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:00 smithi104 ceph-mon[36624]: pgmap v18349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:00.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:00 smithi081 ceph-mon[31453]: pgmap v18349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:01.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:00 smithi153 ceph-mon[36569]: pgmap v18349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:02.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:02 smithi081 ceph-mon[31453]: pgmap v18350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:03.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:02 smithi153 ceph-mon[36569]: pgmap v18350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:03.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:02 smithi104 ceph-mon[36624]: pgmap v18350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:03.985+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:49:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:03.985+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:49:03.999 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:03.985+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:49:04.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:04 smithi081 ceph-mon[31453]: pgmap v18351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:05.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:04 smithi153 ceph-mon[36569]: pgmap v18351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:05.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:04 smithi104 ceph-mon[36624]: pgmap v18351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:06.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:06 smithi081 ceph-mon[31453]: pgmap v18352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:07.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:06 smithi153 ceph-mon[36569]: pgmap v18352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:07.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:06 smithi104 ceph-mon[36624]: pgmap v18352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:08.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:08 smithi081 ceph-mon[31453]: pgmap v18353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:09.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:08 smithi153 ceph-mon[36569]: pgmap v18353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:09.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:08 smithi104 ceph-mon[36624]: pgmap v18353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:10.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:10 smithi081 ceph-mon[31453]: pgmap v18354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:11.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:10 smithi153 ceph-mon[36569]: pgmap v18354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:11.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:10 smithi104 ceph-mon[36624]: pgmap v18354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:13.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:12 smithi153 ceph-mon[36569]: pgmap v18355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:13.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:12 smithi104 ceph-mon[36624]: pgmap v18355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:13.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:12 smithi081 ceph-mon[31453]: pgmap v18355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:14.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:13 smithi153 ceph-mon[36569]: pgmap v18356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:14.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:13 smithi104 ceph-mon[36624]: pgmap v18356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:14.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:13 smithi081 ceph-mon[31453]: pgmap v18356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:16.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:16 smithi081 ceph-mon[31453]: pgmap v18357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:17.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:16 smithi153 ceph-mon[36569]: pgmap v18357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:17.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:16 smithi104 ceph-mon[36624]: pgmap v18357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:18.403 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:18.391+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:49:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:18.391+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:49:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:18.392+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 265 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:49:18.404 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:18.392+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 265 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:49:18.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:18 smithi081 ceph-mon[31453]: pgmap v18358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:19.002+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:49:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:19.002+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:49:19.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:19.002+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:49:19.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:18 smithi153 ceph-mon[36569]: pgmap v18358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:19.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:18 smithi104 ceph-mon[36624]: pgmap v18358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:20.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:20 smithi081 ceph-mon[31453]: pgmap v18359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:21.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:20 smithi153 ceph-mon[36569]: pgmap v18359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:21.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:20 smithi104 ceph-mon[36624]: pgmap v18359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:22.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:22 smithi081 ceph-mon[31453]: pgmap v18360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:23.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:22 smithi153 ceph-mon[36569]: pgmap v18360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:23.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:22 smithi104 ceph-mon[36624]: pgmap v18360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:24.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:24 smithi081 ceph-mon[31453]: pgmap v18361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:25.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:24 smithi153 ceph-mon[36569]: pgmap v18361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:25.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:24 smithi104 ceph-mon[36624]: pgmap v18361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:27.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:26 smithi153 ceph-mon[36569]: pgmap v18362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:27.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:26 smithi104 ceph-mon[36624]: pgmap v18362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:27.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:26 smithi081 ceph-mon[31453]: pgmap v18362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:29.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:28 smithi153 ceph-mon[36569]: pgmap v18363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:29.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:28 smithi104 ceph-mon[36624]: pgmap v18363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:29.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:28 smithi081 ceph-mon[31453]: pgmap v18363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:30.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:29 smithi153 ceph-mon[36569]: pgmap v18364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:30.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:29 smithi081 ceph-mon[31453]: pgmap v18364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:30.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:29 smithi104 ceph-mon[36624]: pgmap v18364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:32.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:32 smithi081 ceph-mon[31453]: pgmap v18365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:33.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:32 smithi153 ceph-mon[36569]: pgmap v18365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:33.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:32 smithi104 ceph-mon[36624]: pgmap v18365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:34.012+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:49:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:34.013+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:49:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:34.013+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:49:34.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:34 smithi081 ceph-mon[31453]: pgmap v18366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:35.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:34 smithi153 ceph-mon[36569]: pgmap v18366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:35.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:34 smithi104 ceph-mon[36624]: pgmap v18366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:36.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:36 smithi081 ceph-mon[31453]: pgmap v18367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:37.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:36 smithi153 ceph-mon[36569]: pgmap v18367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:37.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:36 smithi104 ceph-mon[36624]: pgmap v18367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:39.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:38 smithi153 ceph-mon[36569]: pgmap v18368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:39.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:38 smithi104 ceph-mon[36624]: pgmap v18368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:39.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:38 smithi081 ceph-mon[31453]: pgmap v18368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:41.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:40 smithi153 ceph-mon[36569]: pgmap v18369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:41.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:40 smithi104 ceph-mon[36624]: pgmap v18369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:41.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:40 smithi081 ceph-mon[31453]: pgmap v18369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:43.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:42 smithi153 ceph-mon[36569]: pgmap v18370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:43.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:49:43.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:43.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:43.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:42 smithi104 ceph-mon[36624]: pgmap v18370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:43.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:49:43.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:43.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:43.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:42 smithi081 ceph-mon[31453]: pgmap v18370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:43.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:49:43.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:43.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:44.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:49:44.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:49:44.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:43 smithi153 ceph-mon[36569]: pgmap v18371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:44.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:43 smithi153 ceph-mon[36569]: pgmap v18372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:44.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:44.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:49:44.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:49:44.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:43 smithi104 ceph-mon[36624]: pgmap v18371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:44.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:43 smithi104 ceph-mon[36624]: pgmap v18372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:44.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:44.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:49:44.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:49:44.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:43 smithi081 ceph-mon[31453]: pgmap v18371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:44.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:43 smithi081 ceph-mon[31453]: pgmap v18372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:44.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:49:46.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:46 smithi153 ceph-mon[36569]: pgmap v18373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:46.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:46 smithi104 ceph-mon[36624]: pgmap v18373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:46.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:46 smithi081 ceph-mon[31453]: pgmap v18373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:48.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:48 smithi153 ceph-mon[36569]: pgmap v18374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:48.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:48 smithi104 ceph-mon[36624]: pgmap v18374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:48.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:48 smithi081 ceph-mon[31453]: pgmap v18374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:49.027+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:49:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:49.027+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:49:49.041 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:49:49.027+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:49:50.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:50 smithi153 ceph-mon[36569]: pgmap v18375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:50.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:50 smithi104 ceph-mon[36624]: pgmap v18375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:50.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:50 smithi081 ceph-mon[31453]: pgmap v18375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:52.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:52 smithi153 ceph-mon[36569]: pgmap v18376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:52.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:52 smithi104 ceph-mon[36624]: pgmap v18376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:52.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:52 smithi081 ceph-mon[31453]: pgmap v18376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:54.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:54 smithi153 ceph-mon[36569]: pgmap v18377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:54.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:54 smithi104 ceph-mon[36624]: pgmap v18377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:54.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:54 smithi081 ceph-mon[31453]: pgmap v18377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:56.418 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:56 smithi081 ceph-mon[31453]: pgmap v18378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:56.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:56 smithi153 ceph-mon[36569]: pgmap v18378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:56.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:56 smithi104 ceph-mon[36624]: pgmap v18378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:58.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:49:58 smithi153 ceph-mon[36569]: pgmap v18379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:58.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:49:58 smithi104 ceph-mon[36624]: pgmap v18379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:49:58.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:49:58 smithi081 ceph-mon[31453]: pgmap v18379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:00.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:00 smithi153 ceph-mon[36569]: pgmap v18380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:00.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:50:00.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:00 smithi104 ceph-mon[36624]: pgmap v18380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:00.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:50:00.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:00 smithi081 ceph-mon[31453]: pgmap v18380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:00.679 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T06:50:02.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:02 smithi153 ceph-mon[36569]: pgmap v18381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:02.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:02 smithi104 ceph-mon[36624]: pgmap v18381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:02.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:02 smithi081 ceph-mon[31453]: pgmap v18381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:04.035+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:50:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:04.035+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:50:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:04.035+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:50:04.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:04 smithi153 ceph-mon[36569]: pgmap v18382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:04.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:04 smithi104 ceph-mon[36624]: pgmap v18382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:04.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:04 smithi081 ceph-mon[31453]: pgmap v18382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:06.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:06 smithi153 ceph-mon[36569]: pgmap v18383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:06.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:06 smithi104 ceph-mon[36624]: pgmap v18383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:06.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:06 smithi081 ceph-mon[31453]: pgmap v18383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:08.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:08 smithi153 ceph-mon[36569]: pgmap v18384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:08.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:08 smithi104 ceph-mon[36624]: pgmap v18384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:08.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:08 smithi081 ceph-mon[31453]: pgmap v18384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:10.569 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:10 smithi153 ceph-mon[36569]: pgmap v18385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:10.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:10 smithi104 ceph-mon[36624]: pgmap v18385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:10.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:10 smithi081 ceph-mon[31453]: pgmap v18385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:12.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:12 smithi104 ceph-mon[36624]: pgmap v18386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:12.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:12 smithi081 ceph-mon[31453]: pgmap v18386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:12.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:12 smithi153 ceph-mon[36569]: pgmap v18386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:14.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:14 smithi104 ceph-mon[36624]: pgmap v18387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:14.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:14 smithi081 ceph-mon[31453]: pgmap v18387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:14.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:14 smithi153 ceph-mon[36569]: pgmap v18387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:16.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:16 smithi104 ceph-mon[36624]: pgmap v18388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:16.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:16 smithi081 ceph-mon[31453]: pgmap v18388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:16.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:16 smithi153 ceph-mon[36569]: pgmap v18388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:18.595 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:18 smithi104 ceph-mon[36624]: pgmap v18389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:18.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:18 smithi081 ceph-mon[31453]: pgmap v18389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:18.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:18 smithi153 ceph-mon[36569]: pgmap v18389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:19.050+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:50:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:19.050+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:50:19.063 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:19.050+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:50:20.594 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:20 smithi104 ceph-mon[36624]: pgmap v18390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:20.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:20 smithi081 ceph-mon[31453]: pgmap v18390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:20.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:20 smithi153 ceph-mon[36569]: pgmap v18390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:22.621 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:22 smithi153 ceph-mon[36569]: pgmap v18391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:22.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:22 smithi081 ceph-mon[31453]: pgmap v18391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:22.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:22 smithi104 ceph-mon[36624]: pgmap v18391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:24.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:24 smithi081 ceph-mon[31453]: pgmap v18392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:24.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:24 smithi153 ceph-mon[36569]: pgmap v18392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:24.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:24 smithi104 ceph-mon[36624]: pgmap v18392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:26.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:26 smithi081 ceph-mon[31453]: pgmap v18393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:26.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:26 smithi153 ceph-mon[36569]: pgmap v18393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:26.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:26 smithi104 ceph-mon[36624]: pgmap v18393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:28.675 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:28 smithi081 ceph-mon[31453]: pgmap v18394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:28.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:28 smithi153 ceph-mon[36569]: pgmap v18394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:28.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:28 smithi104 ceph-mon[36624]: pgmap v18394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:30.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:30 smithi081 ceph-mon[31453]: pgmap v18395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:30.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:30 smithi153 ceph-mon[36569]: pgmap v18395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:30.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:30 smithi104 ceph-mon[36624]: pgmap v18395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:32.647 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:32 smithi104 ceph-mon[36624]: pgmap v18396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:32.668 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:32 smithi081 ceph-mon[31453]: pgmap v18396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:32.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:32 smithi153 ceph-mon[36569]: pgmap v18396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:34.066+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:50:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:34.066+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:50:34.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:34.066+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:50:34.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:34 smithi081 ceph-mon[31453]: pgmap v18397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:34.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:34 smithi153 ceph-mon[36569]: pgmap v18397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:34.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:34 smithi104 ceph-mon[36624]: pgmap v18397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:36.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:36 smithi081 ceph-mon[31453]: pgmap v18398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:36.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:36 smithi153 ceph-mon[36569]: pgmap v18398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:36.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:36 smithi104 ceph-mon[36624]: pgmap v18398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:38.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:38 smithi081 ceph-mon[31453]: pgmap v18399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:38.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:38 smithi153 ceph-mon[36569]: pgmap v18399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:38.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:38 smithi104 ceph-mon[36624]: pgmap v18399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:40.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:40 smithi081 ceph-mon[31453]: pgmap v18400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:40.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:40 smithi153 ceph-mon[36569]: pgmap v18400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:40.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:40 smithi104 ceph-mon[36624]: pgmap v18400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:42.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:42 smithi081 ceph-mon[31453]: pgmap v18401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:42.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:42 smithi153 ceph-mon[36569]: pgmap v18401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:42.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:42 smithi104 ceph-mon[36624]: pgmap v18401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:44.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:44 smithi081 ceph-mon[31453]: pgmap v18402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:44.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:50:44.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:44 smithi153 ceph-mon[36569]: pgmap v18402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:44.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:50:44.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:44 smithi104 ceph-mon[36624]: pgmap v18402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:44.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:50:45.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:50:45.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:50:45.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:50:45.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:50:45.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:50:45.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:50:45.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:50:45.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:50:45.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:50:46.672 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:46 smithi081 ceph-mon[31453]: pgmap v18403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:46.672 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:46 smithi081 ceph-mon[31453]: pgmap v18404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:46.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:46 smithi153 ceph-mon[36569]: pgmap v18403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:46.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:46 smithi153 ceph-mon[36569]: pgmap v18404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:46.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:46 smithi104 ceph-mon[36624]: pgmap v18403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:46.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:46 smithi104 ceph-mon[36624]: pgmap v18404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:48.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:48 smithi153 ceph-mon[36569]: pgmap v18405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:48.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:48 smithi104 ceph-mon[36624]: pgmap v18405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:48.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:48 smithi081 ceph-mon[31453]: pgmap v18405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:49.081+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:50:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:49.081+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:50:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:50:49.081+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:50:50.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:50 smithi153 ceph-mon[36569]: pgmap v18406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:50.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:50 smithi104 ceph-mon[36624]: pgmap v18406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:50.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:50 smithi081 ceph-mon[31453]: pgmap v18406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:52.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:52 smithi153 ceph-mon[36569]: pgmap v18407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:52.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:52 smithi104 ceph-mon[36624]: pgmap v18407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:52.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:52 smithi081 ceph-mon[31453]: pgmap v18407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:54.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:54 smithi153 ceph-mon[36569]: pgmap v18408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:54.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:54 smithi104 ceph-mon[36624]: pgmap v18408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:54.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:54 smithi081 ceph-mon[31453]: pgmap v18408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:56.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:56 smithi153 ceph-mon[36569]: pgmap v18409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:56.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:56 smithi104 ceph-mon[36624]: pgmap v18409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:56.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:56 smithi081 ceph-mon[31453]: pgmap v18409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:58.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:50:58 smithi153 ceph-mon[36569]: pgmap v18410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:58.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:50:58 smithi104 ceph-mon[36624]: pgmap v18410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:50:58.918 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:50:58 smithi081 ceph-mon[31453]: pgmap v18410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:00.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:00 smithi153 ceph-mon[36569]: pgmap v18411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:00.824 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:00 smithi081 ceph-mon[31453]: pgmap v18411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:00.844 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:00 smithi104 ceph-mon[36624]: pgmap v18411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:02.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:02 smithi153 ceph-mon[36569]: pgmap v18412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:02.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:02 smithi104 ceph-mon[36624]: pgmap v18412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:02.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:02 smithi081 ceph-mon[31453]: pgmap v18412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:04.089+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:51:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:04.089+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:51:04.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:04.089+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:51:04.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:04 smithi153 ceph-mon[36569]: pgmap v18413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:04.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:04 smithi104 ceph-mon[36624]: pgmap v18413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:04.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:04 smithi081 ceph-mon[31453]: pgmap v18413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:06.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:06 smithi153 ceph-mon[36569]: pgmap v18414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:06.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:06 smithi104 ceph-mon[36624]: pgmap v18414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:06.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:06 smithi081 ceph-mon[31453]: pgmap v18414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:08.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:08 smithi153 ceph-mon[36569]: pgmap v18415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:08.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:08 smithi104 ceph-mon[36624]: pgmap v18415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:08.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:08 smithi081 ceph-mon[31453]: pgmap v18415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:10.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:10 smithi153 ceph-mon[36569]: pgmap v18416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:10.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:10 smithi104 ceph-mon[36624]: pgmap v18416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:10.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:10 smithi081 ceph-mon[31453]: pgmap v18416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:12.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:12 smithi153 ceph-mon[36569]: pgmap v18417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:12.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:12 smithi104 ceph-mon[36624]: pgmap v18417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:12.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:12 smithi081 ceph-mon[31453]: pgmap v18417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:14.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:14 smithi153 ceph-mon[36569]: pgmap v18418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:14.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:14 smithi104 ceph-mon[36624]: pgmap v18418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:14.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:14 smithi081 ceph-mon[31453]: pgmap v18418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:16.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:16 smithi153 ceph-mon[36569]: pgmap v18419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:16.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:16 smithi104 ceph-mon[36624]: pgmap v18419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:16.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:16 smithi081 ceph-mon[31453]: pgmap v18419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:18.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:18 smithi153 ceph-mon[36569]: pgmap v18420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:18.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:18 smithi104 ceph-mon[36624]: pgmap v18420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:18.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:18 smithi081 ceph-mon[31453]: pgmap v18420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:19.104+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:51:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:19.104+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:51:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:19.104+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:51:20.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:20 smithi153 ceph-mon[36569]: pgmap v18421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:20.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:20 smithi104 ceph-mon[36624]: pgmap v18421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:20.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:20 smithi081 ceph-mon[31453]: pgmap v18421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:22.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:22 smithi153 ceph-mon[36569]: pgmap v18422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:22.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:22 smithi104 ceph-mon[36624]: pgmap v18422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:22.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:22 smithi081 ceph-mon[31453]: pgmap v18422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:24.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:24 smithi153 ceph-mon[36569]: pgmap v18423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:24.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:24 smithi104 ceph-mon[36624]: pgmap v18423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:24.922 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:24 smithi081 ceph-mon[31453]: pgmap v18423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:26.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:26 smithi153 ceph-mon[36569]: pgmap v18424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:26.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:26 smithi104 ceph-mon[36624]: pgmap v18424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:26.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:26 smithi081 ceph-mon[31453]: pgmap v18424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:28.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:28 smithi153 ceph-mon[36569]: pgmap v18425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:28.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:28 smithi104 ceph-mon[36624]: pgmap v18425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:28.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:28 smithi081 ceph-mon[31453]: pgmap v18425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:30.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:30 smithi081 ceph-mon[31453]: pgmap v18426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:30.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:30 smithi153 ceph-mon[36569]: pgmap v18426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:30.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:30 smithi104 ceph-mon[36624]: pgmap v18426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:32.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:32 smithi153 ceph-mon[36569]: pgmap v18427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:32.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:32 smithi104 ceph-mon[36624]: pgmap v18427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:32.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:32 smithi081 ceph-mon[31453]: pgmap v18427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:34.132 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:34.118+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:51:34.132 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:34.118+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:51:34.132 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:34.118+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:51:34.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:34 smithi153 ceph-mon[36569]: pgmap v18428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:34.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:34 smithi104 ceph-mon[36624]: pgmap v18428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:34.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:34 smithi081 ceph-mon[31453]: pgmap v18428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:36.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:36 smithi153 ceph-mon[36569]: pgmap v18429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:36.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:36 smithi104 ceph-mon[36624]: pgmap v18429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:36.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:36 smithi081 ceph-mon[31453]: pgmap v18429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:38.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:38 smithi153 ceph-mon[36569]: pgmap v18430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:38.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:38 smithi104 ceph-mon[36624]: pgmap v18430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:38.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:38 smithi081 ceph-mon[31453]: pgmap v18430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:40.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:40 smithi104 ceph-mon[36624]: pgmap v18431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:40.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:40 smithi081 ceph-mon[31453]: pgmap v18431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:41.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:40 smithi153 ceph-mon[36569]: pgmap v18431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:42.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:42 smithi104 ceph-mon[36624]: pgmap v18432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:42.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:42 smithi081 ceph-mon[31453]: pgmap v18432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:43.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:42 smithi153 ceph-mon[36569]: pgmap v18432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:44.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:44 smithi104 ceph-mon[36624]: pgmap v18433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:44.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:44 smithi081 ceph-mon[31453]: pgmap v18433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:45.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:44 smithi153 ceph-mon[36569]: pgmap v18433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:45.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:51:45.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:51:46.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:51:46.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:46 smithi104 ceph-mon[36624]: pgmap v18434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:46.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:51:46.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:51:46.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:51:46.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:46 smithi081 ceph-mon[31453]: pgmap v18434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:46.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:51:46.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:51:46.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:51:47.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:46 smithi153 ceph-mon[36569]: pgmap v18434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:47.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:51:47.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:51:47.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:51:47.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:47 smithi081 ceph-mon[31453]: pgmap v18435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:47.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:47 smithi081 ceph-mon[31453]: pgmap v18436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:48.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:47 smithi153 ceph-mon[36569]: pgmap v18435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:48.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:47 smithi153 ceph-mon[36569]: pgmap v18436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:48.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:47 smithi104 ceph-mon[36624]: pgmap v18435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:48.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:47 smithi104 ceph-mon[36624]: pgmap v18436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:49.134+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:51:49.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:49.134+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:51:49.148 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:51:49.134+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:51:49.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:49 smithi081 ceph-mon[31453]: pgmap v18437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:50.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:49 smithi153 ceph-mon[36569]: pgmap v18437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:50.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:49 smithi104 ceph-mon[36624]: pgmap v18437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:51.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:51 smithi081 ceph-mon[31453]: pgmap v18438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:52.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:51 smithi153 ceph-mon[36569]: pgmap v18438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:52.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:51 smithi104 ceph-mon[36624]: pgmap v18438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:53.872 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:53 smithi153 ceph-mon[36569]: pgmap v18439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:53.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:53 smithi081 ceph-mon[31453]: pgmap v18439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:54.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:53 smithi104 ceph-mon[36624]: pgmap v18439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:55.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:55 smithi081 ceph-mon[31453]: pgmap v18440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:56.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:55 smithi153 ceph-mon[36569]: pgmap v18440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:56.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:55 smithi104 ceph-mon[36624]: pgmap v18440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:57.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:57 smithi081 ceph-mon[31453]: pgmap v18441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:58.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:57 smithi153 ceph-mon[36569]: pgmap v18441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:58.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:57 smithi104 ceph-mon[36624]: pgmap v18441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:51:59.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:51:59 smithi081 ceph-mon[31453]: pgmap v18442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:00.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:51:59 smithi153 ceph-mon[36569]: pgmap v18442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:00.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:51:59 smithi104 ceph-mon[36624]: pgmap v18442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:01.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:01 smithi081 ceph-mon[31453]: pgmap v18443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:02.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:01 smithi153 ceph-mon[36569]: pgmap v18443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:02.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:01 smithi104 ceph-mon[36624]: pgmap v18443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:03.898 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:03 smithi104 ceph-mon[36624]: pgmap v18444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:03.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:03 smithi081 ceph-mon[31453]: pgmap v18444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:04.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:03 smithi153 ceph-mon[36569]: pgmap v18444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:04.162 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:04.149+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:52:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:04.149+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:52:04.163 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:04.149+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:52:05.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:05 smithi081 ceph-mon[31453]: pgmap v18445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:06.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:05 smithi153 ceph-mon[36569]: pgmap v18445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:06.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:05 smithi104 ceph-mon[36624]: pgmap v18445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:07.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:07 smithi081 ceph-mon[31453]: pgmap v18446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:08.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:07 smithi153 ceph-mon[36569]: pgmap v18446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:08.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:07 smithi104 ceph-mon[36624]: pgmap v18446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:09.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:09 smithi081 ceph-mon[31453]: pgmap v18447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:10.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:09 smithi153 ceph-mon[36569]: pgmap v18447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:10.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:09 smithi104 ceph-mon[36624]: pgmap v18447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:12.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:11 smithi153 ceph-mon[36569]: pgmap v18448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:12.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:11 smithi104 ceph-mon[36624]: pgmap v18448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:12.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:11 smithi081 ceph-mon[31453]: pgmap v18448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:14.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:13 smithi153 ceph-mon[36569]: pgmap v18449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:14.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:13 smithi104 ceph-mon[36624]: pgmap v18449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:14.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:13 smithi081 ceph-mon[31453]: pgmap v18449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:16.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:15 smithi153 ceph-mon[36569]: pgmap v18450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:16.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:15 smithi104 ceph-mon[36624]: pgmap v18450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:16.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:15 smithi081 ceph-mon[31453]: pgmap v18450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:18.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:17 smithi153 ceph-mon[36569]: pgmap v18451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:18.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:17 smithi104 ceph-mon[36624]: pgmap v18451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:18.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:17 smithi081 ceph-mon[31453]: pgmap v18451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:18.421+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:52:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:18.421+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:52:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:18.421+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 266 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:52:18.434 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:18.421+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 266 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:52:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:19.153+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:52:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:19.154+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:52:19.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:19.154+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:52:20.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:19 smithi153 ceph-mon[36569]: pgmap v18452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:20.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:19 smithi104 ceph-mon[36624]: pgmap v18452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:20.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:19 smithi081 ceph-mon[31453]: pgmap v18452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:22.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:21 smithi153 ceph-mon[36569]: pgmap v18453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:22.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:21 smithi104 ceph-mon[36624]: pgmap v18453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:22.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:21 smithi081 ceph-mon[31453]: pgmap v18453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:24.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:23 smithi153 ceph-mon[36569]: pgmap v18454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:24.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:23 smithi104 ceph-mon[36624]: pgmap v18454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:24.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:23 smithi081 ceph-mon[31453]: pgmap v18454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:26.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:25 smithi153 ceph-mon[36569]: pgmap v18455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:26.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:25 smithi104 ceph-mon[36624]: pgmap v18455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:26.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:25 smithi081 ceph-mon[31453]: pgmap v18455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:28.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:27 smithi153 ceph-mon[36569]: pgmap v18456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:28.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:27 smithi104 ceph-mon[36624]: pgmap v18456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:28.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:27 smithi081 ceph-mon[31453]: pgmap v18456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:29.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:28 smithi153 ceph-mon[36569]: pgmap v18457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:29.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:28 smithi104 ceph-mon[36624]: pgmap v18457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:29.177 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:28 smithi081 ceph-mon[31453]: pgmap v18457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:31.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:31 smithi153 ceph-mon[36569]: pgmap v18458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:31.595 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:31 smithi104 ceph-mon[36624]: pgmap v18458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:31.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:31 smithi081 ceph-mon[31453]: pgmap v18458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:33.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:33 smithi153 ceph-mon[36569]: pgmap v18459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:33.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:33 smithi104 ceph-mon[36624]: pgmap v18459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:33.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:33 smithi081 ceph-mon[31453]: pgmap v18459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:34.160+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:52:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:34.160+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:52:34.174 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:34.160+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:52:35.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:35 smithi153 ceph-mon[36569]: pgmap v18460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:35.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:35 smithi104 ceph-mon[36624]: pgmap v18460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:35.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:35 smithi081 ceph-mon[31453]: pgmap v18460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:37.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:37 smithi153 ceph-mon[36569]: pgmap v18461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:37.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:37 smithi104 ceph-mon[36624]: pgmap v18461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:37.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:37 smithi081 ceph-mon[31453]: pgmap v18461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:39.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:39 smithi153 ceph-mon[36569]: pgmap v18462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:39.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:39 smithi104 ceph-mon[36624]: pgmap v18462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:39.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:39 smithi081 ceph-mon[31453]: pgmap v18462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:41.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:41 smithi153 ceph-mon[36569]: pgmap v18463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:41.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:41 smithi104 ceph-mon[36624]: pgmap v18463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:41.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:41 smithi081 ceph-mon[31453]: pgmap v18463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:43.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:43 smithi153 ceph-mon[36569]: pgmap v18464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:43.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:43 smithi104 ceph-mon[36624]: pgmap v18464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:43.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:43 smithi081 ceph-mon[31453]: pgmap v18464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:45.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:45 smithi153 ceph-mon[36569]: pgmap v18465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:45.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:45 smithi104 ceph-mon[36624]: pgmap v18465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:45.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:45 smithi081 ceph-mon[31453]: pgmap v18465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:47.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:47 smithi153 ceph-mon[36569]: pgmap v18466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:47.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:52:47.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:47 smithi104 ceph-mon[36624]: pgmap v18466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:47.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:52:47.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:47 smithi081 ceph-mon[31453]: pgmap v18466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:47.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:52:48.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:52:48.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:52:48.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:52:48.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:52:48.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:52:48.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:52:48.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:52:48.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:52:48.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:52:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:49.174+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:52:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:49.174+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:52:49.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:52:49.174+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:52:49.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:49 smithi153 ceph-mon[36569]: pgmap v18467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:49.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:49 smithi153 ceph-mon[36569]: pgmap v18468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:49.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:49 smithi104 ceph-mon[36624]: pgmap v18467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:49.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:49 smithi104 ceph-mon[36624]: pgmap v18468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:49.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:49 smithi081 ceph-mon[31453]: pgmap v18467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:49.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:49 smithi081 ceph-mon[31453]: pgmap v18468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:51.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:51 smithi153 ceph-mon[36569]: pgmap v18469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:51.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:51 smithi104 ceph-mon[36624]: pgmap v18469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:51.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:51 smithi081 ceph-mon[31453]: pgmap v18469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:53.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:53 smithi153 ceph-mon[36569]: pgmap v18470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:53.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:53 smithi104 ceph-mon[36624]: pgmap v18470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:53.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:53 smithi081 ceph-mon[31453]: pgmap v18470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:55.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:55 smithi153 ceph-mon[36569]: pgmap v18471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:55.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:55 smithi104 ceph-mon[36624]: pgmap v18471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:55.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:55 smithi081 ceph-mon[31453]: pgmap v18471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:57.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:57 smithi153 ceph-mon[36569]: pgmap v18472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:57.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:57 smithi104 ceph-mon[36624]: pgmap v18472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:57.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:57 smithi081 ceph-mon[31453]: pgmap v18472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:59.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:52:59 smithi153 ceph-mon[36569]: pgmap v18473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:59.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:52:59 smithi104 ceph-mon[36624]: pgmap v18473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:52:59.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:52:59 smithi081 ceph-mon[31453]: pgmap v18473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:01.819 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:01 smithi153 ceph-mon[36569]: pgmap v18474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:01.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:01 smithi104 ceph-mon[36624]: pgmap v18474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:01.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:01 smithi081 ceph-mon[31453]: pgmap v18474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:03.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:03 smithi153 ceph-mon[36569]: pgmap v18475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:03.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:03 smithi104 ceph-mon[36624]: pgmap v18475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:03.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:03 smithi081 ceph-mon[31453]: pgmap v18475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:04.191 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:04.178+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:53:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:04.178+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:53:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:04.178+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:53:05.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:05 smithi153 ceph-mon[36569]: pgmap v18476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:05.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:05 smithi104 ceph-mon[36624]: pgmap v18476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:05.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:05 smithi081 ceph-mon[31453]: pgmap v18476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:07.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:07 smithi153 ceph-mon[36569]: pgmap v18477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:07.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:07 smithi104 ceph-mon[36624]: pgmap v18477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:07.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:07 smithi081 ceph-mon[31453]: pgmap v18477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:09.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:09 smithi153 ceph-mon[36569]: pgmap v18478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:09.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:09 smithi104 ceph-mon[36624]: pgmap v18478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:09.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:09 smithi081 ceph-mon[31453]: pgmap v18478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:11.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:11 smithi153 ceph-mon[36569]: pgmap v18479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:11.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:11 smithi104 ceph-mon[36624]: pgmap v18479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:11.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:11 smithi081 ceph-mon[31453]: pgmap v18479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:13.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:13 smithi153 ceph-mon[36569]: pgmap v18480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:13.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:13 smithi104 ceph-mon[36624]: pgmap v18480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:13.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:13 smithi081 ceph-mon[31453]: pgmap v18480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:15.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:15 smithi104 ceph-mon[36624]: pgmap v18481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:15.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:15 smithi081 ceph-mon[31453]: pgmap v18481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:16.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:15 smithi153 ceph-mon[36569]: pgmap v18481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:17.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:17 smithi104 ceph-mon[36624]: pgmap v18482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:17.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:17 smithi081 ceph-mon[31453]: pgmap v18482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:18.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:17 smithi153 ceph-mon[36569]: pgmap v18482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:19.186+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:53:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:19.186+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:53:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:19.186+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:53:19.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:19 smithi104 ceph-mon[36624]: pgmap v18483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:19.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:19 smithi081 ceph-mon[31453]: pgmap v18483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:20.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:19 smithi153 ceph-mon[36569]: pgmap v18483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:21.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:21 smithi104 ceph-mon[36624]: pgmap v18484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:21.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:21 smithi081 ceph-mon[31453]: pgmap v18484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:22.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:21 smithi153 ceph-mon[36569]: pgmap v18484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:23.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:23 smithi104 ceph-mon[36624]: pgmap v18485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:23.872 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:23 smithi153 ceph-mon[36569]: pgmap v18485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:23.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:23 smithi081 ceph-mon[31453]: pgmap v18485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:25.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:25 smithi081 ceph-mon[31453]: pgmap v18486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:26.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:25 smithi153 ceph-mon[36569]: pgmap v18486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:26.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:25 smithi104 ceph-mon[36624]: pgmap v18486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:27.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:27 smithi081 ceph-mon[31453]: pgmap v18487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:28.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:27 smithi153 ceph-mon[36569]: pgmap v18487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:28.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:27 smithi104 ceph-mon[36624]: pgmap v18487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:29.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:29 smithi081 ceph-mon[31453]: pgmap v18488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:30.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:29 smithi153 ceph-mon[36569]: pgmap v18488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:30.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:29 smithi104 ceph-mon[36624]: pgmap v18488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:31.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:31 smithi081 ceph-mon[31453]: pgmap v18489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:32.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:31 smithi153 ceph-mon[36569]: pgmap v18489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:32.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:31 smithi104 ceph-mon[36624]: pgmap v18489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:33.898 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:33 smithi104 ceph-mon[36624]: pgmap v18490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:33.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:33 smithi081 ceph-mon[31453]: pgmap v18490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:34.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:33 smithi153 ceph-mon[36569]: pgmap v18490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:34.202+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:53:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:34.202+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:53:34.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:34.202+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:53:35.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:35 smithi081 ceph-mon[31453]: pgmap v18491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:36.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:35 smithi153 ceph-mon[36569]: pgmap v18491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:36.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:35 smithi104 ceph-mon[36624]: pgmap v18491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:37.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:37 smithi081 ceph-mon[31453]: pgmap v18492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:38.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:37 smithi153 ceph-mon[36569]: pgmap v18492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:38.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:37 smithi104 ceph-mon[36624]: pgmap v18492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:39.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:39 smithi081 ceph-mon[31453]: pgmap v18493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:40.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:39 smithi153 ceph-mon[36569]: pgmap v18493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:40.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:39 smithi104 ceph-mon[36624]: pgmap v18493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:41.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:41 smithi081 ceph-mon[31453]: pgmap v18494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:42.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:41 smithi153 ceph-mon[36569]: pgmap v18494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:42.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:41 smithi104 ceph-mon[36624]: pgmap v18494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:43.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:43 smithi081 ceph-mon[31453]: pgmap v18495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:44.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:43 smithi153 ceph-mon[36569]: pgmap v18495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:44.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:43 smithi104 ceph-mon[36624]: pgmap v18495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:45.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:45 smithi081 ceph-mon[31453]: pgmap v18496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:46.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:45 smithi153 ceph-mon[36569]: pgmap v18496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:46.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:45 smithi104 ceph-mon[36624]: pgmap v18496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:47.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:47 smithi081 ceph-mon[31453]: pgmap v18497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:48.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:47 smithi153 ceph-mon[36569]: pgmap v18497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:48.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:47 smithi104 ceph-mon[36624]: pgmap v18497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:48.435+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:53:48.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:48.435+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:53:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:48.436+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 267 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:53:48.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:48.436+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 267 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:53:48.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:53:49.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:53:49.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:53:49.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:49.218+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:53:49.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:49.218+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:53:49.231 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:53:49.218+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:53:50.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:49 smithi153 ceph-mon[36569]: pgmap v18498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:50.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:53:50.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:53:50.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:53:50.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:49 smithi104 ceph-mon[36624]: pgmap v18498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:50.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:53:50.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:53:50.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:53:50.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:49 smithi081 ceph-mon[31453]: pgmap v18498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:50.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:53:50.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:53:50.101 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:53:51.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:50 smithi153 ceph-mon[36569]: pgmap v18499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:51.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:50 smithi153 ceph-mon[36569]: pgmap v18500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:51.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:50 smithi104 ceph-mon[36624]: pgmap v18499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:51.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:50 smithi104 ceph-mon[36624]: pgmap v18500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:51.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:50 smithi081 ceph-mon[31453]: pgmap v18499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:51.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:50 smithi081 ceph-mon[31453]: pgmap v18500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:53.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:52 smithi153 ceph-mon[36569]: pgmap v18501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:53.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:52 smithi104 ceph-mon[36624]: pgmap v18501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:53.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:52 smithi081 ceph-mon[31453]: pgmap v18501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:55.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:54 smithi153 ceph-mon[36569]: pgmap v18502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:55.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:54 smithi104 ceph-mon[36624]: pgmap v18502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:55.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:54 smithi081 ceph-mon[31453]: pgmap v18502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:57.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:56 smithi153 ceph-mon[36569]: pgmap v18503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:57.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:56 smithi104 ceph-mon[36624]: pgmap v18503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:57.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:56 smithi081 ceph-mon[31453]: pgmap v18503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:59.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:53:58 smithi153 ceph-mon[36569]: pgmap v18504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:59.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:53:58 smithi104 ceph-mon[36624]: pgmap v18504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:53:59.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:53:58 smithi081 ceph-mon[31453]: pgmap v18504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:01.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:00 smithi153 ceph-mon[36569]: pgmap v18505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:01.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:00 smithi104 ceph-mon[36624]: pgmap v18505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:01.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:00 smithi081 ceph-mon[31453]: pgmap v18505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:02.973 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:02 smithi081 ceph-mon[31453]: pgmap v18506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:03.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:02 smithi153 ceph-mon[36569]: pgmap v18506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:03.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:02 smithi104 ceph-mon[36624]: pgmap v18506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:04.233+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:54:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:04.233+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:54:04.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:04.233+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:54:05.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:04 smithi153 ceph-mon[36569]: pgmap v18507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:05.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:04 smithi104 ceph-mon[36624]: pgmap v18507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:05.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:04 smithi081 ceph-mon[31453]: pgmap v18507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:07.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:06 smithi153 ceph-mon[36569]: pgmap v18508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:07.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:06 smithi104 ceph-mon[36624]: pgmap v18508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:07.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:06 smithi081 ceph-mon[31453]: pgmap v18508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:09.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:08 smithi153 ceph-mon[36569]: pgmap v18509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:09.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:08 smithi104 ceph-mon[36624]: pgmap v18509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:09.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:08 smithi081 ceph-mon[31453]: pgmap v18509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:11.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:10 smithi153 ceph-mon[36569]: pgmap v18510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:11.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:10 smithi104 ceph-mon[36624]: pgmap v18510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:11.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:10 smithi081 ceph-mon[31453]: pgmap v18510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:12 smithi153 ceph-mon[36569]: pgmap v18511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:13.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:12 smithi104 ceph-mon[36624]: pgmap v18511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:13.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:12 smithi081 ceph-mon[31453]: pgmap v18511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:14.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:13 smithi153 ceph-mon[36569]: pgmap v18512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:14.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:13 smithi104 ceph-mon[36624]: pgmap v18512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:14.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:13 smithi081 ceph-mon[31453]: pgmap v18512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:16.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:16 smithi153 ceph-mon[36569]: pgmap v18513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:16.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:16 smithi104 ceph-mon[36624]: pgmap v18513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:16.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:16 smithi081 ceph-mon[31453]: pgmap v18513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:18.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:18 smithi153 ceph-mon[36569]: pgmap v18514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:18.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:18 smithi104 ceph-mon[36624]: pgmap v18514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:18.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:18 smithi081 ceph-mon[31453]: pgmap v18514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:19.248+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:54:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:19.248+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:54:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:19.248+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:54:20.669 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:20 smithi081 ceph-mon[31453]: pgmap v18515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:20.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:20 smithi153 ceph-mon[36569]: pgmap v18515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:20.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:20 smithi104 ceph-mon[36624]: pgmap v18515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:22.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:22 smithi153 ceph-mon[36569]: pgmap v18516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:22.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:22 smithi104 ceph-mon[36624]: pgmap v18516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:22.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:22 smithi081 ceph-mon[31453]: pgmap v18516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:24.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:24 smithi153 ceph-mon[36569]: pgmap v18517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:24.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:24 smithi104 ceph-mon[36624]: pgmap v18517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:24.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:24 smithi081 ceph-mon[31453]: pgmap v18517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:26.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:26 smithi153 ceph-mon[36569]: pgmap v18518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:26.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:26 smithi104 ceph-mon[36624]: pgmap v18518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:26.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:26 smithi081 ceph-mon[31453]: pgmap v18518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:28.820 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:28 smithi153 ceph-mon[36569]: pgmap v18519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:28.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:28 smithi104 ceph-mon[36624]: pgmap v18519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:28.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:28 smithi081 ceph-mon[31453]: pgmap v18519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:30.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:30 smithi104 ceph-mon[36624]: pgmap v18520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:30.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:30 smithi081 ceph-mon[31453]: pgmap v18520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:31.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:30 smithi153 ceph-mon[36569]: pgmap v18520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:32.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:32 smithi104 ceph-mon[36624]: pgmap v18521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:32.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:32 smithi081 ceph-mon[31453]: pgmap v18521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:33.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:32 smithi153 ceph-mon[36569]: pgmap v18521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:34.264+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:54:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:34.264+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:54:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:34.264+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:54:34.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:34 smithi104 ceph-mon[36624]: pgmap v18522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:34.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:34 smithi081 ceph-mon[31453]: pgmap v18522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:35.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:34 smithi153 ceph-mon[36569]: pgmap v18522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:36.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:36 smithi104 ceph-mon[36624]: pgmap v18523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:36.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:36 smithi081 ceph-mon[31453]: pgmap v18523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:37.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:36 smithi153 ceph-mon[36569]: pgmap v18523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:38.845 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:38 smithi104 ceph-mon[36624]: pgmap v18524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:38.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:38 smithi081 ceph-mon[31453]: pgmap v18524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:39.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:38 smithi153 ceph-mon[36569]: pgmap v18524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:40.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:40 smithi081 ceph-mon[31453]: pgmap v18525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:41.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:40 smithi153 ceph-mon[36569]: pgmap v18525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:41.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:40 smithi104 ceph-mon[36624]: pgmap v18525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:42.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:42 smithi081 ceph-mon[31453]: pgmap v18526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:43.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:42 smithi153 ceph-mon[36569]: pgmap v18526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:43.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:42 smithi104 ceph-mon[36624]: pgmap v18526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:44.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:44 smithi081 ceph-mon[31453]: pgmap v18527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:45.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:44 smithi153 ceph-mon[36569]: pgmap v18527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:45.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:44 smithi104 ceph-mon[36624]: pgmap v18527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:46.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:46 smithi081 ceph-mon[31453]: pgmap v18528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:47.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:46 smithi153 ceph-mon[36569]: pgmap v18528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:47.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:46 smithi104 ceph-mon[36624]: pgmap v18528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:48.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:48 smithi081 ceph-mon[31453]: pgmap v18529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:49.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:48 smithi153 ceph-mon[36569]: pgmap v18529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:49.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:48 smithi104 ceph-mon[36624]: pgmap v18529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:49.278+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:54:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:49.278+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:54:49.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:54:49.278+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:54:50.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: pgmap v18530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:50.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: pgmap v18530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: pgmap v18530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:51.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:54:51.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:54:51.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:54:51.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:54:51.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:54:51.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:51.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:54:51.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:54:51.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:54:52.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:54:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:54:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:54:52.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:52 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:54:52.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:52 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:54:52.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:52 smithi081 ceph-mon[31453]: pgmap v18531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:52.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:52 smithi081 ceph-mon[31453]: pgmap v18532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:53.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:52 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:54:53.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:52 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:54:53.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:52 smithi153 ceph-mon[36569]: pgmap v18531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:53.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:52 smithi153 ceph-mon[36569]: pgmap v18532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:53.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:52 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T06:54:53.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:52 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T06:54:53.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:52 smithi104 ceph-mon[36624]: pgmap v18531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:53.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:52 smithi104 ceph-mon[36624]: pgmap v18532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:54.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:54 smithi081 ceph-mon[31453]: pgmap v18533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:55.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:54 smithi153 ceph-mon[36569]: pgmap v18533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:55.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:54 smithi104 ceph-mon[36624]: pgmap v18533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:56.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:56 smithi081 ceph-mon[31453]: pgmap v18534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:57.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:56 smithi153 ceph-mon[36569]: pgmap v18534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:57.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:56 smithi104 ceph-mon[36624]: pgmap v18534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:58.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:54:58 smithi081 ceph-mon[31453]: pgmap v18535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:59.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:54:58 smithi153 ceph-mon[36569]: pgmap v18535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:54:59.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:54:58 smithi104 ceph-mon[36624]: pgmap v18535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:00.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:00 smithi081 ceph-mon[31453]: pgmap v18536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:01.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:00 smithi153 ceph-mon[36569]: pgmap v18536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:01.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:00 smithi104 ceph-mon[36624]: pgmap v18536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:02.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:02 smithi081 ceph-mon[31453]: pgmap v18537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:03.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:02 smithi153 ceph-mon[36569]: pgmap v18537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:03.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:02 smithi104 ceph-mon[36624]: pgmap v18537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:04.285+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:55:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:04.285+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:55:04.300 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:04.285+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:55:05.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:04 smithi153 ceph-mon[36569]: pgmap v18538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:05.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:04 smithi104 ceph-mon[36624]: pgmap v18538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:05.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:04 smithi081 ceph-mon[31453]: pgmap v18538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:06.973 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:06 smithi081 ceph-mon[31453]: pgmap v18539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:07.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:06 smithi153 ceph-mon[36569]: pgmap v18539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:07.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:06 smithi104 ceph-mon[36624]: pgmap v18539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:09.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:08 smithi153 ceph-mon[36569]: pgmap v18540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:09.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:08 smithi104 ceph-mon[36624]: pgmap v18540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:09.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:08 smithi081 ceph-mon[31453]: pgmap v18540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:11.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:10 smithi153 ceph-mon[36569]: pgmap v18541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:11.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:10 smithi104 ceph-mon[36624]: pgmap v18541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:11.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:10 smithi081 ceph-mon[31453]: pgmap v18541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:12 smithi153 ceph-mon[36569]: pgmap v18542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:13.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:12 smithi104 ceph-mon[36624]: pgmap v18542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:13.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:12 smithi081 ceph-mon[31453]: pgmap v18542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:15.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:14 smithi153 ceph-mon[36569]: pgmap v18543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:15.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:14 smithi104 ceph-mon[36624]: pgmap v18543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:15.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:14 smithi081 ceph-mon[31453]: pgmap v18543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:17.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:16 smithi153 ceph-mon[36569]: pgmap v18544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:17.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:16 smithi104 ceph-mon[36624]: pgmap v18544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:17.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:16 smithi081 ceph-mon[31453]: pgmap v18544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:18.450+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:55:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:18.450+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:55:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:18.451+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 268 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:55:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:18.451+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 268 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:55:19.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:18 smithi153 ceph-mon[36569]: pgmap v18545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:19.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:18 smithi104 ceph-mon[36624]: pgmap v18545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:19.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:18 smithi081 ceph-mon[31453]: pgmap v18545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:19.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:19.301+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:55:19.315 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:19.301+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:55:19.316 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:19.301+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:55:21.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:20 smithi153 ceph-mon[36569]: pgmap v18546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:21.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:20 smithi104 ceph-mon[36624]: pgmap v18546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:21.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:20 smithi081 ceph-mon[31453]: pgmap v18546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:23.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:22 smithi153 ceph-mon[36569]: pgmap v18547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:23.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:22 smithi104 ceph-mon[36624]: pgmap v18547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:23.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:22 smithi081 ceph-mon[31453]: pgmap v18547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:25.074 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:24 smithi153 ceph-mon[36569]: pgmap v18548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:25.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:24 smithi104 ceph-mon[36624]: pgmap v18548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:25.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:24 smithi081 ceph-mon[31453]: pgmap v18548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:27.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:26 smithi153 ceph-mon[36569]: pgmap v18549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:27.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:26 smithi104 ceph-mon[36624]: pgmap v18549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:27.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:26 smithi081 ceph-mon[31453]: pgmap v18549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:29.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:28 smithi153 ceph-mon[36569]: pgmap v18550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:29.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:28 smithi104 ceph-mon[36624]: pgmap v18550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:29.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:28 smithi081 ceph-mon[31453]: pgmap v18550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:31.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:30 smithi153 ceph-mon[36569]: pgmap v18551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:31.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:30 smithi104 ceph-mon[36624]: pgmap v18551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:31.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:30 smithi081 ceph-mon[31453]: pgmap v18551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:32.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:31 smithi153 ceph-mon[36569]: pgmap v18552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:32.071 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:31 smithi081 ceph-mon[31453]: pgmap v18552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:32.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:31 smithi104 ceph-mon[36624]: pgmap v18552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:34.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:34.317+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:55:34.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:34.317+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:55:34.331 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:34.317+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:55:34.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:34 smithi081 ceph-mon[31453]: pgmap v18553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:34.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:34 smithi153 ceph-mon[36569]: pgmap v18553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:34.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:34 smithi104 ceph-mon[36624]: pgmap v18553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:36.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:36 smithi081 ceph-mon[31453]: pgmap v18554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:36.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:36 smithi153 ceph-mon[36569]: pgmap v18554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:36.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:36 smithi104 ceph-mon[36624]: pgmap v18554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:38.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:38 smithi081 ceph-mon[31453]: pgmap v18555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:38.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:38 smithi153 ceph-mon[36569]: pgmap v18555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:38.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:38 smithi104 ceph-mon[36624]: pgmap v18555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:40.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:40 smithi081 ceph-mon[31453]: pgmap v18556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:40.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:40 smithi153 ceph-mon[36569]: pgmap v18556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:40.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:40 smithi104 ceph-mon[36624]: pgmap v18556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:42.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:42 smithi081 ceph-mon[31453]: pgmap v18557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:42.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:42 smithi153 ceph-mon[36569]: pgmap v18557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:42.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:42 smithi104 ceph-mon[36624]: pgmap v18557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:44.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:44 smithi081 ceph-mon[31453]: pgmap v18558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:44.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:44 smithi153 ceph-mon[36569]: pgmap v18558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:44.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:44 smithi104 ceph-mon[36624]: pgmap v18558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:46.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:46 smithi081 ceph-mon[31453]: pgmap v18559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:46.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:46 smithi153 ceph-mon[36569]: pgmap v18559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:46.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:46 smithi104 ceph-mon[36624]: pgmap v18559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:48.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:48 smithi081 ceph-mon[31453]: pgmap v18560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:48.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:48 smithi153 ceph-mon[36569]: pgmap v18560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:48.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:48 smithi104 ceph-mon[36624]: pgmap v18560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:49.319+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:55:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:49.319+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:55:49.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:55:49.319+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:55:50.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:50 smithi081 ceph-mon[31453]: pgmap v18561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:50.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:50 smithi153 ceph-mon[36569]: pgmap v18561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:50.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:50 smithi104 ceph-mon[36624]: pgmap v18561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:52.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:52 smithi153 ceph-mon[36569]: pgmap v18562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:52.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:55:52.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:52 smithi104 ceph-mon[36624]: pgmap v18562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:52.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:55:52.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:52 smithi081 ceph-mon[31453]: pgmap v18562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:52.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:55:53.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:55:53.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:55:53.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:53 smithi153 ceph-mon[36569]: pgmap v18563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:53.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:53 smithi153 ceph-mon[36569]: pgmap v18564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:53.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:55:53.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:55:53.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:55:53.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:53 smithi104 ceph-mon[36624]: pgmap v18563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:53.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:53 smithi104 ceph-mon[36624]: pgmap v18564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:53.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:55:53.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:55:53.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:55:53.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:53 smithi081 ceph-mon[31453]: pgmap v18563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:53.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:53 smithi081 ceph-mon[31453]: pgmap v18564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:53.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:55:55.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:55 smithi081 ceph-mon[31453]: pgmap v18565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:56.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:55 smithi153 ceph-mon[36569]: pgmap v18565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:56.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:55 smithi104 ceph-mon[36624]: pgmap v18565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:57.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:57 smithi081 ceph-mon[31453]: pgmap v18566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:58.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:57 smithi153 ceph-mon[36569]: pgmap v18566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:58.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:57 smithi104 ceph-mon[36624]: pgmap v18566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:55:59.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:55:59 smithi081 ceph-mon[31453]: pgmap v18567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:00.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:55:59 smithi153 ceph-mon[36569]: pgmap v18567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:00.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:55:59 smithi104 ceph-mon[36624]: pgmap v18567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:01.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:01 smithi081 ceph-mon[31453]: pgmap v18568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:02.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:01 smithi153 ceph-mon[36569]: pgmap v18568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:02.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:01 smithi104 ceph-mon[36624]: pgmap v18568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:03.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:03 smithi081 ceph-mon[31453]: pgmap v18569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:04.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:03 smithi153 ceph-mon[36569]: pgmap v18569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:04.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:03 smithi104 ceph-mon[36624]: pgmap v18569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:04.325+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:56:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:04.325+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:56:04.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:04.325+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:56:05.898 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:05 smithi104 ceph-mon[36624]: pgmap v18570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:05.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:05 smithi081 ceph-mon[31453]: pgmap v18570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:06.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:05 smithi153 ceph-mon[36569]: pgmap v18570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:07.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:07 smithi081 ceph-mon[31453]: pgmap v18571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:08.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:07 smithi153 ceph-mon[36569]: pgmap v18571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:08.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:07 smithi104 ceph-mon[36624]: pgmap v18571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:09.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:09 smithi081 ceph-mon[31453]: pgmap v18572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:10.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:09 smithi153 ceph-mon[36569]: pgmap v18572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:10.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:09 smithi104 ceph-mon[36624]: pgmap v18572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:11.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:11 smithi081 ceph-mon[31453]: pgmap v18573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:12.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:11 smithi153 ceph-mon[36569]: pgmap v18573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:12.165 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:11 smithi104 ceph-mon[36624]: pgmap v18573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:13.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:13 smithi081 ceph-mon[31453]: pgmap v18574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:14.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:13 smithi153 ceph-mon[36569]: pgmap v18574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:14.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:13 smithi104 ceph-mon[36624]: pgmap v18574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:15.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:15 smithi081 ceph-mon[31453]: pgmap v18575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:16.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:15 smithi153 ceph-mon[36569]: pgmap v18575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:16.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:15 smithi104 ceph-mon[36624]: pgmap v18575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:17.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:17 smithi081 ceph-mon[31453]: pgmap v18576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:18.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:17 smithi153 ceph-mon[36569]: pgmap v18576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:18.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:17 smithi104 ceph-mon[36624]: pgmap v18576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:19.341+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:56:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:19.341+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:56:19.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:19.341+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:56:19.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:19 smithi081 ceph-mon[31453]: pgmap v18577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:20.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:19 smithi153 ceph-mon[36569]: pgmap v18577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:20.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:19 smithi104 ceph-mon[36624]: pgmap v18577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:22.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:21 smithi153 ceph-mon[36569]: pgmap v18578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:22.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:21 smithi104 ceph-mon[36624]: pgmap v18578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:22.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:21 smithi081 ceph-mon[31453]: pgmap v18578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:24.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:23 smithi153 ceph-mon[36569]: pgmap v18579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:24.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:23 smithi104 ceph-mon[36624]: pgmap v18579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:24.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:23 smithi081 ceph-mon[31453]: pgmap v18579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:26.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:25 smithi153 ceph-mon[36569]: pgmap v18580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:26.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:25 smithi104 ceph-mon[36624]: pgmap v18580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:26.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:25 smithi081 ceph-mon[31453]: pgmap v18580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:28.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:27 smithi153 ceph-mon[36569]: pgmap v18581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:28.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:27 smithi104 ceph-mon[36624]: pgmap v18581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:28.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:27 smithi081 ceph-mon[31453]: pgmap v18581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:30.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:29 smithi153 ceph-mon[36569]: pgmap v18582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:30.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:29 smithi104 ceph-mon[36624]: pgmap v18582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:30.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:29 smithi081 ceph-mon[31453]: pgmap v18582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:31.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:31 smithi153 ceph-mon[36569]: pgmap v18583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:31.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:31 smithi104 ceph-mon[36624]: pgmap v18583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:31.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:31 smithi081 ceph-mon[31453]: pgmap v18583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:33.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:33 smithi081 ceph-mon[31453]: pgmap v18584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:34.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:33 smithi153 ceph-mon[36569]: pgmap v18584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:34.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:33 smithi104 ceph-mon[36624]: pgmap v18584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:34.350+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:56:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:34.350+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:56:34.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:34.350+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:56:35.899 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:35 smithi104 ceph-mon[36624]: pgmap v18585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:35.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:35 smithi081 ceph-mon[31453]: pgmap v18585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:36.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:35 smithi153 ceph-mon[36569]: pgmap v18585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:37.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:37 smithi081 ceph-mon[31453]: pgmap v18586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:38.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:37 smithi153 ceph-mon[36569]: pgmap v18586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:38.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:37 smithi104 ceph-mon[36624]: pgmap v18586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:39.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:39 smithi081 ceph-mon[31453]: pgmap v18587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:40.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:39 smithi153 ceph-mon[36569]: pgmap v18587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:40.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:39 smithi104 ceph-mon[36624]: pgmap v18587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:41.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:41 smithi081 ceph-mon[31453]: pgmap v18588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:42.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:41 smithi153 ceph-mon[36569]: pgmap v18588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:42.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:41 smithi104 ceph-mon[36624]: pgmap v18588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:43.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:43 smithi081 ceph-mon[31453]: pgmap v18589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:44.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:43 smithi153 ceph-mon[36569]: pgmap v18589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:44.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:43 smithi104 ceph-mon[36624]: pgmap v18589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:45.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:45 smithi081 ceph-mon[31453]: pgmap v18590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:46.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:45 smithi153 ceph-mon[36569]: pgmap v18590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:46.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:45 smithi104 ceph-mon[36624]: pgmap v18590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:47.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:47 smithi081 ceph-mon[31453]: pgmap v18591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:48.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:47 smithi153 ceph-mon[36569]: pgmap v18591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:48.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:47 smithi104 ceph-mon[36624]: pgmap v18591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:49.361+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:56:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:49.361+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:56:49.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:56:49.361+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:56:50.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:49 smithi153 ceph-mon[36569]: pgmap v18592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:50.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:49 smithi104 ceph-mon[36624]: pgmap v18592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:50.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:49 smithi081 ceph-mon[31453]: pgmap v18592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:52.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:51 smithi153 ceph-mon[36569]: pgmap v18593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:52.098 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:51 smithi104 ceph-mon[36624]: pgmap v18593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:52.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:51 smithi081 ceph-mon[31453]: pgmap v18593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:54.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:53 smithi153 ceph-mon[36569]: pgmap v18594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:54.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:56:54.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:53 smithi104 ceph-mon[36624]: pgmap v18594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:54.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:56:54.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:53 smithi081 ceph-mon[31453]: pgmap v18594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:54.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:56:55.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:56:55.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:56:55.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:56:55.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:56:55.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:56:55.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:56:55.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:56:55.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:56:55.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:56:56.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:55 smithi153 ceph-mon[36569]: pgmap v18595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:56.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:55 smithi153 ceph-mon[36569]: pgmap v18596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:56.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:55 smithi104 ceph-mon[36624]: pgmap v18595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:56.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:55 smithi104 ceph-mon[36624]: pgmap v18596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:56.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:55 smithi081 ceph-mon[31453]: pgmap v18595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:56.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:55 smithi081 ceph-mon[31453]: pgmap v18596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:58.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:57 smithi153 ceph-mon[36569]: pgmap v18597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:58.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:57 smithi104 ceph-mon[36624]: pgmap v18597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:56:58.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:57 smithi081 ceph-mon[31453]: pgmap v18597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:00.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:56:59 smithi153 ceph-mon[36569]: pgmap v18598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:00.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:56:59 smithi104 ceph-mon[36624]: pgmap v18598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:00.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:56:59 smithi081 ceph-mon[31453]: pgmap v18598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:02.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:01 smithi153 ceph-mon[36569]: pgmap v18599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:02.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:01 smithi104 ceph-mon[36624]: pgmap v18599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:02.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:01 smithi081 ceph-mon[31453]: pgmap v18599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:03.973 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:03 smithi081 ceph-mon[31453]: pgmap v18600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:04.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:03 smithi153 ceph-mon[36569]: pgmap v18600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:04.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:03 smithi104 ceph-mon[36624]: pgmap v18600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:04.372+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:57:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:04.372+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:57:04.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:04.372+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:57:06.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:05 smithi153 ceph-mon[36569]: pgmap v18601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:06.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:05 smithi104 ceph-mon[36624]: pgmap v18601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:06.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:05 smithi081 ceph-mon[31453]: pgmap v18601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:08.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:07 smithi153 ceph-mon[36569]: pgmap v18602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:08.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:07 smithi104 ceph-mon[36624]: pgmap v18602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:08.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:07 smithi081 ceph-mon[31453]: pgmap v18602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:10.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:09 smithi153 ceph-mon[36569]: pgmap v18603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:10.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:09 smithi104 ceph-mon[36624]: pgmap v18603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:10.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:09 smithi081 ceph-mon[31453]: pgmap v18603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:12.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:11 smithi153 ceph-mon[36569]: pgmap v18604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:11 smithi104 ceph-mon[36624]: pgmap v18604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:12.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:11 smithi081 ceph-mon[31453]: pgmap v18604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:14.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:13 smithi153 ceph-mon[36569]: pgmap v18605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:14.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:13 smithi104 ceph-mon[36624]: pgmap v18605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:14.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:13 smithi081 ceph-mon[31453]: pgmap v18605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:16.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:15 smithi153 ceph-mon[36569]: pgmap v18606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:16.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:15 smithi104 ceph-mon[36624]: pgmap v18606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:16.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:15 smithi081 ceph-mon[31453]: pgmap v18606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:18.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:17 smithi153 ceph-mon[36569]: pgmap v18607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:18.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:17 smithi104 ceph-mon[36624]: pgmap v18607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:18.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:17 smithi081 ceph-mon[31453]: pgmap v18607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:19.387+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:57:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:19.387+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:57:19.402 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:19.387+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:57:20.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:19 smithi153 ceph-mon[36569]: pgmap v18608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:20.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:19 smithi104 ceph-mon[36624]: pgmap v18608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:20.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:19 smithi081 ceph-mon[31453]: pgmap v18608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:22.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:21 smithi153 ceph-mon[36569]: pgmap v18609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:22.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:21 smithi104 ceph-mon[36624]: pgmap v18609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:22.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:21 smithi081 ceph-mon[31453]: pgmap v18609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:24.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:23 smithi153 ceph-mon[36569]: pgmap v18610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:24.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:23 smithi104 ceph-mon[36624]: pgmap v18610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:24.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:23 smithi081 ceph-mon[31453]: pgmap v18610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:25.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:24 smithi104 ceph-mon[36624]: pgmap v18611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:25.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:24 smithi081 ceph-mon[31453]: pgmap v18611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:25.320 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:24 smithi153 ceph-mon[36569]: pgmap v18611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:27.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:27 smithi081 ceph-mon[31453]: pgmap v18612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:27.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:27 smithi153 ceph-mon[36569]: pgmap v18612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:27.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:27 smithi104 ceph-mon[36624]: pgmap v18612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:29.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:29 smithi081 ceph-mon[31453]: pgmap v18613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:29.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:29 smithi153 ceph-mon[36569]: pgmap v18613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:29.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:29 smithi104 ceph-mon[36624]: pgmap v18613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:31.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:31 smithi081 ceph-mon[31453]: pgmap v18614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:31.570 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:31 smithi153 ceph-mon[36569]: pgmap v18614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:31.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:31 smithi104 ceph-mon[36624]: pgmap v18614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:33.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:33 smithi081 ceph-mon[31453]: pgmap v18615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:33.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:33 smithi153 ceph-mon[36569]: pgmap v18615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:33.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:33 smithi104 ceph-mon[36624]: pgmap v18615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:34.417 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:34.403+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:57:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:34.403+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:57:34.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:34.403+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:57:35.420 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:35 smithi081 ceph-mon[31453]: pgmap v18616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:35.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:35 smithi153 ceph-mon[36569]: pgmap v18616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:35.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:35 smithi104 ceph-mon[36624]: pgmap v18616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:37.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:37 smithi153 ceph-mon[36569]: pgmap v18617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:37.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:37 smithi104 ceph-mon[36624]: pgmap v18617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:37.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:37 smithi081 ceph-mon[31453]: pgmap v18617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:39.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:39 smithi153 ceph-mon[36569]: pgmap v18618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:39.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:39 smithi104 ceph-mon[36624]: pgmap v18618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:39.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:39 smithi081 ceph-mon[31453]: pgmap v18618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:41.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:41 smithi153 ceph-mon[36569]: pgmap v18619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:41.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:41 smithi104 ceph-mon[36624]: pgmap v18619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:41.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:41 smithi081 ceph-mon[31453]: pgmap v18619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:43.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:43 smithi153 ceph-mon[36569]: pgmap v18620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:43.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:43 smithi104 ceph-mon[36624]: pgmap v18620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:43.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:43 smithi081 ceph-mon[31453]: pgmap v18620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:45.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:45 smithi153 ceph-mon[36569]: pgmap v18621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:45.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:45 smithi104 ceph-mon[36624]: pgmap v18621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:45.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:45 smithi081 ceph-mon[31453]: pgmap v18621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:47.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:47 smithi153 ceph-mon[36569]: pgmap v18622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:47.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:47 smithi104 ceph-mon[36624]: pgmap v18622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:47.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:47 smithi081 ceph-mon[31453]: pgmap v18622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:49.416+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:57:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:49.416+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:57:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:57:49.416+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:57:49.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:49 smithi153 ceph-mon[36569]: pgmap v18623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:49.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:49 smithi104 ceph-mon[36624]: pgmap v18623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:49.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:49 smithi081 ceph-mon[31453]: pgmap v18623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:51.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:51 smithi153 ceph-mon[36569]: pgmap v18624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:51.598 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:51 smithi104 ceph-mon[36624]: pgmap v18624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:51.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:51 smithi081 ceph-mon[31453]: pgmap v18624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:53.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:53 smithi153 ceph-mon[36569]: pgmap v18625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:53.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:53 smithi104 ceph-mon[36624]: pgmap v18625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:53.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:53 smithi081 ceph-mon[31453]: pgmap v18625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:55.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:55 smithi153 ceph-mon[36569]: pgmap v18626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:55.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:57:55.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:55 smithi104 ceph-mon[36624]: pgmap v18626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:55.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:57:55.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:55 smithi081 ceph-mon[31453]: pgmap v18626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:55.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:57:56.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:57:56.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:57:56.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:56 smithi153 ceph-mon[36569]: pgmap v18627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:56.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:56 smithi153 ceph-mon[36569]: pgmap v18628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:56.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:57:56.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:57:56.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:57:56.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:56 smithi104 ceph-mon[36624]: pgmap v18627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:56.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:56 smithi104 ceph-mon[36624]: pgmap v18628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:56.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:57:56.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:57:56.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:57:56.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:56 smithi081 ceph-mon[31453]: pgmap v18627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:56.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:56 smithi081 ceph-mon[31453]: pgmap v18628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:56.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:57:59.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:57:58 smithi153 ceph-mon[36569]: pgmap v18629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:59.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:57:58 smithi104 ceph-mon[36624]: pgmap v18629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:57:59.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:57:58 smithi081 ceph-mon[31453]: pgmap v18629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:01.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:00 smithi153 ceph-mon[36569]: pgmap v18630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:01.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:00 smithi104 ceph-mon[36624]: pgmap v18630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:01.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:00 smithi081 ceph-mon[31453]: pgmap v18630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:03.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:02 smithi153 ceph-mon[36569]: pgmap v18631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:03.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:02 smithi104 ceph-mon[36624]: pgmap v18631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:03.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:02 smithi081 ceph-mon[31453]: pgmap v18631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:04.432+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:58:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:04.432+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:58:04.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:04.432+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:58:05.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:04 smithi153 ceph-mon[36569]: pgmap v18632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:05.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:04 smithi104 ceph-mon[36624]: pgmap v18632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:05.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:04 smithi081 ceph-mon[31453]: pgmap v18632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:07.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:06 smithi153 ceph-mon[36569]: pgmap v18633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:07.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:06 smithi104 ceph-mon[36624]: pgmap v18633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:07.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:06 smithi081 ceph-mon[31453]: pgmap v18633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:09.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:08 smithi153 ceph-mon[36569]: pgmap v18634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:09.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:08 smithi104 ceph-mon[36624]: pgmap v18634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:09.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:08 smithi081 ceph-mon[31453]: pgmap v18634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:11.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:10 smithi153 ceph-mon[36569]: pgmap v18635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:11.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:10 smithi104 ceph-mon[36624]: pgmap v18635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:11.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:10 smithi081 ceph-mon[31453]: pgmap v18635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:13.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:12 smithi153 ceph-mon[36569]: pgmap v18636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:12 smithi104 ceph-mon[36624]: pgmap v18636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:13.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:12 smithi081 ceph-mon[31453]: pgmap v18636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:15.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:14 smithi153 ceph-mon[36569]: pgmap v18637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:15.106 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:14 smithi104 ceph-mon[36624]: pgmap v18637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:15.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:14 smithi081 ceph-mon[31453]: pgmap v18637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:17.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:16 smithi153 ceph-mon[36569]: pgmap v18638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:17.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:16 smithi104 ceph-mon[36624]: pgmap v18638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:17.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:16 smithi081 ceph-mon[31453]: pgmap v18638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:18.481+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:58:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:18.481+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:58:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:18.482+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 269 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T06:58:18.496 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:18.482+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 269 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T06:58:19.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:18 smithi153 ceph-mon[36569]: pgmap v18639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:19.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:18 smithi104 ceph-mon[36624]: pgmap v18639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:19.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:18 smithi081 ceph-mon[31453]: pgmap v18639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:19.434+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:58:19.449 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:19.435+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:58:19.450 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:19.435+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:58:21.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:20 smithi153 ceph-mon[36569]: pgmap v18640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:21.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:20 smithi104 ceph-mon[36624]: pgmap v18640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:21.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:20 smithi081 ceph-mon[31453]: pgmap v18640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:23.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:22 smithi153 ceph-mon[36569]: pgmap v18641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:23.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:22 smithi104 ceph-mon[36624]: pgmap v18641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:23.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:22 smithi081 ceph-mon[31453]: pgmap v18641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:25.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:24 smithi153 ceph-mon[36569]: pgmap v18642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:25.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:24 smithi104 ceph-mon[36624]: pgmap v18642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:25.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:24 smithi081 ceph-mon[31453]: pgmap v18642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:26.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:25 smithi104 ceph-mon[36624]: pgmap v18643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:26.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:25 smithi081 ceph-mon[31453]: pgmap v18643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:26.321 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:25 smithi153 ceph-mon[36569]: pgmap v18643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:29.024 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:28 smithi153 ceph-mon[36569]: pgmap v18644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:29.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:28 smithi104 ceph-mon[36624]: pgmap v18644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:29.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:28 smithi081 ceph-mon[31453]: pgmap v18644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:31.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:30 smithi153 ceph-mon[36569]: pgmap v18645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:31.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:30 smithi104 ceph-mon[36624]: pgmap v18645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:31.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:30 smithi081 ceph-mon[31453]: pgmap v18645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:33.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:32 smithi153 ceph-mon[36569]: pgmap v18646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:33.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:32 smithi104 ceph-mon[36624]: pgmap v18646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:33.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:32 smithi081 ceph-mon[31453]: pgmap v18646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:34.437+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:58:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:34.437+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:58:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:34.437+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:58:35.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:34 smithi153 ceph-mon[36569]: pgmap v18647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:35.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:34 smithi104 ceph-mon[36624]: pgmap v18647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:35.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:34 smithi081 ceph-mon[31453]: pgmap v18647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:37.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:36 smithi153 ceph-mon[36569]: pgmap v18648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:37.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:36 smithi104 ceph-mon[36624]: pgmap v18648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:37.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:36 smithi081 ceph-mon[31453]: pgmap v18648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:39.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:38 smithi153 ceph-mon[36569]: pgmap v18649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:39.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:38 smithi104 ceph-mon[36624]: pgmap v18649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:39.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:38 smithi081 ceph-mon[31453]: pgmap v18649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:41.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:40 smithi153 ceph-mon[36569]: pgmap v18650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:41.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:40 smithi104 ceph-mon[36624]: pgmap v18650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:41.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:40 smithi081 ceph-mon[31453]: pgmap v18650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:43.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:42 smithi153 ceph-mon[36569]: pgmap v18651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:43.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:42 smithi104 ceph-mon[36624]: pgmap v18651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:43.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:42 smithi081 ceph-mon[31453]: pgmap v18651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:45.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:44 smithi153 ceph-mon[36569]: pgmap v18652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:45.098 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:44 smithi104 ceph-mon[36624]: pgmap v18652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:45.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:44 smithi081 ceph-mon[31453]: pgmap v18652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:47.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:46 smithi153 ceph-mon[36569]: pgmap v18653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:47.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:46 smithi104 ceph-mon[36624]: pgmap v18653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:47.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:46 smithi081 ceph-mon[31453]: pgmap v18653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:49.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:48 smithi153 ceph-mon[36569]: pgmap v18654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:49.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:48 smithi104 ceph-mon[36624]: pgmap v18654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:49.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:48 smithi081 ceph-mon[31453]: pgmap v18654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:49.453+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:58:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:49.453+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:58:49.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:58:49.453+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:58:51.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:50 smithi153 ceph-mon[36569]: pgmap v18655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:51.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:50 smithi104 ceph-mon[36624]: pgmap v18655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:51.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:50 smithi081 ceph-mon[31453]: pgmap v18655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:52.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:51 smithi081 ceph-mon[31453]: pgmap v18656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:52.321 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:51 smithi153 ceph-mon[36569]: pgmap v18656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:52.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:51 smithi104 ceph-mon[36624]: pgmap v18656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:55.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:54 smithi153 ceph-mon[36569]: pgmap v18657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:55.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:54 smithi104 ceph-mon[36624]: pgmap v18657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:55.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:54 smithi081 ceph-mon[31453]: pgmap v18657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:57.006 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:56 smithi153 ceph-mon[36569]: pgmap v18658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:57.006 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:58:57.006 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:57.007 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:57.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:56 smithi104 ceph-mon[36624]: pgmap v18658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:57.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:58:57.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:57.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:57.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:56 smithi081 ceph-mon[31453]: pgmap v18658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:57.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T06:58:57.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:57.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:58.068 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:58.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:58.069 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:57 smithi081 ceph-mon[31453]: pgmap v18659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:58.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:58.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:58.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:57 smithi153 ceph-mon[36569]: pgmap v18659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:58.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:58.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:58.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:57 smithi104 ceph-mon[36624]: pgmap v18659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:58:59.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:58:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:58:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:58:59.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:58:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:00.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:00 smithi153 ceph-mon[36569]: pgmap v18660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:00.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:00 smithi104 ceph-mon[36624]: pgmap v18660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:00.886 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:00 smithi081 ceph-mon[31453]: pgmap v18660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:02.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:02.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:02.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:02 smithi153 ceph-mon[36569]: pgmap v18661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:02.596 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:02.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:02.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:02 smithi104 ceph-mon[36624]: pgmap v18661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:02.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:02.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:02.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:02 smithi081 ceph-mon[31453]: pgmap v18661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:03.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:59:03.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:59:03.571 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:03 smithi153 ceph-mon[36569]: pgmap v18662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:03.572 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:03 smithi153 ceph-mon[36569]: pgmap v18663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:03.572 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:03.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:59:03.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:59:03.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:03 smithi104 ceph-mon[36624]: pgmap v18662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:03.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:03 smithi104 ceph-mon[36624]: pgmap v18663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:03.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:03.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T06:59:03.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T06:59:03.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:03 smithi081 ceph-mon[31453]: pgmap v18662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:03.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:03 smithi081 ceph-mon[31453]: pgmap v18663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:03.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T06:59:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:04.461+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:59:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:04.461+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:59:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:04.461+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:59:05.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:05 smithi081 ceph-mon[31453]: pgmap v18664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:06.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:05 smithi153 ceph-mon[36569]: pgmap v18664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:06.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:05 smithi104 ceph-mon[36624]: pgmap v18664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:07.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:07 smithi081 ceph-mon[31453]: pgmap v18665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:08.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:07 smithi153 ceph-mon[36569]: pgmap v18665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:08.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:07 smithi104 ceph-mon[36624]: pgmap v18665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:09.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:09 smithi081 ceph-mon[31453]: pgmap v18666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:10.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:09 smithi153 ceph-mon[36569]: pgmap v18666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:10.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:09 smithi104 ceph-mon[36624]: pgmap v18666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:11.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:11 smithi081 ceph-mon[31453]: pgmap v18667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:12.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:11 smithi153 ceph-mon[36569]: pgmap v18667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:11 smithi104 ceph-mon[36624]: pgmap v18667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:13.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:13 smithi081 ceph-mon[31453]: pgmap v18668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:14.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:13 smithi153 ceph-mon[36569]: pgmap v18668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:14.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:13 smithi104 ceph-mon[36624]: pgmap v18668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:15.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:15 smithi081 ceph-mon[31453]: pgmap v18669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:16.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:15 smithi153 ceph-mon[36569]: pgmap v18669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:16.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:15 smithi104 ceph-mon[36624]: pgmap v18669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:18.017 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:17 smithi081 ceph-mon[31453]: pgmap v18670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:18.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:17 smithi153 ceph-mon[36569]: pgmap v18670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:18.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:17 smithi104 ceph-mon[36624]: pgmap v18670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:19.473+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:59:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:19.473+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:59:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:19.473+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:59:20.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:19 smithi153 ceph-mon[36569]: pgmap v18671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:20.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:19 smithi104 ceph-mon[36624]: pgmap v18671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:20.185 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:19 smithi081 ceph-mon[31453]: pgmap v18671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:22.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:21 smithi153 ceph-mon[36569]: pgmap v18672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:22.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:21 smithi104 ceph-mon[36624]: pgmap v18672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:22.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:21 smithi081 ceph-mon[31453]: pgmap v18672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:24.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:23 smithi153 ceph-mon[36569]: pgmap v18673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:24.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:23 smithi104 ceph-mon[36624]: pgmap v18673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:24.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:23 smithi081 ceph-mon[31453]: pgmap v18673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:26.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:25 smithi153 ceph-mon[36569]: pgmap v18674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:26.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:25 smithi104 ceph-mon[36624]: pgmap v18674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:26.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:25 smithi081 ceph-mon[31453]: pgmap v18674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:28.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:27 smithi153 ceph-mon[36569]: pgmap v18675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:28.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:27 smithi104 ceph-mon[36624]: pgmap v18675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:28.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:27 smithi081 ceph-mon[31453]: pgmap v18675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:30.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:29 smithi153 ceph-mon[36569]: pgmap v18676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:30.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:29 smithi104 ceph-mon[36624]: pgmap v18676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:30.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:29 smithi081 ceph-mon[31453]: pgmap v18676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:32.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:31 smithi153 ceph-mon[36569]: pgmap v18677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:32.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:31 smithi104 ceph-mon[36624]: pgmap v18677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:32.118 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:31 smithi081 ceph-mon[31453]: pgmap v18677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:34.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:33 smithi153 ceph-mon[36569]: pgmap v18678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:34.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:33 smithi104 ceph-mon[36624]: pgmap v18678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:34.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:33 smithi081 ceph-mon[31453]: pgmap v18678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:34.485+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:59:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:34.485+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:59:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:34.485+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:59:36.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:35 smithi153 ceph-mon[36569]: pgmap v18679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:36.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:35 smithi104 ceph-mon[36624]: pgmap v18679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:36.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:35 smithi081 ceph-mon[31453]: pgmap v18679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:38.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:37 smithi153 ceph-mon[36569]: pgmap v18680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:38.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:37 smithi104 ceph-mon[36624]: pgmap v18680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:38.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:37 smithi081 ceph-mon[31453]: pgmap v18680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:40.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:39 smithi153 ceph-mon[36569]: pgmap v18681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:40.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:39 smithi104 ceph-mon[36624]: pgmap v18681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:40.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:39 smithi081 ceph-mon[31453]: pgmap v18681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:42.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:41 smithi153 ceph-mon[36569]: pgmap v18682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:42.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:41 smithi104 ceph-mon[36624]: pgmap v18682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:42.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:41 smithi081 ceph-mon[31453]: pgmap v18682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:44.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:43 smithi153 ceph-mon[36569]: pgmap v18683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:44.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:43 smithi104 ceph-mon[36624]: pgmap v18683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:44.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:43 smithi081 ceph-mon[31453]: pgmap v18683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:46.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:45 smithi153 ceph-mon[36569]: pgmap v18684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:46.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:45 smithi104 ceph-mon[36624]: pgmap v18684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:46.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:45 smithi081 ceph-mon[31453]: pgmap v18684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:48.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:47 smithi153 ceph-mon[36569]: pgmap v18685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:48.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:47 smithi104 ceph-mon[36624]: pgmap v18685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:48.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:47 smithi081 ceph-mon[31453]: pgmap v18685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:49.500+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T06:59:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:49.500+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T06:59:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T06:59:49.500+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T06:59:50.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:49 smithi153 ceph-mon[36569]: pgmap v18686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:50.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:49 smithi104 ceph-mon[36624]: pgmap v18686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:50.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:49 smithi081 ceph-mon[31453]: pgmap v18686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:51 smithi153 ceph-mon[36569]: pgmap v18687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:52.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:51 smithi104 ceph-mon[36624]: pgmap v18687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:52.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:51 smithi081 ceph-mon[31453]: pgmap v18687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:54.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:53 smithi153 ceph-mon[36569]: pgmap v18688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:54.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:53 smithi104 ceph-mon[36624]: pgmap v18688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:54.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:53 smithi081 ceph-mon[31453]: pgmap v18688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:56.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:55 smithi153 ceph-mon[36569]: pgmap v18689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:56.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:55 smithi104 ceph-mon[36624]: pgmap v18689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:56.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:55 smithi081 ceph-mon[31453]: pgmap v18689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:58.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:57 smithi153 ceph-mon[36569]: pgmap v18690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:58.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:57 smithi104 ceph-mon[36624]: pgmap v18690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T06:59:58.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:57 smithi081 ceph-mon[31453]: pgmap v18690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:00.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 06:59:59 smithi153 ceph-mon[36569]: pgmap v18691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:00.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 06:59:59 smithi104 ceph-mon[36624]: pgmap v18691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:00.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 06:59:59 smithi081 ceph-mon[31453]: pgmap v18691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:01.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:00:01.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:00:01.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:00:02.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:01 smithi153 ceph-mon[36569]: pgmap v18692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:02.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:01 smithi104 ceph-mon[36624]: pgmap v18692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:02.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:01 smithi081 ceph-mon[31453]: pgmap v18692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:04.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:03 smithi153 ceph-mon[36569]: pgmap v18693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:04.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:00:04.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:04.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:04.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:03 smithi104 ceph-mon[36624]: pgmap v18693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:04.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:00:04.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:04.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:04.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:03 smithi081 ceph-mon[31453]: pgmap v18693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:04.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:00:04.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:04.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:04.515+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:00:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:04.515+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:00:04.531 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:04.515+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:00:05.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:00:05.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:00:05.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:04 smithi081 ceph-mon[31453]: pgmap v18694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:05.074 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:04 smithi081 ceph-mon[31453]: pgmap v18695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:05.075 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:05.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:00:05.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:00:05.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:04 smithi104 ceph-mon[36624]: pgmap v18694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:05.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:04 smithi104 ceph-mon[36624]: pgmap v18695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:05.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:05.321 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:00:05.321 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:00:05.321 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:04 smithi153 ceph-mon[36569]: pgmap v18694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:05.321 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:04 smithi153 ceph-mon[36569]: pgmap v18695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:05.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:00:07.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:07 smithi081 ceph-mon[31453]: pgmap v18696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:07.799 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:07 smithi104 ceph-mon[36624]: pgmap v18696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:07.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:07 smithi153 ceph-mon[36569]: pgmap v18696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:09.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:09 smithi081 ceph-mon[31453]: pgmap v18697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:09.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:09 smithi153 ceph-mon[36569]: pgmap v18697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:09.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:09 smithi104 ceph-mon[36624]: pgmap v18697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:11.597 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:11 smithi104 ceph-mon[36624]: pgmap v18698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:11.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:11 smithi081 ceph-mon[31453]: pgmap v18698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:11.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:11 smithi153 ceph-mon[36569]: pgmap v18698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:13.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:13 smithi081 ceph-mon[31453]: pgmap v18699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:13.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:13 smithi153 ceph-mon[36569]: pgmap v18699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:13.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:13 smithi104 ceph-mon[36624]: pgmap v18699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:15.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:15 smithi081 ceph-mon[31453]: pgmap v18700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:15.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:15 smithi153 ceph-mon[36569]: pgmap v18700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:15.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:15 smithi104 ceph-mon[36624]: pgmap v18700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:17.670 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:17 smithi081 ceph-mon[31453]: pgmap v18701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:17.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:17 smithi153 ceph-mon[36569]: pgmap v18701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:17.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:17 smithi104 ceph-mon[36624]: pgmap v18701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:19.474 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:19 smithi081 ceph-mon[31453]: pgmap v18702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:19.523+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:00:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:19.523+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:00:19.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:19.523+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:00:19.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:19 smithi153 ceph-mon[36569]: pgmap v18702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:19.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:19 smithi104 ceph-mon[36624]: pgmap v18702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:21.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:21 smithi081 ceph-mon[31453]: pgmap v18703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:21.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:21 smithi153 ceph-mon[36569]: pgmap v18703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:21.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:21 smithi104 ceph-mon[36624]: pgmap v18703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:23.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:23 smithi081 ceph-mon[31453]: pgmap v18704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:23.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:23 smithi153 ceph-mon[36569]: pgmap v18704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:23.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:23 smithi104 ceph-mon[36624]: pgmap v18704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:25.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:25 smithi081 ceph-mon[31453]: pgmap v18705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:25.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:25 smithi153 ceph-mon[36569]: pgmap v18705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:25.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:25 smithi104 ceph-mon[36624]: pgmap v18705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:27.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:27 smithi081 ceph-mon[31453]: pgmap v18706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:27.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:27 smithi153 ceph-mon[36569]: pgmap v18706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:27.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:27 smithi104 ceph-mon[36624]: pgmap v18706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:29.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:29 smithi081 ceph-mon[31453]: pgmap v18707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:29.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:29 smithi153 ceph-mon[36569]: pgmap v18707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:29.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:29 smithi104 ceph-mon[36624]: pgmap v18707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:31.572 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:31 smithi153 ceph-mon[36569]: pgmap v18708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:31.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:31 smithi081 ceph-mon[31453]: pgmap v18708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:31.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:31 smithi104 ceph-mon[36624]: pgmap v18708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:33.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:33 smithi153 ceph-mon[36569]: pgmap v18709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:33.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:33 smithi104 ceph-mon[36624]: pgmap v18709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:33.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:33 smithi081 ceph-mon[31453]: pgmap v18709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:34.527+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:00:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:34.527+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:00:34.542 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:34.527+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:00:35.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:35 smithi153 ceph-mon[36569]: pgmap v18710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:35.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:35 smithi104 ceph-mon[36624]: pgmap v18710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:35.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:35 smithi081 ceph-mon[31453]: pgmap v18710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:37.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:37 smithi153 ceph-mon[36569]: pgmap v18711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:37.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:37 smithi104 ceph-mon[36624]: pgmap v18711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:37.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:37 smithi081 ceph-mon[31453]: pgmap v18711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:39.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:39 smithi153 ceph-mon[36569]: pgmap v18712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:39.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:39 smithi104 ceph-mon[36624]: pgmap v18712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:39.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:39 smithi081 ceph-mon[31453]: pgmap v18712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:41.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:41 smithi153 ceph-mon[36569]: pgmap v18713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:41.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:41 smithi104 ceph-mon[36624]: pgmap v18713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:41.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:41 smithi081 ceph-mon[31453]: pgmap v18713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:43.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:43 smithi153 ceph-mon[36569]: pgmap v18714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:43.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:43 smithi104 ceph-mon[36624]: pgmap v18714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:43.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:43 smithi081 ceph-mon[31453]: pgmap v18714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:45.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:45 smithi153 ceph-mon[36569]: pgmap v18715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:45.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:45 smithi104 ceph-mon[36624]: pgmap v18715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:45.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:45 smithi081 ceph-mon[31453]: pgmap v18715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:47.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:47 smithi153 ceph-mon[36569]: pgmap v18716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:47.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:47 smithi104 ceph-mon[36624]: pgmap v18716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:47.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:47 smithi081 ceph-mon[31453]: pgmap v18716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:49.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:49.543+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:00:49.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:49.543+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:00:49.558 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:00:49.543+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:00:49.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:49 smithi153 ceph-mon[36569]: pgmap v18717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:49.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:49 smithi104 ceph-mon[36624]: pgmap v18717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:49.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:49 smithi081 ceph-mon[31453]: pgmap v18717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:51.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:51 smithi153 ceph-mon[36569]: pgmap v18718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:51.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:51 smithi104 ceph-mon[36624]: pgmap v18718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:51.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:51 smithi081 ceph-mon[31453]: pgmap v18718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:53.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:53 smithi153 ceph-mon[36569]: pgmap v18719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:53.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:53 smithi104 ceph-mon[36624]: pgmap v18719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:53.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:53 smithi081 ceph-mon[31453]: pgmap v18719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:55.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:55 smithi153 ceph-mon[36569]: pgmap v18720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:55.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:55 smithi104 ceph-mon[36624]: pgmap v18720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:55.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:55 smithi081 ceph-mon[31453]: pgmap v18720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:57.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:57 smithi153 ceph-mon[36569]: pgmap v18721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:57.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:57 smithi104 ceph-mon[36624]: pgmap v18721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:57.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:57 smithi081 ceph-mon[31453]: pgmap v18721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:59.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:00:59 smithi153 ceph-mon[36569]: pgmap v18722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:59.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:00:59 smithi104 ceph-mon[36624]: pgmap v18722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:00:59.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:00:59 smithi081 ceph-mon[31453]: pgmap v18722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:01.755 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:01 smithi153 ceph-mon[36569]: pgmap v18723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:01.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:01 smithi104 ceph-mon[36624]: pgmap v18723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:01.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:01 smithi081 ceph-mon[31453]: pgmap v18723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:03.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:03 smithi153 ceph-mon[36569]: pgmap v18724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:03 smithi104 ceph-mon[36624]: pgmap v18724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:03.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:03 smithi081 ceph-mon[31453]: pgmap v18724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:04.573 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:04.558+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:01:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:04.558+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:01:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:04.558+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:01:05.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:05 smithi081 ceph-mon[31453]: pgmap v18725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:05.671 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:01:05.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:05 smithi153 ceph-mon[36569]: pgmap v18725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:05.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:01:05.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:05 smithi104 ceph-mon[36624]: pgmap v18725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:05.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:01:06.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:01:06.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:01:06.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:01:06.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:01:06.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:01:06.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:01:06.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:01:06.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:01:06.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:01:07.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:07 smithi153 ceph-mon[36569]: pgmap v18726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:07.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:07 smithi153 ceph-mon[36569]: pgmap v18727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:07.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:07 smithi104 ceph-mon[36624]: pgmap v18726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:07.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:07 smithi104 ceph-mon[36624]: pgmap v18727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:07.923 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:07 smithi081 ceph-mon[31453]: pgmap v18726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:07.923 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:07 smithi081 ceph-mon[31453]: pgmap v18727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:09.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:09 smithi153 ceph-mon[36569]: pgmap v18728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:09.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:09 smithi104 ceph-mon[36624]: pgmap v18728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:09.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:09 smithi081 ceph-mon[31453]: pgmap v18728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:11.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:11 smithi153 ceph-mon[36569]: pgmap v18729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:11.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:11 smithi104 ceph-mon[36624]: pgmap v18729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:11.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:11 smithi081 ceph-mon[31453]: pgmap v18729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:12.186+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000815) for more than 900000000 us, fault. 2024-04-13T07:01:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:12.186+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006110 tx=0x7f16f40ba000 comp rx=0 tx=0).stop 2024-04-13T07:01:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:12.186+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T07:01:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:12.186+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T07:01:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:12.186+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T07:01:12.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:12.187+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T07:01:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:12.188+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ce5f0 tx=0x7f16f4032000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T07:01:13.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:13 smithi153 ceph-mon[36569]: pgmap v18730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:13.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:13 smithi104 ceph-mon[36624]: pgmap v18730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:13.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:13 smithi081 ceph-mon[31453]: pgmap v18730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:15.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:15 smithi153 ceph-mon[36569]: pgmap v18731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:15.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:15 smithi104 ceph-mon[36624]: pgmap v18731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:15.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:15 smithi081 ceph-mon[31453]: pgmap v18731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:17.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:17 smithi153 ceph-mon[36569]: pgmap v18732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:17.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:17 smithi104 ceph-mon[36624]: pgmap v18732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:17.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:17 smithi081 ceph-mon[31453]: pgmap v18732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:18.511+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:01:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:18.511+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:01:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:18.511+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 270 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:01:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:18.511+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 270 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:01:19.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:19.561+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:01:19.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:19.561+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:01:19.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:19.561+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:01:19.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:19 smithi153 ceph-mon[36569]: pgmap v18733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:19.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:19 smithi104 ceph-mon[36624]: pgmap v18733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:19.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:19 smithi081 ceph-mon[31453]: pgmap v18733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:21.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:21 smithi153 ceph-mon[36569]: pgmap v18734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:21.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:21 smithi104 ceph-mon[36624]: pgmap v18734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:21.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:21 smithi081 ceph-mon[31453]: pgmap v18734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:23.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:23 smithi153 ceph-mon[36569]: pgmap v18735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:23.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:23 smithi104 ceph-mon[36624]: pgmap v18735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:23.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:23 smithi081 ceph-mon[31453]: pgmap v18735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:25.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:25 smithi153 ceph-mon[36569]: pgmap v18736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:25.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:25 smithi104 ceph-mon[36624]: pgmap v18736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:25.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:25 smithi081 ceph-mon[31453]: pgmap v18736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:27.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:27 smithi104 ceph-mon[36624]: pgmap v18737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:27.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:27 smithi081 ceph-mon[31453]: pgmap v18737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:28.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:27 smithi153 ceph-mon[36569]: pgmap v18737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:29.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:29 smithi104 ceph-mon[36624]: pgmap v18738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:29.874 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:29 smithi153 ceph-mon[36569]: pgmap v18738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:29.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:29 smithi081 ceph-mon[31453]: pgmap v18738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:31.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:31 smithi104 ceph-mon[36624]: pgmap v18739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:31.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:31 smithi081 ceph-mon[31453]: pgmap v18739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:32.074 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:31 smithi153 ceph-mon[36569]: pgmap v18739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:33.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:33 smithi104 ceph-mon[36624]: pgmap v18740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:33.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:33 smithi081 ceph-mon[31453]: pgmap v18740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:34.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:33 smithi153 ceph-mon[36569]: pgmap v18740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:34.576+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:01:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:34.576+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:01:34.592 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:34.576+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:01:35.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:35 smithi104 ceph-mon[36624]: pgmap v18741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:35.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:35 smithi081 ceph-mon[31453]: pgmap v18741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:36.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:35 smithi153 ceph-mon[36569]: pgmap v18741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:37.922 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:37 smithi081 ceph-mon[31453]: pgmap v18742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:38.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:37 smithi153 ceph-mon[36569]: pgmap v18742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:38.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:37 smithi104 ceph-mon[36624]: pgmap v18742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:39.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:39 smithi081 ceph-mon[31453]: pgmap v18743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:40.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:39 smithi153 ceph-mon[36569]: pgmap v18743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:40.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:39 smithi104 ceph-mon[36624]: pgmap v18743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:41.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:41 smithi081 ceph-mon[31453]: pgmap v18744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:42.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:41 smithi153 ceph-mon[36569]: pgmap v18744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:42.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:41 smithi104 ceph-mon[36624]: pgmap v18744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:43.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:43 smithi081 ceph-mon[31453]: pgmap v18745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:44.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:43 smithi153 ceph-mon[36569]: pgmap v18745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:44.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:43 smithi104 ceph-mon[36624]: pgmap v18745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:45.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:45 smithi081 ceph-mon[31453]: pgmap v18746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:46.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:45 smithi153 ceph-mon[36569]: pgmap v18746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:46.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:45 smithi104 ceph-mon[36624]: pgmap v18746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:47.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:47 smithi081 ceph-mon[31453]: pgmap v18747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:48.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:47 smithi153 ceph-mon[36569]: pgmap v18747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:48.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:47 smithi104 ceph-mon[36624]: pgmap v18747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:49.606 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:49.591+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:01:49.607 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:49.591+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:01:49.607 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:01:49.591+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:01:49.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:49 smithi081 ceph-mon[31453]: pgmap v18748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:50.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:49 smithi153 ceph-mon[36569]: pgmap v18748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:50.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:49 smithi104 ceph-mon[36624]: pgmap v18748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:51.894 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:51 smithi081 ceph-mon[31453]: pgmap v18749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:52.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:51 smithi153 ceph-mon[36569]: pgmap v18749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:52.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:51 smithi104 ceph-mon[36624]: pgmap v18749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:53.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:53 smithi081 ceph-mon[31453]: pgmap v18750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:54.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:53 smithi153 ceph-mon[36569]: pgmap v18750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:54.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:53 smithi104 ceph-mon[36624]: pgmap v18750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:55.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:55 smithi081 ceph-mon[31453]: pgmap v18751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:56.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:55 smithi153 ceph-mon[36569]: pgmap v18751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:56.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:55 smithi104 ceph-mon[36624]: pgmap v18751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:57.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:57 smithi081 ceph-mon[31453]: pgmap v18752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:58.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:57 smithi153 ceph-mon[36569]: pgmap v18752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:58.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:57 smithi104 ceph-mon[36624]: pgmap v18752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:01:59.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:01:59 smithi081 ceph-mon[31453]: pgmap v18753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:00.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:01:59 smithi153 ceph-mon[36569]: pgmap v18753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:00.098 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:01:59 smithi104 ceph-mon[36624]: pgmap v18753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:01.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:01 smithi081 ceph-mon[31453]: pgmap v18754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:02.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:01 smithi153 ceph-mon[36569]: pgmap v18754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:02.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:01 smithi104 ceph-mon[36624]: pgmap v18754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:03.518+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:02:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:03.518+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:02:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:03.518+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 271 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:02:03.534 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:03.518+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 271 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:02:03.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:03 smithi081 ceph-mon[31453]: pgmap v18755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:04.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:03 smithi153 ceph-mon[36569]: pgmap v18755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:04.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:03 smithi104 ceph-mon[36624]: pgmap v18755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:04.598+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:02:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:04.598+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:02:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:04.598+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:02:06.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:05 smithi153 ceph-mon[36569]: pgmap v18756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:06.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:05 smithi104 ceph-mon[36624]: pgmap v18756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:06.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:05 smithi081 ceph-mon[31453]: pgmap v18756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:07.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:02:07.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:02:07.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:02:07.999 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:07 smithi104 ceph-mon[36624]: pgmap v18757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:07.999 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:02:07.999 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:02:07.999 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:02:08.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:07 smithi153 ceph-mon[36569]: pgmap v18757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:08.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:02:08.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:02:08.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:02:08.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:07 smithi081 ceph-mon[31453]: pgmap v18757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:08.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:02:08.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:02:08.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:02:09.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:08 smithi153 ceph-mon[36569]: pgmap v18758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:09.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:08 smithi153 ceph-mon[36569]: pgmap v18759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:09.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:08 smithi104 ceph-mon[36624]: pgmap v18758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:09.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:08 smithi104 ceph-mon[36624]: pgmap v18759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:09.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:08 smithi081 ceph-mon[31453]: pgmap v18758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:09.174 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:08 smithi081 ceph-mon[31453]: pgmap v18759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:11.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:10 smithi153 ceph-mon[36569]: pgmap v18760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:11.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:10 smithi104 ceph-mon[36624]: pgmap v18760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:11.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:10 smithi081 ceph-mon[31453]: pgmap v18760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:13.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:12 smithi153 ceph-mon[36569]: pgmap v18761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:13.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:12 smithi104 ceph-mon[36624]: pgmap v18761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:12 smithi081 ceph-mon[31453]: pgmap v18761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:15.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:14 smithi153 ceph-mon[36569]: pgmap v18762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:15.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:14 smithi104 ceph-mon[36624]: pgmap v18762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:15.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:14 smithi081 ceph-mon[31453]: pgmap v18762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:17.073 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:16 smithi153 ceph-mon[36569]: pgmap v18763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:17.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:16 smithi104 ceph-mon[36624]: pgmap v18763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:17.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:16 smithi081 ceph-mon[31453]: pgmap v18763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:19.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:18 smithi153 ceph-mon[36569]: pgmap v18764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:19.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:18 smithi104 ceph-mon[36624]: pgmap v18764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:19.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:18 smithi081 ceph-mon[31453]: pgmap v18764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:19.614+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:19.615+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:02:19.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:19.615+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:02:21.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:20 smithi153 ceph-mon[36569]: pgmap v18765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:21.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:20 smithi104 ceph-mon[36624]: pgmap v18765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:21.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:20 smithi081 ceph-mon[31453]: pgmap v18765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:23.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:22 smithi153 ceph-mon[36569]: pgmap v18766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:23.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:22 smithi104 ceph-mon[36624]: pgmap v18766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:23.126 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:22 smithi081 ceph-mon[31453]: pgmap v18766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:25.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:24 smithi153 ceph-mon[36569]: pgmap v18767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:25.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:24 smithi104 ceph-mon[36624]: pgmap v18767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:25.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:24 smithi081 ceph-mon[31453]: pgmap v18767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:27.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:26 smithi153 ceph-mon[36569]: pgmap v18768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:27.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:26 smithi104 ceph-mon[36624]: pgmap v18768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:27.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:26 smithi081 ceph-mon[31453]: pgmap v18768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:29.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:28 smithi153 ceph-mon[36569]: pgmap v18769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:29.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:28 smithi104 ceph-mon[36624]: pgmap v18769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:29.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:28 smithi081 ceph-mon[31453]: pgmap v18769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:31.074 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:30 smithi153 ceph-mon[36569]: pgmap v18770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:31.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:30 smithi104 ceph-mon[36624]: pgmap v18770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:31.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:30 smithi081 ceph-mon[31453]: pgmap v18770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:33.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:32 smithi153 ceph-mon[36569]: pgmap v18771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:33.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:32 smithi104 ceph-mon[36624]: pgmap v18771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:33.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:32 smithi081 ceph-mon[31453]: pgmap v18771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:34.626+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:02:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:34.626+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:02:34.642 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:34.626+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:02:35.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:34 smithi153 ceph-mon[36569]: pgmap v18772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:35.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:34 smithi104 ceph-mon[36624]: pgmap v18772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:35.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:34 smithi081 ceph-mon[31453]: pgmap v18772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:37.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:36 smithi153 ceph-mon[36569]: pgmap v18773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:37.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:36 smithi104 ceph-mon[36624]: pgmap v18773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:37.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:36 smithi081 ceph-mon[31453]: pgmap v18773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:39.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:38 smithi153 ceph-mon[36569]: pgmap v18774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:39.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:38 smithi104 ceph-mon[36624]: pgmap v18774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:39.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:38 smithi081 ceph-mon[31453]: pgmap v18774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:41.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:40 smithi153 ceph-mon[36569]: pgmap v18775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:41.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:40 smithi104 ceph-mon[36624]: pgmap v18775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:41.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:40 smithi081 ceph-mon[31453]: pgmap v18775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:43.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:42 smithi153 ceph-mon[36569]: pgmap v18776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:43.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:42 smithi104 ceph-mon[36624]: pgmap v18776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:43.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:42 smithi081 ceph-mon[31453]: pgmap v18776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:45.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:44 smithi153 ceph-mon[36569]: pgmap v18777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:45.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:44 smithi104 ceph-mon[36624]: pgmap v18777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:45.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:44 smithi081 ceph-mon[31453]: pgmap v18777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:47.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:46 smithi153 ceph-mon[36569]: pgmap v18778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:47.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:46 smithi104 ceph-mon[36624]: pgmap v18778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:47.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:46 smithi081 ceph-mon[31453]: pgmap v18778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:49.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:48 smithi153 ceph-mon[36569]: pgmap v18779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:49.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:48 smithi104 ceph-mon[36624]: pgmap v18779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:49.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:48 smithi081 ceph-mon[31453]: pgmap v18779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:49.642+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:02:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:49.642+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:02:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:02:49.642+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:02:51.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:50 smithi153 ceph-mon[36569]: pgmap v18780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:51.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:50 smithi104 ceph-mon[36624]: pgmap v18780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:51.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:50 smithi081 ceph-mon[31453]: pgmap v18780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:53.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:52 smithi153 ceph-mon[36569]: pgmap v18781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:53.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:52 smithi104 ceph-mon[36624]: pgmap v18781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:53.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:52 smithi081 ceph-mon[31453]: pgmap v18781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:55.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:54 smithi153 ceph-mon[36569]: pgmap v18782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:55.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:54 smithi104 ceph-mon[36624]: pgmap v18782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:55.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:54 smithi081 ceph-mon[31453]: pgmap v18782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:57.072 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:56 smithi153 ceph-mon[36569]: pgmap v18783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:57.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:56 smithi104 ceph-mon[36624]: pgmap v18783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:57.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:56 smithi081 ceph-mon[31453]: pgmap v18783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:59.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:02:58 smithi104 ceph-mon[36624]: pgmap v18784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:59.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:02:58 smithi081 ceph-mon[31453]: pgmap v18784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:02:59.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:02:58 smithi153 ceph-mon[36569]: pgmap v18784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:01.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:00 smithi104 ceph-mon[36624]: pgmap v18785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:01.124 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:00 smithi153 ceph-mon[36569]: pgmap v18785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:01.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:00 smithi081 ceph-mon[31453]: pgmap v18785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:03.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:02 smithi104 ceph-mon[36624]: pgmap v18786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:03.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:02 smithi081 ceph-mon[31453]: pgmap v18786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:03.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:02 smithi153 ceph-mon[36569]: pgmap v18786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:04.649+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:03:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:04.649+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:03:04.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:04.649+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:03:05.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:04 smithi104 ceph-mon[36624]: pgmap v18787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:05.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:04 smithi081 ceph-mon[31453]: pgmap v18787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:05.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:04 smithi153 ceph-mon[36569]: pgmap v18787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:07.097 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:06 smithi104 ceph-mon[36624]: pgmap v18788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:07.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:06 smithi081 ceph-mon[31453]: pgmap v18788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:07.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:06 smithi153 ceph-mon[36569]: pgmap v18788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:08.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:03:08.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:03:08.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:03:09.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:08 smithi081 ceph-mon[31453]: pgmap v18789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:09.249 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:08 smithi104 ceph-mon[36624]: pgmap v18789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:09.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:08 smithi153 ceph-mon[36569]: pgmap v18789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:10.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:03:10.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:03:10.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:09 smithi081 ceph-mon[31453]: pgmap v18790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:10.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:09 smithi081 ceph-mon[31453]: pgmap v18791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:10.172 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:03:10.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:03:10.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:03:10.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:09 smithi153 ceph-mon[36569]: pgmap v18790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:10.323 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:09 smithi153 ceph-mon[36569]: pgmap v18791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:10.323 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:03:10.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:03:10.348 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:03:10.348 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:09 smithi104 ceph-mon[36624]: pgmap v18790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:10.348 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:09 smithi104 ceph-mon[36624]: pgmap v18791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:10.348 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:03:12.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:11 smithi153 ceph-mon[36569]: pgmap v18792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:12.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:11 smithi104 ceph-mon[36624]: pgmap v18792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:12.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:11 smithi081 ceph-mon[31453]: pgmap v18792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:14.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:13 smithi153 ceph-mon[36569]: pgmap v18793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:14.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:13 smithi104 ceph-mon[36624]: pgmap v18793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:14.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:13 smithi081 ceph-mon[31453]: pgmap v18793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:16.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:15 smithi153 ceph-mon[36569]: pgmap v18794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:16.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:15 smithi104 ceph-mon[36624]: pgmap v18794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:16.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:15 smithi081 ceph-mon[31453]: pgmap v18794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:18.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:17 smithi153 ceph-mon[36569]: pgmap v18795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:18.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:17 smithi104 ceph-mon[36624]: pgmap v18795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:18.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:17 smithi081 ceph-mon[31453]: pgmap v18795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:19.665+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:03:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:19.665+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:03:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:19.665+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:03:20.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:19 smithi153 ceph-mon[36569]: pgmap v18796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:20.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:19 smithi104 ceph-mon[36624]: pgmap v18796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:20.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:19 smithi081 ceph-mon[31453]: pgmap v18796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:22.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:21 smithi153 ceph-mon[36569]: pgmap v18797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:22.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:21 smithi104 ceph-mon[36624]: pgmap v18797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:22.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:21 smithi081 ceph-mon[31453]: pgmap v18797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:24.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:23 smithi153 ceph-mon[36569]: pgmap v18798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:24.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:23 smithi104 ceph-mon[36624]: pgmap v18798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:24.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:23 smithi081 ceph-mon[31453]: pgmap v18798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:26.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:25 smithi153 ceph-mon[36569]: pgmap v18799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:26.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:25 smithi104 ceph-mon[36624]: pgmap v18799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:26.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:25 smithi081 ceph-mon[31453]: pgmap v18799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:28.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:27 smithi153 ceph-mon[36569]: pgmap v18800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:28.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:27 smithi104 ceph-mon[36624]: pgmap v18800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:28.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:27 smithi081 ceph-mon[31453]: pgmap v18800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:30.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:29 smithi153 ceph-mon[36569]: pgmap v18801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:30.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:29 smithi104 ceph-mon[36624]: pgmap v18801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:30.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:29 smithi081 ceph-mon[31453]: pgmap v18801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:32.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:31 smithi153 ceph-mon[36569]: pgmap v18802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:32.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:31 smithi104 ceph-mon[36624]: pgmap v18802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:32.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:31 smithi081 ceph-mon[31453]: pgmap v18802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:34.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:34 smithi153 ceph-mon[36569]: pgmap v18803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:34.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:34 smithi104 ceph-mon[36624]: pgmap v18803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:34.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:34 smithi081 ceph-mon[31453]: pgmap v18803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:34.696 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:34.681+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:03:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:34.681+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:03:34.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:34.681+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:03:36.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:36 smithi153 ceph-mon[36569]: pgmap v18804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:36.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:36 smithi104 ceph-mon[36624]: pgmap v18804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:36.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:36 smithi081 ceph-mon[31453]: pgmap v18804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:38.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:38 smithi153 ceph-mon[36569]: pgmap v18805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:38.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:38 smithi104 ceph-mon[36624]: pgmap v18805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:38.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:38 smithi081 ceph-mon[31453]: pgmap v18805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:40.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:40 smithi153 ceph-mon[36569]: pgmap v18806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:40.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:40 smithi104 ceph-mon[36624]: pgmap v18806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:40.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:40 smithi081 ceph-mon[31453]: pgmap v18806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:42.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:42 smithi153 ceph-mon[36569]: pgmap v18807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:42.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:42 smithi104 ceph-mon[36624]: pgmap v18807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:42.422 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:42 smithi081 ceph-mon[31453]: pgmap v18807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:44.322 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:44 smithi153 ceph-mon[36569]: pgmap v18808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:44.347 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:44 smithi104 ceph-mon[36624]: pgmap v18808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:44.421 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:44 smithi081 ceph-mon[31453]: pgmap v18808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:46.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:46 smithi153 ceph-mon[36569]: pgmap v18809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:46.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:46 smithi104 ceph-mon[36624]: pgmap v18809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:46.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:46 smithi081 ceph-mon[31453]: pgmap v18809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:48.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:48 smithi153 ceph-mon[36569]: pgmap v18810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:48.848 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:48 smithi104 ceph-mon[36624]: pgmap v18810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:48.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:48 smithi081 ceph-mon[31453]: pgmap v18810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:49.694+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:03:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:49.694+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:03:49.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:03:49.694+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:03:50.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:50 smithi153 ceph-mon[36569]: pgmap v18811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:50.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:50 smithi104 ceph-mon[36624]: pgmap v18811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:50.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:50 smithi081 ceph-mon[31453]: pgmap v18811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:52.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:52 smithi153 ceph-mon[36569]: pgmap v18812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:52.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:52 smithi104 ceph-mon[36624]: pgmap v18812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:52.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:52 smithi081 ceph-mon[31453]: pgmap v18812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:54.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:54 smithi153 ceph-mon[36569]: pgmap v18813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:54.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:54 smithi104 ceph-mon[36624]: pgmap v18813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:54.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:54 smithi081 ceph-mon[31453]: pgmap v18813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:56.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:56 smithi153 ceph-mon[36569]: pgmap v18814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:56.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:56 smithi104 ceph-mon[36624]: pgmap v18814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:56.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:56 smithi081 ceph-mon[31453]: pgmap v18814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:58.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:03:58 smithi153 ceph-mon[36569]: pgmap v18815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:58.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:03:58 smithi104 ceph-mon[36624]: pgmap v18815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:03:58.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:03:58 smithi081 ceph-mon[31453]: pgmap v18815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:00.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:00 smithi153 ceph-mon[36569]: pgmap v18816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:00.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:00 smithi104 ceph-mon[36624]: pgmap v18816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:00.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:00 smithi081 ceph-mon[31453]: pgmap v18816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:02.822 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:02 smithi153 ceph-mon[36569]: pgmap v18817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:02.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:02 smithi104 ceph-mon[36624]: pgmap v18817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:02.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:02 smithi081 ceph-mon[31453]: pgmap v18817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:04.706+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:04:04.721 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:04.706+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:04:04.722 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:04.706+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:04:04.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:04 smithi153 ceph-mon[36569]: pgmap v18818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:04.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:04 smithi104 ceph-mon[36624]: pgmap v18818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:04.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:04 smithi081 ceph-mon[31453]: pgmap v18818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:06.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:06 smithi153 ceph-mon[36569]: pgmap v18819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:06.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:06 smithi104 ceph-mon[36624]: pgmap v18819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:06.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:06 smithi081 ceph-mon[31453]: pgmap v18819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:08.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:08 smithi153 ceph-mon[36569]: pgmap v18820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:08.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:08 smithi104 ceph-mon[36624]: pgmap v18820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:08.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:08 smithi081 ceph-mon[31453]: pgmap v18820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:09.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:04:09.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:04:09.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:04:10.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:10 smithi153 ceph-mon[36569]: pgmap v18821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:10.857 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:04:10.857 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:04:10.857 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:04:10.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:10 smithi104 ceph-mon[36624]: pgmap v18821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:10.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:04:10.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:04:10.858 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:04:10.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:10 smithi081 ceph-mon[31453]: pgmap v18821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:10.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:04:10.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:04:10.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:04:11.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:11 smithi153 ceph-mon[36569]: pgmap v18822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:11.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:11 smithi153 ceph-mon[36569]: pgmap v18823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:11.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:11 smithi104 ceph-mon[36624]: pgmap v18822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:11.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:11 smithi104 ceph-mon[36624]: pgmap v18823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:11.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:11 smithi081 ceph-mon[31453]: pgmap v18822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:11.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:11 smithi081 ceph-mon[31453]: pgmap v18823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:13.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:13 smithi153 ceph-mon[36569]: pgmap v18824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:13.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:13 smithi104 ceph-mon[36624]: pgmap v18824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:13.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:13 smithi081 ceph-mon[31453]: pgmap v18824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:15.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:15 smithi153 ceph-mon[36569]: pgmap v18825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:15.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:15 smithi104 ceph-mon[36624]: pgmap v18825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:15.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:15 smithi081 ceph-mon[31453]: pgmap v18825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:17.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:17 smithi153 ceph-mon[36569]: pgmap v18826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:17.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:17 smithi104 ceph-mon[36624]: pgmap v18826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:17.921 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:17 smithi081 ceph-mon[31453]: pgmap v18826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:18.541+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:04:18.556 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:18.541+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:04:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:18.542+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 272 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:04:18.557 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:18.542+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 272 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:04:19.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:19.721+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:04:19.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:19.721+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:04:19.736 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:19.721+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:04:19.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:19 smithi153 ceph-mon[36569]: pgmap v18827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:19.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:19 smithi104 ceph-mon[36624]: pgmap v18827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:19.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:19 smithi081 ceph-mon[31453]: pgmap v18827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:21.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:21 smithi153 ceph-mon[36569]: pgmap v18828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:21.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:21 smithi104 ceph-mon[36624]: pgmap v18828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:21.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:21 smithi081 ceph-mon[31453]: pgmap v18828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:23.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:23 smithi153 ceph-mon[36569]: pgmap v18829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:23.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:23 smithi104 ceph-mon[36624]: pgmap v18829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:23.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:23 smithi081 ceph-mon[31453]: pgmap v18829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:25.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:25 smithi153 ceph-mon[36569]: pgmap v18830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:25.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:25 smithi104 ceph-mon[36624]: pgmap v18830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:25.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:25 smithi081 ceph-mon[31453]: pgmap v18830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:27.821 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:27 smithi153 ceph-mon[36569]: pgmap v18831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:27.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:27 smithi104 ceph-mon[36624]: pgmap v18831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:27.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:27 smithi081 ceph-mon[31453]: pgmap v18831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:29.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:29 smithi104 ceph-mon[36624]: pgmap v18832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:29.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:29 smithi081 ceph-mon[31453]: pgmap v18832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:30.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:29 smithi153 ceph-mon[36569]: pgmap v18832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:31.850 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:31 smithi104 ceph-mon[36624]: pgmap v18833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:31.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:31 smithi081 ceph-mon[31453]: pgmap v18833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:32.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:31 smithi153 ceph-mon[36569]: pgmap v18833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:33.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:33 smithi104 ceph-mon[36624]: pgmap v18834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:33.873 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:33 smithi153 ceph-mon[36569]: pgmap v18834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:33.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:33 smithi081 ceph-mon[31453]: pgmap v18834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:34.724+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:04:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:34.724+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:04:34.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:34.724+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:04:35.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:35 smithi104 ceph-mon[36624]: pgmap v18835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:35.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:35 smithi081 ceph-mon[31453]: pgmap v18835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:36.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:35 smithi153 ceph-mon[36569]: pgmap v18835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:37.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:37 smithi104 ceph-mon[36624]: pgmap v18836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:37.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:37 smithi081 ceph-mon[31453]: pgmap v18836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:38.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:37 smithi153 ceph-mon[36569]: pgmap v18836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:39.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:39 smithi081 ceph-mon[31453]: pgmap v18837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:40.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:39 smithi153 ceph-mon[36569]: pgmap v18837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:40.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:39 smithi104 ceph-mon[36624]: pgmap v18837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:41.898 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:41 smithi104 ceph-mon[36624]: pgmap v18838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:41.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:41 smithi081 ceph-mon[31453]: pgmap v18838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:42.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:41 smithi153 ceph-mon[36569]: pgmap v18838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:43.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:43 smithi081 ceph-mon[31453]: pgmap v18839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:44.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:43 smithi153 ceph-mon[36569]: pgmap v18839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:44.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:43 smithi104 ceph-mon[36624]: pgmap v18839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:45.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:45 smithi081 ceph-mon[31453]: pgmap v18840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:46.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:45 smithi153 ceph-mon[36569]: pgmap v18840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:46.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:45 smithi104 ceph-mon[36624]: pgmap v18840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:47.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:47 smithi081 ceph-mon[31453]: pgmap v18841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:48.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:47 smithi153 ceph-mon[36569]: pgmap v18841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:48.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:47 smithi104 ceph-mon[36624]: pgmap v18841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:49.739+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:04:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:49.739+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:04:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:04:49.739+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:04:49.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:49 smithi081 ceph-mon[31453]: pgmap v18842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:50.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:49 smithi153 ceph-mon[36569]: pgmap v18842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:50.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:49 smithi104 ceph-mon[36624]: pgmap v18842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:51.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:51 smithi081 ceph-mon[31453]: pgmap v18843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:52.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:51 smithi153 ceph-mon[36569]: pgmap v18843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:52.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:51 smithi104 ceph-mon[36624]: pgmap v18843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:53.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:53 smithi081 ceph-mon[31453]: pgmap v18844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:54.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:53 smithi153 ceph-mon[36569]: pgmap v18844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:54.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:53 smithi104 ceph-mon[36624]: pgmap v18844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:55.920 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:55 smithi081 ceph-mon[31453]: pgmap v18845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:56.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:55 smithi153 ceph-mon[36569]: pgmap v18845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:56.098 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:55 smithi104 ceph-mon[36624]: pgmap v18845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:57.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:57 smithi081 ceph-mon[31453]: pgmap v18846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:58.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:57 smithi153 ceph-mon[36569]: pgmap v18846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:58.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:57 smithi104 ceph-mon[36624]: pgmap v18846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:04:59.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:04:59 smithi081 ceph-mon[31453]: pgmap v18847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:00.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:04:59 smithi153 ceph-mon[36569]: pgmap v18847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:00.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:04:59 smithi104 ceph-mon[36624]: pgmap v18847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:01.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:01 smithi081 ceph-mon[31453]: pgmap v18848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:02.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:01 smithi153 ceph-mon[36569]: pgmap v18848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:02.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:01 smithi104 ceph-mon[36624]: pgmap v18848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:03.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:03 smithi081 ceph-mon[31453]: pgmap v18849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:04.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:03 smithi153 ceph-mon[36569]: pgmap v18849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:04.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:03 smithi104 ceph-mon[36624]: pgmap v18849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:04.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:04.752+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:05:04.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:04.752+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:05:04.766 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:04.752+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:05:05.919 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:05 smithi081 ceph-mon[31453]: pgmap v18850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:06.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:05 smithi153 ceph-mon[36569]: pgmap v18850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:06.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:05 smithi104 ceph-mon[36624]: pgmap v18850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:08.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:07 smithi153 ceph-mon[36569]: pgmap v18851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:08.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:07 smithi104 ceph-mon[36624]: pgmap v18851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:08.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:07 smithi081 ceph-mon[31453]: pgmap v18851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:10.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:09 smithi153 ceph-mon[36569]: pgmap v18852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:10.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:09 smithi104 ceph-mon[36624]: pgmap v18852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:10.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:09 smithi081 ceph-mon[31453]: pgmap v18852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:12.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: pgmap v18853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:12.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:05:12.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: pgmap v18853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:12.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:05:12.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: pgmap v18853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:12.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:05:12.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:12.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:05:13.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:05:13.071 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:05:13.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:05:13.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:05:13.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:05:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:05:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:05:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:05:13.096 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:05:13.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:05:13.171 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:05:14.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:13 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:05:14.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:13 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:05:14.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:13 smithi153 ceph-mon[36569]: pgmap v18854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:14.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:13 smithi153 ceph-mon[36569]: pgmap v18855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:14.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:13 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:05:14.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:13 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:05:14.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:13 smithi104 ceph-mon[36624]: pgmap v18854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:14.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:13 smithi104 ceph-mon[36624]: pgmap v18855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:14.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:13 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:05:14.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:13 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:05:14.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:13 smithi081 ceph-mon[31453]: pgmap v18854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:14.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:13 smithi081 ceph-mon[31453]: pgmap v18855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:16.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:15 smithi153 ceph-mon[36569]: pgmap v18856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:16.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:15 smithi104 ceph-mon[36624]: pgmap v18856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:16.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:15 smithi081 ceph-mon[31453]: pgmap v18856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:18.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:17 smithi153 ceph-mon[36569]: pgmap v18857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:18.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:17 smithi104 ceph-mon[36624]: pgmap v18857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:18.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:17 smithi081 ceph-mon[31453]: pgmap v18857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:19.768+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:05:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:19.768+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:05:19.781 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:19.768+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:05:20.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:19 smithi153 ceph-mon[36569]: pgmap v18858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:20.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:19 smithi104 ceph-mon[36624]: pgmap v18858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:20.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:19 smithi081 ceph-mon[31453]: pgmap v18858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:22.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:21 smithi153 ceph-mon[36569]: pgmap v18859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:22.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:21 smithi104 ceph-mon[36624]: pgmap v18859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:22.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:21 smithi081 ceph-mon[31453]: pgmap v18859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:24.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:23 smithi153 ceph-mon[36569]: pgmap v18860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:24.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:23 smithi104 ceph-mon[36624]: pgmap v18860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:24.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:23 smithi081 ceph-mon[31453]: pgmap v18860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:26.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:25 smithi153 ceph-mon[36569]: pgmap v18861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:26.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:25 smithi104 ceph-mon[36624]: pgmap v18861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:26.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:25 smithi081 ceph-mon[31453]: pgmap v18861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:28.070 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:27 smithi153 ceph-mon[36569]: pgmap v18862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:28.095 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:27 smithi104 ceph-mon[36624]: pgmap v18862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:28.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:27 smithi081 ceph-mon[31453]: pgmap v18862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:30.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:29 smithi153 ceph-mon[36569]: pgmap v18863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:30.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:29 smithi104 ceph-mon[36624]: pgmap v18863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:30.169 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:29 smithi081 ceph-mon[31453]: pgmap v18863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:32.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:31 smithi153 ceph-mon[36569]: pgmap v18864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:32.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:31 smithi104 ceph-mon[36624]: pgmap v18864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:32.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:31 smithi081 ceph-mon[31453]: pgmap v18864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:34.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:33 smithi153 ceph-mon[36569]: pgmap v18865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:34.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:33 smithi104 ceph-mon[36624]: pgmap v18865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:34.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:33 smithi081 ceph-mon[31453]: pgmap v18865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:34.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:34.773+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:05:34.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:34.773+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:05:34.787 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:34.773+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:05:36.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:35 smithi153 ceph-mon[36569]: pgmap v18866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:36.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:35 smithi104 ceph-mon[36624]: pgmap v18866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:36.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:35 smithi081 ceph-mon[31453]: pgmap v18866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:38.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:37 smithi153 ceph-mon[36569]: pgmap v18867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:38.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:37 smithi104 ceph-mon[36624]: pgmap v18867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:38.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:37 smithi081 ceph-mon[31453]: pgmap v18867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:40.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:39 smithi153 ceph-mon[36569]: pgmap v18868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:40.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:39 smithi104 ceph-mon[36624]: pgmap v18868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:40.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:39 smithi081 ceph-mon[31453]: pgmap v18868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:42.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:41 smithi153 ceph-mon[36569]: pgmap v18869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:42.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:41 smithi104 ceph-mon[36624]: pgmap v18869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:42.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:41 smithi081 ceph-mon[31453]: pgmap v18869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:44.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:43 smithi153 ceph-mon[36569]: pgmap v18870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:44.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:43 smithi104 ceph-mon[36624]: pgmap v18870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:44.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:43 smithi081 ceph-mon[31453]: pgmap v18870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:46.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:45 smithi153 ceph-mon[36569]: pgmap v18871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:46.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:45 smithi104 ceph-mon[36624]: pgmap v18871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:46.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:45 smithi081 ceph-mon[31453]: pgmap v18871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:48.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:47 smithi153 ceph-mon[36569]: pgmap v18872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:48.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:47 smithi104 ceph-mon[36624]: pgmap v18872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:48.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:47 smithi081 ceph-mon[31453]: pgmap v18872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:49.789+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:05:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:49.789+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:05:49.802 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:05:49.789+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:05:50.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:49 smithi153 ceph-mon[36569]: pgmap v18873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:50.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:49 smithi104 ceph-mon[36624]: pgmap v18873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:50.170 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:49 smithi081 ceph-mon[31453]: pgmap v18873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:52.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:51 smithi153 ceph-mon[36569]: pgmap v18874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:52.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:51 smithi104 ceph-mon[36624]: pgmap v18874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:52.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:51 smithi081 ceph-mon[31453]: pgmap v18874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:54.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:53 smithi153 ceph-mon[36569]: pgmap v18875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:54.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:53 smithi104 ceph-mon[36624]: pgmap v18875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:54.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:53 smithi081 ceph-mon[31453]: pgmap v18875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:56.069 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:55 smithi153 ceph-mon[36569]: pgmap v18876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:56.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:55 smithi104 ceph-mon[36624]: pgmap v18876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:56.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:55 smithi081 ceph-mon[31453]: pgmap v18876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:58.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:57 smithi153 ceph-mon[36569]: pgmap v18877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:58.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:57 smithi104 ceph-mon[36624]: pgmap v18877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:05:58.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:57 smithi081 ceph-mon[31453]: pgmap v18877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:00.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:05:59 smithi153 ceph-mon[36569]: pgmap v18878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:00.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:05:59 smithi104 ceph-mon[36624]: pgmap v18878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:00.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:05:59 smithi081 ceph-mon[31453]: pgmap v18878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:02.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:01 smithi104 ceph-mon[36624]: pgmap v18879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:02.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:01 smithi081 ceph-mon[31453]: pgmap v18879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:02.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:01 smithi153 ceph-mon[36569]: pgmap v18879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:04.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:03 smithi104 ceph-mon[36624]: pgmap v18880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:04.120 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:03 smithi153 ceph-mon[36569]: pgmap v18880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:04.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:03 smithi081 ceph-mon[31453]: pgmap v18880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:04.792+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:06:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:04.792+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:06:04.804 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:04.792+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:06:06.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:05 smithi104 ceph-mon[36624]: pgmap v18881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:06.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:05 smithi081 ceph-mon[31453]: pgmap v18881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:06.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:05 smithi153 ceph-mon[36569]: pgmap v18881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:08.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:07 smithi104 ceph-mon[36624]: pgmap v18882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:08.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:07 smithi081 ceph-mon[31453]: pgmap v18882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:08.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:07 smithi153 ceph-mon[36569]: pgmap v18882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:10.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:09 smithi104 ceph-mon[36624]: pgmap v18883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:10.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:09 smithi081 ceph-mon[31453]: pgmap v18883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:10.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:09 smithi153 ceph-mon[36569]: pgmap v18883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:12.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:11 smithi081 ceph-mon[31453]: pgmap v18884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:12.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:11 smithi153 ceph-mon[36569]: pgmap v18884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:12.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:11 smithi104 ceph-mon[36624]: pgmap v18884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:13.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:06:13.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:06:13.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:06:14.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:13 smithi081 ceph-mon[31453]: pgmap v18885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:14.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:06:14.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:06:14.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:06:14.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:13 smithi153 ceph-mon[36569]: pgmap v18885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:14.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:06:14.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:06:14.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:06:14.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:13 smithi104 ceph-mon[36624]: pgmap v18885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:14.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:06:14.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:06:14.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:06:15.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:14 smithi081 ceph-mon[31453]: pgmap v18886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:15.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:14 smithi081 ceph-mon[31453]: pgmap v18887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:15.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:14 smithi153 ceph-mon[36569]: pgmap v18886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:15.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:14 smithi153 ceph-mon[36569]: pgmap v18887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:15.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:14 smithi104 ceph-mon[36624]: pgmap v18886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:15.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:14 smithi104 ceph-mon[36624]: pgmap v18887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:17.168 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:16 smithi081 ceph-mon[31453]: pgmap v18888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:17.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:16 smithi153 ceph-mon[36569]: pgmap v18888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:17.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:16 smithi104 ceph-mon[36624]: pgmap v18888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:19.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:18 smithi081 ceph-mon[31453]: pgmap v18889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:19.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:18 smithi153 ceph-mon[36569]: pgmap v18889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:19.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:18 smithi104 ceph-mon[36624]: pgmap v18889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:19.807+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:06:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:19.807+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:06:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:19.807+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:06:20.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:19 smithi081 ceph-mon[31453]: pgmap v18890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:20.318 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:19 smithi153 ceph-mon[36569]: pgmap v18890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:20.343 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:19 smithi104 ceph-mon[36624]: pgmap v18890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:22.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:22 smithi081 ceph-mon[31453]: pgmap v18891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:23.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:22 smithi153 ceph-mon[36569]: pgmap v18891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:23.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:22 smithi104 ceph-mon[36624]: pgmap v18891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:24.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:24 smithi081 ceph-mon[31453]: pgmap v18892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:25.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:24 smithi153 ceph-mon[36569]: pgmap v18892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:25.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:24 smithi104 ceph-mon[36624]: pgmap v18892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:26.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:26 smithi081 ceph-mon[31453]: pgmap v18893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:27.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:26 smithi153 ceph-mon[36569]: pgmap v18893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:27.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:26 smithi104 ceph-mon[36624]: pgmap v18893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:28.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:28 smithi081 ceph-mon[31453]: pgmap v18894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:29.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:28 smithi153 ceph-mon[36569]: pgmap v18894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:29.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:28 smithi104 ceph-mon[36624]: pgmap v18894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:30.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:30 smithi081 ceph-mon[31453]: pgmap v18895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:31.068 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:30 smithi153 ceph-mon[36569]: pgmap v18895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:31.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:30 smithi104 ceph-mon[36624]: pgmap v18895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:32.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:32 smithi081 ceph-mon[31453]: pgmap v18896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:33.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:32 smithi153 ceph-mon[36569]: pgmap v18896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:33.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:32 smithi104 ceph-mon[36624]: pgmap v18896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:34.834 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:34.823+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:06:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:34.823+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:06:34.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:34.823+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:06:34.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:34 smithi081 ceph-mon[31453]: pgmap v18897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:35.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:34 smithi153 ceph-mon[36569]: pgmap v18897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:35.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:34 smithi104 ceph-mon[36624]: pgmap v18897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:36.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:36 smithi081 ceph-mon[31453]: pgmap v18898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:37.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:36 smithi153 ceph-mon[36569]: pgmap v18898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:37.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:36 smithi104 ceph-mon[36624]: pgmap v18898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:38.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:38 smithi081 ceph-mon[31453]: pgmap v18899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:39.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:38 smithi153 ceph-mon[36569]: pgmap v18899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:39.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:38 smithi104 ceph-mon[36624]: pgmap v18899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:40.917 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:40 smithi081 ceph-mon[31453]: pgmap v18900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:41.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:40 smithi153 ceph-mon[36569]: pgmap v18900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:41.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:40 smithi104 ceph-mon[36624]: pgmap v18900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:43.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:42 smithi153 ceph-mon[36569]: pgmap v18901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:43.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:42 smithi104 ceph-mon[36624]: pgmap v18901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:43.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:42 smithi081 ceph-mon[31453]: pgmap v18901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:45.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:44 smithi153 ceph-mon[36569]: pgmap v18902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:45.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:44 smithi104 ceph-mon[36624]: pgmap v18902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:45.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:44 smithi081 ceph-mon[31453]: pgmap v18902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:47.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:46 smithi153 ceph-mon[36569]: pgmap v18903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:47.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:46 smithi104 ceph-mon[36624]: pgmap v18903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:47.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:46 smithi081 ceph-mon[31453]: pgmap v18903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:49.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:48 smithi153 ceph-mon[36569]: pgmap v18904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:49.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:48 smithi104 ceph-mon[36624]: pgmap v18904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:49.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:48 smithi081 ceph-mon[31453]: pgmap v18904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:49.831+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:06:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:49.831+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:06:49.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:06:49.831+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:06:51.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:50 smithi153 ceph-mon[36569]: pgmap v18905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:51.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:50 smithi104 ceph-mon[36624]: pgmap v18905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:51.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:50 smithi081 ceph-mon[31453]: pgmap v18905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:53.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:52 smithi153 ceph-mon[36569]: pgmap v18906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:53.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:52 smithi104 ceph-mon[36624]: pgmap v18906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:53.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:52 smithi081 ceph-mon[31453]: pgmap v18906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:55.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:54 smithi153 ceph-mon[36569]: pgmap v18907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:55.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:54 smithi104 ceph-mon[36624]: pgmap v18907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:55.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:54 smithi081 ceph-mon[31453]: pgmap v18907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:57.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:56 smithi153 ceph-mon[36569]: pgmap v18908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:57.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:56 smithi104 ceph-mon[36624]: pgmap v18908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:57.167 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:56 smithi081 ceph-mon[31453]: pgmap v18908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:59.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:06:58 smithi153 ceph-mon[36569]: pgmap v18909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:59.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:06:58 smithi104 ceph-mon[36624]: pgmap v18909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:06:59.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:06:58 smithi081 ceph-mon[31453]: pgmap v18909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:01.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:00 smithi153 ceph-mon[36569]: pgmap v18910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:01.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:00 smithi104 ceph-mon[36624]: pgmap v18910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:01.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:00 smithi081 ceph-mon[31453]: pgmap v18910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:03.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:02 smithi153 ceph-mon[36569]: pgmap v18911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:03.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:02 smithi104 ceph-mon[36624]: pgmap v18911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:03.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:02 smithi081 ceph-mon[31453]: pgmap v18911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:04.835+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:07:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:04.835+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:07:04.846 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:04.835+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:07:05.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:04 smithi153 ceph-mon[36569]: pgmap v18912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:05.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:04 smithi104 ceph-mon[36624]: pgmap v18912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:05.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:04 smithi081 ceph-mon[31453]: pgmap v18912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:07.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:06 smithi153 ceph-mon[36569]: pgmap v18913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:07.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:06 smithi104 ceph-mon[36624]: pgmap v18913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:07.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:06 smithi081 ceph-mon[31453]: pgmap v18913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:09.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:08 smithi153 ceph-mon[36569]: pgmap v18914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:09.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:08 smithi104 ceph-mon[36624]: pgmap v18914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:09.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:08 smithi081 ceph-mon[31453]: pgmap v18914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:11.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:10 smithi153 ceph-mon[36569]: pgmap v18915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:11.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:10 smithi104 ceph-mon[36624]: pgmap v18915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:11.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:10 smithi081 ceph-mon[31453]: pgmap v18915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:13.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:12 smithi153 ceph-mon[36569]: pgmap v18916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:13.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:12 smithi104 ceph-mon[36624]: pgmap v18916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:13.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:12 smithi081 ceph-mon[31453]: pgmap v18916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:15.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:14 smithi153 ceph-mon[36569]: pgmap v18917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:15.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:07:15.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:14 smithi104 ceph-mon[36624]: pgmap v18917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:15.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:07:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:14 smithi081 ceph-mon[31453]: pgmap v18917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:15.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:07:16.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:07:16.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:07:16.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:07:16.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:07:16.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:07:16.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:07:16.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:07:16.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:07:16.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:07:17.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:16 smithi153 ceph-mon[36569]: pgmap v18918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:17.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:16 smithi153 ceph-mon[36569]: pgmap v18919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:17.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:16 smithi104 ceph-mon[36624]: pgmap v18918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:17.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:16 smithi104 ceph-mon[36624]: pgmap v18919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:17.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:16 smithi081 ceph-mon[31453]: pgmap v18918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:17.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:16 smithi081 ceph-mon[31453]: pgmap v18919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:18.582 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:18.572+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:07:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:18.572+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:07:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:18.572+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 273 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:07:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:18.572+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 273 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:07:19.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:18 smithi153 ceph-mon[36569]: pgmap v18920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:19.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:18 smithi104 ceph-mon[36624]: pgmap v18920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:19.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:18 smithi081 ceph-mon[31453]: pgmap v18920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:19.847+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:07:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:19.847+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:07:19.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:19.847+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:07:21.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:20 smithi153 ceph-mon[36569]: pgmap v18921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:21.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:20 smithi104 ceph-mon[36624]: pgmap v18921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:21.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:20 smithi081 ceph-mon[31453]: pgmap v18921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:23.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:22 smithi153 ceph-mon[36569]: pgmap v18922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:23.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:22 smithi104 ceph-mon[36624]: pgmap v18922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:23.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:22 smithi081 ceph-mon[31453]: pgmap v18922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:25.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:24 smithi153 ceph-mon[36569]: pgmap v18923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:25.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:24 smithi104 ceph-mon[36624]: pgmap v18923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:25.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:24 smithi081 ceph-mon[31453]: pgmap v18923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:27.067 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:26 smithi153 ceph-mon[36569]: pgmap v18924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:27.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:26 smithi104 ceph-mon[36624]: pgmap v18924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:27.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:26 smithi081 ceph-mon[31453]: pgmap v18924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:29.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:28 smithi153 ceph-mon[36569]: pgmap v18925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:29.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:28 smithi104 ceph-mon[36624]: pgmap v18925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:29.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:28 smithi081 ceph-mon[31453]: pgmap v18925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:31.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:30 smithi153 ceph-mon[36569]: pgmap v18926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:31.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:30 smithi104 ceph-mon[36624]: pgmap v18926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:31.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:30 smithi081 ceph-mon[31453]: pgmap v18926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:33.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:32 smithi104 ceph-mon[36624]: pgmap v18927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:33.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:32 smithi081 ceph-mon[31453]: pgmap v18927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:33.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:32 smithi153 ceph-mon[36569]: pgmap v18927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:34.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:34.849+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:07:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:34.849+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:07:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:34.849+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:07:35.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:34 smithi104 ceph-mon[36624]: pgmap v18928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:35.118 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:34 smithi153 ceph-mon[36569]: pgmap v18928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:35.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:34 smithi081 ceph-mon[31453]: pgmap v18928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:37.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:36 smithi104 ceph-mon[36624]: pgmap v18929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:37.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:36 smithi081 ceph-mon[31453]: pgmap v18929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:37.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:36 smithi153 ceph-mon[36569]: pgmap v18929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:39.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:38 smithi104 ceph-mon[36624]: pgmap v18930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:39.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:38 smithi081 ceph-mon[31453]: pgmap v18930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:39.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:38 smithi153 ceph-mon[36569]: pgmap v18930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:41.092 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:40 smithi104 ceph-mon[36624]: pgmap v18931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:41.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:40 smithi081 ceph-mon[31453]: pgmap v18931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:41.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:40 smithi153 ceph-mon[36569]: pgmap v18931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:43.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:42 smithi081 ceph-mon[31453]: pgmap v18932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:43.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:42 smithi153 ceph-mon[36569]: pgmap v18932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:43.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:42 smithi104 ceph-mon[36624]: pgmap v18932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:45.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:44 smithi081 ceph-mon[31453]: pgmap v18933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:45.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:44 smithi153 ceph-mon[36569]: pgmap v18933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:45.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:44 smithi104 ceph-mon[36624]: pgmap v18933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:47.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:46 smithi081 ceph-mon[31453]: pgmap v18934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:47.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:46 smithi153 ceph-mon[36569]: pgmap v18934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:47.342 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:46 smithi104 ceph-mon[36624]: pgmap v18934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:49.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:48 smithi081 ceph-mon[31453]: pgmap v18935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:49.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:48 smithi153 ceph-mon[36569]: pgmap v18935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:49.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:48 smithi104 ceph-mon[36624]: pgmap v18935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:49.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:49.862+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:07:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:49.863+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:07:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:07:49.863+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:07:51.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:50 smithi081 ceph-mon[31453]: pgmap v18936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:51.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:50 smithi153 ceph-mon[36569]: pgmap v18936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:51.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:50 smithi104 ceph-mon[36624]: pgmap v18936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:53.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:52 smithi081 ceph-mon[31453]: pgmap v18937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:53.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:52 smithi153 ceph-mon[36569]: pgmap v18937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:53.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:52 smithi104 ceph-mon[36624]: pgmap v18937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:55.166 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:54 smithi081 ceph-mon[31453]: pgmap v18938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:55.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:54 smithi153 ceph-mon[36569]: pgmap v18938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:55.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:54 smithi104 ceph-mon[36624]: pgmap v18938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:57.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:56 smithi081 ceph-mon[31453]: pgmap v18939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:57.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:56 smithi153 ceph-mon[36569]: pgmap v18939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:57.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:56 smithi104 ceph-mon[36624]: pgmap v18939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:59.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:07:58 smithi081 ceph-mon[31453]: pgmap v18940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:59.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:07:58 smithi153 ceph-mon[36569]: pgmap v18940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:07:59.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:07:58 smithi104 ceph-mon[36624]: pgmap v18940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:01.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:00 smithi081 ceph-mon[31453]: pgmap v18941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:01.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:00 smithi153 ceph-mon[36569]: pgmap v18941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:01.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:00 smithi104 ceph-mon[36624]: pgmap v18941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:03.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:02 smithi081 ceph-mon[31453]: pgmap v18942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:03.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:02 smithi153 ceph-mon[36569]: pgmap v18942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:03.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:02 smithi104 ceph-mon[36624]: pgmap v18942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:04.864+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:08:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:04.864+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:08:04.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:04.864+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:08:05.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:04 smithi081 ceph-mon[31453]: pgmap v18943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:05.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:04 smithi153 ceph-mon[36569]: pgmap v18943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:05.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:04 smithi104 ceph-mon[36624]: pgmap v18943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:06.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:05 smithi081 ceph-mon[31453]: pgmap v18944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:06.316 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:05 smithi153 ceph-mon[36569]: pgmap v18944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:06.341 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:05 smithi104 ceph-mon[36624]: pgmap v18944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:08.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:08 smithi081 ceph-mon[31453]: pgmap v18945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:08.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:08 smithi153 ceph-mon[36569]: pgmap v18945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:08.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:08 smithi104 ceph-mon[36624]: pgmap v18945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:10.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:10 smithi081 ceph-mon[31453]: pgmap v18946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:10.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:10 smithi153 ceph-mon[36569]: pgmap v18946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:10.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:10 smithi104 ceph-mon[36624]: pgmap v18946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:12.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:12 smithi081 ceph-mon[31453]: pgmap v18947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:12.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:12 smithi153 ceph-mon[36569]: pgmap v18947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:12.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:12 smithi104 ceph-mon[36624]: pgmap v18947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:14.415 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:14 smithi081 ceph-mon[31453]: pgmap v18948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:14.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:14 smithi153 ceph-mon[36569]: pgmap v18948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:14.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:14 smithi104 ceph-mon[36624]: pgmap v18948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:16.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:16 smithi153 ceph-mon[36569]: pgmap v18949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:16.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:08:16.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:16 smithi104 ceph-mon[36624]: pgmap v18949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:16.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:08:16.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:16 smithi081 ceph-mon[31453]: pgmap v18949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:16.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:08:17.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:08:17.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:08:17.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:17 smithi153 ceph-mon[36569]: pgmap v18950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:17.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:17 smithi153 ceph-mon[36569]: pgmap v18951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:17.566 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:08:17.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:08:17.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:08:17.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:17 smithi104 ceph-mon[36624]: pgmap v18950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:17.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:17 smithi104 ceph-mon[36624]: pgmap v18951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:17.592 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:08:17.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:08:17.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:08:17.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:17 smithi081 ceph-mon[31453]: pgmap v18950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:17.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:17 smithi081 ceph-mon[31453]: pgmap v18951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:17.666 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:08:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:19.878+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:08:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:19.878+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:08:19.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:19.878+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:08:20.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:19 smithi153 ceph-mon[36569]: pgmap v18952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:20.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:19 smithi104 ceph-mon[36624]: pgmap v18952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:20.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:19 smithi081 ceph-mon[31453]: pgmap v18952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:22.034 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:21 smithi081 ceph-mon[31453]: pgmap v18953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:22.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:21 smithi153 ceph-mon[36569]: pgmap v18953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:22.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:21 smithi104 ceph-mon[36624]: pgmap v18953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:24.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:23 smithi153 ceph-mon[36569]: pgmap v18954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:24.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:23 smithi104 ceph-mon[36624]: pgmap v18954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:24.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:23 smithi081 ceph-mon[31453]: pgmap v18954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:26.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:25 smithi153 ceph-mon[36569]: pgmap v18955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:26.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:25 smithi104 ceph-mon[36624]: pgmap v18955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:26.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:25 smithi081 ceph-mon[31453]: pgmap v18955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:28.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:27 smithi153 ceph-mon[36569]: pgmap v18956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:28.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:27 smithi104 ceph-mon[36624]: pgmap v18956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:28.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:27 smithi081 ceph-mon[31453]: pgmap v18956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:30.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:29 smithi153 ceph-mon[36569]: pgmap v18957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:30.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:29 smithi104 ceph-mon[36624]: pgmap v18957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:30.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:29 smithi081 ceph-mon[31453]: pgmap v18957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:31.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:31 smithi153 ceph-mon[36569]: pgmap v18958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:31.841 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:31 smithi104 ceph-mon[36624]: pgmap v18958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:31.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:31 smithi081 ceph-mon[31453]: pgmap v18958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:34.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:33 smithi153 ceph-mon[36569]: pgmap v18959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:34.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:33 smithi104 ceph-mon[36624]: pgmap v18959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:34.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:33 smithi081 ceph-mon[31453]: pgmap v18959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:34.887+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:08:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:34.887+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:08:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:34.887+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:08:36.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:35 smithi153 ceph-mon[36569]: pgmap v18960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:36.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:35 smithi104 ceph-mon[36624]: pgmap v18960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:36.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:35 smithi081 ceph-mon[31453]: pgmap v18960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:38.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:37 smithi153 ceph-mon[36569]: pgmap v18961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:38.070 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:37 smithi081 ceph-mon[31453]: pgmap v18961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:38.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:37 smithi104 ceph-mon[36624]: pgmap v18961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:40.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:39 smithi153 ceph-mon[36569]: pgmap v18962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:40.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:39 smithi104 ceph-mon[36624]: pgmap v18962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:40.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:39 smithi081 ceph-mon[31453]: pgmap v18962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:42.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:41 smithi153 ceph-mon[36569]: pgmap v18963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:42.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:41 smithi104 ceph-mon[36624]: pgmap v18963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:42.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:41 smithi081 ceph-mon[31453]: pgmap v18963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:44.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:43 smithi153 ceph-mon[36569]: pgmap v18964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:44.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:43 smithi104 ceph-mon[36624]: pgmap v18964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:44.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:43 smithi081 ceph-mon[31453]: pgmap v18964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:46.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:45 smithi153 ceph-mon[36569]: pgmap v18965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:46.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:45 smithi104 ceph-mon[36624]: pgmap v18965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:46.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:45 smithi081 ceph-mon[31453]: pgmap v18965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:48.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:47 smithi153 ceph-mon[36569]: pgmap v18966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:48.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:47 smithi104 ceph-mon[36624]: pgmap v18966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:48.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:47 smithi081 ceph-mon[31453]: pgmap v18966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:49.902+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:08:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:49.902+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:08:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:08:49.902+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:08:50.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:49 smithi153 ceph-mon[36569]: pgmap v18967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:50.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:49 smithi104 ceph-mon[36624]: pgmap v18967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:50.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:49 smithi081 ceph-mon[31453]: pgmap v18967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:52.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:51 smithi153 ceph-mon[36569]: pgmap v18968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:52.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:51 smithi104 ceph-mon[36624]: pgmap v18968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:52.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:51 smithi081 ceph-mon[31453]: pgmap v18968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:54.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:53 smithi153 ceph-mon[36569]: pgmap v18969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:54.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:53 smithi104 ceph-mon[36624]: pgmap v18969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:54.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:53 smithi081 ceph-mon[31453]: pgmap v18969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:56.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:55 smithi153 ceph-mon[36569]: pgmap v18970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:56.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:55 smithi104 ceph-mon[36624]: pgmap v18970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:56.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:55 smithi081 ceph-mon[31453]: pgmap v18970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:58.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:57 smithi153 ceph-mon[36569]: pgmap v18971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:58.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:57 smithi104 ceph-mon[36624]: pgmap v18971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:08:58.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:57 smithi081 ceph-mon[31453]: pgmap v18971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:00.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:08:59 smithi153 ceph-mon[36569]: pgmap v18972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:00.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:08:59 smithi104 ceph-mon[36624]: pgmap v18972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:00.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:08:59 smithi081 ceph-mon[31453]: pgmap v18972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:02.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:01 smithi153 ceph-mon[36569]: pgmap v18973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:02.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:01 smithi104 ceph-mon[36624]: pgmap v18973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:02.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:01 smithi081 ceph-mon[31453]: pgmap v18973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:04.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:03 smithi153 ceph-mon[36569]: pgmap v18974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:04.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:03 smithi104 ceph-mon[36624]: pgmap v18974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:04.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:03 smithi081 ceph-mon[31453]: pgmap v18974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:04.915+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:09:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:04.915+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:09:04.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:04.915+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:09:06.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:05 smithi153 ceph-mon[36569]: pgmap v18975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:06.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:05 smithi104 ceph-mon[36624]: pgmap v18975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:06.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:05 smithi081 ceph-mon[31453]: pgmap v18975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:08.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:07 smithi153 ceph-mon[36569]: pgmap v18976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:08.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:07 smithi104 ceph-mon[36624]: pgmap v18976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:08.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:07 smithi081 ceph-mon[31453]: pgmap v18976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:10.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:09 smithi153 ceph-mon[36569]: pgmap v18977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:10.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:09 smithi104 ceph-mon[36624]: pgmap v18977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:10.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:09 smithi081 ceph-mon[31453]: pgmap v18977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:12.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:11 smithi153 ceph-mon[36569]: pgmap v18978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:12.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:11 smithi104 ceph-mon[36624]: pgmap v18978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:12.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:11 smithi081 ceph-mon[31453]: pgmap v18978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:14.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:13 smithi153 ceph-mon[36569]: pgmap v18979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:14.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:13 smithi104 ceph-mon[36624]: pgmap v18979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:14.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:13 smithi081 ceph-mon[31453]: pgmap v18979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:16.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:15 smithi153 ceph-mon[36569]: pgmap v18980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:16.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:15 smithi104 ceph-mon[36624]: pgmap v18980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:16.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:15 smithi081 ceph-mon[31453]: pgmap v18980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:18.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:17 smithi153 ceph-mon[36569]: pgmap v18981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:09:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.065 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:17 smithi104 ceph-mon[36624]: pgmap v18981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:18.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:09:18.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:17 smithi081 ceph-mon[31453]: pgmap v18981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:18.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:09:18.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:18.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:19.931+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:09:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:19.931+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:09:19.939 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:19.931+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:09:20.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:19 smithi104 ceph-mon[36624]: pgmap v18982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:20.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:20.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:20.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:19 smithi081 ceph-mon[31453]: pgmap v18982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:20.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:20.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:20.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:19 smithi153 ceph-mon[36569]: pgmap v18982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:20.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:20.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:21.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:21.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:21.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:21.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:21.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:21.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:22.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:21 smithi081 ceph-mon[31453]: pgmap v18983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:22.315 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:21 smithi153 ceph-mon[36569]: pgmap v18983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:22.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:21 smithi104 ceph-mon[36624]: pgmap v18983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:23.564 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:23.564 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:23.565 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:23 smithi153 ceph-mon[36569]: pgmap v18984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:23.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:23.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:23.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:23 smithi104 ceph-mon[36624]: pgmap v18984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:23.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:23.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:23.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:23 smithi081 ceph-mon[31453]: pgmap v18984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:25.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:25.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:25.590 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:09:25.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:09:25.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:25 smithi104 ceph-mon[36624]: pgmap v18985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:25.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:25 smithi104 ceph-mon[36624]: pgmap v18986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:25.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:25.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:25.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:25.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:09:25.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:09:25.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:25 smithi081 ceph-mon[31453]: pgmap v18985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:25.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:25 smithi081 ceph-mon[31453]: pgmap v18986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:25.665 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:25.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:25.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:25.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:09:25.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:09:25.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:25 smithi153 ceph-mon[36569]: pgmap v18985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:25.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:25 smithi153 ceph-mon[36569]: pgmap v18986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:25.815 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:09:27.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:27 smithi081 ceph-mon[31453]: pgmap v18987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:27.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:27 smithi153 ceph-mon[36569]: pgmap v18987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:27.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:27 smithi104 ceph-mon[36624]: pgmap v18987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:29.667 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:29 smithi081 ceph-mon[31453]: pgmap v18988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:29.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:29 smithi153 ceph-mon[36569]: pgmap v18988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:29.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:29 smithi104 ceph-mon[36624]: pgmap v18988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:31.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:31 smithi153 ceph-mon[36569]: pgmap v18989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:31.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:31 smithi104 ceph-mon[36624]: pgmap v18989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:31.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:31 smithi081 ceph-mon[31453]: pgmap v18989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:33.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:33 smithi153 ceph-mon[36569]: pgmap v18990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:33.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:33 smithi104 ceph-mon[36624]: pgmap v18990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:33.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:33 smithi081 ceph-mon[31453]: pgmap v18990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:34.944+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:09:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:34.944+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:09:34.952 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:34.944+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:09:35.814 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:35 smithi153 ceph-mon[36569]: pgmap v18991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:35.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:35 smithi104 ceph-mon[36624]: pgmap v18991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:35.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:35 smithi081 ceph-mon[31453]: pgmap v18991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:37.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:37 smithi104 ceph-mon[36624]: pgmap v18992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:37.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:37 smithi081 ceph-mon[31453]: pgmap v18992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:38.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:37 smithi153 ceph-mon[36569]: pgmap v18992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:39.664 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:39 smithi081 ceph-mon[31453]: pgmap v18993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:39.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:39 smithi104 ceph-mon[36624]: pgmap v18993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:40.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:39 smithi153 ceph-mon[36569]: pgmap v18993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:41.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:41 smithi104 ceph-mon[36624]: pgmap v18994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:41.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:41 smithi081 ceph-mon[31453]: pgmap v18994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:42.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:41 smithi153 ceph-mon[36569]: pgmap v18994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:43.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:43 smithi104 ceph-mon[36624]: pgmap v18995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:43.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:43 smithi081 ceph-mon[31453]: pgmap v18995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:44.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:43 smithi153 ceph-mon[36569]: pgmap v18995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:45.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:45 smithi104 ceph-mon[36624]: pgmap v18996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:45.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:45 smithi081 ceph-mon[31453]: pgmap v18996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:46.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:45 smithi153 ceph-mon[36569]: pgmap v18996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:47.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:47 smithi081 ceph-mon[31453]: pgmap v18997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:48.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:47 smithi153 ceph-mon[36569]: pgmap v18997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:48.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:47 smithi104 ceph-mon[36624]: pgmap v18997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:49.914 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:49 smithi081 ceph-mon[31453]: pgmap v18998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:49.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:49.952+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:09:49.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:49.952+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:09:49.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:09:49.952+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:09:50.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:49 smithi153 ceph-mon[36569]: pgmap v18998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:50.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:49 smithi104 ceph-mon[36624]: pgmap v18998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:51.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:51 smithi081 ceph-mon[31453]: pgmap v18999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:52.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:51 smithi153 ceph-mon[36569]: pgmap v18999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:52.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:51 smithi104 ceph-mon[36624]: pgmap v18999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:53.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:53 smithi081 ceph-mon[31453]: pgmap v19000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:54.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:53 smithi153 ceph-mon[36569]: pgmap v19000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:54.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:53 smithi104 ceph-mon[36624]: pgmap v19000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:55.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:55 smithi081 ceph-mon[31453]: pgmap v19001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:56.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:55 smithi153 ceph-mon[36569]: pgmap v19001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:56.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:55 smithi104 ceph-mon[36624]: pgmap v19001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:57.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:57 smithi081 ceph-mon[31453]: pgmap v19002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:58.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:57 smithi153 ceph-mon[36569]: pgmap v19002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:58.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:57 smithi104 ceph-mon[36624]: pgmap v19002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:09:59.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:09:59 smithi081 ceph-mon[31453]: pgmap v19003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:00.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:09:59 smithi153 ceph-mon[36569]: pgmap v19003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:00.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:09:59 smithi104 ceph-mon[36624]: pgmap v19003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:00.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:10:01.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:10:01.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:10:01.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:01 smithi081 ceph-mon[31453]: pgmap v19004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:02.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:01 smithi153 ceph-mon[36569]: pgmap v19004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:02.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:01 smithi104 ceph-mon[36624]: pgmap v19004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:03.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:03 smithi081 ceph-mon[31453]: pgmap v19005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:04.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:03 smithi153 ceph-mon[36569]: pgmap v19005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:04.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:03 smithi104 ceph-mon[36624]: pgmap v19005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:04.963+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:10:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:04.963+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:10:04.971 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:04.963+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:10:05.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:05 smithi081 ceph-mon[31453]: pgmap v19006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:06.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:05 smithi153 ceph-mon[36569]: pgmap v19006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:06.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:05 smithi104 ceph-mon[36624]: pgmap v19006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:07.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:07 smithi081 ceph-mon[31453]: pgmap v19007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:08.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:07 smithi153 ceph-mon[36569]: pgmap v19007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:08.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:07 smithi104 ceph-mon[36624]: pgmap v19007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:09.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:09 smithi081 ceph-mon[31453]: pgmap v19008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:10.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:09 smithi153 ceph-mon[36569]: pgmap v19008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:10.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:09 smithi104 ceph-mon[36624]: pgmap v19008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:11.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:11 smithi081 ceph-mon[31453]: pgmap v19009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:12.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:11 smithi153 ceph-mon[36569]: pgmap v19009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:12.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:11 smithi104 ceph-mon[36624]: pgmap v19009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:14.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:13 smithi153 ceph-mon[36569]: pgmap v19010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:14.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:13 smithi104 ceph-mon[36624]: pgmap v19010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:14.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:13 smithi081 ceph-mon[31453]: pgmap v19010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:16.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:15 smithi153 ceph-mon[36569]: pgmap v19011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:16.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:15 smithi104 ceph-mon[36624]: pgmap v19011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:16.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:15 smithi081 ceph-mon[31453]: pgmap v19011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:18.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:17 smithi153 ceph-mon[36569]: pgmap v19012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:18.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:17 smithi104 ceph-mon[36624]: pgmap v19012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:18.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:17 smithi081 ceph-mon[31453]: pgmap v19012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:18.603+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:10:18.610 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:18.603+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:10:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:18.604+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 274 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:10:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:18.604+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:10:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:18.604+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 274 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:10:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:18.604+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:10:18.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:18.604+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 275 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:10:18.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:18.604+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 275 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:19.970+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:19.970+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:10:19.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:19.970+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:10:20.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:19 smithi153 ceph-mon[36569]: pgmap v19013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:20.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:19 smithi104 ceph-mon[36624]: pgmap v19013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:20.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:19 smithi081 ceph-mon[31453]: pgmap v19013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:22.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:21 smithi153 ceph-mon[36569]: pgmap v19014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:22.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:21 smithi104 ceph-mon[36624]: pgmap v19014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:22.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:21 smithi081 ceph-mon[31453]: pgmap v19014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:24.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:23 smithi153 ceph-mon[36569]: pgmap v19015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:24.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:23 smithi104 ceph-mon[36624]: pgmap v19015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:24.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:23 smithi081 ceph-mon[31453]: pgmap v19015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:25.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:10:25.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:10:25.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:10:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:25 smithi153 ceph-mon[36569]: pgmap v19016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:26.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:25 smithi104 ceph-mon[36624]: pgmap v19016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:26.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:26.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:26.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:25 smithi081 ceph-mon[31453]: pgmap v19016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:26.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:26.165 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: pgmap v19017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: pgmap v19018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: pgmap v19019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.314 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: pgmap v19017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.339 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:10:27.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:10:27.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: pgmap v19018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: pgmap v19019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.340 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: pgmap v19017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.413 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:27.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:10:27.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:10:27.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: pgmap v19018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: pgmap v19019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:27.414 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:10:29.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:29 smithi104 ceph-mon[36624]: pgmap v19020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:29.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:29 smithi081 ceph-mon[31453]: pgmap v19020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:30.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:29 smithi153 ceph-mon[36569]: pgmap v19020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:31.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:31 smithi104 ceph-mon[36624]: pgmap v19021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:31.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:31 smithi081 ceph-mon[31453]: pgmap v19021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:32.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:31 smithi153 ceph-mon[36569]: pgmap v19021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:33.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:33 smithi081 ceph-mon[31453]: pgmap v19022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:34.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:33 smithi153 ceph-mon[36569]: pgmap v19022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:34.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:33 smithi104 ceph-mon[36624]: pgmap v19022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:34.982+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:10:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:34.982+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:10:34.989 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:34.982+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:10:35.866 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:35 smithi153 ceph-mon[36569]: pgmap v19023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:35.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:35 smithi081 ceph-mon[31453]: pgmap v19023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:36.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:35 smithi104 ceph-mon[36624]: pgmap v19023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:37.813 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:37 smithi153 ceph-mon[36569]: pgmap v19024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:37.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:37 smithi081 ceph-mon[31453]: pgmap v19024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:38.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:37 smithi104 ceph-mon[36624]: pgmap v19024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:39.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:39 smithi081 ceph-mon[31453]: pgmap v19025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:40.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:39 smithi153 ceph-mon[36569]: pgmap v19025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:40.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:39 smithi104 ceph-mon[36624]: pgmap v19025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:41.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:41 smithi081 ceph-mon[31453]: pgmap v19026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:42.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:41 smithi153 ceph-mon[36569]: pgmap v19026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:42.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:41 smithi104 ceph-mon[36624]: pgmap v19026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:43.891 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:43 smithi104 ceph-mon[36624]: pgmap v19027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:43.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:43 smithi081 ceph-mon[31453]: pgmap v19027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:44.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:43 smithi153 ceph-mon[36569]: pgmap v19027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:45.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:45 smithi081 ceph-mon[31453]: pgmap v19028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:46.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:45 smithi153 ceph-mon[36569]: pgmap v19028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:46.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:45 smithi104 ceph-mon[36624]: pgmap v19028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:47.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:47 smithi081 ceph-mon[31453]: pgmap v19029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:48.010 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:47 smithi153 ceph-mon[36569]: pgmap v19029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:48.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:47 smithi104 ceph-mon[36624]: pgmap v19029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:49.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:49 smithi081 ceph-mon[31453]: pgmap v19030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:50.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:49.993+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:10:50.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:49.993+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:10:50.001 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:10:49.993+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:10:50.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:49 smithi153 ceph-mon[36569]: pgmap v19030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:50.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:49 smithi104 ceph-mon[36624]: pgmap v19030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:51.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:51 smithi081 ceph-mon[31453]: pgmap v19031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:52.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:51 smithi153 ceph-mon[36569]: pgmap v19031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:52.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:51 smithi104 ceph-mon[36624]: pgmap v19031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:53.912 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:53 smithi081 ceph-mon[31453]: pgmap v19032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:54.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:53 smithi153 ceph-mon[36569]: pgmap v19032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:54.090 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:53 smithi104 ceph-mon[36624]: pgmap v19032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:55.913 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:55 smithi081 ceph-mon[31453]: pgmap v19033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:56.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:55 smithi153 ceph-mon[36569]: pgmap v19033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:56.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:55 smithi104 ceph-mon[36624]: pgmap v19033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:58.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:57 smithi153 ceph-mon[36569]: pgmap v19034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:58.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:57 smithi104 ceph-mon[36624]: pgmap v19034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:10:58.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:57 smithi081 ceph-mon[31453]: pgmap v19034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:00.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:10:59 smithi153 ceph-mon[36569]: pgmap v19035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:00.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:10:59 smithi104 ceph-mon[36624]: pgmap v19035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:00.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:10:59 smithi081 ceph-mon[31453]: pgmap v19035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:02.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:01 smithi153 ceph-mon[36569]: pgmap v19036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:02.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:01 smithi104 ceph-mon[36624]: pgmap v19036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:02.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:01 smithi081 ceph-mon[31453]: pgmap v19036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:04.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:03 smithi153 ceph-mon[36569]: pgmap v19037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:04.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:03 smithi104 ceph-mon[36624]: pgmap v19037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:04.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:03 smithi081 ceph-mon[31453]: pgmap v19037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:05.003+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:11:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:05.003+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:11:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:05.003+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:11:06.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:05 smithi153 ceph-mon[36569]: pgmap v19038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:06.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:05 smithi104 ceph-mon[36624]: pgmap v19038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:06.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:05 smithi081 ceph-mon[31453]: pgmap v19038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:08.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:07 smithi153 ceph-mon[36569]: pgmap v19039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:08.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:07 smithi104 ceph-mon[36624]: pgmap v19039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:08.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:07 smithi081 ceph-mon[31453]: pgmap v19039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:09.966 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:09 smithi081 ceph-mon[31453]: pgmap v19040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:10.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:09 smithi153 ceph-mon[36569]: pgmap v19040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:10.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:09 smithi104 ceph-mon[36624]: pgmap v19040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:12.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:11 smithi153 ceph-mon[36569]: pgmap v19041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:12.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:11 smithi104 ceph-mon[36624]: pgmap v19041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:12.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:11 smithi081 ceph-mon[31453]: pgmap v19041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:14.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:13 smithi153 ceph-mon[36569]: pgmap v19042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:14.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:13 smithi104 ceph-mon[36624]: pgmap v19042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:14.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:13 smithi081 ceph-mon[31453]: pgmap v19042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:16.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:15 smithi153 ceph-mon[36569]: pgmap v19043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:16.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:15 smithi104 ceph-mon[36624]: pgmap v19043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:16.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:15 smithi081 ceph-mon[31453]: pgmap v19043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:18.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:17 smithi153 ceph-mon[36569]: pgmap v19044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:18.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:17 smithi104 ceph-mon[36624]: pgmap v19044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:18.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:17 smithi081 ceph-mon[31453]: pgmap v19044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:20.018+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:11:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:20.018+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:11:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:20.018+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:11:20.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:19 smithi153 ceph-mon[36569]: pgmap v19045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:20.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:19 smithi104 ceph-mon[36624]: pgmap v19045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:20.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:19 smithi081 ceph-mon[31453]: pgmap v19045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:22.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:21 smithi153 ceph-mon[36569]: pgmap v19046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:22.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:21 smithi104 ceph-mon[36624]: pgmap v19046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:22.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:21 smithi081 ceph-mon[31453]: pgmap v19046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:24.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:23 smithi153 ceph-mon[36569]: pgmap v19047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:24.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:23 smithi104 ceph-mon[36624]: pgmap v19047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:24.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:23 smithi081 ceph-mon[31453]: pgmap v19047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:26.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:25 smithi153 ceph-mon[36569]: pgmap v19048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:26.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:25 smithi104 ceph-mon[36624]: pgmap v19048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:26.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:25 smithi081 ceph-mon[31453]: pgmap v19048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:28.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:27 smithi153 ceph-mon[36569]: pgmap v19049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:28.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:11:28.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:27 smithi104 ceph-mon[36624]: pgmap v19049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:28.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:11:28.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:27 smithi081 ceph-mon[31453]: pgmap v19049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:28.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:11:29.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:11:29.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:11:29.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:11:29.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:11:29.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:11:29.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:11:29.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:11:29.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:11:29.163 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:11:30.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:29 smithi153 ceph-mon[36569]: pgmap v19050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:30.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:29 smithi153 ceph-mon[36569]: pgmap v19051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:30.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:29 smithi104 ceph-mon[36624]: pgmap v19050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:30.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:29 smithi104 ceph-mon[36624]: pgmap v19051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:30.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:29 smithi081 ceph-mon[31453]: pgmap v19050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:30.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:29 smithi081 ceph-mon[31453]: pgmap v19051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:32.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:31 smithi153 ceph-mon[36569]: pgmap v19052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:32.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:31 smithi104 ceph-mon[36624]: pgmap v19052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:32.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:31 smithi081 ceph-mon[31453]: pgmap v19052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:34.066 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:33 smithi153 ceph-mon[36569]: pgmap v19053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:34.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:33 smithi104 ceph-mon[36624]: pgmap v19053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:34.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:33 smithi081 ceph-mon[31453]: pgmap v19053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:35.019+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:11:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:35.019+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:11:35.026 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:35.019+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:11:36.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:35 smithi153 ceph-mon[36569]: pgmap v19054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:36.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:35 smithi104 ceph-mon[36624]: pgmap v19054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:36.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:35 smithi081 ceph-mon[31453]: pgmap v19054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:38.063 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:37 smithi153 ceph-mon[36569]: pgmap v19055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:38.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:37 smithi104 ceph-mon[36624]: pgmap v19055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:38.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:37 smithi081 ceph-mon[31453]: pgmap v19055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:40.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:39 smithi153 ceph-mon[36569]: pgmap v19056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:40.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:39 smithi104 ceph-mon[36624]: pgmap v19056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:40.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:39 smithi081 ceph-mon[31453]: pgmap v19056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:42.012 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:41 smithi081 ceph-mon[31453]: pgmap v19057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:42.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:41 smithi153 ceph-mon[36569]: pgmap v19057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:42.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:41 smithi104 ceph-mon[36624]: pgmap v19057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:44.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:43 smithi153 ceph-mon[36569]: pgmap v19058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:44.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:43 smithi104 ceph-mon[36624]: pgmap v19058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:44.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:43 smithi081 ceph-mon[31453]: pgmap v19058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:46.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:45 smithi153 ceph-mon[36569]: pgmap v19059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:46.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:45 smithi104 ceph-mon[36624]: pgmap v19059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:46.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:45 smithi081 ceph-mon[31453]: pgmap v19059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:48.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:47 smithi153 ceph-mon[36569]: pgmap v19060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:48.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:47 smithi104 ceph-mon[36624]: pgmap v19060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:48.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:47 smithi081 ceph-mon[31453]: pgmap v19060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:50.035+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:11:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:50.036+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:11:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:11:50.036+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:11:50.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:49 smithi153 ceph-mon[36569]: pgmap v19061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:50.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:49 smithi104 ceph-mon[36624]: pgmap v19061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:50.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:49 smithi081 ceph-mon[31453]: pgmap v19061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:52.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:51 smithi104 ceph-mon[36624]: pgmap v19062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:52.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:51 smithi081 ceph-mon[31453]: pgmap v19062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:52.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:51 smithi153 ceph-mon[36569]: pgmap v19062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:54.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:53 smithi104 ceph-mon[36624]: pgmap v19063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:54.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:53 smithi081 ceph-mon[31453]: pgmap v19063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:54.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:53 smithi153 ceph-mon[36569]: pgmap v19063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:56.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:55 smithi104 ceph-mon[36624]: pgmap v19064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:56.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:55 smithi081 ceph-mon[31453]: pgmap v19064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:56.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:55 smithi153 ceph-mon[36569]: pgmap v19064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:58.088 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:57 smithi104 ceph-mon[36624]: pgmap v19065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:58.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:57 smithi081 ceph-mon[31453]: pgmap v19065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:11:58.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:57 smithi153 ceph-mon[36569]: pgmap v19065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:00.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:11:59 smithi081 ceph-mon[31453]: pgmap v19066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:00.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:11:59 smithi153 ceph-mon[36569]: pgmap v19066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:00.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:11:59 smithi104 ceph-mon[36624]: pgmap v19066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:02.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:01 smithi081 ceph-mon[31453]: pgmap v19067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:02.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:01 smithi153 ceph-mon[36569]: pgmap v19067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:02.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:01 smithi104 ceph-mon[36624]: pgmap v19067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:04.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:03 smithi081 ceph-mon[31453]: pgmap v19068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:04.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:03 smithi153 ceph-mon[36569]: pgmap v19068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:04.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:03 smithi104 ceph-mon[36624]: pgmap v19068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:05.046+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:12:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:05.046+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:12:05.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:05.046+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:12:06.114 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:05 smithi153 ceph-mon[36569]: pgmap v19069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:06.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:05 smithi081 ceph-mon[31453]: pgmap v19069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:06.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:05 smithi104 ceph-mon[36624]: pgmap v19069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:08.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:07 smithi081 ceph-mon[31453]: pgmap v19070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:08.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:07 smithi153 ceph-mon[36569]: pgmap v19070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:08.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:07 smithi104 ceph-mon[36624]: pgmap v19070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:10.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:09 smithi081 ceph-mon[31453]: pgmap v19071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:10.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:09 smithi153 ceph-mon[36569]: pgmap v19071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:10.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:09 smithi104 ceph-mon[36624]: pgmap v19071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:12.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:11 smithi081 ceph-mon[31453]: pgmap v19072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:12.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:11 smithi153 ceph-mon[36569]: pgmap v19072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:12.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:11 smithi104 ceph-mon[36624]: pgmap v19072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:14.140 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:13 smithi104 ceph-mon[36624]: pgmap v19073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:14.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:13 smithi081 ceph-mon[31453]: pgmap v19073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:14.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:13 smithi153 ceph-mon[36569]: pgmap v19073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:16.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:15 smithi081 ceph-mon[31453]: pgmap v19074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:16.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:15 smithi153 ceph-mon[36569]: pgmap v19074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:16.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:15 smithi104 ceph-mon[36624]: pgmap v19074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:18.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:17 smithi081 ceph-mon[31453]: pgmap v19075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:18.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:17 smithi153 ceph-mon[36569]: pgmap v19075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:18.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:17 smithi104 ceph-mon[36624]: pgmap v19075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:20.061+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:12:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:20.061+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:12:20.067 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:20.061+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:12:20.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:19 smithi081 ceph-mon[31453]: pgmap v19076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:20.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:19 smithi153 ceph-mon[36569]: pgmap v19076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:20.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:19 smithi104 ceph-mon[36624]: pgmap v19076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:22.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:21 smithi081 ceph-mon[31453]: pgmap v19077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:22.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:21 smithi153 ceph-mon[36569]: pgmap v19077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:22.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:21 smithi104 ceph-mon[36624]: pgmap v19077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:24.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:23 smithi081 ceph-mon[31453]: pgmap v19078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:24.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:23 smithi153 ceph-mon[36569]: pgmap v19078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:24.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:23 smithi104 ceph-mon[36624]: pgmap v19078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:26.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:25 smithi081 ceph-mon[31453]: pgmap v19079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:26.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:25 smithi153 ceph-mon[36569]: pgmap v19079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:26.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:25 smithi104 ceph-mon[36624]: pgmap v19079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:28.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:27 smithi153 ceph-mon[36569]: pgmap v19080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:28.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:27 smithi104 ceph-mon[36624]: pgmap v19080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:28.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:27 smithi081 ceph-mon[31453]: pgmap v19080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:29.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:12:29.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:12:29.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:12:30.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:29 smithi153 ceph-mon[36569]: pgmap v19081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:30.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:12:30.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:12:30.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:12:30.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:29 smithi104 ceph-mon[36624]: pgmap v19081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:30.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:12:30.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:12:30.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:12:30.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:29 smithi081 ceph-mon[31453]: pgmap v19081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:30.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:12:30.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:12:30.349 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:12:31.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:30 smithi153 ceph-mon[36569]: pgmap v19082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:31.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:30 smithi153 ceph-mon[36569]: pgmap v19083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:31.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:30 smithi104 ceph-mon[36624]: pgmap v19082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:31.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:30 smithi104 ceph-mon[36624]: pgmap v19083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:31.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:30 smithi081 ceph-mon[31453]: pgmap v19082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:31.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:30 smithi081 ceph-mon[31453]: pgmap v19083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:33.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:32 smithi153 ceph-mon[36569]: pgmap v19084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:33.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:32 smithi104 ceph-mon[36624]: pgmap v19084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:33.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:32 smithi081 ceph-mon[31453]: pgmap v19084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:35.072+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:12:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:35.072+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:12:35.078 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:35.072+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:12:35.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:34 smithi153 ceph-mon[36569]: pgmap v19085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:35.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:34 smithi104 ceph-mon[36624]: pgmap v19085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:35.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:34 smithi081 ceph-mon[31453]: pgmap v19085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:37.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:36 smithi153 ceph-mon[36569]: pgmap v19086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:37.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:36 smithi104 ceph-mon[36624]: pgmap v19086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:37.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:36 smithi081 ceph-mon[31453]: pgmap v19086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:39.214 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:38 smithi081 ceph-mon[31453]: pgmap v19087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:39.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:38 smithi153 ceph-mon[36569]: pgmap v19087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:39.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:38 smithi104 ceph-mon[36624]: pgmap v19087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:41.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:40 smithi153 ceph-mon[36569]: pgmap v19088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:41.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:40 smithi104 ceph-mon[36624]: pgmap v19088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:41.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:40 smithi081 ceph-mon[31453]: pgmap v19088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:42.312 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:42 smithi153 ceph-mon[36569]: pgmap v19089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:42.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:42 smithi104 ceph-mon[36624]: pgmap v19089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:42.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:42 smithi081 ceph-mon[31453]: pgmap v19089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:44.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:44 smithi081 ceph-mon[31453]: pgmap v19090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:45.041 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:44 smithi104 ceph-mon[36624]: pgmap v19090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:45.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:44 smithi153 ceph-mon[36569]: pgmap v19090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:46.911 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:46 smithi081 ceph-mon[31453]: pgmap v19091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:47.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:46 smithi153 ceph-mon[36569]: pgmap v19091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:47.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:46 smithi104 ceph-mon[36624]: pgmap v19091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:49.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:48 smithi153 ceph-mon[36569]: pgmap v19092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:49.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:48 smithi104 ceph-mon[36624]: pgmap v19092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:49.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:48 smithi081 ceph-mon[31453]: pgmap v19092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:50.080+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:12:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:50.080+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:12:50.086 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:12:50.080+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:12:51.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:50 smithi153 ceph-mon[36569]: pgmap v19093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:51.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:50 smithi104 ceph-mon[36624]: pgmap v19093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:51.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:50 smithi081 ceph-mon[31453]: pgmap v19093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:53.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:52 smithi153 ceph-mon[36569]: pgmap v19094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:53.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:52 smithi104 ceph-mon[36624]: pgmap v19094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:53.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:52 smithi081 ceph-mon[31453]: pgmap v19094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:55.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:54 smithi153 ceph-mon[36569]: pgmap v19095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:55.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:54 smithi104 ceph-mon[36624]: pgmap v19095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:55.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:54 smithi081 ceph-mon[31453]: pgmap v19095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:57.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:56 smithi153 ceph-mon[36569]: pgmap v19096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:57.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:56 smithi104 ceph-mon[36624]: pgmap v19096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:57.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:56 smithi081 ceph-mon[31453]: pgmap v19096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:59.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:12:58 smithi153 ceph-mon[36569]: pgmap v19097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:59.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:12:58 smithi104 ceph-mon[36624]: pgmap v19097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:12:59.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:12:58 smithi081 ceph-mon[31453]: pgmap v19097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:01.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:00 smithi153 ceph-mon[36569]: pgmap v19098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:01.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:00 smithi104 ceph-mon[36624]: pgmap v19098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:01.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:00 smithi081 ceph-mon[31453]: pgmap v19098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:03.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:02 smithi153 ceph-mon[36569]: pgmap v19099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:03.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:02 smithi104 ceph-mon[36624]: pgmap v19099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:03.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:02 smithi081 ceph-mon[31453]: pgmap v19099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:05.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:04 smithi153 ceph-mon[36569]: pgmap v19100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:05.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:04 smithi104 ceph-mon[36624]: pgmap v19100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:05.088+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:13:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:05.088+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:13:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:05.088+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:13:05.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:04 smithi081 ceph-mon[31453]: pgmap v19100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:07.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:06 smithi153 ceph-mon[36569]: pgmap v19101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:07.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:06 smithi104 ceph-mon[36624]: pgmap v19101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:07.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:06 smithi081 ceph-mon[31453]: pgmap v19101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:09.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:08 smithi153 ceph-mon[36569]: pgmap v19102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:09.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:08 smithi104 ceph-mon[36624]: pgmap v19102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:09.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:08 smithi081 ceph-mon[31453]: pgmap v19102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:11.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:10 smithi153 ceph-mon[36569]: pgmap v19103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:11.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:10 smithi104 ceph-mon[36624]: pgmap v19103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:11.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:10 smithi081 ceph-mon[31453]: pgmap v19103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:13.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:12 smithi153 ceph-mon[36569]: pgmap v19104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:13.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:12 smithi104 ceph-mon[36624]: pgmap v19104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:13.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:12 smithi081 ceph-mon[31453]: pgmap v19104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:15.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:14 smithi153 ceph-mon[36569]: pgmap v19105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:15.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:14 smithi104 ceph-mon[36624]: pgmap v19105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:15.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:14 smithi081 ceph-mon[31453]: pgmap v19105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:17.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:16 smithi153 ceph-mon[36569]: pgmap v19106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:17.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:16 smithi104 ceph-mon[36624]: pgmap v19106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:17.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:16 smithi081 ceph-mon[31453]: pgmap v19106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:18.631+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:13:18.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:18.631+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:13:18.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:18.632+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 276 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:13:18.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:18.632+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 276 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:13:19.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:18 smithi153 ceph-mon[36569]: pgmap v19107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:19.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:18 smithi104 ceph-mon[36624]: pgmap v19107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:19.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:18 smithi081 ceph-mon[31453]: pgmap v19107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:20.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:20.094+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:13:20.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:20.094+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:13:20.101 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:20.094+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:13:21.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:20 smithi153 ceph-mon[36569]: pgmap v19108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:21.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:20 smithi104 ceph-mon[36624]: pgmap v19108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:21.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:20 smithi081 ceph-mon[31453]: pgmap v19108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:23.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:22 smithi153 ceph-mon[36569]: pgmap v19109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:23.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:22 smithi104 ceph-mon[36624]: pgmap v19109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:23.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:22 smithi081 ceph-mon[31453]: pgmap v19109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:25.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:24 smithi153 ceph-mon[36569]: pgmap v19110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:25.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:24 smithi104 ceph-mon[36624]: pgmap v19110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:25.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:24 smithi081 ceph-mon[31453]: pgmap v19110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:27.062 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:26 smithi153 ceph-mon[36569]: pgmap v19111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:27.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:26 smithi104 ceph-mon[36624]: pgmap v19111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:27.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:26 smithi081 ceph-mon[31453]: pgmap v19111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:29.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:28 smithi153 ceph-mon[36569]: pgmap v19112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:29.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:28 smithi104 ceph-mon[36624]: pgmap v19112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:29.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:28 smithi081 ceph-mon[31453]: pgmap v19112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:31.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:30 smithi153 ceph-mon[36569]: pgmap v19113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:31.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:13:31.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:30 smithi104 ceph-mon[36624]: pgmap v19113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:31.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:13:31.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:30 smithi081 ceph-mon[31453]: pgmap v19113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:31.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:13:32.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:13:32.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:13:32.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:13:32.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:13:32.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:13:32.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:13:32.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:13:32.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:13:32.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:13:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:32 smithi153 ceph-mon[36569]: pgmap v19114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:33.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:32 smithi153 ceph-mon[36569]: pgmap v19115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:33.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:32 smithi104 ceph-mon[36624]: pgmap v19114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:33.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:32 smithi104 ceph-mon[36624]: pgmap v19115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:33.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:32 smithi081 ceph-mon[31453]: pgmap v19114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:33.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:32 smithi081 ceph-mon[31453]: pgmap v19115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:35.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:34 smithi153 ceph-mon[36569]: pgmap v19116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:35.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:34 smithi104 ceph-mon[36624]: pgmap v19116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:35.111+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:13:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:35.111+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:13:35.116 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:35.111+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:13:35.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:34 smithi081 ceph-mon[31453]: pgmap v19116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:37.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:36 smithi153 ceph-mon[36569]: pgmap v19117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:37.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:36 smithi104 ceph-mon[36624]: pgmap v19117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:37.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:36 smithi081 ceph-mon[31453]: pgmap v19117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:39.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:38 smithi153 ceph-mon[36569]: pgmap v19118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:39.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:38 smithi104 ceph-mon[36624]: pgmap v19118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:39.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:38 smithi081 ceph-mon[31453]: pgmap v19118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:41.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:40 smithi153 ceph-mon[36569]: pgmap v19119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:41.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:40 smithi104 ceph-mon[36624]: pgmap v19119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:41.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:40 smithi081 ceph-mon[31453]: pgmap v19119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:43.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:42 smithi153 ceph-mon[36569]: pgmap v19120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:43.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:42 smithi104 ceph-mon[36624]: pgmap v19120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:43.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:42 smithi081 ceph-mon[31453]: pgmap v19120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:45.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:44 smithi104 ceph-mon[36624]: pgmap v19121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:45.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:44 smithi081 ceph-mon[31453]: pgmap v19121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:45.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:44 smithi153 ceph-mon[36569]: pgmap v19121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:47.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:46 smithi104 ceph-mon[36624]: pgmap v19122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:47.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:46 smithi081 ceph-mon[31453]: pgmap v19122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:47.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:46 smithi153 ceph-mon[36569]: pgmap v19122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:49.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:48 smithi104 ceph-mon[36624]: pgmap v19123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:49.162 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:48 smithi081 ceph-mon[31453]: pgmap v19123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:49.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:48 smithi153 ceph-mon[36569]: pgmap v19123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:50.126+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:13:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:50.126+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:13:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:13:50.126+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:13:51.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:50 smithi104 ceph-mon[36624]: pgmap v19124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:51.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:50 smithi081 ceph-mon[31453]: pgmap v19124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:51.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:50 smithi153 ceph-mon[36569]: pgmap v19124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:53.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:52 smithi081 ceph-mon[31453]: pgmap v19125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:53.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:52 smithi153 ceph-mon[36569]: pgmap v19125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:53.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:52 smithi104 ceph-mon[36624]: pgmap v19125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:55.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:54 smithi081 ceph-mon[31453]: pgmap v19126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:55.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:54 smithi153 ceph-mon[36569]: pgmap v19126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:55.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:54 smithi104 ceph-mon[36624]: pgmap v19126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:57.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:56 smithi081 ceph-mon[31453]: pgmap v19127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:57.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:56 smithi153 ceph-mon[36569]: pgmap v19127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:57.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:56 smithi104 ceph-mon[36624]: pgmap v19127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:59.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:13:58 smithi081 ceph-mon[31453]: pgmap v19128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:59.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:13:58 smithi153 ceph-mon[36569]: pgmap v19128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:13:59.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:13:58 smithi104 ceph-mon[36624]: pgmap v19128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:01.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:00 smithi081 ceph-mon[31453]: pgmap v19129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:01.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:00 smithi153 ceph-mon[36569]: pgmap v19129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:01.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:00 smithi104 ceph-mon[36624]: pgmap v19129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:03.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:02 smithi081 ceph-mon[31453]: pgmap v19130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:03.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:02 smithi153 ceph-mon[36569]: pgmap v19130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:03.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:02 smithi104 ceph-mon[36624]: pgmap v19130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:05.142 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:05.138+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:14:05.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:05.138+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:14:05.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:05.138+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:14:05.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:04 smithi081 ceph-mon[31453]: pgmap v19131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:05.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:04 smithi153 ceph-mon[36569]: pgmap v19131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:05.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:04 smithi104 ceph-mon[36624]: pgmap v19131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:07.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:06 smithi081 ceph-mon[31453]: pgmap v19132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:07.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:06 smithi153 ceph-mon[36569]: pgmap v19132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:07.388 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:06 smithi104 ceph-mon[36624]: pgmap v19132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:09.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:08 smithi081 ceph-mon[31453]: pgmap v19133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:09.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:08 smithi153 ceph-mon[36569]: pgmap v19133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:09.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:08 smithi104 ceph-mon[36624]: pgmap v19133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:11.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:10 smithi081 ceph-mon[31453]: pgmap v19134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:11.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:10 smithi153 ceph-mon[36569]: pgmap v19134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:11.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:10 smithi104 ceph-mon[36624]: pgmap v19134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:13.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:12 smithi081 ceph-mon[31453]: pgmap v19135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:13.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:12 smithi153 ceph-mon[36569]: pgmap v19135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:13.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:12 smithi104 ceph-mon[36624]: pgmap v19135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:15.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:14 smithi081 ceph-mon[31453]: pgmap v19136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:15.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:14 smithi153 ceph-mon[36569]: pgmap v19136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:15.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:14 smithi104 ceph-mon[36624]: pgmap v19136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:17.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:16 smithi081 ceph-mon[31453]: pgmap v19137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:17.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:16 smithi153 ceph-mon[36569]: pgmap v19137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:17.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:16 smithi104 ceph-mon[36624]: pgmap v19137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:19.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:18 smithi081 ceph-mon[31453]: pgmap v19138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:19.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:18 smithi153 ceph-mon[36569]: pgmap v19138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:19.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:18 smithi104 ceph-mon[36624]: pgmap v19138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:20.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:20.151+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:14:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:20.151+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:14:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:20.151+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:14:21.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:20 smithi153 ceph-mon[36569]: pgmap v19139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:21.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:20 smithi104 ceph-mon[36624]: pgmap v19139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:21.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:20 smithi081 ceph-mon[31453]: pgmap v19139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:23.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:22 smithi153 ceph-mon[36569]: pgmap v19140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:23.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:22 smithi104 ceph-mon[36624]: pgmap v19140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:23.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:22 smithi081 ceph-mon[31453]: pgmap v19140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:25.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:24 smithi153 ceph-mon[36569]: pgmap v19141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:25.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:24 smithi104 ceph-mon[36624]: pgmap v19141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:25.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:24 smithi081 ceph-mon[31453]: pgmap v19141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:27.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:26 smithi153 ceph-mon[36569]: pgmap v19142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:27.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:26 smithi104 ceph-mon[36624]: pgmap v19142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:27.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:26 smithi081 ceph-mon[31453]: pgmap v19142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:29.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:28 smithi153 ceph-mon[36569]: pgmap v19143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:29.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:28 smithi104 ceph-mon[36624]: pgmap v19143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:29.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:28 smithi081 ceph-mon[31453]: pgmap v19143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:31.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:30 smithi153 ceph-mon[36569]: pgmap v19144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:31.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:30 smithi104 ceph-mon[36624]: pgmap v19144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:31.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:30 smithi081 ceph-mon[31453]: pgmap v19144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:32.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:14:32.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:14:32.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:14:33.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:32 smithi153 ceph-mon[36569]: pgmap v19145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:33.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:14:33.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:14:33.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:14:33.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:32 smithi104 ceph-mon[36624]: pgmap v19145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:33.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:14:33.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:14:33.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:14:33.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:32 smithi081 ceph-mon[31453]: pgmap v19145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:33.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:14:33.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:14:33.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:14:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:33 smithi153 ceph-mon[36569]: pgmap v19146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:33 smithi153 ceph-mon[36569]: pgmap v19147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:34.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:33 smithi104 ceph-mon[36624]: pgmap v19146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:34.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:33 smithi104 ceph-mon[36624]: pgmap v19147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:33 smithi081 ceph-mon[31453]: pgmap v19146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:33 smithi081 ceph-mon[31453]: pgmap v19147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:35.167+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:14:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:35.167+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:14:35.171 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:35.167+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:14:36.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:35 smithi153 ceph-mon[36569]: pgmap v19148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:36.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:35 smithi104 ceph-mon[36624]: pgmap v19148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:36.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:35 smithi081 ceph-mon[31453]: pgmap v19148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:38.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:37 smithi153 ceph-mon[36569]: pgmap v19149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:38.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:37 smithi104 ceph-mon[36624]: pgmap v19149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:38.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:37 smithi081 ceph-mon[31453]: pgmap v19149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:40.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:39 smithi153 ceph-mon[36569]: pgmap v19150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:40.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:39 smithi104 ceph-mon[36624]: pgmap v19150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:40.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:39 smithi081 ceph-mon[31453]: pgmap v19150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:41.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:41 smithi153 ceph-mon[36569]: pgmap v19151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:41.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:41 smithi104 ceph-mon[36624]: pgmap v19151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:41.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:41 smithi081 ceph-mon[31453]: pgmap v19151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:44.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:43 smithi153 ceph-mon[36569]: pgmap v19152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:44.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:43 smithi104 ceph-mon[36624]: pgmap v19152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:44.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:43 smithi081 ceph-mon[31453]: pgmap v19152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:46.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:45 smithi153 ceph-mon[36569]: pgmap v19153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:46.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:45 smithi104 ceph-mon[36624]: pgmap v19153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:46.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:45 smithi081 ceph-mon[31453]: pgmap v19153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:48.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:47 smithi153 ceph-mon[36569]: pgmap v19154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:48.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:47 smithi104 ceph-mon[36624]: pgmap v19154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:48.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:47 smithi081 ceph-mon[31453]: pgmap v19154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:50.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:49 smithi153 ceph-mon[36569]: pgmap v19155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:50.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:49 smithi104 ceph-mon[36624]: pgmap v19155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:50.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:49 smithi081 ceph-mon[31453]: pgmap v19155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:50.174+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:14:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:50.174+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:14:50.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:14:50.174+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:14:52.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:51 smithi153 ceph-mon[36569]: pgmap v19156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:52.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:51 smithi104 ceph-mon[36624]: pgmap v19156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:52.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:51 smithi081 ceph-mon[31453]: pgmap v19156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:54.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:53 smithi104 ceph-mon[36624]: pgmap v19157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:54.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:53 smithi081 ceph-mon[31453]: pgmap v19157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:54.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:53 smithi153 ceph-mon[36569]: pgmap v19157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:56.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:55 smithi104 ceph-mon[36624]: pgmap v19158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:56.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:55 smithi081 ceph-mon[31453]: pgmap v19158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:56.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:55 smithi153 ceph-mon[36569]: pgmap v19158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:58.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:57 smithi081 ceph-mon[31453]: pgmap v19159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:58.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:57 smithi153 ceph-mon[36569]: pgmap v19159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:14:58.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:57 smithi104 ceph-mon[36624]: pgmap v19159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:00.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:14:59 smithi081 ceph-mon[31453]: pgmap v19160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:00.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:14:59 smithi153 ceph-mon[36569]: pgmap v19160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:00.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:14:59 smithi104 ceph-mon[36624]: pgmap v19160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:02.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:01 smithi081 ceph-mon[31453]: pgmap v19161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:02.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:01 smithi153 ceph-mon[36569]: pgmap v19161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:02.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:01 smithi104 ceph-mon[36624]: pgmap v19161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:04.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:03 smithi081 ceph-mon[31453]: pgmap v19162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:04.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:03 smithi153 ceph-mon[36569]: pgmap v19162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:04.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:03 smithi104 ceph-mon[36624]: pgmap v19162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:05.183+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:15:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:05.183+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:15:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:05.183+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:15:06.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:05 smithi081 ceph-mon[31453]: pgmap v19163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:06.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:05 smithi153 ceph-mon[36569]: pgmap v19163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:06.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:05 smithi104 ceph-mon[36624]: pgmap v19163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:08.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:07 smithi081 ceph-mon[31453]: pgmap v19164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:08.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:07 smithi153 ceph-mon[36569]: pgmap v19164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:08.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:07 smithi104 ceph-mon[36624]: pgmap v19164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:10.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:09 smithi081 ceph-mon[31453]: pgmap v19165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:10.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:09 smithi153 ceph-mon[36569]: pgmap v19165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:10.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:09 smithi104 ceph-mon[36624]: pgmap v19165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:12.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:11 smithi081 ceph-mon[31453]: pgmap v19166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:12.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:11 smithi153 ceph-mon[36569]: pgmap v19166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:12.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:11 smithi104 ceph-mon[36624]: pgmap v19166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:14.161 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:13 smithi081 ceph-mon[31453]: pgmap v19167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:14.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:13 smithi153 ceph-mon[36569]: pgmap v19167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:14.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:13 smithi104 ceph-mon[36624]: pgmap v19167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:16.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:15 smithi081 ceph-mon[31453]: pgmap v19168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:16.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:15 smithi153 ceph-mon[36569]: pgmap v19168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:16.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:15 smithi104 ceph-mon[36624]: pgmap v19168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:18.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:17 smithi081 ceph-mon[31453]: pgmap v19169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:18.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:17 smithi153 ceph-mon[36569]: pgmap v19169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:18.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:17 smithi104 ceph-mon[36624]: pgmap v19169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:20.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:19 smithi081 ceph-mon[31453]: pgmap v19170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:20.196+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:15:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:20.196+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:15:20.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:20.197+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:15:20.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:19 smithi153 ceph-mon[36569]: pgmap v19170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:20.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:19 smithi104 ceph-mon[36624]: pgmap v19170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:22.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:21 smithi081 ceph-mon[31453]: pgmap v19171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:22.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:21 smithi153 ceph-mon[36569]: pgmap v19171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:22.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:21 smithi104 ceph-mon[36624]: pgmap v19171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:24.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:23 smithi081 ceph-mon[31453]: pgmap v19172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:24.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:23 smithi153 ceph-mon[36569]: pgmap v19172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:24.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:23 smithi104 ceph-mon[36624]: pgmap v19172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:26.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:25 smithi081 ceph-mon[31453]: pgmap v19173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:26.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:25 smithi153 ceph-mon[36569]: pgmap v19173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:26.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:25 smithi104 ceph-mon[36624]: pgmap v19173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:28.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:27 smithi153 ceph-mon[36569]: pgmap v19174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:28.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:27 smithi104 ceph-mon[36624]: pgmap v19174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:28.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:27 smithi081 ceph-mon[31453]: pgmap v19174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:30.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:29 smithi153 ceph-mon[36569]: pgmap v19175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:30.338 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:29 smithi104 ceph-mon[36624]: pgmap v19175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:30.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:29 smithi081 ceph-mon[31453]: pgmap v19175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:32.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:31 smithi153 ceph-mon[36569]: pgmap v19176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:32.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:31 smithi104 ceph-mon[36624]: pgmap v19176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:32.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:31 smithi081 ceph-mon[31453]: pgmap v19176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:34.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:33 smithi153 ceph-mon[36569]: pgmap v19177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:34.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:15:34.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:33 smithi104 ceph-mon[36624]: pgmap v19177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:15:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:33 smithi081 ceph-mon[31453]: pgmap v19177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:15:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:34.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:35.204+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:15:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:35.204+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:15:35.208 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:35.204+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:15:35.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:34 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:15:35.257 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:34 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:15:35.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:34 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:15:35.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:34 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:15:35.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:34 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:15:35.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:34 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:15:36.302 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:35 smithi081 ceph-mon[31453]: pgmap v19178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:36.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:35 smithi153 ceph-mon[36569]: pgmap v19178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:36.336 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:35 smithi104 ceph-mon[36624]: pgmap v19178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:38.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:37 smithi153 ceph-mon[36569]: pgmap v19179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:38.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:38.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:38.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:37 smithi104 ceph-mon[36624]: pgmap v19179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:38.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:38.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:38.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.323 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:37 smithi081 ceph-mon[31453]: pgmap v19179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:38.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:38.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:38.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:38.411 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:39.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:38 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:15:39.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:38 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:15:39.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:38 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:15:39.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:38 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:15:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:38 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:15:39.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:38 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:15:40.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:39 smithi081 ceph-mon[31453]: pgmap v19180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:40.211 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:39 smithi153 ceph-mon[36569]: pgmap v19180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:40.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:39 smithi104 ceph-mon[36624]: pgmap v19180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:15:41.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:15:41.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:15:41.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:15:41.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: pgmap v19181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:41.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: pgmap v19182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:41.587 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:41.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:41.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:41.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:15:41.660 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:15:41.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:15:41.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:15:41.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: pgmap v19181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:41.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: pgmap v19182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:41.661 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:15:41.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:15:41.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:15:41.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:15:41.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:15:41.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: pgmap v19181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:41.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: pgmap v19182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:41.811 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:15:43.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:43 smithi153 ceph-mon[36569]: pgmap v19183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:43.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:43 smithi104 ceph-mon[36624]: pgmap v19183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:43.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:43 smithi081 ceph-mon[31453]: pgmap v19183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:45.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:45 smithi153 ceph-mon[36569]: pgmap v19184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:45.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:45 smithi104 ceph-mon[36624]: pgmap v19184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:45.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:45 smithi081 ceph-mon[31453]: pgmap v19184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:47.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:47 smithi153 ceph-mon[36569]: pgmap v19185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:47.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:47 smithi104 ceph-mon[36624]: pgmap v19185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:47.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:47 smithi081 ceph-mon[31453]: pgmap v19185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:49.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:49 smithi104 ceph-mon[36624]: pgmap v19186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:49.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:49 smithi153 ceph-mon[36569]: pgmap v19186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:49.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:49 smithi081 ceph-mon[31453]: pgmap v19186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:50.217+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:15:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:50.217+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:15:50.221 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:15:50.217+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:15:51.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:51 smithi153 ceph-mon[36569]: pgmap v19187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:51.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:51 smithi104 ceph-mon[36624]: pgmap v19187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:51.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:51 smithi081 ceph-mon[31453]: pgmap v19187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:53.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:53 smithi153 ceph-mon[36569]: pgmap v19188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:53.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:53 smithi104 ceph-mon[36624]: pgmap v19188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:53.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:53 smithi081 ceph-mon[31453]: pgmap v19188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:55.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:55 smithi153 ceph-mon[36569]: pgmap v19189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:55.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:55 smithi104 ceph-mon[36624]: pgmap v19189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:55.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:55 smithi081 ceph-mon[31453]: pgmap v19189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:57.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:57 smithi153 ceph-mon[36569]: pgmap v19190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:57.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:57 smithi104 ceph-mon[36624]: pgmap v19190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:57.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:57 smithi081 ceph-mon[31453]: pgmap v19190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:59.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:15:59 smithi153 ceph-mon[36569]: pgmap v19191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:59.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:15:59 smithi104 ceph-mon[36624]: pgmap v19191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:15:59.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:15:59 smithi081 ceph-mon[31453]: pgmap v19191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:01.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:01 smithi153 ceph-mon[36569]: pgmap v19192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:01.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:01 smithi104 ceph-mon[36624]: pgmap v19192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:01.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:01 smithi081 ceph-mon[31453]: pgmap v19192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:03.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:03 smithi153 ceph-mon[36569]: pgmap v19193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:03.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:03 smithi104 ceph-mon[36624]: pgmap v19193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:03.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:03 smithi081 ceph-mon[31453]: pgmap v19193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:05.232+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:16:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:05.232+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:16:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:05.232+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:16:05.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:05 smithi153 ceph-mon[36569]: pgmap v19194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:05.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:05 smithi104 ceph-mon[36624]: pgmap v19194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:05.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:05 smithi081 ceph-mon[31453]: pgmap v19194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:07.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:07 smithi153 ceph-mon[36569]: pgmap v19195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:07.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:07 smithi104 ceph-mon[36624]: pgmap v19195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:07.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:07 smithi081 ceph-mon[31453]: pgmap v19195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:09.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:09 smithi153 ceph-mon[36569]: pgmap v19196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:09.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:09 smithi104 ceph-mon[36624]: pgmap v19196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:09.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:09 smithi081 ceph-mon[31453]: pgmap v19196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:11.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:11 smithi153 ceph-mon[36569]: pgmap v19197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:11.837 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:11 smithi104 ceph-mon[36624]: pgmap v19197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:11.910 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:11 smithi081 ceph-mon[31453]: pgmap v19197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:12.189+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 msgr2=0x7f16e80b1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000716) for more than 900000000 us, fault. 2024-04-13T07:16:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:12.189+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ce5f0 tx=0x7f16f4032000 comp rx=0 tx=0).stop 2024-04-13T07:16:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:12.189+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 msgr2=0x7f16e80b1a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T07:16:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:12.189+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4f50 0x7f16e80b1a60 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T07:16:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:12.189+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T07:16:12.193 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:12.190+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T07:16:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:12.191+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006530 tx=0x7f16f404e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T07:16:13.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:13 smithi153 ceph-mon[36569]: pgmap v19198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:13.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:13 smithi104 ceph-mon[36624]: pgmap v19198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:13.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:13 smithi081 ceph-mon[31453]: pgmap v19198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:15.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:15 smithi153 ceph-mon[36569]: pgmap v19199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:15.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:15 smithi104 ceph-mon[36624]: pgmap v19199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:15.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:15 smithi081 ceph-mon[31453]: pgmap v19199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:17.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:17 smithi153 ceph-mon[36569]: pgmap v19200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:17.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:17 smithi104 ceph-mon[36624]: pgmap v19200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:17.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:17 smithi081 ceph-mon[31453]: pgmap v19200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:18.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:18.662+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:16:18.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:18.662+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:16:18.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:18.663+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 277 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:16:18.666 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:18.663+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 277 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:16:19.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:19 smithi153 ceph-mon[36569]: pgmap v19201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:19.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:19 smithi104 ceph-mon[36624]: pgmap v19201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:19.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:19 smithi081 ceph-mon[31453]: pgmap v19201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:20.242+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:16:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:20.242+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:16:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:20.242+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:16:21.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:21 smithi153 ceph-mon[36569]: pgmap v19202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:21.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:21 smithi104 ceph-mon[36624]: pgmap v19202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:21.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:21 smithi081 ceph-mon[31453]: pgmap v19202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:23.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:23 smithi153 ceph-mon[36569]: pgmap v19203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:23.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:23 smithi104 ceph-mon[36624]: pgmap v19203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:23.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:23 smithi081 ceph-mon[31453]: pgmap v19203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:25.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:25 smithi153 ceph-mon[36569]: pgmap v19204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:25.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:25 smithi104 ceph-mon[36624]: pgmap v19204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:25.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:25 smithi081 ceph-mon[31453]: pgmap v19204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:27.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:27 smithi153 ceph-mon[36569]: pgmap v19205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:27.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:27 smithi104 ceph-mon[36624]: pgmap v19205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:27.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:27 smithi081 ceph-mon[31453]: pgmap v19205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:29.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:29 smithi153 ceph-mon[36569]: pgmap v19206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:29.835 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:29 smithi104 ceph-mon[36624]: pgmap v19206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:29.909 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:29 smithi081 ceph-mon[31453]: pgmap v19206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:32.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:31 smithi153 ceph-mon[36569]: pgmap v19207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:32.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:31 smithi104 ceph-mon[36624]: pgmap v19207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:32.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:31 smithi081 ceph-mon[31453]: pgmap v19207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:34.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:33 smithi153 ceph-mon[36569]: pgmap v19208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:34.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:33 smithi104 ceph-mon[36624]: pgmap v19208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:34.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:33 smithi081 ceph-mon[31453]: pgmap v19208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:35.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:35.244+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:16:35.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:35.244+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:16:35.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:35.244+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:16:36.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:35 smithi153 ceph-mon[36569]: pgmap v19209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:36.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:35 smithi104 ceph-mon[36624]: pgmap v19209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:36.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:35 smithi081 ceph-mon[31453]: pgmap v19209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:38.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:37 smithi153 ceph-mon[36569]: pgmap v19210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:38.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:37 smithi104 ceph-mon[36624]: pgmap v19210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:38.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:37 smithi081 ceph-mon[31453]: pgmap v19210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:40.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:39 smithi153 ceph-mon[36569]: pgmap v19211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:40.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:39 smithi104 ceph-mon[36624]: pgmap v19211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:40.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:39 smithi081 ceph-mon[31453]: pgmap v19211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:41.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:16:41.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:16:41.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:16:42.012 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:41 smithi153 ceph-mon[36569]: pgmap v19212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:42.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:41 smithi104 ceph-mon[36624]: pgmap v19212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:42.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:41 smithi081 ceph-mon[31453]: pgmap v19212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:44.060 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:43 smithi153 ceph-mon[36569]: pgmap v19213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:44.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:43 smithi104 ceph-mon[36624]: pgmap v19213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:44.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:43 smithi081 ceph-mon[31453]: pgmap v19213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:45.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:45 smithi153 ceph-mon[36569]: pgmap v19214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:45.957 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:45.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:45.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:16:45.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:16:45.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:46.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:45 smithi104 ceph-mon[36624]: pgmap v19214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:46.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:46.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:46.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:16:46.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:16:46.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:46.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:45 smithi081 ceph-mon[31453]: pgmap v19214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:46.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:46.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:46.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:16:46.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:16:46.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:16:47.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:46 smithi153 ceph-mon[36569]: pgmap v19215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:47.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:46 smithi153 ceph-mon[36569]: pgmap v19216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:47.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:46 smithi104 ceph-mon[36624]: pgmap v19215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:47.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:46 smithi104 ceph-mon[36624]: pgmap v19216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:47.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:46 smithi081 ceph-mon[31453]: pgmap v19215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:47.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:46 smithi081 ceph-mon[31453]: pgmap v19216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:49.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:48 smithi153 ceph-mon[36569]: pgmap v19217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:49.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:48 smithi104 ceph-mon[36624]: pgmap v19217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:49.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:48 smithi081 ceph-mon[31453]: pgmap v19217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:50.254 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:50.250+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:16:50.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:50.250+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:16:50.255 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:16:50.250+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:16:51.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:50 smithi153 ceph-mon[36569]: pgmap v19218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:51.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:50 smithi104 ceph-mon[36624]: pgmap v19218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:51.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:50 smithi081 ceph-mon[31453]: pgmap v19218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:53.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:52 smithi153 ceph-mon[36569]: pgmap v19219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:53.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:52 smithi104 ceph-mon[36624]: pgmap v19219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:53.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:52 smithi081 ceph-mon[31453]: pgmap v19219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:55.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:54 smithi153 ceph-mon[36569]: pgmap v19220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:55.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:54 smithi104 ceph-mon[36624]: pgmap v19220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:55.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:54 smithi081 ceph-mon[31453]: pgmap v19220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:57.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:56 smithi153 ceph-mon[36569]: pgmap v19221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:57.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:56 smithi104 ceph-mon[36624]: pgmap v19221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:57.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:56 smithi081 ceph-mon[31453]: pgmap v19221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:59.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:16:58 smithi153 ceph-mon[36569]: pgmap v19222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:59.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:16:58 smithi104 ceph-mon[36624]: pgmap v19222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:16:59.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:16:58 smithi081 ceph-mon[31453]: pgmap v19222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:01.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:00 smithi153 ceph-mon[36569]: pgmap v19223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:01.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:00 smithi104 ceph-mon[36624]: pgmap v19223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:01.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:00 smithi081 ceph-mon[31453]: pgmap v19223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:03.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:02 smithi153 ceph-mon[36569]: pgmap v19224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:03.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:02 smithi104 ceph-mon[36624]: pgmap v19224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:03.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:02 smithi081 ceph-mon[31453]: pgmap v19224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:05.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:04 smithi153 ceph-mon[36569]: pgmap v19225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:05.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:04 smithi104 ceph-mon[36624]: pgmap v19225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:05.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:04 smithi081 ceph-mon[31453]: pgmap v19225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:05.258+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:17:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:05.258+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:17:05.262 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:05.258+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:17:07.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:06 smithi153 ceph-mon[36569]: pgmap v19226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:07.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:06 smithi104 ceph-mon[36624]: pgmap v19226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:07.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:06 smithi081 ceph-mon[31453]: pgmap v19226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:09.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:08 smithi153 ceph-mon[36569]: pgmap v19227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:09.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:08 smithi104 ceph-mon[36624]: pgmap v19227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:09.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:08 smithi081 ceph-mon[31453]: pgmap v19227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:11.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:10 smithi104 ceph-mon[36624]: pgmap v19228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:11.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:10 smithi081 ceph-mon[31453]: pgmap v19228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:11.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:10 smithi153 ceph-mon[36569]: pgmap v19228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:13.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:12 smithi104 ceph-mon[36624]: pgmap v19229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:13.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:12 smithi153 ceph-mon[36569]: pgmap v19229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:13.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:12 smithi081 ceph-mon[31453]: pgmap v19229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:15.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:14 smithi104 ceph-mon[36624]: pgmap v19230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:15.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:14 smithi081 ceph-mon[31453]: pgmap v19230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:15.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:14 smithi153 ceph-mon[36569]: pgmap v19230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:17.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:16 smithi104 ceph-mon[36624]: pgmap v19231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:17.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:16 smithi081 ceph-mon[31453]: pgmap v19231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:16 smithi153 ceph-mon[36569]: pgmap v19231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:19.085 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:18 smithi104 ceph-mon[36624]: pgmap v19232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:19.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:18 smithi081 ceph-mon[31453]: pgmap v19232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:18 smithi153 ceph-mon[36569]: pgmap v19232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:20.274+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:17:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:20.274+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:17:20.277 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:20.274+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:17:21.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:20 smithi081 ceph-mon[31453]: pgmap v19233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:20 smithi153 ceph-mon[36569]: pgmap v19233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:21.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:20 smithi104 ceph-mon[36624]: pgmap v19233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:23.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:22 smithi081 ceph-mon[31453]: pgmap v19234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:22 smithi153 ceph-mon[36569]: pgmap v19234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:23.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:22 smithi104 ceph-mon[36624]: pgmap v19234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:25.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:24 smithi081 ceph-mon[31453]: pgmap v19235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:25.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:24 smithi153 ceph-mon[36569]: pgmap v19235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:25.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:24 smithi104 ceph-mon[36624]: pgmap v19235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:27.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:26 smithi081 ceph-mon[31453]: pgmap v19236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:27.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:26 smithi153 ceph-mon[36569]: pgmap v19236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:27.337 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:26 smithi104 ceph-mon[36624]: pgmap v19236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:29.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:28 smithi081 ceph-mon[31453]: pgmap v19237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:28 smithi153 ceph-mon[36569]: pgmap v19237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:29.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:28 smithi104 ceph-mon[36624]: pgmap v19237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:31.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:30 smithi081 ceph-mon[31453]: pgmap v19238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:31.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:30 smithi153 ceph-mon[36569]: pgmap v19238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:31.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:30 smithi104 ceph-mon[36624]: pgmap v19238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:33.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:32 smithi081 ceph-mon[31453]: pgmap v19239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:32 smithi153 ceph-mon[36569]: pgmap v19239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:33.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:32 smithi104 ceph-mon[36624]: pgmap v19239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:35.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:34 smithi081 ceph-mon[31453]: pgmap v19240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:35.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:35.290+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:17:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:35.290+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:17:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:35.290+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:17:35.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:34 smithi153 ceph-mon[36569]: pgmap v19240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:35.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:34 smithi104 ceph-mon[36624]: pgmap v19240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:37.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:36 smithi081 ceph-mon[31453]: pgmap v19241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:37.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:36 smithi153 ceph-mon[36569]: pgmap v19241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:37.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:36 smithi104 ceph-mon[36624]: pgmap v19241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:39.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:38 smithi081 ceph-mon[31453]: pgmap v19242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:39.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:38 smithi153 ceph-mon[36569]: pgmap v19242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:39.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:38 smithi104 ceph-mon[36624]: pgmap v19242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:41.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:40 smithi081 ceph-mon[31453]: pgmap v19243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:41.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:40 smithi153 ceph-mon[36569]: pgmap v19243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:40 smithi104 ceph-mon[36624]: pgmap v19243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:42 smithi081 ceph-mon[31453]: pgmap v19244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:43.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:42 smithi153 ceph-mon[36569]: pgmap v19244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:43.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:42 smithi104 ceph-mon[36624]: pgmap v19244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:45.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:44 smithi081 ceph-mon[31453]: pgmap v19245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:45.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:44 smithi153 ceph-mon[36569]: pgmap v19245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:45.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:44 smithi104 ceph-mon[36624]: pgmap v19245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:47.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:46 smithi081 ceph-mon[31453]: pgmap v19246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:47.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:17:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:46 smithi153 ceph-mon[36569]: pgmap v19246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:47.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:17:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:46 smithi104 ceph-mon[36624]: pgmap v19246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:47.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:17:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:17:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:17:48.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:17:48.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:17:48.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:17:48.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:17:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:17:48.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:17:48.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:17:49.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:48 smithi153 ceph-mon[36569]: pgmap v19247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:49.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:48 smithi153 ceph-mon[36569]: pgmap v19248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:48 smithi104 ceph-mon[36624]: pgmap v19247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:49.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:48 smithi104 ceph-mon[36624]: pgmap v19248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:48 smithi081 ceph-mon[31453]: pgmap v19247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:49.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:48 smithi081 ceph-mon[31453]: pgmap v19248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:50.296+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:17:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:50.296+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:17:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:17:50.296+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:17:51.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:50 smithi153 ceph-mon[36569]: pgmap v19249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:51.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:50 smithi104 ceph-mon[36624]: pgmap v19249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:51.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:50 smithi081 ceph-mon[31453]: pgmap v19249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:52 smithi153 ceph-mon[36569]: pgmap v19250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:52 smithi104 ceph-mon[36624]: pgmap v19250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:53.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:52 smithi081 ceph-mon[31453]: pgmap v19250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:55.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:54 smithi153 ceph-mon[36569]: pgmap v19251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:55.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:54 smithi104 ceph-mon[36624]: pgmap v19251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:55.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:54 smithi081 ceph-mon[31453]: pgmap v19251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:57.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:56 smithi153 ceph-mon[36569]: pgmap v19252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:56 smithi104 ceph-mon[36624]: pgmap v19252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:57.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:56 smithi081 ceph-mon[31453]: pgmap v19252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:59.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:17:58 smithi153 ceph-mon[36569]: pgmap v19253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:17:58 smithi104 ceph-mon[36624]: pgmap v19253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:17:59.339 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:17:58 smithi081 ceph-mon[31453]: pgmap v19253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:01.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:00 smithi153 ceph-mon[36569]: pgmap v19254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:00 smithi104 ceph-mon[36624]: pgmap v19254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:00 smithi081 ceph-mon[31453]: pgmap v19254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:03.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:02 smithi153 ceph-mon[36569]: pgmap v19255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:03.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:02 smithi104 ceph-mon[36624]: pgmap v19255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:03.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:02 smithi081 ceph-mon[31453]: pgmap v19255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:05.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:04 smithi153 ceph-mon[36569]: pgmap v19256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:05.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:05.311+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:18:05.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:05.311+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:18:05.314 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:05.311+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:18:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:04 smithi104 ceph-mon[36624]: pgmap v19256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:04 smithi081 ceph-mon[31453]: pgmap v19256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:07.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:06 smithi153 ceph-mon[36569]: pgmap v19257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:07.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:06 smithi104 ceph-mon[36624]: pgmap v19257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:07.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:06 smithi081 ceph-mon[31453]: pgmap v19257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:09.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:08 smithi153 ceph-mon[36569]: pgmap v19258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:09.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:08 smithi104 ceph-mon[36624]: pgmap v19258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:09.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:08 smithi081 ceph-mon[31453]: pgmap v19258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:11.226 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:10 smithi081 ceph-mon[31453]: pgmap v19259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:11.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:10 smithi153 ceph-mon[36569]: pgmap v19259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:11.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:10 smithi104 ceph-mon[36624]: pgmap v19259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:13.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:12 smithi081 ceph-mon[31453]: pgmap v19260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:12 smithi153 ceph-mon[36569]: pgmap v19260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:13.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:12 smithi104 ceph-mon[36624]: pgmap v19260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:15.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:14 smithi153 ceph-mon[36569]: pgmap v19261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:15.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:14 smithi104 ceph-mon[36624]: pgmap v19261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:15.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:14 smithi081 ceph-mon[31453]: pgmap v19261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:17.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:16 smithi153 ceph-mon[36569]: pgmap v19262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:17.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:16 smithi104 ceph-mon[36624]: pgmap v19262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:17.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:16 smithi081 ceph-mon[31453]: pgmap v19262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:19.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:18 smithi153 ceph-mon[36569]: pgmap v19263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:19.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:18 smithi104 ceph-mon[36624]: pgmap v19263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:19.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:18 smithi081 ceph-mon[31453]: pgmap v19263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:20.316+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:18:20.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:20.316+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:18:20.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:20.316+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:18:21.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:20 smithi153 ceph-mon[36569]: pgmap v19264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:21.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:20 smithi104 ceph-mon[36624]: pgmap v19264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:21.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:20 smithi081 ceph-mon[31453]: pgmap v19264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:23.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:23 smithi153 ceph-mon[36569]: pgmap v19265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:23.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:23 smithi104 ceph-mon[36624]: pgmap v19265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:23.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:23 smithi081 ceph-mon[31453]: pgmap v19265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:25.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:25 smithi153 ceph-mon[36569]: pgmap v19266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:25.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:25 smithi104 ceph-mon[36624]: pgmap v19266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:25.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:25 smithi081 ceph-mon[31453]: pgmap v19266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:27.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:27 smithi153 ceph-mon[36569]: pgmap v19267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:27 smithi104 ceph-mon[36624]: pgmap v19267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:27.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:27 smithi081 ceph-mon[31453]: pgmap v19267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:29.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:29 smithi153 ceph-mon[36569]: pgmap v19268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:29 smithi104 ceph-mon[36624]: pgmap v19268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:29.409 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:29 smithi081 ceph-mon[31453]: pgmap v19268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:31.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:31 smithi153 ceph-mon[36569]: pgmap v19269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:31.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:31 smithi104 ceph-mon[36624]: pgmap v19269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:31.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:31 smithi081 ceph-mon[31453]: pgmap v19269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:33.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:33 smithi153 ceph-mon[36569]: pgmap v19270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:33.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:33 smithi104 ceph-mon[36624]: pgmap v19270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:33.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:33 smithi081 ceph-mon[31453]: pgmap v19270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:33.684+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:18:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:33.684+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:18:33.688 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:33.685+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 278 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:18:33.688 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:33.685+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 278 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:18:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:34 smithi081 ceph-mon[31453]: pgmap v19271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:34.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:34 smithi153 ceph-mon[36569]: pgmap v19271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:34.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:34 smithi104 ceph-mon[36624]: pgmap v19271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:35.331+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:18:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:35.331+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:18:35.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:35.331+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:18:36.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:36 smithi153 ceph-mon[36569]: pgmap v19272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:36.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:36 smithi104 ceph-mon[36624]: pgmap v19272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:36.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:36 smithi081 ceph-mon[31453]: pgmap v19272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:38.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:38 smithi153 ceph-mon[36569]: pgmap v19273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:38.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:38 smithi104 ceph-mon[36624]: pgmap v19273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:38.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:38 smithi081 ceph-mon[31453]: pgmap v19273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:40.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:40 smithi153 ceph-mon[36569]: pgmap v19274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:40.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:40 smithi104 ceph-mon[36624]: pgmap v19274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:40.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:40 smithi081 ceph-mon[31453]: pgmap v19274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:42.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:42 smithi153 ceph-mon[36569]: pgmap v19275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:42.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:42 smithi104 ceph-mon[36624]: pgmap v19275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:42.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:42 smithi081 ceph-mon[31453]: pgmap v19275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:44 smithi081 ceph-mon[31453]: pgmap v19276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:44.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:44 smithi153 ceph-mon[36569]: pgmap v19276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:44.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:44 smithi104 ceph-mon[36624]: pgmap v19276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:46.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:46 smithi104 ceph-mon[36624]: pgmap v19277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:46 smithi081 ceph-mon[31453]: pgmap v19277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:46.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:46 smithi153 ceph-mon[36569]: pgmap v19277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:48.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:48 smithi104 ceph-mon[36624]: pgmap v19278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:48.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:18:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:48 smithi081 ceph-mon[31453]: pgmap v19278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:18:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:48 smithi153 ceph-mon[36569]: pgmap v19278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:48.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:18:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:18:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:18:49.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:49 smithi104 ceph-mon[36624]: pgmap v19279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:49.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:49 smithi104 ceph-mon[36624]: pgmap v19280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:49.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:18:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:18:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:18:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:49 smithi081 ceph-mon[31453]: pgmap v19279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:49 smithi081 ceph-mon[31453]: pgmap v19280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:49.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:18:49.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:18:49.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:18:49.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:49 smithi153 ceph-mon[36569]: pgmap v19279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:49.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:49 smithi153 ceph-mon[36569]: pgmap v19280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:49.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:18:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:50.336+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:18:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:50.336+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:18:50.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:18:50.336+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:18:52.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:51 smithi153 ceph-mon[36569]: pgmap v19281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:52.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:51 smithi104 ceph-mon[36624]: pgmap v19281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:52.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:51 smithi081 ceph-mon[31453]: pgmap v19281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:54.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:53 smithi104 ceph-mon[36624]: pgmap v19282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:53 smithi081 ceph-mon[31453]: pgmap v19282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:54.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:53 smithi153 ceph-mon[36569]: pgmap v19282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:56.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:55 smithi104 ceph-mon[36624]: pgmap v19283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:56.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:55 smithi081 ceph-mon[31453]: pgmap v19283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:55 smithi153 ceph-mon[36569]: pgmap v19283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:58.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:57 smithi104 ceph-mon[36624]: pgmap v19284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:57 smithi081 ceph-mon[31453]: pgmap v19284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:18:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:57 smithi153 ceph-mon[36569]: pgmap v19284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:00.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:18:59 smithi104 ceph-mon[36624]: pgmap v19285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:00.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:18:59 smithi081 ceph-mon[31453]: pgmap v19285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:00.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:18:59 smithi153 ceph-mon[36569]: pgmap v19285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:02.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:01 smithi104 ceph-mon[36624]: pgmap v19286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:02.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:01 smithi081 ceph-mon[31453]: pgmap v19286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:01 smithi153 ceph-mon[36569]: pgmap v19286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:04.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:03 smithi081 ceph-mon[31453]: pgmap v19287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:03 smithi153 ceph-mon[36569]: pgmap v19287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:04.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:03 smithi104 ceph-mon[36624]: pgmap v19287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:05.354 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:05.352+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:19:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:05.352+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:19:05.355 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:05.352+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:19:06.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:05 smithi081 ceph-mon[31453]: pgmap v19288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:05 smithi153 ceph-mon[36569]: pgmap v19288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:05 smithi104 ceph-mon[36624]: pgmap v19288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:08.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:07 smithi081 ceph-mon[31453]: pgmap v19289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:07 smithi153 ceph-mon[36569]: pgmap v19289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:08.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:07 smithi104 ceph-mon[36624]: pgmap v19289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:09 smithi081 ceph-mon[31453]: pgmap v19290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:09 smithi153 ceph-mon[36569]: pgmap v19290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:10.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:09 smithi104 ceph-mon[36624]: pgmap v19290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:12.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:11 smithi081 ceph-mon[31453]: pgmap v19291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:12.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:11 smithi153 ceph-mon[36569]: pgmap v19291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:11 smithi104 ceph-mon[36624]: pgmap v19291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:14.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:13 smithi081 ceph-mon[31453]: pgmap v19292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:14.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:13 smithi153 ceph-mon[36569]: pgmap v19292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:14.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:13 smithi104 ceph-mon[36624]: pgmap v19292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:16.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:15 smithi081 ceph-mon[31453]: pgmap v19293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:16.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:15 smithi153 ceph-mon[36569]: pgmap v19293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:16.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:15 smithi104 ceph-mon[36624]: pgmap v19293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:18.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:17 smithi081 ceph-mon[31453]: pgmap v19294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:18.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:17 smithi153 ceph-mon[36569]: pgmap v19294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:18.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:17 smithi104 ceph-mon[36624]: pgmap v19294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:18.693+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:19:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:18.693+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:19:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:18.693+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 279 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:19:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:18.693+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 279 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:19:20.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:19 smithi081 ceph-mon[31453]: pgmap v19295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:19 smithi153 ceph-mon[36569]: pgmap v19295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:20.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:19 smithi104 ceph-mon[36624]: pgmap v19295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:20.368+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:19:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:20.368+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:19:20.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:20.368+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:19:22.137 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:21 smithi104 ceph-mon[36624]: pgmap v19296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:22.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:21 smithi081 ceph-mon[31453]: pgmap v19296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:22.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:21 smithi153 ceph-mon[36569]: pgmap v19296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:24.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:23 smithi081 ceph-mon[31453]: pgmap v19297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:24.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:23 smithi153 ceph-mon[36569]: pgmap v19297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:24.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:23 smithi104 ceph-mon[36624]: pgmap v19297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:26.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:25 smithi081 ceph-mon[31453]: pgmap v19298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:25 smithi153 ceph-mon[36569]: pgmap v19298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:26.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:25 smithi104 ceph-mon[36624]: pgmap v19298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:28.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:27 smithi081 ceph-mon[31453]: pgmap v19299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:27 smithi153 ceph-mon[36569]: pgmap v19299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:28.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:27 smithi104 ceph-mon[36624]: pgmap v19299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:30.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:29 smithi081 ceph-mon[31453]: pgmap v19300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:30.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:29 smithi153 ceph-mon[36569]: pgmap v19300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:30.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:29 smithi104 ceph-mon[36624]: pgmap v19300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:31 smithi153 ceph-mon[36569]: pgmap v19301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:32.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:31 smithi104 ceph-mon[36624]: pgmap v19301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:32.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:31 smithi081 ceph-mon[31453]: pgmap v19301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:34.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:33 smithi153 ceph-mon[36569]: pgmap v19302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:34.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:33 smithi104 ceph-mon[36624]: pgmap v19302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:34.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:33 smithi081 ceph-mon[31453]: pgmap v19302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:35.371+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:19:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:35.371+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:19:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:35.371+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:19:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:35 smithi153 ceph-mon[36569]: pgmap v19303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:36.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:35 smithi104 ceph-mon[36624]: pgmap v19303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:36.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:35 smithi081 ceph-mon[31453]: pgmap v19303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:37 smithi153 ceph-mon[36569]: pgmap v19304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:38.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:37 smithi104 ceph-mon[36624]: pgmap v19304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:38.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:37 smithi081 ceph-mon[31453]: pgmap v19304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:40.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:39 smithi153 ceph-mon[36569]: pgmap v19305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:40.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:39 smithi104 ceph-mon[36624]: pgmap v19305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:40.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:39 smithi081 ceph-mon[31453]: pgmap v19305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:41 smithi153 ceph-mon[36569]: pgmap v19306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:42.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:41 smithi104 ceph-mon[36624]: pgmap v19306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:42.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:41 smithi081 ceph-mon[31453]: pgmap v19306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:44.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:43 smithi153 ceph-mon[36569]: pgmap v19307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:44.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:43 smithi104 ceph-mon[36624]: pgmap v19307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:44.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:43 smithi081 ceph-mon[31453]: pgmap v19307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:46.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:45 smithi081 ceph-mon[31453]: pgmap v19308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:46.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:45 smithi153 ceph-mon[36569]: pgmap v19308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:46.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:45 smithi104 ceph-mon[36624]: pgmap v19308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:47 smithi153 ceph-mon[36569]: pgmap v19309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:48.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:47 smithi104 ceph-mon[36624]: pgmap v19309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:48.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:47 smithi081 ceph-mon[31453]: pgmap v19309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:49 smithi153 ceph-mon[36569]: pgmap v19310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:19:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:49 smithi104 ceph-mon[36624]: pgmap v19310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:19:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.388 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:50.386+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:19:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:50.387+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:19:50.389 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:19:50.387+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:19:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:49 smithi081 ceph-mon[31453]: pgmap v19310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:19:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:50.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:51 smithi153 ceph-mon[36569]: pgmap v19311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:52.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:52.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:51 smithi104 ceph-mon[36624]: pgmap v19311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:52.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:51 smithi081 ceph-mon[31453]: pgmap v19311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:52.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:53.309 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:53 smithi153 ceph-mon[36569]: pgmap v19312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:53.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:53 smithi104 ceph-mon[36624]: pgmap v19312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:53.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:53 smithi081 ceph-mon[31453]: pgmap v19312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:55.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:55 smithi153 ceph-mon[36569]: pgmap v19313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:55.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:55.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:55.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:55 smithi104 ceph-mon[36624]: pgmap v19313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:55.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:55 smithi081 ceph-mon[31453]: pgmap v19313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:56.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:19:56.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:19:56.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:56 smithi153 ceph-mon[36569]: pgmap v19314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:56.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:56 smithi153 ceph-mon[36569]: pgmap v19315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:56.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:56.584 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:19:56.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:19:56.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:56 smithi104 ceph-mon[36624]: pgmap v19314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:56.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:56 smithi104 ceph-mon[36624]: pgmap v19315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:56.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:56.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:19:56.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:19:56.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:56 smithi081 ceph-mon[31453]: pgmap v19314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:56.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:56 smithi081 ceph-mon[31453]: pgmap v19315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:56.659 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:19:58.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:19:58 smithi081 ceph-mon[31453]: pgmap v19316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:59.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:19:58 smithi153 ceph-mon[36569]: pgmap v19316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:19:59.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:19:58 smithi104 ceph-mon[36624]: pgmap v19316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:00 smithi081 ceph-mon[31453]: pgmap v19317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:00.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:20:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:00 smithi153 ceph-mon[36569]: pgmap v19317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:01.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:20:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:00 smithi104 ceph-mon[36624]: pgmap v19317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:01.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:20:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:02 smithi081 ceph-mon[31453]: pgmap v19318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:03.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:02 smithi153 ceph-mon[36569]: pgmap v19318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:03.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:02 smithi104 ceph-mon[36624]: pgmap v19318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:04.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:04 smithi081 ceph-mon[31453]: pgmap v19319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:05.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:04 smithi153 ceph-mon[36569]: pgmap v19319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:05.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:04 smithi104 ceph-mon[36624]: pgmap v19319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:05.402+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:20:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:05.402+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:20:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:05.402+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:20:06.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:06 smithi081 ceph-mon[31453]: pgmap v19320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:07.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:06 smithi153 ceph-mon[36569]: pgmap v19320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:07.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:06 smithi104 ceph-mon[36624]: pgmap v19320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:08.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:08 smithi081 ceph-mon[31453]: pgmap v19321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:08 smithi153 ceph-mon[36569]: pgmap v19321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:09.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:08 smithi104 ceph-mon[36624]: pgmap v19321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:11.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:10 smithi153 ceph-mon[36569]: pgmap v19322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:11.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:10 smithi104 ceph-mon[36624]: pgmap v19322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:11.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:10 smithi081 ceph-mon[31453]: pgmap v19322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:13.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:12 smithi153 ceph-mon[36569]: pgmap v19323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:13.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:12 smithi104 ceph-mon[36624]: pgmap v19323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:13.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:12 smithi081 ceph-mon[31453]: pgmap v19323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:15.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:14 smithi153 ceph-mon[36569]: pgmap v19324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:15.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:14 smithi104 ceph-mon[36624]: pgmap v19324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:15.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:14 smithi081 ceph-mon[31453]: pgmap v19324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:16.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:16 smithi081 ceph-mon[31453]: pgmap v19325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:17.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:16 smithi153 ceph-mon[36569]: pgmap v19325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:17.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:16 smithi104 ceph-mon[36624]: pgmap v19325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:19.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:18 smithi153 ceph-mon[36569]: pgmap v19326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:19.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:18 smithi104 ceph-mon[36624]: pgmap v19326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:18 smithi081 ceph-mon[31453]: pgmap v19326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:20.415+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:20:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:20.415+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:20:20.418 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:20.415+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:20:21.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:20 smithi153 ceph-mon[36569]: pgmap v19327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:21.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:20 smithi104 ceph-mon[36624]: pgmap v19327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:21.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:20 smithi081 ceph-mon[31453]: pgmap v19327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:23.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:22 smithi153 ceph-mon[36569]: pgmap v19328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:23.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:22 smithi104 ceph-mon[36624]: pgmap v19328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:23.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:22 smithi081 ceph-mon[31453]: pgmap v19328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:25.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:24 smithi153 ceph-mon[36569]: pgmap v19329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:25.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:24 smithi104 ceph-mon[36624]: pgmap v19329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:25.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:24 smithi081 ceph-mon[31453]: pgmap v19329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:27.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:26 smithi153 ceph-mon[36569]: pgmap v19330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:27.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:26 smithi104 ceph-mon[36624]: pgmap v19330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:27.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:26 smithi081 ceph-mon[31453]: pgmap v19330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:29.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:28 smithi104 ceph-mon[36624]: pgmap v19331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:29.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:28 smithi081 ceph-mon[31453]: pgmap v19331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:28 smithi153 ceph-mon[36569]: pgmap v19331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:31.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:30 smithi104 ceph-mon[36624]: pgmap v19332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:30 smithi081 ceph-mon[31453]: pgmap v19332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:30 smithi153 ceph-mon[36569]: pgmap v19332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:33.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:32 smithi104 ceph-mon[36624]: pgmap v19333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:33.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:32 smithi081 ceph-mon[31453]: pgmap v19333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:32 smithi153 ceph-mon[36569]: pgmap v19333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:35.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:34 smithi104 ceph-mon[36624]: pgmap v19334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:35.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:34 smithi081 ceph-mon[31453]: pgmap v19334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:34 smithi153 ceph-mon[36569]: pgmap v19334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:35.433 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:35.431+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:20:35.433 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:35.431+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:20:35.433 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:35.431+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:20:37.084 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:36 smithi104 ceph-mon[36624]: pgmap v19335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:37.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:36 smithi081 ceph-mon[31453]: pgmap v19335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:36 smithi153 ceph-mon[36569]: pgmap v19335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:39.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:38 smithi081 ceph-mon[31453]: pgmap v19336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:38 smithi153 ceph-mon[36569]: pgmap v19336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:38 smithi104 ceph-mon[36624]: pgmap v19336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:41.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:40 smithi081 ceph-mon[31453]: pgmap v19337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:40 smithi153 ceph-mon[36569]: pgmap v19337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:41.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:40 smithi104 ceph-mon[36624]: pgmap v19337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:42 smithi081 ceph-mon[31453]: pgmap v19338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:42 smithi153 ceph-mon[36569]: pgmap v19338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:43.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:42 smithi104 ceph-mon[36624]: pgmap v19338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:45.111 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:44 smithi153 ceph-mon[36569]: pgmap v19339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:45.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:44 smithi081 ceph-mon[31453]: pgmap v19339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:44 smithi104 ceph-mon[36624]: pgmap v19339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:47.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:46 smithi081 ceph-mon[31453]: pgmap v19340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:46 smithi153 ceph-mon[36569]: pgmap v19340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:46 smithi104 ceph-mon[36624]: pgmap v19340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:49.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:48 smithi081 ceph-mon[31453]: pgmap v19341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:48 smithi153 ceph-mon[36569]: pgmap v19341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:48 smithi104 ceph-mon[36624]: pgmap v19341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:50.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:50.444+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:20:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:50.444+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:20:50.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:20:50.444+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:20:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:50 smithi081 ceph-mon[31453]: pgmap v19342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:50 smithi153 ceph-mon[36569]: pgmap v19342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:50 smithi104 ceph-mon[36624]: pgmap v19342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:52 smithi081 ceph-mon[31453]: pgmap v19343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:52 smithi153 ceph-mon[36569]: pgmap v19343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:52 smithi104 ceph-mon[36624]: pgmap v19343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:55.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:54 smithi104 ceph-mon[36624]: pgmap v19344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:54 smithi081 ceph-mon[31453]: pgmap v19344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:54 smithi153 ceph-mon[36569]: pgmap v19344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:56.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:20:56.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:20:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:20:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:56 smithi081 ceph-mon[31453]: pgmap v19345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:57.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:56 smithi153 ceph-mon[36569]: pgmap v19345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:56 smithi104 ceph-mon[36624]: pgmap v19345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:57.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:20:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:20:58.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:20:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:20:58.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:20:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:20:58.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:20:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:58 smithi081 ceph-mon[31453]: pgmap v19346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:59.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:20:58 smithi081 ceph-mon[31453]: pgmap v19347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:58 smithi153 ceph-mon[36569]: pgmap v19346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:59.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:20:58 smithi153 ceph-mon[36569]: pgmap v19347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:58 smithi104 ceph-mon[36624]: pgmap v19346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:20:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:20:58 smithi104 ceph-mon[36624]: pgmap v19347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:00 smithi153 ceph-mon[36569]: pgmap v19348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:00 smithi104 ceph-mon[36624]: pgmap v19348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:01.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:00 smithi081 ceph-mon[31453]: pgmap v19348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:03.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:02 smithi153 ceph-mon[36569]: pgmap v19349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:02 smithi104 ceph-mon[36624]: pgmap v19349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:02 smithi081 ceph-mon[31453]: pgmap v19349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:05.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:04 smithi153 ceph-mon[36569]: pgmap v19350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:05.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:04 smithi104 ceph-mon[36624]: pgmap v19350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:05.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:04 smithi081 ceph-mon[31453]: pgmap v19350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:05.450+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:21:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:05.450+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:21:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:05.450+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:21:07.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:06 smithi153 ceph-mon[36569]: pgmap v19351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:06 smithi104 ceph-mon[36624]: pgmap v19351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:06 smithi081 ceph-mon[31453]: pgmap v19351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:09.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:08 smithi153 ceph-mon[36569]: pgmap v19352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:08 smithi104 ceph-mon[36624]: pgmap v19352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:08 smithi081 ceph-mon[31453]: pgmap v19352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:11.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:10 smithi153 ceph-mon[36569]: pgmap v19353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:10 smithi104 ceph-mon[36624]: pgmap v19353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:10 smithi081 ceph-mon[31453]: pgmap v19353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:13.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:12 smithi153 ceph-mon[36569]: pgmap v19354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:12 smithi104 ceph-mon[36624]: pgmap v19354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:12 smithi081 ceph-mon[31453]: pgmap v19354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:15.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:14 smithi153 ceph-mon[36569]: pgmap v19355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:14 smithi104 ceph-mon[36624]: pgmap v19355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:14 smithi081 ceph-mon[31453]: pgmap v19355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:17.211 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:16 smithi081 ceph-mon[31453]: pgmap v19356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:17.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:16 smithi153 ceph-mon[36569]: pgmap v19356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:16 smithi104 ceph-mon[36624]: pgmap v19356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:19.241 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:18 smithi081 ceph-mon[31453]: pgmap v19357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:19.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:18 smithi153 ceph-mon[36569]: pgmap v19357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:18 smithi104 ceph-mon[36624]: pgmap v19357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:20.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:20.465+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:21:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:20.465+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:21:20.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:20.465+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:21:21.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:20 smithi153 ceph-mon[36569]: pgmap v19358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:20 smithi104 ceph-mon[36624]: pgmap v19358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:20 smithi081 ceph-mon[31453]: pgmap v19358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:23.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:22 smithi153 ceph-mon[36569]: pgmap v19359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:22 smithi104 ceph-mon[36624]: pgmap v19359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:22 smithi081 ceph-mon[31453]: pgmap v19359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:25.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:24 smithi153 ceph-mon[36569]: pgmap v19360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:24 smithi104 ceph-mon[36624]: pgmap v19360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:24 smithi081 ceph-mon[31453]: pgmap v19360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:27.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:26 smithi153 ceph-mon[36569]: pgmap v19361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:27.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:26 smithi104 ceph-mon[36624]: pgmap v19361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:26 smithi081 ceph-mon[31453]: pgmap v19361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:29.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:28 smithi153 ceph-mon[36569]: pgmap v19362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:29.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:28 smithi104 ceph-mon[36624]: pgmap v19362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:29.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:28 smithi081 ceph-mon[31453]: pgmap v19362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:31.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:30 smithi153 ceph-mon[36569]: pgmap v19363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:30 smithi104 ceph-mon[36624]: pgmap v19363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:30 smithi081 ceph-mon[31453]: pgmap v19363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:33.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:32 smithi153 ceph-mon[36569]: pgmap v19364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:32 smithi104 ceph-mon[36624]: pgmap v19364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:32 smithi081 ceph-mon[31453]: pgmap v19364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:35.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:34 smithi153 ceph-mon[36569]: pgmap v19365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:34 smithi104 ceph-mon[36624]: pgmap v19365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:34 smithi081 ceph-mon[31453]: pgmap v19365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:35.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:35.480+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:21:35.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:35.480+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:21:35.482 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:35.480+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:21:37.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:37 smithi153 ceph-mon[36569]: pgmap v19366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:37 smithi104 ceph-mon[36624]: pgmap v19366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:37.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:36 smithi081 ceph-mon[31453]: pgmap v19366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:39.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:39 smithi153 ceph-mon[36569]: pgmap v19367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:39.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:39 smithi104 ceph-mon[36624]: pgmap v19367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:39 smithi081 ceph-mon[31453]: pgmap v19367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:41.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:41 smithi153 ceph-mon[36569]: pgmap v19368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:41.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:41 smithi104 ceph-mon[36624]: pgmap v19368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:41 smithi081 ceph-mon[31453]: pgmap v19368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:43.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:43 smithi153 ceph-mon[36569]: pgmap v19369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:43 smithi104 ceph-mon[36624]: pgmap v19369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:43 smithi081 ceph-mon[31453]: pgmap v19369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:45.313 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:45 smithi153 ceph-mon[36569]: pgmap v19370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:45.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:45 smithi104 ceph-mon[36624]: pgmap v19370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:45.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:45 smithi081 ceph-mon[31453]: pgmap v19370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:47.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:47 smithi153 ceph-mon[36569]: pgmap v19371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:47.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:47 smithi104 ceph-mon[36624]: pgmap v19371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:47.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:47 smithi081 ceph-mon[31453]: pgmap v19371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:49.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:49 smithi153 ceph-mon[36569]: pgmap v19372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:49.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:49 smithi104 ceph-mon[36624]: pgmap v19372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:49 smithi081 ceph-mon[31453]: pgmap v19372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:50.487 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:50.486+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:21:50.488 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:50.486+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:21:50.488 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:21:50.486+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:21:51.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:51 smithi153 ceph-mon[36569]: pgmap v19373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:51 smithi104 ceph-mon[36624]: pgmap v19373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:51.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:51 smithi081 ceph-mon[31453]: pgmap v19373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:53.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:53 smithi153 ceph-mon[36569]: pgmap v19374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:53.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:53 smithi104 ceph-mon[36624]: pgmap v19374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:53.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:53 smithi081 ceph-mon[31453]: pgmap v19374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:55.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:55 smithi153 ceph-mon[36569]: pgmap v19375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:55 smithi104 ceph-mon[36624]: pgmap v19375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:55 smithi081 ceph-mon[31453]: pgmap v19375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:57.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:57 smithi153 ceph-mon[36569]: pgmap v19376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:57 smithi104 ceph-mon[36624]: pgmap v19376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:57.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:57 smithi081 ceph-mon[31453]: pgmap v19376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:21:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:21:58.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:21:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:59 smithi104 ceph-mon[36624]: pgmap v19377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:21:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:21:59.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:21:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:21:59.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:59 smithi081 ceph-mon[31453]: pgmap v19377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:21:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:21:59.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:21:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:21:59.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:59 smithi153 ceph-mon[36569]: pgmap v19377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:21:59.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:21:59.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:21:59.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:21:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:22:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:00 smithi104 ceph-mon[36624]: pgmap v19378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:00 smithi104 ceph-mon[36624]: pgmap v19379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:00 smithi081 ceph-mon[31453]: pgmap v19378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:00 smithi081 ceph-mon[31453]: pgmap v19379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:00 smithi153 ceph-mon[36569]: pgmap v19378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:00.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:00 smithi153 ceph-mon[36569]: pgmap v19379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:01 smithi081 ceph-mon[31453]: pgmap v19380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:01.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:01 smithi153 ceph-mon[36569]: pgmap v19380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:01.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:01 smithi104 ceph-mon[36624]: pgmap v19380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:04.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:03 smithi104 ceph-mon[36624]: pgmap v19381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:04.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:03 smithi081 ceph-mon[31453]: pgmap v19381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:04.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:03 smithi153 ceph-mon[36569]: pgmap v19381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:05.491+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:22:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:05.491+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:22:05.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:05.491+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:22:06.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:05 smithi104 ceph-mon[36624]: pgmap v19382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:06.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:05 smithi081 ceph-mon[31453]: pgmap v19382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:05 smithi153 ceph-mon[36569]: pgmap v19382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:08.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:07 smithi104 ceph-mon[36624]: pgmap v19383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:08.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:07 smithi081 ceph-mon[31453]: pgmap v19383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:08.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:07 smithi153 ceph-mon[36569]: pgmap v19383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:10.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:09 smithi104 ceph-mon[36624]: pgmap v19384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:10.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:09 smithi081 ceph-mon[31453]: pgmap v19384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:09 smithi153 ceph-mon[36569]: pgmap v19384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:11 smithi104 ceph-mon[36624]: pgmap v19385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:12.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:11 smithi081 ceph-mon[31453]: pgmap v19385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:11 smithi153 ceph-mon[36569]: pgmap v19385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:14.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:13 smithi081 ceph-mon[31453]: pgmap v19386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:13 smithi153 ceph-mon[36569]: pgmap v19386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:13 smithi104 ceph-mon[36624]: pgmap v19386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:16.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:15 smithi153 ceph-mon[36569]: pgmap v19387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:16.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:15 smithi081 ceph-mon[31453]: pgmap v19387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:15 smithi104 ceph-mon[36624]: pgmap v19387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:17 smithi081 ceph-mon[31453]: pgmap v19388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:17 smithi153 ceph-mon[36569]: pgmap v19388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:17 smithi104 ceph-mon[36624]: pgmap v19388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:18.722+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:22:18.724 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:18.722+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:22:18.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:18.723+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 280 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:22:18.725 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:18.723+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 280 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:22:20.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:19 smithi081 ceph-mon[31453]: pgmap v19389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:20.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:19 smithi153 ceph-mon[36569]: pgmap v19389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:19 smithi104 ceph-mon[36624]: pgmap v19389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:20.504+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:22:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:20.504+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:22:20.506 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:20.504+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:22:22.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:21 smithi081 ceph-mon[31453]: pgmap v19390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:21 smithi153 ceph-mon[36569]: pgmap v19390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:21 smithi104 ceph-mon[36624]: pgmap v19390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:24.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:23 smithi081 ceph-mon[31453]: pgmap v19391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:23 smithi153 ceph-mon[36569]: pgmap v19391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:23 smithi104 ceph-mon[36624]: pgmap v19391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:26.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:25 smithi104 ceph-mon[36624]: pgmap v19392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:25 smithi081 ceph-mon[31453]: pgmap v19392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:26.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:25 smithi153 ceph-mon[36569]: pgmap v19392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:28.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:27 smithi081 ceph-mon[31453]: pgmap v19393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:27 smithi153 ceph-mon[36569]: pgmap v19393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:27 smithi104 ceph-mon[36624]: pgmap v19393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:30.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:29 smithi081 ceph-mon[31453]: pgmap v19394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:29 smithi153 ceph-mon[36569]: pgmap v19394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:29 smithi104 ceph-mon[36624]: pgmap v19394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:31 smithi081 ceph-mon[31453]: pgmap v19395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:32.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:31 smithi153 ceph-mon[36569]: pgmap v19395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:31 smithi104 ceph-mon[36624]: pgmap v19395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:34.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:33 smithi081 ceph-mon[31453]: pgmap v19396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:33 smithi153 ceph-mon[36569]: pgmap v19396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:33 smithi104 ceph-mon[36624]: pgmap v19396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:35.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:35.517+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:22:35.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:35.517+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:22:35.518 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:35.517+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:22:36.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:35 smithi081 ceph-mon[31453]: pgmap v19397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:36.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:35 smithi153 ceph-mon[36569]: pgmap v19397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:35 smithi104 ceph-mon[36624]: pgmap v19397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:38.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:37 smithi153 ceph-mon[36569]: pgmap v19398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:37 smithi104 ceph-mon[36624]: pgmap v19398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:37 smithi081 ceph-mon[31453]: pgmap v19398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:39 smithi153 ceph-mon[36569]: pgmap v19399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:39 smithi104 ceph-mon[36624]: pgmap v19399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:39 smithi081 ceph-mon[31453]: pgmap v19399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:42.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:41 smithi153 ceph-mon[36569]: pgmap v19400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:41 smithi104 ceph-mon[36624]: pgmap v19400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:42.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:41 smithi081 ceph-mon[31453]: pgmap v19400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:43 smithi153 ceph-mon[36569]: pgmap v19401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:44.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:43 smithi104 ceph-mon[36624]: pgmap v19401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:43 smithi081 ceph-mon[31453]: pgmap v19401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:45 smithi153 ceph-mon[36569]: pgmap v19402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:46.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:45 smithi104 ceph-mon[36624]: pgmap v19402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:45 smithi081 ceph-mon[31453]: pgmap v19402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:48.210 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:47 smithi081 ceph-mon[31453]: pgmap v19403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:48.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:47 smithi153 ceph-mon[36569]: pgmap v19403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:48.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:47 smithi104 ceph-mon[36624]: pgmap v19403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:50.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:49 smithi153 ceph-mon[36569]: pgmap v19404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:50.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:49 smithi104 ceph-mon[36624]: pgmap v19404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:50.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:49 smithi081 ceph-mon[31453]: pgmap v19404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:50.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:50.528+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:22:50.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:50.528+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:22:50.530 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:22:50.528+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:22:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:51 smithi153 ceph-mon[36569]: pgmap v19405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:51 smithi104 ceph-mon[36624]: pgmap v19405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:52.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:51 smithi081 ceph-mon[31453]: pgmap v19405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:53 smithi153 ceph-mon[36569]: pgmap v19406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:54.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:53 smithi104 ceph-mon[36624]: pgmap v19406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:54.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:53 smithi081 ceph-mon[31453]: pgmap v19406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:55 smithi153 ceph-mon[36569]: pgmap v19407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:56.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:55 smithi104 ceph-mon[36624]: pgmap v19407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:56.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:55 smithi081 ceph-mon[31453]: pgmap v19407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:57 smithi153 ceph-mon[36569]: pgmap v19408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:58.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:57 smithi104 ceph-mon[36624]: pgmap v19408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:22:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:57 smithi081 ceph-mon[31453]: pgmap v19408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:00.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:59 smithi153 ceph-mon[36569]: pgmap v19409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:00.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:22:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:23:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:59 smithi104 ceph-mon[36624]: pgmap v19409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:00.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:22:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:23:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:59 smithi081 ceph-mon[31453]: pgmap v19409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:00.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:22:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:23:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:23:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:23:01.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:23:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:23:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:23:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:23:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:23:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:23:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:23:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:01 smithi153 ceph-mon[36569]: pgmap v19410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:02.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:01 smithi153 ceph-mon[36569]: pgmap v19411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:01 smithi104 ceph-mon[36624]: pgmap v19410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:02.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:01 smithi104 ceph-mon[36624]: pgmap v19411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:01 smithi081 ceph-mon[31453]: pgmap v19410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:02.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:01 smithi081 ceph-mon[31453]: pgmap v19411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:03 smithi153 ceph-mon[36569]: pgmap v19412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:03 smithi104 ceph-mon[36624]: pgmap v19412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:03 smithi081 ceph-mon[31453]: pgmap v19412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:05.536+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:23:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:05.536+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:23:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:05.536+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:23:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:05 smithi153 ceph-mon[36569]: pgmap v19413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:05 smithi104 ceph-mon[36624]: pgmap v19413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:05 smithi081 ceph-mon[31453]: pgmap v19413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:07 smithi153 ceph-mon[36569]: pgmap v19414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:08.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:07 smithi104 ceph-mon[36624]: pgmap v19414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:08.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:07 smithi081 ceph-mon[31453]: pgmap v19414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:10.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:09 smithi153 ceph-mon[36569]: pgmap v19415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:10.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:09 smithi104 ceph-mon[36624]: pgmap v19415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:10.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:09 smithi081 ceph-mon[31453]: pgmap v19415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:12.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:12 smithi153 ceph-mon[36569]: pgmap v19416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:12.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:12 smithi104 ceph-mon[36624]: pgmap v19416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:12.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:12 smithi081 ceph-mon[31453]: pgmap v19416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:14.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:14 smithi153 ceph-mon[36569]: pgmap v19417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:14.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:14 smithi104 ceph-mon[36624]: pgmap v19417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:14 smithi081 ceph-mon[31453]: pgmap v19417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:16.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:16 smithi153 ceph-mon[36569]: pgmap v19418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:16.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:16 smithi104 ceph-mon[36624]: pgmap v19418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:16.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:16 smithi081 ceph-mon[31453]: pgmap v19418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:18.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:18 smithi153 ceph-mon[36569]: pgmap v19419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:18.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:18 smithi104 ceph-mon[36624]: pgmap v19419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:18.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:18 smithi081 ceph-mon[31453]: pgmap v19419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:20 smithi153 ceph-mon[36569]: pgmap v19420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:20.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:20 smithi104 ceph-mon[36624]: pgmap v19420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:20.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:20 smithi081 ceph-mon[31453]: pgmap v19420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:20.552+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:23:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:20.552+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:23:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:20.552+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:23:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:22 smithi153 ceph-mon[36569]: pgmap v19421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:22.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:22 smithi104 ceph-mon[36624]: pgmap v19421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:22.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:22 smithi081 ceph-mon[31453]: pgmap v19421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:24 smithi153 ceph-mon[36569]: pgmap v19422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:24.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:24 smithi104 ceph-mon[36624]: pgmap v19422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:24.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:24 smithi081 ceph-mon[31453]: pgmap v19422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:26 smithi153 ceph-mon[36569]: pgmap v19423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:26.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:26 smithi104 ceph-mon[36624]: pgmap v19423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:26.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:26 smithi081 ceph-mon[31453]: pgmap v19423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:28.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:28 smithi153 ceph-mon[36569]: pgmap v19424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:28.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:28 smithi104 ceph-mon[36624]: pgmap v19424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:28 smithi081 ceph-mon[31453]: pgmap v19424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:30 smithi153 ceph-mon[36569]: pgmap v19425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:30 smithi104 ceph-mon[36624]: pgmap v19425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:30.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:30 smithi081 ceph-mon[31453]: pgmap v19425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:32 smithi153 ceph-mon[36569]: pgmap v19426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:32.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:32 smithi104 ceph-mon[36624]: pgmap v19426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:32.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:32 smithi081 ceph-mon[31453]: pgmap v19426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:34.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:34 smithi104 ceph-mon[36624]: pgmap v19427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:34.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:34 smithi081 ceph-mon[31453]: pgmap v19427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:34 smithi153 ceph-mon[36569]: pgmap v19427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:35.563+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:23:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:35.563+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:23:35.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:35.563+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:23:36.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:36 smithi104 ceph-mon[36624]: pgmap v19428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:36.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:36 smithi081 ceph-mon[31453]: pgmap v19428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:36.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:36 smithi153 ceph-mon[36569]: pgmap v19428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:38 smithi104 ceph-mon[36624]: pgmap v19429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:38 smithi081 ceph-mon[31453]: pgmap v19429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:38 smithi153 ceph-mon[36569]: pgmap v19429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:40.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:40 smithi104 ceph-mon[36624]: pgmap v19430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:40 smithi081 ceph-mon[31453]: pgmap v19430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:40 smithi153 ceph-mon[36569]: pgmap v19430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:42.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:42 smithi104 ceph-mon[36624]: pgmap v19431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:42.410 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:42 smithi081 ceph-mon[31453]: pgmap v19431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:42 smithi153 ceph-mon[36569]: pgmap v19431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:44.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:44 smithi081 ceph-mon[31453]: pgmap v19432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:44.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:44 smithi153 ceph-mon[36569]: pgmap v19432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:44.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:44 smithi104 ceph-mon[36624]: pgmap v19432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:46.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:46 smithi153 ceph-mon[36569]: pgmap v19433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:46.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:46 smithi081 ceph-mon[31453]: pgmap v19433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:46.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:46 smithi104 ceph-mon[36624]: pgmap v19433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:48.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:48 smithi081 ceph-mon[31453]: pgmap v19434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:48 smithi153 ceph-mon[36569]: pgmap v19434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:48.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:48 smithi104 ceph-mon[36624]: pgmap v19434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:49.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:49 smithi081 ceph-mon[31453]: pgmap v19435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:49 smithi153 ceph-mon[36569]: pgmap v19435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:49.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:49 smithi104 ceph-mon[36624]: pgmap v19435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:50.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:50.578+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:23:50.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:50.578+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:23:50.579 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:23:50.578+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:23:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:51 smithi153 ceph-mon[36569]: pgmap v19436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:51.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:51 smithi104 ceph-mon[36624]: pgmap v19436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:51 smithi081 ceph-mon[31453]: pgmap v19436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:53 smithi153 ceph-mon[36569]: pgmap v19437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:53.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:53 smithi104 ceph-mon[36624]: pgmap v19437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:53.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:53 smithi081 ceph-mon[31453]: pgmap v19437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:55 smithi153 ceph-mon[36569]: pgmap v19438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:55 smithi104 ceph-mon[36624]: pgmap v19438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:55.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:55 smithi081 ceph-mon[31453]: pgmap v19438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:57 smithi153 ceph-mon[36569]: pgmap v19439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:57.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:57 smithi104 ceph-mon[36624]: pgmap v19439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:57.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:57 smithi081 ceph-mon[31453]: pgmap v19439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:23:59 smithi153 ceph-mon[36569]: pgmap v19440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:59.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:23:59 smithi104 ceph-mon[36624]: pgmap v19440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:23:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:23:59 smithi081 ceph-mon[31453]: pgmap v19440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:01 smithi153 ceph-mon[36569]: pgmap v19441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:24:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:01 smithi104 ceph-mon[36624]: pgmap v19441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:01.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:24:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:01 smithi081 ceph-mon[31453]: pgmap v19441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:01.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:24:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:24:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:24:02.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:02 smithi153 ceph-mon[36569]: pgmap v19442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:02.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:02 smithi153 ceph-mon[36569]: pgmap v19443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:02.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:24:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:24:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:24:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:02 smithi104 ceph-mon[36624]: pgmap v19442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:02 smithi104 ceph-mon[36624]: pgmap v19443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:02.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:24:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:24:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:24:02.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:02 smithi081 ceph-mon[31453]: pgmap v19442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:02 smithi081 ceph-mon[31453]: pgmap v19443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:02.908 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:24:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:04 smithi153 ceph-mon[36569]: pgmap v19444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:04 smithi104 ceph-mon[36624]: pgmap v19444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:04 smithi081 ceph-mon[31453]: pgmap v19444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:05.593 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:05.592+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:24:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:05.592+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:24:05.594 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:05.592+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:24:07.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:07 smithi153 ceph-mon[36569]: pgmap v19445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:07 smithi104 ceph-mon[36624]: pgmap v19445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:07 smithi081 ceph-mon[31453]: pgmap v19445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:09 smithi153 ceph-mon[36569]: pgmap v19446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:09.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:09 smithi104 ceph-mon[36624]: pgmap v19446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:09.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:09 smithi081 ceph-mon[31453]: pgmap v19446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:11.272 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:11 smithi081 ceph-mon[31453]: pgmap v19447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:11 smithi153 ceph-mon[36569]: pgmap v19447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:11.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:11 smithi104 ceph-mon[36624]: pgmap v19447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:13.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:13 smithi153 ceph-mon[36569]: pgmap v19448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:13 smithi104 ceph-mon[36624]: pgmap v19448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:13 smithi081 ceph-mon[31453]: pgmap v19448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:15 smithi153 ceph-mon[36569]: pgmap v19449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:15 smithi104 ceph-mon[36624]: pgmap v19449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:15 smithi081 ceph-mon[31453]: pgmap v19449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:17 smithi153 ceph-mon[36569]: pgmap v19450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:17 smithi104 ceph-mon[36624]: pgmap v19450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:17.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:17 smithi081 ceph-mon[31453]: pgmap v19450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:19.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:19 smithi153 ceph-mon[36569]: pgmap v19451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:19.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:19 smithi104 ceph-mon[36624]: pgmap v19451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:19.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:19 smithi081 ceph-mon[31453]: pgmap v19451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:20.608+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:24:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:20.608+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:24:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:20.608+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:24:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:21 smithi153 ceph-mon[36569]: pgmap v19452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:21.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:21 smithi104 ceph-mon[36624]: pgmap v19452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:21.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:21 smithi081 ceph-mon[31453]: pgmap v19452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:23 smithi153 ceph-mon[36569]: pgmap v19453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:23.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:23 smithi104 ceph-mon[36624]: pgmap v19453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:23.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:23 smithi081 ceph-mon[31453]: pgmap v19453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:25 smithi153 ceph-mon[36569]: pgmap v19454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:25.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:25 smithi104 ceph-mon[36624]: pgmap v19454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:25.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:25 smithi081 ceph-mon[31453]: pgmap v19454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:27 smithi153 ceph-mon[36569]: pgmap v19455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:27.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:27 smithi104 ceph-mon[36624]: pgmap v19455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:27.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:27 smithi081 ceph-mon[31453]: pgmap v19455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:29.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:29 smithi104 ceph-mon[36624]: pgmap v19456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:29.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:29 smithi081 ceph-mon[31453]: pgmap v19456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:29 smithi153 ceph-mon[36569]: pgmap v19456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:31.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:31 smithi104 ceph-mon[36624]: pgmap v19457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:31.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:31 smithi081 ceph-mon[31453]: pgmap v19457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:31 smithi153 ceph-mon[36569]: pgmap v19457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:33 smithi104 ceph-mon[36624]: pgmap v19458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:33.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:33 smithi081 ceph-mon[31453]: pgmap v19458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:33 smithi153 ceph-mon[36569]: pgmap v19458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:35.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:35 smithi104 ceph-mon[36624]: pgmap v19459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:35.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:35 smithi081 ceph-mon[31453]: pgmap v19459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:35 smithi153 ceph-mon[36569]: pgmap v19459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:35.617+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:24:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:35.617+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:24:35.619 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:35.617+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:24:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:37 smithi104 ceph-mon[36624]: pgmap v19460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:37 smithi081 ceph-mon[31453]: pgmap v19460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:37 smithi153 ceph-mon[36569]: pgmap v19460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:39 smithi081 ceph-mon[31453]: pgmap v19461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:39 smithi153 ceph-mon[36569]: pgmap v19461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:39.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:39 smithi104 ceph-mon[36624]: pgmap v19461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:41 smithi081 ceph-mon[31453]: pgmap v19462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:41 smithi153 ceph-mon[36569]: pgmap v19462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:41 smithi104 ceph-mon[36624]: pgmap v19462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:43.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:43 smithi081 ceph-mon[31453]: pgmap v19463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:43 smithi153 ceph-mon[36569]: pgmap v19463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:43.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:43 smithi104 ceph-mon[36624]: pgmap v19463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:45 smithi081 ceph-mon[31453]: pgmap v19464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:45 smithi153 ceph-mon[36569]: pgmap v19464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:45.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:45 smithi104 ceph-mon[36624]: pgmap v19464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:47.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:47 smithi153 ceph-mon[36569]: pgmap v19465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:47 smithi081 ceph-mon[31453]: pgmap v19465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:47.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:47 smithi104 ceph-mon[36624]: pgmap v19465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:48 smithi153 ceph-mon[36569]: pgmap v19466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:48 smithi104 ceph-mon[36624]: pgmap v19466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:48 smithi081 ceph-mon[31453]: pgmap v19466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:50.633+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:24:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:50.633+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:24:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:24:50.633+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:24:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:51 smithi153 ceph-mon[36569]: pgmap v19467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:51.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:51 smithi104 ceph-mon[36624]: pgmap v19467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:51.361 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:51 smithi081 ceph-mon[31453]: pgmap v19467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:53 smithi153 ceph-mon[36569]: pgmap v19468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:53 smithi104 ceph-mon[36624]: pgmap v19468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:53 smithi081 ceph-mon[31453]: pgmap v19468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:55 smithi153 ceph-mon[36569]: pgmap v19469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:55.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:55 smithi104 ceph-mon[36624]: pgmap v19469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:55 smithi081 ceph-mon[31453]: pgmap v19469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:57 smithi153 ceph-mon[36569]: pgmap v19470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:57.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:57 smithi104 ceph-mon[36624]: pgmap v19470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:57 smithi081 ceph-mon[31453]: pgmap v19470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:24:59 smithi153 ceph-mon[36569]: pgmap v19471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:59.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:24:59 smithi104 ceph-mon[36624]: pgmap v19471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:24:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:24:59 smithi081 ceph-mon[31453]: pgmap v19471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:01 smithi153 ceph-mon[36569]: pgmap v19472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:01.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:01 smithi104 ceph-mon[36624]: pgmap v19472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:01 smithi081 ceph-mon[31453]: pgmap v19472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:03 smithi153 ceph-mon[36569]: pgmap v19473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:25:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:03 smithi104 ceph-mon[36624]: pgmap v19473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:25:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:03 smithi081 ceph-mon[31453]: pgmap v19473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:03.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:25:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:25:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:25:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:25:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:25:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:25:04.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:25:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:25:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:25:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:25:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:05 smithi104 ceph-mon[36624]: pgmap v19474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:05 smithi104 ceph-mon[36624]: pgmap v19475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:05 smithi081 ceph-mon[31453]: pgmap v19474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:05 smithi081 ceph-mon[31453]: pgmap v19475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:05 smithi153 ceph-mon[36569]: pgmap v19474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:05 smithi153 ceph-mon[36569]: pgmap v19475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:05.649+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:25:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:05.649+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:25:05.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:05.649+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:25:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:07 smithi104 ceph-mon[36624]: pgmap v19476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:07.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:07 smithi081 ceph-mon[31453]: pgmap v19476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:07 smithi153 ceph-mon[36569]: pgmap v19476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:09 smithi081 ceph-mon[31453]: pgmap v19477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:09 smithi153 ceph-mon[36569]: pgmap v19477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:09.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:09 smithi104 ceph-mon[36624]: pgmap v19477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:11.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:11 smithi081 ceph-mon[31453]: pgmap v19478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:11 smithi153 ceph-mon[36569]: pgmap v19478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:11.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:11 smithi104 ceph-mon[36624]: pgmap v19478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:13.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:13 smithi081 ceph-mon[31453]: pgmap v19479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:13 smithi153 ceph-mon[36569]: pgmap v19479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:13 smithi104 ceph-mon[36624]: pgmap v19479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:15.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:15 smithi081 ceph-mon[31453]: pgmap v19480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:15 smithi153 ceph-mon[36569]: pgmap v19480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:15 smithi104 ceph-mon[36624]: pgmap v19480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:17.360 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:17 smithi153 ceph-mon[36569]: pgmap v19481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:17 smithi081 ceph-mon[31453]: pgmap v19481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:17 smithi104 ceph-mon[36624]: pgmap v19481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:18.752+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:25:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:18.752+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:25:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:18.753+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 281 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:25:18.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:18.753+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 281 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:25:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:19 smithi081 ceph-mon[31453]: pgmap v19482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:19 smithi153 ceph-mon[36569]: pgmap v19482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:19.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:19 smithi104 ceph-mon[36624]: pgmap v19482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:20.657+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:25:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:20.657+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:25:20.658 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:20.657+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:25:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:21 smithi081 ceph-mon[31453]: pgmap v19483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:21 smithi153 ceph-mon[36569]: pgmap v19483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:21.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:21 smithi104 ceph-mon[36624]: pgmap v19483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:22 smithi081 ceph-mon[31453]: pgmap v19484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:22 smithi153 ceph-mon[36569]: pgmap v19484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:22 smithi104 ceph-mon[36624]: pgmap v19484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:24 smithi081 ceph-mon[31453]: pgmap v19485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:24 smithi153 ceph-mon[36569]: pgmap v19485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:24 smithi104 ceph-mon[36624]: pgmap v19485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:26 smithi081 ceph-mon[31453]: pgmap v19486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:26 smithi153 ceph-mon[36569]: pgmap v19486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:26 smithi104 ceph-mon[36624]: pgmap v19486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:28.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:28 smithi104 ceph-mon[36624]: pgmap v19487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:28 smithi081 ceph-mon[31453]: pgmap v19487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:28 smithi153 ceph-mon[36569]: pgmap v19487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:30 smithi081 ceph-mon[31453]: pgmap v19488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:30 smithi153 ceph-mon[36569]: pgmap v19488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:30 smithi104 ceph-mon[36624]: pgmap v19488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:32 smithi081 ceph-mon[31453]: pgmap v19489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:32 smithi153 ceph-mon[36569]: pgmap v19489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:32 smithi104 ceph-mon[36624]: pgmap v19489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:34 smithi081 ceph-mon[31453]: pgmap v19490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:34 smithi153 ceph-mon[36569]: pgmap v19490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:35.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:34 smithi104 ceph-mon[36624]: pgmap v19490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:35.673 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:35.673+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:25:35.674 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:35.673+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:25:35.674 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:35.673+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:25:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:36 smithi081 ceph-mon[31453]: pgmap v19491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:37.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:36 smithi153 ceph-mon[36569]: pgmap v19491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:36 smithi104 ceph-mon[36624]: pgmap v19491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:38 smithi081 ceph-mon[31453]: pgmap v19492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:38 smithi153 ceph-mon[36569]: pgmap v19492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:38 smithi104 ceph-mon[36624]: pgmap v19492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:40 smithi081 ceph-mon[31453]: pgmap v19493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:40 smithi153 ceph-mon[36569]: pgmap v19493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:40 smithi104 ceph-mon[36624]: pgmap v19493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:42 smithi153 ceph-mon[36569]: pgmap v19494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:42 smithi104 ceph-mon[36624]: pgmap v19494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:42 smithi081 ceph-mon[31453]: pgmap v19494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:44 smithi153 ceph-mon[36569]: pgmap v19495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:44 smithi104 ceph-mon[36624]: pgmap v19495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:44 smithi081 ceph-mon[31453]: pgmap v19495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:46 smithi153 ceph-mon[36569]: pgmap v19496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:46 smithi104 ceph-mon[36624]: pgmap v19496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:46 smithi081 ceph-mon[31453]: pgmap v19496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:49.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:48 smithi153 ceph-mon[36569]: pgmap v19497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:48 smithi104 ceph-mon[36624]: pgmap v19497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:48 smithi081 ceph-mon[31453]: pgmap v19497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:50.688+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:25:50.689 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:50.688+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:25:50.690 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:25:50.688+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:25:51.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:50 smithi081 ceph-mon[31453]: pgmap v19498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:50 smithi153 ceph-mon[36569]: pgmap v19498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:50 smithi104 ceph-mon[36624]: pgmap v19498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:52 smithi153 ceph-mon[36569]: pgmap v19499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:52 smithi104 ceph-mon[36624]: pgmap v19499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:52 smithi081 ceph-mon[31453]: pgmap v19499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:54 smithi153 ceph-mon[36569]: pgmap v19500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:54 smithi104 ceph-mon[36624]: pgmap v19500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:55.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:54 smithi081 ceph-mon[31453]: pgmap v19500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:57.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:56 smithi153 ceph-mon[36569]: pgmap v19501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:56 smithi104 ceph-mon[36624]: pgmap v19501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:56 smithi081 ceph-mon[31453]: pgmap v19501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:59.304 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:25:58 smithi081 ceph-mon[31453]: pgmap v19502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:25:58 smithi153 ceph-mon[36569]: pgmap v19502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:25:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:25:58 smithi104 ceph-mon[36624]: pgmap v19502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:00 smithi153 ceph-mon[36569]: pgmap v19503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:00 smithi104 ceph-mon[36624]: pgmap v19503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:00 smithi081 ceph-mon[31453]: pgmap v19503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:03.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:02 smithi153 ceph-mon[36569]: pgmap v19504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:02 smithi104 ceph-mon[36624]: pgmap v19504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:02 smithi081 ceph-mon[31453]: pgmap v19504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:26:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:26:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:26:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: pgmap v19505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: pgmap v19505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: pgmap v19505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:05.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:05.704+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:26:05.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:05.704+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:26:05.705 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:05.704+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:26:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:26:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:26:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:26:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:26:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: pgmap v19506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: pgmap v19507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:06.308 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:26:06.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:26:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: pgmap v19506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: pgmap v19507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:06.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:26:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:26:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:26:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:26:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:26:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: pgmap v19506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: pgmap v19507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:06.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:26:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:07 smithi153 ceph-mon[36569]: pgmap v19508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:07.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:07 smithi104 ceph-mon[36624]: pgmap v19508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:07 smithi081 ceph-mon[31453]: pgmap v19508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:09 smithi153 ceph-mon[36569]: pgmap v19509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:09 smithi104 ceph-mon[36624]: pgmap v19509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:09 smithi081 ceph-mon[31453]: pgmap v19509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:11 smithi153 ceph-mon[36569]: pgmap v19510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:11 smithi104 ceph-mon[36624]: pgmap v19510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:11.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:11 smithi081 ceph-mon[31453]: pgmap v19510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:13 smithi153 ceph-mon[36569]: pgmap v19511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:13 smithi104 ceph-mon[36624]: pgmap v19511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:13.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:13 smithi081 ceph-mon[31453]: pgmap v19511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:15.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:15 smithi081 ceph-mon[31453]: pgmap v19512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:15 smithi153 ceph-mon[36569]: pgmap v19512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:15 smithi104 ceph-mon[36624]: pgmap v19512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:17.559 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:17 smithi153 ceph-mon[36569]: pgmap v19513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:17 smithi104 ceph-mon[36624]: pgmap v19513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:17 smithi081 ceph-mon[31453]: pgmap v19513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:19 smithi153 ceph-mon[36569]: pgmap v19514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:19 smithi104 ceph-mon[36624]: pgmap v19514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:19 smithi081 ceph-mon[31453]: pgmap v19514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:20.718 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:20.717+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:26:20.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:20.717+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:26:20.719 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:20.717+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:26:21.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:21 smithi081 ceph-mon[31453]: pgmap v19515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:21 smithi153 ceph-mon[36569]: pgmap v19515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:21 smithi104 ceph-mon[36624]: pgmap v19515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:23 smithi153 ceph-mon[36569]: pgmap v19516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:23 smithi104 ceph-mon[36624]: pgmap v19516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:23 smithi081 ceph-mon[31453]: pgmap v19516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:25.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:25 smithi153 ceph-mon[36569]: pgmap v19517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:25 smithi104 ceph-mon[36624]: pgmap v19517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:25 smithi081 ceph-mon[31453]: pgmap v19517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:27.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:27 smithi153 ceph-mon[36569]: pgmap v19518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:27 smithi104 ceph-mon[36624]: pgmap v19518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:27 smithi081 ceph-mon[31453]: pgmap v19518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:29.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:29 smithi153 ceph-mon[36569]: pgmap v19519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:29 smithi104 ceph-mon[36624]: pgmap v19519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:29 smithi081 ceph-mon[31453]: pgmap v19519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:31 smithi081 ceph-mon[31453]: pgmap v19520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:31.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:31 smithi153 ceph-mon[36569]: pgmap v19520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:31 smithi104 ceph-mon[36624]: pgmap v19520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:33 smithi153 ceph-mon[36569]: pgmap v19521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:33 smithi104 ceph-mon[36624]: pgmap v19521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:33 smithi081 ceph-mon[31453]: pgmap v19521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:35.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:35 smithi153 ceph-mon[36569]: pgmap v19522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:35 smithi104 ceph-mon[36624]: pgmap v19522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:35 smithi081 ceph-mon[31453]: pgmap v19522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:35.719+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:26:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:35.719+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:26:35.720 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:35.719+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:26:37.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:37 smithi153 ceph-mon[36569]: pgmap v19523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:37 smithi104 ceph-mon[36624]: pgmap v19523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:37 smithi081 ceph-mon[31453]: pgmap v19523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:39.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:39 smithi153 ceph-mon[36569]: pgmap v19524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:39 smithi104 ceph-mon[36624]: pgmap v19524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:39 smithi081 ceph-mon[31453]: pgmap v19524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:41 smithi153 ceph-mon[36569]: pgmap v19525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:41 smithi104 ceph-mon[36624]: pgmap v19525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:41 smithi081 ceph-mon[31453]: pgmap v19525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:43.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:43 smithi153 ceph-mon[36569]: pgmap v19526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:43 smithi104 ceph-mon[36624]: pgmap v19526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:43 smithi081 ceph-mon[31453]: pgmap v19526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:45 smithi153 ceph-mon[36569]: pgmap v19527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:45 smithi104 ceph-mon[36624]: pgmap v19527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:45 smithi081 ceph-mon[31453]: pgmap v19527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:47.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:47 smithi153 ceph-mon[36569]: pgmap v19528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:47 smithi104 ceph-mon[36624]: pgmap v19528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:47 smithi081 ceph-mon[31453]: pgmap v19528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:48.768+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:26:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:48.768+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:26:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:48.768+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 282 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:26:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:48.768+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 282 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:26:49.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:49 smithi153 ceph-mon[36569]: pgmap v19529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:49 smithi104 ceph-mon[36624]: pgmap v19529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:49 smithi081 ceph-mon[31453]: pgmap v19529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:50.732+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:26:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:50.732+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:26:50.733 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:26:50.732+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:26:51.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:51 smithi081 ceph-mon[31453]: pgmap v19530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:51 smithi153 ceph-mon[36569]: pgmap v19530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:51 smithi104 ceph-mon[36624]: pgmap v19530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:53.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:53 smithi153 ceph-mon[36569]: pgmap v19531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:53 smithi104 ceph-mon[36624]: pgmap v19531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:53 smithi081 ceph-mon[31453]: pgmap v19531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:55 smithi153 ceph-mon[36569]: pgmap v19532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:55 smithi104 ceph-mon[36624]: pgmap v19532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:55 smithi081 ceph-mon[31453]: pgmap v19532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:57.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:57 smithi153 ceph-mon[36569]: pgmap v19533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:57 smithi104 ceph-mon[36624]: pgmap v19533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:57 smithi081 ceph-mon[31453]: pgmap v19533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:59.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:26:59 smithi153 ceph-mon[36569]: pgmap v19534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:26:59 smithi104 ceph-mon[36624]: pgmap v19534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:26:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:26:59 smithi081 ceph-mon[31453]: pgmap v19534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:01 smithi153 ceph-mon[36569]: pgmap v19535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:01 smithi104 ceph-mon[36624]: pgmap v19535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:01 smithi081 ceph-mon[31453]: pgmap v19535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:03.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:03 smithi153 ceph-mon[36569]: pgmap v19536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:03 smithi104 ceph-mon[36624]: pgmap v19536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:03 smithi081 ceph-mon[31453]: pgmap v19536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:05 smithi153 ceph-mon[36569]: pgmap v19537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:05 smithi104 ceph-mon[36624]: pgmap v19537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:05 smithi081 ceph-mon[31453]: pgmap v19537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:05.747+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:05.747+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:27:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:05.747+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:27:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:27:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:27:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:27:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:27:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:27:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:07 smithi153 ceph-mon[36569]: pgmap v19538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:07 smithi153 ceph-mon[36569]: pgmap v19539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:27:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:27:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:27:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:07 smithi104 ceph-mon[36624]: pgmap v19538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:07 smithi104 ceph-mon[36624]: pgmap v19539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:07.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:27:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:27:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:27:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:07 smithi081 ceph-mon[31453]: pgmap v19538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:07 smithi081 ceph-mon[31453]: pgmap v19539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:07.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:27:10.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:09 smithi153 ceph-mon[36569]: pgmap v19540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:09 smithi104 ceph-mon[36624]: pgmap v19540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:09 smithi081 ceph-mon[31453]: pgmap v19540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:11 smithi153 ceph-mon[36569]: pgmap v19541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:11 smithi104 ceph-mon[36624]: pgmap v19541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:11 smithi081 ceph-mon[31453]: pgmap v19541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:13 smithi153 ceph-mon[36569]: pgmap v19542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:13 smithi104 ceph-mon[36624]: pgmap v19542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:13 smithi081 ceph-mon[31453]: pgmap v19542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:15 smithi153 ceph-mon[36569]: pgmap v19543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:15 smithi104 ceph-mon[36624]: pgmap v19543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:15 smithi081 ceph-mon[31453]: pgmap v19543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:18.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:17 smithi153 ceph-mon[36569]: pgmap v19544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:17 smithi104 ceph-mon[36624]: pgmap v19544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:17 smithi081 ceph-mon[31453]: pgmap v19544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:19 smithi153 ceph-mon[36569]: pgmap v19545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:19 smithi104 ceph-mon[36624]: pgmap v19545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:19 smithi081 ceph-mon[31453]: pgmap v19545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:20.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:20.763+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:27:20.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:20.763+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:27:20.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:20.763+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:27:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:21 smithi153 ceph-mon[36569]: pgmap v19546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:21 smithi104 ceph-mon[36624]: pgmap v19546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:21 smithi081 ceph-mon[31453]: pgmap v19546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:23 smithi153 ceph-mon[36569]: pgmap v19547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:23 smithi104 ceph-mon[36624]: pgmap v19547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:23 smithi081 ceph-mon[31453]: pgmap v19547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:25 smithi153 ceph-mon[36569]: pgmap v19548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:25 smithi104 ceph-mon[36624]: pgmap v19548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:25 smithi081 ceph-mon[31453]: pgmap v19548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:27 smithi153 ceph-mon[36569]: pgmap v19549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:27 smithi104 ceph-mon[36624]: pgmap v19549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:27 smithi081 ceph-mon[31453]: pgmap v19549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:30.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:29 smithi153 ceph-mon[36569]: pgmap v19550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:29 smithi104 ceph-mon[36624]: pgmap v19550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:29 smithi081 ceph-mon[31453]: pgmap v19550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:31 smithi153 ceph-mon[36569]: pgmap v19551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:31 smithi104 ceph-mon[36624]: pgmap v19551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:31 smithi081 ceph-mon[31453]: pgmap v19551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:34.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:33 smithi153 ceph-mon[36569]: pgmap v19552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:33 smithi104 ceph-mon[36624]: pgmap v19552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:33 smithi081 ceph-mon[31453]: pgmap v19552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:35.764+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:27:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:35.764+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:27:35.765 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:35.764+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:27:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:35 smithi153 ceph-mon[36569]: pgmap v19553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:35 smithi104 ceph-mon[36624]: pgmap v19553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:35 smithi081 ceph-mon[31453]: pgmap v19553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:37 smithi153 ceph-mon[36569]: pgmap v19554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:37 smithi104 ceph-mon[36624]: pgmap v19554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:37 smithi081 ceph-mon[31453]: pgmap v19554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:39 smithi153 ceph-mon[36569]: pgmap v19555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:39 smithi104 ceph-mon[36624]: pgmap v19555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:39 smithi081 ceph-mon[31453]: pgmap v19555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:41 smithi153 ceph-mon[36569]: pgmap v19556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:41 smithi104 ceph-mon[36624]: pgmap v19556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:41 smithi081 ceph-mon[31453]: pgmap v19556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:43 smithi104 ceph-mon[36624]: pgmap v19557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:43 smithi081 ceph-mon[31453]: pgmap v19557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:43 smithi153 ceph-mon[36569]: pgmap v19557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:45 smithi104 ceph-mon[36624]: pgmap v19558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:45 smithi081 ceph-mon[31453]: pgmap v19558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:45 smithi153 ceph-mon[36569]: pgmap v19558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:47 smithi104 ceph-mon[36624]: pgmap v19559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:48.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:47 smithi153 ceph-mon[36569]: pgmap v19559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:48.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:47 smithi081 ceph-mon[31453]: pgmap v19559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:49 smithi104 ceph-mon[36624]: pgmap v19560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:49 smithi081 ceph-mon[31453]: pgmap v19560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:49 smithi153 ceph-mon[36569]: pgmap v19560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:50.779 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:50.779+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:27:50.780 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:50.779+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:27:50.780 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:27:50.779+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:27:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:51 smithi104 ceph-mon[36624]: pgmap v19561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:51 smithi081 ceph-mon[31453]: pgmap v19561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:51 smithi153 ceph-mon[36569]: pgmap v19561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:53 smithi081 ceph-mon[31453]: pgmap v19562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:53 smithi153 ceph-mon[36569]: pgmap v19562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:53 smithi104 ceph-mon[36624]: pgmap v19562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:55 smithi081 ceph-mon[31453]: pgmap v19563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:56.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:55 smithi153 ceph-mon[36569]: pgmap v19563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:55 smithi104 ceph-mon[36624]: pgmap v19563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:57 smithi081 ceph-mon[31453]: pgmap v19564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:57 smithi153 ceph-mon[36569]: pgmap v19564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:27:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:57 smithi104 ceph-mon[36624]: pgmap v19564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:00.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:27:59 smithi104 ceph-mon[36624]: pgmap v19565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:27:59 smithi081 ceph-mon[31453]: pgmap v19565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:27:59 smithi153 ceph-mon[36569]: pgmap v19565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:01 smithi081 ceph-mon[31453]: pgmap v19566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:01 smithi153 ceph-mon[36569]: pgmap v19566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:01 smithi104 ceph-mon[36624]: pgmap v19566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:03 smithi081 ceph-mon[31453]: pgmap v19567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:03 smithi153 ceph-mon[36569]: pgmap v19567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:04.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:03 smithi104 ceph-mon[36624]: pgmap v19567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:05.789+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:28:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:05.789+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:28:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:05.789+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:28:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:05 smithi081 ceph-mon[31453]: pgmap v19568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:05 smithi153 ceph-mon[36569]: pgmap v19568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:05 smithi104 ceph-mon[36624]: pgmap v19568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:07 smithi081 ceph-mon[31453]: pgmap v19569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:28:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:07 smithi153 ceph-mon[36569]: pgmap v19569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:28:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:07 smithi104 ceph-mon[36624]: pgmap v19569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:28:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:28:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:28:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:28:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:28:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:28:09.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:28:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:28:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:28:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:28:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:09 smithi081 ceph-mon[31453]: pgmap v19570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:09 smithi081 ceph-mon[31453]: pgmap v19571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:09 smithi153 ceph-mon[36569]: pgmap v19570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:09 smithi153 ceph-mon[36569]: pgmap v19571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:09 smithi104 ceph-mon[36624]: pgmap v19570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:09 smithi104 ceph-mon[36624]: pgmap v19571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:11 smithi081 ceph-mon[31453]: pgmap v19572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:11 smithi153 ceph-mon[36569]: pgmap v19572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:11 smithi104 ceph-mon[36624]: pgmap v19572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:13 smithi081 ceph-mon[31453]: pgmap v19573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:13 smithi153 ceph-mon[36569]: pgmap v19573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:13 smithi104 ceph-mon[36624]: pgmap v19573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:15 smithi081 ceph-mon[31453]: pgmap v19574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:15 smithi153 ceph-mon[36569]: pgmap v19574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:15 smithi104 ceph-mon[36624]: pgmap v19574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:17 smithi081 ceph-mon[31453]: pgmap v19575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:17 smithi153 ceph-mon[36569]: pgmap v19575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:17 smithi104 ceph-mon[36624]: pgmap v19575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:18.782+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:28:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:18.782+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:28:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:18.783+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 283 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:28:18.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:18.783+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 283 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:28:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:19 smithi153 ceph-mon[36569]: pgmap v19576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:19 smithi104 ceph-mon[36624]: pgmap v19576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:20.359 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:19 smithi081 ceph-mon[31453]: pgmap v19576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:20.805+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:28:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:20.805+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:28:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:20.805+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:28:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:21 smithi081 ceph-mon[31453]: pgmap v19577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:21 smithi153 ceph-mon[36569]: pgmap v19577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:21 smithi104 ceph-mon[36624]: pgmap v19577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:23 smithi153 ceph-mon[36569]: pgmap v19578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:23 smithi104 ceph-mon[36624]: pgmap v19578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:23 smithi081 ceph-mon[31453]: pgmap v19578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:25 smithi153 ceph-mon[36569]: pgmap v19579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:25 smithi104 ceph-mon[36624]: pgmap v19579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:25 smithi081 ceph-mon[31453]: pgmap v19579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:27 smithi153 ceph-mon[36569]: pgmap v19580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:27 smithi104 ceph-mon[36624]: pgmap v19580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:27 smithi081 ceph-mon[31453]: pgmap v19580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:29 smithi153 ceph-mon[36569]: pgmap v19581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:30.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:29 smithi104 ceph-mon[36624]: pgmap v19581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:29 smithi081 ceph-mon[31453]: pgmap v19581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:31 smithi153 ceph-mon[36569]: pgmap v19582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:31 smithi104 ceph-mon[36624]: pgmap v19582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:31 smithi081 ceph-mon[31453]: pgmap v19582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:34.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:33 smithi153 ceph-mon[36569]: pgmap v19583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:33 smithi104 ceph-mon[36624]: pgmap v19583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:33 smithi081 ceph-mon[31453]: pgmap v19583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:35.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:35.819+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:28:35.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:35.819+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:28:35.820 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:35.819+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:28:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:35 smithi153 ceph-mon[36569]: pgmap v19584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:35 smithi104 ceph-mon[36624]: pgmap v19584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:35 smithi081 ceph-mon[31453]: pgmap v19584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:37 smithi153 ceph-mon[36569]: pgmap v19585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:37 smithi104 ceph-mon[36624]: pgmap v19585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:37 smithi081 ceph-mon[31453]: pgmap v19585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:39 smithi153 ceph-mon[36569]: pgmap v19586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:39 smithi104 ceph-mon[36624]: pgmap v19586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:39 smithi081 ceph-mon[31453]: pgmap v19586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:42.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:41 smithi153 ceph-mon[36569]: pgmap v19587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:41 smithi104 ceph-mon[36624]: pgmap v19587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:41 smithi081 ceph-mon[31453]: pgmap v19587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:43 smithi153 ceph-mon[36569]: pgmap v19588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:43 smithi104 ceph-mon[36624]: pgmap v19588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:43 smithi081 ceph-mon[31453]: pgmap v19588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:46 smithi153 ceph-mon[36569]: pgmap v19589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:46 smithi104 ceph-mon[36624]: pgmap v19589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:46 smithi081 ceph-mon[31453]: pgmap v19589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:48 smithi153 ceph-mon[36569]: pgmap v19590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:48 smithi104 ceph-mon[36624]: pgmap v19590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:48 smithi081 ceph-mon[31453]: pgmap v19590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:50 smithi153 ceph-mon[36569]: pgmap v19591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:50 smithi104 ceph-mon[36624]: pgmap v19591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:50 smithi081 ceph-mon[31453]: pgmap v19591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:50.835+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:50.835+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:28:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:28:50.835+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:28:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:52 smithi153 ceph-mon[36569]: pgmap v19592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:52 smithi104 ceph-mon[36624]: pgmap v19592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:52 smithi081 ceph-mon[31453]: pgmap v19592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:54 smithi153 ceph-mon[36569]: pgmap v19593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:54 smithi104 ceph-mon[36624]: pgmap v19593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:54 smithi081 ceph-mon[31453]: pgmap v19593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:56 smithi153 ceph-mon[36569]: pgmap v19594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:56 smithi104 ceph-mon[36624]: pgmap v19594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:56 smithi081 ceph-mon[31453]: pgmap v19594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:58.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:28:58 smithi153 ceph-mon[36569]: pgmap v19595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:28:58 smithi104 ceph-mon[36624]: pgmap v19595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:28:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:28:58 smithi081 ceph-mon[31453]: pgmap v19595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:00 smithi153 ceph-mon[36569]: pgmap v19596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:00 smithi104 ceph-mon[36624]: pgmap v19596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:00 smithi081 ceph-mon[31453]: pgmap v19596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:02 smithi153 ceph-mon[36569]: pgmap v19597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:02 smithi104 ceph-mon[36624]: pgmap v19597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:02 smithi081 ceph-mon[31453]: pgmap v19597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:04 smithi153 ceph-mon[36569]: pgmap v19598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:04 smithi104 ceph-mon[36624]: pgmap v19598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:04 smithi081 ceph-mon[31453]: pgmap v19598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:05.842+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:29:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:05.842+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:29:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:05.842+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:29:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:06 smithi153 ceph-mon[36569]: pgmap v19599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:06 smithi104 ceph-mon[36624]: pgmap v19599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:06 smithi081 ceph-mon[31453]: pgmap v19599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:08 smithi104 ceph-mon[36624]: pgmap v19600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:08 smithi081 ceph-mon[31453]: pgmap v19600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:08 smithi153 ceph-mon[36569]: pgmap v19600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:29:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:29:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:29:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:10 smithi104 ceph-mon[36624]: pgmap v19601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:29:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:29:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:29:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:10 smithi081 ceph-mon[31453]: pgmap v19601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:29:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:29:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:29:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:10 smithi153 ceph-mon[36569]: pgmap v19601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:29:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:29:10.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:29:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:11 smithi104 ceph-mon[36624]: pgmap v19602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:11 smithi104 ceph-mon[36624]: pgmap v19603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:11 smithi081 ceph-mon[31453]: pgmap v19602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:11 smithi081 ceph-mon[31453]: pgmap v19603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:11 smithi153 ceph-mon[36569]: pgmap v19602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:11.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:11 smithi153 ceph-mon[36569]: pgmap v19603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:13 smithi081 ceph-mon[31453]: pgmap v19604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:13 smithi153 ceph-mon[36569]: pgmap v19604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:13 smithi104 ceph-mon[36624]: pgmap v19604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:15 smithi081 ceph-mon[31453]: pgmap v19605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:15 smithi153 ceph-mon[36569]: pgmap v19605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:15 smithi104 ceph-mon[36624]: pgmap v19605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:17 smithi081 ceph-mon[31453]: pgmap v19606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:17 smithi153 ceph-mon[36569]: pgmap v19606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:17 smithi104 ceph-mon[36624]: pgmap v19606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:19.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:19 smithi153 ceph-mon[36569]: pgmap v19607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:19 smithi081 ceph-mon[31453]: pgmap v19607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:19 smithi104 ceph-mon[36624]: pgmap v19607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:20.848+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:29:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:20.849+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:29:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:20.849+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:29:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:21 smithi081 ceph-mon[31453]: pgmap v19608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:21 smithi153 ceph-mon[36569]: pgmap v19608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:21 smithi104 ceph-mon[36624]: pgmap v19608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:23 smithi081 ceph-mon[31453]: pgmap v19609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:23 smithi153 ceph-mon[36569]: pgmap v19609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:23 smithi104 ceph-mon[36624]: pgmap v19609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:25 smithi081 ceph-mon[31453]: pgmap v19610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:25 smithi153 ceph-mon[36569]: pgmap v19610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:25 smithi104 ceph-mon[36624]: pgmap v19610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:27 smithi081 ceph-mon[31453]: pgmap v19611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:27 smithi153 ceph-mon[36569]: pgmap v19611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:27 smithi104 ceph-mon[36624]: pgmap v19611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:29 smithi081 ceph-mon[31453]: pgmap v19612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:29 smithi153 ceph-mon[36569]: pgmap v19612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:29 smithi104 ceph-mon[36624]: pgmap v19612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:31.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:31 smithi104 ceph-mon[36624]: pgmap v19613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:31 smithi081 ceph-mon[31453]: pgmap v19613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:31 smithi153 ceph-mon[36569]: pgmap v19613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:33 smithi081 ceph-mon[31453]: pgmap v19614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:33 smithi153 ceph-mon[36569]: pgmap v19614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:33 smithi104 ceph-mon[36624]: pgmap v19614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:35 smithi081 ceph-mon[31453]: pgmap v19615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:35 smithi153 ceph-mon[36569]: pgmap v19615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:35 smithi104 ceph-mon[36624]: pgmap v19615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:35.864+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:29:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:35.864+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:29:35.864 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:35.864+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:29:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:37 smithi081 ceph-mon[31453]: pgmap v19616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:37 smithi153 ceph-mon[36569]: pgmap v19616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:37 smithi104 ceph-mon[36624]: pgmap v19616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:39 smithi081 ceph-mon[31453]: pgmap v19617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:39 smithi153 ceph-mon[36569]: pgmap v19617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:39 smithi104 ceph-mon[36624]: pgmap v19617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:41 smithi081 ceph-mon[31453]: pgmap v19618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:41 smithi153 ceph-mon[36569]: pgmap v19618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:41 smithi104 ceph-mon[36624]: pgmap v19618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:43 smithi153 ceph-mon[36569]: pgmap v19619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:43 smithi104 ceph-mon[36624]: pgmap v19619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:43 smithi081 ceph-mon[31453]: pgmap v19619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:45 smithi153 ceph-mon[36569]: pgmap v19620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:45 smithi104 ceph-mon[36624]: pgmap v19620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:45 smithi081 ceph-mon[31453]: pgmap v19620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:46 smithi153 ceph-mon[36569]: pgmap v19621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:46 smithi104 ceph-mon[36624]: pgmap v19621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:46 smithi081 ceph-mon[31453]: pgmap v19621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:49.259 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:48 smithi153 ceph-mon[36569]: pgmap v19622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:48 smithi104 ceph-mon[36624]: pgmap v19622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:48 smithi081 ceph-mon[31453]: pgmap v19622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:50.877+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:29:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:50.877+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:29:50.878 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:29:50.877+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:29:51.239 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:50 smithi081 ceph-mon[31453]: pgmap v19623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:50 smithi153 ceph-mon[36569]: pgmap v19623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:50 smithi104 ceph-mon[36624]: pgmap v19623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:52 smithi153 ceph-mon[36569]: pgmap v19624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:52 smithi104 ceph-mon[36624]: pgmap v19624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:52 smithi081 ceph-mon[31453]: pgmap v19624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:54 smithi153 ceph-mon[36569]: pgmap v19625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:54 smithi104 ceph-mon[36624]: pgmap v19625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:54 smithi081 ceph-mon[31453]: pgmap v19625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:57 smithi153 ceph-mon[36569]: pgmap v19626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:57 smithi104 ceph-mon[36624]: pgmap v19626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:57 smithi081 ceph-mon[31453]: pgmap v19626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:29:59 smithi153 ceph-mon[36569]: pgmap v19627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:29:59 smithi104 ceph-mon[36624]: pgmap v19627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:29:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:29:59 smithi081 ceph-mon[31453]: pgmap v19627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:30:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:30:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:30:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:01 smithi153 ceph-mon[36569]: pgmap v19628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:01 smithi104 ceph-mon[36624]: pgmap v19628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:01 smithi081 ceph-mon[31453]: pgmap v19628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:03 smithi153 ceph-mon[36569]: pgmap v19629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:03 smithi104 ceph-mon[36624]: pgmap v19629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:03 smithi081 ceph-mon[31453]: pgmap v19629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:05 smithi153 ceph-mon[36569]: pgmap v19630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:05 smithi104 ceph-mon[36624]: pgmap v19630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:05 smithi081 ceph-mon[31453]: pgmap v19630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:05.882+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:30:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:05.882+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:30:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:05.882+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:30:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:07 smithi153 ceph-mon[36569]: pgmap v19631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:07.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:07 smithi104 ceph-mon[36624]: pgmap v19631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:07 smithi081 ceph-mon[31453]: pgmap v19631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:09 smithi153 ceph-mon[36569]: pgmap v19632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:09 smithi104 ceph-mon[36624]: pgmap v19632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:09 smithi081 ceph-mon[31453]: pgmap v19632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:11 smithi153 ceph-mon[36569]: pgmap v19633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:30:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:11 smithi104 ceph-mon[36624]: pgmap v19633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:30:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:11 smithi081 ceph-mon[31453]: pgmap v19633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:30:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:13 smithi104 ceph-mon[36624]: pgmap v19634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:13 smithi081 ceph-mon[31453]: pgmap v19634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:13 smithi153 ceph-mon[36569]: pgmap v19634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:14 smithi153 ceph-mon[36569]: pgmap v19635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:14 smithi104 ceph-mon[36624]: pgmap v19635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:14 smithi081 ceph-mon[31453]: pgmap v19635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:16 smithi081 ceph-mon[31453]: pgmap v19636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:16 smithi153 ceph-mon[36569]: pgmap v19636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:16 smithi104 ceph-mon[36624]: pgmap v19636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:30:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:30:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:17 smithi081 ceph-mon[31453]: pgmap v19637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:17 smithi081 ceph-mon[31453]: pgmap v19638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:30:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:30:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:17 smithi153 ceph-mon[36569]: pgmap v19637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:17 smithi153 ceph-mon[36569]: pgmap v19638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:30:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:30:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:17 smithi104 ceph-mon[36624]: pgmap v19637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:17 smithi104 ceph-mon[36624]: pgmap v19638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:30:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:19 smithi153 ceph-mon[36569]: pgmap v19639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:19 smithi104 ceph-mon[36624]: pgmap v19639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:19 smithi081 ceph-mon[31453]: pgmap v19639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:20.897+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:30:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:20.897+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:30:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:20.897+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:30:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:21 smithi153 ceph-mon[36569]: pgmap v19640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:21 smithi104 ceph-mon[36624]: pgmap v19640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:21 smithi081 ceph-mon[31453]: pgmap v19640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:23 smithi153 ceph-mon[36569]: pgmap v19641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:23 smithi104 ceph-mon[36624]: pgmap v19641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:23 smithi081 ceph-mon[31453]: pgmap v19641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:25 smithi153 ceph-mon[36569]: pgmap v19642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:25 smithi104 ceph-mon[36624]: pgmap v19642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:25 smithi081 ceph-mon[31453]: pgmap v19642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:27 smithi153 ceph-mon[36569]: pgmap v19643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:27 smithi104 ceph-mon[36624]: pgmap v19643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:27 smithi081 ceph-mon[31453]: pgmap v19643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:29 smithi153 ceph-mon[36569]: pgmap v19644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:29 smithi104 ceph-mon[36624]: pgmap v19644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:29 smithi081 ceph-mon[31453]: pgmap v19644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:31 smithi153 ceph-mon[36569]: pgmap v19645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:31 smithi104 ceph-mon[36624]: pgmap v19645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:31 smithi081 ceph-mon[31453]: pgmap v19645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:33 smithi153 ceph-mon[36569]: pgmap v19646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:33 smithi104 ceph-mon[36624]: pgmap v19646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:33 smithi081 ceph-mon[31453]: pgmap v19646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:35.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:35.913+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:30:35.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:35.913+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:30:35.913 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:35.913+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:30:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:35 smithi153 ceph-mon[36569]: pgmap v19647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:35 smithi104 ceph-mon[36624]: pgmap v19647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:35 smithi081 ceph-mon[31453]: pgmap v19647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:37 smithi153 ceph-mon[36569]: pgmap v19648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:37 smithi104 ceph-mon[36624]: pgmap v19648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:37 smithi081 ceph-mon[31453]: pgmap v19648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:39 smithi153 ceph-mon[36569]: pgmap v19649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:39 smithi104 ceph-mon[36624]: pgmap v19649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:39 smithi081 ceph-mon[31453]: pgmap v19649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:41 smithi153 ceph-mon[36569]: pgmap v19650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:41 smithi104 ceph-mon[36624]: pgmap v19650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:41 smithi081 ceph-mon[31453]: pgmap v19650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:43 smithi153 ceph-mon[36569]: pgmap v19651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:43 smithi104 ceph-mon[36624]: pgmap v19651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:43 smithi081 ceph-mon[31453]: pgmap v19651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:45 smithi153 ceph-mon[36569]: pgmap v19652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:45 smithi104 ceph-mon[36624]: pgmap v19652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:45 smithi081 ceph-mon[31453]: pgmap v19652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:47 smithi153 ceph-mon[36569]: pgmap v19653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:47 smithi104 ceph-mon[36624]: pgmap v19653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:47 smithi081 ceph-mon[31453]: pgmap v19653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:49 smithi104 ceph-mon[36624]: pgmap v19654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:50.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:49 smithi081 ceph-mon[31453]: pgmap v19654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:49 smithi153 ceph-mon[36569]: pgmap v19654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:50.928+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:30:50.928 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:50.928+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:30:50.929 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:30:50.928+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:30:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:51 smithi104 ceph-mon[36624]: pgmap v19655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:52.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:51 smithi153 ceph-mon[36569]: pgmap v19655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:51 smithi081 ceph-mon[31453]: pgmap v19655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:53 smithi104 ceph-mon[36624]: pgmap v19656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:53 smithi081 ceph-mon[31453]: pgmap v19656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:53 smithi153 ceph-mon[36569]: pgmap v19656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:55 smithi104 ceph-mon[36624]: pgmap v19657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:55 smithi081 ceph-mon[31453]: pgmap v19657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:55 smithi153 ceph-mon[36569]: pgmap v19657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:57 smithi104 ceph-mon[36624]: pgmap v19658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:57 smithi081 ceph-mon[31453]: pgmap v19658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:30:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:57 smithi153 ceph-mon[36569]: pgmap v19658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:30:59 smithi081 ceph-mon[31453]: pgmap v19659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:30:59 smithi153 ceph-mon[36569]: pgmap v19659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:00.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:30:59 smithi104 ceph-mon[36624]: pgmap v19659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:02.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:01 smithi104 ceph-mon[36624]: pgmap v19660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:01 smithi081 ceph-mon[31453]: pgmap v19660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:01 smithi153 ceph-mon[36569]: pgmap v19660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:04.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:03 smithi081 ceph-mon[31453]: pgmap v19661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:03 smithi153 ceph-mon[36569]: pgmap v19661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:03 smithi104 ceph-mon[36624]: pgmap v19661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:05.943+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:05.943+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:31:05.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:05.943+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:31:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:05 smithi081 ceph-mon[31453]: pgmap v19662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:05 smithi153 ceph-mon[36569]: pgmap v19662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:05 smithi104 ceph-mon[36624]: pgmap v19662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:07 smithi081 ceph-mon[31453]: pgmap v19663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:07 smithi153 ceph-mon[36569]: pgmap v19663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:07 smithi104 ceph-mon[36624]: pgmap v19663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:10.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:09 smithi081 ceph-mon[31453]: pgmap v19664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:09 smithi153 ceph-mon[36569]: pgmap v19664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:09 smithi104 ceph-mon[36624]: pgmap v19664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:11 smithi081 ceph-mon[31453]: pgmap v19665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:12.192+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 msgr2=0x7f16e80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000648) for more than 900000000 us, fault. 2024-04-13T07:31:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:12.192+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006530 tx=0x7f16f404e000 comp rx=0 tx=0).stop 2024-04-13T07:31:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:12.192+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 msgr2=0x7f16e80b1520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T07:31:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:12.192+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a5330 0x7f16e80b1520 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T07:31:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:12.192+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T07:31:12.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:12.193+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T07:31:12.194 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:12.194+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3fb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006660 tx=0x7f16f408e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T07:31:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:11 smithi153 ceph-mon[36569]: pgmap v19665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:11 smithi104 ceph-mon[36624]: pgmap v19665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:13 smithi081 ceph-mon[31453]: pgmap v19666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:13 smithi153 ceph-mon[36569]: pgmap v19666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:13 smithi104 ceph-mon[36624]: pgmap v19666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:15 smithi081 ceph-mon[31453]: pgmap v19667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:15 smithi153 ceph-mon[36569]: pgmap v19667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:15 smithi104 ceph-mon[36624]: pgmap v19667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:17 smithi081 ceph-mon[31453]: pgmap v19668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:31:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:17 smithi153 ceph-mon[36569]: pgmap v19668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:31:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:17 smithi104 ceph-mon[36624]: pgmap v19668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:31:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:18.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:18.812+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:31:18.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:18.812+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:31:18.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:18.812+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 284 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:31:18.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:18.812+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 284 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:31:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:31:19.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:31:19.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:31:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:31:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:31:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:31:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:31:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:19 smithi081 ceph-mon[31453]: pgmap v19669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:19 smithi081 ceph-mon[31453]: pgmap v19670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:19 smithi153 ceph-mon[36569]: pgmap v19669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:19 smithi153 ceph-mon[36569]: pgmap v19670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:19 smithi104 ceph-mon[36624]: pgmap v19669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:19 smithi104 ceph-mon[36624]: pgmap v19670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:20.949+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:31:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:20.949+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:31:20.949 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:20.949+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:31:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:21 smithi081 ceph-mon[31453]: pgmap v19671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:21 smithi153 ceph-mon[36569]: pgmap v19671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:21 smithi104 ceph-mon[36624]: pgmap v19671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:23 smithi081 ceph-mon[31453]: pgmap v19672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:23 smithi153 ceph-mon[36569]: pgmap v19672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:23 smithi104 ceph-mon[36624]: pgmap v19672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:25 smithi153 ceph-mon[36569]: pgmap v19673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:25 smithi104 ceph-mon[36624]: pgmap v19673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:25 smithi081 ceph-mon[31453]: pgmap v19673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:27 smithi153 ceph-mon[36569]: pgmap v19674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:27 smithi104 ceph-mon[36624]: pgmap v19674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:27 smithi081 ceph-mon[31453]: pgmap v19674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:29 smithi153 ceph-mon[36569]: pgmap v19675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:29 smithi104 ceph-mon[36624]: pgmap v19675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:29 smithi081 ceph-mon[31453]: pgmap v19675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:32 smithi153 ceph-mon[36569]: pgmap v19676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:32 smithi104 ceph-mon[36624]: pgmap v19676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:32 smithi081 ceph-mon[31453]: pgmap v19676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:34 smithi153 ceph-mon[36569]: pgmap v19677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:34 smithi104 ceph-mon[36624]: pgmap v19677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:34 smithi081 ceph-mon[31453]: pgmap v19677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:35.962+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:31:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:35.962+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:31:35.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:35.962+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:31:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:36 smithi153 ceph-mon[36569]: pgmap v19678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:36 smithi104 ceph-mon[36624]: pgmap v19678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:36 smithi081 ceph-mon[31453]: pgmap v19678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:38 smithi153 ceph-mon[36569]: pgmap v19679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:38 smithi104 ceph-mon[36624]: pgmap v19679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:38 smithi081 ceph-mon[31453]: pgmap v19679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:40 smithi104 ceph-mon[36624]: pgmap v19680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:40 smithi081 ceph-mon[31453]: pgmap v19680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:40 smithi153 ceph-mon[36569]: pgmap v19680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:42 smithi104 ceph-mon[36624]: pgmap v19681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:42 smithi081 ceph-mon[31453]: pgmap v19681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:42 smithi153 ceph-mon[36569]: pgmap v19681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:44 smithi104 ceph-mon[36624]: pgmap v19682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:44 smithi081 ceph-mon[31453]: pgmap v19682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:44 smithi153 ceph-mon[36569]: pgmap v19682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:46 smithi104 ceph-mon[36624]: pgmap v19683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:46 smithi081 ceph-mon[31453]: pgmap v19683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:46 smithi153 ceph-mon[36569]: pgmap v19683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:48 smithi104 ceph-mon[36624]: pgmap v19684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:48 smithi081 ceph-mon[31453]: pgmap v19684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:48 smithi153 ceph-mon[36569]: pgmap v19684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:50 smithi081 ceph-mon[31453]: pgmap v19685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:50 smithi153 ceph-mon[36569]: pgmap v19685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:50 smithi104 ceph-mon[36624]: pgmap v19685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:50.965+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:31:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:50.965+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:31:50.965 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:31:50.965+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:31:52.359 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:52 smithi153 ceph-mon[36569]: pgmap v19686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:52 smithi081 ceph-mon[31453]: pgmap v19686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:52 smithi104 ceph-mon[36624]: pgmap v19686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:54 smithi081 ceph-mon[31453]: pgmap v19687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:54 smithi153 ceph-mon[36569]: pgmap v19687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:54 smithi104 ceph-mon[36624]: pgmap v19687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:56 smithi081 ceph-mon[31453]: pgmap v19688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:56 smithi153 ceph-mon[36569]: pgmap v19688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:56 smithi104 ceph-mon[36624]: pgmap v19688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:31:58 smithi081 ceph-mon[31453]: pgmap v19689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:31:58 smithi153 ceph-mon[36569]: pgmap v19689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:31:58.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:31:58 smithi104 ceph-mon[36624]: pgmap v19689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:00 smithi081 ceph-mon[31453]: pgmap v19690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:00 smithi153 ceph-mon[36569]: pgmap v19690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:00 smithi104 ceph-mon[36624]: pgmap v19690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:02.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:02 smithi104 ceph-mon[36624]: pgmap v19691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:02 smithi081 ceph-mon[31453]: pgmap v19691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:02 smithi153 ceph-mon[36569]: pgmap v19691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:04 smithi081 ceph-mon[31453]: pgmap v19692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:04 smithi153 ceph-mon[36569]: pgmap v19692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:04.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:04 smithi104 ceph-mon[36624]: pgmap v19692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:05.981+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:32:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:05.981+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:32:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:05.981+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:32:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:06 smithi081 ceph-mon[31453]: pgmap v19693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:06 smithi153 ceph-mon[36569]: pgmap v19693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:06 smithi104 ceph-mon[36624]: pgmap v19693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:08 smithi081 ceph-mon[31453]: pgmap v19694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:08 smithi153 ceph-mon[36569]: pgmap v19694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:08 smithi104 ceph-mon[36624]: pgmap v19694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:10 smithi081 ceph-mon[31453]: pgmap v19695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:10 smithi153 ceph-mon[36569]: pgmap v19695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:10 smithi104 ceph-mon[36624]: pgmap v19695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:12 smithi081 ceph-mon[31453]: pgmap v19696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:12 smithi153 ceph-mon[36569]: pgmap v19696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:12 smithi104 ceph-mon[36624]: pgmap v19696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:14 smithi081 ceph-mon[31453]: pgmap v19697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:14 smithi153 ceph-mon[36569]: pgmap v19697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:14 smithi104 ceph-mon[36624]: pgmap v19697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:16 smithi081 ceph-mon[31453]: pgmap v19698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:16 smithi153 ceph-mon[36569]: pgmap v19698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:16 smithi104 ceph-mon[36624]: pgmap v19698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:18 smithi081 ceph-mon[31453]: pgmap v19699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:18 smithi153 ceph-mon[36569]: pgmap v19699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:18 smithi104 ceph-mon[36624]: pgmap v19699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:32:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:32:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:32:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:20 smithi153 ceph-mon[36569]: pgmap v19700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:32:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:32:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:32:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:20 smithi104 ceph-mon[36624]: pgmap v19700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:32:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:32:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:32:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:20 smithi081 ceph-mon[31453]: pgmap v19700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:32:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:32:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:32:20.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:20.996+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:32:20.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:20.996+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:32:20.996 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:20.996+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:32:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:21 smithi153 ceph-mon[36569]: pgmap v19701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:21 smithi153 ceph-mon[36569]: pgmap v19702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:21 smithi104 ceph-mon[36624]: pgmap v19701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:21 smithi104 ceph-mon[36624]: pgmap v19702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:21 smithi081 ceph-mon[31453]: pgmap v19701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:21 smithi081 ceph-mon[31453]: pgmap v19702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:23 smithi153 ceph-mon[36569]: pgmap v19703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:23 smithi104 ceph-mon[36624]: pgmap v19703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:23.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:23 smithi081 ceph-mon[31453]: pgmap v19703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:25.458 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:25 smithi081 ceph-mon[31453]: pgmap v19704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:25 smithi153 ceph-mon[36569]: pgmap v19704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:25 smithi104 ceph-mon[36624]: pgmap v19704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:27 smithi153 ceph-mon[36569]: pgmap v19705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:27 smithi104 ceph-mon[36624]: pgmap v19705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:27 smithi081 ceph-mon[31453]: pgmap v19705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:29 smithi153 ceph-mon[36569]: pgmap v19706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:29 smithi104 ceph-mon[36624]: pgmap v19706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:29.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:29 smithi081 ceph-mon[31453]: pgmap v19706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:30 smithi153 ceph-mon[36569]: pgmap v19707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:30 smithi104 ceph-mon[36624]: pgmap v19707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:30 smithi081 ceph-mon[31453]: pgmap v19707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:33 smithi104 ceph-mon[36624]: pgmap v19708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:33 smithi081 ceph-mon[31453]: pgmap v19708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:33 smithi153 ceph-mon[36569]: pgmap v19708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:35 smithi104 ceph-mon[36624]: pgmap v19709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:35 smithi081 ceph-mon[31453]: pgmap v19709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:35 smithi153 ceph-mon[36569]: pgmap v19709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:36.012+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:32:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:36.012+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:32:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:36.012+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:32:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:37 smithi104 ceph-mon[36624]: pgmap v19710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:37 smithi081 ceph-mon[31453]: pgmap v19710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:37 smithi153 ceph-mon[36569]: pgmap v19710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:39 smithi104 ceph-mon[36624]: pgmap v19711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:39 smithi081 ceph-mon[31453]: pgmap v19711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:39 smithi153 ceph-mon[36569]: pgmap v19711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:41 smithi081 ceph-mon[31453]: pgmap v19712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:41 smithi153 ceph-mon[36569]: pgmap v19712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:41 smithi104 ceph-mon[36624]: pgmap v19712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:43 smithi081 ceph-mon[31453]: pgmap v19713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:43 smithi153 ceph-mon[36569]: pgmap v19713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:43.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:43 smithi104 ceph-mon[36624]: pgmap v19713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:45 smithi081 ceph-mon[31453]: pgmap v19714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:45.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:45 smithi153 ceph-mon[36569]: pgmap v19714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:45.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:45 smithi104 ceph-mon[36624]: pgmap v19714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:47 smithi081 ceph-mon[31453]: pgmap v19715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:47 smithi153 ceph-mon[36569]: pgmap v19715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:47 smithi104 ceph-mon[36624]: pgmap v19715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:49 smithi081 ceph-mon[31453]: pgmap v19716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:49 smithi153 ceph-mon[36569]: pgmap v19716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:49 smithi104 ceph-mon[36624]: pgmap v19716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:51.014+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:32:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:51.014+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:32:51.014 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:32:51.014+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:32:51.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:51 smithi081 ceph-mon[31453]: pgmap v19717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:51 smithi153 ceph-mon[36569]: pgmap v19717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:51.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:51 smithi104 ceph-mon[36624]: pgmap v19717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:53 smithi081 ceph-mon[31453]: pgmap v19718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:53 smithi153 ceph-mon[36569]: pgmap v19718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:53.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:53 smithi104 ceph-mon[36624]: pgmap v19718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:55 smithi081 ceph-mon[31453]: pgmap v19719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:55 smithi153 ceph-mon[36569]: pgmap v19719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:55.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:55 smithi104 ceph-mon[36624]: pgmap v19719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:57 smithi081 ceph-mon[31453]: pgmap v19720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:57 smithi153 ceph-mon[36569]: pgmap v19720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:57.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:57 smithi104 ceph-mon[36624]: pgmap v19720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:32:59 smithi081 ceph-mon[31453]: pgmap v19721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:32:59 smithi153 ceph-mon[36569]: pgmap v19721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:32:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:32:59 smithi104 ceph-mon[36624]: pgmap v19721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:01.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:01 smithi081 ceph-mon[31453]: pgmap v19722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:01 smithi153 ceph-mon[36569]: pgmap v19722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:01 smithi104 ceph-mon[36624]: pgmap v19722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:03 smithi081 ceph-mon[31453]: pgmap v19723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:03 smithi153 ceph-mon[36569]: pgmap v19723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:03 smithi104 ceph-mon[36624]: pgmap v19723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:05.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:05 smithi081 ceph-mon[31453]: pgmap v19724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:05 smithi153 ceph-mon[36569]: pgmap v19724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:05 smithi104 ceph-mon[36624]: pgmap v19724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:06.029+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:33:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:06.029+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:33:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:06.029+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:33:07.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:07 smithi081 ceph-mon[31453]: pgmap v19725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:07 smithi153 ceph-mon[36569]: pgmap v19725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:07 smithi104 ceph-mon[36624]: pgmap v19725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:09 smithi081 ceph-mon[31453]: pgmap v19726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:09 smithi153 ceph-mon[36569]: pgmap v19726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:09 smithi104 ceph-mon[36624]: pgmap v19726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:11 smithi153 ceph-mon[36569]: pgmap v19727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:11 smithi104 ceph-mon[36624]: pgmap v19727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:11 smithi081 ceph-mon[31453]: pgmap v19727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:13 smithi153 ceph-mon[36569]: pgmap v19728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:13 smithi104 ceph-mon[36624]: pgmap v19728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:13 smithi081 ceph-mon[31453]: pgmap v19728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:15 smithi081 ceph-mon[31453]: pgmap v19729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:15 smithi153 ceph-mon[36569]: pgmap v19729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:15 smithi104 ceph-mon[36624]: pgmap v19729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:17 smithi153 ceph-mon[36569]: pgmap v19730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:17 smithi104 ceph-mon[36624]: pgmap v19730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:17.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:17 smithi081 ceph-mon[31453]: pgmap v19730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:19 smithi153 ceph-mon[36569]: pgmap v19731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:19 smithi104 ceph-mon[36624]: pgmap v19731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:19 smithi081 ceph-mon[31453]: pgmap v19731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:20 smithi153 ceph-mon[36569]: pgmap v19732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:33:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:20 smithi104 ceph-mon[36624]: pgmap v19732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:33:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:20 smithi081 ceph-mon[31453]: pgmap v19732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:33:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:21.042+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:33:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:21.042+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:33:21.043 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:21.042+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:33:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:33:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:33:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:33:21.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:33:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:33:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:33:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:33:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:33:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:33:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:22 smithi153 ceph-mon[36569]: pgmap v19733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:22 smithi153 ceph-mon[36569]: pgmap v19734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:22 smithi104 ceph-mon[36624]: pgmap v19733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:22.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:22 smithi104 ceph-mon[36624]: pgmap v19734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:22 smithi081 ceph-mon[31453]: pgmap v19733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:22 smithi081 ceph-mon[31453]: pgmap v19734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:24 smithi153 ceph-mon[36569]: pgmap v19735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:24 smithi104 ceph-mon[36624]: pgmap v19735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:24 smithi081 ceph-mon[31453]: pgmap v19735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:26 smithi153 ceph-mon[36569]: pgmap v19736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:26 smithi104 ceph-mon[36624]: pgmap v19736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:26.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:26 smithi081 ceph-mon[31453]: pgmap v19736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:28 smithi153 ceph-mon[36569]: pgmap v19737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:28 smithi104 ceph-mon[36624]: pgmap v19737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:28 smithi081 ceph-mon[31453]: pgmap v19737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:30 smithi104 ceph-mon[36624]: pgmap v19738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:30 smithi081 ceph-mon[31453]: pgmap v19738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:30 smithi153 ceph-mon[36569]: pgmap v19738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:32 smithi104 ceph-mon[36624]: pgmap v19739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:32.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:32 smithi081 ceph-mon[31453]: pgmap v19739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:32 smithi153 ceph-mon[36569]: pgmap v19739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:34 smithi104 ceph-mon[36624]: pgmap v19740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:34 smithi081 ceph-mon[31453]: pgmap v19740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:34 smithi153 ceph-mon[36569]: pgmap v19740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:36.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:36.052+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:33:36.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:36.052+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:33:36.052 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:36.052+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:33:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:36 smithi104 ceph-mon[36624]: pgmap v19741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:36 smithi081 ceph-mon[31453]: pgmap v19741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:36 smithi153 ceph-mon[36569]: pgmap v19741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:38 smithi081 ceph-mon[31453]: pgmap v19742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:38 smithi153 ceph-mon[36569]: pgmap v19742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:38 smithi104 ceph-mon[36624]: pgmap v19742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:40 smithi081 ceph-mon[31453]: pgmap v19743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:40 smithi153 ceph-mon[36569]: pgmap v19743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:40 smithi104 ceph-mon[36624]: pgmap v19743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:43 smithi081 ceph-mon[31453]: pgmap v19744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:43 smithi153 ceph-mon[36569]: pgmap v19744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:43.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:43 smithi104 ceph-mon[36624]: pgmap v19744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:45 smithi081 ceph-mon[31453]: pgmap v19745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:45 smithi153 ceph-mon[36569]: pgmap v19745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:45.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:45 smithi104 ceph-mon[36624]: pgmap v19745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:47 smithi081 ceph-mon[31453]: pgmap v19746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:47 smithi153 ceph-mon[36569]: pgmap v19746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:47 smithi104 ceph-mon[36624]: pgmap v19746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:49.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:49 smithi081 ceph-mon[31453]: pgmap v19747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:49 smithi153 ceph-mon[36569]: pgmap v19747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:49.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:49 smithi104 ceph-mon[36624]: pgmap v19747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:51.058+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:33:51.058 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:51.058+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:33:51.059 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:33:51.058+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:33:51.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:51 smithi081 ceph-mon[31453]: pgmap v19748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:51 smithi153 ceph-mon[36569]: pgmap v19748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:51.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:51 smithi104 ceph-mon[36624]: pgmap v19748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:53 smithi081 ceph-mon[31453]: pgmap v19749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:53 smithi153 ceph-mon[36569]: pgmap v19749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:53.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:53 smithi104 ceph-mon[36624]: pgmap v19749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:55.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:55 smithi081 ceph-mon[31453]: pgmap v19750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:55 smithi153 ceph-mon[36569]: pgmap v19750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:55.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:55 smithi104 ceph-mon[36624]: pgmap v19750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:57 smithi081 ceph-mon[31453]: pgmap v19751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:57 smithi153 ceph-mon[36569]: pgmap v19751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:57.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:57 smithi104 ceph-mon[36624]: pgmap v19751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:59.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:33:59 smithi081 ceph-mon[31453]: pgmap v19752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:33:59 smithi153 ceph-mon[36569]: pgmap v19752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:33:59.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:33:59 smithi104 ceph-mon[36624]: pgmap v19752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:01.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:01 smithi081 ceph-mon[31453]: pgmap v19753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:01 smithi153 ceph-mon[36569]: pgmap v19753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:01.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:01 smithi104 ceph-mon[36624]: pgmap v19753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:03 smithi081 ceph-mon[31453]: pgmap v19754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:03 smithi153 ceph-mon[36569]: pgmap v19754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:03 smithi104 ceph-mon[36624]: pgmap v19754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:05.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:05 smithi081 ceph-mon[31453]: pgmap v19755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:05 smithi153 ceph-mon[36569]: pgmap v19755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:05 smithi104 ceph-mon[36624]: pgmap v19755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:06.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:06.073+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:34:06.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:06.073+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:34:06.074 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:06.073+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:34:07.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:07 smithi081 ceph-mon[31453]: pgmap v19756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:07 smithi153 ceph-mon[36569]: pgmap v19756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:07 smithi104 ceph-mon[36624]: pgmap v19756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:09 smithi153 ceph-mon[36569]: pgmap v19757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:09 smithi104 ceph-mon[36624]: pgmap v19757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:09 smithi081 ceph-mon[31453]: pgmap v19757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:11 smithi153 ceph-mon[36569]: pgmap v19758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:11 smithi104 ceph-mon[36624]: pgmap v19758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:11 smithi081 ceph-mon[31453]: pgmap v19758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:13 smithi153 ceph-mon[36569]: pgmap v19759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:13 smithi104 ceph-mon[36624]: pgmap v19759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:13 smithi081 ceph-mon[31453]: pgmap v19759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:15 smithi153 ceph-mon[36569]: pgmap v19760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:15 smithi104 ceph-mon[36624]: pgmap v19760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:15 smithi081 ceph-mon[31453]: pgmap v19760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:17 smithi081 ceph-mon[31453]: pgmap v19761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:17 smithi153 ceph-mon[36569]: pgmap v19761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:17 smithi104 ceph-mon[36624]: pgmap v19761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:18.842 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:18.842+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:34:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:18.842+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:34:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:18.843+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 285 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:34:18.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:18.843+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 285 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:34:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:19 smithi153 ceph-mon[36569]: pgmap v19762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:19 smithi104 ceph-mon[36624]: pgmap v19762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:19 smithi081 ceph-mon[31453]: pgmap v19762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:21.079+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:34:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:21.079+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:34:21.079 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:21.079+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:34:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:21 smithi153 ceph-mon[36569]: pgmap v19763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:21 smithi104 ceph-mon[36624]: pgmap v19763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:21 smithi081 ceph-mon[31453]: pgmap v19763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:34:22.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:34:22.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:34:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:23 smithi153 ceph-mon[36569]: pgmap v19764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:34:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:34:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:34:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:23 smithi104 ceph-mon[36624]: pgmap v19764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:34:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:34:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:34:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:23 smithi081 ceph-mon[31453]: pgmap v19764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:34:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:34:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:34:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:24 smithi153 ceph-mon[36569]: pgmap v19765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:24 smithi153 ceph-mon[36569]: pgmap v19766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:24 smithi104 ceph-mon[36624]: pgmap v19765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:24 smithi104 ceph-mon[36624]: pgmap v19766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:24 smithi081 ceph-mon[31453]: pgmap v19765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:24 smithi081 ceph-mon[31453]: pgmap v19766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:26 smithi153 ceph-mon[36569]: pgmap v19767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:26 smithi104 ceph-mon[36624]: pgmap v19767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:26 smithi081 ceph-mon[31453]: pgmap v19767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:27 smithi153 ceph-mon[36569]: pgmap v19768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:27.557 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:27 smithi081 ceph-mon[31453]: pgmap v19768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:27 smithi104 ceph-mon[36624]: pgmap v19768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:30 smithi104 ceph-mon[36624]: pgmap v19769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:30 smithi081 ceph-mon[31453]: pgmap v19769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:30 smithi153 ceph-mon[36569]: pgmap v19769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:32.327 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:32 smithi081 ceph-mon[31453]: pgmap v19770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:32 smithi104 ceph-mon[36624]: pgmap v19770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:32 smithi153 ceph-mon[36569]: pgmap v19770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:34 smithi104 ceph-mon[36624]: pgmap v19771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:34.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:34 smithi081 ceph-mon[31453]: pgmap v19771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:34 smithi153 ceph-mon[36569]: pgmap v19771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:36.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:36.092+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:34:36.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:36.092+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:34:36.092 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:36.092+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:34:36.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:36 smithi104 ceph-mon[36624]: pgmap v19772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:36.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:36 smithi081 ceph-mon[31453]: pgmap v19772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:36 smithi153 ceph-mon[36569]: pgmap v19772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:38 smithi081 ceph-mon[31453]: pgmap v19773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:38 smithi153 ceph-mon[36569]: pgmap v19773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:38 smithi104 ceph-mon[36624]: pgmap v19773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:40.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:40 smithi081 ceph-mon[31453]: pgmap v19774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:40 smithi153 ceph-mon[36569]: pgmap v19774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:40 smithi104 ceph-mon[36624]: pgmap v19774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:42.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:42 smithi081 ceph-mon[31453]: pgmap v19775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:42 smithi153 ceph-mon[36569]: pgmap v19775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:42 smithi104 ceph-mon[36624]: pgmap v19775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:44 smithi081 ceph-mon[31453]: pgmap v19776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:44 smithi153 ceph-mon[36569]: pgmap v19776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:44 smithi104 ceph-mon[36624]: pgmap v19776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:46.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:46 smithi081 ceph-mon[31453]: pgmap v19777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:46 smithi153 ceph-mon[36569]: pgmap v19777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:46 smithi104 ceph-mon[36624]: pgmap v19777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:48.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:48 smithi081 ceph-mon[31453]: pgmap v19778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:48 smithi153 ceph-mon[36569]: pgmap v19778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:48 smithi104 ceph-mon[36624]: pgmap v19778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:50 smithi081 ceph-mon[31453]: pgmap v19779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:50 smithi153 ceph-mon[36569]: pgmap v19779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:50.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:50 smithi104 ceph-mon[36624]: pgmap v19779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:51.107+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:34:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:51.107+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:34:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:34:51.107+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:34:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:52 smithi081 ceph-mon[31453]: pgmap v19780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:52 smithi153 ceph-mon[36569]: pgmap v19780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:52.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:52 smithi104 ceph-mon[36624]: pgmap v19780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:54 smithi081 ceph-mon[31453]: pgmap v19781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:54 smithi153 ceph-mon[36569]: pgmap v19781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:54.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:54 smithi104 ceph-mon[36624]: pgmap v19781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:56 smithi081 ceph-mon[31453]: pgmap v19782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:56 smithi153 ceph-mon[36569]: pgmap v19782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:56.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:56 smithi104 ceph-mon[36624]: pgmap v19782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:58.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:34:58 smithi081 ceph-mon[31453]: pgmap v19783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:34:58 smithi153 ceph-mon[36569]: pgmap v19783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:34:58.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:34:58 smithi104 ceph-mon[36624]: pgmap v19783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:00.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:00 smithi081 ceph-mon[31453]: pgmap v19784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:00 smithi153 ceph-mon[36569]: pgmap v19784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:00 smithi104 ceph-mon[36624]: pgmap v19784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:02 smithi081 ceph-mon[31453]: pgmap v19785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:02 smithi153 ceph-mon[36569]: pgmap v19785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:02.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:02 smithi104 ceph-mon[36624]: pgmap v19785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:03.849 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:03.850+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:35:03.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:03.850+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:35:03.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:03.850+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 286 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:35:03.850 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:03.850+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 286 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:35:04.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:04 smithi081 ceph-mon[31453]: pgmap v19786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:04 smithi153 ceph-mon[36569]: pgmap v19786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:04.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:04 smithi104 ceph-mon[36624]: pgmap v19786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:06.110+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:35:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:06.110+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:35:06.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:06.110+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:35:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:06 smithi153 ceph-mon[36569]: pgmap v19787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:06 smithi104 ceph-mon[36624]: pgmap v19787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:06 smithi081 ceph-mon[31453]: pgmap v19787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:08 smithi153 ceph-mon[36569]: pgmap v19788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:08 smithi104 ceph-mon[36624]: pgmap v19788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:08.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:08 smithi081 ceph-mon[31453]: pgmap v19788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:10 smithi153 ceph-mon[36569]: pgmap v19789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:10.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:10 smithi104 ceph-mon[36624]: pgmap v19789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:10.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:10 smithi081 ceph-mon[31453]: pgmap v19789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:12 smithi153 ceph-mon[36569]: pgmap v19790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:12.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:12 smithi104 ceph-mon[36624]: pgmap v19790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:12 smithi081 ceph-mon[31453]: pgmap v19790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:14 smithi153 ceph-mon[36569]: pgmap v19791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:14 smithi104 ceph-mon[36624]: pgmap v19791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:14.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:14 smithi081 ceph-mon[31453]: pgmap v19791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:16 smithi153 ceph-mon[36569]: pgmap v19792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:16 smithi104 ceph-mon[36624]: pgmap v19792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:16.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:16 smithi081 ceph-mon[31453]: pgmap v19792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:18 smithi153 ceph-mon[36569]: pgmap v19793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:18 smithi104 ceph-mon[36624]: pgmap v19793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:18 smithi081 ceph-mon[31453]: pgmap v19793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:20 smithi153 ceph-mon[36569]: pgmap v19794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:20 smithi104 ceph-mon[36624]: pgmap v19794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:20 smithi081 ceph-mon[31453]: pgmap v19794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:21.125+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:35:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:21.125+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:35:21.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:21.125+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:35:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:22 smithi153 ceph-mon[36569]: pgmap v19795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:22.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:22 smithi104 ceph-mon[36624]: pgmap v19795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:22.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:22 smithi081 ceph-mon[31453]: pgmap v19795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:24 smithi153 ceph-mon[36569]: pgmap v19796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:35:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:24 smithi104 ceph-mon[36624]: pgmap v19796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:35:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:24 smithi081 ceph-mon[31453]: pgmap v19796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:35:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:35:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:35:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:35:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:35:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:35:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:35:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:35:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:26 smithi153 ceph-mon[36569]: pgmap v19797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:26 smithi153 ceph-mon[36569]: pgmap v19798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:26 smithi104 ceph-mon[36624]: pgmap v19797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:26 smithi104 ceph-mon[36624]: pgmap v19798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:26 smithi081 ceph-mon[31453]: pgmap v19797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:26 smithi081 ceph-mon[31453]: pgmap v19798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:28 smithi153 ceph-mon[36569]: pgmap v19799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:28 smithi104 ceph-mon[36624]: pgmap v19799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:28 smithi081 ceph-mon[31453]: pgmap v19799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:30 smithi153 ceph-mon[36569]: pgmap v19800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:30 smithi104 ceph-mon[36624]: pgmap v19800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:30 smithi081 ceph-mon[31453]: pgmap v19800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:32 smithi153 ceph-mon[36569]: pgmap v19801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:32 smithi104 ceph-mon[36624]: pgmap v19801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:32 smithi081 ceph-mon[31453]: pgmap v19801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:34 smithi153 ceph-mon[36569]: pgmap v19802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:34 smithi104 ceph-mon[36624]: pgmap v19802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:34 smithi081 ceph-mon[31453]: pgmap v19802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:35 smithi153 ceph-mon[36569]: pgmap v19803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:35 smithi104 ceph-mon[36624]: pgmap v19803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:35.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:35 smithi081 ceph-mon[31453]: pgmap v19803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:36.127 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:36.128+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:35:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:36.128+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:35:36.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:36.128+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:35:37.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:37 smithi081 ceph-mon[31453]: pgmap v19804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:37 smithi153 ceph-mon[36569]: pgmap v19804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:38.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:37 smithi104 ceph-mon[36624]: pgmap v19804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:39 smithi081 ceph-mon[31453]: pgmap v19805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:39 smithi153 ceph-mon[36569]: pgmap v19805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:40.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:39 smithi104 ceph-mon[36624]: pgmap v19805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:41.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:41 smithi081 ceph-mon[31453]: pgmap v19806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:41 smithi153 ceph-mon[36569]: pgmap v19806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:42.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:41 smithi104 ceph-mon[36624]: pgmap v19806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:43.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:43 smithi081 ceph-mon[31453]: pgmap v19807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:43 smithi153 ceph-mon[36569]: pgmap v19807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:44.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:43 smithi104 ceph-mon[36624]: pgmap v19807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:45 smithi153 ceph-mon[36569]: pgmap v19808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:45 smithi104 ceph-mon[36624]: pgmap v19808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:45 smithi081 ceph-mon[31453]: pgmap v19808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:47 smithi153 ceph-mon[36569]: pgmap v19809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:47 smithi104 ceph-mon[36624]: pgmap v19809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:47 smithi081 ceph-mon[31453]: pgmap v19809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:49 smithi153 ceph-mon[36569]: pgmap v19810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:49 smithi104 ceph-mon[36624]: pgmap v19810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:50.151 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:49 smithi081 ceph-mon[31453]: pgmap v19810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:51.143+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:35:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:51.143+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:35:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:35:51.143+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:35:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:51 smithi153 ceph-mon[36569]: pgmap v19811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:52.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:51 smithi104 ceph-mon[36624]: pgmap v19811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:52.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:51 smithi081 ceph-mon[31453]: pgmap v19811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:53 smithi153 ceph-mon[36569]: pgmap v19812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:54.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:53 smithi104 ceph-mon[36624]: pgmap v19812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:54.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:53 smithi081 ceph-mon[31453]: pgmap v19812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:55 smithi153 ceph-mon[36569]: pgmap v19813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:55 smithi104 ceph-mon[36624]: pgmap v19813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:55 smithi081 ceph-mon[31453]: pgmap v19813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:57 smithi153 ceph-mon[36569]: pgmap v19814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:57 smithi104 ceph-mon[36624]: pgmap v19814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:58.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:57 smithi081 ceph-mon[31453]: pgmap v19814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:35:59.957 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:35:59 smithi081 ceph-mon[31453]: pgmap v19815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:35:59 smithi153 ceph-mon[36569]: pgmap v19815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:35:59 smithi104 ceph-mon[36624]: pgmap v19815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:01 smithi153 ceph-mon[36569]: pgmap v19816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:02.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:01 smithi104 ceph-mon[36624]: pgmap v19816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:01 smithi081 ceph-mon[31453]: pgmap v19816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:03 smithi153 ceph-mon[36569]: pgmap v19817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:04.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:03 smithi104 ceph-mon[36624]: pgmap v19817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:04.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:03 smithi081 ceph-mon[31453]: pgmap v19817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:06.023 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:05 smithi081 ceph-mon[31453]: pgmap v19818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:05 smithi153 ceph-mon[36569]: pgmap v19818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:06.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:05 smithi104 ceph-mon[36624]: pgmap v19818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:06.151+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:36:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:06.151+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:36:06.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:06.151+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:36:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:07 smithi153 ceph-mon[36569]: pgmap v19819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:07 smithi104 ceph-mon[36624]: pgmap v19819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:07 smithi081 ceph-mon[31453]: pgmap v19819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:09 smithi153 ceph-mon[36569]: pgmap v19820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:09 smithi104 ceph-mon[36624]: pgmap v19820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:09 smithi081 ceph-mon[31453]: pgmap v19820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:11 smithi153 ceph-mon[36569]: pgmap v19821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:12.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:11 smithi104 ceph-mon[36624]: pgmap v19821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:11 smithi081 ceph-mon[31453]: pgmap v19821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:13 smithi153 ceph-mon[36569]: pgmap v19822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:14.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:13 smithi104 ceph-mon[36624]: pgmap v19822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:14.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:13 smithi081 ceph-mon[31453]: pgmap v19822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:15 smithi153 ceph-mon[36569]: pgmap v19823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:16.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:15 smithi104 ceph-mon[36624]: pgmap v19823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:16.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:15 smithi081 ceph-mon[31453]: pgmap v19823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:17 smithi153 ceph-mon[36569]: pgmap v19824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:18.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:17 smithi104 ceph-mon[36624]: pgmap v19824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:18.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:17 smithi081 ceph-mon[31453]: pgmap v19824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:19 smithi153 ceph-mon[36569]: pgmap v19825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:19 smithi104 ceph-mon[36624]: pgmap v19825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:20.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:19 smithi081 ceph-mon[31453]: pgmap v19825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:21.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:21.167+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:36:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:21.167+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:36:21.167 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:21.167+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:36:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:21 smithi153 ceph-mon[36569]: pgmap v19826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:22.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:21 smithi104 ceph-mon[36624]: pgmap v19826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:22.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:21 smithi081 ceph-mon[31453]: pgmap v19826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:23 smithi153 ceph-mon[36569]: pgmap v19827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:23 smithi104 ceph-mon[36624]: pgmap v19827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:24.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:23 smithi081 ceph-mon[31453]: pgmap v19827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: pgmap v19828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: pgmap v19828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: pgmap v19828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:36:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:36:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:36:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:36:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:36:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:36:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:36:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:36:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:36:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:36:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:36:27.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:36:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:27 smithi153 ceph-mon[36569]: pgmap v19829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:27 smithi153 ceph-mon[36569]: pgmap v19830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:28.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:27 smithi104 ceph-mon[36624]: pgmap v19829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:28.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:27 smithi104 ceph-mon[36624]: pgmap v19830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:27 smithi081 ceph-mon[31453]: pgmap v19829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:27 smithi081 ceph-mon[31453]: pgmap v19830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:29 smithi153 ceph-mon[36569]: pgmap v19831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:29 smithi104 ceph-mon[36624]: pgmap v19831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:29 smithi081 ceph-mon[31453]: pgmap v19831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:32.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:32 smithi081 ceph-mon[31453]: pgmap v19832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:32 smithi153 ceph-mon[36569]: pgmap v19832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:32 smithi104 ceph-mon[36624]: pgmap v19832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:34.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:34 smithi081 ceph-mon[31453]: pgmap v19833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:34 smithi153 ceph-mon[36569]: pgmap v19833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:34 smithi104 ceph-mon[36624]: pgmap v19833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:36.183+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:36:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:36.183+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:36:36.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:36.183+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:36:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:36 smithi153 ceph-mon[36569]: pgmap v19834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:36 smithi104 ceph-mon[36624]: pgmap v19834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:36 smithi081 ceph-mon[31453]: pgmap v19834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:38 smithi153 ceph-mon[36569]: pgmap v19835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:38 smithi104 ceph-mon[36624]: pgmap v19835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:38 smithi081 ceph-mon[31453]: pgmap v19835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:40 smithi153 ceph-mon[36569]: pgmap v19836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:40 smithi104 ceph-mon[36624]: pgmap v19836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:40.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:40 smithi081 ceph-mon[31453]: pgmap v19836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:42 smithi153 ceph-mon[36569]: pgmap v19837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:42 smithi104 ceph-mon[36624]: pgmap v19837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:42 smithi081 ceph-mon[31453]: pgmap v19837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:44 smithi153 ceph-mon[36569]: pgmap v19838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:44 smithi104 ceph-mon[36624]: pgmap v19838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:44.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:44 smithi081 ceph-mon[31453]: pgmap v19838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:46 smithi153 ceph-mon[36569]: pgmap v19839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:46.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:46 smithi104 ceph-mon[36624]: pgmap v19839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:46.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:46 smithi081 ceph-mon[31453]: pgmap v19839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:48 smithi153 ceph-mon[36569]: pgmap v19840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:48 smithi104 ceph-mon[36624]: pgmap v19840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:48 smithi081 ceph-mon[31453]: pgmap v19840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:50 smithi153 ceph-mon[36569]: pgmap v19841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:50.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:50 smithi104 ceph-mon[36624]: pgmap v19841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:50 smithi081 ceph-mon[31453]: pgmap v19841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:51.192+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:36:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:51.193+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:36:51.192 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:36:51.193+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:36:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:52 smithi081 ceph-mon[31453]: pgmap v19842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:52 smithi153 ceph-mon[36569]: pgmap v19842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:52.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:52 smithi104 ceph-mon[36624]: pgmap v19842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:54 smithi153 ceph-mon[36569]: pgmap v19843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:54.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:54 smithi104 ceph-mon[36624]: pgmap v19843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:54.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:54 smithi081 ceph-mon[31453]: pgmap v19843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:56 smithi153 ceph-mon[36569]: pgmap v19844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:56.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:56 smithi104 ceph-mon[36624]: pgmap v19844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:56.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:56 smithi081 ceph-mon[31453]: pgmap v19844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:36:58 smithi153 ceph-mon[36569]: pgmap v19845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:58.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:36:58 smithi104 ceph-mon[36624]: pgmap v19845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:36:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:36:58 smithi081 ceph-mon[31453]: pgmap v19845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:00 smithi153 ceph-mon[36569]: pgmap v19846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:00 smithi104 ceph-mon[36624]: pgmap v19846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:00.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:00 smithi081 ceph-mon[31453]: pgmap v19846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:02 smithi153 ceph-mon[36569]: pgmap v19847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:02.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:02 smithi104 ceph-mon[36624]: pgmap v19847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:02.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:02 smithi081 ceph-mon[31453]: pgmap v19847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:04 smithi153 ceph-mon[36569]: pgmap v19848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:04.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:04 smithi104 ceph-mon[36624]: pgmap v19848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:04.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:04 smithi081 ceph-mon[31453]: pgmap v19848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:06.202+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:37:06.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:06.202+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:37:06.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:06.202+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:37:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:06 smithi153 ceph-mon[36569]: pgmap v19849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:06.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:06 smithi104 ceph-mon[36624]: pgmap v19849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:06 smithi081 ceph-mon[31453]: pgmap v19849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:08 smithi153 ceph-mon[36569]: pgmap v19850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:08 smithi104 ceph-mon[36624]: pgmap v19850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:08.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:08 smithi081 ceph-mon[31453]: pgmap v19850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:10 smithi153 ceph-mon[36569]: pgmap v19851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:10.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:10 smithi104 ceph-mon[36624]: pgmap v19851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:10.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:10 smithi081 ceph-mon[31453]: pgmap v19851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:12 smithi153 ceph-mon[36569]: pgmap v19852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:12.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:12 smithi104 ceph-mon[36624]: pgmap v19852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:12 smithi081 ceph-mon[31453]: pgmap v19852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:14 smithi153 ceph-mon[36569]: pgmap v19853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:14 smithi104 ceph-mon[36624]: pgmap v19853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:14.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:14 smithi081 ceph-mon[31453]: pgmap v19853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:16 smithi153 ceph-mon[36569]: pgmap v19854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:16 smithi104 ceph-mon[36624]: pgmap v19854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:16.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:16 smithi081 ceph-mon[31453]: pgmap v19854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:17 smithi104 ceph-mon[36624]: pgmap v19855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:17.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:17 smithi081 ceph-mon[31453]: pgmap v19855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:17 smithi153 ceph-mon[36569]: pgmap v19855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:18.874+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:37:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:18.874+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:37:18.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:18.874+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 287 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:37:18.874 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:18.874+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 287 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:37:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:19 smithi104 ceph-mon[36624]: pgmap v19856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:19 smithi081 ceph-mon[31453]: pgmap v19856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:19 smithi153 ceph-mon[36569]: pgmap v19856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:21.203+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:37:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:21.203+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:37:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:21.203+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:37:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:21 smithi104 ceph-mon[36624]: pgmap v19857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:21 smithi081 ceph-mon[31453]: pgmap v19857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:21 smithi153 ceph-mon[36569]: pgmap v19857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:23 smithi104 ceph-mon[36624]: pgmap v19858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:23.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:23 smithi081 ceph-mon[31453]: pgmap v19858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:23 smithi153 ceph-mon[36569]: pgmap v19858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:25.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:25 smithi081 ceph-mon[31453]: pgmap v19859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:25 smithi153 ceph-mon[36569]: pgmap v19859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:25.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:25 smithi104 ceph-mon[36624]: pgmap v19859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:27 smithi081 ceph-mon[31453]: pgmap v19860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:37:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:27 smithi153 ceph-mon[36569]: pgmap v19860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:37:27.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:27 smithi104 ceph-mon[36624]: pgmap v19860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:27.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:37:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:37:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:37:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:28 smithi081 ceph-mon[31453]: pgmap v19861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:28 smithi081 ceph-mon[31453]: pgmap v19862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:37:28.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:37:28.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:37:28.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:28 smithi153 ceph-mon[36569]: pgmap v19861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:28.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:28 smithi153 ceph-mon[36569]: pgmap v19862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:28.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:37:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:37:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:37:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:28 smithi104 ceph-mon[36624]: pgmap v19861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:28 smithi104 ceph-mon[36624]: pgmap v19862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:37:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:30 smithi104 ceph-mon[36624]: pgmap v19863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:30 smithi081 ceph-mon[31453]: pgmap v19863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:30 smithi153 ceph-mon[36569]: pgmap v19863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:32 smithi081 ceph-mon[31453]: pgmap v19864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:32 smithi153 ceph-mon[36569]: pgmap v19864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:33.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:32 smithi104 ceph-mon[36624]: pgmap v19864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:34 smithi081 ceph-mon[31453]: pgmap v19865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:34 smithi153 ceph-mon[36569]: pgmap v19865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:35.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:34 smithi104 ceph-mon[36624]: pgmap v19865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:36.207+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:37:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:36.207+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:37:36.207 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:36.207+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:37:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:36 smithi081 ceph-mon[31453]: pgmap v19866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:36 smithi153 ceph-mon[36569]: pgmap v19866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:36 smithi104 ceph-mon[36624]: pgmap v19866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:38 smithi081 ceph-mon[31453]: pgmap v19867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:38 smithi153 ceph-mon[36569]: pgmap v19867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:38 smithi104 ceph-mon[36624]: pgmap v19867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:41.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:40 smithi104 ceph-mon[36624]: pgmap v19868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:40 smithi081 ceph-mon[31453]: pgmap v19868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:40 smithi153 ceph-mon[36569]: pgmap v19868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:42 smithi081 ceph-mon[31453]: pgmap v19869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:42 smithi153 ceph-mon[36569]: pgmap v19869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:42 smithi104 ceph-mon[36624]: pgmap v19869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:44 smithi081 ceph-mon[31453]: pgmap v19870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:44 smithi153 ceph-mon[36569]: pgmap v19870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:44 smithi104 ceph-mon[36624]: pgmap v19870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:46 smithi081 ceph-mon[31453]: pgmap v19871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:46 smithi153 ceph-mon[36569]: pgmap v19871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:46 smithi104 ceph-mon[36624]: pgmap v19871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:48 smithi081 ceph-mon[31453]: pgmap v19872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:48 smithi153 ceph-mon[36569]: pgmap v19872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:48 smithi104 ceph-mon[36624]: pgmap v19872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:50 smithi081 ceph-mon[31453]: pgmap v19873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:51.221 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:51.222+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:37:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:51.222+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:37:51.222 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:37:51.222+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:37:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:50 smithi153 ceph-mon[36569]: pgmap v19873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:50 smithi104 ceph-mon[36624]: pgmap v19873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:52 smithi081 ceph-mon[31453]: pgmap v19874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:53.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:52 smithi153 ceph-mon[36569]: pgmap v19874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:52 smithi104 ceph-mon[36624]: pgmap v19874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:54 smithi081 ceph-mon[31453]: pgmap v19875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:55.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:54 smithi153 ceph-mon[36569]: pgmap v19875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:54 smithi104 ceph-mon[36624]: pgmap v19875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:56 smithi081 ceph-mon[31453]: pgmap v19876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:56 smithi153 ceph-mon[36569]: pgmap v19876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:57.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:56 smithi104 ceph-mon[36624]: pgmap v19876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:37:58 smithi081 ceph-mon[31453]: pgmap v19877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:37:58 smithi153 ceph-mon[36569]: pgmap v19877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:37:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:37:58 smithi104 ceph-mon[36624]: pgmap v19877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:00 smithi153 ceph-mon[36569]: pgmap v19878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:01.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:00 smithi104 ceph-mon[36624]: pgmap v19878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:01.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:00 smithi081 ceph-mon[31453]: pgmap v19878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:03.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:02 smithi081 ceph-mon[31453]: pgmap v19879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:03.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:02 smithi153 ceph-mon[36569]: pgmap v19879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:02 smithi104 ceph-mon[36624]: pgmap v19879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:04 smithi153 ceph-mon[36569]: pgmap v19880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:04 smithi104 ceph-mon[36624]: pgmap v19880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:05.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:04 smithi081 ceph-mon[31453]: pgmap v19880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:06.229+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:38:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:06.229+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:38:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:06.229+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:38:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:06 smithi153 ceph-mon[36569]: pgmap v19881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:07.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:06 smithi104 ceph-mon[36624]: pgmap v19881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:07.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:06 smithi081 ceph-mon[31453]: pgmap v19881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:08 smithi153 ceph-mon[36569]: pgmap v19882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:08 smithi104 ceph-mon[36624]: pgmap v19882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:08 smithi081 ceph-mon[31453]: pgmap v19882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:10 smithi153 ceph-mon[36569]: pgmap v19883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:10 smithi104 ceph-mon[36624]: pgmap v19883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:10 smithi081 ceph-mon[31453]: pgmap v19883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:12 smithi153 ceph-mon[36569]: pgmap v19884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:13.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:12 smithi104 ceph-mon[36624]: pgmap v19884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:12 smithi081 ceph-mon[31453]: pgmap v19884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:14 smithi153 ceph-mon[36569]: pgmap v19885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:15.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:14 smithi104 ceph-mon[36624]: pgmap v19885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:14 smithi081 ceph-mon[31453]: pgmap v19885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:16 smithi153 ceph-mon[36569]: pgmap v19886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:17.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:16 smithi104 ceph-mon[36624]: pgmap v19886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:16 smithi081 ceph-mon[31453]: pgmap v19886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:18 smithi153 ceph-mon[36569]: pgmap v19887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:19.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:18 smithi104 ceph-mon[36624]: pgmap v19887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:18 smithi081 ceph-mon[31453]: pgmap v19887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:21.243+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:38:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:21.243+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:38:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:21.243+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:38:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:20 smithi153 ceph-mon[36569]: pgmap v19888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:21.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:20 smithi104 ceph-mon[36624]: pgmap v19888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:20 smithi081 ceph-mon[31453]: pgmap v19888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:22 smithi153 ceph-mon[36569]: pgmap v19889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:23.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:22 smithi104 ceph-mon[36624]: pgmap v19889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:23.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:22 smithi081 ceph-mon[31453]: pgmap v19889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:25.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:24 smithi081 ceph-mon[31453]: pgmap v19890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:24 smithi153 ceph-mon[36569]: pgmap v19890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:24 smithi104 ceph-mon[36624]: pgmap v19890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:26 smithi153 ceph-mon[36569]: pgmap v19891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:26 smithi104 ceph-mon[36624]: pgmap v19891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:26 smithi081 ceph-mon[31453]: pgmap v19891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:28 smithi153 ceph-mon[36569]: pgmap v19892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:38:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:28 smithi104 ceph-mon[36624]: pgmap v19892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:38:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:28 smithi081 ceph-mon[31453]: pgmap v19892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:38:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:38:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:38:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:38:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:38:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:38:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:38:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:38:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:38:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:38:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:31 smithi153 ceph-mon[36569]: pgmap v19893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:31 smithi153 ceph-mon[36569]: pgmap v19894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:31 smithi104 ceph-mon[36624]: pgmap v19893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:31 smithi104 ceph-mon[36624]: pgmap v19894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:31 smithi081 ceph-mon[31453]: pgmap v19893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:31 smithi081 ceph-mon[31453]: pgmap v19894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:33 smithi153 ceph-mon[36569]: pgmap v19895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:33.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:33 smithi104 ceph-mon[36624]: pgmap v19895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:33 smithi081 ceph-mon[31453]: pgmap v19895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:35 smithi153 ceph-mon[36569]: pgmap v19896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:35.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:35 smithi104 ceph-mon[36624]: pgmap v19896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:35 smithi081 ceph-mon[31453]: pgmap v19896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:36.259+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:38:36.258 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:36.259+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:38:36.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:36.259+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:38:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:37 smithi153 ceph-mon[36569]: pgmap v19897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:37 smithi104 ceph-mon[36624]: pgmap v19897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:37 smithi081 ceph-mon[31453]: pgmap v19897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:39 smithi153 ceph-mon[36569]: pgmap v19898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:39 smithi104 ceph-mon[36624]: pgmap v19898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:39 smithi081 ceph-mon[31453]: pgmap v19898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:41 smithi153 ceph-mon[36569]: pgmap v19899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:41.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:41 smithi104 ceph-mon[36624]: pgmap v19899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:41 smithi081 ceph-mon[31453]: pgmap v19899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:43 smithi153 ceph-mon[36569]: pgmap v19900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:43 smithi104 ceph-mon[36624]: pgmap v19900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:43 smithi081 ceph-mon[31453]: pgmap v19900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:45 smithi153 ceph-mon[36569]: pgmap v19901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:45 smithi104 ceph-mon[36624]: pgmap v19901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:45 smithi081 ceph-mon[31453]: pgmap v19901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:47 smithi153 ceph-mon[36569]: pgmap v19902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:47 smithi104 ceph-mon[36624]: pgmap v19902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:47 smithi081 ceph-mon[31453]: pgmap v19902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:49 smithi153 ceph-mon[36569]: pgmap v19903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:49 smithi104 ceph-mon[36624]: pgmap v19903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:49.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:49 smithi081 ceph-mon[31453]: pgmap v19903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:51.274+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:51.274+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:38:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:38:51.274+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:38:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:51 smithi153 ceph-mon[36569]: pgmap v19904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:51 smithi104 ceph-mon[36624]: pgmap v19904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:51.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:51 smithi081 ceph-mon[31453]: pgmap v19904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:53 smithi104 ceph-mon[36624]: pgmap v19905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:53 smithi081 ceph-mon[31453]: pgmap v19905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:53 smithi153 ceph-mon[36569]: pgmap v19905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:55 smithi104 ceph-mon[36624]: pgmap v19906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:55.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:55 smithi081 ceph-mon[31453]: pgmap v19906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:55 smithi153 ceph-mon[36569]: pgmap v19906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:57.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:57 smithi104 ceph-mon[36624]: pgmap v19907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:57.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:57 smithi081 ceph-mon[31453]: pgmap v19907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:57 smithi153 ceph-mon[36569]: pgmap v19907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:38:59 smithi104 ceph-mon[36624]: pgmap v19908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:59.358 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:38:59 smithi153 ceph-mon[36569]: pgmap v19908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:38:59.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:38:59 smithi081 ceph-mon[31453]: pgmap v19908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:01.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:01 smithi104 ceph-mon[36624]: pgmap v19909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:01.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:01 smithi081 ceph-mon[31453]: pgmap v19909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:01 smithi153 ceph-mon[36569]: pgmap v19909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:03 smithi081 ceph-mon[31453]: pgmap v19910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:03 smithi153 ceph-mon[36569]: pgmap v19910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:03 smithi104 ceph-mon[36624]: pgmap v19910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:05.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:05 smithi081 ceph-mon[31453]: pgmap v19911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:05 smithi153 ceph-mon[36569]: pgmap v19911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:05 smithi104 ceph-mon[36624]: pgmap v19911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:06.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:06.290+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:39:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:06.290+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:39:06.290 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:06.290+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:39:07.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:07 smithi081 ceph-mon[31453]: pgmap v19912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:07 smithi153 ceph-mon[36569]: pgmap v19912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:07.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:07 smithi104 ceph-mon[36624]: pgmap v19912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:09 smithi081 ceph-mon[31453]: pgmap v19913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:09 smithi153 ceph-mon[36569]: pgmap v19913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:09 smithi104 ceph-mon[36624]: pgmap v19913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:11.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:11 smithi104 ceph-mon[36624]: pgmap v19914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:11 smithi081 ceph-mon[31453]: pgmap v19914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:11 smithi153 ceph-mon[36569]: pgmap v19914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:13 smithi081 ceph-mon[31453]: pgmap v19915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:13 smithi153 ceph-mon[36569]: pgmap v19915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:13 smithi104 ceph-mon[36624]: pgmap v19915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:15 smithi081 ceph-mon[31453]: pgmap v19916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:15 smithi153 ceph-mon[36569]: pgmap v19916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:15 smithi104 ceph-mon[36624]: pgmap v19916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:17 smithi081 ceph-mon[31453]: pgmap v19917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:17 smithi153 ceph-mon[36569]: pgmap v19917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:17 smithi104 ceph-mon[36624]: pgmap v19917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:19 smithi081 ceph-mon[31453]: pgmap v19918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:19 smithi153 ceph-mon[36569]: pgmap v19918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:19 smithi104 ceph-mon[36624]: pgmap v19918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:21.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:21.301+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:39:21.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:21.301+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:39:21.302 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:21.301+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:39:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:21 smithi081 ceph-mon[31453]: pgmap v19919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:21 smithi153 ceph-mon[36569]: pgmap v19919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:21 smithi104 ceph-mon[36624]: pgmap v19919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:23.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:23 smithi081 ceph-mon[31453]: pgmap v19920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:23 smithi153 ceph-mon[36569]: pgmap v19920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:23 smithi104 ceph-mon[36624]: pgmap v19920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:25 smithi081 ceph-mon[31453]: pgmap v19921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:25 smithi153 ceph-mon[36569]: pgmap v19921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:25.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:25 smithi104 ceph-mon[36624]: pgmap v19921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:27 smithi081 ceph-mon[31453]: pgmap v19922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:27 smithi153 ceph-mon[36569]: pgmap v19922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:27 smithi104 ceph-mon[36624]: pgmap v19922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:29 smithi081 ceph-mon[31453]: pgmap v19923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:29 smithi153 ceph-mon[36569]: pgmap v19923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:29 smithi104 ceph-mon[36624]: pgmap v19923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:39:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:39:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:39:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:31 smithi153 ceph-mon[36569]: pgmap v19924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:31.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:31 smithi104 ceph-mon[36624]: pgmap v19924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:31.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:31 smithi081 ceph-mon[31453]: pgmap v19924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:32 smithi153 ceph-mon[36569]: pgmap v19925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:39:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:39:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:32 smithi153 ceph-mon[36569]: pgmap v19926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:32 smithi153 ceph-mon[36569]: pgmap v19927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:32 smithi104 ceph-mon[36624]: pgmap v19925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:39:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:39:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:32 smithi104 ceph-mon[36624]: pgmap v19926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:32 smithi104 ceph-mon[36624]: pgmap v19927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:32 smithi081 ceph-mon[31453]: pgmap v19925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:39:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:39:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:32 smithi081 ceph-mon[31453]: pgmap v19926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:32 smithi081 ceph-mon[31453]: pgmap v19927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:39:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:34 smithi104 ceph-mon[36624]: pgmap v19928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:34 smithi081 ceph-mon[31453]: pgmap v19928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:34 smithi153 ceph-mon[36569]: pgmap v19928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:36.306 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:36.307+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:39:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:36.307+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:39:36.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:36.307+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:39:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:36 smithi081 ceph-mon[31453]: pgmap v19929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:36 smithi153 ceph-mon[36569]: pgmap v19929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:37.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:36 smithi104 ceph-mon[36624]: pgmap v19929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:38 smithi081 ceph-mon[31453]: pgmap v19930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:38 smithi153 ceph-mon[36569]: pgmap v19930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:38 smithi104 ceph-mon[36624]: pgmap v19930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:41.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:40 smithi104 ceph-mon[36624]: pgmap v19931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:40 smithi081 ceph-mon[31453]: pgmap v19931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:40 smithi153 ceph-mon[36569]: pgmap v19931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:43.111 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:42 smithi081 ceph-mon[31453]: pgmap v19932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:42 smithi153 ceph-mon[36569]: pgmap v19932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:42 smithi104 ceph-mon[36624]: pgmap v19932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:44 smithi081 ceph-mon[31453]: pgmap v19933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:44 smithi153 ceph-mon[36569]: pgmap v19933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:44 smithi104 ceph-mon[36624]: pgmap v19933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:46 smithi081 ceph-mon[31453]: pgmap v19934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:46 smithi153 ceph-mon[36569]: pgmap v19934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:46 smithi104 ceph-mon[36624]: pgmap v19934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:48 smithi081 ceph-mon[31453]: pgmap v19935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:48 smithi153 ceph-mon[36569]: pgmap v19935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:48 smithi104 ceph-mon[36624]: pgmap v19935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:50 smithi081 ceph-mon[31453]: pgmap v19936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:50 smithi153 ceph-mon[36569]: pgmap v19936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:51.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:51.310+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:39:51.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:51.310+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:39:51.309 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:39:51.310+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:39:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:50 smithi104 ceph-mon[36624]: pgmap v19936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:52 smithi081 ceph-mon[31453]: pgmap v19937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:52 smithi153 ceph-mon[36569]: pgmap v19937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:52 smithi104 ceph-mon[36624]: pgmap v19937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:54 smithi081 ceph-mon[31453]: pgmap v19938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:54 smithi153 ceph-mon[36569]: pgmap v19938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:54 smithi104 ceph-mon[36624]: pgmap v19938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:56 smithi081 ceph-mon[31453]: pgmap v19939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:56 smithi153 ceph-mon[36569]: pgmap v19939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:57.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:56 smithi104 ceph-mon[36624]: pgmap v19939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:39:58 smithi081 ceph-mon[31453]: pgmap v19940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:39:58 smithi153 ceph-mon[36569]: pgmap v19940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:39:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:39:58 smithi104 ceph-mon[36624]: pgmap v19940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:00 smithi081 ceph-mon[31453]: pgmap v19941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:40:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:00 smithi153 ceph-mon[36569]: pgmap v19941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:40:01.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:00 smithi104 ceph-mon[36624]: pgmap v19941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:40:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:02 smithi081 ceph-mon[31453]: pgmap v19942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:02 smithi153 ceph-mon[36569]: pgmap v19942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:02 smithi104 ceph-mon[36624]: pgmap v19942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:04 smithi081 ceph-mon[31453]: pgmap v19943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:04 smithi153 ceph-mon[36569]: pgmap v19943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:04 smithi104 ceph-mon[36624]: pgmap v19943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:06.325+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:40:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:06.325+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:40:06.325 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:06.325+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:40:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:06 smithi153 ceph-mon[36569]: pgmap v19944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:07.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:06 smithi104 ceph-mon[36624]: pgmap v19944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:07.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:06 smithi081 ceph-mon[31453]: pgmap v19944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:08 smithi153 ceph-mon[36569]: pgmap v19945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:08 smithi104 ceph-mon[36624]: pgmap v19945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:08 smithi081 ceph-mon[31453]: pgmap v19945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:11.174 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:10 smithi104 ceph-mon[36624]: pgmap v19946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:10 smithi153 ceph-mon[36569]: pgmap v19946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:10 smithi081 ceph-mon[31453]: pgmap v19946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:12 smithi153 ceph-mon[36569]: pgmap v19947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:12 smithi104 ceph-mon[36624]: pgmap v19947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:12 smithi081 ceph-mon[31453]: pgmap v19947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:14 smithi153 ceph-mon[36569]: pgmap v19948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:15.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:14 smithi104 ceph-mon[36624]: pgmap v19948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:14 smithi081 ceph-mon[31453]: pgmap v19948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:16 smithi153 ceph-mon[36569]: pgmap v19949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:17.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:16 smithi104 ceph-mon[36624]: pgmap v19949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:16 smithi081 ceph-mon[31453]: pgmap v19949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:18.903+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:40:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:18.903+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:40:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:18.904+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 288 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:40:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:18.904+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 288 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:40:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:18 smithi153 ceph-mon[36569]: pgmap v19950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:19.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:18 smithi104 ceph-mon[36624]: pgmap v19950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:18 smithi081 ceph-mon[31453]: pgmap v19950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:20 smithi153 ceph-mon[36569]: pgmap v19951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:21.329+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:40:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:21.329+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:40:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:21.329+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:40:21.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:20 smithi104 ceph-mon[36624]: pgmap v19951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:20 smithi081 ceph-mon[31453]: pgmap v19951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:22 smithi153 ceph-mon[36569]: pgmap v19952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:23.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:22 smithi104 ceph-mon[36624]: pgmap v19952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:23.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:22 smithi081 ceph-mon[31453]: pgmap v19952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:24 smithi153 ceph-mon[36569]: pgmap v19953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:24 smithi104 ceph-mon[36624]: pgmap v19953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:24 smithi081 ceph-mon[31453]: pgmap v19953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:26 smithi153 ceph-mon[36569]: pgmap v19954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:26 smithi104 ceph-mon[36624]: pgmap v19954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:26 smithi081 ceph-mon[31453]: pgmap v19954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:28 smithi153 ceph-mon[36569]: pgmap v19955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:28 smithi104 ceph-mon[36624]: pgmap v19955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:28 smithi081 ceph-mon[31453]: pgmap v19955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:30 smithi153 ceph-mon[36569]: pgmap v19956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:30 smithi104 ceph-mon[36624]: pgmap v19956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:30 smithi081 ceph-mon[31453]: pgmap v19956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:40:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:40:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:40:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:33 smithi153 ceph-mon[36569]: pgmap v19957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:33 smithi104 ceph-mon[36624]: pgmap v19957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:33 smithi081 ceph-mon[31453]: pgmap v19957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:35 smithi153 ceph-mon[36569]: pgmap v19958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:35 smithi104 ceph-mon[36624]: pgmap v19958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:35.583 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:35 smithi081 ceph-mon[31453]: pgmap v19958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:35.583 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:35.583 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:36.344+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:40:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:36.344+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:40:36.344 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:36.344+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:40:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:36 smithi104 ceph-mon[36624]: pgmap v19959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:36 smithi081 ceph-mon[31453]: pgmap v19959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:36 smithi153 ceph-mon[36569]: pgmap v19959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.814 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.814 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.815 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:38 smithi153 ceph-mon[36569]: pgmap v19960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:40:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:40:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:38 smithi104 ceph-mon[36624]: pgmap v19960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:40:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:40:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:38 smithi081 ceph-mon[31453]: pgmap v19960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:40:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:40:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:40:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:39 smithi153 ceph-mon[36569]: pgmap v19961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:39 smithi153 ceph-mon[36569]: pgmap v19962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:39 smithi104 ceph-mon[36624]: pgmap v19961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:39 smithi104 ceph-mon[36624]: pgmap v19962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:39 smithi081 ceph-mon[31453]: pgmap v19961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:39.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:39 smithi081 ceph-mon[31453]: pgmap v19962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:41 smithi153 ceph-mon[36569]: pgmap v19963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:41.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:41 smithi104 ceph-mon[36624]: pgmap v19963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:41.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:41 smithi081 ceph-mon[31453]: pgmap v19963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:43 smithi153 ceph-mon[36569]: pgmap v19964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:43.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:43 smithi104 ceph-mon[36624]: pgmap v19964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:43.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:43 smithi081 ceph-mon[31453]: pgmap v19964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:45 smithi153 ceph-mon[36569]: pgmap v19965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:45 smithi104 ceph-mon[36624]: pgmap v19965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:45 smithi081 ceph-mon[31453]: pgmap v19965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:47 smithi153 ceph-mon[36569]: pgmap v19966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:47.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:47 smithi104 ceph-mon[36624]: pgmap v19966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:47.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:47 smithi081 ceph-mon[31453]: pgmap v19966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:49 smithi153 ceph-mon[36569]: pgmap v19967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:49.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:49 smithi104 ceph-mon[36624]: pgmap v19967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:49.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:49 smithi081 ceph-mon[31453]: pgmap v19967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:51.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:51.348+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:40:51.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:51.348+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:40:51.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:40:51.348+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:40:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:51 smithi153 ceph-mon[36569]: pgmap v19968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:51.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:51 smithi104 ceph-mon[36624]: pgmap v19968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:51.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:51 smithi081 ceph-mon[31453]: pgmap v19968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:53 smithi153 ceph-mon[36569]: pgmap v19969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:53.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:53 smithi104 ceph-mon[36624]: pgmap v19969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:53.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:53 smithi081 ceph-mon[31453]: pgmap v19969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:55 smithi153 ceph-mon[36569]: pgmap v19970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:55.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:55 smithi104 ceph-mon[36624]: pgmap v19970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:55 smithi081 ceph-mon[31453]: pgmap v19970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:57 smithi153 ceph-mon[36569]: pgmap v19971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:57.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:57 smithi104 ceph-mon[36624]: pgmap v19971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:57.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:57 smithi081 ceph-mon[31453]: pgmap v19971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:40:59 smithi153 ceph-mon[36569]: pgmap v19972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:59.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:40:59 smithi104 ceph-mon[36624]: pgmap v19972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:40:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:40:59 smithi081 ceph-mon[31453]: pgmap v19972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:01 smithi153 ceph-mon[36569]: pgmap v19973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:01.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:01 smithi104 ceph-mon[36624]: pgmap v19973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:01 smithi081 ceph-mon[31453]: pgmap v19973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:03 smithi153 ceph-mon[36569]: pgmap v19974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:03.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:03 smithi104 ceph-mon[36624]: pgmap v19974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:03 smithi081 ceph-mon[31453]: pgmap v19974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:05 smithi104 ceph-mon[36624]: pgmap v19975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:05.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:05 smithi081 ceph-mon[31453]: pgmap v19975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:05 smithi153 ceph-mon[36569]: pgmap v19975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:06.355+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:41:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:06.356+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:41:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:06.356+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:41:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:07 smithi104 ceph-mon[36624]: pgmap v19976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:07 smithi081 ceph-mon[31453]: pgmap v19976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:07 smithi153 ceph-mon[36569]: pgmap v19976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:09.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:09 smithi104 ceph-mon[36624]: pgmap v19977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:09.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:09 smithi081 ceph-mon[31453]: pgmap v19977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:09 smithi153 ceph-mon[36569]: pgmap v19977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:11 smithi104 ceph-mon[36624]: pgmap v19978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:11.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:11 smithi081 ceph-mon[31453]: pgmap v19978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:11 smithi153 ceph-mon[36569]: pgmap v19978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:13 smithi081 ceph-mon[31453]: pgmap v19979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:13 smithi153 ceph-mon[36569]: pgmap v19979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:13 smithi104 ceph-mon[36624]: pgmap v19979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:15.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:15 smithi081 ceph-mon[31453]: pgmap v19980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:15 smithi153 ceph-mon[36569]: pgmap v19980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:16.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:15 smithi104 ceph-mon[36624]: pgmap v19980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:17 smithi081 ceph-mon[31453]: pgmap v19981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:17 smithi153 ceph-mon[36569]: pgmap v19981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:18.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:17 smithi104 ceph-mon[36624]: pgmap v19981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:19 smithi081 ceph-mon[31453]: pgmap v19982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:19 smithi153 ceph-mon[36569]: pgmap v19982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:19 smithi104 ceph-mon[36624]: pgmap v19982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:21.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:21.372+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:41:21.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:21.372+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:41:21.372 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:21.372+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:41:21.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:21 smithi081 ceph-mon[31453]: pgmap v19983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:21 smithi153 ceph-mon[36569]: pgmap v19983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:21 smithi104 ceph-mon[36624]: pgmap v19983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:23.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:23 smithi081 ceph-mon[31453]: pgmap v19984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:23 smithi153 ceph-mon[36569]: pgmap v19984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:23 smithi104 ceph-mon[36624]: pgmap v19984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:25.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:25 smithi081 ceph-mon[31453]: pgmap v19985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:25 smithi153 ceph-mon[36569]: pgmap v19985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:26.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:25 smithi104 ceph-mon[36624]: pgmap v19985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:27 smithi081 ceph-mon[31453]: pgmap v19986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:27 smithi153 ceph-mon[36569]: pgmap v19986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:28.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:27 smithi104 ceph-mon[36624]: pgmap v19986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:29 smithi081 ceph-mon[31453]: pgmap v19987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:29 smithi153 ceph-mon[36569]: pgmap v19987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:29 smithi104 ceph-mon[36624]: pgmap v19987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:31 smithi081 ceph-mon[31453]: pgmap v19988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:31 smithi153 ceph-mon[36569]: pgmap v19988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:31 smithi104 ceph-mon[36624]: pgmap v19988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:33 smithi081 ceph-mon[31453]: pgmap v19989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:33 smithi153 ceph-mon[36569]: pgmap v19989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:33 smithi104 ceph-mon[36624]: pgmap v19989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:35 smithi081 ceph-mon[31453]: pgmap v19990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:35 smithi153 ceph-mon[36569]: pgmap v19990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:35 smithi104 ceph-mon[36624]: pgmap v19990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:36.376 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:36.377+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:41:36.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:36.377+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:41:36.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:36.377+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:41:37.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:37 smithi081 ceph-mon[31453]: pgmap v19991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:37 smithi153 ceph-mon[36569]: pgmap v19991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:37 smithi104 ceph-mon[36624]: pgmap v19991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:41:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:41:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:41:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:40 smithi153 ceph-mon[36569]: pgmap v19992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:41:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:41:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:40 smithi104 ceph-mon[36624]: pgmap v19992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:41:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:41:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:40 smithi081 ceph-mon[31453]: pgmap v19992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:41:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:41:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:41:41.483 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:41 smithi104 ceph-mon[36624]: pgmap v19993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:41.484 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:41 smithi104 ceph-mon[36624]: pgmap v19994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:41 smithi153 ceph-mon[36569]: pgmap v19993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:41 smithi153 ceph-mon[36569]: pgmap v19994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:41 smithi081 ceph-mon[31453]: pgmap v19993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:41 smithi081 ceph-mon[31453]: pgmap v19994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:43 smithi153 ceph-mon[36569]: pgmap v19995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:43 smithi104 ceph-mon[36624]: pgmap v19995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:43.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:43 smithi081 ceph-mon[31453]: pgmap v19995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:45 smithi153 ceph-mon[36569]: pgmap v19996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:45 smithi104 ceph-mon[36624]: pgmap v19996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:45 smithi081 ceph-mon[31453]: pgmap v19996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:47 smithi153 ceph-mon[36569]: pgmap v19997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:47 smithi104 ceph-mon[36624]: pgmap v19997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:47 smithi081 ceph-mon[31453]: pgmap v19997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:49 smithi081 ceph-mon[31453]: pgmap v19998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:49 smithi153 ceph-mon[36569]: pgmap v19998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:49 smithi104 ceph-mon[36624]: pgmap v19998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:51.380+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:41:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:51.380+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:41:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:41:51.380+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:41:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:51 smithi153 ceph-mon[36569]: pgmap v19999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:51 smithi104 ceph-mon[36624]: pgmap v19999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:51.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:51 smithi081 ceph-mon[31453]: pgmap v19999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:53 smithi153 ceph-mon[36569]: pgmap v20000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:53 smithi104 ceph-mon[36624]: pgmap v20000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:53 smithi081 ceph-mon[31453]: pgmap v20000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:55 smithi153 ceph-mon[36569]: pgmap v20001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:55 smithi104 ceph-mon[36624]: pgmap v20001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:55 smithi081 ceph-mon[31453]: pgmap v20001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:57 smithi104 ceph-mon[36624]: pgmap v20002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:57.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:57 smithi081 ceph-mon[31453]: pgmap v20002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:57 smithi153 ceph-mon[36569]: pgmap v20002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:41:59 smithi104 ceph-mon[36624]: pgmap v20003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:59.608 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:41:59 smithi153 ceph-mon[36569]: pgmap v20003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:41:59.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:41:59 smithi081 ceph-mon[31453]: pgmap v20003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:01 smithi104 ceph-mon[36624]: pgmap v20004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:01.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:01 smithi081 ceph-mon[31453]: pgmap v20004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:01 smithi153 ceph-mon[36569]: pgmap v20004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:03 smithi104 ceph-mon[36624]: pgmap v20005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:03 smithi081 ceph-mon[31453]: pgmap v20005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:03 smithi153 ceph-mon[36569]: pgmap v20005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:04 smithi081 ceph-mon[31453]: pgmap v20006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:04 smithi153 ceph-mon[36569]: pgmap v20006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:04 smithi104 ceph-mon[36624]: pgmap v20006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:06.388+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:42:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:06.388+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:42:06.388 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:06.388+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:42:07.360 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:07 smithi081 ceph-mon[31453]: pgmap v20007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:07 smithi153 ceph-mon[36569]: pgmap v20007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:07 smithi104 ceph-mon[36624]: pgmap v20007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:09 smithi081 ceph-mon[31453]: pgmap v20008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:09 smithi153 ceph-mon[36569]: pgmap v20008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:09 smithi104 ceph-mon[36624]: pgmap v20008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:11.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:11 smithi104 ceph-mon[36624]: pgmap v20009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:11 smithi081 ceph-mon[31453]: pgmap v20009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:11 smithi153 ceph-mon[36569]: pgmap v20009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:13 smithi081 ceph-mon[31453]: pgmap v20010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:13 smithi153 ceph-mon[36569]: pgmap v20010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:13 smithi104 ceph-mon[36624]: pgmap v20010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:15 smithi081 ceph-mon[31453]: pgmap v20011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:15 smithi153 ceph-mon[36569]: pgmap v20011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:15.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:15 smithi104 ceph-mon[36624]: pgmap v20011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:17 smithi081 ceph-mon[31453]: pgmap v20012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:17 smithi153 ceph-mon[36569]: pgmap v20012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:17 smithi104 ceph-mon[36624]: pgmap v20012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:19 smithi081 ceph-mon[31453]: pgmap v20013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:19 smithi153 ceph-mon[36569]: pgmap v20013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:19 smithi104 ceph-mon[36624]: pgmap v20013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:21.400 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:21.400+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:42:21.400 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:21.400+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:42:21.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:21.400+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:42:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:21 smithi081 ceph-mon[31453]: pgmap v20014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:21 smithi153 ceph-mon[36569]: pgmap v20014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:21 smithi104 ceph-mon[36624]: pgmap v20014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:23 smithi081 ceph-mon[31453]: pgmap v20015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:23 smithi153 ceph-mon[36569]: pgmap v20015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:23 smithi104 ceph-mon[36624]: pgmap v20015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:25 smithi081 ceph-mon[31453]: pgmap v20016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:25 smithi153 ceph-mon[36569]: pgmap v20016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:25 smithi104 ceph-mon[36624]: pgmap v20016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:27 smithi153 ceph-mon[36569]: pgmap v20017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:27 smithi104 ceph-mon[36624]: pgmap v20017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:27.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:27 smithi081 ceph-mon[31453]: pgmap v20017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:29 smithi153 ceph-mon[36569]: pgmap v20018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:29 smithi104 ceph-mon[36624]: pgmap v20018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:29 smithi081 ceph-mon[31453]: pgmap v20018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:31 smithi153 ceph-mon[36569]: pgmap v20019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:31 smithi104 ceph-mon[36624]: pgmap v20019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:31 smithi081 ceph-mon[31453]: pgmap v20019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:33 smithi153 ceph-mon[36569]: pgmap v20020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:33 smithi104 ceph-mon[36624]: pgmap v20020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:33 smithi081 ceph-mon[31453]: pgmap v20020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:35 smithi081 ceph-mon[31453]: pgmap v20021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:35 smithi153 ceph-mon[36569]: pgmap v20021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:35 smithi104 ceph-mon[36624]: pgmap v20021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:36.416+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:42:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:36.416+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:42:36.415 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:36.416+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:42:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:37 smithi153 ceph-mon[36569]: pgmap v20022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:37 smithi104 ceph-mon[36624]: pgmap v20022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:37 smithi081 ceph-mon[31453]: pgmap v20022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:39 smithi153 ceph-mon[36569]: pgmap v20023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:39 smithi104 ceph-mon[36624]: pgmap v20023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:39 smithi081 ceph-mon[31453]: pgmap v20023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:41 smithi153 ceph-mon[36569]: pgmap v20024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:42:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:41 smithi104 ceph-mon[36624]: pgmap v20024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:42:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:41 smithi081 ceph-mon[31453]: pgmap v20024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:41.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:42:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:42:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:42:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:42:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:42:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:42:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:42:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:42:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:42:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:42:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:43 smithi104 ceph-mon[36624]: pgmap v20025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:43 smithi104 ceph-mon[36624]: pgmap v20026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:43 smithi081 ceph-mon[31453]: pgmap v20025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:43 smithi081 ceph-mon[31453]: pgmap v20026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:43 smithi153 ceph-mon[36569]: pgmap v20025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:43 smithi153 ceph-mon[36569]: pgmap v20026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:45 smithi104 ceph-mon[36624]: pgmap v20027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:45 smithi081 ceph-mon[31453]: pgmap v20027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:45 smithi153 ceph-mon[36569]: pgmap v20027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:47 smithi104 ceph-mon[36624]: pgmap v20028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:47 smithi081 ceph-mon[31453]: pgmap v20028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:47 smithi153 ceph-mon[36569]: pgmap v20028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:49 smithi081 ceph-mon[31453]: pgmap v20029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:49 smithi153 ceph-mon[36569]: pgmap v20029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:49 smithi104 ceph-mon[36624]: pgmap v20029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:50 smithi081 ceph-mon[31453]: pgmap v20030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:50 smithi153 ceph-mon[36569]: pgmap v20030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:50 smithi104 ceph-mon[36624]: pgmap v20030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:51.429+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:42:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:51.429+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:42:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:42:51.429+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:42:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:52 smithi153 ceph-mon[36569]: pgmap v20031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:52 smithi104 ceph-mon[36624]: pgmap v20031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:52 smithi081 ceph-mon[31453]: pgmap v20031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:54 smithi153 ceph-mon[36569]: pgmap v20032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:54 smithi104 ceph-mon[36624]: pgmap v20032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:54 smithi081 ceph-mon[31453]: pgmap v20032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:56 smithi153 ceph-mon[36569]: pgmap v20033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:56 smithi104 ceph-mon[36624]: pgmap v20033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:56 smithi081 ceph-mon[31453]: pgmap v20033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:42:58 smithi153 ceph-mon[36569]: pgmap v20034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:42:58 smithi104 ceph-mon[36624]: pgmap v20034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:42:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:42:58 smithi081 ceph-mon[31453]: pgmap v20034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:00 smithi153 ceph-mon[36569]: pgmap v20035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:00 smithi104 ceph-mon[36624]: pgmap v20035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:00 smithi081 ceph-mon[31453]: pgmap v20035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:02 smithi153 ceph-mon[36569]: pgmap v20036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:02 smithi104 ceph-mon[36624]: pgmap v20036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:02 smithi081 ceph-mon[31453]: pgmap v20036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:04 smithi153 ceph-mon[36569]: pgmap v20037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:04 smithi104 ceph-mon[36624]: pgmap v20037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:04 smithi081 ceph-mon[31453]: pgmap v20037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:06.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:06.444+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:43:06.444 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:06.444+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:43:06.445 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:06.444+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:43:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:06 smithi153 ceph-mon[36569]: pgmap v20038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:07.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:06 smithi081 ceph-mon[31453]: pgmap v20038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:07.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:06 smithi104 ceph-mon[36624]: pgmap v20038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:08 smithi153 ceph-mon[36569]: pgmap v20039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:08 smithi104 ceph-mon[36624]: pgmap v20039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:08 smithi081 ceph-mon[31453]: pgmap v20039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:10 smithi153 ceph-mon[36569]: pgmap v20040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:10 smithi104 ceph-mon[36624]: pgmap v20040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:10 smithi081 ceph-mon[31453]: pgmap v20040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:12 smithi153 ceph-mon[36569]: pgmap v20041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:12 smithi104 ceph-mon[36624]: pgmap v20041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:12 smithi081 ceph-mon[31453]: pgmap v20041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:14 smithi153 ceph-mon[36569]: pgmap v20042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:14 smithi104 ceph-mon[36624]: pgmap v20042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:14 smithi081 ceph-mon[31453]: pgmap v20042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:16 smithi153 ceph-mon[36569]: pgmap v20043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:16 smithi104 ceph-mon[36624]: pgmap v20043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:16 smithi081 ceph-mon[31453]: pgmap v20043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:18.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:18.933+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:43:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:18.933+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:43:18.933 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:18.933+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 289 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T07:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:18.933+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:18.933+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 289 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T07:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:18.933+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:43:18.934 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:18.934+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 290 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:43:18.935 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:18.934+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 290 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:43:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:18 smithi153 ceph-mon[36569]: pgmap v20044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:18 smithi104 ceph-mon[36624]: pgmap v20044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:18 smithi081 ceph-mon[31453]: pgmap v20044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:20 smithi153 ceph-mon[36569]: pgmap v20045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:20 smithi104 ceph-mon[36624]: pgmap v20045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:20 smithi081 ceph-mon[31453]: pgmap v20045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:21.453+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:43:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:21.453+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:43:21.453 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:21.453+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:43:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:22 smithi153 ceph-mon[36569]: pgmap v20046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:22 smithi104 ceph-mon[36624]: pgmap v20046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:22 smithi081 ceph-mon[31453]: pgmap v20046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:24 smithi153 ceph-mon[36569]: pgmap v20047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:24 smithi104 ceph-mon[36624]: pgmap v20047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:24 smithi081 ceph-mon[31453]: pgmap v20047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:26 smithi153 ceph-mon[36569]: pgmap v20048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:26 smithi104 ceph-mon[36624]: pgmap v20048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:26 smithi081 ceph-mon[31453]: pgmap v20048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:28 smithi153 ceph-mon[36569]: pgmap v20049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:28 smithi104 ceph-mon[36624]: pgmap v20049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:28 smithi081 ceph-mon[31453]: pgmap v20049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:30 smithi153 ceph-mon[36569]: pgmap v20050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:30 smithi104 ceph-mon[36624]: pgmap v20050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:30 smithi081 ceph-mon[31453]: pgmap v20050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:32 smithi153 ceph-mon[36569]: pgmap v20051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:32 smithi104 ceph-mon[36624]: pgmap v20051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:32 smithi081 ceph-mon[31453]: pgmap v20051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:34 smithi153 ceph-mon[36569]: pgmap v20052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:34 smithi104 ceph-mon[36624]: pgmap v20052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:34 smithi081 ceph-mon[31453]: pgmap v20052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:36.463+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:43:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:36.463+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:43:36.464 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:36.463+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:43:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:36 smithi153 ceph-mon[36569]: pgmap v20053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:36 smithi104 ceph-mon[36624]: pgmap v20053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:37.095 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:36 smithi081 ceph-mon[31453]: pgmap v20053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:38 smithi153 ceph-mon[36569]: pgmap v20054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:38 smithi104 ceph-mon[36624]: pgmap v20054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:38 smithi081 ceph-mon[31453]: pgmap v20054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:40 smithi153 ceph-mon[36569]: pgmap v20055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:40 smithi104 ceph-mon[36624]: pgmap v20055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:40 smithi081 ceph-mon[31453]: pgmap v20055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:42 smithi153 ceph-mon[36569]: pgmap v20056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:43:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:42 smithi104 ceph-mon[36624]: pgmap v20056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:43:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:42 smithi081 ceph-mon[31453]: pgmap v20056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:43:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:43:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:43:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:43 smithi104 ceph-mon[36624]: pgmap v20057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:43 smithi104 ceph-mon[36624]: pgmap v20058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:43:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:43:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:43:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:43 smithi081 ceph-mon[31453]: pgmap v20057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:43 smithi081 ceph-mon[31453]: pgmap v20058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:43:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:43:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:43:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:43 smithi153 ceph-mon[36569]: pgmap v20057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:43 smithi153 ceph-mon[36569]: pgmap v20058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:44.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:43:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:46 smithi153 ceph-mon[36569]: pgmap v20059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:46 smithi104 ceph-mon[36624]: pgmap v20059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:46 smithi081 ceph-mon[31453]: pgmap v20059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:48 smithi153 ceph-mon[36569]: pgmap v20060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:48 smithi104 ceph-mon[36624]: pgmap v20060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:48 smithi081 ceph-mon[31453]: pgmap v20060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:50 smithi153 ceph-mon[36569]: pgmap v20061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:50 smithi104 ceph-mon[36624]: pgmap v20061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:50 smithi081 ceph-mon[31453]: pgmap v20061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:51.467+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:43:51.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:51.467+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:43:51.468 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:43:51.467+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:43:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:52 smithi153 ceph-mon[36569]: pgmap v20062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:52 smithi104 ceph-mon[36624]: pgmap v20062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:52 smithi081 ceph-mon[31453]: pgmap v20062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:54 smithi153 ceph-mon[36569]: pgmap v20063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:54 smithi104 ceph-mon[36624]: pgmap v20063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:54 smithi081 ceph-mon[31453]: pgmap v20063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:56 smithi153 ceph-mon[36569]: pgmap v20064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:56 smithi104 ceph-mon[36624]: pgmap v20064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:56 smithi081 ceph-mon[31453]: pgmap v20064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:43:58 smithi153 ceph-mon[36569]: pgmap v20065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:43:58 smithi104 ceph-mon[36624]: pgmap v20065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:43:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:43:58 smithi081 ceph-mon[31453]: pgmap v20065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:00 smithi153 ceph-mon[36569]: pgmap v20066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:00 smithi104 ceph-mon[36624]: pgmap v20066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:00 smithi081 ceph-mon[31453]: pgmap v20066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:02 smithi153 ceph-mon[36569]: pgmap v20067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:02 smithi104 ceph-mon[36624]: pgmap v20067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:02 smithi081 ceph-mon[31453]: pgmap v20067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:04 smithi153 ceph-mon[36569]: pgmap v20068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:04 smithi104 ceph-mon[36624]: pgmap v20068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:04 smithi081 ceph-mon[31453]: pgmap v20068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:06.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:06.477+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:44:06.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:06.477+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:44:06.477 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:06.477+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:44:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:06 smithi153 ceph-mon[36569]: pgmap v20069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:06 smithi104 ceph-mon[36624]: pgmap v20069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:06 smithi081 ceph-mon[31453]: pgmap v20069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:08 smithi153 ceph-mon[36569]: pgmap v20070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:08 smithi104 ceph-mon[36624]: pgmap v20070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:08 smithi081 ceph-mon[31453]: pgmap v20070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:10 smithi153 ceph-mon[36569]: pgmap v20071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:10 smithi104 ceph-mon[36624]: pgmap v20071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:10 smithi081 ceph-mon[31453]: pgmap v20071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:12 smithi104 ceph-mon[36624]: pgmap v20072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:12 smithi081 ceph-mon[31453]: pgmap v20072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:12 smithi153 ceph-mon[36569]: pgmap v20072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:14 smithi104 ceph-mon[36624]: pgmap v20073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:14 smithi081 ceph-mon[31453]: pgmap v20073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:14 smithi153 ceph-mon[36569]: pgmap v20073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:16 smithi104 ceph-mon[36624]: pgmap v20074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:16 smithi081 ceph-mon[31453]: pgmap v20074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:16 smithi153 ceph-mon[36569]: pgmap v20074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:18 smithi104 ceph-mon[36624]: pgmap v20075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:18 smithi081 ceph-mon[31453]: pgmap v20075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:18.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:18 smithi153 ceph-mon[36569]: pgmap v20075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:20 smithi104 ceph-mon[36624]: pgmap v20076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:20 smithi081 ceph-mon[31453]: pgmap v20076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:20 smithi153 ceph-mon[36569]: pgmap v20076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:21.492+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:44:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:21.492+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:44:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:21.492+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:44:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:22 smithi081 ceph-mon[31453]: pgmap v20077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:22 smithi153 ceph-mon[36569]: pgmap v20077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:22 smithi104 ceph-mon[36624]: pgmap v20077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:24.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:24 smithi081 ceph-mon[31453]: pgmap v20078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:24 smithi153 ceph-mon[36569]: pgmap v20078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:24 smithi104 ceph-mon[36624]: pgmap v20078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:26 smithi081 ceph-mon[31453]: pgmap v20079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:26 smithi153 ceph-mon[36569]: pgmap v20079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:26 smithi104 ceph-mon[36624]: pgmap v20079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:28 smithi081 ceph-mon[31453]: pgmap v20080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:28 smithi153 ceph-mon[36569]: pgmap v20080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:28 smithi104 ceph-mon[36624]: pgmap v20080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:30.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:30 smithi153 ceph-mon[36569]: pgmap v20081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:30 smithi081 ceph-mon[31453]: pgmap v20081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:30 smithi104 ceph-mon[36624]: pgmap v20081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:32 smithi104 ceph-mon[36624]: pgmap v20082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:32 smithi081 ceph-mon[31453]: pgmap v20082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:32 smithi153 ceph-mon[36569]: pgmap v20082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:34 smithi104 ceph-mon[36624]: pgmap v20083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:34 smithi081 ceph-mon[31453]: pgmap v20083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:34 smithi153 ceph-mon[36569]: pgmap v20083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:36.504+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:44:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:36.504+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:44:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:36.504+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:44:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:36 smithi081 ceph-mon[31453]: pgmap v20084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:36 smithi153 ceph-mon[36569]: pgmap v20084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:36 smithi104 ceph-mon[36624]: pgmap v20084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:38 smithi081 ceph-mon[31453]: pgmap v20085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:38 smithi153 ceph-mon[36569]: pgmap v20085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:38 smithi104 ceph-mon[36624]: pgmap v20085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:40 smithi081 ceph-mon[31453]: pgmap v20086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:40 smithi153 ceph-mon[36569]: pgmap v20086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:40 smithi104 ceph-mon[36624]: pgmap v20086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:42.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:42 smithi104 ceph-mon[36624]: pgmap v20087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:42 smithi081 ceph-mon[31453]: pgmap v20087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:42 smithi153 ceph-mon[36569]: pgmap v20087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:44 smithi081 ceph-mon[31453]: pgmap v20088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:44:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:44 smithi153 ceph-mon[36569]: pgmap v20088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:44:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:44 smithi104 ceph-mon[36624]: pgmap v20088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:44:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:44:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:44:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:44:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:44:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:44:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:44:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:44:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:44:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:44:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:46 smithi081 ceph-mon[31453]: pgmap v20089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:46 smithi081 ceph-mon[31453]: pgmap v20090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:46 smithi153 ceph-mon[36569]: pgmap v20089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:46 smithi153 ceph-mon[36569]: pgmap v20090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:46 smithi104 ceph-mon[36624]: pgmap v20089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:46 smithi104 ceph-mon[36624]: pgmap v20090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:47 smithi081 ceph-mon[31453]: pgmap v20091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:47 smithi153 ceph-mon[36569]: pgmap v20091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:47 smithi104 ceph-mon[36624]: pgmap v20091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:49 smithi153 ceph-mon[36569]: pgmap v20092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:49 smithi104 ceph-mon[36624]: pgmap v20092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:49 smithi081 ceph-mon[31453]: pgmap v20092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:51.520+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:44:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:51.520+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:44:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:44:51.520+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:44:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:51 smithi153 ceph-mon[36569]: pgmap v20093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:51 smithi104 ceph-mon[36624]: pgmap v20093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:51 smithi081 ceph-mon[31453]: pgmap v20093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:54.061 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:53 smithi153 ceph-mon[36569]: pgmap v20094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:53 smithi104 ceph-mon[36624]: pgmap v20094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:53 smithi081 ceph-mon[31453]: pgmap v20094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:55 smithi104 ceph-mon[36624]: pgmap v20095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:55 smithi081 ceph-mon[31453]: pgmap v20095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:55 smithi153 ceph-mon[36569]: pgmap v20095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:57 smithi104 ceph-mon[36624]: pgmap v20096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:57 smithi081 ceph-mon[31453]: pgmap v20096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:44:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:57 smithi153 ceph-mon[36569]: pgmap v20096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:44:59 smithi104 ceph-mon[36624]: pgmap v20097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:00.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:44:59 smithi153 ceph-mon[36569]: pgmap v20097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:44:59 smithi081 ceph-mon[31453]: pgmap v20097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:01 smithi104 ceph-mon[36624]: pgmap v20098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:01 smithi081 ceph-mon[31453]: pgmap v20098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:01 smithi153 ceph-mon[36569]: pgmap v20098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:03 smithi081 ceph-mon[31453]: pgmap v20099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:03 smithi153 ceph-mon[36569]: pgmap v20099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:03 smithi104 ceph-mon[36624]: pgmap v20099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:05 smithi081 ceph-mon[31453]: pgmap v20100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:05 smithi153 ceph-mon[36569]: pgmap v20100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:05 smithi104 ceph-mon[36624]: pgmap v20100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:06.521+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:45:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:06.521+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:45:06.521 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:06.521+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:45:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:07 smithi081 ceph-mon[31453]: pgmap v20101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:07 smithi153 ceph-mon[36569]: pgmap v20101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:07 smithi104 ceph-mon[36624]: pgmap v20101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:09 smithi081 ceph-mon[31453]: pgmap v20102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:09 smithi153 ceph-mon[36569]: pgmap v20102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:09 smithi104 ceph-mon[36624]: pgmap v20102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:12.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:11 smithi104 ceph-mon[36624]: pgmap v20103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:11 smithi081 ceph-mon[31453]: pgmap v20103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:11 smithi153 ceph-mon[36569]: pgmap v20103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:13 smithi081 ceph-mon[31453]: pgmap v20104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:13 smithi153 ceph-mon[36569]: pgmap v20104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:13 smithi104 ceph-mon[36624]: pgmap v20104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:15 smithi081 ceph-mon[31453]: pgmap v20105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:15 smithi153 ceph-mon[36569]: pgmap v20105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:15 smithi104 ceph-mon[36624]: pgmap v20105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:18 smithi081 ceph-mon[31453]: pgmap v20106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:18 smithi153 ceph-mon[36569]: pgmap v20106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:18 smithi104 ceph-mon[36624]: pgmap v20106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:20 smithi081 ceph-mon[31453]: pgmap v20107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:20 smithi153 ceph-mon[36569]: pgmap v20107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:20 smithi104 ceph-mon[36624]: pgmap v20107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:21.528+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:45:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:21.528+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:45:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:21.528+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:45:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:22 smithi081 ceph-mon[31453]: pgmap v20108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:22 smithi153 ceph-mon[36569]: pgmap v20108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:22 smithi104 ceph-mon[36624]: pgmap v20108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:24 smithi081 ceph-mon[31453]: pgmap v20109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:24 smithi153 ceph-mon[36569]: pgmap v20109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:24 smithi104 ceph-mon[36624]: pgmap v20109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:25 smithi081 ceph-mon[31453]: pgmap v20110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:25 smithi153 ceph-mon[36569]: pgmap v20110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:25 smithi104 ceph-mon[36624]: pgmap v20110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:27 smithi104 ceph-mon[36624]: pgmap v20111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:27 smithi081 ceph-mon[31453]: pgmap v20111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:27 smithi153 ceph-mon[36569]: pgmap v20111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:29 smithi104 ceph-mon[36624]: pgmap v20112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:30.110 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:29 smithi153 ceph-mon[36569]: pgmap v20112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:29 smithi081 ceph-mon[31453]: pgmap v20112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:31 smithi104 ceph-mon[36624]: pgmap v20113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:31 smithi081 ceph-mon[31453]: pgmap v20113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:31 smithi153 ceph-mon[36569]: pgmap v20113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:33 smithi104 ceph-mon[36624]: pgmap v20114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:33 smithi081 ceph-mon[31453]: pgmap v20114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:33 smithi153 ceph-mon[36569]: pgmap v20114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:35 smithi081 ceph-mon[31453]: pgmap v20115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:35 smithi153 ceph-mon[36569]: pgmap v20115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:35 smithi104 ceph-mon[36624]: pgmap v20115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:36.539 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:36.539+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:45:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:36.539+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:45:36.540 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:36.539+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:45:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:37 smithi081 ceph-mon[31453]: pgmap v20116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:37 smithi153 ceph-mon[36569]: pgmap v20116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:37 smithi104 ceph-mon[36624]: pgmap v20116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:39 smithi081 ceph-mon[31453]: pgmap v20117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:39 smithi153 ceph-mon[36569]: pgmap v20117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:39 smithi104 ceph-mon[36624]: pgmap v20117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:42.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:41 smithi104 ceph-mon[36624]: pgmap v20118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:41 smithi081 ceph-mon[31453]: pgmap v20118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:41 smithi153 ceph-mon[36569]: pgmap v20118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:43 smithi081 ceph-mon[31453]: pgmap v20119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:43 smithi153 ceph-mon[36569]: pgmap v20119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:43 smithi104 ceph-mon[36624]: pgmap v20119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:45 smithi081 ceph-mon[31453]: pgmap v20120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:45:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:45 smithi153 ceph-mon[36569]: pgmap v20120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:45:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:45 smithi104 ceph-mon[36624]: pgmap v20120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:45:47.964 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:47 smithi104 ceph-mon[36624]: pgmap v20121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:47 smithi081 ceph-mon[31453]: pgmap v20121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:47 smithi153 ceph-mon[36569]: pgmap v20121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:50.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:49 smithi081 ceph-mon[31453]: pgmap v20122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:49 smithi153 ceph-mon[36569]: pgmap v20122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:49 smithi104 ceph-mon[36624]: pgmap v20122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:51.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:51.554+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:45:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:51.554+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:45:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:45:51.554+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:45:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:51 smithi081 ceph-mon[31453]: pgmap v20123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:51 smithi153 ceph-mon[36569]: pgmap v20123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:51 smithi104 ceph-mon[36624]: pgmap v20123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:45:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:45:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:53 smithi153 ceph-mon[36569]: pgmap v20124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:53 smithi153 ceph-mon[36569]: pgmap v20125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:45:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:45:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:53 smithi104 ceph-mon[36624]: pgmap v20124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:53 smithi104 ceph-mon[36624]: pgmap v20125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:45:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:45:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:53 smithi081 ceph-mon[31453]: pgmap v20124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:53 smithi081 ceph-mon[31453]: pgmap v20125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:45:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:55 smithi081 ceph-mon[31453]: pgmap v20126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:55 smithi153 ceph-mon[36569]: pgmap v20126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:55 smithi104 ceph-mon[36624]: pgmap v20126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:57 smithi081 ceph-mon[31453]: pgmap v20127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:57 smithi153 ceph-mon[36569]: pgmap v20127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:57 smithi104 ceph-mon[36624]: pgmap v20127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:45:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:45:59 smithi081 ceph-mon[31453]: pgmap v20128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:45:59 smithi153 ceph-mon[36569]: pgmap v20128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:45:59 smithi104 ceph-mon[36624]: pgmap v20128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:01.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:01 smithi153 ceph-mon[36569]: pgmap v20129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:01 smithi081 ceph-mon[31453]: pgmap v20129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:01 smithi104 ceph-mon[36624]: pgmap v20129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:03 smithi081 ceph-mon[31453]: pgmap v20130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:03 smithi153 ceph-mon[36569]: pgmap v20130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:03 smithi104 ceph-mon[36624]: pgmap v20130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:05 smithi081 ceph-mon[31453]: pgmap v20131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:05 smithi153 ceph-mon[36569]: pgmap v20131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:05 smithi104 ceph-mon[36624]: pgmap v20131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:06.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:46:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:06.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:46:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:06.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:46:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:07 smithi081 ceph-mon[31453]: pgmap v20132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:07 smithi153 ceph-mon[36569]: pgmap v20132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:07 smithi104 ceph-mon[36624]: pgmap v20132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:09 smithi081 ceph-mon[31453]: pgmap v20133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:09 smithi153 ceph-mon[36569]: pgmap v20133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:09 smithi104 ceph-mon[36624]: pgmap v20133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:11 smithi081 ceph-mon[31453]: pgmap v20134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:11 smithi153 ceph-mon[36569]: pgmap v20134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:11 smithi104 ceph-mon[36624]: pgmap v20134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:12.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:12.194+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 msgr2=0x7f16e80b3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000587) for more than 900000000 us, fault. 2024-04-13T07:46:12.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:12.194+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3fb0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f16d4006660 tx=0x7f16f408e000 comp rx=0 tx=0).stop 2024-04-13T07:46:12.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:12.194+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 msgr2=0x7f16e80b3fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T07:46:12.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:12.194+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3fb0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T07:46:12.195 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:12.194+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T07:46:12.196 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:12.194+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T07:46:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:12.196+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f16f4010fd0 tx=0x7f16f40bf000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T07:46:13.883 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:13 smithi104 ceph-mon[36624]: pgmap v20135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:13 smithi081 ceph-mon[31453]: pgmap v20135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:13 smithi153 ceph-mon[36569]: pgmap v20135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:15 smithi081 ceph-mon[31453]: pgmap v20136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:15 smithi153 ceph-mon[36569]: pgmap v20136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:15 smithi104 ceph-mon[36624]: pgmap v20136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:17 smithi081 ceph-mon[31453]: pgmap v20137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:17 smithi153 ceph-mon[36569]: pgmap v20137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:17 smithi104 ceph-mon[36624]: pgmap v20137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:18.963+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:46:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:18.963+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:46:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:18.964+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 291 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:46:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:18.964+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 291 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:46:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:19 smithi081 ceph-mon[31453]: pgmap v20138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:19 smithi153 ceph-mon[36569]: pgmap v20138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:19 smithi104 ceph-mon[36624]: pgmap v20138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:21.562+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:46:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:21.562+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:46:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:21.562+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:46:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:21 smithi081 ceph-mon[31453]: pgmap v20139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:21 smithi153 ceph-mon[36569]: pgmap v20139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:21 smithi104 ceph-mon[36624]: pgmap v20139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:23 smithi153 ceph-mon[36569]: pgmap v20140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:23 smithi104 ceph-mon[36624]: pgmap v20140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:23 smithi081 ceph-mon[31453]: pgmap v20140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:25 smithi153 ceph-mon[36569]: pgmap v20141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:25 smithi104 ceph-mon[36624]: pgmap v20141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:25 smithi081 ceph-mon[31453]: pgmap v20141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:27 smithi153 ceph-mon[36569]: pgmap v20142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:27 smithi104 ceph-mon[36624]: pgmap v20142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:28.103 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:27 smithi081 ceph-mon[31453]: pgmap v20142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:29 smithi153 ceph-mon[36569]: pgmap v20143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:29 smithi104 ceph-mon[36624]: pgmap v20143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:29 smithi081 ceph-mon[31453]: pgmap v20143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:32.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:31 smithi153 ceph-mon[36569]: pgmap v20144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:31 smithi104 ceph-mon[36624]: pgmap v20144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:31 smithi081 ceph-mon[31453]: pgmap v20144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:33 smithi153 ceph-mon[36569]: pgmap v20145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:33 smithi104 ceph-mon[36624]: pgmap v20145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:33 smithi081 ceph-mon[31453]: pgmap v20145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:35 smithi153 ceph-mon[36569]: pgmap v20146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:35 smithi104 ceph-mon[36624]: pgmap v20146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:35 smithi081 ceph-mon[31453]: pgmap v20146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:36.563+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:46:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:36.563+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:46:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:36.564+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:46:38.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:37 smithi153 ceph-mon[36569]: pgmap v20147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:37 smithi104 ceph-mon[36624]: pgmap v20147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:37 smithi081 ceph-mon[31453]: pgmap v20147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:39.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:39 smithi081 ceph-mon[31453]: pgmap v20148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:39 smithi153 ceph-mon[36569]: pgmap v20148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:39 smithi104 ceph-mon[36624]: pgmap v20148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:41 smithi153 ceph-mon[36569]: pgmap v20149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:41 smithi104 ceph-mon[36624]: pgmap v20149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:41 smithi081 ceph-mon[31453]: pgmap v20149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:43.975 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:43 smithi081 ceph-mon[31453]: pgmap v20150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:43 smithi153 ceph-mon[36569]: pgmap v20150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:43 smithi104 ceph-mon[36624]: pgmap v20150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:45 smithi153 ceph-mon[36569]: pgmap v20151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:45 smithi104 ceph-mon[36624]: pgmap v20151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:45 smithi081 ceph-mon[31453]: pgmap v20151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:47 smithi153 ceph-mon[36569]: pgmap v20152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:47 smithi104 ceph-mon[36624]: pgmap v20152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:47 smithi081 ceph-mon[31453]: pgmap v20152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:49 smithi153 ceph-mon[36569]: pgmap v20153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:49 smithi104 ceph-mon[36624]: pgmap v20153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:49 smithi081 ceph-mon[31453]: pgmap v20153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:51.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:51.566+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:46:51.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:51.566+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:46:51.567 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:46:51.566+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:46:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:51 smithi153 ceph-mon[36569]: pgmap v20154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:51 smithi104 ceph-mon[36624]: pgmap v20154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:51 smithi081 ceph-mon[31453]: pgmap v20154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:53 smithi153 ceph-mon[36569]: pgmap v20155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:46:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:53 smithi104 ceph-mon[36624]: pgmap v20155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:46:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:53 smithi081 ceph-mon[31453]: pgmap v20155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:46:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:54 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:46:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:54 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:46:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:54 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:46:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:54 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:46:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:54 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:46:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:54 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:46:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:54 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:46:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:54 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:46:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:54 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:46:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:54 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:46:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:54 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:46:55.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:54 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:46:55.775 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:55 smithi153 ceph-mon[36569]: pgmap v20156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:55 smithi104 ceph-mon[36624]: pgmap v20156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:55 smithi081 ceph-mon[31453]: pgmap v20156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: pgmap v20157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: pgmap v20158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: pgmap v20159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: pgmap v20157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:46:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:46:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:46:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:46:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: pgmap v20158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: pgmap v20159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: pgmap v20157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: pgmap v20158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: pgmap v20159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:58.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:46:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:46:59 smithi081 ceph-mon[31453]: pgmap v20160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:46:59 smithi153 ceph-mon[36569]: pgmap v20160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:46:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:46:59 smithi104 ceph-mon[36624]: pgmap v20160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:01 smithi153 ceph-mon[36569]: pgmap v20161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:01 smithi104 ceph-mon[36624]: pgmap v20161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:01 smithi081 ceph-mon[31453]: pgmap v20161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:03 smithi153 ceph-mon[36569]: pgmap v20162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:03 smithi104 ceph-mon[36624]: pgmap v20162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:03 smithi081 ceph-mon[31453]: pgmap v20162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:05 smithi153 ceph-mon[36569]: pgmap v20163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:05 smithi104 ceph-mon[36624]: pgmap v20163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:05 smithi081 ceph-mon[31453]: pgmap v20163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:06.580+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:47:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:06.580+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:47:06.580 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:06.580+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:47:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:07 smithi153 ceph-mon[36569]: pgmap v20164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:07 smithi104 ceph-mon[36624]: pgmap v20164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:07 smithi081 ceph-mon[31453]: pgmap v20164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:09.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:09 smithi081 ceph-mon[31453]: pgmap v20165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:09 smithi153 ceph-mon[36569]: pgmap v20165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:09 smithi104 ceph-mon[36624]: pgmap v20165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:11 smithi153 ceph-mon[36569]: pgmap v20166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:11 smithi104 ceph-mon[36624]: pgmap v20166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:11 smithi081 ceph-mon[31453]: pgmap v20166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:13 smithi153 ceph-mon[36569]: pgmap v20167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:13 smithi104 ceph-mon[36624]: pgmap v20167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:13 smithi081 ceph-mon[31453]: pgmap v20167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:15 smithi081 ceph-mon[31453]: pgmap v20168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:15 smithi153 ceph-mon[36569]: pgmap v20168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:15 smithi104 ceph-mon[36624]: pgmap v20168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:17 smithi153 ceph-mon[36569]: pgmap v20169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:17 smithi104 ceph-mon[36624]: pgmap v20169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:17 smithi081 ceph-mon[31453]: pgmap v20169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:19 smithi153 ceph-mon[36569]: pgmap v20170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:19 smithi104 ceph-mon[36624]: pgmap v20170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:19 smithi081 ceph-mon[31453]: pgmap v20170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:21.595+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:47:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:21.595+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:47:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:21.595+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:47:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:21 smithi153 ceph-mon[36569]: pgmap v20171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:21 smithi104 ceph-mon[36624]: pgmap v20171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:21 smithi081 ceph-mon[31453]: pgmap v20171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:23 smithi153 ceph-mon[36569]: pgmap v20172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:23 smithi104 ceph-mon[36624]: pgmap v20172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:23 smithi081 ceph-mon[31453]: pgmap v20172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:25 smithi153 ceph-mon[36569]: pgmap v20173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:25 smithi104 ceph-mon[36624]: pgmap v20173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:25 smithi081 ceph-mon[31453]: pgmap v20173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:27 smithi153 ceph-mon[36569]: pgmap v20174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:27 smithi104 ceph-mon[36624]: pgmap v20174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:27 smithi081 ceph-mon[31453]: pgmap v20174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:29 smithi153 ceph-mon[36569]: pgmap v20175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:29 smithi104 ceph-mon[36624]: pgmap v20175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:29 smithi081 ceph-mon[31453]: pgmap v20175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:31 smithi153 ceph-mon[36569]: pgmap v20176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:31 smithi104 ceph-mon[36624]: pgmap v20176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:31 smithi081 ceph-mon[31453]: pgmap v20176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:33 smithi153 ceph-mon[36569]: pgmap v20177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:33 smithi104 ceph-mon[36624]: pgmap v20177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:33 smithi081 ceph-mon[31453]: pgmap v20177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:35.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:35 smithi153 ceph-mon[36569]: pgmap v20178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:35 smithi104 ceph-mon[36624]: pgmap v20178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:35 smithi081 ceph-mon[31453]: pgmap v20178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:36.611+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:47:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:36.611+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:47:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:36.611+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:47:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:37 smithi153 ceph-mon[36569]: pgmap v20179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:37 smithi104 ceph-mon[36624]: pgmap v20179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:37 smithi081 ceph-mon[31453]: pgmap v20179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:39.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:39 smithi081 ceph-mon[31453]: pgmap v20180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:39 smithi153 ceph-mon[36569]: pgmap v20180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:39 smithi104 ceph-mon[36624]: pgmap v20180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:41 smithi153 ceph-mon[36569]: pgmap v20181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:41 smithi104 ceph-mon[36624]: pgmap v20181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:41 smithi081 ceph-mon[31453]: pgmap v20181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:43 smithi153 ceph-mon[36569]: pgmap v20182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:43.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:43 smithi104 ceph-mon[36624]: pgmap v20182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:43 smithi081 ceph-mon[31453]: pgmap v20182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:45 smithi153 ceph-mon[36569]: pgmap v20183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:45 smithi104 ceph-mon[36624]: pgmap v20183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:45 smithi081 ceph-mon[31453]: pgmap v20183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:47 smithi153 ceph-mon[36569]: pgmap v20184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:47 smithi104 ceph-mon[36624]: pgmap v20184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:47 smithi081 ceph-mon[31453]: pgmap v20184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:49 smithi153 ceph-mon[36569]: pgmap v20185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:49 smithi104 ceph-mon[36624]: pgmap v20185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:49 smithi081 ceph-mon[31453]: pgmap v20185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:51.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:51.626+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:47:51.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:51.626+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:47:51.627 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:47:51.626+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:47:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:51 smithi153 ceph-mon[36569]: pgmap v20186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:51 smithi104 ceph-mon[36624]: pgmap v20186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:51 smithi081 ceph-mon[31453]: pgmap v20186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:53 smithi153 ceph-mon[36569]: pgmap v20187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:53 smithi104 ceph-mon[36624]: pgmap v20187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:53 smithi081 ceph-mon[31453]: pgmap v20187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:55 smithi153 ceph-mon[36569]: pgmap v20188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:55.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:55 smithi104 ceph-mon[36624]: pgmap v20188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:55 smithi081 ceph-mon[31453]: pgmap v20188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:57 smithi153 ceph-mon[36569]: pgmap v20189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:57 smithi104 ceph-mon[36624]: pgmap v20189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:57 smithi081 ceph-mon[31453]: pgmap v20189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:47:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:47:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:47:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:47:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:47:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:59 smithi153 ceph-mon[36569]: pgmap v20190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:59 smithi153 ceph-mon[36569]: pgmap v20191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:47:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:47:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:47:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:47:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:59 smithi104 ceph-mon[36624]: pgmap v20190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:59 smithi104 ceph-mon[36624]: pgmap v20191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:47:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:47:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:47:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:47:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:59 smithi081 ceph-mon[31453]: pgmap v20190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:59 smithi081 ceph-mon[31453]: pgmap v20191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:47:59.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:47:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:48:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:01 smithi153 ceph-mon[36569]: pgmap v20192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:02.311 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:01 smithi081 ceph-mon[31453]: pgmap v20192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:01 smithi104 ceph-mon[36624]: pgmap v20192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:03 smithi153 ceph-mon[36569]: pgmap v20193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:03 smithi104 ceph-mon[36624]: pgmap v20193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:03 smithi081 ceph-mon[31453]: pgmap v20193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:05 smithi153 ceph-mon[36569]: pgmap v20194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:05 smithi104 ceph-mon[36624]: pgmap v20194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:05 smithi081 ceph-mon[31453]: pgmap v20194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:06.636 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:06.636+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:48:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:06.636+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:48:06.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:06.636+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:48:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:07 smithi153 ceph-mon[36569]: pgmap v20195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:07 smithi104 ceph-mon[36624]: pgmap v20195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:07 smithi081 ceph-mon[31453]: pgmap v20195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:09 smithi153 ceph-mon[36569]: pgmap v20196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:09 smithi104 ceph-mon[36624]: pgmap v20196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:09 smithi081 ceph-mon[31453]: pgmap v20196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:12 smithi153 ceph-mon[36569]: pgmap v20197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:12 smithi104 ceph-mon[36624]: pgmap v20197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:12 smithi081 ceph-mon[31453]: pgmap v20197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:14 smithi153 ceph-mon[36569]: pgmap v20198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:14 smithi104 ceph-mon[36624]: pgmap v20198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:14 smithi081 ceph-mon[31453]: pgmap v20198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:16 smithi153 ceph-mon[36569]: pgmap v20199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:16 smithi104 ceph-mon[36624]: pgmap v20199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:16 smithi081 ceph-mon[31453]: pgmap v20199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:18 smithi153 ceph-mon[36569]: pgmap v20200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:18 smithi104 ceph-mon[36624]: pgmap v20200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:18 smithi081 ceph-mon[31453]: pgmap v20200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:20 smithi153 ceph-mon[36569]: pgmap v20201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:20 smithi104 ceph-mon[36624]: pgmap v20201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:20 smithi081 ceph-mon[31453]: pgmap v20201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:21.640+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:48:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:21.640+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:48:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:21.640+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:48:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:22 smithi153 ceph-mon[36569]: pgmap v20202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:22 smithi104 ceph-mon[36624]: pgmap v20202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:22 smithi081 ceph-mon[31453]: pgmap v20202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:24 smithi153 ceph-mon[36569]: pgmap v20203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:24 smithi104 ceph-mon[36624]: pgmap v20203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:24 smithi081 ceph-mon[31453]: pgmap v20203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:26 smithi153 ceph-mon[36569]: pgmap v20204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:26 smithi104 ceph-mon[36624]: pgmap v20204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:26 smithi081 ceph-mon[31453]: pgmap v20204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:28 smithi153 ceph-mon[36569]: pgmap v20205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:28 smithi104 ceph-mon[36624]: pgmap v20205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:28 smithi081 ceph-mon[31453]: pgmap v20205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:30 smithi153 ceph-mon[36569]: pgmap v20206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:30 smithi104 ceph-mon[36624]: pgmap v20206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:30 smithi081 ceph-mon[31453]: pgmap v20206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:32 smithi153 ceph-mon[36569]: pgmap v20207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:32 smithi104 ceph-mon[36624]: pgmap v20207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:32 smithi081 ceph-mon[31453]: pgmap v20207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:34 smithi104 ceph-mon[36624]: pgmap v20208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:34 smithi081 ceph-mon[31453]: pgmap v20208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:34 smithi153 ceph-mon[36569]: pgmap v20208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:36 smithi104 ceph-mon[36624]: pgmap v20209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:36 smithi081 ceph-mon[31453]: pgmap v20209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:36 smithi153 ceph-mon[36569]: pgmap v20209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:36.644+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:48:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:36.644+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:48:36.645 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:36.644+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:48:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:38 smithi104 ceph-mon[36624]: pgmap v20210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:38 smithi081 ceph-mon[31453]: pgmap v20210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:38 smithi153 ceph-mon[36569]: pgmap v20210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:40 smithi104 ceph-mon[36624]: pgmap v20211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:40 smithi081 ceph-mon[31453]: pgmap v20211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:40 smithi153 ceph-mon[36569]: pgmap v20211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:42 smithi104 ceph-mon[36624]: pgmap v20212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:42 smithi081 ceph-mon[31453]: pgmap v20212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:42 smithi153 ceph-mon[36569]: pgmap v20212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:44 smithi081 ceph-mon[31453]: pgmap v20213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:44 smithi153 ceph-mon[36569]: pgmap v20213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:44 smithi104 ceph-mon[36624]: pgmap v20213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:46 smithi081 ceph-mon[31453]: pgmap v20214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:46 smithi153 ceph-mon[36569]: pgmap v20214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:46 smithi104 ceph-mon[36624]: pgmap v20214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:48 smithi081 ceph-mon[31453]: pgmap v20215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:48 smithi153 ceph-mon[36569]: pgmap v20215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:48 smithi104 ceph-mon[36624]: pgmap v20215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:50 smithi081 ceph-mon[31453]: pgmap v20216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:50 smithi153 ceph-mon[36569]: pgmap v20216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:50 smithi104 ceph-mon[36624]: pgmap v20216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:51.655+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:48:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:51.655+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:48:51.656 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:48:51.655+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:48:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:52 smithi081 ceph-mon[31453]: pgmap v20217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:52 smithi153 ceph-mon[36569]: pgmap v20217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:52 smithi104 ceph-mon[36624]: pgmap v20217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:54 smithi081 ceph-mon[31453]: pgmap v20218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:54 smithi153 ceph-mon[36569]: pgmap v20218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:54 smithi104 ceph-mon[36624]: pgmap v20218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:56 smithi081 ceph-mon[31453]: pgmap v20219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:56 smithi153 ceph-mon[36569]: pgmap v20219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:56 smithi104 ceph-mon[36624]: pgmap v20219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:48:58 smithi081 ceph-mon[31453]: pgmap v20220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:48:58 smithi153 ceph-mon[36569]: pgmap v20220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:48:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:48:58 smithi104 ceph-mon[36624]: pgmap v20220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:00 smithi081 ceph-mon[31453]: pgmap v20221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:00 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:49:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:00 smithi153 ceph-mon[36569]: pgmap v20221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:00 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:49:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:00 smithi104 ceph-mon[36624]: pgmap v20221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:00 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:49:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:49:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:49:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:49:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:49:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:49:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:49:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:49:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:49:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:49:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:02 smithi081 ceph-mon[31453]: pgmap v20222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:02 smithi081 ceph-mon[31453]: pgmap v20223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:02.458 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:02 smithi153 ceph-mon[36569]: pgmap v20222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:02.458 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:02 smithi153 ceph-mon[36569]: pgmap v20223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:02 smithi104 ceph-mon[36624]: pgmap v20222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:02 smithi104 ceph-mon[36624]: pgmap v20223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:04 smithi081 ceph-mon[31453]: pgmap v20224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:04 smithi153 ceph-mon[36569]: pgmap v20224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:04 smithi104 ceph-mon[36624]: pgmap v20224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:06 smithi081 ceph-mon[31453]: pgmap v20225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:06 smithi153 ceph-mon[36569]: pgmap v20225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:06 smithi104 ceph-mon[36624]: pgmap v20225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:06.661+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:49:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:06.661+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:49:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:06.662+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:49:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:08 smithi081 ceph-mon[31453]: pgmap v20226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:08 smithi153 ceph-mon[36569]: pgmap v20226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:08 smithi104 ceph-mon[36624]: pgmap v20226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:10 smithi153 ceph-mon[36569]: pgmap v20227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:10 smithi104 ceph-mon[36624]: pgmap v20227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:10 smithi081 ceph-mon[31453]: pgmap v20227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:12.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:12 smithi153 ceph-mon[36569]: pgmap v20228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:12 smithi104 ceph-mon[36624]: pgmap v20228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:12 smithi081 ceph-mon[31453]: pgmap v20228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:14 smithi153 ceph-mon[36569]: pgmap v20229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:14 smithi104 ceph-mon[36624]: pgmap v20229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:14 smithi081 ceph-mon[31453]: pgmap v20229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:16 smithi153 ceph-mon[36569]: pgmap v20230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:16 smithi104 ceph-mon[36624]: pgmap v20230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:16 smithi081 ceph-mon[31453]: pgmap v20230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:18 smithi153 ceph-mon[36569]: pgmap v20231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:18 smithi104 ceph-mon[36624]: pgmap v20231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:18 smithi081 ceph-mon[31453]: pgmap v20231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:18.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:18.993+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:49:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:18.993+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:49:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:18.994+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 292 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:49:18.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:18.994+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 292 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:49:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:20 smithi081 ceph-mon[31453]: pgmap v20232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:20 smithi153 ceph-mon[36569]: pgmap v20232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:20 smithi104 ceph-mon[36624]: pgmap v20232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:21.673+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:49:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:21.673+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:49:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:21.673+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:49:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:22 smithi153 ceph-mon[36569]: pgmap v20233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:22 smithi104 ceph-mon[36624]: pgmap v20233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:22 smithi081 ceph-mon[31453]: pgmap v20233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:24 smithi153 ceph-mon[36569]: pgmap v20234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:24 smithi104 ceph-mon[36624]: pgmap v20234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:24 smithi081 ceph-mon[31453]: pgmap v20234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:26 smithi153 ceph-mon[36569]: pgmap v20235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:26 smithi104 ceph-mon[36624]: pgmap v20235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:26 smithi081 ceph-mon[31453]: pgmap v20235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:28 smithi153 ceph-mon[36569]: pgmap v20236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:28 smithi104 ceph-mon[36624]: pgmap v20236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:28 smithi081 ceph-mon[31453]: pgmap v20236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:30.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:30 smithi153 ceph-mon[36569]: pgmap v20237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:30 smithi104 ceph-mon[36624]: pgmap v20237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:30 smithi081 ceph-mon[31453]: pgmap v20237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:32 smithi153 ceph-mon[36569]: pgmap v20238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:32 smithi104 ceph-mon[36624]: pgmap v20238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:32 smithi081 ceph-mon[31453]: pgmap v20238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:34 smithi153 ceph-mon[36569]: pgmap v20239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:34 smithi104 ceph-mon[36624]: pgmap v20239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:34 smithi081 ceph-mon[31453]: pgmap v20239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:36 smithi081 ceph-mon[31453]: pgmap v20240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:36 smithi153 ceph-mon[36569]: pgmap v20240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:36 smithi104 ceph-mon[36624]: pgmap v20240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:36.674+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:49:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:36.674+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:49:36.675 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:36.674+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:49:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:38 smithi153 ceph-mon[36569]: pgmap v20241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:38 smithi104 ceph-mon[36624]: pgmap v20241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:38 smithi081 ceph-mon[31453]: pgmap v20241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:40 smithi153 ceph-mon[36569]: pgmap v20242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:40 smithi104 ceph-mon[36624]: pgmap v20242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:40 smithi081 ceph-mon[31453]: pgmap v20242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:42 smithi153 ceph-mon[36569]: pgmap v20243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:42 smithi104 ceph-mon[36624]: pgmap v20243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:42 smithi081 ceph-mon[31453]: pgmap v20243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:44 smithi153 ceph-mon[36569]: pgmap v20244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:44 smithi104 ceph-mon[36624]: pgmap v20244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:44 smithi081 ceph-mon[31453]: pgmap v20244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:46 smithi153 ceph-mon[36569]: pgmap v20245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:46 smithi104 ceph-mon[36624]: pgmap v20245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:46 smithi081 ceph-mon[31453]: pgmap v20245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:48 smithi153 ceph-mon[36569]: pgmap v20246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:48 smithi104 ceph-mon[36624]: pgmap v20246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:48 smithi081 ceph-mon[31453]: pgmap v20246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:50 smithi153 ceph-mon[36569]: pgmap v20247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:50 smithi104 ceph-mon[36624]: pgmap v20247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:50 smithi081 ceph-mon[31453]: pgmap v20247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:51.690+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:49:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:51.690+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:49:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:49:51.690+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:49:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:52 smithi153 ceph-mon[36569]: pgmap v20248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:52 smithi104 ceph-mon[36624]: pgmap v20248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:52 smithi081 ceph-mon[31453]: pgmap v20248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:54 smithi153 ceph-mon[36569]: pgmap v20249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:54 smithi104 ceph-mon[36624]: pgmap v20249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:54 smithi081 ceph-mon[31453]: pgmap v20249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:56.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:56 smithi153 ceph-mon[36569]: pgmap v20250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:56 smithi104 ceph-mon[36624]: pgmap v20250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:56 smithi081 ceph-mon[31453]: pgmap v20250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:49:58 smithi153 ceph-mon[36569]: pgmap v20251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:49:58 smithi104 ceph-mon[36624]: pgmap v20251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:49:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:49:58 smithi081 ceph-mon[31453]: pgmap v20251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:00 smithi153 ceph-mon[36569]: pgmap v20252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:50:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:00 smithi104 ceph-mon[36624]: pgmap v20252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:50:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:00 smithi081 ceph-mon[31453]: pgmap v20252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T07:50:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:50:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:50:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:50:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:02 smithi104 ceph-mon[36624]: pgmap v20253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:50:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:50:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:50:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:02 smithi081 ceph-mon[31453]: pgmap v20253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:50:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:50:02.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:50:02.708 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:02 smithi153 ceph-mon[36569]: pgmap v20253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:02.708 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:50:02.708 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:50:02.708 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:50:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:03 smithi104 ceph-mon[36624]: pgmap v20254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:03 smithi104 ceph-mon[36624]: pgmap v20255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:03 smithi081 ceph-mon[31453]: pgmap v20254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:03 smithi081 ceph-mon[31453]: pgmap v20255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:03 smithi153 ceph-mon[36569]: pgmap v20254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:03 smithi153 ceph-mon[36569]: pgmap v20255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:05 smithi081 ceph-mon[31453]: pgmap v20256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:05 smithi153 ceph-mon[36569]: pgmap v20256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:05 smithi104 ceph-mon[36624]: pgmap v20256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:06.700 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:06.700+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:50:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:06.700+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:50:06.701 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:06.700+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:50:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:07 smithi081 ceph-mon[31453]: pgmap v20257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:07 smithi153 ceph-mon[36569]: pgmap v20257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:07 smithi104 ceph-mon[36624]: pgmap v20257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:09 smithi081 ceph-mon[31453]: pgmap v20258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:09 smithi153 ceph-mon[36569]: pgmap v20258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:09 smithi104 ceph-mon[36624]: pgmap v20258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:11 smithi104 ceph-mon[36624]: pgmap v20259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:11 smithi081 ceph-mon[31453]: pgmap v20259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:11 smithi153 ceph-mon[36569]: pgmap v20259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:13 smithi081 ceph-mon[31453]: pgmap v20260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:13 smithi153 ceph-mon[36569]: pgmap v20260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:13 smithi104 ceph-mon[36624]: pgmap v20260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:15.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:15 smithi104 ceph-mon[36624]: pgmap v20261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:15 smithi081 ceph-mon[31453]: pgmap v20261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:15 smithi153 ceph-mon[36569]: pgmap v20261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:17 smithi081 ceph-mon[31453]: pgmap v20262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:17 smithi153 ceph-mon[36569]: pgmap v20262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:17 smithi104 ceph-mon[36624]: pgmap v20262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:19 smithi081 ceph-mon[31453]: pgmap v20263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:19 smithi153 ceph-mon[36569]: pgmap v20263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:19 smithi104 ceph-mon[36624]: pgmap v20263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:21 smithi081 ceph-mon[31453]: pgmap v20264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:21.711+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:50:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:21.711+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:50:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:21.711+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:50:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:21 smithi153 ceph-mon[36569]: pgmap v20264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:21 smithi104 ceph-mon[36624]: pgmap v20264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:23 smithi081 ceph-mon[31453]: pgmap v20265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:23 smithi153 ceph-mon[36569]: pgmap v20265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:23 smithi104 ceph-mon[36624]: pgmap v20265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:25 smithi081 ceph-mon[31453]: pgmap v20266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:25 smithi153 ceph-mon[36569]: pgmap v20266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:25 smithi104 ceph-mon[36624]: pgmap v20266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:27 smithi081 ceph-mon[31453]: pgmap v20267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:27 smithi153 ceph-mon[36569]: pgmap v20267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:27 smithi104 ceph-mon[36624]: pgmap v20267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:29 smithi081 ceph-mon[31453]: pgmap v20268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:29 smithi153 ceph-mon[36569]: pgmap v20268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:29 smithi104 ceph-mon[36624]: pgmap v20268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:31 smithi081 ceph-mon[31453]: pgmap v20269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:31 smithi153 ceph-mon[36569]: pgmap v20269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:31 smithi104 ceph-mon[36624]: pgmap v20269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:33 smithi081 ceph-mon[31453]: pgmap v20270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:33 smithi153 ceph-mon[36569]: pgmap v20270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:33 smithi104 ceph-mon[36624]: pgmap v20270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:35 smithi153 ceph-mon[36569]: pgmap v20271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:35 smithi104 ceph-mon[36624]: pgmap v20271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:35 smithi081 ceph-mon[31453]: pgmap v20271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:36.722+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:50:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:36.723+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:50:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:36.723+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:50:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:37 smithi153 ceph-mon[36569]: pgmap v20272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:37 smithi104 ceph-mon[36624]: pgmap v20272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:37 smithi081 ceph-mon[31453]: pgmap v20272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:39 smithi153 ceph-mon[36569]: pgmap v20273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:39 smithi104 ceph-mon[36624]: pgmap v20273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:39 smithi081 ceph-mon[31453]: pgmap v20273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:41.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:41 smithi081 ceph-mon[31453]: pgmap v20274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:41 smithi153 ceph-mon[36569]: pgmap v20274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:41 smithi104 ceph-mon[36624]: pgmap v20274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:43 smithi153 ceph-mon[36569]: pgmap v20275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:43 smithi104 ceph-mon[36624]: pgmap v20275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:43 smithi081 ceph-mon[31453]: pgmap v20275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:44 smithi153 ceph-mon[36569]: pgmap v20276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:44 smithi104 ceph-mon[36624]: pgmap v20276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:44 smithi081 ceph-mon[31453]: pgmap v20276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:47 smithi153 ceph-mon[36569]: pgmap v20277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:47 smithi104 ceph-mon[36624]: pgmap v20277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:47 smithi081 ceph-mon[31453]: pgmap v20277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:49 smithi153 ceph-mon[36569]: pgmap v20278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:49 smithi104 ceph-mon[36624]: pgmap v20278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:49 smithi081 ceph-mon[31453]: pgmap v20278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:51 smithi153 ceph-mon[36569]: pgmap v20279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:51 smithi104 ceph-mon[36624]: pgmap v20279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:51 smithi081 ceph-mon[31453]: pgmap v20279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:51.738 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:51.738+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:50:51.738 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:51.738+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:50:51.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:50:51.738+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:50:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:53 smithi153 ceph-mon[36569]: pgmap v20280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:53.319 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:53 smithi081 ceph-mon[31453]: pgmap v20280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:53 smithi104 ceph-mon[36624]: pgmap v20280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:55.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:55 smithi153 ceph-mon[36569]: pgmap v20281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:55 smithi104 ceph-mon[36624]: pgmap v20281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:55 smithi081 ceph-mon[31453]: pgmap v20281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:57 smithi104 ceph-mon[36624]: pgmap v20282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:57 smithi081 ceph-mon[31453]: pgmap v20282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:57 smithi153 ceph-mon[36569]: pgmap v20282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:50:59 smithi104 ceph-mon[36624]: pgmap v20283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:50:59 smithi081 ceph-mon[31453]: pgmap v20283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:50:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:50:59 smithi153 ceph-mon[36569]: pgmap v20283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:01 smithi104 ceph-mon[36624]: pgmap v20284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:01 smithi081 ceph-mon[31453]: pgmap v20284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:01 smithi153 ceph-mon[36569]: pgmap v20284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:03 smithi104 ceph-mon[36624]: pgmap v20285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:51:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.352 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:03 smithi153 ceph-mon[36569]: pgmap v20285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:03.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:51:03.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.353 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:03 smithi081 ceph-mon[31453]: pgmap v20285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:51:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:05 smithi104 ceph-mon[36624]: pgmap v20286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:05 smithi081 ceph-mon[31453]: pgmap v20286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:05 smithi153 ceph-mon[36569]: pgmap v20286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:06.753+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:51:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:06.753+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:51:06.753 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:06.753+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:51:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:07 smithi153 ceph-mon[36569]: pgmap v20287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:07 smithi104 ceph-mon[36624]: pgmap v20287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:07 smithi081 ceph-mon[31453]: pgmap v20287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:08.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:09 smithi081 ceph-mon[31453]: pgmap v20288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:51:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:51:09.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:09 smithi153 ceph-mon[36569]: pgmap v20288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:51:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:51:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:09 smithi104 ceph-mon[36624]: pgmap v20288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:51:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:51:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:51:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:10 smithi153 ceph-mon[36569]: pgmap v20289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:10 smithi153 ceph-mon[36569]: pgmap v20290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:10 smithi104 ceph-mon[36624]: pgmap v20289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:10 smithi104 ceph-mon[36624]: pgmap v20290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:10 smithi081 ceph-mon[31453]: pgmap v20289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:10 smithi081 ceph-mon[31453]: pgmap v20290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:11 smithi153 ceph-mon[36569]: pgmap v20291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:11.809 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:11 smithi081 ceph-mon[31453]: pgmap v20291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:11 smithi104 ceph-mon[36624]: pgmap v20291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:14.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:13 smithi153 ceph-mon[36569]: pgmap v20292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:13 smithi104 ceph-mon[36624]: pgmap v20292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:13 smithi081 ceph-mon[31453]: pgmap v20292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:15 smithi153 ceph-mon[36569]: pgmap v20293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:15 smithi104 ceph-mon[36624]: pgmap v20293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:15 smithi081 ceph-mon[31453]: pgmap v20293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:17 smithi153 ceph-mon[36569]: pgmap v20294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:17 smithi104 ceph-mon[36624]: pgmap v20294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:17 smithi081 ceph-mon[31453]: pgmap v20294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:20.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:19 smithi153 ceph-mon[36569]: pgmap v20295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:19 smithi104 ceph-mon[36624]: pgmap v20295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:19 smithi081 ceph-mon[31453]: pgmap v20295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:21.767+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:51:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:21.767+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:51:21.768 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:21.767+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:51:22.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:21 smithi153 ceph-mon[36569]: pgmap v20296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:21 smithi104 ceph-mon[36624]: pgmap v20296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:21 smithi081 ceph-mon[31453]: pgmap v20296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:23 smithi153 ceph-mon[36569]: pgmap v20297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:23 smithi104 ceph-mon[36624]: pgmap v20297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:23 smithi081 ceph-mon[31453]: pgmap v20297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:25 smithi153 ceph-mon[36569]: pgmap v20298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:25 smithi104 ceph-mon[36624]: pgmap v20298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:25 smithi081 ceph-mon[31453]: pgmap v20298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:27 smithi153 ceph-mon[36569]: pgmap v20299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:27 smithi104 ceph-mon[36624]: pgmap v20299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:27 smithi081 ceph-mon[31453]: pgmap v20299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:29 smithi104 ceph-mon[36624]: pgmap v20300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:29 smithi081 ceph-mon[31453]: pgmap v20300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:30.311 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:29 smithi153 ceph-mon[36569]: pgmap v20300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:31 smithi104 ceph-mon[36624]: pgmap v20301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:31 smithi081 ceph-mon[31453]: pgmap v20301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:31 smithi153 ceph-mon[36569]: pgmap v20301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:34.015+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:51:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:34.015+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:51:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:34.016+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 293 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:51:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:34.016+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 293 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:51:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:33 smithi104 ceph-mon[36624]: pgmap v20302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:34.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:33 smithi153 ceph-mon[36569]: pgmap v20302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:33 smithi081 ceph-mon[31453]: pgmap v20302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:35 smithi104 ceph-mon[36624]: pgmap v20303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:35 smithi081 ceph-mon[31453]: pgmap v20303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:35 smithi153 ceph-mon[36569]: pgmap v20303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:36.783+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:51:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:36.783+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:51:36.783 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:36.783+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:51:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:37 smithi081 ceph-mon[31453]: pgmap v20304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:37 smithi153 ceph-mon[36569]: pgmap v20304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:37 smithi104 ceph-mon[36624]: pgmap v20304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:39 smithi081 ceph-mon[31453]: pgmap v20305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:39 smithi153 ceph-mon[36569]: pgmap v20305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:39 smithi104 ceph-mon[36624]: pgmap v20305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:41 smithi081 ceph-mon[31453]: pgmap v20306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:41 smithi153 ceph-mon[36569]: pgmap v20306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:41 smithi104 ceph-mon[36624]: pgmap v20306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:43 smithi081 ceph-mon[31453]: pgmap v20307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:43 smithi153 ceph-mon[36569]: pgmap v20307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:43 smithi104 ceph-mon[36624]: pgmap v20307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:45 smithi081 ceph-mon[31453]: pgmap v20308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:45 smithi153 ceph-mon[36569]: pgmap v20308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:45 smithi104 ceph-mon[36624]: pgmap v20308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:48.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:47 smithi104 ceph-mon[36624]: pgmap v20309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:47 smithi081 ceph-mon[31453]: pgmap v20309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:47 smithi153 ceph-mon[36569]: pgmap v20309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:49 smithi081 ceph-mon[31453]: pgmap v20310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:49 smithi153 ceph-mon[36569]: pgmap v20310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:49 smithi104 ceph-mon[36624]: pgmap v20310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:51.791+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:51:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:51.791+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:51:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:51:51.791+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:51:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:51 smithi081 ceph-mon[31453]: pgmap v20311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:51 smithi153 ceph-mon[36569]: pgmap v20311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:51 smithi104 ceph-mon[36624]: pgmap v20311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:53 smithi081 ceph-mon[31453]: pgmap v20312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:54.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:53 smithi153 ceph-mon[36569]: pgmap v20312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:53 smithi104 ceph-mon[36624]: pgmap v20312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:55 smithi081 ceph-mon[31453]: pgmap v20313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:55 smithi153 ceph-mon[36569]: pgmap v20313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:55 smithi104 ceph-mon[36624]: pgmap v20313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:57 smithi081 ceph-mon[31453]: pgmap v20314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:57 smithi153 ceph-mon[36569]: pgmap v20314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:51:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:57 smithi104 ceph-mon[36624]: pgmap v20314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:51:59 smithi081 ceph-mon[31453]: pgmap v20315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:51:59 smithi153 ceph-mon[36569]: pgmap v20315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:51:59 smithi104 ceph-mon[36624]: pgmap v20315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:01 smithi081 ceph-mon[31453]: pgmap v20316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:01 smithi153 ceph-mon[36569]: pgmap v20316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:01 smithi104 ceph-mon[36624]: pgmap v20316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:03 smithi081 ceph-mon[31453]: pgmap v20317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:04.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:03 smithi153 ceph-mon[36569]: pgmap v20317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:03 smithi104 ceph-mon[36624]: pgmap v20317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:05 smithi153 ceph-mon[36569]: pgmap v20318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:05 smithi104 ceph-mon[36624]: pgmap v20318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:05 smithi081 ceph-mon[31453]: pgmap v20318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:06.799+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:52:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:06.799+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:52:06.799 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:06.799+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:52:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:07 smithi153 ceph-mon[36569]: pgmap v20319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:07 smithi104 ceph-mon[36624]: pgmap v20319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:07 smithi081 ceph-mon[31453]: pgmap v20319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:09 smithi153 ceph-mon[36569]: pgmap v20320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:52:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:09 smithi104 ceph-mon[36624]: pgmap v20320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:52:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:09 smithi081 ceph-mon[31453]: pgmap v20320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:52:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:52:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:52:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:10 smithi153 ceph-mon[36569]: pgmap v20321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:10 smithi153 ceph-mon[36569]: pgmap v20322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:11.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:11.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:52:11.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:52:11.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:10 smithi081 ceph-mon[31453]: pgmap v20321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:11.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:10 smithi081 ceph-mon[31453]: pgmap v20322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:11.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:52:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:52:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:10 smithi104 ceph-mon[36624]: pgmap v20321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:10 smithi104 ceph-mon[36624]: pgmap v20322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:52:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:13 smithi081 ceph-mon[31453]: pgmap v20323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:13 smithi153 ceph-mon[36569]: pgmap v20323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:13 smithi104 ceph-mon[36624]: pgmap v20323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:15 smithi081 ceph-mon[31453]: pgmap v20324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:15 smithi153 ceph-mon[36569]: pgmap v20324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:15 smithi104 ceph-mon[36624]: pgmap v20324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:17 smithi153 ceph-mon[36569]: pgmap v20325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:17 smithi104 ceph-mon[36624]: pgmap v20325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:17 smithi081 ceph-mon[31453]: pgmap v20325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:19.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:19.023+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:52:19.023 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:19.023+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:52:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:19.024+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 294 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:52:19.024 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:19.024+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 294 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:52:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:19 smithi153 ceph-mon[36569]: pgmap v20326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:19 smithi104 ceph-mon[36624]: pgmap v20326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:19 smithi081 ceph-mon[31453]: pgmap v20326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:21.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:21.806+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:52:21.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:21.806+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:52:21.807 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:21.806+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:52:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:21 smithi153 ceph-mon[36569]: pgmap v20327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:21 smithi104 ceph-mon[36624]: pgmap v20327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:21 smithi081 ceph-mon[31453]: pgmap v20327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:23 smithi153 ceph-mon[36569]: pgmap v20328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:23 smithi104 ceph-mon[36624]: pgmap v20328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:23 smithi081 ceph-mon[31453]: pgmap v20328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:25 smithi153 ceph-mon[36569]: pgmap v20329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:25 smithi104 ceph-mon[36624]: pgmap v20329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:25 smithi081 ceph-mon[31453]: pgmap v20329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:27 smithi153 ceph-mon[36569]: pgmap v20330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:27 smithi104 ceph-mon[36624]: pgmap v20330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:27 smithi081 ceph-mon[31453]: pgmap v20330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:29 smithi153 ceph-mon[36569]: pgmap v20331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:29 smithi104 ceph-mon[36624]: pgmap v20331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:29 smithi081 ceph-mon[31453]: pgmap v20331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:31 smithi153 ceph-mon[36569]: pgmap v20332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:31 smithi104 ceph-mon[36624]: pgmap v20332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:31 smithi081 ceph-mon[31453]: pgmap v20332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:32 smithi153 ceph-mon[36569]: pgmap v20333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:32 smithi104 ceph-mon[36624]: pgmap v20333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:32 smithi081 ceph-mon[31453]: pgmap v20333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:35 smithi153 ceph-mon[36569]: pgmap v20334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:35 smithi104 ceph-mon[36624]: pgmap v20334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:35 smithi081 ceph-mon[31453]: pgmap v20334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:36.819+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:52:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:36.819+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:52:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:36.819+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:52:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:37 smithi153 ceph-mon[36569]: pgmap v20335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:37 smithi104 ceph-mon[36624]: pgmap v20335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:37 smithi081 ceph-mon[31453]: pgmap v20335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:39 smithi153 ceph-mon[36569]: pgmap v20336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:39 smithi104 ceph-mon[36624]: pgmap v20336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:39 smithi081 ceph-mon[31453]: pgmap v20336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:41.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:41 smithi081 ceph-mon[31453]: pgmap v20337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:41 smithi153 ceph-mon[36569]: pgmap v20337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:41 smithi104 ceph-mon[36624]: pgmap v20337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:43.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:43 smithi153 ceph-mon[36569]: pgmap v20338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:43 smithi104 ceph-mon[36624]: pgmap v20338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:43 smithi081 ceph-mon[31453]: pgmap v20338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:45 smithi153 ceph-mon[36569]: pgmap v20339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:45 smithi104 ceph-mon[36624]: pgmap v20339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:45 smithi081 ceph-mon[31453]: pgmap v20339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:47.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:47 smithi153 ceph-mon[36569]: pgmap v20340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:47 smithi104 ceph-mon[36624]: pgmap v20340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:47 smithi081 ceph-mon[31453]: pgmap v20340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:49 smithi153 ceph-mon[36569]: pgmap v20341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:49 smithi104 ceph-mon[36624]: pgmap v20341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:49 smithi081 ceph-mon[31453]: pgmap v20341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:51 smithi153 ceph-mon[36569]: pgmap v20342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:51.829 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:51.829+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:52:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:51.829+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:52:51.830 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:52:51.829+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:52:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:51 smithi104 ceph-mon[36624]: pgmap v20342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:51.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:51 smithi081 ceph-mon[31453]: pgmap v20342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:52 smithi153 ceph-mon[36569]: pgmap v20343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:52 smithi104 ceph-mon[36624]: pgmap v20343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:52 smithi081 ceph-mon[31453]: pgmap v20343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:55 smithi153 ceph-mon[36569]: pgmap v20344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:55 smithi104 ceph-mon[36624]: pgmap v20344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:55 smithi081 ceph-mon[31453]: pgmap v20344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:57 smithi081 ceph-mon[31453]: pgmap v20345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:57 smithi153 ceph-mon[36569]: pgmap v20345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:57 smithi104 ceph-mon[36624]: pgmap v20345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:52:59 smithi153 ceph-mon[36569]: pgmap v20346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:52:59 smithi104 ceph-mon[36624]: pgmap v20346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:52:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:52:59 smithi081 ceph-mon[31453]: pgmap v20346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:01 smithi153 ceph-mon[36569]: pgmap v20347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:01 smithi104 ceph-mon[36624]: pgmap v20347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:01 smithi081 ceph-mon[31453]: pgmap v20347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:03 smithi153 ceph-mon[36569]: pgmap v20348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:03 smithi104 ceph-mon[36624]: pgmap v20348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:03 smithi081 ceph-mon[31453]: pgmap v20348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:05 smithi153 ceph-mon[36569]: pgmap v20349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:05 smithi104 ceph-mon[36624]: pgmap v20349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:05 smithi081 ceph-mon[31453]: pgmap v20349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:06.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:06.843+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:53:06.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:06.843+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:53:06.845 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:06.843+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:53:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:07 smithi153 ceph-mon[36569]: pgmap v20350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:07 smithi104 ceph-mon[36624]: pgmap v20350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:07 smithi081 ceph-mon[31453]: pgmap v20350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:09 smithi153 ceph-mon[36569]: pgmap v20351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:09 smithi104 ceph-mon[36624]: pgmap v20351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:09 smithi081 ceph-mon[31453]: pgmap v20351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:10 smithi153 ceph-mon[36569]: pgmap v20352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:10 smithi104 ceph-mon[36624]: pgmap v20352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:10 smithi081 ceph-mon[31453]: pgmap v20352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:53:11.807 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:53:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:53:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:53:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:53:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:12 smithi153 ceph-mon[36569]: pgmap v20353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:12 smithi153 ceph-mon[36569]: pgmap v20354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:12.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:53:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:53:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:53:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:12 smithi104 ceph-mon[36624]: pgmap v20353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:12 smithi104 ceph-mon[36624]: pgmap v20354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:53:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:53:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:53:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:12 smithi081 ceph-mon[31453]: pgmap v20353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:12 smithi081 ceph-mon[31453]: pgmap v20354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:12.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:53:15.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:14 smithi153 ceph-mon[36569]: pgmap v20355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:14 smithi104 ceph-mon[36624]: pgmap v20355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:14 smithi081 ceph-mon[31453]: pgmap v20355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:16 smithi153 ceph-mon[36569]: pgmap v20356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:16 smithi104 ceph-mon[36624]: pgmap v20356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:16 smithi081 ceph-mon[31453]: pgmap v20356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:18 smithi153 ceph-mon[36569]: pgmap v20357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:18 smithi104 ceph-mon[36624]: pgmap v20357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:18 smithi081 ceph-mon[31453]: pgmap v20357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:20 smithi153 ceph-mon[36569]: pgmap v20358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:20 smithi104 ceph-mon[36624]: pgmap v20358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:20 smithi081 ceph-mon[31453]: pgmap v20358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:21.859 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:21.859+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:53:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:21.859+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:53:21.860 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:21.859+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:53:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:22 smithi153 ceph-mon[36569]: pgmap v20359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:22 smithi104 ceph-mon[36624]: pgmap v20359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:22 smithi081 ceph-mon[31453]: pgmap v20359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:24 smithi153 ceph-mon[36569]: pgmap v20360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:24 smithi104 ceph-mon[36624]: pgmap v20360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:24 smithi081 ceph-mon[31453]: pgmap v20360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:27.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:26 smithi153 ceph-mon[36569]: pgmap v20361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:26 smithi104 ceph-mon[36624]: pgmap v20361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:26 smithi081 ceph-mon[31453]: pgmap v20361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:29.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:29 smithi153 ceph-mon[36569]: pgmap v20362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:29 smithi104 ceph-mon[36624]: pgmap v20362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:29 smithi081 ceph-mon[31453]: pgmap v20362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:31.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:31 smithi153 ceph-mon[36569]: pgmap v20363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:31 smithi104 ceph-mon[36624]: pgmap v20363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:31 smithi081 ceph-mon[31453]: pgmap v20363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:32 smithi153 ceph-mon[36569]: pgmap v20364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:32 smithi104 ceph-mon[36624]: pgmap v20364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:32 smithi081 ceph-mon[31453]: pgmap v20364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:35.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:34 smithi153 ceph-mon[36569]: pgmap v20365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:34 smithi104 ceph-mon[36624]: pgmap v20365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:34 smithi081 ceph-mon[31453]: pgmap v20365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:36.872+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:53:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:36.872+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:53:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:36.872+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:53:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:36 smithi153 ceph-mon[36569]: pgmap v20366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:36 smithi104 ceph-mon[36624]: pgmap v20366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:36 smithi081 ceph-mon[31453]: pgmap v20366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:38 smithi153 ceph-mon[36569]: pgmap v20367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:38 smithi104 ceph-mon[36624]: pgmap v20367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:39.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:38 smithi081 ceph-mon[31453]: pgmap v20367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:41.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:40 smithi153 ceph-mon[36569]: pgmap v20368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:40 smithi104 ceph-mon[36624]: pgmap v20368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:40 smithi081 ceph-mon[31453]: pgmap v20368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:43.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:42 smithi153 ceph-mon[36569]: pgmap v20369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:43.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:42 smithi104 ceph-mon[36624]: pgmap v20369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:42 smithi081 ceph-mon[31453]: pgmap v20369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:45 smithi153 ceph-mon[36569]: pgmap v20370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:45 smithi104 ceph-mon[36624]: pgmap v20370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:45 smithi081 ceph-mon[31453]: pgmap v20370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:47.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:47 smithi153 ceph-mon[36569]: pgmap v20371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:47 smithi104 ceph-mon[36624]: pgmap v20371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:47 smithi081 ceph-mon[31453]: pgmap v20371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:49 smithi153 ceph-mon[36569]: pgmap v20372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:49 smithi104 ceph-mon[36624]: pgmap v20372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:49 smithi081 ceph-mon[31453]: pgmap v20372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:51.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:51 smithi153 ceph-mon[36569]: pgmap v20373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:51 smithi104 ceph-mon[36624]: pgmap v20373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:51 smithi081 ceph-mon[31453]: pgmap v20373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:51.879+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:53:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:51.879+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:53:51.881 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:53:51.879+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:53:53.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:53 smithi153 ceph-mon[36569]: pgmap v20374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:53.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:53 smithi104 ceph-mon[36624]: pgmap v20374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:53 smithi081 ceph-mon[31453]: pgmap v20374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:55 smithi153 ceph-mon[36569]: pgmap v20375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:55.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:55 smithi104 ceph-mon[36624]: pgmap v20375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:55 smithi081 ceph-mon[31453]: pgmap v20375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:57 smithi153 ceph-mon[36569]: pgmap v20376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:57 smithi104 ceph-mon[36624]: pgmap v20376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:57 smithi081 ceph-mon[31453]: pgmap v20376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:59.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:53:59 smithi153 ceph-mon[36569]: pgmap v20377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:53:59 smithi104 ceph-mon[36624]: pgmap v20377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:53:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:53:59 smithi081 ceph-mon[31453]: pgmap v20377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:01.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:01 smithi153 ceph-mon[36569]: pgmap v20378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:01 smithi104 ceph-mon[36624]: pgmap v20378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:01 smithi081 ceph-mon[31453]: pgmap v20378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:03 smithi153 ceph-mon[36569]: pgmap v20379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:03 smithi104 ceph-mon[36624]: pgmap v20379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:03 smithi081 ceph-mon[31453]: pgmap v20379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:05 smithi153 ceph-mon[36569]: pgmap v20380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:05 smithi104 ceph-mon[36624]: pgmap v20380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:05 smithi081 ceph-mon[31453]: pgmap v20380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:06.890+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:54:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:06.890+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:54:06.890 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:06.890+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:54:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:07 smithi104 ceph-mon[36624]: pgmap v20381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:07 smithi081 ceph-mon[31453]: pgmap v20381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:07.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:07 smithi153 ceph-mon[36569]: pgmap v20381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:09 smithi104 ceph-mon[36624]: pgmap v20382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:09 smithi081 ceph-mon[31453]: pgmap v20382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:09 smithi153 ceph-mon[36569]: pgmap v20382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:11 smithi104 ceph-mon[36624]: pgmap v20383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:11 smithi081 ceph-mon[31453]: pgmap v20383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:11 smithi153 ceph-mon[36569]: pgmap v20383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:13 smithi104 ceph-mon[36624]: pgmap v20384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:54:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:13 smithi081 ceph-mon[31453]: pgmap v20384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:54:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:13 smithi153 ceph-mon[36569]: pgmap v20384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:54:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:54:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:54:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:14 smithi081 ceph-mon[31453]: pgmap v20385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:14 smithi081 ceph-mon[31453]: pgmap v20386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:14.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:54:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:54:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:54:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:14 smithi153 ceph-mon[36569]: pgmap v20385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:14 smithi153 ceph-mon[36569]: pgmap v20386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:14.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:54:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:54:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:54:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:14 smithi104 ceph-mon[36624]: pgmap v20385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:14 smithi104 ceph-mon[36624]: pgmap v20386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:54:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:15 smithi153 ceph-mon[36569]: pgmap v20387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:15 smithi104 ceph-mon[36624]: pgmap v20387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:15 smithi081 ceph-mon[31453]: pgmap v20387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:17 smithi153 ceph-mon[36569]: pgmap v20388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:17 smithi104 ceph-mon[36624]: pgmap v20388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:17 smithi081 ceph-mon[31453]: pgmap v20388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:19 smithi153 ceph-mon[36569]: pgmap v20389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:19 smithi104 ceph-mon[36624]: pgmap v20389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:19 smithi081 ceph-mon[31453]: pgmap v20389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:21 smithi153 ceph-mon[36569]: pgmap v20390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:21 smithi104 ceph-mon[36624]: pgmap v20390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:21.898 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:21.898+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:54:21.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:21.898+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:54:21.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:21.898+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:54:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:21 smithi081 ceph-mon[31453]: pgmap v20390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:23 smithi153 ceph-mon[36569]: pgmap v20391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:23 smithi104 ceph-mon[36624]: pgmap v20391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:23 smithi081 ceph-mon[31453]: pgmap v20391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:25.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:25 smithi153 ceph-mon[36569]: pgmap v20392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:25 smithi104 ceph-mon[36624]: pgmap v20392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:25 smithi081 ceph-mon[31453]: pgmap v20392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:27 smithi153 ceph-mon[36569]: pgmap v20393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:27 smithi104 ceph-mon[36624]: pgmap v20393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:27 smithi081 ceph-mon[31453]: pgmap v20393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:29 smithi153 ceph-mon[36569]: pgmap v20394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:29 smithi104 ceph-mon[36624]: pgmap v20394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:29 smithi081 ceph-mon[31453]: pgmap v20394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:31 smithi153 ceph-mon[36569]: pgmap v20395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:31 smithi104 ceph-mon[36624]: pgmap v20395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:31 smithi081 ceph-mon[31453]: pgmap v20395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:33 smithi153 ceph-mon[36569]: pgmap v20396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:33 smithi104 ceph-mon[36624]: pgmap v20396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:33 smithi081 ceph-mon[31453]: pgmap v20396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:35 smithi153 ceph-mon[36569]: pgmap v20397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:35 smithi104 ceph-mon[36624]: pgmap v20397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:35 smithi081 ceph-mon[31453]: pgmap v20397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:36.914+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:54:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:36.914+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:54:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:36.914+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:54:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:37 smithi153 ceph-mon[36569]: pgmap v20398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:37 smithi104 ceph-mon[36624]: pgmap v20398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:37 smithi081 ceph-mon[31453]: pgmap v20398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:39 smithi153 ceph-mon[36569]: pgmap v20399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:39 smithi104 ceph-mon[36624]: pgmap v20399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:39 smithi081 ceph-mon[31453]: pgmap v20399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:41 smithi153 ceph-mon[36569]: pgmap v20400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:41 smithi104 ceph-mon[36624]: pgmap v20400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:41 smithi081 ceph-mon[31453]: pgmap v20400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:43 smithi153 ceph-mon[36569]: pgmap v20401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:43 smithi104 ceph-mon[36624]: pgmap v20401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:43 smithi081 ceph-mon[31453]: pgmap v20401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:45 smithi153 ceph-mon[36569]: pgmap v20402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:45 smithi104 ceph-mon[36624]: pgmap v20402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:45 smithi081 ceph-mon[31453]: pgmap v20402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:47 smithi153 ceph-mon[36569]: pgmap v20403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:47 smithi104 ceph-mon[36624]: pgmap v20403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:47 smithi081 ceph-mon[31453]: pgmap v20403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:49 smithi153 ceph-mon[36569]: pgmap v20404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:49 smithi104 ceph-mon[36624]: pgmap v20404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:49 smithi081 ceph-mon[31453]: pgmap v20404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:51 smithi153 ceph-mon[36569]: pgmap v20405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:51 smithi104 ceph-mon[36624]: pgmap v20405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:51 smithi081 ceph-mon[31453]: pgmap v20405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:51.922+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:54:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:51.922+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:54:51.923 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:54:51.922+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:54:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:53 smithi153 ceph-mon[36569]: pgmap v20406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:53 smithi104 ceph-mon[36624]: pgmap v20406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:53 smithi081 ceph-mon[31453]: pgmap v20406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:55 smithi153 ceph-mon[36569]: pgmap v20407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:55 smithi104 ceph-mon[36624]: pgmap v20407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:55 smithi081 ceph-mon[31453]: pgmap v20407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:57 smithi153 ceph-mon[36569]: pgmap v20408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:57 smithi104 ceph-mon[36624]: pgmap v20408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:57 smithi081 ceph-mon[31453]: pgmap v20408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:54:59 smithi153 ceph-mon[36569]: pgmap v20409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:54:59 smithi104 ceph-mon[36624]: pgmap v20409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:54:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:54:59 smithi081 ceph-mon[31453]: pgmap v20409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:01.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:01 smithi153 ceph-mon[36569]: pgmap v20410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:01 smithi104 ceph-mon[36624]: pgmap v20410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:01 smithi081 ceph-mon[31453]: pgmap v20410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:03.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:03 smithi153 ceph-mon[36569]: pgmap v20411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:03 smithi104 ceph-mon[36624]: pgmap v20411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:03 smithi081 ceph-mon[31453]: pgmap v20411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:05 smithi153 ceph-mon[36569]: pgmap v20412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:05 smithi104 ceph-mon[36624]: pgmap v20412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:05 smithi081 ceph-mon[31453]: pgmap v20412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:06.932+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:55:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:06.932+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:55:06.932 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:06.932+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:55:07.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:07 smithi153 ceph-mon[36569]: pgmap v20413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:07 smithi104 ceph-mon[36624]: pgmap v20413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:07 smithi081 ceph-mon[31453]: pgmap v20413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:09 smithi153 ceph-mon[36569]: pgmap v20414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:09 smithi104 ceph-mon[36624]: pgmap v20414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:09 smithi081 ceph-mon[31453]: pgmap v20414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:11.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:11 smithi153 ceph-mon[36569]: pgmap v20415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:11 smithi104 ceph-mon[36624]: pgmap v20415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:11 smithi081 ceph-mon[31453]: pgmap v20415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:13 smithi153 ceph-mon[36569]: pgmap v20416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:13 smithi104 ceph-mon[36624]: pgmap v20416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:13 smithi081 ceph-mon[31453]: pgmap v20416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:55:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:55:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:55:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:55:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:55:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:15 smithi153 ceph-mon[36569]: pgmap v20417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:15 smithi153 ceph-mon[36569]: pgmap v20418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:55:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:55:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:55:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:15 smithi104 ceph-mon[36624]: pgmap v20417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:15 smithi104 ceph-mon[36624]: pgmap v20418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:55:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:55:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:55:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:15 smithi081 ceph-mon[31453]: pgmap v20417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:15 smithi081 ceph-mon[31453]: pgmap v20418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:15.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:55:18.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:17 smithi153 ceph-mon[36569]: pgmap v20419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:17 smithi104 ceph-mon[36624]: pgmap v20419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:17 smithi081 ceph-mon[31453]: pgmap v20419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:19.053+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:55:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:19.053+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:19.053+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 295 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:19.054+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 295 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:55:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:20 smithi153 ceph-mon[36569]: pgmap v20420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:20 smithi104 ceph-mon[36624]: pgmap v20420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:20 smithi081 ceph-mon[31453]: pgmap v20420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:21.947+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:55:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:21.947+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:55:21.948 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:21.947+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:55:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:22 smithi153 ceph-mon[36569]: pgmap v20421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:22 smithi104 ceph-mon[36624]: pgmap v20421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:22 smithi081 ceph-mon[31453]: pgmap v20421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:24 smithi153 ceph-mon[36569]: pgmap v20422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:24 smithi104 ceph-mon[36624]: pgmap v20422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:24 smithi081 ceph-mon[31453]: pgmap v20422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:26 smithi153 ceph-mon[36569]: pgmap v20423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:26 smithi104 ceph-mon[36624]: pgmap v20423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:26 smithi081 ceph-mon[31453]: pgmap v20423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:28.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:28 smithi153 ceph-mon[36569]: pgmap v20424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:28 smithi104 ceph-mon[36624]: pgmap v20424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:28 smithi081 ceph-mon[31453]: pgmap v20424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:30.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:30 smithi153 ceph-mon[36569]: pgmap v20425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:30 smithi104 ceph-mon[36624]: pgmap v20425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:30 smithi081 ceph-mon[31453]: pgmap v20425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:32 smithi153 ceph-mon[36569]: pgmap v20426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:32 smithi104 ceph-mon[36624]: pgmap v20426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:32 smithi081 ceph-mon[31453]: pgmap v20426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:34 smithi153 ceph-mon[36569]: pgmap v20427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:34 smithi104 ceph-mon[36624]: pgmap v20427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:34 smithi081 ceph-mon[31453]: pgmap v20427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:36 smithi153 ceph-mon[36569]: pgmap v20428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:36 smithi104 ceph-mon[36624]: pgmap v20428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:36 smithi081 ceph-mon[31453]: pgmap v20428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:36.962 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:36.961+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:55:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:36.961+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:55:36.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:36.961+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:55:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:38 smithi153 ceph-mon[36569]: pgmap v20429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:38 smithi104 ceph-mon[36624]: pgmap v20429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:38 smithi081 ceph-mon[31453]: pgmap v20429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:40 smithi104 ceph-mon[36624]: pgmap v20430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:40 smithi081 ceph-mon[31453]: pgmap v20430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:40 smithi153 ceph-mon[36569]: pgmap v20430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:42 smithi104 ceph-mon[36624]: pgmap v20431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:42 smithi081 ceph-mon[31453]: pgmap v20431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:42 smithi153 ceph-mon[36569]: pgmap v20431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:44 smithi104 ceph-mon[36624]: pgmap v20432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:44 smithi081 ceph-mon[31453]: pgmap v20432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:44 smithi153 ceph-mon[36569]: pgmap v20432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:46 smithi104 ceph-mon[36624]: pgmap v20433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:46 smithi081 ceph-mon[31453]: pgmap v20433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:46 smithi153 ceph-mon[36569]: pgmap v20433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:48 smithi104 ceph-mon[36624]: pgmap v20434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:48 smithi081 ceph-mon[31453]: pgmap v20434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:48 smithi153 ceph-mon[36569]: pgmap v20434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:50 smithi081 ceph-mon[31453]: pgmap v20435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:50 smithi153 ceph-mon[36569]: pgmap v20435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:50 smithi104 ceph-mon[36624]: pgmap v20435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:51.975+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:55:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:51.975+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:55:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:55:51.975+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:55:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:52 smithi081 ceph-mon[31453]: pgmap v20436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:52 smithi153 ceph-mon[36569]: pgmap v20436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:52 smithi104 ceph-mon[36624]: pgmap v20436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:54 smithi081 ceph-mon[31453]: pgmap v20437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:54 smithi153 ceph-mon[36569]: pgmap v20437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:54 smithi104 ceph-mon[36624]: pgmap v20437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:56 smithi081 ceph-mon[31453]: pgmap v20438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:56 smithi153 ceph-mon[36569]: pgmap v20438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:56 smithi104 ceph-mon[36624]: pgmap v20438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:55:58 smithi081 ceph-mon[31453]: pgmap v20439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:55:58 smithi153 ceph-mon[36569]: pgmap v20439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:55:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:55:58 smithi104 ceph-mon[36624]: pgmap v20439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:00 smithi081 ceph-mon[31453]: pgmap v20440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:00 smithi153 ceph-mon[36569]: pgmap v20440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:00 smithi104 ceph-mon[36624]: pgmap v20440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:02 smithi081 ceph-mon[31453]: pgmap v20441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:02 smithi153 ceph-mon[36569]: pgmap v20441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:02 smithi104 ceph-mon[36624]: pgmap v20441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:04 smithi081 ceph-mon[31453]: pgmap v20442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:04 smithi153 ceph-mon[36569]: pgmap v20442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:04 smithi104 ceph-mon[36624]: pgmap v20442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:06 smithi081 ceph-mon[31453]: pgmap v20443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:06 smithi153 ceph-mon[36569]: pgmap v20443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:06 smithi104 ceph-mon[36624]: pgmap v20443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:06.982+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:56:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:06.982+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:56:06.982 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:06.982+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:56:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:08 smithi081 ceph-mon[31453]: pgmap v20444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:08 smithi153 ceph-mon[36569]: pgmap v20444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:08 smithi104 ceph-mon[36624]: pgmap v20444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:10 smithi081 ceph-mon[31453]: pgmap v20445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:10 smithi153 ceph-mon[36569]: pgmap v20445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:10 smithi104 ceph-mon[36624]: pgmap v20445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:12 smithi081 ceph-mon[31453]: pgmap v20446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:12 smithi153 ceph-mon[36569]: pgmap v20446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:12 smithi104 ceph-mon[36624]: pgmap v20446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:14 smithi081 ceph-mon[31453]: pgmap v20447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:14 smithi153 ceph-mon[36569]: pgmap v20447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:14 smithi104 ceph-mon[36624]: pgmap v20447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:16 smithi081 ceph-mon[31453]: pgmap v20448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:56:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:16 smithi153 ceph-mon[36569]: pgmap v20448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:16.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:56:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:16 smithi104 ceph-mon[36624]: pgmap v20448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:56:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:56:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:56:17.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:56:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:56:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:56:17.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:56:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:56:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:56:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:56:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:18 smithi153 ceph-mon[36569]: pgmap v20449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:18.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:18 smithi153 ceph-mon[36569]: pgmap v20450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:18 smithi104 ceph-mon[36624]: pgmap v20449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:18 smithi104 ceph-mon[36624]: pgmap v20450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:18 smithi081 ceph-mon[31453]: pgmap v20449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:18 smithi081 ceph-mon[31453]: pgmap v20450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:20 smithi081 ceph-mon[31453]: pgmap v20451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:20 smithi153 ceph-mon[36569]: pgmap v20451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:20 smithi104 ceph-mon[36624]: pgmap v20451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:21.993+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:56:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:21.993+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:56:21.993 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:21.993+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:56:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:22 smithi153 ceph-mon[36569]: pgmap v20452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:22 smithi104 ceph-mon[36624]: pgmap v20452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:22 smithi081 ceph-mon[31453]: pgmap v20452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:24 smithi153 ceph-mon[36569]: pgmap v20453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:24 smithi104 ceph-mon[36624]: pgmap v20453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:24 smithi081 ceph-mon[31453]: pgmap v20453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:26 smithi153 ceph-mon[36569]: pgmap v20454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:26 smithi104 ceph-mon[36624]: pgmap v20454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:26 smithi081 ceph-mon[31453]: pgmap v20454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:28 smithi153 ceph-mon[36569]: pgmap v20455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:28 smithi104 ceph-mon[36624]: pgmap v20455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:28 smithi081 ceph-mon[31453]: pgmap v20455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:30 smithi153 ceph-mon[36569]: pgmap v20456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:30 smithi104 ceph-mon[36624]: pgmap v20456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:30 smithi081 ceph-mon[31453]: pgmap v20456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:32 smithi153 ceph-mon[36569]: pgmap v20457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:32 smithi104 ceph-mon[36624]: pgmap v20457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:32 smithi081 ceph-mon[31453]: pgmap v20457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:34 smithi153 ceph-mon[36569]: pgmap v20458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:34 smithi104 ceph-mon[36624]: pgmap v20458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:34 smithi081 ceph-mon[31453]: pgmap v20458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:36 smithi153 ceph-mon[36569]: pgmap v20459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:36 smithi104 ceph-mon[36624]: pgmap v20459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:36 smithi081 ceph-mon[31453]: pgmap v20459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:37.008 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:37.008+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:56:37.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:37.008+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:56:37.009 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:37.008+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:56:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:38 smithi153 ceph-mon[36569]: pgmap v20460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:38 smithi104 ceph-mon[36624]: pgmap v20460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:38 smithi081 ceph-mon[31453]: pgmap v20460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:40 smithi153 ceph-mon[36569]: pgmap v20461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:40 smithi104 ceph-mon[36624]: pgmap v20461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:40 smithi081 ceph-mon[31453]: pgmap v20461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:42 smithi153 ceph-mon[36569]: pgmap v20462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:42 smithi104 ceph-mon[36624]: pgmap v20462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:42 smithi081 ceph-mon[31453]: pgmap v20462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:44 smithi153 ceph-mon[36569]: pgmap v20463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:44 smithi104 ceph-mon[36624]: pgmap v20463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:44 smithi081 ceph-mon[31453]: pgmap v20463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:46.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:46 smithi153 ceph-mon[36569]: pgmap v20464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:46 smithi104 ceph-mon[36624]: pgmap v20464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:46 smithi081 ceph-mon[31453]: pgmap v20464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:48 smithi153 ceph-mon[36569]: pgmap v20465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:48 smithi104 ceph-mon[36624]: pgmap v20465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:48.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:48 smithi081 ceph-mon[31453]: pgmap v20465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:50.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:50 smithi153 ceph-mon[36569]: pgmap v20466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:50 smithi104 ceph-mon[36624]: pgmap v20466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:50 smithi081 ceph-mon[31453]: pgmap v20466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:52.017 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:52.017+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:56:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:52.017+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:56:52.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:56:52.017+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:56:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:52 smithi153 ceph-mon[36569]: pgmap v20467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:52 smithi104 ceph-mon[36624]: pgmap v20467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:52 smithi081 ceph-mon[31453]: pgmap v20467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:54 smithi153 ceph-mon[36569]: pgmap v20468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:54 smithi104 ceph-mon[36624]: pgmap v20468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:54 smithi081 ceph-mon[31453]: pgmap v20468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:56 smithi153 ceph-mon[36569]: pgmap v20469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:56 smithi104 ceph-mon[36624]: pgmap v20469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:56 smithi081 ceph-mon[31453]: pgmap v20469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:56:58 smithi153 ceph-mon[36569]: pgmap v20470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:56:58 smithi104 ceph-mon[36624]: pgmap v20470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:56:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:56:58 smithi081 ceph-mon[31453]: pgmap v20470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:00 smithi153 ceph-mon[36569]: pgmap v20471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:00 smithi104 ceph-mon[36624]: pgmap v20471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:00 smithi081 ceph-mon[31453]: pgmap v20471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:02 smithi153 ceph-mon[36569]: pgmap v20472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:02 smithi104 ceph-mon[36624]: pgmap v20472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:02 smithi081 ceph-mon[31453]: pgmap v20472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:04 smithi153 ceph-mon[36569]: pgmap v20473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:04 smithi104 ceph-mon[36624]: pgmap v20473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:04 smithi081 ceph-mon[31453]: pgmap v20473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:06 smithi153 ceph-mon[36569]: pgmap v20474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:06 smithi104 ceph-mon[36624]: pgmap v20474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:06 smithi081 ceph-mon[31453]: pgmap v20474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:07.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:07.019+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:57:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:07.019+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:57:07.020 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:07.019+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:57:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:08 smithi153 ceph-mon[36569]: pgmap v20475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:08 smithi104 ceph-mon[36624]: pgmap v20475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:08 smithi081 ceph-mon[31453]: pgmap v20475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:10 smithi153 ceph-mon[36569]: pgmap v20476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:10 smithi104 ceph-mon[36624]: pgmap v20476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:10 smithi081 ceph-mon[31453]: pgmap v20476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:12 smithi153 ceph-mon[36569]: pgmap v20477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:12 smithi104 ceph-mon[36624]: pgmap v20477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:12 smithi081 ceph-mon[31453]: pgmap v20477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:14 smithi104 ceph-mon[36624]: pgmap v20478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:14 smithi081 ceph-mon[31453]: pgmap v20478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:14.809 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:14 smithi153 ceph-mon[36569]: pgmap v20478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:16 smithi104 ceph-mon[36624]: pgmap v20479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:16.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:16 smithi081 ceph-mon[31453]: pgmap v20479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:16 smithi153 ceph-mon[36569]: pgmap v20479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:57:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:57:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: pgmap v20480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:57:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:57:18.658 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:18 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: pgmap v20480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:57:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:57:18.808 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:18 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:57:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: pgmap v20480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:57:18.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:18 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:57:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:19 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:57:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:19 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:57:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:19 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:57:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:19 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:57:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:19 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:57:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:19 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:57:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:19 smithi081 ceph-mon[31453]: pgmap v20481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:19 smithi081 ceph-mon[31453]: pgmap v20482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:19 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:57:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:19 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:57:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:19 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:57:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:19 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:57:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:19 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:57:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:19 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:57:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:19 smithi153 ceph-mon[36569]: pgmap v20481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:19 smithi153 ceph-mon[36569]: pgmap v20482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:19 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T07:57:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:19 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T07:57:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:19 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T07:57:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:19 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T07:57:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:19 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T07:57:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:19 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T07:57:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:19 smithi104 ceph-mon[36624]: pgmap v20481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:19.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:19 smithi104 ceph-mon[36624]: pgmap v20482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:21.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:21 smithi104 ceph-mon[36624]: pgmap v20483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:21 smithi081 ceph-mon[31453]: pgmap v20483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:21 smithi153 ceph-mon[36569]: pgmap v20483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:22.022 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:22.021+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:57:22.022 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:22.021+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:57:22.022 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:22.021+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:57:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:23 smithi081 ceph-mon[31453]: pgmap v20484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:23 smithi153 ceph-mon[36569]: pgmap v20484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:23 smithi104 ceph-mon[36624]: pgmap v20484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:25 smithi081 ceph-mon[31453]: pgmap v20485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:25 smithi153 ceph-mon[36569]: pgmap v20485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:25 smithi104 ceph-mon[36624]: pgmap v20485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:27 smithi081 ceph-mon[31453]: pgmap v20486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:27.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:27 smithi153 ceph-mon[36569]: pgmap v20486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:27 smithi104 ceph-mon[36624]: pgmap v20486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:29 smithi081 ceph-mon[31453]: pgmap v20487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:29 smithi153 ceph-mon[36569]: pgmap v20487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:29 smithi104 ceph-mon[36624]: pgmap v20487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:31 smithi081 ceph-mon[31453]: pgmap v20488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:31.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:31 smithi153 ceph-mon[36569]: pgmap v20488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:31 smithi104 ceph-mon[36624]: pgmap v20488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:33 smithi081 ceph-mon[31453]: pgmap v20489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:33 smithi153 ceph-mon[36569]: pgmap v20489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:33 smithi104 ceph-mon[36624]: pgmap v20489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:35 smithi081 ceph-mon[31453]: pgmap v20490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:35 smithi153 ceph-mon[36569]: pgmap v20490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:35 smithi104 ceph-mon[36624]: pgmap v20490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:37.027+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:57:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:37.027+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:57:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:37.027+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:57:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:37 smithi081 ceph-mon[31453]: pgmap v20491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:37 smithi153 ceph-mon[36569]: pgmap v20491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:37 smithi104 ceph-mon[36624]: pgmap v20491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:39.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:39 smithi153 ceph-mon[36569]: pgmap v20492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:39 smithi104 ceph-mon[36624]: pgmap v20492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:39 smithi081 ceph-mon[31453]: pgmap v20492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:41.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:41 smithi153 ceph-mon[36569]: pgmap v20493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:41 smithi104 ceph-mon[36624]: pgmap v20493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:41 smithi081 ceph-mon[31453]: pgmap v20493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:43.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:43 smithi081 ceph-mon[31453]: pgmap v20494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:43 smithi153 ceph-mon[36569]: pgmap v20494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:43 smithi104 ceph-mon[36624]: pgmap v20494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:45 smithi153 ceph-mon[36569]: pgmap v20495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:45 smithi104 ceph-mon[36624]: pgmap v20495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:45 smithi081 ceph-mon[31453]: pgmap v20495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:47 smithi153 ceph-mon[36569]: pgmap v20496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:47 smithi104 ceph-mon[36624]: pgmap v20496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:47 smithi081 ceph-mon[31453]: pgmap v20496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:49.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:49 smithi153 ceph-mon[36569]: pgmap v20497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:49 smithi104 ceph-mon[36624]: pgmap v20497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:49 smithi081 ceph-mon[31453]: pgmap v20497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:51.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:51 smithi153 ceph-mon[36569]: pgmap v20498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:51 smithi104 ceph-mon[36624]: pgmap v20498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:51 smithi081 ceph-mon[31453]: pgmap v20498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:52.037+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:57:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:52.037+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:57:52.038 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:57:52.037+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:57:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:52 smithi081 ceph-mon[31453]: pgmap v20499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:52 smithi153 ceph-mon[36569]: pgmap v20499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:52 smithi104 ceph-mon[36624]: pgmap v20499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:55.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:55 smithi153 ceph-mon[36569]: pgmap v20500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:55.649 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:55 smithi104 ceph-mon[36624]: pgmap v20500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:55 smithi081 ceph-mon[31453]: pgmap v20500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:57 smithi153 ceph-mon[36569]: pgmap v20501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:57 smithi104 ceph-mon[36624]: pgmap v20501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:57 smithi081 ceph-mon[31453]: pgmap v20501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:57:59 smithi153 ceph-mon[36569]: pgmap v20502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:57:59 smithi104 ceph-mon[36624]: pgmap v20502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:57:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:57:59 smithi081 ceph-mon[31453]: pgmap v20502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:01.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:01 smithi153 ceph-mon[36569]: pgmap v20503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:01 smithi104 ceph-mon[36624]: pgmap v20503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:01 smithi081 ceph-mon[31453]: pgmap v20503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:03 smithi153 ceph-mon[36569]: pgmap v20504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:03 smithi104 ceph-mon[36624]: pgmap v20504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:03 smithi081 ceph-mon[31453]: pgmap v20504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:05.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:05 smithi153 ceph-mon[36569]: pgmap v20505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:05 smithi104 ceph-mon[36624]: pgmap v20505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:05 smithi081 ceph-mon[31453]: pgmap v20505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:07.052+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:58:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:07.052+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:58:07.053 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:07.052+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:58:07.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:07 smithi153 ceph-mon[36569]: pgmap v20506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:07 smithi104 ceph-mon[36624]: pgmap v20506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:07 smithi081 ceph-mon[31453]: pgmap v20506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:09.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:09 smithi153 ceph-mon[36569]: pgmap v20507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:09 smithi104 ceph-mon[36624]: pgmap v20507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:09 smithi081 ceph-mon[31453]: pgmap v20507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:11 smithi153 ceph-mon[36569]: pgmap v20508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:11 smithi104 ceph-mon[36624]: pgmap v20508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:11 smithi081 ceph-mon[31453]: pgmap v20508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:13.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:13 smithi081 ceph-mon[31453]: pgmap v20509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:13 smithi153 ceph-mon[36569]: pgmap v20509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:13 smithi104 ceph-mon[36624]: pgmap v20509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:15.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:15 smithi153 ceph-mon[36569]: pgmap v20510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:15 smithi104 ceph-mon[36624]: pgmap v20510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:15 smithi081 ceph-mon[31453]: pgmap v20510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:17 smithi153 ceph-mon[36569]: pgmap v20511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:17.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:17 smithi104 ceph-mon[36624]: pgmap v20511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:17 smithi081 ceph-mon[31453]: pgmap v20511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:19.082+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:58:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:19.082+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:58:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:19.082+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 296 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:58:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:19.082+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 296 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:58:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:19 smithi153 ceph-mon[36569]: pgmap v20512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:58:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:19 smithi104 ceph-mon[36624]: pgmap v20512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:58:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:19 smithi081 ceph-mon[31453]: pgmap v20512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:58:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:58:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:58:20.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:58:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:58:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:58:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:58:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:58:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:58:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:58:21.483 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:21 smithi104 ceph-mon[36624]: pgmap v20513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:21.484 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:21 smithi104 ceph-mon[36624]: pgmap v20514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:21 smithi153 ceph-mon[36569]: pgmap v20513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:21.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:21 smithi153 ceph-mon[36569]: pgmap v20514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:21 smithi081 ceph-mon[31453]: pgmap v20513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:21 smithi081 ceph-mon[31453]: pgmap v20514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:22.058+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:58:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:22.058+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:58:22.059 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:22.058+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:58:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:23 smithi153 ceph-mon[36569]: pgmap v20515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:23 smithi104 ceph-mon[36624]: pgmap v20515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:23 smithi081 ceph-mon[31453]: pgmap v20515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:25 smithi153 ceph-mon[36569]: pgmap v20516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:25 smithi104 ceph-mon[36624]: pgmap v20516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:25 smithi081 ceph-mon[31453]: pgmap v20516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:27 smithi153 ceph-mon[36569]: pgmap v20517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:27 smithi104 ceph-mon[36624]: pgmap v20517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:27 smithi081 ceph-mon[31453]: pgmap v20517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:29 smithi153 ceph-mon[36569]: pgmap v20518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:29 smithi104 ceph-mon[36624]: pgmap v20518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:29 smithi081 ceph-mon[31453]: pgmap v20518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:31 smithi153 ceph-mon[36569]: pgmap v20519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:31 smithi104 ceph-mon[36624]: pgmap v20519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:31 smithi081 ceph-mon[31453]: pgmap v20519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:33.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:33 smithi153 ceph-mon[36569]: pgmap v20520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:33.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:33 smithi104 ceph-mon[36624]: pgmap v20520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:33 smithi081 ceph-mon[31453]: pgmap v20520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:35 smithi153 ceph-mon[36569]: pgmap v20521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:35 smithi104 ceph-mon[36624]: pgmap v20521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:35 smithi081 ceph-mon[31453]: pgmap v20521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:37.070+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:58:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:37.070+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:58:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:37.070+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:58:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:37 smithi153 ceph-mon[36569]: pgmap v20522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:37 smithi104 ceph-mon[36624]: pgmap v20522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:37 smithi081 ceph-mon[31453]: pgmap v20522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:39 smithi153 ceph-mon[36569]: pgmap v20523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:39 smithi104 ceph-mon[36624]: pgmap v20523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:39 smithi081 ceph-mon[31453]: pgmap v20523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:41 smithi153 ceph-mon[36569]: pgmap v20524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:41 smithi104 ceph-mon[36624]: pgmap v20524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:41 smithi081 ceph-mon[31453]: pgmap v20524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:43 smithi153 ceph-mon[36569]: pgmap v20525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:43 smithi104 ceph-mon[36624]: pgmap v20525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:43 smithi081 ceph-mon[31453]: pgmap v20525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:45 smithi153 ceph-mon[36569]: pgmap v20526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:45 smithi104 ceph-mon[36624]: pgmap v20526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:45 smithi081 ceph-mon[31453]: pgmap v20526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:47 smithi153 ceph-mon[36569]: pgmap v20527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:47 smithi104 ceph-mon[36624]: pgmap v20527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:47 smithi081 ceph-mon[31453]: pgmap v20527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:49 smithi153 ceph-mon[36569]: pgmap v20528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:49 smithi104 ceph-mon[36624]: pgmap v20528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:49.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:49 smithi081 ceph-mon[31453]: pgmap v20528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:51 smithi104 ceph-mon[36624]: pgmap v20529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:51 smithi081 ceph-mon[31453]: pgmap v20529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:51 smithi153 ceph-mon[36569]: pgmap v20529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:52.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:52.085+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:58:52.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:52.085+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:58:52.085 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:58:52.085+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:58:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:53 smithi104 ceph-mon[36624]: pgmap v20530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:53 smithi081 ceph-mon[31453]: pgmap v20530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:53.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:53 smithi153 ceph-mon[36569]: pgmap v20530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:55 smithi104 ceph-mon[36624]: pgmap v20531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:55 smithi081 ceph-mon[31453]: pgmap v20531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:55 smithi153 ceph-mon[36569]: pgmap v20531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:57 smithi104 ceph-mon[36624]: pgmap v20532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:57 smithi081 ceph-mon[31453]: pgmap v20532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:57 smithi153 ceph-mon[36569]: pgmap v20532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:58:59 smithi104 ceph-mon[36624]: pgmap v20533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:58:59 smithi081 ceph-mon[31453]: pgmap v20533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:58:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:58:59 smithi153 ceph-mon[36569]: pgmap v20533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:01 smithi081 ceph-mon[31453]: pgmap v20534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:01 smithi153 ceph-mon[36569]: pgmap v20534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:01 smithi104 ceph-mon[36624]: pgmap v20534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:03 smithi081 ceph-mon[31453]: pgmap v20535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:03 smithi153 ceph-mon[36569]: pgmap v20535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:03 smithi104 ceph-mon[36624]: pgmap v20535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:05 smithi081 ceph-mon[31453]: pgmap v20536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:05.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:05 smithi153 ceph-mon[36569]: pgmap v20536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:05 smithi104 ceph-mon[36624]: pgmap v20536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:07.094+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:59:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:07.094+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:59:07.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:07.094+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:59:07.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:07 smithi153 ceph-mon[36569]: pgmap v20537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:07 smithi081 ceph-mon[31453]: pgmap v20537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:07 smithi104 ceph-mon[36624]: pgmap v20537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:09 smithi081 ceph-mon[31453]: pgmap v20538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:09.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:09 smithi153 ceph-mon[36569]: pgmap v20538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:09 smithi104 ceph-mon[36624]: pgmap v20538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:11 smithi081 ceph-mon[31453]: pgmap v20539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:11 smithi153 ceph-mon[36569]: pgmap v20539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:11 smithi104 ceph-mon[36624]: pgmap v20539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:13 smithi081 ceph-mon[31453]: pgmap v20540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:13 smithi153 ceph-mon[36569]: pgmap v20540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:13 smithi104 ceph-mon[36624]: pgmap v20540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:15 smithi081 ceph-mon[31453]: pgmap v20541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:15 smithi153 ceph-mon[36569]: pgmap v20541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:15 smithi104 ceph-mon[36624]: pgmap v20541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:17 smithi081 ceph-mon[31453]: pgmap v20542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:17.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:17 smithi153 ceph-mon[36569]: pgmap v20542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:17 smithi104 ceph-mon[36624]: pgmap v20542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:19.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:19 smithi081 ceph-mon[31453]: pgmap v20543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:19 smithi153 ceph-mon[36569]: pgmap v20543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:19 smithi104 ceph-mon[36624]: pgmap v20543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:20 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:59:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:20 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:59:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:20 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T07:59:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:21 smithi081 ceph-mon[31453]: pgmap v20544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:59:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:59:21.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:21 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:59:21.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:21 smithi104 ceph-mon[36624]: pgmap v20544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:21.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:59:21.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:59:21.734 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:21 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:59:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:21 smithi153 ceph-mon[36569]: pgmap v20544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T07:59:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T07:59:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:21 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T07:59:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:22.098+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:59:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:22.098+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:59:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:22.098+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:59:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:22 smithi153 ceph-mon[36569]: pgmap v20545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:22 smithi153 ceph-mon[36569]: pgmap v20546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:22 smithi104 ceph-mon[36624]: pgmap v20545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:22 smithi104 ceph-mon[36624]: pgmap v20546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:22 smithi081 ceph-mon[31453]: pgmap v20545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:22 smithi081 ceph-mon[31453]: pgmap v20546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:24 smithi153 ceph-mon[36569]: pgmap v20547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:24 smithi104 ceph-mon[36624]: pgmap v20547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:24 smithi081 ceph-mon[31453]: pgmap v20547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:26 smithi153 ceph-mon[36569]: pgmap v20548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:26 smithi104 ceph-mon[36624]: pgmap v20548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:26.856 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:26 smithi081 ceph-mon[31453]: pgmap v20548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:28 smithi153 ceph-mon[36569]: pgmap v20549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:28 smithi104 ceph-mon[36624]: pgmap v20549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:28 smithi081 ceph-mon[31453]: pgmap v20549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:30 smithi153 ceph-mon[36569]: pgmap v20550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:30 smithi104 ceph-mon[36624]: pgmap v20550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:30 smithi081 ceph-mon[31453]: pgmap v20550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:32 smithi153 ceph-mon[36569]: pgmap v20551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:32 smithi104 ceph-mon[36624]: pgmap v20551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:32 smithi081 ceph-mon[31453]: pgmap v20551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:34 smithi153 ceph-mon[36569]: pgmap v20552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:34 smithi104 ceph-mon[36624]: pgmap v20552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:34 smithi081 ceph-mon[31453]: pgmap v20552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:36 smithi153 ceph-mon[36569]: pgmap v20553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:36 smithi104 ceph-mon[36624]: pgmap v20553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:36 smithi081 ceph-mon[31453]: pgmap v20553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:37.100+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:59:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:37.100+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:59:37.100 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:37.100+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:59:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:38 smithi153 ceph-mon[36569]: pgmap v20554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:38 smithi104 ceph-mon[36624]: pgmap v20554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:38 smithi081 ceph-mon[31453]: pgmap v20554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:40.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:40 smithi153 ceph-mon[36569]: pgmap v20555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:40 smithi104 ceph-mon[36624]: pgmap v20555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:40 smithi081 ceph-mon[31453]: pgmap v20555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:42 smithi081 ceph-mon[31453]: pgmap v20556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:42 smithi153 ceph-mon[36569]: pgmap v20556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:42 smithi104 ceph-mon[36624]: pgmap v20556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:44 smithi153 ceph-mon[36569]: pgmap v20557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:44 smithi104 ceph-mon[36624]: pgmap v20557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:44 smithi081 ceph-mon[31453]: pgmap v20557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:46.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:46 smithi153 ceph-mon[36569]: pgmap v20558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:46 smithi104 ceph-mon[36624]: pgmap v20558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:46 smithi081 ceph-mon[31453]: pgmap v20558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:48.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:48 smithi153 ceph-mon[36569]: pgmap v20559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:48 smithi104 ceph-mon[36624]: pgmap v20559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:48 smithi081 ceph-mon[31453]: pgmap v20559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:49.094 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:49.095+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:59:49.095 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:49.095+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 297 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T07:59:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:49.096+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:59:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:49.097+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 297 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T07:59:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:50 smithi153 ceph-mon[36569]: pgmap v20560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:50 smithi104 ceph-mon[36624]: pgmap v20560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:50 smithi081 ceph-mon[31453]: pgmap v20560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:52.113+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T07:59:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:52.113+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T07:59:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T07:59:52.113+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T07:59:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:52 smithi153 ceph-mon[36569]: pgmap v20561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:52 smithi104 ceph-mon[36624]: pgmap v20561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:52 smithi081 ceph-mon[31453]: pgmap v20561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:54.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:54 smithi153 ceph-mon[36569]: pgmap v20562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:54 smithi104 ceph-mon[36624]: pgmap v20562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:54 smithi081 ceph-mon[31453]: pgmap v20562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:56 smithi153 ceph-mon[36569]: pgmap v20563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:56 smithi104 ceph-mon[36624]: pgmap v20563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:56 smithi081 ceph-mon[31453]: pgmap v20563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 07:59:58 smithi153 ceph-mon[36569]: pgmap v20564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 07:59:58 smithi104 ceph-mon[36624]: pgmap v20564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T07:59:58.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 07:59:58 smithi081 ceph-mon[31453]: pgmap v20564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:00 smithi153 ceph-mon[36569]: pgmap v20565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:00:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:00 smithi104 ceph-mon[36624]: pgmap v20565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:00:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:00 smithi081 ceph-mon[31453]: pgmap v20565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:00:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:02 smithi153 ceph-mon[36569]: pgmap v20566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:02 smithi104 ceph-mon[36624]: pgmap v20566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:02 smithi081 ceph-mon[31453]: pgmap v20566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:04 smithi153 ceph-mon[36569]: pgmap v20567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:04 smithi104 ceph-mon[36624]: pgmap v20567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:04 smithi081 ceph-mon[31453]: pgmap v20567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:06 smithi153 ceph-mon[36569]: pgmap v20568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:06 smithi104 ceph-mon[36624]: pgmap v20568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:06 smithi081 ceph-mon[31453]: pgmap v20568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:07.129+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:00:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:07.129+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:00:07.129 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:07.129+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:00:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:08 smithi153 ceph-mon[36569]: pgmap v20569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:08 smithi104 ceph-mon[36624]: pgmap v20569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:08 smithi081 ceph-mon[31453]: pgmap v20569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:10.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:10 smithi153 ceph-mon[36569]: pgmap v20570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:10 smithi104 ceph-mon[36624]: pgmap v20570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:10 smithi081 ceph-mon[31453]: pgmap v20570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:12.805 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:12 smithi081 ceph-mon[31453]: pgmap v20571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:12 smithi153 ceph-mon[36569]: pgmap v20571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:12 smithi104 ceph-mon[36624]: pgmap v20571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:14.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:14 smithi153 ceph-mon[36569]: pgmap v20572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:14 smithi104 ceph-mon[36624]: pgmap v20572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:14 smithi081 ceph-mon[31453]: pgmap v20572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:15 smithi104 ceph-mon[36624]: pgmap v20573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:15.869 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:15 smithi081 ceph-mon[31453]: pgmap v20573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:15 smithi153 ceph-mon[36569]: pgmap v20573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:18 smithi081 ceph-mon[31453]: pgmap v20574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:18 smithi153 ceph-mon[36569]: pgmap v20574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:18 smithi104 ceph-mon[36624]: pgmap v20574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:20 smithi153 ceph-mon[36569]: pgmap v20575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:20 smithi104 ceph-mon[36624]: pgmap v20575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:20 smithi081 ceph-mon[31453]: pgmap v20575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:22.144 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:22.144+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:00:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:22.144+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:00:22.145 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:22.144+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:00:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:22 smithi153 ceph-mon[36569]: pgmap v20576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:00:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:22 smithi104 ceph-mon[36624]: pgmap v20576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:00:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:22 smithi081 ceph-mon[31453]: pgmap v20576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:00:23.486 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:00:23.487 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:00:23.487 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:00:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:00:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:00:23.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:00:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:00:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:00:23.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:00:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:24 smithi153 ceph-mon[36569]: pgmap v20577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:24.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:24 smithi153 ceph-mon[36569]: pgmap v20578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:24 smithi104 ceph-mon[36624]: pgmap v20577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:24.583 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:24 smithi104 ceph-mon[36624]: pgmap v20578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:24 smithi081 ceph-mon[31453]: pgmap v20577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:24 smithi081 ceph-mon[31453]: pgmap v20578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:26 smithi153 ceph-mon[36569]: pgmap v20579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:26 smithi104 ceph-mon[36624]: pgmap v20579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:26 smithi081 ceph-mon[31453]: pgmap v20579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:28 smithi153 ceph-mon[36569]: pgmap v20580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:28 smithi104 ceph-mon[36624]: pgmap v20580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:28 smithi081 ceph-mon[31453]: pgmap v20580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:30 smithi153 ceph-mon[36569]: pgmap v20581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:30 smithi104 ceph-mon[36624]: pgmap v20581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:30 smithi081 ceph-mon[31453]: pgmap v20581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:32 smithi153 ceph-mon[36569]: pgmap v20582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:32 smithi104 ceph-mon[36624]: pgmap v20582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:32.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:32 smithi081 ceph-mon[31453]: pgmap v20582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:34 smithi153 ceph-mon[36569]: pgmap v20583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:34 smithi104 ceph-mon[36624]: pgmap v20583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:34 smithi081 ceph-mon[31453]: pgmap v20583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:36 smithi153 ceph-mon[36569]: pgmap v20584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:36 smithi104 ceph-mon[36624]: pgmap v20584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:36 smithi081 ceph-mon[31453]: pgmap v20584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:37.154+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:00:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:37.154+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:00:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:37.154+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:00:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:38 smithi153 ceph-mon[36569]: pgmap v20585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:38 smithi104 ceph-mon[36624]: pgmap v20585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:38 smithi081 ceph-mon[31453]: pgmap v20585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:40.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:40 smithi153 ceph-mon[36569]: pgmap v20586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:40 smithi104 ceph-mon[36624]: pgmap v20586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:40 smithi081 ceph-mon[31453]: pgmap v20586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:42 smithi153 ceph-mon[36569]: pgmap v20587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:42 smithi104 ceph-mon[36624]: pgmap v20587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:42 smithi081 ceph-mon[31453]: pgmap v20587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:44 smithi153 ceph-mon[36569]: pgmap v20588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:44 smithi104 ceph-mon[36624]: pgmap v20588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:44 smithi081 ceph-mon[31453]: pgmap v20588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:46 smithi153 ceph-mon[36569]: pgmap v20589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:46 smithi104 ceph-mon[36624]: pgmap v20589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:46 smithi081 ceph-mon[31453]: pgmap v20589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:48 smithi153 ceph-mon[36569]: pgmap v20590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:48 smithi104 ceph-mon[36624]: pgmap v20590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:48 smithi081 ceph-mon[31453]: pgmap v20590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:50 smithi153 ceph-mon[36569]: pgmap v20591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:50 smithi104 ceph-mon[36624]: pgmap v20591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:50 smithi081 ceph-mon[31453]: pgmap v20591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:52.168+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:00:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:52.168+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:00:52.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:00:52.168+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:00:52.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:52 smithi153 ceph-mon[36569]: pgmap v20592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:52 smithi104 ceph-mon[36624]: pgmap v20592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:52 smithi081 ceph-mon[31453]: pgmap v20592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:54 smithi153 ceph-mon[36569]: pgmap v20593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:54 smithi104 ceph-mon[36624]: pgmap v20593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:54 smithi081 ceph-mon[31453]: pgmap v20593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:56 smithi153 ceph-mon[36569]: pgmap v20594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:56 smithi104 ceph-mon[36624]: pgmap v20594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:56 smithi081 ceph-mon[31453]: pgmap v20594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:00:58 smithi153 ceph-mon[36569]: pgmap v20595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:00:58 smithi104 ceph-mon[36624]: pgmap v20595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:00:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:00:58 smithi081 ceph-mon[31453]: pgmap v20595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:00 smithi153 ceph-mon[36569]: pgmap v20596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:00 smithi104 ceph-mon[36624]: pgmap v20596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:00 smithi081 ceph-mon[31453]: pgmap v20596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:02.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:02 smithi153 ceph-mon[36569]: pgmap v20597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:02 smithi104 ceph-mon[36624]: pgmap v20597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:02 smithi081 ceph-mon[31453]: pgmap v20597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:04.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:04 smithi153 ceph-mon[36569]: pgmap v20598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:04 smithi104 ceph-mon[36624]: pgmap v20598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:04 smithi081 ceph-mon[31453]: pgmap v20598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:06.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:06 smithi153 ceph-mon[36569]: pgmap v20599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:06 smithi104 ceph-mon[36624]: pgmap v20599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:06 smithi081 ceph-mon[31453]: pgmap v20599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:07.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:07.170+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:01:07.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:07.170+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:01:07.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:07.170+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:01:08.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:08 smithi153 ceph-mon[36569]: pgmap v20600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:08 smithi104 ceph-mon[36624]: pgmap v20600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:08 smithi081 ceph-mon[31453]: pgmap v20600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:10 smithi153 ceph-mon[36569]: pgmap v20601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:10 smithi104 ceph-mon[36624]: pgmap v20601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:10 smithi081 ceph-mon[31453]: pgmap v20601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:12.197 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:12.197+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 msgr2=0x7f16e80a5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000584) for more than 900000000 us, fault. 2024-04-13T08:01:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:12.197+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f16f4010fd0 tx=0x7f16f40bf000 comp rx=0 tx=0).stop 2024-04-13T08:01:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:12.197+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 msgr2=0x7f16e80a5010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T08:01:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:12.197+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T08:01:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:12.197+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T08:01:12.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:12.198+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T08:01:12.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:12.199+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f16f4098ec0 tx=0x7f16f40ba000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T08:01:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:12 smithi153 ceph-mon[36569]: pgmap v20602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:12 smithi104 ceph-mon[36624]: pgmap v20602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:12 smithi081 ceph-mon[31453]: pgmap v20602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:14 smithi104 ceph-mon[36624]: pgmap v20603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:14 smithi081 ceph-mon[31453]: pgmap v20603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:14 smithi153 ceph-mon[36569]: pgmap v20603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:16 smithi104 ceph-mon[36624]: pgmap v20604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:16 smithi081 ceph-mon[31453]: pgmap v20604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:16 smithi153 ceph-mon[36569]: pgmap v20604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:18 smithi104 ceph-mon[36624]: pgmap v20605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:18 smithi081 ceph-mon[31453]: pgmap v20605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:18 smithi153 ceph-mon[36569]: pgmap v20605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:19.110+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:01:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:19.110+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:01:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:19.110+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 298 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:01:19.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:19.110+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 298 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:01:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:20 smithi104 ceph-mon[36624]: pgmap v20606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:20 smithi081 ceph-mon[31453]: pgmap v20606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:20 smithi153 ceph-mon[36569]: pgmap v20606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:22.185+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:01:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:22.185+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:01:22.186 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:22.185+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:01:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:22 smithi104 ceph-mon[36624]: pgmap v20607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:22 smithi081 ceph-mon[31453]: pgmap v20607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:22 smithi153 ceph-mon[36569]: pgmap v20607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:01:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:01:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:01:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:24 smithi104 ceph-mon[36624]: pgmap v20608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:24 smithi153 ceph-mon[36569]: pgmap v20608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:24 smithi081 ceph-mon[31453]: pgmap v20608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:26 smithi153 ceph-mon[36569]: pgmap v20609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:26 smithi104 ceph-mon[36624]: pgmap v20609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:26 smithi081 ceph-mon[31453]: pgmap v20609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:26.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:28 smithi153 ceph-mon[36569]: pgmap v20610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:28.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:28 smithi104 ceph-mon[36624]: pgmap v20610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:28 smithi081 ceph-mon[31453]: pgmap v20610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:30 smithi153 ceph-mon[36569]: pgmap v20611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:01:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:01:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:30 smithi153 ceph-mon[36569]: pgmap v20612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:30 smithi153 ceph-mon[36569]: pgmap v20613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:30 smithi104 ceph-mon[36624]: pgmap v20611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:01:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:01:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:30 smithi104 ceph-mon[36624]: pgmap v20612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:30 smithi104 ceph-mon[36624]: pgmap v20613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:30 smithi081 ceph-mon[31453]: pgmap v20611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:01:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:01:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:30 smithi081 ceph-mon[31453]: pgmap v20612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:30 smithi081 ceph-mon[31453]: pgmap v20613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:30.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:01:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:31 smithi081 ceph-mon[31453]: pgmap v20614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:31 smithi153 ceph-mon[36569]: pgmap v20614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:31 smithi104 ceph-mon[36624]: pgmap v20614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:33 smithi081 ceph-mon[31453]: pgmap v20615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:33 smithi153 ceph-mon[36569]: pgmap v20615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:33 smithi104 ceph-mon[36624]: pgmap v20615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:35 smithi081 ceph-mon[31453]: pgmap v20616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:35 smithi153 ceph-mon[36569]: pgmap v20616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:35 smithi104 ceph-mon[36624]: pgmap v20616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:37.200+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:01:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:37.200+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:01:37.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:37.200+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:01:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:37 smithi081 ceph-mon[31453]: pgmap v20617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:37 smithi153 ceph-mon[36569]: pgmap v20617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:37 smithi104 ceph-mon[36624]: pgmap v20617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:39 smithi081 ceph-mon[31453]: pgmap v20618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:39 smithi153 ceph-mon[36569]: pgmap v20618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:39 smithi104 ceph-mon[36624]: pgmap v20618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:41 smithi081 ceph-mon[31453]: pgmap v20619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:42.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:41 smithi153 ceph-mon[36569]: pgmap v20619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:41 smithi104 ceph-mon[36624]: pgmap v20619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:43 smithi081 ceph-mon[31453]: pgmap v20620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:43 smithi153 ceph-mon[36569]: pgmap v20620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:43 smithi104 ceph-mon[36624]: pgmap v20620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:45 smithi081 ceph-mon[31453]: pgmap v20621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:45 smithi153 ceph-mon[36569]: pgmap v20621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:45 smithi104 ceph-mon[36624]: pgmap v20621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:47 smithi153 ceph-mon[36569]: pgmap v20622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:47 smithi104 ceph-mon[36624]: pgmap v20622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:47 smithi081 ceph-mon[31453]: pgmap v20622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:49 smithi081 ceph-mon[31453]: pgmap v20623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:50.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:49 smithi153 ceph-mon[36569]: pgmap v20623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:49 smithi104 ceph-mon[36624]: pgmap v20623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:51 smithi081 ceph-mon[31453]: pgmap v20624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:51 smithi153 ceph-mon[36569]: pgmap v20624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:51 smithi104 ceph-mon[36624]: pgmap v20624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:52.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:52.215+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:01:52.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:52.215+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:01:52.215 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:01:52.215+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:01:53.885 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:53 smithi104 ceph-mon[36624]: pgmap v20625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:53 smithi081 ceph-mon[31453]: pgmap v20625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:53 smithi153 ceph-mon[36569]: pgmap v20625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:55 smithi081 ceph-mon[31453]: pgmap v20626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:55 smithi153 ceph-mon[36569]: pgmap v20626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:55 smithi104 ceph-mon[36624]: pgmap v20626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:01:58 smithi081 ceph-mon[31453]: pgmap v20627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:01:58 smithi153 ceph-mon[36569]: pgmap v20627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:01:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:01:58 smithi104 ceph-mon[36624]: pgmap v20627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:00 smithi081 ceph-mon[31453]: pgmap v20628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:00 smithi153 ceph-mon[36569]: pgmap v20628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:00 smithi104 ceph-mon[36624]: pgmap v20628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:02 smithi081 ceph-mon[31453]: pgmap v20629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:02 smithi153 ceph-mon[36569]: pgmap v20629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:02 smithi104 ceph-mon[36624]: pgmap v20629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:04 smithi081 ceph-mon[31453]: pgmap v20630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:04 smithi153 ceph-mon[36569]: pgmap v20630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:04 smithi104 ceph-mon[36624]: pgmap v20630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:06 smithi081 ceph-mon[31453]: pgmap v20631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:06 smithi153 ceph-mon[36569]: pgmap v20631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:06 smithi104 ceph-mon[36624]: pgmap v20631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:07.228+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:02:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:07.228+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:02:07.228 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:07.228+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:02:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:08 smithi081 ceph-mon[31453]: pgmap v20632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:08 smithi153 ceph-mon[36569]: pgmap v20632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:08 smithi104 ceph-mon[36624]: pgmap v20632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:10 smithi153 ceph-mon[36569]: pgmap v20633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:10 smithi104 ceph-mon[36624]: pgmap v20633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:10 smithi081 ceph-mon[31453]: pgmap v20633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:12 smithi153 ceph-mon[36569]: pgmap v20634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:12 smithi104 ceph-mon[36624]: pgmap v20634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:12 smithi081 ceph-mon[31453]: pgmap v20634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:14 smithi153 ceph-mon[36569]: pgmap v20635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:14 smithi104 ceph-mon[36624]: pgmap v20635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:14 smithi081 ceph-mon[31453]: pgmap v20635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:16.958 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:16 smithi081 ceph-mon[31453]: pgmap v20636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:16 smithi153 ceph-mon[36569]: pgmap v20636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:16 smithi104 ceph-mon[36624]: pgmap v20636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:19.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:18 smithi153 ceph-mon[36569]: pgmap v20637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:18 smithi104 ceph-mon[36624]: pgmap v20637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:18 smithi081 ceph-mon[31453]: pgmap v20637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:20 smithi153 ceph-mon[36569]: pgmap v20638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:20 smithi104 ceph-mon[36624]: pgmap v20638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:20 smithi081 ceph-mon[31453]: pgmap v20638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:22.244+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:02:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:22.244+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:02:22.244 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:22.244+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:02:23.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:22 smithi153 ceph-mon[36569]: pgmap v20639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:22 smithi104 ceph-mon[36624]: pgmap v20639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:22 smithi081 ceph-mon[31453]: pgmap v20639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:24 smithi153 ceph-mon[36569]: pgmap v20640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:24 smithi104 ceph-mon[36624]: pgmap v20640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:24 smithi081 ceph-mon[31453]: pgmap v20640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:26 smithi153 ceph-mon[36569]: pgmap v20641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:26 smithi104 ceph-mon[36624]: pgmap v20641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:26 smithi081 ceph-mon[31453]: pgmap v20641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:29.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:28 smithi153 ceph-mon[36569]: pgmap v20642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:28 smithi104 ceph-mon[36624]: pgmap v20642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:28 smithi081 ceph-mon[31453]: pgmap v20642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:30 smithi153 ceph-mon[36569]: pgmap v20643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:02:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:30 smithi104 ceph-mon[36624]: pgmap v20643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:02:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:30 smithi081 ceph-mon[31453]: pgmap v20643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:02:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:31.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:02:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:02:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:31 smithi153 ceph-mon[36569]: pgmap v20644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:31 smithi153 ceph-mon[36569]: pgmap v20645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:02:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:02:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:31 smithi104 ceph-mon[36624]: pgmap v20644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:31 smithi104 ceph-mon[36624]: pgmap v20645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:02:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:02:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:31 smithi081 ceph-mon[31453]: pgmap v20644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:31 smithi081 ceph-mon[31453]: pgmap v20645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:02:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:34 smithi153 ceph-mon[36569]: pgmap v20646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:34 smithi104 ceph-mon[36624]: pgmap v20646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:34 smithi081 ceph-mon[31453]: pgmap v20646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:36 smithi153 ceph-mon[36569]: pgmap v20647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:36 smithi104 ceph-mon[36624]: pgmap v20647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:36 smithi081 ceph-mon[31453]: pgmap v20647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:37.248+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:02:37.248 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:37.248+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:02:37.249 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:37.248+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:02:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:38 smithi153 ceph-mon[36569]: pgmap v20648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:38 smithi104 ceph-mon[36624]: pgmap v20648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:38 smithi081 ceph-mon[31453]: pgmap v20648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:40 smithi153 ceph-mon[36569]: pgmap v20649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:40 smithi104 ceph-mon[36624]: pgmap v20649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:40 smithi081 ceph-mon[31453]: pgmap v20649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:42.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:42 smithi153 ceph-mon[36569]: pgmap v20650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:42 smithi104 ceph-mon[36624]: pgmap v20650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:42 smithi081 ceph-mon[31453]: pgmap v20650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:44 smithi153 ceph-mon[36569]: pgmap v20651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:44 smithi104 ceph-mon[36624]: pgmap v20651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:44 smithi081 ceph-mon[31453]: pgmap v20651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:46 smithi153 ceph-mon[36569]: pgmap v20652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:46 smithi104 ceph-mon[36624]: pgmap v20652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:46 smithi081 ceph-mon[31453]: pgmap v20652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:48.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:48 smithi153 ceph-mon[36569]: pgmap v20653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:48 smithi104 ceph-mon[36624]: pgmap v20653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:48 smithi081 ceph-mon[31453]: pgmap v20653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:50 smithi153 ceph-mon[36569]: pgmap v20654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:50.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:50 smithi104 ceph-mon[36624]: pgmap v20654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:50 smithi081 ceph-mon[31453]: pgmap v20654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:52.256+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:02:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:52.256+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:02:52.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:02:52.256+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:02:52.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:52 smithi153 ceph-mon[36569]: pgmap v20655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:52.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:52 smithi104 ceph-mon[36624]: pgmap v20655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:52 smithi081 ceph-mon[31453]: pgmap v20655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:54.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:54 smithi153 ceph-mon[36569]: pgmap v20656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:54.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:54 smithi104 ceph-mon[36624]: pgmap v20656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:54 smithi081 ceph-mon[31453]: pgmap v20656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:56.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:56 smithi153 ceph-mon[36569]: pgmap v20657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:56 smithi104 ceph-mon[36624]: pgmap v20657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:56 smithi081 ceph-mon[31453]: pgmap v20657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:02:58 smithi153 ceph-mon[36569]: pgmap v20658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:02:58 smithi104 ceph-mon[36624]: pgmap v20658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:02:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:02:58 smithi081 ceph-mon[31453]: pgmap v20658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:00 smithi153 ceph-mon[36569]: pgmap v20659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:00 smithi104 ceph-mon[36624]: pgmap v20659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:00 smithi081 ceph-mon[31453]: pgmap v20659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:02 smithi153 ceph-mon[36569]: pgmap v20660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:02 smithi104 ceph-mon[36624]: pgmap v20660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:02 smithi081 ceph-mon[31453]: pgmap v20660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:04 smithi153 ceph-mon[36569]: pgmap v20661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:04 smithi104 ceph-mon[36624]: pgmap v20661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:04 smithi081 ceph-mon[31453]: pgmap v20661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:06 smithi153 ceph-mon[36569]: pgmap v20662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:06 smithi104 ceph-mon[36624]: pgmap v20662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:06 smithi081 ceph-mon[31453]: pgmap v20662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:07.264+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:03:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:07.264+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:03:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:07.264+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:03:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:08 smithi104 ceph-mon[36624]: pgmap v20663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:08 smithi081 ceph-mon[31453]: pgmap v20663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:08 smithi153 ceph-mon[36569]: pgmap v20663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:10 smithi104 ceph-mon[36624]: pgmap v20664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:10.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:10 smithi153 ceph-mon[36569]: pgmap v20664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:10.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:10 smithi081 ceph-mon[31453]: pgmap v20664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:12 smithi104 ceph-mon[36624]: pgmap v20665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:12 smithi081 ceph-mon[31453]: pgmap v20665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:12 smithi153 ceph-mon[36569]: pgmap v20665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:14 smithi104 ceph-mon[36624]: pgmap v20666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:14 smithi081 ceph-mon[31453]: pgmap v20666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:14 smithi153 ceph-mon[36569]: pgmap v20666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:16 smithi104 ceph-mon[36624]: pgmap v20667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:16 smithi081 ceph-mon[31453]: pgmap v20667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:16 smithi153 ceph-mon[36569]: pgmap v20667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:18 smithi081 ceph-mon[31453]: pgmap v20668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:18 smithi153 ceph-mon[36569]: pgmap v20668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:18 smithi104 ceph-mon[36624]: pgmap v20668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:20 smithi081 ceph-mon[31453]: pgmap v20669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:20 smithi153 ceph-mon[36569]: pgmap v20669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:20 smithi104 ceph-mon[36624]: pgmap v20669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:22.274+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:03:22.275 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:22.274+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:03:22.276 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:22.274+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:03:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:22 smithi081 ceph-mon[31453]: pgmap v20670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:22 smithi153 ceph-mon[36569]: pgmap v20670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:22 smithi104 ceph-mon[36624]: pgmap v20670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:24.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:24 smithi104 ceph-mon[36624]: pgmap v20671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:24 smithi081 ceph-mon[31453]: pgmap v20671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:24 smithi153 ceph-mon[36569]: pgmap v20671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:26 smithi081 ceph-mon[31453]: pgmap v20672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:26.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:26 smithi153 ceph-mon[36569]: pgmap v20672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:26 smithi104 ceph-mon[36624]: pgmap v20672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:28.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:28 smithi081 ceph-mon[31453]: pgmap v20673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:28 smithi153 ceph-mon[36569]: pgmap v20673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:28 smithi104 ceph-mon[36624]: pgmap v20673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:30 smithi081 ceph-mon[31453]: pgmap v20674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:30 smithi153 ceph-mon[36569]: pgmap v20674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:30 smithi104 ceph-mon[36624]: pgmap v20674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:32 smithi081 ceph-mon[31453]: pgmap v20675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:03:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:32 smithi153 ceph-mon[36569]: pgmap v20675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:32.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:03:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:32 smithi104 ceph-mon[36624]: pgmap v20675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:03:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:03:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:03:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:03:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:03:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:03:33.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:03:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:03:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:03:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:03:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:34 smithi081 ceph-mon[31453]: pgmap v20676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:34 smithi081 ceph-mon[31453]: pgmap v20677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:34 smithi153 ceph-mon[36569]: pgmap v20676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:34 smithi153 ceph-mon[36569]: pgmap v20677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:34 smithi104 ceph-mon[36624]: pgmap v20676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:34 smithi104 ceph-mon[36624]: pgmap v20677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:36.459 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:36 smithi081 ceph-mon[31453]: pgmap v20678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:36 smithi153 ceph-mon[36569]: pgmap v20678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:36 smithi104 ceph-mon[36624]: pgmap v20678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:37.287 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:37.287+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:03:37.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:37.287+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:03:37.288 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:37.287+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:03:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:38 smithi081 ceph-mon[31453]: pgmap v20679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:38 smithi153 ceph-mon[36569]: pgmap v20679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:38 smithi104 ceph-mon[36624]: pgmap v20679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:40 smithi081 ceph-mon[31453]: pgmap v20680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:40 smithi153 ceph-mon[36569]: pgmap v20680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:40 smithi104 ceph-mon[36624]: pgmap v20680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:42 smithi081 ceph-mon[31453]: pgmap v20681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:42.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:42 smithi153 ceph-mon[36569]: pgmap v20681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:42 smithi104 ceph-mon[36624]: pgmap v20681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:44 smithi153 ceph-mon[36569]: pgmap v20682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:44 smithi104 ceph-mon[36624]: pgmap v20682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:44 smithi081 ceph-mon[31453]: pgmap v20682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:46.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:46 smithi081 ceph-mon[31453]: pgmap v20683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:46 smithi153 ceph-mon[36569]: pgmap v20683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:46 smithi104 ceph-mon[36624]: pgmap v20683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:48.736 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:48 smithi081 ceph-mon[31453]: pgmap v20684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:48 smithi153 ceph-mon[36569]: pgmap v20684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:48 smithi104 ceph-mon[36624]: pgmap v20684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:50 smithi153 ceph-mon[36569]: pgmap v20685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:50 smithi104 ceph-mon[36624]: pgmap v20685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:50 smithi081 ceph-mon[31453]: pgmap v20685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:52.295 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:52.295+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:03:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:52.295+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:03:52.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:03:52.295+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:03:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:52 smithi153 ceph-mon[36569]: pgmap v20686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:52 smithi104 ceph-mon[36624]: pgmap v20686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:52 smithi081 ceph-mon[31453]: pgmap v20686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:54 smithi153 ceph-mon[36569]: pgmap v20687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:54.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:54 smithi104 ceph-mon[36624]: pgmap v20687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:54 smithi081 ceph-mon[31453]: pgmap v20687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:56 smithi153 ceph-mon[36569]: pgmap v20688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:56 smithi104 ceph-mon[36624]: pgmap v20688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:56 smithi081 ceph-mon[31453]: pgmap v20688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:03:58 smithi153 ceph-mon[36569]: pgmap v20689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:03:58 smithi104 ceph-mon[36624]: pgmap v20689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:03:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:03:58 smithi081 ceph-mon[31453]: pgmap v20689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:00 smithi153 ceph-mon[36569]: pgmap v20690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:00 smithi104 ceph-mon[36624]: pgmap v20690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:00 smithi081 ceph-mon[31453]: pgmap v20690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:02 smithi153 ceph-mon[36569]: pgmap v20691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:02 smithi104 ceph-mon[36624]: pgmap v20691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:02 smithi081 ceph-mon[31453]: pgmap v20691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:04 smithi153 ceph-mon[36569]: pgmap v20692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:04 smithi104 ceph-mon[36624]: pgmap v20692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:04 smithi081 ceph-mon[31453]: pgmap v20692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:06 smithi153 ceph-mon[36569]: pgmap v20693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:06 smithi104 ceph-mon[36624]: pgmap v20693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:06 smithi081 ceph-mon[31453]: pgmap v20693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:07.310+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:04:07.310 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:07.310+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:04:07.311 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:07.310+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:04:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:08 smithi153 ceph-mon[36569]: pgmap v20694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:08 smithi104 ceph-mon[36624]: pgmap v20694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:08 smithi081 ceph-mon[31453]: pgmap v20694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:10 smithi153 ceph-mon[36569]: pgmap v20695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:10 smithi104 ceph-mon[36624]: pgmap v20695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:10 smithi081 ceph-mon[31453]: pgmap v20695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:12 smithi153 ceph-mon[36569]: pgmap v20696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:12 smithi104 ceph-mon[36624]: pgmap v20696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:12 smithi081 ceph-mon[31453]: pgmap v20696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:14 smithi153 ceph-mon[36569]: pgmap v20697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:14 smithi104 ceph-mon[36624]: pgmap v20697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:14 smithi081 ceph-mon[31453]: pgmap v20697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:16 smithi153 ceph-mon[36569]: pgmap v20698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:16 smithi104 ceph-mon[36624]: pgmap v20698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:16 smithi081 ceph-mon[31453]: pgmap v20698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:18 smithi153 ceph-mon[36569]: pgmap v20699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:18 smithi104 ceph-mon[36624]: pgmap v20699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:18 smithi081 ceph-mon[31453]: pgmap v20699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:19.139+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:04:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:19.139+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:04:19.140 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:19.140+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 299 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:04:19.141 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:19.140+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 299 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:04:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:20 smithi153 ceph-mon[36569]: pgmap v20700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:20 smithi104 ceph-mon[36624]: pgmap v20700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:20 smithi081 ceph-mon[31453]: pgmap v20700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:22.322+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:04:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:22.322+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:04:22.322 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:22.322+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:04:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:22 smithi081 ceph-mon[31453]: pgmap v20701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:22 smithi153 ceph-mon[36569]: pgmap v20701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:22 smithi104 ceph-mon[36624]: pgmap v20701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:24 smithi153 ceph-mon[36569]: pgmap v20702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:24 smithi104 ceph-mon[36624]: pgmap v20702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:24 smithi081 ceph-mon[31453]: pgmap v20702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:26 smithi153 ceph-mon[36569]: pgmap v20703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:26 smithi104 ceph-mon[36624]: pgmap v20703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:26 smithi081 ceph-mon[31453]: pgmap v20703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:28 smithi153 ceph-mon[36569]: pgmap v20704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:28 smithi104 ceph-mon[36624]: pgmap v20704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:28 smithi081 ceph-mon[31453]: pgmap v20704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:30 smithi153 ceph-mon[36569]: pgmap v20705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:30 smithi104 ceph-mon[36624]: pgmap v20705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:30 smithi081 ceph-mon[31453]: pgmap v20705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:32 smithi153 ceph-mon[36569]: pgmap v20706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:32 smithi104 ceph-mon[36624]: pgmap v20706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:32 smithi081 ceph-mon[31453]: pgmap v20706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:04:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:04:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:04:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:34 smithi104 ceph-mon[36624]: pgmap v20707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:04:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:04:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:04:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:34 smithi081 ceph-mon[31453]: pgmap v20707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:04:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:04:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:04:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:34 smithi153 ceph-mon[36569]: pgmap v20707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:04:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:04:35.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:04:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:35 smithi104 ceph-mon[36624]: pgmap v20708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:35 smithi104 ceph-mon[36624]: pgmap v20709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:35 smithi081 ceph-mon[31453]: pgmap v20708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:35 smithi081 ceph-mon[31453]: pgmap v20709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:35 smithi153 ceph-mon[36569]: pgmap v20708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:36.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:35 smithi153 ceph-mon[36569]: pgmap v20709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:37.337+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:04:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:37.337+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:04:37.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:37.337+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:04:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:37 smithi104 ceph-mon[36624]: pgmap v20710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:37 smithi081 ceph-mon[31453]: pgmap v20710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:37 smithi153 ceph-mon[36569]: pgmap v20710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:39 smithi104 ceph-mon[36624]: pgmap v20711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:39 smithi081 ceph-mon[31453]: pgmap v20711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:39 smithi153 ceph-mon[36569]: pgmap v20711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:41 smithi104 ceph-mon[36624]: pgmap v20712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:41.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:41 smithi153 ceph-mon[36569]: pgmap v20712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:41 smithi081 ceph-mon[31453]: pgmap v20712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:43 smithi081 ceph-mon[31453]: pgmap v20713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:43 smithi153 ceph-mon[36569]: pgmap v20713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:43 smithi104 ceph-mon[36624]: pgmap v20713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:45 smithi081 ceph-mon[31453]: pgmap v20714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:45 smithi153 ceph-mon[36569]: pgmap v20714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:45 smithi104 ceph-mon[36624]: pgmap v20714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:47 smithi081 ceph-mon[31453]: pgmap v20715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:47 smithi153 ceph-mon[36569]: pgmap v20715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:47 smithi104 ceph-mon[36624]: pgmap v20715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:49 smithi081 ceph-mon[31453]: pgmap v20716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:49 smithi153 ceph-mon[36569]: pgmap v20716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:49 smithi104 ceph-mon[36624]: pgmap v20716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:51 smithi081 ceph-mon[31453]: pgmap v20717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:51 smithi153 ceph-mon[36569]: pgmap v20717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:51 smithi104 ceph-mon[36624]: pgmap v20717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:52.338 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:52.338+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:04:52.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:52.338+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:04:52.339 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:04:52.338+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:04:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:52 smithi081 ceph-mon[31453]: pgmap v20718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:52 smithi153 ceph-mon[36569]: pgmap v20718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:52 smithi104 ceph-mon[36624]: pgmap v20718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:55 smithi081 ceph-mon[31453]: pgmap v20719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:55.785 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:55 smithi104 ceph-mon[36624]: pgmap v20719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:55 smithi153 ceph-mon[36569]: pgmap v20719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:57 smithi081 ceph-mon[31453]: pgmap v20720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:57 smithi153 ceph-mon[36569]: pgmap v20720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:57 smithi104 ceph-mon[36624]: pgmap v20720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:04:59 smithi081 ceph-mon[31453]: pgmap v20721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:04:59 smithi153 ceph-mon[36569]: pgmap v20721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:04:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:04:59 smithi104 ceph-mon[36624]: pgmap v20721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:01 smithi081 ceph-mon[31453]: pgmap v20722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:01 smithi153 ceph-mon[36569]: pgmap v20722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:01 smithi104 ceph-mon[36624]: pgmap v20722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:03 smithi081 ceph-mon[31453]: pgmap v20723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:03 smithi153 ceph-mon[36569]: pgmap v20723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:03 smithi104 ceph-mon[36624]: pgmap v20723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:05 smithi081 ceph-mon[31453]: pgmap v20724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:05 smithi153 ceph-mon[36569]: pgmap v20724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:05 smithi104 ceph-mon[36624]: pgmap v20724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:07.346+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:05:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:07.346+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:05:07.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:07.346+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:05:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:07 smithi081 ceph-mon[31453]: pgmap v20725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:07 smithi153 ceph-mon[36569]: pgmap v20725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:07 smithi104 ceph-mon[36624]: pgmap v20725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:09 smithi081 ceph-mon[31453]: pgmap v20726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:09 smithi153 ceph-mon[36569]: pgmap v20726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:09 smithi104 ceph-mon[36624]: pgmap v20726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:11 smithi081 ceph-mon[31453]: pgmap v20727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:11 smithi153 ceph-mon[36569]: pgmap v20727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:11 smithi104 ceph-mon[36624]: pgmap v20727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:13 smithi081 ceph-mon[31453]: pgmap v20728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:13 smithi153 ceph-mon[36569]: pgmap v20728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:13 smithi104 ceph-mon[36624]: pgmap v20728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:15 smithi081 ceph-mon[31453]: pgmap v20729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:15 smithi153 ceph-mon[36569]: pgmap v20729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:15 smithi104 ceph-mon[36624]: pgmap v20729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:17 smithi153 ceph-mon[36569]: pgmap v20730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:17 smithi104 ceph-mon[36624]: pgmap v20730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:17 smithi081 ceph-mon[31453]: pgmap v20730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:19 smithi153 ceph-mon[36569]: pgmap v20731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:19 smithi104 ceph-mon[36624]: pgmap v20731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:19 smithi081 ceph-mon[31453]: pgmap v20731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:21.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:21 smithi153 ceph-mon[36569]: pgmap v20732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:21 smithi104 ceph-mon[36624]: pgmap v20732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:21 smithi081 ceph-mon[31453]: pgmap v20732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:22.358 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:22.358+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:05:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:22.358+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:05:22.359 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:22.358+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:05:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:23 smithi153 ceph-mon[36569]: pgmap v20733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:23 smithi104 ceph-mon[36624]: pgmap v20733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:23 smithi081 ceph-mon[31453]: pgmap v20733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:25 smithi153 ceph-mon[36569]: pgmap v20734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:25 smithi104 ceph-mon[36624]: pgmap v20734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:25 smithi081 ceph-mon[31453]: pgmap v20734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:27 smithi153 ceph-mon[36569]: pgmap v20735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:27 smithi104 ceph-mon[36624]: pgmap v20735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:27 smithi081 ceph-mon[31453]: pgmap v20735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:29.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:29 smithi153 ceph-mon[36569]: pgmap v20736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:29 smithi104 ceph-mon[36624]: pgmap v20736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:29 smithi081 ceph-mon[31453]: pgmap v20736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:31 smithi153 ceph-mon[36569]: pgmap v20737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:31 smithi104 ceph-mon[36624]: pgmap v20737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:31 smithi081 ceph-mon[31453]: pgmap v20737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:33 smithi153 ceph-mon[36569]: pgmap v20738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:33 smithi104 ceph-mon[36624]: pgmap v20738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:33 smithi081 ceph-mon[31453]: pgmap v20738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:35 smithi153 ceph-mon[36569]: pgmap v20739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:35.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:05:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:35 smithi104 ceph-mon[36624]: pgmap v20739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:05:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:35 smithi081 ceph-mon[31453]: pgmap v20739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:35.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:05:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:05:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:05:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:05:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:05:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:05:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:05:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:05:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:05:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:05:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:37.361+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:05:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:37.361+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:05:37.362 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:37.361+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:05:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:37 smithi153 ceph-mon[36569]: pgmap v20740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:37.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:37 smithi153 ceph-mon[36569]: pgmap v20741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:37 smithi104 ceph-mon[36624]: pgmap v20740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:37 smithi104 ceph-mon[36624]: pgmap v20741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:37 smithi081 ceph-mon[31453]: pgmap v20740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:37 smithi081 ceph-mon[31453]: pgmap v20741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:39 smithi153 ceph-mon[36569]: pgmap v20742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:39 smithi104 ceph-mon[36624]: pgmap v20742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:39 smithi081 ceph-mon[31453]: pgmap v20742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:41 smithi153 ceph-mon[36569]: pgmap v20743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:41 smithi104 ceph-mon[36624]: pgmap v20743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:41 smithi081 ceph-mon[31453]: pgmap v20743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:43 smithi153 ceph-mon[36569]: pgmap v20744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:43 smithi104 ceph-mon[36624]: pgmap v20744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:43 smithi081 ceph-mon[31453]: pgmap v20744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:45 smithi153 ceph-mon[36569]: pgmap v20745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:45 smithi104 ceph-mon[36624]: pgmap v20745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:45 smithi081 ceph-mon[31453]: pgmap v20745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:47 smithi153 ceph-mon[36569]: pgmap v20746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:47 smithi104 ceph-mon[36624]: pgmap v20746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:47 smithi081 ceph-mon[31453]: pgmap v20746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:49 smithi153 ceph-mon[36569]: pgmap v20747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:49 smithi104 ceph-mon[36624]: pgmap v20747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:49 smithi081 ceph-mon[31453]: pgmap v20747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:51 smithi153 ceph-mon[36569]: pgmap v20748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:51 smithi104 ceph-mon[36624]: pgmap v20748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:51 smithi081 ceph-mon[31453]: pgmap v20748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:52.369+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:05:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:52.369+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:05:52.369 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:05:52.369+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:05:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:53 smithi153 ceph-mon[36569]: pgmap v20749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:53 smithi104 ceph-mon[36624]: pgmap v20749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:53 smithi081 ceph-mon[31453]: pgmap v20749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:55 smithi153 ceph-mon[36569]: pgmap v20750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:55 smithi104 ceph-mon[36624]: pgmap v20750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:55 smithi081 ceph-mon[31453]: pgmap v20750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:57 smithi153 ceph-mon[36569]: pgmap v20751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:57 smithi104 ceph-mon[36624]: pgmap v20751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:57 smithi081 ceph-mon[31453]: pgmap v20751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:05:59 smithi153 ceph-mon[36569]: pgmap v20752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:05:59 smithi104 ceph-mon[36624]: pgmap v20752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:05:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:05:59 smithi081 ceph-mon[31453]: pgmap v20752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:01 smithi153 ceph-mon[36569]: pgmap v20753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:01 smithi104 ceph-mon[36624]: pgmap v20753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:01 smithi081 ceph-mon[31453]: pgmap v20753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:03 smithi153 ceph-mon[36569]: pgmap v20754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:03 smithi104 ceph-mon[36624]: pgmap v20754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:03 smithi081 ceph-mon[31453]: pgmap v20754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:05 smithi153 ceph-mon[36569]: pgmap v20755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:05 smithi104 ceph-mon[36624]: pgmap v20755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:05 smithi081 ceph-mon[31453]: pgmap v20755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:07.385+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:06:07.385 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:07.385+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:06:07.386 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:07.385+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:06:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:07 smithi104 ceph-mon[36624]: pgmap v20756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:07 smithi081 ceph-mon[31453]: pgmap v20756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:07 smithi153 ceph-mon[36569]: pgmap v20756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:09 smithi104 ceph-mon[36624]: pgmap v20757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:09 smithi081 ceph-mon[31453]: pgmap v20757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:09 smithi153 ceph-mon[36569]: pgmap v20757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:11 smithi081 ceph-mon[31453]: pgmap v20758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:11 smithi104 ceph-mon[36624]: pgmap v20758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:11.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:11 smithi153 ceph-mon[36569]: pgmap v20758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:13 smithi104 ceph-mon[36624]: pgmap v20759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:13 smithi081 ceph-mon[31453]: pgmap v20759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:13 smithi153 ceph-mon[36569]: pgmap v20759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:15 smithi104 ceph-mon[36624]: pgmap v20760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:15 smithi081 ceph-mon[31453]: pgmap v20760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:15 smithi153 ceph-mon[36569]: pgmap v20760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:17 smithi081 ceph-mon[31453]: pgmap v20761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:17 smithi153 ceph-mon[36569]: pgmap v20761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:17 smithi104 ceph-mon[36624]: pgmap v20761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:19 smithi081 ceph-mon[31453]: pgmap v20762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:19 smithi153 ceph-mon[36569]: pgmap v20762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:19 smithi104 ceph-mon[36624]: pgmap v20762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:21 smithi081 ceph-mon[31453]: pgmap v20763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:21 smithi153 ceph-mon[36569]: pgmap v20763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:21 smithi104 ceph-mon[36624]: pgmap v20763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:22.400 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:22.400+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:06:22.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:22.400+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:06:22.401 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:22.400+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:06:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:23 smithi081 ceph-mon[31453]: pgmap v20764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:23 smithi153 ceph-mon[36569]: pgmap v20764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:23 smithi104 ceph-mon[36624]: pgmap v20764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:25.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:25 smithi104 ceph-mon[36624]: pgmap v20765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:25 smithi081 ceph-mon[31453]: pgmap v20765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:25 smithi153 ceph-mon[36569]: pgmap v20765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:27 smithi081 ceph-mon[31453]: pgmap v20766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:27 smithi153 ceph-mon[36569]: pgmap v20766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:27 smithi104 ceph-mon[36624]: pgmap v20766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:29 smithi081 ceph-mon[31453]: pgmap v20767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:29 smithi153 ceph-mon[36569]: pgmap v20767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:29 smithi104 ceph-mon[36624]: pgmap v20767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:31 smithi081 ceph-mon[31453]: pgmap v20768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:31 smithi153 ceph-mon[36569]: pgmap v20768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:31 smithi104 ceph-mon[36624]: pgmap v20768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:33 smithi081 ceph-mon[31453]: pgmap v20769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:33 smithi153 ceph-mon[36569]: pgmap v20769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:33 smithi104 ceph-mon[36624]: pgmap v20769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:34 smithi153 ceph-mon[36569]: pgmap v20770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:35.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:34 smithi104 ceph-mon[36624]: pgmap v20770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:34 smithi081 ceph-mon[31453]: pgmap v20770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:36 smithi153 ceph-mon[36569]: pgmap v20771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:06:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:36 smithi104 ceph-mon[36624]: pgmap v20771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:06:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:36 smithi081 ceph-mon[31453]: pgmap v20771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:06:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:37.415+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:06:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:37.415+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:06:37.415 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:37.415+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:06:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:06:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:06:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:37 smithi153 ceph-mon[36569]: pgmap v20772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:37 smithi153 ceph-mon[36569]: pgmap v20773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:38.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:06:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:06:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:06:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:37 smithi104 ceph-mon[36624]: pgmap v20772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:37 smithi104 ceph-mon[36624]: pgmap v20773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:38.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:06:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:06:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:06:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:37 smithi081 ceph-mon[31453]: pgmap v20772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:37 smithi081 ceph-mon[31453]: pgmap v20773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:38.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:06:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:40 smithi153 ceph-mon[36569]: pgmap v20774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:40 smithi104 ceph-mon[36624]: pgmap v20774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:40 smithi081 ceph-mon[31453]: pgmap v20774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:42 smithi153 ceph-mon[36569]: pgmap v20775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:42 smithi104 ceph-mon[36624]: pgmap v20775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:42 smithi081 ceph-mon[31453]: pgmap v20775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:44 smithi153 ceph-mon[36569]: pgmap v20776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:44 smithi104 ceph-mon[36624]: pgmap v20776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:44 smithi081 ceph-mon[31453]: pgmap v20776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:46 smithi153 ceph-mon[36569]: pgmap v20777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:46 smithi104 ceph-mon[36624]: pgmap v20777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:46 smithi081 ceph-mon[31453]: pgmap v20777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:48.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:48 smithi081 ceph-mon[31453]: pgmap v20778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:48 smithi153 ceph-mon[36569]: pgmap v20778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:48 smithi104 ceph-mon[36624]: pgmap v20778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:50 smithi153 ceph-mon[36569]: pgmap v20779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:50 smithi104 ceph-mon[36624]: pgmap v20779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:50 smithi081 ceph-mon[31453]: pgmap v20779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:52.430+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:06:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:52.430+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:06:52.431 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:06:52.430+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:06:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:52 smithi153 ceph-mon[36569]: pgmap v20780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:52 smithi104 ceph-mon[36624]: pgmap v20780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:52 smithi081 ceph-mon[31453]: pgmap v20780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:54 smithi153 ceph-mon[36569]: pgmap v20781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:54 smithi104 ceph-mon[36624]: pgmap v20781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:54 smithi081 ceph-mon[31453]: pgmap v20781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:56 smithi153 ceph-mon[36569]: pgmap v20782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:56 smithi104 ceph-mon[36624]: pgmap v20782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:56 smithi081 ceph-mon[31453]: pgmap v20782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:06:58 smithi153 ceph-mon[36569]: pgmap v20783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:06:58 smithi104 ceph-mon[36624]: pgmap v20783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:06:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:06:58 smithi081 ceph-mon[31453]: pgmap v20783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:00.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:00 smithi153 ceph-mon[36569]: pgmap v20784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:00 smithi104 ceph-mon[36624]: pgmap v20784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:00 smithi081 ceph-mon[31453]: pgmap v20784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:02 smithi153 ceph-mon[36569]: pgmap v20785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:02 smithi104 ceph-mon[36624]: pgmap v20785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:02 smithi081 ceph-mon[31453]: pgmap v20785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:04 smithi153 ceph-mon[36569]: pgmap v20786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:04 smithi104 ceph-mon[36624]: pgmap v20786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:04 smithi081 ceph-mon[31453]: pgmap v20786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:06.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:06 smithi153 ceph-mon[36569]: pgmap v20787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:06 smithi104 ceph-mon[36624]: pgmap v20787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:06 smithi081 ceph-mon[31453]: pgmap v20787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:07.446 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:07.446+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:07:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:07.446+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:07:07.447 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:07.446+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:07:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:08 smithi153 ceph-mon[36569]: pgmap v20788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:08 smithi104 ceph-mon[36624]: pgmap v20788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:08 smithi081 ceph-mon[31453]: pgmap v20788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:10 smithi153 ceph-mon[36569]: pgmap v20789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:10 smithi104 ceph-mon[36624]: pgmap v20789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:10 smithi081 ceph-mon[31453]: pgmap v20789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:12 smithi153 ceph-mon[36569]: pgmap v20790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:12 smithi104 ceph-mon[36624]: pgmap v20790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:12 smithi081 ceph-mon[31453]: pgmap v20790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:14 smithi153 ceph-mon[36569]: pgmap v20791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:14 smithi104 ceph-mon[36624]: pgmap v20791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:14 smithi081 ceph-mon[31453]: pgmap v20791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:16 smithi153 ceph-mon[36569]: pgmap v20792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:16 smithi104 ceph-mon[36624]: pgmap v20792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:16 smithi081 ceph-mon[31453]: pgmap v20792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:18.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:18 smithi153 ceph-mon[36569]: pgmap v20793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:18 smithi104 ceph-mon[36624]: pgmap v20793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:18 smithi081 ceph-mon[31453]: pgmap v20793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:19.169+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:07:19.169 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:19.169+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:07:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:19.170+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 300 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:07:19.170 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:19.170+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 300 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:07:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:20 smithi153 ceph-mon[36569]: pgmap v20794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:20 smithi104 ceph-mon[36624]: pgmap v20794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:20 smithi081 ceph-mon[31453]: pgmap v20794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:22.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:22.462+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:07:22.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:22.462+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:07:22.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:22.462+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:07:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:22 smithi153 ceph-mon[36569]: pgmap v20795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:22 smithi104 ceph-mon[36624]: pgmap v20795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:22 smithi081 ceph-mon[31453]: pgmap v20795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:24.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:24 smithi153 ceph-mon[36569]: pgmap v20796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:24 smithi104 ceph-mon[36624]: pgmap v20796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:24 smithi081 ceph-mon[31453]: pgmap v20796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:26.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:26 smithi153 ceph-mon[36569]: pgmap v20797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:26 smithi104 ceph-mon[36624]: pgmap v20797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:26 smithi081 ceph-mon[31453]: pgmap v20797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:28 smithi153 ceph-mon[36569]: pgmap v20798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:28 smithi104 ceph-mon[36624]: pgmap v20798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:28 smithi081 ceph-mon[31453]: pgmap v20798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:30 smithi104 ceph-mon[36624]: pgmap v20799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:30 smithi081 ceph-mon[31453]: pgmap v20799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:30 smithi153 ceph-mon[36569]: pgmap v20799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:32 smithi104 ceph-mon[36624]: pgmap v20800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:32 smithi081 ceph-mon[31453]: pgmap v20800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:32 smithi153 ceph-mon[36569]: pgmap v20800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:34 smithi104 ceph-mon[36624]: pgmap v20801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:34 smithi081 ceph-mon[31453]: pgmap v20801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:34 smithi153 ceph-mon[36569]: pgmap v20801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:36 smithi104 ceph-mon[36624]: pgmap v20802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:36 smithi081 ceph-mon[31453]: pgmap v20802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:36 smithi153 ceph-mon[36569]: pgmap v20802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:37.471+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:07:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:37.471+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:07:37.471 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:37.471+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:07:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: pgmap v20803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:38.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: pgmap v20803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: pgmap v20803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:07:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:07:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:07:39.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:39 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:07:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:07:40.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:39 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:07:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:07:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:07:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:07:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:07:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:07:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:07:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:07:40.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:39 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:07:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:40 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:07:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:40 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:07:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:40 smithi081 ceph-mon[31453]: pgmap v20804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:40 smithi081 ceph-mon[31453]: pgmap v20805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:40 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:07:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:40 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:07:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:40 smithi153 ceph-mon[36569]: pgmap v20804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:40 smithi153 ceph-mon[36569]: pgmap v20805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:40 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:07:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:40 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:07:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:40 smithi104 ceph-mon[36624]: pgmap v20804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:40 smithi104 ceph-mon[36624]: pgmap v20805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:42.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:42 smithi153 ceph-mon[36569]: pgmap v20806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:42 smithi081 ceph-mon[31453]: pgmap v20806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:42 smithi104 ceph-mon[36624]: pgmap v20806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:44 smithi081 ceph-mon[31453]: pgmap v20807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:44 smithi153 ceph-mon[36569]: pgmap v20807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:44 smithi104 ceph-mon[36624]: pgmap v20807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:46 smithi081 ceph-mon[31453]: pgmap v20808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:46 smithi153 ceph-mon[36569]: pgmap v20808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:46 smithi104 ceph-mon[36624]: pgmap v20808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:48 smithi081 ceph-mon[31453]: pgmap v20809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:48 smithi153 ceph-mon[36569]: pgmap v20809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:48 smithi104 ceph-mon[36624]: pgmap v20809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:50 smithi081 ceph-mon[31453]: pgmap v20810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:50 smithi153 ceph-mon[36569]: pgmap v20810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:50 smithi104 ceph-mon[36624]: pgmap v20810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:52.484 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:52.484+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:07:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:52.484+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:07:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:07:52.484+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:07:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:52 smithi081 ceph-mon[31453]: pgmap v20811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:52 smithi153 ceph-mon[36569]: pgmap v20811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:52 smithi104 ceph-mon[36624]: pgmap v20811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:54 smithi081 ceph-mon[31453]: pgmap v20812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:54 smithi153 ceph-mon[36569]: pgmap v20812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:54 smithi104 ceph-mon[36624]: pgmap v20812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:56 smithi081 ceph-mon[31453]: pgmap v20813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:57.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:56 smithi153 ceph-mon[36569]: pgmap v20813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:56 smithi104 ceph-mon[36624]: pgmap v20813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:58 smithi081 ceph-mon[31453]: pgmap v20814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:58 smithi153 ceph-mon[36569]: pgmap v20814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:07:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:58 smithi104 ceph-mon[36624]: pgmap v20814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:07:59 smithi153 ceph-mon[36569]: pgmap v20815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:07:59 smithi104 ceph-mon[36624]: pgmap v20815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:07:59 smithi081 ceph-mon[31453]: pgmap v20815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:02 smithi153 ceph-mon[36569]: pgmap v20816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:02 smithi104 ceph-mon[36624]: pgmap v20816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:02 smithi081 ceph-mon[31453]: pgmap v20816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:04.177+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:08:04.177 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:04.177+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:08:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:04.177+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 301 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:08:04.178 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:04.177+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 301 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:08:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:04 smithi153 ceph-mon[36569]: pgmap v20817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:04 smithi104 ceph-mon[36624]: pgmap v20817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:04.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:04 smithi081 ceph-mon[31453]: pgmap v20817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:06.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:06 smithi153 ceph-mon[36569]: pgmap v20818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:06 smithi104 ceph-mon[36624]: pgmap v20818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:06 smithi081 ceph-mon[31453]: pgmap v20818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:07.499+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:08:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:07.500+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:08:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:07.500+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:08:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:08 smithi153 ceph-mon[36569]: pgmap v20819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:08 smithi104 ceph-mon[36624]: pgmap v20819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:08 smithi081 ceph-mon[31453]: pgmap v20819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:10.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:10 smithi153 ceph-mon[36569]: pgmap v20820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:10 smithi104 ceph-mon[36624]: pgmap v20820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:10 smithi081 ceph-mon[31453]: pgmap v20820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:12 smithi153 ceph-mon[36569]: pgmap v20821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:12 smithi104 ceph-mon[36624]: pgmap v20821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:12 smithi081 ceph-mon[31453]: pgmap v20821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:14 smithi104 ceph-mon[36624]: pgmap v20822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:14 smithi081 ceph-mon[31453]: pgmap v20822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:14 smithi153 ceph-mon[36569]: pgmap v20822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:16 smithi104 ceph-mon[36624]: pgmap v20823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:16 smithi081 ceph-mon[31453]: pgmap v20823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:16 smithi153 ceph-mon[36569]: pgmap v20823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:18 smithi104 ceph-mon[36624]: pgmap v20824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:18 smithi081 ceph-mon[31453]: pgmap v20824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:18 smithi153 ceph-mon[36569]: pgmap v20824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:20 smithi104 ceph-mon[36624]: pgmap v20825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:20 smithi081 ceph-mon[31453]: pgmap v20825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:20 smithi153 ceph-mon[36569]: pgmap v20825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:22 smithi104 ceph-mon[36624]: pgmap v20826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:22 smithi081 ceph-mon[31453]: pgmap v20826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:22.504+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:08:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:22.504+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:08:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:22.504+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:08:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:22 smithi153 ceph-mon[36569]: pgmap v20826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:24 smithi081 ceph-mon[31453]: pgmap v20827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:24 smithi153 ceph-mon[36569]: pgmap v20827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:24 smithi104 ceph-mon[36624]: pgmap v20827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:26.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:26 smithi104 ceph-mon[36624]: pgmap v20828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:26 smithi081 ceph-mon[31453]: pgmap v20828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:26 smithi153 ceph-mon[36569]: pgmap v20828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:28 smithi081 ceph-mon[31453]: pgmap v20829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:28.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:28 smithi153 ceph-mon[36569]: pgmap v20829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:28 smithi104 ceph-mon[36624]: pgmap v20829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:30 smithi081 ceph-mon[31453]: pgmap v20830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:30 smithi153 ceph-mon[36569]: pgmap v20830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:30 smithi104 ceph-mon[36624]: pgmap v20830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:32 smithi081 ceph-mon[31453]: pgmap v20831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:32 smithi153 ceph-mon[36569]: pgmap v20831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:32 smithi104 ceph-mon[36624]: pgmap v20831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:34 smithi081 ceph-mon[31453]: pgmap v20832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:34 smithi153 ceph-mon[36569]: pgmap v20832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:34 smithi104 ceph-mon[36624]: pgmap v20832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:36 smithi081 ceph-mon[31453]: pgmap v20833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:36 smithi153 ceph-mon[36569]: pgmap v20833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:36 smithi104 ceph-mon[36624]: pgmap v20833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:37.519+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:08:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:37.519+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:08:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:37.519+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:08:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:38 smithi081 ceph-mon[31453]: pgmap v20834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:38 smithi153 ceph-mon[36569]: pgmap v20834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:38 smithi104 ceph-mon[36624]: pgmap v20834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:40 smithi081 ceph-mon[31453]: pgmap v20835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:08:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:40 smithi153 ceph-mon[36569]: pgmap v20835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:40.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:08:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:40 smithi104 ceph-mon[36624]: pgmap v20835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:08:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:08:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:08:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:41 smithi081 ceph-mon[31453]: pgmap v20836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:41 smithi081 ceph-mon[31453]: pgmap v20837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:41.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:08:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:08:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:08:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:41 smithi153 ceph-mon[36569]: pgmap v20836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:41 smithi153 ceph-mon[36569]: pgmap v20837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:41.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:08:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:08:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:08:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:41 smithi104 ceph-mon[36624]: pgmap v20836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:41 smithi104 ceph-mon[36624]: pgmap v20837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:08:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:43 smithi104 ceph-mon[36624]: pgmap v20838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:43 smithi081 ceph-mon[31453]: pgmap v20838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:43 smithi153 ceph-mon[36569]: pgmap v20838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:45 smithi081 ceph-mon[31453]: pgmap v20839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:45 smithi153 ceph-mon[36569]: pgmap v20839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:45 smithi104 ceph-mon[36624]: pgmap v20839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:47 smithi081 ceph-mon[31453]: pgmap v20840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:47 smithi153 ceph-mon[36569]: pgmap v20840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:47 smithi104 ceph-mon[36624]: pgmap v20840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:49 smithi081 ceph-mon[31453]: pgmap v20841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:49 smithi153 ceph-mon[36569]: pgmap v20841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:49 smithi104 ceph-mon[36624]: pgmap v20841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:51 smithi081 ceph-mon[31453]: pgmap v20842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:51 smithi153 ceph-mon[36569]: pgmap v20842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:51 smithi104 ceph-mon[36624]: pgmap v20842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:52.522+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:08:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:52.522+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:08:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:08:52.522+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:08:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:53 smithi081 ceph-mon[31453]: pgmap v20843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:53 smithi153 ceph-mon[36569]: pgmap v20843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:53 smithi104 ceph-mon[36624]: pgmap v20843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:55 smithi081 ceph-mon[31453]: pgmap v20844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:55 smithi153 ceph-mon[36569]: pgmap v20844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:55 smithi104 ceph-mon[36624]: pgmap v20844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:57.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:57 smithi104 ceph-mon[36624]: pgmap v20845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:57 smithi081 ceph-mon[31453]: pgmap v20845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:58.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:57 smithi153 ceph-mon[36569]: pgmap v20845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:08:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:08:59 smithi081 ceph-mon[31453]: pgmap v20846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:00.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:08:59 smithi153 ceph-mon[36569]: pgmap v20846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:08:59 smithi104 ceph-mon[36624]: pgmap v20846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:01 smithi081 ceph-mon[31453]: pgmap v20847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:01 smithi153 ceph-mon[36569]: pgmap v20847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:01 smithi104 ceph-mon[36624]: pgmap v20847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:03 smithi081 ceph-mon[31453]: pgmap v20848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:03 smithi153 ceph-mon[36569]: pgmap v20848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:03 smithi104 ceph-mon[36624]: pgmap v20848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:05 smithi081 ceph-mon[31453]: pgmap v20849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:05 smithi153 ceph-mon[36569]: pgmap v20849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:05 smithi104 ceph-mon[36624]: pgmap v20849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:07.537+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:09:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:07.537+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:09:07.538 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:07.537+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:09:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:07 smithi081 ceph-mon[31453]: pgmap v20850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:07 smithi153 ceph-mon[36569]: pgmap v20850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:07 smithi104 ceph-mon[36624]: pgmap v20850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:09 smithi081 ceph-mon[31453]: pgmap v20851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:09 smithi153 ceph-mon[36569]: pgmap v20851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:09 smithi104 ceph-mon[36624]: pgmap v20851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:11 smithi081 ceph-mon[31453]: pgmap v20852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:11 smithi153 ceph-mon[36569]: pgmap v20852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:11 smithi104 ceph-mon[36624]: pgmap v20852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:13 smithi153 ceph-mon[36569]: pgmap v20853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:13 smithi104 ceph-mon[36624]: pgmap v20853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:13 smithi081 ceph-mon[31453]: pgmap v20853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:14 smithi153 ceph-mon[36569]: pgmap v20854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:14 smithi104 ceph-mon[36624]: pgmap v20854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:14 smithi081 ceph-mon[31453]: pgmap v20854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:17 smithi081 ceph-mon[31453]: pgmap v20855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:17 smithi153 ceph-mon[36569]: pgmap v20855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:17 smithi104 ceph-mon[36624]: pgmap v20855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:19 smithi081 ceph-mon[31453]: pgmap v20856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:19 smithi153 ceph-mon[36569]: pgmap v20856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:19 smithi104 ceph-mon[36624]: pgmap v20856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:21 smithi081 ceph-mon[31453]: pgmap v20857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:21 smithi153 ceph-mon[36569]: pgmap v20857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:21 smithi104 ceph-mon[36624]: pgmap v20857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:22.553+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:09:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:22.553+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:09:22.553 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:22.553+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:09:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:23 smithi081 ceph-mon[31453]: pgmap v20858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:24.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:23 smithi153 ceph-mon[36569]: pgmap v20858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:23 smithi104 ceph-mon[36624]: pgmap v20858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:25.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:25 smithi081 ceph-mon[31453]: pgmap v20859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:25 smithi153 ceph-mon[36569]: pgmap v20859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:25 smithi104 ceph-mon[36624]: pgmap v20859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:27 smithi081 ceph-mon[31453]: pgmap v20860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:27 smithi153 ceph-mon[36569]: pgmap v20860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:27 smithi104 ceph-mon[36624]: pgmap v20860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:29.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:29 smithi081 ceph-mon[31453]: pgmap v20861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:29 smithi153 ceph-mon[36569]: pgmap v20861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:29 smithi104 ceph-mon[36624]: pgmap v20861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:31 smithi081 ceph-mon[31453]: pgmap v20862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:31 smithi153 ceph-mon[36569]: pgmap v20862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:31 smithi104 ceph-mon[36624]: pgmap v20862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:33 smithi081 ceph-mon[31453]: pgmap v20863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:33 smithi153 ceph-mon[36569]: pgmap v20863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:33 smithi104 ceph-mon[36624]: pgmap v20863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:35 smithi153 ceph-mon[36569]: pgmap v20864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:35 smithi104 ceph-mon[36624]: pgmap v20864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:35 smithi081 ceph-mon[31453]: pgmap v20864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:37.565+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:09:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:37.566+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:09:37.566 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:37.566+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:09:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:37 smithi153 ceph-mon[36569]: pgmap v20865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:37 smithi104 ceph-mon[36624]: pgmap v20865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:37 smithi081 ceph-mon[31453]: pgmap v20865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:38 smithi153 ceph-mon[36569]: pgmap v20866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:38 smithi104 ceph-mon[36624]: pgmap v20866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:38 smithi081 ceph-mon[31453]: pgmap v20866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:41 smithi081 ceph-mon[31453]: pgmap v20867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:41 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:41 smithi153 ceph-mon[36569]: pgmap v20867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:41 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:41 smithi104 ceph-mon[36624]: pgmap v20867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:41 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:09:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:43 smithi081 ceph-mon[31453]: pgmap v20868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:09:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:09:43.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:43 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:43 smithi153 ceph-mon[36569]: pgmap v20868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:09:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:09:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:43 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:43 smithi104 ceph-mon[36624]: pgmap v20868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:09:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:09:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:43 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:09:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:44 smithi081 ceph-mon[31453]: pgmap v20869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:44 smithi081 ceph-mon[31453]: pgmap v20870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:44.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:44 smithi153 ceph-mon[36569]: pgmap v20869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:44.958 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:44 smithi153 ceph-mon[36569]: pgmap v20870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:44 smithi104 ceph-mon[36624]: pgmap v20869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:44 smithi104 ceph-mon[36624]: pgmap v20870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:46 smithi081 ceph-mon[31453]: pgmap v20871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:46 smithi153 ceph-mon[36569]: pgmap v20871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:46 smithi104 ceph-mon[36624]: pgmap v20871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:48 smithi081 ceph-mon[31453]: pgmap v20872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:48 smithi153 ceph-mon[36569]: pgmap v20872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:48 smithi104 ceph-mon[36624]: pgmap v20872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:50 smithi081 ceph-mon[31453]: pgmap v20873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:50 smithi153 ceph-mon[36569]: pgmap v20873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:50 smithi104 ceph-mon[36624]: pgmap v20873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:52.569+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:09:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:52.569+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:09:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:09:52.569+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:09:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:52 smithi081 ceph-mon[31453]: pgmap v20874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:52 smithi153 ceph-mon[36569]: pgmap v20874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:52 smithi104 ceph-mon[36624]: pgmap v20874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:54.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:54 smithi081 ceph-mon[31453]: pgmap v20875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:54 smithi153 ceph-mon[36569]: pgmap v20875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:54 smithi104 ceph-mon[36624]: pgmap v20875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:56 smithi081 ceph-mon[31453]: pgmap v20876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:56 smithi153 ceph-mon[36569]: pgmap v20876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:56 smithi104 ceph-mon[36624]: pgmap v20876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:09:58 smithi153 ceph-mon[36569]: pgmap v20877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:09:58 smithi104 ceph-mon[36624]: pgmap v20877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:09:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:09:58 smithi081 ceph-mon[31453]: pgmap v20877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:00 smithi153 ceph-mon[36569]: pgmap v20878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:10:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:00 smithi104 ceph-mon[36624]: pgmap v20878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:10:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:00 smithi081 ceph-mon[31453]: pgmap v20878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:10:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:02 smithi153 ceph-mon[36569]: pgmap v20879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:02 smithi104 ceph-mon[36624]: pgmap v20879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:02 smithi081 ceph-mon[31453]: pgmap v20879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:03 smithi153 ceph-mon[36569]: pgmap v20880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:03 smithi104 ceph-mon[36624]: pgmap v20880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:03 smithi081 ceph-mon[31453]: pgmap v20880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:06 smithi153 ceph-mon[36569]: pgmap v20881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:06 smithi104 ceph-mon[36624]: pgmap v20881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:06 smithi081 ceph-mon[31453]: pgmap v20881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:07.576+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:10:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:07.576+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:10:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:07.576+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:10:08.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:08 smithi153 ceph-mon[36569]: pgmap v20882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:08 smithi104 ceph-mon[36624]: pgmap v20882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:08 smithi081 ceph-mon[31453]: pgmap v20882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:10 smithi153 ceph-mon[36569]: pgmap v20883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:10 smithi104 ceph-mon[36624]: pgmap v20883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:10 smithi081 ceph-mon[31453]: pgmap v20883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:12 smithi153 ceph-mon[36569]: pgmap v20884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:12 smithi104 ceph-mon[36624]: pgmap v20884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:12 smithi081 ceph-mon[31453]: pgmap v20884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:14 smithi153 ceph-mon[36569]: pgmap v20885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:14 smithi104 ceph-mon[36624]: pgmap v20885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:14 smithi081 ceph-mon[31453]: pgmap v20885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:16 smithi153 ceph-mon[36569]: pgmap v20886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:16 smithi104 ceph-mon[36624]: pgmap v20886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:16 smithi081 ceph-mon[31453]: pgmap v20886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:18 smithi153 ceph-mon[36569]: pgmap v20887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:18 smithi104 ceph-mon[36624]: pgmap v20887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:18 smithi081 ceph-mon[31453]: pgmap v20887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:19.199+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:10:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:19.199+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:10:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:19.199+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 302 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:10:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:19.199+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 302 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:10:20.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:20 smithi153 ceph-mon[36569]: pgmap v20888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:20 smithi104 ceph-mon[36624]: pgmap v20888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:20 smithi081 ceph-mon[31453]: pgmap v20888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:22 smithi153 ceph-mon[36569]: pgmap v20889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:22 smithi104 ceph-mon[36624]: pgmap v20889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:22.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:22 smithi081 ceph-mon[31453]: pgmap v20889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:22.590+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:10:22.590 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:22.590+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:10:22.591 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:22.590+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:10:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:24 smithi153 ceph-mon[36569]: pgmap v20890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:24 smithi104 ceph-mon[36624]: pgmap v20890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:24 smithi081 ceph-mon[31453]: pgmap v20890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:26 smithi153 ceph-mon[36569]: pgmap v20891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:26 smithi104 ceph-mon[36624]: pgmap v20891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:26 smithi081 ceph-mon[31453]: pgmap v20891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:28 smithi104 ceph-mon[36624]: pgmap v20892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:28 smithi081 ceph-mon[31453]: pgmap v20892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:28 smithi153 ceph-mon[36569]: pgmap v20892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:30 smithi104 ceph-mon[36624]: pgmap v20893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:30.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:30 smithi081 ceph-mon[31453]: pgmap v20893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:30 smithi153 ceph-mon[36569]: pgmap v20893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:32 smithi104 ceph-mon[36624]: pgmap v20894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:32.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:32 smithi081 ceph-mon[31453]: pgmap v20894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:32 smithi153 ceph-mon[36569]: pgmap v20894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:34 smithi104 ceph-mon[36624]: pgmap v20895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:34 smithi081 ceph-mon[31453]: pgmap v20895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:34 smithi153 ceph-mon[36569]: pgmap v20895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:36 smithi104 ceph-mon[36624]: pgmap v20896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:36 smithi081 ceph-mon[31453]: pgmap v20896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:36 smithi153 ceph-mon[36569]: pgmap v20896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:37.601+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:10:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:37.601+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:10:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:37.601+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:10:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:38 smithi081 ceph-mon[31453]: pgmap v20897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:38.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:38 smithi153 ceph-mon[36569]: pgmap v20897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:38 smithi104 ceph-mon[36624]: pgmap v20897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:40.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:40 smithi081 ceph-mon[31453]: pgmap v20898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:40 smithi153 ceph-mon[36569]: pgmap v20898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:40 smithi104 ceph-mon[36624]: pgmap v20898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:42 smithi081 ceph-mon[31453]: pgmap v20899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:42.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:42 smithi153 ceph-mon[36569]: pgmap v20899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:42 smithi104 ceph-mon[36624]: pgmap v20899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:44.358 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:44 smithi153 ceph-mon[36569]: pgmap v20900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:44.358 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:10:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:44 smithi081 ceph-mon[31453]: pgmap v20900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:10:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:44 smithi104 ceph-mon[36624]: pgmap v20900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: pgmap v20901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: pgmap v20902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:45.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:10:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:10:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: pgmap v20901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: pgmap v20902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:45.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:10:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:10:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: pgmap v20901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: pgmap v20902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:45.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:10:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:47 smithi153 ceph-mon[36569]: pgmap v20903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:47 smithi104 ceph-mon[36624]: pgmap v20903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:47 smithi081 ceph-mon[31453]: pgmap v20903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:49 smithi153 ceph-mon[36569]: pgmap v20904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:49 smithi104 ceph-mon[36624]: pgmap v20904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:49 smithi081 ceph-mon[31453]: pgmap v20904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:51 smithi104 ceph-mon[36624]: pgmap v20905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:51 smithi081 ceph-mon[31453]: pgmap v20905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:52.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:51 smithi153 ceph-mon[36569]: pgmap v20905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:52.614+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:10:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:52.614+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:10:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:10:52.614+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:10:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:53 smithi104 ceph-mon[36624]: pgmap v20906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:53 smithi081 ceph-mon[31453]: pgmap v20906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:53 smithi153 ceph-mon[36569]: pgmap v20906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:55 smithi104 ceph-mon[36624]: pgmap v20907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:55 smithi081 ceph-mon[31453]: pgmap v20907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:55 smithi153 ceph-mon[36569]: pgmap v20907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:57 smithi104 ceph-mon[36624]: pgmap v20908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:57 smithi081 ceph-mon[31453]: pgmap v20908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:57 smithi153 ceph-mon[36569]: pgmap v20908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:10:59 smithi104 ceph-mon[36624]: pgmap v20909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:10:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:10:59 smithi081 ceph-mon[31453]: pgmap v20909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:10:59 smithi153 ceph-mon[36569]: pgmap v20909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:01 smithi081 ceph-mon[31453]: pgmap v20910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:01 smithi153 ceph-mon[36569]: pgmap v20910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:01 smithi104 ceph-mon[36624]: pgmap v20910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:03 smithi081 ceph-mon[31453]: pgmap v20911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:03 smithi153 ceph-mon[36569]: pgmap v20911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:03 smithi104 ceph-mon[36624]: pgmap v20911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:05 smithi081 ceph-mon[31453]: pgmap v20912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:05 smithi153 ceph-mon[36569]: pgmap v20912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:05 smithi104 ceph-mon[36624]: pgmap v20912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:07.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:07.629+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:11:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:07.629+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:11:07.630 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:07.629+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:11:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:07 smithi081 ceph-mon[31453]: pgmap v20913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:07 smithi153 ceph-mon[36569]: pgmap v20913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:07 smithi104 ceph-mon[36624]: pgmap v20913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:09 smithi081 ceph-mon[31453]: pgmap v20914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:09 smithi153 ceph-mon[36569]: pgmap v20914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:09 smithi104 ceph-mon[36624]: pgmap v20914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:11 smithi081 ceph-mon[31453]: pgmap v20915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:11 smithi153 ceph-mon[36569]: pgmap v20915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:11 smithi104 ceph-mon[36624]: pgmap v20915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:13 smithi081 ceph-mon[31453]: pgmap v20916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:13 smithi153 ceph-mon[36569]: pgmap v20916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:13 smithi104 ceph-mon[36624]: pgmap v20916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:15 smithi081 ceph-mon[31453]: pgmap v20917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:15 smithi153 ceph-mon[36569]: pgmap v20917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:15 smithi104 ceph-mon[36624]: pgmap v20917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:17 smithi081 ceph-mon[31453]: pgmap v20918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:17 smithi153 ceph-mon[36569]: pgmap v20918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:17 smithi104 ceph-mon[36624]: pgmap v20918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:19 smithi081 ceph-mon[31453]: pgmap v20919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:19 smithi153 ceph-mon[36569]: pgmap v20919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:19 smithi104 ceph-mon[36624]: pgmap v20919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:21 smithi081 ceph-mon[31453]: pgmap v20920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:21 smithi153 ceph-mon[36569]: pgmap v20920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:21 smithi104 ceph-mon[36624]: pgmap v20920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:22.637 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:22.637+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:11:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:22.637+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:11:22.638 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:22.637+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:11:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:23 smithi081 ceph-mon[31453]: pgmap v20921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:23 smithi153 ceph-mon[36569]: pgmap v20921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:23 smithi104 ceph-mon[36624]: pgmap v20921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:25 smithi081 ceph-mon[31453]: pgmap v20922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:25 smithi153 ceph-mon[36569]: pgmap v20922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:25 smithi104 ceph-mon[36624]: pgmap v20922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:27 smithi081 ceph-mon[31453]: pgmap v20923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:27 smithi153 ceph-mon[36569]: pgmap v20923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:27 smithi104 ceph-mon[36624]: pgmap v20923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:29 smithi153 ceph-mon[36569]: pgmap v20924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:29 smithi104 ceph-mon[36624]: pgmap v20924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:29 smithi081 ceph-mon[31453]: pgmap v20924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:32.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:31 smithi153 ceph-mon[36569]: pgmap v20925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:31 smithi104 ceph-mon[36624]: pgmap v20925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:31 smithi081 ceph-mon[31453]: pgmap v20925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:33 smithi153 ceph-mon[36569]: pgmap v20926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:33 smithi104 ceph-mon[36624]: pgmap v20926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:33 smithi081 ceph-mon[31453]: pgmap v20926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:35 smithi153 ceph-mon[36569]: pgmap v20927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:35 smithi104 ceph-mon[36624]: pgmap v20927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:35 smithi081 ceph-mon[31453]: pgmap v20927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:37.651+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:11:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:37.651+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:11:37.651 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:37.651+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:11:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:37 smithi153 ceph-mon[36569]: pgmap v20928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:37 smithi104 ceph-mon[36624]: pgmap v20928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:37 smithi081 ceph-mon[31453]: pgmap v20928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:39 smithi153 ceph-mon[36569]: pgmap v20929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:39 smithi104 ceph-mon[36624]: pgmap v20929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:39 smithi081 ceph-mon[31453]: pgmap v20929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:40 smithi153 ceph-mon[36569]: pgmap v20930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:40 smithi104 ceph-mon[36624]: pgmap v20930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:40 smithi081 ceph-mon[31453]: pgmap v20930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:43 smithi104 ceph-mon[36624]: pgmap v20931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:43 smithi081 ceph-mon[31453]: pgmap v20931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:44.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:43 smithi153 ceph-mon[36569]: pgmap v20931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:45.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:45 smithi153 ceph-mon[36569]: pgmap v20932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:45.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:11:45.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:45.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:45 smithi081 ceph-mon[31453]: pgmap v20932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:11:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:45 smithi104 ceph-mon[36624]: pgmap v20932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:11:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:47 smithi081 ceph-mon[31453]: pgmap v20933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:47 smithi153 ceph-mon[36569]: pgmap v20933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:48.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:47 smithi104 ceph-mon[36624]: pgmap v20933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:49.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:48 smithi153 ceph-mon[36569]: pgmap v20934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:48 smithi104 ceph-mon[36624]: pgmap v20934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:48 smithi081 ceph-mon[31453]: pgmap v20934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:51.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:51 smithi153 ceph-mon[36569]: pgmap v20935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:51 smithi104 ceph-mon[36624]: pgmap v20935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:51 smithi081 ceph-mon[31453]: pgmap v20935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:11:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:11:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:52 smithi081 ceph-mon[31453]: pgmap v20936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:52 smithi081 ceph-mon[31453]: pgmap v20937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:52.657 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:52.659+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:11:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:52.659+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:11:52.660 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:11:52.659+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:11:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:11:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:11:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:52 smithi153 ceph-mon[36569]: pgmap v20936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:52 smithi153 ceph-mon[36569]: pgmap v20937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:52.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:11:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:11:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:52 smithi104 ceph-mon[36624]: pgmap v20936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:52 smithi104 ceph-mon[36624]: pgmap v20937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:11:55.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:54 smithi153 ceph-mon[36569]: pgmap v20938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:54 smithi104 ceph-mon[36624]: pgmap v20938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:54 smithi081 ceph-mon[31453]: pgmap v20938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:56 smithi153 ceph-mon[36569]: pgmap v20939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:56 smithi104 ceph-mon[36624]: pgmap v20939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:56 smithi081 ceph-mon[31453]: pgmap v20939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:11:58 smithi153 ceph-mon[36569]: pgmap v20940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:11:58 smithi104 ceph-mon[36624]: pgmap v20940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:11:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:11:58 smithi081 ceph-mon[31453]: pgmap v20940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:00 smithi153 ceph-mon[36569]: pgmap v20941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:00 smithi104 ceph-mon[36624]: pgmap v20941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:00 smithi081 ceph-mon[31453]: pgmap v20941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:02 smithi153 ceph-mon[36569]: pgmap v20942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:02 smithi104 ceph-mon[36624]: pgmap v20942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:02 smithi081 ceph-mon[31453]: pgmap v20942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:05.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:04 smithi153 ceph-mon[36569]: pgmap v20943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:04 smithi104 ceph-mon[36624]: pgmap v20943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:04 smithi081 ceph-mon[31453]: pgmap v20943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:06 smithi153 ceph-mon[36569]: pgmap v20944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:06 smithi104 ceph-mon[36624]: pgmap v20944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:06 smithi081 ceph-mon[31453]: pgmap v20944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:07.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:07.670+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:12:07.670 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:07.670+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:12:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:07.670+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:12:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:07 smithi153 ceph-mon[36569]: pgmap v20945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:07 smithi104 ceph-mon[36624]: pgmap v20945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:07 smithi081 ceph-mon[31453]: pgmap v20945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:10 smithi153 ceph-mon[36569]: pgmap v20946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:10 smithi104 ceph-mon[36624]: pgmap v20946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:11.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:10 smithi081 ceph-mon[31453]: pgmap v20946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:12 smithi153 ceph-mon[36569]: pgmap v20947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:12 smithi104 ceph-mon[36624]: pgmap v20947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:13.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:12 smithi081 ceph-mon[31453]: pgmap v20947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:14 smithi153 ceph-mon[36569]: pgmap v20948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:14 smithi104 ceph-mon[36624]: pgmap v20948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:14 smithi081 ceph-mon[31453]: pgmap v20948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:16 smithi153 ceph-mon[36569]: pgmap v20949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:16 smithi104 ceph-mon[36624]: pgmap v20949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:16 smithi081 ceph-mon[31453]: pgmap v20949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:18 smithi153 ceph-mon[36569]: pgmap v20950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:18 smithi104 ceph-mon[36624]: pgmap v20950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:18 smithi081 ceph-mon[31453]: pgmap v20950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:21.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:20 smithi153 ceph-mon[36569]: pgmap v20951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:20 smithi104 ceph-mon[36624]: pgmap v20951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:20 smithi081 ceph-mon[31453]: pgmap v20951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:21 smithi153 ceph-mon[36569]: pgmap v20952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:22.058 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:21 smithi081 ceph-mon[31453]: pgmap v20952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:21 smithi104 ceph-mon[36624]: pgmap v20952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:22.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:22.681+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:12:22.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:22.681+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:12:22.681 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:22.681+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:12:25.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:24 smithi153 ceph-mon[36569]: pgmap v20953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:24 smithi104 ceph-mon[36624]: pgmap v20953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:25.095 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:24 smithi081 ceph-mon[31453]: pgmap v20953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:26 smithi153 ceph-mon[36569]: pgmap v20954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:26 smithi104 ceph-mon[36624]: pgmap v20954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:26 smithi081 ceph-mon[31453]: pgmap v20954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:28 smithi153 ceph-mon[36569]: pgmap v20955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:28 smithi104 ceph-mon[36624]: pgmap v20955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:28 smithi081 ceph-mon[31453]: pgmap v20955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:30 smithi153 ceph-mon[36569]: pgmap v20956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:30 smithi104 ceph-mon[36624]: pgmap v20956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:30 smithi081 ceph-mon[31453]: pgmap v20956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:32 smithi153 ceph-mon[36569]: pgmap v20957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:32 smithi104 ceph-mon[36624]: pgmap v20957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:32 smithi081 ceph-mon[31453]: pgmap v20957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:33 smithi153 ceph-mon[36569]: pgmap v20958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:33 smithi104 ceph-mon[36624]: pgmap v20958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:33 smithi081 ceph-mon[31453]: pgmap v20958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:36 smithi153 ceph-mon[36569]: pgmap v20959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:36 smithi104 ceph-mon[36624]: pgmap v20959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:36 smithi081 ceph-mon[31453]: pgmap v20959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:37.696+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:12:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:37.696+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:12:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:37.696+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:12:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:38 smithi153 ceph-mon[36569]: pgmap v20960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:38 smithi104 ceph-mon[36624]: pgmap v20960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:38 smithi081 ceph-mon[31453]: pgmap v20960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:40.967 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:40 smithi081 ceph-mon[31453]: pgmap v20961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:40 smithi153 ceph-mon[36569]: pgmap v20961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:40 smithi104 ceph-mon[36624]: pgmap v20961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:42 smithi153 ceph-mon[36569]: pgmap v20962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:42 smithi104 ceph-mon[36624]: pgmap v20962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:42 smithi081 ceph-mon[31453]: pgmap v20962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:43 smithi153 ceph-mon[36569]: pgmap v20963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:43 smithi104 ceph-mon[36624]: pgmap v20963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:43 smithi081 ceph-mon[31453]: pgmap v20963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:46 smithi153 ceph-mon[36569]: pgmap v20964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:46 smithi104 ceph-mon[36624]: pgmap v20964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:46 smithi081 ceph-mon[31453]: pgmap v20964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:48 smithi153 ceph-mon[36569]: pgmap v20965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:48 smithi104 ceph-mon[36624]: pgmap v20965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:48 smithi081 ceph-mon[31453]: pgmap v20965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:50 smithi153 ceph-mon[36569]: pgmap v20966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:50 smithi104 ceph-mon[36624]: pgmap v20966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:50 smithi081 ceph-mon[31453]: pgmap v20966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:52.710+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:12:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:52.710+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:12:52.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:12:52.710+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:12:52.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:52 smithi081 ceph-mon[31453]: pgmap v20967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:52.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:12:52.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:52.960 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:52 smithi153 ceph-mon[36569]: pgmap v20967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:12:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:52 smithi104 ceph-mon[36624]: pgmap v20967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:12:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:12:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:12:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:53 smithi153 ceph-mon[36569]: pgmap v20968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:53 smithi153 ceph-mon[36569]: pgmap v20969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:12:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:12:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:53 smithi104 ceph-mon[36624]: pgmap v20968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:53 smithi104 ceph-mon[36624]: pgmap v20969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:12:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:12:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:53 smithi081 ceph-mon[31453]: pgmap v20968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:53 smithi081 ceph-mon[31453]: pgmap v20969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:12:56.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:56 smithi153 ceph-mon[36569]: pgmap v20970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:56.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:56 smithi104 ceph-mon[36624]: pgmap v20970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:56 smithi081 ceph-mon[31453]: pgmap v20970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:12:58 smithi153 ceph-mon[36569]: pgmap v20971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:12:58 smithi104 ceph-mon[36624]: pgmap v20971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:12:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:12:58 smithi081 ceph-mon[31453]: pgmap v20971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:00.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:00 smithi153 ceph-mon[36569]: pgmap v20972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:00.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:00 smithi104 ceph-mon[36624]: pgmap v20972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:00.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:00 smithi081 ceph-mon[31453]: pgmap v20972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:02 smithi153 ceph-mon[36569]: pgmap v20973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:02.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:02 smithi104 ceph-mon[36624]: pgmap v20973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:02 smithi081 ceph-mon[31453]: pgmap v20973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:04 smithi153 ceph-mon[36569]: pgmap v20974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:04.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:04 smithi104 ceph-mon[36624]: pgmap v20974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:04 smithi081 ceph-mon[31453]: pgmap v20974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:06 smithi153 ceph-mon[36569]: pgmap v20975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:06 smithi104 ceph-mon[36624]: pgmap v20975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:06 smithi081 ceph-mon[31453]: pgmap v20975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:07.727+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:13:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:07.727+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:13:07.727 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:07.727+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:13:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:08 smithi153 ceph-mon[36569]: pgmap v20976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:08 smithi104 ceph-mon[36624]: pgmap v20976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:08 smithi081 ceph-mon[31453]: pgmap v20976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:10 smithi153 ceph-mon[36569]: pgmap v20977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:10 smithi104 ceph-mon[36624]: pgmap v20977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:10 smithi081 ceph-mon[31453]: pgmap v20977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:12 smithi153 ceph-mon[36569]: pgmap v20978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:12 smithi104 ceph-mon[36624]: pgmap v20978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:12 smithi081 ceph-mon[31453]: pgmap v20978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:14 smithi153 ceph-mon[36569]: pgmap v20979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:14.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:14 smithi104 ceph-mon[36624]: pgmap v20979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:14.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:14 smithi081 ceph-mon[31453]: pgmap v20979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:16 smithi153 ceph-mon[36569]: pgmap v20980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:16.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:16 smithi104 ceph-mon[36624]: pgmap v20980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:16 smithi081 ceph-mon[31453]: pgmap v20980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:18 smithi153 ceph-mon[36569]: pgmap v20981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:18.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:18 smithi104 ceph-mon[36624]: pgmap v20981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:18.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:18 smithi081 ceph-mon[31453]: pgmap v20981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:19.229+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:13:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:19.229+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:13:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:19.229+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 303 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:13:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:19.229+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 303 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:13:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:20 smithi153 ceph-mon[36569]: pgmap v20982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:20.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:20 smithi104 ceph-mon[36624]: pgmap v20982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:20.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:20 smithi081 ceph-mon[31453]: pgmap v20982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:22.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:22 smithi153 ceph-mon[36569]: pgmap v20983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:22.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:22 smithi104 ceph-mon[36624]: pgmap v20983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:22 smithi081 ceph-mon[31453]: pgmap v20983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:22.734+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:13:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:22.734+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:13:22.734 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:22.734+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:13:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:24 smithi153 ceph-mon[36569]: pgmap v20984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:24 smithi104 ceph-mon[36624]: pgmap v20984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:24.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:24 smithi081 ceph-mon[31453]: pgmap v20984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:26 smithi153 ceph-mon[36569]: pgmap v20985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:26 smithi104 ceph-mon[36624]: pgmap v20985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:26.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:26 smithi081 ceph-mon[31453]: pgmap v20985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:28 smithi081 ceph-mon[31453]: pgmap v20986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:28 smithi153 ceph-mon[36569]: pgmap v20986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:28.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:28 smithi104 ceph-mon[36624]: pgmap v20986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:30 smithi153 ceph-mon[36569]: pgmap v20987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:30.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:30 smithi104 ceph-mon[36624]: pgmap v20987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:30.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:30 smithi081 ceph-mon[31453]: pgmap v20987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:32.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:32 smithi153 ceph-mon[36569]: pgmap v20988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:32.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:32 smithi104 ceph-mon[36624]: pgmap v20988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:32.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:32 smithi081 ceph-mon[31453]: pgmap v20988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:34 smithi153 ceph-mon[36569]: pgmap v20989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:34 smithi104 ceph-mon[36624]: pgmap v20989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:34.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:34 smithi081 ceph-mon[31453]: pgmap v20989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:36 smithi153 ceph-mon[36569]: pgmap v20990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:36 smithi104 ceph-mon[36624]: pgmap v20990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:36.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:36 smithi081 ceph-mon[31453]: pgmap v20990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:37.737+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:13:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:37.737+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:13:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:37.737+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:13:38.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:38 smithi104 ceph-mon[36624]: pgmap v20991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:38.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:38 smithi081 ceph-mon[31453]: pgmap v20991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:38 smithi153 ceph-mon[36569]: pgmap v20991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:40 smithi104 ceph-mon[36624]: pgmap v20992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:40 smithi081 ceph-mon[31453]: pgmap v20992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:40 smithi153 ceph-mon[36569]: pgmap v20992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:42 smithi104 ceph-mon[36624]: pgmap v20993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:42.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:42 smithi081 ceph-mon[31453]: pgmap v20993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:42 smithi153 ceph-mon[36569]: pgmap v20993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:44 smithi104 ceph-mon[36624]: pgmap v20994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:44.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:44 smithi081 ceph-mon[31453]: pgmap v20994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:44.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:44 smithi153 ceph-mon[36569]: pgmap v20994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:46 smithi104 ceph-mon[36624]: pgmap v20995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:46.609 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:46 smithi153 ceph-mon[36569]: pgmap v20995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:46.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:46 smithi081 ceph-mon[31453]: pgmap v20995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:48.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:48 smithi081 ceph-mon[31453]: pgmap v20996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:48 smithi153 ceph-mon[36569]: pgmap v20996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:48 smithi104 ceph-mon[36624]: pgmap v20996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:50.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:50 smithi081 ceph-mon[31453]: pgmap v20997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:50.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:50 smithi153 ceph-mon[36569]: pgmap v20997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:50 smithi104 ceph-mon[36624]: pgmap v20997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:52.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:52 smithi081 ceph-mon[31453]: pgmap v20998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:52.748+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:13:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:52.749+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:13:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:13:52.749+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:13:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:52 smithi153 ceph-mon[36569]: pgmap v20998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:52 smithi104 ceph-mon[36624]: pgmap v20998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:54 smithi081 ceph-mon[31453]: pgmap v20999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:54.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:54 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:13:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:54 smithi153 ceph-mon[36569]: pgmap v20999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:54 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:13:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:54 smithi104 ceph-mon[36624]: pgmap v20999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:54 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:13:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:13:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:13:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:13:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:13:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:13:55.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:13:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:13:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:13:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:13:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:56 smithi081 ceph-mon[31453]: pgmap v21000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:56.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:56 smithi081 ceph-mon[31453]: pgmap v21001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:56 smithi153 ceph-mon[36569]: pgmap v21000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:56 smithi153 ceph-mon[36569]: pgmap v21001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:56 smithi104 ceph-mon[36624]: pgmap v21000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:56 smithi104 ceph-mon[36624]: pgmap v21001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:58.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:13:58 smithi081 ceph-mon[31453]: pgmap v21002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:13:58 smithi153 ceph-mon[36569]: pgmap v21002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:13:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:13:58 smithi104 ceph-mon[36624]: pgmap v21002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:00.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:00 smithi104 ceph-mon[36624]: pgmap v21003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:00.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:00 smithi081 ceph-mon[31453]: pgmap v21003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:00 smithi153 ceph-mon[36569]: pgmap v21003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:02.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:02 smithi081 ceph-mon[31453]: pgmap v21004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:02 smithi153 ceph-mon[36569]: pgmap v21004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:02.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:02 smithi104 ceph-mon[36624]: pgmap v21004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:04.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:04 smithi081 ceph-mon[31453]: pgmap v21005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:04 smithi153 ceph-mon[36569]: pgmap v21005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:04.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:04 smithi104 ceph-mon[36624]: pgmap v21005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:06 smithi081 ceph-mon[31453]: pgmap v21006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:06 smithi153 ceph-mon[36569]: pgmap v21006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:06 smithi104 ceph-mon[36624]: pgmap v21006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:07.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:07.763+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:14:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:07.763+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:14:07.764 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:07.763+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:14:08.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:08 smithi081 ceph-mon[31453]: pgmap v21007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:08 smithi153 ceph-mon[36569]: pgmap v21007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:08 smithi104 ceph-mon[36624]: pgmap v21007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:10 smithi081 ceph-mon[31453]: pgmap v21008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:10 smithi153 ceph-mon[36569]: pgmap v21008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:10 smithi104 ceph-mon[36624]: pgmap v21008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:12.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:12 smithi081 ceph-mon[31453]: pgmap v21009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:12 smithi153 ceph-mon[36569]: pgmap v21009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:12 smithi104 ceph-mon[36624]: pgmap v21009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:14 smithi153 ceph-mon[36569]: pgmap v21010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:14 smithi104 ceph-mon[36624]: pgmap v21010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:14 smithi081 ceph-mon[31453]: pgmap v21010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:16 smithi153 ceph-mon[36569]: pgmap v21011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:16 smithi104 ceph-mon[36624]: pgmap v21011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:16 smithi081 ceph-mon[31453]: pgmap v21011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:18 smithi153 ceph-mon[36569]: pgmap v21012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:18 smithi104 ceph-mon[36624]: pgmap v21012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:18 smithi081 ceph-mon[31453]: pgmap v21012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:20 smithi153 ceph-mon[36569]: pgmap v21013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:20 smithi104 ceph-mon[36624]: pgmap v21013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:20 smithi081 ceph-mon[31453]: pgmap v21013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:22.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:22 smithi081 ceph-mon[31453]: pgmap v21014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:22.772 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:22.772+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:14:22.773 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:22.772+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:14:22.773 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:22.772+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:14:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:22 smithi153 ceph-mon[36569]: pgmap v21014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:22 smithi104 ceph-mon[36624]: pgmap v21014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:24 smithi153 ceph-mon[36569]: pgmap v21015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:24 smithi104 ceph-mon[36624]: pgmap v21015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:24 smithi081 ceph-mon[31453]: pgmap v21015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:26 smithi153 ceph-mon[36569]: pgmap v21016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:26 smithi104 ceph-mon[36624]: pgmap v21016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:26 smithi081 ceph-mon[31453]: pgmap v21016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:28 smithi153 ceph-mon[36569]: pgmap v21017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:28 smithi104 ceph-mon[36624]: pgmap v21017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:28 smithi081 ceph-mon[31453]: pgmap v21017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:30 smithi153 ceph-mon[36569]: pgmap v21018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:30 smithi104 ceph-mon[36624]: pgmap v21018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:30 smithi081 ceph-mon[31453]: pgmap v21018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:32 smithi153 ceph-mon[36569]: pgmap v21019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:32 smithi104 ceph-mon[36624]: pgmap v21019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:32 smithi081 ceph-mon[31453]: pgmap v21019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:34 smithi153 ceph-mon[36569]: pgmap v21020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:34 smithi104 ceph-mon[36624]: pgmap v21020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:34 smithi081 ceph-mon[31453]: pgmap v21020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:36 smithi153 ceph-mon[36569]: pgmap v21021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:36 smithi104 ceph-mon[36624]: pgmap v21021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:36 smithi081 ceph-mon[31453]: pgmap v21021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:37.773+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:14:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:37.773+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:14:37.774 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:37.773+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:14:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:38 smithi153 ceph-mon[36569]: pgmap v21022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:38 smithi104 ceph-mon[36624]: pgmap v21022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:38 smithi081 ceph-mon[31453]: pgmap v21022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:40 smithi153 ceph-mon[36569]: pgmap v21023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:40 smithi104 ceph-mon[36624]: pgmap v21023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:40 smithi081 ceph-mon[31453]: pgmap v21023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:42 smithi153 ceph-mon[36569]: pgmap v21024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:42 smithi104 ceph-mon[36624]: pgmap v21024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:42 smithi081 ceph-mon[31453]: pgmap v21024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:44 smithi153 ceph-mon[36569]: pgmap v21025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:44 smithi104 ceph-mon[36624]: pgmap v21025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:44.871 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:44 smithi081 ceph-mon[31453]: pgmap v21025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:46 smithi153 ceph-mon[36569]: pgmap v21026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:46 smithi104 ceph-mon[36624]: pgmap v21026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:46 smithi081 ceph-mon[31453]: pgmap v21026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:48 smithi153 ceph-mon[36569]: pgmap v21027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:48 smithi104 ceph-mon[36624]: pgmap v21027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:48 smithi081 ceph-mon[31453]: pgmap v21027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:50 smithi153 ceph-mon[36569]: pgmap v21028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:50.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:50 smithi104 ceph-mon[36624]: pgmap v21028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:50 smithi081 ceph-mon[31453]: pgmap v21028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:52.786+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:14:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:52.786+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:14:52.786 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:14:52.786+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:14:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:52 smithi153 ceph-mon[36569]: pgmap v21029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:52.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:52 smithi104 ceph-mon[36624]: pgmap v21029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:52 smithi081 ceph-mon[31453]: pgmap v21029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:54.810 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:54 smithi153 ceph-mon[36569]: pgmap v21030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:54 smithi104 ceph-mon[36624]: pgmap v21030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:54 smithi081 ceph-mon[31453]: pgmap v21030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:55 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:14:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:55 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:14:55.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:55 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:14:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:56 smithi153 ceph-mon[36569]: pgmap v21031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:14:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:14:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:14:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:56 smithi104 ceph-mon[36624]: pgmap v21031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:14:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:14:56.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:14:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:56 smithi081 ceph-mon[31453]: pgmap v21031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:14:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:14:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:14:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:57 smithi153 ceph-mon[36569]: pgmap v21032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:57 smithi153 ceph-mon[36569]: pgmap v21033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:57 smithi104 ceph-mon[36624]: pgmap v21032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:57 smithi104 ceph-mon[36624]: pgmap v21033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:57 smithi081 ceph-mon[31453]: pgmap v21032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:57 smithi081 ceph-mon[31453]: pgmap v21033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:59.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:14:59 smithi153 ceph-mon[36569]: pgmap v21034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:14:59 smithi104 ceph-mon[36624]: pgmap v21034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:14:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:14:59 smithi081 ceph-mon[31453]: pgmap v21034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:01 smithi153 ceph-mon[36569]: pgmap v21035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:01 smithi104 ceph-mon[36624]: pgmap v21035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:01 smithi081 ceph-mon[31453]: pgmap v21035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:03 smithi153 ceph-mon[36569]: pgmap v21036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:03 smithi104 ceph-mon[36624]: pgmap v21036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:03 smithi081 ceph-mon[31453]: pgmap v21036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:05 smithi153 ceph-mon[36569]: pgmap v21037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:05 smithi104 ceph-mon[36624]: pgmap v21037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:05 smithi081 ceph-mon[31453]: pgmap v21037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:07.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:07.794+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:15:07.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:07.794+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:15:07.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:07.794+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:15:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:07 smithi104 ceph-mon[36624]: pgmap v21038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:07 smithi081 ceph-mon[31453]: pgmap v21038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:07 smithi153 ceph-mon[36569]: pgmap v21038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:09 smithi104 ceph-mon[36624]: pgmap v21039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:09 smithi081 ceph-mon[31453]: pgmap v21039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:09 smithi153 ceph-mon[36569]: pgmap v21039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:11 smithi104 ceph-mon[36624]: pgmap v21040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:11 smithi081 ceph-mon[31453]: pgmap v21040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:11 smithi153 ceph-mon[36569]: pgmap v21040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:13 smithi104 ceph-mon[36624]: pgmap v21041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:13 smithi081 ceph-mon[31453]: pgmap v21041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:13 smithi153 ceph-mon[36569]: pgmap v21041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:15 smithi104 ceph-mon[36624]: pgmap v21042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:15 smithi081 ceph-mon[31453]: pgmap v21042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:15 smithi153 ceph-mon[36569]: pgmap v21042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:17.859 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:17 smithi153 ceph-mon[36569]: pgmap v21043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:17 smithi081 ceph-mon[31453]: pgmap v21043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:17 smithi104 ceph-mon[36624]: pgmap v21043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:19 smithi081 ceph-mon[31453]: pgmap v21044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:19 smithi153 ceph-mon[36569]: pgmap v21044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:19 smithi104 ceph-mon[36624]: pgmap v21044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:21 smithi081 ceph-mon[31453]: pgmap v21045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:21 smithi153 ceph-mon[36569]: pgmap v21045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:21 smithi104 ceph-mon[36624]: pgmap v21045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:22.800+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:15:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:22.800+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:15:22.800 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:22.800+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:15:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:23 smithi081 ceph-mon[31453]: pgmap v21046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:23 smithi153 ceph-mon[36569]: pgmap v21046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:23 smithi104 ceph-mon[36624]: pgmap v21046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:25 smithi081 ceph-mon[31453]: pgmap v21047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:25 smithi153 ceph-mon[36569]: pgmap v21047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:25 smithi104 ceph-mon[36624]: pgmap v21047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:27 smithi081 ceph-mon[31453]: pgmap v21048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:27 smithi153 ceph-mon[36569]: pgmap v21048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:27 smithi104 ceph-mon[36624]: pgmap v21048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:29 smithi081 ceph-mon[31453]: pgmap v21049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:29 smithi153 ceph-mon[36569]: pgmap v21049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:29 smithi104 ceph-mon[36624]: pgmap v21049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:31.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:31 smithi104 ceph-mon[36624]: pgmap v21050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:31 smithi081 ceph-mon[31453]: pgmap v21050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:31 smithi153 ceph-mon[36569]: pgmap v21050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:33 smithi153 ceph-mon[36569]: pgmap v21051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:33 smithi104 ceph-mon[36624]: pgmap v21051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:33 smithi081 ceph-mon[31453]: pgmap v21051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:35 smithi081 ceph-mon[31453]: pgmap v21052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:35 smithi153 ceph-mon[36569]: pgmap v21052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:35 smithi104 ceph-mon[36624]: pgmap v21052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:37 smithi081 ceph-mon[31453]: pgmap v21053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:37 smithi153 ceph-mon[36569]: pgmap v21053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:37.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:37.812+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:15:37.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:37.812+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:15:37.812 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:37.812+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:15:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:37 smithi104 ceph-mon[36624]: pgmap v21053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:39.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:39 smithi081 ceph-mon[31453]: pgmap v21054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:39 smithi153 ceph-mon[36569]: pgmap v21054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:39 smithi104 ceph-mon[36624]: pgmap v21054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:41 smithi081 ceph-mon[31453]: pgmap v21055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:41 smithi153 ceph-mon[36569]: pgmap v21055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:41 smithi104 ceph-mon[36624]: pgmap v21055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:43 smithi081 ceph-mon[31453]: pgmap v21056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:43 smithi153 ceph-mon[36569]: pgmap v21056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:43 smithi104 ceph-mon[36624]: pgmap v21056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:45 smithi081 ceph-mon[31453]: pgmap v21057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:45 smithi153 ceph-mon[36569]: pgmap v21057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:45 smithi104 ceph-mon[36624]: pgmap v21057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:47 smithi081 ceph-mon[31453]: pgmap v21058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:47 smithi153 ceph-mon[36569]: pgmap v21058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:47 smithi104 ceph-mon[36624]: pgmap v21058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:49 smithi081 ceph-mon[31453]: pgmap v21059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:49 smithi153 ceph-mon[36569]: pgmap v21059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:49 smithi104 ceph-mon[36624]: pgmap v21059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:51 smithi081 ceph-mon[31453]: pgmap v21060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:51 smithi153 ceph-mon[36569]: pgmap v21060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:51 smithi104 ceph-mon[36624]: pgmap v21060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:52.827+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:15:52.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:52.827+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:15:52.828 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:15:52.827+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:15:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:53 smithi081 ceph-mon[31453]: pgmap v21061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:53 smithi153 ceph-mon[36569]: pgmap v21061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:53 smithi104 ceph-mon[36624]: pgmap v21061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:55 smithi081 ceph-mon[31453]: pgmap v21062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:55 smithi153 ceph-mon[36569]: pgmap v21062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:55 smithi104 ceph-mon[36624]: pgmap v21062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:57 smithi081 ceph-mon[31453]: pgmap v21063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:57 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:15:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:57 smithi153 ceph-mon[36569]: pgmap v21063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:57.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:57 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:15:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:57 smithi104 ceph-mon[36624]: pgmap v21063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:57 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:15:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:15:59 smithi104 ceph-mon[36624]: pgmap v21064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:15:59 smithi153 ceph-mon[36569]: pgmap v21064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:15:59.863 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:15:59 smithi081 ceph-mon[31453]: pgmap v21064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:01.800 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:01 smithi104 ceph-mon[36624]: pgmap v21065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:01 smithi153 ceph-mon[36569]: pgmap v21065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:01 smithi081 ceph-mon[31453]: pgmap v21065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:02 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:02 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:02 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:03 smithi081 ceph-mon[31453]: pgmap v21066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:03 smithi153 ceph-mon[36569]: pgmap v21066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:03 smithi104 ceph-mon[36624]: pgmap v21066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:16:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:16:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:04 smithi081 ceph-mon[31453]: pgmap v21067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:04 smithi081 ceph-mon[31453]: pgmap v21068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:05.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:04 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:16:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:16:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:04 smithi153 ceph-mon[36569]: pgmap v21067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:04 smithi153 ceph-mon[36569]: pgmap v21068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:05.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:04 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:16:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:16:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:04 smithi104 ceph-mon[36624]: pgmap v21067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:04 smithi104 ceph-mon[36624]: pgmap v21068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:04 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:16:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:06 smithi153 ceph-mon[36569]: pgmap v21069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:07.328 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:06 smithi081 ceph-mon[31453]: pgmap v21069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:06 smithi104 ceph-mon[36624]: pgmap v21069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:07.843+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:16:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:07.843+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:16:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:07.843+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:16:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:08 smithi153 ceph-mon[36569]: pgmap v21070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:08 smithi104 ceph-mon[36624]: pgmap v21070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:08 smithi081 ceph-mon[31453]: pgmap v21070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:10 smithi153 ceph-mon[36569]: pgmap v21071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:10 smithi104 ceph-mon[36624]: pgmap v21071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:11.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:10 smithi081 ceph-mon[31453]: pgmap v21071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:12.200+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 msgr2=0x7f16e80b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000611) for more than 900000000 us, fault. 2024-04-13T08:16:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:12.200+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f16f4098ec0 tx=0x7f16f40ba000 comp rx=0 tx=0).stop 2024-04-13T08:16:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:12.200+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 msgr2=0x7f16e80b3c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T08:16:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:12.200+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T08:16:12.200 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:12.200+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T08:16:12.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:12.201+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T08:16:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:12.202+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f40bd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T08:16:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:12 smithi104 ceph-mon[36624]: pgmap v21072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:12 smithi153 ceph-mon[36569]: pgmap v21072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:12 smithi081 ceph-mon[31453]: pgmap v21072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:14 smithi153 ceph-mon[36569]: pgmap v21073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:15.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:14 smithi104 ceph-mon[36624]: pgmap v21073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:15.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:14 smithi081 ceph-mon[31453]: pgmap v21073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:16 smithi081 ceph-mon[31453]: pgmap v21074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:16 smithi153 ceph-mon[36569]: pgmap v21074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:16 smithi104 ceph-mon[36624]: pgmap v21074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:19.258+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:16:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:19.258+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:16:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:19.258+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 304 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:16:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:19.259+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:16:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:19.259+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 304 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:16:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:19.259+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:16:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:19.259+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 305 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:16:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:19.259+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 305 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:16:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:18 smithi153 ceph-mon[36569]: pgmap v21075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:18 smithi104 ceph-mon[36624]: pgmap v21075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:19.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:18 smithi081 ceph-mon[31453]: pgmap v21075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:20 smithi153 ceph-mon[36569]: pgmap v21076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:20 smithi104 ceph-mon[36624]: pgmap v21076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:20 smithi081 ceph-mon[31453]: pgmap v21076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:22.853+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:16:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:22.853+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:16:22.854 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:22.853+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:16:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:22 smithi153 ceph-mon[36569]: pgmap v21077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:22 smithi104 ceph-mon[36624]: pgmap v21077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:22 smithi081 ceph-mon[31453]: pgmap v21077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:25 smithi081 ceph-mon[31453]: pgmap v21078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:25 smithi153 ceph-mon[36569]: pgmap v21078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:25 smithi104 ceph-mon[36624]: pgmap v21078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:27 smithi153 ceph-mon[36569]: pgmap v21079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:27 smithi104 ceph-mon[36624]: pgmap v21079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:27 smithi081 ceph-mon[31453]: pgmap v21079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:29 smithi153 ceph-mon[36569]: pgmap v21080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:29 smithi104 ceph-mon[36624]: pgmap v21080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:29 smithi081 ceph-mon[31453]: pgmap v21080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:31 smithi153 ceph-mon[36569]: pgmap v21081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:31.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:31 smithi104 ceph-mon[36624]: pgmap v21081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:31.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:31 smithi081 ceph-mon[31453]: pgmap v21081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:33 smithi153 ceph-mon[36569]: pgmap v21082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:33 smithi104 ceph-mon[36624]: pgmap v21082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:33 smithi081 ceph-mon[31453]: pgmap v21082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:35 smithi153 ceph-mon[36569]: pgmap v21083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:35 smithi104 ceph-mon[36624]: pgmap v21083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:35 smithi081 ceph-mon[31453]: pgmap v21083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:37 smithi153 ceph-mon[36569]: pgmap v21084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:37 smithi104 ceph-mon[36624]: pgmap v21084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:37 smithi081 ceph-mon[31453]: pgmap v21084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:37.857+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:16:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:37.857+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:16:37.857 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:37.857+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:16:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:39 smithi153 ceph-mon[36569]: pgmap v21085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:39.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:39 smithi104 ceph-mon[36624]: pgmap v21085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:39.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:39 smithi081 ceph-mon[31453]: pgmap v21085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:41 smithi153 ceph-mon[36569]: pgmap v21086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:41 smithi104 ceph-mon[36624]: pgmap v21086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:41 smithi081 ceph-mon[31453]: pgmap v21086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:43.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:43 smithi104 ceph-mon[36624]: pgmap v21087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:43.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:43 smithi081 ceph-mon[31453]: pgmap v21087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:43 smithi153 ceph-mon[36569]: pgmap v21087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:45 smithi104 ceph-mon[36624]: pgmap v21088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:45.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:45 smithi081 ceph-mon[31453]: pgmap v21088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:45 smithi153 ceph-mon[36569]: pgmap v21088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:47 smithi104 ceph-mon[36624]: pgmap v21089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:47.358 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:47 smithi153 ceph-mon[36569]: pgmap v21089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:47 smithi081 ceph-mon[31453]: pgmap v21089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:49 smithi104 ceph-mon[36624]: pgmap v21090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:49 smithi081 ceph-mon[31453]: pgmap v21090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:49 smithi153 ceph-mon[36569]: pgmap v21090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:51.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:51 smithi104 ceph-mon[36624]: pgmap v21091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:51.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:51 smithi081 ceph-mon[31453]: pgmap v21091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:51 smithi153 ceph-mon[36569]: pgmap v21091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:52.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:52.872+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:16:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:52.872+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:16:52.873 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:16:52.872+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:16:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:53 smithi081 ceph-mon[31453]: pgmap v21092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:53 smithi153 ceph-mon[36569]: pgmap v21092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:53 smithi104 ceph-mon[36624]: pgmap v21092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:55 smithi081 ceph-mon[31453]: pgmap v21093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:55 smithi153 ceph-mon[36569]: pgmap v21093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:55 smithi104 ceph-mon[36624]: pgmap v21093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:57 smithi081 ceph-mon[31453]: pgmap v21094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:57 smithi153 ceph-mon[36569]: pgmap v21094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:57 smithi104 ceph-mon[36624]: pgmap v21094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:59.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:16:59 smithi081 ceph-mon[31453]: pgmap v21095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:16:59 smithi153 ceph-mon[36569]: pgmap v21095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:16:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:16:59 smithi104 ceph-mon[36624]: pgmap v21095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:01.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:01 smithi104 ceph-mon[36624]: pgmap v21096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:01.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:01 smithi081 ceph-mon[31453]: pgmap v21096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:01 smithi153 ceph-mon[36569]: pgmap v21096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:03.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:03 smithi081 ceph-mon[31453]: pgmap v21097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:03 smithi153 ceph-mon[36569]: pgmap v21097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:03 smithi104 ceph-mon[36624]: pgmap v21097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:05 smithi081 ceph-mon[31453]: pgmap v21098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:17:05.515 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:05 smithi153 ceph-mon[36569]: pgmap v21098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:05.515 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:17:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:05 smithi104 ceph-mon[36624]: pgmap v21098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:17:07.400 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:07 smithi153 ceph-mon[36569]: pgmap v21099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:07 smithi081 ceph-mon[31453]: pgmap v21099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:07 smithi104 ceph-mon[36624]: pgmap v21099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:07.882+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:17:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:07.882+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:17:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:07.882+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:17:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:09 smithi153 ceph-mon[36569]: pgmap v21100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:09 smithi081 ceph-mon[31453]: pgmap v21100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:09 smithi104 ceph-mon[36624]: pgmap v21100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:10.654 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:10.655 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:10.655 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:17:10.655 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:17:10.655 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:10 smithi153 ceph-mon[36569]: pgmap v21101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:10.655 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:10 smithi153 ceph-mon[36569]: pgmap v21102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:10.655 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:17:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:17:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:10 smithi081 ceph-mon[31453]: pgmap v21101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:10 smithi081 ceph-mon[31453]: pgmap v21102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:10.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:17:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:17:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:10 smithi104 ceph-mon[36624]: pgmap v21101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:10 smithi104 ceph-mon[36624]: pgmap v21102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:17:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:12 smithi153 ceph-mon[36569]: pgmap v21103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:12 smithi104 ceph-mon[36624]: pgmap v21103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:12 smithi081 ceph-mon[31453]: pgmap v21103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:14 smithi153 ceph-mon[36569]: pgmap v21104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:14 smithi104 ceph-mon[36624]: pgmap v21104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:14 smithi081 ceph-mon[31453]: pgmap v21104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:16 smithi153 ceph-mon[36569]: pgmap v21105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:16 smithi104 ceph-mon[36624]: pgmap v21105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:16 smithi081 ceph-mon[31453]: pgmap v21105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:18 smithi153 ceph-mon[36569]: pgmap v21106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:18 smithi104 ceph-mon[36624]: pgmap v21106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:18 smithi081 ceph-mon[31453]: pgmap v21106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:20.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:20 smithi153 ceph-mon[36569]: pgmap v21107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:20 smithi104 ceph-mon[36624]: pgmap v21107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:20 smithi081 ceph-mon[31453]: pgmap v21107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:22 smithi153 ceph-mon[36569]: pgmap v21108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:22 smithi104 ceph-mon[36624]: pgmap v21108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:22.886 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:22.886+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:17:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:22.886+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:17:22.887 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:22.887+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:17:22.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:22 smithi081 ceph-mon[31453]: pgmap v21108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:24 smithi153 ceph-mon[36569]: pgmap v21109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:24 smithi104 ceph-mon[36624]: pgmap v21109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:24 smithi081 ceph-mon[31453]: pgmap v21109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:26 smithi153 ceph-mon[36569]: pgmap v21110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:26 smithi104 ceph-mon[36624]: pgmap v21110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:26 smithi081 ceph-mon[31453]: pgmap v21110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:28 smithi153 ceph-mon[36569]: pgmap v21111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:28 smithi104 ceph-mon[36624]: pgmap v21111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:28 smithi081 ceph-mon[31453]: pgmap v21111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:30 smithi153 ceph-mon[36569]: pgmap v21112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:30 smithi104 ceph-mon[36624]: pgmap v21112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:30 smithi081 ceph-mon[31453]: pgmap v21112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:32 smithi153 ceph-mon[36569]: pgmap v21113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:32 smithi104 ceph-mon[36624]: pgmap v21113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:32 smithi081 ceph-mon[31453]: pgmap v21113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:34 smithi153 ceph-mon[36569]: pgmap v21114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:34 smithi104 ceph-mon[36624]: pgmap v21114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:34 smithi081 ceph-mon[31453]: pgmap v21114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:36 smithi153 ceph-mon[36569]: pgmap v21115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:36 smithi104 ceph-mon[36624]: pgmap v21115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:36.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:36 smithi081 ceph-mon[31453]: pgmap v21115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:37.893+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:17:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:37.893+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:17:37.894 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:37.894+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:17:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:38 smithi153 ceph-mon[36569]: pgmap v21116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:38 smithi104 ceph-mon[36624]: pgmap v21116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:38 smithi081 ceph-mon[31453]: pgmap v21116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:40 smithi104 ceph-mon[36624]: pgmap v21117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:40 smithi081 ceph-mon[31453]: pgmap v21117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:40 smithi153 ceph-mon[36569]: pgmap v21117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:42 smithi104 ceph-mon[36624]: pgmap v21118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:42 smithi081 ceph-mon[31453]: pgmap v21118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:42 smithi153 ceph-mon[36569]: pgmap v21118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:44 smithi104 ceph-mon[36624]: pgmap v21119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:44 smithi081 ceph-mon[31453]: pgmap v21119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:44 smithi153 ceph-mon[36569]: pgmap v21119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:46 smithi104 ceph-mon[36624]: pgmap v21120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:46 smithi081 ceph-mon[31453]: pgmap v21120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:47.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:46 smithi153 ceph-mon[36569]: pgmap v21120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:48 smithi104 ceph-mon[36624]: pgmap v21121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:48.861 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:48 smithi153 ceph-mon[36569]: pgmap v21121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:48 smithi081 ceph-mon[31453]: pgmap v21121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:50 smithi081 ceph-mon[31453]: pgmap v21122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:50 smithi153 ceph-mon[36569]: pgmap v21122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:50 smithi104 ceph-mon[36624]: pgmap v21122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:52.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:52.897+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:17:52.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:52.897+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:17:52.897 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:17:52.897+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:17:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:52 smithi081 ceph-mon[31453]: pgmap v21123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:52 smithi153 ceph-mon[36569]: pgmap v21123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:52 smithi104 ceph-mon[36624]: pgmap v21123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:54 smithi081 ceph-mon[31453]: pgmap v21124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:54 smithi153 ceph-mon[36569]: pgmap v21124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:54 smithi104 ceph-mon[36624]: pgmap v21124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:56 smithi081 ceph-mon[31453]: pgmap v21125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:56 smithi153 ceph-mon[36569]: pgmap v21125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:56 smithi104 ceph-mon[36624]: pgmap v21125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:17:58 smithi081 ceph-mon[31453]: pgmap v21126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:17:58 smithi153 ceph-mon[36569]: pgmap v21126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:17:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:17:58 smithi104 ceph-mon[36624]: pgmap v21126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:00 smithi081 ceph-mon[31453]: pgmap v21127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:01.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:00 smithi153 ceph-mon[36569]: pgmap v21127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:00 smithi104 ceph-mon[36624]: pgmap v21127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:02.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:02 smithi104 ceph-mon[36624]: pgmap v21128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:02 smithi081 ceph-mon[31453]: pgmap v21128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:02 smithi153 ceph-mon[36569]: pgmap v21128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:04 smithi081 ceph-mon[31453]: pgmap v21129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:04 smithi153 ceph-mon[36569]: pgmap v21129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:04 smithi104 ceph-mon[36624]: pgmap v21129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:06 smithi081 ceph-mon[31453]: pgmap v21130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:06 smithi153 ceph-mon[36569]: pgmap v21130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:06 smithi104 ceph-mon[36624]: pgmap v21130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:07.898+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:18:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:07.898+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:18:07.899 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:07.898+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:18:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:08 smithi081 ceph-mon[31453]: pgmap v21131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:08 smithi153 ceph-mon[36569]: pgmap v21131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:08 smithi104 ceph-mon[36624]: pgmap v21131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: pgmap v21132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:18:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:10.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:10 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: pgmap v21132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:10 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: pgmap v21132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:18:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:10 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:18:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:18:11.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:18:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:18:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:18:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:18:11.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:11 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:18:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:18:12.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:11 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:18:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:18:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:18:12.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:11 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:18:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:12 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:18:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:12 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:18:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:12 smithi081 ceph-mon[31453]: pgmap v21133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:12 smithi081 ceph-mon[31453]: pgmap v21134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:12 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:18:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:12 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:18:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:12 smithi153 ceph-mon[36569]: pgmap v21133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:13.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:12 smithi153 ceph-mon[36569]: pgmap v21134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:12 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:18:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:12 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:18:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:12 smithi104 ceph-mon[36624]: pgmap v21133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:12 smithi104 ceph-mon[36624]: pgmap v21134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:14 smithi081 ceph-mon[31453]: pgmap v21135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:14 smithi153 ceph-mon[36569]: pgmap v21135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:14 smithi104 ceph-mon[36624]: pgmap v21135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:16 smithi153 ceph-mon[36569]: pgmap v21136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:16 smithi104 ceph-mon[36624]: pgmap v21136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:16 smithi081 ceph-mon[31453]: pgmap v21136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:18 smithi153 ceph-mon[36569]: pgmap v21137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:18 smithi104 ceph-mon[36624]: pgmap v21137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:18 smithi081 ceph-mon[31453]: pgmap v21137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:20 smithi153 ceph-mon[36569]: pgmap v21138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:20 smithi104 ceph-mon[36624]: pgmap v21138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:20 smithi081 ceph-mon[31453]: pgmap v21138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:22.911+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:18:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:22.911+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:18:22.911 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:22.911+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:18:22.983 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:22 smithi081 ceph-mon[31453]: pgmap v21139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:22 smithi153 ceph-mon[36569]: pgmap v21139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:23.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:22 smithi104 ceph-mon[36624]: pgmap v21139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:24 smithi153 ceph-mon[36569]: pgmap v21140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:24 smithi104 ceph-mon[36624]: pgmap v21140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:24 smithi081 ceph-mon[31453]: pgmap v21140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:26 smithi153 ceph-mon[36569]: pgmap v21141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:26 smithi104 ceph-mon[36624]: pgmap v21141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:26 smithi081 ceph-mon[31453]: pgmap v21141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:28 smithi153 ceph-mon[36569]: pgmap v21142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:28 smithi104 ceph-mon[36624]: pgmap v21142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:28 smithi081 ceph-mon[31453]: pgmap v21142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:30 smithi153 ceph-mon[36569]: pgmap v21143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:30 smithi104 ceph-mon[36624]: pgmap v21143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:30 smithi081 ceph-mon[31453]: pgmap v21143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:32 smithi153 ceph-mon[36569]: pgmap v21144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:32 smithi104 ceph-mon[36624]: pgmap v21144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:32 smithi081 ceph-mon[31453]: pgmap v21144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:34 smithi153 ceph-mon[36569]: pgmap v21145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:34 smithi104 ceph-mon[36624]: pgmap v21145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:34 smithi081 ceph-mon[31453]: pgmap v21145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:36 smithi153 ceph-mon[36569]: pgmap v21146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:36 smithi104 ceph-mon[36624]: pgmap v21146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:36 smithi081 ceph-mon[31453]: pgmap v21146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:37.925+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:18:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:37.925+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:18:37.925 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:37.925+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:18:39.059 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:38 smithi153 ceph-mon[36569]: pgmap v21147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:38 smithi104 ceph-mon[36624]: pgmap v21147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:38 smithi081 ceph-mon[31453]: pgmap v21147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:40 smithi153 ceph-mon[36569]: pgmap v21148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:40 smithi104 ceph-mon[36624]: pgmap v21148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:40 smithi081 ceph-mon[31453]: pgmap v21148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:42 smithi153 ceph-mon[36569]: pgmap v21149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:42 smithi104 ceph-mon[36624]: pgmap v21149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:42 smithi081 ceph-mon[31453]: pgmap v21149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:44 smithi153 ceph-mon[36569]: pgmap v21150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:44 smithi104 ceph-mon[36624]: pgmap v21150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:44 smithi081 ceph-mon[31453]: pgmap v21150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:46 smithi153 ceph-mon[36569]: pgmap v21151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:46 smithi104 ceph-mon[36624]: pgmap v21151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:46 smithi081 ceph-mon[31453]: pgmap v21151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:48 smithi153 ceph-mon[36569]: pgmap v21152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:48 smithi104 ceph-mon[36624]: pgmap v21152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:48 smithi081 ceph-mon[31453]: pgmap v21152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:50 smithi153 ceph-mon[36569]: pgmap v21153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:50 smithi104 ceph-mon[36624]: pgmap v21153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:50 smithi081 ceph-mon[31453]: pgmap v21153: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:52.940+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:18:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:52.940+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:18:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:18:52.940+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:18:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:52 smithi153 ceph-mon[36569]: pgmap v21154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:52 smithi104 ceph-mon[36624]: pgmap v21154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:52 smithi081 ceph-mon[31453]: pgmap v21154: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:54 smithi153 ceph-mon[36569]: pgmap v21155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:54 smithi104 ceph-mon[36624]: pgmap v21155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:54 smithi081 ceph-mon[31453]: pgmap v21155: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:56 smithi153 ceph-mon[36569]: pgmap v21156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:56 smithi104 ceph-mon[36624]: pgmap v21156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:56 smithi081 ceph-mon[31453]: pgmap v21156: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:18:58 smithi153 ceph-mon[36569]: pgmap v21157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:18:58 smithi104 ceph-mon[36624]: pgmap v21157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:18:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:18:58 smithi081 ceph-mon[31453]: pgmap v21157: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:00 smithi153 ceph-mon[36569]: pgmap v21158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:00 smithi104 ceph-mon[36624]: pgmap v21158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:00 smithi081 ceph-mon[31453]: pgmap v21158: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:02 smithi153 ceph-mon[36569]: pgmap v21159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:02 smithi104 ceph-mon[36624]: pgmap v21159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:02 smithi081 ceph-mon[31453]: pgmap v21159: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:03 smithi153 ceph-mon[36569]: pgmap v21160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:04.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:03 smithi104 ceph-mon[36624]: pgmap v21160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:04.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:03 smithi081 ceph-mon[31453]: pgmap v21160: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:06 smithi081 ceph-mon[31453]: pgmap v21161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:06 smithi153 ceph-mon[36569]: pgmap v21161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:06.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:06 smithi104 ceph-mon[36624]: pgmap v21161: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:07.954 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:07.954+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:19:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:07.954+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:19:07.955 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:07.954+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:19:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:08 smithi081 ceph-mon[31453]: pgmap v21162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:08 smithi153 ceph-mon[36569]: pgmap v21162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:08.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:08 smithi104 ceph-mon[36624]: pgmap v21162: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:10 smithi081 ceph-mon[31453]: pgmap v21163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:10 smithi153 ceph-mon[36569]: pgmap v21163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:10.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:10 smithi104 ceph-mon[36624]: pgmap v21163: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:12 smithi081 ceph-mon[31453]: pgmap v21164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:12 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:19:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:12 smithi153 ceph-mon[36569]: pgmap v21164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:12 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:19:12.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:12 smithi104 ceph-mon[36624]: pgmap v21164: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:12.687 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:12 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:19:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:19:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:19:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:13 smithi153 ceph-mon[36569]: pgmap v21165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:13 smithi153 ceph-mon[36569]: pgmap v21166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:13.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:19:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:19:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:19:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:13 smithi104 ceph-mon[36624]: pgmap v21165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:13 smithi104 ceph-mon[36624]: pgmap v21166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:19:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:19:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:19:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:13 smithi081 ceph-mon[31453]: pgmap v21165: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:13 smithi081 ceph-mon[31453]: pgmap v21166: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:19:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:15 smithi081 ceph-mon[31453]: pgmap v21167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:15 smithi153 ceph-mon[36569]: pgmap v21167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:15 smithi104 ceph-mon[36624]: pgmap v21167: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:17 smithi081 ceph-mon[31453]: pgmap v21168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:17 smithi153 ceph-mon[36569]: pgmap v21168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:17 smithi104 ceph-mon[36624]: pgmap v21168: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:19.289+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039d0 con 0x7f1704076980 2024-04-13T08:19:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:19.289+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:19:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:19.289+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 306 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:19:19.289 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:19.289+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 306 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039d0 con 0x7f1704076980 2024-04-13T08:19:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:19 smithi081 ceph-mon[31453]: pgmap v21169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:19 smithi153 ceph-mon[36569]: pgmap v21169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:19 smithi104 ceph-mon[36624]: pgmap v21169: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:21 smithi081 ceph-mon[31453]: pgmap v21170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:21 smithi153 ceph-mon[36569]: pgmap v21170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:21 smithi104 ceph-mon[36624]: pgmap v21170: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:22.960+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:19:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:22.960+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:19:22.960 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:22.960+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:19:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:23 smithi081 ceph-mon[31453]: pgmap v21171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:23 smithi153 ceph-mon[36569]: pgmap v21171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:23 smithi104 ceph-mon[36624]: pgmap v21171: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:25 smithi081 ceph-mon[31453]: pgmap v21172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:25 smithi153 ceph-mon[36569]: pgmap v21172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:25 smithi104 ceph-mon[36624]: pgmap v21172: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:27 smithi153 ceph-mon[36569]: pgmap v21173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:27 smithi104 ceph-mon[36624]: pgmap v21173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:27 smithi081 ceph-mon[31453]: pgmap v21173: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:29 smithi153 ceph-mon[36569]: pgmap v21174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:29 smithi104 ceph-mon[36624]: pgmap v21174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:29 smithi081 ceph-mon[31453]: pgmap v21174: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:31 smithi153 ceph-mon[36569]: pgmap v21175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:31 smithi104 ceph-mon[36624]: pgmap v21175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:31 smithi081 ceph-mon[31453]: pgmap v21175: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:33 smithi153 ceph-mon[36569]: pgmap v21176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:33 smithi104 ceph-mon[36624]: pgmap v21176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:33 smithi081 ceph-mon[31453]: pgmap v21176: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:35 smithi153 ceph-mon[36569]: pgmap v21177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:35 smithi104 ceph-mon[36624]: pgmap v21177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:35 smithi081 ceph-mon[31453]: pgmap v21177: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:37.974+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:19:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:37.974+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:19:37.974 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:37.974+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:19:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:37 smithi153 ceph-mon[36569]: pgmap v21178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:37 smithi104 ceph-mon[36624]: pgmap v21178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:37 smithi081 ceph-mon[31453]: pgmap v21178: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:39 smithi153 ceph-mon[36569]: pgmap v21179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:39 smithi104 ceph-mon[36624]: pgmap v21179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:39 smithi081 ceph-mon[31453]: pgmap v21179: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:41 smithi153 ceph-mon[36569]: pgmap v21180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:41 smithi104 ceph-mon[36624]: pgmap v21180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:41 smithi081 ceph-mon[31453]: pgmap v21180: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:43 smithi153 ceph-mon[36569]: pgmap v21181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:43 smithi104 ceph-mon[36624]: pgmap v21181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:43 smithi081 ceph-mon[31453]: pgmap v21181: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:45 smithi153 ceph-mon[36569]: pgmap v21182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:45 smithi104 ceph-mon[36624]: pgmap v21182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:45 smithi081 ceph-mon[31453]: pgmap v21182: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:47 smithi153 ceph-mon[36569]: pgmap v21183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:47 smithi104 ceph-mon[36624]: pgmap v21183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:47 smithi081 ceph-mon[31453]: pgmap v21183: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:49 smithi153 ceph-mon[36569]: pgmap v21184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:49 smithi104 ceph-mon[36624]: pgmap v21184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:49 smithi081 ceph-mon[31453]: pgmap v21184: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:51 smithi153 ceph-mon[36569]: pgmap v21185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:51 smithi104 ceph-mon[36624]: pgmap v21185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:51 smithi081 ceph-mon[31453]: pgmap v21185: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:52.986+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:19:52.986 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:52.986+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:19:52.987 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:19:52.986+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:19:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:53 smithi153 ceph-mon[36569]: pgmap v21186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:53 smithi104 ceph-mon[36624]: pgmap v21186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:53 smithi081 ceph-mon[31453]: pgmap v21186: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:55 smithi153 ceph-mon[36569]: pgmap v21187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:55 smithi104 ceph-mon[36624]: pgmap v21187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:55 smithi081 ceph-mon[31453]: pgmap v21187: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:57 smithi153 ceph-mon[36569]: pgmap v21188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:57 smithi104 ceph-mon[36624]: pgmap v21188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:19:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:57 smithi081 ceph-mon[31453]: pgmap v21188: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:19:59 smithi153 ceph-mon[36569]: pgmap v21189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:19:59 smithi104 ceph-mon[36624]: pgmap v21189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:19:59 smithi081 ceph-mon[31453]: pgmap v21189: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:20:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:20:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:20:02.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:01 smithi153 ceph-mon[36569]: pgmap v21190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:01 smithi104 ceph-mon[36624]: pgmap v21190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:01 smithi081 ceph-mon[31453]: pgmap v21190: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:03 smithi153 ceph-mon[36569]: pgmap v21191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:03 smithi104 ceph-mon[36624]: pgmap v21191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:03 smithi081 ceph-mon[31453]: pgmap v21191: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:05 smithi153 ceph-mon[36569]: pgmap v21192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:05 smithi104 ceph-mon[36624]: pgmap v21192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:05 smithi081 ceph-mon[31453]: pgmap v21192: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:08.000+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:20:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:08.000+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:20:08.000 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:08.000+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:20:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:07 smithi153 ceph-mon[36569]: pgmap v21193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:07 smithi104 ceph-mon[36624]: pgmap v21193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:07 smithi081 ceph-mon[31453]: pgmap v21193: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:09 smithi153 ceph-mon[36569]: pgmap v21194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:09 smithi104 ceph-mon[36624]: pgmap v21194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:09 smithi081 ceph-mon[31453]: pgmap v21194: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:12.039 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:11 smithi081 ceph-mon[31453]: pgmap v21195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:11 smithi153 ceph-mon[36569]: pgmap v21195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:11 smithi104 ceph-mon[36624]: pgmap v21195: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:13 smithi153 ceph-mon[36569]: pgmap v21196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:13 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:20:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:13 smithi104 ceph-mon[36624]: pgmap v21196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:13 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:20:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:13 smithi081 ceph-mon[31453]: pgmap v21196: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:13 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:20:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:20:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:20:15.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:14 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:20:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:20:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:20:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:14 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:20:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:20:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:20:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:14 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:20:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:15 smithi153 ceph-mon[36569]: pgmap v21197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:15 smithi153 ceph-mon[36569]: pgmap v21198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:15 smithi104 ceph-mon[36624]: pgmap v21197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:15 smithi104 ceph-mon[36624]: pgmap v21198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:15 smithi081 ceph-mon[31453]: pgmap v21197: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:15 smithi081 ceph-mon[31453]: pgmap v21198: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:16 smithi081 ceph-mon[31453]: pgmap v21199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:16 smithi153 ceph-mon[36569]: pgmap v21199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:16 smithi104 ceph-mon[36624]: pgmap v21199: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:19.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:19 smithi153 ceph-mon[36569]: pgmap v21200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:19.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:19 smithi104 ceph-mon[36624]: pgmap v21200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:19.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:19 smithi081 ceph-mon[31453]: pgmap v21200: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:21 smithi153 ceph-mon[36569]: pgmap v21201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:21.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:21 smithi104 ceph-mon[36624]: pgmap v21201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:21.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:21 smithi081 ceph-mon[31453]: pgmap v21201: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:23.015 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:23.015+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:20:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:23.015+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:20:23.016 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:23.015+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:20:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:23 smithi153 ceph-mon[36569]: pgmap v21202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:23.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:23 smithi104 ceph-mon[36624]: pgmap v21202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:23 smithi081 ceph-mon[31453]: pgmap v21202: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:25 smithi153 ceph-mon[36569]: pgmap v21203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:25.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:25 smithi104 ceph-mon[36624]: pgmap v21203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:25.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:25 smithi081 ceph-mon[31453]: pgmap v21203: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:27 smithi153 ceph-mon[36569]: pgmap v21204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:27.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:27 smithi104 ceph-mon[36624]: pgmap v21204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:27.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:27 smithi081 ceph-mon[31453]: pgmap v21204: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:29 smithi153 ceph-mon[36569]: pgmap v21205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:29.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:29 smithi104 ceph-mon[36624]: pgmap v21205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:29.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:29 smithi081 ceph-mon[31453]: pgmap v21205: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:31 smithi153 ceph-mon[36569]: pgmap v21206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:31.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:31 smithi104 ceph-mon[36624]: pgmap v21206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:31.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:31 smithi081 ceph-mon[31453]: pgmap v21206: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:33.384 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:33 smithi104 ceph-mon[36624]: pgmap v21207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:33 smithi081 ceph-mon[31453]: pgmap v21207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:33 smithi153 ceph-mon[36569]: pgmap v21207: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:35 smithi153 ceph-mon[36569]: pgmap v21208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:35.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:35 smithi104 ceph-mon[36624]: pgmap v21208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:35.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:35 smithi081 ceph-mon[31453]: pgmap v21208: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:37 smithi153 ceph-mon[36569]: pgmap v21209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:37.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:37 smithi104 ceph-mon[36624]: pgmap v21209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:37.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:37 smithi081 ceph-mon[31453]: pgmap v21209: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:38.016+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:20:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:38.017+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:20:38.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:38.017+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:20:39.527 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:39 smithi081 ceph-mon[31453]: pgmap v21210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:39 smithi153 ceph-mon[36569]: pgmap v21210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:39.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:39 smithi104 ceph-mon[36624]: pgmap v21210: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:41 smithi153 ceph-mon[36569]: pgmap v21211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:41.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:41 smithi104 ceph-mon[36624]: pgmap v21211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:41.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:41 smithi081 ceph-mon[31453]: pgmap v21211: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:43 smithi153 ceph-mon[36569]: pgmap v21212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:43 smithi104 ceph-mon[36624]: pgmap v21212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:43.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:43 smithi081 ceph-mon[31453]: pgmap v21212: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:45 smithi153 ceph-mon[36569]: pgmap v21213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:45 smithi104 ceph-mon[36624]: pgmap v21213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:45 smithi081 ceph-mon[31453]: pgmap v21213: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:47 smithi153 ceph-mon[36569]: pgmap v21214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:47 smithi104 ceph-mon[36624]: pgmap v21214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:47 smithi081 ceph-mon[31453]: pgmap v21214: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:49.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:49 smithi153 ceph-mon[36569]: pgmap v21215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:49 smithi104 ceph-mon[36624]: pgmap v21215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:49 smithi081 ceph-mon[31453]: pgmap v21215: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:51 smithi153 ceph-mon[36569]: pgmap v21216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:51 smithi104 ceph-mon[36624]: pgmap v21216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:51.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:51 smithi081 ceph-mon[31453]: pgmap v21216: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:53.033+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:20:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:53.033+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:20:53.033 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:20:53.033+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:20:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:53 smithi153 ceph-mon[36569]: pgmap v21217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:53 smithi104 ceph-mon[36624]: pgmap v21217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:53.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:53 smithi081 ceph-mon[31453]: pgmap v21217: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:55 smithi153 ceph-mon[36569]: pgmap v21218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:55 smithi104 ceph-mon[36624]: pgmap v21218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:55 smithi081 ceph-mon[31453]: pgmap v21218: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:57 smithi153 ceph-mon[36569]: pgmap v21219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:57.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:57 smithi104 ceph-mon[36624]: pgmap v21219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:57 smithi081 ceph-mon[31453]: pgmap v21219: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:20:59 smithi153 ceph-mon[36569]: pgmap v21220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:59.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:20:59 smithi104 ceph-mon[36624]: pgmap v21220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:20:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:20:59 smithi081 ceph-mon[31453]: pgmap v21220: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:01 smithi153 ceph-mon[36569]: pgmap v21221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:01.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:01 smithi104 ceph-mon[36624]: pgmap v21221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:01 smithi081 ceph-mon[31453]: pgmap v21221: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:03 smithi153 ceph-mon[36569]: pgmap v21222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:03.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:03 smithi104 ceph-mon[36624]: pgmap v21222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:03 smithi081 ceph-mon[31453]: pgmap v21222: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:05 smithi153 ceph-mon[36569]: pgmap v21223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:05.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:05 smithi104 ceph-mon[36624]: pgmap v21223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:05 smithi081 ceph-mon[31453]: pgmap v21223: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:07.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:07 smithi104 ceph-mon[36624]: pgmap v21224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:07 smithi081 ceph-mon[31453]: pgmap v21224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:07 smithi153 ceph-mon[36569]: pgmap v21224: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:08.049+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:21:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:08.049+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:21:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:08.049+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:21:09.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:09 smithi104 ceph-mon[36624]: pgmap v21225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:09.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:09 smithi081 ceph-mon[31453]: pgmap v21225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:09 smithi153 ceph-mon[36569]: pgmap v21225: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:11.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:11 smithi104 ceph-mon[36624]: pgmap v21226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:11.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:11 smithi081 ceph-mon[31453]: pgmap v21226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:11 smithi153 ceph-mon[36569]: pgmap v21226: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:13.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:13 smithi104 ceph-mon[36624]: pgmap v21227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:13.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:13 smithi081 ceph-mon[31453]: pgmap v21227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:13 smithi153 ceph-mon[36569]: pgmap v21227: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:15 smithi104 ceph-mon[36624]: pgmap v21228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:21:15.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:15 smithi081 ceph-mon[31453]: pgmap v21228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:21:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:15 smithi153 ceph-mon[36569]: pgmap v21228: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:15.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:21:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:21:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:21:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:16 smithi081 ceph-mon[31453]: pgmap v21229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:16 smithi081 ceph-mon[31453]: pgmap v21230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:16.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:21:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:21:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:21:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:16 smithi153 ceph-mon[36569]: pgmap v21229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:16 smithi153 ceph-mon[36569]: pgmap v21230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:16.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:21:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:21:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:21:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:16 smithi104 ceph-mon[36624]: pgmap v21229: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:16 smithi104 ceph-mon[36624]: pgmap v21230: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:21:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:17 smithi081 ceph-mon[31453]: pgmap v21231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:17 smithi153 ceph-mon[36569]: pgmap v21231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:17 smithi104 ceph-mon[36624]: pgmap v21231: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:20.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:19 smithi153 ceph-mon[36569]: pgmap v21232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:19 smithi081 ceph-mon[31453]: pgmap v21232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:19 smithi104 ceph-mon[36624]: pgmap v21232: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:21 smithi081 ceph-mon[31453]: pgmap v21233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:21 smithi153 ceph-mon[36569]: pgmap v21233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:21 smithi104 ceph-mon[36624]: pgmap v21233: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:23.065+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:21:23.064 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:23.065+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:21:23.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:23.065+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:21:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:23 smithi081 ceph-mon[31453]: pgmap v21234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:23 smithi153 ceph-mon[36569]: pgmap v21234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:23 smithi104 ceph-mon[36624]: pgmap v21234: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:25 smithi081 ceph-mon[31453]: pgmap v21235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:25 smithi153 ceph-mon[36569]: pgmap v21235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:25 smithi104 ceph-mon[36624]: pgmap v21235: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:27 smithi081 ceph-mon[31453]: pgmap v21236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:27 smithi153 ceph-mon[36569]: pgmap v21236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:27 smithi104 ceph-mon[36624]: pgmap v21236: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:29 smithi081 ceph-mon[31453]: pgmap v21237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:29 smithi153 ceph-mon[36569]: pgmap v21237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:29 smithi104 ceph-mon[36624]: pgmap v21237: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:31 smithi081 ceph-mon[31453]: pgmap v21238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:31 smithi153 ceph-mon[36569]: pgmap v21238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:31 smithi104 ceph-mon[36624]: pgmap v21238: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:34.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:33 smithi104 ceph-mon[36624]: pgmap v21239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:33 smithi081 ceph-mon[31453]: pgmap v21239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:33 smithi153 ceph-mon[36569]: pgmap v21239: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:35 smithi081 ceph-mon[31453]: pgmap v21240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:35 smithi153 ceph-mon[36569]: pgmap v21240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:35 smithi104 ceph-mon[36624]: pgmap v21240: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:38.072+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:21:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:38.073+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:21:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:38.073+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:21:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:37 smithi081 ceph-mon[31453]: pgmap v21241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:37 smithi153 ceph-mon[36569]: pgmap v21241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:37 smithi104 ceph-mon[36624]: pgmap v21241: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:39 smithi081 ceph-mon[31453]: pgmap v21242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:39 smithi153 ceph-mon[36569]: pgmap v21242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:39 smithi104 ceph-mon[36624]: pgmap v21242: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:41 smithi081 ceph-mon[31453]: pgmap v21243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:41 smithi153 ceph-mon[36569]: pgmap v21243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:42.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:41 smithi104 ceph-mon[36624]: pgmap v21243: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:43 smithi081 ceph-mon[31453]: pgmap v21244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:43 smithi153 ceph-mon[36569]: pgmap v21244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:43 smithi104 ceph-mon[36624]: pgmap v21244: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:45 smithi081 ceph-mon[31453]: pgmap v21245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:45 smithi153 ceph-mon[36569]: pgmap v21245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:45 smithi104 ceph-mon[36624]: pgmap v21245: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:47 smithi081 ceph-mon[31453]: pgmap v21246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:47 smithi153 ceph-mon[36569]: pgmap v21246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:48.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:47 smithi104 ceph-mon[36624]: pgmap v21246: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:49 smithi081 ceph-mon[31453]: pgmap v21247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:49 smithi153 ceph-mon[36569]: pgmap v21247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:49 smithi104 ceph-mon[36624]: pgmap v21247: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:51 smithi081 ceph-mon[31453]: pgmap v21248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:51 smithi153 ceph-mon[36569]: pgmap v21248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:51 smithi104 ceph-mon[36624]: pgmap v21248: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:53.088+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:21:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:53.088+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:21:53.088 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:21:53.088+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:21:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:53 smithi081 ceph-mon[31453]: pgmap v21249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:53 smithi153 ceph-mon[36569]: pgmap v21249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:53 smithi104 ceph-mon[36624]: pgmap v21249: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:55 smithi081 ceph-mon[31453]: pgmap v21250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:55 smithi153 ceph-mon[36569]: pgmap v21250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:55 smithi104 ceph-mon[36624]: pgmap v21250: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:57 smithi081 ceph-mon[31453]: pgmap v21251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:57 smithi153 ceph-mon[36569]: pgmap v21251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:21:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:57 smithi104 ceph-mon[36624]: pgmap v21251: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:21:59 smithi081 ceph-mon[31453]: pgmap v21252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:21:59 smithi153 ceph-mon[36569]: pgmap v21252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:21:59 smithi104 ceph-mon[36624]: pgmap v21252: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:01 smithi081 ceph-mon[31453]: pgmap v21253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:01 smithi153 ceph-mon[36569]: pgmap v21253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:01 smithi104 ceph-mon[36624]: pgmap v21253: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:04.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:03 smithi104 ceph-mon[36624]: pgmap v21254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:03 smithi081 ceph-mon[31453]: pgmap v21254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:03 smithi153 ceph-mon[36569]: pgmap v21254: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:05 smithi081 ceph-mon[31453]: pgmap v21255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:05 smithi153 ceph-mon[36569]: pgmap v21255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:06.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:05 smithi104 ceph-mon[36624]: pgmap v21255: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:08.102+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:22:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:08.102+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:22:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:08.102+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:22:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:07 smithi081 ceph-mon[31453]: pgmap v21256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:07 smithi153 ceph-mon[36569]: pgmap v21256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:07 smithi104 ceph-mon[36624]: pgmap v21256: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:09 smithi081 ceph-mon[31453]: pgmap v21257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:09 smithi153 ceph-mon[36569]: pgmap v21257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:10.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:09 smithi104 ceph-mon[36624]: pgmap v21257: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:11 smithi081 ceph-mon[31453]: pgmap v21258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:11 smithi153 ceph-mon[36569]: pgmap v21258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:11 smithi104 ceph-mon[36624]: pgmap v21258: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:13 smithi081 ceph-mon[31453]: pgmap v21259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:13 smithi153 ceph-mon[36569]: pgmap v21259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:14.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:13 smithi104 ceph-mon[36624]: pgmap v21259: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:15 smithi081 ceph-mon[31453]: pgmap v21260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:15 smithi153 ceph-mon[36569]: pgmap v21260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:16.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:15 smithi104 ceph-mon[36624]: pgmap v21260: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:22:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:16 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:17.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:22:17.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:17.186 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:16 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:22:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:16 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:18.215 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:18.216 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:18.216 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:17 smithi081 ceph-mon[31453]: pgmap v21261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:17 smithi153 ceph-mon[36569]: pgmap v21261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:17 smithi104 ceph-mon[36624]: pgmap v21261: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:19.317+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:22:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:19.317+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:22:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:19.317+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 307 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:22:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:19.317+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 307 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:22:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:19 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:19 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:19 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:20 smithi153 ceph-mon[36569]: pgmap v21262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:20 smithi104 ceph-mon[36624]: pgmap v21262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:20 smithi081 ceph-mon[31453]: pgmap v21262: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:22.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:22 smithi081 ceph-mon[31453]: pgmap v21263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:22 smithi153 ceph-mon[36569]: pgmap v21263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:22 smithi104 ceph-mon[36624]: pgmap v21263: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:23.110+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:22:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:23.110+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:22:23.111 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:23.110+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:22:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:22:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:22:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:23 smithi081 ceph-mon[31453]: pgmap v21264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:23 smithi081 ceph-mon[31453]: pgmap v21265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:23.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:22:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:22:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:23 smithi153 ceph-mon[36569]: pgmap v21264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:23 smithi153 ceph-mon[36569]: pgmap v21265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:23.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:22:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:22:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:23 smithi104 ceph-mon[36624]: pgmap v21264: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:23 smithi104 ceph-mon[36624]: pgmap v21265: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:22:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:25 smithi153 ceph-mon[36569]: pgmap v21266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:25 smithi104 ceph-mon[36624]: pgmap v21266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:25 smithi081 ceph-mon[31453]: pgmap v21266: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:27 smithi153 ceph-mon[36569]: pgmap v21267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:27 smithi104 ceph-mon[36624]: pgmap v21267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:27 smithi081 ceph-mon[31453]: pgmap v21267: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:29 smithi153 ceph-mon[36569]: pgmap v21268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:30.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:29 smithi104 ceph-mon[36624]: pgmap v21268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:30.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:29 smithi081 ceph-mon[31453]: pgmap v21268: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:31 smithi153 ceph-mon[36569]: pgmap v21269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:32.140 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:31 smithi104 ceph-mon[36624]: pgmap v21269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:32.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:31 smithi081 ceph-mon[31453]: pgmap v21269: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:33 smithi153 ceph-mon[36569]: pgmap v21270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:33 smithi104 ceph-mon[36624]: pgmap v21270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:33 smithi081 ceph-mon[31453]: pgmap v21270: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:35 smithi153 ceph-mon[36569]: pgmap v21271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:35 smithi104 ceph-mon[36624]: pgmap v21271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:35 smithi081 ceph-mon[31453]: pgmap v21271: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:37 smithi153 ceph-mon[36569]: pgmap v21272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:37 smithi104 ceph-mon[36624]: pgmap v21272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:38.119+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:22:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:38.120+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:22:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:38.120+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:22:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:37 smithi081 ceph-mon[31453]: pgmap v21272: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:39 smithi153 ceph-mon[36569]: pgmap v21273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:39 smithi104 ceph-mon[36624]: pgmap v21273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:39 smithi081 ceph-mon[31453]: pgmap v21273: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:41 smithi153 ceph-mon[36569]: pgmap v21274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:42.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:41 smithi104 ceph-mon[36624]: pgmap v21274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:42.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:41 smithi081 ceph-mon[31453]: pgmap v21274: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:43 smithi153 ceph-mon[36569]: pgmap v21275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:43 smithi104 ceph-mon[36624]: pgmap v21275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:43 smithi081 ceph-mon[31453]: pgmap v21275: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:45 smithi153 ceph-mon[36569]: pgmap v21276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:45 smithi104 ceph-mon[36624]: pgmap v21276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:45 smithi081 ceph-mon[31453]: pgmap v21276: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:47 smithi153 ceph-mon[36569]: pgmap v21277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:47 smithi104 ceph-mon[36624]: pgmap v21277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:47 smithi081 ceph-mon[31453]: pgmap v21277: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:49 smithi104 ceph-mon[36624]: pgmap v21278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:49 smithi081 ceph-mon[31453]: pgmap v21278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:49 smithi153 ceph-mon[36569]: pgmap v21278: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:51 smithi104 ceph-mon[36624]: pgmap v21279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:52.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:51 smithi153 ceph-mon[36569]: pgmap v21279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:51 smithi081 ceph-mon[31453]: pgmap v21279: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:53.135+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:22:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:53.135+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:22:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:22:53.135+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:22:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:53 smithi104 ceph-mon[36624]: pgmap v21280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:53 smithi081 ceph-mon[31453]: pgmap v21280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:53 smithi153 ceph-mon[36569]: pgmap v21280: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:55 smithi104 ceph-mon[36624]: pgmap v21281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:55 smithi081 ceph-mon[31453]: pgmap v21281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:55 smithi153 ceph-mon[36569]: pgmap v21281: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:56 smithi081 ceph-mon[31453]: pgmap v21282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:56 smithi153 ceph-mon[36569]: pgmap v21282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:22:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:56 smithi104 ceph-mon[36624]: pgmap v21282: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:22:59 smithi153 ceph-mon[36569]: pgmap v21283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:22:59 smithi104 ceph-mon[36624]: pgmap v21283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:22:59 smithi081 ceph-mon[31453]: pgmap v21283: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:01 smithi153 ceph-mon[36569]: pgmap v21284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:01 smithi104 ceph-mon[36624]: pgmap v21284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:01 smithi081 ceph-mon[31453]: pgmap v21284: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:03 smithi153 ceph-mon[36569]: pgmap v21285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:04.071 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:03 smithi081 ceph-mon[31453]: pgmap v21285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:04.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:03 smithi104 ceph-mon[36624]: pgmap v21285: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:06.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:05 smithi153 ceph-mon[36569]: pgmap v21286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:06.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:05 smithi104 ceph-mon[36624]: pgmap v21286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:06.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:05 smithi081 ceph-mon[31453]: pgmap v21286: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:07 smithi153 ceph-mon[36569]: pgmap v21287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:07 smithi104 ceph-mon[36624]: pgmap v21287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:08.151+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:23:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:08.151+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:23:08.151 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:08.151+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:23:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:07 smithi081 ceph-mon[31453]: pgmap v21287: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:09 smithi153 ceph-mon[36569]: pgmap v21288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:09 smithi104 ceph-mon[36624]: pgmap v21288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:10.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:09 smithi081 ceph-mon[31453]: pgmap v21288: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:11 smithi153 ceph-mon[36569]: pgmap v21289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:11 smithi104 ceph-mon[36624]: pgmap v21289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:11 smithi081 ceph-mon[31453]: pgmap v21289: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:13 smithi153 ceph-mon[36569]: pgmap v21290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:13 smithi104 ceph-mon[36624]: pgmap v21290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:14.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:13 smithi081 ceph-mon[31453]: pgmap v21290: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:15 smithi104 ceph-mon[36624]: pgmap v21291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:15 smithi081 ceph-mon[31453]: pgmap v21291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:15 smithi153 ceph-mon[36569]: pgmap v21291: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:17 smithi104 ceph-mon[36624]: pgmap v21292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:17 smithi081 ceph-mon[31453]: pgmap v21292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:17 smithi153 ceph-mon[36569]: pgmap v21292: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:19 smithi104 ceph-mon[36624]: pgmap v21293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:19 smithi081 ceph-mon[31453]: pgmap v21293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:19 smithi153 ceph-mon[36569]: pgmap v21293: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:21 smithi104 ceph-mon[36624]: pgmap v21294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:22.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:21 smithi153 ceph-mon[36569]: pgmap v21294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:22.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:21 smithi081 ceph-mon[31453]: pgmap v21294: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:23.166+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:23:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:23.166+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:23:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:23.166+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:23:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:23 smithi104 ceph-mon[36624]: pgmap v21295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:23:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:23 smithi081 ceph-mon[31453]: pgmap v21295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:23:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:23 smithi153 ceph-mon[36569]: pgmap v21295: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:23:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:24.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:23:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:23:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:23:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:23:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:23:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:23:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:23:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:25 smithi081 ceph-mon[31453]: pgmap v21296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:25 smithi081 ceph-mon[31453]: pgmap v21297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:25 smithi153 ceph-mon[36569]: pgmap v21296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:25 smithi153 ceph-mon[36569]: pgmap v21297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:25 smithi104 ceph-mon[36624]: pgmap v21296: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:25 smithi104 ceph-mon[36624]: pgmap v21297: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:26 smithi081 ceph-mon[31453]: pgmap v21298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:26 smithi153 ceph-mon[36569]: pgmap v21298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:26 smithi104 ceph-mon[36624]: pgmap v21298: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:29 smithi153 ceph-mon[36569]: pgmap v21299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:29 smithi104 ceph-mon[36624]: pgmap v21299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:29 smithi081 ceph-mon[31453]: pgmap v21299: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:31 smithi153 ceph-mon[36569]: pgmap v21300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:31 smithi104 ceph-mon[36624]: pgmap v21300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:31 smithi081 ceph-mon[31453]: pgmap v21300: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:33 smithi153 ceph-mon[36569]: pgmap v21301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:33 smithi104 ceph-mon[36624]: pgmap v21301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:33.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:33 smithi081 ceph-mon[31453]: pgmap v21301: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:35.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:35 smithi153 ceph-mon[36569]: pgmap v21302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:35.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:35 smithi104 ceph-mon[36624]: pgmap v21302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:35 smithi081 ceph-mon[31453]: pgmap v21302: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:37.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:37 smithi153 ceph-mon[36569]: pgmap v21303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:37.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:37 smithi104 ceph-mon[36624]: pgmap v21303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:37 smithi081 ceph-mon[31453]: pgmap v21303: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:38.180+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:23:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:38.180+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:23:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:38.180+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:23:39.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:39 smithi153 ceph-mon[36569]: pgmap v21304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:39.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:39 smithi104 ceph-mon[36624]: pgmap v21304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:39 smithi081 ceph-mon[31453]: pgmap v21304: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:41.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:41 smithi153 ceph-mon[36569]: pgmap v21305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:41.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:41 smithi104 ceph-mon[36624]: pgmap v21305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:41 smithi081 ceph-mon[31453]: pgmap v21305: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:43.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:43 smithi153 ceph-mon[36569]: pgmap v21306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:43.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:43 smithi104 ceph-mon[36624]: pgmap v21306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:43.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:43 smithi081 ceph-mon[31453]: pgmap v21306: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:45.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:45 smithi153 ceph-mon[36569]: pgmap v21307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:45.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:45 smithi104 ceph-mon[36624]: pgmap v21307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:45 smithi081 ceph-mon[31453]: pgmap v21307: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:47 smithi153 ceph-mon[36569]: pgmap v21308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:47 smithi104 ceph-mon[36624]: pgmap v21308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:47 smithi081 ceph-mon[31453]: pgmap v21308: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:49 smithi153 ceph-mon[36569]: pgmap v21309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:49.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:49 smithi104 ceph-mon[36624]: pgmap v21309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:49 smithi081 ceph-mon[31453]: pgmap v21309: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:51 smithi153 ceph-mon[36569]: pgmap v21310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:51.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:51 smithi104 ceph-mon[36624]: pgmap v21310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:51 smithi081 ceph-mon[31453]: pgmap v21310: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:53.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:53.182+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:23:53.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:53.182+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:23:53.182 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:23:53.182+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:23:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:53 smithi153 ceph-mon[36569]: pgmap v21311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:53.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:53 smithi104 ceph-mon[36624]: pgmap v21311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:53.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:53 smithi081 ceph-mon[31453]: pgmap v21311: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:55 smithi153 ceph-mon[36569]: pgmap v21312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:55.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:55 smithi104 ceph-mon[36624]: pgmap v21312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:55.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:55 smithi081 ceph-mon[31453]: pgmap v21312: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:57.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:57 smithi153 ceph-mon[36569]: pgmap v21313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:57 smithi104 ceph-mon[36624]: pgmap v21313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:57 smithi081 ceph-mon[31453]: pgmap v21313: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:23:59 smithi153 ceph-mon[36569]: pgmap v21314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:23:59 smithi104 ceph-mon[36624]: pgmap v21314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:23:59.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:23:59 smithi081 ceph-mon[31453]: pgmap v21314: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:01 smithi153 ceph-mon[36569]: pgmap v21315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:01 smithi104 ceph-mon[36624]: pgmap v21315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:01.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:01 smithi081 ceph-mon[31453]: pgmap v21315: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:03 smithi153 ceph-mon[36569]: pgmap v21316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:03 smithi104 ceph-mon[36624]: pgmap v21316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:03.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:03 smithi081 ceph-mon[31453]: pgmap v21316: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:05 smithi153 ceph-mon[36569]: pgmap v21317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:05 smithi104 ceph-mon[36624]: pgmap v21317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:05.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:05 smithi081 ceph-mon[31453]: pgmap v21317: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:07 smithi153 ceph-mon[36569]: pgmap v21318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:07 smithi104 ceph-mon[36624]: pgmap v21318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:07 smithi081 ceph-mon[31453]: pgmap v21318: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:08.197+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:24:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:08.197+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:24:08.198 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:08.197+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:24:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:09 smithi153 ceph-mon[36569]: pgmap v21319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:09.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:09 smithi104 ceph-mon[36624]: pgmap v21319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:09.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:09 smithi081 ceph-mon[31453]: pgmap v21319: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:11 smithi153 ceph-mon[36569]: pgmap v21320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:11.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:11 smithi104 ceph-mon[36624]: pgmap v21320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:11.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:11 smithi081 ceph-mon[31453]: pgmap v21320: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:13 smithi153 ceph-mon[36569]: pgmap v21321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:13 smithi104 ceph-mon[36624]: pgmap v21321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:13.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:13 smithi081 ceph-mon[31453]: pgmap v21321: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:15 smithi153 ceph-mon[36569]: pgmap v21322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:15 smithi104 ceph-mon[36624]: pgmap v21322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:15.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:15 smithi081 ceph-mon[31453]: pgmap v21322: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:17 smithi153 ceph-mon[36569]: pgmap v21323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:17.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:17 smithi104 ceph-mon[36624]: pgmap v21323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:17.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:17 smithi081 ceph-mon[31453]: pgmap v21323: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:19 smithi153 ceph-mon[36569]: pgmap v21324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:19.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:19 smithi104 ceph-mon[36624]: pgmap v21324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:19.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:19 smithi081 ceph-mon[31453]: pgmap v21324: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:21 smithi153 ceph-mon[36569]: pgmap v21325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:21 smithi104 ceph-mon[36624]: pgmap v21325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:21.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:21 smithi081 ceph-mon[31453]: pgmap v21325: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:23.199+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:24:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:23.199+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:24:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:23.199+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:24:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:23 smithi104 ceph-mon[36624]: pgmap v21326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:23.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:23 smithi081 ceph-mon[31453]: pgmap v21326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:23 smithi153 ceph-mon[36569]: pgmap v21326: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:25 smithi104 ceph-mon[36624]: pgmap v21327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:24:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:25 smithi081 ceph-mon[31453]: pgmap v21327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:24:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:25 smithi153 ceph-mon[36569]: pgmap v21327: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:26.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:24:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:24:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:24:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:24:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:24:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:24:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:24:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:24:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:24:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:24:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:27 smithi104 ceph-mon[36624]: pgmap v21328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:27.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:27 smithi104 ceph-mon[36624]: pgmap v21329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:27 smithi081 ceph-mon[31453]: pgmap v21328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:27 smithi081 ceph-mon[31453]: pgmap v21329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:27 smithi153 ceph-mon[36569]: pgmap v21328: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:27 smithi153 ceph-mon[36569]: pgmap v21329: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:29 smithi104 ceph-mon[36624]: pgmap v21330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:29 smithi081 ceph-mon[31453]: pgmap v21330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:29 smithi153 ceph-mon[36569]: pgmap v21330: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:31 smithi081 ceph-mon[31453]: pgmap v21331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:31 smithi153 ceph-mon[36569]: pgmap v21331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:31 smithi104 ceph-mon[36624]: pgmap v21331: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:33.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:33 smithi081 ceph-mon[31453]: pgmap v21332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:33 smithi153 ceph-mon[36569]: pgmap v21332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:34.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:33 smithi104 ceph-mon[36624]: pgmap v21332: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:34.340 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:34.340+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:24:34.340 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:34.340+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:24:34.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:34.341+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 308 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:24:34.341 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:34.341+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 308 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:24:35.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:35 smithi104 ceph-mon[36624]: pgmap v21333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:35 smithi081 ceph-mon[31453]: pgmap v21333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:35 smithi153 ceph-mon[36569]: pgmap v21333: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:37 smithi081 ceph-mon[31453]: pgmap v21334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:37 smithi153 ceph-mon[36569]: pgmap v21334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:37 smithi104 ceph-mon[36624]: pgmap v21334: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:38.201 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:38.201+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:24:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:38.201+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:24:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:38.201+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:24:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:39 smithi081 ceph-mon[31453]: pgmap v21335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:39 smithi153 ceph-mon[36569]: pgmap v21335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:39 smithi104 ceph-mon[36624]: pgmap v21335: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:41.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:41 smithi081 ceph-mon[31453]: pgmap v21336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:41 smithi153 ceph-mon[36569]: pgmap v21336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:41 smithi104 ceph-mon[36624]: pgmap v21336: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:43.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:43 smithi081 ceph-mon[31453]: pgmap v21337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:43 smithi153 ceph-mon[36569]: pgmap v21337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:44.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:43 smithi104 ceph-mon[36624]: pgmap v21337: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:45 smithi081 ceph-mon[31453]: pgmap v21338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:45 smithi153 ceph-mon[36569]: pgmap v21338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:45 smithi104 ceph-mon[36624]: pgmap v21338: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:47 smithi081 ceph-mon[31453]: pgmap v21339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:47 smithi153 ceph-mon[36569]: pgmap v21339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:47 smithi104 ceph-mon[36624]: pgmap v21339: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:49.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:49 smithi081 ceph-mon[31453]: pgmap v21340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:49 smithi153 ceph-mon[36569]: pgmap v21340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:49 smithi104 ceph-mon[36624]: pgmap v21340: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:51 smithi081 ceph-mon[31453]: pgmap v21341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:51 smithi153 ceph-mon[36569]: pgmap v21341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:51 smithi104 ceph-mon[36624]: pgmap v21341: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:53.217+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:24:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:53.217+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:24:53.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:24:53.217+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:24:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:53 smithi153 ceph-mon[36569]: pgmap v21342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:53 smithi104 ceph-mon[36624]: pgmap v21342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:53 smithi081 ceph-mon[31453]: pgmap v21342: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:55 smithi153 ceph-mon[36569]: pgmap v21343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:55 smithi104 ceph-mon[36624]: pgmap v21343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:55 smithi081 ceph-mon[31453]: pgmap v21343: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:57.959 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:57 smithi081 ceph-mon[31453]: pgmap v21344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:57 smithi153 ceph-mon[36569]: pgmap v21344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:24:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:57 smithi104 ceph-mon[36624]: pgmap v21344: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:24:59 smithi153 ceph-mon[36569]: pgmap v21345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:24:59 smithi104 ceph-mon[36624]: pgmap v21345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:24:59 smithi081 ceph-mon[31453]: pgmap v21345: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:01 smithi153 ceph-mon[36569]: pgmap v21346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:02.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:01 smithi104 ceph-mon[36624]: pgmap v21346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:01 smithi081 ceph-mon[31453]: pgmap v21346: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:03 smithi153 ceph-mon[36569]: pgmap v21347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:03 smithi104 ceph-mon[36624]: pgmap v21347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:03 smithi081 ceph-mon[31453]: pgmap v21347: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:05 smithi153 ceph-mon[36569]: pgmap v21348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:05 smithi104 ceph-mon[36624]: pgmap v21348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:05 smithi081 ceph-mon[31453]: pgmap v21348: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:07.975 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:07 smithi081 ceph-mon[31453]: pgmap v21349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:07 smithi153 ceph-mon[36569]: pgmap v21349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:07 smithi104 ceph-mon[36624]: pgmap v21349: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:08.232+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:25:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:08.232+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:25:08.232 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:08.232+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:25:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:09 smithi153 ceph-mon[36569]: pgmap v21350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:09 smithi104 ceph-mon[36624]: pgmap v21350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:09 smithi081 ceph-mon[31453]: pgmap v21350: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:11 smithi153 ceph-mon[36569]: pgmap v21351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:11 smithi104 ceph-mon[36624]: pgmap v21351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:11 smithi081 ceph-mon[31453]: pgmap v21351: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:13 smithi153 ceph-mon[36569]: pgmap v21352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:13 smithi104 ceph-mon[36624]: pgmap v21352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:13 smithi081 ceph-mon[31453]: pgmap v21352: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:16.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:15 smithi153 ceph-mon[36569]: pgmap v21353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:15 smithi104 ceph-mon[36624]: pgmap v21353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:15 smithi081 ceph-mon[31453]: pgmap v21353: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:17 smithi153 ceph-mon[36569]: pgmap v21354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:17 smithi104 ceph-mon[36624]: pgmap v21354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:17 smithi081 ceph-mon[31453]: pgmap v21354: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:19.347 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:19.347+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:25:19.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:19.348+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:25:19.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:19.348+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 309 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:25:19.348 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:19.348+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 309 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:25:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:19 smithi153 ceph-mon[36569]: pgmap v21355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:19 smithi104 ceph-mon[36624]: pgmap v21355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:19 smithi081 ceph-mon[31453]: pgmap v21355: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:21 smithi153 ceph-mon[36569]: pgmap v21356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:22.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:21 smithi104 ceph-mon[36624]: pgmap v21356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:21 smithi081 ceph-mon[31453]: pgmap v21356: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:23.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:23.247+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:25:23.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:23.247+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:25:23.247 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:23.247+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:25:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:23 smithi153 ceph-mon[36569]: pgmap v21357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:24.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:23 smithi104 ceph-mon[36624]: pgmap v21357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:23 smithi081 ceph-mon[31453]: pgmap v21357: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:25 smithi153 ceph-mon[36569]: pgmap v21358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:25 smithi104 ceph-mon[36624]: pgmap v21358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:25 smithi081 ceph-mon[31453]: pgmap v21358: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:25:27.057 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:25:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:25:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:27 smithi153 ceph-mon[36569]: pgmap v21359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:25:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:25:28.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:25:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:27 smithi104 ceph-mon[36624]: pgmap v21359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:25:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:25:28.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:25:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:27 smithi081 ceph-mon[31453]: pgmap v21359: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:25:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:25:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:25:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:28 smithi153 ceph-mon[36569]: pgmap v21360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:28 smithi153 ceph-mon[36569]: pgmap v21361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:28 smithi104 ceph-mon[36624]: pgmap v21360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:28 smithi104 ceph-mon[36624]: pgmap v21361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:28 smithi081 ceph-mon[31453]: pgmap v21360: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:28 smithi081 ceph-mon[31453]: pgmap v21361: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:30 smithi153 ceph-mon[36569]: pgmap v21362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:30 smithi104 ceph-mon[36624]: pgmap v21362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:30 smithi081 ceph-mon[31453]: pgmap v21362: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:32 smithi153 ceph-mon[36569]: pgmap v21363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:32 smithi104 ceph-mon[36624]: pgmap v21363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:32 smithi081 ceph-mon[31453]: pgmap v21363: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:35.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:34 smithi153 ceph-mon[36569]: pgmap v21364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:34 smithi104 ceph-mon[36624]: pgmap v21364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:34 smithi081 ceph-mon[31453]: pgmap v21364: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:36 smithi153 ceph-mon[36569]: pgmap v21365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:37.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:36 smithi104 ceph-mon[36624]: pgmap v21365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:36 smithi081 ceph-mon[31453]: pgmap v21365: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:38.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:38.256+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:25:38.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:38.256+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:25:38.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:38.256+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:25:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:38 smithi153 ceph-mon[36569]: pgmap v21366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:39.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:38 smithi104 ceph-mon[36624]: pgmap v21366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:38 smithi081 ceph-mon[31453]: pgmap v21366: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:40 smithi153 ceph-mon[36569]: pgmap v21367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:40 smithi104 ceph-mon[36624]: pgmap v21367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:40 smithi081 ceph-mon[31453]: pgmap v21367: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:42 smithi153 ceph-mon[36569]: pgmap v21368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:42 smithi104 ceph-mon[36624]: pgmap v21368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:42 smithi081 ceph-mon[31453]: pgmap v21368: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:44 smithi104 ceph-mon[36624]: pgmap v21369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:44 smithi081 ceph-mon[31453]: pgmap v21369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:44 smithi153 ceph-mon[36569]: pgmap v21369: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:46 smithi104 ceph-mon[36624]: pgmap v21370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:47.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:46 smithi081 ceph-mon[31453]: pgmap v21370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:46 smithi153 ceph-mon[36569]: pgmap v21370: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:48 smithi104 ceph-mon[36624]: pgmap v21371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:48 smithi081 ceph-mon[31453]: pgmap v21371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:48 smithi153 ceph-mon[36569]: pgmap v21371: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:50 smithi104 ceph-mon[36624]: pgmap v21372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:50 smithi081 ceph-mon[31453]: pgmap v21372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:50 smithi153 ceph-mon[36569]: pgmap v21372: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:52 smithi104 ceph-mon[36624]: pgmap v21373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:53.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:52 smithi153 ceph-mon[36569]: pgmap v21373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:52 smithi081 ceph-mon[31453]: pgmap v21373: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:53.264+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:25:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:53.264+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:25:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:25:53.264+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:25:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:54 smithi081 ceph-mon[31453]: pgmap v21374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:54 smithi153 ceph-mon[36569]: pgmap v21374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:54 smithi104 ceph-mon[36624]: pgmap v21374: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:56 smithi081 ceph-mon[31453]: pgmap v21375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:56 smithi153 ceph-mon[36569]: pgmap v21375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:57.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:56 smithi104 ceph-mon[36624]: pgmap v21375: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:59.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:25:58 smithi081 ceph-mon[31453]: pgmap v21376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:25:58 smithi153 ceph-mon[36569]: pgmap v21376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:25:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:25:58 smithi104 ceph-mon[36624]: pgmap v21376: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:00 smithi081 ceph-mon[31453]: pgmap v21377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:00 smithi153 ceph-mon[36569]: pgmap v21377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:00 smithi104 ceph-mon[36624]: pgmap v21377: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:02 smithi081 ceph-mon[31453]: pgmap v21378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:02 smithi153 ceph-mon[36569]: pgmap v21378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:02 smithi104 ceph-mon[36624]: pgmap v21378: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:04 smithi081 ceph-mon[31453]: pgmap v21379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:04 smithi153 ceph-mon[36569]: pgmap v21379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:05.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:04 smithi104 ceph-mon[36624]: pgmap v21379: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:07.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:06 smithi104 ceph-mon[36624]: pgmap v21380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:06 smithi081 ceph-mon[31453]: pgmap v21380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:06 smithi153 ceph-mon[36569]: pgmap v21380: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:08.280+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:26:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:08.280+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:26:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:08.280+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:26:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:07 smithi153 ceph-mon[36569]: pgmap v21381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:07 smithi104 ceph-mon[36624]: pgmap v21381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:07 smithi081 ceph-mon[31453]: pgmap v21381: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:10 smithi153 ceph-mon[36569]: pgmap v21382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:10 smithi104 ceph-mon[36624]: pgmap v21382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:10 smithi081 ceph-mon[31453]: pgmap v21382: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:12 smithi153 ceph-mon[36569]: pgmap v21383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:12 smithi104 ceph-mon[36624]: pgmap v21383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:12 smithi081 ceph-mon[31453]: pgmap v21383: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:14 smithi153 ceph-mon[36569]: pgmap v21384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:14 smithi104 ceph-mon[36624]: pgmap v21384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:14.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:14 smithi081 ceph-mon[31453]: pgmap v21384: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:16 smithi153 ceph-mon[36569]: pgmap v21385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:16 smithi104 ceph-mon[36624]: pgmap v21385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:16 smithi081 ceph-mon[31453]: pgmap v21385: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:18 smithi153 ceph-mon[36569]: pgmap v21386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:18 smithi104 ceph-mon[36624]: pgmap v21386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:18 smithi081 ceph-mon[31453]: pgmap v21386: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:20 smithi153 ceph-mon[36569]: pgmap v21387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:20 smithi104 ceph-mon[36624]: pgmap v21387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:20.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:20 smithi081 ceph-mon[31453]: pgmap v21387: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:22 smithi153 ceph-mon[36569]: pgmap v21388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:22 smithi104 ceph-mon[36624]: pgmap v21388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:22 smithi081 ceph-mon[31453]: pgmap v21388: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:23.296+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:26:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:23.296+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:26:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:23.296+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:26:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:24 smithi153 ceph-mon[36569]: pgmap v21389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:24 smithi104 ceph-mon[36624]: pgmap v21389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:24 smithi081 ceph-mon[31453]: pgmap v21389: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:26 smithi153 ceph-mon[36569]: pgmap v21390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:26.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:26 smithi104 ceph-mon[36624]: pgmap v21390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:26 smithi081 ceph-mon[31453]: pgmap v21390: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:28 smithi153 ceph-mon[36569]: pgmap v21391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:28 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:26:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:28 smithi104 ceph-mon[36624]: pgmap v21391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:28 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:26:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:28 smithi081 ceph-mon[31453]: pgmap v21391: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:28 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:26:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:26:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:26:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:26:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:26:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:26:29.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:26:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:26:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:26:29.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:26:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:30 smithi153 ceph-mon[36569]: pgmap v21392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:30 smithi153 ceph-mon[36569]: pgmap v21393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:30 smithi104 ceph-mon[36624]: pgmap v21392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:30.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:30 smithi104 ceph-mon[36624]: pgmap v21393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:30 smithi081 ceph-mon[31453]: pgmap v21392: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:30 smithi081 ceph-mon[31453]: pgmap v21393: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:32 smithi153 ceph-mon[36569]: pgmap v21394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:32 smithi104 ceph-mon[36624]: pgmap v21394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:32.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:32 smithi081 ceph-mon[31453]: pgmap v21394: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:34 smithi153 ceph-mon[36569]: pgmap v21395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:34 smithi104 ceph-mon[36624]: pgmap v21395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:34 smithi081 ceph-mon[31453]: pgmap v21395: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:36.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:36 smithi153 ceph-mon[36569]: pgmap v21396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:36 smithi104 ceph-mon[36624]: pgmap v21396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:36 smithi081 ceph-mon[31453]: pgmap v21396: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:38.303 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:38.303+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:26:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:38.303+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:26:38.304 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:38.303+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:26:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:38 smithi104 ceph-mon[36624]: pgmap v21397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:38 smithi081 ceph-mon[31453]: pgmap v21397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:38 smithi153 ceph-mon[36569]: pgmap v21397: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:40 smithi104 ceph-mon[36624]: pgmap v21398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:40.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:40 smithi081 ceph-mon[31453]: pgmap v21398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:40 smithi153 ceph-mon[36569]: pgmap v21398: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:42.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:42 smithi104 ceph-mon[36624]: pgmap v21399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:42.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:42 smithi081 ceph-mon[31453]: pgmap v21399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:42 smithi153 ceph-mon[36569]: pgmap v21399: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:44 smithi104 ceph-mon[36624]: pgmap v21400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:44 smithi081 ceph-mon[31453]: pgmap v21400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:44 smithi153 ceph-mon[36569]: pgmap v21400: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:46 smithi081 ceph-mon[31453]: pgmap v21401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:46 smithi153 ceph-mon[36569]: pgmap v21401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:46 smithi104 ceph-mon[36624]: pgmap v21401: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:48 smithi081 ceph-mon[31453]: pgmap v21402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:48 smithi153 ceph-mon[36569]: pgmap v21402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:48 smithi104 ceph-mon[36624]: pgmap v21402: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:50 smithi081 ceph-mon[31453]: pgmap v21403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:50 smithi153 ceph-mon[36569]: pgmap v21403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:50 smithi104 ceph-mon[36624]: pgmap v21403: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:52 smithi081 ceph-mon[31453]: pgmap v21404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:52 smithi153 ceph-mon[36569]: pgmap v21404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:52 smithi104 ceph-mon[36624]: pgmap v21404: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:53.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:53.313+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:26:53.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:53.313+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:26:53.313 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:26:53.313+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:26:54.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:54 smithi153 ceph-mon[36569]: pgmap v21405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:54 smithi081 ceph-mon[31453]: pgmap v21405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:54 smithi104 ceph-mon[36624]: pgmap v21405: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:56 smithi081 ceph-mon[31453]: pgmap v21406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:56 smithi153 ceph-mon[36569]: pgmap v21406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:56 smithi104 ceph-mon[36624]: pgmap v21406: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:26:58 smithi081 ceph-mon[31453]: pgmap v21407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:26:58 smithi153 ceph-mon[36569]: pgmap v21407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:26:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:26:58 smithi104 ceph-mon[36624]: pgmap v21407: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:00 smithi081 ceph-mon[31453]: pgmap v21408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:00 smithi153 ceph-mon[36569]: pgmap v21408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:00 smithi104 ceph-mon[36624]: pgmap v21408: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:02.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:02 smithi081 ceph-mon[31453]: pgmap v21409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:03.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:02 smithi153 ceph-mon[36569]: pgmap v21409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:02 smithi104 ceph-mon[36624]: pgmap v21409: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:04 smithi081 ceph-mon[31453]: pgmap v21410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:04 smithi153 ceph-mon[36569]: pgmap v21410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:04 smithi104 ceph-mon[36624]: pgmap v21410: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:06 smithi081 ceph-mon[31453]: pgmap v21411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:06 smithi153 ceph-mon[36569]: pgmap v21411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:06 smithi104 ceph-mon[36624]: pgmap v21411: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:08.323+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:27:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:08.323+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:27:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:08.323+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:27:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:08 smithi081 ceph-mon[31453]: pgmap v21412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:08 smithi153 ceph-mon[36569]: pgmap v21412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:08 smithi104 ceph-mon[36624]: pgmap v21412: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:10.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:10 smithi081 ceph-mon[31453]: pgmap v21413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:10 smithi153 ceph-mon[36569]: pgmap v21413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:11.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:10 smithi104 ceph-mon[36624]: pgmap v21413: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:12.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:12 smithi081 ceph-mon[31453]: pgmap v21414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:12 smithi153 ceph-mon[36569]: pgmap v21414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:12 smithi104 ceph-mon[36624]: pgmap v21414: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:14 smithi153 ceph-mon[36569]: pgmap v21415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:14 smithi104 ceph-mon[36624]: pgmap v21415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:15.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:14 smithi081 ceph-mon[31453]: pgmap v21415: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:16 smithi153 ceph-mon[36569]: pgmap v21416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:16 smithi104 ceph-mon[36624]: pgmap v21416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:17.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:16 smithi081 ceph-mon[31453]: pgmap v21416: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:18 smithi153 ceph-mon[36569]: pgmap v21417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:18 smithi104 ceph-mon[36624]: pgmap v21417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:19.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:18 smithi081 ceph-mon[31453]: pgmap v21417: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:20 smithi153 ceph-mon[36569]: pgmap v21418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:21.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:20 smithi104 ceph-mon[36624]: pgmap v21418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:20 smithi081 ceph-mon[31453]: pgmap v21418: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:22 smithi153 ceph-mon[36569]: pgmap v21419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:22 smithi104 ceph-mon[36624]: pgmap v21419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:22 smithi081 ceph-mon[31453]: pgmap v21419: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:23.327+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:27:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:23.327+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:27:23.327 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:23.327+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:27:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:24 smithi153 ceph-mon[36569]: pgmap v21420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:25.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:24 smithi104 ceph-mon[36624]: pgmap v21420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:24 smithi081 ceph-mon[31453]: pgmap v21420: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:26 smithi153 ceph-mon[36569]: pgmap v21421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:26 smithi104 ceph-mon[36624]: pgmap v21421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:26 smithi081 ceph-mon[31453]: pgmap v21421: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:28 smithi153 ceph-mon[36569]: pgmap v21422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:28 smithi104 ceph-mon[36624]: pgmap v21422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:29.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:28 smithi081 ceph-mon[31453]: pgmap v21422: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:29 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:27:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:29 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:27:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:29 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:27:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:30 smithi153 ceph-mon[36569]: pgmap v21423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:27:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:27:31.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:30 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:27:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:30 smithi104 ceph-mon[36624]: pgmap v21423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:27:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:27:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:30 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:27:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:30 smithi081 ceph-mon[31453]: pgmap v21423: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:27:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:27:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:30 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:27:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:31 smithi153 ceph-mon[36569]: pgmap v21424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:31 smithi153 ceph-mon[36569]: pgmap v21425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:31 smithi104 ceph-mon[36624]: pgmap v21424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:31 smithi104 ceph-mon[36624]: pgmap v21425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:31 smithi081 ceph-mon[31453]: pgmap v21424: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:31 smithi081 ceph-mon[31453]: pgmap v21425: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:33 smithi153 ceph-mon[36569]: pgmap v21426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:33 smithi104 ceph-mon[36624]: pgmap v21426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:34.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:33 smithi081 ceph-mon[31453]: pgmap v21426: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:35 smithi153 ceph-mon[36569]: pgmap v21427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:35 smithi104 ceph-mon[36624]: pgmap v21427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:35 smithi081 ceph-mon[31453]: pgmap v21427: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:37 smithi153 ceph-mon[36569]: pgmap v21428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:37 smithi104 ceph-mon[36624]: pgmap v21428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:38.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:37 smithi081 ceph-mon[31453]: pgmap v21428: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:38.334 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:38.334+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:27:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:38.334+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:27:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:38.334+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:27:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:39 smithi153 ceph-mon[36569]: pgmap v21429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:40.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:39 smithi104 ceph-mon[36624]: pgmap v21429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:39 smithi081 ceph-mon[31453]: pgmap v21429: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:41 smithi153 ceph-mon[36569]: pgmap v21430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:41 smithi104 ceph-mon[36624]: pgmap v21430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:42.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:41 smithi081 ceph-mon[31453]: pgmap v21430: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:43 smithi153 ceph-mon[36569]: pgmap v21431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:44.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:43 smithi104 ceph-mon[36624]: pgmap v21431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:44.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:43 smithi081 ceph-mon[31453]: pgmap v21431: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:45 smithi153 ceph-mon[36569]: pgmap v21432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:46.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:45 smithi104 ceph-mon[36624]: pgmap v21432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:46.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:45 smithi081 ceph-mon[31453]: pgmap v21432: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:47 smithi153 ceph-mon[36569]: pgmap v21433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:47 smithi104 ceph-mon[36624]: pgmap v21433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:48.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:47 smithi081 ceph-mon[31453]: pgmap v21433: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:49 smithi153 ceph-mon[36569]: pgmap v21434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:49 smithi104 ceph-mon[36624]: pgmap v21434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:49 smithi081 ceph-mon[31453]: pgmap v21434: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:51 smithi153 ceph-mon[36569]: pgmap v21435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:51 smithi104 ceph-mon[36624]: pgmap v21435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:51 smithi081 ceph-mon[31453]: pgmap v21435: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:53.350+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:27:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:53.350+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:27:53.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:27:53.350+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:27:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:53 smithi153 ceph-mon[36569]: pgmap v21436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:53 smithi104 ceph-mon[36624]: pgmap v21436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:53 smithi081 ceph-mon[31453]: pgmap v21436: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:55 smithi153 ceph-mon[36569]: pgmap v21437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:55 smithi104 ceph-mon[36624]: pgmap v21437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:56.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:55 smithi081 ceph-mon[31453]: pgmap v21437: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:57 smithi153 ceph-mon[36569]: pgmap v21438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:58.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:57 smithi104 ceph-mon[36624]: pgmap v21438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:27:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:57 smithi081 ceph-mon[31453]: pgmap v21438: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:27:59 smithi153 ceph-mon[36569]: pgmap v21439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:00.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:27:59 smithi104 ceph-mon[36624]: pgmap v21439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:00.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:27:59 smithi081 ceph-mon[31453]: pgmap v21439: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:01 smithi153 ceph-mon[36569]: pgmap v21440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:02.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:01 smithi104 ceph-mon[36624]: pgmap v21440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:02.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:01 smithi081 ceph-mon[31453]: pgmap v21440: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:03 smithi153 ceph-mon[36569]: pgmap v21441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:03 smithi104 ceph-mon[36624]: pgmap v21441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:04.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:03 smithi081 ceph-mon[31453]: pgmap v21441: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:05 smithi153 ceph-mon[36569]: pgmap v21442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:06.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:05 smithi104 ceph-mon[36624]: pgmap v21442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:05 smithi081 ceph-mon[31453]: pgmap v21442: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:07 smithi153 ceph-mon[36569]: pgmap v21443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:07 smithi104 ceph-mon[36624]: pgmap v21443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:07 smithi081 ceph-mon[31453]: pgmap v21443: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:08.365+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:28:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:08.365+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:28:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:08.365+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:28:10.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:09 smithi104 ceph-mon[36624]: pgmap v21444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:09 smithi081 ceph-mon[31453]: pgmap v21444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:09 smithi153 ceph-mon[36569]: pgmap v21444: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:12.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:11 smithi104 ceph-mon[36624]: pgmap v21445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:12.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:11 smithi081 ceph-mon[31453]: pgmap v21445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:11 smithi153 ceph-mon[36569]: pgmap v21445: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:14.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:13 smithi104 ceph-mon[36624]: pgmap v21446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:13 smithi081 ceph-mon[31453]: pgmap v21446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:13 smithi153 ceph-mon[36569]: pgmap v21446: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:16.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:15 smithi104 ceph-mon[36624]: pgmap v21447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:16.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:15 smithi081 ceph-mon[31453]: pgmap v21447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:15 smithi153 ceph-mon[36569]: pgmap v21447: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:17 smithi104 ceph-mon[36624]: pgmap v21448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:17 smithi081 ceph-mon[31453]: pgmap v21448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:17 smithi153 ceph-mon[36569]: pgmap v21448: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:19.377+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:28:19.377 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:19.377+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:28:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:19.378+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 310 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:28:19.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:19.378+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 310 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:28:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:19 smithi081 ceph-mon[31453]: pgmap v21449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:19 smithi153 ceph-mon[36569]: pgmap v21449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:20.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:19 smithi104 ceph-mon[36624]: pgmap v21449: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:21 smithi081 ceph-mon[31453]: pgmap v21450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:21 smithi153 ceph-mon[36569]: pgmap v21450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:21 smithi104 ceph-mon[36624]: pgmap v21450: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:23.377+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:28:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:23.377+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:28:23.378 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:23.377+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:28:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:23 smithi081 ceph-mon[31453]: pgmap v21451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:23 smithi153 ceph-mon[36569]: pgmap v21451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:23 smithi104 ceph-mon[36624]: pgmap v21451: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:26.109 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:25 smithi153 ceph-mon[36569]: pgmap v21452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:25 smithi081 ceph-mon[31453]: pgmap v21452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:25 smithi104 ceph-mon[36624]: pgmap v21452: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:28.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:27 smithi081 ceph-mon[31453]: pgmap v21453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:27 smithi153 ceph-mon[36569]: pgmap v21453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:28.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:27 smithi104 ceph-mon[36624]: pgmap v21453: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:29 smithi081 ceph-mon[31453]: pgmap v21454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:29 smithi153 ceph-mon[36569]: pgmap v21454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:29 smithi104 ceph-mon[36624]: pgmap v21454: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: pgmap v21455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:28:32.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:31 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: pgmap v21455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:31 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: pgmap v21455: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:31 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:28:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:28:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:28:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:28:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:28:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:28:33.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:32 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:28:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:28:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:28:33.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:32 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:28:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:28:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:28:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:28:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:28:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:28:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:28:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:28:33.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:32 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:28:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:33 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:28:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:33 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:28:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:33 smithi081 ceph-mon[31453]: pgmap v21456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:34.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:33 smithi081 ceph-mon[31453]: pgmap v21457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:33 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:28:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:33 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:28:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:33 smithi153 ceph-mon[36569]: pgmap v21456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:33 smithi153 ceph-mon[36569]: pgmap v21457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:33 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:28:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:33 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:28:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:33 smithi104 ceph-mon[36624]: pgmap v21456: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:33 smithi104 ceph-mon[36624]: pgmap v21457: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:36.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:35 smithi081 ceph-mon[31453]: pgmap v21458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:35 smithi153 ceph-mon[36569]: pgmap v21458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:36.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:35 smithi104 ceph-mon[36624]: pgmap v21458: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:38.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:37 smithi081 ceph-mon[31453]: pgmap v21459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:37 smithi153 ceph-mon[36569]: pgmap v21459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:37 smithi104 ceph-mon[36624]: pgmap v21459: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:38.379+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:28:38.379 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:38.379+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:28:38.380 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:38.379+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:28:40.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:39 smithi081 ceph-mon[31453]: pgmap v21460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:39 smithi153 ceph-mon[36569]: pgmap v21460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:39 smithi104 ceph-mon[36624]: pgmap v21460: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:40 smithi153 ceph-mon[36569]: pgmap v21461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:41.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:40 smithi104 ceph-mon[36624]: pgmap v21461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:40 smithi081 ceph-mon[31453]: pgmap v21461: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:43 smithi153 ceph-mon[36569]: pgmap v21462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:43.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:43 smithi104 ceph-mon[36624]: pgmap v21462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:43.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:43 smithi081 ceph-mon[31453]: pgmap v21462: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:45 smithi153 ceph-mon[36569]: pgmap v21463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:45.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:45 smithi104 ceph-mon[36624]: pgmap v21463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:45.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:45 smithi081 ceph-mon[31453]: pgmap v21463: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:47.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:47 smithi104 ceph-mon[36624]: pgmap v21464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:47.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:47 smithi081 ceph-mon[31453]: pgmap v21464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:47.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:47 smithi153 ceph-mon[36569]: pgmap v21464: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:49.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:49 smithi104 ceph-mon[36624]: pgmap v21465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:49.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:49 smithi081 ceph-mon[31453]: pgmap v21465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:49.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:49 smithi153 ceph-mon[36569]: pgmap v21465: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:51.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:51 smithi104 ceph-mon[36624]: pgmap v21466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:51.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:51 smithi081 ceph-mon[31453]: pgmap v21466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:51.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:51 smithi153 ceph-mon[36569]: pgmap v21466: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:53.395+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:28:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:53.395+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:28:53.395 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:28:53.395+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:28:53.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:53 smithi104 ceph-mon[36624]: pgmap v21467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:53.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:53 smithi081 ceph-mon[31453]: pgmap v21467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:53.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:53 smithi153 ceph-mon[36569]: pgmap v21467: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:55.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:55 smithi104 ceph-mon[36624]: pgmap v21468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:55.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:55 smithi081 ceph-mon[31453]: pgmap v21468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:55.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:55 smithi153 ceph-mon[36569]: pgmap v21468: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:57.608 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:57 smithi153 ceph-mon[36569]: pgmap v21469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:57.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:57 smithi081 ceph-mon[31453]: pgmap v21469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:57.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:57 smithi104 ceph-mon[36624]: pgmap v21469: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:59.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:28:59 smithi081 ceph-mon[31453]: pgmap v21470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:28:59 smithi153 ceph-mon[36569]: pgmap v21470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:28:59.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:28:59 smithi104 ceph-mon[36624]: pgmap v21470: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:01.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:01 smithi081 ceph-mon[31453]: pgmap v21471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:01.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:01 smithi153 ceph-mon[36569]: pgmap v21471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:01.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:01 smithi104 ceph-mon[36624]: pgmap v21471: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:03.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:03 smithi081 ceph-mon[31453]: pgmap v21472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:03 smithi153 ceph-mon[36569]: pgmap v21472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:03 smithi104 ceph-mon[36624]: pgmap v21472: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:05 smithi081 ceph-mon[31453]: pgmap v21473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:05 smithi153 ceph-mon[36569]: pgmap v21473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:05.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:05 smithi104 ceph-mon[36624]: pgmap v21473: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:07.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:07 smithi081 ceph-mon[31453]: pgmap v21474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:07.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:07 smithi153 ceph-mon[36569]: pgmap v21474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:07.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:07 smithi104 ceph-mon[36624]: pgmap v21474: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:08.397+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:29:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:08.397+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:29:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:08.397+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:29:09.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:09 smithi081 ceph-mon[31453]: pgmap v21475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:09.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:09 smithi153 ceph-mon[36569]: pgmap v21475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:09.833 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:09 smithi104 ceph-mon[36624]: pgmap v21475: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:11.633 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:11 smithi104 ceph-mon[36624]: pgmap v21476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:11.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:11 smithi081 ceph-mon[31453]: pgmap v21476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:11.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:11 smithi153 ceph-mon[36569]: pgmap v21476: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:13.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:13 smithi081 ceph-mon[31453]: pgmap v21477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:13.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:13 smithi153 ceph-mon[36569]: pgmap v21477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:13.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:13 smithi104 ceph-mon[36624]: pgmap v21477: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:15.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:15 smithi081 ceph-mon[31453]: pgmap v21478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:15.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:15 smithi153 ceph-mon[36569]: pgmap v21478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:15.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:15 smithi104 ceph-mon[36624]: pgmap v21478: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:17.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:17 smithi081 ceph-mon[31453]: pgmap v21479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:17.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:17 smithi153 ceph-mon[36569]: pgmap v21479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:17.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:17 smithi104 ceph-mon[36624]: pgmap v21479: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:19.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:19 smithi081 ceph-mon[31453]: pgmap v21480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:19.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:19 smithi153 ceph-mon[36569]: pgmap v21480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:19.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:19 smithi104 ceph-mon[36624]: pgmap v21480: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:21.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:21 smithi081 ceph-mon[31453]: pgmap v21481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:21.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:21 smithi153 ceph-mon[36569]: pgmap v21481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:21.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:21 smithi104 ceph-mon[36624]: pgmap v21481: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:23.412 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:23.412+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:29:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:23.412+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:29:23.413 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:23.412+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:29:23.426 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:23 smithi081 ceph-mon[31453]: pgmap v21482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:23.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:23 smithi153 ceph-mon[36569]: pgmap v21482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:23.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:23 smithi104 ceph-mon[36624]: pgmap v21482: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:25.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:25 smithi153 ceph-mon[36569]: pgmap v21483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:25.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:25 smithi104 ceph-mon[36624]: pgmap v21483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:25.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:25 smithi081 ceph-mon[31453]: pgmap v21483: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:27.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:27 smithi153 ceph-mon[36569]: pgmap v21484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:27.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:27 smithi104 ceph-mon[36624]: pgmap v21484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:27.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:27 smithi081 ceph-mon[31453]: pgmap v21484: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:29.709 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:29 smithi081 ceph-mon[31453]: pgmap v21485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:29.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:29 smithi153 ceph-mon[36569]: pgmap v21485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:29.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:29 smithi104 ceph-mon[36624]: pgmap v21485: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:31.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:31 smithi153 ceph-mon[36569]: pgmap v21486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:31.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:31 smithi104 ceph-mon[36624]: pgmap v21486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:31.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:31 smithi081 ceph-mon[31453]: pgmap v21486: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:33.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:33 smithi081 ceph-mon[31453]: pgmap v21487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:33.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:33 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:29:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:33 smithi153 ceph-mon[36569]: pgmap v21487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:33 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:29:33.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:33 smithi104 ceph-mon[36624]: pgmap v21487: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:33.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:33 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:29:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:29:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:29:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:34 smithi153 ceph-mon[36569]: pgmap v21488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:34 smithi153 ceph-mon[36569]: pgmap v21489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:34.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:29:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:29:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:29:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:34 smithi104 ceph-mon[36624]: pgmap v21488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:34 smithi104 ceph-mon[36624]: pgmap v21489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:29:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:29:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:29:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:34 smithi081 ceph-mon[31453]: pgmap v21488: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:34 smithi081 ceph-mon[31453]: pgmap v21489: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:34.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:29:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:36 smithi153 ceph-mon[36569]: pgmap v21490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:36 smithi104 ceph-mon[36624]: pgmap v21490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:37.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:36 smithi081 ceph-mon[31453]: pgmap v21490: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:37 smithi153 ceph-mon[36569]: pgmap v21491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:38.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:37 smithi104 ceph-mon[36624]: pgmap v21491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:38.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:37 smithi081 ceph-mon[31453]: pgmap v21491: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:38.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:38.421+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:29:38.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:38.421+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:29:38.421 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:38.421+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:29:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:39 smithi153 ceph-mon[36569]: pgmap v21492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:40.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:39 smithi104 ceph-mon[36624]: pgmap v21492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:39 smithi081 ceph-mon[31453]: pgmap v21492: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:41 smithi153 ceph-mon[36569]: pgmap v21493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:42.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:41 smithi104 ceph-mon[36624]: pgmap v21493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:41 smithi081 ceph-mon[31453]: pgmap v21493: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:43 smithi153 ceph-mon[36569]: pgmap v21494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:44.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:43 smithi104 ceph-mon[36624]: pgmap v21494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:43 smithi081 ceph-mon[31453]: pgmap v21494: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:45 smithi153 ceph-mon[36569]: pgmap v21495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:45 smithi104 ceph-mon[36624]: pgmap v21495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:45 smithi081 ceph-mon[31453]: pgmap v21495: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:47 smithi153 ceph-mon[36569]: pgmap v21496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:48.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:47 smithi104 ceph-mon[36624]: pgmap v21496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:47 smithi081 ceph-mon[31453]: pgmap v21496: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:49 smithi153 ceph-mon[36569]: pgmap v21497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:50.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:49 smithi104 ceph-mon[36624]: pgmap v21497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:50.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:49 smithi081 ceph-mon[31453]: pgmap v21497: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:52.310 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:51 smithi153 ceph-mon[36569]: pgmap v21498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:52.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:51 smithi104 ceph-mon[36624]: pgmap v21498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:52.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:51 smithi081 ceph-mon[31453]: pgmap v21498: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:53.435+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:53.435+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:29:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:29:53.435+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:29:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:53 smithi153 ceph-mon[36569]: pgmap v21499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:54.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:53 smithi104 ceph-mon[36624]: pgmap v21499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:54.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:53 smithi081 ceph-mon[31453]: pgmap v21499: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:55 smithi153 ceph-mon[36569]: pgmap v21500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:56.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:55 smithi104 ceph-mon[36624]: pgmap v21500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:56.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:55 smithi081 ceph-mon[31453]: pgmap v21500: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:57 smithi153 ceph-mon[36569]: pgmap v21501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:58.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:57 smithi104 ceph-mon[36624]: pgmap v21501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:29:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:57 smithi081 ceph-mon[31453]: pgmap v21501: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:29:59 smithi153 ceph-mon[36569]: pgmap v21502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:00.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:29:59 smithi104 ceph-mon[36624]: pgmap v21502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:00.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:29:59 smithi081 ceph-mon[31453]: pgmap v21502: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:30:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:30:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:30:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:01 smithi153 ceph-mon[36569]: pgmap v21503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:01 smithi104 ceph-mon[36624]: pgmap v21503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:01 smithi081 ceph-mon[31453]: pgmap v21503: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:04.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:03 smithi081 ceph-mon[31453]: pgmap v21504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:03 smithi153 ceph-mon[36569]: pgmap v21504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:04.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:03 smithi104 ceph-mon[36624]: pgmap v21504: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:05 smithi153 ceph-mon[36569]: pgmap v21505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:06.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:05 smithi104 ceph-mon[36624]: pgmap v21505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:06.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:05 smithi081 ceph-mon[31453]: pgmap v21505: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:07 smithi153 ceph-mon[36569]: pgmap v21506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:07 smithi104 ceph-mon[36624]: pgmap v21506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:08.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:07 smithi081 ceph-mon[31453]: pgmap v21506: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:08.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:08.450+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:30:08.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:08.450+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:30:08.451 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:08.450+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:30:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:09 smithi153 ceph-mon[36569]: pgmap v21507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:09 smithi104 ceph-mon[36624]: pgmap v21507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:10.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:09 smithi081 ceph-mon[31453]: pgmap v21507: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:11 smithi153 ceph-mon[36569]: pgmap v21508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:12.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:11 smithi104 ceph-mon[36624]: pgmap v21508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:12.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:11 smithi081 ceph-mon[31453]: pgmap v21508: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:13 smithi153 ceph-mon[36569]: pgmap v21509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:13 smithi104 ceph-mon[36624]: pgmap v21509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:13 smithi081 ceph-mon[31453]: pgmap v21509: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:15 smithi153 ceph-mon[36569]: pgmap v21510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:16.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:15 smithi104 ceph-mon[36624]: pgmap v21510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:15 smithi081 ceph-mon[31453]: pgmap v21510: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:17 smithi153 ceph-mon[36569]: pgmap v21511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:18.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:17 smithi104 ceph-mon[36624]: pgmap v21511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:17 smithi081 ceph-mon[31453]: pgmap v21511: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:19 smithi153 ceph-mon[36569]: pgmap v21512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:20.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:19 smithi104 ceph-mon[36624]: pgmap v21512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:20.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:19 smithi081 ceph-mon[31453]: pgmap v21512: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:21 smithi153 ceph-mon[36569]: pgmap v21513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:22.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:21 smithi104 ceph-mon[36624]: pgmap v21513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:22.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:21 smithi081 ceph-mon[31453]: pgmap v21513: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:23.462+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:30:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:23.462+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:30:23.462 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:23.462+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:30:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:23 smithi153 ceph-mon[36569]: pgmap v21514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:23 smithi104 ceph-mon[36624]: pgmap v21514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:23 smithi081 ceph-mon[31453]: pgmap v21514: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:25 smithi153 ceph-mon[36569]: pgmap v21515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:25 smithi104 ceph-mon[36624]: pgmap v21515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:26.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:25 smithi081 ceph-mon[31453]: pgmap v21515: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:27 smithi153 ceph-mon[36569]: pgmap v21516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:27 smithi104 ceph-mon[36624]: pgmap v21516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:28.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:27 smithi081 ceph-mon[31453]: pgmap v21516: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:29 smithi153 ceph-mon[36569]: pgmap v21517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:29 smithi104 ceph-mon[36624]: pgmap v21517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:29 smithi081 ceph-mon[31453]: pgmap v21517: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:31 smithi153 ceph-mon[36569]: pgmap v21518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:31 smithi104 ceph-mon[36624]: pgmap v21518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:32.345 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:31 smithi081 ceph-mon[31453]: pgmap v21518: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:33 smithi153 ceph-mon[36569]: pgmap v21519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:34.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:33 smithi104 ceph-mon[36624]: pgmap v21519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:34.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:33 smithi081 ceph-mon[31453]: pgmap v21519: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:34 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:30:35.308 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:34 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:30:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:34 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:30:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:30:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:30:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:35 smithi153 ceph-mon[36569]: pgmap v21520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:35 smithi153 ceph-mon[36569]: pgmap v21521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:36.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:35 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:30:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:30:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:30:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:35 smithi104 ceph-mon[36624]: pgmap v21520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:35 smithi104 ceph-mon[36624]: pgmap v21521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:36.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:35 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:30:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:30:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:30:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:35 smithi081 ceph-mon[31453]: pgmap v21520: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:35 smithi081 ceph-mon[31453]: pgmap v21521: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:35 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:30:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:38.478+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:30:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:38.478+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:30:38.478 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:38.478+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:30:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:38 smithi153 ceph-mon[36569]: pgmap v21522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:38.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:38 smithi104 ceph-mon[36624]: pgmap v21522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:38 smithi081 ceph-mon[31453]: pgmap v21522: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:40 smithi153 ceph-mon[36569]: pgmap v21523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:40 smithi104 ceph-mon[36624]: pgmap v21523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:40 smithi081 ceph-mon[31453]: pgmap v21523: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:42 smithi153 ceph-mon[36569]: pgmap v21524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:42 smithi104 ceph-mon[36624]: pgmap v21524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:42 smithi081 ceph-mon[31453]: pgmap v21524: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:44 smithi153 ceph-mon[36569]: pgmap v21525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:44.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:44 smithi104 ceph-mon[36624]: pgmap v21525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:44.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:44 smithi081 ceph-mon[31453]: pgmap v21525: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:46 smithi153 ceph-mon[36569]: pgmap v21526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:46 smithi104 ceph-mon[36624]: pgmap v21526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:46.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:46 smithi081 ceph-mon[31453]: pgmap v21526: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:48 smithi153 ceph-mon[36569]: pgmap v21527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:48.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:48 smithi104 ceph-mon[36624]: pgmap v21527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:48.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:48 smithi081 ceph-mon[31453]: pgmap v21527: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:50 smithi153 ceph-mon[36569]: pgmap v21528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:50 smithi104 ceph-mon[36624]: pgmap v21528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:50 smithi081 ceph-mon[31453]: pgmap v21528: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:52 smithi153 ceph-mon[36569]: pgmap v21529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:52 smithi104 ceph-mon[36624]: pgmap v21529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:52 smithi081 ceph-mon[31453]: pgmap v21529: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:53.486+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:30:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:53.486+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:30:53.486 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:30:53.486+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:30:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:54 smithi153 ceph-mon[36569]: pgmap v21530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:54 smithi104 ceph-mon[36624]: pgmap v21530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:54 smithi081 ceph-mon[31453]: pgmap v21530: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:56 smithi153 ceph-mon[36569]: pgmap v21531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:56 smithi104 ceph-mon[36624]: pgmap v21531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:56 smithi081 ceph-mon[31453]: pgmap v21531: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:30:58 smithi153 ceph-mon[36569]: pgmap v21532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:30:58 smithi104 ceph-mon[36624]: pgmap v21532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:30:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:30:58 smithi081 ceph-mon[31453]: pgmap v21532: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:00 smithi153 ceph-mon[36569]: pgmap v21533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:00.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:00 smithi104 ceph-mon[36624]: pgmap v21533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:00 smithi081 ceph-mon[31453]: pgmap v21533: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:02 smithi153 ceph-mon[36569]: pgmap v21534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:02 smithi104 ceph-mon[36624]: pgmap v21534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:02 smithi081 ceph-mon[31453]: pgmap v21534: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:04 smithi153 ceph-mon[36569]: pgmap v21535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:04 smithi104 ceph-mon[36624]: pgmap v21535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:04 smithi081 ceph-mon[31453]: pgmap v21535: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:06 smithi153 ceph-mon[36569]: pgmap v21536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:06 smithi104 ceph-mon[36624]: pgmap v21536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:06.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:06 smithi081 ceph-mon[31453]: pgmap v21536: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:08.494+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:31:08.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:08.494+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:31:08.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:08.494+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:31:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:08 smithi153 ceph-mon[36569]: pgmap v21537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:08 smithi104 ceph-mon[36624]: pgmap v21537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:08.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:08 smithi081 ceph-mon[31453]: pgmap v21537: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:10 smithi153 ceph-mon[36569]: pgmap v21538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:10 smithi104 ceph-mon[36624]: pgmap v21538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:10 smithi081 ceph-mon[31453]: pgmap v21538: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:12.202+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 msgr2=0x7f16e80a5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000616) for more than 900000000 us, fault. 2024-04-13T08:31:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:12.202+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f16d4005850 tx=0x7f16f40bd040 comp rx=0 tx=0).stop 2024-04-13T08:31:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:12.202+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 msgr2=0x7f16e80a5010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T08:31:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:12.202+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b15c0 0x7f16e80a5010 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T08:31:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:12.202+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T08:31:12.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:12.203+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T08:31:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:12.204+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c530 tx=0x7f16f40a6000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T08:31:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:12 smithi153 ceph-mon[36569]: pgmap v21539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:12.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:12 smithi104 ceph-mon[36624]: pgmap v21539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:12 smithi081 ceph-mon[31453]: pgmap v21539: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:14 smithi153 ceph-mon[36569]: pgmap v21540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:14 smithi104 ceph-mon[36624]: pgmap v21540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:14 smithi081 ceph-mon[31453]: pgmap v21540: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:16 smithi153 ceph-mon[36569]: pgmap v21541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:16.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:16 smithi104 ceph-mon[36624]: pgmap v21541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:16.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:16 smithi081 ceph-mon[31453]: pgmap v21541: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:18 smithi153 ceph-mon[36569]: pgmap v21542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:18.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:18 smithi104 ceph-mon[36624]: pgmap v21542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:18 smithi081 ceph-mon[31453]: pgmap v21542: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:19.407+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:31:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:19.407+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:31:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:19.408+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 311 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:31:19.407 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:19.408+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 311 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:31:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:20 smithi153 ceph-mon[36569]: pgmap v21543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:20.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:20 smithi104 ceph-mon[36624]: pgmap v21543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:20 smithi081 ceph-mon[31453]: pgmap v21543: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:22.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:22 smithi153 ceph-mon[36569]: pgmap v21544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:22 smithi104 ceph-mon[36624]: pgmap v21544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:22.907 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:22 smithi081 ceph-mon[31453]: pgmap v21544: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:23.499+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:31:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:23.499+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:31:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:23.499+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:31:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:24 smithi153 ceph-mon[36569]: pgmap v21545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:24.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:24 smithi104 ceph-mon[36624]: pgmap v21545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:24.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:24 smithi081 ceph-mon[31453]: pgmap v21545: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:26 smithi153 ceph-mon[36569]: pgmap v21546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:26 smithi104 ceph-mon[36624]: pgmap v21546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:26.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:26 smithi081 ceph-mon[31453]: pgmap v21546: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:28 smithi153 ceph-mon[36569]: pgmap v21547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:28.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:28 smithi104 ceph-mon[36624]: pgmap v21547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:28 smithi081 ceph-mon[31453]: pgmap v21547: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:30 smithi104 ceph-mon[36624]: pgmap v21548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:30.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:30 smithi081 ceph-mon[31453]: pgmap v21548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:30 smithi153 ceph-mon[36569]: pgmap v21548: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:32.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:32 smithi104 ceph-mon[36624]: pgmap v21549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:32.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:32 smithi081 ceph-mon[31453]: pgmap v21549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:32 smithi153 ceph-mon[36569]: pgmap v21549: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:34.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:34 smithi104 ceph-mon[36624]: pgmap v21550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:34 smithi081 ceph-mon[31453]: pgmap v21550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:34 smithi153 ceph-mon[36569]: pgmap v21550: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:36 smithi104 ceph-mon[36624]: pgmap v21551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:36.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:36 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:31:36.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:36 smithi081 ceph-mon[31453]: pgmap v21551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:36.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:36 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:31:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:36 smithi153 ceph-mon[36569]: pgmap v21551: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:36 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:31:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:31:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:31:37.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:31:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:31:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:31:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:31:38.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:31:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:31:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:31:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:38.511+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:31:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:38.511+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:31:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:38.511+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:31:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:38 smithi081 ceph-mon[31453]: pgmap v21552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:38.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:38 smithi081 ceph-mon[31453]: pgmap v21553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:38 smithi153 ceph-mon[36569]: pgmap v21552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:38 smithi153 ceph-mon[36569]: pgmap v21553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:39.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:38 smithi104 ceph-mon[36624]: pgmap v21552: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:39.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:38 smithi104 ceph-mon[36624]: pgmap v21553: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:40 smithi081 ceph-mon[31453]: pgmap v21554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:40 smithi153 ceph-mon[36569]: pgmap v21554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:40 smithi104 ceph-mon[36624]: pgmap v21554: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:42.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:42 smithi104 ceph-mon[36624]: pgmap v21555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:42 smithi081 ceph-mon[31453]: pgmap v21555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:42 smithi153 ceph-mon[36569]: pgmap v21555: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:44 smithi081 ceph-mon[31453]: pgmap v21556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:44 smithi153 ceph-mon[36569]: pgmap v21556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:44 smithi104 ceph-mon[36624]: pgmap v21556: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:46 smithi081 ceph-mon[31453]: pgmap v21557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:46 smithi153 ceph-mon[36569]: pgmap v21557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:46 smithi104 ceph-mon[36624]: pgmap v21557: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:48.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:48 smithi081 ceph-mon[31453]: pgmap v21558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:48 smithi153 ceph-mon[36569]: pgmap v21558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:48 smithi104 ceph-mon[36624]: pgmap v21558: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:50.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:50 smithi081 ceph-mon[31453]: pgmap v21559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:50 smithi153 ceph-mon[36569]: pgmap v21559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:50 smithi104 ceph-mon[36624]: pgmap v21559: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:52.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:52 smithi081 ceph-mon[31453]: pgmap v21560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:52 smithi153 ceph-mon[36569]: pgmap v21560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:52 smithi104 ceph-mon[36624]: pgmap v21560: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:53.513+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:31:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:53.513+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:31:53.514 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:31:53.513+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:31:54.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:54 smithi081 ceph-mon[31453]: pgmap v21561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:55.064 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:54 smithi153 ceph-mon[36569]: pgmap v21561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:55.083 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:54 smithi104 ceph-mon[36624]: pgmap v21561: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:56.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:56 smithi081 ceph-mon[31453]: pgmap v21562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:56 smithi153 ceph-mon[36569]: pgmap v21562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:56 smithi104 ceph-mon[36624]: pgmap v21562: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:58.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:31:58 smithi081 ceph-mon[31453]: pgmap v21563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:31:58 smithi153 ceph-mon[36569]: pgmap v21563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:31:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:31:58 smithi104 ceph-mon[36624]: pgmap v21563: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:00 smithi081 ceph-mon[31453]: pgmap v21564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:00 smithi153 ceph-mon[36569]: pgmap v21564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:00 smithi104 ceph-mon[36624]: pgmap v21564: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:02 smithi081 ceph-mon[31453]: pgmap v21565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:02 smithi153 ceph-mon[36569]: pgmap v21565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:02 smithi104 ceph-mon[36624]: pgmap v21565: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:04 smithi153 ceph-mon[36569]: pgmap v21566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:04 smithi104 ceph-mon[36624]: pgmap v21566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:05.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:04 smithi081 ceph-mon[31453]: pgmap v21566: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:06 smithi153 ceph-mon[36569]: pgmap v21567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:06 smithi104 ceph-mon[36624]: pgmap v21567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:06 smithi081 ceph-mon[31453]: pgmap v21567: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:08.529+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:32:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:08.529+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:32:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:08.529+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:32:09.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:08 smithi153 ceph-mon[36569]: pgmap v21568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:08 smithi104 ceph-mon[36624]: pgmap v21568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:08 smithi081 ceph-mon[31453]: pgmap v21568: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:10 smithi153 ceph-mon[36569]: pgmap v21569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:10 smithi104 ceph-mon[36624]: pgmap v21569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:10 smithi081 ceph-mon[31453]: pgmap v21569: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:12 smithi153 ceph-mon[36569]: pgmap v21570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:12 smithi104 ceph-mon[36624]: pgmap v21570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:12 smithi081 ceph-mon[31453]: pgmap v21570: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:14 smithi153 ceph-mon[36569]: pgmap v21571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:14 smithi104 ceph-mon[36624]: pgmap v21571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:14 smithi081 ceph-mon[31453]: pgmap v21571: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:16 smithi153 ceph-mon[36569]: pgmap v21572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:16 smithi104 ceph-mon[36624]: pgmap v21572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:16 smithi081 ceph-mon[31453]: pgmap v21572: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:18 smithi153 ceph-mon[36569]: pgmap v21573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:19.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:18 smithi104 ceph-mon[36624]: pgmap v21573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:18 smithi081 ceph-mon[31453]: pgmap v21573: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:20 smithi153 ceph-mon[36569]: pgmap v21574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:20 smithi104 ceph-mon[36624]: pgmap v21574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:21.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:20 smithi081 ceph-mon[31453]: pgmap v21574: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:22 smithi153 ceph-mon[36569]: pgmap v21575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:22 smithi104 ceph-mon[36624]: pgmap v21575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:22 smithi081 ceph-mon[31453]: pgmap v21575: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:23.545+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:32:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:23.545+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:32:23.545 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:23.545+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:32:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:24 smithi153 ceph-mon[36569]: pgmap v21576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:24 smithi104 ceph-mon[36624]: pgmap v21576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:24 smithi081 ceph-mon[31453]: pgmap v21576: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:26 smithi153 ceph-mon[36569]: pgmap v21577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:27.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:26 smithi104 ceph-mon[36624]: pgmap v21577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:26 smithi081 ceph-mon[31453]: pgmap v21577: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:28 smithi153 ceph-mon[36569]: pgmap v21578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:29.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:28 smithi104 ceph-mon[36624]: pgmap v21578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:28 smithi081 ceph-mon[31453]: pgmap v21578: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:30 smithi153 ceph-mon[36569]: pgmap v21579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:30 smithi104 ceph-mon[36624]: pgmap v21579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:31.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:30 smithi081 ceph-mon[31453]: pgmap v21579: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:32 smithi153 ceph-mon[36569]: pgmap v21580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:32 smithi104 ceph-mon[36624]: pgmap v21580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:32 smithi081 ceph-mon[31453]: pgmap v21580: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:34 smithi153 ceph-mon[36569]: pgmap v21581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:35.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:34 smithi104 ceph-mon[36624]: pgmap v21581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:35.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:34 smithi081 ceph-mon[31453]: pgmap v21581: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:36 smithi153 ceph-mon[36569]: pgmap v21582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:36 smithi104 ceph-mon[36624]: pgmap v21582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:36 smithi081 ceph-mon[31453]: pgmap v21582: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:37 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:32:38.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:37 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:32:38.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:37 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:32:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:38.560+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:32:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:38.560+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:32:38.560 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:38.560+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:38 smithi153 ceph-mon[36569]: pgmap v21583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:38 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:38 smithi104 ceph-mon[36624]: pgmap v21583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:38 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:38 smithi081 ceph-mon[31453]: pgmap v21583: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:39.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:38 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:40 smithi153 ceph-mon[36569]: pgmap v21584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:40 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:40 smithi104 ceph-mon[36624]: pgmap v21584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:40 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:40 smithi081 ceph-mon[31453]: pgmap v21584: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:41.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:40 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:42 smithi153 ceph-mon[36569]: pgmap v21585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:42 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:42 smithi104 ceph-mon[36624]: pgmap v21585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:43.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:42 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:42 smithi081 ceph-mon[31453]: pgmap v21585: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:43.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:42 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:44 smithi153 ceph-mon[36569]: pgmap v21586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:32:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:32:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:44 smithi153 ceph-mon[36569]: pgmap v21587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:44 smithi153 ceph-mon[36569]: pgmap v21588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:44 smithi104 ceph-mon[36624]: pgmap v21586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:32:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:32:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:44 smithi104 ceph-mon[36624]: pgmap v21587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:44 smithi104 ceph-mon[36624]: pgmap v21588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:44 smithi081 ceph-mon[31453]: pgmap v21586: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:32:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:32:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:44 smithi081 ceph-mon[31453]: pgmap v21587: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:44 smithi081 ceph-mon[31453]: pgmap v21588: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:45.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:32:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:46 smithi153 ceph-mon[36569]: pgmap v21589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:47.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:46 smithi104 ceph-mon[36624]: pgmap v21589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:46 smithi081 ceph-mon[31453]: pgmap v21589: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:48 smithi104 ceph-mon[36624]: pgmap v21590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:48 smithi081 ceph-mon[31453]: pgmap v21590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:48 smithi153 ceph-mon[36569]: pgmap v21590: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:49.423+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:32:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:49.423+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:32:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:49.423+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 312 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:32:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:49.423+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 312 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:32:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:50 smithi104 ceph-mon[36624]: pgmap v21591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:50 smithi081 ceph-mon[31453]: pgmap v21591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:50 smithi153 ceph-mon[36569]: pgmap v21591: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:52 smithi104 ceph-mon[36624]: pgmap v21592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:53.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:52 smithi081 ceph-mon[31453]: pgmap v21592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:52 smithi153 ceph-mon[36569]: pgmap v21592: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:53.572+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:32:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:53.572+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:32:53.572 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:32:53.572+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:32:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:54 smithi104 ceph-mon[36624]: pgmap v21593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:54 smithi081 ceph-mon[31453]: pgmap v21593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:54 smithi153 ceph-mon[36569]: pgmap v21593: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:57.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:56 smithi104 ceph-mon[36624]: pgmap v21594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:56 smithi081 ceph-mon[31453]: pgmap v21594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:56 smithi153 ceph-mon[36569]: pgmap v21594: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:59.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:32:58 smithi153 ceph-mon[36569]: pgmap v21595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:32:58 smithi081 ceph-mon[31453]: pgmap v21595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:32:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:32:58 smithi104 ceph-mon[36624]: pgmap v21595: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:00 smithi081 ceph-mon[31453]: pgmap v21596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:00 smithi153 ceph-mon[36569]: pgmap v21596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:01.334 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:00 smithi104 ceph-mon[36624]: pgmap v21596: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:02 smithi081 ceph-mon[31453]: pgmap v21597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:02 smithi153 ceph-mon[36569]: pgmap v21597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:03.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:02 smithi104 ceph-mon[36624]: pgmap v21597: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:04 smithi081 ceph-mon[31453]: pgmap v21598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:04 smithi153 ceph-mon[36569]: pgmap v21598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:04 smithi104 ceph-mon[36624]: pgmap v21598: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:06 smithi081 ceph-mon[31453]: pgmap v21599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:06 smithi153 ceph-mon[36569]: pgmap v21599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:07.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:06 smithi104 ceph-mon[36624]: pgmap v21599: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:08.584+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:33:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:08.584+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:33:08.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:08.584+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:33:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:08 smithi081 ceph-mon[31453]: pgmap v21600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:08 smithi153 ceph-mon[36569]: pgmap v21600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:08 smithi104 ceph-mon[36624]: pgmap v21600: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:10 smithi081 ceph-mon[31453]: pgmap v21601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:10 smithi153 ceph-mon[36569]: pgmap v21601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:10 smithi104 ceph-mon[36624]: pgmap v21601: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:13.133 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:12 smithi104 ceph-mon[36624]: pgmap v21602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:12 smithi081 ceph-mon[31453]: pgmap v21602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:12 smithi153 ceph-mon[36569]: pgmap v21602: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:15.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:14 smithi104 ceph-mon[36624]: pgmap v21603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:14 smithi081 ceph-mon[31453]: pgmap v21603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:14 smithi153 ceph-mon[36569]: pgmap v21603: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:16 smithi081 ceph-mon[31453]: pgmap v21604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:16 smithi153 ceph-mon[36569]: pgmap v21604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:16 smithi104 ceph-mon[36624]: pgmap v21604: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:18 smithi081 ceph-mon[31453]: pgmap v21605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:18 smithi153 ceph-mon[36569]: pgmap v21605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:19.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:18 smithi104 ceph-mon[36624]: pgmap v21605: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:20 smithi081 ceph-mon[31453]: pgmap v21606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:20 smithi153 ceph-mon[36569]: pgmap v21606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:21.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:20 smithi104 ceph-mon[36624]: pgmap v21606: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:23.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:22 smithi081 ceph-mon[31453]: pgmap v21607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:22 smithi153 ceph-mon[36569]: pgmap v21607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:23.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:22 smithi104 ceph-mon[36624]: pgmap v21607: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:23.598+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:33:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:23.598+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:33:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:23.598+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:33:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:24 smithi081 ceph-mon[31453]: pgmap v21608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:25.164 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:24 smithi104 ceph-mon[36624]: pgmap v21608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:24 smithi153 ceph-mon[36569]: pgmap v21608: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:26 smithi081 ceph-mon[31453]: pgmap v21609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:26 smithi153 ceph-mon[36569]: pgmap v21609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:26 smithi104 ceph-mon[36624]: pgmap v21609: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:28 smithi153 ceph-mon[36569]: pgmap v21610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:28 smithi104 ceph-mon[36624]: pgmap v21610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:28 smithi081 ceph-mon[31453]: pgmap v21610: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:31.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:30 smithi081 ceph-mon[31453]: pgmap v21611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:30 smithi153 ceph-mon[36569]: pgmap v21611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:30 smithi104 ceph-mon[36624]: pgmap v21611: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:32 smithi153 ceph-mon[36569]: pgmap v21612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:33.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:32 smithi104 ceph-mon[36624]: pgmap v21612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:32 smithi081 ceph-mon[31453]: pgmap v21612: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:34 smithi153 ceph-mon[36569]: pgmap v21613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:34 smithi104 ceph-mon[36624]: pgmap v21613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:34 smithi081 ceph-mon[31453]: pgmap v21613: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:36 smithi153 ceph-mon[36569]: pgmap v21614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:37.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:36 smithi104 ceph-mon[36624]: pgmap v21614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:36 smithi081 ceph-mon[31453]: pgmap v21614: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:38.607+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:33:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:38.607+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:33:38.608 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:38.607+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:33:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:38 smithi153 ceph-mon[36569]: pgmap v21615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:38 smithi104 ceph-mon[36624]: pgmap v21615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:38 smithi081 ceph-mon[31453]: pgmap v21615: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:40 smithi153 ceph-mon[36569]: pgmap v21616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:41.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:40 smithi104 ceph-mon[36624]: pgmap v21616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:40 smithi081 ceph-mon[31453]: pgmap v21616: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:42 smithi153 ceph-mon[36569]: pgmap v21617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:42 smithi104 ceph-mon[36624]: pgmap v21617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:42 smithi081 ceph-mon[31453]: pgmap v21617: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:44 smithi153 ceph-mon[36569]: pgmap v21618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:33:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:45.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:44 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:44 smithi104 ceph-mon[36624]: pgmap v21618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:33:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:44 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:44 smithi081 ceph-mon[31453]: pgmap v21618: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:33:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:44 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:33:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:33:46.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:45 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:33:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:33:46.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:45 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:46.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:33:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:33:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:45 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:33:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:46 smithi153 ceph-mon[36569]: pgmap v21619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:46 smithi153 ceph-mon[36569]: pgmap v21620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:46 smithi104 ceph-mon[36624]: pgmap v21619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:47.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:46 smithi104 ceph-mon[36624]: pgmap v21620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:46 smithi081 ceph-mon[31453]: pgmap v21619: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:46 smithi081 ceph-mon[31453]: pgmap v21620: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:47 smithi153 ceph-mon[36569]: pgmap v21621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:48.335 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:47 smithi104 ceph-mon[36624]: pgmap v21621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:47 smithi081 ceph-mon[31453]: pgmap v21621: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:50 smithi153 ceph-mon[36569]: pgmap v21622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:50.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:50 smithi104 ceph-mon[36624]: pgmap v21622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:50.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:50 smithi081 ceph-mon[31453]: pgmap v21622: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:52 smithi153 ceph-mon[36569]: pgmap v21623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:52 smithi104 ceph-mon[36624]: pgmap v21623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:52.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:52 smithi081 ceph-mon[31453]: pgmap v21623: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:53.623+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:33:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:53.623+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:33:53.623 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:33:53.623+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:33:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:54 smithi153 ceph-mon[36569]: pgmap v21624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:54 smithi104 ceph-mon[36624]: pgmap v21624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:54.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:54 smithi081 ceph-mon[31453]: pgmap v21624: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:56 smithi153 ceph-mon[36569]: pgmap v21625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:56 smithi104 ceph-mon[36624]: pgmap v21625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:56.872 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:56 smithi081 ceph-mon[31453]: pgmap v21625: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:33:58 smithi153 ceph-mon[36569]: pgmap v21626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:58.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:33:58 smithi104 ceph-mon[36624]: pgmap v21626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:33:58.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:33:58 smithi081 ceph-mon[31453]: pgmap v21626: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:00.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:00 smithi153 ceph-mon[36569]: pgmap v21627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:00.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:00 smithi104 ceph-mon[36624]: pgmap v21627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:00.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:00 smithi081 ceph-mon[31453]: pgmap v21627: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:02.708 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:02 smithi081 ceph-mon[31453]: pgmap v21628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:02.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:02 smithi153 ceph-mon[36569]: pgmap v21628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:02.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:02 smithi104 ceph-mon[36624]: pgmap v21628: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:04.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:04 smithi153 ceph-mon[36569]: pgmap v21629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:04.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:04 smithi104 ceph-mon[36624]: pgmap v21629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:04.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:04 smithi081 ceph-mon[31453]: pgmap v21629: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:06.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:06 smithi153 ceph-mon[36569]: pgmap v21630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:06 smithi104 ceph-mon[36624]: pgmap v21630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:06 smithi081 ceph-mon[31453]: pgmap v21630: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:08.629+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:34:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:08.629+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:34:08.629 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:08.629+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:34:08.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:08 smithi153 ceph-mon[36569]: pgmap v21631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:08.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:08 smithi104 ceph-mon[36624]: pgmap v21631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:08 smithi081 ceph-mon[31453]: pgmap v21631: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:10.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:10 smithi153 ceph-mon[36569]: pgmap v21632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:10.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:10 smithi104 ceph-mon[36624]: pgmap v21632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:10 smithi081 ceph-mon[31453]: pgmap v21632: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:12 smithi081 ceph-mon[31453]: pgmap v21633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:12.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:12 smithi153 ceph-mon[36569]: pgmap v21633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:12.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:12 smithi104 ceph-mon[36624]: pgmap v21633: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:14.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:14 smithi153 ceph-mon[36569]: pgmap v21634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:14.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:14 smithi104 ceph-mon[36624]: pgmap v21634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:14 smithi081 ceph-mon[31453]: pgmap v21634: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:16.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:16 smithi153 ceph-mon[36569]: pgmap v21635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:16.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:16 smithi104 ceph-mon[36624]: pgmap v21635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:16 smithi081 ceph-mon[31453]: pgmap v21635: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:18 smithi153 ceph-mon[36569]: pgmap v21636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:18 smithi104 ceph-mon[36624]: pgmap v21636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:18 smithi081 ceph-mon[31453]: pgmap v21636: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:19.437 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:19.438+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:34:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:19.438+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:34:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:19.439+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 313 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:34:19.438 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:19.439+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 313 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:34:20.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:20 smithi153 ceph-mon[36569]: pgmap v21637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:20.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:20 smithi104 ceph-mon[36624]: pgmap v21637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:20 smithi081 ceph-mon[31453]: pgmap v21637: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:22.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:22 smithi153 ceph-mon[36569]: pgmap v21638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:22.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:22 smithi104 ceph-mon[36624]: pgmap v21638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:22 smithi081 ceph-mon[31453]: pgmap v21638: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:23.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:23.634+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:34:23.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:23.634+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:34:23.634 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:23.634+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:34:24.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:24 smithi153 ceph-mon[36569]: pgmap v21639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:24.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:24 smithi104 ceph-mon[36624]: pgmap v21639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:24 smithi081 ceph-mon[31453]: pgmap v21639: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:26.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:26 smithi153 ceph-mon[36569]: pgmap v21640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:26.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:26 smithi104 ceph-mon[36624]: pgmap v21640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:26.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:26 smithi081 ceph-mon[31453]: pgmap v21640: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:28.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:28 smithi153 ceph-mon[36569]: pgmap v21641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:28.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:28 smithi104 ceph-mon[36624]: pgmap v21641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:28.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:28 smithi081 ceph-mon[31453]: pgmap v21641: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:30.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:30 smithi153 ceph-mon[36569]: pgmap v21642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:30.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:30 smithi104 ceph-mon[36624]: pgmap v21642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:30.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:30 smithi081 ceph-mon[31453]: pgmap v21642: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:32.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:32 smithi153 ceph-mon[36569]: pgmap v21643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:32.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:32 smithi104 ceph-mon[36624]: pgmap v21643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:32.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:32 smithi081 ceph-mon[31453]: pgmap v21643: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:34.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:34 smithi153 ceph-mon[36569]: pgmap v21644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:34.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:34 smithi104 ceph-mon[36624]: pgmap v21644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:34.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:34 smithi081 ceph-mon[31453]: pgmap v21644: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:36.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:36 smithi153 ceph-mon[36569]: pgmap v21645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:36.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:36 smithi104 ceph-mon[36624]: pgmap v21645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:36.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:36 smithi081 ceph-mon[31453]: pgmap v21645: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:38.649 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:38.649+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:34:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:38.649+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:34:38.650 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:38.649+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:34:38.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:38 smithi153 ceph-mon[36569]: pgmap v21646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:38.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:38 smithi104 ceph-mon[36624]: pgmap v21646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:38.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:38 smithi081 ceph-mon[31453]: pgmap v21646: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:40.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:40 smithi153 ceph-mon[36569]: pgmap v21647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:40.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:40 smithi104 ceph-mon[36624]: pgmap v21647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:40.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:40 smithi081 ceph-mon[31453]: pgmap v21647: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:42 smithi153 ceph-mon[36569]: pgmap v21648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:42.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:42 smithi104 ceph-mon[36624]: pgmap v21648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:42.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:42 smithi081 ceph-mon[31453]: pgmap v21648: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:44 smithi153 ceph-mon[36569]: pgmap v21649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:44.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:44 smithi104 ceph-mon[36624]: pgmap v21649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:44.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:44 smithi081 ceph-mon[31453]: pgmap v21649: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:46 smithi153 ceph-mon[36569]: pgmap v21650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:46 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:34:46.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:46 smithi104 ceph-mon[36624]: pgmap v21650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:46.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:46 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:34:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:46 smithi081 ceph-mon[31453]: pgmap v21650: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:46.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:46 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:34:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:34:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:34:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:47 smithi104 ceph-mon[36624]: pgmap v21651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:47 smithi104 ceph-mon[36624]: pgmap v21652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:47.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:47 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:34:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:34:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:34:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:47 smithi081 ceph-mon[31453]: pgmap v21651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:47 smithi081 ceph-mon[31453]: pgmap v21652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:47.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:47 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:34:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:34:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:34:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:47 smithi153 ceph-mon[36569]: pgmap v21651: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:47 smithi153 ceph-mon[36569]: pgmap v21652: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:47 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:34:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:49 smithi153 ceph-mon[36569]: pgmap v21653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:50.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:49 smithi104 ceph-mon[36624]: pgmap v21653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:49 smithi081 ceph-mon[31453]: pgmap v21653: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:51 smithi153 ceph-mon[36569]: pgmap v21654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:51 smithi104 ceph-mon[36624]: pgmap v21654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:51 smithi081 ceph-mon[31453]: pgmap v21654: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:53 smithi153 ceph-mon[36569]: pgmap v21655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:53 smithi104 ceph-mon[36624]: pgmap v21655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:53.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:53 smithi081 ceph-mon[31453]: pgmap v21655: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:53.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:53.665+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:34:53.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:53.665+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:34:53.665 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:34:53.665+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:34:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:55 smithi153 ceph-mon[36569]: pgmap v21656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:56.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:55 smithi104 ceph-mon[36624]: pgmap v21656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:55 smithi081 ceph-mon[31453]: pgmap v21656: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:57 smithi153 ceph-mon[36569]: pgmap v21657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:58.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:57 smithi104 ceph-mon[36624]: pgmap v21657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:58.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:57 smithi081 ceph-mon[31453]: pgmap v21657: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:34:59 smithi153 ceph-mon[36569]: pgmap v21658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:59.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:34:59 smithi104 ceph-mon[36624]: pgmap v21658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:34:59.336 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:34:59 smithi081 ceph-mon[31453]: pgmap v21658: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:01 smithi153 ceph-mon[36569]: pgmap v21659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:02.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:01 smithi104 ceph-mon[36624]: pgmap v21659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:01 smithi081 ceph-mon[31453]: pgmap v21659: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:03 smithi153 ceph-mon[36569]: pgmap v21660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:04.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:03 smithi104 ceph-mon[36624]: pgmap v21660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:04.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:03 smithi081 ceph-mon[31453]: pgmap v21660: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:04 smithi153 ceph-mon[36569]: pgmap v21661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:04 smithi104 ceph-mon[36624]: pgmap v21661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:05.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:04 smithi081 ceph-mon[31453]: pgmap v21661: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:07 smithi153 ceph-mon[36569]: pgmap v21662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:08.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:07 smithi104 ceph-mon[36624]: pgmap v21662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:08.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:07 smithi081 ceph-mon[31453]: pgmap v21662: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:08.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:08.671+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:35:08.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:08.671+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:35:08.671 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:08.671+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:35:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:09 smithi153 ceph-mon[36569]: pgmap v21663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:09 smithi104 ceph-mon[36624]: pgmap v21663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:09 smithi081 ceph-mon[31453]: pgmap v21663: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:11 smithi153 ceph-mon[36569]: pgmap v21664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:12.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:11 smithi104 ceph-mon[36624]: pgmap v21664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:12.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:11 smithi081 ceph-mon[31453]: pgmap v21664: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:13 smithi153 ceph-mon[36569]: pgmap v21665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:13.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:13 smithi104 ceph-mon[36624]: pgmap v21665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:13 smithi081 ceph-mon[31453]: pgmap v21665: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:15 smithi153 ceph-mon[36569]: pgmap v21666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:15 smithi104 ceph-mon[36624]: pgmap v21666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:16.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:15 smithi081 ceph-mon[31453]: pgmap v21666: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:16 smithi153 ceph-mon[36569]: pgmap v21667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:17.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:16 smithi104 ceph-mon[36624]: pgmap v21667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:16 smithi081 ceph-mon[31453]: pgmap v21667: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:19 smithi153 ceph-mon[36569]: pgmap v21668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:19.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:19 smithi104 ceph-mon[36624]: pgmap v21668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:19 smithi081 ceph-mon[31453]: pgmap v21668: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:21.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:21 smithi153 ceph-mon[36569]: pgmap v21669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:21.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:21 smithi104 ceph-mon[36624]: pgmap v21669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:21 smithi081 ceph-mon[31453]: pgmap v21669: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:23 smithi153 ceph-mon[36569]: pgmap v21670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:23 smithi104 ceph-mon[36624]: pgmap v21670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:23.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:23 smithi081 ceph-mon[31453]: pgmap v21670: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:23.686+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:35:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:23.686+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:35:23.687 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:23.686+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:35:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:25 smithi153 ceph-mon[36569]: pgmap v21671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:25 smithi104 ceph-mon[36624]: pgmap v21671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:25 smithi081 ceph-mon[31453]: pgmap v21671: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:27 smithi153 ceph-mon[36569]: pgmap v21672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:27 smithi104 ceph-mon[36624]: pgmap v21672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:27 smithi081 ceph-mon[31453]: pgmap v21672: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:29 smithi153 ceph-mon[36569]: pgmap v21673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:29 smithi104 ceph-mon[36624]: pgmap v21673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:29.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:29 smithi081 ceph-mon[31453]: pgmap v21673: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:31 smithi153 ceph-mon[36569]: pgmap v21674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:31 smithi104 ceph-mon[36624]: pgmap v21674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:31 smithi081 ceph-mon[31453]: pgmap v21674: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:33 smithi153 ceph-mon[36569]: pgmap v21675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:33 smithi104 ceph-mon[36624]: pgmap v21675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:33 smithi081 ceph-mon[31453]: pgmap v21675: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:35 smithi153 ceph-mon[36569]: pgmap v21676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:35.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:35 smithi104 ceph-mon[36624]: pgmap v21676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:35 smithi081 ceph-mon[31453]: pgmap v21676: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:37 smithi153 ceph-mon[36569]: pgmap v21677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:37 smithi104 ceph-mon[36624]: pgmap v21677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:37 smithi081 ceph-mon[31453]: pgmap v21677: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:38.700+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:35:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:38.700+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:35:38.700 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:38.700+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:35:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:39 smithi153 ceph-mon[36569]: pgmap v21678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:39 smithi104 ceph-mon[36624]: pgmap v21678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:39 smithi081 ceph-mon[31453]: pgmap v21678: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:41 smithi153 ceph-mon[36569]: pgmap v21679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:41.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:41 smithi104 ceph-mon[36624]: pgmap v21679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:41 smithi081 ceph-mon[31453]: pgmap v21679: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:43 smithi153 ceph-mon[36569]: pgmap v21680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:43 smithi104 ceph-mon[36624]: pgmap v21680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:43 smithi081 ceph-mon[31453]: pgmap v21680: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:45 smithi153 ceph-mon[36569]: pgmap v21681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:45 smithi104 ceph-mon[36624]: pgmap v21681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:45 smithi081 ceph-mon[31453]: pgmap v21681: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:47 smithi153 ceph-mon[36569]: pgmap v21682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:47 smithi104 ceph-mon[36624]: pgmap v21682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:47 smithi081 ceph-mon[31453]: pgmap v21682: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:48 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:35:48.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:48 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:35:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:48 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:35:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:35:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:35:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:49 smithi153 ceph-mon[36569]: pgmap v21683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:49 smithi153 ceph-mon[36569]: pgmap v21684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:49.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:35:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:35:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:35:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:49 smithi104 ceph-mon[36624]: pgmap v21683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:49 smithi104 ceph-mon[36624]: pgmap v21684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:35:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:35:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:35:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:49 smithi081 ceph-mon[31453]: pgmap v21683: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:49 smithi081 ceph-mon[31453]: pgmap v21684: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:49.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:35:51.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:51 smithi081 ceph-mon[31453]: pgmap v21685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:51 smithi153 ceph-mon[36569]: pgmap v21685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:52.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:51 smithi104 ceph-mon[36624]: pgmap v21685: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:53.711+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:35:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:53.711+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:35:53.711 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:35:53.711+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:35:53.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:53 smithi081 ceph-mon[31453]: pgmap v21686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:53 smithi153 ceph-mon[36569]: pgmap v21686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:54.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:53 smithi104 ceph-mon[36624]: pgmap v21686: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:55.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:55 smithi081 ceph-mon[31453]: pgmap v21687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:55 smithi153 ceph-mon[36569]: pgmap v21687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:56.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:55 smithi104 ceph-mon[36624]: pgmap v21687: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:57.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:57 smithi081 ceph-mon[31453]: pgmap v21688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:57 smithi153 ceph-mon[36569]: pgmap v21688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:57 smithi104 ceph-mon[36624]: pgmap v21688: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:35:59.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:35:59 smithi081 ceph-mon[31453]: pgmap v21689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:35:59 smithi153 ceph-mon[36569]: pgmap v21689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:35:59 smithi104 ceph-mon[36624]: pgmap v21689: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:01.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:01 smithi081 ceph-mon[31453]: pgmap v21690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:01 smithi153 ceph-mon[36569]: pgmap v21690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:02.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:01 smithi104 ceph-mon[36624]: pgmap v21690: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:03 smithi081 ceph-mon[31453]: pgmap v21691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:03 smithi153 ceph-mon[36569]: pgmap v21691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:04.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:03 smithi104 ceph-mon[36624]: pgmap v21691: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:05.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:05 smithi081 ceph-mon[31453]: pgmap v21692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:05 smithi153 ceph-mon[36569]: pgmap v21692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:06.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:05 smithi104 ceph-mon[36624]: pgmap v21692: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:07.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:07 smithi081 ceph-mon[31453]: pgmap v21693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:07 smithi153 ceph-mon[36569]: pgmap v21693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:08.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:07 smithi104 ceph-mon[36624]: pgmap v21693: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:08.722+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:36:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:08.722+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:36:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:08.722+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:36:09.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:09 smithi081 ceph-mon[31453]: pgmap v21694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:09 smithi153 ceph-mon[36569]: pgmap v21694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:09 smithi104 ceph-mon[36624]: pgmap v21694: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:11.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:11 smithi081 ceph-mon[31453]: pgmap v21695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:11 smithi153 ceph-mon[36569]: pgmap v21695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:12.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:11 smithi104 ceph-mon[36624]: pgmap v21695: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:13 smithi153 ceph-mon[36569]: pgmap v21696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:14.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:13 smithi104 ceph-mon[36624]: pgmap v21696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:14.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:13 smithi081 ceph-mon[31453]: pgmap v21696: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:15 smithi153 ceph-mon[36569]: pgmap v21697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:16.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:15 smithi104 ceph-mon[36624]: pgmap v21697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:16.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:15 smithi081 ceph-mon[31453]: pgmap v21697: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:17 smithi153 ceph-mon[36569]: pgmap v21698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:18.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:17 smithi104 ceph-mon[36624]: pgmap v21698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:18.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:17 smithi081 ceph-mon[31453]: pgmap v21698: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:19 smithi153 ceph-mon[36569]: pgmap v21699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:19 smithi104 ceph-mon[36624]: pgmap v21699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:20.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:19 smithi081 ceph-mon[31453]: pgmap v21699: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:21 smithi153 ceph-mon[36569]: pgmap v21700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:22.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:21 smithi104 ceph-mon[36624]: pgmap v21700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:22.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:21 smithi081 ceph-mon[31453]: pgmap v21700: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:23.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:23.739+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:36:23.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:23.739+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:36:23.739 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:23.739+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:36:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:23 smithi153 ceph-mon[36569]: pgmap v21701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:23 smithi104 ceph-mon[36624]: pgmap v21701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:24.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:23 smithi081 ceph-mon[31453]: pgmap v21701: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:25 smithi153 ceph-mon[36569]: pgmap v21702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:25 smithi104 ceph-mon[36624]: pgmap v21702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:25 smithi081 ceph-mon[31453]: pgmap v21702: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:27 smithi153 ceph-mon[36569]: pgmap v21703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:28.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:27 smithi104 ceph-mon[36624]: pgmap v21703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:27 smithi081 ceph-mon[31453]: pgmap v21703: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:29 smithi153 ceph-mon[36569]: pgmap v21704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:29 smithi104 ceph-mon[36624]: pgmap v21704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:29 smithi081 ceph-mon[31453]: pgmap v21704: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:31 smithi153 ceph-mon[36569]: pgmap v21705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:32.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:31 smithi104 ceph-mon[36624]: pgmap v21705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:32.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:31 smithi081 ceph-mon[31453]: pgmap v21705: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:33.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:33 smithi153 ceph-mon[36569]: pgmap v21706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:33.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:33 smithi104 ceph-mon[36624]: pgmap v21706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:33.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:33 smithi081 ceph-mon[31453]: pgmap v21706: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:35.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:35 smithi081 ceph-mon[31453]: pgmap v21707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:35 smithi153 ceph-mon[36569]: pgmap v21707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:36.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:35 smithi104 ceph-mon[36624]: pgmap v21707: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:37.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:37 smithi081 ceph-mon[31453]: pgmap v21708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:37 smithi153 ceph-mon[36569]: pgmap v21708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:38.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:37 smithi104 ceph-mon[36624]: pgmap v21708: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:38.749+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:36:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:38.749+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:36:38.749 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:38.749+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:36:39.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:39 smithi081 ceph-mon[31453]: pgmap v21709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:39 smithi153 ceph-mon[36569]: pgmap v21709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:40.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:39 smithi104 ceph-mon[36624]: pgmap v21709: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:41.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:41 smithi081 ceph-mon[31453]: pgmap v21710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:41 smithi153 ceph-mon[36569]: pgmap v21710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:42.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:41 smithi104 ceph-mon[36624]: pgmap v21710: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:43.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:43 smithi081 ceph-mon[31453]: pgmap v21711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:44.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:43 smithi153 ceph-mon[36569]: pgmap v21711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:44.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:43 smithi104 ceph-mon[36624]: pgmap v21711: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:45.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:45 smithi081 ceph-mon[31453]: pgmap v21712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:46.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:45 smithi153 ceph-mon[36569]: pgmap v21712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:45 smithi104 ceph-mon[36624]: pgmap v21712: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:48.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:47 smithi153 ceph-mon[36569]: pgmap v21713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:48.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:47 smithi104 ceph-mon[36624]: pgmap v21713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:47 smithi081 ceph-mon[31453]: pgmap v21713: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:49 smithi153 ceph-mon[36569]: pgmap v21714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:50.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:49 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:36:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:49 smithi104 ceph-mon[36624]: pgmap v21714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:50.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:49 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:36:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:49 smithi081 ceph-mon[31453]: pgmap v21714: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:49 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:36:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:36:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:36:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:36:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:36:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:36:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:36:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:36:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:36:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:36:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:51 smithi153 ceph-mon[36569]: pgmap v21715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:52.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:51 smithi153 ceph-mon[36569]: pgmap v21716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:52.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:51 smithi104 ceph-mon[36624]: pgmap v21715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:52.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:51 smithi104 ceph-mon[36624]: pgmap v21716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:52.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:51 smithi081 ceph-mon[31453]: pgmap v21715: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:52.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:51 smithi081 ceph-mon[31453]: pgmap v21716: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:53.763 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:53.764+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:36:53.764 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:53.764+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:36:53.764 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:36:53.764+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:36:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:53 smithi153 ceph-mon[36569]: pgmap v21717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:54.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:53 smithi104 ceph-mon[36624]: pgmap v21717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:54.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:53 smithi081 ceph-mon[31453]: pgmap v21717: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:56.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:55 smithi153 ceph-mon[36569]: pgmap v21718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:56.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:55 smithi104 ceph-mon[36624]: pgmap v21718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:56.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:55 smithi081 ceph-mon[31453]: pgmap v21718: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:58.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:57 smithi153 ceph-mon[36569]: pgmap v21719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:58.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:57 smithi104 ceph-mon[36624]: pgmap v21719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:36:58.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:57 smithi081 ceph-mon[31453]: pgmap v21719: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:00.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:36:59 smithi153 ceph-mon[36569]: pgmap v21720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:00.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:36:59 smithi104 ceph-mon[36624]: pgmap v21720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:00.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:36:59 smithi081 ceph-mon[31453]: pgmap v21720: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:02.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:01 smithi153 ceph-mon[36569]: pgmap v21721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:02.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:01 smithi104 ceph-mon[36624]: pgmap v21721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:01 smithi081 ceph-mon[31453]: pgmap v21721: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:04.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:03 smithi153 ceph-mon[36569]: pgmap v21722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:04.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:03 smithi104 ceph-mon[36624]: pgmap v21722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:04.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:03 smithi081 ceph-mon[31453]: pgmap v21722: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:06.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:05 smithi153 ceph-mon[36569]: pgmap v21723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:06.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:05 smithi104 ceph-mon[36624]: pgmap v21723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:05 smithi081 ceph-mon[31453]: pgmap v21723: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:08.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:07 smithi153 ceph-mon[36569]: pgmap v21724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:08.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:07 smithi104 ceph-mon[36624]: pgmap v21724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:07 smithi081 ceph-mon[31453]: pgmap v21724: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:08.777+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:37:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:08.777+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:37:08.778 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:08.777+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:37:10.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:09 smithi153 ceph-mon[36569]: pgmap v21725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:10.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:09 smithi104 ceph-mon[36624]: pgmap v21725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:09 smithi081 ceph-mon[31453]: pgmap v21725: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:12.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:11 smithi153 ceph-mon[36569]: pgmap v21726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:12.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:11 smithi104 ceph-mon[36624]: pgmap v21726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:11 smithi081 ceph-mon[31453]: pgmap v21726: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:14.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:13 smithi153 ceph-mon[36569]: pgmap v21727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:14.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:13 smithi104 ceph-mon[36624]: pgmap v21727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:14.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:13 smithi081 ceph-mon[31453]: pgmap v21727: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:16.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:15 smithi153 ceph-mon[36569]: pgmap v21728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:16.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:15 smithi104 ceph-mon[36624]: pgmap v21728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:16.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:15 smithi081 ceph-mon[31453]: pgmap v21728: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:17 smithi153 ceph-mon[36569]: pgmap v21729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:17 smithi104 ceph-mon[36624]: pgmap v21729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:18.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:17 smithi081 ceph-mon[31453]: pgmap v21729: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:19.466 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:19.467+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:37:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:19.467+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:37:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:19.468+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 314 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:37:19.467 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:19.468+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 314 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:37:20.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:19 smithi153 ceph-mon[36569]: pgmap v21730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:20.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:19 smithi104 ceph-mon[36624]: pgmap v21730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:20.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:19 smithi081 ceph-mon[31453]: pgmap v21730: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:22.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:21 smithi153 ceph-mon[36569]: pgmap v21731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:22.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:21 smithi104 ceph-mon[36624]: pgmap v21731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:21 smithi081 ceph-mon[31453]: pgmap v21731: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:23.793 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:23.793+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:37:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:23.794+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:37:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:23.794+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:37:24.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:23 smithi153 ceph-mon[36569]: pgmap v21732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:23 smithi104 ceph-mon[36624]: pgmap v21732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:24.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:23 smithi081 ceph-mon[31453]: pgmap v21732: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:26.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:25 smithi153 ceph-mon[36569]: pgmap v21733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:26.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:25 smithi104 ceph-mon[36624]: pgmap v21733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:25 smithi081 ceph-mon[31453]: pgmap v21733: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:28.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:27 smithi153 ceph-mon[36569]: pgmap v21734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:28.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:27 smithi104 ceph-mon[36624]: pgmap v21734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:27 smithi081 ceph-mon[31453]: pgmap v21734: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:30.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:29 smithi153 ceph-mon[36569]: pgmap v21735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:30.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:29 smithi104 ceph-mon[36624]: pgmap v21735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:29 smithi081 ceph-mon[31453]: pgmap v21735: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:32.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:31 smithi153 ceph-mon[36569]: pgmap v21736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:32.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:31 smithi104 ceph-mon[36624]: pgmap v21736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:32.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:31 smithi081 ceph-mon[31453]: pgmap v21736: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:34.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:33 smithi153 ceph-mon[36569]: pgmap v21737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:34.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:33 smithi104 ceph-mon[36624]: pgmap v21737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:34.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:33 smithi081 ceph-mon[31453]: pgmap v21737: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:36.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:35 smithi153 ceph-mon[36569]: pgmap v21738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:36.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:35 smithi104 ceph-mon[36624]: pgmap v21738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:36.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:35 smithi081 ceph-mon[31453]: pgmap v21738: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:38.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:37 smithi153 ceph-mon[36569]: pgmap v21739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:38.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:37 smithi104 ceph-mon[36624]: pgmap v21739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:38.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:37 smithi081 ceph-mon[31453]: pgmap v21739: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:38.806+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:37:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:38.806+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:37:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:38.806+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:37:40.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:39 smithi153 ceph-mon[36569]: pgmap v21740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:40.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:39 smithi104 ceph-mon[36624]: pgmap v21740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:40.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:39 smithi081 ceph-mon[31453]: pgmap v21740: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:42.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:41 smithi153 ceph-mon[36569]: pgmap v21741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:42.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:41 smithi104 ceph-mon[36624]: pgmap v21741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:42.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:41 smithi081 ceph-mon[31453]: pgmap v21741: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:44.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:43 smithi104 ceph-mon[36624]: pgmap v21742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:44.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:43 smithi081 ceph-mon[31453]: pgmap v21742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:43 smithi153 ceph-mon[36569]: pgmap v21742: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:46.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:45 smithi104 ceph-mon[36624]: pgmap v21743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:45 smithi081 ceph-mon[31453]: pgmap v21743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:45 smithi153 ceph-mon[36569]: pgmap v21743: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:48.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:47 smithi104 ceph-mon[36624]: pgmap v21744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:47 smithi081 ceph-mon[31453]: pgmap v21744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:47 smithi153 ceph-mon[36569]: pgmap v21744: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:50.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:49 smithi104 ceph-mon[36624]: pgmap v21745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:50.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:49 smithi081 ceph-mon[31453]: pgmap v21745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:49 smithi153 ceph-mon[36569]: pgmap v21745: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:50 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:37:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:50 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:37:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:50 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:37:52.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:51 smithi081 ceph-mon[31453]: pgmap v21746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:52.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:37:52.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:37:52.160 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:51 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:37:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:51 smithi153 ceph-mon[36569]: pgmap v21746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:37:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:37:52.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:51 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:37:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:51 smithi104 ceph-mon[36624]: pgmap v21746: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:37:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:37:52.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:51 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:37:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:52 smithi081 ceph-mon[31453]: pgmap v21747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:52 smithi081 ceph-mon[31453]: pgmap v21748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:52 smithi153 ceph-mon[36569]: pgmap v21747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:52 smithi153 ceph-mon[36569]: pgmap v21748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:52 smithi104 ceph-mon[36624]: pgmap v21747: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:52 smithi104 ceph-mon[36624]: pgmap v21748: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:53.813+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:37:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:53.813+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:37:53.814 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:37:53.813+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:37:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:54 smithi081 ceph-mon[31453]: pgmap v21749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:54 smithi153 ceph-mon[36569]: pgmap v21749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:54 smithi104 ceph-mon[36624]: pgmap v21749: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:56 smithi081 ceph-mon[31453]: pgmap v21750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:56 smithi153 ceph-mon[36569]: pgmap v21750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:57.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:56 smithi104 ceph-mon[36624]: pgmap v21750: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:37:58 smithi081 ceph-mon[31453]: pgmap v21751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:37:58 smithi153 ceph-mon[36569]: pgmap v21751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:37:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:37:58 smithi104 ceph-mon[36624]: pgmap v21751: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:01.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:00 smithi081 ceph-mon[31453]: pgmap v21752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:00 smithi153 ceph-mon[36569]: pgmap v21752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:01.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:00 smithi104 ceph-mon[36624]: pgmap v21752: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:03.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:02 smithi081 ceph-mon[31453]: pgmap v21753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:02 smithi153 ceph-mon[36569]: pgmap v21753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:02 smithi104 ceph-mon[36624]: pgmap v21753: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:04 smithi081 ceph-mon[31453]: pgmap v21754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:04 smithi153 ceph-mon[36569]: pgmap v21754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:04 smithi104 ceph-mon[36624]: pgmap v21754: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:06 smithi081 ceph-mon[31453]: pgmap v21755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:06 smithi153 ceph-mon[36569]: pgmap v21755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:06 smithi104 ceph-mon[36624]: pgmap v21755: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:08.816+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:38:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:08.816+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:38:08.816 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:08.816+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:38:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:08 smithi081 ceph-mon[31453]: pgmap v21756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:08 smithi153 ceph-mon[36569]: pgmap v21756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:08 smithi104 ceph-mon[36624]: pgmap v21756: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:10 smithi081 ceph-mon[31453]: pgmap v21757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:10 smithi153 ceph-mon[36569]: pgmap v21757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:11.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:10 smithi104 ceph-mon[36624]: pgmap v21757: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:12 smithi081 ceph-mon[31453]: pgmap v21758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:12 smithi153 ceph-mon[36569]: pgmap v21758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:12 smithi104 ceph-mon[36624]: pgmap v21758: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:14 smithi153 ceph-mon[36569]: pgmap v21759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:15.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:14 smithi104 ceph-mon[36624]: pgmap v21759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:14 smithi081 ceph-mon[31453]: pgmap v21759: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:16 smithi153 ceph-mon[36569]: pgmap v21760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:17.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:16 smithi104 ceph-mon[36624]: pgmap v21760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:16 smithi081 ceph-mon[31453]: pgmap v21760: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:18.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:18 smithi153 ceph-mon[36569]: pgmap v21761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:18.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:18 smithi104 ceph-mon[36624]: pgmap v21761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:18.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:18 smithi081 ceph-mon[31453]: pgmap v21761: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:20 smithi153 ceph-mon[36569]: pgmap v21762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:20 smithi104 ceph-mon[36624]: pgmap v21762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:20 smithi081 ceph-mon[31453]: pgmap v21762: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:22 smithi153 ceph-mon[36569]: pgmap v21763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:22 smithi104 ceph-mon[36624]: pgmap v21763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:22 smithi081 ceph-mon[31453]: pgmap v21763: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:23.832+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:38:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:23.832+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:38:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:23.832+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:38:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:24 smithi153 ceph-mon[36569]: pgmap v21764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:24 smithi104 ceph-mon[36624]: pgmap v21764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:24 smithi081 ceph-mon[31453]: pgmap v21764: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:27.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:26 smithi153 ceph-mon[36569]: pgmap v21765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:26 smithi104 ceph-mon[36624]: pgmap v21765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:26 smithi081 ceph-mon[31453]: pgmap v21765: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:28 smithi153 ceph-mon[36569]: pgmap v21766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:28 smithi104 ceph-mon[36624]: pgmap v21766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:28 smithi081 ceph-mon[31453]: pgmap v21766: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:30 smithi153 ceph-mon[36569]: pgmap v21767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:31.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:30 smithi104 ceph-mon[36624]: pgmap v21767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:30 smithi081 ceph-mon[31453]: pgmap v21767: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:32 smithi153 ceph-mon[36569]: pgmap v21768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:33.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:32 smithi104 ceph-mon[36624]: pgmap v21768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:33.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:32 smithi081 ceph-mon[31453]: pgmap v21768: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:34 smithi153 ceph-mon[36569]: pgmap v21769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:34 smithi104 ceph-mon[36624]: pgmap v21769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:34 smithi081 ceph-mon[31453]: pgmap v21769: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:36 smithi153 ceph-mon[36569]: pgmap v21770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:36 smithi104 ceph-mon[36624]: pgmap v21770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:36 smithi081 ceph-mon[31453]: pgmap v21770: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:38.847+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:38:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:38.847+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:38:38.847 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:38.847+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:38:38.961 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:38 smithi081 ceph-mon[31453]: pgmap v21771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:38 smithi153 ceph-mon[36569]: pgmap v21771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:38 smithi104 ceph-mon[36624]: pgmap v21771: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:40 smithi153 ceph-mon[36569]: pgmap v21772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:40 smithi104 ceph-mon[36624]: pgmap v21772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:40 smithi081 ceph-mon[31453]: pgmap v21772: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:42 smithi153 ceph-mon[36569]: pgmap v21773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:42 smithi104 ceph-mon[36624]: pgmap v21773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:42 smithi081 ceph-mon[31453]: pgmap v21773: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:44 smithi153 ceph-mon[36569]: pgmap v21774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:44 smithi104 ceph-mon[36624]: pgmap v21774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:44 smithi081 ceph-mon[31453]: pgmap v21774: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:46 smithi153 ceph-mon[36569]: pgmap v21775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:46 smithi104 ceph-mon[36624]: pgmap v21775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:46 smithi081 ceph-mon[31453]: pgmap v21775: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:48 smithi153 ceph-mon[36569]: pgmap v21776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:48 smithi104 ceph-mon[36624]: pgmap v21776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:49.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:48 smithi081 ceph-mon[31453]: pgmap v21776: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:50 smithi153 ceph-mon[36569]: pgmap v21777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:51.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:50 smithi104 ceph-mon[36624]: pgmap v21777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:50 smithi081 ceph-mon[31453]: pgmap v21777: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: pgmap v21778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:38:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:52 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: pgmap v21778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:52 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: pgmap v21778: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:38:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:52 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:53.856+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:38:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:53.856+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:38:53.856 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:38:53.856+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:38:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:38:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:38:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:38:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:38:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:38:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:38:54.057 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:38:54.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:38:54.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:38:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:38:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:38:54.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:38:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:38:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:38:54.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:38:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:38:54.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:38:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:54 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:38:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:54 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:38:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:54 smithi153 ceph-mon[36569]: pgmap v21779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:54 smithi153 ceph-mon[36569]: pgmap v21780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:54 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:38:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:54 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:38:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:54 smithi104 ceph-mon[36624]: pgmap v21779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:55.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:54 smithi104 ceph-mon[36624]: pgmap v21780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:54 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:38:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:54 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:38:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:54 smithi081 ceph-mon[31453]: pgmap v21779: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:55.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:54 smithi081 ceph-mon[31453]: pgmap v21780: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:56 smithi153 ceph-mon[36569]: pgmap v21781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:56 smithi104 ceph-mon[36624]: pgmap v21781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:56 smithi081 ceph-mon[31453]: pgmap v21781: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:38:58 smithi153 ceph-mon[36569]: pgmap v21782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:38:58 smithi104 ceph-mon[36624]: pgmap v21782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:38:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:38:58 smithi081 ceph-mon[31453]: pgmap v21782: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:00 smithi153 ceph-mon[36569]: pgmap v21783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:00 smithi104 ceph-mon[36624]: pgmap v21783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:00 smithi081 ceph-mon[31453]: pgmap v21783: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:02 smithi153 ceph-mon[36569]: pgmap v21784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:02 smithi104 ceph-mon[36624]: pgmap v21784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:02 smithi081 ceph-mon[31453]: pgmap v21784: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:04 smithi153 ceph-mon[36569]: pgmap v21785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:04 smithi104 ceph-mon[36624]: pgmap v21785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:04 smithi081 ceph-mon[31453]: pgmap v21785: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:07.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:06 smithi153 ceph-mon[36569]: pgmap v21786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:06 smithi104 ceph-mon[36624]: pgmap v21786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:06 smithi081 ceph-mon[31453]: pgmap v21786: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:08.872+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:39:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:08.872+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:39:08.872 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:08.872+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:39:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:08 smithi153 ceph-mon[36569]: pgmap v21787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:08 smithi104 ceph-mon[36624]: pgmap v21787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:08 smithi081 ceph-mon[31453]: pgmap v21787: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:10 smithi153 ceph-mon[36569]: pgmap v21788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:10 smithi104 ceph-mon[36624]: pgmap v21788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:10 smithi081 ceph-mon[31453]: pgmap v21788: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:12 smithi153 ceph-mon[36569]: pgmap v21789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:12 smithi104 ceph-mon[36624]: pgmap v21789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:12 smithi081 ceph-mon[31453]: pgmap v21789: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:15.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:14 smithi104 ceph-mon[36624]: pgmap v21790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:14 smithi081 ceph-mon[31453]: pgmap v21790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:14 smithi153 ceph-mon[36569]: pgmap v21790: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:16 smithi104 ceph-mon[36624]: pgmap v21791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:16 smithi081 ceph-mon[31453]: pgmap v21791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:16 smithi153 ceph-mon[36569]: pgmap v21791: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:19.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:18 smithi104 ceph-mon[36624]: pgmap v21792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:18 smithi081 ceph-mon[31453]: pgmap v21792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:18 smithi153 ceph-mon[36569]: pgmap v21792: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:20 smithi104 ceph-mon[36624]: pgmap v21793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:20 smithi081 ceph-mon[31453]: pgmap v21793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:20 smithi153 ceph-mon[36569]: pgmap v21793: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:22 smithi104 ceph-mon[36624]: pgmap v21794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:22 smithi081 ceph-mon[31453]: pgmap v21794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:22 smithi153 ceph-mon[36569]: pgmap v21794: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:23.887+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:39:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:23.888+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:39:23.888 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:23.888+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:39:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:24 smithi081 ceph-mon[31453]: pgmap v21795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:24 smithi153 ceph-mon[36569]: pgmap v21795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:24 smithi104 ceph-mon[36624]: pgmap v21795: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:26 smithi081 ceph-mon[31453]: pgmap v21796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:26 smithi153 ceph-mon[36569]: pgmap v21796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:26 smithi104 ceph-mon[36624]: pgmap v21796: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:28 smithi081 ceph-mon[31453]: pgmap v21797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:28 smithi153 ceph-mon[36569]: pgmap v21797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:28 smithi104 ceph-mon[36624]: pgmap v21797: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:30 smithi081 ceph-mon[31453]: pgmap v21798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:30 smithi153 ceph-mon[36569]: pgmap v21798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:30 smithi104 ceph-mon[36624]: pgmap v21798: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:32 smithi081 ceph-mon[31453]: pgmap v21799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:32 smithi153 ceph-mon[36569]: pgmap v21799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:33.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:32 smithi104 ceph-mon[36624]: pgmap v21799: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:34 smithi081 ceph-mon[31453]: pgmap v21800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:34 smithi153 ceph-mon[36569]: pgmap v21800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:35.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:34 smithi104 ceph-mon[36624]: pgmap v21800: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:36 smithi081 ceph-mon[31453]: pgmap v21801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:36 smithi153 ceph-mon[36569]: pgmap v21801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:36 smithi104 ceph-mon[36624]: pgmap v21801: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:38.902+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:39:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:38.902+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:39:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:38.902+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:39:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:38 smithi081 ceph-mon[31453]: pgmap v21802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:38 smithi153 ceph-mon[36569]: pgmap v21802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:38 smithi104 ceph-mon[36624]: pgmap v21802: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:40 smithi081 ceph-mon[31453]: pgmap v21803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:40 smithi153 ceph-mon[36569]: pgmap v21803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:41.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:40 smithi104 ceph-mon[36624]: pgmap v21803: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:42 smithi081 ceph-mon[31453]: pgmap v21804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:42 smithi153 ceph-mon[36569]: pgmap v21804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:42 smithi104 ceph-mon[36624]: pgmap v21804: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:44 smithi081 ceph-mon[31453]: pgmap v21805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:44 smithi153 ceph-mon[36569]: pgmap v21805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:45.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:44 smithi104 ceph-mon[36624]: pgmap v21805: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:46 smithi081 ceph-mon[31453]: pgmap v21806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:46 smithi153 ceph-mon[36569]: pgmap v21806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:46 smithi104 ceph-mon[36624]: pgmap v21806: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:48 smithi081 ceph-mon[31453]: pgmap v21807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:48 smithi153 ceph-mon[36569]: pgmap v21807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:48 smithi104 ceph-mon[36624]: pgmap v21807: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:50 smithi081 ceph-mon[31453]: pgmap v21808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:50 smithi153 ceph-mon[36569]: pgmap v21808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:50 smithi104 ceph-mon[36624]: pgmap v21808: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:52 smithi153 ceph-mon[36569]: pgmap v21809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:52 smithi104 ceph-mon[36624]: pgmap v21809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:52 smithi081 ceph-mon[31453]: pgmap v21809: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:53.917 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:53.918+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:39:53.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:53.918+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:39:53.918 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:39:53.918+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:39:54.248 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:53 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:39:54.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:53 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:39:54.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:53 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:39:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:54 smithi153 ceph-mon[36569]: pgmap v21810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:54 smithi104 ceph-mon[36624]: pgmap v21810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:55.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:54 smithi081 ceph-mon[31453]: pgmap v21810: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:56 smithi153 ceph-mon[36569]: pgmap v21811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:39:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:39:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:56 smithi153 ceph-mon[36569]: pgmap v21812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:56 smithi153 ceph-mon[36569]: pgmap v21813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.807 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:56 smithi104 ceph-mon[36624]: pgmap v21811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:39:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:39:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:56 smithi104 ceph-mon[36624]: pgmap v21812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:56 smithi104 ceph-mon[36624]: pgmap v21813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:56 smithi081 ceph-mon[31453]: pgmap v21811: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:39:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:39:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:56 smithi081 ceph-mon[31453]: pgmap v21812: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:56 smithi081 ceph-mon[31453]: pgmap v21813: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:56.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:39:58.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:39:58 smithi081 ceph-mon[31453]: pgmap v21814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:59.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:39:58 smithi153 ceph-mon[36569]: pgmap v21814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:39:59.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:39:58 smithi104 ceph-mon[36624]: pgmap v21814: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:00 smithi081 ceph-mon[31453]: pgmap v21815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:00.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:40:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:00 smithi153 ceph-mon[36569]: pgmap v21815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:01.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:40:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:00 smithi104 ceph-mon[36624]: pgmap v21815: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:40:02.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:02 smithi081 ceph-mon[31453]: pgmap v21816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:03.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:02 smithi153 ceph-mon[36569]: pgmap v21816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:02 smithi104 ceph-mon[36624]: pgmap v21816: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:04.906 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:04 smithi081 ceph-mon[31453]: pgmap v21817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:05.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:04 smithi153 ceph-mon[36569]: pgmap v21817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:05.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:04 smithi104 ceph-mon[36624]: pgmap v21817: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:06.858 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:06 smithi153 ceph-mon[36569]: pgmap v21818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:06.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:06 smithi081 ceph-mon[31453]: pgmap v21818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:06 smithi104 ceph-mon[36624]: pgmap v21818: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:08.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:08 smithi081 ceph-mon[31453]: pgmap v21819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:08.931+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:40:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:08.931+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:40:08.930 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:08.931+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:40:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:08 smithi153 ceph-mon[36569]: pgmap v21819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:09.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:08 smithi104 ceph-mon[36624]: pgmap v21819: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:10.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:10 smithi081 ceph-mon[31453]: pgmap v21820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:10 smithi153 ceph-mon[36569]: pgmap v21820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:10 smithi104 ceph-mon[36624]: pgmap v21820: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:12.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:12 smithi081 ceph-mon[31453]: pgmap v21821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:12 smithi153 ceph-mon[36569]: pgmap v21821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:13.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:12 smithi104 ceph-mon[36624]: pgmap v21821: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:14.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:14 smithi081 ceph-mon[31453]: pgmap v21822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:14 smithi153 ceph-mon[36569]: pgmap v21822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:15.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:14 smithi104 ceph-mon[36624]: pgmap v21822: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:16.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:16 smithi081 ceph-mon[31453]: pgmap v21823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:16 smithi153 ceph-mon[36569]: pgmap v21823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:17.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:16 smithi104 ceph-mon[36624]: pgmap v21823: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:18 smithi081 ceph-mon[31453]: pgmap v21824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:18 smithi153 ceph-mon[36569]: pgmap v21824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:19.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:18 smithi104 ceph-mon[36624]: pgmap v21824: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:19.495+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:40:19.494 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:19.495+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:40:19.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:19.495+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 315 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:40:19.495 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:19.495+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 315 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:40:20.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:20 smithi081 ceph-mon[31453]: pgmap v21825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:20 smithi153 ceph-mon[36569]: pgmap v21825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:20 smithi104 ceph-mon[36624]: pgmap v21825: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:22.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:22 smithi081 ceph-mon[31453]: pgmap v21826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:22 smithi153 ceph-mon[36569]: pgmap v21826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:22 smithi104 ceph-mon[36624]: pgmap v21826: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:23.943 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:23.944+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:40:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:23.944+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:40:23.944 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:23.944+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:40:24.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:24 smithi081 ceph-mon[31453]: pgmap v21827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:24 smithi153 ceph-mon[36569]: pgmap v21827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:24 smithi104 ceph-mon[36624]: pgmap v21827: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:26 smithi153 ceph-mon[36569]: pgmap v21828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:26 smithi104 ceph-mon[36624]: pgmap v21828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:26 smithi081 ceph-mon[31453]: pgmap v21828: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:28 smithi153 ceph-mon[36569]: pgmap v21829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:28 smithi104 ceph-mon[36624]: pgmap v21829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:28 smithi081 ceph-mon[31453]: pgmap v21829: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:30 smithi153 ceph-mon[36569]: pgmap v21830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:30 smithi104 ceph-mon[36624]: pgmap v21830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:30 smithi081 ceph-mon[31453]: pgmap v21830: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:32 smithi153 ceph-mon[36569]: pgmap v21831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:32 smithi104 ceph-mon[36624]: pgmap v21831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:32 smithi081 ceph-mon[31453]: pgmap v21831: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:34 smithi153 ceph-mon[36569]: pgmap v21832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:34 smithi104 ceph-mon[36624]: pgmap v21832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:34 smithi081 ceph-mon[31453]: pgmap v21832: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:36 smithi153 ceph-mon[36569]: pgmap v21833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:36 smithi104 ceph-mon[36624]: pgmap v21833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:36 smithi081 ceph-mon[31453]: pgmap v21833: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:38.947+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:40:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:38.947+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:40:38.947 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:38.947+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:40:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:38 smithi153 ceph-mon[36569]: pgmap v21834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:38 smithi104 ceph-mon[36624]: pgmap v21834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:38 smithi081 ceph-mon[31453]: pgmap v21834: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:40 smithi153 ceph-mon[36569]: pgmap v21835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:41.091 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:40 smithi104 ceph-mon[36624]: pgmap v21835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:40 smithi081 ceph-mon[31453]: pgmap v21835: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:42 smithi153 ceph-mon[36569]: pgmap v21836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:42 smithi104 ceph-mon[36624]: pgmap v21836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:43.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:42 smithi081 ceph-mon[31453]: pgmap v21836: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:44 smithi153 ceph-mon[36569]: pgmap v21837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:44 smithi104 ceph-mon[36624]: pgmap v21837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:44 smithi081 ceph-mon[31453]: pgmap v21837: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:46 smithi153 ceph-mon[36569]: pgmap v21838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:46 smithi104 ceph-mon[36624]: pgmap v21838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:46 smithi081 ceph-mon[31453]: pgmap v21838: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:48 smithi153 ceph-mon[36569]: pgmap v21839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:48 smithi104 ceph-mon[36624]: pgmap v21839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:48 smithi081 ceph-mon[31453]: pgmap v21839: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:50 smithi153 ceph-mon[36569]: pgmap v21840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:50 smithi104 ceph-mon[36624]: pgmap v21840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:51.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:50 smithi081 ceph-mon[31453]: pgmap v21840: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:52 smithi153 ceph-mon[36569]: pgmap v21841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:52 smithi104 ceph-mon[36624]: pgmap v21841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:52 smithi081 ceph-mon[31453]: pgmap v21841: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:53.963+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:40:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:53.963+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:40:53.963 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:40:53.963+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:40:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:54 smithi153 ceph-mon[36569]: pgmap v21842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:54 smithi104 ceph-mon[36624]: pgmap v21842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:54 smithi081 ceph-mon[31453]: pgmap v21842: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:56 smithi153 ceph-mon[36569]: pgmap v21843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:57.058 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:56 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:40:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:56 smithi104 ceph-mon[36624]: pgmap v21843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:56 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:40:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:56 smithi081 ceph-mon[31453]: pgmap v21843: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:57.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:56 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:40:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:40:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:40:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: pgmap v21844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: pgmap v21845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:58.408 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:40:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:40:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:40:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: pgmap v21844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: pgmap v21845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:58.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:40:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: pgmap v21844: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: pgmap v21845: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:40:58.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:40:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:41:00.526 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:00 smithi081 ceph-mon[31453]: pgmap v21846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:00.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:00 smithi153 ceph-mon[36569]: pgmap v21846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:00.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:00 smithi104 ceph-mon[36624]: pgmap v21846: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:02.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:02 smithi153 ceph-mon[36569]: pgmap v21847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:02.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:02 smithi104 ceph-mon[36624]: pgmap v21847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:02.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:02 smithi081 ceph-mon[31453]: pgmap v21847: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:04.502+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:41:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:04.502+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:41:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:04.502+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 316 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:41:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:04.503+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 316 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:41:04.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:04 smithi153 ceph-mon[36569]: pgmap v21848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:04.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:04 smithi104 ceph-mon[36624]: pgmap v21848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:04.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:04 smithi081 ceph-mon[31453]: pgmap v21848: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:06 smithi153 ceph-mon[36569]: pgmap v21849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:06.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:06 smithi104 ceph-mon[36624]: pgmap v21849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:06 smithi081 ceph-mon[31453]: pgmap v21849: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:08.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:08 smithi153 ceph-mon[36569]: pgmap v21850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:08.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:08 smithi104 ceph-mon[36624]: pgmap v21850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:08.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:08 smithi081 ceph-mon[31453]: pgmap v21850: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:08.979+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:41:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:08.979+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:41:08.978 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:08.979+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:41:10.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:10 smithi153 ceph-mon[36569]: pgmap v21851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:10.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:10 smithi104 ceph-mon[36624]: pgmap v21851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:10.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:10 smithi081 ceph-mon[31453]: pgmap v21851: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:12.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:12 smithi153 ceph-mon[36569]: pgmap v21852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:12.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:12 smithi104 ceph-mon[36624]: pgmap v21852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:12.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:12 smithi081 ceph-mon[31453]: pgmap v21852: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:14.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:14 smithi153 ceph-mon[36569]: pgmap v21853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:14.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:14 smithi104 ceph-mon[36624]: pgmap v21853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:14.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:14 smithi081 ceph-mon[31453]: pgmap v21853: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:16.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:16 smithi153 ceph-mon[36569]: pgmap v21854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:16.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:16 smithi104 ceph-mon[36624]: pgmap v21854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:16.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:16 smithi081 ceph-mon[31453]: pgmap v21854: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:18.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:18 smithi153 ceph-mon[36569]: pgmap v21855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:18.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:18 smithi104 ceph-mon[36624]: pgmap v21855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:18.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:18 smithi081 ceph-mon[31453]: pgmap v21855: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:20 smithi153 ceph-mon[36569]: pgmap v21856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:20.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:20 smithi104 ceph-mon[36624]: pgmap v21856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:20.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:20 smithi081 ceph-mon[31453]: pgmap v21856: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:22 smithi153 ceph-mon[36569]: pgmap v21857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:22.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:22 smithi104 ceph-mon[36624]: pgmap v21857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:22.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:22 smithi081 ceph-mon[31453]: pgmap v21857: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:23.994+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:41:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:23.994+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:41:23.994 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:23.994+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:41:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:24 smithi153 ceph-mon[36569]: pgmap v21858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:24 smithi104 ceph-mon[36624]: pgmap v21858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:24.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:24 smithi081 ceph-mon[31453]: pgmap v21858: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:26 smithi153 ceph-mon[36569]: pgmap v21859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:26.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:26 smithi104 ceph-mon[36624]: pgmap v21859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:26 smithi081 ceph-mon[31453]: pgmap v21859: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:28 smithi153 ceph-mon[36569]: pgmap v21860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:28.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:28 smithi104 ceph-mon[36624]: pgmap v21860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:28.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:28 smithi081 ceph-mon[31453]: pgmap v21860: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:30 smithi153 ceph-mon[36569]: pgmap v21861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:30 smithi104 ceph-mon[36624]: pgmap v21861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:30.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:30 smithi081 ceph-mon[31453]: pgmap v21861: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:32 smithi153 ceph-mon[36569]: pgmap v21862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:32 smithi104 ceph-mon[36624]: pgmap v21862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:32.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:32 smithi081 ceph-mon[31453]: pgmap v21862: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:34.557 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:34 smithi153 ceph-mon[36569]: pgmap v21863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:34 smithi104 ceph-mon[36624]: pgmap v21863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:34.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:34 smithi081 ceph-mon[31453]: pgmap v21863: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:36 smithi153 ceph-mon[36569]: pgmap v21864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:36 smithi104 ceph-mon[36624]: pgmap v21864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:36.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:36 smithi081 ceph-mon[31453]: pgmap v21864: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:38 smithi153 ceph-mon[36569]: pgmap v21865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:38 smithi104 ceph-mon[36624]: pgmap v21865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:38.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:38 smithi081 ceph-mon[31453]: pgmap v21865: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:38.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:38.995+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:41:38.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:38.995+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:41:38.995 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:38.995+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:41:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:40 smithi153 ceph-mon[36569]: pgmap v21866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:40 smithi104 ceph-mon[36624]: pgmap v21866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:40.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:40 smithi081 ceph-mon[31453]: pgmap v21866: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:42.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:42 smithi104 ceph-mon[36624]: pgmap v21867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:42.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:42 smithi081 ceph-mon[31453]: pgmap v21867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:42.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:42 smithi153 ceph-mon[36569]: pgmap v21867: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:44.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:44 smithi104 ceph-mon[36624]: pgmap v21868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:44.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:44 smithi081 ceph-mon[31453]: pgmap v21868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:44.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:44 smithi153 ceph-mon[36569]: pgmap v21868: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:46 smithi104 ceph-mon[36624]: pgmap v21869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:46.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:46 smithi081 ceph-mon[31453]: pgmap v21869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:46.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:46 smithi153 ceph-mon[36569]: pgmap v21869: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:48 smithi104 ceph-mon[36624]: pgmap v21870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:48.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:48 smithi081 ceph-mon[31453]: pgmap v21870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:48.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:48 smithi153 ceph-mon[36569]: pgmap v21870: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:50.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:50 smithi104 ceph-mon[36624]: pgmap v21871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:50.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:50 smithi081 ceph-mon[31453]: pgmap v21871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:50.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:50 smithi153 ceph-mon[36569]: pgmap v21871: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:52.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:52 smithi081 ceph-mon[31453]: pgmap v21872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:52.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:52 smithi153 ceph-mon[36569]: pgmap v21872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:52.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:52 smithi104 ceph-mon[36624]: pgmap v21872: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:54.003+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:41:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:54.003+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:41:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:41:54.003+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:41:54.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:54 smithi081 ceph-mon[31453]: pgmap v21873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:54.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:54 smithi153 ceph-mon[36569]: pgmap v21873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:54.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:54 smithi104 ceph-mon[36624]: pgmap v21873: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:56.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:56 smithi081 ceph-mon[31453]: pgmap v21874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:56.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:56 smithi153 ceph-mon[36569]: pgmap v21874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:56.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:56 smithi104 ceph-mon[36624]: pgmap v21874: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:58 smithi081 ceph-mon[31453]: pgmap v21875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:58.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:58 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:41:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:58 smithi153 ceph-mon[36569]: pgmap v21875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:58.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:58 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:41:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:58 smithi104 ceph-mon[36624]: pgmap v21875: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:58.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:58 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:41:59.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:41:59.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:41:59.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:59 smithi081 ceph-mon[31453]: pgmap v21876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:59.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:59 smithi081 ceph-mon[31453]: pgmap v21877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:59.558 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:41:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:41:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:41:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:41:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:59 smithi153 ceph-mon[36569]: pgmap v21876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:59 smithi153 ceph-mon[36569]: pgmap v21877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:59.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:41:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:41:59.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:41:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:41:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:59 smithi104 ceph-mon[36624]: pgmap v21876: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:59 smithi104 ceph-mon[36624]: pgmap v21877: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:41:59.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:41:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:42:02.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:01 smithi104 ceph-mon[36624]: pgmap v21878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:02.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:01 smithi081 ceph-mon[31453]: pgmap v21878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:01 smithi153 ceph-mon[36569]: pgmap v21878: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:04.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:03 smithi104 ceph-mon[36624]: pgmap v21879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:04.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:03 smithi081 ceph-mon[31453]: pgmap v21879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:03 smithi153 ceph-mon[36569]: pgmap v21879: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:06.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:05 smithi153 ceph-mon[36569]: pgmap v21880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:06.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:05 smithi081 ceph-mon[31453]: pgmap v21880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:06.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:05 smithi104 ceph-mon[36624]: pgmap v21880: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:07 smithi081 ceph-mon[31453]: pgmap v21881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:07 smithi153 ceph-mon[36569]: pgmap v21881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:08.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:07 smithi104 ceph-mon[36624]: pgmap v21881: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:09.019+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:42:09.018 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:09.019+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:42:09.019 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:09.019+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:42:10.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:09 smithi081 ceph-mon[31453]: pgmap v21882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:09 smithi153 ceph-mon[36569]: pgmap v21882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:09 smithi104 ceph-mon[36624]: pgmap v21882: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:12.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:11 smithi081 ceph-mon[31453]: pgmap v21883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:11 smithi153 ceph-mon[36569]: pgmap v21883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:12.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:11 smithi104 ceph-mon[36624]: pgmap v21883: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:14.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:13 smithi081 ceph-mon[31453]: pgmap v21884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:13 smithi153 ceph-mon[36569]: pgmap v21884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:13 smithi104 ceph-mon[36624]: pgmap v21884: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:16.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:15 smithi081 ceph-mon[31453]: pgmap v21885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:15 smithi153 ceph-mon[36569]: pgmap v21885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:16.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:15 smithi104 ceph-mon[36624]: pgmap v21885: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:18.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:17 smithi081 ceph-mon[31453]: pgmap v21886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:17 smithi153 ceph-mon[36569]: pgmap v21886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:18.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:17 smithi104 ceph-mon[36624]: pgmap v21886: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:20.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:19 smithi104 ceph-mon[36624]: pgmap v21887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:20.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:19 smithi081 ceph-mon[31453]: pgmap v21887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:19 smithi153 ceph-mon[36569]: pgmap v21887: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:22.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:21 smithi081 ceph-mon[31453]: pgmap v21888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:21 smithi153 ceph-mon[36569]: pgmap v21888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:22.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:21 smithi104 ceph-mon[36624]: pgmap v21888: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:24.034+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:42:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:24.034+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:42:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:24.034+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:42:24.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:23 smithi081 ceph-mon[31453]: pgmap v21889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:23 smithi153 ceph-mon[36569]: pgmap v21889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:24.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:23 smithi104 ceph-mon[36624]: pgmap v21889: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:25 smithi081 ceph-mon[31453]: pgmap v21890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:25 smithi153 ceph-mon[36569]: pgmap v21890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:26.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:25 smithi104 ceph-mon[36624]: pgmap v21890: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:27 smithi081 ceph-mon[31453]: pgmap v21891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:27 smithi153 ceph-mon[36569]: pgmap v21891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:28.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:27 smithi104 ceph-mon[36624]: pgmap v21891: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:30.159 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:29 smithi081 ceph-mon[31453]: pgmap v21892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:29 smithi153 ceph-mon[36569]: pgmap v21892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:29 smithi104 ceph-mon[36624]: pgmap v21892: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:32.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:31 smithi081 ceph-mon[31453]: pgmap v21893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:31 smithi153 ceph-mon[36569]: pgmap v21893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:31 smithi104 ceph-mon[36624]: pgmap v21893: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:33 smithi153 ceph-mon[36569]: pgmap v21894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:34.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:33 smithi104 ceph-mon[36624]: pgmap v21894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:34.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:33 smithi081 ceph-mon[31453]: pgmap v21894: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:35 smithi153 ceph-mon[36569]: pgmap v21895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:36.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:35 smithi104 ceph-mon[36624]: pgmap v21895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:36.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:35 smithi081 ceph-mon[31453]: pgmap v21895: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:37 smithi153 ceph-mon[36569]: pgmap v21896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:37 smithi104 ceph-mon[36624]: pgmap v21896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:37 smithi081 ceph-mon[31453]: pgmap v21896: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:39.048 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:39.049+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:42:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:39.049+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:42:39.049 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:39.049+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:42:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:39 smithi153 ceph-mon[36569]: pgmap v21897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:40.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:39 smithi104 ceph-mon[36624]: pgmap v21897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:40.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:39 smithi081 ceph-mon[31453]: pgmap v21897: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:42.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:41 smithi081 ceph-mon[31453]: pgmap v21898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:42.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:41 smithi153 ceph-mon[36569]: pgmap v21898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:42.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:41 smithi104 ceph-mon[36624]: pgmap v21898: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:43 smithi153 ceph-mon[36569]: pgmap v21899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:44.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:43 smithi104 ceph-mon[36624]: pgmap v21899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:43 smithi081 ceph-mon[31453]: pgmap v21899: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:45 smithi153 ceph-mon[36569]: pgmap v21900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:45 smithi104 ceph-mon[36624]: pgmap v21900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:45 smithi081 ceph-mon[31453]: pgmap v21900: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:47 smithi153 ceph-mon[36569]: pgmap v21901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:48.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:47 smithi104 ceph-mon[36624]: pgmap v21901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:48.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:47 smithi081 ceph-mon[31453]: pgmap v21901: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:49 smithi153 ceph-mon[36569]: pgmap v21902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:50.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:49 smithi104 ceph-mon[36624]: pgmap v21902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:49 smithi081 ceph-mon[31453]: pgmap v21902: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:51 smithi153 ceph-mon[36569]: pgmap v21903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:52.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:51 smithi104 ceph-mon[36624]: pgmap v21903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:51 smithi081 ceph-mon[31453]: pgmap v21903: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:54.058+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:42:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:54.058+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:42:54.057 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:42:54.058+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:42:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:53 smithi153 ceph-mon[36569]: pgmap v21904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:54.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:53 smithi104 ceph-mon[36624]: pgmap v21904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:53 smithi081 ceph-mon[31453]: pgmap v21904: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:55 smithi153 ceph-mon[36569]: pgmap v21905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:56.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:55 smithi104 ceph-mon[36624]: pgmap v21905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:55 smithi081 ceph-mon[31453]: pgmap v21905: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:57 smithi153 ceph-mon[36569]: pgmap v21906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:58.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:57 smithi104 ceph-mon[36624]: pgmap v21906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:42:58.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:57 smithi081 ceph-mon[31453]: pgmap v21906: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:00.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:59 smithi153 ceph-mon[36569]: pgmap v21907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:00.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:43:00.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.204 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:42:59 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:59 smithi104 ceph-mon[36624]: pgmap v21907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:00.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:43:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:42:59 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:59 smithi081 ceph-mon[31453]: pgmap v21907: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:43:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:00.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:42:59 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:01 smithi153 ceph-mon[36569]: pgmap v21908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:02.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:01 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:01 smithi104 ceph-mon[36624]: pgmap v21908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:02.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:01 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:01 smithi081 ceph-mon[31453]: pgmap v21908: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:02.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:01 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:03 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:03.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:03 smithi153 ceph-mon[36569]: pgmap v21909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:03 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:03.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:03 smithi104 ceph-mon[36624]: pgmap v21909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:03 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:03.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:03 smithi081 ceph-mon[31453]: pgmap v21909: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:05 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:05.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:05 smithi081 ceph-mon[31453]: pgmap v21910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:05 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:05.806 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:05 smithi153 ceph-mon[36569]: pgmap v21910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:05 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:05.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:05 smithi104 ceph-mon[36624]: pgmap v21910: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:06.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:43:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:43:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:06 smithi081 ceph-mon[31453]: pgmap v21911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:06 smithi081 ceph-mon[31453]: pgmap v21912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:06.656 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:06.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:43:06.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:43:06.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:06 smithi153 ceph-mon[36569]: pgmap v21911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:06.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:06 smithi153 ceph-mon[36569]: pgmap v21912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:06.707 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:06.831 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:43:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:43:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:06 smithi104 ceph-mon[36624]: pgmap v21911: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:06 smithi104 ceph-mon[36624]: pgmap v21912: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:06.832 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:43:09.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:08 smithi153 ceph-mon[36569]: pgmap v21913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:09.065+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:43:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:09.065+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:43:09.065 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:09.065+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:43:09.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:08 smithi104 ceph-mon[36624]: pgmap v21913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:08 smithi081 ceph-mon[31453]: pgmap v21913: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:11.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:10 smithi153 ceph-mon[36569]: pgmap v21914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:11.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:10 smithi104 ceph-mon[36624]: pgmap v21914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:10 smithi081 ceph-mon[31453]: pgmap v21914: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:13.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:12 smithi153 ceph-mon[36569]: pgmap v21915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:13.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:12 smithi104 ceph-mon[36624]: pgmap v21915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:12 smithi081 ceph-mon[31453]: pgmap v21915: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:15.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:14 smithi153 ceph-mon[36569]: pgmap v21916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:15.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:14 smithi104 ceph-mon[36624]: pgmap v21916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:14 smithi081 ceph-mon[31453]: pgmap v21916: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:16.999 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:16 smithi081 ceph-mon[31453]: pgmap v21917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:17.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:16 smithi153 ceph-mon[36569]: pgmap v21917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:17.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:16 smithi104 ceph-mon[36624]: pgmap v21917: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:18.905 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:18 smithi081 ceph-mon[31453]: pgmap v21918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:19.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:18 smithi153 ceph-mon[36569]: pgmap v21918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:19.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:18 smithi104 ceph-mon[36624]: pgmap v21918: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:19.524+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:43:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:19.524+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:43:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:19.525+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 317 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:43:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:19.525+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 317 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:43:21.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:20 smithi153 ceph-mon[36569]: pgmap v21919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:21.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:20 smithi104 ceph-mon[36624]: pgmap v21919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:20 smithi081 ceph-mon[31453]: pgmap v21919: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:23.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:22 smithi153 ceph-mon[36569]: pgmap v21920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:23.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:22 smithi104 ceph-mon[36624]: pgmap v21920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:22 smithi081 ceph-mon[31453]: pgmap v21920: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:24.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:24.071+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:43:24.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:24.071+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:43:24.071 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:24.071+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:43:25.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:24 smithi153 ceph-mon[36569]: pgmap v21921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:25.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:24 smithi104 ceph-mon[36624]: pgmap v21921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:24 smithi081 ceph-mon[31453]: pgmap v21921: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:27.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:26 smithi153 ceph-mon[36569]: pgmap v21922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:27.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:26 smithi104 ceph-mon[36624]: pgmap v21922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:26 smithi081 ceph-mon[31453]: pgmap v21922: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:28 smithi153 ceph-mon[36569]: pgmap v21923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:28 smithi104 ceph-mon[36624]: pgmap v21923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:28 smithi081 ceph-mon[31453]: pgmap v21923: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:30 smithi153 ceph-mon[36569]: pgmap v21924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:30 smithi104 ceph-mon[36624]: pgmap v21924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:30 smithi081 ceph-mon[31453]: pgmap v21924: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:32 smithi153 ceph-mon[36569]: pgmap v21925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:32 smithi104 ceph-mon[36624]: pgmap v21925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:32 smithi081 ceph-mon[31453]: pgmap v21925: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:34 smithi153 ceph-mon[36569]: pgmap v21926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:34 smithi104 ceph-mon[36624]: pgmap v21926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:34 smithi081 ceph-mon[31453]: pgmap v21926: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:36 smithi153 ceph-mon[36569]: pgmap v21927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:36 smithi104 ceph-mon[36624]: pgmap v21927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:37.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:36 smithi081 ceph-mon[31453]: pgmap v21927: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:38 smithi153 ceph-mon[36569]: pgmap v21928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:39.075 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:39.076+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:43:39.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:39.076+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:43:39.076 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:39.076+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:43:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:38 smithi104 ceph-mon[36624]: pgmap v21928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:38 smithi081 ceph-mon[31453]: pgmap v21928: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:40 smithi153 ceph-mon[36569]: pgmap v21929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:40 smithi104 ceph-mon[36624]: pgmap v21929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:40 smithi081 ceph-mon[31453]: pgmap v21929: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:42 smithi153 ceph-mon[36569]: pgmap v21930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:42 smithi104 ceph-mon[36624]: pgmap v21930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:42 smithi081 ceph-mon[31453]: pgmap v21930: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:44 smithi153 ceph-mon[36569]: pgmap v21931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:44 smithi104 ceph-mon[36624]: pgmap v21931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:44 smithi081 ceph-mon[31453]: pgmap v21931: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:46 smithi153 ceph-mon[36569]: pgmap v21932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:46 smithi104 ceph-mon[36624]: pgmap v21932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:46 smithi081 ceph-mon[31453]: pgmap v21932: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:48 smithi153 ceph-mon[36569]: pgmap v21933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:49.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:48 smithi104 ceph-mon[36624]: pgmap v21933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:48 smithi081 ceph-mon[31453]: pgmap v21933: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:50 smithi153 ceph-mon[36569]: pgmap v21934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:50 smithi104 ceph-mon[36624]: pgmap v21934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:50 smithi081 ceph-mon[31453]: pgmap v21934: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:52 smithi153 ceph-mon[36569]: pgmap v21935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:52 smithi104 ceph-mon[36624]: pgmap v21935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:52 smithi081 ceph-mon[31453]: pgmap v21935: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:54.081+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:43:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:54.081+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:43:54.081 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:43:54.081+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:43:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:54 smithi153 ceph-mon[36569]: pgmap v21936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:54 smithi104 ceph-mon[36624]: pgmap v21936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:55.158 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:54 smithi081 ceph-mon[31453]: pgmap v21936: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:56 smithi153 ceph-mon[36569]: pgmap v21937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:56 smithi104 ceph-mon[36624]: pgmap v21937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:56 smithi081 ceph-mon[31453]: pgmap v21937: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:43:58 smithi104 ceph-mon[36624]: pgmap v21938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:43:58 smithi081 ceph-mon[31453]: pgmap v21938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:43:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:43:58 smithi153 ceph-mon[36569]: pgmap v21938: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:01.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:00 smithi104 ceph-mon[36624]: pgmap v21939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:00 smithi081 ceph-mon[31453]: pgmap v21939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:00 smithi153 ceph-mon[36569]: pgmap v21939: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:03.080 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:02 smithi081 ceph-mon[31453]: pgmap v21940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:03.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:02 smithi104 ceph-mon[36624]: pgmap v21940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:02 smithi153 ceph-mon[36569]: pgmap v21940: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:04 smithi104 ceph-mon[36624]: pgmap v21941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:04 smithi081 ceph-mon[31453]: pgmap v21941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:04 smithi153 ceph-mon[36569]: pgmap v21941: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:07.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:06 smithi104 ceph-mon[36624]: pgmap v21942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:44:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:07.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:06 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:07.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:06 smithi153 ceph-mon[36569]: pgmap v21942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:07.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:44:07.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:07.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:06 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:06 smithi081 ceph-mon[31453]: pgmap v21942: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:44:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:07.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:06 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:44:08.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:44:08.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:07 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:44:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:44:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:07 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:08.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:44:08.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:44:08.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:07 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:44:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:09.096+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:44:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:09.096+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:44:09.096 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:09.096+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:44:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:08 smithi081 ceph-mon[31453]: pgmap v21943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:09.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:08 smithi081 ceph-mon[31453]: pgmap v21944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:08 smithi153 ceph-mon[36569]: pgmap v21943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:08 smithi153 ceph-mon[36569]: pgmap v21944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:08 smithi104 ceph-mon[36624]: pgmap v21943: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:08 smithi104 ceph-mon[36624]: pgmap v21944: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:10 smithi081 ceph-mon[31453]: pgmap v21945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:10 smithi153 ceph-mon[36569]: pgmap v21945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:10 smithi104 ceph-mon[36624]: pgmap v21945: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:12 smithi081 ceph-mon[31453]: pgmap v21946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:12 smithi153 ceph-mon[36569]: pgmap v21946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:12 smithi104 ceph-mon[36624]: pgmap v21946: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:14 smithi081 ceph-mon[31453]: pgmap v21947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:14 smithi153 ceph-mon[36569]: pgmap v21947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:15.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:14 smithi104 ceph-mon[36624]: pgmap v21947: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:16 smithi081 ceph-mon[31453]: pgmap v21948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:16 smithi153 ceph-mon[36569]: pgmap v21948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:17.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:16 smithi104 ceph-mon[36624]: pgmap v21948: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:18 smithi081 ceph-mon[31453]: pgmap v21949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:18 smithi153 ceph-mon[36569]: pgmap v21949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:19.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:18 smithi104 ceph-mon[36624]: pgmap v21949: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:21.134 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:20 smithi104 ceph-mon[36624]: pgmap v21950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:20 smithi081 ceph-mon[31453]: pgmap v21950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:20 smithi153 ceph-mon[36569]: pgmap v21950: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:22 smithi081 ceph-mon[31453]: pgmap v21951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:22 smithi153 ceph-mon[36569]: pgmap v21951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:23.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:22 smithi104 ceph-mon[36624]: pgmap v21951: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:24.109+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:44:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:24.109+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:44:24.109 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:24.109+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:44:25.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:24 smithi081 ceph-mon[31453]: pgmap v21952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:24 smithi153 ceph-mon[36569]: pgmap v21952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:24 smithi104 ceph-mon[36624]: pgmap v21952: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:26 smithi081 ceph-mon[31453]: pgmap v21953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:26 smithi153 ceph-mon[36569]: pgmap v21953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:27.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:26 smithi104 ceph-mon[36624]: pgmap v21953: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:28 smithi081 ceph-mon[31453]: pgmap v21954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:28 smithi153 ceph-mon[36569]: pgmap v21954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:29.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:28 smithi104 ceph-mon[36624]: pgmap v21954: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:30 smithi153 ceph-mon[36569]: pgmap v21955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:30 smithi104 ceph-mon[36624]: pgmap v21955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:30 smithi081 ceph-mon[31453]: pgmap v21955: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:33.287 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:32 smithi081 ceph-mon[31453]: pgmap v21956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:32 smithi153 ceph-mon[36569]: pgmap v21956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:33.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:32 smithi104 ceph-mon[36624]: pgmap v21956: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:34 smithi153 ceph-mon[36569]: pgmap v21957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:35.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:34 smithi104 ceph-mon[36624]: pgmap v21957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:34 smithi081 ceph-mon[31453]: pgmap v21957: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:36 smithi153 ceph-mon[36569]: pgmap v21958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:36 smithi104 ceph-mon[36624]: pgmap v21958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:36 smithi081 ceph-mon[31453]: pgmap v21958: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:39.125+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:44:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:39.125+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:44:39.125 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:39.125+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:44:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:38 smithi153 ceph-mon[36569]: pgmap v21959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:38 smithi104 ceph-mon[36624]: pgmap v21959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:38 smithi081 ceph-mon[31453]: pgmap v21959: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:40 smithi153 ceph-mon[36569]: pgmap v21960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:41.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:40 smithi104 ceph-mon[36624]: pgmap v21960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:40 smithi081 ceph-mon[31453]: pgmap v21960: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:43.208 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:42 smithi081 ceph-mon[31453]: pgmap v21961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:42 smithi153 ceph-mon[36569]: pgmap v21961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:42 smithi104 ceph-mon[36624]: pgmap v21961: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:44 smithi153 ceph-mon[36569]: pgmap v21962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:45.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:44 smithi104 ceph-mon[36624]: pgmap v21962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:44 smithi081 ceph-mon[31453]: pgmap v21962: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:46 smithi153 ceph-mon[36569]: pgmap v21963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:46 smithi104 ceph-mon[36624]: pgmap v21963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:46 smithi081 ceph-mon[31453]: pgmap v21963: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:48 smithi153 ceph-mon[36569]: pgmap v21964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:49.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:48 smithi104 ceph-mon[36624]: pgmap v21964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:49.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:48 smithi081 ceph-mon[31453]: pgmap v21964: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:50 smithi153 ceph-mon[36569]: pgmap v21965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:50 smithi104 ceph-mon[36624]: pgmap v21965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:51.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:50 smithi081 ceph-mon[31453]: pgmap v21965: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:52 smithi153 ceph-mon[36569]: pgmap v21966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:52 smithi104 ceph-mon[36624]: pgmap v21966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:52 smithi081 ceph-mon[31453]: pgmap v21966: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:54.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:54.128+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:44:54.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:54.128+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:44:54.128 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:44:54.128+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:44:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:54 smithi153 ceph-mon[36569]: pgmap v21967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:54 smithi104 ceph-mon[36624]: pgmap v21967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:55.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:54 smithi081 ceph-mon[31453]: pgmap v21967: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:56 smithi153 ceph-mon[36569]: pgmap v21968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:57.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:56 smithi104 ceph-mon[36624]: pgmap v21968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:57.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:56 smithi081 ceph-mon[31453]: pgmap v21968: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:44:58 smithi153 ceph-mon[36569]: pgmap v21969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:44:58 smithi104 ceph-mon[36624]: pgmap v21969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:44:59.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:44:58 smithi081 ceph-mon[31453]: pgmap v21969: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:00 smithi153 ceph-mon[36569]: pgmap v21970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:01.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:00 smithi104 ceph-mon[36624]: pgmap v21970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:01.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:00 smithi081 ceph-mon[31453]: pgmap v21970: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:02 smithi153 ceph-mon[36569]: pgmap v21971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:02 smithi104 ceph-mon[36624]: pgmap v21971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:02 smithi081 ceph-mon[31453]: pgmap v21971: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:04 smithi153 ceph-mon[36569]: pgmap v21972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:04 smithi104 ceph-mon[36624]: pgmap v21972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:05.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:04 smithi081 ceph-mon[31453]: pgmap v21972: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:07 smithi153 ceph-mon[36569]: pgmap v21973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:07.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:07 smithi104 ceph-mon[36624]: pgmap v21973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:07.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:07 smithi081 ceph-mon[31453]: pgmap v21973: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:08.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:08 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:45:08.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:08 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:45:08.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:08 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:45:09.136 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:09.137+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:45:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:09.137+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:45:09.137 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:09.137+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:45:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:09 smithi153 ceph-mon[36569]: pgmap v21974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:45:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:45:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:45:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:09 smithi104 ceph-mon[36624]: pgmap v21974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:45:09.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:45:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:45:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:09 smithi081 ceph-mon[31453]: pgmap v21974: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:45:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:45:09.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:45:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:10 smithi153 ceph-mon[36569]: pgmap v21975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:10 smithi153 ceph-mon[36569]: pgmap v21976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:10 smithi104 ceph-mon[36624]: pgmap v21975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:10 smithi104 ceph-mon[36624]: pgmap v21976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:10 smithi081 ceph-mon[31453]: pgmap v21975: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:10 smithi081 ceph-mon[31453]: pgmap v21976: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:12 smithi153 ceph-mon[36569]: pgmap v21977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:12.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:12 smithi104 ceph-mon[36624]: pgmap v21977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:12.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:12 smithi081 ceph-mon[31453]: pgmap v21977: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:14 smithi153 ceph-mon[36569]: pgmap v21978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:14 smithi104 ceph-mon[36624]: pgmap v21978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:14.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:14 smithi081 ceph-mon[31453]: pgmap v21978: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:16 smithi153 ceph-mon[36569]: pgmap v21979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:16.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:16 smithi104 ceph-mon[36624]: pgmap v21979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:16 smithi081 ceph-mon[31453]: pgmap v21979: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:18 smithi153 ceph-mon[36569]: pgmap v21980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:18.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:18 smithi104 ceph-mon[36624]: pgmap v21980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:18.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:18 smithi081 ceph-mon[31453]: pgmap v21980: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:20.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:20 smithi081 ceph-mon[31453]: pgmap v21981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:20.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:20 smithi104 ceph-mon[36624]: pgmap v21981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:20.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:20 smithi153 ceph-mon[36569]: pgmap v21981: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:22.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:22 smithi104 ceph-mon[36624]: pgmap v21982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:22.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:22 smithi081 ceph-mon[31453]: pgmap v21982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:22.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:22 smithi153 ceph-mon[36569]: pgmap v21982: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:24.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:24.147+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:45:24.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:24.147+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:45:24.147 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:24.147+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:45:24.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:24 smithi104 ceph-mon[36624]: pgmap v21983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:24.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:24 smithi081 ceph-mon[31453]: pgmap v21983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:24 smithi153 ceph-mon[36569]: pgmap v21983: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:26 smithi104 ceph-mon[36624]: pgmap v21984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:26.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:26 smithi081 ceph-mon[31453]: pgmap v21984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:26.555 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:26 smithi153 ceph-mon[36569]: pgmap v21984: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:28.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:28 smithi104 ceph-mon[36624]: pgmap v21985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:28.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:28 smithi081 ceph-mon[31453]: pgmap v21985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:28 smithi153 ceph-mon[36569]: pgmap v21985: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:30 smithi081 ceph-mon[31453]: pgmap v21986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:30 smithi153 ceph-mon[36569]: pgmap v21986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:30.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:30 smithi104 ceph-mon[36624]: pgmap v21986: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:32.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:32 smithi081 ceph-mon[31453]: pgmap v21987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:32 smithi153 ceph-mon[36569]: pgmap v21987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:32.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:32 smithi104 ceph-mon[36624]: pgmap v21987: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:34.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:34 smithi081 ceph-mon[31453]: pgmap v21988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:34 smithi153 ceph-mon[36569]: pgmap v21988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:34.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:34 smithi104 ceph-mon[36624]: pgmap v21988: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:36.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:36 smithi081 ceph-mon[31453]: pgmap v21989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:36 smithi153 ceph-mon[36569]: pgmap v21989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:36.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:36 smithi104 ceph-mon[36624]: pgmap v21989: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:38.361 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:38 smithi153 ceph-mon[36569]: pgmap v21990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:38 smithi081 ceph-mon[31453]: pgmap v21990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:38 smithi104 ceph-mon[36624]: pgmap v21990: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:39.156+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:45:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:39.156+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:45:39.156 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:39.156+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:45:40.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:40 smithi081 ceph-mon[31453]: pgmap v21991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:40 smithi153 ceph-mon[36569]: pgmap v21991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:40.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:40 smithi104 ceph-mon[36624]: pgmap v21991: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:41 smithi081 ceph-mon[31453]: pgmap v21992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:41 smithi153 ceph-mon[36569]: pgmap v21992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:41 smithi104 ceph-mon[36624]: pgmap v21992: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:43 smithi153 ceph-mon[36569]: pgmap v21993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:44.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:43 smithi104 ceph-mon[36624]: pgmap v21993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:44.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:43 smithi081 ceph-mon[31453]: pgmap v21993: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:45 smithi153 ceph-mon[36569]: pgmap v21994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:45 smithi104 ceph-mon[36624]: pgmap v21994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:46.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:45 smithi081 ceph-mon[31453]: pgmap v21994: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:47 smithi153 ceph-mon[36569]: pgmap v21995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:48.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:47 smithi104 ceph-mon[36624]: pgmap v21995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:48.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:47 smithi081 ceph-mon[31453]: pgmap v21995: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:49 smithi153 ceph-mon[36569]: pgmap v21996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:50.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:49 smithi104 ceph-mon[36624]: pgmap v21996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:49 smithi081 ceph-mon[31453]: pgmap v21996: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:51 smithi153 ceph-mon[36569]: pgmap v21997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:52.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:51 smithi104 ceph-mon[36624]: pgmap v21997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:52.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:51 smithi081 ceph-mon[31453]: pgmap v21997: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:54.172+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:45:54.172 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:54.172+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:45:54.173 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:45:54.172+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:45:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:53 smithi153 ceph-mon[36569]: pgmap v21998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:54.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:53 smithi104 ceph-mon[36624]: pgmap v21998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:53 smithi081 ceph-mon[31453]: pgmap v21998: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:55 smithi153 ceph-mon[36569]: pgmap v21999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:56.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:55 smithi104 ceph-mon[36624]: pgmap v21999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:55 smithi081 ceph-mon[31453]: pgmap v21999: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:57 smithi153 ceph-mon[36569]: pgmap v22000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:58.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:57 smithi104 ceph-mon[36624]: pgmap v22000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:45:58.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:57 smithi081 ceph-mon[31453]: pgmap v22000: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:45:59 smithi153 ceph-mon[36569]: pgmap v22001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:00.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:45:59 smithi104 ceph-mon[36624]: pgmap v22001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:00.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:45:59 smithi081 ceph-mon[31453]: pgmap v22001: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:01 smithi153 ceph-mon[36569]: pgmap v22002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:02.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:01 smithi104 ceph-mon[36624]: pgmap v22002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:01 smithi081 ceph-mon[31453]: pgmap v22002: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:03 smithi153 ceph-mon[36569]: pgmap v22003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:04.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:03 smithi104 ceph-mon[36624]: pgmap v22003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:04.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:03 smithi081 ceph-mon[31453]: pgmap v22003: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:05 smithi153 ceph-mon[36569]: pgmap v22004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:06.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:05 smithi104 ceph-mon[36624]: pgmap v22004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:06.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:05 smithi081 ceph-mon[31453]: pgmap v22004: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:07 smithi153 ceph-mon[36569]: pgmap v22005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:08.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:07 smithi104 ceph-mon[36624]: pgmap v22005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:08.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:07 smithi081 ceph-mon[31453]: pgmap v22005: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:09.187 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:09.188+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:46:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:09.188+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:46:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:09.188+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:46:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:09 smithi153 ceph-mon[36569]: pgmap v22006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:09 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:46:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:09 smithi104 ceph-mon[36624]: pgmap v22006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:09 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:46:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:09 smithi081 ceph-mon[31453]: pgmap v22006: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:09 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:46:12.118 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:11 smithi104 ceph-mon[36624]: pgmap v22007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:12.205+0000 7f1703e00640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 msgr2=0x7f16e80b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000638) for more than 900000000 us, fault. 2024-04-13T08:46:12.204 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:12.205+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c530 tx=0x7f16f40a6000 comp rx=0 tx=0).stop 2024-04-13T08:46:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:12.205+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 msgr2=0x7f16e80b3c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-13T08:46:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:12.205+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80a4c30 0x7f16e80b3c80 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-13T08:46:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:12.205+0000 7f1701600640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 0x7f16e80a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-13T08:46:12.205 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:12.206+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 0x7f16e80a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-13T08:46:12.206 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:12.207+0000 7f1703e00640 1 --2- 172.21.15.81:0/759453957 >> [v2:172.21.15.81:6800/3972789130,v1:172.21.15.81:6801/3972789130] conn(0x7f16e80b1520 0x7f16e80a5010 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f16f409c9f0 tx=0x7f16f4022000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-13T08:46:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:11 smithi153 ceph-mon[36569]: pgmap v22007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:12.316 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:11 smithi081 ceph-mon[31453]: pgmap v22007: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:13 smithi153 ceph-mon[36569]: pgmap v22008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:13 smithi104 ceph-mon[36624]: pgmap v22008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:13 smithi081 ceph-mon[31453]: pgmap v22008: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:15 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:15.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:15.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:15 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:15.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:15.396 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:15 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:16 smithi153 ceph-mon[36569]: pgmap v22009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:16.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:16 smithi104 ceph-mon[36624]: pgmap v22009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:16 smithi081 ceph-mon[31453]: pgmap v22009: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:46:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:46:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:17 smithi153 ceph-mon[36569]: pgmap v22010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:17 smithi153 ceph-mon[36569]: pgmap v22011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:18.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:18.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:46:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:46:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:17 smithi104 ceph-mon[36624]: pgmap v22010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:17 smithi104 ceph-mon[36624]: pgmap v22011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:18.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:46:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:46:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:17 smithi081 ceph-mon[31453]: pgmap v22010: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:18.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:17 smithi081 ceph-mon[31453]: pgmap v22011: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:18.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:46:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:19.555+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:46:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:19.555+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:46:19.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:19.556+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 318 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:46:19.555 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:19.556+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 318 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:46:20.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:19 smithi104 ceph-mon[36624]: pgmap v22012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:20.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:19 smithi081 ceph-mon[31453]: pgmap v22012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:19 smithi153 ceph-mon[36569]: pgmap v22012: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:22.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:21 smithi104 ceph-mon[36624]: pgmap v22013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:22.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:21 smithi081 ceph-mon[31453]: pgmap v22013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:22.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:21 smithi153 ceph-mon[36569]: pgmap v22013: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:24.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:23 smithi104 ceph-mon[36624]: pgmap v22014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:24.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:23 smithi081 ceph-mon[31453]: pgmap v22014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:24.202+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:46:24.202 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:24.202+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:46:24.203 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:24.202+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:46:24.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:23 smithi153 ceph-mon[36569]: pgmap v22014: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:26.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:25 smithi081 ceph-mon[31453]: pgmap v22015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:25 smithi153 ceph-mon[36569]: pgmap v22015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:25 smithi104 ceph-mon[36624]: pgmap v22015: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:28.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:27 smithi081 ceph-mon[31453]: pgmap v22016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:28.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:27 smithi153 ceph-mon[36569]: pgmap v22016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:28.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:27 smithi104 ceph-mon[36624]: pgmap v22016: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:30.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:29 smithi081 ceph-mon[31453]: pgmap v22017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:30.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:29 smithi153 ceph-mon[36569]: pgmap v22017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:30.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:29 smithi104 ceph-mon[36624]: pgmap v22017: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:32.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:31 smithi081 ceph-mon[31453]: pgmap v22018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:32.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:31 smithi153 ceph-mon[36569]: pgmap v22018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:32.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:31 smithi104 ceph-mon[36624]: pgmap v22018: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:34.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:33 smithi081 ceph-mon[31453]: pgmap v22019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:34.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:33 smithi153 ceph-mon[36569]: pgmap v22019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:34.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:33 smithi104 ceph-mon[36624]: pgmap v22019: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:36.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:35 smithi081 ceph-mon[31453]: pgmap v22020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:36.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:35 smithi153 ceph-mon[36569]: pgmap v22020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:36.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:35 smithi104 ceph-mon[36624]: pgmap v22020: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:38.164 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:37 smithi081 ceph-mon[31453]: pgmap v22021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:38.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:37 smithi153 ceph-mon[36569]: pgmap v22021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:38.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:37 smithi104 ceph-mon[36624]: pgmap v22021: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:39.217+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:46:39.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:39.217+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:46:39.217 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:39.217+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:46:40.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:39 smithi081 ceph-mon[31453]: pgmap v22022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:40.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:39 smithi153 ceph-mon[36569]: pgmap v22022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:40.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:39 smithi104 ceph-mon[36624]: pgmap v22022: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:42.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:41 smithi081 ceph-mon[31453]: pgmap v22023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:42.305 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:41 smithi153 ceph-mon[36569]: pgmap v22023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:42.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:41 smithi104 ceph-mon[36624]: pgmap v22023: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:44.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:43 smithi081 ceph-mon[31453]: pgmap v22024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:44.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:43 smithi153 ceph-mon[36569]: pgmap v22024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:44.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:43 smithi104 ceph-mon[36624]: pgmap v22024: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:46.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:45 smithi081 ceph-mon[31453]: pgmap v22025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:46.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:45 smithi153 ceph-mon[36569]: pgmap v22025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:46.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:45 smithi104 ceph-mon[36624]: pgmap v22025: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:48.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:47 smithi081 ceph-mon[31453]: pgmap v22026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:48.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:47 smithi153 ceph-mon[36569]: pgmap v22026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:48.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:47 smithi104 ceph-mon[36624]: pgmap v22026: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:50.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:49 smithi081 ceph-mon[31453]: pgmap v22027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:50.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:49 smithi153 ceph-mon[36569]: pgmap v22027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:50.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:49 smithi104 ceph-mon[36624]: pgmap v22027: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:52.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:51 smithi081 ceph-mon[31453]: pgmap v22028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:52.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:51 smithi153 ceph-mon[36569]: pgmap v22028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:52.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:51 smithi104 ceph-mon[36624]: pgmap v22028: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:54.228 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:54.229+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:46:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:54.229+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:46:54.229 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:46:54.229+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:46:54.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:53 smithi153 ceph-mon[36569]: pgmap v22029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:54.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:53 smithi104 ceph-mon[36624]: pgmap v22029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:54.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:53 smithi081 ceph-mon[31453]: pgmap v22029: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:56.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:55 smithi153 ceph-mon[36569]: pgmap v22030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:56.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:55 smithi104 ceph-mon[36624]: pgmap v22030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:56.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:55 smithi081 ceph-mon[31453]: pgmap v22030: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:58.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:57 smithi153 ceph-mon[36569]: pgmap v22031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:58.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:57 smithi104 ceph-mon[36624]: pgmap v22031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:46:58.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:57 smithi081 ceph-mon[31453]: pgmap v22031: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:00.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:46:59 smithi153 ceph-mon[36569]: pgmap v22032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:00.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:46:59 smithi104 ceph-mon[36624]: pgmap v22032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:00.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:46:59 smithi081 ceph-mon[31453]: pgmap v22032: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:02.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:01 smithi153 ceph-mon[36569]: pgmap v22033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:02.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:01 smithi104 ceph-mon[36624]: pgmap v22033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:02.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:01 smithi081 ceph-mon[31453]: pgmap v22033: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:04.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:03 smithi153 ceph-mon[36569]: pgmap v22034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:04.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:03 smithi104 ceph-mon[36624]: pgmap v22034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:04.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:03 smithi081 ceph-mon[31453]: pgmap v22034: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:06.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:05 smithi153 ceph-mon[36569]: pgmap v22035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:06.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:05 smithi104 ceph-mon[36624]: pgmap v22035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:06.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:05 smithi081 ceph-mon[31453]: pgmap v22035: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:08.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:07 smithi153 ceph-mon[36569]: pgmap v22036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:08.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:07 smithi104 ceph-mon[36624]: pgmap v22036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:08.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:07 smithi081 ceph-mon[31453]: pgmap v22036: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:09.241 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:09.241+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:47:09.241 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:09.241+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:47:09.241 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:09.241+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:47:10.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:09 smithi153 ceph-mon[36569]: pgmap v22037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:10.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:09 smithi104 ceph-mon[36624]: pgmap v22037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:10.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:09 smithi081 ceph-mon[31453]: pgmap v22037: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:12.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:11 smithi153 ceph-mon[36569]: pgmap v22038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:12.365 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:11 smithi104 ceph-mon[36624]: pgmap v22038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:12.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:11 smithi081 ceph-mon[31453]: pgmap v22038: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:14.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:13 smithi153 ceph-mon[36569]: pgmap v22039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:14.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:13 smithi104 ceph-mon[36624]: pgmap v22039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:14.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:13 smithi081 ceph-mon[31453]: pgmap v22039: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:16.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:15 smithi153 ceph-mon[36569]: pgmap v22040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:16.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:15 smithi104 ceph-mon[36624]: pgmap v22040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:16.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:15 smithi081 ceph-mon[31453]: pgmap v22040: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:17 smithi153 ceph-mon[36569]: pgmap v22041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:18.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:17 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:47:18.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:17 smithi104 ceph-mon[36624]: pgmap v22041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:18.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:17 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:47:18.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:17 smithi081 ceph-mon[31453]: pgmap v22041: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:18.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:17 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:47:20.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:19 smithi153 ceph-mon[36569]: pgmap v22042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:20.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:19 smithi104 ceph-mon[36624]: pgmap v22042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:20.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:19 smithi081 ceph-mon[31453]: pgmap v22042: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:22.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:21 smithi153 ceph-mon[36569]: pgmap v22043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:22.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:21 smithi104 ceph-mon[36624]: pgmap v22043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:22.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:21 smithi081 ceph-mon[31453]: pgmap v22043: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:47:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:47:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:22 smithi153 ceph-mon[36569]: pgmap v22044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:22 smithi153 ceph-mon[36569]: pgmap v22045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:23.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:22 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:47:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:47:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:22 smithi104 ceph-mon[36624]: pgmap v22044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:22 smithi104 ceph-mon[36624]: pgmap v22045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:22 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:47:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:47:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:22 smithi081 ceph-mon[31453]: pgmap v22044: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:22 smithi081 ceph-mon[31453]: pgmap v22045: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:22 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:47:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:24.256+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:47:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:24.256+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:47:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:24.256+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:47:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:25 smithi104 ceph-mon[36624]: pgmap v22046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:25 smithi081 ceph-mon[31453]: pgmap v22046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:25 smithi153 ceph-mon[36569]: pgmap v22046: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:27.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:27 smithi081 ceph-mon[31453]: pgmap v22047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:27.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:27 smithi153 ceph-mon[36569]: pgmap v22047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:27.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:27 smithi104 ceph-mon[36624]: pgmap v22047: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:29.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:29 smithi081 ceph-mon[31453]: pgmap v22048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:29.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:29 smithi153 ceph-mon[36569]: pgmap v22048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:29.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:29 smithi104 ceph-mon[36624]: pgmap v22048: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:31.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:31 smithi081 ceph-mon[31453]: pgmap v22049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:31.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:31 smithi153 ceph-mon[36569]: pgmap v22049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:31.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:31 smithi104 ceph-mon[36624]: pgmap v22049: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:33.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:33 smithi081 ceph-mon[31453]: pgmap v22050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:33.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:33 smithi153 ceph-mon[36569]: pgmap v22050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:33.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:33 smithi104 ceph-mon[36624]: pgmap v22050: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:35.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:35 smithi081 ceph-mon[31453]: pgmap v22051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:35.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:35 smithi153 ceph-mon[36569]: pgmap v22051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:35.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:35 smithi104 ceph-mon[36624]: pgmap v22051: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:37 smithi081 ceph-mon[31453]: pgmap v22052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:37.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:37 smithi153 ceph-mon[36569]: pgmap v22052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:37.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:37 smithi104 ceph-mon[36624]: pgmap v22052: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:39.262+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:47:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:39.262+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:47:39.263 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:39.262+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:47:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:39 smithi081 ceph-mon[31453]: pgmap v22053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:39.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:39 smithi153 ceph-mon[36569]: pgmap v22053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:39.591 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:39 smithi104 ceph-mon[36624]: pgmap v22053: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:41.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:41 smithi081 ceph-mon[31453]: pgmap v22054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:41.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:41 smithi153 ceph-mon[36569]: pgmap v22054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:41.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:41 smithi104 ceph-mon[36624]: pgmap v22054: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:43 smithi081 ceph-mon[31453]: pgmap v22055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:43.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:43 smithi153 ceph-mon[36569]: pgmap v22055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:43.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:43 smithi104 ceph-mon[36624]: pgmap v22055: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:45.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:45 smithi081 ceph-mon[31453]: pgmap v22056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:45.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:45 smithi153 ceph-mon[36569]: pgmap v22056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:45.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:45 smithi104 ceph-mon[36624]: pgmap v22056: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:47.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:47 smithi081 ceph-mon[31453]: pgmap v22057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:47.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:47 smithi153 ceph-mon[36569]: pgmap v22057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:47.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:47 smithi104 ceph-mon[36624]: pgmap v22057: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:48.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:48 smithi081 ceph-mon[31453]: pgmap v22058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:48 smithi153 ceph-mon[36569]: pgmap v22058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:48.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:48 smithi104 ceph-mon[36624]: pgmap v22058: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:51.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:51 smithi081 ceph-mon[31453]: pgmap v22059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:51.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:51 smithi153 ceph-mon[36569]: pgmap v22059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:51.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:51 smithi104 ceph-mon[36624]: pgmap v22059: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:53 smithi081 ceph-mon[31453]: pgmap v22060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:53.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:53 smithi153 ceph-mon[36569]: pgmap v22060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:53.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:53 smithi104 ceph-mon[36624]: pgmap v22060: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:54.278+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:47:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:54.278+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:47:54.278 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:47:54.278+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:47:55.385 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:55 smithi104 ceph-mon[36624]: pgmap v22061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:55.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:55 smithi081 ceph-mon[31453]: pgmap v22061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:55.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:55 smithi153 ceph-mon[36569]: pgmap v22061: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:57.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:57 smithi081 ceph-mon[31453]: pgmap v22062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:57.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:57 smithi153 ceph-mon[36569]: pgmap v22062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:57.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:57 smithi104 ceph-mon[36624]: pgmap v22062: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:59.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:47:59 smithi081 ceph-mon[31453]: pgmap v22063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:59.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:47:59 smithi153 ceph-mon[36569]: pgmap v22063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:47:59.605 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:47:59 smithi104 ceph-mon[36624]: pgmap v22063: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:01.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:01 smithi081 ceph-mon[31453]: pgmap v22064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:01.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:01 smithi153 ceph-mon[36569]: pgmap v22064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:01.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:01 smithi104 ceph-mon[36624]: pgmap v22064: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:03 smithi081 ceph-mon[31453]: pgmap v22065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:03.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:03 smithi153 ceph-mon[36569]: pgmap v22065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:03.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:03 smithi104 ceph-mon[36624]: pgmap v22065: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:05.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:05 smithi081 ceph-mon[31453]: pgmap v22066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:05.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:05 smithi153 ceph-mon[36569]: pgmap v22066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:05.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:05 smithi104 ceph-mon[36624]: pgmap v22066: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:06.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:06 smithi081 ceph-mon[31453]: pgmap v22067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:06.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:06 smithi153 ceph-mon[36569]: pgmap v22067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:06.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:06 smithi104 ceph-mon[36624]: pgmap v22067: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:09.283 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:09.283+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:48:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:09.284+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:48:09.284 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:09.284+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:48:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:09 smithi081 ceph-mon[31453]: pgmap v22068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:09.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:09 smithi153 ceph-mon[36569]: pgmap v22068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:09.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:09 smithi104 ceph-mon[36624]: pgmap v22068: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:11 smithi081 ceph-mon[31453]: pgmap v22069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:11.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:11 smithi153 ceph-mon[36569]: pgmap v22069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:11.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:11 smithi104 ceph-mon[36624]: pgmap v22069: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:13.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:13 smithi081 ceph-mon[31453]: pgmap v22070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:13.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:13 smithi153 ceph-mon[36569]: pgmap v22070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:13.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:13 smithi104 ceph-mon[36624]: pgmap v22070: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:15.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:15 smithi081 ceph-mon[31453]: pgmap v22071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:15.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:15 smithi153 ceph-mon[36569]: pgmap v22071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:15.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:15 smithi104 ceph-mon[36624]: pgmap v22071: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:17.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:17 smithi081 ceph-mon[31453]: pgmap v22072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:17.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:17 smithi153 ceph-mon[36569]: pgmap v22072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:17.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:17 smithi104 ceph-mon[36624]: pgmap v22072: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:19 smithi081 ceph-mon[31453]: pgmap v22073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:19.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:19 smithi153 ceph-mon[36569]: pgmap v22073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:19.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:19 smithi104 ceph-mon[36624]: pgmap v22073: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:21.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:21 smithi081 ceph-mon[31453]: pgmap v22074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:21.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:21 smithi153 ceph-mon[36569]: pgmap v22074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:21.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:21 smithi104 ceph-mon[36624]: pgmap v22074: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:23.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:23 smithi081 ceph-mon[31453]: pgmap v22075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:23 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:48:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:23 smithi153 ceph-mon[36569]: pgmap v22075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:23.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:23 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:48:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:23 smithi104 ceph-mon[36624]: pgmap v22075: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:23.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:23 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:48:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:24.292+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:48:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:24.293+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:48:24.292 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:24.293+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:48:24.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:48:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:48:24.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:48:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:48:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:48:24.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:48:24.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:48:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:48:24.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:48:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:25 smithi081 ceph-mon[31453]: pgmap v22076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:25.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:25 smithi081 ceph-mon[31453]: pgmap v22077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:25.483 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:25 smithi104 ceph-mon[36624]: pgmap v22076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:25.483 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:25 smithi104 ceph-mon[36624]: pgmap v22077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:25 smithi153 ceph-mon[36569]: pgmap v22076: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:25.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:25 smithi153 ceph-mon[36569]: pgmap v22077: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:26 smithi153 ceph-mon[36569]: pgmap v22078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:26.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:26 smithi104 ceph-mon[36624]: pgmap v22078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:26.655 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:26 smithi081 ceph-mon[31453]: pgmap v22078: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:29.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:28 smithi153 ceph-mon[36569]: pgmap v22079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:29.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:28 smithi104 ceph-mon[36624]: pgmap v22079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:28 smithi081 ceph-mon[31453]: pgmap v22079: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:31.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:30 smithi153 ceph-mon[36569]: pgmap v22080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:31.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:30 smithi104 ceph-mon[36624]: pgmap v22080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:30 smithi081 ceph-mon[31453]: pgmap v22080: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:33.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:32 smithi153 ceph-mon[36569]: pgmap v22081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:33.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:32 smithi104 ceph-mon[36624]: pgmap v22081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:33.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:32 smithi081 ceph-mon[31453]: pgmap v22081: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:35.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:34 smithi153 ceph-mon[36569]: pgmap v22082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:35.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:34 smithi104 ceph-mon[36624]: pgmap v22082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:35.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:34 smithi081 ceph-mon[31453]: pgmap v22082: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:37.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:36 smithi153 ceph-mon[36569]: pgmap v22083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:37.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:36 smithi104 ceph-mon[36624]: pgmap v22083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:37.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:36 smithi081 ceph-mon[31453]: pgmap v22083: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:39.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:38 smithi153 ceph-mon[36569]: pgmap v22084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:39.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:38 smithi104 ceph-mon[36624]: pgmap v22084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:39.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:38 smithi081 ceph-mon[31453]: pgmap v22084: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:39.308+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:48:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:39.308+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:48:39.308 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:39.308+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:48:41.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:40 smithi153 ceph-mon[36569]: pgmap v22085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:41.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:40 smithi104 ceph-mon[36624]: pgmap v22085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:41.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:40 smithi081 ceph-mon[31453]: pgmap v22085: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:43.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:42 smithi153 ceph-mon[36569]: pgmap v22086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:43.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:42 smithi104 ceph-mon[36624]: pgmap v22086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:43.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:42 smithi081 ceph-mon[31453]: pgmap v22086: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:45.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:44 smithi153 ceph-mon[36569]: pgmap v22087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:45.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:44 smithi104 ceph-mon[36624]: pgmap v22087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:45.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:44 smithi081 ceph-mon[31453]: pgmap v22087: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:47.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:46 smithi153 ceph-mon[36569]: pgmap v22088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:47.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:46 smithi104 ceph-mon[36624]: pgmap v22088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:47.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:46 smithi081 ceph-mon[31453]: pgmap v22088: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:49.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:48 smithi153 ceph-mon[36569]: pgmap v22089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:49.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:48 smithi104 ceph-mon[36624]: pgmap v22089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:49.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:48 smithi081 ceph-mon[31453]: pgmap v22089: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:51.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:50 smithi153 ceph-mon[36569]: pgmap v22090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:51.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:50 smithi104 ceph-mon[36624]: pgmap v22090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:51.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:50 smithi081 ceph-mon[31453]: pgmap v22090: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:53.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:52 smithi153 ceph-mon[36569]: pgmap v22091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:53.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:52 smithi104 ceph-mon[36624]: pgmap v22091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:53.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:52 smithi081 ceph-mon[31453]: pgmap v22091: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:54.320+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:48:54.319 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:54.320+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:48:54.320 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:48:54.320+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:48:55.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:54 smithi153 ceph-mon[36569]: pgmap v22092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:55.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:54 smithi104 ceph-mon[36624]: pgmap v22092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:55.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:54 smithi081 ceph-mon[31453]: pgmap v22092: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:57.056 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:56 smithi153 ceph-mon[36569]: pgmap v22093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:57.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:56 smithi104 ceph-mon[36624]: pgmap v22093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:57.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:56 smithi081 ceph-mon[31453]: pgmap v22093: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:59.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:48:58 smithi104 ceph-mon[36624]: pgmap v22094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:59.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:48:58 smithi081 ceph-mon[31453]: pgmap v22094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:48:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:48:58 smithi153 ceph-mon[36569]: pgmap v22094: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:01.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:00 smithi104 ceph-mon[36624]: pgmap v22095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:01.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:00 smithi081 ceph-mon[31453]: pgmap v22095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:00 smithi153 ceph-mon[36569]: pgmap v22095: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:03.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:02 smithi104 ceph-mon[36624]: pgmap v22096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:03.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:02 smithi081 ceph-mon[31453]: pgmap v22096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:02 smithi153 ceph-mon[36569]: pgmap v22096: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:05.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:04 smithi104 ceph-mon[36624]: pgmap v22097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:05.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:04 smithi081 ceph-mon[31453]: pgmap v22097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:04 smithi153 ceph-mon[36569]: pgmap v22097: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:07.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:06 smithi104 ceph-mon[36624]: pgmap v22098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:07.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:06 smithi081 ceph-mon[31453]: pgmap v22098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:06 smithi153 ceph-mon[36569]: pgmap v22098: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:09.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:08 smithi081 ceph-mon[31453]: pgmap v22099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:08 smithi153 ceph-mon[36569]: pgmap v22099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:08 smithi104 ceph-mon[36624]: pgmap v22099: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:09.333+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:49:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:09.333+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:49:09.333 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:09.333+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:49:11.108 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:10 smithi153 ceph-mon[36569]: pgmap v22100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:11.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:10 smithi081 ceph-mon[31453]: pgmap v22100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:11.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:10 smithi104 ceph-mon[36624]: pgmap v22100: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:13.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:12 smithi081 ceph-mon[31453]: pgmap v22101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:12 smithi153 ceph-mon[36569]: pgmap v22101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:12 smithi104 ceph-mon[36624]: pgmap v22101: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:15.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:14 smithi081 ceph-mon[31453]: pgmap v22102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:14 smithi153 ceph-mon[36569]: pgmap v22102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:15.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:14 smithi104 ceph-mon[36624]: pgmap v22102: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:17.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:16 smithi081 ceph-mon[31453]: pgmap v22103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:17.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:16 smithi153 ceph-mon[36569]: pgmap v22103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:17.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:16 smithi104 ceph-mon[36624]: pgmap v22103: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:19.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:18 smithi081 ceph-mon[31453]: pgmap v22104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:18 smithi153 ceph-mon[36569]: pgmap v22104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:19.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:18 smithi104 ceph-mon[36624]: pgmap v22104: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:19.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:19.585+0000 7f16fb400640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:49:19.584 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:19.585+0000 7f16f0c00640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:49:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:19.585+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 319 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f16d40039a0 con 0x7f1704076980 2024-04-13T08:49:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:19.586+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 319 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f16d00064c0 con 0x563c9e245d20 2024-04-13T08:49:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:19.586+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:49:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:19.586+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:49:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:19.586+0000 7f16f2000640 1 -- 172.21.15.81:0/861362842 <== mon.1 v2:172.21.15.153:3300/0 320 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f16d00062f0 con 0x563c9e245d20 2024-04-13T08:49:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:19.586+0000 7f1701600640 1 -- 172.21.15.81:0/759453957 <== mon.2 v2:172.21.15.104:3300/0 320 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f16d4003770 con 0x7f1704076980 2024-04-13T08:49:21.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:20 smithi081 ceph-mon[31453]: pgmap v22105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:20 smithi153 ceph-mon[36569]: pgmap v22105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:21.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:20 smithi104 ceph-mon[36624]: pgmap v22105: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:23.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:22 smithi081 ceph-mon[31453]: pgmap v22106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:22 smithi153 ceph-mon[36569]: pgmap v22106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:23.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:22 smithi104 ceph-mon[36624]: pgmap v22106: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:24.349+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:49:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:24.349+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:49:24.349 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:24.349+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: pgmap v22107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:24 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: pgmap v22107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:49:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:24 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: pgmap v22107: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:24 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:49:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:49:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:49:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:49:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:49:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:49:26.157 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:25 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:49:26.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:49:26.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:49:26.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:49:26.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:49:26.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.233 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:49:26.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:49:26.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:25 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:49:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi153 to 174.1M 2024-04-13T08:49:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi104 to 174.1M 2024-04-13T08:49:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi153 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-04-13T08:49:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi104 to 182632038: error parsing value: Value '182632038' is below minimum 939524096 2024-04-13T08:49:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-13T08:49:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:49:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:49:26.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:25 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:49:27.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:26 smithi081 ceph-mon[31453]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:49:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:26 smithi081 ceph-mon[31453]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:49:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:26 smithi081 ceph-mon[31453]: pgmap v22108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:27.156 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:26 smithi081 ceph-mon[31453]: pgmap v22109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:26 smithi153 ceph-mon[36569]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:49:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:26 smithi153 ceph-mon[36569]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:49:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:26 smithi153 ceph-mon[36569]: pgmap v22108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:27.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:26 smithi153 ceph-mon[36569]: pgmap v22109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:26 smithi104 ceph-mon[36624]: Adjusting osd_memory_target on smithi081 to 176.4M 2024-04-13T08:49:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:26 smithi104 ceph-mon[36624]: Unable to set osd_memory_target on smithi081 to 185061273: error parsing value: Value '185061273' is below minimum 939524096 2024-04-13T08:49:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:26 smithi104 ceph-mon[36624]: pgmap v22108: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:26 smithi104 ceph-mon[36624]: pgmap v22109: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:29.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:28 smithi081 ceph-mon[31453]: pgmap v22110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:29.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:28 smithi153 ceph-mon[36569]: pgmap v22110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:29.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:28 smithi104 ceph-mon[36624]: pgmap v22110: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:31.155 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:30 smithi081 ceph-mon[31453]: pgmap v22111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:31.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:30 smithi153 ceph-mon[36569]: pgmap v22111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:31.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:30 smithi104 ceph-mon[36624]: pgmap v22111: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:33.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:32 smithi153 ceph-mon[36569]: pgmap v22112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:33.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:32 smithi104 ceph-mon[36624]: pgmap v22112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:33.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:32 smithi081 ceph-mon[31453]: pgmap v22112: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:35.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:34 smithi153 ceph-mon[36569]: pgmap v22113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:35.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:34 smithi104 ceph-mon[36624]: pgmap v22113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:35.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:34 smithi081 ceph-mon[31453]: pgmap v22113: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:37.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:36 smithi153 ceph-mon[36569]: pgmap v22114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:37.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:36 smithi104 ceph-mon[36624]: pgmap v22114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:37.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:36 smithi081 ceph-mon[31453]: pgmap v22114: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:39.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:38 smithi153 ceph-mon[36569]: pgmap v22115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:39.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:38 smithi104 ceph-mon[36624]: pgmap v22115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:39.351+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:49:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:39.351+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:49:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:39.351+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:49:39.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:38 smithi081 ceph-mon[31453]: pgmap v22115: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:41.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:40 smithi153 ceph-mon[36569]: pgmap v22116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:41.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:40 smithi104 ceph-mon[36624]: pgmap v22116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:41.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:40 smithi081 ceph-mon[31453]: pgmap v22116: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:43.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:42 smithi153 ceph-mon[36569]: pgmap v22117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:43.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:42 smithi104 ceph-mon[36624]: pgmap v22117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:43.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:42 smithi081 ceph-mon[31453]: pgmap v22117: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:45.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:44 smithi081 ceph-mon[31453]: pgmap v22118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:45.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:44 smithi153 ceph-mon[36569]: pgmap v22118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:45.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:44 smithi104 ceph-mon[36624]: pgmap v22118: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:47.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:46 smithi153 ceph-mon[36569]: pgmap v22119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:47.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:46 smithi104 ceph-mon[36624]: pgmap v22119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:47.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:46 smithi081 ceph-mon[31453]: pgmap v22119: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:49.209 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:48 smithi081 ceph-mon[31453]: pgmap v22120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:49.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:48 smithi153 ceph-mon[36569]: pgmap v22120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:49.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:48 smithi104 ceph-mon[36624]: pgmap v22120: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:51.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:50 smithi153 ceph-mon[36569]: pgmap v22121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:51.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:50 smithi104 ceph-mon[36624]: pgmap v22121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:51.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:50 smithi081 ceph-mon[31453]: pgmap v22121: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:53.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:52 smithi153 ceph-mon[36569]: pgmap v22122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:53.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:52 smithi104 ceph-mon[36624]: pgmap v22122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:53.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:52 smithi081 ceph-mon[31453]: pgmap v22122: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:54.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:54.366+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:49:54.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:54.366+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:49:54.366 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:49:54.366+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:49:55.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:54 smithi153 ceph-mon[36569]: pgmap v22123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:55.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:54 smithi104 ceph-mon[36624]: pgmap v22123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:55.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:54 smithi081 ceph-mon[31453]: pgmap v22123: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:57.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:56 smithi153 ceph-mon[36569]: pgmap v22124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:57.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:56 smithi104 ceph-mon[36624]: pgmap v22124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:57.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:56 smithi081 ceph-mon[31453]: pgmap v22124: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:59.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:49:58 smithi153 ceph-mon[36569]: pgmap v22125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:59.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:49:58 smithi104 ceph-mon[36624]: pgmap v22125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:49:59.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:49:58 smithi081 ceph-mon[31453]: pgmap v22125: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:00 smithi153 ceph-mon[36569]: pgmap v22126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:01.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:00 smithi153 ceph-mon[36569]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:50:01.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:00 smithi104 ceph-mon[36624]: pgmap v22126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:01.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:00 smithi104 ceph-mon[36624]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:50:01.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:00 smithi081 ceph-mon[31453]: pgmap v22126: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:01.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:00 smithi081 ceph-mon[31453]: overall HEALTH_WARN 12 failed cephadm daemon(s) 2024-04-13T08:50:03.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:02 smithi153 ceph-mon[36569]: pgmap v22127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:03.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:02 smithi104 ceph-mon[36624]: pgmap v22127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:03.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:02 smithi081 ceph-mon[31453]: pgmap v22127: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:05.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:04 smithi153 ceph-mon[36569]: pgmap v22128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:05.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:04 smithi104 ceph-mon[36624]: pgmap v22128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:05.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:04 smithi081 ceph-mon[31453]: pgmap v22128: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:07.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:06 smithi153 ceph-mon[36569]: pgmap v22129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:07.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:06 smithi104 ceph-mon[36624]: pgmap v22129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:07.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:06 smithi081 ceph-mon[31453]: pgmap v22129: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:09.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:09 smithi153 ceph-mon[36569]: pgmap v22130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:09.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:09 smithi104 ceph-mon[36624]: pgmap v22130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:09.367+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:50:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:09.367+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:50:09.370 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:09.367+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:50:09.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:09 smithi081 ceph-mon[31453]: pgmap v22130: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:11.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:11 smithi104 ceph-mon[36624]: pgmap v22131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:11.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:11 smithi153 ceph-mon[36569]: pgmap v22131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:11.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:11 smithi081 ceph-mon[31453]: pgmap v22131: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:13.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:13 smithi153 ceph-mon[36569]: pgmap v22132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:13.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:13 smithi104 ceph-mon[36624]: pgmap v22132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:13.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:13 smithi081 ceph-mon[31453]: pgmap v22132: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:15.303 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:15 smithi081 ceph-mon[31453]: pgmap v22133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:15.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:15 smithi153 ceph-mon[36569]: pgmap v22133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:15.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:15 smithi104 ceph-mon[36624]: pgmap v22133: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:17.307 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:17 smithi153 ceph-mon[36569]: pgmap v22134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:17.333 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:17 smithi104 ceph-mon[36624]: pgmap v22134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:17.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:17 smithi081 ceph-mon[31453]: pgmap v22134: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:19.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:19 smithi153 ceph-mon[36569]: pgmap v22135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:19.331 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:19 smithi104 ceph-mon[36624]: pgmap v22135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:19.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:19 smithi081 ceph-mon[31453]: pgmap v22135: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:21.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:21 smithi153 ceph-mon[36569]: pgmap v22136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:21.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:21 smithi104 ceph-mon[36624]: pgmap v22136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:21.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:21 smithi081 ceph-mon[31453]: pgmap v22136: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:23.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:23 smithi153 ceph-mon[36569]: pgmap v22137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:23.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:23 smithi104 ceph-mon[36624]: pgmap v22137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:23.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:23 smithi081 ceph-mon[31453]: pgmap v22137: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:24.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:24.368+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:50:24.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:24.368+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:50:24.368 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:24.368+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:50:25.306 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:25 smithi153 ceph-mon[36569]: pgmap v22138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:25.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:25 smithi104 ceph-mon[36624]: pgmap v22138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:25.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:25 smithi081 ceph-mon[31453]: pgmap v22138: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:26.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:26 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:50:26.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:26 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:50:26.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:26 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-13T08:50:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:27 smithi104 ceph-mon[36624]: pgmap v22139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:50:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:50:27.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:27 smithi104 ceph-mon[36624]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:50:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:27 smithi081 ceph-mon[31453]: pgmap v22139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:50:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:50:27.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:27 smithi081 ceph-mon[31453]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:50:27.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:27 smithi153 ceph-mon[36569]: pgmap v22139: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:27.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-13T08:50:27.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-13T08:50:27.558 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:27 smithi153 ceph-mon[36569]: from='mgr.14152 172.21.15.81:0/623024170' entity='mgr.x' 2024-04-13T08:50:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:28 smithi104 ceph-mon[36624]: pgmap v22140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:28.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:28 smithi104 ceph-mon[36624]: pgmap v22141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:28 smithi081 ceph-mon[31453]: pgmap v22140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:28.407 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:28 smithi081 ceph-mon[31453]: pgmap v22141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:28 smithi153 ceph-mon[36569]: pgmap v22140: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:28.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:28 smithi153 ceph-mon[36569]: pgmap v22141: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:30.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:30 smithi104 ceph-mon[36624]: pgmap v22142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:30.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:30 smithi081 ceph-mon[31453]: pgmap v22142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:30.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:30 smithi153 ceph-mon[36569]: pgmap v22142: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:32.332 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:32 smithi104 ceph-mon[36624]: pgmap v22143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:32.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:32 smithi081 ceph-mon[31453]: pgmap v22143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:32.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:32 smithi153 ceph-mon[36569]: pgmap v22143: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:34.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:34 smithi081 ceph-mon[31453]: pgmap v22144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:34.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:34 smithi153 ceph-mon[36569]: pgmap v22144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:34.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:34 smithi104 ceph-mon[36624]: pgmap v22144: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:36.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:36 smithi081 ceph-mon[31453]: pgmap v22145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:36.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:36 smithi153 ceph-mon[36569]: pgmap v22145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:36.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:36 smithi104 ceph-mon[36624]: pgmap v22145: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:38.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:38 smithi081 ceph-mon[31453]: pgmap v22146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:38.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:38 smithi153 ceph-mon[36569]: pgmap v22146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:38.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:38 smithi104 ceph-mon[36624]: pgmap v22146: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:39.383 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:39.383+0000 7f1708e00640 1 -- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 msgr2=0x563c9e2c4b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.104:6838/1944172329 2024-04-13T08:50:39.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:39.383+0000 7f1708e00640 1 --2- 172.21.15.81:0/861362842 >> [v2:172.21.15.104:6838/1944172329,v1:172.21.15.104:6839/1944172329] conn(0x563c9e2c2750 0x563c9e2c4b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-13T08:50:39.384 INFO:teuthology.task.background_exec.ubuntu@smithi081.front.sepia.ceph.com.smithi081.stderr:2024-04-13T08:50:39.383+0000 7f16f2000640 1 client.25321 ms_handle_refused on v2:172.21.15.104:6838/1944172329 2024-04-13T08:50:40.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:40 smithi081 ceph-mon[31453]: pgmap v22147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:40.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:40 smithi153 ceph-mon[36569]: pgmap v22147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:40.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:40 smithi104 ceph-mon[36624]: pgmap v22147: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:42.358 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:42 smithi153 ceph-mon[36569]: pgmap v22148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:42.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:42 smithi081 ceph-mon[31453]: pgmap v22148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:42.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:42 smithi104 ceph-mon[36624]: pgmap v22148: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:44.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:44 smithi081 ceph-mon[31453]: pgmap v22149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:44.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:44 smithi153 ceph-mon[36569]: pgmap v22149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:44.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:44 smithi104 ceph-mon[36624]: pgmap v22149: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:46.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:46 smithi081 ceph-mon[31453]: pgmap v22150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:46.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:46 smithi153 ceph-mon[36569]: pgmap v22150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:46.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:46 smithi104 ceph-mon[36624]: pgmap v22150: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:48.406 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:48 smithi081 ceph-mon[31453]: pgmap v22151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:48.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:48 smithi153 ceph-mon[36569]: pgmap v22151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:48.582 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:48 smithi104 ceph-mon[36624]: pgmap v22151: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:50.405 INFO:journalctl@ceph.mon.a.smithi081.stdout:Apr 13 08:50:50 smithi081 ceph-mon[31453]: pgmap v22152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:50.556 INFO:journalctl@ceph.mon.c.smithi153.stdout:Apr 13 08:50:50 smithi153 ceph-mon[36569]: pgmap v22152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:50.581 INFO:journalctl@ceph.mon.b.smithi104.stdout:Apr 13 08:50:50 smithi104 ceph-mon[36624]: pgmap v22152: 129 pgs: 129 active+clean; 515 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-13T08:50:51.402 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-04-13T08:50:51.402 DEBUG:teuthology.task.console_log:Killing console logger for smithi081 2024-04-13T08:50:51.403 DEBUG:teuthology.task.console_log:Killing console logger for smithi104 2024-04-13T08:50:51.403 DEBUG:teuthology.task.console_log:Killing console logger for smithi153 2024-04-13T08:50:51.403 DEBUG:teuthology.exit:Finished running handlers